News
Tesla Smart Summon patent highlights progress in 3D labeling for full self-driving features
A recently published Tesla patent application details the machine learning methods behind Smart Summon, specifically highlighting the progress being made with 3D labeling in training data.
The application, titled “Autonomous and User Controlled Vehicle Summon to a Target,” utilizes machine learning methods explicitly detailed in two other recent Tesla patent publications in its functionality. This series of three inventions altogether describes an automated way of generating training data which is then used by a machine learning model to accomplish an expansive list of self-driving capabilities in Summon.
“Traditionally, much of the effort to curate a training data set is done manually by reviewing potential training data and properly labeling the features associated with the data,” Tesla’s first application in the series states. “The effort required to create a training set with accurate labels can be significant and is often tedious… Therefore, there exists a need to improve the process for generating training data with accurate labeled features.”
-
- A method flow chart from Tesla’s autonomous 3D labeling patent. | Image: Tesla/USPTO
-
- A method flow chart from Tesla’s Smart Summon patent application. | Image: Tesla/USPTO
The application goes on to describe how labeled training data is made autonomously in their invention using sensors and the collection of what’s called a “time series,” i.e., a series of images captured over a period of time.
“Using data captured by sensors on a vehicle to capture the environment of the vehicle and vehicle operating parameters, a training data set is created,” it explains. “In some embodiments, a three-dimensional representation of a feature, such as a lane line, is created from the group of time series elements that corresponds to the ground truth… As one example, a series of images for a time period, such as 30 seconds, is used to determine the actual path of a vehicle lane line over the time period the vehicle travels…a single image of the group and the actual path taken can be used as training data to predict the path of the vehicle.”
Tesla CEO Elon Musk has previously mentioned that better labeling is one of the keys to speeding up the rollout of self-driving functionality and features like Reverse Summon. “We need to finish work on Autopilot core foundation code & 3D labeling, then functionality will happen quickly. Not long now,” Musk wrote on Twitter in March this year. With better labeling (more accurate training data) comes safer and more capable software due to improved predictions from the modeling.
When it comes to Tesla’s Smart Summon, prediction modeling is essential considering there isn’t a driver in the vehicle during its operation. The patent publication covering Summon embodies the first application’s time series functionality and a second application’s implementation of the time series’ training data in its methods, demonstrating one of the numerous potential uses for the machine learning invention. Hints about future developments using Smart Summon are also detailed in the application. Examples include:
- Syncing the Smart Summon with a calendar so the vehicle “automatically navigates to arrive at the location at the ending time, such as the end of a dinner party, a wedding, a restaurant reservation, etc.”
- Implementing a multi-part destination into the Summon instructions such as waypoints at an airport to pick up multiple passengers.
- Monitoring the heartbeat of a Summon user to ensure they are maintaining a connection with the vehicle while operating the feature.
- Customizing the vehicle’s arrival settings such as interior lighting, exterior lighting, hazard lights, welcome music, and climate control preferences.
One of the more unique bits about the Smart Summon patent application is the appearance of Elon Musk as an inventor. While the CEO is known to be intimately involved in nearly all aspects of vehicle design, software features, and business operations, his name is unexpectedly absent from most of the company’s inventions. However, this is apparently on purpose. “I generally try my best not to be on patents,” he revealed on Twitter in reply to a post about the Smart Summon application. Notably, inventorship is a legal definition based on the conception of an invention, i.e., not the person/people who suggested or directed its creation, but the person/people who devised the means to accomplish it.
Prior to the most recent patent publication, Musk contributed inventorship to the door and body styling of the Model X. He also contributed the same to both the design and function of Tesla’s vehicle charge inlets.
Tesla has launched a direct campaign targeting its customers in New Jersey, sending emails that warn of pending legislation that could effectively block true driverless technology in the state.
The email focuses on Senate Bill S.1677 and Assembly Bill A.3968, measures intended to create a three-year autonomous vehicle pilot program but laden with requirements that Tesla argues make unsupervised Robotaxis impossible.
Tesla is sending out this email to New Jersey Tesla owners, warning them that NJ could block autonomous vehicles, and to take action.
“Proposed legislation moving through Trenton right now would impose restrictions so severe that true driverless deployment would remain illegal.… pic.twitter.com/2bmY646AUL
— Sawyer Merritt (@SawyerMerritt) June 16, 2026
According to the email, the bills impose “restrictions so severe that true driverless deployment would remain illegal.” Specific hurdles include mandates for human safety drivers during operations, multimillion-dollar insurance minimums, reportedly $5 million, and thresholds like 100,000 miles of demonstrated safe autonomous driving before any driverless approval.
Tesla contends these are arbitrary barriers that ignore real-world performance data and favor entrenched competitors over innovative technologies like its Full Self-Driving (FSD) system.
The push comes as Tesla has started expanding Robotaxi operations in states like Texas, where unsupervised vehicles are already providing rides in several cities. New Jersey, by contrast, risks falling behind. The company highlights in the email communication that more than 94 percent of serious crashes result from human error, meaning impairment, distraction, or fatigue. These are all problems that Robotaxis eliminate entirely.
In 2025, New Jersey recorded 582 traffic deaths, underscoring the human cost of delayed adoption.
Tesla’s outreach stresses the transformative potential of robotaxis. For families, they could offer safer school runs without drowsy or distracted drivers. For seniors and people with disabilities, robotaxis promise independence and reliable mobility.
In areas with limited public transit, they could deliver affordable, on-demand transportation, reducing congestion, emissions, and overall transportation costs. Economically, the company warns that restrictive rules could cost New Jersey jobs, innovation investment, and billions in potential growth as autonomous ride-hailing scales elsewhere.
Supporters of the legislation, including Sen. Andrew Zwicker, describe the pilot as a cautious framework with strong safety oversight, including incident reporting, expert task forces, and restrictions in sensitive zones like school areas. They view it as balancing innovation with public protection.
Tesla and pro-AV advocates counter that the bill lacks technology neutrality, creates insurmountable entry barriers for commercial deployment, and prioritizes process over outcomes — effectively functioning as a de facto ban on services like Robotaxi.
This latest clash echoes Tesla’s past battles in New Jersey over direct vehicle sales. The email directs owners to Tesla’s advocacy platform, where they can send customized messages to legislators calling for amendments: outcome-based safety standards, open competition, and clear pathways for fully driverless commercial operations.
As hearings approach, Tesla’s campaign frames the issue as a choice between protecting the status quo and embracing life-saving progress. With robotaxi technology already proving itself in permissive states, New Jersey owners are being asked to ensure their state doesn’t lock out the future of transportation.
Turn-by-turn navigation is not new technology.
For over two decades, drivers have relied on Garmin, TomTom, and later smartphone apps like Google Maps and Waze to receive precise, reliable directions. These systems have guided millions safely through unfamiliar cities, highways, and backroads with remarkable effectiveness. They handle real-time traffic, construction detours, and complex intersections with minimal fuss.
Yet Tesla, the company that promised revolutionary Full Self-Driving (FSD), continues to struggle with this foundational capability. As FSD (Supervised) v14.3.4 has started rolling out to cars this week, navigation remains its glaring Achilles’ heel, undermining the entire autonomous vision.
Tesla Summon got insanely good in FSD v14.3.2 — Navigation? Not so much
Tesla’s FSD excels in many driving behaviors—smooth acceleration, confident lane changes in ideal conditions, and responsive handling of visible obstacles. However, when it comes to following a route accurately, the system falters repeatedly.
Owners report wrong turns, missed exits, inefficient routing through local roads instead of highways, phantom speed limit errors, and even directing vehicles to building rear entrances. Interventions for navigation issues often outnumber those for core driving maneuvers. Tesla has begun surveying owners specifically about these errors, acknowledging the problem after years of complaints.
Navigation is perhaps my biggest complaint when it comes to FSD, because sometimes, we do know better. Some of us have been living in our areas for our entire lives, but even those who have not have years or even decades of experience driving on local roads. We might know a little better about routing.
But the navigation mistakes are more than just FSD potentially taking a slightly different route that may or may not save you a few minutes. Sometimes, they’re genuinely mind-boggling.
This isn’t just annoying; it cascades into broader failures. A flawed route plan confuses the AI’s decision-making, leading to hesitant behavior, unnecessary disengagements, or dangerous maneuvers like attempting impossible U-turns or ignoring clear ramps. In a system meant to operate with minimal supervision, unreliable navigation erodes trust.
More often than not, false or plain incorrect navigation is what causes me to interrupt FSD operation. Unfortunately, I believe the latest FSD version is the worst example of it, and it leads me to believe that Tesla might be making some changes; they’ve just made them in the wrong direction.
It makes you wonder: Why is a company that has done so much with the progress of FSD and autonomy struggling so much with navigation, something that is not new and has been around a long time?
Multiple Data Sources
First, Tesla’s navigation relies on a fragile patchwork of multiple data sources—Google Maps, TomTom, OpenStreetMap, Valhalla, and its own fleet-derived data—stitched together rather than a single authoritative map. When these conflict on lane geometry, road status, or turn details, the system hesitates or chooses incorrectly.
Traditional GPS providers maintain centralized, regularly validated databases with professional curation and rapid updates. Tesla’s hybrid approach, while innovative in crowdsourcing, introduces inconsistencies that a purely vision-based or end-to-end AI approach may not easily reconcile in real time.
Persistent Learning
FSD seems to struggle with persistent learning from driver interventions.
Unlike consumer apps that quickly adapt to repeated corrections or user preferences (e.g., avoiding certain routes or remembering habitual detours), Tesla’s FSD often fails to internalize fixes on the same trip or across similar scenarios. Owners note making the same manual override multiple times without the routing engine updating its behavior meaningfully.
This stems from the neural architecture prioritizing real-time perception and control over long-term route memory and personalization, making navigation feel rigid and “opinionated” compared to the adaptive logic in Waze or Google Maps.
I noticed that when I asked Grok to try and get me home a certain way (a way that FSD routinely took in the past because it was the most efficient), it had to place a waypoint between my location at the time and my house. When I went to edit the waypoint out, as Grok had placed it for a way to get FSD to get off the highway at the right exit, it was stumped again, rerouted, and took a longer way home.
The next thing I’ve noticed, and this might be controversial, is that Nav has gotten even worse.
I think that might actually be a good thing; Tesla seems to be adjusting it. They just need to adjust it the opposite way.
The car is taking extremely strange routes to very… https://t.co/UHg3tVfNA2
— TESLARATI (@Teslarati) June 16, 2026
Reasoning, Scaling, and Intuition
Third, scaling navigation for unsupervised or robotaxi ambitions requires not just accuracy but adaptability and user-like reasoning. Current FSD often defaults to single routes that ignore driver preferences or real-world nuances like time-of-day traffic patterns. It fails to match the intuitive, context-aware planning that traditional systems have refined over the years.
Resolving navigation is critical for several reasons. Practically, it is the backbone of any autonomous journey: without trustworthy routing, the car cannot reliably reach destinations, rendering FSD useless for robotaxis or hands-free commutes. Safety depends on it—mismatched plans create hesitation in merges or intersections, increasing accident risk.
Economically, Tesla’s valuation and future hinge on FSD delivering unsupervised driving; persistent navigation flaws delay regulatory approval and erode consumer confidence. For owners who paid premiums for FSD, these issues represent unfulfilled promises. While it is unlikely Tesla will lose too many customers due to bad navigation, some will be frustrated with the constant need for human input.
Tesla has achieved miracles in electric vehicles and battery tech. Mastering turn-by-turn—technology Garmin nailed in the early 2000s—should not be this hard. By investing in tighter data integration, faster learning loops from interventions, and more intuitive routing algorithms, Tesla could close this gap.
Until then, FSD’s navigation struggles highlight a humbling truth: even the most ambitious innovator must sometimes master the basics before conquering the future.
A Tesla Cybertruck driver in the United Kingdom had their all-electric pickup seized by local police in the Greater Manchester area after the department cited “legitimate concerns.”
Last Thursday, police saw the pickup on the roads and decided to pull the driver over. Greater Manchester Police said:
“Whilst this may seem trivial to some, legitimate concerns exist around the safety of other road users or pedestrians if they were involved in a collision with the Cybertruck.”
🚨 A Tesla Cybertruck, which is illegal to drive in the UK due to safety concerns, has been seized by police in Greater Manchester
“Whilst this may seem trivial to some, legitimate concerns exist around the safety of other road users or pedestrians if they were involved in a… pic.twitter.com/cqhdPok3DM
— TESLARATI (@Teslarati) June 16, 2026
The Cybertruck in question was, according to the BBC, registered and insured abroad and was confiscated. The driver, who is a UK resident, was reported.
The Greater Manchester Police Department then added:
“The Tesla Cybertruck is not road-legal in the UK and does not hold a certificate of conformity.”
The Cybertruck cannot be legally driven in the UK because it has no UK Type Approval for operation in the country. This is due to some safety concerns, which are related to its angular shape and design. The stainless steel exoskeleton has sharp edges and projections that violate UK/EU rules on pedestrian protection.
Tesla has considered creating what it referred to as an “international version” that would be approved for operation in Europe. However, there has been no real movement on that front by the company, as it has been focused on the Robotaxi rollout primarily.
Tesla urges New Jersey owners to oppose new bill that could block Robotaxi
Tesla’s Navigation Nightmare: Why the easiest part of FSD might be the hardest
