News
Tesla is patenting a clever way to train Autopilot with augmented camera images
Tesla is currently tackling what could only be described as its biggest challenge to date. In his Master Plan, Part Deux, CEO Elon Musk envisioned a fleet of zero-emissions vehicles that are capable of driving on their own. Tesla has made steps towards this goal with improvements and refinements to its Autopilot and Full Self-Driving suites, but a lot of work remains to be done.
As noted by Tesla during its Autonomy Day presentation last year, attaining Full Self-Driving is largely a matter of training the neural networks used by the company. Tesla adopts what could be described as a somewhat organic approach for autonomy, with the company using a system that is centered on cameras and artificial intelligence — the equivalent of a human primarily using the eyes and brain to drive.
Tesla’s camera-centric approach may be quite controversial due to Elon Musk’s strong stance against LiDAR, but it is gaining ground, with other autonomous vehicle companies such as MobilEye developing FSD systems that rely primarily on visual data and a trained neural network. This approach does come with its challenges, as training neural networks requires tons of data. Tesla emphasized this point as much during its Autonomy Day presentation.
With this in mind, it is pertinent for the electric car maker to train its neural networks in a way that is as efficient as possible with zero compromises. To help accomplish this, Tesla seems to be looking into the utilization of augmented data, as described in a recently published patent titled “Systems and Methods for Training Machine Models with Augmented Data.”
Teslas are equipped with a suite of cameras that provide 360-degree visual coverage for the vehicle. In the patent’s description, Tesla noted that images used for neural network training are usually captured by various sensors, which, at times, have different characteristics. An example of this may lie in a Tesla’s three forward-facing cameras, each of which has a different field of view and range as the other two.
Tesla’s recent patent describes a system that allows the company to process these images in an optimized manner. Part of how this is done is through augmentation, which opens the doors to flexible and widespread neural network training, even when it involves vehicles equipped with differently-specced cameras. The electric car maker describes this process as such:
“Augmentation may provide generalization and greater robustness to the model prediction, particularly when images are clouded, occluded, or otherwise do not provide clear views of the detectable objects. These approaches may be particularly useful for object detection and in autonomous vehicles. This approach may also be beneficial for other situations in which the same camera configurations may be deployed to many devices. Since these devices may have a consistent set of sensors in a consistent orientation, the training data may be collected with a given configuration, a model may be trained with augmented data from the collected training data, and the trained model may be deployed to devices having the same configuration.”
Among the most notable aspects of Tesla’s recent patent is the use of “cutouts,” which allow Tesla’s neural networks to be trained using an optimized set of images. This was something that was discussed by former Tesla Autopilot engineer Eshak Mir in a Third Row Podcast interview, where he hinted at a system adopted in the electric car maker’s ongoing Autopilot rewrite that helped lay out “all the camera images” from a vehicle “into one view.” Such a process has the potential to help Tesla with 3D labeling, especially since the images used for neural network training are stitched together. Tesla’s patent seems to reference a system that is very similar to that described by the former Autopilot engineer.
“As a further example, the images may be augmented with a“cutout” function that removes a portion of the original image. The removed portion of the image may then be replaced with other image content, such as a specified color, blur, noise, or from another image. The number, size, region, and replacement content for cutouts may be varied and may be based on the label of the image (e.g., the region of interest in the image, or a bounding box for an object).”
Tesla is aiming to release a feature-complete version of its Full Self-Driving suite as soon as possible. Elon Musk remains optimistic about this, despite the company missing its initial timeline that was set at the end of 2019. That being said, Elon Musk did mention previously that Tesla is working on a foundational rewrite of Autopilot. In a tweet early last month, Musk stated that an essential part of the rewrite involves work on Autopilot’s core foundation code and 3D labeling. Once done, the CEO indicated that additional functionalities could be rolled out quickly. This recent patent, if any, seems to give a glimpse at how these improvements are being done.
Elon Musk
Tesla shows what Roadster drivers will see on screen before thruster activation
Tesla’s Roadster video reveals a fighter jet inspired cockpit with a 5,500 PSI thruster gauge.
Tesla has given future owners their clearest look yet at the cockpit of its next generation Roadster. In a 34 second video posted to X on Friday afternoon, the company walked through the car’s touchscreen with a caption: “Inspired by fighter jet cockpits, Roadster’s interface was designed around the pilot.”
The clip opens on a black screen with red graphics and a 3D wireframe of the car, which sits beside the Roadster’s squared off yoke. Tapping parts of the model brings up controls for the rear wing, including low drag and high downforce positions, along with a charging page with a charge limit bar and a small music player. A row of tabs along the bottom reads Home, Vehicle, Listen, Climate and Navigate.
Most of the video is spent in Track Mode. The dashboard shows a live circuit map, a list of session lap times, brake and motor temperatures, and pressure and temperature readouts for each tire. A setup page lets the driver pick Auto, Low Drag, High Downforce or Custom aero modes, adjust aero balance and brake balance, and switch stability and traction control off. Along the bottom, a status bar reads “Thruster · 5,500 PSI” next to a “Boost Ready” indicator.
That gauge ties the software to the hardware Tesla showed off near Waco, Texas on October 15. The Thrust and Rocket versions use cold gas thrusters fed by carbon overwrapped pressure vessels, and Tesla lists 0 to 60 mph times of under 0.9 seconds and under 0.59 seconds for them, as Teslarati detailed in its breakdown of the three Roadster configurations. The base Performance car relies on a fan and ground sealing system for grip, and the aero settings on screen match the active wings and suction setup covered in our look at how the Roadster generates downforce.
The Roadster layout is a departure from the rest of Tesla’s lineup. Instead of a map centered screen, the Roadster puts the car itself at the center of the home page, and Tesla’s Roadster page lists two smaller forward displays alongside the main touchscreen. Tesla has not said whether any of the Track Mode telemetry will make its way to the Model 3 Performance or other vehicles.
The interface could still change before deliveries. Elon Musk said at the event that Roadster production will begin in 2028, and Tesla says the flagship Rocket version is not approved for public road use. Reservations are open with a $50,000 payment, while Thrust is listed as a limited release and Rocket is invite only.
Those who want to see the car in person will get their first chance this weekend. The Roadster goes on display in the lobby of the Petersen Automotive Museum in Los Angeles from October 24 to 27, according to Sawyer Merritt, the first time the public will be able to view it up close.
Elon Musk
Tesla reveals Optimus Academy, where robots teach themselves
Tesla has begun building production quality Optimus robots in Fremont and revealed its training plan.
Tesla has started building Optimus on the high volume line it set up inside its Fremont factory, and the company says the robots coming off it are the first that meet the bar for a real product. “The bots coming off this line are the first production-quality bots that we have manufactured,” Tesla wrote in its third quarter shareholder letter, adding that it has “proven that the current manufacturing process for Optimus is suitable for a commercial product.”
The line sits in space that built the Model S and Model X until this spring. Tesla cleared the old vehicle line in 46 days, as Teslarati reported in July, to make room for robot production sized for one million units a year. On Wednesday’s earnings call, Elon Musk said most of the equipment for that capacity is already in Fremont, with output set to climb toward roughly 20,000 robots a week as the line ramps.
Musk described the ramp as a classic S curve. “The growth of Optimus is an S curve, but the initial curve is a shallow slope,” he said. The team is now refining the robot’s AI, electromechanical design, reliability and service life, and Musk expects that work to be finished by the end of next year, when Tesla plans to start delivering Optimus to outside companies for dangerous, repetitive and around the clock jobs.
The most interesting part of the call was how Tesla plans to teach the robots. Its cars send real driving data back to the AI team from millions of vehicles, but Optimus has no fleet like that yet. “A big difference with Optimus versus the cars is we don’t have this vast fleet of cars from which we can gather training data,” Musk said. Tesla’s answer is an Optimus Academy, where 10,000 to 20,000 robots will practice a wide range of tasks and feed what they learn into the next version of the model. Every unit off the Fremont line becomes a source of that training data.
Smarter hardware is on the way too. Tesla plans to put its AI5 chip into Optimus next year, and Musk said a new Optimus video should arrive within one or two months showing the robot completing tasks driven by AI rather than preprogrammed motions. He took aim at rival humanoid demos that only run scripted routines, saying those would not hold up in the real world.
The supply chain is lining up behind the program. In September, Tesla certified Tuopu, Joyson and Sanhua as mass production partners for Optimus components. A far larger Optimus factory is also rising at Giga Texas, where drone footage from Joe Tegtmeyer showed the steel frame nearing completion, and Musk has pegged that plant’s long term target at 10 million robots a year.
Tesla executives also said Optimus production costs are trending toward the expected manufacturing cost of Cybercab, a sign the robot is being engineered for volume from the start. The hands remain the toughest piece of the design, a point Musk also made at the X Takeover event earlier this month, as Teslarati covered at the time, and that is where much of the current engineering effort is focused.
Elon Musk
The Starship moon landing number NASA and Elon Musk can’t agree on
Musk disputes NASA estimate of 22 Starship launches for Moon landing, predicting 6 to 8.
Elon Musk is disputing a new NASA safety panel estimate of how many Starship launches it will take to put astronauts on the Moon, saying the real number will be less than half the panel’s figure.
On Thursday, NASA’s Aerospace Safety Advisory Panel held its quarterly public meeting. Panel member Charlie Precourt, a former NASA astronaut, flagged the fueling process for SpaceX’s lunar lander as a major risk for Artemis IV, the mission planned to land a crew on the Moon in 2028. “We need to keep our eyes on numerous required fuelings for the lander,” Precourt said, according to Ars Technica. “I think the last estimate is around 22 flights.”
Ars Technica’s Eric Berger posted the comment on X, and Musk replied. “That’s way too high,” Musk wrote. “Actual number will be more like 6 to 8.”
The disagreement comes down to a basic problem with sending a ship as large as Starship to the Moon. Starship burns nearly all of its propellant just reaching Earth orbit, so it cannot continue on to the Moon without stopping for a refill. SpaceX’s answer is a gas station in space. First, a Starship that works as a storage tank, called a depot, is launched into orbit. Then a series of tanker Starships launch one at a time, dock with the depot and pump their leftover propellant into it. Once the depot is full, the lunar lander version of Starship launches, docks, fills up and heads for the Moon.
Every tanker trip is its own full rocket launch, which is why the count matters. More launches mean more time, more cost and more chances for something to go wrong before astronauts ever leave Earth. Berger estimated that even if SpaceX reaches twice weekly Starship launches by the end of 2027, filling a depot at the panel’s number would still take roughly three months.
How many tankers are needed depends mostly on how much propellant each one can carry to orbit and how much is lost along the way. Starship’s fuel and oxygen are stored at extremely cold temperatures, and some of it slowly boils off while waiting in orbit. More is lost in each transfer. NASA appears to be working from what Starship can lift today, while Musk’s range assumes a more capable rocket.
Musk has pushed back on higher counts before. In 2021, after a government document suggested 16 launches, he called that “extremely unlikely” and said eight tankers would be the most required.
Much of the answer depends on hardware SpaceX has not finished flying. As Teslarati noted earlier this year, the company plans to carry out the refueling process with its upgraded V3 Starship, and Precourt described Artemis IV as carrying “a large number of firsts.” He also praised NASA Administrator Jared Isaacman’s team for working to clear obstacles for both SpaceX and Blue Origin, whose Blue Moon lander does not rely on orbital refueling for its first mission.
Tesla shows what Roadster drivers will see on screen before thruster activation
Tesla reveals Optimus Academy, where robots teach themselves
