Teaching AI Self-Driving Cars to Drive Using Simulations

January 4. 2019. 7 mins read

Back in September, one of the world’s most brilliant, ballsy, and just downright crazy billionaires, Elon Musk, said that our current reality is most likely a simulation created by a much more advanced civilization. Of course, that was the same talk show on which we later learned that Musk had ripped on a fat blunt, so you can draw your own conclusions about the validity of that particular theory. Obviously, humans make use of simulations for all sorts of purposes, from teaching pilots how to fly to training AI algorithms how to tell the difference between cats and dogs. A startup like Improbable is trying to simulate the entire world, so if we can’t make it to Mars, at least we can plug into an alternative realm based on 1980s pop culture, when real Earth becomes an unbearable dystopia. Until then, we can also use simulations to help teach AI self-driving cars how to drive.

Long Road Ahead for AI Self-Driving Cars

You might think, judging by the never-ending headlines about autonomous vehicles, that self-driving cars are ready to hit the road and cause even worse traffic jams from LA to New York. We’re not even close. Sure, there are some limited deployments, like an autonomous bus that shuttles retirees around the golf course clubhouse, or self-driving tractors on carrot farms. What we mostly have now are what are called Advanced Driver Assistance Systems (ADAS), with some limited automation capability, such as automatic braking, along with various alerts to help with driver awareness. There is a hierarchy for AI self-driving cars that goes from Level 0 to Level 5. At Level 5, you can kick back behind the wheel and drink Bud Lite while watching the latest skin flix movies on Showtime. The Tesla Autopilot, on the other hand, is around Level 2 autonomy.

Levels of self-driving autonomy.
Credit: Applitools

Getting to Level 5 autonomy – particularly at scale – requires a lot of things to come together. We’ll need super-fast telecommunications through 5G networks so autonomous vehicles don’t need all of the expensive hardware required for intense decision-making computations every second on the road. High-definition mapping and specialized sensors like LiDAR are also part of the big picture of helping AI self-driving cars navigate. And, of course, they’ll have to know how to drive in the first place. That’s mainly been accomplished manually by driving millions of real miles to provide the data needed to teach the algorithms that driving isn’t really like Grand Theft Auto (though some companies actually are using the video game for synthetic training data).

AI Self-Driving Cars Enter the Fuzzy Matrix

Not only is it time-consuming and expensive to build these datasets, but it’s dangerous, too. That’s especially the case in Arizona where people, who have maybe been out in the sun a bit too long, are attacking autonomous test vehicles with rocks and knives. The company under attack by these neo-Luddites, Waymo (owned by Alphabet, aka, Google), is one of the leaders in developing AI self-driving cars. Waymo has also developed its own Matrix-like simulation for training autonomous vehicles how to drive called Carcraft, according to Business Insider. As of last year, the 25,000 cars in its virtual fleet had driven more than five billion miles. BI quotes the chief architect of Waymo’s virtual world, James Stout, in explaining how it works:

We have different universes and worlds running. And we’re testing small variations over and over while creating situations that the vehicles have never seen before. … Fuzzing comes into play.

Fuzzing begins with a driving scenario, like a four-way stop, that draws upon the millions of real-world experiences already collected by Waymo vehicles. The data are then “fuzzed,” morphed into a nearly endless number of scenarios that are analyzed. The lessons learned from Carcraft are then looped back into the AI self-driving car pilots. So, it’s a bit like Neo in the Matrix downloading the complete works of Jackie Chan and suddenly becoming a martial arts expert.

Enter the Matrix with Waymo.
Enter the Matrix with Waymo. Credit: Waymo

AI chip manufacturer NVIDIA (NVDA) has been at the forefront of the autonomous vehicle revolution, providing specialized hardware and software to some of the major automakers out there such as Volkswagen and Volvo. It also offers its own simulation platform called DRIVE Constellation, which simulate an AI self-driving car’s sensors, including cameras, LiDAR, and radar. NVIDIA GPUs then generate photo-realistic data streams that create a wide range of testing environments and scenarios. Platforms like Carcraft and Constellation are especially useful for testing autonomous vehicles in difficult and dangerous conditions, such as rainstorms or snowstorms, as well as different road surfaces or when the glare on your pockmarked windshield creates near-blinding glare.

Startups Teach AI Self-Drive Cars How to Drive

The bright minds at CB Insights noted in their recent report on 2019 mobility trends that some automakers, rather than build in-house simulation platforms or buy into the NVIDIA DRIVE system, are turning to startups for simulation services. As you can see in the chart below, CB Insights still ranks simulation technology fairly low on its widespread adoption by industry:

A roadmap to the trends guiding mobility automation in 2019.
A roadmap to the trends guiding mobility automation in 2019. Credit: CB Insights

Now, let’s take a look at four of these startups that are likely looking for deals with major auto manufacturers.

Click for company websiteFounded in 2016, Israel-based Cognata has raised a total of $23.5 million, including an $18.5 million Series B in October 2018. Most of its investors also participated in a $5 million Series A in 2017, including the venture capital arm of Airbus. Cognata uses “computer vision and deep-learning algorithms to automatically generate a whole city-simulator including buildings, roads, lane marks, traffic signs, and even trees and bushes.” It then shakes the whole thing up by adding layers of real-world traffic data, data from vehicle sensors, and finally throws in weather conditions based on historical data to stress-test the system.

Congnata's simulation engine takes to the mean streets of San Francisco.
Congnata’s simulation engine takes to the mean streets of San Francisco. Credit: Cognata

The young startup scored a major coup last year when it inked a multi-year deal with Audi to speed up development of the automaker’s AI self-driving car line.

Click for company websiteFounded in 2017, Applied Intuition out of Sunnyvale, California, took in its first round of VC funding last September, with the marquee firm Andreessen Horowitz co-leading the $11.5 million Series A. Bloomberg notes that Applied Intuition lets customers simulate more than 100,000 different road scenarios in its elaborate 3D game world, which may include even more twists and turns than Netflix’s new choose-your-own-adventure Black Mirror movie. A dashboard provides different insights, such as “how each virtual intersection and obstacle affected a car’s speed acceleration rate and something called ‘calculated longitudinal jerk,’ a measure of how comfortable the ride would be for passengers,” Bloomberg reported.

A simulation developed by Applied Intuition.
A simulation developed by Applied Intuition. Credit: Applied Intuition

Applied Intuition works with a range of clients, from a $50 billion multinational automaker looking to accelerate its AV efforts to a Silicon Valley startup focused on AI self-driving trucks. The latter, for instance, requires simulating a wide range of lane-change scenarios that trucks have to deal with, like motorists who think it’s a good idea to try to speed past a 40-ton truck the moment its blinker comes on.  

Update 11/12/2021: Applied Intuition has raised $175 million in Series D funding to serve the increased demand for its software tools for autonomous system development and to support additional use cases across industries. This brings the company’s total funding to $351.5 million to date.

Click for company websiteFounded in 2017, Silicon Valley-based Parallel Domain raised a $2.9 million Seed round last May, with participation from Toyota, among others. Jim Adler of Toyota AI Ventures writes of the company:

Parallel Domain automates the creation of rich virtual worlds to enable developers to deploy autonomous technology faster, more safely, and at a lower cost. The company’s software enables developers to generate a seemingly infinite variety of detailed simulation environments and dynamic scenarios based on both real and fictional locations.

So, we’re probably all starting to get the idea at this point. The company has announced a partnership with NIO, the billion-dollar Chinese electric car startup that went public last year to challenge Tesla and its philosopher-king Musk. Parallel Domain also has a partnership with NVIDIA to use its hardware and software systems.

Click for company websiteFounded in 2016, Metamoto is another Silicon Valley startup building virtual worlds to simulate driving conditions for autonomous vehicles. The company took in $2 million from a Seed round in July 2017 and reportedly started working on fundraising for a Series A last summer. Whereas its rival Parallel Domain plans to sell its platform directly to autonomous vehicle companies, per TechCrunch, Metamoto plans to offer Simulation as a Service (SaaS). Its offering includes three components: Director, a simulator for testing across a spectrum of edge cases; Designer, a tool to build customized scenarios via a visual editor; and Analyzer, a tool to see how many people were killed how well the AI driver did during the simulation.

And if none of the above options are affordable for the AV startup on a shoestring, Intel Labs, the Toyota Research Institute and the Computer Vision Center in Barcelona, Spain, got together to create an open-source simulator for AI self-driving cars.


Sticks and stones may break a few windshields, but they won’t stop automation throughout the transportation sector. The electric train has left the station. Roughly between 30,000 and 40,000 people die every year in the United States, so the bar for AI self-driving cars isn’t really that high. Still, a report by RAND Corporation back in 2016 said autonomous vehicles would need to drive billions of miles to demonstrate reliable safety. Simulations offer a valuable shortcut for getting there quickly.


Leave a Reply

Your email address will not be published.