Why Self-Driving Cars Still Need You Behind the Wheel
Because the only thing autonomous right now is the hype
Hey Folks 👋
Self-driving cars have always been a part of popular fiction. People have long fantasized about cars that could drive on their own, pick them up and drive them off.
Humans pick up driving pretty fast but the same is not the case with machines. Turns out, even companies like Apple have spent nearly a decade and billions of dollars into figuring out a self-driving solution.
Even though some like Apple and Cruise have quit, many are still working on solving self-driving. In fact Chinese and American companies are working hard towards developing successful their self-driving solutions.
Some may think of these investments as unnecessary but it’s pretty likely that we may create something really interesting out of these experiments.
Full self-driving isn’t a reality today but it can very well become one that may fundamentally change how humans travel.
That’s why I’m doing a 3-part series going over the ins and outs of self-driving vehicles, the LiDAR vs Software debate, why Apple and Cruise quit and what the future holds for self-driving cars.
Making Sense of Self-Driving Jargon
The self-driving industry is known to use terms that are really confusing if someone doesn’t know the context behind those terms.
When our Bashir hears terms like Full Self-Driving (FSD), Autopilot and Robotaxis, he thinks of a car without a steering wheel that would automatically take him to his destination without him having to do anything.
But the real world doesn’t work like that (at least for now). From a practical standpoint, most of these terms seem like wordplay to some extent.
Tesla having the biggest fleet of deployed vehicles used the following terms to describe its self-driving system:
Autopilot
Full Self-Driving / (FSD Supervised)
But neither of the terms above describes full self-driving.
Tesla Autopilot
Tesla on its website explains Autopilot as an “advanced driver assistance system”. It claims that it helps enhance safety and convenience for the driver behind the wheel.
Autopilot being a driver assistance system means that it is to be used only with a fully attentive driver behind the wheel.
Autopilot’s main features include:
Traffic-Aware Cruise Control - Through it the car maintains a set driving speed and slows down or accelerates as needed to maintain the following distance from the vehicle in front of you
Autosteer - Through it the car maintains a set speed if there is not a vehicle in front of you or a set following distance if there is a vehicle in front of you and your vehicle detects lane markings, road edges and the presence of vehicles and objects to intelligently keep your vehicle in its driving lane
Autopilot also has a range of safety features like Automatic Emergency Braking, Collision Warnings, Blind Spot Monitoring, Lane Departure Avoidance, etc.
Autopilot comes in Tesla vehicles built after October 2014.
So, Autopilot is Tesla’s basic assistance system that helps Bashir by the wheel in driving but doesn’t drive the vehicle by itself.
Full Self-Driving (Supervised)
FSD is Autopilot’s big brother. It is also a “driver assistance feature” but it’s an advanced one.
FSD (Supervised), introduced in early 2024 has the following features:
Drives to you - Through the Tesla app your car will navigate parking spaces and move around objects as required to find you in a parking lot. It also allows the car to move the vehicle in and out of a tight parking space
Drives for You - Using the on-board cameras this feature allows the car to drive on its own but under human supervision at all times. It allows lane changing and driving on most roadways like residential and city streets.
FSD (Supervised) allows the drive behind the wheels to take a slow breath but still requires active control of the steering wheel. It is not fully autonomous and will disengage if the drive leaves the steering wheel.
Tesla clearly says that “None of these advanced driver assistance features make your Tesla vehicle fully autonomous or replace you as the driver”.
This means that neither Autopilot nor FSD will work without a human present. They need human supervision at all times or they will disengage.
So, FSD (Supervised) is Autopilot’s big brother that adds on top of Autopilot a bunch of additional functionalities but it too requires a fully attentive driver behind the wheel at all times.
Robotaxis
Robotaxis use more advanced technology than Tesla’s rather basic self-driving features but they also have a very limited amount of places they could go to.
Around the world only a few cities have robotaxis designed for them. Robotaxis rely on high-cost hardware and detailed geofenced maps, unlike Tesla’s camera-only approach.
Waymo currently operates Robotaxi services in Phoenix, San Francisco, LA, and Austin while companies like Baidu operate in more than 15 cities like Beijing, Wuhan, Shanghai across China.
The 6 Levels of Self-Driving
While companies continue to use confusing marketing terms, we can use a common yardstick to judge the capability of a company’s self-driving offering through SAE Vehicle Autonomy Levels.
SAE Vehicle Autonomy Levels divides the self-driving capabilities of a vehicle into 6 levels. Below I will go through each level:
Level 0 (No Automation)
This is the lowest level of automation. As the name suggests this level means that there is no self-driving or driver assistance capability present in the vehicle.
In Level 0, a fully attentive human is who drives the car. The vehicle’s direction and speed rest on the driver. Most roads, especially in countries like ours are Level 0.
Level 1 (Driver Assistance)
Level 1 is where many features like cruise control and adaptive cruise control exist.
Level 1 features a single automated system for things like keeping the car at a safe distance to other cars while the driver does the rest like steering or braking.
Level 2 (Partial Automation)
Level 2 is where the Advanced Driver Assistance Systems (ADAS) help the driver by sharing information as well as automating some portion of the driving.
In Level 2, the vehicle can control both acceleration, steering, and deceleration. It can also help in lane changing and remaining in a lane.
Tesla Autopilot and FSD (Supervised) also fall in this level.
Just like the earlier levels, Level 2 requires the human driver to be present and ready to take over at all times.
Level 3 (Conditional Automation)
Level 3 is where the vehicles can monitor environmental changes which allows them to make better decisions.
Level 3 systems are where the vehicle can perform most basic driving tasks but still they require human drivers to be present and take over if need arises.
Level 4 (High Automation)
Things start to get interesting from Level 4 because a large part of driver’s tasks are automated and the human is not required in most cases.
Most of the driving tasks are automated in this level under given conditions without any input from the driver. If something goes wrong the system has protections in place.
Level 4 systems have safe stop functionality that allows the vehicle’s driving system to fall back on itself when necessary.
Although Level 4 can handle most driving tasks there can still be some conditions which this system cannot handle.
Level 5 (Full Automation)
This is the pinnacle of autonomous driving technology. This is what we call true self-driving. A vehicle that drives by itself in all conditions.
A vehicle following SAE Level 5 is worthy of the name ‘self-driving’. These systems are designed to operate with no human intervention at any stage.
Level 5 vehicles can go anywhere, at any time, in any conditions, completely by themselves.
The only sad thing about SAE Level 5 is that we have yet to see a single vehicle following this level and it may still be quite some time until we see one.
The Big Players & their Approach
At the moment we can say that there are three big players in the self-driving industry, if we judge by the number of cars deployed on roads. Those are Tesla, Waymo and Baidu.
Tesla is an automaker that isn’t primarily a self-driving startup but it has heavily invested in self-driving technology.
Tesla is also different in its approach because they’re the only big player that isn’t using LiDAR or advanced sensors for their self-driving features.
Tesla Autopilot and FSD (Supervised) run only on a bunch of cameras with a computer that runs a machine learning algorithm.
Waymo and Baidu are similar as both of them use a combination of LiDAR and other expensive sensors along with geofencing for their robotaxis.
The result is that their self-driving service is Level 4 but significantly expensive and limited to just a couple of cities while Tesla’s approach isn’t theoretically limited in that sense (it still has regulatory and reliability limits).
Note: Although Tesla ranks lower on the SAE level compared to Waymo, we’re still discussing it because of the number of cars on the road.
Because of its fleet, Tesla has the most significant market position and it’s also working to solve self-driving and reaching higher SAE levels.
Tech behind Self-Driving Cars
When it comes to the actual self-driving technology, there are two schools of thought across the many companies. That are:
Computer Vision
LiDAR & Sensors
Computer Vision
We humans use your eyes and our brains to drive our cars. Computer Vision is a field of AI that teaches computers to do just that.
Through the use of cameras to see, Computer Vision systems are trained on a vast dataset of pictures and videos of humans driving.
Computer Vision systems are trained on a lot of data. That training allows them to draw patterns that they could further use in applications like self-driving.
Through Computer Vision self-driving vehicles make sense of the world around them.
The visual input from car’s cameras and other sensors allows it to identify other cars, traffic signs, bicycles, pedestrians and other entities on the road.
Tesla largely believes self-driving is a computer vision problem and can be solved through vision-based software alone without the need of radars.
That’s why the company is a big proponent of Computer Vision and hates using LiDAR for self-driving mainly because they’re significantly more expensive.
LiDAR & Sensors
Light Detection And Ranging (LiDAR) looks like a radar and also pretty much works like one too but it is a bit different because instead of radio waves like a normal radar it emits pulses of infrared light.
Basically LiDAR shoots lasers that we humans can’t see and it waits for the sent lasers to get back to it and the delay in receiving those lasers allows it to calculate a 3-D map.
Basically, LiDAR emits millions of invisible infrared laser pulses per second to create a 3-D map.
In a second, LIDAR could shoot millions of invisible lasers all of which when compiled get us a real time map. That detailed map is then used to spot objects.
In addition to LIDARs which are an industry favorite, companies combine functionality of other sensors too including cameras.
This allows them to get an even enriched environmental map around the self-driving car. The cameras share a 2D map while sensors and LiDAR create the 3D map.
Waymo, Baidu and Zoox, all make use of LiDARs which are significantly more expensive when compared to cameras but costs have fallen dramatically in recent years.
Many experts trust it for pure vision in addition to Computer Vision but companies like Tesla rely only on Computer Vision.
In our next article we’ll go over the details between both of the approaches.
Current State of Self-Driving Cars
Based on the SAE Vehicle Autonomy Levels, most driver assistance systems like Tesla Autopilot, FSD (Supervised) are Level 2.
Mercedes-Benz’s DRIVE PILOT and BMW Personal Pilot are Level 3 operating only in certain markets like Germany and some U.S. states.
Waymo and Baidu robotaxis fall in Level 4. Amazon’s subsidiary Zoox’s robotaxis are Level 4 too.
At this moment, we don’t have any SAE Level 5 car driving on the roads.
So, anyone claiming ‘Full Self-Driving’ at this point is probably lying to you.
Self-Driving is a hard problem and it’s realistic to assume that we still have quite a few years before reaching full self-driving (if we ever could). Some experts estimate full Level 5 autonomy to be a decade away.
Up Next: Diving Deep into Self-Driving Technology
In the next article in this series, I’ll dive deep into the two major technical schools of thought when it comes to self-driving approaches: LiDAR vs No-LiDAR.
People like Elon Musk have publicly shown their disdain for LiDAR, going so far as to say that “LiDAR is lame” (in the context of self-driving cars).
However, this hasn’t stopped companies like Waymo or Baidu in dispatching thousands of cars with LiDARs and other expensive sensors around multiple cities. And early reports are encouraging.
Waymo’s hardware-first and Tesla’s software-first approach is going to be the topic of our next article.
I would really appreciate it if you’d leave any suggestions or topics around self-driving that you want covered.