Teen Programmer Uses Real Self-Driving Car Tech in 'GTA V'

Cars that can drive themselves in certain situations are an expensive luxury, but programmers, hobbyists, and gearheads across the world are working to drive down the cost of the technology and make it easier to use. Comma.ai, founded by infamous hacker George Hotz (AKA "geohot"). is one of those companies. Drivers can install the "comma two" kit in supported models and take the technology for a spin.

For those not ready to make the leap, programmer Leon Hillman has taken Comma.ai’s open source driving software and retrofitted it to control vehicles in Grand Theft Auto V. What that means is they have figured out a way to put self-driving cars into a video game from 2013, relieving the player of the compulsion to, uh, play the game.

Comma.ai lets people retrofit recent model Honda and Toyota cars with software that brings automated lane centering and adaptive cruise control for use on highways and in stop-and-go traffic to the vehicles. The software, Openpilot, is open source so anybody can manually add support to more vehicles, if they have the ability to do so.

Comma.ai places a camera system inside the car that views the road and the driver, a tablet to give driver feedback, and a cabling system that lets the system connect up to the car and run Openpilot. “Openpilot works by taking the radar data integrated with supported car models and combining it with the camera built into comma hardware, to determine what acceleration, braking, and steering events are required,” Comma.ai’s website explained.

Leon "littlemountainman” Hillmann, a programmer living in Munich, augmented the Openpilot software to work in Grand Theft Auto V. According to his GitHub, the hardest part was getting Openpilot and Grand Theft Auto to communicate with each other. Openpilot collects data both from its cameras and from a car’s on-board diagnostics port. (OBD)

“Cars in GTA just don’t have a [control area network bus] or an [inertial measurement unit] or anything like that that would make our life easier,” Hillmann said on GitHub. “So because controls wouldn’t work without the sensors and would crash then, I first started by making my own GTA car interface which is essentially based off of a Honda civic from 2018.”

It took Hillmann two weeks to code the initial input for the steering wheel, gas, and break. After that, he spent a few days making the whole thing work with an Xbox controller. Now, he’s got a version of Openpilot that works with Grand Theft Auto.

“So what you need is two PCs. One PC with Windows and all the Xbox drivers installed and one laptop or PC with Ubuntu 16.04 running openpilot with a webcam,” he said on Github. “I would recommend at least a 1080p webcam for any kind of openpilot webcam thing. I used a Logitech C920 but the quality still wasn’t really outstanding.”

The idea of using video games to train real-world self-driving car AI is an old one. The stakes are much lower and it’s always better to suffer the consequences of bad code when human lives aren’t on the line.