Nvidia’s GameGAN generates games like Pac-Man by watching videos

Two years ago, Nvidia researchers detailed AI that could generate visuals and combine them with a game engine to create an interactive, navigable environment. In something of a follow-up to that work, scientists at the company, the Vector Institute, MIT, and the University of Toronto this week published a paper describing GameGAN, a system that can synthesize a functional version of a game without an underlying engine.

While game generation on its face might not seem like the most practical application of AI, algorithms like GameGAN could be one day used to produce simulators for training robotic systems. Before it’s deployed to the real world, robot-controlling AI typically undergoes extensive testing in simulated environments, which comprise procedural models that synthesize scenes and behavior trees specifying in-simulation agents’ behaviors. Writing these models and trees requires both time and highly skilled domain experts, which translates to an uptick in expenditures by companies looking to transfer models to real-world robots.