Have you ever wondered if AI could be used in games? Certainly there have. Lots of it. I can think of hundreds, if not thousands, of examples of the use of machine learning and artificial intelligence in game development. But even with all the imagination, we are still amazed at what Nvidia's research team has accomplished today: an AI that can recreate Pac-Man with just a few pixels and keystrokes.

The generative adversary network (GAN) outlined in the research paper, nicknamed GameGAN, can take pixels and input data from a video game and recreate a similar copy; the AI actually recreates the game based on prior events, player actions, and environmental generates a new frame for each on-screen event based on randomness

.

Its first task was to create the video game classic Pac-Man. Don't be fooled by its simple appearance. Even a game like Pac-Man is a daunting task, a far cry from the complexity of today's AAA titles. This is because in order to build a game, the AI must first understand its rules.

To do that, it needs gameplay data, and lots of it. Nvidia has collected enough data to build a second AI that can play Pac-Man over and over again. It took 50,000 "episodes" of gameplay (millions of frames total) to give GameGAN enough information to reproduce the game logic.

And that's just the basic information; GameGAN learns only from the pixel information on the screen and the key presses registered during gameplay. None of the basic code needed to build and run the game is passed to the AI.

From there, GameGAN was taught to generate its own copycat game rule sets (across several Nvidia DGX systems). For example, players must learn that walls block PacMan's moves in order to keep him from breaking the game each turn.

Three main modules in the neural network help Pacman achieve this. The team also realized a game mode called "egocentric," in which Pac-Man takes the lead role and moves around the map without being able to see what is ahead.

"We think this could seed a variety of uses for AI-assisted game building," says Reb LeBaredian, NVIDIA's vice president of simulation technology, in a preliminary briefing. Before we can do any of these things, we need an AI smart enough to understand the basic rules, and once the AI understands the rules, then we can do other things with the AI, like building levels."

GameGAN can also use existing information to create new levels and environments, and researchers believe it may even one day help developers and users develop and simplify the levels and characters they create.

"It can help game developers as well as end users," says Sanja Fidler, AI director at Nvidia. 'The neural network has to separate the components and learn about those different components in order to learn to recreate the game. One can imagine end users actually modifying the game in this way." [The research team behind GameGAN consists of Nvidia, the University of Toronto, the Vector Institute, and MIT. The full research paper can be found here. Their work joins other Nvidia-led projects such as GauGAN, a photorealistic image generation app, Image Inpainting, a photo reconstruction tool, and GANimal, a pet face swapping app. Yes, AI can be quite strange.

What's next for GameGAN, you might ask, and Nvidia researchers have not yet ruled out 3D video games. We are told, "This technology does not fundamentally limit 2D." Nor is this technology limited to video games. Researchers also believe that advanced AI capable of learning virtual rule sets is critical to the development of AI for real-world applications, minimizing the work of generating reality-faithful simulations.

The possibilities for game development and AI are nearly endless and seem to fit each other perfectly; we feel that GameGAN is just the beginning of the relationship between AI and video games. It offers a glimpse (and just a bit sobering) of what will be possible with more complex and advanced AI algorithms in the future.

Those who want to try GameGAN for themselves can try it out at Nvidia's AI Playground starting later this year. There, you'll also find the other AI-powered apps mentioned above, which you can try out at any time.