At the Siggraph computing event this year, Nvidia’s research isn’t just focused on creating worlds that look real — but that behave realistically and respond in real time.
That shift is the throughline across Nvidia’s 21 accepted technical papers — becoming the foundation of real-time systems that generate virtual worlds and drive machine training in the real world.
Whether the output is a game, film, robot or factory digital twin, the goal is the same: expand the canvas of creativity with AI-generated worlds that are grounded in 3D, governed by physics and directed by creators.
The clearest proof is in MotionBricks: a real-time motion model — trained on more than 350,000 motion clips, running at game-engine speeds — that lets creators direct and connect character movements. The same model that drives the animated character on screen drives a Unitree G1 humanoid robot in the room, using computer graphics and simulation to accelerate physical AI development.
Rev Lebaredian, vice president of physical AI simulation technology at Nvidia, said in a press briefing, “What you’ll see with all of the research that we’re doing is that the throughline, the theme through all of them, is the use of AI to to help us build these real time systems to generate physically accurate virtual worlds for.”
He added, “for both virtual purposes, for enjoying inside the virtual world, but also to build AI systems and and real things inside the real world. These research papers are all grounded in 3D, with physics, and it’s all directable by creators.”
Lebaredian said you can think of of MotionBricks like a foundation model that generates lifelike character motion that is very seamless and smooth, but it does it in real time.
“This can drive both an animated character that’s inside a virtual world or in a virtual experience, and even more exciting, it can generate the motion for a real humanoid robot, online inside the robot’s brain,” he said.
GPC, a framework for training generative controllers on large-scale motion datasets, extends that idea. NVIDIA pretrains a single controller on large-scale human motion, giving it transferable motor skills that carry over to new tasks. Think of it as the start of a foundation model for motor control.
To build virtual worlds in which to test these movements, ArtiFixer turns messy real-world 3D captures into clean, complete virtual scenes. It also includes a new method for predicting photoreal global illumination straight from a scene’s geometry — without tracing a single ray.
To make those virtual worlds behave as they would in the real one, a new solver brings hard-to-simulate materials — such as snow, sand and elastic solids — to life inside the Nvidia Newton physics engine.
Lebaredian said ArtiFixer is a model that can take a rough, very incomplete and noisy 3D scan of the real world, and turn it into a clean and complete 3D scene.
“It can even fill in areas that have gaps, doing hole filling inside the world. It’s quite remarkable,” he said.
And to keep creators in control, the VideoNeuMat pipeline gives them reusable, relightable
materials to pull out of generative video models, while the ARDY autoregressive diffusion model lets them steer 3D character motion in real time from a text prompt.
The papers linked above are openly available, with the code and models free to download.