“We know what OnLive tried to do years ago,” Freeman said. “But what if we could stream fully interactive content with no lag, no loss, no frame drops or no downloads, and provide an instant interactive experience at the highest quality, and actually at a much lower cost using today’s infrastructure. What we’re the most excited about is the interactivity. So VR, MR, AR content is really possible with our technology. And since we’re not downloading all the content ever, we eliminate piracy out of the equation. This is not traditional cloud gaming as others have approached it. It’s an entirely new way of streaming data.”
The mad doctor
Jenkins got his medical degree from the Harvard University Medical School. But he became very interested in computational models of human vision and computer vision and real-time graphics. He wrote software for like massive 3D data set processing that was used by defense contractor Northrop Grumman for rendering imagery.
“We are focused on the 3D data itself, not video data, which is really what Stadia and GeForce Now do,” Jenkins said. “They run the entire game, of course in a data center, and then they compress the video frames, and then it gets sent to the user. And as you know, this was possible many years ago, but not necessarily practical. It does have a very high cost per user and almost an unprecedentedly high cost per user in the data center. We estimate that Stadia dedicates $1,500 dollars worth of equipment to one user at one time.”
The compression of the game content into video form isn’t that good, and so it takes up a lot of bandwidth. There is also a lot of latency, or interaction delays, in a cloud game as the player’s laptop sends an input, it travels to the data center, gets computed there, and then gets sent back down to the computer in the form of a video. At 720p resolution, this lag is OK. But not with today’s 4K TVs.
How to fix game streaming
Instant Interactive’s software can fix this, as it is integrated into the Unreal Engine and will enable game publishers, game developers, and game distributors to give their users a better experience, Jenkins said.
“We’re actually streaming the game engine content itself in a very sort of agile way to the gaming console to PC or mobile device,” Jenkins said. “We take the game itself, all the levels that make it up a game, and we process them offline through our GPEG encoder, and we turn it into data, which is GPEG. That packet data can sit on any server. It doesn’t have to have a GPU, basically any CDN server.”
He added, “What it does is it streams of data into our GPEG client software, which is a plug-in for Unreal Engine. So we have integrated this into Unreal Engine. It’s written to be integrated into other engines as available. And basically instead of downloading the entire game, we just, we initially very quickly stream the just the data that that game needs in the moment. It’s all prefetched. So there’s, you know, the user doesn’t experience any latency.”
The software takes a given scene and breaks it up into cells. Then the software precomputes exactly what surfaces which texture triangles are visible from each box. And then we encode the change between the boxes. All this data sits on a server, and the server predicts where a user is inside a game and then only shows the parts of a scene that the user can see. It also predicts what the user might see in the next few moments and it precomputes that to show it to the user. The game engine doesn’t have to render the rest of the scene.
“So instead of downloading your Call of Duty games as a 140GB download, we could really get this started with a few 10s of megabytes of data,” Jenkins said. “There’s a real opportunity here to improve the performance of the game by managing very precisely what the game engine actually has to render at any one point in time.”
Reaching into the past
Computer scientists have studied this concept for decades. John Carmack, the graphics guru who once worked at id Software on games like Doom and Quake, was the first one actually use this kind of technology in a game, in the Quake game engine from the 1990s.
“He actually accelerated the game by like a factor of three or four in the frame rate like by using this pre-computed visibility,” Jenkins said.
While others left this technology behind, Jenkins did not. And he applied it brought it into the modern world.
Instant Interactive wants to license what it has created — middleware — to other companies.
“We’d like to see it used by many different distributors and producers of games from Valve to Epic to Activision and EA,” Freeman said.”The bandwidth requirement is actually much more reasonable than video-based game streaming because we’re not using any compression. We’re actually exploiting the structure of the game data itself.”
Jenkins said the GPEG can work with better interactivity, instant access, efficient and lossless transmission, and fast delivery of 4K and VR content.
“This is middleware that is really designed to help the game engine run better,” Jenkins said. “And say you are watching Netflix and decide to jump off the cinematographer’s camera rail, for a different narrative or to pick up a brief challenge, and become more immersed and engaged in the story. Video-based streaming doesn’t really allow you to do this. Within a matter of seconds, you’re actually playing other content. A 22-minute video episode could become an hour or more of engagement every week.”