Niantic Spatial, the team pioneering AI that understands the physical world, is showcasing a compelling glimpse of what can happen when spatial intelligence and augmented reality meet. It’s basically a companion critter from Niantic’s Peridot game.
In partnership with Snap (the maker of AR glasses hardware) and Hume AI (voice AI startup), Niantic Spatial has developed location-aware companions for Spectacles, blending Snap’s AR glasses, Niantic Spatial’s Large Geospatial Model, and Hume’s Empathic Voice Interface (EVI) for natural, emotionally intelligent conversation.
Its flagship demo, Project Jade, features Dot—a companion that guides users through the real world with deep contextual understanding and rich, natural dialogue.
Creating human-like AR interactions
As the new era of spatial computing dawns, traditional screen-based navigation becomes
insufficient. Drawing on a 25-year legacy of leadership in geospatial AI, mapping, and AR, from Google Earth to Pokémon GO, Niantic Spatial understands our present world requires a new form of wayfinding.
The team knew the solution wasn’t just better directions; the real solution is creating an authentic guide with a true understanding of space—one that feels natural and helps
people connect more deeply with the world as they navigate.
Niantic Spatial’s vision for this guide is built on three crucial pillars:
● First-person awareness: an AI that literally sees what the user sees, and from the
same perspective, through AR glasses.
● Context-rich storytelling: dynamic insights and understanding tied to the environment.
● Emotional intelligence: conversation that feels like exploring with a knowledgeable,
empathetic friend.
When you ask Dot about a hanging art installation in an airport terminal, for example, it doesn’t just recite facts—it shares the artist’s vision with genuine enthusiasm, adapting its tone to your level of interest.
Achieving this requires more than cutting-edge mapping and hardware. It demands an
emotionally adaptive voice.
Niantic Spatial’s partnership with Hume
Hume’s Empathic Voice Interface (EVI) powers the natural, emotionally aware voice behind
Niantic Spatial’s location-aware companions, making interactions with them feel personal:
● Sub-second responsiveness for seamless, natural dialogue.
● Emotionally adaptive delivery that shifts tone based on user engagement, becoming
enthusiastic when sharing a story and calm when providing directions.
● Seamless API integration with Niantic Spatial’s geospatial AI stack, including its Visual
Positioning System (VPS) and Large Geospatial Model.
As Asim Ahmed (Head of Product Marketing at Niantic Spatial), said in a statement, “Our vision with this experience is to transform the traditional map from a static tool into a living companion that makes you love the journey. We provide the foundational geospatial AI for this, a ‘smart map’ that understands the world with unmatched precision…but for that map to truly come to life, it needs a storyteller’s personality. Hume’s technology provides that critical emotional layer, giving a powerful, empathetic voice to the world’s hidden stories.”
Niantic Spatial’s geospatial AI serves as the core of the experience, brought to life through
Snap’s AR hardware and given that distinct personality by Hume. This powerful combination transforms Dot from a simple guide into a genuine companion and represents a fundamental shift in how we interact with digital spaces.
You can read more about the development of Project Jade on Niantic Spatial’s website.
Niantic Spatial said that at Snap’s Lens Fest, the team’s AI companion, Peridot, is evolving. In an experience tailored for the event, it’s moving beyond just understanding its surroundings to actively navigating them alongside you.
Attendees can ask Dot about interesting sights and featured experiences, and it will physically guide them there. It will share context along the way, conversing through natural, emotionally aware and conversational AI powered by Hume AI. With similar demos now active at major Niantic Spatial offices in San Francisco, Seattle, London, and Tokyo, more people can check it out.
This demo marks the next waypoint on a journey that Niantic Spatial began at AWE with Project Jade. Our first step was to give our companion a suite of senses: sight via Snap’s AR glasses, Spectacles, a deep understanding of the world via our Visual Positioning System (VPS), and an AI brain for memory and knowledge. This allowed Dot to hold an intelligent conversation about a single location.
With today’s update, we add the final piece. By giving it the ability to navigate, we’re evolving our spatially-aware companion into one that can embark on a journey with you.
This leap required us to solve one of the most fundamental challenges of AR navigation:
teaching a digital companion the invisible rules of a physical space. How do you teach it to
navigate a crowd or differentiate a path from a physical object? Our solution is a proprietary internal tool that ingests 2D floorplans to algorithmically generate a comprehensive pathfinding map, that can further be modified. This map teaches our companion all the possible routes it can take, ensuring it navigates spaces intelligently without walking through walls or into restricted areas, while our computer vision algorithms can provide the real-time semantic understanding needed to avoid obstacles and hazards.
But a great companion needs more than just a sense of direction; it needs personality. To make Dot even more delightful and lifelike than in its first iteration, we’ve collaborated with our partners at Hume AI to integrate EVI, their advanced conversational AI, that delivers sub-second, emotionally adaptive dialogue in real time, enabling Dot to respond naturally like a true companion. This gives Dot a more nuanced and empathetic voice, transforming the experience from simply receiving information into something that feels more like exploring with a curious and trusted friend.
Imagine a future where your companion in a new city isn’t a blue arrow on a screen, but a
character who walks alongside you, sharing stories about architecture and pointing out hidden gems you’d otherwise miss. Imagine walking through a busy airport terminal, where your companion draws your attention to a massive hanging art installation, explaining the artist’s vision and how it was constructed.
Niantic Spatial is working on a Large Geospatial Model that will allow us to deliver these experiences to scale: more coverage, higher fidelity, lower cost, and greater repeatability.
Project Jade and its underlying technology is an exciting example of what’s possible, enabling a new wave of experiences, from advanced AR companions to intelligent robotics, that help us find more connection, more understanding, and more joy in the world around us, said Asim Ahmed, Head of Product Marketing at Niantic Spatial.
Niantic Spatial’s work on the project
Alicia Berry, executive producer at Niantic Spatial, said her company created a special demo just for the Snap Lens Fest event. It uses an AR navigation app for Snap, utilizing Niantic’s Visual Positioning System (VPS) for the first time on a wearable AR headset.
The app, built in under two and a half months, serves as a tour guide for Lens Fest, providing information about exhibits and locations. The technology, which has been tested in a small demo at Augmented World Expo, aims to enhance real-world experiences. Berry described the experience as “magical.”
This time, Niantic Spatial also collaborated with Hume AI for voice-driven, emotion-rich synthetic voices. The platform aims to provide contextual information in various applications, including gaming and non-game uses.
For Project Jade, Niantic Spatial integrated its VPS system with Snap’s ecosystem. So it then built an experience where Peridot could be your tour guide for the event. The team built it in 2.5 months.
“Before this, has been very challenging to build something like this because of how people interact with sound and light, but with the right moment in time, the tech has aligned, and we can do this today,” Berry said. “it’s the first time this experience has been seen.”
The goal is to have the Niantic Spatial tech working for the Snap consumer launch, which is scheduled for next year.
Backing out a bit, Niantic Spatial is the geospatial model company. Scopely, the maker of Monopoly Go, bought Niantic, the maker of the Pokemon Go games, and all of the games that Niantic made. But it didn’t buy Niantic Spatial, which has the mapping platform underlying the games, and so it spun out as an independent firm.
“We are working on understanding for computers and humans to understand the real world, to be able to provide context to what your camera is seeing or what your eye is seeing. What we’re really trying to do is find a way to meet people’s expectations” related to cameras figuring out what where you are in the world and what is in front of you.
“What Niantic Spatial is working on is the system for applications like this one to be able to understand where you are, what you’re seeing, what you’re doing, and provide you contextual information that is helpful,” said Berry.
For things like smart AI glasses, that means figuring out what is in front of you and then having the AI add to your experience, like telling you the history of a statue in front of you.
“In our experience building games, we think that this is a foundation for future experiences like MMOs,” Berry said. “Imagine a real world quest system, where you don’t to do scans, and it just knows where you are. There are many games that have come before this who have tried to do it, but as of now, it hasn’t been easy to understand the real world.”
Niantic Spatial is exploring the non-game uses, Berry said, like remote collaboration, expert troubleshooting live at industrial or construction sites, delivery logistics and more. Niantic Spatial has a service in the market now, focused on its AR platform.
“What’s coming next is the ability to do things like understand where you are in the real world without GPS or without having prescanned,” Berry said.
Niantic Spatial is partnering on the Meta Quest VR headsets too. One of the goals is to design systems where people no longer have to push buttons on glasses. That’s harder than it sounds.
Too often, other AI tech had canned voices that were very flat. They don’t produce the right emotions to make people more engaged. But Hume has tech that is trained to detect emotions behind spoken words. So Hume created a synthetic voice for the Dot character. The latency was low and the voice sounds different from session to session. That breathes more life into the character.
Hume’s view of the project
Alan Cowen, CEO and chief scientist at Hume.AI, discussed his company’s partnership with Niantic Spatial to create an AR avatar that can interact with its environment and users. Hume.AI’s speech-to-speech foundation model, launched in April 2022, enables customizable voice responses without requiring fine-tuning.
The technology, which can respond in 300 milliseconds, is being used in a demo similar to Pokémon Go but with talking characters from Peridot. Hume.AI’s model also includes emotional intelligence and can be used in gaming, assistive technology, and educational applications. The company, with 45 employees, aims to provide immersive AI experiences.
Cowen said that Niantic is integrating Hume.AI’s API to enable voice queries and responses based on context. The AI can respond with appropriate emotions, making the character’s voice more realistic and engaging.
Hume.AI’s technology handles the intelligence behind the responses, making the experience more dynamic and personalized. Hume.AI’s speech-to-speech foundation model was the first commercial speech LLM launched last year.
Hume.AI’s model can interface with external APIs for more complex reasoning and web searches. Cowen emphasized the importance of building safety guardrails into the technology, such as detecting toxic language. Hume.AI collaborates with the Human Initiative, a nonprofit that provides guidelines for ethical AI use.
The company ensures compliance with these guidelines through its terms of use and monitoring of usage.
“Hume builds intelligent voice models. So you can think of it as an LLM that can speak and actually understands what it’s saying. That has a lot of implications for having realistic voices. It knows what emotions somebody might be expressing at a given time,” Cowen said. “If you’re in a game, it knows what a character might be experiencing and how they would express that with their voice. So it just leads to much more realistic kinds of experiences with voice-to-voice AI.”
The goal was to work with Niantic Spatial to bring an AR avatar into the world that was aware of its environment.
“It can speak to you if you’re talking to another intelligent being, and not just an NPC that used to be in a game with preprogrammed lines. And I think that this is going to be a really, really exciting, immersive experience,” Cowen said. “It’s something that is really well suited to our technology.”
“Niantic Spatial plugs into Hume.AI’s API, and that allows them to send queries to their users, voice queries to us, and we send responses,” Cowen said. “And those responses are guided by the prompts that they use, the context they inject in, so we handle all of the intelligence. How would somebody respond to this query, knowing that this building or this site is in their context? We form the vocal response.”
Hume.AI is the first company to release a commercial speech LLM, which debuted in April 2024. The tech can be customized to power interesting characters. In this case, the world of Peridot characters is combined with the tech, resulting in a virtual tour guide.
“We provide the emotional intelligence piece that really is at the core of user experience, of exploring the world with a knowledgeable, empathetic friend who can tell you a story behind this art installation in an airport terminal, or the history of this boardwalk,” Cowen said.
Cowen got a doctorate in psychology and did research at Google. He left to start Hume in 2021 to bring the expertise to the world and build new models.
“One of the areas I’m most excited about is having an immersive experience and having an NPC that you trust,” he said. “You have to earn this trust from the user, which means reflecting that you understand the user’s emotions.”