Carnegie Mellon researchers use Twitch to collect sounds for AI research

Carnegie Mellon researchers have designed a livestreaming video game that collects audio from players to populate a database for AI research. The team’s game — Rolling Rhapsody — is specifically designed to be played on Twitch, and it tasks streamers with rolling a ball across a map to collect “treasure” while viewers record sound from their homes via an app.

It’s researchers’ belief that recordings of domestic sounds like thudding from a bedroom door or a coughing fit could be used to create a range of useful technologies. For instance, Google drew on audio from thousands of its own meetings and YouTube videos to train the noise-canceling algorithm in Google Meet. Meanwhile, a team of Carnegie Mellon researchers created a “sound-action-vision” corpus to anticipate where objects will move when subjected to physical force.