Using Stems for Customized, Interactive Music Experiences

Interactive audio is a growing trend in fitness, gaming, and social apps, focused on matching music to the movement and actions of a listener. The concept itself has been around for a while — you’ve likely already experienced it in games like Dance, Dance Revolution (DDR) or Guitar Hero (unless you had an older sibling who wouldn’t let you play). When you pluck the plastic string or stomp on the correct arrow, the music rewards you by progressing the song or adding in new instrument layers. In both, you’ve composed a custom song with your actions.

‍

But interactive audio isn’t just about helping interact with and build audio experiences, as is the case in those games. It can mean a fitness app subtly changing the music to keep pace with your movements, or a social or AR app syncing audio to what your body is doing.

‍

Stems are at the core of these customized audio experiences. If you want to make the drums really snap! when the AR fitness app user is punching the sky, you need to be able to work with the drum stems. If you want all the music except the bass to drop out as the hero of a game enters the dark and foreboding cave, you need the bass stems.

‍

That’s where AudioShake comes in. Our patented technology can create standardized audio assets that are ideal for a programming environment — avoiding the prohibitive amount of work that would be required if a game or app wanted to work with thousands of tracks.

‍

Our partners at Artiphon recently turned to AudioShake to help with their latest app, Minibeats. Minibeats is a collaborative music video app that uses AR effects, stems, and responsive visuals to bring videos to life. In a partnership with Rhino Records, they released three interactive Snapchat lenses for the song “Green Onions’’ by Booker T. Jones. Using AudioShake stems, the song responds to a user’s movements and interaction with the animated lens, creating a custom audio-visual experience.

‍

‍

In addition to Minibeats, AudioShake has partnered with a variety of companies heading the charge on interactive audio across gaming, social media, and more, including:

• Reactional is an engine that can bring any music into a game and allow the entire game — visuals, music and sounds — to react live to that music. Stems are used to manipulate these gaming soundscapes based on the characters movements, location, progress, etc.
• Pixelynx has built a music metaverse that let’s fans engage with music in new ways. Their game SoundBlocks is a new take on music-related UGC. Players collect stem files and use them to build music sculptures that can be programmed to generate new mini-games and experiences.