Adaptive Music and Video Games: A Love Story <3

Hey folks! I’m starting this on Friday (a little late, to be honest), but it may not be done till tomorrow. Sorry in advance!

So. Adaptive music. It sounds like something you’d hear at a Borg nightclub. But seriously, what IS adaptive music??
It’s music that adapts.


Well, that was a short blog post. See you next week!…

Kidding, of course. Adaptive music, however, is no joke. It’s something new (“new”) in video games that is changing the way we think about writing video game music AND how we think about a video game environment. Interested yet? Of COURSE you are! If you didn’t have a teensy-weensy soft spot for music in your already-full-of-nostalgic-video-games heart, then you’d have stopped reading this blog a long time ago.

To give you the proper context, we’ve got to go waaaay back in video game music history. Okay, not *way* back, but about 25 years back, when good old Mario came out with its addictive, 8-bitty soundtrack. It made do with a very small amount of music by shaking up what you heard and when you heard it. Going down pipes changed the tune, going back up restarted everyone’s favorite Mario soundtrack from the beginning, and running out of time sped it up. These things lent different feelings to the gameplay and stretched out how long you paid attention to (and enjoyed) the less-than-5-minute total soundtrack. Nifty, right? (This summary has been stolen mercilessly from my fellow blogger at Classical Gaming.)

But times have changed, and for the more powerful, more modern games, new soundtrack options have become available. Right? Absolutely! Chiptune no more, let’s get ourselves some digital audio! Games for the PS and N64 (and maybe Dreamcast? Anyone? Who knows…) supported digital audio recordings and thus made the amount of data spent on soundtracks a lot more significant. Gave a lot more possibilities for exploiting music in a creative way. The games like Final Fantasy used the rich, orchestral sounds of their audio to create an awesome aural background that engulfs the player and gives them a thrilling, intense experience. However, one can’t help but notice the progression: –World Map music plays…….--PSHEEEEEEOOOOOOOOOOOOSHHHHHHHBattle music plays–Dun-dun-dun-dunnnn-dunn-dunn-dunnn-dadunnnn–World Map music plays– etc.

It’s mechanical! The devs said, “This thing is happening and requires this kind of music. Play a sound effect and switch tracks.” The breaks in the experience make it feel somehow scripted, as if you’re not guided from one moment to the next by your own actions, but rather that the game is deciding for you what’s going on, and you’d damn well better enjoy it and grind some levels!

Then came Legend of Zelda: Ocarina of Time. This isn’t the first example of adaptive music, but it’s an enormously popular one and it’s one I’ll use because EVERYBODY and their mother has played it at least once. (This may or may not have something to do with the fact that I haven’t had console gaming in over a year and my first order of business was to do an OoT playthrough when I got back.)

When you’re walking around, you’ve got your general, run-of-the-mill Kakariko Village song (or something). Then, the music dies down… there’s an aura of suspense, of tension. You find a Deku Baba! Attack! Hi-yah! The music changes yet again, and fuels your righteous rage against an unfortunately located garden plant!! Victory! Another brief pause and the soothing music of the village resumes, making your psychopathic hatred of local flora seem like a distant bad dream.

It’s smooth! Seamless, almost. It’s what we call adaptive music, because the music doesn’t merely switch tracks as your position or situation changes, but smoothly alters the audio background to coincide with your present level of tension. If the enemy is unseen, the adaptive music can even offer CUES to the player about what’s lurking nearby. The music ADAPTS to the player’s actions in a much more organic way than Mario or Final Fantasy can boast.

It’s an evolution in the music world, and while it may not seem earth-shattering to you now, think about it in terms of gameplay instead. In the old Megaman games, you had screens with a puzzle or platforming challenge. When you moved to the edge of the screen, gameplay stopped and the screen moved to the next puzzle/challenge/fight. In Megaman X, the transitions disappeared and you got this huge, open-worldy, huge-level feeling that made the gameplay seem more continuous (and as a result, more engaging and complete). Awesome! (Check out Egoraptor’s video on the two games. Worth seeing even though I’ve posted it before!)

Same deal with the music, of course. Instead of your slap-in-the-face you-are-in-a-battle-now notification of Final Fantasy, you’ve got this smooth, eerie transition that makes that huge world of Hyrule seem a lot less compartmentalized, and gives you a more hair-raising clue to when you’re in danger.

But Ocarina of Time is a crude, early example of adaptive music. It basically has two unnamed plumbers sitting behind the screen watching you play Link and saying, “Welp, he’s gettin’ ready to fight the Tektite, better fire up the battle theme and git’er ready to go!” And then, when you engage said Tektite, they just press the play button, with that wee little silent transition in between. We’ll call that the “cross-fading” version. It’s the simplest because it requires two tracks: the track that’s playing normally, and the more intense battle music track. Brilliant! Good job, Nintendo! Doesn’t require a lot of power, doesn’t require a lot of memory, and the only issue is that you’ve got a brief pause between the two tracks, which is acceptable in a LOT of cases, Zelda included.

What could we do if we had, say, MORE power?!? Another awesome example is… Portal 2! Much newer and spiffier. This uses a fabulous little thing I like to call “layering.” Actually… everyone likes to call it layering. It is what it sounds like: you have multiple loops of music that go well together, but have different moods or qualities. Portal 2’s layering is both rare and obvious which is why I like to use it. The minimalistic electronic soundtrack isn’t exactly memorable (except the last turret aria), but when you get on your little jumpy launcher things, the music changes. A more crass, bright, unfiltered electronic sound cuts in as you gain speed, and fades out as you slow down and land. The two layers of sound are going simultaneously and you don’t hear the 2nd until you get up to a certain velocity. Easy peasy to program, but it makes a huge difference in the gameplay. You could also imagine a gritty 3rd-person shooter, where you have different layers of sound for the intensity of the action. It’s easily controllable by proximity and visibility to enemies. Everything in the game produces numbers, and you’ve just got to pick a few of them to drive your music. The freedom and artistic license is enormous! You’ve got your sneaky tremolo strings… and then as you get more intense, you get brass swells that lend a bit of gravity to your situation. Then, as you engage the evil alien enemy (there ALWAYS is one), you get your taiko drums banging away like it’s The Dark Knight. As things get really hairy and you’ve got to break out your miraculous medpacks, you get intense strings doing their fast notes up and down, and you’ve got this cacophonous symphony pounding at you and letting you know: “DUDE, YOU’RE IN TROUBLE NOW, MAN.” Layering.

The issue with that is that you’ve got 4 or 5 layers, and that means 4 or 5 tracks that have to be all going simultaneously and 4 or 5 tracks that have to be perfectly synchronized. Processor heavy, and most games use that power on rendering instead. Gotta make music on a budget. For Zelda, it was really small, and it’s getting forever larger, but you’ve got to pick your battles.

Keep in mind that “layering” doesn’t mean that two tracks have to be going simultaneously out loud, only that they have to be going simultaneously in the system. In Legend of Zelda: Skyward Sword, in the time travel sections, when you’re inside the time bubble, there’s a lively, organic sounding track, and as soon as you get outside of one, the layered track playing simultaneously takes over, more ancient and decrepit-sounding. Only one is playing out loud at a time, but they’re both going alongside each other in the system, meaning the transition between the two is as seamless as your gameplay experience. Feel free to run in and out of time bubbles as you please. The possibilities are huge and you’ll notice them being exploited if you listen and pay attention!

The happy medium is transitions. Sort of. In this sense, you have several loops of music that repeat for the certain moods/situations. These loops have what we’ll call “exit points,” which are breaks in the music where the music can transition from one mood to the next. This means you’ve only got one track at a time (or two if you’re crossfading), and it’s totally smooth and gorgeous. The problem is that you’ve got to have a TON of tracks. Every mood has to be able to transition to EVERY other mood at EVERY exit point, and that requires TONS of little transition tracks that do it smoothly. It’s a composing nightmare, even when you do have enough memory to hold all the tracks.

That’s the current state of adaptive music. We’ve got these tools to compose, we use them to the best of our abilities to fit each situation. We’re on a processor and memory budget, but with those in mind we as game composers can create a seemless, rich, interactive audio environment within which our players can feel absorbed and engaged. It’s all about the experience, and adaptive music helps achieve that.

The question is this: where do we go from here? As our budgets get bigger, do we simply add more layers? Or do we expand our possibilities in other directions? We have huge games like Mass Effect, Fable, and Skyrim where one can be truly good or mostly evil, and these decisions last. In an enormous open world, why not use adaptive music to change what the mood of the game is based on those lasting decisions? It’s feasible. It’s a large task but it would make those types of games feel as if your decisions are more than just points in Renegade or Paragon tabs, but as if they have an impact on the world (and your experience). Why not generate the music THROUGH the game? Instead of playing mp3s, why not set up a framework and let the game program build its own soundtrack with the composer’s guidelines? The player would never get bored with the music because they’d never hear the same thing twice. There are so many possibilities for the future of adaptive and interactive music. I sometimes wonder what they’ll be, but more often I’m filled with the desire to help shape that future myself.

Weirdly personal. Go play some games! Listen for their adaptive music. Write me with what you find! Sorry for the monster post, it’s worth it and when I learned about this stuff, it blew my frickin’ mind. Hope it blew yours as well! Till Monday 🙂


P.S. Done at 11:20pm GMT -8:00. It’s still Friday for me! Huzzah!


About Isaac Smith

I write about music, technology, video games, and probably many other subjects that don't bear mentioning here. Either way, most of it's worth reading, and you may even enjoy yourself!

Posted on August 9, 2013, in Miscellaneous and tagged , , , , , , , , , , , , , , , , , . Bookmark the permalink. Leave a comment.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: