Generating Audio UI for Android

By Peter Drescher
February 24, 2011 | Comments: 4

One futuristic aspect of phones and tablets is how much they resemble the Personal Access Data Devices (PADD) used by the crew on Star Trek. A touch sensitive screen, wirelessly connected to the computer network, displaying a wide range of information, I love it when sci-fi invents reality ...

But the thing I liked most about them is how they sounded! Everything I know about Audio UI, I learned from Star Trek, and their devices produced quite wonderful soundtracks. They perfectly reflected the user's experience, and were clearly audible even in high noise environments (like Engineering).

Of course, they had the advantage of being fictional, but wouldn't it be great if our devices sounded like that? I've been talking about "using game audio techniques to generate system sounds" for three years now, but without much luck getting it implemented. I tried selling the idea to Danger; they were too busy. I tried selling it to Microsoft; no dice (too innovative?) I even tried selling it to Google; wow, they were REALLY uninterested.

Because it's hard for others to hear what's only in my head (or as I like to say, "talking about music is like smelling a painting"), I wrote notepadd as a proof of concept. It's a sonified version of the example.android.notepad code, and generates audio for app in/out, note left/right, and menu events.

Each UI sound is made by playing a random sequence of very short .ogg files, each varied by pitch and volume. The result is a series of "unique, but sonically related" sounds, played in response to user interaction. This not only produces a lively soundtrack, it eliminates the annoying effect of "playing the same damn sound over and over and over again, every time you hit that button!"

I'd patent the idea, but it's nothing new. In case you weren't paying attention, this is exactly how game audio has been produced for the last 20 years! Nobody plays the same gunfire sample over and over anymore. Instead, gunshots are generated by mixing together a bunch of audio files according to an algorithm. That's all I'm doing with notepadd ...

Of course, the iPhone deals with the problem of repetitive UI sounds the old-school way: silence. And so now, by convention, most cell phones run silent too, except for those annoying ringtones. That's a shame, because in the movies, computers and mobile devices always make UI sounds, and they always sound great!

I just want reality to sound a bit more like science fiction ...

SO if you've got a Nexus (or other open android phone), download the app, install it, play around with it, and let me know what you think in the comments. It also supports multiple audio themes, including a fun Cybertron setting ...

- pdx

p.s. if you're interested, the Eclipse project is available on my Twittering Machine code page.

Update April 2012: Now Available On Google Play!

You might also be interested in:

4 Comments

Thanks for the sharing. I'll try download it first and hope i can make a comment about it.

Regards,
ovgadget

Very nice. I don't have an Android device, so I cannot play around with it.
However, I extracted the sounds, to 'smell the paint' a bit.
What about these .ogg files? Are they copyrighted?

three sets of .ogg files (the "native" audio format for Android) are included in code, one for each theme:
a) Next Generation: Star Trek sounds have been floating around on the web for years, and IMHO, represent some of the best UI sounds ever. The bleeps used here are from the LCARS site ...
b) Cybertron: One of my "hobbies" is collecting movie lines and iconic sound effects from DVDs (makes a great ringtone!) These snippets were (I think!) mostly pulled off the first Transformer movie ...
c) The Art of Noise: So many sound effects are simply shaped noise, and I believe this style of Audio UI tends to be less attention grabbing (and thus less annoying) in the long run. This sound set is edited from a rather extensive library of noises I have recorded over the years ...

Remember: this is only a "proof of concept" app, intended to demonstrate the Generated Audio UI concept. If I were to actually implement the idea into a real product, I would of course record and produce original sounds, depending on the design direction (for a detailed description of that process, see my first Audio UI article "Singing With Your Thumbs")

But I've always felt there was an untapped market opportunity there: Licensed Audio UI themes. Make Your iPad Sound Like A Star Trek Computer or Iron Man's Suit! Systems Alerts by Majel Barrett, or Homer Simpson! Of course, there are many apps that will produce licensed sounds (not to mention fart and barnyard animal noises), but I'm talking about audio played during, and in response to, user interaction with the operating system.

Final Note: My response to your comment was delayed because I was at the Game Developer's Conference in San Francisco last week, where I showed off my little app to some of my game audio colleagues. It's no surprise they all understood immediately what I was doing (mixing multiple samples to vary playback) and why, because they've all been doing the exact same thing for years! All the middleware game audio engines (Wwise, FMOD, et al) do this kind of thing as a basic function. It's called a "container" or "bucket"; you load in a bunch of audio files, then play them according to some algorithm, modified by gameplay parameters.

I'm just trying to use those techniques to make operating systems sound more like the ones in the movies ...

Hi Peter. It's a pity I don't own an Android device yet, but I'll sure hear your demos when I do. But here's a story: A couple of years ago, I was doing research for my master's degree dissertation, and by then I thought that doing audio UI was something only video games have been doing through all these years.

But guess what? I found out there is a huge research field called "auditory displays" which have been investigating the relation between audio, GUIs and user experience for as long as two decades! These guys have made experiments and written solid theory on the matter and no one knows about it, not even game audio folks.

Since you are interested by the subject, you shold head over to http://www.icad.org/knowledgebase and take a look at some of the stuff that's being researched. You won't regret!

Like you, I think that sonification could add another dimension (no star trek pun intended) to software. I hope the big guys open their eyes (ears?) for this some time...

News Topics

Recommended for You

Got a Question?