Hello and welcome.

A couple of weeks ago I participated in the IEEE Present Around The World Competition at the University of Essex and I came second. I’m terrible at answering questions and this was particularly difficult because what I presented was (and is) still in the stage of a project idea, with very little background research behind it. But I am quite proud of the presentation - I plan to record it to show it off in its whole glory, but for now I’ll offer the slides and my notes for it here. What this form of sharing will miss is the main part of the presentation, the sound effects and music. So look out for the updated recorded version.

A quick overview of the project, I plan to start working on this end of June, hopefully in partnership with more knowledgable people in the areas of interest, particularly sounds and the Deaf community. To get the main idea, I’ll let you look at and read the following notes.

This next part I signed in American Sign Language. Good thing about doing an introduction in ASL, nobody notices if you completely mess up because of the nerves and by the end of it you’re all good to speak for the rest of the presentation!

Many think that the way we experience the world defines us. And sounds, music, play a big part in that. When you’re driving your car, turning on the radio and that one song - one song - sticks into your mind for the rest of the day. You’re doing work later, sitting at your desk, scrolling through endless documents and that song comes back in your mind. song playing But you have to go out and walk the dog and song playing. And then you’re back home, cooking dinner for the kids, but that song… playing song. Some people don’t get that.

• quick comment: the song chosen for the PAWT presentation was along the lines of “I like to move it move it”… -

A bunch of us like to play games. How many of you have actually done that - playing games? Hop scotch, tic tac toe, that crazy game where you form lines and you gotta run max speed into the other line and break your teeth - maybe that’s just me. Imagine how being in the court yard as a kid would be, unable to hear the others shouting that a ball is about to hit you from behind. Unable to hear the mean kids sneaking up on you.

Then there’s those of us that like to play video games. And these are becoming more and more incredible, with Virtual Reality bringing you into different amazing worlds. But even without that, you can get so immersed into one game, because of its story, or the fun things you get to do, or maybe just because it looks so much like your life, your story, that you can’t help feeling part of the game. Modern games, they rely a lot on music and sound effects for immersion. They build these incredible soundtracks that lift your spirit or terrify you, designed to evoke emotions and challenge you to pour your soul into the game. Some people don’t get that.

But it’s not about a defect. No, I’ve learned that the Deaf don’t think of themselves as less and we shouldn’t either. They have a rich beautiful culture and they are proud. They reinvent the world and make it work for them, instead of against them. Movies, theatre, songs… they still enjoy that, in such an expressive manner it can move you deeply.

It’s about being a community. And often, it is games that brings us together. So why do we limit their access to this world, why not bring them in instead?

There are efforts in the games industry for Deaf accessibility. Subtitles are used in some games for character dialogue, comic book style expressive words or screen flashes and colours in time with the beat of the music in others. But in the majority of games, the game experience is so much more than that.

I’ve discovered recently that the facilities offered for the blind are much greater in this regard. It’s hard to portray pictures in sounds, but there can be games of sound only. And there is a large collection of such very specific games, ranging from shooters to adventure and even puzzle games.

The Blind are actually very good at understanding sounds in video games and navigating by them. One example is the gamer Sightless Kombat, who climbed the competitive ladder to the top in the game “Killer Instinct” - a fighting game. He is able to identify gameplay by sounds only.

The project I am going to kickstart uses the larger library of games meant for the blind to understand sounds and translate them into visual cues and emotions for the Deaf.

Work in AI focused a lot on screen capture these days. Google DeepMind is everywhere and their work IS fascinating, how their algorithms learn to respond to pixels.

But having an agent learn and react to audio stimuli, that is more of a niche subject. In order to provide an interface for the Deaf regarding the dimension they miss, we first need to understand this dimension.

So let’s start with isolating it. We reduce the games to only sounds. The sound of footsteps as you move around. A narrator telling their story. A clock ticking in the background. Shouts from the left. A whoosh of a sword. A pulse from the right. Can an agent understand they are being attacked and they need to hurry to the exit at their right?

Then can we translate this understanding to visual information? Something as basic as an animated character in a corner reacting to the environment as you play the game. The ultimate goal is to create a tool that can be easily applied to any game, without the need to specifically tailor games for it. This would make all games accessible for the deaf community and not make game designers crinkle their nose at the thought of extra work in placing audio cues in key spots.

My area is game-playing Artificial Intelligence, but I know nothing of sounds and little of accessibility in games. So I’ve asked for help from two students in the same programme who are much more experienced in these areas than I am, hopefully turning this into a great collaboration. A community effort for another community.

This next part is something I didn’t do in PAWT, fearing the time limit. But the plan was to ask the audience to close their eyes, listen to the sounds I played and raise their left hand if the sound they heard they thought of as bad (endangering, annoying, would like to run away from etc.), raise the right hand if the sound seemed good or clap their hands if they identified the exit of the level in our little escape game we’d be playing …

… And of course the clapping would lead perfectly right to this!