Rebecca Fiebrink uses machine learning to invent new musical instruments. She wants to create instruments that anyone can play, tailored to their abilities.

Suppose you want to create a new kind of instrument that you play by waving your hand. A fist means sound a hooter. Waggling your fingers means ring a bell. You need to write a program that can recognise different hand movements and that is really hard. Rebecca wants anyone to be able to do it!

She has written a program that learns. It watches, listens … senses what is happening. It then learns to link what it senses with the sound it should make.

To create your hand-waving instrument, you pick a sound, the hooter, then repeatedly move your hand in the way you want to trigger it: making a fist, say. It gradually links the pattern it is sensing with that sound. Then you select the next sound, the bell, and waggle your fingers – until it has seen enough examples so it can tell it apart from other movements. You do this with each new hand movement. Very quickly you have a working instrument to try.

It has learnt what instrument you want by example. You can try it and, if it doesn’t quite do what you want, change it by showing it new examples. That is different to normal machine learning programs, as there is no right pattern for it to learn when you start. It helps you work out what you want by tinkering. It helps you be creative!

You could turn a whole room into an instrument where every movement does something: create an orchestra as you run around, jumping, twisting and turning. Playing an instrument can just be playing!

---

Fun to do: Design your own instrument.

Draw a table showing which hand shapes link with which sounds (like fist – hooter, waggle fingers – bell, …). Get a friend to act as the program following your table to play a tune as you make the hand movements.

---

Read Issue 1 of A Bit of cs4fn

---