School for Poetic Computation, Fall 2015


03 November, 2015 | Physical Computing

A sound-making glove: how to encode emotion into an object otherwise devoid of personality?


I wanted all of the sound-making to be self-contained to the Arduino (Teensy 3.1). There's a piezo buzzer hiding under the breadboard, and I used the built-in Arduino function tone

tone(pinNumber, frequency, duration);

I saw two options: I could synthesize an utterance by changing the pitch over time, probably a combination of sine functions, or build a table of pitches that were harvested from a recording of a sound. The latter sounded much more emotive.

I used a Chris Wilson's pitch detection for javascript and fed it recordings of my voice. Notice how the data jumps between different harmonics (twice the frequency), that was annoying. The more I made my voice like a pure sine wave the better the capture.

Flex Sensors

The next challenge was to establish contexts for recognizing gestures. The flex sensors were easiest:

the growling won't happen until a second after making a fist. so each boolean should also have a time stamp that saves the moment which the boolean switched to true

int flex1Mark = (flex1Max - flex1Min)*.5 + flex1Min;
int flex2Mark = (flex2Max - flex2Min)*.5 + flex2Min;
if(fist && flex1 > flex1Mark || flex2 > flex2Mark){
	fist = false;
	fistBegin = 0;
if(!fist && flex1 < flex1Mark && flex2 < flex2Mark){
	fist = true;
	fistBegin = millis();
if(!growl && fistBegin && fistBegin + 1000 < millis()){
	growl = true;
	growlBegin = millis();
	boolean growloffset = ((int)(millis()/20.0))%2;
	tone(4,200 - 100*growloffset,10);
		growl = false;
		growlBegin = 0;


the threshold is a value somewhere between bent and unbent (in this case: .5 or halfway), and since each flex sensor is always updating it's values for bent and unbent, always be updating the threshold.

if the hand is making a fist, but now any of the fingers become straightened, the hand is no longer making a fist.

if the hand isn't making a fist, but all the fingers are flexed past the threshold, the hand is now making a fist. Also, record the time that the hand started making a fist, this will become important later.

if the glove is not growling, but the fist has been made for longer than one second, begin growling.

if the glove is growling, but a fist is no longer being made, stop growling.

Abstractly, each gesture goes:

if(gesture 1 is on){
	run action associated with gesture 1
	if(case to turn off gesture 1) { turn it off }
else (if gesture 1 is not on){
	if(case to turn on gesture 1) { turn it on and remember the time that this happened }

Now I'm imagining javascript function pointers, and each "case to turn on gesture" is a pointer, could be to a variable, or a function, contained in each gesture object. code would feel pretty happy then. Save this for the day Arduinos go beyond C.


For a gesture like shaking, we don't need quaternion computed IMU orientation, just simple accelerometer data.

int ACCEL_INTERVAL 100 // milliseconds
static unsigned long dAccelLastMillis = 0;
if(millis() > dAccelLastMillis + ACCEL_INTERVAL){
	dAccelX = (accelX-lastAccelX);
	dAccelY = (accelY-lastAccelY);
	dAccelZ = (accelZ-lastAccelZ);
	lastAccelX = accelX;
	lastAccelY = accelY;
	lastAccelZ = accelZ;

First step is to get the change in the accelerometer between each read period. This takes care of removing the effect of gravity, we'll only be looking at changes to the accelerometer data over time (dx/dt).

float D_ACCEL_MARK 0.25
if(!shaking && !growling){
	if(dAccelX > D_ACCEL_MARK || dAccelX < -D_ACCEL_MARK ||
	   dAccelY > D_ACCEL_MARK || dAccelY < -D_ACCEL_MARK ||
	   dAccelZ > D_ACCEL_MARK || dAccelZ < -D_ACCEL_MARK)
	if(!motions && (motionsLast + motionsDelayTime) < millis()){
		motionsBegin = millis();
		motions = true;
		if(motionsBegin < millis() - 200){
			motions = false;
			motionsLast = millis();
			// play motion sound

The gesture that's tied to the accelerometer is wrapped inside a statement that says only do this gesture if these other gestures (shaking and growling) aren't happening, and it's triggered by an acceleration that's one fourth of a G, one quarter the effect of gravity, or about 2.5 m/s^2.


There are somedays that I feel like my frequency-only piezo-driven sound synthesis is cute in a nostalgic minimalist way (it's actually the worst attempt at sound synthesis ever). Perhaps my favorite take-away was that underneath all this messy code there's the potential for a formal system for managing gesture objects. The trouble is I'm stuck in a simple language running on a micro controller, there's an opportunity to revisit this in another framework and build out this formal system for gesture handling to sensor input.