Kinect + Max / MSP

In this tutorial, you will learn to make uncoordinated and awkward movements like this…

and this…

Here is a brief tutorial on using the Kinect sensor with Max / MSP.  I’ve only tested this on a Macintosh Macbook Pro 2010 version and using Max/MSP 5 with Jitter, but it should work just fine on Windows as well.  The basic setup involves using “Synapse for Kinect” to stream hand/body/foot positions from an external program to Max/MSP over UDP.  UDP is an internet protocol that doesn’t require acknowledgements, such as TCP.  Once in Max/MSP, the UDP packets are parsed by some javascript functions that I wrote and normalized to the normal range of MIDI values (0-127).  In the Max / MSP patch that I supply, I chose to use the hand/foot detection that is based on distance the hands/feet are away from the body.  You can modify the patch to allow for detection based on raw location on the screen and other methods, by reading the documentation on the Synapse site. However, if you do this, you will probably have to modify my javascript functions to normalize the values to the MIDI range.  For my Kinect to MIDI transformation, the hand position to MIDI number translations are as follows:

  1. Hands straight forward (toward the sensor) register a value of 127. Vice versa, hands pointed away from the camera (behind you), have a value of 0.
  2. Hands directly to the side of the body, away from the body, are 127, while hands at the center of the chest are 0.
  3. Hands up high are at 127, hands at the ground, are at 0.

Basic Steps:

  1. Install Synapse for Kinect.  (Note: I had also installed libfreekinect, but I do not think this is required.  Let me know if just installing Synapse for Kinect does not work!)
  2. Download my Max/MSP patch. kinect-to-ableton sends MIDI control signals to other programs, while jit-synapse-kinect uses the kinect to control simple oscillators in Max/MSP. You will also find several custom javascript Max objects.  Just double click on them to open them up and see what they are doing.  Most of them are just normalizing the numbers from Synapse to numbers in the MIDI range.
  3. Run Ableton Live or similar program that can accept MIDI input to modify oscillators, tune parameters, etc.  With Ableton live you can use the MIDI map mode to specify which parameters are affected by which MIDI messages.

Let me know if you have any questions by contacting me.

UPDATES:

6/3/2012 – Updated the kinect-to-midi.zip file to include the kinectNormalize.js file that was originally missing.