frey

Sounds Like Light, Lights Like Sound

Technical Details

Download this text as a pdf document.

Sounds Like Light, Lights Like Sound uses computer vision technology, open source software including PureData and OpenCV, and hand-built LED lighting controlled through Wiring/Arduino boards. Source code for everything is available on request.

Video capture and analysis

A wide-angle lens CCTV camera and an infrared lightsource are mounted above a space, pointed downwards. They feed an image, even in pitch black conditions, to a video capture card plugged into an old Pentium4 2.4GHz laptop. Home-made computer vision software built using a collection of open source libraries including OpenCV, WOscLib, pthreads, inilib processes the image to find positions of people moving through the room. Person-motion data is transmitted to listening applications using OpenSoundControl packets.

Sound generation

A complex set of PureData patches processes the incoming data to determine what sounds to make. The system has a number of states it moves through as a kind of linear narrative. The current state’s audio processing is modulated by the position of the person in the space in either Cartesian (X-Y) or polar (r-theta) coordinates, or both, with additional modulation available in the size of the person (in effect, how outstretched their arms are).

Light generation

Internal details of the sound generation system are reprocessed through another complex set of PureData patches to create control data for five 24-bit RGB-controllable LED-based light sources installed around the room. The control data is sent to a Wiring or Arduino board, which subsequently operates lower-level PWM control for each of the lights.

Go back to the main Sounds Like Light, Lights Like Sound project page.