Musical Amoebae

Electronic music instrument Reactable turns images to sounds. The experimental synthesizer was developed by a research team of the Pompeu Fabra University of Barcelona (Spain).

The Pompeu Fabra University was established 1990 by the regional government of the Spanish province of Catalonia with a vocation for excellence, renewal and the modernisation of the university system of Catalonia. The university focuses on social sciences, biomedical sciences as well as media and communications. Today, the campus of the Pompeu Fabra university counts around 10,000 students.

The Music Technology Group (MTG) of the Universitat Pompeu Fabra of Barcelona, part of its Departament of Information and Communication Technologies and of its Audiovisual Institute, is specialized in sound and music computing. With more than 40 researchers coming from different and complementary disciplines, the MTG carries out research in topics such as sound processing and synthesis; music content analysis, description and transformation; interactive music systems; or computational models of perceptual and music cognition.

The Reactable: a Table as a Music Instrument

Digital image processing opens up unlimited possibilities to artists, as demonstrated by the experimental musical instrument Reactable, developed by researchers from the University of Barcelona using cameras from Allied Vision Technologies.

The Reactable looks like an illuminated, round table with a diameter of about 1 m (3 ft.) and a glass top. To operate the Reactable, various Plexiglas objects are placed on the table, related to each other and moved. These objects fulfill different functions based upon their geometric shape. For example, square shaped elements generate basic tones, while round objects act as sound filters, which modulate these basic tones. The symbol on the selected elements determines the type of the basic tone and/or the filter; the spatial relationship of the objects to each other determines the extent to which one element affects another. A special collection of symbols was conceived for the Reactable which meet the system requirements for easy and fast recognition. Due to the fact that they look like one celled organisms, the symbols were given the nickname "amoebae" by the researchers.

The Reactable projects markings onto the surface of the table to make the instrument easier to operate. These not only confirm to the musician that the object has been recognized by the system, but also provide additional information regarding the status of the generated tone and its interaction with neighboring objects. This allows the artist to see the connections and a dynamic graphic presentation of the generated sound waves on the table. The kicker: The musician can change individual sound parameters by touching the projected information with his finger.

Digital Image Processing Turns Images into Sounds

The Reactable has ambitious image processing to thank for the fact that it is so easy and fun to use: An Allied Vision Technologies digital camera monitors everything that takes place on the table from underneath the glass plate. The specially developed image processing system ReacTIVision analyzes the images and derives corresponding sound information from the position of the objects. This is then transferred to the speakers as an audio signal and graphically projected onto the tabletop.

The Requirements

Alternative technologies would have been available for this project for determining the position of the objects on the surface of the table – such as ultrasound or RFID. Digital image processing prevailed, because this technology offers significant advantages, says Sergi Jordà, director of the project at the Pompeu Fabra University of Barcelona: “With a camera and a constant bandwidth, we can monitor the entire surface, regardless of how many objects are on the table. In addition, image processing determines not only the position of the objects but also their orientation (rotation) and can recognize human fingers on the tabletop. With this, the optical solution offers many more fine tuning possibilities and nuances than alternative technologies and places almost no limits on musical creativity."

Fast and Reliable: FireWire Digitalkameras from Allied Vision Technologies

What was critical for implementation, however, were the image transfer and processing speeds, as well as the sensitivity of the camera for the environmental lighting conditions. The instrument must react very quickly to the commands of the musician, especially if it is meant to play in synchronicity with a band. “With the Guppy F-046B digital FireWire cameras by Allied Vision Technologies, we have a high data rate with more than 60 frames per second available to us, which makes the short reaction time of the instrument possible”, says Jordà. The stability and reliability of the system is also extremely important: A breakdown during a concert by an international star like Björk is out of the question. “The AVT cameras are also impressive in terms of lifetime and stability", say the researchers. In order to lessen the sensitivity to environmental light, the AVT cameras work in the near infrared spectrum (NIR). A special LED light is used in conjunction with a filter on the camera for this.

Practical Implementation: Hard Test on Björk World Tour

The Reactable is not yet being mass produced, but the actual prototypes have proven that the system can also function reliably in a live concert. As the first famous customer, popstar Björk implements a Reactable on the world tour of her new album “Volta”. In this way not only does a wider audience have the chance to get to know the Reactable; the technical reliability of the system is also going to be subjected to a hard test. The Volta Tour 2007 with the Reactable will have concerts in Amsterdam, Barcelona, Baton Rouge, Brussels, Bucharest, Lisbon, Madrid, Milan, New York and Rome, among others. The goal of the inventors is to use the public attention to market the innovative instrument and begin to mass produce it.