The Subtle Body Project is an interactive multimedia biofeedback system that uses the human body and brain to generate data and to control the media elements of the project installation. This project seeks to integrate the convergence of the disparate disciplines of interactive media, physical computing, biofeedback research, sound design, video mixing, stagecraft and theatrics in the attempt to create a profound and artful expression of the subtle sensitivity of the human body to create a multi-sensory and multi-modal interactive installation and performance environment using the physical body as the content generating device. The arc of the experience for the user will be directly controlled by their brainwave activity and their skin conductivity. To achieve this aim, an array of highly sensitive sensors will be attached to the user’s body to collect data from the brain, the skin and muscle movement of the user. There will be an additional array of sensors used to monitor the movement and location of the user as they participate in the experience.

     The Subtle Body Project has undergone several major revisions since it’s initial proposal in June of 2008. Although the focus of the project remains that of the human body and its interaction with sensors and MIDI, it has grown substantially in scope. The Project is now employing from 7 different data streams from 10 biosensors attached to the body of the user. These sensors are collecting a diverse range of data including Brainwave (EEG), Eye motion (EOG) and Muscle (EMG), Galvanic Skin Response (GSR), Body Motion and Proximity (PNG). This sensor data is captured through several microcontrollers, both manufactured devices and devices built by the Subtle Body team.  The sensor data is then piped into Max/MSP, which serves as our central data processing hub. The data is heavily processed using sequencers, scaling devices and metronomes and routed out to audio to the audio application Ableton Live which is in turn hosting multiple instances of VST synthesizers such as Atmosphere, Predator, Absynth, and Reason among others for audio processing. The same data channels generated by the biosensor array are also sent out to the video management application Jitter, also from Cycling 74 (the makers of MAX/MSP), for the visual component of the experience.  The video elements interact with the user in a variety of ways depending on the phase of the experience. The project is using the filtering, spatial mapping, video mixing and playback features found in Jitter.

Video assets have been collected and created by the Subtle Body team, processed and edited in Final Cut Pro and After Effects and then exported to Jitter for interactive playback on the experience. There are over 80 video cues used in the project.

     The result of this effort is a three phase experience in which the user experiences the raw data of their bodies electromagnetic energy, a smoothed artistic interpretation of this energy data and finally an elaborate third section in which the user is controlling the playback of audio and video clips with energy gathered from their brain, their body and the blinking of their eyes in a real time audio/visual remix mash up.

The final version of the Subtle Body project is a more streamlined and elegant design that reflects the research and development conducted on the project in the past year. The project remains an interactive biofeedback process in which the user interacts with the electromagnetic energy of their body and brain to generate and engage with audio and visual elements. The following is an in depth explanation of the technical and aesthetic solutions employed in the project. This includes hardware and software, code, audio processing and sound design, creation and management of visual assets and the design of the final presentation.     

Technical Components

Sensors Overview

     The Subtle Body Project is using several sensors to collect data from the users physical and mental state.  There are 8 individual sensors or electrodes being attached to the user with an additional microprocessor/sensor system for motion and a final potentiometer for the user to select their choice of phase for the experience.

     EEG brainwave sensor data will be captured by three head mounted sensors and collected via the medical grade hardware device known as the Atlantis 2. This data is then processed by the Brain Master software, which separates each brainwave into data streams of amplitude and frequency. This sensor data is sent from a Dell PC via a flash server to our main computer hub, a Macintosh Power Mac G5 where it is converted to discreet data channels via Max/MSP.

     Additional EEG and EOG sensor data is collected via the Bio-Emo array of sensors produced by the ICubeX company from Toronto, Canada.  The sensors are housed in a headband with the sensors attached to the forehead of the User. This data is collected through a small microcontroller called the Wi-Micro Dig that converts raw numeric data into MIDI information. 

     The GSR sensor data is collected through another ICubeX sensor array called the BioEmo, which consists of 2 small fingertip sensors and a small micro processor housed in a small wristband. This data is also sent out to the Micro Dig microprocessor. The Micro Dig data is then sent to our Mac via USB where it is received by the ICubeX interface and split into discreet MIDI channels and then sent to MAX/MSP.

     The PNG motion sensor is a device constructed by Mohammed Mohanna using a small Arduino microprocessor and a basic breadboard. The data from this sensor is sent to our main computer via USB and is received and processed by MAX/MSP.

Sensor Technical Specifications

The Brainmaster EEG sensor array

     The Brainmaster EEG sensor array consists of a set of five 9 mm gold cup electrodes with built-in 48-inch lead wires. These provide two channels of EEG data and a ground connection. These terminate in an extension cable providing a connection from User to the BrainMaster 2E module. This configuration is made with 5 DIN touch proof leads, with shielded cable. The shield is connected to “ground” connection at the input of the 2E module. This configuration provides clean signals over extended distances; with less concern and artifact due to electrode lead wires.

     These signals are then inputted into the Brainmaster ATLANTIS II hardware-processing device.  The Atlantis II processes 2 input and 2 output signals, with two channels of EEG and two channels of AUX signals for additional biofeedback. It features include continuous real-time impedance monitoring, total immersion with photic, vibrotactile and auditory feedback. It is bus powered via USB and sends data to our computers via USB 2.0. It should be noted that this is primarily a medical grade device that we have acquired from the Brainmaster company for use in this project. We do not claim to have medical knowledge of this product or of the human brain. Nor is this project meant to be clinical in any way. It is a purely an art installation.

The Subtle Body Project is using a variety of sensor arrays from the I-Cube X company to gather data from the body of the user. This data comes in the form of EEG, GSR and EOG sensors. The EEG captured by the BioWave is an AC signal that is generated by brain activity and is captured by sensors on the scalp and the forehead. The burst of 10Hz alpha wave activity is generated by a few seconds of eye closure. Alpha usually appears in the BioWave EEG as a sinusoidal burst with a center frequency at about 10 Hz, and at much greater amplitude than the continuous background beta wave activity. Alpha is one of the brain wave frequencies that can be generated intentionally and is therefore useful for interactive biocontroller applications.

     The EOG is a DC signal that is generated by the eye and changes polarity with the direction of eye motion. The BioWave makes EOG available for biocontroller MIDI applications such as panning sound or moving video objects across a screen. The EMG is an AC signal that is generated as a result of muscle contraction generated by the clenching of the temporalis muscles in the jaw of the human. This produces a large signal from the forehead sensors of the BioWave.

BIO WAVE Sensor by I-CubeX

     The BioWave headband is a sophisticated bioelectric signal acquisition device manufactured by the I-CubeX company. It is designed to capture three types of data: Brainwave (EEG), Eye motion (EOG) and Muscle Movement (EMG). These three types of signal output will be described in more detail in subsequent sections. The BioWave sensors are housed in a headband made of elastic materials and the sensors are housed in a polyurethane foam strip. This foam section also houses a small micro controller that and provides initial processing including signal amplification. The sensor module has 3 gold sensor contacts that are designed to capture EEG, EOG, and EMG and each of these signals requires slightly different use of the headband. The band includes a 1meter cable with a connector that plugs into the Wi-MicroDig, ICUBEX’s processing unit. This allows freedom of motion in the User while they are wearing the Bio Wave.

BIO-EMO Sensor by I-CubeX

     The electromagnetic skin response from the User is collected via the biosensor know as the Bio-Emo. It is also manufactured by the I-CubeX company.

     This GSR sensor configuration measures skin resistance and skin impedance (across 2 finger copper electrodes), which relates to the level of arousal and/or emotional excitation in the user. This is the same measurement method used for the famed lie detector test. This sensor’s construction is similar to that of the BioWave sensor in that it contains a small polyurethane foam strip, and in this instance, two copper sensors extending from the foam strip. This device also contains a small micro controller housed in the wristband that provides signal processing including a notch filter to reduce power grid noise (50 or 60 Hz) and signal amplification.

     The two above mentioned ICUBEX sensors are processed further through the use of the MIcroDig hardware device.

Wi-MicroDig Microcontroller

     The MicroDig is an easily configurable thumb-sized hardware device that encodes up to 8 analog sensor signals to multimedia industry compatible MIDI messages with high resolution and transmits these messages to a computer in real-time for analysis and/or control purposes. The interface has 8 inputs of each 10 bits resolution (1024 steps of each 4.9 mV) that can sample at up to 1500 Hz with milliseconds latency to capture very subtle gestures and movements.

Updateable firmware from ICUBEX enables the MicroDig to operate in both standalone mode (sensor data is processed before it is transmitted) as well as host mode (raw sensor data is transmitted). Standalone mode includes various sensor processing and mapping features such as gesture recognition that can be conveniently configured using one of the ICUBEX editors. The MicroDig encodes analog (0-5V) sensor signals to multimedia industry compliant MIDI messages, and transmits them using a physical MIDI interface.

PNG Motion Sensor

     The Parallax PING ultrasonic distance sensor provides precise measurements of distance using an echolocation algorithm. The sensor transmits an ultrasonic burst and provides an output pulse that corresponds to the time required for the burst echo to return to the sensor. By measuring the echo pulse width, the distance to the target can be calculated. The range of the sensor is approximately 2 cm (0.8 inches) to 3 meters (3.3 yards). It connects easily to the Arduino microcontrollers requiring only one I/O pin. The PING sensor has a male 3-pin header used to supply ground, power (+5 VDC) and signal. We have plugged the header directly into the breadboard to power the unit.

Arduino Microcontroller

     The Arduino is a small Microcontroller made in Italy. It operates in an open source environment and is ideal for controlling sensors.  Data is transferred via a USB connection and the device is bus powered. The microcontroller houses 6 analog inputs and 13 digital inputs. We are using digital inputs for our PNG motion sensor connection. We built a simple circuit on a breadboard to connect the Arduino and the motion sensor. The Microcontroller Runs at baud rate 9600 for connection to USB.

Data Processing

     All of the sensor data is then processed in MAX/MSP for our audio and visual components. In Max/MSP, we have created several data filters and step sequencers to control and arrange the sensor data. For instance we have created a series of logic algorithms for the GSR sensor array in which each numeric change in the electromagnetic activity of the User launches a new sequencer that plays a range of notes from 1 to 5. These notes are harmonically related to each other and to all the other sonified sensor data in the experience. This technique is used through out the Max patch to continually smooth the incoming data. This serves two functions simultaneously; to control the floating streams of data that are coming in from the sensors and also to take that pruned data and turn it into audio sequences and control variables for video manipulation.

The Brain Master Atlantis 2 System

     The Brain Master Atlantis 2 is a clinical biofeedback system used for the improvement of the mental functioning and for the monitoring of brain waves. What makes the Atlantis 2 system different than other EEG devices is its portability that gives the user the opportunity to use it anywhere. Also, unlike other EEG devices, nonprofessional home users as well as medical professionals can use it. 

Brainmaster System (continued)

Hardware: The Atlantis 2 hardware provides two channels of EEG and two AUX signals for additional biofeedback modalities. The hardware does not need any batteries or power cords, it powered from USB connectivity.

Software: The Brainmaster 2.5SE is the main software for the system, used for monitoring brain wave activities from the EEG device .The software can display EEG data in many forms depending on what type of graphical display the user or technician needs. The system can display the EEG data in any combination of numbers, graphs, or charts. There are two main components built inside the 2.5SE software that we have used for our project to improve brain wave signals and to establish communication with other software applications that we are using in our project. The first component is the Event Wizard. Based on the real time processing provided by the Brain Master 2.5 SE software, this tool allows us to design control variables and feedback with a variety of options. For our project, we have built a function in the Event Wizard to display the amplitude of the brain waves instead of using the default settings of the Brain Master 2.5 SE that analyzes only frequencies of brainwaves. As these frequencies range from 4 to 10 Hz, they are in a range that is nearly unusable for the project. The second component is using the Brain Master flash player. In the flash player, we have built a flash file that sends the EEG data to from the Brainmaster software, on a Dell PC to Max/MSP, which is on another Apple computer.

The I-Cube Editor Software

     We are using EOG muscle motion sensors, GSR sensors and a potentiometer that are made by the I- Cube X company for our project. We achieve connectivity with from these sensors to the computer using the Micro Dig microcontroller and the I- Cube X editor software. The I-Cube X editor software receives sensor data from the Micro Dig as MIDI information. Through the editor software, we can control the behaviors of each sensor by setting up thresholds or by changing the sampling rate. Also, the editor software is responsible for sending the sensor data to Max/MSP.

Figure 13 : I-Cube Editor Software

The Arduino Code

     We have used the Arduino microcontroller to connect the parallax PNG ultrasonic sensor directly to Max/MSP using a simple circuit. The following is the source code that we have embedded in the Arduino microcontroller to have the motion sensor function in the way needed. The code is supported by comments that explain the functions.

int pingPin = 7;

void setup()

{

  Serial.begin(9600);

}

void loop()

{

  long duration, inches, cm;

  // The PING))) is triggered by a HIGH pulse of 2 or more microseconds.

  // We give a short LOW pulse beforehand to ensure a clean HIGH pulse.

  pinMode(pingPin, OUTPUT);

  digitalWrite(pingPin, LOW);

  delayMicroseconds(2);

  digitalWrite(pingPin, HIGH);

  delayMicroseconds(5);

  digitalWrite(pingPin, LOW);

  // The same pin is used to read the signal from the PING))): a HIGH

  // pulse whose duration is the time (in microseconds) from the sending

  // of the ping to the reception of its echo off of an object.

  pinMode(pingPin, INPUT);

  duration = pulseIn(pingPin, HIGH);

  // convert the time into a distance

  inches = microsecondsToInches(duration);

  cm = microsecondsToCentimeters(duration);

  Serial.print(inches);

// Serial.print(“in, “);

  //Serial.print(cm);

  //Serial.print(“cm”);

  Serial.println();

  delay(100);

}

long microsecondsToInches(long microseconds)

{

  // According to Parallax’s datasheet for the PING))), there are

  // 73.746 microseconds per inch (i.e. sound travels at 1130 feet per

  // second).  This gives the distance travelled by the ping, outbound

  // and return, so we divide by 2 to get the distance of the obstacle.

  // See: http://www.parallax.com/dl/docs/prod/acc/28015-PING-v1.3.pdf

  return microseconds / 74 / 2;

}

long microsecondsToCentimeters(long microseconds)

{

  // The speed of sound is 340 m/s or 29 microseconds per centimeter.

  // The ping travels out and back, so to find the distance of the

  // object we take half of the distance travelled.

  return microseconds / 29 / 2;

}

Communication between Actionscript and MAX/MSP

(Sending EEG data to MAX/MSP)

     This Actionscript 2 file has been created to receive EEG data by Max/MSP on the Mac from the Brainmaster software on the Dell PC. This code is a modification of the fla code used to run the Flash player embedded in the Brain Master software which used to run flash games based on EEG data. The fla file that we have created is stored in the Brain Master flash player. The main function of this fla file is to send EEG data to Max/MSP and Jitter using XML socket functions built in the Actionscript file. These functions are used to establish communication between Max/MS/Jitter and Actionscript.   Max/MSP/Jitter then receives the EEG data through a Flashserver external. The externals can be found at http://www.nullmedium.de/dev/flashserver/