machine_in_the_middle (2021)
The set up for machine_in_the_middle.On the left Steve Davis is seated, equipped with the EEG headset and EMS electrodes attached to his face. To the right the computer system processes and visualises Davis' brain activity while interpreting his emotions and controlling the EMS device.
How it works:
machine_in_the_middle uses an OpenBCI 3D printed headset with 8 electrodes and an OpenBCI Cyton board to capture and amplify Steve Davis’ electrophysiological (EEG) data and send it in real time to Brainflow open source software and to a Machine Learning algorithm trained on the LUMED emotion recognition dataset (Loughborough University Multimodal Emotion Dataset-2).
The Machine Learning software determines Davis’ approximate emotional state and sends the result to an Arduino and relays which activate Electrical Muscle Stimulation (EMS) pads attached to Davis’ face. The EMS pads contracted specific facial muscles to animate his face into the expression determined by the Machine Learning system.
Emotion recognition:
The emotion recognition system utilised two parameters from the EEG data: Arousal (or how relaxed the subject was) and Valence (how much pleasure the subject was experiencing). Both values are expressed as a fraction or float (0.0 - 1.0).
machine_in_the_middle emotions classification, using a two-dimensional classification of valence and arousal
The classifications of valence and arousal are then used to create a two-dimensional classification of emotion. Each pairing of the valence and arousal value produces a unique emotional category, which ultimately add up to a set of four possible emotions that the system can classify brain activity with. We called these “Stressed”, “Happy”, “Sad”, “Relaxed”.
It might be more appropriate to conceptualise these four emotional states as a new type of emotion, an “emotion-as-number” and to consider it as parallel to the emotional states we experience at a human visceral level. Much in the way that we understand a social media ‘friend’ or ‘like’ as a parallel to actually liking something or knowing someone.
Electrical Muscle Stimulation:
Each of the four emotional states had a corresponding facial expression created by Electrical Muscle Stimulation (EMS) pads.
machine_in_the_middle EMS pads and TENS machines
To animate Steve Davis’ facial expression, machine_in_the_middle uses 3 transcutaneous electrical nerve stimulation (TENS) machines to deliver electrical muscle stimulation to the face via 12 electroconductive sticky pads. The 12 pads vary in size and are strategically placed around the participant’s jaw, chin, and forehead, such that the system has access to a wide enough range of facial muscles to produce stressed, happy, sad, and relaxed facial expressions.
The placement of the pads on and the voltage setting was critical to target each muscle accurately. While Davis was wearing the system, the TENS machines were on but current / stimulation is cut at a relay board which all the pads are connected to. Once the emotion classifier determines an emotion, the classification is sent to the Arduino controlling the relay board, which then opens the circuit between the TENS machine and the pads associated with the desired facial expression, contracting the muscles and animating the face.