Emotion Recognition - Benjamin Jeffries

Project: Emotion Recognition
Industry: Interaction Design / Design Research / Human-Computer Interaction / Speculative Design
Institution: Goldsmiths, University of London
Services: Design research

Interaction design

Human-computer interaction

Motion graphics

Data visualisation

Video production
Tutor: Sarah Pennington
Designer: Benjamin Jeffries

An investigation into how emotion recognition technologies shape digital communication and the limitations of teaching machines to interpret human emotion.

Context

Emotion Recognition was an independent research project completed as part of the MA Design: Expanded Practice at Goldsmiths, University of London.

Developed during the COVID-19 pandemic, the project explored how digital communication platforms mediate emotional interaction. As video conferencing became central to work, education, and social life, the project questioned whether emotion recognition technologies could bridge the growing disconnect between people communicating through screens.

Rather than evaluating the technology itself, the project examined its assumptions, biases, and ability to represent the complexity of human emotion.

A historical criminal-anthropology plate of composite portraits labelled by number of components, captioned ‘prevalent types of features among men convicted of larceny’ - an early attempt to read character from the face.

Challenge

Video conferencing reduced many of the non-verbal cues that shape face-to-face communication, often leaving interactions feeling detached and emotionally limited.

The challenge was exploring whether emotion recognition systems could restore emotional presence while exposing the ethical, cultural, and technical limitations of reducing human emotion to measurable data.

Our Approach

Exploring Human-Computer Interaction

The project began with a series of mediated interaction experiments, including conversations between people and AI assistants, pre-recorded voices, and automated dialogue systems. These studies examined how technology alters emotional communication.

Mapping Emotional Expression

Using Hupont et al.’s framework of six primary emotions, I photographed and analysed facial expressions from twelve participants. Layering these images created composite visualisations that revealed both shared characteristics and significant variation between individuals.

Prototyping Emotion Recognition

Using Adobe After Effects and facial mesh tracking, I developed an interactive prototype that analysed facial expressions in real time during a recorded tutorial. The prototype classified emotional states while exposing the limitations and ambiguity of automated interpretation.

Communicating Through Visualisation

The project concluded with a video installation combining motion graphics, facial composites, tracking overlays, and recorded interactions to communicate both the possibilities and shortcomings of emotion recognition technologies.

Composite Studies - twelve participants layered across six primary emotions

Anger
Disgust
Fear
Happiness
Sadness
Surprise

Facial Charts - averaged measurements for each emotional state

Anger
Disgust
Fear
Happiness
Sadness
Surprise

The Work

Conducted independent research into emotion recognition, human-computer interaction, and digital communication during the COVID-19 pandemic.

Designed a series of interaction experiments exploring communication between people and AI systems, including mediated conversations using Siri and Alexa.

Captured and analysed facial expression data from twelve participants to create composite visualisations of six primary emotions.

Developed a real-time emotion recognition prototype using facial mesh tracking and Adobe After Effects.

Produced a motion graphics film combining facial composites, tracking overlays, and live demonstrations to communicate the research.

Impact

Exploring the Limits of AI

The project questioned whether emotion recognition systems can accurately interpret human emotion, highlighting the complexity, subjectivity, and cultural variation that challenge automated classification.
Visualising Invisible Processes

By exposing facial tracking, emotion mapping, and algorithmic interpretation, the project made normally invisible AI processes visible, helping audiences understand how these systems analyse human behaviour.
Combining Research and Motion Design

The project translated research into a visual narrative through data visualisation, animation, and interactive prototyping, demonstrating how complex technological concepts can be communicated through design rather than traditional academic outputs.