Capture emotion expressions from video, audio, and text with a single pipeline.


Cite this software

What mexca can do for you

mexca is an open-source Python package which aims to capture human emotion expressions from videos in a single pipeline. The package implements the customizable yet easy-to-use Multimodal Emotion eXpression Capture Amsterdam (MEXCA) pipeline for extracting emotion expression features from videos. It contains building blocks that can be used to extract features for individual modalities (i.e., facial expressions, voice, and dialogue/spoken text). The blocks can also be integrated into a single pipeline to extract the features from all modalities at once. Next to extracting features, mexca can also identify the speakers shown in the video by clustering speaker and face representations. This allows users to compare emotion expressions across speakers, time, and contexts.

The package contains five components that can be used to build the MEXCA pipeline:

  • FaceExtractor: Detects faces, encodes them into an embedding space, clusters the embeddings to link reoccuring faces, and extracts facial landmarks and action units.
  • SpeakerIdentifier: Performs speaker diarization, that is, detects speech and speech segments, encodes speakers into an embedding space, and clusters the embeddings. Attempts to answer the question: “Who speaks when?”.
  • VoiceExtractor: Extracts voice features, such as pitch, associated with emotion expressions.
  • AudioTranscriber: Transcribes detected speech segments to text.
  • SentimentExtractor: Predicts sentiment scores for the transcribed text.
Logo of mexca
Programming languages
  • Python 99%
  • Dockerfile 1%
</>Source code

Participating organisations

Social Sciences & Humanities
Social Sciences & Humanities
Netherlands eScience Center
University of Amsterdam

Reference papers



Malte Lüken
Malte Lüken
Research Software Engineer
Netherlands eScience Center
Research Software Engineer
Netherlands eScience Center
Gijs Schumacher
Christian Pipal
Kody Moodley
Research Software Engineer
Netherlands eScience Center

Related projects


Multimodal Emotion Expression Capture Amsterdam

Updated 1 month ago
In progress

Vocalisations of Disgust

Vocalisations of Disgust across Behavioural Contexts: A Computational Analysis

Updated 5 months ago

Different kinds of laughter

A machine learning approach

Updated 19 months ago

Related software



Interactive visualization and exploration of multi modal time-series data and video

Updated 23 months ago
1 5