Sign in


Capture emotion expressions from video, audio, and text with a single pipeline.


Cite this software

What mexca can do for you

mexca is an open-source Python package which aims to capture human emotion expressions from videos in a single pipeline. The package implements the customizable yet easy-to-use Multimodal Emotion eXpression Capture Amsterdam (MEXCA) pipeline for extracting emotion expression features from videos. It contains building blocks that can be used to extract features for individual modalities (i.e., facial expressions, voice, and dialogue/spoken text). The blocks can also be integrated into a single pipeline to extract the features from all modalities at once. Next to extracting features, mexca can also identify the speakers shown in the video by clustering speaker and face representations. This allows users to compare emotion expressions across speakers, time, and contexts.

The package contains five components that can be used to build the MEXCA pipeline:

  • FaceExtractor: Detects faces, encodes them into an embedding space, clusters the embeddings to link reoccuring faces, and extracts facial landmarks and action units.
  • SpeakerIdentifier: Performs speaker diarization, that is, detects speech and speech segments, encodes speakers into an embedding space, and clusters the embeddings. Attempts to answer the question: “Who speaks when?”.
  • VoiceExtractor: Extracts voice features, such as pitch, associated with emotion expressions.
  • AudioTranscriber: Transcribes detected speech segments to text.
  • SentimentExtractor: Predicts sentiment scores for the transcribed text.
Logo of mexca
Programming languages
  • Python 99%
  • Dockerfile 1%
  • Apache-2.0
</>Source code

Participating organisations

Social Sciences & Humanities
Social Sciences & Humanities
Netherlands eScience Center
University of Amsterdam


Malte Lüken
Malte Lüken
Research Software Engineer
Netherlands eScience Center
Eva Viviani
Research Software Engineer
Netherlands eScience Center
Gijs Schumacher
Christian Pipal
Kody Moodley
Kody Moodley
Research Software Engineer
Netherlands eScience Center

Related projects


Multimodal Emotion Expression Capture Amsterdam

Updated 9 months ago

Vocalisations of Disgust

Vocalisations of Disgust across Behavioural Contexts: A Computational Analysis

Updated 7 months ago

Different kinds of laughter

A machine learning approach

Updated 9 months ago

Related tools



Interactive visualization and exploration of multi modal time-series data and video

Updated 14 months ago
1 5