mexca

Capture emotion expressions from video, audio, and text with a single pipeline.

4
mentions
5
contributors

Cite this software

What mexca can do for you

mexca is an open-source Python package which aims to capture human emotion expressions from videos in a single pipeline. The package implements the customizable yet easy-to-use Multimodal Emotion eXpression Capture Amsterdam (MEXCA) pipeline for extracting emotion expression features from videos. It contains building blocks that can be used to extract features for individual modalities (i.e., facial expressions, voice, and dialogue/spoken text). The blocks can also be integrated into a single pipeline to extract the features from all modalities at once. Next to extracting features, mexca can also identify the speakers shown in the video by clustering speaker and face representations. This allows users to compare emotion expressions across speakers, time, and contexts.

The package contains five components that can be used to build the MEXCA pipeline:

  • FaceExtractor: Detects faces, encodes them into an embedding space, clusters the embeddings to link reoccuring faces, and extracts facial landmarks and action units.
  • SpeakerIdentifier: Performs speaker diarization, that is, detects speech and speech segments, encodes speakers into an embedding space, and clusters the embeddings. Attempts to answer the question: “Who speaks when?”.
  • VoiceExtractor: Extracts voice features, such as pitch, associated with emotion expressions.
  • AudioTranscriber: Transcribes detected speech segments to text.
  • SentimentExtractor: Predicts sentiment scores for the transcribed text.
Logo of mexca
Keywords
Programming languages
  • Python 99%
  • Dockerfile 1%
License
</>Source code

Participating organisations

Social Sciences & Humanities
Social Sciences & Humanities
Netherlands eScience Center
University of Amsterdam

Reference papers

Mentions

Contributors

Malte Lüken
Malte Lüken
Research Software Engineer
Netherlands eScience Center
EV
Research Software Engineer
Netherlands eScience Center
GS
Gijs Schumacher
CP
Christian Pipal
Kody Moodley
Research Software Engineer
Netherlands eScience Center

Related projects

MEXCA

Multimodal Emotion Expression Capture Amsterdam

Updated 1 month ago
In progress

Vocalisations of Disgust

Vocalisations of Disgust across Behavioural Contexts: A Computational Analysis

Updated 5 months ago
Finished

Different kinds of laughter

A machine learning approach

Updated 19 months ago
Finished

Related software

Emo-spectre

EM

Interactive visualization and exploration of multi modal time-series data and video

Updated 23 months ago
1 5