Speech Emotion Recognition

Speech carries more than words. Changes in pitch, energy, rhythm, and tone can give listeners clues about how an utterance sounds emotionally.

The Speech Emotion Recognition project explores how machine learning can identify those patterns in audio and assign an emotion label. Try the product: tonelabb.vercel.app

The project is built around the RAVDESS speech dataset, a collection of acted emotional speech. Its current classification task covers six labels: neutral, happy, sad, angry, fearful, and disgust.

Because the examples are acted, the system’s output should be understood as a prediction based on patterns in that dataset - not as a reliable reading of a speaker’s actual internal state.

Preparing speech for machine learning

Before a model can classify an audio clip, the clip needs to be converted into a consistent form. The project loads audio as mono, resamples it to 22,050 Hz, trims silence, and adjusts it to three seconds by cropping or padding. It then normalizes the waveform.

The project prepares two kinds of audio features. The first uses Mel-frequency cepstral coefficients (MFCCs). For each of 40 coefficients, it calculates the mean and standard deviation over time, producing an 80-value feature vector. This gives conventional classifiers a compact summary of the sound.

The second representation is a log-mel spectrogram. It describes how sound energy changes over time across frequency bands. The project resizes each spectrogram to a 128-by-128 image for its convolutional neural network.

Comparing two modeling approaches

The MFCC training workflow compares a support vector machine, a random forest, and - when installed - an XGBoost classifier. Each model is evaluated on validation and test data. The project chooses the deployable model using validation macro-F1, a measure that considers performance across classes rather than letting the most frequent class dominate.

A separate workflow trains a small convolutional neural network on log-mel spectrograms. It uses validation accuracy to guide learning-rate adjustments and early stopping. The CNN is saved as its own model artifact.

At present, the API uses the MFCC pipeline. The CNN training workflow is available in the repository, but the API does not load or serve its model. Keeping that distinction clear helps users understand which model is actually making predictions.

Keeping the evaluation speaker-independent

A useful evaluation should check whether a model can handle voices it did not encounter during training. To support that, the data loader assigns each actor’s recordings to just one split: training, validation, or test. The configured validation and test portions are each 15 percent.

This speaker-independent split reduces the chance that the same person’s voice appears in both training and evaluation data. It makes the evaluation more meaningful than a random clip-level split, though it still measures performance on this dataset and its acted recordings.

Making predictions through an API

Once a trained MFCC pipeline is available, the FastAPI service provides an HTTP endpoint for audio classification. A client uploads an audio file to POST /predict; the service validates the file, checks its duration and silence level, and returns the predicted label, the top three class probabilities, the clip duration, and a disclaimer.

The endpoint accepts WAV, MP3, FLAC, OGG, and M4A files up to 20 MB. The service also provides a health endpoint and interactive API documentation.

Recent predictions can be retrieved through another endpoint when Supabase has been configured. The model artifact can be present on the server or downloaded from Supabase Storage at startup. The API can also log predictions to a Supabase table. Those features depend on external Supabase configuration, including storage access and a prediction table whose schema is not defined in the repository.

A practical foundation, with clear boundaries

The project brings together dataset preparation, feature engineering, model training, evaluation, and API serving in one codebase. It offers two approaches to modeling audio and includes a speaker-aware split that supports a more careful evaluation.

Its scope also has limits. RAVDESS consists of acted speech, so results on it do not establish how well the model will perform on spontaneous conversation or across different recording conditions and populations. The repository does not claim a fixed accuracy figure; training produces evaluation reports for the models that are run.

That makes the project a useful foundation for experimentation and demonstration. Its most responsible use is as a study of emotion classification from acted speech, with predicted labels treated as model outputs - not facts about what a person feels.

Where speech emotion recognition could be useful

Speech emotion recognition can support products that need to respond to the tone of an interaction as well as its words. Potential applications include:

• Customer support: Flag calls that sound frustrated so a human agent can review them or offer extra help. The model’s label should be treated as a cue, not a verdict about the caller.
• Voice interfaces: Help a voice assistant adjust its response style when speech sounds excited, calm, or upset.
• Media and accessibility tools: Add emotion-related tags to dialogue in audio or video, giving listeners another way to navigate or understand content.
• Call quality analysis: Help teams explore patterns across recorded interactions and identify conversations that may need follow-up.
• Research and prototyping: Let developers experiment with audio features, model evaluation, and speech-based interfaces.

These are possible uses of the technology, not claims that this project has been validated for those settings. The model was built with acted RAVDESS speech, so applying it to real conversations would require evaluation on representative recordings, attention to consent and privacy, and a way for people to correct or challenge its predictions.

A predicted emotion should not be used by itself for consequential decisions about a person.