Start Detecting Non-Speech Sound Events with the Modulate API Today

First things first: Why

A transcript is a record of what was said but it is not a record of what happened, meaning the signals that matter most are often not just words. For example, a support call transcript that captures a customer saying "hold on, sorry" and throws away the crying in the background. Or a transcript of a lobby altercation says nothing about the glass that broke. If your audio pipeline only reads words from a transcript, every non-speech signal in the recording is gone before you get a chance to look at it. 

This leaves developers pointing reviewers at thousands of files instead of a smaller, more manageable set. For instance, at a contact center, “Which of these 40,000 calls had someone crying” is a lot less helpful than “which of these 40 calls had someone crying.” 

Due to the API’s audio-native architecture, our models have an edge when detecting acoustic signals beyond voice with things like AI Music Detection, or even understanding full conversations with context no one else in the industry can provide using Velma Triage. However, Audio Event Detection as the newest addition to the Modulate API takes another giant leap forward: it further expands our catalog past speech and traditional transcription by explicitly exposing and scoring non-speech audio based on common customer use cases. This includes areas like Trust & Safety, Media Tooling, and Contact Centers/CS, to name a few. 

Audio Event Detection is now live

Now generally available, Modulate’s Audio Event Detection API scores an audio file against 42 non-speech sound events in one synchronous request: laughter, crying, and coughing, household and office noises, gunshots and shattering glass, animals, musical instruments, and many more. One POST, one JSON response, a probability for every supported event. 

Listen to it before you write any code

The fastest way to find out whether the model catches what you care about is to feed it your own audio. Drag a file into the live demo and you get the same scores the API returns, without a key or a client. Two things to know while you are poking at it: the demo hides anything below 50 percent, which is a display choice rather than a recommended threshold, and it scores the clip as a whole, exactly like the endpoint does.

One call, one response

Everything runs through a single REST endpoint. Get an API key here, pass it as an X-API-Key header, and attach a file:

curl -X POST https://platform.modulate.ai/api/velma-2-audio-event-classifier \
  -H "X-API-Key: $MODULATE_API_KEY" \
  -F "upload_file=@audio.mp3"

The same call in Python, with the ranking logic you will want on top of it:

import os, requests

response = requests.post(
    "https://platform.modulate.ai/api/velma-2-audio-event-classifier",
    headers={"X-API-Key": os.environ["MODULATE_API_KEY"]},
    files={"upload_file": open("audio.mp3", "rb")},
)
response.raise_for_status()
result = response.json()

probs = result["probs"]

# The 41 shared keys sum to 1, so rank them rather than thresholding each one.
shared = {event: score for event, score in probs.items() if event != "cry"}
top_event, top_score = max(shared.items(), key=lambda item: item[1])

print(f"{result['duration_ms']} ms")
print(f"most prominent event: {top_event} ({top_score:.3f})")
print(f"crying: {probs['cry']:.3f}")

A response for a two second clip of laughter, trimmed for readability:

{
  "probs": {
    "cry": 0.01,
    "Laughter": 0.913,
    "Applause": 0.03,
    "Cough": 0.01,
    "Finger_snapping": 0.005,
    "Acoustic_guitar": 0.002,
    "Gunshot_or_gunfire": 0.001,
    "Shatter": 0.001
  },
  "duration_ms": 2000
}

A note on interpreting this output: “Laughter”: 0.913 translates to a 91.3% probability of laughter with extremely low probability of other events that could share sound characteristics. This indicates with high confidence that someone is laughing

All 42 keys are present on every response no matter what is in the audio, so there are no conditionals in your parsing. Only the values move. The rest of the surface area fits in a table:

MOV, MP4, and WebM are on that list on purpose. If your source material is video, you skip the extraction step and post the file as it sits.

The one thing to get right about the scores

probs always carries 42 keys, but they are not all on the same scale, and this is where a first integration usually goes wrong.

cry is an independent probability. Thresholding it is meaningful: 0.94 means the model is confident the clip contains crying, and that value does not move based on what else is in the audio.

The other 41 keys are drawn from a single shared distribution that sums to 1 across them, so they rank the most prominent event rather than reporting independent detections. A clip with both applause and laughter splits probability between Applause and Laughter, and neither lands where it would have alone. Read those 41 by taking the highest, keep a separate threshold for cry, and do not compare across the two. cry against Laughter is a comparison of two different quantities.

Keep in mind that key names are case sensitive so make sure to copy them from the API reference rather than retyping them. Standard error conventions also apply: If you are scanning at volume, hold in-flight requests at or under your quota and retry on 429 instead of fanning out and hoping.

Three patterns that work well

Sweep an archive for exceptions. At $0.02 per hour you can afford to run every file you have, then look only at the clips that came back with something. This is the shape most teams start with: 40,000 recordings in, a few hundred worth a human's attention out.

Split the file when you need locality. Scores apply to the clip as a whole, so if you need to know roughly where an event happened, segment first and call once per chunk, using the chunk index as your timestamp.

ffmpeg -i input.wav -f segment -segment_time 10 -c copy chunk_%03d.wav

Ten second chunks give you ten second resolution. It multiplies your request count, but the arithmetic usually still works out.

Layer it onto what you already run. It’s easy to incorporate additional models to get a more complete set of data. Audio Event Detection reads the room, Emotion Detection reads the speaker, and speech-to-text reads the words. Same audio and same key means adding a layer is another call rather than another vendor, and Modulate makes it seamless to only use the models you need. Which API should I use? lays out the choice between them.

Known limits and where this is heading

Worth knowing before you design around it. All three of these are shaped by what developers ask us for, so if one of them is blocking you, that is useful information.

• Whether, not when. No timestamps yet. Chunking, above, is the workaround, and finer output granularity is the request we hear most.
• A fixed set of 42 events, and no Speech label. Ordinary conversation returns near zero across the board, which is what makes this good at finding exceptions rather than describing a normal call. Sirens, alarms, shouting, and door slams are not in the set today, and label coverage grows based on what customers need.
• Batch only. One synchronous call per file, so it sits in a processing pipeline rather than a live path.

Start detecting audio events

Drag a file into the demo to see whether the model hears what you need, then grab a key and ship it. The integration is one request and about ten lines of parsing.

Need a sound that is not in the 42 available options, or see a signal that we’re missing? Tell us at support@modulate.ai.

‍