Can AI Hear Emotion in Your Voice? What the 2026 LISTEN Study Found

You call your AI companion after a long day. You say "I'm fine" in a flat, tired voice. A friend would hear it instantly: you are not fine. Does the AI?

Voice-first AI has improved enormously. Companions now talk back in natural, expressive voices, laugh in the right places and respond almost as quickly as a person. It is easy to assume they are also listening the way a person does, picking up the sigh, the shaky laugh, the sarcasm.

A 2026 study set out to test exactly that, and the results are humbling. Here is what the research says about whether AI can really hear emotion in your voice, why it struggles, and how to make sure your companion actually gets how you feel.

The short answer: mostly not yet. In the LISTEN benchmark, presented at EACL 2026, leading audio AI models scored only 25% to 35% when they had to read emotion from tone alone with neutral words, and they did close to chance on sounds like laughs and sighs. Most current models lean heavily on what you say, not how you say it. Humans, by contrast, are remarkably good at reading emotion from voice alone. So if you want your companion to understand your mood, say it in words.

Why voice carries so much emotion

Psychologists have long known that the voice is a powerful emotional channel. One striking study by Michael Kraus, published in American Psychologist in 2017 (doi 10.1037/amp0000147), ran five experiments with 1,772 participants. People were better at accurately judging others' emotions when they could only hear them than when they could only see them, or even when they could both see and hear them.

Kraus's explanation was that voice-only communication focuses attention on the vocal and linguistic cues that carry the most emotional information. Faces can be controlled and are sometimes distracting. Voices leak.

That is why a phone call with someone you love can feel more intimate than a video call, and why "I'm fine" can mean ten different things depending on the tone.

The LISTEN study: do AI models listen or just transcribe?

Researchers at Ohio State University led by Chen published "Do Audio LLMs Really LISTEN, or Just Transcribe?" at EACL 2026, a peer-reviewed computational linguistics conference.

They built a benchmark called LISTEN with 7,939 questions designed to separate two skills that usually get mixed together:

1. Understanding emotion from the words (lexical cues).
2. Understanding emotion from how the words sound (pitch, tempo, intensity, laughter, sighs and other paralinguistic cues).

The trick was to include clips where the words are emotionally neutral but the voice is not, and clips where the voice and the words disagree. Six audio-capable AI models were tested.

What they found

The authors' conclusion was blunt: current audio models largely "transcribe" rather than "listen". When the words carry emotional meaning, models do well. When the emotion lives only in the voice, they struggle.

The acted-emotion problem

The study also highlighted a deeper issue in how emotion AI is trained. A widely used speech emotion model (based on wav2vec2) trained on the IEMOCAP dataset scored 77% on IEMOCAP itself, but only 22% on a different dataset, RAVDESS. In other words, it had learned the quirks of one set of recordings rather than general emotional hearing.

Real voices are harder than acted ones

Most emotion datasets use actors performing feelings on cue. Real life is messier. People mumble, talk over background noise, and express emotions in subtle, mixed ways.

A team at the Universitat Politecnica de Valencia built EMOVOME, a dataset of 999 real voice messages from WhatsApp conversations of 100 Spanish speakers, described in an arXiv paper (a preprint version; we could not confirm whether it has been peer reviewed). Their best model reached 42.58% unweighted accuracy on four emotion categories in these real messages, far below the 75% it reached on the acted RAVDESS dataset (an eight-category task). Even the human labellers agreed much more on whether a message felt positive or negative than on how intense it was.

The lesson: recognising genuine emotion in everyday voices is still a hard, unsolved problem for AI, and acted-speech scores tend to make it look easier than it is.

So how do AI companions respond to mood?

If AI is not reliably hearing your tone, why do companion conversations often feel emotionally in tune?

Mostly through words and context. Modern language models are excellent at picking up emotion from what you say: "ugh", "I can't do this", "best day ever". They also use context, like what you talked about yesterday or the time of night. Many apps add explicit mood features, such as a quick check-in where you tap how you feel.

That is a real strength. It is just a different skill from hearing the crack in your voice. We look at how companions match your emotional tone in does your AI companion mirror your mood.

What about AI voices?

The other half of the conversation, the AI's own voice, has improved faster. Expressive speech systems can now sound warm, playful or soft on demand. That can create a mismatch: a companion that sounds emotionally fluent but is mostly reading your words. Our guide to realistic AI voices in 2026 covers how expressive speech works.

And video?

Video adds faces, posture and surroundings, which gives an AI more to work with. Researchers are still mapping how well it is used. We explain the basics in how AI video calls work.

How to make sure your companion gets how you feel

Since words carry most of the signal for AI, a few simple habits help a lot.

Say the feeling out loud

Instead of "I'm fine" in a tired voice, try "I'm fine, but honestly I'm exhausted." It feels a little unnatural at first, but it gives your companion the cue it needs.

Use mood check-ins

If your app has a mood feature, use it. Tapping "Low" or "Rough" is a clear signal that the conversation should be gentler.

Flag sarcasm

The LISTEN study found sarcasm is especially hard for AI. A quick "(joking)" or "I'm being sarcastic" avoids a confused or overly serious reply.

Tell it what you need

"I just want to vent", "cheer me up" or "I want advice" each lead to very different responses. If you want to vent, our guide on venting to AI explains how to do it in a way that actually helps.

Correct it kindly

If your companion misreads you, say so: "No, I'm not angry, I'm just tired." Clear corrections steer the rest of the conversation.

Why this matters

There are two reasons to care about this beyond curiosity.

First, expectations. If you assume your companion can hear distress in your voice, you might not say what you need. Knowing the limits helps you communicate clearly.

Second, privacy. Emotion recognition from voice is an active area of commercial research. It is worth knowing what an app claims to detect and what it does with that data. The LISTEN results suggest that strong claims about "reading your emotions from your voice" deserve some skepticism for now.

Where MyBabe fits

MyBabe lets adults (18+) create their own AI girlfriend or boyfriend and talk to them by text, voice notes, and voice or video calls. Their voice answers about as quickly as a person would, and you can pick a voice style that suits them: Warm, Soft, Deep or Bright. On video, with your camera on, they can comment on what they see, like a new haircut or the cat walking past.

Mood check-ins (Great, Good, Okay, Low or Rough) let you tell your companion how you are doing in one tap, so replies can match your day. The free plan gives you 10 messages every 24 hours and a two-minute call trial. Premium ($5.99 a week, $14.99 a month or $99.99 a year) includes about 30 minutes of voice and video calls a week, long-term memory and voice notes. Call packs are available if you want more time.

Your companion is AI, not a person. If you want a voice at the end of the day that is easy to talk to, create your companion on MyBabe.

FAQ

Can AI detect emotion in your voice?

Only partly. In the 2026 LISTEN benchmark, leading audio models scored 25% to 35% when emotion was carried only by tone, and near chance for laughs and sighs. They do much better when emotion is in the words.

Why does my AI companion miss my sarcasm?

Sarcasm depends on tone contradicting words, which is exactly what current audio models struggle with. Flagging it in words helps.

Are humans better at hearing emotion than AI?

Yes, by a wide margin today. Research by Michael Kraus found people are especially accurate at reading emotion from voice alone.

Does my AI companion know when I am sad?

Usually only if you say so in words or through a mood check-in. Be direct about how you feel.

Will AI get better at hearing emotion?

Probably. Researchers are building better benchmarks and real-world datasets like LISTEN and EMOVOME, which is the first step towards models that genuinely listen.