Spotify Patents A Voice Assistant That Can Read Your Emotions

Spotify appears to be interested in challenging Alexa and Siri with its own state-of-the-art voice assistant. Last month, the streaming service was awarded a patent on new ways to make a voice assistant more responsive to human emotions.

Originally filed in 2018, the patent describes a voice assistant for a “media playback device” that could, for example, recognize when a user sounds sad and acknowledge sympathetically (“you seem a little down”), offer encouragement (“cheer up”) or tough love (“snap out of it!”). If a user sounds like they are in a rush, the voice assistant might try to keep pace by speaking faster or responding in fewer words.

The theoretical voice assistant could listen to a user’s vocalized reactions to music and stop playing songs that seem to evoke anger, sadness, or other negative emotions. 

“Spotify’s patent… claims to interpret the emotion conveyed in human voice commands as part of the input, within the utterance itself, and apply intelligence to infer a logical outcome that may not have been explicitly requested,” said Simon Forrest, principal technology analyst at Futuresource Consulting. 

“An example is the user uttering ‘ugh’ when a song starts playing and for the voice assistant to recognize that this implies a negative emotional response and stop the playback, or switch to another song, even though no specific ‘stop’ or ‘skip’ command has been issued,” Forrest said.

The technology described in the Spotify patent would be able to discern emotions from the tone, cadence, volume, and pitch of a user’s voice. However, Spotify’s current capabilities of speech emotion recognition are unknown.

Rahul Telang, professor of Information Systems and Management at Carnegie Mellon University’s Heinz College, said speech emotion recognition still is in its early stages.

“It’s an emerging field; I don’t think there is something widely used in the market today,” Telang said. “Two or three years down the line, it could become mainstream.” 

Amazon and Google also have published patents related to speech emotion recognition in recent years, as global sales of their smart speakers have surged. Last year, Bloomberg reported that Amazon is working on a voice-activated wearable device that can read human emotions.

“Detection of emotional states is becoming essential to improvements in voice recognition technology … and is a crucial step along the way to true conversational AI,” said Forrest.

Telang said it is unclear if introducing an emotionally intelligent voice assistant would give Spotify an edge over competing services like Apple Music.

“It sounds great on paper, but whether it leads to the appropriate loyalty and returns is up in the air,” Telang said.

Spotify shares some user data with advertisers and music industry partners. Although the new patent doesn’t say whether information about users’ emotions could be used in this way, Telang said this might be a concern if Spotify actually adds these capabilities.

“If the user worries about the information being shared, they might be spooked rather than fall in love with [Spotify],” Telang said. “It’s a double-edged sword.”

Spotify did not respond to multiple requests for comment on the patent.

It could be a while before Spotify brings conversational AI to streaming music, but other enhancements to its voice user interface might be arriving soon. App researcher Jane Manchung Wong recently uncovered code for a voice activation feature —  “Hey Spotify” — within the Spotify mobile app.