Wirebook › Bonus lesson
How does a phone know a song from hearing it?
You're in a shop, a song comes on, and you hold up your phone. A few seconds later it tells you the name of the song, even with people chatting and trolleys rattling. It doesn't have a tiny ear that remembers every song ever. It turns those few seconds into a star map of the loudest notes, makes short fingerprints from pairs of stars, and looks them up in a giant index, like finding a word in the back of a book. Here you can watch that happen with six made-up tunes.
Step 1: turn sound into a picture
A microphone only records air wobbling back and forth thousands of times a second. The app chops the recording into slices a fraction of a second long and works out which notes (frequencies) are in each slice and how loud they are. Laid side by side, the slices make a picture called a spectrogram: time across, pitch up the side, loudness as brightness. A tune shows up as a bright wiggly line, with fainter copies above it (the higher "overtones" every instrument makes).
Step 2: keep only the brightest stars
Noise in a shop covers the picture with a dim fog, but it rarely beats the music at the music's loudest spots. So the app throws away almost everything and keeps just a few peaks per slice, the spots louder than everything around them. What's left looks like a map of stars in the night sky. A song's star map stays much the same whether it's played loudly, quietly, through a cheap speaker or over chatter, and that's the trick that makes it work in a noisy café.
Step 3: pairs of stars make a fingerprint
One star on its own isn't special; lots of songs have that note somewhere. A pair is much more unusual: "this note, then that note, this far apart in time". The app pairs each star with a few stars just after it and turns each pair into a short number, an audio fingerprint. It doesn't matter when you started listening, because the pair only records the gap between the two stars, not the clock time. The service has done the same for every song in its library and filed every fingerprint in an index, along with which song it came from and when.
Step 4: votes that line up
Your phone sends its few hundred fingerprints, not the recording, and each one is looked up. Many match by luck in random songs, at random moments. But for the real song they all agree on the same moment: "Toast Rocket, about 4 seconds in". When enough votes pile up on one song at one moment, the app answers. That's why it often needs a few more seconds in a loud place: it's waiting for more votes.
Why humming usually doesn't work
Song recognition matches the exact recording: the same notes at the same pitch with the same timing. If you hum it, your notes are a little higher or lower, a little faster or slower, and without the drums and backing, so almost none of the star pairs match. Try the "hums it instead" box above. "Hum to search" features exist, but they use a different trick: they follow the shape of the tune (up, up, down, long note) instead of exact stars, and they guess rather than know.
Is my phone always listening for songs?
Most song apps only listen after you press the button, and they send fingerprints rather than the sound itself. Some phones have a setting that recognises songs playing nearby by themselves; those keep a small list of popular songs' fingerprints on the phone and match without sending anything. If you'd rather it didn't, the setting can be turned off. Either way, the app isn't keeping every song you've ever heard in your phone. A fingerprint is tiny compared with the music: this page's whole library of six tunes is about four thousand little numbers.
Related lessons: how sound and pictures are squeezed to travel · how apps decide what to show you next · how a phone recognises your face.
Grown-up twin: Tooltern's help page on why song recognition apps miss a song, and what to try.