Audio fingerprinting versus reading the screen: how track detection actually works

Audio fingerprinting works by turning a few seconds of sound into a small set of landmarks, usually the loudest points on a spectrogram, and looking those landmarks up in an index built from millions of known recordings. If enough landmarks from the clip line up with one recording at a consistent time offset, that recording is the match. It is fast, it needs only a short sample, and it copes with compression and background noise because it only cares about the strongest peaks. It fails when the peaks have moved, which is what pitch shifting and mixing do, and when the recording has never been indexed, which is the case for every unreleased track.

Reading the screen is the other way to identify a track in a video: pull frames from the clip and read the title and artist off a CDJ, a laptop or a text overlay. It does not care about pitch, mixing or release status. It cares whether the text is in shot. The two methods fail in different places, which is why the best detection uses both.

Fingerprinting, step by step

Spectrogram. The audio is cut into short overlapping windows, a few tens of milliseconds each, and each window is transformed into a set of frequency bands and their energies. Stack those and you have a picture of the sound with time along one axis and frequency along the other.

Peak picking. Most of that picture is thrown away. The algorithm keeps only points that are louder than everything around them in both time and frequency, spread so that every region of the spectrogram contributes a few. These survive noise well: adding crowd or hiss raises the floor but rarely changes which points are the local maxima.

Hashing. The approach Shazam published in 2003 pairs each peak with a handful of peaks shortly after it, and stores each pair as a compact hash of the two frequencies and the time gap between them. A three-second clip yields hundreds of these. Each hash is a small, specific claim about the recording: “there is a peak at this frequency followed this many milliseconds later by a peak at that frequency.”

Lookup and voting. Every hash from the clip is looked up in the index, which maps hashes to every recording and position where they occur. For each candidate recording, the algorithm looks at the difference between where each hash appears in the clip and where it appears in the recording. A true match produces a pile of hashes with the same offset. A false candidate produces scattered offsets. The recording with the tallest pile wins, and the height of the pile is the confidence.

That is the whole idea. Everything else is engineering: how many peaks to keep, how far apart to pair them, how to shard an index of billions of hashes, and how to decide the pile is tall enough.

Why it survives some things and not others

Noise, compression and a bad speaker mostly add energy or remove detail. The strongest peaks are still the strongest peaks, so the hashes still match. That is why Shazam works in a pub.

Pitch and tempo changes move the peaks. Speed a track up by four percent and every peak shifts up in frequency and the time gaps shrink by the same factor. The hashes are quantised coarsely enough that very small changes sometimes still land in the same bucket, but as the shift grows the matches thin out until the pile is no taller than chance. Master tempo, which keeps pitch while changing speed, still changes the time gaps. Some commercial services add tolerance for this, with mixed results.

Mixing overlays a second set of peaks. During a blend, the peak picker chooses whichever track is louder in each region, and the clip’s hashes become a shuffle of two recordings, neither of which gets enough votes. Loops, filters and delay do similar damage on a smaller scale.

Unreleased tracks are simply absent. The index is built from recordings that have been delivered to the service, typically through distributors and labels. Anything played from a private file has never been seen. No tuning changes that.

The main systems and how they differ

Shazam is the consumer standard, owned by Apple, with the largest catalogue of released music and a design optimised for short samples from a phone microphone. It is the one most people mean when they say “fingerprinting”.

ACRCloud is a commercial recognition service used by broadcasters, monitoring companies and apps. It uses a peak-based approach of its own and maintains a large catalogue, and it works well with short samples of clean audio. It also offers separate models for humming and for cover versions, which are different problems from exact matching.

AudD is another commercial API in the same family, with its own catalogue and its own tolerances. Services like this disagree with each other more often than you would expect, particularly on remixes and regional releases, which is the practical case for querying more than one.

Chromaprint and AcoustID work differently and are built for a different job. Chromaprint, the open-source fingerprinter, does not pick peaks. It computes chroma features, which describe how much energy sits in each of the twelve pitch classes over time, and compresses those into a sequence of 32-bit values. Two fingerprints are compared by how many bits differ. It is designed for whole files, and its main use is tagging a music library against the MusicBrainz database through AcoustID, where the fingerprints come from users submitting their own files. It is excellent at “is this file the same recording as that file” and poor at “what is this twelve-second clip from a club”.

That said, Chromaprint’s whole-file comparison is useful for a related problem: telling whether two unidentified clips contain the same unknown track. Nobody needs to know what the track is to know that the same one has been recorded twice.

Verifying a match

A fingerprint hit is a claim, and on dirty audio the claim is sometimes wrong in a specific way: right artist, wrong track; original when you heard the remix; radio edit when the DJ played the extended mix. Verification means taking the claimed artist and title and looking them up in a store catalogue such as Spotify or Apple Music, which confirms the recording exists as named, gives you the correct spelling and versions, and lets a second signal check the version. Without that step, a confident-looking result on a noisy clip is only as good as the noisiest hash.

Reading the screen

The second method treats the video as a document. Frames are extracted from the clip, a few per second or at scene changes, and each is examined for text that names the track. In DJ footage that text takes a few forms.

• A CDJ or laptop screen showing the title and artist of the loaded track, described in reading the deck.
• A text overlay the creator added, often stylised, sometimes appearing for only a second.
• A caption banner such as “Track ID:” or “Now playing:”.
• A platform-generated music sticker, which is really metadata rather than pixels but appears in the same place.

Traditional OCR, of the kind in a scanner, handles plain black text on white. It does badly on a transparent overlay across a moving crowd, a deck screen at an angle with glare, or a handwritten caption. A vision model reads those the way a person would, using context: it knows that on a Pioneer display the title is the large line at the top and the artist sits beneath it, that “BPM” and “KEY” mark a deck panel rather than a title, and that a line reading “ID?” is a question, not an answer. It returns structured fields, artist and title, with a confidence and a note of where it read them.

The failure modes are different from fingerprinting. Vision fails when the screen is out of shot, out of focus, or showing the cued deck rather than the playing one. It fails when the overlay names a different track as a joke, or misspells the artist. And it fails on the majority of clips, which simply do not contain any text. But when it works, it works on unreleased material, pitched audio, and long blends, because it never listens to the audio at all.

Why fusion beats either alone

Fingerprinting and screen reading fail in opposite places, so combining them covers most of the ground. The useful design is not “try one, then the other” but running both and cross-checking. When the fingerprint says one thing and the screen says the same, the confidence is close to certain. When the fingerprint finds the original and the screen reads “Extended Mix”, the screen wins on version. When the fingerprint fails and the screen reads a filename like “untitled 03 master”, the track is unreleased and the filename is the lead.

Two more signals belong in the same pile. The caption and comments are text that people wrote about the track, which is often the answer in plain words. The platform’s own metadata, such as a tagged sound on TikTok or Instagram, is a licensing match the platform has already done. None of the four is reliable alone. Together they identify most clips that can be identified.

That is the design behind Detect Audio: five signals run at the same time, audio fingerprinting through several services with samples at more than one point in the clip, a Claude vision model reading the frames, caption parsing, platform metadata and comment mining, then fused with cross-confirmation and verified against Spotify and Apple Music. Unidentified clips are fingerprinted with Chromaprint and compared with each other, so the community can see when the same unknown track has turned up in several people’s recordings.

What this means when you are trying to identify something

Sample where the audio is cleanest, because peaks are what matter. Do not trust a single service, because their catalogues and tolerances differ. Look for a screen, because it is the only method that works before release. And when everything fails, keep the clip. The audio in it will not change, but the index will, and the track that matches nothing in September will match in the spring. For the vocabulary around all of this, the glossary is the place to look.