How face recognition search works

Face recognition is a family of techniques that measure a face and compare the measurement with other measurements. Modern systems do not store a picture of your face and hold it up against other pictures. They run each face through a neural network that outputs a fixed-length list of numbers, and then compare lists. Everything a face search engine can and cannot do follows from that one design choice, so this page walks through it in order: how a face is found in a photo, how it becomes numbers, how two sets of numbers are scored, and what gets in the way.

What does a face recognition system actually do?

It answers one narrow question: how similar are these two faces? It does not answer who is this. A name only enters the picture when a human, or a database that a human built, attaches a name to one of the faces. A face search engine like FaceSearch attaches a public web page instead, which is why its results are pages where a similar face appears rather than identities. Keeping that distinction in mind prevents the most common misreading of a result, which is treating a high similarity score as proof that two photos show the same person.

How does a photo become numbers?

Every system follows roughly the same five stages. The names differ between vendors and papers, but the pipeline below is the one described in the research that today's engines are built on, including the 2015 FaceNet paper from Google that popularised the embedding approach.

1. 1

   Detection

   A detector scans the image for face-shaped regions and returns a box around each one. Small, turned or heavily shadowed faces may not be detected at all, and a photo with no detected face cannot be searched.

2. 2

   Landmarking and alignment

   Key points such as the eye centres, nose tip and mouth corners are located. The face is rotated, scaled and cropped so those points sit in standard positions. Alignment is why a slightly tilted selfie still matches a straight-on passport photo.

3. 3

   Normalisation

   Brightness and contrast are evened out and the crop is resized to the fixed input size the network expects, often around 112 or 160 pixels square. Detail that was not in the original cannot be invented here, which is why resolution matters.

4. 4

   Embedding

   A neural network trained on millions of labelled faces turns the aligned crop into a vector: a list of a few hundred numbers (FaceNet used 128) that encode the geometry and texture the network learned to find distinctive. Two photos of the same person should produce vectors that sit close together.

5. 5

   Comparison

   The distance or angle between two vectors becomes a similarity score. In a search engine, the query vector is compared with the stored vectors of every indexed face and the closest ones come back first, each attached to the page it was found on.

What is a face embedding?

An embedding is the vector from stage four: a point in a space with a few hundred dimensions. The network is trained so that photos of the same person land near each other and photos of different people land far apart, whatever the lighting, expression or camera. The numbers have no human meaning individually; there is no entry for nose length or eye colour. What matters is the distance between points.

• It is not a photo. An embedding cannot be turned back into a recognisable image by any ordinary means, and it carries nothing but what the network learned to compare. This is why FaceSearch can state that it keeps no face database built from uploads: the upload is compared, not collected.
• It is model-specific. Embeddings from two different networks cannot be compared with each other. Every engine has its own, which is one reason results differ so much between engines on the same photo.
• It is lossy. Compressing a face into a few hundred numbers throws detail away. Two people whose faces differ only in ways the network did not learn to weigh will land close together.

How is similarity scored, and what is a threshold?

The comparison produces a number, often expressed as a cosine similarity between 0 and 1 or as a distance where smaller is closer. A threshold turns that number into a decision: above it, the system calls the pair a match. Where the threshold sits is a policy choice, not a fact about faces. Set it strict and the system misses genuine matches (false negatives). Set it loose and it accepts strangers (false positives). Verification systems such as phone unlock sit at the strict end. A search engine deliberately sits looser, because its job is to surface candidates for a human to check, and it ranks them rather than deciding.

Why do look-alikes and relatives score high?

Because the embedding encodes what the network found distinctive across millions of training faces, and some people simply share those features. Siblings, parents and children often land close together, identical twins almost on top of each other, and complete strangers occasionally within the match range by chance. The larger the index, the more chances for a coincidence: a search over hundreds of millions of faces will find someone who resembles you even if you have never been photographed in public. The doppelgänger effect page goes into how to tell a look-alike from a real match.

What makes a face harder to match?

Anything that changes the pixels the network sees without changing the person. The table lists the usual culprits, roughly in the order they cause trouble in practice.

Verification, identification and search: what is the difference?

These three jobs use the same embeddings but answer different questions, and conflating them is behind most bad claims about face recognition.

How is a web-scale face index built?

A face search engine runs the same pipeline over images on publicly accessible web pages, at scale. A crawler fetches pages, a detector finds faces in the images on them, and each face becomes an embedding stored with the URL it came from. A query runs the pipeline on your photo and asks the index for the nearest stored vectors. Three consequences follow. Pages behind a login, private profiles and direct messages are never crawled, so they are never in the index. Pages change, so a stored result can point at an image that has since been removed. And a face that appeared online last week may not be indexed yet, because crawling the web is slower than searching it.

What does no match mean?

Only that nothing in the index sat close enough to the query embedding. It does not mean the face is not online. The photo may be behind a login, on a page not yet crawled, too small or too turned to produce a usable embedding, or of a person who has simply never been photographed in public. For the same reason, a clean result on a dating profile is not evidence that the person is real; AI-generated faces have no original to find. A no-match result rules nothing out, and a second search with a different photo is often worth the credit.

How accurate is face recognition?

On good-quality, front-facing photos, the leading algorithms tested by the US National Institute of Standards and Technology (NIST) in its Face Recognition Technology Evaluation make very few errors, and accuracy has improved sharply since deep learning replaced hand-built features around 2014. Accuracy drops as image quality drops, which is why the factors above matter more than any headline number. NIST's 2019 study of demographic effects (NISTIR 8280) also found that error rates differed between demographic groups for most of the algorithms it tested, with false positive rates for some groups between 10 and 100 times higher than for others in some algorithms, while the most accurate algorithms showed much smaller differences. Vendors rarely publish their own figures in a comparable form, so treat any accuracy percentage on a marketing page as unverified; FaceSearch does not publish one for that reason.

How does this apply to FaceSearch?

FaceSearch runs exactly this pipeline over publicly accessible pages. Your upload has its metadata stripped, is re-encoded, and is turned into an embedding that is compared with the index; it is never sold and never used to build a face database. Results come back in rank order with the page each one was found on, and a deep search reaches further across the index at the cost of a deep credit. Because the system cannot know who anyone is, our rules on responsible use treat every result as a lead to verify, never as an identification, and the privacy page sets out what happens to the photo you upload.