Why your skin looks different in every photo

Anyone who has photographed their own face twice in ten minutes knows the problem. In the bathroom, under the warm bulb above the mirror, the skin looks blotchy and tired and the pores are obvious. By a window a few minutes later, the same face looks even and calm. Nothing about the skin changed in those minutes. Everything about the photograph did.

This is an ordinary experience, and it is also a serious measurement problem, because a great deal now rests on the assumption that it is not one. Point a phone at a face, the pitch goes, and receive a number: a score, a grade, an age. The promise treats the photograph as a window, something the skin is simply seen through. It is closer to a reading taken by a cheap instrument in an uncontrolled room, and the first honest thing to say about reading skin from a camera is that the camera is a weak sensor for skin.

That does not mean nothing can be learned from it. It means the intelligence involved is not the kind the pitch implies. It is not a technique for pulling hidden truth out of a selfie. It is the far less glamorous work of removing what is misleading, measuring the little that physics genuinely leaves behind, and being honest about how uncertain the answer remains.

What is actually in a photograph of skin

A photograph of a face is a record of light that left a bulb or the sky, struck skin, bounced off it, passed through a lens, landed on a sensor, and was then processed by software before anyone saw it. Skin is one term in that chain, and not the dominant one.

The light source alone changes everything. A tungsten bulb throws a warm, red-heavy spectrum; an overcast window throws a cool, blue-heavy one; an office ceiling throws something else again. The skin reflects whatever it is given. Then the camera intervenes: automatic white balance tries to guess what the light was and to correct for it, exposure decides how bright the result should be, and compression discards whatever the algorithm judges the eye will not miss. Add the angle of the face, the distance, the lens, the sensor, and whatever is on the skin, and a large share of what separates two photographs of the same face is not the face.

The correction the camera is attempting has a name, and a long literature. Human vision performs it well enough that most people never notice: a white shirt looks white indoors under a bulb and outdoors at noon, though the light reaching the eye is wildly different in each case. This is colour constancy, and as the review by Foster sets out, a quarter century of work has clarified how the effect operates without fully solving how to achieve it for the variable surfaces and illuminations of the real world (Foster, 2011). A camera's automatic white balance is an approximate, fallible guess at a problem that is genuinely hard.

So the first correction to the naive picture is a large one. In an uncontrolled photograph, most of the variance is the room and the device. Skin is the minority signal.

What the physics genuinely allows

None of this makes a camera useless, and it would be a different kind of dishonesty to pretend so. Real, physically grounded signal does survive, and it survives for reasons that can be stated plainly.

Start with what happens to light at the surface. Anderson, describing the optics of photographing skin, separated reflected light into two components. A few percent of the light striking the face bounces straight off the boundary between air and the outermost skin, without ever entering the tissue. This is glare, and it carries information about surface texture and nothing about what lies beneath. The rest enters the skin, scatters among its structures, and comes back out carrying the cues that matter: pigmentation, redness, vessels, what sits within the tissue rather than on it (Anderson, 1991).

The elegant part is that these two components can be told apart, because they behave differently under polarized light. Glare preserves the plane of polarization; the light that has been scattered inside the skin does not. Photograph the face through a polarizer aligned with the illumination and surface detail is enhanced. Cross the polarizers, and Anderson found that wrinkles and surface detail disappear while vasculature and pigmented lesions come into view. The glare can be subtracted, physically, before the image is ever analyzed.

Then there is what the returning light carries. Skin colour is dominated by two pigments that absorb light in different ways: melanin, and the haemoglobin in the blood beneath. Because their absorption differs, their contributions to an image are not hopelessly entangled. Tsumura, Haneishi and Miyake showed that the spatial distributions of melanin and haemoglobin can be separated from an ordinary colour photograph of skin by independent component analysis, on a model with three assumptions: that two pigments drive the colour variation, that their quantities vary independently across the skin, and that they combine linearly in the optical density domain (Tsumura et al., 1999).

This is worth dwelling on, because it is the honest answer to a fair question. Redness and pigment are genuinely separable signals, not marketing. A brown mark and a flushed patch are different physical things, and light knows the difference. What the method requires is that its assumptions roughly hold, and in an uncontrolled photograph under an unknown light, they hold less well. The capability is real. So is its boundary.

One photograph is one noisy observation

Suppose the glare has been handled and the pigments separated. A single photograph is still a single measurement, and the problem underneath it does not go away. The two photographs from the bathroom and the window remain irreconcilable, and nothing in the processing so far says which of them was right.

The difficulty is structural. Many different combinations of illumination and surface can produce exactly the same pixels. A slightly darker skin under slightly brighter light and a slightly lighter skin under slightly dimmer light are, to a sensor, indistinguishable. Recovering the surface from the sensor's response is therefore what statisticians call an inverse problem: the observation does not determine the answer on its own.

The way this is handled is instructive, because it is precisely the shape of honest inference. Brainard and Freeman formulated colour constancy in exactly these terms, using Bayes's rule. One begins with prior knowledge about which illuminants and which surfaces actually occur in the world, combines it with the sensor's response, and obtains not a fact but a posterior distribution: a range of possible answers with probabilities attached. From that distribution an estimate can be drawn. Their paper is unusually candid about the limits of the exercise, noting conditions under which even an optimal estimator returns poor estimates, as when the surfaces in a scene are spectrally biased (Brainard & Freeman, 1997).

That is the honest description of what any system reading skin from a photograph is doing. Not extracting. Estimating, under assumptions, with a quantifiable uncertainty and identifiable circumstances in which it will be wrong.

It is worth noticing which circumstance they name. An estimator of this kind reasons about the light by reasoning about the whole scene, and it does that best when the scene offers a variety of surfaces to compare. A photograph taken at arm's length of a face offers almost none. Nearly every surface in the frame is skin, of one colour, lit by one source. That is close to the condition under which Brainard and Freeman warn the estimate degrades, and it is also a precise description of a selfie. The most common way people photograph their skin is close to the least favourable case for working out what the light was doing.

There is a second, older point layered on top. Every measurement carries error, a matter Bland and Altman put at the centre of how clinical measurements should be understood, and repeating a measurement is how that error is characterized rather than assumed away (Bland & Altman, 1996). A single reading tells you where a noisy needle happened to land. Several readings, taken over time, begin to tell you where the needle is pointing.

This is where a photograph stops being a poor instrument and becomes a useful one. Any one image is a weak observation dominated by circumstance. A sequence of images, taken over weeks, lets the circumstance average out and the underlying state emerge, because the room changes from day to day while the skin changes slowly. The estimate sharpens. The uncertainty around it narrows. What was a guess about today becomes a defensible statement about a trend, which is the same reason a single day of skin tells you so little. One image is noise in the way one day is noise.

Where the line falls

The limits are as concrete as the capabilities, and they follow from the same physics.

Nothing recovers information the sensor never captured. If a highlight is blown to white or a colour channel is clipped, what was there is gone, and no processing restores it. Nothing invents detail finer than the pixels the lens resolved. Nothing, from a single frame, can distinguish a change in the skin from a change in the light, because on that evidence the two are the same event. And no measurement of colour and texture, however careful, converts a correlate into a diagnosis. What can be measured is an estimate of a physical property, not a verdict about health.

Those constraints suggest a simple test. When a system reports a confident number from one uncontrolled photograph, with no interval around it and no account of the lighting it was taken in, the confidence is not evidence of a better model. It is evidence that the confounds have been hidden rather than handled. The tell of a serious instrument is that it tells you how sure it is.

The constraints also suggest something a person can do, which is more useful than anything a model can do afterwards. A confound that is prevented never has to be corrected. Photographs taken in the same light, at the same time of day, from roughly the same distance, on the same device, with the skin bare, differ from one another far more because of the skin than because of the room. That is the whole trick, and it is unglamorous. The comparison, not the single image, is what carries information, and a comparison is only as good as the conditions it holds constant.

How Mela reads a photograph

Mela treats a photograph as what it is: one noisy observation of a slow-moving thing. It asks for consistent conditions where consistency is achievable, because controlling a confound is always cheaper than correcting it. It leans on the physics that genuinely separates signals rather than on the appearance of a face. It holds every reading as an estimate with an uncertainty attached, revises that estimate as images accumulate, and reports the trend rather than the snapshot. When the evidence will not support a statement, the honest output is that it is too early to say, and that is the output given.

None of this replaces a clinician. A photograph, read carefully, describes correlates of skin condition over time. Anything sudden, severe, painful, or persistent, and any lesion or mole that changes, belongs with a doctor and not with a camera.

The intelligence is the humility

A camera is a real sensor. Light that has passed through skin comes back changed, and those changes can be measured, separated, and followed. That is not nothing; it is the basis of a great deal of dermatological imaging.

What a camera is not is an oracle. Between the skin and the number lies a room, a bulb, a lens, and a stack of software, and the honest reading is the one that accounts for all of them and then admits what remains uncertain. The apps that promise a verdict from a single selfie are not solving that problem. They are declining to mention it.

The useful reading is quieter. It says what it saw, it says how confident it is, it waits for more evidence before it changes its mind, and it tells you when the light was bad. Skin looks different in every photograph. A reading worth trusting is one that knows it.