2026-08-30 · 6 min read

Teaching a diffusion model to remember one face

Claudia is the tutor you chat with inside lepochi. She travels — one city a week — and every morning she posts a photo of wherever she is, writes a short journal entry about it, and then messages you about her day in the language you're learning.

All of it runs on one machine in my house. No GPU, no cloud image API. Stable Diffusion on twelve CPU cores, a local vision model to describe what it made, and a local language model to write the journal. The photos never leave the building.

The hard part wasn't generating images. It was generating the same person twice.

Diffusion models have no memory

Every render starts from fresh noise and reconstructs a face from a text description. "Petite Chinese woman with long black hair" describes millions of people, and the model picks a different one each time. Some mornings Claudia had a bob, some mornings she didn't. One week she was in Barcelona and came back European — because I'd never actually said she was Chinese, so the model inferred her appearance from the scenery.

That's not a bug you fix with better prompting. It's the central limitation, and it's why an entire category of tooling exists: IP-Adapter, InstantID, PhotoMaker, LoRA training. All of them answer the same question — how do you carry identity between two runs that share nothing?

I picked PhotoMaker, which fuses reference-face embeddings directly into the text embedding at a trigger word. You give it a few photos of a person and it conditions every render on them.

It took four failed attempts to get one working image. Each failure was a different real bug, and three of them had been quietly sabotaging the pipeline for weeks.

The negative prompt that never ran

I had a carefully written negative prompt — no deformed hands, no text, no watermarks — and after one particularly bad render, guards against the model drifting the character's ethnicity toward whatever country she was visiting.

None of it had ever been applied. Not once.

We were running SDXL-Turbo at --cfg-scale 1.0. Turbo is a distilled model that trades quality for speed; it's how you get an image in eight steps instead of forty. But cfg 1.0 means classifier-free guidance is off, and classifier-free guidance is the mechanism by which a negative prompt exists at all. It works by computing one prediction with your prompt and one without, then pushing away from the second. At cfg 1.0 there is no second prediction to push away from.

So every anti-artifact term I'd written had been decorative. Worse, prompt adherence in general is weak without guidance, which explained a lot of mornings where Claudia ignored the scene entirely.

The fix was to stop using Turbo for this path. PhotoMaker needs real guidance anyway — its own documentation asks for cfg 5.0 — so the identity work and the quality work turned out to be the same change.

Lesson: a distilled fast model isn't a cheaper version of the base model. It has different semantics, and some of your configuration silently stops meaning anything.

The undocumented square-image requirement

With PhotoMaker wired up, every render aborted:

sd::Tensor error: Tensor index out of range: shape=[234, 223, 3, 1]

None of my reference photos were 234×223. I bisected by reference image — all failed. By resolution — all failed. By step count — which was itself a bad test, because PhotoMaker computes the step where identity merging begins as a fraction of total steps, so --steps 1 collapses that to zero and breaks configurations that otherwise work.

The thing that cracked it was testing the library's own bundled example faces. Those worked. So it wasn't the build; it was my images.

Their examples were 1024×1024 and 512×512. Every one of mine was something like 577×592 or 795×677 — ordinary crops. PhotoMaker's preprocessing assumes square reference images, and no documentation says so. Non-square input produces a malformed tensor and kills the render.

Thirty lines of centre-cropping fixed it permanently.

Lesson: when a library works with its own fixtures and not your data, stop bisecting your data and start diffing the shape of it against theirs.

The face restorer that erased the face

Small diffusion renders produce bad eyes. The standard fix is GFPGAN, a face restoration network, and it worked — eyes went from smeared to sharp with proper catchlights.

It was also making every render look like plastic. That "old video game character" sheen: waxy skin, no pores, uncanny smoothness.

GFPGAN rebuilds a face by projecting it toward its own learned prior of what faces look like. That's exactly why it fixes broken eyes. It's also why it homogenises: it pulls every face toward an average one. I had added a component that repaired the symptom of low-quality renders while quietly working against the identity consistency I was trying to achieve.

Once the render quality improved — forty steps with real guidance instead of eight without — the eyes were fine unaided, so I dropped it from that path entirely. If it's ever needed again, CodeFormer has an explicit fidelity dial that GFPGAN lacks.

Lesson: a post-process that "fixes" your output may be fighting a different goal you care about more. Ask what it's optimising toward.

Two failures that were only reporting bugs

Twice I lost hours to problems that were trivial except that the error message was useless.

A render died after exactly forty-five minutes with an empty error string. The cause was a subprocess timeout I'd sized for the old eight-step profile and never raised when the work grew to two hours. A process killed by a signal writes no stderr, so reporting stderr alone reported nothing. Now the failure path prints signal, exit status and elapsed time when stderr is empty.

And a level-of-detail image cache silently served full-size masters no matter what size was requested. The derivatives were being generated correctly — they were just never sent. I'd put the cache in a directory called .lod, and Express's sendFile refuses any path containing a dot-segment by default. It errored, fell through to the static handler, and served the original. My own error handling was designed to degrade quietly, which is exactly what made it invisible.

Lesson: graceful degradation and silent failure are the same code path. Log the difference.

What actually worked

The final pipeline, running unattended at 05:00 every morning:

RealVisXL rather than base SDXL — same architecture, finetuned on

photographs, and the single biggest jump in "is this a photo or a render".

PhotoMaker conditioned on a handful of square-normalised reference

portraits, at forty steps and cfg 5.

A local vision model describes the result, so the tutor can talk about

what's actually in her photo rather than what was requested.

A local language model writes the journal entry from that description,

plus a beginner-level version in each language the app teaches.

Text-to-speech renders the reading aloud, so learners hear it.

About an hour per image on CPU. It runs while the house is asleep.

The trade nobody tells you about

Identity conditioning and prompt adherence pull against each other. PhotoMaker has a style-strength parameter that decides when identity fusion begins — low values follow the reference face faithfully, high values leave more room for the prompt.

At a low setting, Claudia is unmistakably herself and completely ignores the scene: I asked for a rooftop restaurant and got a portrait against a hedge. The reference photos don't only encode a face, they encode context — framing, clothing, the kind of background those photos happened to have. Turn identity up and the scene evaporates.

There's no correct answer, only a dial. Which is a fittingly honest place to end up: after all the debugging, the last problem isn't a bug at all. It's a choice about how much of the character you're willing to trade for the world she stands in.