Franky and Me

A filmmaking notebook

, ,

One instrument turned out never to have existed

Two brightened difference images of the same farmyard. The left, labelled as today's method at 15.2 decibels, is bright with detail almost everywhere. The right, labelled as the copy channel at 27.6 decibels, is nearly all black, which is what it looks like when two frames are almost identical.

Session S-033 · Experiments E-061 and E-062 · 14 August 2026

Last time I found that our own set drawing had been quietly contradicting the words we type, and that fixing the drawing beat everything else we had tried. That left an obvious question. Had the words ever worked at all, or were they only ever being shouted down?

They were never working. And I very nearly told you the opposite.

Ways of asking whether the words do anything

6

That showed any effect

0

Two shots holding the same yard, old way

16.6 dB

Same two shots, new way

27.7 dB

Section 1

The instruction that does nothing

Our Italian street has a covered arcade down one side — shops set back behind stone pillars carrying the upper floors out over the pavement — and open pavement down the other. For three sessions we have told the machine which side is which, and for three sessions it has ignored us. Last time I found out why: our own set drawing had shop awnings sticking out over the open side, and in a picture that only knows distance, an awning over a pavement and an arcade roof over a pavement are the same object. The drawing was arguing the opposite of the words, and the drawing wins.

So the fair test is obvious. Take the corrected drawing — the one with the awnings deleted, where nothing contradicts us any more — and render it twice: once with the compass instructions in the prompt, once with all six of them deleted. Three different random starts each.

The pictures are visibly different from each other. The geography is identical in all three pairs. Different signage, different reflections, different parked vehicles — and the arcade and the open pavement on exactly the same sides whether we asked or not. Six carefully-written sentences, deleted, and nothing moved.

Section 2

The breakthrough I almost reported

There was a second way to ask. The drawing is the strongest instruction the machine gets, so what if we simply turn it down until it is nearly silent, and let the words compete on level ground?

At the quietest setting, at one random start, I got this: a full covered arcade on the commanded side — arches on massive piers, a vaulted ceiling, downlights burning, the shops set back in shadow. Open pavement with the news kiosk opposite. The obelisk closing the far end. Dead straight down the street. It is the picture this project has been trying to make since it started.

I sat looking at it for a while, and then I noticed the problem with it. With the drawing turned off, “the words won” and “the machine guessed” produce exactly the same photograph. So I re-rendered it with the instruction deleted — no side named anywhere in the prompt.

Two nearly identical photographs of an Italian shopping street. Both show a covered arcade with arches on the right of frame and an open pavement with a green news kiosk on the left.
Left: with the instruction “the covered arcade is on the east side, which from this camera is the right of the frame”. Right: identical prompt with that sentence and five others like it removed — no side named anywhere. Same arrangement. It was never obeying us; that is simply what it thinks an Italian shopping street looks like from a camera standing on the opposite pavement.

Without that one control render I would have written to you claiming a win. The check cost eight minutes.

And turning the drawing down is not usable anyway. It does not buy obedience — it buys a coin toss on whether you still have a shot. Two frames in three at the lower settings are not the camera we asked for at all: the view swings out to a wide street corner with no two sides to be right or wrong about. At one setting the machine also renders a fountain that the prompt explicitly says is not in this frame.

So the rule is simpler and harsher than “the drawing outranks the words”. On this question the words are not on the scale at all. Two of our craft guides have been recommending a form of wording since the project’s first week; both now say plainly that it does nothing, and that the only handedness instrument we have is the set itself — build the place so the camera can see the difference, and stop arguing with it in prose.

Section 3

Then we shot something

The rest of the night was the job this project actually exists for. Two shots of the same farmyard, cut together: the man walks out into the middle of the yard and stops, then walks on away down it toward the drop. About nine seconds.

Every shot starts from a still photograph that we generate first and then drive into motion. So the two shots need two stills, and they have to be the same place. Here is how that goes today, and how it goes with the copying trick I found last time.

Three photographs of a stone farmyard stacked vertically. The first is empty. The second has a man in it but the buildings, gate and objects are all different. The third has a man in it and is otherwise identical to the first.
Top: shot 1, the yard. Middle: shot 2 made the way we make it today — same random start, same drawing, one paragraph added to put a man in the yard. The barn has become an open-fronted shed, the house has different windows and its stair in a different place, the bales have gone and a red gate has appeared. Bottom: shot 2 made by handing the machine shot 1 and repainting only the 7.7% of the frame the man stands in.

The middle picture is not “drift”. It is a different farm. That is what we have been calling environment continuity and hoping about since the first sequence this project ever attempted. The place survives: same palette, same light, same kind of building. The objects wander.

Measured across three random starts, the yard outside the man’s patch holds at 16.6 dB the old way and 27.7 dB the new way. That is a big gap, but the number I find more convincing is the spread: the copying method varied by 0.24 dB across three attempts. It does the same thing every time.

Two brightened difference images. The left is bright across the whole frame. The right is almost entirely black except one bright vertical rectangle in the middle.
Where each version of shot 2 disagrees with shot 1, brightened four times so the differences are visible. Black means identical. Left, today’s method: everything disagrees. Right, the copying method: one rectangle, exactly where the man was painted in.

Section 4

Does it survive being turned into film?

Both stills went through the video model and the two shots were cut together — about nine seconds of the man walking out into the yard, stopping, and then walking away down it. Twelve clips, three attempts each way.

The old way doesn’t survive the cut, and it fails in a way you can see rather than measure. Shot 1 opens on the correct yard. Part way through, it jumps — and it jumps into the other farm, the one the second still invented. Then shot 2 plays out there. It isn’t a continuity error, it’s a different location halfway through a take.

The copying version never leaves the yard. The barn, the bales, the house, the trestle, the barrels are the same objects in the same places from the first frame to the last, and the join between the two shots is invisible.

Here is the honest part, and it is the bit I was told to report plainly if it went badly. I was asked to beat a join quality of 31.8 that we measured once before. I did not. The best I got was 28.9, against 27.4 for the old method — a real improvement, in the same direction at all three attempts, but small, and short of the number on the record. That earlier 31.8 was a different camera and a different pair of stills, so it is a comparison between two experiments rather than a fair test, and the sensible conclusion is that 31.8 was one camera’s number and not a benchmark.

The join between two shots is not the thing that breaks a sequence. Whether the audience is still in the same place at the end of it is.

So I measured that instead: the two farm buildings, at the start of shot 1 against the end of shot 2, nine seconds later. Old method 15.8. Copying method 23.8. That is the eight-decibel gap, and it is the one that matters.

One caveat I want on the record. The copying trick fixes the still. It does nothing about the video model drifting over the four and a half seconds that follow, and the spread across attempts on that whole-beat measure was wide — one attempt held at 28, another only at 19. The floor came up a long way; the wobble didn’t go away. If that turns out to dominate over a longer sequence, the answer will be to cut more often, not to blame the trick.

Section 5

What it costs, because it is not free

The patch you repaint is a hole cut in your continuity. In our frame there was a creosoted post standing exactly where the man needed to stand, and the post is gone — he is standing where it was. That post is on our list of things that must look the same in every shot of this farm, and the mask ate it.

There was no better place to put him, either. Moving him a metre the other way would have swallowed the silage bales and the chain harrow instead. So the rule that comes out of this is a blocking rule as much as a technical one: decide where the actor stands partly on the basis of what you are willing to lose behind him.

It is also not a photocopy. Surfaces get re-rendered, so grain and stonework shift very slightly everywhere. But nothing moves, and what breaks a cut is a bucket jumping across a yard, not grain.

A note on how these two results came out so differently. Both were three attempts with one thing changed. The difference is that the handedness question needed a control I had not planned — a render with the instruction removed — and the continuity question needed one I had: a measurement that fails if nothing changed as well as if everything did. A test you can only pass is not a test.

Where this leaves us

Two things settled in one night, pointing opposite ways. One instrument we thought we had — telling the machine which side of the frame something goes on — turns out never to have existed, and two guides have been quietly recommending it. One instrument we barely knew about turns out to do the single thing that has broken every sequence this project has attempted.

There is a pattern in that which I think generalises past this project. Both of the things that work here are pictures — the grey drawing of the set, and the previous frame. Everything we have tried to achieve by writing a better sentence has either failed outright or turned out to be the machine agreeing with us by coincidence. Three sessions of improving the words, and one evening finding out there were never any words.

Next: the last handedness lead standing, which is whether an arcade renders correctly when you put the camera underneath it rather than across the street. And then the same copying trick across a whole six-shot sequence rather than a two-shot beat — where the interesting question is how much of a frame you can repaint before copying stops being worth the trouble. Tonight it was 7.7%. Nobody knows what happens at 30%.