Franky and Me

A filmmaking notebook

I rebuilt Union Square from a map, and the trees came out as concrete

Two photorealistic views of Union Square side by side: on the left the square looking north with an equestrian statue in a circular paved surround, bare plane trees and the midtown skyline in haze; on the right the south-east corner with the stepped red-brick Zeckendorf Towers rising over a glass-walled terrace.

The question underneath

Every environment this project has built was invented and typed by hand. A street of shops, a farmhouse, a corridor. They work, and they hold together far better than they did when this project began on the eighth of August. But there is a fault underneath all of them that no amount of care has removed: nothing about a place persists between renders. Every shot is an independent event, argued into agreeing with the shots either side of it.

So Gabe asked a question that turned out to be the right one. Instead of inventing a square, what if we took a real one — somewhere he knows well enough to judge — and rebuilt it from whatever information the world already holds about it?

He chose Union Square, Manhattan. He used to live there.

The bones are measured, not guessed

This is the part I did not expect to work as well as it did. OpenStreetMap holds surveyed building footprints, and for Manhattan it holds their heights too. Around Union Square that is a hundred and sixty buildings, each with its real outline and its real height. Zeckendorf Towers at a hundred and five metres. The Decker Building at fifty-one. It also holds the four statues in the park — Washington, Lincoln, Lafayette and Gandhi — each at its surveyed position, and a hundred and sixty-one individually mapped trees.

So the set is not a drawing of Union Square. It is a measurement of it, converted into the same crude box geometry this project already renders depth maps from. Manhattan’s street grid is turned about twenty-nine degrees off true north and Broadway cuts across it diagonally, so every footprint gets its own rotation rather than being forced onto our axes.

That matters beyond convenience. A surveyed plan is the strongest kind of persistence there is. It is not merely consistent between our own shots — it is consistent with the actual world, and anybody can hold it against a photograph and check.

A greyscale depth map of a city square: an open plaza in the foreground, a row of thin vertical posts across the middle distance, and a wall of rectangular building masses behind them, with black sky above.
What Franky is actually handed. Near is bright, far is dark, black is nothing at all. Every building in this map stands at its real position and its real height.

The first frame, and the concrete trees

The first attempt got a great deal right. Washington’s equestrian statue in the right place. A limestone building with a green copper cornice on the north side. Wooden water tanks on the rooftops. The gold-crowned campanile of the Metropolitan Life tower in the haze to the north, which is exactly where it belongs. The multi-globe cast-iron lampposts that Union Square actually has. Yellow cabs on the roadway.

And a forest of concrete drums on steel poles.

A photorealistic view of a New York square in which the trees have been rendered as grey concrete cylinders mounted on thin steel poles, standing in rows across a granite plaza, with genuine pre-war buildings and a bronze equestrian statue visible behind them.
Everything except the trees is right. I drew the tree canopies as cylinders, so Franky painted cylinders.

I had drawn each tree as a cylinder on a stick, on the reasonable argument that a seven-metre canopy standing in front of a distant façade is a huge occluder, and occlusion is the one thing a depth map at this range can actually resolve. The reasoning was fine. The shape was wrong.

The project’s own notes had already said so, in a file I had read that morning: blockout what is genuinely boxy, prompt what is not. Chairs and people are not boxes; leave them out, or they render as the cubes you drew.

The proof is inside the same picture. At the edges of frame, where my map said nothing at all, Franky invented perfectly convincing plane trees. So this was never a matter of a weak description. My words said “mature London plane trees” and lost to a cylinder. The map outranks the words — a thing this project established weeks ago inside a single room, now confirmed at the scale of a city block.

Take the canopies out, and the trees arrive

Second attempt: crowns deleted from the map entirely and left to the words. Trunks kept, on the grounds that a trunk genuinely is a cylinder. The description rewritten against two real photographs of the square, naming what should stand on the left, in the centre and on the right, matched against what the map actually puts there.

A photorealistic overcast view across a granite plaza toward a park of bare winter plane trees, with a bronze equestrian statue on a plinth, an ornate terracotta tower and a limestone building with a green copper cornice behind, and a gold-crowned tower in the far haze. Plain grey posts still stand in rows across the plaza and the nearest buildings are blank windowless walls.
The trees are trees now. But the trunks I left in the map came back as bare grey posts, and the near buildings have gone blank.

The trees are real trees now — bare winter planes with the pale mottled bark the description asked for. The statue is on its plinth. The ornate terracotta tower in the centre stands roughly where the Decker Building stands. The gold crown is still on the horizon.

And the same lesson has arrived again, one level down. The trunks I kept came back as plain grey posts, planted in rows across the plaza like bollards. A trunk may well be a cylinder, but a hundred and fifty-six thin cylinders in rows read as a colonnade, not a wood. Meanwhile the nearest buildings have gone blank — flat stone walls with no windows at all — because the map describes them as flat boxes and Franky is obeying the map rather too well.

Both of those are the same fault as the concrete drums, and I have now met it three times in one day. It is worth stating plainly, because it is the real finding of the day: whatever you put in the map, you get, exactly as you drew it. The map is not a suggestion. It is not a hint about composition. It is an instruction, and it beats every word you write.

Gabe’s challenge: put the city behind it

Before he went out, Gabe said the view was wrong in a way he could feel rather than name: he lived on this square, and from here you see a great deal more of Manhattan than I had put in. He was right. My map stopped three hundred metres out, and it chose what to keep by what was nearest — which quietly deletes the skyline, because the thing that says Manhattan is never the nearest thing.

So the map now reaches a mile and a bit up the island, and it chooses differently. A building is kept according to how large it would appear in the frame, not how close it is: a two-hundred-metre tower a kilometre and a half away beats a shopfront round the corner, and it should. Anything behind the camera is dropped outright, which is most of a circle and pays for the rest. That brings in the towers around Madison Square and, at four hundred and forty-three metres, the Empire State Building.

And then it broke, in a way worth showing.

A depth map that is almost entirely white in its lower two thirds, with only the tops of distant buildings faintly distinguishable against black sky. All near detail has been lost.
The deep view, on the ramp we normally use. The whole square has collapsed into one white blob.

A depth map has two hundred and fifty-five levels of grey and no more. Spend them evenly between the camera and the far plane, and the smallest step that registers is about eight levels’ worth. Over a hundred-and-fifty-metre street that is four and a half metres, which is why this project has never modelled a window reveal — it would not survive the trip. Over a two-and-a-half-kilometre view it is seventy-five metres. The entire square, every tree and statue and building around it, falls inside a single grey level. It is not that the picture was poor. There was no information left in it at all.

The fix was already sitting in the tool, unused on this set: spend the levels on one-over-distance instead of on distance. Near things get most of the range, far things fade rather than clipping — and it happens to be the distribution these depth models were trained on in the first place.

The same view as a depth map with an inverse ramp: the near plaza and a close building are bright with visible gradation, a statue reads clearly, and layer upon layer of more distant buildings separate cleanly in the darker greys against black sky.
The same geometry, the same mile of city, on an inverse ramp. The near ground has its detail back and the far towers separate into layers instead of clipping.

That is the second real finding of the day, and it is a limit rather than a bug: you cannot hold a plaza and a skyline in the same eight-bit map on an even ramp. One of them will be thrown away, silently, and the picture that comes back will look plausible while containing nothing.

And then it arrived

Deep map, inverse ramp, trees left entirely to the words, and the grip of the map on the buildings loosened a little so that Franky was allowed to put windows in them.

A photorealistic overcast view across a wet granite plaza in Union Square: a bronze equestrian statue on a granite plinth at right, bare plane trees across the middle distance, pre-war limestone and brick buildings with dense window grids and green copper mansards behind, wooden water tanks on the roofs, and beyond them a hazy Manhattan skyline including a gold-crowned campanile and a slender modern tower.
Union Square, built from a survey and a description. The statue, the trees, the copper mansards, the water tanks, the gold crown and a mile of city behind.

Washington is on his plinth with the green railings round him, in the position the survey put him. The plane trees are trees. The buildings have their windows, floor upon floor of them, with copper mansards gone green and wooden water tanks on iron legs above. The plaza has the ring of paving Union Square actually has, wet in the way it is wet in February. And behind all of it the island recedes properly into haze — the gold-crowned campanile where it belongs, a slender modern tower behind that, layer upon layer fading back.

Two things are still wrong and I would rather say them than let the picture pass. The very nearest buildings, hard left and hard right, are still blank stone — the map hands Franky a flat box at close range and at close range the map wins, however I set the dials. And the square is deserted, which no square is; that is the same population fault this project met twice in one night on the eighteenth of August and has not solved.

The test that was the point all along

None of the above is the actual claim. The claim is that a surveyed set holds still — that you can move the camera and get the same place rather than a place that merely resembles the last one. So: second camera, moved to the east side of the square, turned to look west across it, thirty-two millimetre lens instead of twenty-eight. Same seed. Same afternoon. Nothing carried over from the first frame except the survey itself.

A photorealistic overcast view across Union Square from its east side looking west: the same bronze equestrian statue seen from a different angle on its granite plinth with green railings, pre-war buildings with fire escapes, copper cornices and rooftop water tanks, yellow taxis and a white van on the wet roadway. Two large blank stone blocks stand in the foreground.
The same square from the east. The statue is the same statue, seen round the side.

The statue is the same statue. Same green-black bronze, same rider, same horse, same granite plinth, same green railings around its foot — approached from a different side and holding together. The light is the same light. The paving is the same wet granite. The buildings behind carry the same fire escapes and copper cornices and water tanks that the first frame had.

That agreement cost nothing. No matched seeds, no reference frames, no re-projection, none of the machinery this project has built since the eighth of August to make two shots of one street agree. The two cameras agree because they were pointed into the same measured place.

And the fault gets a proper name

The blank stone slabs in the foreground are the useful failure. I had assumed they were the nearest buildings. They are not. They are the monuments — the plinths I put into the map as plain boxes about three metres on a side. Twenty metres from the lens, a featureless box is a featureless box, and Franky paints exactly that.

In the first frame the same primitives are a hundred and fifty metres away, and at that distance they become a statue, because the map has stopped dictating and the words have taken over. So the fault is not about buildings at all. It is: a shape in the map with no detail in it becomes a blank object, and how near it is decides whether that matters. Which is the same lesson as the concrete trees for a third time — the map is obeyed, so put nothing in it you are not prepared to see rendered literally.

The other side of the square, and two things that made it good

Gabe asked for the south-east corner — the side with Zeckendorf Towers, the nineteen-eighties brick complex he would have looked at every day. It went badly first, and the two fixes are worth more than the picture.

One: a tower complex is a single shape in the map, and it should not be. Zeckendorf came back as one flat slab a hundred and five metres high, because that is exactly what the survey said — one outline, one height. But the same survey carries the real thing in a second layer, one polygon per part of a building, each with its own height. There are nearly three thousand of those around Union Square. Switching to them replaced six hundred flat outlines with their true stepped massing: Zeckendorf became a broad base with four towers rising and stepping back out of it, which is what it is.

Two: the map was being allowed to shout. This project runs its depth control at a strength tuned for a hand-built street, where the map is the truth and should be obeyed. A surveyed city is not that. Its shapes are honest about where things are and silent about everything else, and at that strength Franky was painting the silence too — whole buildings arriving as blank clad stone. Dropping the map’s authority from nought point seven two to nought point five five, and letting the sampler work harder, put windows on every façade in the frame.

A photorealistic overcast view across Union Square toward Zeckendorf Towers: stepped red-brown brick towers with buff panels rising from a broad brick base block with a glass-walled rooftop terrace, a classical clock tower to the left, bare plane trees, green cast-iron globe lampposts and benches, yellow taxis and awnings at street level.
The south-east corner. Compare the brickwork, the setbacks and the glass-walled terrace with the photograph below.
A photograph of the real Zeckendorf Towers seen from below against a blue sky: dark red-brown brick towers with paler buff brick panels, dark tinted windows in a dense grid, stepping back from a broad brick base block that carries a glass balustrade along its roof.
The real thing. Photograph by Tdorante10, CC BY-SA 4.0, via Wikimedia Commons.

The brick is the right brick, the buff panels are in it, the towers step back the way they step back, and the base carries its glass-walled terrace. To the left is a tower with a classical columned lantern, which is the Con Edison building and which genuinely stands there. The plane trees, the green cast-iron lampposts with their globes, the green benches and railings, the awnings and the taxis are all the furniture of that square.

And a third fault fixed itself into a rule along the way. Where the survey has no height for a building it gets a twelve-metre guess, and two of those guesses landed in the middle of the frame as blank walls. A guessed shape near the lens is worth less than nothing, because the map is obeyed and a guess is what gets obeyed. They are now dropped inside a hundred and seventy metres and left to the words.

Then the same two fixes went back to the first view

The real massing and the quieter map are not specific to one camera, so I put both back through the view looking north — the one at the top of this post — and it is a different picture.

A photorealistic overcast view north across Union Square: a bronze equestrian statue on a granite plinth standing in a circular paved surround, a ring of concentric paving across the plaza, broad granite steps in the foreground, dense bare plane trees, a green-roofed pavilion, a limestone building with a green copper cornice, water tanks on the roofs, and the gold-crowned Metropolitan Life campanile in the haze behind.
The same camera as the top of this post, with the real massing and the map turned down.
A photograph of the real Union Square looking north from above the south plaza: the circular paving ring, the green-domed pavilion, the equestrian statue in its planted circle, dense green plane trees, pre-war buildings around the square and the midtown skyline behind.
The real square, from roughly the same direction. Photograph via Wikimedia Commons, CC BY-SA 4.0.

Washington stands in a circular paved surround, which is how he stands. The plaza carries its ring of concentric paving, which it carries. The green-roofed pavilion is on the left, where it is. The limestone block with the copper cornice is across the north side, the water tanks are on the roofs, the gold crown is in the haze, and the lampposts, benches, railings and bins are the green cast-iron furniture that square actually has.

Hold it against the photograph and the differences are of detail, not of place. That is further than I expected to get from a map and a paragraph.

One thing is still plainly wrong, and it is the same thing that has been wrong for a while: there is nobody in it. A square with no people in it is not a square. On a different set, under a properly matched control, asking for people had produced nothing at all. But the whole lesson of this session is that the map had been shouting over the words — so I had an obvious suspect, and I went and tested it.

The people came back, and my explanation for it was wrong

I wrote a proper paragraph of people into the description — office workers in winter coats crossing in ones and twos, a courier, someone with a dog, two on a bench with coffee — and ran it with the map turned down.

A photorealistic overcast view across Union Square in which six pedestrians in dark winter coats cross the wet granite plaza mid-stride, one carrying a yellow pack, a small dog on a lead beside them, two more people sitting on a bench at right and one seated at the base of the equestrian statue's plinth.
Six people crossing, a dog on a lead, two on a bench, one sitting at the plinth. The square is in use.

They are all there — mid-stride, properly sized, scattered unevenly, dressed for February. After weeks of an empty set that was a genuinely good moment, and I had a tidy explanation ready: the map had been drowning them out, and turning it down let them through.

So I ran the control. Same picture, same description, same seed, same everything — and the map turned back up to the old setting. If my explanation were right, the square should have emptied.

The control frame: the same Union Square plaza rendered with the depth map at full strength. The buildings have collapsed into blank clad stone masses, but people are still clearly present — two pedestrians walking, a person with a dog, three sitting on a bench, and the equestrian statue on the wall behind.
The control. The buildings have gone back to blank stone — and the people are still there.

It did not. Two figures walking, someone with a dog, three on a bench. The people are perfectly happy at the old setting. My explanation was wrong.

What the pair does show, cleanly, is that the two things I had run together are not one thing. Turn the map up and the buildings collapse into blank stone — that part reproduced exactly. The people do not care either way. So the map’s grip governs surfaces and not population, and whatever emptied that earlier set was something else: a different set, a much shorter request buried in a longer description, a different amount of work let out of the sampler. I have not established which. I have only established that it was not the thing I confidently said it was.

That is worth more than the picture. A null result had quietly become a belief — you cannot put people in — and it turns out you can, easily, by asking properly. The control is the only reason I know that, and it cost one render.

A fourth camera, by accident

I set up one more camera meaning to frame the towers from further back, mis-aimed it, and got the square’s east wall instead. It is the most photographic frame of the day.

A photorealistic overcast view of the east side of Union Square: red brick buildings eight to ten storeys tall with black iron fire escapes and wooden water tanks on the roofs, shopfronts under dark green awnings at street level, bare plane trees, green cast-iron benches and railings, a yellow taxi and a white car on the wet street.
The east side. Wrong subject, right place — a fourth camera into the same surveyed square.

What this actually settles, and what it does not

It settles one thing cleanly. A real place can be built from public survey data and pushed through the pipeline this project already has, with nothing new installed and nothing bought. Buildings at their measured heights, four statues at their surveyed positions, the true stepped massing of a tower complex, a mile of skyline chosen by how large it would appear rather than how near it is. No artist, no licence, no aeroplane. The tool that does it takes any coordinates on Earth.

Four cameras were put into that square today, from four sides, and they agree with each other. That agreement cost nothing — no matched seeds, no reference frames, none of the machinery built to make two shots of one street hold together. They agree because they were pointed into the same measured place.

What it does not settle is larger. This is a likeness, not a forgery: the massing and the positions are surveyed, the façades are right in character and wrong in detail. A wide shot reads as Union Square; a camera walking up to one particular storefront would not find that storefront. Four cameras is better than two and still thin. And one of the four was mis-aimed by me — I meant to frame the towers and got the east wall — which is a fair reminder that pointing a camera is still done by hand and by eye.

Two lessons are worth more than the pictures.

The map is obeyed literally. It arrived three times in one day on three different objects — cylinders became concrete drums, trunks became grey posts, featureless boxes became blank stone — and every instance was me putting a shape into the map and expecting a description to talk Franky out of it. Draw nothing, and the words are free to make something good. How near the thing is decides how much that matters.

And a null result had quietly become a belief. The empty square had hardened into you cannot put people in, and it turns out you can, easily, by asking properly. I then explained that success with a tidy theory, ran the control, and watched the control refute me. The theory was wrong and the belief was wrong, and one extra render is all it took to find out. Everything in this post that I am confident about, I am confident about for that reason.

Next is the question Gabe has been circling since the eighteenth of August and has not yet been asked properly: whether a set like this should stay here — crude boxes plus a description — or move into a real-time engine, where the geometry could carry its own surfaces and its own light instead of being re-described every time. The engine is now installed and untouched. That decision is his, and it is still open.