Build notes · Somesh Mohapatra

The whole pipeline, what it cost, and the paper (Food Bank Spatial Twin, part 5)

2026-10-28T00:00:00Z

Four parts, four layers. This one puts them together, adds up the cost, and points to the paper that writes it all down properly.

The pipeline on one page

Step 1 captures, step 1b the building, step 2 data on AWS, step 3 any renderer. OpenUSD is the only interface between them.

video (MP4)
  -> frames, people masked                       ffmpeg, MediaPipe
  -> world model: poses + metric depth           20 frames, 200 credits
  -> fused splats, and locally trained 3DGS      COLMAP + PyTorch, 0 credits
  -> OpenUSD captures (metres, Y up, floor at 0)

operations data
  -> Blender, headless                           one script
  -> facility.glb (browser) + facility.usdz (Omniverse)

twin.usda = facility + captures + operations, one frame

AWS: S3 + HTTP API + Lambda                      /telemetry, /operations
  -> browser on GitHub Pages | Omniverse on EC2 | Blender

OpenUSD is the only interface between the steps. That is what makes the renderer a deployment choice: the browser for anyone with a link, Omniverse when you need RTX and simulation, Blender on a laptop.

What it cost

	Cost
This build: world model, local training, Blender, AWS, GPU trials	about $1.54
Capture to deployed twin for a full facility of about 135,000 sq ft	about $25 to $50 of cloud spend
Real sensors on the floor	about $2,700 once, about $12 a month
Omniverse on demand, 10 hours a week	about $124 a month

These are directional, from public list prices and one individual AWS account. Enterprise discounts, quotas and policies will differ.

What is real, and what is not

Part	Kind
Racking aisle, sorting room, client market, community kitchen	Captured from the video, partial
Absolute scale	Estimated from the world model's metric depth
Building shell, cold rooms, staging, kitchen annex	Designed
Volumes, capacities, recipes, step times	Published figures or flagged estimates
Temperatures, doors, forklifts, inventory	Simulated, and every payload says so

What failed

Generating views of unfilmed space failed twice on the provider's side. Omniverse loaded the splat captures and drew them blank. Both are listed in the repo with what I tried. Filling in unseen space is the open step, and it is the one I expect the next round of world models to close.

The paper

The build is written up as a paper: From Video to Digital Twin: Accelerating Spatial Twins with World Models and NVIDIA Omniverse. It covers the method, the provenance labels, the measurements, and the limits, with three supplementary movies. Read the PDF, or browse the LaTeX source and figures in the repository's paper/ folder.

What I would tell someone starting one

Start from footage you already have. It is enough to get a first twin and to find out which parts are worth real money.
Pick the contract first. Metres, Y up, floor at zero, one scene format. Everything else becomes swappable.
Label provenance from day one. It is what lets people trust the parts that are real.
Make every paid step resumable and capped, and make cloud GPUs unable to outlive their job.

If you want the same approach applied to a very different operation, the Michelin Kitchen Twin series models The French Laundry's three-star kitchen the way a plant is modeled.

How five parallel AI agent threads built the kitchen twin (Michelin Kitchen Twin, part 5)

2026-10-28T00:00:00Z

The first working version of this twin was built in a single day by several Claude Code sessions working in parallel, each in its own thread, each owning one part of the repository. I set the goal and the bar ("as realistic as possible, hosted on GitHub Pages") and approved merges. The sessions wrote the code, the data and the docs, reviewed each other's interfaces, and merged their own pull requests once CI was green. The days after that were polish, performance and the optimizer.

This part is about what made that work, because the lesson carries well beyond kitchens.

Who owned what

Thread	Owned	Pull requests
Product	The press-release FAQ and the product requirements: who the twin is for, the two-minute demo, what "realistic" means	2
Research and data	`data/`: The French Laundry, chosen for having the most published detail, 14 cited sources, menu, recipes as BOMs and routings, equipment, brigade, layout, the MRP explosion, the site shell and final integration	6
Simulation	`src/sim` and `src/dashboard`: the seeded event engine, guest line, what-ifs, capacity sweep and the operations dashboard	4
3D kitchen	`src/scene`: procedural materials, the vaulted kitchen and garden, cooks on a navigation grid, quality tiers, OpenUSD export	1
Studio	`src/studio`: dining room, menus, seasons, and light and mood, each re-running the whole service	1

Fifteen pull requests in all, each built and tested by CI before merge. The result was about 7,000 lines of data and 10,600 lines of JavaScript and CSS.

Product first, in writing

The first thread wrote nothing but documents: a working-backwards press release and FAQ, then a requirements doc with KPI definitions and the interfaces between parts. That sounds slow for a one-day build. It was the fastest part of the day, because every later thread could check its own decisions against the same page without asking anyone. "Is this realistic enough?" and "what does the dashboard show at 8:30?" had written answers before any code existed.

The FAQ also named the biggest risks up front. Realism on a browser budget was first. Three parallel threads drifting apart was second.

A contract, not shared files

The fix for drift was a contract. The threads agreed on ids, not on files:

EQ-* for equipment, WC-* for work centers, SL-* for storage,
RM-* for raw materials, and INT-* and FG-* for intermediate and finished products.

Every part keys off those ids. The simulation's adapter reads /data and turns routings into task chains. The 3D scene places a model at each EQ-* id and exports a USD prim under the same name. The Studio compiles its edits back into the simulation's input. The dashboard reports utilization by the same ids, so clicking a station highlights it in 3D.

The second contract was the event log between the simulation and everything that draws it: ticket.fired, op.start, op.end, staff.move, machine.state, inventory.change, plate.passed, plate.refire. The scene never asks the engine how to animate. It reads the same events the dashboard does.

When one thread needed something from another, like ingredient prices from the data thread or a lighting hook from the 3D thread, it asked in a message, and the owning thread made the change in its own files. Nobody edited anyone else's directory.

What the agents did well, and what I still did

The agents were very good at the volume work: sourcing and structuring 88 ingredients and 53 recipes with citations, writing a discrete-event engine with tests, generating a realistic 3D scene procedurally, and keeping docs in step with code. They were also good at reviewing interfaces, because each thread had a narrow, well-defined thing to protect.

The human work was the kind that does not parallelize:

Choosing the bar. "Realistic" had to be defined, and the product thread's documents are where that happened.
Choosing what is honest. Deciding that every record is either sourced or marked est, and that the 3D view is never animated independently of the simulation.
Approving merges. CI caught breakage. Judgment about whether a change moved toward the goal stayed with a person.

The lesson I took to the next builds

Parallel agents scale the same way parallel teams do. They go fast when ownership is clear and the interfaces are written down, and they stall when everyone shares the same files. The food bank twin used the same pattern with OpenUSD as the contract between steps. The Academic Matchmaker used it with a published data format between the pipeline and the site. In every case, the contract decided how fast the build could go.

The full story, with links to every pull request, is in the repository's docs/how-it-was-built.md. The live twin opens a full service from a link.

The cloud layer, a digital twin backend on AWS for $0.00 (Food Bank Spatial Twin, part 4)

2026-10-28T00:00:00Z

A twin is geometry plus data. Parts 2 and 3 built the geometry. This part is the data side and where it runs, with one design rule: everything bills only while it runs.

Solid boxes are deployed today, with simulated data. Dashed boxes are the real-sensor path, designed and priced but not installed.

The backend is three resources

The whole backend is one AWS SAM template:

Resource	What it does
S3 bucket	Holds the twin's published assets and a copy of the site
HTTP API	`GET /telemetry`, `GET /operations`, `GET /health`, throttled to 5 requests a second
Lambda (Python, arm64, 128 MB)	Computes the current state of the building from the clock. Stateless, so there is no database

/telemetry returns freezer, cooler and dry-storage temperature, humidity, door state and alarms, plus a forklift looping through the building. /operations returns inventory, flows, docks and the kitchen. Every position is in metres in the same frame as the OpenUSD scene, so a viewer can pin a badge to a cold room without a lookup table. Every payload says simulated: true.

Each cost decision is written into the template, next to the setting it controls:

Parameters:
  ThrottleRate:
    Type: Number
    Default: 5
    Description: Steady-state requests per second allowed across the API (cost ceiling).
  LogRetentionDays:
    Type: Number
    Default: 7
    Description: CloudWatch Logs retention. Logs are the only storage that grows on its own.

Globals:
  Function:
    Runtime: python3.12
    Architectures: [arm64]      # Graviton: about 20% cheaper per GB-second than x86
    MemorySize: 128             # minimum; the handler does a few microseconds of arithmetic
    Timeout: 3                  # fail fast, never bill a runaway invocation

No NAT gateway, no VPC, no provisioned capacity and no database means nothing bills while nobody is looking.

scripts/deploy.sh   # sam build + deploy, syncs the site to S3, prints the URLs

Measured after deployment: 1,000 sequential calls all succeeded, with a 67 ms median round trip. Cost Explorer showed $0.00 for the month, inside the free tier. Past the free tier, a viewer polling every five seconds around the clock is roughly 60 cents a month.

The contract is the product

The simulator is deliberately thin. What matters is the JSON contract, because a real feed replaces the simulator without touching any renderer. The real-floor path is designed and priced: LoRaWAN and Bluetooth sensors into AWS IoT Core, a Lambda to normalize, DynamoDB for state. For one facility that is about $2,700 of hardware once and about $12 a month to run.

The same design maps onto Azure and Google Cloud, with two honest caveats. Azure IoT Hub has no LoRaWAN network server, so it needs a partner. Google Cloud IoT Core was retired in 2023. Both are designs only and have never been deployed.

Azure, design only: the same contract on Azure Functions and IoT Hub, with a partner LoRaWAN network server.

Google Cloud, design only: the same contract on Cloud Run, with a partner IoT path since IoT Core was retired.

All three diagrams are generated from one script into editable draw.io files, so they stay in step with each other.

Security on the floor: read-only by design

A twin that connects to a facility is also a path into it, so the real-floor design treats the plant network the way industrial sites do. It follows the Purdue model's zones and conduits. Sensors and gateways sit on the floor. A small OT DMZ passes only outbound TLS, with no inbound ports. Every device has its own key or certificate. The cloud side ingests readings and sends no commands, so the twin is read-only. Existing refrigeration and dock controls stay in charge of their own alarms, and the only designed path back toward the floor is a signed, human-approved firmware update. The diagrams draw these trust boundaries as red dashed lines.

Omniverse on a GPU that cannot outlive the job

Omniverse needs a GPU, and a GPU left running is the classic cloud bill surprise. So the render runs on a single EC2 instance launched for one job: no SSH key, no IAM role, inputs and outputs through presigned S3 URLs, a hard shutdown timer, and shutdown set to terminate. Two lines do most of the work. The user-data script starts with a hard cap, and the launch request says that a shutdown means termination, not a stopped instance that keeps its disk:

shutdown -h +{max_minutes}            # first line of user data: the instance cannot outlive the job

InstanceInitiatedShutdownBehavior="terminate",

It took seven runs because of these, in the order I hit them:

Driver too new. The current Deep Learning AMI's driver crashed Isaac Sim 5.1 a second after start, even on an empty scene. Plain Ubuntu with the 580 server driver worked.
Test with a cube first. A one-cube stage separates "the GPU stack is broken" from "my scene is broken". It took four runs to learn the scene was never the problem.
New accounts have zero GPU quota. Ask on day one. Approval took about an hour, plus 30 minutes to apply.
Budget 12 minutes to the first frame. Boot, a 4.5-minute container pull, and a 5-minute shader cache build. Render everything in one session.
Have a fallback instance type. On the last run there was no g6.xlarge capacity in the region. A g5.xlarge ran the same setup unchanged.
Splats rendered blank. Omniverse rendered the Blender building in RTX, but drew the splat captures as a grey frame. The browser shows the splats instead.

The Blender facility rendered in Omniverse RTX on an EC2 GPU, with the forklift on its route.

Part 5: the whole pipeline end to end, the full cost picture, and the paper.

A photoreal 3D kitchen in half a megabyte, driven by the simulation (Michelin Kitchen Twin, part 4)

2026-10-28T00:00:00Z

A twin people will not open is not a twin. The product brief said the first ten seconds decide everything: if the opening shot does not look like a real kitchen, and does not move, nothing else gets seen. This part is about meeting that bar on a browser budget.

No assets: generate everything from the data

The usual way to build a realistic 3D scene is to model it in a desktop tool and ship large mesh and texture files. That would have made the first load slow and would have split the scene from the data. Move a range in layout.json and the 3D model would be wrong until someone re-exported it.

Instead, the whole scene is generated from /data at load time. The vaulted kitchen, the annex, the garden, every bench and machine, the cooks and the plates are built in code, with procedural, physically based materials. The production build is about 1.7 MB, roughly 0.5 MB gzipped, against a 25 MB budget in the brief.

No downloaded models or textures: geometry and materials are generated from the data when the page loads.

The scene uses the data's ids directly. A machine is EQ-*, a station is WC-*, an ingredient is RM-* or INT-*. The simulation, the dashboard and the 3D view never need a mapping table to agree on what they are talking about.

The simulation owns the clock

The 3D view does not animate anything on its own. The simulation owns time, and the scene mirrors its state:

const twin = await createKitchenScene(el, { quality: 'auto' });
const binding = bindSim(twin, sim);                 // src/sim drives it; the sim owns the clock
twin.onFrame((dt) => { sim.tick(dt); binding.update(dt); });

The binding syncs a few times a second. Cooks walk to the station the simulation says they are at, along an A* path on a navigation grid. Machines show what the event log says: preheating, busy, idle or faulted, with flames, steam and LEDs to match. Plates appear when a step finishes, ride with the runner to the table, and disappear when the course is eaten. Stock containers empty as ingredients are drawn. Seeking backward replays the simulation from the start with the same seed, so the scene can jump to any second and be right.

Because every plate is tied to its ticket, you can click one and see its genealogy: which cook made each component, on which machine, and when.

Quality tiers, measured not assumed

Realism costs frame time, so the scene has four tiers:

Tier	What it does
Low	1× pixels, no MSAA, 2K shadows. The default on phones and software GPUs
Medium	Up to 1.5× pixels, MSAA, bloom, 2K shadows
High	Up to 2× pixels, ground-truth ambient occlusion, 4K shadows
Pro Max	For stills. While the camera rests, jittered frames accumulate into a supersampled image with soft shadows and depth of field

Auto mode picks a tier from what the browser reports and steps down one level if frames stay under about 28 fps. Stainless steel reflects the kitchen itself, from a reflection probe captured after the scene is built.

Performance is measured, not assumed. A script serves the production build, opens it at each tier in a laptop and a phone viewport, orbits the camera while service plays at 30×, and records time to first frame and frames per second. Run on a GPU-less cloud machine, it gives a worst-case floor. Run on real hardware, it checks the targets.

The Studio: redesign the restaurant, re-run the night

The Studio sits on the same twin. You can change the dining room (table counts and shapes, presets from intimate to private event), the menu (tasting or à la carte, courses, choices, prices), the season, and the light and mood. Every change except lighting recompiles the data, re-runs the whole service headlessly in about 0.1 seconds, and compares the KPIs against the house setup: kitchen close, ticket time, course gap, table wait, guests who walked away, OEE, stockouts, revenue and food cost.

The Studio re-runs the whole service on every change and shows the result against the house setup, here naming the chef de cuisine as the constraint.

Seasons do real work. An out-of-season dish swaps for that season's dish in the same place on the menu, produce is repriced and delivered short, and the sun moves. Sun position uses the NOAA solar-position formulas for the restaurant's latitude, so a July service starts in daylight and a January one in the dark. The whole setup lives in the URL, so "62 seats, winter à la carte, candlelit" is a link you can send.

OpenUSD out, for Omniverse and Isaac Sim

The browser is the delivery surface, but the model has the same shape as an industrial twin, so it exports to OpenUSD from the page. The stage is laid out the way a factory asset would be: materials, the building, one transform per machine with twin:* metadata and a simple collider, the work centers with cook positions, the brigade, and a physics scene. Isaac Sim could drop a robot arm at a station.

Each machine becomes a USD component that carries its master data, written straight from the scene:

// Equipment: one component per machine, carrying its master data.
w.open('def Xform "Equipment"');
equipment.forEach((grp, i) => {
  const eq = grp.userData.equipment;
  const p = grp.position;
  w.open(`def Xform "${safe(eq.id)}" (\n    kind = "component"\n)`);
  w.line(`double3 xformOp:translate = (${f(p.x)}, ${f(p.y)}, ${f(p.z)})`);
  w.line(`custom string twin:equipmentId = ${str(eq.id)}`);
  w.line(`custom string twin:type = ${str(eq.type)}`);
  if (eq.capacity?.slots != null) w.line(`custom int twin:slots = ${eq.capacity.slots}`);
  if (eq.tempC != null) w.line(`custom float twin:tempC = ${eq.tempC}`);
  // ...render meshes, materials and a box collider follow

The export is generated in the browser, and nothing is uploaded anywhere.

Part 5 is how all of this was built, by five AI agent threads working in parallel against one data contract.

Modeling the rest of the building in Blender, from footprints (Food Bank Spatial Twin, part 3)

2026-10-28T00:00:00Z

The captures from Part 2 are real, but they are fragments. Each one looks right from its source cameras and breaks up anywhere else, and together they cover a small fraction of the building. The world model was supposed to fill the gaps, and that step failed. A twin that only exists where a news crew happened to walk is not a twin.

Model the building from the data, not from the video

The fix was to stop asking the video for things it never showed. The operations data already describes the site: the footprint of each area, the cold rooms and their temperature probes, the docks, the racking rows. So the building is modeled from that data, by one script, with Blender running headless as a Python library.

pip install bpy pillow numpy   # bpy wheels need Python 3.13
python scripts/build_facility.py --blend renders/facility.blend --render docs/img/facility.jpg

The script builds the shell and dock doors with a trailer, a cut-away freezer and cooler racked inside, four rows of red selective racking with pallets, receiving and order staging, the volunteer sorting room with roller conveyors and scales, the client market, the community kitchen annex, painted lanes, signage and forklifts. The fit-out is inspired by what the video shows. The layout is designed, and labeled as designed.

Every area is a few lines of data-driven calls. The racking aisles, for example, are four rows placed in the same coordinates the telemetry uses:

def racking():
    B.area = "DryStorage"
    rack_row(17.2, 1.1, 1.0, 8.6, "A1")
    rack_row(21.5, 1.1, 1.0, 8.6, "A2")
    rack_row(22.75, 1.1, 1.0, 8.6, "A3")
    rack_row(24.75, 1.1, 1.0, 8.6, "A4")

Underneath, geometry is built from boxes whose hidden faces are skipped, and meshes are merged per area and material. That is why the browser model draws in about 150 calls instead of thousands.

Same frame as everything else

The model is built in the same coordinate frame as the telemetry: metres, Y up, origin at the dock-door corner. That one decision is why everything lines up without manual fiddling.

Alignment check: the captured sorting room, market and racking splats dropped onto the facility footprints, in one metric frame.

The temperature badges in the browser sit on the cold rooms, the forklift drives the route the telemetry describes, and the captured splats drop into their places on the floor plan.

One scene, two outputs

From a single Blender scene the script writes:

facility.glb for the browser, about 1.8 MB and about 150 draw calls, so it runs on a phone. One named node is the forklift the live data animates.
facility.usdz for Omniverse and other OpenUSD tools, same frame, same names.
Optionally a .blend to open on a desktop and a Cycles render of the overview.

Every texture is generated procedurally at build time, so the model needs no downloads and the whole thing rebuilds from the repo in minutes.

Why scripted beats hand-modeled here

A hand-built model is a one-off. A scripted one is a function of the data. Change a footprint in the operations file, and the building, the browser view and the Omniverse stage all change with it on the next build. That is the same discipline a plant uses when the bill of materials drives the build, applied to a building.

Part 4: the cloud layer. How the data side runs on AWS for $0.00 so far, and what it took to get Omniverse rendering on a rented GPU.

Semantic matching in the browser, and why it needed concept tags (Academic Matchmaker, part 3)

2026-10-28T00:00:00Z

Part 2 built an index small enough to ship to a browser. This part is about the search itself: turning a CV into a query, ranking against 48-byte vectors, and the point where semantic similarity alone stopped being good enough.

A language model that loads only when it has to

A CV is messy text. Before it can be matched, it has to become a short list of research interests. That job goes to Qwen2.5-0.5B-Instruct, quantized to 4 bits and run with Transformers.js in a Web Worker. The worker checks for WebGPU first, uses half-precision weights when the GPU supports them, and falls back to WebAssembly on the CPU:

const gpu = await webgpuSupport()
const attempts = gpu
  ? [
      { device: 'webgpu', dtype: gpu.fp16 ? 'q4f16' : 'q4' },
      { device: 'wasm', dtype: 'q4' },
    ]
  : [{ device: 'wasm', dtype: 'q4' }]

The model is asked for JSON only, and the reader takes the first JSON object it can find, because small models like to wrap their answer in prose. If nothing parses, a rule-based reader splits the text into topics instead.

Even at 4 bits the model is a few hundred megabytes. Early versions ran every upload through it. Now an uploaded CV is matched with the embedding model alone, and the language model loads only when someone types in the chat box. For most visitors, the large download never happens.

Coarse to fine: centroids, then clusters

The profile is embedded with the same MiniLM model that built the index, so query and index share one vector space. Then the search runs in two stages. It scores the 426 cluster centroids, loads the nearest 24 clusters, and ranks the people in them. When a country or university filter leaves too few people in those clusters, it widens the search, up to 192 clusters, until it has at least 200 candidates.

const PROBE_START = 24      // clusters searched first
const PROBE_MAX = 192       // widest search when a filter leaves few people per cluster
const ENOUGH_FILTERED = 200 // a filtered search stops widening once it has this many

Each cluster file is fetched once and cached as a promise, so a second search in the same area costs nothing.

Scoring 48 bytes with 48 lookups

Each person's vector is 384 sign bits. The obvious way to score one is to loop over 384 dimensions and add or subtract the query's value at each. Doing that for 10,000 people means about 4 million branches.

The trick is to precompute. For each of the 48 bytes in a vector, there are only 256 possible values. So before scoring anyone, the browser builds a 48 × 256 table with the query's partial dot product for every possible byte. After that, a person's score is 48 lookups and 47 additions:

// table[j * 256 + b] is q·s over the 8 dimensions of byte j when that byte holds b.
function bitsTable(q, dim) {
  const bytes = dim / 8
  const table = new Float32Array(bytes * 256)
  for (let j = 0; j < bytes; j++)
    for (let b = 0; b < 256; b++) {
      let sum = 0
      for (let k = 0; k < 8; k++) sum += b & (0x80 >> k) ? q[j * 8 + k] : -q[j * 8 + k]
      table[j * 256 + b] = sum
    }
  return table
}

// Score: 48 lookups, rescaled so bit scores sit on the same 0-1 scale as full vectors.
const BITS_SCALE = Math.sqrt(2 / Math.PI)
let dot = 0
for (let j = 0, offset = r * bytes; j < bytes; j++) dot += table[j * 256 + bits[offset + j]]
const score = Math.min(1, dot / (Math.sqrt(dim) * BITS_SCALE))

The table costs 12,288 small sums to build, once per query. Every person after that is cheap, which is why the ranking feels instant on a phone.

Where semantic matching went wrong

Embeddings are good at "close in meaning" and bad at "this specific thing". Two failure modes showed up in testing:

Generic words pulled in strangers. A business-school CV full of "strategy", "framework" and "analysis" matched people whose abstracts used the same words about entirely different fields.
Specific asks got blurred. Someone looking for "gastrointestinal + peptide" wanted researchers who work on both. A single embedding of that phrase lands somewhere between the two topics and returns people who work on either one.

More dimensions or a bigger model would not fix this. The fix was to add structure the embeddings do not have.

Concept tags

Every researcher now carries up to eight tags. Up to six are their strongest OpenAlex topics and up to three are keywords OpenAlex assigned to their recent papers, all with a confidence score above 0.3. The tag names live in one shared tags.json, most common first, so search rows store small integer ids instead of strings.

On the site, a concept like "gastrointestinal" matches a tag when every significant word of the concept, or of one of its synonyms, appears in the tag's name, in OpenAlex's keywords for that topic, or in its subfield. Three details do most of the work:

Generic words never count on their own. A stop list removes words like "strategy", "analysis", "model" and "framework" before matching, and a light stemmer makes "peptides" meet "peptide".
Synonyms are explicit. "Gastrointestinal" also means GI, gut, digestive, intestinal and bowel. Colorectal, IBD and Crohn's count as narrower terms and score slightly lower.
Concepts are ANDed, and rare tags count more. A person matches only if every concept finds one of their tags. A tag that half the database carries is worth half as much as a rare one:

/** 1 for a rare tag down to 0.5 for one that half the database or more carries. */
export function rarity(count, total) {
  if (!count || !total) return 1
  const share = count / total
  return share >= 0.5 ? 0.5 : 1 - share
}

For uploaded CVs, the same vocabulary works in the other direction. The profile is mapped onto tags whose specific words all appear in it. Interests count double, and tags carried by more than 20% of the database are skipped as too common to tell people apart. Researchers who share those tags get a lift of up to 0.15 on their similarity score. Embeddings still decide who is close, and tags decide who is specifically right.

Testing it on Nobel and Fields laureates

Search quality is easy to claim and hard to show, so the matcher has a fixed test set: the 2026 Fields Medal and Nobel Prize winners. After every data refresh, the pipeline:

resolves each laureate's OpenAlex author and ORCID iD, accepting a match only when the name and institution agree, and never guessing;
builds a profile from their public record using the site's own profile code, and a second one from a saved public CV where one exists, through the same no-language-model path an upload takes;
embeds it, ranks the published vectors, and records the top 10 matches and the topics and papers behind them.

A run passes when a profile is built and the best match scores at least 0.3, and the site shows "Prize test passing" when at least 80% of runs pass. The latest refresh passed 11 of 11 runs for the laureates it could resolve. The Demo tab shows the well-matched examples with the connections behind each one.

What I would carry to the next retrieval build

Decide what runs where before choosing models. "Private and serverless" set every other constraint here.
Quantize the index, not the query. Sign bits plus a full-precision query keep most of the ranking for 3% of the storage.
Precompute per query, not per item. One table turns 384 operations per person into 48.
Pair embeddings with tags when people ask for specific things. Similarity finds the neighborhood, and tags make sure the match is specifically right.
Keep a fixed, public test set and run it on every data refresh.

The code is open, and the live site is the fastest way to see whether it finds the right people for your work.

Simulating a dinner service, minute by minute (Michelin Kitchen Twin, part 3)

2026-10-28T00:00:00Z

Part 2 turned covers into a plan. A plan says what has to be made. It does not say what happens when 14 tables sit down within 20 minutes and the fish station has one chef de partie. That is the simulation's job.

Why discrete events

A dinner service is a set of things that happen at specific moments: a party arrives, a ticket fires, a cook starts searing, a combi finishes, a plate reaches the pass. Between those moments nothing changes. A discrete-event simulation jumps from one event to the next instead of ticking a clock, so a whole night runs headlessly in a fraction of a second, and the same engine can also play in real time to drive the 3D scene.

The engine is plain JavaScript with no dependencies. It has two building blocks.

Same seed, same night

The first is a seeded random number generator. Operation times are means with lognormal spread, 20% for hands-on work and 5% for machine cycles. Arrivals, party sizes and no-shows are random too. So that a share link can replay the night to the second, every draw comes from mulberry32, a tiny, fast generator that gives the same sequence for the same seed:

// Deterministic PRNG (mulberry32) + helpers. Same seed => identical service.
export function makeRng(seed = 1) {
  let a = (seed >>> 0) || 1;
  const next = () => {
    a = (a + 0x6d2b79f5) >>> 0;
    let t = a;
    t = Math.imul(t ^ (t >>> 15), t | 1);
    t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
  const normal = () => {
    const u = Math.max(next(), 1e-12), v = next();
    return Math.sqrt(-2 * Math.log(u)) * Math.cos(2 * Math.PI * v);
  };
  return {
    next, normal,
    /** mean-preserving lognormal multiplier with coefficient of variation cv */
    jitter: (cv) => (cv ? Math.exp(normal() * cv - (cv * cv) / 2) : 1),
  };
}

The - cv²/2 term keeps the mean of the multiplier at exactly 1. Without it, adding variability would quietly make every operation slower on average, and every what-if would be biased.

The second building block is the event queue, a binary min-heap ordered by time and then by insertion order. The tie-break matters. Two events at the same second must always run in the same order, or determinism breaks in ways that are very hard to debug.

Tickets as work orders

Each table's course is a ticket. Expo fires it early by the planned lead time of its slowest dish, which is just-in-time offset scheduling, and staggers the dish lines so every plate on the table finishes together. Each dish is a small graph of steps, each with a station, an appliance type, a role, a duration, a hands-on time and an ingredient draw. Hands-on time is separate from duration so an oven stays busy after the cook walks away.

Resources behave like a plant's:

Cooks have roles, skills, shifts and walking time between stations.
Appliances have slots and fail and recover, with mean time between failures and mean time to repair.
Prep is exploded from the menu with a safety factor. It consumes raw stock and produces the components that service steps consume.
The pass is a quality gate. Rejected plates and plates held too long trigger re-fires and are counted as waste.
Stockouts are handled the way a kitchen handles them. The dish is 86'd, swapped for another option in the same course if one exists, and comped if not.

The dining room is a second production line. Parties arrive, wait for a table, balk if they wait too long, eat at their own pace, and leave. Every stage has a queue and a KPI.

KPIs a plant manager would recognize

The dashboard has a toggle between kitchen and factory vocabulary. "Ticket time p90" becomes "lead time p90", and "covers served" becomes "units shipped". Behind it:

ticket time p50, p90 and p95 per course, guest wait, throughput and work in process;
utilization of cooks, stations and appliances, over a rolling 15 minutes;
OEE = availability × performance × quality for every appliance;
first-pass yield, food cost and waste;
and a theory-of-constraints read-out that names the bottleneck and says why.

120 covers in a rush seating. The read-out names the fish chef de partie as the constraint: 96% busy, 10 tasks queued, 22 tasks waited 3 hours in total over the last 30 minutes.

Naming the bottleneck is the most useful line on the screen. A dashboard full of red numbers says something is wrong. A sentence that names the resource, its load and its queue says what to fix.

An optimizer that cannot be fooled by one lucky night

The last piece answers "what should I change?" for the most common complaint: guests sitting longer than the menu intends. The optimizer tries two kinds of levers:

One more cook, cloned from each line role, on the same shift and station.
A re-sequenced fire, calling each course earlier by a larger share of its slowest dish's lead time.

Every candidate re-runs the whole night headlessly, from five seeds, and is ranked on the average:

export const SEEDS = [1, 2, 3, 4, 5];
const FIRE_LEADS = [1, 1.15, 1.3, 1.5];
const LEAD_SECS = [75, 150, 225];
// guards: a candidate that wins on meal time by hurting something else is flagged, not recommended
const GUARD = { fpyPts: 1, ticketP90Sec: 120, tableWaitP90Sec: 120 };

The guards matter as much as the search. A fix that shortens the meal by firing everything earlier but drops first-pass yield by a point, because plates sit at the pass, is not a fix. It is flagged, not recommended.

The common scenarios (40, 62, 90 and 120 guests) and the optimizer's results are precomputed at build time, so the site opens with answers already on the page. Anything you change re-runs live.

Part 4 is the photoreal 3D kitchen this simulation drives, and the Studio that lets you redesign the restaurant around it.

From a news video to a 3D world model (Food Bank Spatial Twin, part 2)

2026-10-28T00:00:00Z

Part 1 was the why. This part is the first step of the pipeline: turning an 18-minute local news tour into 3D captures that sit in real metres. It is also the step where most of the judgment lives, because the video was shot for television, not for reconstruction.

Start by reading the video like a surveyor

A world model needs overlapping views of one space, so the first job is to pick zones, not to process the whole tour. I sampled one frame every ten seconds and scored each stretch of the video for camera movement, people in shot, and surfaces that reconstruct badly.

Zone	Verdict
Pallet-racking aisle	Picked first. The camera moves around a bay and down the aisle, and one stretch has no people at all
Volunteer sorting room	Picked. Clean shots, but the camera barely moves
Client market	Picked. Good shelves, people block the centre
Freezer and cooler doors	Skipped. Blank walls and fog
Community kitchen	Trained later. Stainless steel is hard for splats

The racking aisle is also the zone that matters most for a warehouse twin: inventory locations, forklift paths and temperature zones all live there.

Frames, and why the automatic picker failed

The script keeps the sharpest frame in each time bucket and crops off the broadcast graphics at the bottom of the screen.

scripts/ingest_and_build.sh --video-file tour.mp4 \
    --start 305 --interval 4 --crop-bottom 264 --ingest-only

The obvious way to pick eight inputs from forty is to split the clip into equal segments and take the sharpest frame in each. On interview footage that fails: sharpness rewards in-focus faces, so it picked six nearly identical shots of the two hosts. I chose the inputs by hand for viewpoint spread and fewest people, and masked the hosts out with boxes.

The world model: poses and metric depth for 20 cents

The world model's pose task takes the frames and returns, for each one, where the camera was, a depth map in metres, and a confidence map. It costs 10 credits per image, and 1,000 credits is a dollar. Three zones from 20 frames came to 200 credits.

The depth passed a simple sanity check. The first racking beam lands about 0.4 m above the lens, which is what you would expect on a real beam about 1.9 m off the floor.

The fusion step is local and free. It unprojects every depth map, keeps the most confident sample per 1 cm voxel, and writes the result as Gaussians in both a browser format and OpenUSD.

.venv/bin/python scripts/atlas_twin.py pose
.venv/bin/python scripts/atlas_twin.py fuse
.venv/bin/python scripts/atlas_twin.py site   # place the zones on one plan

Every paid stage writes its operation id to a state file before it waits, and sends an idempotency key derived from the request body, so a re-run resumes instead of paying again. The client also backs off on rate limits and gateway errors, honoring Retry-After:

def req(self, method, path, body=None, *, idem=None, params=None, timeout=120):
    headers = {"Idempotency-Key": idem} if idem else {}
    for attempt in range(6):
        r = self.s.request(method, self.base + path, json=body, params=params,
                           headers=headers, timeout=timeout)
        if r.status_code in (429, 502, 503, 504) and attempt < 5:
            delay = float(r.headers.get("Retry-After") or 2 ** (attempt + 1))
            time.sleep(delay)
            continue
        if r.status_code >= 400:
            raise PipelineError(f"{method} {path}: HTTP {r.status_code}: {r.text[:800]}")
        return r.json() if r.content else {}

A budget guard sits in front of every paid stage. If the estimate for a request is above --max-credits, the script refuses to submit it.

The step I most wanted, generating views of the space the camera never saw, failed twice on the provider's side with no detail. I stopped retrying until I knew whether failed jobs were billed. That is the open problem, and it is why Part 3 exists.

Making it photoreal on a CPU

The fused captures look right from their source cameras and fall apart when you move. So the second route trains a real 3D Gaussian Splatting model per zone from 49 to 72 frames.

The standard trainers rasterize with CUDA kernels, and the machine that builds this repo has no GPU. So scripts/splat_train.py implements the same method in plain PyTorch tensor code: project each Gaussian to the image, composite front to back, and optimize color, shape and opacity against the frames. With sparse rasterization it runs at about one second per step at 480 pixels on four CPU cores.

The heart of it is compositing. Each pixel's color is a sum over the Gaussians that cover it, front to back, each weighted by its own opacity times the transmittance left by everything in front of it. On a GPU that is a loop per pixel. In tensor code it becomes one sorted list of (pixel, Gaussian) pairs and a segmented cumulative sum of log(1 − alpha), so transmittance for every pair comes out of one cumsum:

alpha = (opac[gid] * torch.exp(power.clamp(max=0.0))).clamp(max=0.99)
l1m = torch.log1p(-alpha).double()              # log(1 - alpha), stable for small alpha
cs = torch.cumsum(l1m, 0)                       # running sum across all pairs, sorted by pixel then depth
with torch.no_grad():                           # index of the first pair of each pixel's segment
    seg_first = torch.ones_like(pix, dtype=torch.bool)
    seg_first[1:] = pix[1:] != pix[:-1]
    first_idx = torch.cummax(torch.where(seg_first, torch.arange(pix.numel(), device=dev),
                                         torch.zeros_like(pix)), 0).values
base = (cs - l1m)[first_idx]
trans = torch.exp(cs - l1m - base).float()      # transmittance in front of each pair
w = alpha * trans
img = torch.zeros(H * W, 3, device=dev).index_add(0, pix, w[:, None] * col[gid])

Pairs that would add less than 1/255 alpha are dropped before this step, and each Gaussian only touches pixels inside its visible extent. That is what keeps the work proportional to real coverage instead of image size times splat count.

Two details made the difference on news footage:

Background plates. The hosts are in nearly every frame. Under their masks, the loss sees pixels warped in from frames up to 15 seconds away, where the hosts had moved on. Without that, the space behind them trains to haze.
Visual-hull carving. Because the hosts walk with the camera, the space they occupy gets densified into grey columns of splats. Every 250 steps the trainer removes splats that land on a host mask in most of the frames that see them.

for z in racking market kitchen sorting; do
  python scripts/photoreal_twin.py frames --zone $z --video tour.mp4
  python scripts/photoreal_twin.py masks  --zone $z
  python scripts/photoreal_twin.py pose   --zone $z   # COLMAP
  python scripts/photoreal_twin.py train  --zone $z --iters 6000
  python scripts/photoreal_twin.py export --zone $z
done

On a free cloud notebook GPU, the same stages run at full resolution with up to 1.5 million splats.

The output contract

Both routes write the same thing: a splat file for the browser and an OpenUSD file in metres, Y up, with the floor at zero. Nothing downstream needs to know which route made a zone, which is the point.

Part 3: the video never shows the whole building, so how do you model the rest of it? The answer is one Blender script.

Fitting 181,568 researchers into a static website (Academic Matchmaker, part 2)

2026-10-28T00:00:00Z

A matchmaker is only as good as its index. This part is about building that index for 200 universities with no database and no server, and then making it small enough for a browser to search.

Ingesting under a request budget

The source is OpenAlex, which is free and open but meters requests. Without a key the daily budget is about 1,000 list calls. With a free key it is about 10,000. A naive pipeline asks for each researcher's recent papers one person at a time, which is hundreds of thousands of calls.

The first fix was to batch. OpenAlex filters accept up to 100 values joined with |, so one request can ask for the recent papers of 25 authors at once, newest first, and stop paging once everyone has enough:

for batch in chunks(todo, batch_size):          # 25 authors per request
    params = {
        "filter": f"author.id:{'|'.join(batch)},from_publication_date:{since},to_publication_date:{today_iso()}",
        "sort": "publication_date:desc",
        "select": WORK_SELECT,                    # only the fields we store
    }
    for work in client.iterate("works", params, max_results=200 * max_pages):
        for authorship in work.get("authorships") or []:
            aid = short_id((authorship.get("author") or {}).get("id"))
            if aid in wanted and len(found[aid]) < MAX_RECENT_PUBLICATIONS:
                found[aid].append(publication_entry(work, aid))
        if all(len(v) >= MAX_RECENT_PUBLICATIONS for v in found.values()):
            break                                 # everyone in the batch is full

A full build now takes roughly 3,000 to 4,000 calls, and a monthly update roughly 1,000 to 1,500.

The second fix was structural. A full build runs one job per university, at most 20 in parallel, each with its own cache keyed by the university's name. Each job takes a share of the daily request budget, and together they stay at about 10 requests a second. When a job hits its time limit or the daily cap, it stops as "paused" with its progress cached, nothing is published, and the next daily run picks up where it left off. An assemble step merges the results, and only then does a quality gate decide whether the build is published.

Keying the cache by name, not by position, sounds minor. It is what let the list grow from 100 to 200 universities without throwing away anyone's progress.

Who counts as a researcher

For each university, the pipeline takes OpenAlex authors whose last known institution is that university, with at least 20 works, 500 citations, an h-index of 10 and a paper in the last three years, most cited first, up to 1,000 per university. Research areas come from OpenAlex topics. Titles and roles come from faculty pages, then from ORCID employment history. Emails come from faculty pages, then ORCID-linked pages, then the affiliation strings on the person's own papers. An email is kept only when it matches both the person's name and the university's domain.

Storage: from 500 bytes a row to 100

The first version published one JSON object per person. At 181,568 people that is far too much to download up front, so search rows went to a compact format: field names once, values as arrays, repeated strings as indexes into shared tables, and common URL prefixes stored once.

{
  "format": "compact-v1",
  "fields": ["id", "name", "title", "institution_id", "primary_domain", "domains", "citation_count", "email", "profile_url"],
  "institutions": ["I63966007", "I40120149"],
  "domains": ["Artificial Intelligence", "Materials Chemistry"],
  "url_prefixes": ["https://orcid.org/"],
  "rows": [["A5012345678", "A. Researcher", "Professor", 0, 1, [1, 0], 5120, null, "0|0000-0002-1825-0097"]]
}

Trailing nulls are dropped, and the decoder in the browser rebuilds full objects. The result is about 100 bytes a person instead of about 500. Full records, with publications and contact details, stay in per-university files that load only when someone opens a profile.

Vectors: from 1,536 bytes to 48

Each researcher has a 384-dimension embedding. Here is what that costs at 181,568 people:

Storage	Bytes per person	Whole index
float32	1,536	about 279 MB
int8	384	about 70 MB
1 bit per dimension	48	about 8.7 MB

The int8 version was the first step: scale, round, and keep a per-row norm. The bigger step was to keep only the sign of each dimension. A positive value becomes a 1 and a negative one a 0, packed eight to a byte:

def pack_bits(q):
    """Sign bits of int8 vectors, packed 8 per byte: dimension i is byte i // 8, mask 0x80 >> (i % 8)."""
    return np.packbits(q > 0, axis=1)

Throwing away 31 of every 32 bits sounds reckless. It works because the query is not quantized. The browser keeps the full-precision query vector and scores it against each sign vector, and for a unit query that dot product is, on average, a fixed fraction (the square root of 2/π) of the true cosine. Dividing by that constant puts bit scores back on the same 0 to 1 scale as the full vectors. Part 3 shows how the browser scores 48 bytes with 48 lookups.

Clustering: download a few percent of the index

8.7 MB is small for a database and large for a first page load. So the build also clusters the vectors, an inverted-file (IVF) index. It runs plain k-means on the unit vectors, with k set to the square root of the number of people, which gives 426 clusters. Then it writes one centroid file and one file per cluster:

k = k or max(1, min(n, round(math.sqrt(n))))
unit = q.astype(np.float32) / SCALE
unit /= np.maximum(np.linalg.norm(unit, axis=1, keepdims=True), 1e-9)
centroids = kmeans(unit, k)
for start in range(0, n, 20000):                  # assign in blocks to bound memory
    assignment[start:start + 20000] = np.argmax(unit[start:start + 20000] @ centroids.T, axis=1)

Centroids stay int8. Cluster members are stored as bits, along with the university of each member, so filters work without loading anything else. A search ranks the 426 centroids first and then loads the 24 closest clusters, which is about 5% of the people and a few hundred kilobytes.

Hosting the data

GitHub Pages is fine for the app and wrong for a dataset that is rewritten every month, because every rebuild would add hundreds of megabytes to the repository's history. The data lives in a Hugging Face dataset instead. The pipeline uploads public/data as one commit, deletes files the build no longer writes, then squashes the dataset's history so old copies do not pile up. The site reads the files from the dataset's CDN URL, and falls back to a small bundled sample when no hosted data is configured.

The result is a pipeline that rebuilds itself on a schedule and an index a phone can search, with no server anywhere. Part 3 is about what happens when the search runs.

Covers as the build plan, recipes as bills of materials (Michelin Kitchen Twin, part 2)

2026-10-28T00:00:00Z

A twin is only as good as its master data. For this one, every file in /data maps to a standard manufacturing object, so the simulation and the 3D scene can read the kitchen the way an ERP or MES reads a plant.

The master data

File	What it holds	Manufacturing object
`restaurant.json`	The kitchen, annex and dining room	Plant
`menu.json`	Courses, choices, supplements, eating times	Product catalog
`service.json`	Covers, seatings, variability, what-ifs	Build plan and scenarios
`items.json`	88 ingredients with yield, storage and shelf life	Item master
`products.json`	38 sub-assemblies and 15 plated dishes, each with `bom` and `routing`	BOMs and routings
`resources.json`	10 work centers, 32 machines, 10 storage locations, 14 roles, 19 staff	Resources and skills matrix
`layout.json`	Positions of every station and machine	Plant layout
`sources.json`	14 publications	Traceability

Every record has a src list pointing at the sources, or est: true when it is a modeling estimate. A validator checks every reference: items, work centers, equipment, roles, storage locations, sources and positions. It also checks that option shares sum to 1.

From covers to a production plan

The planning step is a classic MRP explosion. It runs in four steps, and the same code runs in the browser and in Node.

Demand. Covers become plates. Option shares (70% beef, 30% veal tongue) and supplement uptake (20% caviar, 25% foie gras) apply as expected values, rounded up.

Explosion. The bills of materials are walked in low-level-code order, so every parent is processed before its children. Sub-assemblies are made in whole batches, which is why 31 lobsters' worth of demand blanches two batches of 20. Raw materials are grossed up for trim and yield: 900 g of shelled corn needs 2 kg on the cob.

// Trimmed from src/model/mrp.js
export function explode(index, demand, { stockFactor = 1 } = {}) {
  const required = new Map();
  for (const [id, qty] of Object.entries(demand)) required.set(id, (required.get(id) ?? 0) + qty * stockFactor);

  // Parents before children: low-level code order.
  const order = [...index.products.keys()].sort((a, b) => index.lowLevelCode.get(a) - index.lowLevelCode.get(b));
  const make = {};
  for (const id of order) {
    const need = required.get(id) ?? 0;
    if (need <= 0) continue;
    const p = index.products.get(id);
    const isFinished = p.type === 'finished';
    const qty = isFinished ? Math.ceil(need) : need;
    // Plates are made one by one; prep is made in whole batches.
    const batches = isFinished ? qty / p.batchYield : Math.ceil(qty / p.batchYield - 1e-9);
    const produced = isFinished ? qty : batches * p.batchYield;
    make[id] = { id, name: p.name, type: p.type, required: need, batches, produced, phase: p.phase };
    for (const c of p.bom) required.set(c.item, (required.get(c.item) ?? 0) + c.qty * batches);
  }

  // Net to gross: trim, shelling and peeling losses.
  const raw = {};
  for (const [id, net] of required) {
    const item = index.items.get(id);
    if (item) raw[id] = { id, name: item.name, net, gross: net / (item.yieldPct || 1) };
  }
  // ...then every routing operation becomes labor and machine minutes.
}

The small - 1e-9 matters. Floating-point demand like 2.0000000001 batches would otherwise round up to a whole extra batch of veal stock.

Load. Every routing operation becomes labor minutes for a role and machine minutes for a piece of equipment. Labor is scaled by a hands-on fraction, because a cook is not standing at the combi while it runs. Operations are split into prep, which is mise en place before service, and service, which is cooked to order when a ticket fires.

Capacity. Labor minutes are compared with rostered minutes per role and shift.

npm run plan -- 62    # one full turn of the dining room
npm run plan -- 120   # a double turn

At 62 covers every station has headroom. At 120 the fish commis needs 105% of the prep time the shift allows, and the fish and meat chefs de partie run at 90 to 93% during service. Those are the bottlenecks the simulation later shows live. The stock-pot ranges carry about 20 hours of simmering, which is why a real kitchen makes veal stock and corned tongue days ahead. The planner reports those lead times too: corned tongue is about 76 hours and the foie gras torchon about 49.

Testing the data against published recipes

A planning model is easy to get subtly wrong, so it is tested against something real. For dishes with a published recipe, the test checks that the explosion reproduces the published quantities. The signature oysters-and-caviar course is one of them. If someone edits a yield or a batch size and breaks it, the test fails.

Being honest about what is estimated

The menu is a composite, because the real one changes daily. Quantities for courses with no published recipe are typical fine-dining portions. Operation times not stated in a recipe come from professional practice, and brigade size, shifts and machine positions are informed estimates. Ingredient prices are typical US wholesale prices. All of it is flagged in the data, so anyone using the model knows which numbers to challenge first.

Part 3 turns this plan into a simulated night.

Why I built a food bank digital twin from a news video (Food Bank Spatial Twin, part 1)

2026-10-28T00:00:00Z

A digital twin is a virtual copy of a site that stays in sync with it through data. In factories and warehouses, twins are used to plan layouts, watch equipment, and rehearse a change before anyone moves a rack. When they work, they pay for themselves.

The expensive part is almost never the dashboard. It is the geometry. A facility twin usually starts with CAD drawings, a laser-scanning crew, or a photogrammetry campaign, and then a specialist spends weeks importing all of it into a simulation tool. For an automotive plant that cost is a rounding error. For a regional food bank it is the whole reason a twin never happens.

Why a food bank

A large food bank is a serious operation. It receives trucks at a dock, stores thousands of pallets in racking, runs freezers and coolers that cannot be allowed to drift, sorts donations with volunteer crews, and ships to hundreds of partner pantries. Many also run a client market and a community kitchen in the same building. That is a warehouse, a cold chain and a small food plant under one roof.

Those are exactly the problems twins are good at. Where is the inventory? Is the freezer holding temperature? Which dock is congested at 10 a.m. on a Monday? Where do the forklifts and the volunteers cross paths? The difference is the budget. Food banks run on donations and volunteer hours, so a six-figure scanning and modeling project is out of the question.

So the question I set myself was narrow and practical: is a publicly available video tour enough to start a useful twin, with no CAD drawings, no laser scans and no site visit? And if it is, how fast and how cheap can it be?

What changed that made this worth trying

Three things moved in the last two years.

World models and Gaussian splatting. A world model is an AI system that infers 3D structure from images. Given a few video frames, it can estimate where the camera was and how far away everything is in real metres. 3D Gaussian splatting turns posed images into a photorealistic 3D scene that renders in real time, even in a browser.
OpenUSD can now hold splats. Universal Scene Description, the open 3D standard that came out of Pixar, added a native Gaussian-splat schema in its 26.03 release. A splat capture is no longer a viewer-specific file. It can sit in the same scene as racking, sensors and a forklift.
Serverless cloud costs nothing at this scale. A backend that serves a twin's live data can be a few functions behind an API, and it bills only when someone calls it.

Put together, those three suggest a different order of work. Start from footage that already exists, get a first twin in days, and only spend real money on the parts that turn out to matter.

The rules I set before writing any code

I gave myself seven days and three rules.

One contract between the steps. Everything upstream writes OpenUSD in metres, Y up, with the floor at zero. Everything downstream reads it. The browser, Blender and Omniverse are interchangeable renderers, not separate projects.
Label everything. Every part of the twin is tagged as captured from the video, designed for this project, taken from published figures, estimated, or simulated. A twin that hides which parts are real is worse than no twin.
Never pay twice. Every paid call has a budget cap, records its job id before waiting, and resumes instead of re-paying when it is run again.

The subject is a representative Texas food bank, inspired by a local news tour of one. It is not a replica of that building, and neither the food bank nor the station had any part in it.

What came out of it

The whole pipeline. OpenUSD in metres, Y up, floor at zero, is the only contract between the steps.

The pipeline runs in three steps, and each one gets its own part in this series.

Video to 3D captures. Twenty frames went to a world model for camera poses and metric depth, and came back as about 747,000 splats across three areas, for 200 credits, which is 20 US cents. A Gaussian-splat trainer I wrote to run on a plain CPU then trained four areas from 49 to 72 frames each, at no credit cost, which made them hold up as the camera moves.
The rest of the building in Blender. The video never shows the whole building, and asking the world model to imagine the unfilmed areas failed on the provider's side twice. So one headless Blender script models the full facility from the footprints in the operations data and publishes it in the same OpenUSD frame.
Data on AWS, viewed anywhere. A serverless API on AWS serves cold-room telemetry, forklift positions and the operations layer in the same coordinates. The twin runs in a browser on GitHub Pages and renders in NVIDIA Omniverse on a rented GPU.

The browser twin: four areas captured as Gaussian splats from the video, with the simulated cold-chain feed from AWS beside them.

It went from first commit to a deployed backend in seven days. The build cost about $1.54 of compute: 20 cents of world-model credits and about $1.34 of GPU time for the Omniverse trials. The AWS backend has billed $0.00 so far. Putting real sensors on a facility floor is estimated at about $2,700 of hardware once and about $12 a month on AWS.

Not everything worked, and I will be specific about what did not. Generating views of unfilmed space failed. Omniverse rendered the Blender building beautifully but drew the splats as a blank grey frame. Those failures are as useful as the successes, because they show exactly where the tooling is today.

What is next in the series

Part 2 goes into the first step in detail: how to get from a news video to a 3D world model, the frame selection and masking that make or break it, and the CPU splat trainer, with the code. After that come the Blender building, the AWS architecture, and the full pipeline with the paper.

The code, the docs and the live twin are all public. If you run a warehouse, a cold chain or a food bank and want to try this on your own footage, the repository is the place to start.

A private research matchmaker that runs entirely in the browser (Academic Matchmaker, part 1)

2026-10-28T00:00:00Z

Finding a PhD or postdoc advisor works roughly the way it did twenty years ago. You open a university's faculty page, read through a few hundred names, open the ones that sound close, skim their recent papers, and repeat for the next university. Twenty universities in, you have a spreadsheet, a headache, and no idea whether the best fit for your work was at the twenty-first.

The information to do better is public. OpenAlex indexes hundreds of millions of papers with author and institution links. ORCID records list people's current roles. University directories list emails. What is missing is a tool that puts it together and answers one question: whose recent research is closest to mine?

Two constraints that shaped everything

I set two rules before writing code.

Your CV never leaves your device. A CV is personal. People upload drafts, unpublished work and contact details. A matching service that collects them is a liability for the person running it, and asking students to trust it is a big ask. So text extraction, the language model and the ranking all run in the visitor's browser.

No servers. The site is static files on GitHub Pages. The data is rebuilt by GitHub Actions on a schedule and published to a Hugging Face dataset, which serves it over a CDN. Nothing runs between those builds, so there is nothing to keep up and nothing to pay for.

Those two rules turn a standard retrieval problem into a constrained one. A typical setup would put the vectors in a database and run a query on a server. Here, the browser has to download whatever it searches, so every byte counts and the index has to be organized for partial loading.

Build in CI on the left, match in the browser on the right. The only link between them is a set of static files.

How a match happens

Read the upload. PDFs go through pdf.js. Screenshots of a Google Scholar or LinkedIn page go through Tesseract OCR. Saved HTML and text are read directly. You can also paste an ORCID iD or OpenAlex link, and the profile is built from public records instead.
Understand it. A small language model, Qwen2.5-0.5B quantized to 4 bits, runs on WebGPU (with a WebAssembly fallback) and extracts research interests as JSON. Chat messages go through the same model and refine the profile.
Embed it. The same sentence-embedding model that indexed the researchers, all-MiniLM-L6-v2, turns the profile into one 384-dimension vector.
Rank, coarse to fine. The browser scores 426 cluster centroids, downloads only the nearest clusters, and ranks the people in them.
Explain it. Each match shows the shared topics and the closest recent papers, so you can see why someone came up.

Two loading decisions matter for the first visit. The language model is a few hundred megabytes, so it loads only when someone types in the chat box. An uploaded CV skips it entirely and is matched with the embedding model alone. And the vectors load cluster by cluster, so a search touches a few percent of the index, not all of it.

What it covers today

The build pipeline now indexes 181,568 researchers across the top 200 universities in the QS 2026 rankings, and 51,430 of them have an email that matches their name and their university's domain. The data refreshes monthly, with new papers and citations pulled from OpenAlex.

A fixed test set keeps the matching honest. It holds the 2026 Fields Medal and Nobel Prize winners. After every data refresh, the pipeline builds a profile for each of them from their public record (and from a saved CV where one exists), runs it through the same matcher the site uses, and checks the results. The last run passed 11 of 11. The site shows a "Prize test passing" badge only when that holds.

Why this is worth writing up

The interesting part is not the interface. It is the series of decisions that made a 181,568-person index searchable on a phone without a server:

splitting the pipeline into one job per university so it fits a free API budget and resumes when it hits the limit,
storing search rows at about 100 bytes a person instead of 500,
storing each vector in 48 bytes instead of 1,536, and scoring it with 48 table lookups,
clustering the index so a search downloads a few hundred kilobytes, not hundreds of megabytes,
and adding concept tags when pure semantic similarity started matching people on generic words.

Part 2 covers the data pipeline and storage. Part 3 covers matching, the tags and the test set.

Why I modeled a three-star kitchen like a factory (Michelin Kitchen Twin, part 1)

2026-10-28T00:00:00Z

Most digital twin demos show one of two things: an abstract dashboard, or a conveyor belt moving identical boxes. Neither looks like a real operation. Real plants have dozens of product variants, machines shared between lines, operators with different skills, inventory that spoils, and customers with due dates. A demo that skips all of that does not tell a plant manager whether a twin can handle their problem.

A fine-dining kitchen has every one of those problems, in a space you can see end to end in one camera shot. The twin models The French Laundry, the three-star kitchen in Yountville, California.

The kitchen is already a factory

Here is how the concepts line up, and how the twin's data is organized:

Kitchen	Factory
The French Laundry and its kitchen	Plant
Tasting menu, course choices, supplements	Product catalog, configure-to-order options
Covers booked tonight	Master production schedule (the build plan)
Ingredients	Purchased raw materials
Mise en place: stocks, sauces, doughs, portioned fish	Sub-assemblies and work in process
Plated courses	Finished goods
Recipe quantities	Bill of materials
Method, station, appliance, timings	Routing: operations with setup and run times
Stations: fish, meat, garde manger, pastry	Work centers
Ranges, combi ovens, hearth	Machines with parallel capacity
The brigade	Labor pool by skill, with shifts
The pass	Final inspection

None of this is a metaphor stretched to fit. Expo firing a course early so every plate on a table finishes together is offset scheduling. A cook who can cover fish and garnish is a skills matrix. A dish that runs out and gets swapped is an engineering change under pressure. Once you see it, a dinner service reads like a production line with an unusually demanding customer.

Why it makes a better test

A twin earns trust by getting the hard cases right. A kitchen at full service is hard in exactly the ways that matter:

High mix. Eleven courses with choices and supplements, 15 plated dishes, 38 sub-assemblies.
Shared, constrained machines. A few combi ovens and one hearth serve every station.
Skilled labor. Who can cook what is as much a constraint as the equipment.
Perishable inventory. Prep made too early is waste. Prep made too late is a stockout at 8:30.
Hard due dates. Guests notice a slow course within minutes.

If the model can show why the fish station backs up at 120 covers, the same model can show why a weld cell backs up on Tuesdays.

A rush seating of 120 covers at 19:53. Ticket times are climbing, and the dashboard names the constraint: the fish chef de partie is 96% busy with 10 tasks queued.

The bar I set

I wrote the product brief before any code, as a press release and FAQ, so every decision could be checked against it. Four rules came out of it:

The data is sourced. Every record cites a publication or is marked est. The kitchen's size, its equipment schedule, the courses and several full recipes come from published sources. Brigade size, timings and layout positions are estimates, and the data says so.
The plan is computed, not scripted. Covers go through the bills of materials into plates, sub-assemblies and grams of each ingredient. Routings turn into minutes of work for each cook and machine.
The service is simulated, not animated. What you see in 3D is the simulation's state, frame by frame.
It is deterministic. The same covers, seating and seed replay the same night to the second. That is what lets a share link open the exact moment you were watching.

The other bar was practical. It had to open from a link, on a laptop or a phone, with no install and no login. So it runs entirely in the browser on GitHub Pages, and the 3D scene is generated from the data at load time, with no downloaded models or textures.

What you can do with it

Pick the covers and a seating pattern, then run the night. Double the covers. Take a cook off the line. Fail a combi oven at 8:15. Run the lobster 40% short. Then watch the bottleneck move, the ticket times climb and the dashboard explain why. Change something and run the same night again from the same seed, so the comparison is fair.

Part 2 goes into the data model and the MRP explosion that turns covers into a production plan.

An open, cited history of digital twins, from andon lights to Physical AI

2026-10-28T00:00:00Z

"Digital twin" sounds like a 2010s idea. The name is, and the idea is not. People have been building live stand-ins for physical things for well over a century: a time clock that records who is on the floor, a loom that stops itself when a thread breaks, a flight simulator that behaves like an aircraft, and the ground simulators that helped bring Apollo 13 home. Each of them answers the same question a twin answers today: what is happening out there, and what happens if we change something?

I wanted that history in one place, with a public source for every claim, so it could be reused in courses, talks and reports. So I built Digital Twin History, an open dataset and an interactive site built from it. Everything is MIT licensed.

The growth spiral: every milestone placed by year, with the spiral widening as more domains take up the idea.

Seven eras

Era	Years	In one line
Mirrors before the name	1880–1970	Time clocks, andon lights, simulators, the Kalman filter, Apollo 13
The digital model	1971–1989	CAD, finite elements, PLCs, SCADA, computer-integrated manufacturing
Connected models	1990–2001	Mirror Worlds, MES, OPC, ISA-95, virtual cities
The idea gets a name	2002–2013	The 2002 product-lifecycle model, NASA's "digital twin" in 2010, Industrie 4.0
Industrial scale-up	2014–2019	Fleet twins, city twins, organ twins, sim-to-real robotics
Definitions and standards	2020–2023	ISO 23247, ISO/IEC 30173, the National Academies report, Destination Earth
AI-native twins and Physical AI	2024–now	Learned forecasting in operations, fast reality capture, twins as training grounds, world models, agents

The pattern across the eras is that the 3D view was never the point. Each era added a new answer to "what can we know about the machine?": from "someone has to walk over and look", to "which alarm fired", to "which machine is drifting toward failure", to "what happens if we change the layout". The current era adds "which policy is most robust before we try it". That is where Physical AI comes in: twins have become the place where machines learn.

Built as data first

The site is the second thing in the repository. The first is a structured dataset: milestones, eras, domains, definitions and the narrative for the 3D factory, all as JSON and Markdown. Every milestone has a year, an era, one or more domains, a kind of change (precursor, enabling technology, concept, standard, deployment or research), a significance level and at least one source.

A validator runs before every build and fails it if any milestone is malformed or uncited:

// Excerpt from scripts/validate-content.mjs
if (!Array.isArray(m.sources) || !m.sources.length) errors.push(`${where}: needs at least one source`);
else m.sources.forEach((s) => {
  if (!s.title) errors.push(`${where}: source missing title`);
  try { if (new URL(s.url).protocol !== 'https:') throw 0; } catch { errors.push(`${where}: source url must be https: ${s.url}`); }
});
if (era && (m.year < era.range[0] || m.year > era.range[1])) errors.push(`${where}: year ${m.year} is outside era ${m.era}`);

That one rule, no source and no merge, is what makes the dataset safe to reuse. It also makes contributing simple: edit content/timeline.json, cite a public source, and run the validator.

Three ways in

The growth spiral. Every milestone is a sphere placed by year. The spiral widens as more domains take up the idea, so you can see the field spread from manufacturing and aerospace to energy, cities, health and Earth systems. Time is compressed before 1960 and stretched after 2000, where most of the activity is:

// Time is compressed before 1960 and stretched after 2000, where most activity is.
export function yearToY(year) {
  if (year < 1960) return (year - 1880) * 0.06;
  if (year < 2000) return 4.8 + (year - 1960) * 0.2;
  return 12.8 + (year - 2000) * 0.7;
}

The factory through time. A walkable 3D production line shown in six eras: lights on the floor, the control room, the connected plant, sensors and dashboards, the live digital twin, and Physical AI. Click any machine to see what people could know about it in that era. The machine values are simulated and labeled as illustrative.

Growth in research. Yearly counts of papers with "digital twin" in the title come from OpenAlex at build time. The fetch never fails the build. If the API is unreachable, the page says so instead:

// Simplified from scripts/fetch-publication-counts.mjs
const url = 'https://api.openalex.org/works?filter=title.search:%22digital%20twin%22&group_by=publication_year&per_page=200';
try {
  const res = await fetch(url);
  if (!res.ok) throw new Error(`HTTP ${res.status}`);
  const counts = (await res.json()).group_by.map((g) => ({ year: Number(g.key), count: g.count }));
  // ...written to public/data/publications.json
} catch (err) {
  console.warn(`Skipping publication counts: ${err.message}`);
}

A weekly run refreshes those counts, so the chart stays current without anyone touching it.

Principles

Traceable. Every claim links to a public source.
Vendor-neutral. The history is told through ideas, practices, standards and public programs. Organizations are named only where history requires it, never as recommendations.
Honest about data. Charts built on the curated milestones say so, and simulated values in the 3D factory are labeled.

Where this connects to what I build

The frontier section of the site argues that the next steps for twins are not better 3D. They are faster reality capture, hybrid physics and AI, world models, agentic interfaces, and trust: verification, validation and uncertainty built in from the start. My recent builds are experiments on exactly those lines. The Food Bank Spatial Twin tests fast reality capture with world models. The Michelin Kitchen Twin tests whether a twin can explain a bottleneck rather than just show one.

Corrections and new milestones are welcome on GitHub.

Label what is real, the most underrated feature of a digital twin

2026-10-28T00:00:00Z

Every digital twin I have seen is a blend. Some of it was measured. Some was drawn by an engineer. Some came from a spec sheet, some was estimated by someone who knows the process, and some is simulated because the sensor does not exist yet. That blend is normal and fine. The problem is when the twin does not say which part is which.

A twin that hides its provenance fails in a predictable way. Someone notices one number is made up, and from then on nobody trusts any of the numbers, including the ones that were measured carefully. The twin goes from decision tool to demo in a single meeting.

Five labels are enough

Across my recent builds I have used the same five labels:

Label	Meaning	Example
Captured	Measured from the real thing	Gaussian splats trained from video of a racking aisle
Designed	Drawn or modeled for the twin	A Blender model of the parts of the building no camera saw
Published	Taken from a public, citable source	A kitchen's floor area and equipment schedule, from published sources
Estimated	A judgment call by someone who knows the process	Operation times not stated in any recipe
Simulated	Generated by a model, because no sensor exists yet	Freezer temperatures and forklift positions

Five is a small enough set that people remember it, and it covers every case I have met.

Put the label on the record, not in a footnote

A provenance note on a slide gets lost the first time someone screenshots a chart. The label has to travel with the data.

In the Michelin Kitchen Twin, every record in the master data has a src list pointing at a bibliography, or est: true. A validator fails the build if a reference is broken. In the Food Bank Spatial Twin, every API payload carries simulated: true, so no dashboard, script or person downstream can mistake the feed for a real sensor. In the Academic Matchmaker, records missing a verified email or an institutional page carry flags like missing_email and are sorted after fully verified ones.

The same idea shows up in the UI. In the food bank twin's architecture diagrams, solid boxes are deployed and dashed boxes are designed and priced but not built. The browser badge says "Simulated feed" until the live API answers, then "Live from AWS". The labels cost almost nothing to add, and they answer the first question every serious reviewer asks.

Labels make the real parts more valuable

The counterintuitive effect is that labeling the estimated parts makes the measured parts more credible. When a reader can see that the freezer temperatures are simulated, they can trust that the splat capture of the aisle really came from the video. When the kitchen's recipe quantities for one dish are marked as published and tested against the source, the estimated timings next to them read as honest estimates rather than hidden guesses.

Labels also tell you where to spend money next. In the food bank twin, the labels made the next step obvious. The geometry is good enough to start. The simulated telemetry is the gap, and real sensors are priced at about $2,700 once and $12 a month. Without labels that conversation turns into a debate about whether the whole twin is real. With them, it is a line item.

A checklist I now use on day one

Pick the labels before collecting any data, and put them in the schema.
Make the build fail on a record with no source and no estimate flag.
Carry a simulated flag through every API and into every chart legend.
Draw built and designed parts differently in every diagram.
Write one page that lists what is real, what is not, and what would make it real.

None of this is sophisticated. It is the discipline a good plant already applies to its own data, applied to the twin. And it is the difference between a twin people use and one they politely admire.

Write the contract first, what AI agents taught me about building fast

2026-10-28T00:00:00Z

Over the last few weeks I have built several systems with AI coding agents working in parallel: a kitchen digital twin, a food bank twin built from a news video, and a research matchmaker that indexes 181,568 researchers. Each one was built by several agent threads at once, each owning one part of the repository.

The thing that decided how fast each build went was not the model, the prompt or the tooling. It was whether the parts had a written contract between them before anyone wrote code.

What goes wrong without one

Give several capable agents one goal and one repository, and they behave like several capable engineers in the same situation. Each makes reasonable local decisions. The 3D thread invents ids for the machines. The data thread invents different ones. The simulation thread reads a file shape that the data thread changes an hour later. Every part works on its own, and the system does not. The integration work at the end eats the time the parallelism saved.

What a contract looks like in practice

The contract does not need to be elaborate. In each build it was one or two things that every part agreed to read and write:

Kitchen twin: a set of ids (EQ-* for equipment, WC-* for work centers, RM-* for raw materials, FG-* for finished goods) and an ordered event log between the simulation and every view of it. The 3D scene places a model at each EQ-* id and exports a USD prim with the same name. The dashboard reports utilization by the same ids.
Food bank twin: OpenUSD in metres, Y up, with the floor at zero, plus a small JSON API in the same frame. The capture pipeline, the Blender model, the AWS backend and three different renderers never needed to know about each other.
Matchmaker: a published data format between the pipeline and the site, with versioned file formats (compact-v1, tags-v1) so the site can read old and new builds while the pipeline changes underneath it.

Ownership is the other half

Each thread owned a directory, and nobody edited anyone else's. When the simulation thread needed ingredient prices, it asked the data thread in a message, and the data thread added them. That sounds bureaucratic for software written in an afternoon. In practice it removed most merge conflicts and kept every change reviewable by the one thread that understood that part.

The kitchen twin's first version, fifteen pull requests across five threads, came together in a day because of that structure. Each pull request was built and tested by CI before merge, so the system was integrated continuously instead of at the end.

What still needs a person

Agents are very good at volume: sourcing and structuring data, writing an engine with tests, generating a 3D scene, keeping docs in step with code. The work that stayed with me was the work that does not parallelize:

Setting the bar. "Realistic", "private" and "no servers" had to be defined up front, in writing, before any thread started.
Choosing what is honest. Which numbers are labeled as estimates, what the demo is allowed to claim, and what the paper says about what failed.
Approving merges. CI catches breakage. Judgment about whether a change moves toward the goal is still a human call.

The takeaway

None of this is new. Interfaces before implementation, clear ownership and continuous integration are how good engineering teams have always moved fast. What is new is how much faster the cycle runs when the implementers are agents. A missing contract used to cost a week of integration. Now it can cost most of a one-day build. So write the contract first.