Adron's Composite Code

GitHub PRs are slow. Keep shipping anyway.

2026-11-05T04:00:00-08:00

The issue is ready, the code is ready, and then the PR page spins. A check stays queued. A webhook arrives late. The review that should have taken ten minutes turns into an afternoon of refreshing GitHub and wondering which part of the pipeline is actually broken.

That is not hypothetical in October 2026. GitHub’s August 17 incident report says a 7 hour, 47 minute outage disrupted the site, APIs, Actions, pull requests, and issues. Its September availability report describes database pressure and work to contain failures in pull requests. The status history also records October delays across PRs, Issues, Actions, and webhooks. GitHub is doing capacity work. A development team still needs a plan for the hours when its work is waiting on that capacity.

Find the actual wait

“GitHub is slow” describes the feeling, not the bottleneck. Track issue ready to first code change, code complete to first review, review complete to checks, checks complete to merge, and merge to deploy. Look at the median and the ugly tail. A slow PR page is one problem. A two day review queue is another. A CI job that queues for an hour and runs for three minutes has a different remedy from a test suite that takes an hour on a ready runner.

This measurement matters because some delays are in our own process. If a task cannot start until somebody writes an elaborate issue, or a trivial change waits behind a weekly approval meeting, GitHub uptime will not fix it. Write the acceptance criteria and the risk in a short issue or local task note, then let implementation start. The artifact should help the next person understand the work, not become a ceremony the work must survive.

Move the fast feedback close to the code

Make the build, focused tests, formatting, and relevant static checks runnable with one documented local command. Run them before opening the PR. A remote check is still useful as the shared attestation, but it should confirm a result developers can reproduce, not be the first place anyone discovers a compiler error. Cache dependencies carefully, keep the quick checks quick, and reserve expensive matrices for changes that need them.

Review can begin from a branch or a patch while the PR UI is unhealthy. A reviewer can read git diff, run the relevant tests, and write findings in a shared local note. When GitHub recovers, put that discussion on the PR so the decision remains visible with the change. Do not mistake a green local run for the required check on a protected branch. Branch protection is a team contract, and an outage is a poor reason to silently discard it.

Keep a second path for coordination

Issues are a convenient source of truth until the issue service is unavailable. Keep enough task context in the repository or another team owned system to continue: a task ID, expected behavior, owner, and decision log. It can be a small Markdown file. When GitHub returns, reconcile it to the canonical issue. Avoid two competing permanent backlogs.

Git itself is distributed. If GitHub’s web services are degraded but Git operations still work, teams can continue committing and exchanging branches through the remote. If Git operations are unavailable too, local commits keep the work safe. For a prepared contingency between trusted machines, Git bundles can carry repository history without a live server. A second forge or internal mirror can provide a more complete continuity path, but only if the team has already rehearsed access, review, and reconciliation. Setting one up during an incident is not a shortcut.

Self hosted Actions runners solve a different problem: where jobs execute. They still connect to GitHub for job assignments, so they do not remove the Actions control plane dependency. A locally runnable test script does. A separate CI system can, if its triggers and repository access are independent too.

Reduce the amount of GitHub work per change

A giant PR asks the platform and the reviewer to do too much at once. Keep changes small enough to inspect and merge. Separate mechanical cleanup from behavior changes. Use feature flags or compatible incremental changes so one PR need not contain the entire release. Avoid automations that trigger redundant workflows on every push, comment, label, and issue edit. They add queue pressure and make it harder to tell which signal matters.

This is also where agent output can hurt. An agent that opens a PR for every tiny correction can swamp reviewers and checks. One that generates a thousand file change is worse. The useful middle is a coherent slice with a clear behavior change and a reproducible local result.

Decide what happens during an outage before one happens

For ordinary work, keep committing, testing, and reviewing locally, then publish when service recovers. For urgent production work, use the team’s documented emergency path, including independent review, recorded test evidence, and a later audit trail. If branch rules require a specific bypass, make that an explicit incident decision by the people authorized to make it. Do not give agents broad bypass permission so they can “keep shipping.”

GitHub should remain a useful place to coordinate and verify work. It does not need to be the only place work can happen. The practical goal is a pipeline that can keep making safe progress when a PR page, webhook, or runner queue has a bad day.

AI code quality: Ship fast without the fires

2026-11-04T04:00:00-08:00

The October 2025 DynamoDB disruption spread through AWS after a race condition in automated DNS management erased the service’s regional endpoint. Its published cause was an automation defect, not AI-written code. That distinction matters. If we blame every outage on an LLM, we miss the specific mechanisms that made the outage possible.

The AI-related cases are narrower and messier. Reporting connected an AI coding tool to an AWS Cost Explorer interruption, while Amazon attributes that event to misconfigured access controls and disputes the reported cause. In separate Amazon retail incidents, the company says an engineer followed advice an AI tool inferred from an outdated internal wiki. These cases do not prove that faster AI development caused AWS’s broad outage. They do show how stale context and excess authority can give an ordinary mistake operational reach.

There is a cleaner example. In 2025, Replit acknowledged that its agent deleted data from a user’s application database while development and production data were insufficiently separated. The data was restored, but the production application could have been broken during the recovery window. Replit’s response was to separate the databases by default and keep the agent away from production data during development. The system had given a development tool production reach.

Most failures look less dramatic. An agent adds a retry around a payment call without an idempotency key. It makes a slow query pass the unit suite by mocking the database. It replaces a boring, explicit validation rule with a generic helper that accepts an empty value. Each diff can look reasonable. Each can make the application worse. Generate enough of those diffs, and the review queue becomes an exercise in finding one dangerous sentence in a book that keeps rewriting itself.

The old gate is getting squeezed

The traditional bargain was that a developer made a change, a reviewer read it, CI ran, and the team shipped. Those steps still matter. The trouble is the arrival rate. Agents can produce plausible changes faster than a team can understand them. A reviewer staring at a giant generated PR is likely to check syntax and style, then miss the changed behavior. A test suite assembled by the same agent can reinforce the same mistaken assumption.

The answer cannot be “add three more approvers.” That makes the queue longer without making anyone better informed. The real unit of review needs to get smaller, and the evidence attached to each change needs to get better. I argued for small, scoped changes before the current volume of agent output made the problem obvious. A reviewable diff has one job, names its expected behavior, and says which failure modes it considered. A generated feature that rewrites twelve unrelated files needs to be split before it reaches a reviewer.

Put checks where mistakes enter

Start before code generation. Give the agent the current source of truth: the domain terms, the relevant files, the interface it must preserve, and the operations it may perform. If the design is unclear, ask for a plan and a list of unknowns first. My ask, plan, confirm workflow is useful here because it makes a change in authority visible. A read only investigation should not quietly become a production edit.

Next, make every implementation change answer three concrete questions in its PR description:

What behavior changed for a user or another system?
What happens when the dependency is slow, unavailable, or returns an unexpected result?
How will we know the change is wrong after deployment, and how will we back it out?

Those questions are more valuable than a declaration that “all tests pass.” Tests need to exercise the behavior, including a failed request, a duplicate request, and a partially completed operation where those cases apply. A retry that works on the happy path can duplicate an order. A migration that passes locally can lock a table under production load. The test plan should follow the risk, not the quantity of generated code.

Then put hard boundaries around the places where a wrong answer is expensive. Development agents do not need production credentials. Migration agents do not need permission to destroy live data to draft a schema change. Deployments that alter durable state deserve a separate authorization step and a recovery rehearsal. The Replit incident makes that point more clearly than any policy memo.

Review the behavior, then the shape of the code

“Clean code” is too easy to reduce to formatting. I want a reviewer to trace the new behavior through the caller, the domain rule, the persistence boundary, and the observable result. Does the code use the team’s vocabulary? Does it duplicate a rule already implemented elsewhere? Does it leave a clear place for the next change? Is the error path understandable to someone on call at 2 a.m.?

An independent review helps when it has a specific lens. Ask one reviewer to look for changed invariants and another to check operational failure modes. Give both the actual diff and current contract. Do not ask the authoring agent to certify its own work with a cheerful paragraph. In additive, mutative, and destructive changes, I used those categories to make the impact of a diff easier to see. They also help decide how deep the review and rollout need to be.

Finally, use production as a feedback loop without treating customers as the test suite. Release behind a flag when practical. Start with a small cohort. Watch error rate, latency, saturation, and business outcomes. Have a known rollback path. If the change modifies data, confirm that rollback restores behavior even when old code meets new data. An automatic rollback triggered by a reliable signal beats a heroic incident call after everyone has gone home.

Change the throughput target

The target is not generated lines per engineer or PRs merged per day. It is useful behavior delivered without borrowing trouble from next week. Track how often a change needs a follow-up fix, how long failures take to detect, and how easily the team can restore service. If those numbers get worse while code output rises, the pipeline is accelerating in the wrong direction.

AI can make the first draft cheap. Understanding the system, containing the blast radius, and learning from what shipped are still the work. A faster keyboard makes those jobs more urgent.

Long-running coding agents: When to stop and split

2026-11-03T04:00:00-08:00

I have written about Herdr as a place to see all the agents and about why I give spaces, tabs, and panes specific jobs. A multiplexer helps me find the agent that is blocked on a question. It does not tell me whether the agent should still be working on that question at all.

That distinction matters when several agents are active. Six sessions can all report “working” while two implement the same feature, one waits on a test process that will never exit, and another repeatedly rewrites a file to satisfy an impossible prompt. The dashboard is truthful about process state. It is silent about whether the work is useful.

Give each run a finish line

Before starting an agent, write down its owner, scope, expected output, and stop condition. “Improve the billing service” has no finish line. “Add an idempotency check to POST /charges, show the duplicate request test, and stop before changing the database schema” does. The run can finish, fail, or ask for a decision. Each state has something a human can inspect.

I also set a budget appropriate to the task: elapsed time, tool calls or tokens if the platform exposes them, files touched, and number of attempted approaches. These are tripwires, not productivity scores. An agent spending twenty minutes on a slow integration test may be doing exactly the right thing. An agent spending twenty minutes alternating between two incompatible implementations is not.

A useful checkpoint is a short record of what has changed, what has been verified, what remains, and what is blocked. The evidence should be commands and artifacts, not just the agent’s confidence. If a long run cannot say what changed since the previous checkpoint, I want to inspect it before buying another hour of execution.

Long running can be legitimate

Some work has a real long pole. A full build, a large data migration rehearsal, a cross platform test matrix, or a carefully scoped investigation may need hours. The clue is visible progress against a fixed plan. The agent reaches milestones, the diff stays within scope, test results accumulate, and the remaining work gets smaller. The clock alone is a poor judge.

Other runs are long because the task contains too many different verbs. The agent is asked to research a domain, design an API, implement it, update dependencies, test it, deploy it, and review its own work. Each phase changes what “done” means. The conversation grows, old assumptions remain in context, and the agent starts using its own earlier guesses as facts. That is where I stop and split the work.

My practical test is to ask whether the current run can be reviewed without replaying the entire transcript. If the answer is no, ask for a checkpoint and create a new task with the verified facts. Split at a decision boundary: investigation into a brief, brief into a design, design into one implementation slice, implementation into an independent review. Do not split a single atomic edit into five agents just because parallelism is available. The handoff can cost more than the edit.

Supervise the queue, not every keystroke

With several agents, I want one visible register of active work. A simple table is enough: task, owner, branch or environment, paths or interfaces owned, state, last checkpoint, and next decision. The multiplexer tells me where the process is. The register tells me what the process is allowed to change. When an agent goes blocked, I answer the decision once and record it where the next session will find it.

I keep a limit on simultaneous writers. More agents do not automatically mean more completed work. They create more integration edges and more reviewer load. Read only investigators and reviewers can often run alongside an implementer. Two agents changing the same contract, deployment configuration, or migration sequence need sequencing even if their files are separate. Worktrees isolate files; they do not isolate architecture, a shared database, or the main branch after merge.

The review agent gets a different job from the writer. It reads a committed diff and reports behavior risks with file and line evidence. It does not silently fix what it finds. A fix becomes another scoped implementation task. That keeps the person supervising the queue from confusing an agent’s reported success with independent verification.

A run needs an end, even when the project does not

“Keep going until everything is perfect” is an expensive prompt and an impossible stop condition. I prefer a bounded assignment that can end with a useful handoff: what landed, what failed, what remains, and the exact next slice. A fresh session can consume that handoff plus the source files. It does not need the entire history of abandoned attempts.

There are exceptions. An agent supervising a known batch job may stay alive to watch it, provided it has a timeout, a clear alert condition, and no open ended permission to change the system when something looks strange. A long lived process should be easier to audit than a short one, because it has more opportunities to drift.

A punch list for a busy team

Name one owner and one finish line for every agent run.
Record the paths, interfaces, and environments each writer may touch.
Limit concurrent writers to the number the team can integrate and review.
Set checkpoints and tripwires for time, tool use, diff size, and repeated failed approaches.
Split work at design, implementation, and review decisions. Preserve verified facts in a short handoff.
Give production resources separate permissions from development resources.
Run meaningful behavior and failure tests before merging, then review the combined result.
Ship with observability and a recovery path, especially for data and deployment changes.

The aim is not to keep every agent busy. It is to keep the codebase coherent and the application boring to operate. A blocked agent is easy to spot with the right tool. A busy agent making the wrong change needs a clear assignment, evidence, and someone willing to stop the run.

AI agent collisions: Worktrees aren't enough

2026-11-02T04:00:00-08:00

Two developers editing the same service at once is older than Git. One changes an API, the other writes a caller against the old shape. One migration lands, the other assumes the previous schema. The collision may wait until integration, but nobody invented it with coding agents.

Agents compress the timeline. They can start the overlapping work within seconds, finish competing implementations before either human sees a diff, and spend tokens repeatedly “fixing” each other’s changes. A human collision can cost far more than the token bill, sometimes an order of magnitude more once meetings, rework, and a delayed release are counted. That is a scenario, not a universal ratio. The new problem is how quickly both kinds of cost can accumulate while everyone thinks parallel work is progress.

I have already written the case for giving each agent its own worktree. Set that option aside here. Even with separate directories, two agents can change the same behavior, rely on incompatible contracts, or point tests at the same database. We need ways to manage the work itself.

Assign ownership at the level that matters

File ownership is a useful start, but interfaces are the real collision surface. If one agent changes the response shape of GET /orders and another changes the frontend that consumes it, they are collaborating on one contract. Give one task ownership of that contract and sequence the dependent implementation after the new shape is agreed. If parallel work is valuable, freeze the interface first and let both agents build against a shared contract test.

The same rule applies to domain terms, schemas, feature flags, and deployment configuration. A small task register can say: agent A owns the API contract and tests; agent B owns the UI after the contract is approved; agent C performs a read only review. It need not be a complicated scheduler. Its purpose is to stop two plausible plans from quietly disagreeing.

Use short leases on hot areas. A lease says who is editing a shared contract, when they expect to hand it off, and where the result will appear. It is a social protocol with a visible record, not a file lock that magically prevents bad code. If an agent exceeds the lease, review progress and renew or split the task. Do not let an abandoned session hold the architecture hostage all day.

Choose the kind of parallelism deliberately

Some tasks are naturally parallel: read only investigation, documentation of settled behavior, tests for a frozen contract, or independent review. Others are sequential: a migration followed by code that assumes it, a public API change followed by client updates, or two fixes to the same state machine. Forcing the second group into parallel execution creates integration work disguised as speed.

When the work can be split, split by stable boundary. “Agent A handles orders, agent B handles billing” may work if the interface between them is already explicit. “Agent A edits the first half of the file, agent B edits the second half” probably will not. File lines are not architecture. A good boundary gives each agent a testable outcome and few shared assumptions.

Another option is a single writer with several readers. Ask separate agents to investigate edge cases, inspect the current behavior, or critique the plan. Feed those findings to the writer after a human checks them. That gets the benefit of parallel thought without parallel mutation. It is particularly useful in a sensitive area where three implementations would be hard to reconcile.

Isolate the shared things that are not code

Separate branches do nothing for a shared development database, a common staging environment, one cloud account, or the same test tenant. Give agent runs separate databases or schemas where practical, unique test data, distinct ports, and credentials limited to their task. If isolation is not available, schedule the shared resource as a single writer and make its current owner visible.

Generated artifacts need the same care. Two agents can each pass tests locally, then regenerate the same client library from different versions of a schema. A dependency lockfile can change under both tasks. A shared cache can hide an integration failure. Decide which task owns generation, pin the input, and test the combined result after both changes are present.

Detect conflict before merge time

At task start, have each writer declare the files, interfaces, and resources it expects to change. Recheck that declaration when the diff expands. If an agent discovers it needs a boundary owned by another task, it stops and asks. This is a cheap rule, and it prevents a lot of expensive “helpful” edits.

At integration time, do more than resolve text conflicts. Run contract tests and the affected application path against the combined code. Look for two implementations of the same behavior, changed assumptions about defaults, and migrations whose order matters. A clean merge only proves Git could combine the files. It says nothing about whether the resulting application makes sense.

When a collision does occur, preserve both task intents before choosing a fix. Ask what behavior each agent was trying to protect, then select the contract the product actually needs. Blindly asking agents to reconcile each other’s diffs often creates a third implementation that nobody intended.

Keep parallel work cheaper than serial work

A small number of clear assignments, visible ownership, bounded runs, and one integration pass will beat a large crowd of agents whose outputs need arbitration. Count rework and review time alongside token use. If adding another agent makes the queue of conflicting diffs longer, reduce concurrency or stabilize the interface first.

Collisions were always part of software development. Agents make them arrive while the coffee is still hot. The fix is to make the boundaries, contracts, and shared resources explicit before the work starts, then check the integrated behavior before calling it done.

Spaces, Tabs, Panes, and Precise Words: Why I Break Agent Work Out the Way I Do

2026-10-09T12:37:57-07:00

In the previous post I toured Herdr and how I lay it out: one space per folder, one tab per kind of work, agents in tall columns, shells in short stacks, all of it spread across a 5120×1440 ultrawide. That’s the what. This one is the why, and then the part that actually decides whether any of it pays off: how I talk to the agents once they’re in their panes.

Short version: the layout isn’t decoration. It’s context management for two kinds of brains, mine and the models’, and the prompts are the same idea at a smaller scale. Every tool in this stack, Claude Code, Codex, Cursor, and the human, gets worse the more unrelated junk is in its head. So I keep the junk out, structurally first and then verbally.

Part 1: Why I break it out like this

Spaces are folders because the folder is the agent’s whole world

An agent’s working directory isn’t a detail. It’s the edge of its universe. Whatever it can read, grep, build, and break starts at that folder. So when I make one space per folder, I’m not organizing windows. I’m drawing the same boundary the agent already lives inside, and making it visible to me.

That buys three things:

The sidebar is a readout of my working memory. Ten spaces means ten things in flight. If a folder isn’t in the sidebar, I’m not working on it. When I’m done, the space goes away. There’s no “misc” space and no “scratch” space, because “misc” is where context goes to die.
Blast radius is legible. If something is going sideways in ledger-service, the dot next to ledger-service says so. Nothing in that space can be quietly editing storefront-web, because it doesn’t live there.
It pairs with worktrees. When I need two implementation agents on the same repo at once, each gets its own worktree and its own branch, and in Herdr that’s just another folder, so another space (herdr worktree create does this directly). I made the full case in Giving Every Agent Its Own Branch: Git Worktrees for Parallel AI Work: “Run two coding agents against a single working tree and they will fight.” Spaces-as-folders keeps the referee’s view honest.

Tabs are modes because each mode is a different verb

I name tabs domain-modeling, implementation, and pr-reviews because those are three different jobs. Each one comes with a different posture from me and a different register of prompt to the agent.

Tab	The verb	My posture	What the agent is allowed to do
`domain-modeling`	think	Skeptical, Socratic	Read, reason, write a model. No code. Plan mode.
`implementation`	change	Directive, fenced	Edit inside an explicit scope, after an approved plan
`pr-reviews`	judge	Adversarial, evidentiary	Read the diff, report findings. No fixing.

The failure mode I’m designing against is verb drift. You start a conversation reviewing a PR, you say “huh, good catch,” and three messages later the “reviewer” is rewriting the module, now with the confidence of something that just graded its own homework. Separate tabs, and separate agent sessions in them, make the verb change explicit. If I want a fix, I go to the implementation tab and ask for one there, with a scope. It’s the same rule I put on the read-only reviewer agents in Ask, Plan, Confirm: Making Agents Stop Before They Start: the reviewer doesn’t get to “slide from ‘here is a finding’ into ‘and I fixed it’ without crossing the same line everyone else crosses.” The tab is that line, drawn on screen.

There’s a human benefit too. When I switch to a tab named pr-reviews, my brain switches hats before I’ve read a word. Mode labels are cheap and they work.

A pane is one agent, one context window, one job

Every agent pane gets one job for the life of that conversation. When the job is done, the conversation ends, and the next job gets a fresh agent. Context windows are big now, but they aren’t free and they aren’t clean. Every stale plan, abandoned approach, and “actually, ignore that” sits in there and pulls on the next answer. A pane per job is the cheapest context hygiene there is.

The panes next to an agent are there for verification, not decoration:

In domain-modeling, the glossary and domain notes sit beside the agent, so I can check that it’s using the words we agreed on.
In implementation, a diffstat and a shell sit beside the agents, so I can see what actually changed, not what the agent says changed.
In pr-reviews, the raw diff sits beside the reviewer, so every file:line it cites can be checked by moving my eyes and nothing else.

Trust, but keep the evidence in the same field of view.

Tall for agents, short for shells, and why 32:9 matters

Agents emit long, structured output (plans, findings, tables), so they get full-height vertical splits. Shells and watchers are glanceable, so they get short horizontal stacks in a utility column. That isn’t a style choice. It’s matching pane shape to how much reading each pane needs.

The ultrawide makes it work without compromise. At 5120×1440 I get three or four full-height agent columns and the utility column and the sidebar, with nothing hidden behind a zoom or another tab. That matters for one reason above all: a blocked agent is a cost that accrues silently. Herdr’s sidebar tells me that something is blocked. Having it physically on screen tells me why, in peripheral vision, without moving. The less effort it takes to notice, the shorter the stall.

Part 2: Keeping Claude, Codex, and Cursor effective and accurate

All of the above is plumbing. The water is language.

I’ve been on this soapbox a while. In Precision in Words, Precision in Code: The Power of Writing in Modern Development I argued that “Writing isn’t just ‘extra work’—it’s the work that clarifies, simplifies, and accelerates everything else,” and in AI Prompt Engineering: Mastering Language Constructs I went into how the actual constructs, imperatives, conditionals, and contextual markers, shape what comes back. With agents that act, not just answer, the stakes go up. A vague word in a chat gets you a vague paragraph. A vague word in an agent prompt gets you a nine-file diff.

Here’s what I actually do. Every example below is a real prompt and a real response, captured for the screenshots in this repo with Claude Code in plan mode against throwaway demo projects.

1. Use the domain’s words, and ban the overloaded ones

Accurate language starts with the same word meaning the same thing every time: in the docs, in the code, in the prompt, in the review. That’s just ubiquitous language from domain-driven design, and agents need it more than humans do, because they fill every gap with the most statistically likely meaning, which is rarely your meaning.

So I keep a glossary in the notes repo (more on that in Notes Repos as Shared Project Context, where the glossary is described as “cheap insurance against confident mistakes”), and I put the words in the prompt, including the ones that are off limits:

Read docs/domain.md and internal/ledger/entry.go. Using only the domain terms Account, Entry, Journal, and Posting (never ‘transaction’), describe a Posting aggregate: its invariants, what it owns, and what it must reject. Do not write code. Output a short markdown model.

“Transaction” is banned because in a ledger it’s hopelessly overloaded: a database transaction, a business event, a bank-statement line. Pick one meaning, give it its own word, and forbid the ambiguous one. Here’s what came back:

Two things to notice. The model sticks to Account, Entry, Journal, Posting the whole way through. And it marks which rules it inferred from general double-entry practice versus which came from the repo: “Rules marked (inferred) come from standard double-entry practice, not the repo.” That distinction is gold in domain modeling, and it showed up because the prompt was explicit about where the truth lives (those two files).

2. Lead with scope, as the first word

My implementation prompts literally start with the word Scope:.

Scope: cmd/main.go only. Plan a minimal CLI entrypoint that loads a Journal from a JSON file and prints the balance per Account. Do not touch internal/ledger. Keep the planned diff under 50 lines.

Scope first, because everything after it gets read in its light. Then a positive fence (only this file), a negative fence (not that package), and a size limit stated as a number, not a vibe. “Keep it small” means nothing. “Under 50 lines” is checkable. I went deeper on why scope and diff size are the main levers in Hurting or Helping Devs?.

The payoff:

The repo had no go.mod, so cmd/main.go couldn’t import internal/ledger. An unfenced agent “fixes” that by adding a module file, rewiring imports, and maybe stubbing a Journal type in the package I’d fenced off. This one stopped and asked, offered the in-scope option as the recommendation, and named the out-of-scope option as out of scope: “This touches a second file, which goes beyond the cmd/main.go-only scope.” Herdr flagged it blocked, and I could see it on the ultrawide without switching anything. That’s the whole system working: precise words create the stop, and the layout makes the stop cheap.

3. Pick verbs that mean exactly one thing

plan, implement, review, and fix are different verbs, and I never let one stand in for another.

Plan means produce a plan and change nothing. I back it with tooling: agents start in plan mode (claude --permission-mode plan; Codex and Cursor have their own read-only and ask-first modes), so the verb and the permission agree.
Implement means change things inside the approved plan’s scope, and only after an explicit yes. Approval is scoped to the plan that was approved. This is the three-beat gate from Ask, Plan, Confirm.
Review means read and report. It doesn’t mean fix.
Fix means a new implementation task, with its own scope, in the implementation tab.

When the verb is precise, the agent’s job is precise, and so is my evaluation of whether it did the job.

4. Specify the shape of the answer

If I can’t check an answer quickly, I’ll check it badly. So I specify the output format in a way that’s easy to verify against the pane next door:

Review the diff main…feature/posting-aggregate. Classify every change as additive, mutative, or destructive. Flag any naming that drifts from docs/domain.md. Report findings as a numbered list with file:line. Do not propose rewrites outside the diff.

“Additive, mutative, or destructive” isn’t decoration either. It’s the vocabulary from Additive vs. Mutative vs. Destructive Code Changes (and Why AI Agents Love the Wrong One at 2:13AM), and giving the reviewer that vocabulary makes it classify instead of just commenting.

And because the question was precise, the reviewer had attention left over for what mattered. Under “side notes” it flagged that any Direction other than the exact string "debit", including "" or "Debit", gets counted as a credit, and that an empty Posting passes validation as balanced. Neither is a naming issue. Both are exactly the kind of quiet behavioral drift that post warned about: code that looks right and behaves like an alien artifact. A vague “please review this PR” tends to bury that under eleven style nits.

The same pattern works for any review lens. On the storefront demo, “integer cents only, no floating point until display formatting. Numbered findings with file:line, severity first” produced a short, ranked list I could verify in about thirty seconds.

5. Front-load the context instead of looping for it

A thin prompt plus five rounds of correction is worse than one well-fed prompt, and more expensive. I made that argument at length in “Loop Engineering” Is Mostly Just Broken SDLC Wearing a Costume: “A well-fed single pass beats a starved five-pass loop most of the time.” In practice that means every prompt names where the truth lives: the files to read, the glossary, the plan in the notes repo, the decision that was already made and shouldn’t be re-litigated.

It also means the environment has to be ready. An agent that can’t build, test, or reach the systems it needs will improvise, and improvisation is the opposite of accuracy. The checklist I use for that is in Agent-Ready Local Environments and MCP: “An agent with broad access and no rules will eventually do something technically impressive and socially awful.”

6. Same template, any agent

Claude Code, Codex, and Cursor have different personalities and different defaults, but the prompt shape carries across them. Herdr doesn’t care which one is in a pane. It detects them all, so I can put a different tool in the reviewer pane than the one that wrote the code. A second model has different blind spots, and that’s the point of a second opinion. The template:

Scope: <path(s)> only. <Do not touch: path(s)>.
Context: Read <files / glossary / plan>. Use the terms <A, B, C>; never <overloaded term>.
Task (<verb: plan | implement | review>): <one sentence, one job>.
Constraints: <additive only | under N lines | no new dependencies | tests first>.
Output: <numbered list | file:line | severity first | markdown model | test names first>.
If anything above is ambiguous or blocks you, stop and ask.

That last line matters most. It turns “the agent is stuck” from a silent failure into a blocked state, which Herdr turns into a dot I can see from across the room.

7. When an agent goes blocked or done

The loop on my side is short and boring, on purpose:

Blocked: read the question in place (it’s already on screen), answer it, and if the answer is a decision, write it into the notes repo’s decisions file so the next session doesn’t ask again.
Done: read the output against the evidence pane next to it. If it’s a plan, approve it or correct it. If it’s findings, open a scoped implementation task in the other tab for anything worth fixing.
Finished job: end that agent’s conversation. New job, new agent, clean context.

The point of all of it

The spaces, tabs, panes, and ultrawide exist to keep context small, boundaries visible, and stalls cheap. The prompting exists to keep meaning precise. Neither works without the other. A gorgeous Herdr layout full of agents with vague instructions is just a very wide way to watch things go wrong in parallel, and the most precise prompt in the world doesn’t help if the agent has been blocked on it since lunch and you never noticed.

Draw the boundaries on screen. Draw them again in words. Then let the agents work.

References

Herdr

herdr.dev · Docs · Concepts · Agents · Socket API · GitHub

Previously on Composite Code

Precision in Words, Precision in Code: The Power of Writing in Modern Development (Dec 2024)
AI Prompt Engineering: Mastering Language Constructs (Feb 2025)
Architects Evolving: How AI Is Reshaping the Role (Nov 2025)
AI Is Forcing Docs To Finally Grow Up (Dec 2025)
Additive vs. Mutative vs. Destructive Code Changes (and Why AI Agents Love the Wrong One at 2:13AM) (Feb 2026)
Hurting or Helping Devs? (Apr 2026)
The Gap Is Widening and It’s Not Slowing Down (May 2026)
“Loop Engineering” Is Mostly Just Broken SDLC Wearing a Costume (Jul 2026)
Giving Every Agent Its Own Branch: Git Worktrees for Parallel AI Work (Aug 2026)
Ask, Plan, Confirm: Making Agents Stop Before They Start (Aug 2026)
Agent-Ready Local Environments and MCP (Sep 2026)
Notes Repos as Shared Project Context (Sep 2026)

Elsewhere

Martin Fowler, Ubiquitous Language

Herding Agents: How I Actually Use Herdr (on a Ridiculously Wide Screen)

2026-10-09T10:49:10-07:00

One coding agent is a tool. Two is a pair. Six is a zoo, and somebody has to know which animal is chewing on the fence.

For most of this year that somebody was me, armed with a pile of terminal windows, a couple of tmux sessions I half remembered setting up, and the nagging feeling that some agent somewhere had been politely waiting on a yes/no question for twenty minutes. Agents don’t tap you on the shoulder. They sit there, cursor blinking, and burn the most expensive thing in the whole setup, which is wall-clock time.

Herdr fixed most of that for me. This post covers what it is, how it works, and how I lay it out day to day. There’s a companion post (TBD) on why I split things up the way I do and how I prompt inside the panes. This one is the tour.

What Herdr is

Herdr bills itself as “the runtime your coding agents live on.” Less marketing, more mechanics: it’s a terminal workspace manager built for AI coding agents. If you’ve used tmux, you already have about 70% of the mental model. It’s:

A single Rust binary. No Electron and no browser tab pretending to be a terminal. brew install herdr, or curl -fsSL https://herdr.dev/install.sh | sh, and you’re done.
Something that runs inside your terminal. iTerm, Ghostty, WezTerm, kitty, Alacritty, even inside tmux if you enjoy nesting dolls. It doesn’t replace your terminal. It organizes what’s in it.
A client/server pair. A background server owns the terminal sessions. The thing you look at is just a client. Close the window, detach, drop SSH, and the agents keep working. Reattach and they’re right where you left them. (I learned this the fun way while building the screenshots for this post: the demo window got closed out from under me mid-run, and every agent was still sitting there when I reattached.)
Agent-aware. This is the actual point. Herdr detects Claude Code, Codex, Cursor’s agent, Copilot CLI, OpenCode, Devin and a pile of others, then classifies each one’s state so you don’t have to go look.

The model: session › space › tab › pane

Herdr’s concepts page lays out four nouns. The precision matters, so here they are the way I think about them.

Noun	What Herdr says it is	What I use it for
Session	A persistent server namespace. `herdr` attaches to the default one; `herdr --session <name>` gives you an isolated one.	My one long-lived “everything” session, plus throwaway named sessions when I’m experimenting (like the demo session for this post’s screenshots).
Workspace (the sidebar calls them spaces)	“The top-level project container. Use one workspace per repo, task, or investigation.”	One space per folder I’m working in. Full stop.
Tab	“A layout inside a workspace.”	One tab per kind of work: `pr-reviews`, `domain-modeling`, `implementation`.
Pane	“A real terminal.”	One agent, or one shell, or one thing I’m watching. Arranged in vertical or horizontal splits.

Then there are the agent states, which make the whole thing worth it:

Blocked: the agent needs input, approval, or a decision. This is the one you care about.
Working: it’s chewing.
Done: it finished and you haven’t looked yet.
Idle: it finished and you have looked.
Unknown: Herdr can’t tell, usually because it’s a plain shell.

Those states roll up into the sidebar. Each space gets a dot, and there’s an agents list underneath that shows every agent across every space. One glance and I know who’s blocked, who’s working, and who’s done and quietly waiting for me to read their output.

That’s the shoulder tap agents never gave me.

How it works (enough to be dangerous)

Keyboard

The default prefix is ctrl+b, deliberately tmux-flavored, and the keymap is “prefix-first” so Herdr doesn’t steal keystrokes from your shell, editor, or the agent. The ones I actually use:

Keys	Does
`prefix` `v`	Split vertical: new pane to the right, side by side
`prefix` `-`	Split horizontal: new pane below, stacked
`prefix` `h` / `j` / `k` / `l`	Move focus left / down / up / right
`prefix` `z`	Zoom the current pane (and again to unzoom)
`prefix` `r`	Resize mode
`prefix` `c`	New tab
`prefix` `shift+t`	Rename tab
`prefix` `1`…`9`	Jump to tab N
`prefix` `shift+n`	New workspace (space)
`prefix` `w`	Workspace picker
`prefix` `b`	Toggle the sidebar
`prefix` `q`	Detach (everything keeps running)
`prefix` `?`	Help, for when you forget all of the above

Run herdr --default-config to see the full annotated config, including how to remap any of it. Mouse works too: click, drag, split. I won’t judge.

The CLI and socket API

Here’s where it stops being “tmux with a status light.” Everything you can do with a keystroke you can also do from the command line over Herdr’s socket API. That includes the agents themselves, since they’re running in panes Herdr owns. Every screenshot in this post came from a layout I scripted like this, in an isolated session so my real one never got touched:

# a space per folder
herdr workspace create --cwd ~/Codez/ledger-service --label ledger-service

# name the first tab for the kind of work happening in it
herdr tab rename w3:t1 domain-modeling

# agent on the left, reference material on the right, split again downward
herdr pane split w3:p1 --direction right --ratio 0.5
herdr pane split w3:p2 --direction down  --ratio 0.5

# start an agent and hand it a scoped prompt
herdr pane run w3:p1 "claude --permission-mode plan"
herdr agent prompt w3:p1 "Read docs/domain.md and internal/ledger/entry.go. Using only the domain terms Account, Entry, Journal, and Posting (never 'transaction'), describe a Posting aggregate..."

# and then, the good part
herdr agent list      # who is working / blocked / done
herdr agent wait w3:p1 --until blocked --until done   # block until it needs me
herdr pane read w3:p1 # read what it said, without switching to it

That’s enough rope for agents to spawn panes, prompt each other, and notice when a sibling is blocked. I’ll admit that’s both very cool and the setup for a future post titled something like “How My Agents Unionized.”

Integrations

herdr integration install claude (or codex, copilot, opencode, and so on) drops a small hook into each agent so it reports its lifecycle to Herdr directly, rather than Herdr inferring state by watching the screen. herdr integration status shows what’s wired up. Do this. The state dots get noticeably more accurate.

The rest of the toolbox

Worktrees: herdr worktree create makes a git worktree and opens it as its own space. It fits the one-branch-per-agent pattern I wrote about in Giving Every Agent Its Own Branch: Git Worktrees for Parallel AI Work.
Remote machines: herdr --remote <ssh-target> puts panes on other boxes in the same sidebar, and they reconnect independently. See Connecting machines.
Session state: layouts persist and restore. Processes don’t survive a server restart (nothing does, that’s physics), but the shape of your workspace does. See Session state.

How I actually use it

That’s the brochure. Here’s what my screen looks like.

One space per folder

Every folder I’m actively working in gets its own space, named after the folder. As I type this, my sidebar has ten of them: interlinedlist, then interlinedlist-ios, interlinedlist-macos-native, interlinedlist-windows-app, interlinedlist-android, interlinedlist-architecture-migration, a couple of client and side projects, and the repo this post lives in.

Is that a lot? Sure. Is it more than what’s actually in my head? No. It’s exactly what’s in my head, and that’s the point. The sidebar is a direct readout of my working memory. If it isn’t a folder I’m working in, it doesn’t get a space. When I stop working in a folder, the space goes away. My config.toml even sorts the agent list by space (agent_panel_sort = "spaces") so the agents group the same way my brain does.

It’s mostly for my own mental organization, and I’m fine with that. The agents don’t care what the sidebar looks like. I do.

One tab per kind of work

Inside a space, tabs aren’t “more room.” Tabs are modes. I name them for the kind of work being done:

domain-modeling: one agent, plus the glossary, domain notes, and recent history open next to it. The agent’s job is to think, not type.
implementation: one or more agents, each scoped to a slice of the code, with a diffstat or test runner and a plain shell beside them.
pr-reviews: a reviewing agent next to the actual diff, so I can check its claims against the code without switching windows.

Most spaces have just one tab, whichever mode that folder is in today. The busy ones have all three. Switching tabs is switching hats, and the tab label tells me which hat is on before I’ve read a single line.

One or more terminals, split vertically or horizontally

Within a tab I keep a simple rule: agents get tall columns, shells get short stacks.

Agents produce long, scrolling output: plans, numbered findings, diffs they want to show me. That wants height. So agents get vertical splits (prefix v), side by side, full height. Shells, test runners, git log, diffstats and log tails are glanceable. They get horizontal splits (prefix -) stacked in the right-hand column.

The ultrawide changes the math

Here’s the cheat. I run all of this on a 5120×1440, 32:9 ultrawide: two 2560×1440 monitors fused into one with no bezel down the middle. At normal terminal font sizes, that’s room for three or four full-height agent columns plus a stacked utility column and the sidebar, all visible at once, nobody squished.

On a regular 16:9 screen, that same implementation tab looks like this:

Workable. Tight. Add a third agent and you’re zooming panes in and out all day. On the ultrawide it’s this:

The difference isn’t really “more stuff.” It’s that nothing is hidden. When an agent goes blocked, I don’t need to find it. It’s already on screen, in my peripheral vision, with a red dot in the sidebar to back it up. Herdr tells me who needs attention. The ultrawide means I can see why without changing anything.

That screenshot is an agent I’d told to touch cmd/main.go only. It found the repo had no go.mod, which meant staying in scope required a decision. So it stopped, went blocked, and asked. Herdr flagged it, and I could see the question without switching anything. Total cost: one keystroke. The alternative, an agent “helpfully” adding a module file and rewiring imports across a package I’d explicitly fenced off, costs a lot more than one keystroke. Why that worked, and how I write prompts so it keeps working, is the next post (TBD).

The short version

Herdr is a terminal workspace manager for coding agents: one Rust binary, runs in your terminal, background server, agent-state detection.
Spaces are folders. One per folder you’re working in. It’s a mirror of your head, not a filing cabinet.
Tabs are modes of work. pr-reviews, domain-modeling, implementation.
Panes: agents tall, shells short.
The sidebar tells you who’s blocked. The ultrawide lets you see why without moving.

Install it, run herdr, hit ctrl+b ?, and give it a day. Worst case, you’ve got a nice tmux. Best case, you stop finding agents that have been patiently blocked since lunch.

References

Herdr: herdr.dev · Docs · Concepts · Configuration & keybindings · Agents · Socket API · Source on GitHub
Related on Composite Code:
Screenshots were captured on Herdr 0.8.0 in an isolated --session, against throwaway demo repos, with Claude Code in plan mode (nothing was written to disk by the agents).

Notes Repos as Shared Project Context

2026-09-25T15:01:30-07:00

The first thing most people put in a new repository is code. That is fine. It is also incomplete.

When I start a real project these days, something I expect Cursor, Claude, Codex, and I to grind on together for more than an afternoon, I create a notes repo early. Sometimes it sits beside the product repo. Sometimes it lives inside the product repo as a deliberate notes/ or docs/project/ tree. Either way, the job is the same: give the humans and the agents one durable place to put the stuff that does not belong in source files but absolutely belongs in the work.

This is not documentation for show. It is shared working memory.

Why a notes repo, not a pile of chat

Chat is ephemeral. Agent transcripts are useful, but they are not a system of record. Plans get buried. Decisions get restated wrong three sessions later. Somebody new (human or model) shows up and has to rediscover who owns what, what “done” means, and which external systems are in play.

A notes repo fixes that by making the context something you can open and search. Cursor can open it. Claude can read it. Codex can search it. I can edit it when the plan changes. Everyone is looking at the same markdown, not reconstructing the project from memory and half-remembered Slack threads.

I treat it as the base layer under the code. The product repo holds the build. The notes repo holds the thinking that keeps the build pointed in the right direction.

What actually goes in it

I keep the structure boring on purpose. Fancy wikis rot. Flat, named markdown files get used.

Project brief. What we are building, who it is for, what it is not. One page. If you cannot say it in a page, you do not have a project yet. You have a mood.

Current plan. The active plan, not a museum of every plan that ever existed. Short enough to approve in one breath. Specific enough that “yes” means something. When the plan changes, rewrite it. Do not append a novel of abandoned approaches unless those approaches teach a real constraint.

Decisions. Lightweight ADRs, or just dated notes: we chose X because Y, and Z is explicitly out of scope. Agents love to re-litigate settled choices. Write the settlement down.

Team map. Who is who. Humans, roles, ownership. Which agent lanes exist if you have specialized agents. Who reviews security. Who can approve schema changes. “The team” includes the tools now, so name them and their jobs.

Glossary. Product words, domain words, the three acronyms everybody uses differently. Cheap insurance against confident mistakes.

Working notes. Scratch space for the current thread of work: open questions, links to tickets, sketches, “we tried this and it failed because…”. This is the joint notebook Cursor, Claude, Codex, and I write into while we work.

Pointers out. Links to the product repos, design files, staging URLs, runbooks, and the environment/setup notes. The notes repo should know where the rest of the world lives without trying to duplicate it.

I do not put secrets here. I put the names of secrets and where they are supposed to live. That distinction matters.

How I use it with agents

The pattern is simple: before an agent starts implementing, it reads the brief, the current plan, and the relevant decision notes. After a meaningful chunk of work, it updates the working notes or the plan so the next session does not start from zero.

That pairs cleanly with the ask-plan-confirm habit I already use. The notes repo is where the approved plan lives between sessions. The agent does not get to invent a new goal because the chat scrolled away.

A few practical rules that hold up:

One current plan file. Archive old plans if you must, but do not make the agent guess which file is live.
Prefer short files with clear names over one giant NOTES.md.
Write for the next reader who was not in the room. That reader might be you in two weeks, or an agent with a fresh context window.
Update notes as part of the work, not as a cleanup chore after merge. If it is optional, it will not happen.

What this saves

It saves re-explaining the project every morning. It saves the “wait, who owns auth?” loop. It saves agents from optimizing the wrong goal because the real goal lived in a Slack thread from Thursday.

It also saves me from being the only continuity process on the team. Continuity is a file. Version it. Diff it. Argue with it in a pull request if the decision is big enough.

Start smaller than you think

You do not need a knowledge management platform. You need a repo, a README that says what the repo is for, and a handful of markdown files that stay honest.

Create it when the project becomes real. Keep it next to the code in your mental model. Make every agent treat it as required reading before they touch the product tree.

Code is the artifact. Notes are the shared brain that keeps the artifact from wandering off.

Toggle Switches When the Thing Behind the Switch Is a Whole System

2026-09-24T18:00:31-07:00

There is a version of a feature toggle that looks wonderfully simple in a pull request:

if (features.newThing) {
  doNewThing();
}

Then newThing turns out to be a data pipeline, a shipping promise, or a screen people have already started using. Now the switch has an owner, a scope, a failure mode, and a date when we ought to remove it. The if statement is the least interesting part.

I want to work through three concrete examples. The first moves a flat-file order feed through Databricks, then cuts a tenant over to normalized PostgreSQL behind a data API. The second turns on expedited shipping, with the same decision implemented in TypeScript and Go. The third releases a saved-filters interface in React and SwiftUI. Each is a different kind of switch, and each can surprise you if you treat it as a Boolean sprinkled through the codebase.

The examples are deliberately small enough to read, but the boundaries are real: immutable inputs, idempotency, tenant scoping, stable rollout decisions, and the difference between hiding a button and actually controlling a capability. Let’s get into it.

1. The order feed: Databricks today, PostgreSQL data API tomorrow

Imagine a partner dropping one immutable CSV file per batch into object storage. Each row is an item on an order:

order_id,customer_id,customer_name,sku,quantity,event_at
o-100,c-7,Ada,rail-pass,2,2026-09-24T10:00:00Z
o-100,c-7,Ada,seat-upgrade,1,2026-09-24T10:00:00Z
o-101,c-8,Grace,rail-pass,1,2026-09-24T10:01:00Z

The existing path loads the file into a Databricks Delta bronze table, then produces a current-order table for queries. The proposed path reads that same object, validates it, and sends a complete batch to a private data API. The API writes three normalized PostgreSQL tables: customers, orders, and order_items.

immutable CSV in object storage
            |
       batch worker
            |
     tenant route snapshot
       /           \
  Databricks     PostgreSQL data API
  COPY INTO       validate + transaction
  bronze/curated  customers/orders/items
       \           /
       read adapter for the tenant

The switch is a tenant route, not a random choice made separately for each row. For a given batch, resolve the route once and put it in the batch log. Changing the route while a file is half processed would create an impressively confusing incident.

The route and the file contract

I keep the control-plane data separate from the pipeline code. In this example the configuration is a checked, versioned snapshot supplied to the worker; in production it might come from a flag service. The important bit is that the worker receives one decision and uses it for the whole operation.

// pipeline/route.ts
export type PipelinePath = "databricks" | "postgres";

export interface RouteSnapshot {
  version: number;
  defaultPath: PipelinePath;
  tenants: Record<string, PipelinePath>;
}

export interface BatchRef {
  tenantId: string;
  batchId: string;
  fileName: string; // a basename under the tenant's immutable S3 prefix
  sha256: string;   // digest of the file's bytes, recorded when uploaded
}

export function choosePath(
  config: RouteSnapshot,
  tenantId: string,
): { path: PipelinePath; configVersion: number } {
  return {
    path: config.tenants[tenantId] ?? config.defaultPath,
    configVersion: config.version,
  };
}

export function checkBatchRef(batch: BatchRef): void {
  if (!/^[a-z0-9-]{1,64}$/.test(batch.tenantId) ||
      !/^[a-zA-Z0-9-]{1,100}$/.test(batch.batchId) ||
      !/^[a-zA-Z0-9-]+\.csv$/.test(batch.fileName) ||
      !/^[a-f0-9]{64}$/.test(batch.sha256)) {
    throw new Error("Invalid batch reference");
  }
}

The file key is a basename by design. The worker constructs the object path from a fixed bucket and tenant prefix. That keeps a caller from turning a batch submission into “please read whatever URL I hand you.” The SHA-256 is about replay identity: batch-42 with new bytes is an error, not a cute way to overwrite history.

The existing Databricks path

Here is the setup on the lakehouse side. The external location and warehouse already have access to the bucket. I’m showing Databricks SQL because COPY INTO is a good fit for an incremental feed of files, and the loaded-file tracking makes retries tractable. If your feed is millions of files, Databricks points you toward Auto Loader instead.

CREATE TABLE IF NOT EXISTS main.orders.order_rows_bronze (
  order_id STRING,
  customer_id STRING,
  customer_name STRING,
  sku STRING,
  quantity STRING,
  event_at STRING
) USING DELTA;

CREATE TABLE IF NOT EXISTS main.orders.order_rows_current (
  order_id STRING,
  customer_id STRING,
  customer_name STRING,
  sku STRING,
  quantity INT,
  event_at TIMESTAMP
) USING DELTA;

The TypeScript adapter submits a SQL statement to a warehouse and waits for completion. For this example every query returns either no rows or a small lookup result; large query results need the API’s external-links disposition and a separate paging design.

// pipeline/databricks.ts
type StatementResult = {
  statement_id?: string;
  status: { state: string; error?: { message: string } };
  result?: { data_array?: string[][] };
};

export class DatabricksSql {
  constructor(
    private readonly host: string,
    private readonly token: string,
    private readonly warehouseId: string,
  ) {}

  private async request(path: string, init?: RequestInit): Promise<StatementResult> {
    const response = await fetch(`${this.host}${path}`, {
      ...init,
      headers: {
        Authorization: `Bearer ${this.token}`,
        "Content-Type": "application/json",
        ...init?.headers,
      },
    });
    if (!response.ok) throw new Error(`Databricks HTTP ${response.status}`);
    return response.json() as Promise<StatementResult>;
  }

  async execute(statement: string, parameters: { name: string; value: string }[] = []) {
    let result = await this.request("/api/2.0/sql/statements", {
      method: "POST",
      body: JSON.stringify({
        warehouse_id: this.warehouseId,
        statement,
        parameters,
        wait_timeout: "10s",
        disposition: "INLINE",
        format: "JSON_ARRAY",
      }),
    });

    const deadline = Date.now() + 120_000;
    while (result.status.state === "PENDING" || result.status.state === "RUNNING") {
      if (!result.statement_id || Date.now() > deadline) {
        throw new Error("Databricks statement exceeded worker deadline");
      }
      await new Promise(resolve => setTimeout(resolve, 1_000));
      result = await this.request(`/api/2.0/sql/statements/${result.statement_id}`);
    }
    if (result.status.state !== "SUCCEEDED") {
      throw new Error(result.status.error?.message ?? `Statement ${result.status.state}`);
    }
    return result.result?.data_array ?? [];
  }
}

export async function loadDatabricksBatch(
  sql: DatabricksSql,
  batch: BatchRef,
): Promise<void> {
  checkBatchRef(batch);
  const prefix = `s3://example-order-feed/${batch.tenantId}/`;

  // Only the basename is interpolated; checkBatchRef restricts its alphabet.
  await sql.execute(`
    COPY INTO main.orders.order_rows_bronze
    FROM '${prefix}'
    FILEFORMAT = CSV
    FILES = ('${batch.fileName}')
    FORMAT_OPTIONS ('header' = 'true')
  `);

  // The source file is a complete order snapshot. Deduplicate source rows
  // before MERGE: two matching source rows for one target key are ambiguous.
  await sql.execute(`
    MERGE INTO main.orders.order_rows_current AS target
    USING (
      SELECT order_id, customer_id, customer_name, sku,
             CAST(quantity AS INT) AS quantity,
             CAST(event_at AS TIMESTAMP) AS event_at
      FROM main.orders.order_rows_bronze
      QUALIFY ROW_NUMBER() OVER (
        PARTITION BY order_id, sku ORDER BY CAST(event_at AS TIMESTAMP) DESC
      ) = 1
    ) AS source
    ON target.order_id = source.order_id AND target.sku = source.sku
    WHEN MATCHED AND source.event_at >= target.event_at THEN UPDATE SET
      customer_id = source.customer_id,
      customer_name = source.customer_name,
      quantity = source.quantity,
      event_at = source.event_at
    WHEN NOT MATCHED THEN INSERT
      (order_id, customer_id, customer_name, sku, quantity, event_at)
      VALUES (source.order_id, source.customer_id, source.customer_name,
              source.sku, source.quantity, source.event_at)
  `);
}

There is a deliberate simplification here: the merge scans bronze and models upserts, not item deletion. If a later snapshot can remove an item, the source contract needs tombstones or a replace-whole-order operation. No toggle solves an undefined deletion contract. Also, in a real lakehouse I would key these tables by tenant, or put each tenant in its own governed schema; the sample’s order_id must be globally unique for the shown merge.

The PostgreSQL path and its data API

On the new path, the database is an implementation detail of the data API. The worker never gets a PostgreSQL connection string. A private, authenticated service receives a complete batch; the service owns validation, idempotency, and the transaction.

CREATE TABLE customers (
  tenant_id text NOT NULL,
  customer_id text NOT NULL,
  customer_name text NOT NULL,
  PRIMARY KEY (tenant_id, customer_id)
);

CREATE TABLE orders (
  tenant_id text NOT NULL,
  order_id text NOT NULL,
  customer_id text NOT NULL,
  event_at timestamptz NOT NULL,
  PRIMARY KEY (tenant_id, order_id),
  FOREIGN KEY (tenant_id, customer_id)
    REFERENCES customers (tenant_id, customer_id)
);

CREATE TABLE order_items (
  tenant_id text NOT NULL,
  order_id text NOT NULL,
  sku text NOT NULL,
  quantity integer NOT NULL CHECK (quantity > 0),
  PRIMARY KEY (tenant_id, order_id, sku),
  FOREIGN KEY (tenant_id, order_id)
    REFERENCES orders (tenant_id, order_id) ON DELETE CASCADE
);

CREATE TABLE ingestion_batches (
  tenant_id text NOT NULL,
  batch_id text NOT NULL,
  sha256 char(64) NOT NULL,
  committed_at timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (tenant_id, batch_id)
);

The flat file repeats customer names and order IDs. The relational model stores each customer and order once, then each item under that order. This normalization is useful for operational queries and constraints; it is not a claim that PostgreSQL is automatically the better analytics engine. The switch is about the workload we actually need to serve.

The worker downloads the immutable object and parses it. This uses @aws-sdk/client-s3 and csv-parse/sync. I cap the file at 10,000 rows here so the data API can handle one transaction; larger feeds should use bounded chunks with a manifest and a stronger commit protocol.

// pipeline/postgres-path.ts
import { S3Client, GetObjectCommand } from "@aws-sdk/client-s3";
import { parse } from "csv-parse/sync";
import { createHash } from "node:crypto";

export type OrderRow = {
  order_id: string;
  customer_id: string;
  customer_name: string;
  sku: string;
  quantity: number;
  event_at: string;
};

const s3 = new S3Client({});

export async function loadPostgresBatch(
  batch: BatchRef,
  apiBase: string,
  serviceToken: string,
): Promise<void> {
  checkBatchRef(batch);
  const key = `${batch.tenantId}/${batch.fileName}`;
  const object = await s3.send(new GetObjectCommand({
    Bucket: "example-order-feed", Key: key,
  }));
  if (!object.Body) throw new Error("Empty object body");
  const bytes = await object.Body.transformToByteArray();
  const digest = createHash("sha256").update(bytes).digest("hex");
  if (digest !== batch.sha256) throw new Error("Batch content changed");

  const records = parse(Buffer.from(bytes), {
    columns: true, skip_empty_lines: true, bom: true,
  }) as Record<string, string>[];
  if (records.length === 0 || records.length > 10_000) {
    throw new Error("Batch size outside accepted range");
  }
  const rows: OrderRow[] = records.map(row => ({
    order_id: row.order_id,
    customer_id: row.customer_id,
    customer_name: row.customer_name,
    sku: row.sku,
    quantity: Number(row.quantity),
    event_at: row.event_at,
  }));

  const response = await fetch(`${apiBase}/v1/batches`, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${serviceToken}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ ...batch, rows }),
  });
  if (!response.ok) throw new Error(`Data API rejected batch: ${response.status}`);
}

The API side is where the transaction belongs. zod checks the wire shape; PostgreSQL constraints still carry the final integrity guarantee. The gateway authenticates the service token and supplies the tenant identity; it must verify that the tenant in the request body matches that identity. I’m leaving gateway wiring out of this excerpt, but I would not expose this route to the public internet as an unauthenticated import endpoint.

// data-api/batches.ts (Fastify + pg + zod)
import Fastify from "fastify";
import pg from "pg";
import { z } from "zod";

const pool = new pg.Pool({ connectionString: process.env.DATABASE_URL });
const app = Fastify();
const rowSchema = z.object({
  order_id: z.string().min(1).max(100),
  customer_id: z.string().min(1).max(100),
  customer_name: z.string().min(1).max(200),
  sku: z.string().min(1).max(100),
  quantity: z.number().int().positive(),
  event_at: z.string().datetime({ offset: true }),
});
const batchSchema = z.object({
  tenantId: z.string().regex(/^[a-z0-9-]{1,64}$/),
  batchId: z.string().min(1).max(100),
  fileName: z.string().endsWith(".csv"),
  sha256: z.string().regex(/^[a-f0-9]{64}$/),
  rows: z.array(rowSchema).min(1).max(10_000),
});

app.post("/v1/batches", async (request, reply) => {
  const parsed = batchSchema.safeParse(request.body);
  if (!parsed.success) {
    return reply.code(400).send({ error: "Invalid batch payload" });
  }
  const input = parsed.data;
  // Gateway auth must bind this tenantId to the authenticated caller.
  const byOrder = new Map<string, typeof input.rows>();
  for (const row of input.rows) {
    const group = byOrder.get(row.order_id) ?? [];
    group.push(row);
    byOrder.set(row.order_id, group);
  }
  for (const rows of byOrder.values()) {
    const first = rows[0];
    const skus = new Set<string>();
    for (const row of rows) {
      if (row.customer_id !== first.customer_id ||
          row.customer_name !== first.customer_name ||
          row.event_at !== first.event_at || skus.has(row.sku)) {
        return reply.code(400).send({ error: "Inconsistent order snapshot" });
      }
      skus.add(row.sku);
    }
  }

  const client = await pool.connect();
  try {
    await client.query("BEGIN");
    const inserted = await client.query(
      `INSERT INTO ingestion_batches (tenant_id, batch_id, sha256)
       VALUES ($1, $2, $3) ON CONFLICT DO NOTHING RETURNING batch_id`,
      [input.tenantId, input.batchId, input.sha256],
    );
    if (inserted.rowCount === 0) {
      const existing = await client.query(
        `SELECT sha256 FROM ingestion_batches
         WHERE tenant_id = $1 AND batch_id = $2`,
        [input.tenantId, input.batchId],
      );
      await client.query("COMMIT");
      if (existing.rows[0]?.sha256 !== input.sha256) {
        return reply.code(409).send({ error: "Batch ID reused with new bytes" });
      }
      return reply.send({ status: "already-committed" });
    }

    for (const [orderId, rows] of byOrder) {
      const first = rows[0];
      await client.query(
        `INSERT INTO customers (tenant_id, customer_id, customer_name)
         VALUES ($1, $2, $3)
         ON CONFLICT (tenant_id, customer_id)
         DO UPDATE SET customer_name = EXCLUDED.customer_name`,
        [input.tenantId, first.customer_id, first.customer_name],
      );
      await client.query(
        `INSERT INTO orders (tenant_id, order_id, customer_id, event_at)
         VALUES ($1, $2, $3, $4)
         ON CONFLICT (tenant_id, order_id) DO UPDATE SET
           customer_id = EXCLUDED.customer_id,
           event_at = EXCLUDED.event_at
         WHERE orders.event_at <= EXCLUDED.event_at`,
        [input.tenantId, orderId, first.customer_id, first.event_at],
      );
      const current = await client.query(
        `SELECT event_at FROM orders WHERE tenant_id = $1 AND order_id = $2
         FOR UPDATE`,
        [input.tenantId, orderId],
      );
      if (new Date(current.rows[0].event_at).toISOString() !==
          new Date(first.event_at).toISOString()) continue; // stale snapshot

      await client.query(
        `DELETE FROM order_items WHERE tenant_id = $1 AND order_id = $2`,
        [input.tenantId, orderId],
      );
      for (const row of rows) {
        await client.query(
          `INSERT INTO order_items (tenant_id, order_id, sku, quantity)
           VALUES ($1, $2, $3, $4)`,
          [input.tenantId, orderId, row.sku, row.quantity],
        );
      }
    }
    await client.query("COMMIT");
    return reply.send({ status: "committed", orders: byOrder.size });
  } catch (error) {
    await client.query("ROLLBACK");
    throw error;
  } finally {
    client.release();
  }
});

Notice the batch marker is inserted inside the same transaction as the rows. If the transaction fails, the marker disappears too. Retrying the file then does useful work instead of being mistaken for a success. The FOR UPDATE also serializes replacement of one order’s items. For exact ties in event_at, the upstream contract should supply a monotonically increasing revision; timestamps alone cannot tell which different snapshot wins.

The read half of the data API keeps PostgreSQL behind the boundary too:

app.get<{ Params: { orderId: string } }>(
  "/v1/orders/:orderId",
  async (request, reply) => {
    // tenantId comes from authenticated gateway context, never a query param.
    const tenantId = request.headers["x-verified-tenant-id"] as string;
    const order = await pool.query(
      `SELECT o.order_id, o.event_at, c.customer_id, c.customer_name
       FROM orders o JOIN customers c
         ON c.tenant_id = o.tenant_id AND c.customer_id = o.customer_id
       WHERE o.tenant_id = $1 AND o.order_id = $2`,
      [tenantId, request.params.orderId],
    );
    if (order.rowCount === 0) return reply.code(404).send();
    const items = await pool.query(
      `SELECT sku, quantity FROM order_items
       WHERE tenant_id = $1 AND order_id = $2 ORDER BY sku`,
      [tenantId, request.params.orderId],
    );
    return { ...order.rows[0], items: items.rows };
  },
);

That x-verified-tenant-id header is only safe if a trusted gateway strips any client-supplied copy and injects its own value. In a direct deployment, put authentication and tenant extraction in a Fastify hook instead. The point is to show where tenant ownership is enforced, because “the new API is private” is not an authorization strategy.

The consumer needs the same route decision. Here the Databricks query uses a named parameter and the PostgreSQL side uses the data API. The adapter presents one order shape to its caller. This example assumes the order ID is globally unique in Databricks; if it is only unique per tenant, add tenant_id to the Delta tables and both predicates.

// pipeline/read-order.ts
export type OrderView = {
  orderId: string;
  customerId: string;
  customerName: string;
  eventAt: string;
  items: { sku: string; quantity: number }[];
};

export async function getOrder(
  config: RouteSnapshot,
  tenantId: string,
  orderId: string,
  deps: { databricks: DatabricksSql; apiBase: string; serviceToken: string },
): Promise<OrderView | null> {
  if (choosePath(config, tenantId).path === "postgres") {
    const response = await fetch(
      `${deps.apiBase}/v1/orders/${encodeURIComponent(orderId)}`,
      { headers: { Authorization: `Bearer ${deps.serviceToken}` } },
    );
    if (response.status === 404) return null;
    if (!response.ok) throw new Error(`Data API HTTP ${response.status}`);
    const row = await response.json() as {
      order_id: string; customer_id: string; customer_name: string;
      event_at: string; items: { sku: string; quantity: number }[];
    };
    return {
      orderId: row.order_id, customerId: row.customer_id,
      customerName: row.customer_name, eventAt: row.event_at,
      items: row.items,
    };
  }

  const rows = await deps.databricks.execute(`
    SELECT order_id, customer_id, customer_name, sku,
           CAST(quantity AS STRING), CAST(event_at AS STRING)
    FROM main.orders.order_rows_current WHERE order_id = :order_id
    ORDER BY sku
  `, [{ name: "order_id", value: orderId }]);
  if (rows.length === 0) return null;
  return {
    orderId: rows[0][0], customerId: rows[0][1],
    customerName: rows[0][2], eventAt: rows[0][5],
    items: rows.map(row => ({ sku: row[3], quantity: Number(row[4]) })),
  };
}

Finally, the worker uses its pinned route. Read traffic should use the tenant route only after the PostgreSQL side has been backfilled and checked.

// pipeline/worker.ts
export async function processBatch(
  config: RouteSnapshot,
  batch: BatchRef,
  deps: { databricks: DatabricksSql; apiBase: string; serviceToken: string },
) {
  checkBatchRef(batch);
  const decision = choosePath(config, batch.tenantId);
  // Persist {batchId, sha256, path, configVersion} in your job ledger.
  if (decision.path === "databricks") {
    await loadDatabricksBatch(deps.databricks, batch);
  } else {
    await loadPostgresBatch(batch, deps.apiBase, deps.serviceToken);
  }
  return decision;
}

My cutover order would be: add the new API and schema, backfill it from immutable files, compare order counts and sampled contents, run both paths in a controlled shadow period, then route one tenant’s writes and reads to PostgreSQL. Measure lag, API errors, rejected rows, and mismatches by tenant. Keep the original feed while rollback is needed. If writes have happened only on the new path, flipping the read switch back to Databricks can show stale data; rollback needs replay to the old path or a freeze until it catches up. That’s the part the tidy if statement never tells you.

2. Expedited shipping: a feature flag with a bill attached

Here’s a second example: turn on an expedited-shipping offer for a percentage of accounts. The new path does not merely paint a badge. It changes the checkout quote, so we need one stable decision per order and a record of the rule that produced the price.

The rule is: the account is included in the rollout, the destination is in the supported region, the cart subtotal is at least 7,500 cents, and the feature has not been killed globally. I use a stable FNV-1a hash of accountId for the rollout. It is small and portable across TypeScript and Go; it is not a security primitive. A production flag service can own the assignment instead, provided both stacks read the same assignment.

TypeScript implementation

// checkout/expedited.ts
export type ShippingFlag = {
  enabled: boolean;
  killSwitch: boolean;
  rolloutPercent: number; // integer 0..100
  revision: string;
};

export type QuoteInput = {
  accountId: string;
  subtotalCents: number;
  region: "US-LOWER-48" | "US-OTHER" | "INTERNATIONAL";
};

export type ShippingQuote = {
  standardCents: number;
  expeditedCents: number | null;
  decision: {
    offered: boolean;
    flagRevision: string;
    reason: string;
  };
};

export function bucket(accountId: string): number {
  let hash = 0x811c9dc5;
  for (const byte of new TextEncoder().encode(accountId)) {
    hash ^= byte;
    hash = Math.imul(hash, 0x01000193) >>> 0;
  }
  return hash % 100;
}

export function quoteShipping(input: QuoteInput, flag: ShippingFlag): ShippingQuote {
  if (!Number.isSafeInteger(input.subtotalCents) || input.subtotalCents < 0 ||
      !Number.isInteger(flag.rolloutPercent) ||
      flag.rolloutPercent < 0 || flag.rolloutPercent > 100) {
    throw new Error("Invalid quote input or flag configuration");
  }
  const reason = !flag.enabled || flag.killSwitch ? "disabled"
    : bucket(input.accountId) >= flag.rolloutPercent ? "outside-rollout"
    : input.region !== "US-LOWER-48" ? "unsupported-region"
    : input.subtotalCents < 7_500 ? "below-minimum"
    : "eligible";

  return {
    standardCents: 799,
    expeditedCents: reason === "eligible" ? 1499 : null,
    decision: {
      offered: reason === "eligible",
      flagRevision: flag.revision,
      reason,
    },
  };
}

// At checkout creation, persist the quoted cents and decision with the order.
// At payment confirmation, charge that stored quote after its normal expiry
// and inventory checks. Do not ask the flag again halfway through checkout.

This is where teams often reach for Math.random() and accidentally give the same customer a different offer on every refresh. The bucket makes the rollout stable. The saved decision makes an in-progress checkout stable even if the operator moves from 10% to 20% while a customer is entering their card details.

The same boundary in Go

If another service computes a quote in Go, it must agree on byte encoding, hash, threshold, region names, and cents. This uses only the standard library.

package shipping

import (
    "errors"
    "hash/fnv"
)

type Flag struct {
    Enabled        bool
    KillSwitch     bool
    RolloutPercent int
    Revision       string
}

type Input struct {
    AccountID     string
    SubtotalCents int64
    Region        string
}

type Decision struct {
    Offered      bool
    FlagRevision string
    Reason       string
}

type Quote struct {
    StandardCents  int64
    ExpeditedCents *int64
    Decision       Decision
}

func Bucket(accountID string) int {
    h := fnv.New32a()
    _, _ = h.Write([]byte(accountID)) // UTF-8 bytes, as in TextEncoder
    return int(h.Sum32() % 100)
}

func QuoteShipping(in Input, flag Flag) (Quote, error) {
    if in.SubtotalCents < 0 || flag.RolloutPercent < 0 ||
        flag.RolloutPercent > 100 {
        return Quote{}, errors.New("invalid quote input or flag configuration")
    }

    reason := "eligible"
    switch {
    case !flag.Enabled || flag.KillSwitch:
        reason = "disabled"
    case Bucket(in.AccountID) >= flag.RolloutPercent:
        reason = "outside-rollout"
    case in.Region != "US-LOWER-48":
        reason = "unsupported-region"
    case in.SubtotalCents < 7500:
        reason = "below-minimum"
    }

    result := Quote{
        StandardCents: 799,
        Decision: Decision{
            Offered: reason == "eligible", FlagRevision: flag.Revision,
            Reason: reason,
        },
    }
    if result.Decision.Offered {
        expedited := int64(1499)
        result.ExpeditedCents = &expedited
    }
    return result, nil
}

I would put a shared set of fixture inputs in both test suites: account IDs with ASCII and non-ASCII characters, rollout at 0 and 100, subtotal at 7,499 and 7,500 cents, each region, and the kill switch. The tests should assert that TypeScript and Go assign the same account to the same bucket. This is one of those boring cross-language details that becomes a very exciting checkout bug if you skip it.

The rollout sequence is straightforward: ship both implementations dark, compare decisions against fixtures, enable internal accounts, then 1%, 10%, and onward while watching quote errors, conversion, fulfillment capacity, and customer support reports. The kill switch stops new offers. Existing orders retain their stored price and promise; changing those would be a different business operation. When the feature is permanent, remove the rollout branch and leave the eligibility rule as normal checkout code.

3. Saved filters: the interface switch and the user’s own toggle

The third example is a saved-filters panel for an order list. There are two switches here that people tend to conflate. The feature flag says whether this account has access to saved filters. The user preference says whether the available panel is currently shown. If the feature flag is off, the preference cannot manufacture access.

The server returns a capability document after authentication:

{
  "savedFilters": true,
  "revision": "ui-2026-09-24-3"
}

The endpoint that creates or lists saved filters must evaluate the account’s capability again. A React component that omits a button is a nicer screen, not an access control check. The same applies on iOS.

TypeScript and React

The browser fetches the capability, treats unknown as off, and lets the person hide or show the panel with a visible toggle. The preference is local to the browser in this version; if we want it to follow a user across devices, that becomes a server-side preference with its own API and migration.

// SavedFiltersFeature.tsx
import { useEffect, useState } from "react";

type Capabilities = { savedFilters: boolean; revision: string };
type Filter = { id: string; name: string; query: string };

export function SavedFiltersFeature() {
  const [capability, setCapability] = useState<Capabilities | null>(null);
  const [filters, setFilters] = useState<Filter[]>([]);
  const [open, setOpen] = useState(
    () => localStorage.getItem("saved-filters-open") === "true",
  );
  const [error, setError] = useState<string | null>(null);

  useEffect(() => {
    const controller = new AbortController();
    fetch("/v1/me/features", { signal: controller.signal, credentials: "include" })
      .then(response => {
        if (!response.ok) throw new Error("Could not load features");
        return response.json() as Promise<Capabilities>;
      })
      .then(setCapability)
      .catch(err => {
        if (err.name !== "AbortError") setError(err.message);
      });
    return () => controller.abort();
  }, []);

  useEffect(() => {
    if (!capability?.savedFilters || !open) return;
    const controller = new AbortController();
    fetch("/v1/saved-filters", { signal: controller.signal, credentials: "include" })
      .then(response => {
        if (!response.ok) throw new Error("Could not load saved filters");
        return response.json() as Promise<Filter[]>;
      })
      .then(setFilters)
      .catch(err => {
        if (err.name !== "AbortError") setError(err.message);
      });
    return () => controller.abort();
  }, [capability?.savedFilters, open]);

  if (!capability?.savedFilters) {
    return error ? <p role="alert">{error}</p> : null;
  }

  return <section aria-label="Saved filters">
    <label>
      <input type="checkbox" checked={open} onChange={event => {
        const next = event.target.checked;
        setOpen(next);
        localStorage.setItem("saved-filters-open", String(next));
      }} />
      Show saved filters
    </label>
    {error && <p role="alert">{error}</p>}
    {open && <ul>{filters.map(filter =>
      <li key={filter.id}><button type="button" onClick={() => {
        // The order list owns applying filter.query, after validating its DSL.
        window.dispatchEvent(new CustomEvent("apply-saved-filter", {
          detail: { id: filter.id },
        }));
      }}>{filter.name}</button></li>,
    )}</ul>}
  </section>;
}

The button passes a filter ID, not arbitrary SQL from storage. The order list asks its own API to apply that ID under the current account. If the flag goes off while this page is open, a fresh capability fetch on navigation or a short-lived cache will remove the panel; the API check shuts off server access immediately.

The same feature in SwiftUI

On iOS, @AppStorage is the user preference. The capability still comes from the server. A small model loads it and, only when allowed and opened, loads the filters.

import SwiftUI

struct Capabilities: Decodable {
    let savedFilters: Bool
    let revision: String
}

struct SavedFilter: Decodable, Identifiable {
    let id: String
    let name: String
    let query: String
}

@MainActor
final class SavedFiltersModel: ObservableObject {
    @Published private(set) var capability: Capabilities?
    @Published private(set) var filters: [SavedFilter] = []
    @Published private(set) var errorMessage: String?

    // apiBase and URLSession are injected so previews/tests can use fixtures.
    private let apiBase: URL
    private let session: URLSession

    init(apiBase: URL, session: URLSession = .shared) {
        self.apiBase = apiBase
        self.session = session
    }

    func loadCapability() async {
        do {
            let (data, response) = try await session.data(
                from: apiBase.appending(path: "v1/me/features"))
            guard (response as? HTTPURLResponse)?.statusCode == 200 else {
                throw URLError(.badServerResponse)
            }
            capability = try JSONDecoder().decode(Capabilities.self, from: data)
            if capability?.savedFilters != true { filters = [] }
            errorMessage = nil
        } catch {
            capability = nil // unknown is off
            filters = []
            errorMessage = "Features could not be loaded."
        }
    }

    func loadFilters() async {
        guard capability?.savedFilters == true else { return }
        do {
            let (data, response) = try await session.data(
                from: apiBase.appending(path: "v1/saved-filters"))
            guard (response as? HTTPURLResponse)?.statusCode == 200 else {
                throw URLError(.badServerResponse)
            }
            filters = try JSONDecoder().decode([SavedFilter].self, from: data)
            errorMessage = nil
        } catch {
            filters = []
            errorMessage = "Saved filters could not be loaded."
        }
    }
}

struct SavedFiltersView: View {
    @StateObject private var model: SavedFiltersModel
    @AppStorage("savedFiltersOpen") private var isOpen = false
    let applyFilter: (String) -> Void

    init(apiBase: URL, applyFilter: @escaping (String) -> Void) {
        _model = StateObject(wrappedValue: SavedFiltersModel(apiBase: apiBase))
        self.applyFilter = applyFilter
    }

    var body: some View {
        Group {
            if model.capability?.savedFilters == true {
                Section("Saved filters") {
                    Toggle("Show saved filters", isOn: $isOpen)
                    if isOpen {
                        ForEach(model.filters) { filter in
                            Button(filter.name) { applyFilter(filter.id) }
                        }
                    }
                }
            }
            if let error = model.errorMessage {
                Text(error).foregroundStyle(.red)
            }
        }
        .task { await model.loadCapability() }
        .task(id: isOpen && model.capability?.savedFilters == true) {
            if isOpen { await model.loadFilters() }
        }
    }
}

The app’s authenticated URLSession would carry the user’s credentials; the sample leaves that wiring to the host app. When someone taps a filter, the host sends the ID to the order-list API rather than trusting the locally decoded query. I would test both clients with the same capability responses: on, off, failed request, and revocation after the screen has loaded. I would also test the API endpoint directly with the feature disabled. The server is the actual gate.

The switch has a lifecycle

Across these examples, I would keep a little record for every flag: owner, default, scope, rollout plan, telemetry, rollback behavior, and removal date. The pipeline route is scoped to a tenant and batch. The shipping offer is scoped to a stable account cohort and then frozen into an order quote. The interface flag is scoped to an authenticated account, while the visible on/off control is a separate user preference.

That distinction is the useful mental model. A toggle switch is an operational decision point, and the rest of the system has to agree on what was decided. Give it one boundary, persist decisions when they affect money or durable data, measure the new path, and remove the temporary branch after the migration is over. Otherwise the switch becomes one more permanent mystery in the codebase, which is a lousy reward for trying to ship safely.

References

Agent-Ready Local Environments and MCP

2026-09-24T12:37:01-07:00

Getting a coding agent productive is less about the prompt and more about whether the machine in front of it can actually build, run, and reach the systems the work depends on.

Cursor, Claude, Codex, and the rest are fast when the local world is already honest: dependencies install, the app boots, tests can run, and the external surfaces (GitHub, GitLab, Slack, Outlook, Teams, whatever is in play) are reachable through clear, permissioned paths. When those pieces are missing, the agent spends its tokens rediscovering your laptop instead of doing the job.

This post is about making that setup cheap and repeatable.

The real prerequisite is a bootable story

Every repo should answer, in files the agent can read, a small set of questions:

What do I need installed on this machine?
How do I get from clone to a running local system?
Which services does this project talk to, and which of those am I expected to use locally?
Where do secrets live, and what is not allowed?
How do I know the setup worked?

If those answers only exist in your head, the agent will invent something almost right. Almost right is expensive.

I keep this in the product repo as first-class docs and scripts: README, CONTRIBUTING, docs/setup, scripts/bootstrap, .env.example. When the work spans more than one codebase, I also mirror the “where the rest of the world lives” map in the project notes repo.

Make setup mechanical

Agents are excellent at following a checklist. They are mediocre at inferring your undocumented brew taps and tribal knowledge.

What works well right now:

A single bootstrap path. One script or documented sequence: install language runtimes, install dependencies, copy env templates, start local dependencies, run a smoke check. Prefer boring and explicit over clever.

Pinned, discoverable tooling. Version files (.nvmrc, .node-version, mise.toml, asdf configs, rust-toolchain, etc.) beat README poetry. The agent should be able to detect the toolchain without asking you which Node major you meant last quarter.

.env.example that matches reality. Every required variable named. No secret values. Short comments for where to get each one. If a variable is only for production, say so.

Smoke tests for the environment. A command that proves the local stack is alive: npm run doctor, make verify, a compose healthcheck, a migration status call. Green means “safe to start feature work.” Red means “fix the machine, not the feature.”

Dev containers or explicit host setup. Pick one and document it. Either the agent works inside a known container, or it works on the host with a known bootstrap. Mixing both without saying which is canonical creates two slightly broken worlds.

This is the same instinct as my old dev-setup-osx habit, updated for a world where the “new developer” might be an agent that showed up thirty seconds ago.

Tell the agent where the working systems are

Local build is only half the map. Most real work touches other systems: source hosts, chat, mail, issue trackers, cloud consoles, internal APIs.

Write down the topology.

Product repos and their remotes (GitHub, GitLab, both, mirrors).
Which environments exist: local, staging, production, preview apps.
Which human collaboration surfaces matter: Slack channels, Teams teams, Outlook lists, Linear/Jira projects.
Which of those the agent is allowed to read or write.
How authentication is supposed to happen for each.

Do not make the agent guess that “the deploy” means GitHub Actions in one repo and a GitLab pipeline in another. Put the pointers in markdown next to the code or in the notes repo. Ambiguity here turns into the wrong PR opened against the wrong remote, or a “fix” that never lands where humans look.

MCP is how agents reach the rest of the desk

Model Context Protocol servers are the practical bridge between the coding agent and the tools already on your desk. Used well, they turn “go check the thread / issue / inbox / pipeline” into a first-class action instead of a copy-paste scavenger hunt.

The useful pattern is not “connect everything.” It is “connect the systems this project actually depends on, with the least privilege that still helps.”

A sane MCP layout for project work often includes:

Source control: GitHub and/or GitLab for issues, PRs, checks, and releases.
Chat: Slack or Teams for the channels where decisions and unblockers live.
Mail/calendar: Outlook or similar when the work is genuinely gated on threads or meetings, not because it is fun to give an agent your inbox.
Project tracking: whatever holds the tickets, if that is not already the git host.
Docs and runbooks: if they live outside the repo and agents keep asking for them.

Wire these at the user or project level in Cursor (and equivalents elsewhere), then document in the repo which MCP servers are expected for this project and what they are for. An agent that knows “Teams is available for the eng channel, GitHub is available for PRs, Outlook is not in scope for this repo” wastes less time and takes fewer weird actions.

Authentication matters. Prefer the product’s normal OAuth/device flows. Keep tokens out of the notes repo and out of prompts. If a server needs auth, say so in setup docs and stop there. Do not paste credentials into markdown “for convenience.”

Boundaries beat cleverness

An agent with broad access and no rules will eventually do something technically impressive and socially awful: comment in the wrong channel, open a PR against the wrong fork, or dig through mail that was never part of the task.

Give it clear limits:

Read-only by default where write access is not required.
Explicit allow-lists of repos, channels, and projects.
Repo instructions that say when to use MCP versus when to stay local.
The same ask-plan-confirm gate you use for code changes when the action leaves the laptop (posting, labeling, merging, emailing).

Local environment setup gets the agent building. MCP gets the agent collaborating. Boundaries keep both from becoming a mess you have to unwind.

A minimal checklist I actually use

When I stand up a project for agent-assisted work, I want at least this:

Clone, bootstrap, and smoke test documented and scripted.
Toolchain versions pinned in-repo.
.env.example complete; real secrets elsewhere.
Notes/brief that name the remotes, environments, and collaboration surfaces.
MCP servers configured for the systems in play, with purpose notes in the project docs.
Clear write/read expectations for each integration.
A first prompt that points the agent at setup docs before feature work.

None of that is fancy. All of it is what makes Cursor, Claude, Codex, and friends look brilliant on day one instead of lost in your PATH.

The goal is simple: when an agent sits down at the project, the local world boots, the surrounding systems are findable, and the rules of engagement are already written down.

Twenty-ish Terminal Commands for Getting Around the File System

2026-09-11T09:49:05-07:00

Recently I was digging around in the terminal and wanted an ls command that behaved like a fully opened tree view. That led me back through a bunch of the file-system commands I use all the time, along with a few switches that make them far more useful.

This is the resulting quick reference. I’m writing this on macOS with zsh, but most of it applies directly to Linux too. A few tools—particularly tree and fd—may need to be installed first.

First, the Many Faces of `ls`

The plain command lists the current directory.

ls

Add -l for the vertical, long-listing format. This shows permissions, link count, owner, group, size, modified date, and name.

ls -l

Add -a to include hidden entries—names beginning with a dot—and -h to make sizes human-readable. The switches can be combined.

ls -lah

To descend through every directory and list its contents, add -R for recursive.

ls -laRh

That gets us the fully opened contents, although the output is grouped by directory rather than drawn as a visual tree.

1. `pwd` — Where Am I?

When I’ve wandered six directories deep and forgotten where I am, pwd prints the full path to the working directory.

pwd

2. `cd` — Change Directory

Move into a directory, up one level, back to the previous directory, or straight home.

cd projects
cd ..
cd -
cd ~

Paths containing spaces need quotes, as in cd "My Projects".

3. `pushd` and `popd` — Directory Bookmarks, Sort Of

pushd changes directories while saving the current location on a stack. popd takes me back. This is excellent when bouncing between two distant parts of a repository.

pushd ~/Code/my-project/docs
popd

4. `tree` — The Actual Tree View

Unlike recursive ls, tree draws the hierarchy. -a includes hidden entries, -L 2 limits output to two levels, and -h prints readable sizes.

tree -a -h -L 2

On macOS, install it with brew install tree if it is not already available.

Finding and Inspecting Things

5. `find` — Search the Directory Tree

Find every Markdown file below the current directory. Here . means “start here,” -type f limits results to files, and -name supplies the filename pattern.

find . -type f -name "*.md"

Use -type d to find directories instead.

6. `fd` — A Friendlier Find

fd provides a concise, fast alternative when installed. This finds Markdown files while including hidden paths but excluding .git.

fd --hidden --exclude .git '\.md$'

Install it on macOS with brew install fd.

7. `stat` — All the File Details

stat reports metadata such as size, permissions, timestamps, and inode information.

stat quick-article.md

The exact output differs between macOS and Linux, but the intent is the same.

8. `file` — What Is This Thing?

Extensions can lie. file inspects the contents and reports what it believes the file actually is.

file mysterious-download

9. `du` — What Is Using the Space?

du measures disk usage. -s summarizes instead of listing everything below the target, and -h makes the result readable.

du -sh .
du -sh ./*

10. `df` — How Much Disk Is Left?

Where du examines files and directories, df reports free and used space on mounted file systems. Again, -h keeps the numbers readable.

df -h

11. `less` — Read Without Flooding the Terminal

For a file longer than one screen, use less. Search with /text, advance with the space bar, move backward with b, and quit with q.

less application.log

12. `head` and `tail` — Inspect the Edges

These show the beginning or end of a file. -n 20 asks for twenty lines. tail -f keeps watching as new lines arrive, which is particularly handy for logs.

head -n 20 application.log
tail -n 20 application.log
tail -f application.log

Creating and Managing Files

13. `mkdir` — Create Directories

The useful switch here is -p: it creates missing parent directories and does not complain if the path already exists.

mkdir -p notes/terminal/examples

14. `touch` — Create an Empty File

If the file does not exist, touch creates it. If it does exist, touch updates its timestamps without changing the contents.

touch notes.md

15. `cp` — Copy

Copy a file with plain cp. Use -R to copy a directory recursively and -i to ask before overwriting something.

cp -i notes.md notes-backup.md
cp -Ri source-folder destination-folder

16. `mv` — Move or Rename

mv handles both jobs. I often include -i for an overwrite prompt.

mv -i draft.md published.md
mv -i published.md archive/

17. `ln` — Create a Link

ln -s creates a symbolic link. The first path is the existing target; the second is the new link.

ln -s ~/Code/my-project/current-config.json config.json

18. `rmdir` — Remove an Empty Directory

rmdir only removes empty directories. That limitation makes it useful when I want the command to refuse anything containing files.

rmdir old-empty-folder

19. `rm` — Remove Files Carefully

Plain rm removes files. -i asks before each removal. Recursive -R removes directories and their contents, so double-check the path before pressing Return.

rm -i unwanted.txt
rm -Ri unwanted-folder

There generally is no built-in undo. On macOS, moving something to the Trash in Finder is often a better choice when I’m not absolutely sure.

20. `open` — Hand It to macOS

open opens a file in its default application. Give it a directory and Finder opens there; use -a to choose an application.

open README.md
open .
open -a "Visual Studio Code" .

On Linux, the closest general equivalent is usually xdg-open.

A Few Combinations I Actually Use

Here are the quick combinations I tend to reach for most often.

# Everything here, including hidden entries, with readable details.
ls -lah

# Everything below here, recursively, in long format.
ls -laRh

# A manageable visual overview, two levels deep.
tree -a -L 2

# Find the biggest immediate entries in the current directory.
du -sh ./* | sort -h

# Jump somewhere temporarily, inspect it, and jump back.
pushd ~/Code/some-project
ls -lah
popd

That’s the lot: twenty-ish commands and a pile of switches that turn the terminal into a quick file-system navigator. The commands themselves are only half of the story. Run man ls, man find, or man followed by any of the built-in commands above to spelunk through everything else they can do.

Dashing Arrivals: Watch Every Bus, Train, and Ferry in Puget Sound Move in Real Time

2026-08-31T18:29:09-07:00

There’s a particular kind of magic in watching a whole city move at once. Not one bus, not one train — all of them, gliding across the map in real time, each one a little rectangle of somebody’s commute home.

That’s Dashing Arrivals — a live transit map for the Puget Sound region. Open it up and you’re looking at every active bus, train, and ferry in service right now, from a King County Metro coach on Rainier Avenue to a Sound Transit Link train sliding through the Rainier Valley to a Washington State Ferry crossing to Bainbridge. On a typical weekday afternoon that’s over 1,100 vehicles on screen, all updating live.

This post is a tour of what it is and what you can do with it.

The whole region at a glance

The default view drops you over Puget Sound with everything turned on. A quick legend does a lot of work here: color tells you the agency and shape tells you the type (bus, train, or ferry), and every icon is rotated to face the direction it’s actually traveling. Green is King County Metro, blue/indigo is Sound Transit, red is Pierce Transit, purple is Kitsap, and so on — seven agencies in all, plus the ferries.

Pan, zoom, and the icons resize smoothly so trains read a little larger than buses, which read a little larger than ferries. Between updates, vehicles don’t teleport — they animate from their last position to the next one, so the whole map has a calm, continuous, living quality instead of a jumpy refresh.

Click any vehicle for the details

Tap a vehicle and it tells you who it is: the route, the agency, the fleet number, its heading (as a compass direction and degrees), its speed, and when it last reported in. Here’s a 1 Line Link train in the Rainier Valley, heading south at 159°.

See that Show route ▸ link at the top of the popup? That’s where it gets fun.

See the entire route

Click Show route and the map draws the vehicle’s full line — the shape it follows and every stop along the way — right under the live vehicles still moving on it.

Here’s the 1 Line, Link light rail’s spine, threading from the north down through downtown, Beacon Hill, and the Rainier Valley. The blue train icons strung along the highlighted line are the actual trains in service on it right now.

And the newer 2 Line, running across Lake Washington on I‑90 to Bellevue and up to Downtown Redmond. (Fun bit of local history: the 2 Line effectively replaced the old ST Express 550 bus between Seattle and Bellevue — which is why you won’t find a 550 on the map anymore.)

Rail beyond Link: the Sounder

Light rail isn’t the only train out there. Pick a Sounder commuter-rail train and you get the full BNSF corridor.

This is the S Line, running from King Street Station in Seattle south through Tukwila, Kent, Auburn, and on toward Tacoma and Lakewood. Sounder only runs during peak commute windows, so catching one on the map is a small, satisfying reward for looking at the right time — and down in Tacoma you can see Pierce Transit’s red buses fill in the local network.

Buses, of course — every route

The same trick works for any bus. Click one, hit Show route, and the corridor lights up in that agency’s color.

The RapidRide C Line from West Seattle into downtown, in Metro green.

And the RapidRide E Line, a ruler-straight shot down Aurora Avenue from Shoreline to downtown — one of the busiest bus corridors in the state. Whether it’s a lettered RapidRide line, a numbered local route, or a Community Transit Swift line, the route overlay works the same way.

Filter the firehose

Eleven-hundred vehicles is a lot. The Show and Agencies panel lets you dial it in — toggle whole modes or individual agencies on and off.

Here I’ve switched off buses to leave just trains and ferries. Suddenly the shape of the rail network jumps out — the Link lines tracing north–south and across the lake — alongside the ferries stitching the Sound together to Bainbridge, Bremerton, Kingston, and Vashon. It’s the whole regional rail-and-water map, drawn entirely by the vehicles themselves.

When’s my ride? Live arrivals at any stop

Zoom in and the stops appear. Click one to get a live arrivals board.

This is the Symphony stop downtown, with real-time predictions counting down — Due, Due, 1 min, 4 min, 6 min… — for 1 Line and 2 Line trains toward Lynnwood City Center. Each prediction carries a live indicator, and any alerts that affect this specific stop are pinned right at the top.

Know before you go: service alerts

The pill at the top of the screen keeps a running count of active service alerts across the region — detours, stop relocations, reduced service, construction. Open it and you can read them all: each card shows the agency, the affected routes, the reroute instructions, and a link out to the agency’s own detail page. It’s the regional “what’s disrupted right now” board in one place.

Light or dark, your call

Prefer a bright map? A theme toggle switches between a sleek dark basemap and a clean light one (with a System option that follows your OS). Same live data, different mood.

Under the hood

For the curious, Dashing Arrivals is a modern, open-source web app:

Live data comes from the OneBusAway Puget Sound regional API, decoded from GTFS‑realtime vehicle-position feeds. A single regional feed covers King County Metro, Sound Transit, Community Transit, Pierce Transit, Kitsap Transit, and Everett Transit.
Ferries come from a separate source — the WSDOT Washington State Ferries vessel-location API — and are normalized in alongside everything else.
The map is MapLibre GL on an open OpenFreeMap basemap (no proprietary map key required), with the app built on Next.js and React and deployed on Vercel.
Vehicle positions are fetched on a short polling loop and cached server-side, then eased between updates on the client so motion looks smooth. Icons are tinted per agency, sized by zoom, and rotated to the reported heading.

Eventually the GTFS feed is going to have some changes, and I plan to put those fixes in then, but in the meantime this is a solid way to explore the Seattle and Puget Sound area’s transit options!

Go watch the city move

That’s the whole pitch: one map, every vehicle, in real time, with routes, stops, arrivals, and alerts a click away. It’s genuinely useful for catching a bus — and, if you’re the type who likes watching systems work, it’s a little bit hypnotic.

Try it for yourself at dashingarrivals.com.

Screenshots captured live from dashingarrivals.com. Basemap © OpenFreeMap / OpenMapTiles, data © OpenStreetMap contributors. Real-time transit data via OneBusAway and WSDOT.

Ask, Plan, Confirm: Making Agents Stop Before They Start

2026-08-20T08:00:00-07:00

The scariest thing about a genuinely capable coding agent is how quickly it commits. You type two sentences, and forty seconds later there are changes across nine files, half of which you did not want and one of which quietly changed a query you spent a week hardening. The agent was not wrong about how to implement the thing. It was wrong about what the thing was, and it never stopped to check.

I have watched this happen enough times that I stopped treating it as a prompting problem and started treating it as a workflow problem. A better one-shot prompt is not the fix. A gate is: the agent is not allowed to edit files until it has asked what it needs to ask, shown me a plan, and gotten a yes.

Why speed is the problem

Human engineers have a built-in pause. Before a senior developer touches your codebase they ask a couple of questions, sketch the approach, maybe drop a comment on the ticket. That pause is where the wrong-work gets caught, cheaply, in a sentence, instead of expensively, in a diff you have to read and reject.

Agents removed the pause. That is most of their value and most of their danger in the same motion. An agent that implements immediately optimizes for the wrong thing: it treats “produce a diff” as the goal, when the goal was “produce the diff we agreed on.” The gap between those two is where the deleted work and the silent scope creep live.

That gap is not a knowledge problem. The agent knows how to write the code. It just never checked that it was writing the right code.

The same task, gated and ungated. Skipping the pause trades a small, predictable cost for a large, unpredictable one.

So I gave the pause back, deliberately, as a standing instruction on every agent in the InterlinedList repo.

The three beats

The workflow is written down in .claude/workflows/plan-first.md and every agent links to it. It is three beats, in order, and implementation is gated behind all three.

The three beats, in order. Nothing is edited until all three pass, and work that grows past the approved plan loops back to re-plan rather than quietly expanding.

Ask. Surface what the prompt left open before committing to an approach. Ambiguous scope, unstated edge cases, a product decision hiding inside a technical request, whether a feature should be tier-gated. The migrations agent asks about column types and nullability and whether anything destructive is implied. The Next.js agent asks which surfaces are in and out. The rule has an escape hatch, because asking three questions about a one-line copy fix is its own kind of annoying: skip the questions only when the request is genuinely unambiguous and low-risk. When in doubt, ask. A pointed question is cheaper than a wrong build every single time.

Plan. Before touching files, lay out the shape of the change: the files and routes and components you will touch, the ones you will deliberately leave alone, the approach, any migration (additive, always), the tests the change needs, and anything risky. In this repo “risky” has a specific meaning: auth, IDOR, subscription gating, SSRF, secret handling, anything destructive or hard to reverse. The plan is a decision aid, not a document. It should be short enough to read in one breath and specific enough that approving it means something.

Confirm. Implement only after an explicit yes. If the plan changes in the back-and-forth, restate the revised version and get the yes again. And the part that actually matters over a long session: approval is scoped to the plan that was approved. If the work grows past it, the agent stops and re-plans instead of quietly expanding. That last clause is what keeps a “small fix” from turning into an afternoon of changes I never signed off on.

What it looks like per agent

I did not want one generic paragraph pasted eight times. The gate is the same, but what you ask about depends on the job, so each agent got the beats written for its lane.

One gate, written for each lane. The two read-only reviewers pick the full gate back up the moment they move from finding to fixing.

The migrations agent plans the exact idempotent migration.sql and confirms it is purely additive before it applies anything. The unit-testing agent asks which behaviors to lock in and which boundaries to mock, then lists the cases each test file will assert. The e2e agent names the flows, the auth and seed prerequisites, and the breakpoints that matter. The docs agent confirms which of the three docs is in scope and whether a new page is needed. The blog agent (yes, this one) settles the angle and the section arc and which real code it will verify claims against before drafting a word.

The two read-only reviewers are the interesting edge. Security and UX do not implement, so there is no edit to gate. For them the gate degrades to its first beat: confirm the review scope if it is ambiguous (which routes, how deep, which breakpoints), then produce findings. But the moment the user says “now fix what you found,” they are implementers, and the full ask-plan-confirm gate snaps back on before they touch code. The reviewer does not get to slide from “here is a finding” into “and I fixed it” without crossing the same line everyone else crosses.

The obvious objection

This is slower. That is the point, and it is also not as true as it sounds. The plan step costs you a few seconds and one read. Rejecting a forty-second nine-file diff that went the wrong direction costs you the read plus the reject plus the re-prompt plus the nagging worry about what it touched that you did not catch. The gate front-loads a small, predictable cost to avoid a larger, unpredictable one. Over a day of handoffs it is not close.

It also composes with the other habit I built into these agents: every one of them does its work in an isolated git worktree, on its own branch, torn down when the task lands. Plan first, then do the approved work in a sandbox that cannot collide with anyone else. The worktree contains the blast radius. The plan makes sure there is not supposed to be a blast in the first place.

The pause was always the expensive part of good engineering. Worth teaching the machines to keep it.

I’m Adron, brainstorming and building InterlinedList.

Giving Every Agent Its Own Branch: Git Worktrees for Parallel AI Work

2026-08-19T08:00:00-07:00

Run two coding agents against a single working tree and they will fight. One is halfway through editing app/api/messages/route.ts while the other checks out a different branch underneath it. The index lock flickers. A git stash from one session swallows the other’s uncommitted work. I have lost real edits this way, and every time the root cause was the same: one working tree, one HEAD, two writers.

The InterlinedList repo already had matching agents for the jobs I hand off most: a Next.js implementer, a migrations specialist, unit and e2e testers, a docs writer, security and UX reviewers. They are good at their lanes. What they were missing was a lane in the literal sense. They all drove on one road.

I found the pattern I wanted written up in Augment’s guide to git worktrees for parallel AI execution, and it maps almost one to one onto how I already think about agents. This is the writeup of what I built on top of it for this repo: the directory convention, the seven scripts that manage the lifecycle, and the wiring that makes every agent use them without being reminded.

The one-tree problem

A normal clone gives you a single working directory backed by one .git. That is fine for one person doing one thing. The moment you parallelize, the shared mutable state (the working files, the index, the current branch) becomes the bottleneck. You cannot have the migrations agent on agent/add-webhooks and the docs agent on agent/help-refresh at the same instant, because “the branch” is a property of the whole checkout.

Two writers, one mutable checkout. The collisions have nothing to do with the work: an index lock flickers, a git stash swallows the other session’s edits, HEAD thrashes between branches.

Git solved this in 2015 with git worktree. A worktree is a second (third, fourth) working directory attached to the same repository. Each one has its own files, its own index, and its own checked-out branch, while sharing one object store on disk. The object store is the expensive part, so you share it. A directory of files is cheap, so you duplicate it per task. That is exactly the tradeoff you want for parallel agents.

Share the expensive part, duplicate the cheap part. The object store lives on disk once; the working files, the index, the checked-out branch, and even the dev-server port are private to each worktree.

The convention

Every agent works in .trees/<task-id> on a branch named agent/<task-id>, cut from origin/develop (this repo integrates on develop, not main). So a task called add-list-webhooks lives at .trees/add-list-webhooks on branch agent/add-list-webhooks. The directory name and the branch name always agree because both are derived from the same sanitized slug.

The .trees/ container is gitignored. Worktrees are workspace, not history:

# .gitignore
# Agent worktrees (see scripts/worktrees/)
.trees/

That single ignore line is the whole footprint the pattern leaves in the tracked tree. Everything else is scripts and instructions.

The scripts

I did not want agents (or me) typing raw git worktree incantations and getting the branch name wrong, or forgetting to copy .env.local, or leaving stale metadata behind. So the lifecycle lives in scripts/worktrees/ as a small set of focused shell scripts. Here is every one of them.

The whole lifecycle before the per-script detail. Create, inspect while working, remove when it lands, sweep the merged trees on a schedule.

_lib.sh is the shared library the others source. It holds the functions that keep conventions consistent: wt_repo_root resolves the primary checkout even when you call it from inside a linked worktree (it reads git rev-parse --git-common-dir and walks up), wt_sanitize lowercases a task id and strips it to [a-z0-9._-], wt_resolve_base fetches and prefers origin/<base> over a local branch, wt_port_for hashes a branch name into a stable dev port, and wt_is_locked reads the porcelain worktree list to check lock state. Nothing in it is clever. It exists so the clever bits are written once.

wt-create.sh <task-id> [base] is the one agents call first. It creates the worktree and makes it ready to work in, in one shot:

scripts/worktrees/wt-create.sh add-list-webhooks

Under the hood it resolves the base ref (default develop), ensures .trees/ is in .gitignore, turns on git rerere so repeated conflict resolutions replay across parallel merges, runs git worktree add -b agent/<slug> .trees/<slug> origin/<base>, copies the root .env.local into the worktree, appends a deterministic DEV_PORT derived from the branch name, runs npm ci --prefer-offline, and finally locks the worktree so other sessions can see it is in use. Flags let you opt out where it makes sense: --no-install skips the dependency install for a quick plumbing check, --no-lock leaves it unlocked, and --baseline runs the test suite right after setup so a green baseline proves any later failure came from the agent’s change rather than a pre-existing break.

The port assignment is worth a sentence. Two agents both running next dev on 3000 is another collision, a quieter one. wt_port_for runs the branch name through cksum and maps it into the 3100 to 9998 range, so each worktree gets a stable, distinct port written into its own .env.local. Start the server with npm run dev -- -p "$DEV_PORT" and two dev servers coexist.

wt-list.sh answers “what is running right now.” It prunes stale metadata, then prints one row per worktree with path, branch, lock state, and short HEAD:

PATH                                    BRANCH                LOCKED  HEAD
/Users/adron/Codez/interlinedlist       feature/fix-blog-image  -     58afa91562
/Users/adron/Codez/interlinedlist/.trees/add-list-webhooks  agent/add-list-webhooks  yes  b78faf9e21

Before an agent touches a shared file, it can look here and see who else is holding what. Git will not warn you about two branches editing the same file, so this list plus discipline about non-overlapping file domains is the actual safety mechanism.

wt-lock.sh <task-id> [reason] and wt-unlock.sh <task-id> are thin wrappers over git worktree lock/unlock. A lock is advisory: it resists prune and move, and it is the signal in wt-list.sh that says “an agent is live in here, do not reap this.”

wt-remove.sh <task-id> is the teardown. It unlocks if needed, runs git worktree remove, and prunes. This matters more than it looks: deleting a worktree with rm -rf leaves dangling metadata in .git/worktrees/ that haunts you until the next prune. The script never does that. --force discards uncommitted changes on purpose, and --delete-branch drops agent/<task-id> in the same step when the work has landed.

wt-cleanup.sh [base] is the bulk sweep. It walks every worktree physically under .trees/, and for each one whose branch is already an ancestor of origin/develop (in other words, merged), it removes the worktree and deletes the branch. It deliberately skips anything outside .trees/, so the primary checkout and any sibling worktrees I keep elsewhere on disk are never touched. This is the script you point a daily cron or a post-merge hook at.

Each of these is also exposed as an npm script, so npm run wt:create -- add-list-webhooks, npm run wt:list, and npm run wt:remove -- add-list-webhooks all work if you prefer that entry point. There is a scripts/worktrees/README.md documenting the whole set alongside the code.

Wiring it into the agents

Scripts nobody runs are decoration. The point was to make every agent reach for a worktree by default, so I added the instruction in three places at three levels of specificity.

At the top, CLAUDE.md now states the standing rule: every agent works in an isolated worktree, torn down with the lifecycle scripts. That is the repo-wide contract.

In the middle, two shared protocol docs under .claude/workflows/ hold the full detail: worktrees.md spells out the create-work-remove lifecycle, the boundaries (shared object store, non-overlapping files, and the important caveat that the database is shared even though the files are not), and plan-first.md covers the companion habit I wrote about separately. Every agent and skill links to these rather than repeating them.

At the leaf, each of the eight agent definitions in .claude/agents/ got a “Work in an isolated git worktree (required)” section written for its job. The implementers (Next.js, migrations, tests, docs, blog) get the full create-work-remove flow. The two read-only reviewers (security, UX) get a variant that tells them to cd into the worktree under review and read its diff, and explicitly not to create, lock, or remove anything. The migrations agent gets an extra warning in bold, because the worktree isolates schema.prisma and the migration files but not the Postgres instance: db:migrate still hits localhost and db:migrate:deploy still hits production from any worktree. That is the one place the isolation is a lie, and the agent needs to know it.

The five paired skills in .claude/skills/ (the ones that back the implementer agents) got a short “Worktree-first, plan-first” block near the top pointing at the same protocol docs, so whether the work comes in through the agent or the skill, the instruction is there.

Proving it works

I ran the whole lifecycle before committing any of it. Create a worktree from develop, confirm it is locked and has its port, unlock and re-lock it, remove it with the branch, and verify the cleanup pass leaves the sibling worktrees alone:

scripts/worktrees/wt-create.sh wt-smoke-test --no-install
scripts/worktrees/wt-list.sh          # shows agent/wt-smoke-test, LOCKED yes
scripts/worktrees/wt-unlock.sh wt-smoke-test
scripts/worktrees/wt-lock.sh wt-smoke-test "smoke re-lock"
scripts/worktrees/wt-remove.sh wt-smoke-test --delete-branch
scripts/worktrees/wt-cleanup.sh develop   # safe no-op, siblings untouched

Every step did what it said, the .gitignore guard refused to duplicate the .trees/ line it found already present, and the sibling worktree I keep for feed-perf work was never in scope for cleanup. That last part was the thing I most wanted to confirm, because a cleanup script that reaches outside its sandbox is worse than no cleanup script.

Where agents still get confused

Isolating the filesystem fixes the filesystem. It does nothing about the fact that the work itself overlaps, and there are a handful of ways an agent still gets lost.

The database is one instance, and every worktree writes to it. I flagged this to the migrations agent in bold, but it deserves more than a warning. If the migrations agent adds a column on agent/add-webhooks, that column now exists in the same localhost Postgres every other worktree points at. The docs agent three trees over never sees the changed schema.prisma, yet its queries hit the mutated database anyway. Additive-only migrations keep this survivable most of the time, since an extra column nobody reads is harmless. But the moment two agents touch the same table, or one runs db:migrate:deploy and reaches production from what looked like a sandbox, the isolation is a fiction. The files are private. The database is not.

Green in isolation, red on merge. This is the one that bites. Agent A changes a function signature in lib/lists/queries.ts. Agent B, on its own branch, calls that function from a route it owns. Their files never overlap, so wt-list.sh shows no conflict and git stays quiet. Both test suites pass, because A’s worktree still holds B’s old caller and B’s worktree still holds A’s old signature. It’s all green right up until both branches land on develop, and then the integration is broken in a way neither agent could see from inside its own tree. Worktrees convert a loud, immediate collision into a quiet one that surfaces later. Often a fair trade, but a trade.

Stale base drift. Every worktree is cut from origin/develop the moment it’s created. Agents aren’t always short-lived. Let one run for a few hours while three others merge back, and it’s now building on a develop that no longer exists. It will reintroduce a helper that got deleted upstream, or write against an API another agent already reshaped. The worktree has no idea the ground moved under it. Nothing in the create-work-remove loop forces a re-fetch, so a long-running agent drifts out of date without noticing.

“Does this already exist?” stops having one answer. With five branches in flight, whether feature X is “already built” depends on which tree you grep. An agent that checks the primary checkout won’t see work-in-progress on another branch, and it will happily build a second copy. I’ve watched two sessions independently implement overlapping halves of the same feature, each sure it was first, because neither branch was visible to the other and neither said up front what it was about to touch. wt-list.sh tells you which branches exist. It says nothing about what each one intends to change.

The lock is a suggestion. wt-lock resists prune and move, and it flags a tree as live in the list, but it will not stop another agent from opening the same file on its own branch and editing away. The real guard against two agents clobbering one file is the up-front decomposition plus the discipline to run wt-list.sh and honor what it shows. An agent that skips the check has no seatbelt, just the shape of one.

Abandoned trees pile up. wt-cleanup.sh only sweeps branches already merged into develop. A worktree from a crashed or cancelled session is unmerged, still locked, and invisible to the sweep. It sits on disk with a full node_modules until someone removes it by hand. And an agent resuming a task can trip over a half-finished tree from an earlier run and read its stale state as current work.

Losing the current directory. The workflow says cd .trees/<task-id> and do everything there. Shell state doesn’t always survive between tool calls, and an absolute path into the primary checkout looks identical to one into a worktree. An agent that loses track of where it is can read the main checkout’s copy of a file, reason about it as though it were its branch’s version, and edit the wrong tree. The isolation holds only as long as the agent keeps its bearings.

What this buys

The honest version: worktrees do not make independent tasks independent. If two agents both need to edit the same route, isolating their filesystems just delays the merge conflict, it does not prevent it. The decomposition still has to be real. What the pattern removes is the accidental collision, the kind that has nothing to do with the work and everything to do with sharing one mutable checkout. Those were most of my pain, and now they are gone by construction.

Advantages and disadvantages

Advantages

Accidental collisions disappear. Each agent gets its own files, index, and HEAD, so index-lock flicker, a stray git stash eating another session’s edits, and checkout thrash stop happening.
Every task lands on its own agent/<task-id> branch, which keeps review and merge clean and stops one task’s half-finished mess from bleeding into another’s diff.
The object store is shared, so the costly part of the repo lives on disk once and spinning up another tree is cheap.
Each worktree gets a deterministic DEV_PORT, so several next dev servers run side by side instead of fighting over 3000.
The lifecycle is scripted and wired into every agent definition, so the right setup and teardown happen without anyone remembering the incantation.
git rerere is on by default, so a conflict you resolve once replays across the parallel merges that hit it again.
Cleanup is fenced to .trees/, so the bulk sweep never reaches the primary checkout or the sibling worktrees I keep elsewhere.

Disadvantages

The database isn’t isolated, and neither is anything else global (production through db:migrate:deploy, OAuth apps, third-party rate limits). File isolation quietly implies an isolation that isn’t there.
Overlapping edits on separate branches turn into silent, deferred conflicts and semantic breakage that pass every isolated test and only show up at integration.
A long-running worktree drifts from a moving develop, and nothing in the loop forces the re-fetch that would catch it up.
Locks are advisory, so the actual protection against two agents editing one file is decomposition plus discipline, not anything git enforces.
With several branches live, “does this already exist” has no single answer, and agents duplicate each other’s work when branches can’t see one another.
Every worktree carries a full node_modules, so disk use and npm ci time multiply with each active task.
Crashed or abandoned sessions leave locked, unmerged trees that the merged-only cleanup won’t reap, so someone clears them by hand.
Coordination is still manual: the tooling shows which branches exist, not which files each agent means to touch.

The next habit I want in every agent sits upstream of all of this: stop and plan before touching a single file. That one is worth its own post.

More solutions to the confusion above are coming in the next few posts. Subscribe so you don’t miss ’em: drop your email into the box just below this post, or grab the RSS feed.

I’m Adron, brainstorming and building InterlinedList.

A Research Agent on a Budget Leash, and How I’d Sequence the Whole Build

2026-08-18T06:30:00-07:00

Part 8 of 8: Durable AI for InterlinedList

Back in Part 1, I said the whole series was one argument dressed up as eight posts. LLM tokens are the expensive thing, Temporal is not, and Temporal’s mechanics structurally push the token bill down. I also said the way I’d build it was to brainstorm against the real code first, then hand the agent structure instead of hope. This is where those two threads tie together.

The capstone I keep circling back to is the one that sounds the most like science fiction and is actually the most dangerous to ship carelessly: “research topic X from my saved links and draft a document with citations.” The agent retrieves. It reasons. It drafts. And if I’m not careful, it also burns through a month of my inference budget in one runaway loop while I’m asleep.

So this post is two things. The agent itself, and the playbook for building everything that came before it. The agent is the thing you build last, and the order it sits on top of is the part I actually want to argue for.

The Capstone Is Just the Earlier Pieces, Composed

This part should feel earned by now. The agentic-research-to-draft phase, the last of the proposals, doesn’t invent anything. It composes.

The link content from Part 5 (the crawl-and-cache that turns your saved URLs into readable text) is the agent’s source material. The retrieval from Part 5’s second half, the pgvector search over your own docs, lists, and messages, is how the agent finds the relevant three paragraphs instead of stuffing forty links into context. And the durable generation from Part 3, the workflow that survives a timeout and asks before it writes, is the drafting step.

Compose those three and you get an agent loop: retrieve, reason, retrieve again, draft, cite. Each of those steps is already an activity I’d have built and tested for its own feature. The agent is a workflow that calls them in a loop the model steers.

Which is exactly where it gets scary. A loop the model steers has no natural stopping point. It’ll keep deciding it needs one more search, one more fetch, one more pass at the draft. That’s not a bug in the model. That’s what an agent is. The bug would be letting it run unsupervised.

Temporal Supervising an LLM Agent Is a Nice Recursion

I like this framing because it’s a little bit funny. Temporal is an orchestrator whose entire job is supervising unreliable, long-running, retry-heavy work. An LLM agent is unreliable, long-running, retry-heavy work. So the agent runs inside a Temporal workflow, and the workflow is the adult in the room.

Three guardrails, all first-class workflow logic, none of them prompt-hope:

A hard token budget. Before every model call, the workflow calls assertBudget, the same primitive from Part 7 that reads the AiGeneration ledger and throws a non-retryable BudgetExceeded when the user’s daily or monthly ceiling is hit. The agent can want another search all it likes. If the budget’s gone, the call never fires.

A max-step cap. The loop counts its own iterations. Twelve steps, or whatever I tune it to, and it’s done. It drafts with what it has rather than spiraling.

A wall-clock timeout. Temporal’s workflow timeout means even a wedged agent eventually stops, cleanly, with whatever it produced so far.

And here’s the piece that makes the whole thing cost-sane instead of terrifying: every step is memoized. If the agent fails at step 9 of a 12-step loop (provider hiccup, worker restart, whatever) the workflow resumes at step 9. It does not re-run steps 1 through 8. It does not re-pay for the eight model calls it already made. In a serverless handler, a mid-loop failure means starting the whole expensive agent over. Here it means retrying one step.

Sketching the loop, this is the shape I’d hand Claude to fill in:

// Proposed: the supervised agent loop as a Temporal workflow.
export async function researchAgentWorkflow(input: ResearchInput): Promise<Draft> {
  const MAX_STEPS = 12;
  const context: Snippet[] = [];

  for (let step = 0; step < MAX_STEPS; step++) {
    // Governor first: a hard ceiling the agent cannot talk past.
    await assertBudget(input.userId);

    // Cheap model plans the next move over the small retrieved context.
    const move = await planNextStep({
      model: "claude-haiku-4-5",
      goal: input.topic,
      context,
    });

    if (move.type === "done") break;

    // Retrieval is mechanical and nearly free. No expensive tier here.
    const hits = await retrieveFromLinks(input.userId, move.query);
    context.push(...hits); // stays small: top-k snippets, not whole pages
  }

  // Expensive tier runs exactly once, on the final synthesis.
  return await draftWithCitations({
    model: "claude-opus-4-8",
    topic: input.topic,
    context,
  });
}

The cost story lives in the model choices, not the prose. Retrieval and planning run on a cheap tier: Haiku for the “what should I look for next” decisions, which are frequent and small. Opus 4.8, the expensive tier, runs once, on the final synthesis, over a retrieval-augmented context that stayed deliberately small the whole way through. I’m not paying Opus prices to decide which link to read next. I pay them for the draft, and that’s the one thing in the loop I think is worth it.

No clever prompt is doing that. The structure is.

How I’d Sequence the Whole Build

If you’ve read the series straight through, you might be tempted to build the shiniest thing first. Don’t. The agent ships last, and the ordering underneath it is the real deliverable of this post, because every step ships value on its own and nothing is a big-bang.

Here’s the order I’d hand myself:

1. The foundation phase: the Temporal worker and the cost primitives. The worker, the client wiring, the llmCall activity with the model cascade, assertBudget, the response cache. This is the groundwork everything else stands on, and nothing works without it. Mostly infra, not code, and it’s a week.

2. The durable publish and idempotency fix, pure win, no LLM. This is the one from Part 4: the every-minute cron that can double-publish because a partial failure re-runs on the next tick. Rebuild it as a Temporal Schedule with the message id as the workflow id, so a given message can never be published twice. Zero model tokens. It fixes a current production hazard. If I could only ship one thing from this whole series, it’d be this.

3. Durable Generate plus confirm. Part 3. Long docs stop blowing the function budget, and a wrong list schema never gets silently written because the workflow pauses on a signal and waits for a human. This is the in-product generate button, on solid footing.

4. Link crawl, then retrieval. Part 5. Crawl the links into a cache (the cheapest, highest-cache-value piece in the set) then layer pgvector retrieval on top. This is “where did I save that article?”

5. Batch enrichment, then prompt-to-list. Part 6. The fan-out that tags everything and expands anything without re-billing completed calls, then the “build a list from a sentence” flow composed on top of it.

6. The AI calendar and digests. Part 4’s second half. Now that publishing is durable, the calendar planner and the scheduled digests have solid ground to schedule onto.

7. The cost dashboard, then the agent. Part 7’s dashboard makes “cheap” visible: tokens, cache-hit rate, budget remaining. I want that governor on-screen before I ship the most token-hungry feature in the whole set. Then, and only then, the agentic capstone.

The rule underneath the whole ordering is that value ships at every step. The publish fix helps users who never touch AI. The link cache is useful before any retrieval sits on it. The dashboard is useful before the agent exists. I’m never in a state where I’ve spent three weeks and have nothing a user can hold.

The Verification Is Where the Trust Actually Lives

The meta-point of the whole series is this one, so I’ll say it flat. You do not trust an autonomous agent because the prompt was good. You trust it because it runs inside guardrails, and because tests prove the properties you care about are there.

So the verification for the agentic capstone, and everything under it, is not an afterthought. It’s the thing that makes any of this shippable:

Vitest unit tests for the activity logic and the cost math. Does the cascade pick the cheap model for planning? Does assertBudget throw at the right ledger sum? Pure functions, no DB, fast.

Temporal replay tests that prove a workflow resumes correctly after a simulated worker crash. This is how I prove, not hope, that a retry doesn’t restart the expensive agent from step zero.

An idempotency test that starts two publish workflows with the same message id and asserts exactly one publish. That’s the double-post race, closed and locked.

Cost-assertion tests, the ones I care about most for the agent. Fail an activity mid-pipeline and assert the already-completed model calls are not re-invoked on retry. Memoization is money, and this test is how I know the money’s saved.

Playwright E2E for the generate to confirm to write flow, so the human-in-the-loop gate works in a real browser and not just in my head.

A security pass before any PR: SSRF on the crawler (everything through safeFetch), IDOR on every workflow write (where: { id, userId }), prompt-injection defaults (artifacts private by default so a poisoned link can’t publish on your behalf), and budget-bypass attempts.

And every schema change (the LinkContent table, the pgvector column, the response cache) goes through the strict additive migration workflow. Hand-written, idempotent, applied to both databases. No prisma db push, no shortcuts. That workflow has broken production before, precisely when someone skipped it.

What I Actually Learned Working With Claude

The lesson closes the loop the series opened. Supervise the agent with structure, let the tests carry the trust, and it turns out that’s also how I’ve been building with Claude the whole time.

You don’t hand an agent a vague goal and hope. You hand it constraints: a budget it can’t exceed, a step count it can’t blow past, a timeout it can’t outlast. Then you make it write the tests that prove those constraints hold, the replay test, the idempotency test, the cost-assertion test. What makes the agent trustworthy is the wall around it, not how nicely I phrased the ask.

And that’s true one level up too, which I didn’t expect going in. Sequencing the build so each step is independently verifiable (the publish fix before the calendar, the cache before retrieval, the dashboard before the agent) is the same move as capping the agent’s steps. Give the work boundaries, make the boundaries checkable, prove them before moving on. Whether the thing on the leash is a research agent or my own plan to build one, the leash is what I trust.

I’ve been brainstorming this out loud for eight posts. The pieces are grounded in code that already exists: the safeFetch guards, the DSL, the crosspost fan-out, the cron that really does risk a double-post. None of it is shipped. All of it is buildable, in the order above, with value at every step. So I’m going to start at step one and build the durable publish fix, because it’s a pure win with no model tokens and it closes a real hazard. I’d love for you to follow along, as I am going to move forward now and implement this. If you’re interested, subscribe!

Adron brainstorming and working on InterlinedList.

Making LLM Features Cheap Enough to Actually Ship

2026-08-17T09:30:00-07:00

Part 7 of 8: Durable AI for InterlinedList

Six posts in, and I’ve been dancing around a number. Every proposal I’ve sketched (durable generate, the calendar planner, the link crawler, batch enrichment) carries a cost, and the expensive part is never Temporal. It’s the tokens. A worker box and a managed Temporal namespace run me on the order of tens of dollars a month (obviously at this time), flat, whether they process ten jobs or ten thousand. The model bill is the one that scales with usage. It’s also the one that decides whether an AI feature is worth shipping or is a slow-motion budget fire.

So this post is the payoff. Durability isn’t just nice, durability is money. The mechanics I keep reaching for aren’t in the plan because they’re clever. They’re there because each one structurally drives the token bill down, and does it in a way I can measure instead of squint at and hope for. Keeping this cheap relative to the Claude and ChatGPT APIs was never a footnote I tacked onto the series. It’s been the point the whole way through.

None of this is live yet. AI assistance in the product is still marked Coming Soon. This is me brainstorming the cost layer I’d build, and how I’d hand it to Claude to build with me.

The Single Biggest Saver Is Not Re-Paying for Work You Already Did

Start with the one that matters most. Durable memoization.

When a Temporal workflow runs an activity (say, an llmCall that generates a document section) the result gets persisted the moment it completes. If the workflow later fails and retries, it does not re-run that activity. It replays the history, sees the completed result, and moves on.

Think about what that means on a six-step generation pipeline that dies at step five. Without durability, a retry starts over: steps one through four re-invoke the model, and you pay for all four again just to get back to where you already were. With Temporal, the retry re-runs step five and only step five. Steps one through four are already-billed calls that never happen twice.

That’s the difference between retries as a cost multiplier and retries that are free. On a batch enrichment job that fails at item 47 of 50, you re-bill item 47, not items 1 through 46. I wrote a whole post about that in Part 6, and it’s the same mechanic every proposal here leans on. Memoization is why a flaky provider or a worker restart doesn’t torch the budget.

Cheapest Model First, Escalate Only on Demand

The next lever is refusing to reach for the expensive tier by default. The llmCall activity would run a model cascade: cheapest capable tier first (Haiku-class, Flash-class), escalating only when the task actually needs it.

Most of what these features do is mechanical. Tagging a message. Summarizing a row. Extracting a field. That work does not need a frontier model, and paying frontier rates for it is just lighting money on fire. So the cascade defaults to cheap, and it reserves the Opus-class tier for the thing the user explicitly asked for: long-form synthesis, a research draft, a document they told me they want written well.

Paired with that: per-target max_tokens ceilings, so a runaway generation can’t balloon. A message caps around 1k, a list around 4k, a document around 16k. The ceiling isn’t a polite suggestion the model is vaguely aware of. The activity enforces it as a hard cap. So you get the cheap tier by default, a small ceiling per target, and the expensive model only shows up when you asked for it.

A Gate That Throws Before It Spends

Cascades and ceilings keep individual calls cheap. But the scenario that scares me is unbounded spend: a runaway agent loop, or a leaked user key someone else is now happily burning. So the plan puts a budget gate before every call, not after.

An assertBudget activity reads accumulated spend from the AiGeneration ledger and throws a non-retryable BudgetExceeded when the user is over their ceiling. Non-retryable is the important word. A normal failure retries; this one halts the workflow cold, because retrying a budget breach just tries again to spend money you’ve already said no to.

Roughly what I’d hand to Claude as the shape:

// activity: runs on the worker, before any billable llmCall
export async function assertBudget(userId: string, estimate: TokenEstimate): Promise<void> {
  const spent = await getSpendThisPeriod(userId); // counts only, from AiGeneration
  const remaining = budgetFor(userId) - spent;

  if (estimate.maxTokens > remaining) {
    // non-retryable: a budget breach must not loop and re-attempt the spend
    throw ApplicationFailure.nonRetryable(
      `Budget exceeded: need ${estimate.maxTokens}, have ${remaining}`,
      "BudgetExceeded",
    );
  }
}

A workflow calls assertBudget and then llmCall, in that order, every time. Worst case is a bounded overspend of one call, never an open tap.

Never Bill the Same Request Twice

Then there’s the plain-dumb-obvious saver: caching. A response cache keyed by hash(system + prompt + model). If an identical request comes through (same system prompt, same user prompt, same model) it’s served from cache and never re-billed. Regenerate the same summary twice, pay once.

Same idea one layer down for link work: URL-hash dedup. When the crawler from Part 5 fetches and embeds a link, it keys on a hash of the URL. A link shared across ten messages, three docs, and a list row gets fetched, extracted, and embedded exactly once, ever. Embeddings are already orders of magnitude cheaper than generation, so computing them once and caching forever makes the cheapest part of the pipeline round down to nothing.

And to keep from wasting tokens on retries you caused yourself: worker concurrency and task-queue rate caps tuned to stay under provider limits. Blow past a rate limit and you get a 429 storm, and every retry in that storm is wall-clock and, on some providers, tokens down the drain. Obeying the limit is cheaper than fighting it.

One more, because it’s the most satisfying: confirm-before-spend. For the prompt-to-list flow, the workflow generates the schema, then pauses on a Signal and waits. It does not spend a single row-generation token until the user confirms the schema is right. Get the schema wrong and you’ve spent one cheap schema call, not fifty row calls against a shape nobody wanted. That pause is impossible in a serverless handler and native in Temporal.

The Ledger Counts, It Doesn’t Read

Every one of these levers needs a source of truth, and that’s the AiGeneration ledger. Critical design decision: it records counts and status only. Tokens in, tokens out, which model, succeeded or failed. It never stores the prompt text or the model output.

That’s not laziness. One table does three jobs. Privacy, because I’m not warehousing what people wrote or what the model said back. The quota counter, because assertBudget reads its sums off it. And the cost dashboard, because those same rows drive a per-user “AI usage and spend” view: tokens, calls, cache-hit rate, budget remaining. When something looks expensive, the ledger doubles as the debugging trail. You can see that a workflow made forty calls without ever seeing what it said.

Build the Guardrails With Claude, Then Make It Prove Them

Now the part I’ve been wanting to get to, because this is where working with an LLM changed how I’d approach any of it.

Memoization and caching only save money if they’re in the right places. Put a cache breakpoint one step too early and you re-bill everything after it. Miss a memoization boundary and a retry quietly re-runs a paid call. The savings are entirely a function of where the boundaries land, and eyeballing a six-step pipeline for those boundaries is the kind of thing I get wrong.

So the first move is a diagnostic question, not a code request. I’d point Claude at the actual workflow and ask: “walk this pipeline and tell me exactly where a retry would re-bill the model.” Then let its answer drive placement. It reads the activity boundaries, traces the retry path, and tells me which calls are already durably memoized and which ones a mid-pipeline failure would re-invoke. That answer is the map. The caching and memoization go where the map says, not where I guessed.

The second move matters more: make it prove the savings. Memoization you can’t prove is just a hope with good intentions. So I’d have Claude write a Temporal replay test (using TestWorkflowEnvironment) that fails an activity partway through the pipeline and then asserts the already-completed LLM activities are not invoked again on retry.

it("does not re-bill completed LLM activities on retry", async () => {
  const llmCall = vi.fn()
    .mockResolvedValueOnce("step-1 result")
    .mockResolvedValueOnce("step-2 result")
    .mockRejectedValueOnce(new Error("provider blip at step 3")) // fail mid-pipeline
    .mockResolvedValue("step-3 result (retry)");

  await worker.runUntil(client.workflow.execute(generatePipeline, { /* ... */ }));

  // steps 1 and 2 completed before the failure, so Temporal replays their
  // memoized results and MUST NOT call the model for them again.
  const stepsCalled = llmCall.mock.calls.map((c) => c[0].step);
  expect(stepsCalled.filter((s) => s === 1)).toHaveLength(1);
  expect(stepsCalled.filter((s) => s === 2)).toHaveLength(1);
  expect(stepsCalled.filter((s) => s === 3)).toHaveLength(2); // failed once, retried
});

That test is the proof. It fails loudly the day someone refactors the pipeline in a way that breaks memoization and starts silently re-billing steps one and two. The savings stop being a story I tell about the architecture and start being a property CI enforces. Same loop for the response cache: a test that fires two identical requests and asserts the second one hits cache and never reaches the model.

Ask the agent where the money leaks. Let its answer place the guardrails. Then make it write the test that locks them in. That loop works because a cost regression is invisible until the bill shows up weeks later, and a test drags it into the light on the exact commit that caused it, while you can still git blame your way back to it.

Where This Lands

Part 8 is the capstone: the agentic research-to-draft flow, the single most token-hungry thing in the whole set. It’s the reason every governor in this post has to exist before it ships. I’ll also lay out how I’d sequence the entire build, starting from the foundation phase (the Temporal wiring and the shared cost primitives everything else reuses) and working all the way up to the agent, and how I’d verify each piece as it lands.

Adron brainstorming and working on InterlinedList.

Tag Everything, Expand Anything, Without Re-Billing the Model

2026-08-16T09:30:00-07:00

Part 6 of 8: Durable AI for InterlinedList

Every serverless AI feature I’ve built hits the same wall the second someone wants it at scale. “Can it tag all my untagged messages?” Sure, one message. “All of them?” Now we’re talking about a job that runs for minutes, touches hundreds of items, and will absolutely fall over partway through. And when it falls over, the naive version starts from zero and re-bills you for every item it already finished. Nobody wrote that behavior on purpose. It’s just what you get by default, and it’s expensive.

This is the post in the Durable AI series where durability pays off the most. Not “nice to have.” It’s the biggest cost saver in the whole series, and I’m putting that up front so you don’t skim past it.

The Jobs That Don’t Fit in a Request

There’s a whole class of work InterlinedList can’t do today and won’t be able to do inside a Vercel function, no matter how the AI plan lands. It’s the bulk stuff:

“Add 20 rows to this list matching X.”
“Tag every untagged message.”
“Summarize each finished book in my reading list into its own document.”
“Backfill link previews across all my docs.”

What these share is that they’re multi-minute, many-item, and retry-heavy. A serverless handler has a hard duration ceiling and no durable state between invocations. If item 34 of 50 throws a 429 from the model provider, the whole request dies and there’s no clean way to pick up at 35. So you either don’t offer the feature, or you ship a fragile version that quietly loses work and double-charges for the rest.

Today’s ground truth matters here, so let me be precise. Lists are a Subscriber feature. A list’s row data is stored as JSONB in ListDataRow.rowData, validated against a user-defined schema. The validator that gates every row lives in lib/lists/dsl-validator.ts. None of the batch AI machinery below exists yet: this is a brainstorm I’d hand to Claude to build, and AI assistance in the product is still “Coming Soon.” I’m designing it out loud.

The Shape: Fan Out, Cap Concurrency, Track Progress

The design I keep landing on is a generic fan-out workflow. One workflow supervises the whole batch; one item equals one activity. Each activity is independent and idempotent. It does its work, writes its result, and doesn’t care whether its neighbors succeeded or failed.

Temporal is the durable engine underneath, for reasons that are specific and not hand-wavy:

Partial progress is preserved. A completed activity’s result is persisted. When the workflow retries after a failure at item 47 of 50, it re-runs item 47 and only item 47. Items 1 through 46 already succeeded and their results are memoized, so it doesn’t touch them. On a batch of 50 LLM calls where one flakes, you re-bill one item, not fifty. This is the money argument, and I like it because it’s structural. You get it from the shape of the thing, not from remembering to code it.

Concurrency is capped. You don’t fire 50 model calls at once and trigger a 429 storm. A 429 storm is expensive twice over: the retries burn wall-clock, and on some providers they burn tokens too. Cap concurrency to a small number and you respect the provider’s rate limit (and your own BYO-key limit), so the retries never happen at all.

The job survives a worker restart. Deploy mid-batch, crash the worker, whatever. The workflow resumes from where it was, not from zero.

Here’s the constraint made concrete: a generic batchEnrich workflow that fans out over items with a bounded concurrency window.

// Workflow: supervises the batch, never touches the model or the DB directly.
export async function batchEnrich(input: BatchEnrichInput): Promise<void> {
  const { items, concurrency } = input; // e.g. concurrency = 5
  const running: Promise<void>[] = [];

  for (const item of items) {
    // enrichItem is an ACTIVITY: its result is durably memoized.
    // On a workflow retry, already-completed items are NOT re-invoked.
    // Temporal replays their recorded results instead of re-billing them.
    const p = enrichItemActivity(item).then(() => {
      recordProgress(item.id); // drives the live progress UI
    });
    running.push(p);

    // Cap in-flight work so we never exceed the provider / BYO-key rate limit.
    if (running.length >= concurrency) {
      await Promise.race(running);
      // prune settled promises so the window stays at `concurrency`
    }
  }

  await Promise.all(running);
}

The per-item activity is where the real work (and the idempotency) lives. It reuses code that already ships in the repo:

// Activity: one item. Independent, idempotent, individually retryable.
export async function enrichItemActivity(item: EnrichItem): Promise<void> {
  // 1. Cheap-model call for the mechanical part (tag / summarize).
  const enriched = await llmCall(item.prompt, { model: "cheap" });

  // 2. Validate against the list's real schema before writing anything.
  //    validateFormData lives in lib/lists/dsl-validator.ts, the same gate
  //    the app uses for every human-entered row.
  const result = validateFormData(item.fields, enriched.rowData);
  if (!result.isValid) throw new NonRetryableError(result.errors);

  // 3. Write through the existing materialize helpers, not raw Prisma.
  //    lib/materialize/build-list.ts / build-doc.ts already know how to
  //    turn structured data into rows and documents with ownership intact.
  await writeThroughMaterialize(item.target, enriched);
}

Nothing here reinvents validation or writing. validateFormData is the existing row validator. lib/materialize/build-list.ts and build-doc.ts are the existing builders that turn structured content into lists and documents. The persistence writers I sketched back in the foundation phase, the groundwork everything in this series stands on, handle the actual writes with IDOR checks intact. The workflow’s only job is orchestration: fan out, cap, track, resume.

The Cost Levers, Stacked

Cost is the spine running through this whole series, so let me connect the dots. Three levers push the LLM bill down, and this one feature stacks all three:

Batching. Tagging is mechanical and small. Instead of N calls to tag N messages, one call tags a batch of them: you collapse the per-request overhead and the token duplication of repeating the same system prompt N times over. Fan-out doesn’t have to mean one-model-call-per-item. The fan-out unit can itself be a batch.

A cheap-model default. Tagging and summarizing don’t need a frontier model. They need a competent, fast, cheap one. Claude Haiku 4.5 runs at $1 per million input tokens and $5 per million output; Opus 4.8 is $5 and $25. That’s a 5x swing for mechanical work where the cheap tier does the job fine. The default model for a batch-tag job should be the cheap one, and you escalate only for the rare task that needs it.

No redo on retry. Memoization again: the durable engine never re-invokes an already-billed call on a retry. This is the lever that costs you nothing to pull, because it’s baked into how the workflow resumes.

Put those together and a “tag everything” run over a few hundred messages costs cents instead of dollars, and a mid-run failure costs you one retry instead of a full re-run.

What I’d Tell Claude Before It Writes Anything

There’s a “working with Claude” lesson buried in all this, and honestly it’s why I wanted to write this post at all.

Resumability is a design constraint, not a feature you bolt on later. Ask an agent to “write a batch tagging job” and say nothing else, and you get a for loop that calls the model N times and stores results at the end. It works in the demo. Then it falls over on the first real batch, loses partial work, and re-bills everything on retry. Now you’re spending a day retrofitting durability into code that was never shaped for it, and durability doesn’t retrofit cleanly, because it changes what a “unit of work” even is.

So I state the constraints before the code, right in the prompt:

Each item is an independent, idempotent activity. The workflow must resume mid-batch without re-running finished items: a failure at item 47 of 50 re-processes only item 47. Cap concurrency at N to stay under the provider rate limit. Validate every item against the list schema before writing, and write through the existing materialize helpers, not raw Prisma.

Hand it that, and Claude generates the right thing: the generic batchEnrich workflow, the per-target activity adapters (list rows go through build-list, documents through build-doc, message tags through the tag writer), and the batched prompt that tags many items in one call. The constraints do the architectural work; the agent fills in the wiring. I own the seam (what’s an activity, what’s idempotent, where the concurrency cap lives) and let it own the code.

The failure mode I’m avoiding is the one where the constraint shows up after the code. State “don’t re-bill completed items” up front and it shapes the whole structure. State it after, and you’re rewriting.

The Surface

On the product side this becomes a “run a bulk action” panel: pick a target (a list, your messages, a set of docs), describe the enrichment, and watch live progress as items complete. Cancel is a Temporal Signal, a real mid-flight stop rather than a “please ignore the results” fiction. The workflow receives the signal, stops fanning out new items, lets the in-flight ones finish or abort, and reports what it got done. Progress and cancel both fall out of a workflow that already tracks per-item state.

Part 7 is the payoff post: the cost governor, where all of these levers get formalized into a budget ledger, a model cascade, and the response cache that make “cheap enough to ship” measurable instead of hoped-for.

Adron brainstorming and working on InterlinedList.

Where Did I Save That Article? Crawl, Cache, Retrieve

2026-08-15T06:30:00-07:00

Part 5 of 8: Durable AI for InterlinedList

I have a problem, and I bet you have it too. I read something good, I drop the link into a post or a document or a row in some list, and then three weeks later I want it back and I have no idea where it went. Was it a message? A doc? A row in that reading-list I built? So I scroll my own feed like an archaeologist. It’s a slow-moving disaster. The whole point of writing something down is being able to get it back, and I’d built the writing-down part beautifully while ignoring the getting-it-back part entirely.

So this is the post where I brainstorm fixing that. Two pieces come from the Temporal plan, and they run in order. First, the link-crawl-and-cache phase turns saved links into a real cache. Then the retrieval phase makes that cache searchable, semantic and keyword, with cited snippets, so “where did I save that article about X” actually returns the article about X. None of this is shipped. AI assistance in the product is still “Coming Soon.” This is me thinking out loud with the intent to build.

What Actually Exists Today (Less Than You’d Hope)

Before I design anything, I made myself write down what the codebase does right now, because designing against a fantasy version of your own product is how you build the wrong thing.

Today, link metadata gets fetched for messages only. When you post something with a URL, lib/messages/metadata-fetcher.ts pulls the Open Graph tags and the result gets stored inline on the message as a JSONB blob: Message.linkMetadata. That’s it. There’s no cache table. There’s no dedup, so if you and forty other people all post the same great article, that URL gets fetched forty-one times. Documents and list rows (both Subscriber features) don’t get scanned for links at all. And search? Search today is Postgres ILIKE substring matching across three separate routes: messages, lists, documents. It’s a contains query. Type “kubernetes” and you get rows where the literal string “kubernetes” appears. Miss the spelling, miss the synonym, miss the concept, and you get nothing. There is zero vector infrastructure anywhere in the repo. I checked.

That’s the honest starting line. It’s also a decent starting line, because two of the hard parts are already solved and I’d be a fool to rewrite them.

The Crawler: Reuse the Fetcher, Reuse the Guard

So, the caching step. A durable workflow scans a user’s messages, documents, and list rows for URLs, fetches each one, extracts the readable content, and stores it in a new LinkContent cache table keyed by a hash of the URL. Deduplicated across all your content, and optionally across all users. It runs incrementally: only new or changed URLs get fetched, so the second run is nearly free.

The building blocks already exist. lib/messages/link-detector.ts gives me extractUrls and detectLinks. lib/messages/metadata-fetcher.ts gives me fetchLinkMetadata. And critically, lib/security/ssrf.ts gives me safeFetch and assertUrlAllowed, the SSRF guard the codebase already fought to get right. This work is mostly generalizing the message-only path to also cover docs and list rows, then bolting a real cache behind it so a URL is fetched once and never again.

This is the single most important thing to say to Claude when you hand it this task, so I’ll say it here first: build on safeFetch and assertUrlAllowed. Do not write a new fetch().

I cannot overstate this. Left to its own devices, an LLM asked to “write a crawler that fetches URLs and extracts content” will cheerfully write you a fresh fetch(url) with a nice timeout and some error handling, and it’ll look completely reasonable, and it’ll have reintroduced the server-side request forgery hole this codebase already closed. It won’t check whether the URL resolves to 169.254.169.254 and hands back your cloud metadata. It won’t block the redirect that bounces from a public host to localhost. The existing safeFetch does all of that, and it re-validates on every redirect hop. A brand-new fetcher is a security regression wearing the costume of a feature.

So the instruction isn’t “write a crawler.” The instruction is:

// Crawler activity: the reuse contract, spelled out for the agent.
// 1. detectLinks/extractUrls from lib/messages/link-detector.ts to find the URLs
// 2. hash each URL, check the LinkContent cache, skip if already fetched & unchanged
// 3. fetch ONLY through safeFetch from lib/security/ssrf.ts (never a raw fetch)
// 4. extract readable text mechanically (HTML to text), NO LLM in this path
// 5. upsert into LinkContent keyed by urlHash; dedup means fetch-once-ever
async function crawlUserLinks(userId: string) {
  const urls = await collectUrlsFromMessagesDocsAndRows(userId); // reuse extractUrls
  for (const url of dedupeByHash(urls)) {
    if (await cacheHasFresh(url)) continue;         // incremental: skip known URLs
    const html = await safeFetch(url);              // SSRF guard, NOT a new fetch()
    await upsertLinkContent(url, extractReadableText(html));
  }
}

Note what’s not in there: a language model. HTML-to-readable-text is mechanical. You strip the tags, you pull the article body, you’re done, no tokens spent. An optional cheap-model summary of each page could sit on top later, opt-in, for people who want it, but the base extraction costs nothing per link beyond the fetch. Pair that with URL-hash dedup (a shared link is fetched once, ever, across the whole system) and this becomes the cheapest proposal in the whole plan for what you get out of it. Which is why I like it. It’s mostly plumbing, mostly reuse, and it barely touches an LLM.

Retrieval: The Part That Needs New Ground

The crawl phase gives me the cache. The search-over-my-links phase is what makes it findable. This is “search over my link index and my own docs, lists, and messages, returning cited snippets.”

And I have to be honest that this one needs real new infrastructure, because the vector story doesn’t exist yet: not partially, not disabled behind a flag, not anywhere. So the retrieval phase needs a pgvector extension, an embedding column or table, an embed activity that turns text into vectors, and a search route that ranks by both vector similarity and the existing ILIKE match. Hybrid. It supplements today’s substring search rather than ripping it out. Keyword matching is still the right answer for exact-string lookups, and it needs no model at all.

The cost math here is the reassuring part. Embeddings are orders of magnitude cheaper than text generation: you’re paying to turn text into a vector, not to write prose. You compute each embedding once and cache it, sharing the same cache the crawl phase already built. And because retrieval comes first, any eventual answer-synthesis step runs over a small, already-narrowed set of retrieved snippets instead of dumping your whole corpus into a prompt. Keep the expensive context small and this stays affordable.

The incremental pipeline is what keeps it fresh without a nightly full re-scan: on every content write (new post, edited doc, added row) enqueue a re-embed activity for just that thing. Batch the embeddings for throughput. The index stays current, and you never re-embed the 99% of your corpus that didn’t change.

The Second Lesson: Don’t Let the Agent Touch the Database Directly

The pgvector migration is where the other working-with-Claude lesson lives, and it’s a governance one.

This repo has a strict database workflow, and it’s strict for a reason the CLAUDE.md states plainly: violating it has broken production before. Additive-only. Idempotent SQL. Hand-written migration files applied to both databases. No prisma db push, no prisma migrate dev, no raw DDL fired at the database from a script.

An LLM does not know this in its bones. Ask it to “add a vector column and an embeddings table” and its instinct (a perfectly good instinct in most repos) is to edit the schema and run prisma db push to sync it. In this repo that’s a loaded gun. So the instruction to the agent isn’t just “add the schema.” It’s: route this through the db-migrations workflow. Write an idempotent, additive migration file (CREATE EXTENSION IF NOT EXISTS vector, CREATE TABLE IF NOT EXISTS, the column adds guarded) and do not run db push or migrate dev against the database. The good news is the migration is cleanly additive: there’s no existing vector infra to alter or drop, so it’s all new tables and columns, which is exactly the shape this workflow handles best.

Two guardrails, then, and I’d hand both to the agent up front rather than catch them in review: reuse safeFetch instead of writing a fetcher, and route the schema change through the migration workflow instead of letting it push. Both are cases where the model’s default behavior looks reasonable and is wrong for this codebase. Tell it about the thing that already exists, and tell it not to reinvent it.

The payoff, if I build it, is the thing I wanted in the first place. I type “that article about durable execution I saved somewhere,” and I get it back with the snippet and a link to wherever I stashed it, whether that was a throwaway post or a row buried in a list. My scattered links stop being a graveyard I dig through and turn into something I can actually ask questions of. Subscriber-gated, computed once and cached, and cheap enough that I’d leave it running without watching the bill.

Next up in Part 6, I want to point this same durable-fan-out machinery at bulk work: tag every untagged message, add twenty rows matching a criterion, backfill previews across everything, and do it without re-billing the model for the forty-six items that already succeeded before item forty-seven fell over.

Adron brainstorming and working on InterlinedList.

The Cron That Double-Posts and the Calendar I Actually Want

2026-08-14T06:30:00-07:00

Part 4 of 8: Durable AI for InterlinedList

Most of this series is me thinking out loud about features that don’t exist yet. This post is different as the bug is real. It’s in the code today. 😬

Scheduled publishing is a Subscriber feature, and so is the cross-posting it rides on. You queue a post, set a time, and a cron fans it out to Bluesky, Mastodon, LinkedIn, and X when the moment arrives. Both sit behind the subscription, so this bug hits paying subscribers. The machinery that runs it has a hole in it I don’t love.

The Cron That Can Post Twice

Walk through what the code does. app/api/cron/publish-scheduled-messages/route.ts runs every minute. It selects every message where scheduledAt <= now, then loops over them. For each one it fans out to whatever platforms the user configured, collects the resulting URLs, and then, as the last step, updates the message row to null out scheduledAt so the message is no longer “due.”

Read that ordering again. The cross-post happens first. The “mark it done” write happens second. No lock. No idempotency key. No transaction spanning the fan-out and the update. Those two operations are just sequential, and hopeful.

So picture the fan-out succeeding. Your post lands on Bluesky and Mastodon. Great. Now the message.update that’s supposed to clear scheduledAt hits a reaped Neon connection, or a transient network blip, or any of the ten boring things that make a database write fail once in a while. The update throws. The message row still has scheduledAt <= now.

One minute later the cron runs again. The message is still due. It fans out again. Same post, second time, to every platform. That’s your double-post.

The code isn’t naive about this: there’s a withPrismaRetry wrapper on that critical update and a comment that literally says “if this fails the message stays due and is re-published.” Someone saw the cliff and put up a guardrail. But a retry is a mitigation, not a fix. It narrows the window without closing it. If the retries exhaust, or the process dies between the fan-out and the update, you’re back on the cliff. And the fan-out itself is best-effort per platform (LinkedIn can fail while Bluesky succeeds, with no compensation), so a re-run doesn’t even cleanly re-do the same thing. It re-does some of it.

This is a slow-moving disaster. It doesn’t fire on every post. It fires on the unlucky ones, occasionally, in production, and it makes your platform look like it can’t be trusted to post your thing exactly once.

The Working-With-Claude Lesson: Point at the Bug Class, Not the Bug

If you take one thing from this post, take this part, not the fix. It’s worth more.

If I open a session and say “fix the double-post in publish-scheduled-messages,” I get a patch that treats the symptom. Maybe it wraps more things in retries. Maybe it adds a published boolean and checks it. Narrow prompt, narrow fix, and I’ve probably left three other copies of the same hazard sitting in the codebase untouched.

So I don’t open it that way. I say: “Audit this scheduled-publish path for concurrency and idempotency hazards.”

That framing is doing real work. It doesn’t presume I already know what’s wrong. It asks the agent to reason about the class of failure: what happens under a partial failure, what happens if two invocations overlap, what’s the ordering of side effects versus state writes, where’s the operation that isn’t idempotent. The double-publish race falls out of that audit as one instance of a general pattern (side effect before commit, no dedup key). And the audit usually surfaces siblings I hadn’t thought to ask about.

This is the two-step I keep coming back to with Claude on anything gnarly. Audit for the class, then delegate the conversion. First get the agent to characterize the whole shape of the problem. Then, once you both understand it, hand over the mechanical rewrite. Collapse those into one “fix it” prompt and you’ll get a bandage every time.

Most of the value is in the audit. The conversion is mostly typing, and Claude is very good at the typing once you’ve done the thinking with it.

The Fix: A Schedule, a Workflow ID, and a Saga

So this is the conversion I’d delegate. It’s a proposal. None of it is built yet.

Replace the every-minute cron with a Temporal Schedule that starts one workflow per due message. The whole trick is what you use for the workflow ID.

// Proposed: the message id IS the workflow id.
// Temporal refuses to start a second workflow with an id that's
// already running or already completed. That's the dedup.
await client.workflow.start(publishScheduledMessage, {
  workflowId: `publish-message:${message.id}`,
  taskQueue: "publishing",
  args: [{ messageId: message.id }],
});

Temporal deduplicates by workflow ID. If a workflow with publish-message:abc123 has already run (or is running right now), a second start with that same ID is rejected or returns the existing handle, depending on the reuse policy you pick. The message cannot be published twice, because you cannot start its publish workflow twice. The race doesn’t get narrower. It closes. There’s no second racer allowed on the track, so there’s nothing left to race.

That’s a different guarantee than “we retried hard enough that it usually works.” One is a hope with good odds. The other is impossible by construction, and I’ll take impossible.

Then model the fan-out as a saga instead of a hopeful loop. One activity per platform: postToBluesky, postToMastodon, postToLinkedIn, postToTwitter, the exact post-status.ts functions the cron already calls, plus the same splitTextForPlatform and resolveLinkedInTarget logic. Each activity retries independently with backoff, and each one’s success is durably recorded before the next runs. If LinkedIn fails after Bluesky succeeded, the workflow knows Bluesky is done and won’t re-do it. It records the half-failure coherently and runs compensation, instead of silently re-posting the whole set on a blind re-run.

And the thing that makes this an easy call: publishing uses zero LLM. No model in this loop at all. No tokens, no inference cost, nothing to govern with a budget. It’s pure durability engineering, a straight win against a present-day production hazard with no ongoing cost attached. If anything in this series jumps the queue, it’s this.

The Calendar I Actually Want

Once publishing is durable, the thing I’ve wanted for ages becomes buildable. This part is squarely future, gated behind AI assistance that’s still marked Coming Soon.

I want to type “give me a month of posts about distributed systems, three a week” and get a filled calendar back. Not a wizard. Not twelve separate button clicks. A month.

The cheap way to build that leans on batching. One LLM call produces all N drafts at once, and batching many items into a single prompt is dramatically cheaper than N separate calls (the cost thesis that runs through this whole series). The workflow takes those drafts, schedules them across the month, and fills the gaps. Then, to keep it cheap, you can reject or regenerate an individual slot. Don’t like Thursday’s post? Regenerate just that one. Only the rejected slot re-bills. You never repay for the twelve you kept.

The same shape covers scheduled digests: a Temporal Schedule that drafts you a weekly summary doc from the week’s messages. But put a cheap guard before the model runs. Did this user post anything this week? If not, skip the LLM entirely. You never pay to generate an empty digest for a quiet week. The guard is a database count. The model is the expensive part. You gate the expensive part behind the cheap check, which is the move I keep making all through this series: spend tokens last, and only when there’s something worth spending them on.

The calendar is the feature I daydream about. The double-publish fix is the one I’d ship first, because it’s a real hole in production right now and closing it costs nothing but the work.

Part 5 goes hunting through everything you’ve saved: the links buried in your messages, docs, and list rows, and how I’d crawl and cache them so “where did I save that article?” finally has an answer.

Adron brainstorming and working on InterlinedList.

A Generate Button That Survives a Timeout and Asks Before It Writes

2026-08-13T06:30:00-07:00

Part 3 of 8: Durable AI for InterlinedList

In Part 2 I talked myself into a durable worker sitting next to Vercel, reusing the lib/ code the app already trusts. That was the plumbing. Now for the first feature I’d actually run through it, a Generate button, and the two things I won’t ship it without.

One: it has to survive a timeout. Two: it has to ask before it writes.

I keep calling those hard requirements because they are. They’re the reason this design points at a durable workflow instead of another POST handler. And to be clear up front, since AI assistance in the product is still marked Coming Soon: none of this exists yet. I’m brainstorming out loud before I hand pieces of it to Claude.

Two Ways a Naive Generate Button Fails

Picture the obvious version. A route that takes a prompt, calls a model, writes the result. Works great in the demo. Then two things happen.

First, someone asks for a long document. Sixteen thousand tokens of generated Markdown streaming back through a serverless function that has a hard duration ceiling. The function dies mid-stream. The user gets nothing, and depending on where it died, maybe a half-written doc. That’s not a fluke. Any generation big enough to be worth doing is big enough to hit the ceiling.

Second, and this is the one that scares me, the model writes straight into your data. For a plain message, fine, you can delete it. For a List schema, that’s a slow-moving disaster. A list’s schema is a set of ListProperty rows parsed from the DSL, and once rows of data hang off those properties, unwinding a wrong guess gets expensive fast. An AI that invents seven columns and commits them before you’ve looked is a terrible first impression. Undoing a bad schema is the wrong goal. You want to never have written it in the first place.

So the design has to fix both: don’t let a long job die in a function, and don’t write anything the user hasn’t confirmed.

Suggest, Then Generate: Two Endpoints on Purpose

The split I keep coming back to is two endpoints, and the difference between them carries the whole safety story.

/suggest returns a validated artifact and writes nothing. /generate writes.

/suggest is the default path. It produces the thing (a doc draft, a list schema, a set of rows), validates it, and hands it back to the UI for a human to look at. Nothing touches the database. /generate is the second half, and it only runs after a person confirms.

In the durable version, this isn’t two disconnected calls. It’s one workflow that pauses in the middle. The route starts a workflow and returns an id. The workflow generates the artifact, then blocks on a Signal (Temporal’s native “wait for an external event”) until a confirm arrives. Long documents don’t block a function while this happens; the UI polls the workflow and reads progress through a Query. The confirm route (/generate/[workflowId]/confirm) sends the signal, the workflow wakes up, and then it writes.

// Sketch of the workflow body (not shipped, this is the design I'd hand off).
export async function generateArtifactWorkflow(input: GenerateInput) {
  // 1. Draft on a cheap tier. Result is memoized by Temporal.
  const artifact = await generateArtifactActivity(input);   // Haiku/Flash draft
  const validated = await validateArtifactActivity(artifact); // untrusted, checked

  // 2. Pause here. No function held open. No tokens spent waiting.
  setHandler(confirmSignal, (decision) => { confirmation = decision; });
  await condition(() => confirmation !== undefined);

  if (confirmation === "reject") return { status: "discarded" };

  // 3. Only now do we write, reusing the same lib/ writers the app trusts.
  return await writeArtifactActivity(validated, input.userId);
}

The part I like: while that workflow is parked on the signal, it costs nothing. No held-open request, no tokens burning, no worker thread stuck. Temporal parks it and moves on. You cannot do that inside a serverless handler, and it’s exactly the behavior a confirm gate needs.

Prompt to List Is Where This Really Pays Off

The flagship case, the one I’d build the whole pattern around, is “build me a list from a sentence.”

You type “a reading list with title, author, status, and a rating out of five.” The workflow generates a DSL schema, not rows. It renders that schema for you to confirm. You look at it, maybe you fix the rating field, you say yes. Only then does it batch-generate rows to populate the list.

Two phases, and the confirm barrier sits between them for a reason: no row-generation tokens get spent until the schema is confirmed. If the model guessed the schema wrong, you caught it before paying to fill fifty rows into the wrong shape. So confirm-before-spend does double duty here, better UX and a cost control at once, which is the same thread running through this whole series. The cheapest token is the one you never send.

And I’m not inventing new validation to do this. The DSL already has the pieces. validateDSLSchema and parseDSLSchema both live in lib/lists/dsl-parser.ts today, and they’re what turns DSL into ListProperty rows for real, human-authored lists right now. The generated schema goes through the exact same door. Then lib/materialize/build-list.ts builds the actual list. The AI path reuses the trusted path; it doesn’t get a shortcut around it. (Lists are a Subscriber feature, so this whole flow lives behind that gate. The workflow checks it before writing, same as every other list-creating action.)

Model Output Is Untrusted Input

One mental flip makes the rest of the design fall out easily. Model output is untrusted input. Treat it exactly like a request body from a stranger.

You’d never take a JSON blob off the wire and write it to your database without validating the envelope shape, checking the types, and confirming the user is allowed to do the thing. A model’s output gets the identical treatment: envelope shape first, then DSL validation, then per-row validation, then ownership and subscription gating. Only something that survives all four gates gets written. If the model hallucinates a field type the DSL doesn’t support, validation rejects it the way it’d reject any malformed request. The model doesn’t get trusted just because it’s ours.

How I’d Hand This to Claude

If there’s one working-with-an-LLM lesson in this post, it’s this: design the contract, delegate the wiring.

There’s a trust boundary running through this feature, and I want to own every inch of it myself. Three pieces I write by hand: the typed artifact envelope (what a generated doc/list/schema/row is shaped like), the validators (envelope, then DSL, then per-row, then gating), and the ownership and subscription checks. That’s where a subtle mistake means a wrong schema gets written or a free user slips past a gate. I don’t delegate that. A plausible-looking version isn’t good enough. I want to have written it.

Everything around that contract is the kind of thing I’d hand to Claude. The workflow body. The signal handler and the condition wait. The polling glue and the Query that streams progress. The confirm route. It’s durable-execution boilerplate: Claude writes it well, and it’s fast to verify against a contract I already defined.

The hand-off is concrete. I give Claude the typed artifact schema and one rule stated plainly: “validate model output like a request body (envelope, then DSL, then per-row, then gating) and never write it through raw. Here are the validators; call them, don’t reimplement them.” Then I let it build the durable plumbing around that. I own the trust boundary, Claude owns the machinery that carries data across it, and the review stays easy because I already know where the sharp edge is. I made sure it’s mine.

A Generate button that can’t die mid-job and won’t write behind your back is the piece I want first. Next in Part 4, I’ll turn to the cron that already double-posts in production today, and the content calendar I actually want built on top of a scheduler that doesn’t.

Adron brainstorming and working on InterlinedList.

Where Does Temporal Even Run When You’re All-In on Vercel?

2026-08-12T06:30:00-07:00

Part 2 of 8: Durable AI for InterlinedList

In Part 1 I talked myself into Temporal as the orchestration layer for the AI features I want to build. Durable memoization, a model cascade, a hard budget gate: all the mechanics that push the LLM bill down. Good story. Then I hit the question that stops most “let’s just add Temporal” plans cold, and it stopped me too.

Where does it actually run?

InterlinedList is 100% serverless on Vercel. Every route is a function that spins up, does its thing, and gets torn down. That model is wonderful right up until you need something that never stops running. And Temporal needs exactly that.

The Two Parts That Don’t Fit

Temporal has two moving pieces, and neither of them is a serverless function.

The first is the Temporal Service, the durable-state backend. It’s the thing that remembers where every workflow is, what’s completed, what’s pending, and what to retry. It’s stateful infrastructure. You don’t invoke it and let it die.

The second is the Worker. A worker is a long-lived process that polls a task queue, picks up work, and runs your activities. Polls it continuously. As in, forever. A Vercel function that shuts down after a request cannot be a Temporal worker, because the polling loop dies the instant the function returns. You can’t flag your way out of this in a config file. It’s a straight-up mismatch between “poll a queue for hours” and “respond in seconds then vanish.”

So for a while this felt like a dead end. Adding Temporal seemed to mean un-serverless-ing the whole app, and I did not build InterlinedList on Vercel just to go stand up a fleet of always-on boxes.

One New Always-On Thing. That’s It.

The resolution I kept circling back to is smaller than the fear suggested. I introduce exactly one new always-on component: a small @temporalio/worker process running somewhere cheap and boring, like Fly.io, Railway, or a small VM. Something in the five-to-twenty-dollars-a-month range. Alongside it, either self-hosted Temporal OSS or a low-tier Temporal Cloud namespace for the service.

That’s the entire footprint. One worker, one service. Vercel doesn’t change shape at all.

What does change is what the Vercel routes do. They stop doing work and become thin clients. Authenticate with the getCurrentUser / getCurrentUserOrSyncToken helpers I already have, gate with isSubscriber, start a workflow, then hand the UI a workflow id to poll. The route never does the long-running work itself. It just points at it.

The route sketch is almost embarrassingly short:

export async function POST(request: NextRequest) {
  const user = await getCurrentUserOrSyncToken(request);
  if (!user) return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
  if (!isSubscriber(user.customerStatus)) return NextResponse.json({ error: "Forbidden" }, { status: 403 });

  const handle = await client.workflow.start(generateArtifactWorkflow, {
    taskQueue: "ai",
    workflowId: `gen-${user.id}-${crypto.randomUUID()}`,
    args: [{ userId: user.id, prompt }],
  });
  return NextResponse.json({ workflowId: handle.workflowId });
}

Auth, gate, start the workflow, return the id. That 16k-token document generation that would blow right past the function’s duration budget? It never runs in the function. The function kicks it off and gets out of the way.

The Reuse Win: No Second Codebase

This is the part that made me stop worrying. The worker is not a new codebase. It imports the same lib/ code the app already runs.

The activities (the actual units of work Temporal executes) call into modules that already exist or are already planned. safeFetch from lib/security/ssrf.ts for any link crawling, so the SSRF guards apply exactly as they do today. detectLinks from lib/messages/link-detector.ts. The post-status.ts modules under lib/bluesky/ and lib/mastodon/ for cross-post fan-out. The DSL parser and validator in lib/lists/dsl-*.ts for anything list-shaped. And the AI writer library from the companion genai plan, lib/ai/, which doesn’t exist yet but will be wrapped as activities rather than rewritten.

Zero business-logic duplication. That matters more than it sounds. It means the IDOR checks and subscription gating I already trust stay exactly where they live. A worker activity that writes a list row runs through the same where: { id, userId } ownership filter as the web route. I’m not maintaining two versions of “can this user do this” and praying they stay in sync. There’s one version, and the worker just calls it.

What the Foundation Actually Contains

Before any feature ships, there’s a foundation phase, the groundwork everything else in the plan stands on. It’s the worker, the service, the client wiring, and then a small set of cost-control primitives that every later proposal reuses:

An llmCall activity with a model-cascade policy (cheap model first, escalate only when the work demands it) and per-target max_tokens ceilings: roughly 1k for a message, 4k for a list, 16k for a document.
An assertBudget activity that reads spend before it lets a call through, so a runaway job throws instead of burning credits.
A response cache keyed by hash(system + prompt + model), so identical requests never re-bill.
Task queues with rate limits tuned to whatever the provider or the user’s BYO key allows.

Net-new dependencies are modest: @temporalio/client on the Vercel side, the @temporalio/worker / workflow / activity trio on the worker host, and one LLM SDK. That’s the whole shopping list. (AI assistance in the product is still Coming Soon. This is me brainstorming the machine that would power it, not shipping it.)

Scaffold the Unfamiliar, Own the Verification

I’ve been doing a lot of this alongside Claude lately, and one lesson keeps repeating.

A durable-execution worker is new territory for me. When something’s unfamiliar the instinct is either to avoid it or to throw the whole thing at the model and hope. Both are wrong. What works is to spec the seams and delegate the boilerplate.

I give Claude the hard constraints, not vibes. “The app is serverless on Vercel and stays that way. Reuse this lib/ code, here are the exact modules. Here are the auth and gating seams. The worker imports, it does not reimplement.” With those rails, Claude is great at generating the worker scaffold and the client wiring, the boilerplate I’d get subtly wrong on the first pass because the framework is new to me.

But I do not let it hand me generated infrastructure and call it done. I own correctness, and for a durable workflow, correctness has a specific proof: a Temporal replay test using TestWorkflowEnvironment that simulates a worker crash mid-workflow and asserts the workflow resumes deterministically, picking up from the last completed activity instead of re-running (and re-billing) the ones already done.

That test does the real work. It’s how “a retry doesn’t re-pay for the LLM call” stops being a claim in a blog post and turns into something I’ve watched go green in CI. The model writes the code I don’t know how to write yet, and I write the test that proves it’s right.

With the foundation standing and provably resumable, Part 3 turns it into the first real feature: a Generate button that survives a timeout and asks before it writes a single thing.

Adron brainstorming and working on InterlinedList.