Part 7 of 8: Durable AI for InterlinedList

Six posts in, and I’ve been dancing around a number. Every proposal I’ve sketched (durable generate, the calendar planner, the link crawler, batch enrichment) carries a cost, and the expensive part is never Temporal. It’s the tokens. A worker box and a managed Temporal namespace run me on the order of tens of dollars a month (obviously at this time), flat, whether they process ten jobs or ten thousand. The model bill is the one that scales with usage. It’s also the one that decides whether an AI feature is worth shipping or is a slow-motion budget fire.

So this post is the payoff. Durability isn’t just nice, durability is money. The mechanics I keep reaching for aren’t in the plan because they’re clever. They’re there because each one structurally drives the token bill down, and does it in a way I can measure instead of squint at and hope for. Keeping this cheap relative to the Claude and ChatGPT APIs was never a footnote I tacked onto the series. It’s been the point the whole way through.

None of this is live yet. AI assistance in the product is still marked Coming Soon. This is me brainstorming the cost layer I’d build, and how I’d hand it to Claude to build with me.

The Single Biggest Saver Is Not Re-Paying for Work You Already Did

Start with the one that matters most. Durable memoization.

When a Temporal workflow runs an activity (say, an llmCall that generates a document section) the result gets persisted the moment it completes. If the workflow later fails and retries, it does not re-run that activity. It replays the history, sees the completed result, and moves on.

Think about what that means on a six-step generation pipeline that dies at step five. Without durability, a retry starts over: steps one through four re-invoke the model, and you pay for all four again just to get back to where you already were. With Temporal, the retry re-runs step five and only step five. Steps one through four are already-billed calls that never happen twice.

That’s the difference between retries as a cost multiplier and retries that are free. On a batch enrichment job that fails at item 47 of 50, you re-bill item 47, not items 1 through 46. I wrote a whole post about that in Part 6, and it’s the same mechanic every proposal here leans on. Memoization is why a flaky provider or a worker restart doesn’t torch the budget.

Cheapest Model First, Escalate Only on Demand

The next lever is refusing to reach for the expensive tier by default. The llmCall activity would run a model cascade: cheapest capable tier first (Haiku-class, Flash-class), escalating only when the task actually needs it.

Most of what these features do is mechanical. Tagging a message. Summarizing a row. Extracting a field. That work does not need a frontier model, and paying frontier rates for it is just lighting money on fire. So the cascade defaults to cheap, and it reserves the Opus-class tier for the thing the user explicitly asked for: long-form synthesis, a research draft, a document they told me they want written well.

Paired with that: per-target max_tokens ceilings, so a runaway generation can’t balloon. A message caps around 1k, a list around 4k, a document around 16k. The ceiling isn’t a polite suggestion the model is vaguely aware of. The activity enforces it as a hard cap. So you get the cheap tier by default, a small ceiling per target, and the expensive model only shows up when you asked for it.

A Gate That Throws Before It Spends

Cascades and ceilings keep individual calls cheap. But the scenario that scares me is unbounded spend: a runaway agent loop, or a leaked user key someone else is now happily burning. So the plan puts a budget gate before every call, not after.

An assertBudget activity reads accumulated spend from the AiGeneration ledger and throws a non-retryable BudgetExceeded when the user is over their ceiling. Non-retryable is the important word. A normal failure retries; this one halts the workflow cold, because retrying a budget breach just tries again to spend money you’ve already said no to.

Roughly what I’d hand to Claude as the shape:

// activity: runs on the worker, before any billable llmCall
export async function assertBudget(userId: string, estimate: TokenEstimate): Promise<void> {
  const spent = await getSpendThisPeriod(userId); // counts only, from AiGeneration
  const remaining = budgetFor(userId) - spent;

  if (estimate.maxTokens > remaining) {
    // non-retryable: a budget breach must not loop and re-attempt the spend
    throw ApplicationFailure.nonRetryable(
      `Budget exceeded: need ${estimate.maxTokens}, have ${remaining}`,
      "BudgetExceeded",
    );
  }
}

A workflow calls assertBudget and then llmCall, in that order, every time. Worst case is a bounded overspend of one call, never an open tap.

Never Bill the Same Request Twice

Then there’s the plain-dumb-obvious saver: caching. A response cache keyed by hash(system + prompt + model). If an identical request comes through (same system prompt, same user prompt, same model) it’s served from cache and never re-billed. Regenerate the same summary twice, pay once.

Same idea one layer down for link work: URL-hash dedup. When the crawler from Part 5 fetches and embeds a link, it keys on a hash of the URL. A link shared across ten messages, three docs, and a list row gets fetched, extracted, and embedded exactly once, ever. Embeddings are already orders of magnitude cheaper than generation, so computing them once and caching forever makes the cheapest part of the pipeline round down to nothing.

And to keep from wasting tokens on retries you caused yourself: worker concurrency and task-queue rate caps tuned to stay under provider limits. Blow past a rate limit and you get a 429 storm, and every retry in that storm is wall-clock and, on some providers, tokens down the drain. Obeying the limit is cheaper than fighting it.

One more, because it’s the most satisfying: confirm-before-spend. For the prompt-to-list flow, the workflow generates the schema, then pauses on a Signal and waits. It does not spend a single row-generation token until the user confirms the schema is right. Get the schema wrong and you’ve spent one cheap schema call, not fifty row calls against a shape nobody wanted. That pause is impossible in a serverless handler and native in Temporal.

The Ledger Counts, It Doesn’t Read

Every one of these levers needs a source of truth, and that’s the AiGeneration ledger. Critical design decision: it records counts and status only. Tokens in, tokens out, which model, succeeded or failed. It never stores the prompt text or the model output.

That’s not laziness. One table does three jobs. Privacy, because I’m not warehousing what people wrote or what the model said back. The quota counter, because assertBudget reads its sums off it. And the cost dashboard, because those same rows drive a per-user “AI usage and spend” view: tokens, calls, cache-hit rate, budget remaining. When something looks expensive, the ledger doubles as the debugging trail. You can see that a workflow made forty calls without ever seeing what it said.

Build the Guardrails With Claude, Then Make It Prove Them

Now the part I’ve been wanting to get to, because this is where working with an LLM changed how I’d approach any of it.

Memoization and caching only save money if they’re in the right places. Put a cache breakpoint one step too early and you re-bill everything after it. Miss a memoization boundary and a retry quietly re-runs a paid call. The savings are entirely a function of where the boundaries land, and eyeballing a six-step pipeline for those boundaries is the kind of thing I get wrong.

So the first move is a diagnostic question, not a code request. I’d point Claude at the actual workflow and ask: “walk this pipeline and tell me exactly where a retry would re-bill the model.” Then let its answer drive placement. It reads the activity boundaries, traces the retry path, and tells me which calls are already durably memoized and which ones a mid-pipeline failure would re-invoke. That answer is the map. The caching and memoization go where the map says, not where I guessed.

The second move matters more: make it prove the savings. Memoization you can’t prove is just a hope with good intentions. So I’d have Claude write a Temporal replay test (using TestWorkflowEnvironment) that fails an activity partway through the pipeline and then asserts the already-completed LLM activities are not invoked again on retry.

it("does not re-bill completed LLM activities on retry", async () => {
  const llmCall = vi.fn()
    .mockResolvedValueOnce("step-1 result")
    .mockResolvedValueOnce("step-2 result")
    .mockRejectedValueOnce(new Error("provider blip at step 3")) // fail mid-pipeline
    .mockResolvedValue("step-3 result (retry)");

  await worker.runUntil(client.workflow.execute(generatePipeline, { /* ... */ }));

  // steps 1 and 2 completed before the failure, so Temporal replays their
  // memoized results and MUST NOT call the model for them again.
  const stepsCalled = llmCall.mock.calls.map((c) => c[0].step);
  expect(stepsCalled.filter((s) => s === 1)).toHaveLength(1);
  expect(stepsCalled.filter((s) => s === 2)).toHaveLength(1);
  expect(stepsCalled.filter((s) => s === 3)).toHaveLength(2); // failed once, retried
});

That test is the proof. It fails loudly the day someone refactors the pipeline in a way that breaks memoization and starts silently re-billing steps one and two. The savings stop being a story I tell about the architecture and start being a property CI enforces. Same loop for the response cache: a test that fires two identical requests and asserts the second one hits cache and never reaches the model.

Ask the agent where the money leaks. Let its answer place the guardrails. Then make it write the test that locks them in. That loop works because a cost regression is invisible until the bill shows up weeks later, and a test drags it into the light on the exact commit that caused it, while you can still git blame your way back to it.

Where This Lands

Part 8 is the capstone: the agentic research-to-draft flow, the single most token-hungry thing in the whole set. It’s the reason every governor in this post has to exist before it ships. I’ll also lay out how I’d sequence the entire build, starting from the foundation phase (the Temporal wiring and the shared cost primitives everything else reuses) and working all the way up to the agent, and how I’d verify each piece as it lands.

Adron brainstorming and working on InterlinedList.

Part 5 of 8: Durable AI for InterlinedList

I have a problem, and I bet you have it too. I read something good, I drop the link into a post or a document or a row in some list, and then three weeks later I want it back and I have no idea where it went. Was it a message? A doc? A row in that reading-list I built? So I scroll my own feed like an archaeologist. It’s a slow-moving disaster. The whole point of writing something down is being able to get it back, and I’d built the writing-down part beautifully while ignoring the getting-it-back part entirely.

So this is the post where I brainstorm fixing that. Two pieces come from the Temporal plan, and they run in order. First, the link-crawl-and-cache phase turns saved links into a real cache. Then the retrieval phase makes that cache searchable, semantic and keyword, with cited snippets, so “where did I save that article about X” actually returns the article about X. None of this is shipped. AI assistance in the product is still “Coming Soon.” This is me thinking out loud with the intent to build.

What Actually Exists Today (Less Than You’d Hope)

Before I design anything, I made myself write down what the codebase does right now, because designing against a fantasy version of your own product is how you build the wrong thing.

Today, link metadata gets fetched for messages only. When you post something with a URL, lib/messages/metadata-fetcher.ts pulls the Open Graph tags and the result gets stored inline on the message as a JSONB blob: Message.linkMetadata. That’s it. There’s no cache table. There’s no dedup, so if you and forty other people all post the same great article, that URL gets fetched forty-one times. Documents and list rows (both Subscriber features) don’t get scanned for links at all. And search? Search today is Postgres ILIKE substring matching across three separate routes: messages, lists, documents. It’s a contains query. Type “kubernetes” and you get rows where the literal string “kubernetes” appears. Miss the spelling, miss the synonym, miss the concept, and you get nothing. There is zero vector infrastructure anywhere in the repo. I checked.

That’s the honest starting line. It’s also a decent starting line, because two of the hard parts are already solved and I’d be a fool to rewrite them.

The Crawler: Reuse the Fetcher, Reuse the Guard

So, the caching step. A durable workflow scans a user’s messages, documents, and list rows for URLs, fetches each one, extracts the readable content, and stores it in a new LinkContent cache table keyed by a hash of the URL. Deduplicated across all your content, and optionally across all users. It runs incrementally: only new or changed URLs get fetched, so the second run is nearly free.

The building blocks already exist. lib/messages/link-detector.ts gives me extractUrls and detectLinks. lib/messages/metadata-fetcher.ts gives me fetchLinkMetadata. And critically, lib/security/ssrf.ts gives me safeFetch and assertUrlAllowed, the SSRF guard the codebase already fought to get right. This work is mostly generalizing the message-only path to also cover docs and list rows, then bolting a real cache behind it so a URL is fetched once and never again.

This is the single most important thing to say to Claude when you hand it this task, so I’ll say it here first: build on safeFetch and assertUrlAllowed. Do not write a new fetch().

I cannot overstate this. Left to its own devices, an LLM asked to “write a crawler that fetches URLs and extracts content” will cheerfully write you a fresh fetch(url) with a nice timeout and some error handling, and it’ll look completely reasonable, and it’ll have reintroduced the server-side request forgery hole this codebase already closed. It won’t check whether the URL resolves to 169.254.169.254 and hands back your cloud metadata. It won’t block the redirect that bounces from a public host to localhost. The existing safeFetch does all of that, and it re-validates on every redirect hop. A brand-new fetcher is a security regression wearing the costume of a feature.

So the instruction isn’t “write a crawler.” The instruction is:

// Crawler activity: the reuse contract, spelled out for the agent.
// 1. detectLinks/extractUrls from lib/messages/link-detector.ts to find the URLs
// 2. hash each URL, check the LinkContent cache, skip if already fetched & unchanged
// 3. fetch ONLY through safeFetch from lib/security/ssrf.ts (never a raw fetch)
// 4. extract readable text mechanically (HTML to text), NO LLM in this path
// 5. upsert into LinkContent keyed by urlHash; dedup means fetch-once-ever
async function crawlUserLinks(userId: string) {
  const urls = await collectUrlsFromMessagesDocsAndRows(userId); // reuse extractUrls
  for (const url of dedupeByHash(urls)) {
    if (await cacheHasFresh(url)) continue;         // incremental: skip known URLs
    const html = await safeFetch(url);              // SSRF guard, NOT a new fetch()
    await upsertLinkContent(url, extractReadableText(html));
  }
}

Note what’s not in there: a language model. HTML-to-readable-text is mechanical. You strip the tags, you pull the article body, you’re done, no tokens spent. An optional cheap-model summary of each page could sit on top later, opt-in, for people who want it, but the base extraction costs nothing per link beyond the fetch. Pair that with URL-hash dedup (a shared link is fetched once, ever, across the whole system) and this becomes the cheapest proposal in the whole plan for what you get out of it. Which is why I like it. It’s mostly plumbing, mostly reuse, and it barely touches an LLM.

Retrieval: The Part That Needs New Ground

The crawl phase gives me the cache. The search-over-my-links phase is what makes it findable. This is “search over my link index and my own docs, lists, and messages, returning cited snippets.”

And I have to be honest that this one needs real new infrastructure, because the vector story doesn’t exist yet: not partially, not disabled behind a flag, not anywhere. So the retrieval phase needs a pgvector extension, an embedding column or table, an embed activity that turns text into vectors, and a search route that ranks by both vector similarity and the existing ILIKE match. Hybrid. It supplements today’s substring search rather than ripping it out. Keyword matching is still the right answer for exact-string lookups, and it needs no model at all.

The cost math here is the reassuring part. Embeddings are orders of magnitude cheaper than text generation: you’re paying to turn text into a vector, not to write prose. You compute each embedding once and cache it, sharing the same cache the crawl phase already built. And because retrieval comes first, any eventual answer-synthesis step runs over a small, already-narrowed set of retrieved snippets instead of dumping your whole corpus into a prompt. Keep the expensive context small and this stays affordable.

The incremental pipeline is what keeps it fresh without a nightly full re-scan: on every content write (new post, edited doc, added row) enqueue a re-embed activity for just that thing. Batch the embeddings for throughput. The index stays current, and you never re-embed the 99% of your corpus that didn’t change.

The Second Lesson: Don’t Let the Agent Touch the Database Directly

The pgvector migration is where the other working-with-Claude lesson lives, and it’s a governance one.

This repo has a strict database workflow, and it’s strict for a reason the CLAUDE.md states plainly: violating it has broken production before. Additive-only. Idempotent SQL. Hand-written migration files applied to both databases. No prisma db push, no prisma migrate dev, no raw DDL fired at the database from a script.

An LLM does not know this in its bones. Ask it to “add a vector column and an embeddings table” and its instinct (a perfectly good instinct in most repos) is to edit the schema and run prisma db push to sync it. In this repo that’s a loaded gun. So the instruction to the agent isn’t just “add the schema.” It’s: route this through the db-migrations workflow. Write an idempotent, additive migration file (CREATE EXTENSION IF NOT EXISTS vector, CREATE TABLE IF NOT EXISTS, the column adds guarded) and do not run db push or migrate dev against the database. The good news is the migration is cleanly additive: there’s no existing vector infra to alter or drop, so it’s all new tables and columns, which is exactly the shape this workflow handles best.

Two guardrails, then, and I’d hand both to the agent up front rather than catch them in review: reuse safeFetch instead of writing a fetcher, and route the schema change through the migration workflow instead of letting it push. Both are cases where the model’s default behavior looks reasonable and is wrong for this codebase. Tell it about the thing that already exists, and tell it not to reinvent it.

The payoff, if I build it, is the thing I wanted in the first place. I type “that article about durable execution I saved somewhere,” and I get it back with the snippet and a link to wherever I stashed it, whether that was a throwaway post or a row buried in a list. My scattered links stop being a graveyard I dig through and turn into something I can actually ask questions of. Subscriber-gated, computed once and cached, and cheap enough that I’d leave it running without watching the bill.

Next up in Part 6, I want to point this same durable-fan-out machinery at bulk work: tag every untagged message, add twenty rows matching a criterion, backfill previews across everything, and do it without re-billing the model for the forty-six items that already succeeded before item forty-seven fell over.

Adron brainstorming and working on InterlinedList.

Part 4 of 8: Durable AI for InterlinedList

Most of this series is me thinking out loud about features that don’t exist yet. This post is different as the bug is real. It’s in the code today. 😬

Scheduled publishing is a Subscriber feature, and so is the cross-posting it rides on. You queue a post, set a time, and a cron fans it out to Bluesky, Mastodon, LinkedIn, and X when the moment arrives. Both sit behind the subscription, so this bug hits paying subscribers. The machinery that runs it has a hole in it I don’t love.

The Cron That Can Post Twice

Walk through what the code does. app/api/cron/publish-scheduled-messages/route.ts runs every minute. It selects every message where scheduledAt <= now, then loops over them. For each one it fans out to whatever platforms the user configured, collects the resulting URLs, and then, as the last step, updates the message row to null out scheduledAt so the message is no longer “due.”

Read that ordering again. The cross-post happens first. The “mark it done” write happens second. No lock. No idempotency key. No transaction spanning the fan-out and the update. Those two operations are just sequential, and hopeful.

So picture the fan-out succeeding. Your post lands on Bluesky and Mastodon. Great. Now the message.update that’s supposed to clear scheduledAt hits a reaped Neon connection, or a transient network blip, or any of the ten boring things that make a database write fail once in a while. The update throws. The message row still has scheduledAt <= now.

One minute later the cron runs again. The message is still due. It fans out again. Same post, second time, to every platform. That’s your double-post.

The code isn’t naive about this: there’s a withPrismaRetry wrapper on that critical update and a comment that literally says “if this fails the message stays due and is re-published.” Someone saw the cliff and put up a guardrail. But a retry is a mitigation, not a fix. It narrows the window without closing it. If the retries exhaust, or the process dies between the fan-out and the update, you’re back on the cliff. And the fan-out itself is best-effort per platform (LinkedIn can fail while Bluesky succeeds, with no compensation), so a re-run doesn’t even cleanly re-do the same thing. It re-does some of it.

This is a slow-moving disaster. It doesn’t fire on every post. It fires on the unlucky ones, occasionally, in production, and it makes your platform look like it can’t be trusted to post your thing exactly once.

The Working-With-Claude Lesson: Point at the Bug Class, Not the Bug

If you take one thing from this post, take this part, not the fix. It’s worth more.

If I open a session and say “fix the double-post in publish-scheduled-messages,” I get a patch that treats the symptom. Maybe it wraps more things in retries. Maybe it adds a published boolean and checks it. Narrow prompt, narrow fix, and I’ve probably left three other copies of the same hazard sitting in the codebase untouched.

So I don’t open it that way. I say: “Audit this scheduled-publish path for concurrency and idempotency hazards.”

That framing is doing real work. It doesn’t presume I already know what’s wrong. It asks the agent to reason about the class of failure: what happens under a partial failure, what happens if two invocations overlap, what’s the ordering of side effects versus state writes, where’s the operation that isn’t idempotent. The double-publish race falls out of that audit as one instance of a general pattern (side effect before commit, no dedup key). And the audit usually surfaces siblings I hadn’t thought to ask about.

This is the two-step I keep coming back to with Claude on anything gnarly. Audit for the class, then delegate the conversion. First get the agent to characterize the whole shape of the problem. Then, once you both understand it, hand over the mechanical rewrite. Collapse those into one “fix it” prompt and you’ll get a bandage every time.

Most of the value is in the audit. The conversion is mostly typing, and Claude is very good at the typing once you’ve done the thinking with it.

The Fix: A Schedule, a Workflow ID, and a Saga

So this is the conversion I’d delegate. It’s a proposal. None of it is built yet.

Replace the every-minute cron with a Temporal Schedule that starts one workflow per due message. The whole trick is what you use for the workflow ID.

// Proposed: the message id IS the workflow id.
// Temporal refuses to start a second workflow with an id that's
// already running or already completed. That's the dedup.
await client.workflow.start(publishScheduledMessage, {
  workflowId: `publish-message:${message.id}`,
  taskQueue: "publishing",
  args: [{ messageId: message.id }],
});

Temporal deduplicates by workflow ID. If a workflow with publish-message:abc123 has already run (or is running right now), a second start with that same ID is rejected or returns the existing handle, depending on the reuse policy you pick. The message cannot be published twice, because you cannot start its publish workflow twice. The race doesn’t get narrower. It closes. There’s no second racer allowed on the track, so there’s nothing left to race.

That’s a different guarantee than “we retried hard enough that it usually works.” One is a hope with good odds. The other is impossible by construction, and I’ll take impossible.

Then model the fan-out as a saga instead of a hopeful loop. One activity per platform: postToBluesky, postToMastodon, postToLinkedIn, postToTwitter, the exact post-status.ts functions the cron already calls, plus the same splitTextForPlatform and resolveLinkedInTarget logic. Each activity retries independently with backoff, and each one’s success is durably recorded before the next runs. If LinkedIn fails after Bluesky succeeded, the workflow knows Bluesky is done and won’t re-do it. It records the half-failure coherently and runs compensation, instead of silently re-posting the whole set on a blind re-run.

And the thing that makes this an easy call: publishing uses zero LLM. No model in this loop at all. No tokens, no inference cost, nothing to govern with a budget. It’s pure durability engineering, a straight win against a present-day production hazard with no ongoing cost attached. If anything in this series jumps the queue, it’s this.

The Calendar I Actually Want

Once publishing is durable, the thing I’ve wanted for ages becomes buildable. This part is squarely future, gated behind AI assistance that’s still marked Coming Soon.

I want to type “give me a month of posts about distributed systems, three a week” and get a filled calendar back. Not a wizard. Not twelve separate button clicks. A month.

The cheap way to build that leans on batching. One LLM call produces all N drafts at once, and batching many items into a single prompt is dramatically cheaper than N separate calls (the cost thesis that runs through this whole series). The workflow takes those drafts, schedules them across the month, and fills the gaps. Then, to keep it cheap, you can reject or regenerate an individual slot. Don’t like Thursday’s post? Regenerate just that one. Only the rejected slot re-bills. You never repay for the twelve you kept.

The same shape covers scheduled digests: a Temporal Schedule that drafts you a weekly summary doc from the week’s messages. But put a cheap guard before the model runs. Did this user post anything this week? If not, skip the LLM entirely. You never pay to generate an empty digest for a quiet week. The guard is a database count. The model is the expensive part. You gate the expensive part behind the cheap check, which is the move I keep making all through this series: spend tokens last, and only when there’s something worth spending them on.

The calendar is the feature I daydream about. The double-publish fix is the one I’d ship first, because it’s a real hole in production right now and closing it costs nothing but the work.

Part 5 goes hunting through everything you’ve saved: the links buried in your messages, docs, and list rows, and how I’d crawl and cache them so “where did I save that article?” finally has an answer.

Adron brainstorming and working on InterlinedList.