The launch post worked. A hundred strangers tried the app. Then the provider dashboard showed a bill built for a much larger company.

Do not begin by swapping models or deleting logs. First stop the loss and preserve the evidence.

Cost is production behavior. Trace it with the same care as an error.

An AI cost spike may come from genuine demand, anonymous abuse, a leaked key, repeated retries, a background queue, an agent loop, an oversized context, or a feature whose price never matched its work. Each cause needs a different repair.

Contain the blast radius

If spend is still rising, reduce the system to a safe, useful shape.

Depending on the product, that may mean:

  • disable the expensive route for anonymous users;
  • lower per-user and per-project quotas;
  • pause a background worker or queue consumer;
  • cap concurrency and new job creation;
  • shorten output and tool-call limits;
  • route low-risk work to a bounded fallback;
  • require verified accounts or payment for further use;
  • revoke a key that may have escaped;
  • show a truthful temporary message instead of retrying silently.

Do not take down unrelated parts of the product if one generation route is responsible. Do not leave a harmful money, data, or side-effect path active merely to preserve the launch.

Record the time each containment action begins. You will need it to see whether the cost curve changed.

A provider budget alert may notify you after a threshold. It may not stop requests at that threshold. Know which controls are informational and which are enforced. The application needs its own hard boundaries.

Preserve the evidence before optimizing

Capture the smallest useful incident record:

Cost window: When did spend depart from normal? Provider and project: Which account received the usage? Model or service: What generated the charge? Request count: How many attempts occurred? Input and output units: What volume was processed? Application route: Which feature initiated it? User or tenant: Who was allowed to create the work? Job result: How many useful outputs completed? Retries and tool calls: How much repeated work occurred? Deployment: What version was running? Request IDs: Which provider calls match application traces?

Keep secrets and private prompt content out of the incident note. Store identifiers that let an authorized person retrieve the detail later.

Export or screenshot provider usage before changing project structure. Save relevant logs within the product's retention and privacy rules. Optimization without evidence can lower today's bill while preserving tomorrow's bug.

Reconcile the bill with one product action

Take one successful customer job and follow it from button to result.

Count every billable operation:

  1. initial model request;
  2. embeddings or retrieval;
  3. file conversion or transcription;
  4. tool selection;
  5. each external tool call;
  6. model continuation after the tool result;
  7. validation or critique call;
  8. repair attempt;
  9. image, audio, or document generation;
  10. storage, bandwidth, and background processing.

Then trace a failed job. Failures can cost more than successes when the application retries the expensive step and never reaches a result.

Calculate:

cost per completed customer job =
all variable cost for the cohort / useful jobs delivered

Do not use cost per API call as the business measure. One customer job may need ten sensible calls. Another may generate a hundred calls and no value.

a16z's discussion of AI startup forecasting and pricing notes that foundation-model usage introduces variable cost on every call and that consumption can be uneven across customers. The practical consequence is simple: measure economics at the user and job level, not only on the provider invoice.

Check for a leaked credential

An API key must never be shipped to the browser, mobile bundle, public repository, analytics payload, error page, or shared screenshot.

Search the built client assets and repository history. Inspect provider usage by project, key, IP, model, and time when those dimensions are available. Look for requests that do not match your application shape, region, user traffic, or deployment window.

If exposure is plausible:

  1. revoke the credential;
  2. create a replacement scoped to the correct project and permissions;
  3. store it only in the server-side secret system;
  4. redeploy the consumers;
  5. confirm the old key fails;
  6. review logs and repository history for the source;
  7. contact the provider about abusive usage when appropriate.

Do not merely delete the key from the latest commit. A copied secret remains copied.

Generated code often makes a browser request directly to a model provider because that path is easy to demo. Put a server boundary between the user and the paid service. Authentication, authorization, quota, validation, and cost attribution belong there.

Find duplicate work

The most expensive request may be the same request performed twice.

Check for:

  • a button that remains active after submission;
  • a client retry plus a server retry;
  • a queue message delivered more than once;
  • a timeout that hides a successful upstream request;
  • a webhook redelivered without idempotency;
  • a page refresh that creates a new job;
  • two workers claiming the same record;
  • an effect that runs again after rendering;
  • a failed status poll that starts new work;
  • a user opening two tabs during a slow response.

Assign one durable idempotency key to the customer action. Store the job state before calling the expensive provider. Repeated requests should return or attach to the existing job when that is safe, not create another one.

Retries need a reason, a maximum, and backoff. Retry transient connection failures when the operation can be made safe. Do not retry a validation error, exhausted quota, or unknown side effect until you know what happened.

Put an end on every agent loop

An agent that can call tools has more than one cost dimension. It can reason, call a tool, read the result, reason again, and repeat.

Set explicit ceilings for:

  • model turns;
  • tool calls;
  • repeated use of the same tool;
  • total input and output units;
  • elapsed time;
  • external requests;
  • files or records processed;
  • money or side effects authorized;
  • retries after invalid output.

Stop with a useful partial result when possible. Tell the user which limit was reached and what was preserved.

Detect cycles such as searching for the same phrase, opening the same page, or attempting the same failing mutation. A model instruction that says “do not loop” is not an enforced limit. The orchestrating code must count.

Review the AI-generated code guide to identify other consequential paths where generated implementation may look complete without enforcing the product's invariant.

Bound the input before paying for it

The user's text box may contain a paragraph during testing and an entire repository after launch.

Limit input by the unit that drives cost:

  • characters or tokens;
  • file count and size;
  • document pages;
  • audio or video duration;
  • image count and resolution;
  • retrieved records;
  • conversation history;
  • tool-result size.

Validate before upload or inference where possible. Reject unsupported work with a precise message. Do not accept a giant file, convert it, extract it, embed it, and then reveal that the model cannot process the result.

Trim context deliberately. Separate instructions, relevant customer material, retrieved evidence, conversation state, and tool output. Measure how much each section contributes. Sending the whole history on every turn is easy code and expensive memory.

Preserve the pieces needed for correctness. A shorter prompt that omits a safety or customer constraint is not an optimization.

Cap the output and the repair path

An unconstrained output limit asks the provider to decide when your budget stops.

Set the smallest maximum that fits the result. Use structured output when the job has a known shape. Stop streaming when the user cancels if the provider and runtime support cancellation, while understanding that already-generated work may still be billed.

Validation can improve quality, but recursive repair can multiply cost:

generate -> reject -> regenerate everything -> reject -> regenerate everything

Prefer a bounded path:

generate -> validate -> repair the specific defect once -> return or fail clearly

Track how often repair is needed. If every request needs a second full generation, the second call is part of normal unit cost, not an exception.

Attribute usage before enforcing limits

A global daily limit protects the company by letting one user exhaust the product for everybody.

Attribute expensive work to a stable principal:

  • authenticated user;
  • organization or tenant;
  • API key;
  • paid plan;
  • job type;
  • anonymous device or network as a weaker fallback.

Then combine boundaries:

  • short burst limit;
  • sustained per-user quota;
  • tenant allowance;
  • route-specific cost ceiling;
  • concurrency limit;
  • global emergency brake.

Return a meaningful status and retry time. Do not make rate-limited clients retry immediately and create more traffic.

Cloudflare's Workers rate-limiting documentation supports limits keyed by users and routes, while also warning that its counters are permissive and eventually consistent rather than an exact accounting ledger. The distinction applies broadly: traffic control and billing truth may need different systems.

Queue work that does not need to be immediate

Concurrency turns a popular launch into a cost and reliability spike.

Move long jobs into a queue when the user does not need a synchronous answer. Set a maximum number of consumers. Reject or defer new work when the backlog exceeds the service you can afford. Show the user a real state: queued, running, complete, failed, or cancelled.

Do not poll by repeatedly performing the work. Poll job state.

Give jobs an expiry and a cancellation path. If a deployment leaves incompatible work in the queue, know whether to migrate, drain, retry, or discard it. A queue makes load visible; it does not decide what the company can afford.

Cache only when identity and freshness allow it

Repeated identical work can sometimes reuse a result. Safe candidates include public, deterministic, non-sensitive requests where the inputs, model version, instructions, and relevant sources are part of the cache key.

Do not cache private customer responses into a shared key. Do not serve an old compliance, price, status, or security answer after its source changes. Do not let a cache hide that the underlying workflow is broken.

Cloudflare's guide to controlling AI costs combines caching, per-user or per-key rate limits, and provider-level analytics. Those are useful mechanisms after the product defines which requests are equivalent and safe to reuse.

Route models from evidence, not price alone

Use the least expensive model that passes the job's quality and safety threshold.

Build a small evaluation set from real, permitted production cases:

  • ordinary inputs;
  • difficult inputs;
  • known failures;
  • adversarial or malformed inputs;
  • cases where a wrong answer carries consequence.

Measure correctness, completeness, latency, refusal behavior, and cost per completed job. Test the whole workflow, not a model response in isolation.

A smaller model may handle classification, extraction, formatting, or routing while a stronger model handles the few cases that require it. But cascading through three weak attempts before calling the strong model can cost more and feel worse than one correct request.

Keep a rollback. Provider prices and model behavior change. The cheapest passing route today is a maintained decision, not architecture carved in stone.

Make pricing carry the real work

If one paid customer can create more variable cost than their payment, the launch incident exposed a pricing problem as well as a technical one.

Match the offer to a value-bearing unit customers can understand: completed reports, processed minutes, active projects, successful checks, or included credits. You may still charge a subscription, but place honest usage boundaries around expensive work.

Avoid invented units that conceal the actual limit. Explain what consumes allowance, when it resets, what happens at the boundary, and whether overage is possible.

The free trial versus freemium guide helps decide whether expensive usage belongs in a bounded trial, a durable smaller plan, or a paid pilot. Free access should reduce buying uncertainty without granting an unlimited claim on the company's provider account.

Build a cost launch test

Before the next public push, run these cases:

  1. One ordinary user completes one job.
  2. The same action is submitted twice.
  3. The browser disconnects during generation.
  4. The provider times out after receiving the request.
  5. A queue message is delivered twice.
  6. An agent calls the same failing tool repeatedly.
  7. A user submits the largest permitted input.
  8. An anonymous user reaches the expensive route.
  9. One tenant exhausts its allowance.
  10. The global budget boundary is reached.
  11. The provider returns quota or rate-limit errors.
  12. The product enters and leaves degraded mode.

For each case, record requests made, variable cost, user-visible result, stored job state, retry behavior, and alert received.

Add this to the production checklist for an AI-built app. A successful button click is incomplete evidence if nobody knows what it cost or how many times it ran.

Keep the weekly unit-cost note

Write one page:

Customer job: Useful completion: Variable services involved: Median cost per completion: Worst legitimate cost: Failed-job cost: Free-user cost: Revenue or value unit: Top cost route: Top user or tenant concentration: Retry and duplicate rate: Manual work per completion: Enforced limits: Alert thresholds: One optimization under evaluation:

An unexpected bill is frightening because it turns invisible software behavior into cash.

Make the behavior visible. Stop the repeated work. Bound every user, job, loop, input, output, and queue. Then improve the unit that still produces customer value.

The bill is not merely an infrastructure problem. It is the product telling you what one promise costs.