Short answer: use a prepaid balance rather than a card on file, set alerts well below the number that would hurt, and put a retry ceiling, a concurrency limit and a kill switch in the code. Only the first of those is a hard stop. Alerts tell you after the fact and code fails in the ways you did not anticipate, which is precisely how the overnight run happened.
The story is always roughly the same. Something ran unattended, a loop did not terminate, each iteration cost a fraction of a cent, and by morning there were four hundred dollars of usage and a provider dashboard that updates hourly. Nobody did anything reckless. The system simply had no upper bound.
How it actually happens
- Retry without a ceiling. A call fails, the wrapper retries, the failure is deterministic, and the retry runs forever. Every attempt is billed, including the ones that error.
- The context that grows. An agent loop appends each result to the conversation and resends the whole thing. Cost per step grows with the square of the number of steps, and step forty is not twice step twenty.
- Fan-out. An agent decides to process a list of items concurrently, each item spawns subtasks, and nothing limits the width of the tree.
- A crawl that found a link farm. Scraping and search APIs bill per request and the internet is large.
- A cron job that overlaps itself. A five minute job scheduled every minute is five copies running, all billing.
- Someone else using your key. Less common, more expensive, and covered in our post on rotating a leaked key.
Why you always find out too late
Usage-based billing is postpaid by design. The provider meters, aggregates and charges at the end of a period, which means the feedback loop between the mistake and the consequence can be hours or weeks long. Usage dashboards typically lag as well, sometimes by an hour, sometimes by a day.
Worse, many providers describe their spending controls as budgets or limits when what they really mean is a notification threshold. A notification is not a stop. If the run is unattended, an email at 2am is a record of the damage, not a prevention of it. Read your provider's own wording carefully: the useful question is whether exceeding the limit blocks the next request or merely tells you about it.
Three layers that actually bound the damage
Layer one: a hard financial ceiling. The only genuine stop is a balance that can be exhausted. Prepaid credits cannot overdraw, so the worst case is the amount you funded. Where a provider offers a real hard limit rather than an alert, turn it on. Where it only offers a card on file, consider a virtual card with a monthly cap as a crude substitute.
Layer two: alerts that arrive in time. Set them at fractions of your ceiling, not at your ceiling. Alerting at 75 percent of a monthly budget on the 3rd of the month tells you something is very wrong. Alerting at 100 percent tells you it already happened. Route them somewhere you will see at night, which for most people is a phone notification rather than email.
Layer three: code that gives up. This is the layer you control completely and the one most often missing:
// the four numbers every agent loop needs
const MAX_STEPS = 25; // hard stop on iterations
const MAX_RETRIES = 3; // with exponential backoff
const MAX_CONCURRENCY = 4; // width of any fan-out
const MAX_SPEND_USD = 5; // tracked per run, checked before each call
if (spent > MAX_SPEND_USD) throw new Error('run budget exhausted');Track an estimated cost per call in the run itself and abort when it crosses the ceiling. It does not have to be accurate, it has to be bounded. Add a kill switch too: an environment variable or a row in a table that every loop checks before each call, so you can stop a running job without a deploy.
Rules for anything that runs unattended
- Cap the model output. Set a maximum token count on every request. An unbounded response on an unbounded loop is the expensive combination.
- Use a smaller model for the loop. Route the cheap, repetitive steps to a cheap model and reserve the expensive one for the step that needs it.
- Prevent overlapping runs. Take a lock. A job that can start before the previous one finished will eventually do so.
- Log cost per run, not per month. If you cannot see what one run costs, you cannot notice when it changes.
- Test the failure path. Deliberately make a call fail and watch what your retry logic does. This finds the infinite loop in daylight.
If it already happened
Three things, in order. Stop the bleeding first: revoke or rotate the key the run was using, which halts every process holding it including the ones you have forgotten about. Then work out what actually spent the money by pulling the provider's usage export and grouping by hour and by model, because the story you have in your head is usually wrong by a factor of two in one direction or the other.
Then write to the provider. Support teams see accidental runaway usage constantly and many will waive or credit part of a first, obviously accidental spike, particularly if you can show what changed and what you have put in place to stop it recurring. Nothing is guaranteed, so ask politely, attach the numbers and say what you fixed. The answer is no more often for repeat incidents, which is a fair reason to do the code work before you send the email.
What this looks like on Squiid
Spend controls are the reason Squiid is prepaid rather than postpaid. Your balance is money you have already decided to spend, the gateway meters every call against it, and when the balance reaches zero the gateway stops authorising usage instead of continuing and invoicing you afterwards. Nothing is deleted and nothing is cancelled; calls simply stop being authorised until you top up. The maximum an overnight run can cost you is the balance you funded.
Alerts fire at 75, 90 and 100 percent of the balance, and auto top-up is optional with a cap you set, so it can refill a small amount without becoming an unbounded card charge. Because every service runs through one gateway, the same ceiling covers the model API, the database, the email and everything else, rather than being a setting you had to remember to configure in nine consoles. You can see the full behaviour in the spend controls documentation.
The layered advice still applies. Put the retry ceiling and the kill switch in your code regardless, because a balance that pauses at zero protects your wallet and does not protect your Tuesday.