Skip to content
Adrythm
Book 20 minutes
AI and automation

Batch API

OpenAI Batch API / Message Batches API / batch processing for AI

In short

A batch API lets software send a large group of AI model requests to be processed together within a set window, instead of waiting for each answer. OpenAI and Anthropic both charge half the normal price for batched requests, in exchange for results that can take up to 24 hours.

OpenAI's Batch API takes a file of requests and completes each batch within 24 hours, often sooner, at 50% lower cost. It also draws on a separate pool of significantly higher rate limits. For now the completion window can only be set to 24 hours.

Anthropic's Message Batches API works the same way at the same discount. A batch can hold up to 100,000 requests or 256 MB, whichever comes first. Most batches finish in less than an hour, a batch that has not finished within 24 hours expires, and results stay available for 29 days.

The trade is speed for price. Anything a customer is waiting on, such as a reply in a website chat, needs the normal request and response. Work nobody watches happen, such as summarizing yesterday's call transcripts or tagging a backlog of leads, can wait for a batch.

In practice

A clinic's software writes a summary of every recorded call overnight. Sent as ordinary requests, the nightly run costs $40 in model fees. Sent as a batch, the same run costs $20, and the summaries are normally ready well before the office opens. The figures are a worked example.

Why it matters to you

In custom software that uses AI behind the scenes, the model bill is a running cost that lasts as long as the system does. Moving work that can wait into a batch halves that part of the bill at both providers. The design question is which jobs need an answer in seconds and which can wait hours.

What to ask or check

  1. 01Which AI tasks in this system need an answer while someone waits, and which could run as a batch?
  2. 02Does the running cost estimate assume batch pricing for the jobs that can wait?
  3. 03What happens to work in a batch that expires before it finishes?

What people get wrong

That a batch is the same service with a discount. Batched requests trade speed for price: OpenAI completes each batch within 24 hours, and an Anthropic batch that has not finished within 24 hours expires.

Keep reading

API

An API is the way one piece of software talks to another without a person in the middle. MDN describes it as a contract between the application offering it and whatever uses it. The part that matters commercially is that the contract belongs to whoever publishes it, and it changes.

AI agent

An AI agent is a model that has been granted the ability to take actions, not just produce text. OWASP says the damage one can do comes from three grants: excessive functionality, excessive permissions and excessive autonomy. What it is allowed to do matters more than how good it is.

Retrieval augmented generation

Retrieval augmented generation is how an AI answers from your documents without being trained on them. OWASP describes it as combining a pre-trained model with external knowledge sources at answer time. So the documents sit in a store the system reads from, and who can read that store is the question.

Training data

Training data is what a model learned from. The question owners ask is whether their own data joins it, and there is no single answer: OWASP names three separate stages, pre-training, fine-tuning and embedding. A commitment worth having names the stage and comes in writing.

Hallucination

A hallucination is AI output that sounds right and is not. OWASP describes the model filling gaps in its training data using statistical patterns, without understanding the content, so the answer can be fluent and unfounded at once. Its own first example is an airline that was successfully sued over its chatbot.

Prompt injection

Prompt injection is when text an AI system reads becomes an instruction it follows. OWASP notes the text does not have to be visible to a person, only parsed by the model, and that no fool-proof prevention is known. The UK's national cyber security body says the same.

Want this explained against your own numbers?

Twenty minutes, a straight answer, and no follow-up sequence if you decide not to work with us.

Book 20 minutes