noun · API feature
Batch API
also called batch processing
The slow lane. You hand over a big pile of requests to be done later, usually at a lower cost, instead of one answer right now.
An interface for submitting many model requests as one asynchronous job, trading immediate responses for lower cost or higher throughput.
You submit many requests as one job and the provider processes them when capacity allows, returning results within a stated window rather than instantly. Because nothing needs a live answer, the provider can schedule the work cheaply, and each request succeeds or fails on its own.
Real-life example
You need a summary of fifty thousand old support tickets, and nobody is waiting on the result. Sending them one at a time would hit your rate limit and take all day. You submit them as a batch, go home, and collect the results the next morning.
Where the analogy breaks
-
Batch results are not instant. You get them within a window that can be hours, so it does not suit anything a person is waiting on.
-
Each request in the batch can fail on its own. Check the results line by line and resubmit the ones that errored.
-
It is for volume, not speed. Savings and limits vary by provider, so confirm the details before relying on them.