Many LLM tasks don't need real-time responses: classifying a backlog of documents, generating descriptions for a catalogue, evaluating a test set. Batch APIs suit these jobs.
How Batch APIs Work
Submit many requests at once as a file or list. The provider processes them asynchronously, typically within hours, and you collect results when done.
Benefits
- Lower prices than real-time requests with many providers.
- Higher throughput without managing rate limits.
- Simpler code for large jobs.
Good Uses
- Bulk classification, extraction and summarisation.
- Running evaluation suites.
- Generating synthetic data.
- Periodic reprocessing of content.
Practical Tips
- Include a unique ID with each request to match results.
- Validate outputs and retry failures.
- Test prompts on a small sample before submitting large batches.
- Combine with prompt caching where supported.
When Not to Use
Interactive applications and time-sensitive processing need real-time APIs.
Cost Control
Estimate token usage before submitting, and set budgets.