Integrating Claude and GPT-4 APIs in Next.js Applications
How to integrate the Anthropic Claude and OpenAI APIs into Next.js applications — covering streaming responses, token counting, prompt caching, rate limit handling, and cost control patterns.
By POINTNEXIS Team
Adding LLM capabilities to a Next.js application involves more than calling an API and returning text. Production integrations need streaming for perceived responsiveness, token budget management to control costs, graceful error handling for rate limits and overloads, and caching to avoid redundant API calls.
This guide covers the patterns that matter for production Claude and GPT-4 integrations in Next.js.
Streaming Responses with Route Handlers
Users experience LLM latency viscerally when waiting for a complete response. Streaming sends tokens as they generate, reducing perceived wait time from 5-10 seconds to near-instant first-token latency. Both the Anthropic and OpenAI SDKs support streaming.
In Next.js, create a Route Handler (`app/api/chat/route.ts`) that returns a `ReadableStream`. Use the AI SDK from Vercel (`ai` package) for a thin abstraction that handles both Claude and GPT-4 streaming with `useChat` client-side hook — it manages message state, error handling, and streaming out of the box.
Prompt Caching to Reduce Costs
Claude's prompt caching feature (available on Claude Sonnet and Haiku) caches prefix portions of your prompt, charging only for cache reads on subsequent calls. For applications with a large system prompt or retrieved context that stays constant across requests, caching reduces input token costs by up to 90%.
Mark cache-eligible portions with a `cache_control: { type: 'ephemeral' }` breakpoint in the messages array. System prompts, long instructions, and large document chunks retrieved from your knowledge base are the highest-value cache targets.
Rate Limit Handling and Retry Logic
Both APIs return 429 status codes on rate limit and 529 on overload. Implement exponential backoff with jitter: retry after 1s, 2s, 4s with ±20% random variance to prevent thundering herd from multiple simultaneous retries.
For user-facing features, surface informative loading states rather than blank screens during retries. Queue non-urgent requests (background document processing, batch embeddings) to smooth out peak load rather than hitting limits during real-time interactions.
Token Counting and Cost Control
Use the `countTokens` API endpoint (Anthropic) or `tiktoken` library (OpenAI) before sending large prompts to verify you are within context limits. Set `max_tokens` on every call — never leave it at the default, which may return more tokens than your use case needs.
POINTNEXIS AI integrations include a per-user daily token budget tracked in Redis. When a user approaches their limit, the UI surfaces a friendly message. This prevents runaway costs from scripted abuse or pathological inputs while keeping the experience transparent.