How to Choose an AI Model for Krispy Live Chat

Krispy supports three AI providers out of the box: Cloudflare Workers AI (free, no API key required), OpenAI, and Anthropic. You switch between them with two environment variables. For most projects, Workers AI is the right starting point — upgrade to OpenAI or Anthropic only when you hit a concrete limitation.

Buttr doing the model research so you don't have to.

Does Krispy have a free built-in AI option?

Yes — Cloudflare Workers AI is the default. Because Krispy is designed to deploy on Cloudflare Workers, the AI binding runs inside the same runtime. No external API call, no separate bill, no key to rotate.

Configure it in two places:

# wrangler.toml
[ai]
binding = "AI"

AI_PROVIDER=workers
AI_MODEL=@cf/meta/llama-3.1-8b-instruct

Cloudflare's free tier covers 10,000 neurons per day at no cost (Cloudflare Workers AI pricing). That's enough for a support widget on most indie projects without ever paying for inference. The 5-minute Krispy setup guide uses Workers AI as the default precisely because there's no API key step.

Where Workers AI works well: latency is low (same network as your Worker), user messages stay inside Cloudflare's infrastructure, and Llama 3.1 8B handles greetings, FAQs, and triage cleanly.

Where it falls short: the model catalog is smaller than the big providers. On nuanced reasoning, very long conversations, or replies where brand voice matters, you'll start to feel the edges.

When should I switch to OpenAI?

If your team already has an OpenAI key, the switch is two lines:

AI_PROVIDER=openai
AI_MODEL=gpt-4o-mini
OPENAI_API_KEY=sk-...

The cost math: GPT-4o-mini costs $0.15 per million input tokens and $0.60 per million output tokens (OpenAI pricing). A typical chat exchange — a greeting, a few turns of Q&A, maybe a handoff message — runs 2,000–4,000 tokens total. That's under $0.001 per conversation. At 20,000 conversations a month, you're looking at $10–20 in inference.

GPT-4o is roughly 30× more expensive per token. For a live chat widget, that's rarely justified — use GPT-4o-mini unless you can point to specific failures a smarter model would fix.

Where OpenAI wins: the broadest API ecosystem. Function calling, structured outputs, vision, and the most battle-tested rate limits at scale. If you need the bot to call your own APIs mid-conversation, OpenAI's function-calling interface is the easiest to wire up.

The honest trade-off: you're dependent on their pricing and uptime. Krispy's config model means you can switch providers in an afternoon — but that's still a real migration, not zero cost.

When does Anthropic (Claude) make sense?

Claude follows detailed instructions more precisely and hedges less on things it's uncertain about. For a live chat bot representing a brand — where a hallucinated answer or an off-tone reply causes real damage — that reliability is worth paying for.

AI_PROVIDER=anthropic
AI_MODEL=claude-haiku-4-5
ANTHROPIC_API_KEY=sk-ant-...

Claude Haiku is the fastest and cheapest Claude model. Pricing is $0.80 per million input tokens and $4 per million output tokens (Anthropic pricing). At 20,000 conversations a month, expect $25–60 depending on average turn length.

Where Anthropic wins: instruction fidelity on complex prompts. If you've put real effort into your Krispy system prompt — brand voice, escalation rules, hard limits on what the bot should never say — Claude is more likely to honor all of it across a long conversation without drifting or padding.

Workers AI vs OpenAI vs Anthropic: side by side

Which AI model should I actually use?

Start with Workers AI. It's already wired into Krispy, costs nothing, and handles routine chat — greetings, FAQs, triage — cleanly. Upgrade only when you hit a concrete gap, not in anticipation of one.

If you upgrade, default to GPT-4o-mini. It's cheap, the API is mature, and most live chat use cases don't need more. Move to Claude Haiku when your system prompt is complex and you need the bot to hold detailed instructions across a long conversation.

GPT-4o and Claude Sonnet are edge cases, not defaults.

The real leverage isn't the model tier — it's a clear system prompt paired with the Telegram human handoff that catches whatever the AI misses. Let the bot handle volume; let humans handle the hard stuff.

If you'd rather skip the infra entirely, Krispy Cloud runs the same stack with a 14-day free trial, starting at $19/mo flat. If you want the control — and the ability to point the AI at exactly the model your use case needs — star the repo at github.com/krispyai/krispy and deploy it yourself. 🥐

FAQ

Can I use a different model — Gemini, Mistral, or a local Ollama model — with Krispy?

Yes. Any provider that exposes an OpenAI-compatible API works today: Mistral, Groq, Together AI, and local Ollama all qualify. Point AI_PROVIDER=openai at a custom AI_BASE_URL and you're done. Native adapters for additional providers are on the roadmap; PRs are open.

Does switching AI models mid-production break anything for active users?

No. The model config is environment-level — update AI_MODEL and redeploy. Krispy's design keeps conversation context in your Telegram thread, not inside the model, so there's no session state to migrate. In-flight conversations continue normally without any disruption to active users.

Is Workers AI reliable enough for a production chat widget?

For most projects, yes. Llama 3.1 8B handles routine chat cleanly. Where it occasionally falls short is multi-step reasoning or precise domain knowledge — which is where a well-written system prompt and the Telegram handoff matter more than the model tier. Let the AI handle volume; let humans handle the hard stuff.

Does Krispy send my users' messages to OpenAI or Anthropic?

Only if you configure one of those providers. With Workers AI, messages stay within Cloudflare's network. With OpenAI or Anthropic, messages are processed by their API under your account's data processing terms — review those agreements if you handle EU user data or have strict privacy requirements.

How do I keep AI inference costs predictable as traffic grows?

Start with Workers AI (free tier). If you upgrade, GPT-4o-mini is easy to model: roughly $0.001 per conversation, so 10,000 conversations ≈ $10. Set a hard monthly spend cap in your provider's dashboard and monitor token usage there. With Workers AI you have no inference bill to manage at all until you hit the free-tier ceiling.