Model Comparison

You are an ML engineer evaluating AI model outputs. The user wants to compare responses from multiple AI models side-by-side to assess quality, latency, cost, and correctness.

What to check first

• Install required clients: pip install openai anthropic google-generativeai cohere
• Have API keys ready for each model provider (OpenAI, Anthropic, Google, Cohere)
• Verify your input prompt is deterministic (avoid time-sensitive or random elements)

Steps

1. Define your test prompt and expected evaluation criteria (accuracy, relevance, toxicity, latency)
2. Initialize client objects for each model using their respective SDKs with authentication
3. Make parallel API calls to each model with identical parameters (temperature, max_tokens)
4. Capture response text, token usage, latency, and cost per request
5. Parse and normalize outputs into a structured comparison format
6. Score each response using metrics (BLEU, semantic similarity, custom rubrics)
7. Generate a comparison table or JSON report showing winner by category
8. Log individual model performance for trend analysis across multiple runs

Code

import time
import json
from typing import Dict, List
from dataclasses import dataclass, asdict
import anthropic
import openai
from datetime import datetime

@dataclass
class ModelResponse:
    model_name: str
    response_text: str
    latency_ms: float
    prompt_tokens: int
    completion_tokens: int
    cost_usd: float
    timestamp: str

class ModelComparator:
    def __init__(self):
        self.client_anthropic = anthropic.Anthropic()
        self.client_openai = openai.OpenAI()
        self.pricing = {
            "claude-3-5-sonnet-20241022": {"input": 0.003, "output": 0.015},
            "gpt-4o-mini": {"input": 0.00015, "output": 0.0006},
        }
        self.results: List[ModelResponse] = []

    def query_claude(self, prompt: str, max_tokens: int = 1024) -> ModelResponse:
        start = time.time()
        message = self.client_anthropic.messages.create(
            model="claude-3-5-sonnet-20241022",
            max_tokens=max_tokens,
            messages=[{"role": "user", "content": prompt}]
        )
        latency = (time.time() - start) * 1000
        
        cost = (
            (message.usage.input_tokens * self.pricing["claude-3-5-sonnet-20241022"]["input"]) +
            (message.usage.output_tokens * self.pricing["claude-3-5-sonnet-20241022"]["output"])
        )
        
        return ModelResponse(
            model_name="claude-3-5-sonnet-20241022",
            response_text=

Note: this example was truncated in the source. See the GitHub repo for the latest full version.

Common Pitfalls

• Forgetting to handle rate limits — Anthropic returns 429 errors that need exponential backoff
• Hardcoding the model name in 50 places — use a single config so you can swap models in one place
• Not setting a timeout on API calls — a hanging request can lock your worker indefinitely
• Logging API responses with sensitive data — PII can end up in your logs without realizing
• Treating the API as deterministic — same prompt, different output. Test on multiple runs

When NOT to Use This Skill

• For deterministic tasks where regex or rule-based code would work — LLMs add cost and latency for no benefit
• When you need 100% accuracy on a known schema — use structured output APIs or fine-tuning instead
• For real-time low-latency applications under 100ms — even the fastest LLM is too slow

How to Verify It Worked

• Test with malformed inputs, empty strings, and edge cases — APIs often behave differently than docs suggest
• Verify your error handling on all 4xx and 5xx responses — most code only handles the happy path
• Run a load test with 10x your expected traffic — rate limits hit fast
• Check token usage matches your estimate — surprises here become surprises on your bill

Production Considerations

• Set a daily spend cap on your Anthropic console — prevents runaway costs from bugs or attacks
• Use prompt caching for static parts of your prompts — can cut costs by 50-90%
• Stream responses for any user-facing output — perceived latency drops by 70%
• Have a fallback model ready — if Claude is down, you should be able to swap to a backup with one config change