The Problem: API Counts Hide the Real Cost

Most teams track LLM costs this way:

response = openai.ChatCompletion.create(
  model="gpt-4",
  messages=[...]
)
# Cost = 1 API call

This is backwards. The actual cost depends on how many tokens your prompt and completion use, not how many times you call the API.

For example:

• A short summary request (prompt: 500 tokens, completion: 100 tokens) costs $0.003 with GPT-4
• A long document analysis (prompt: 5,000 tokens, completion: 1,000 tokens) costs $0.030 with GPT-4

That's a 10x difference for just 2 API calls. If you're only counting "API calls," you're flying blind.

Token Pricing Across Providers

Different providers price tokens differently, and the ratios matter:

Model	Prompt (per 1M tokens)	Completion (per 1M tokens)
GPT-3.5 Turbo	$0.50	$1.50
GPT-4	$30.00	$60.00
Claude 3 Opus	$15.00	$75.00
Gemini Pro	$0.50	$1.50

Key insight: Completion tokens cost 2–5x more than prompt tokens. If your app generates long completions, you're paying a premium.

Building a Token Cost Model

Here's the framework:

Step 1: Instrument Your Code

Capture prompt and completion tokens from every API call:

response = openai.ChatCompletion.create(...)
prompt_tokens = response['usage']['prompt_tokens']
completion_tokens = response['usage']['completion_tokens']

# Calculate cost
model_pricing = {
  'gpt-4': {'prompt': 0.03/1000, 'completion': 0.06/1000}
}
cost = (prompt_tokens * model_pricing['gpt-4']['prompt'] +
        completion_tokens * model_pricing['gpt-4']['completion'])

Step 2: Attribute to Business Context

Log tokens with business metadata:

• User ID / customer ID
• Feature / use case (summarization, classification, generation)
• Model used
• Timestamp
• Cost

This lets you answer: "How much do we spend on customer support chat?"

Step 3: Analyze Token Efficiency

Calculate key metrics:

• Tokens per transaction: Are your prompts getting too long?
• Completion ratio: Completion tokens / prompt tokens (higher = more expensive output)
• Cost per feature: Summarization vs. classification vs. generation

Optimization Strategies

1. Prompt Engineering

Fewer tokens in your prompt = lower cost. Example:

• Verbose prompt: "Please analyze the following customer feedback and tell me what the customer is saying about our product quality..." (45 tokens)
• Optimized prompt: "Analyze feedback: sentiment and product quality issues" (10 tokens)

This 4.5x reduction compounds across millions of requests.

2. Model Selection

Use cheaper models where accuracy allows:

• GPT-3.5 Turbo for simple classification
• Claude 3 Haiku for parsing and extraction
• GPT-4 only when you really need reasoning

3. Caching Strategies

If you're hitting the same LLM with similar prompts (like system instructions), cache the responses. With MetaFinOps, you can identify repeated patterns and implement caching.

4. Batch Processing

Process multiple requests together to reduce overhead tokens.

The MetaFinOps Advantage

MetaFinOps handles token cost modeling automatically:

• Provider-agnostic tracking: OpenAI, Anthropic, Google, Cohere, Mistral, self-hosted
• Per-customer attribution: See which customers drive LLM costs
• Real-time alerts: Flag unexpectedly high token counts
• Optimization recommendations: "Your summaries use 2x more completion tokens than industry average"
• Audit trails: Full transparency for compliance and cost allocation

The Bottom Line

Token cost modeling is not optional for AI-native companies. Without it, you're making decisions with incomplete information. A single poorly-optimized feature can cost $50K+/month.

Start capturing token-level costs today. The savings will pay for the infrastructure in weeks.

Related Articles