AI Model Consumption Lab: Compare Tokens, Time and Cost

What this comparison can and cannot answer

Most model comparisons collapse several questions into one ranking. The useful starting point is more precise: what would this exact workload cost under each provider's published standard API rate? The interactive lab answers that calculation. It does not know how many tokens a real model will consume, how quickly it will answer, or whether it will succeed on your task. Those require controlled, repeated measurements.

This article compares API model IDs, not subscription products. Availability varies by account and region. The model registry includes GPT-6 Astra, GPT-6.1 Sol, GPT-6 Luna, GPT-5.6 Sol, Claude Fable/Opus/Sonnet/Haiku 5.x, Gemini 4 Argon and Gemini 3.8 Flash only where the vendor publishes the ID and price. It does not silently substitute another model.

Token consumption, in plain language

A token is a piece of text as represented by a model's tokenizer. Input tokens include the material sent to the API; output tokens are generated by the model. A cached input token may be billed at a lower rate if a provider actually reports a cache hit. Different tokenizers can split identical text differently, so equal word counts do not imply equal token counts.

A context window limits how much material a request can consider; an output limit caps generation. A model's reasoning or thinking tokens may be billed even when their text is not visible. Never estimate hidden reasoning by subtracting a hand-counted visible answer from a provider total unless its usage schema explicitly supports that interpretation.

Time to first token is often used loosely. A rigorous streaming test records time to the first user-visible content, not the first status or tool event. Total response time runs from request start to completion. Output throughput is visible generated tokens divided by the appropriate generation interval when both measurements exist. None is an energy or carbon metric; this page measures API usage and time, not electricity.

Two APIs with the same $/million input and output rates can have different real cost per successful task: tokenization, output length, hidden reasoning, retries and caching differ. For a decision, divide total spend by the number of outputs that pass an agreed rubric, not just by requests attempted.

A reproducible same-prompt experiment

The benchmark specification in this repository contains ten public, original tasks: factual recall with a reference, long-form explanation, code generation, code debugging, logic, structured extraction, business email, summarization, mathematics and formatting compliance. Each task records an exact prompt, prompt ID/version, evaluation criteria and any checkable reference. No private customer data or paid API calls are involved in this release.

1. Freeze the exact prompt text and version before choosing models. Keep the reference answer and scoring rubric separate from generated responses.
2. Record the provider, exact API model ID or snapshot, region, service tier, reasoning effort, system instructions, temperature where supported and streaming configuration.
3. Run at least three trials per model and task, alternating model order to reduce time-of-day effects. Record errors and retries, not only successful runs.
4. Capture first visible streamed content, completion time, input/output usage, reasoning/cache usage where exposed and any actual billing metadata.
5. Score correctness, required content and formatting against the task rubric. Record reviewer and rationale; where practical, review outputs without model names.
6. Publish medians and ranges, not a single fastest run. Disclose settings that cannot be made equivalent across providers.

The local importer accepts a versioned JSON or CSV schema and requires three completed trials of one prompt version and model configuration before showing medians. Importing a dataset does not make it independently verified. A score requires a named reviewer, explicit rubric and trial-specific rationale; user-supplied measurements remain labeled as such. The local dry-run harness validates the prompt manifest and fixture shape without contacting any provider.

Which model should you test first?

Use the cost lab to choose a small shortlist, then run your own task. For simple routing or extraction, begin with the lower-list-price candidates and check whether they satisfy the rubric. For coding, research or high-stakes decisions, test the task-specific failure modes before buying more expensive capacity. A list-price tie is not proof of equivalent output quality.

For an enterprise deployment, add data handling, access control, auditability, deployment region and procurement constraints. For high-volume automation, count retries and human correction time. A model with a low per-token price but frequent failed outputs may cost more per accepted result. There is no responsible overall winner without comparable success and usage evidence.

Important limitations and pricing conditions

• API results are not directly comparable with ChatGPT, Claude or Gemini subscription experiences. Product routing, tools and quota rules can differ.
• The calculator is a text-only, standard paid API scenario. It excludes tools, images/audio/video, taxes, regional uplifts, priority/batch tiers and provider-specific cache-write/storage charges.
• Google's Gemini 4 Argon and 3.8 Flash standard prices are explicitly time-limited in the official price list. GPT-5.6 Sol also has a promotional period. Reverify before using these rates for a later budget.
• Claude Haiku 5.5 changes price above 100,000 prompt tokens, so this calculator caps input at that threshold. Large-context workloads need a separate tier-aware estimate.
• Even with the same prompt, system messages, tokenizer behavior, reasoning settings, network path and hardware allocation vary. A trial should disclose those differences.
• Vendor comparative labels such as “fast” or “slower” are not cross-vendor latency benchmarks. This page withholds speed and quality rankings until real matched trials are available.