Patrick Tech Media
Organizations building generative AI applications usually compare models the same way: dollars per million tokens. It’s the number on every pricing page, so it becomes the number in every spreadsheet.
What changed
It’s the number on every pricing page, so it becomes the number in every spreadsheet. But production workloads don’t buy tokens. They buy outcomes: a resolved support ticket, a completed research brief, a correct financial summary. Between the pricing page and the outcome sit multipliers the sticker price ignores: how often the model is right, how many tokens it needs to get there, and, for agentic workloads, how many turns it takes, because every turn re-sends the growing conversation.
Price and bundle value
In this post, we share results from an open-source benchmarking harness that measures those multipliers across OpenAI models on Amazon Bedrock (gpt-5. 6-luna, gpt-5. 6-terra, and gpt-5. 6-sol) and two widely used cost-efficient models on the OpenAI API (gpt-5. 4-mini and gpt-5. 4-nano). We chose the latter two as the cost-optimized baselines many teams start from, not as like-for-like generational peers, because “we run mini or nano today. Is a newer model on Amazon Bedrock worth it? ” is the question we hear most. We focus on three questions:.
AI features that change the value
The harness evaluates all five models through the same Responses API client, switching the backend and model ID while holding the evaluation logic constant. The results still reflect differences in the models, provider infrastructure, and model-specific configuration. In particular, the models on Amazon Bedrock ran with reasoning disabled, while the OpenAI API baselines ran at their defaults. This is a comparison of practical deployment configurations, not a controlled estimate of intrinsic model capability.
Who should pay attention
We measure three things: single-call accuracy and cost on benchmarks that still separate frontier models (AIME competition mathematics, GPQA Diamond graduate-level science, and MMLU-Pro), multi-turn agent trajectories on live web-research tasks, and rubric-graded professional deliverables. Grading combines deterministic checks with a large language model (LLM) judge (gpt-5. 5, not one of the evaluated models) using frozen prompts whose hashes are recorded in every result file. Each run writes a timestamped result JSON, and every number and chart in this post is generated from those files at build time.
Patrick Tech Media take
How to read the results: sample sizes range from 48–198 items. Treat small gaps as directional unless accompanied by uncertainty estimates, and reproduce the evaluation on your own workload before selecting a model. That is why the useful reading move is not to stop at the headline, but to compare the promise, the workflow change, and the likely cost before deciding anything.
Latest comments
0No comments yet. You can start the conversation.