request sent →
first output token received
Inference API performance · TMLS 2026
Application-level levers for faster inference.
Speed is a product feature
The performance partnership
The inference provider builds the engine.
Your AI application defines the workload.
Performance is what happens when they meet.
Measure what matters · 1
TTFT measures request sent to first output token. Generation speed measures output tokens per second. API end-to-end measures request sent to the last response token. Application end-to-end measures user action to final usable result.
request sent →
first output token received
rate at which
output tokens are produced
request sent →
last response token received
user action →
final usable result rendered
Measure what matters · 2
A request moves from the application through connectivity, admission, orchestration, prompt processing, and decode. Generated tokens then return to the application.
Measure what matters · 3
The application prepares a request that crosses connectivity, admission, orchestration, prompt processing, and decode. The first returned token reveals an illustrative TTFT of 272 milliseconds. Repeated tokens animate a generation-speed gauge between 800 and 1000 tokens per second before settling at 900. The last response token reveals an illustrative API end-to-end latency of 2.3 seconds. One second later, application validation, processing, and rendering reveal an illustrative application end-to-end latency of 3.5 seconds.
Performance numbers are illustrative
Measure what matters · 4
The p50 marker represents a typical request. The p95 marker represents the slow tail, and the p99 marker represents the extreme tail.
Monitor both typical and tail performance. Both matter!
The playbook
01 Route intelligently
Broader model capability often carries a speed and cost premium.
Five models appear in order from lower to higher Artificial Analysis Intelligence Index. A fitted curve shows the typical tradeoff in this snapshot: higher-quality models generate more slowly.
Artificial Analysis model comparison · methodology · snapshot: June 13, 2026
01 Route intelligently
Repeat the decision for every task and every model call.
The same five-model benchmark chart now includes an example task quality bar at index 45. The model and fitted-curve segment below the bar are dimmed. DeepSeek V4 Flash is highlighted as the fastest model in this snapshot that clears the example bar.
Artificial Analysis model comparison · methodology · snapshot: June 13, 2026
01 Route intelligently
When both models qualify, the faster API transforms the user experience.
In an illustrative task where both models meet the quality bar, ten sequential calls each use ten thousand input tokens, generate five hundred output tokens, and include time to first token. Gpt-oss-120b on Cerebras, with 1.62 seconds time to first token and 1,823 tokens per second, takes about nineteen seconds total. GPT-5.5, with 75.1 seconds time to first token and 60 tokens per second, takes about thirteen minutes and fifty-four seconds total. The same agentic task completes about forty-four times faster on the Cerebras route.
gpt-oss-120b provider data · GPT-5.5 data · Artificial Analysis snapshot: June 13, 2026 · scenario assumptions are illustrative
02 Shorten the wire
Connection setup is costly.
In an illustrative request with a 40 millisecond round-trip time, a DNS cache miss takes 20 milliseconds, the TCP handshake takes 40 milliseconds, and full TLS 1.3 negotiation takes 40 milliseconds before the variable request upload.
02 Shorten the wire
Create one client. Reuse its connection pool.
The first request pays for DNS, TCP, TLS, and request upload. It then shifts into the blue transport palette but remains visible. A later request appears below and reuses the established connection, so it can begin with request upload. Developers should create one long-lived client per process, avoid recreating it for every call, and prefer HTTP/2 when supported.
Install: pip install "httpx[http2]" openai
from openai import OpenAI, DefaultHttpxClientllm = OpenAI(http_client=DefaultHttpxClient(http2=True))02 Shorten the wire
Same prompt. Far fewer bytes. Lower latency.
A Cerebras benchmark uses a code-review request with approximately 30 thousand prompt tokens, up to 1,024 output tokens, and Llama 3.1 8B. MessagePack binary encoding followed by gzip reduces the request from 123.4 kilobytes to 2.0 kilobytes. TTFT p50 and p90 improve from 0.63 and 0.70 seconds to 0.43 and 0.49 seconds. API E2E p50 and p90 improve from 0.76 and 0.82 seconds to 0.58 and 0.63 seconds.
…-token code review · max … · … · Benchmarked on Cerebras Inference
import gzip, msgpack
encoded = msgpack.packb(payload)
body = gzip.compress(encoded)
headers = {
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/vnd.msgpack",
"Content-Encoding": "gzip",
}
response = client.post(
CEREBRAS_CHAT_COMPLETIONS_URL,
content=body, headers=headers,
)03 Send less; reuse more
Context grows every turn. Prompt processing grows with it. Curate continuously.
The slide is divided evenly between accumulating context on the left and curating context on the right. Three successive API requests add files, tools, tool outputs, and messages to the accumulating context without removing anything. A conceptual chart below adds one higher time-to-first-token point with each request. The right side remains hidden until the presenter advances. It then shows time to first token rising modestly as context grows and falling after each continuous curation pass. The final state labels unbounded accumulation as the approach to avoid and continuous curation as the preferred approach.
As context grows, so does TTFT
Curate continuously to keep TTFT in check.
03 Send less; reuse more
Stable prefix. Less prompt processing. Lower TTFT.
A first request processes and stores a stable prompt prefix in a provider-managed cache. A later request with the same prefix reuses that work, processes only its variable tail, and has lower time to first token. A request with an early mismatch must process the prefix again and has higher time to first token. On presenter advance, three practices explain how to structure stable prefixes, keep repeated prefixes identical, and monitor cache hit rate from cached token usage.
04 Optimize generation
Generation time is total generated tokens divided by generation speed. After the equation fades away, a small draft model proposes several candidate tokens autoregressively. The target model verifies all candidates in one parallel pass, accepts the matching prefix, and rejects the first mismatch. Longer accepted runs increase generation speed.
A small, fast model guesses ahead. The target model remains authoritative.
Matching prefix tokens are acceptedAt the first rejection, target model takes over.
Higher accepted tokens per step→higher gen speed
Up to 4× speedup (varies by algorithm, workload, and hardware)
The provider handles it. Your API stays the same.
04 Optimize generation
Lower temperature can increase draft-target agreement and acceptance length, increasing generation speed where assisted decoding is supported.
Make the prediction more predictable. Make the draft model succeed more often.
temperature is a sampling parameter supported by most API providers.
It rescales next-token probabilities:
Temperature 0.0 boosted gen speed by …
Unless you need the model to be “creative”,lower temperature is likely to improve gen speed!
04 Optimize generation
The prediction parameter lets the caller provide expected response content. It works best when most of the response is already known and only a small portion changes.
Faster responses when most of the output is already known.
prediction reduces latency by specifying parts of the response that are already known.
instructions = "Change the button color to blue."
html_code = ...
response = client.chat.completions.create(
model="gpt-oss-120b",
messages=[
{"role": "user", "content": instructions},
{"role": "user", "content": html_code},
],
prediction={"type": "content", "content": html_code},
)Predicted Outputs applied to the right tasks can increase gen speed by3x
04 Optimize generation
A strict JSON schema gives the response a predetermined shape. An illustrative order-extraction example compares a typical raw response of approximately 75 output tokens with a structured response of approximately 45 output tokens, a reduction of approximately 40 percent. Schema enforcement also removes schema-repair retries and stabilizes the response contract across model upgrades.
Generate only the data your application needs.
response_format: json_schema constrains the response to a predefined shape, removing explanatory prose and bounding the output.
compact schema→less prose→fewer output tokens
“Extract the order details from this customer message.”
order_idcustomertotal_usd
itemsshipping_addressplaced_date
Looking at the message,
the order details are:
Order ID: 4477
Customer: Jane Smith
Total: $147.50 · Items: 3
Shipping: 742 Evergreen Terrace,
Springfield OR
Placed: March 14, 2026.40% fewer tokens generated
No schema-repair retries
Stable contract across model upgrades
04 Optimize generation
The reasoning effort argument controls how much reasoning work the model performs. Low, medium, and high effort suit progressively more reasoning-heavy tasks. A forthcoming benchmark will compare hidden reasoning-token multipliers relative to low effort, then connect more generated tokens to longer generation and API end-to-end time.
Match reasoning effort to task complexity.
reasoning_effortControls how much reasoning work the model performs before answering.
Answer mostly exists in the input.
extract · classify · route · formatSome inference, comparison, or light planning.
analyze · compare · reconcile constraintsThe answer requires difficult multi-step work.
plan · debug · prove · complex tool useHigh reasoning effort can cost … the tokens, impacting TTFT and API E2E
Use the appropriate effort for the task at hand to boost performance and reduce cost
End of talk
Choose the fastest model that clears the task’s quality bar.
Reuse connections and shrink every payload.
Curate context and preserve cacheable prefixes.
Make output predictable and bound tokens and reasoning.
You do not control the provider’s inference engine.
But, you can shape the workload to let it perform at its best.