Library · topic
Evaluation
Agents fail differently from chat models: trajectories, not single outputs, and nondeterminism that hides in pass@1. These notes cover eval pitfalls, repeated-trial metrics, retrieval-vs-generation attribution, and the latency/cost dimensions that decide whether an agent is usable, not just correct.
- Deep diveverified 2026-07-28
Agent evaluation pitfalls
Agent evaluations should separate model quality from scaffold, tool, and environment failures. Single-run pass rates hide nondeterminism, while overly…
- Deep diveverified 2026-07-28
Latency and streaming agent results
Optimize both time to first useful feedback and total task time: stream text or structured progress, acknowledge long-running work immediately, and ex…
- Deep diveverified 2026-07-28
Retrieval-augmented generation basics
Retrieval-augmented generation embeds a question, retrieves semantically similar passages from a knowledge base, and places those passages in the mode…
- Field noteverified 2026-07-14
Cost control and token economics
Measure cost per completed task, not cost per model call, because cheap models can become expensive when they cause retries or long trajectories. Redu…
- Field noteverified 2026-07-14
Determinism and reproducibility
Lower temperature can reduce sampling variation for extraction and routing, but it does not make a hosted model or a multi-step agent deterministic be…
- Field noteverified 2026-07-14
Production e-commerce agents
E-commerce agents should combine lexical and semantic product retrieval with hard filters for inventory, locale, compatibility, price, and fulfillment…
- Field noteverified 2026-07-14
Offline and online agent evaluation
Offline evals provide repeatable pre-release regression tests over representative tasks, adversarial cases, tool failures, and multi-step trajectories…
- Field noteverified 2026-07-14
Choosing models, prompting, RAG, and fine-tuning
Choose the smallest model that meets measured quality, tool-use, context, latency, modality, privacy, and reliability requirements on your own task di…
- Field noteverified 2026-07-14
Chunking, hybrid retrieval, and reranking
Chunk along semantic boundaries and keep headings, source URL, timestamps, permissions, and neighboring relationships; chunks should be small enough t…
- Field noteverified 2026-07-14
Token budgets change agent capability
UK AI Security Institute evaluations found that increasing an agent's token budget from 1 million to 10 million tokens improved success on software-en…