Library · topic
Evaluation
Agents fail differently from chat models: trajectories, not single outputs, and nondeterminism that hides in pass@1. These notes cover eval pitfalls, repeated-trial metrics, retrieval-vs-generation attribution, and the latency/cost dimensions that decide whether an agent is usable, not just correct.
- Deep diveverified 2026-08-23
Groundedness: measuring and detecting agent hallucination
Groundedness measures whether each claim in an agent's output is entailed by the source material the agent actually saw, and it is computed by decompo…
- Deep diveverified 2026-07-31
Token budgets change agent capability
UK AI Security Institute evaluations found that increasing an agent's token budget from 1 million to 10 million tokens improved success on software-en…
- Deep diveverified 2026-07-29
Cost control and token economics
Measure cost per completed task, not cost per model call, because cheap models can become expensive when they cause retries or long trajectories. Redu…
- Deep diveverified 2026-07-29
Offline and online agent evaluation
Offline evals provide repeatable pre-release regression tests over representative tasks, adversarial cases, tool failures, and multi-step trajectories…
- Deep diveverified 2026-07-29
Chunking, hybrid retrieval, and reranking
Chunk along semantic boundaries and keep headings, source URL, timestamps, permissions, and neighboring relationships; chunks should be small enough t…
- Deep diveverified 2026-07-28
Agent evaluation pitfalls
Agent evaluations should separate model quality from scaffold, tool, and environment failures. Single-run pass rates hide nondeterminism, while overly…
- Deep diveverified 2026-07-28
Latency and streaming agent results
Optimize both time to first useful feedback and total task time: stream text or structured progress, acknowledge long-running work immediately, and ex…
- Deep diveverified 2026-07-28
Retrieval-augmented generation basics
Retrieval-augmented generation embeds a question, retrieves semantically similar passages from a knowledge base, and places those passages in the mode…
- Deep diveverified 2026-07-14
Determinism and reproducibility
Lower temperature can reduce sampling variation for extraction and routing, but it does not make a hosted model or a multi-step agent deterministic be…
- Deep diveverified 2026-07-14
Production e-commerce agents
E-commerce agents should combine lexical and semantic product retrieval with hard filters for inventory, locale, compatibility, price, and fulfillment…
- Deep diveverified 2026-07-14
Choosing models, prompting, RAG, and fine-tuning
Choose the smallest model that meets measured quality, tool-use, context, latency, modality, privacy, and reliability requirements on your own task di…