AI Cost Optimization

AI cost foundations

• Core meters (input, output, cached, batch, and reasoning tokens)
• Secondary meters (tool use, images/audio/video, embeddings)
• Purchasing models (pay-as-you-go, provisioned throughput, self-hosted)
• Economics of LLM inference
• Latency metrics like TTFT and inter-token latency
• Tokenomics in agentic systems
• Cost amplifiers: tool schemas, RAG documents, multi-agent fan-out, etc.

Cost observability

• Context growth and failure modes like context rot
• Traces and spans for LLM applications
• OpenTelemetry GenAI semantic conventions
• Content vs. metadata logging (privacy, PII, retention)
• Sampling strategies
• Attribution hierarchy
• Unit economics such as cost per conversation or task

Gateways and where to optimize

• LLM gateway / LLM mesh like routing and budgets
• Gateway landscape (LiteLLM, Portkey, Kong, Cloudflare, OpenRouter) and trade-offs
• Observability stacks (Langfuse, LangSmith, Helicone, Arize Phoenix, W&B Weave, Braintrust)
• Build vs. buy
• Baseline diagnostics (top drivers, cache hit rate, waste rate)
• When not to optimize
• Optimization stack overview, ranked by ROI

Assignments: Instrument an LLM Gateway

1. Build the mesh

• Build a gateway for a multi-tenant workload
• Emit telemetry for every call (tokens, latency, attribution)
• Connect an observability stack (Langfuse)

2. Diagnose and optimize

• Produce a cost-diagnosis report
• Ship one optimization with a measured before/after

Caching

• Exact-match response caching
• Semantic caching (embed → ANN search → threshold → serve or miss)
• Threshold tuning for cache hits
• Failure taxonomy
• Hardening like scoped keys and verifier models
• Provider prompt caching
• Stable-to-volatile ordering of input prompt
• Cache-busting anti-patterns

Context compression

• Compaction vs. caching
• Agentic compaction (/compact-style)
• Tool-result pruning
• Attention sinks and window management
• Hard compression and soft compression
• KV-cache eviction and quantization
• Context OS

Distillation

• Prompt and context distillation
• Model distillation (soft labels, sequence-level KD, SFT on teacher outputs)
• Limitations and losses

Assignments: Practice Token Efficiency

1. Semantic cache

• Build a semantic cache and sweep similarity thresholds
• Harden with a policy gate and a verifier model
• Build a prefix-cache simulator and detect cache-busting anti-patterns

2. Compaction

• Compare compaction strategies with a needle-recall test
• Measure cost vs. recall per strategy

3. Distillation

• Distill a big-prompt router into a lightweight student
• Test staleness after a policy change and retrain

Model and reasoning effort economics

• Reasoning effort as a cost lever
• Capability overlap and difficulty prediction
• Model complementarity (RouterBench, RouterArena)
• Cost–quality–latency trade-offs

Model routing

• Model cascades (FrugalGPT)
• Expected-cost formulation
• Routing frameworks (WHEN × SIGNAL × COMPUTE)
• Predictive vs. non-predictive routing, hybrid systems
• Ecosystem (RouteLLM, LiteLLM, OpenRouter, Azure Model Router, Bedrock, etc)
• RouteLLM deep dive (router architectures, data augmentation, calibration, serving)
• Deterministic and vector-based routing
• Agent-step routing

Routing evals

• Metrics (PGR, APGR)
• Offline certification (golden sets, replay matrix)
• Programmatic checks and LLM-as-judge labeling
• Online monitoring (routing distribution, shadow scoring, canary, interleaving)

Assignment: Build a Router

• Build a model router from scratch
• Run it against a simulated model pool
• Build evals and certify the router

Multi-agent loops

• Decomposition (context isolation, parallelism, specialization)
• Loop topologies (orchestrator, evaluator, peer debate, blackboard)
• Cost models for multi-agent systems

Runaway loops and state hygiene

• Runaway taxonomy (retry spirals, ping-pong delegation, oscillation, goal drift, amplification)
• Detection (counters, state hashing, semantic repetition, progress metrics, watchdog agents)
• Context rot (attention dilution, behavioral contamination)
• State cleansing (truncation, retention policies, compaction, external memory, fresh restarts)
• Error hygiene (digests, deduplication, failures → constraints)

Planning, speed, and systems thinking

• Semantic convergence (embedding similarity, edit distance, evaluator score)
• Convergence vs. stagnation
• Planning (ReAct vs. plan-then-execute, tree search, pruning, replanning)
• Speculative decoding (draft–verify, acceptance math, agent-level speculation)
• Co-design (tool, model, prompt, eval, infra)

Capstone: Rescue a Research Agent

• Redesign a failing deep research agent
• Apply the full optimization toolkit end to end