Interactive technical brief
The Architecture of Agent Optimization
Memory, retrieval, model, harness: four layers decide what an agent can do and what it costs.
GoodMem is the memory layer, and it fine-tunes its own retrieval models automatically. PAIR Systems, the team behind it, engineers custom LLMs and harnesses when stock parts aren’t enough.
PAIR Systems engineering · July 2026
Preparing the interactive architecture…
Benchmark · Memory
Memory On vs Memory Off
A controlled, paired A/B on a production enterprise BI agent: 17 matched scenarios, three replicates each, identical model and harness, with GoodMem memory as the only variable. Memory also cut agent steps 23%, with no measurable change in answer quality, and heavy retrieval tasks saved 30–66%.
Confidence intervals are nonparametric bootstrap estimates, 10,000 resamples, clustered by scenario. Pilot-scale benchmark; broader multi-tenant validation is in progress.
Benchmark · Retrieval models
Cloud Fine-Tuner on CRUMB
Mean reciprocal rank on eight public CRUMB retrieval tasks, before and after automated fine-tuning. The benchmark tests the embedder; the fine-tuner also tunes rerankers.
| Retrieval task | Base MRR | Fine-tuned | Delta |
|---|---|---|---|
| Clinical trial matchingclinical_trial | 0.633 | 0.746 | |
| Code solution retrievalcode_retrieval | 0.389 | 0.621 | |
| State-specific legal statuteslegal_qa | 0.232 | 0.284 | |
| Scientific paper retrievalpaper_retrieval | 0.464 | 0.514 | |
| Set-based entity retrievalset_operation | 0.198 | 0.256 | |
| Reasoning-heavy community QAstack_exchange | 0.338 | 0.481 | |
| Mathematical theorem retrievaltheorem_retrieval | 0.236 | 0.330 | |
| Vague movie / TV retrievaltip_of_the_tongue | 0.067 | 0.139 |
Eight public CRUMB retrieval tasks, mean reciprocal rank of the base embedder versus the fine-tuner output. Fully automated, with no per-task human tuning. CRUMB was introduced by Killingback and Zamani. Read the CRUMB paper
Benchmark · LLM
BI Harness LLM vs Frontier Models
A 20-scenario production BI agent suite spanning Oracle EBS workloads: accounts payable, sales, inventory, and multi-step drill-downs. Three replicates per scenario, memory off, identical harness, independent LLM judge. The only variable is the model.
| Model | Median tokens | Tool calls | Pass rate | Cost per 1,000 conversations |
|---|---|---|---|---|
| Claude Opus 4.8 (medium thinking)Proprietary frontier model, API only, cannot be self-hosted | 233k | 6 | $2,380$1,780–$3,080 | |
| PAIR Systems BI Harness LLMPAIRCustom-trained for each customer, becomes the customer’s IP. Self-hosts on 2×H100 | 219k | 14 | $67$55–$80 | |
| GLM-5.2744B MoE, 40B active, roughly 16×H100 to self-host at FP8 | 342k | 11 | $260$205–$320 | |
| GPT-OSS-120B117B MoE, 5.1B active, self-hosts on 1×H100; the customer’s prior on-premises harness model | 182k | 8 | $10$8–$12 |
Pass rates and 95% confidence intervals are nonparametric bootstrap estimates (10,000 resamples) over scored replicates (57 to 68 per model). Cost per 1,000 conversations is measured per-conversation token usage priced at posted July 2026 provider rates: Anthropic list for Opus 4.8 ($5 input / $25 output per million tokens), OpenRouter for GLM-5.2 ($0.56/$1.76) and GPT-OSS-120B ($0.036/$0.18), and prevailing open-weight market rates for the PAIR model’s size class ($0.14/$1.00) as its serving proxy. Prompt caching is not modeled. Cross-check: even in the most conservative fully-serial self-hosting scenario (2×H100 at Lambda on-demand rates, $2.99 per GPU-hour), the PAIR model costs $367 per 1,000 conversations, still roughly six times cheaper than Opus 4.8. GPU footprints reflect serving precision as well as parameter count: GPT-OSS-120B ships natively 4-bit quantized (MXFP4), engineered by OpenAI to fit a single 80GB GPU, while the PAIR model’s tested serving configuration is 2×H100.
Engineering note · Harness
Harness Choice vs Model Choice
Zhang and coauthors ran three frontier models through three harness configurations, nine combinations in all. Changing the harness moved agent scores 7.8 times more than changing the model.
PAIR Systems engineers custom harnesses for GoodMem customers: context assembly, tool use, task decomposition, and memory policy.
Variance across the grid: 18.48 percentage points squared from harness configuration, 2.37 from model choice. The 7.8× ratio is what this benchmark measured, not a general constant. Read “Stop Comparing LLM Agents Without Disclosing the Harness”
From architecture to production
Start with memory. Optimize the system above it.
Deploy GoodMem as the context layer, or work with PAIR Systems on the retrieval models, LLMs, and harnesses above it.