Evaluation before automation.

Two systems, one evaluation discipline

The insurance project asked whether automated matching could replace a brittle, manually maintained mapping process. The developer-tooling project asked what information an AI coding agent should receive on every task. The subject matter differed, but the engineering question was the same: what evidence would let the team compare approaches before trusting one?

1. Evaluating automated insurance matching

The Automate Insurance Mappings proof of concept compared insurance records from Zocdoc with EHR-native carrier and plan data. I built the pieces needed to evaluate matching methods alongside the LLM prototype.

• Fetched roughly 721 Zocdoc health carriers through GraphQL.
• Extracted carrier and plan data from Modernizing Medicine through FHIR.
• Built a browser-based curation tool and created roughly 100 manually reviewed mappings.
• Split the ground-truth data by confidence level so uncertain examples stayed visible.
• Implemented Jaccard similarity as a non-LLM baseline.
• Designed MapperEval to compare classical, LLM, and hybrid matching approaches against the same ground truth.

The evaluation dataset gave the team a way to judge the proof of concept and informed SIMS, the production initiative for resolving Zocdoc insurance details to EHR-native records during booking.

2. Auditing repository instructions for coding agents

Root-level repository instructions are always-on context. Every irrelevant command, copied code sample, or stale architectural note consumes context and can change how an agent explores the codebase. I needed a repeatable way to decide what belonged in a CLAUDE.md, what should load only for relevant paths, and what deterministic tooling should enforce instead.

I synthesized public research, Anthropic's documentation, HumanLayer's practical guidance, and internal engineering patterns into an eight-part rubric.

1. IdentityState what the repository is and why it exists.
2. ConcisenessMake every always-loaded instruction justify its context cost.
3. Universal applicabilityKeep only guidance that matters across most tasks.
4. Progressive disclosureLoad specialized guidance when the relevant code is touched.
5. Pointers, not copiesPoint to authoritative code or documentation instead of duplicating it.
6. GuardrailsName the actions that are unsafe, expensive, or hard to reverse.
7. Not a linterLeave deterministic style and formatting checks to deterministic tools.
8. Removal testDelete an instruction if removing it does not change useful behavior.

What changed in the repositories

I applied the rubric to the Interop team's two primary repositories. One instruction file shrank from 137 lines to 100 while gaining a clear identity, a Common Mistakes table, and pointers to extracted build guidance. The other stayed lean at 98 lines while adding a short architecture orientation, explicit rules, and three path-scoped files for CDK, Lambda, and testing work.

I also compared the structural rubric with Anthropic's official CLAUDE.md management plugin. The two checks answered different questions. My rubric evaluated whether the file was designed well; Anthropic's tool cross-checked its claims against the codebase. I recommended running the structural audit first and the accuracy check second.

Recorded output

The internal rubric scores moved from C+ to A- in one repository and from B to A- in the other. Those grades describe conformance to my rubric; they are not model-performance benchmarks. The concrete changes, line counts, merged pull requests, and path-scoped rules are the stronger evidence.

What the research changed

The paper I used was Gloaguen et al.'s Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? Its first version found that generated context files increased inference cost by roughly 20–23% and often reduced task success. Manually written files performed better than generated ones, but still increased steps and cost. That evidence pushed the rubric toward minimal, deliberate, human-reviewed context rather than automatic expansion.

HumanLayer supplied the practical design principles behind universal applicability, progressive disclosure, pointers over copies, and keeping linter work out of the model. Anthropic's documentation supplied the loading model for project instructions and path-scoped rules. Internal guidance supplied repository conventions and safety requirements that public sources could not know.

References

1. Thibaud Gloaguen, Niels Mündler, Mark Müller, Veselin Raychev, and Martin Vechev. Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? arXiv:2602.11988v1, February 2026.
2. Anthropic. How Claude remembers your project. Official Claude Code documentation on project instructions and path-scoped rules.
3. HumanLayer. Writing a good CLAUDE.md, November 25, 2025.
4. Anthropic. CLAUDE.md Management Plugin. Official plugin documentation for codebase-alignment and session-learning checks.
5. Zocdoc internal Claude Code guidance, repository configurations, Jira records, pull requests, and the Interop audit presentation. These materials are not publicly linked.

Evidence boundary

The dates, line counts, deliverables, repository changes, and POC details come from my work journal, which records Jira issues, pull requests, Confluence pages, and local presentation artifacts. The repository grades are self-assessments under the rubric. The journal listed the reusable evaluation skill as in progress in May 2026, so this case study does not present that skill as shipped. Private source code, patient data, internal URLs, and company-only documentation are excluded.