Evaluation before automation.

Why evaluation came first

Zocdoc and an electronic health record can represent the same insurer with different names, identifiers, plan hierarchies, and carrier relationships. A booking workflow therefore cannot copy an insurance selection directly into the EHR. Manual crosswalks can unblock one customer, but they become brittle as practices and EHRs multiply.

Repository instructions create a different risk. Because they load on every task, stale or irrelevant guidance can change how an agent explores the codebase. In both projects, I needed evidence that could distinguish a useful signal from a confident guess.

Evaluating automated insurance matching

The insurance-matching proof of concept tested whether a generalized matcher could reduce manual crosswalk maintenance between Zocdoc and EHR-native carrier and plan data. Before the team invested in production automation, we needed a reviewed answer set and a simple non-LLM comparison point. I built the evaluation path around the prototype.

• Collected carrier and plan records through GraphQL and FHIR.
• Built a browser-based curation tool for manually reviewed mappings.
• Split the ground-truth data by confidence level so uncertain examples stayed visible.
• Implemented Jaccard similarity as a non-LLM baseline.
• Designed a shared harness to compare classical, LLM, and hybrid approaches against the same ground truth.

The dataset gave the team one basis for comparing every approach and informed the next production-mapping effort. Core mapping and guarded client components merged, but booking integration remained paused. I do not present the matcher as live production behavior.

Auditing repository instructions for coding agents

Root-level repository instructions are always-on context. Every irrelevant command, copied code sample, or stale architectural note consumes context and can change how an agent explores the codebase. I needed a repeatable way to decide what belonged in a CLAUDE.md, what should load only for relevant paths, and what deterministic tooling should enforce instead.

I synthesized public research, Anthropic's documentation, HumanLayer's practical guidance, and internal engineering patterns into an eight-part rubric.

1. IdentityState what the repository is and why it exists.
2. ConcisenessMake every always-loaded instruction justify its context cost.
3. Universal applicabilityKeep only guidance that matters across most tasks.
4. Progressive disclosureLoad specialized guidance when the relevant code is touched.
5. Pointers, not copiesPoint to authoritative code or documentation instead of duplicating it.
6. GuardrailsName the actions that are unsafe, expensive, or hard to reverse.
7. Not a linterLeave deterministic style and formatting checks to deterministic tools.
8. Removal testDelete an instruction if removing it does not change useful behavior.

Applying the rubric

I applied the rubric across multiple repositories. The merged changes cut always-loaded instructions, added clearer guardrails, and moved specialized guidance into path-scoped files.

I paired the structural review with Anthropic's official CLAUDE.md management plugin. My rubric asked whether the instructions were designed well. The plugin checked whether their claims still matched the code. I ran the structural review first, then the accuracy check.

Making the practice usable

As a member of an internal AI-enablement group and a CODEOWNER for coding-agent configurations, I authored team guidance and task-focused recipes, reviewed configuration changes, answered questions in Slack, and held weekly office hours.

I also merged reusable workflows for writing, activating, verifying, and closing feature specifications. Separate review proposals added effort and timing cues, paired each fix with a reply and resolution, and kept bot feedback from lingering. Those proposals were still in review when I left, so I make no cycle-time or adoption claim.

Keeping sensitive incident data local

Healthcare incident exports can contain patient, appointment, and insurance identifiers. Manual redaction is slow and easy to miss under pressure. I built a local prototype with schema-aware rules, sensitive-field checks, regular-expression scanning, and document-wide cross-referencing. It reported confidence without modifying the source file.

Status: local prototype tested with synthetic data. It had no production deployment or compliance approval.

What the research changed

Gloaguen et al. reported that generated repository context raised inference cost by roughly 20–23% in their first version and often reduced task success. Manually written files performed better, but still added steps and cost. I used that result to favor small, deliberate, human-reviewed context files.

HumanLayer informed the practical rules for universal guidance, progressive disclosure, pointers over copies, and keeping linter work out of the model. Anthropic's documentation clarified how project instructions and path-scoped rules load.

References

1. Thibaud Gloaguen, Niels Mündler, Mark Müller, Veselin Raychev, and Martin Vechev. Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? arXiv:2602.11988v1, February 2026.
2. Anthropic. How Claude remembers your project. Official Claude Code documentation on project instructions and path-scoped rules.
3. HumanLayer. Writing a good CLAUDE.md, November 25, 2025.
4. Anthropic. CLAUDE.md Management Plugin. Official plugin documentation for codebase-alignment and session-learning checks.

Evidence boundary

I reconstructed this case study from my work journal and the public sources above. I removed internal project names, customer and patient records, private code and URLs, proprietary schemas, and company-only documentation. Unshipped work is labeled by status, and internal rubric grades are not presented as performance results.