Case study
Evidence Bound Pull Request Review
A tested prototype that freezes pull request evidence at one commit, separates private analysis from public research, and returns structured findings for a person to judge.
- Role
- Workflow designer and implementer
- Status
- Working tested prototype with sanitized public evidence
- Year
- 2026
Evidence boundary
Claims and supporting records
I built a review prototype that freezes requirements, changed code, repository context, and approved guidance against the reviewed commit before analysis begins.
Record supporting this claim
Private feedback analysis has read only local access and no web access. General research questions reach a separate public research stage only after a declassification check removes private details.
Record supporting this claim
The workflow returns structured findings with evidence references while a person curates later feedback and keeps authority over every repository action.
Record supporting this claim
Problem
An AI review can inspect the wrong revision, miss the actual requirement, or mix private code with unrestricted public research. It can also produce polished prose without a stable link between a finding, a requirement, and changed code.
The practical problem was not asking a model to review code. The problem was making each review repeatable, private, bounded, and useful to the person who still owns the decision.
Public evidence boundary
This page contains no private repository names, requirement text, review comments, diffs, source excerpts, internal addresses, user data, credentials, or model reports. The public diagram shows only the control and data flow.
Private scripts and tests support the implementation claims. The public diagram explains the architecture without publishing company code or private identifiers.
Ownership
I designed and implemented the evidence packaging, bounded review stages, write controls, structured output, and feedback curation flow. The workflow supports existing repository owners and reviewers. It does not replace their authority.
Commit pinned evidence
Each run freezes the requirements, changed code, selected reference changes, repository context, and active human approved guidance used for that review. The package records the reviewed head commit so later analysis cannot silently drift to newer code.
Feedback reflection proceeds only when the saved review identity, timestamp, and head commit match the source review. Active guidance is copied into a sealed read only snapshot with a digest before a new review begins.
Privacy separation
During feedback reflection, the private stage has read only local tools and no web access. If the private stage needs a public standard, it can propose a general research question.
A declassification check rejects any question that carries private names, source details, or internal context. A separate web only stage receives only approved general questions and returns public standards research to the private workflow.
Bounded review
The research and review stages receive only the tools and file paths needed for their jobs. Exact path checks reject other writes. Time, turn, and cost limits stop work that exceeds the allowed budget.
The review returns a readable report and a structured receipt. Each finding records the affected file and line, its severity, the supporting evidence, and any active guidance used. The workflow never comments, approves, rejects, or merges a pull request.
Curated feedback
The reflection flow compares an earlier structured report with eligible later feedback only when both refer to the same reviewed commit. New reviewing principles begin as inactive candidates.
A person can edit, activate, reject, merge, retire, or restore a candidate. The durable guide stores the principle and its evidence history. It does not store reviewer comments, names, internal addresses, requirement text, or code.
Later comparable feedback can show whether an active principle was caught, partly caught, missed, or not comparable. Silence remains inconclusive because the absence of a comment does not prove that the model was correct.
Prototype result
The implementation reached a working, documented, and tested prototype with one complete review run retained as private evidence. No production adoption is claimed.
This case study does not claim routine team use, organization wide deployment, complete defect detection, faster reviews, or measured productivity improvement.
Limitations
The requirements, changed code, repository integration, tests, and generated reports remain private. The sanitized diagram explains the published architecture. It does not independently prove the private implementation.
The review still depends on the quality of the requirements and repository context. Guardrails make failures easier to detect and contain. They do not turn model output into ground truth.
Reflection
The most important boundary appears after the model responds. A polished finding remains a recommendation until a person checks its evidence and owns the action.
Commit pinned inputs make a later comparison meaningful. Privacy separation makes public research possible without exposing the private review package. Human curation prevents one comment from becoming permanent policy without judgment.