I ran 154 real question-and-answer comparisons to decide whether my AI system should automatically retrieve old context. The system found useful information, but it also returned 8 irrelevant results, 2 stale or conflicting results, and 12 cases that needed manual review.
So I did not turn it on.
What Got Built
- I completed a seven-day memory shadow test. The local index evaluated 154 pairs without injecting its results into the live answer. That let me compare what the system would have remembered against what the working agent actually used, without letting an experimental memory change production decisions.
- I kept WIMPER activity behind its existing stop gate. Research, qualification, email composition, follow-up, and sending remained stopped. The system recorded 366 held sequence touches and 304 held legacy drafts, with 0 outbound actions and 0 replies.
- I refreshed the revenue evidence instead of manufacturing activity. The portfolio review counted 198 tracked DIRECT receipts, revalidated a partner packet, and prepared an approved-draft retirement queue. It did not send an email, publish a social post, or alter a draft.
- I audited 51 WIMPER Institute posts. All 51 passed the structural review, so the correct number of corrections was 0.
- I reviewed 10 queued content drafts through a site-aware gate. All 10 were skipped correctly. None received automatic execution authority.
- I completed the prior Build Chronicler screenshot cycle. Authenticated dashboard views were checked with text recognition before they were embedded. The event stream recorded 4 deployment receipts in the 24-hour window.
What Broke (And How I Fixed It)
The memory system looked good in a fixed benchmark.
It retrieved the expected item within its first three results in 39 of 40 benchmark questions. It put the right result first in 35 of 40. It also passed all 11 deterministic safety checks.
Those numbers were strong enough to justify a real shadow. They were not strong enough to justify automatic use.
A benchmark is a controlled exam. Production questions are the job.
For seven days, the local index watched real interactions from the side. It ran after the live answer, so the retrieved text could not influence what the agent told me or what it did. The test produced 154 evaluated pairs.
Only 28 of those pairs actually needed retrieval. Among the results, 2 supplied useful information that the live answer had missed. Another 13 supported information that was already being used.
But 8 results were irrelevant. Two were stale or conflicted with current truth. Twelve involved sensitive material that required a person to decide whether the context belonged in an answer.
The helpful rate was 53.6%. The irrelevant rate was 28.6%.
That is not a rounding error. If I had promoted the system because 39 of 40 benchmark questions looked good, old or unrelated context could have entered live prompts automatically. A memory system does not need to produce an obviously broken answer to cause damage. It can quietly make the agent more confident in the wrong fact.
The fix was not another prompt telling the AI to be careful.
The fix was to leave the automatic connection unwired. The local index remains available for manual searches, but it cannot inject its own results into a live conversation. The experiment passed the minimum sample gate and failed the semantic-safety and automatic-recall gates. Failing either one meant no promotion.
Two other evidence sources also failed during the daily record.
The GitHub commits API returned HTTP 401, which means its credential was rejected. I recovered an exact count of 2 commits from verified local git history instead of estimating.
The same-day work log was not ready because that job runs later. I used the prior work log, the event bus, and local git receipts, then named the missing source in the record.
Those fallbacks worked because they used primary receipts. They did not turn missing evidence into a flattering guess.
The Lesson
Test retrieval on real work before it can change real work.
Here is what I would tell someone adding memory to an AI agent: do not connect the memory system directly to the prompt on day one. Run it in shadow mode first. Let the live agent answer normally, retrieve context afterward, and compare whether the added material was useful, irrelevant, stale, conflicting, or sensitive.
Make promotion require several gates at once.
A large sample is not enough. High benchmark recall is not enough. Passing code-level safety tests is not enough. Require a minimum sample, useful recall, semantic safety, and a clear rule for sensitive information. If one gate fails, keep the feature manual.
Treat stale memory as a correctness problem, not a storage problem.
Saving more history feels like progress because the archive gets larger. The real question is whether the right fact appears at the right time and still represents current truth. Two stale or conflicting retrievals were enough to block automatic injection because a confidently repeated old fact can be worse than no memory at all.
Separate infrastructure receipts from business outcomes.
The event bus showed 4 site deployments. It showed 0 content.published events. A deployment proves that site infrastructure ran. It does not prove that four new articles reached readers. Keeping those metrics separate prevents technical motion from masquerading as content output.
The Numbers
- Commits: 2 total (0 agent, 2 Matt)
- Agent jobs run: 18
- Prospects added: 0
- Emails sent: 0
- Social posts: 0
- Deployment receipts: 4
- New article publication receipts: 0
- Memory shadow evaluations: 154
- Retrieval-needed pairs: 28
- Useful missing results: 2
- Supportive results already represented in the answer: 13
- Irrelevant results: 8
- Stale or conflicting results: 2
- Sensitive manual-review cases: 12
- Helpful retrieval rate: 53.6%
- Irrelevant retrieval rate: 28.6%
- Benchmark recall within the first 3 results: 39 of 40
- Held WIMPER sequence touches: 366
- Held legacy email drafts: 304
- WIMPER Institute posts audited: 51
The most important number is 2.
The system found 2 useful facts that the live answer had missed. That proves the local index has value. It also returned 2 stale or conflicting results, which proves the same system can add the wrong context.
The correct outcome was not to call the experiment a failure and delete it. It was to keep the useful tool manual until its retrieval and curation improve enough to pass every promotion gate.
What’s Next
Repair the memory retrieval and curation rules, then rerun the seven-day shadow. Automatic context injection stays off until both semantic safety and narrow recall pass.