I released the first public version of my Business Operations Agent Kit. GitHub said the release was live, but a new user would have found zero downloadable files and altered release notes.
I repaired it only after testing the release from the consumer side. The finished gate included 26 tests, four green CI jobs, anonymous downloads, checksums, a fresh install, and a verified refusal path.
What Got Built
- I released Business Operations Agent Kit v0.1.0 as a public Apache-2.0 project. The repository and live demo are public, but the kit is deliberately read-only. It has no live connectors, write access, or hidden path into my business systems.
- I imported 11,914 ChatGPT text messages into private Relationship Recall. The import excluded 672 records carrying health-information or secret signals from model context. Content-hashed snapshots, post-commit cursors, durable 80-record batches, and 87 tests protect the history from silent skips.
- I ran a 240-record ChatGPT reasoning canary in 463 seconds at $0 estimated marginal cost. All 12 reviewed entries stayed faithful to the evidence, with zero unsupported claims, sensitive records, or external actions. I still stopped the proposed scale-up after finding that the sample covered only recent history.
- I made a 30-day Mid-Market Health Plan Renewal Study the primary WIMPER learning motion. Two generic medical-broker research jobs were paused. The system kept 366 due outreach touches and every send on hold while the work moved toward real renewal cases.
- I started two private operations projects behind hard boundaries. The WIMPER and Business Broker Hawaii Operations Brain remained synthetic and schema-only under a $50 monthly cap. The Owner Operations Agent Factory received a verified private off-machine backup, but nothing was deployed or connected to live business data.
What Broke (And How I Fixed It)
The public release had two different kinds of false green.
The first was inside the validation path. The project used JSON Schema, which is a rule sheet for the shape of incoming data. One email rule depended on the caller adding a separate FormatChecker when running that schema.
A caller that omitted the extra checker could submit a malformed recipient and still pass the structural validation. The rule existed in the file, but the safety boundary depended on every future caller remembering an optional setup step.
Independent review caught it before release. I added a controlled-email rule to the semantic validator itself, wrote the regression so it failed on the old behavior, and then made it pass. All 26 tests were green before I crossed the public boundary.
The second false green happened after GitHub reported the release as live.
The release page initially had zero downloadable assets. Its Markdown had also been altered, so the notes a visitor saw were not the exact notes I intended to publish. A provider-side success message proved that a release object existed. It did not prove that a stranger could use it.
I repaired the title and body, uploaded the wheel, source archive, and SHA256SUMS, then tested the release without relying on my logged-in owner view. The downloads worked anonymously. Their hashes matched byte for byte. A fresh environment installed the package, and both the normal path and the refusal path behaved as expected.
That last check matters. A safety-focused tool should prove not only that it can do the allowed thing, but also that it refuses the disallowed thing.
A third problem appeared in a completely different lane.
The 240-record ChatGPT canary looked excellent. Twelve of 12 reviewed entries were evidence-faithful, and the run showed zero unsupported claims. The tempting next move was to process ten more newest-first batches.
Then I checked the age distribution.
The canary covered June through August 2026, while 6,923 of the 11,914 archive records are more than one year old. That is 58.1% of the history. The clean test had not exercised older contradictions, superseded plans, or old assistant-written ideas that should never become current facts.
I withdrew the scale recommendation before running it. A clean canary is evidence about the sample it tested, not the archive it skipped.
The Lesson
Verify a release through the same path a stranger will use.
Here is the checklist I would give an agent before allowing a public software release:
- Clone the repository into a clean directory.
- Run the full test suite and every required CI job.
- Scan the package for private files, secrets, and machine-specific paths.
- Open the live page without owner privileges.
- Download every asset anonymously and verify its checksum.
- Install the downloaded package in a fresh environment.
- Exercise one normal use case and one refusal case.
- Compare the rendered release notes with the approved source text.
A green provider status is one input to that checklist. It is not the final receipt.
Put the safety rule inside the boundary it protects.
If malformed email addresses must be rejected, the core validator should reject them. Do not make safety depend on every caller remembering an optional wrapper, flag, or helper. Optional tooling can add convenience, but the protected operation should fail closed on its own.
Audit sample coverage before scaling a clean canary.
Before expanding an archive-processing job, compare the tested sample with the full population by age, source, risk class, and record type. If 58.1% of the archive sits outside the era you tested, a perfect recent sample does not justify full-history scale.
The Numbers
- Commits: 5 total (0 agent, 5 Matt)
- Agent jobs run: 24
- Prospects added: 0
- Emails sent: 0
- Social posts: 0 canonical receipts; 1 separately verified live post was not normalized into that event count
- Content deployment receipts: 3, representing 1 deployed article
- Public release checks: 26 tests, 4 green CI jobs, anonymous download, checksum, fresh-install, and refusal-path verification
- ChatGPT archive: 11,914 text records imported; 672 sensitive-signaled records excluded from model context
- Canary: 240 records in 463 seconds, 12 of 12 reviewed entries evidence-faithful, $0 estimated marginal cost
- Older archive share not covered by the first canary: 6,923 records, or 58.1%
- Held outreach touches: 366
The useful comparison is one provider-reported release against zero usable downloads before the repair. The release was technically present, but it had not completed the user’s journey.
The other useful comparison is 12 clean reviews against 6,923 older records outside the sample. Quality was not the problem. Coverage was.
The metric ledger had its own honesty boundary. Three deployment events were receipts for one article, while one verified social post had no canonical social.posted receipt. I kept event counts separate from unique artifacts and external outcomes instead of making the numbers look cleaner than the evidence.
What’s Next
Run a read-only, era-stratified audit across older ChatGPT history before scaling intake, and keep the public kit connector-free until each future capability passes the same consumer-side release gate.