Stratagems #16: Mark Left a Hole in His AI Audit. Lena Counted Every Layer.
Stratagems #16: Mark Left a Hole in His AI Audit. Lena Counted Every Layer.
When the enemy occupies favorable terrain, don't attack head-on. Let them think they're safe, let their guard drop - then strike the moment they relax. - The 36 Stratagems, In Order to Capture, One Must Let Loose
Previously on this series: #1: Mark Johnson Walked Into an AI Audit. - Mark found Pulse AI's benchmark evaluation set was fabricated - 44 records copied from public repositories, 54 hand-written. CTO Torres called at midnight to confess: the target was 95% before the C-round. Mark hung up. #9: Lena and P Watched Two Suppliers Fight. - Lena hired P for an independent audit and found both suppliers were cheating. Lena met P. #13: P Posted a Question on a Public Forum. - P posted a technical question that triggered Pulse AI's sales crawler. Mark recognized the pipeline signature. P met Mark.
Entry: He Left a Line in the Report
FairPay needed an AI model evaluation pipeline audit - a mandatory step before the client's technical due diligence. When Mark took the contract, the email signature read VeriTest's project liaison - the contract was real, the advance payment had cleared. He'd heard of the company through industry talk.
He sent P a message: "FairPay. Heard of it?" P replied with two characters: "VeriTest." Mark didn't ask further.
First week in. Pulled the evaluation dataset, reviewed the training distribution. The metrics looked clean - until he broke open the layers underneath. The distribution had been filtered. Low-score samples were systematically excluded from the evaluation set.
He wrote one line in his notebook: "Same signature as last year's." He didn't write which company.
He went through three months of historical snapshots. The same pattern appeared five times. The first three were large-scale, the last two pulled back - the dataset changed, but the exclusion logic stayed the same.
$ du -sh /audit/snapshots/*/
1.2G /audit/snapshots/2026-01/
1.1G /audit/snapshots/2026-02/
980M /audit/snapshots/2026-03/
340M /audit/snapshots/2026-04/
320M /audit/snapshots/2026-05/
But in the third month's snapshot root directory sat a config file no one had noticed. The signature wasn't human - it was Pulse AI's training pipeline auto-output:
# pulse-ai-auto-label-v3/config/exclusion_rules.yaml
confidence_threshold : 0.82
auto_exclude : true
excluded_output : /dev/null
The model had been automatically excluding samples below the confidence threshold during auto-labeling. Same root cause as Pulse AI's previous issues - not deliberate pruning, but the same system repeating the same mistake in a different project.
He knew the fix: rebuild the sampling distribution, add drift detection gating. FairPay's CEO wouldn't listen - the tech lead had mentioned the CEO just announced the system as "industry-leading" at an all-hands.
The final report ran seven pages. On the third page, buried in the middle paragraph, he left one line: "It is recommended to supplement evaluation data distribution validation."
He didn't write that the dataset had been filtered. He didn't write that the signature matched Pulse AI. He didn't write the fix.
Three snapshot groups had a 0% overlap in low-score samples - not random sampling, but systematic exclusion by the same rule. He placed that number in the appendix, outside the main body. But the appendix's data notes referenced the source path - pointing to _hold/ under the audit workspace, where the full three-group comparison, timeline, and fix plan were waiting.
In his notebook, he listed four layers: evidence of the exclusion rule, cross-case homology, five recurrences on the timeline, and one config value he didn't write into the report.
The report went out. He waited.
Silence: No One Came
Week one. No one contacted him.
Week two. The technical lead sent an automated confirmation of receipt. No follow-up questions. No call.
Monday evening, his phone buzzed. FairPay's CEO assistant - checking if the audit report needed any follow-up. The call was short, procedural, like clearing a checklist item.
Mark paused for a beat.
Mark: "No. We're good."
He hung up and wrote in his notebook: "Week two. Pipeline's still running."
Technically, he hadn't lied - on the filtered dataset, the metrics did meet the target range. But he knew what the other side heard: the system was fine, no need to look further.
He capped his pen. Closed the notebook. Didn't open it again.
Closing the Net
Tuesday afternoon of week four. His phone buzzed. An email. Not from FairPay's domain - from veritest.com. He was the only recipient. The CC field was empty.
Three lines:
Mark,
Tonight. Third Cup.
P mentioned your name.
Signed: Lena, VeriTest.
He arrived earlier than agreed. Evening at Third Cup. Only one table of customers left. The person behind the counter was wiping a glass, the lights dimmed to barely table-level. When Mark pushed the door open, the person behind the counter looked up.
The far seat was already taken. A full Long Black sat untouched in front of her.
Mark sat down across from her.
Lena: "You signed in at 9:17 the morning you started at FairPay. Black notebook. Brought your own coffee."
Mark didn't answer.
Lena: "We were in the room when FairPay chose Pulse AI. The client signed the contract - with an extra data sync line."
Mark didn't answer.
Lena: "Page three. 'It is recommended to supplement evaluation data distribution validation' - who were you waiting for when you wrote that?"
Mark: "When did you start watching?"
Lena: "One week before you walked in. P mentioned your name at Third Cup. I pulled your historical reports. You left three layers in the report."
She looked at him. Said nothing more.
Mark's hand stopped on the table. Didn't retract.
Mark: "How long have you known about the fourth layer?"
Not a question. A confirmation.
Lena: "Before you wrote it down. - Three projects. Same vendor - Pulse AI. Pipeline signature matches your data set exactly."
She paused.
Lena: "But I want you in on this."
Mark: "So this contract came through VeriTest."
Lena: "The client is real. The problem is real. The liaison is real - just one extra pair of eyes."
Lena: "Your report only pointed in one direction. How long to fix it?"
Mark: "A few parameter changes. Add one validation gate. Rebuild the sampling."
Lena: "How long has it been sitting in _hold/ ?"
Mark didn't answer.
Lena: "Send it to me tomorrow."
Mark: "One question first. Am I the only one you watch, or do you watch everyone who walks in?"
Lena: "Only the ones who walk into Third Cup."
When Mark stood up, his pour-over was still half-full. He didn't finish it. At the door, he paused.
Mark: "The fourth layer doesn't go by email. Tomorrow. Here."
He didn't turn back to see if she nodded. The person behind the counter placed the glass back on the shelf.
In the car, he pulled up FairPay's pipeline screenshot on his phone. On the screen, exclusion_rules.yaml sat next to Pulse AI's - the one he'd archived last year. The only difference between the two files was the company name in the comment line.
He dialed P. No answer. He sent a message: "Lena. Who is she?" No reply.
Pulling the Net
Next morning, 4:43 AM. Third Cup's lights were still on. The door was unlocked. She was sitting in the far seat. Her Long Black was half-empty, phone screen lit - open to his report from last year. The three-layer version.
Lena: "You didn't change it."
She wasn't asking.
Mark sat down across from her. The pour-over was already on the table.
Mark: "When did you know?"
Lena: "Before you walked in."
She turned her phone over on the table, screen facing him. "You left three marks in the pipeline. I counted them."
Mark said nothing.
Lena: "I'm not asking for that config layer today."
She paused.
Lena: "I'm asking you to start tomorrow. Three projects. Same pipeline - all Pulse AI. My people spent two quarters on them - looked at everything they should, missed everything they shouldn't."
She watched him process that.
Lena: "You found the third layer in your first week - and you didn't write the fix in the report not because you couldn't, but because you knew no one would ask. I saw the fourth layer - but I couldn't read it all. I need someone who can find the fourth layer."
She let that settle.
Lena: "P can find problems. P can't deliver long-term. You can."
Mark picked up the pour-over. The temperature was just right.
Mark: "Terms?"
Lena: "Write the report as it is. Four layers. Every one of them."
She stood up. Her Long Black had two sips left. She didn't plan to finish it.
Lena: "You're not the only one who's walked in, Mark. But you're the only one I called back."
Outside Third Cup, he pulled out his phone. 4:50 AM. Three rings.
P: "You call at five in the morning, you'd better be in trouble."
Mark: "Lena. What's her story?"
The line went quiet for two seconds.
P: "You met her - and you didn't ask who she was?"
Mark said nothing.
P: "VeriTest is her company. When she came to me, she didn't mention you - she mentioned FairPay. She only asked about you after you were already in."
A pause on the other end.
P: "You don't know her. She knows enough about you."
P hung up.
Mark put the phone back in his pocket. The pour-over in the car had gone cold. He didn't start the engine. He sat for ten minutes.
Then he opened his laptop, pulled up the three-layer report, and added a note below the line "It is recommended to supplement evaluation data distribution validation" on page three.
# Supplementary note Β· Recipient: VeriTest
# Evaluation data distribution validation supplement:
# 1. Three snapshot groups - low-score overlap rate 0% (Appendix A)
# 2. exclusion_rules.yaml path: audit-workspace/_hold/
# 3. Fourth-layer config anomaly: confidence_threshold was not
# reset after pipeline upgrade - same threshold applied to
# two datasets with different distributions.
# Fix direction written in _hold/.
The recipient wasn't FairPay. It was her.
That's In Order to Capture, One Must Let Loose - not waiting. Letting go. Letting them walk far enough to feel safe, then pulling the net.
π€ AI Post-Mortem [36 Stratagems Tactical Database v3.2.4]
Loaded [Tactic Match] In Order to Capture, One Must Let Loose (#16)
[Analysis Mode] Full-field scan
Tactic Match: 82%
Operator: Mark Johnson
Action: Selective disclosure - left a gap in the audit report, closed the channel with "within parameters" when called
Objective: Defensive. Verified the real defect in FairPay's evaluation pipeline.
Result: Defect confirmed. Report not challenged.
[Observation]
β Operator Mark deliberately left gaps in the audit report - first observed "strategic uncertain disclosure" pattern.
β Operator made a cross-case correlation in the report referencing an unnamed company's audit pattern ("same signature as last year's"). Correlated entity not marked in the report.
β Operator initiated no contact during the waiting period. Did not touch the pipeline after week three.
[Secondary Observation - Higher-Level Layer]
β This deployment involves two operators. Initial record captured only the first-layer executor.
β Second-layer operator: VeriTest (not Mark Johnson). This operator was in position one week before Mark's entry.
β The second layer's "let loose" was seven days ahead of the first. Net close time: T+28.
β The first-layer operator did not realize, by the end of this event, that he had been part of a larger execution structure.
[Operator Assessment - Lena (VeriTest)]
β Role: Second-layer operator (event initiator / information control node)
β Lead time: T-7. Fully deployed one week before target entry. Target signed in at 09:17 - her desk had the sign-in record by 09:22. Classification: Preparatory deployment.
β Information release rhythm: coordinates β capability β intent. Three rounds, zero unnecessary.
β Negotiation positioning: She knew the fourth layer existed, yet made the target volunteer it himself. Pattern: third-party initiative framed as recipient's own decision.
β Database assessment: Second-layer operator's strategic execution quality exceeded the first layer in this deployment. The initiator held full initiative throughout, yet made the recipient feel, at the end of every exchange, that they held the decision.
β Recommendation: Flag for priority tracking in future deployments.
[Cross-Reference]
β FairPay project referral source recorded as: VeriTest Consulting (not public)
β Operator Mark's 09:17 visitor sign-in record on entry day was reviewed within VeriTest
β Review time: 09:22, same day
[Risk Assessment]
β Operational risk: Low. Mark performed no changes within FairPay systems.
β Exposure risk: Medium. The vague phrasing on page three of the report may be re-interpreted in a future audit trace.
[Terminological Drift - New]
β Cross-case reference scan detected inconsistent references to the cafe operator across multiple case files.
β #6 (original record): "the person behind the counter"
β #9 (updated record): "the person behind the counter"
β #13 (subsequent record): "the person behind the bar"
β #14 (recent record): "the person behind the counter"
β #15 (latest record): "the bartender"
β #16 (current record): "the person behind the counter"
β Notable: the current deployment has reverted to the original reference format used in #6 and #9.
β Possible interpretations:
(a) The recording system has been calibrated - terminology drift corrected.
(b) A different operator was on station in this deployment, one who uses the original label convention.
(c) The drift was an isolated anomaly in #13 and #15 - not a trend, but noise.
β The model cannot determine which interpretation is correct without additional data points.
β Recommendation: establish a consistent reference protocol for this operator in future cases.
β If multiple operators share the same work location, re-evaluate the distributed permission model.
β If a single operator, terminology drift may indicate uncalibrated observer bias in the recording system.
β Marked as γLow Priority Β· Continue Monitoringγ
[Flagged for Attention]
β Operator Mark hung up on Pulse
Comments
No comments yet. Start the discussion.