INC-2026-04-12: domus-ise-linux Runbook Data Loss — Destructive Redirect

Incident Summary

Field Value

Detected

2026-04-12 ~13:30 PST

Mitigated

2026-04-12 ~13:35 PST

Resolved

2026-04-12 ~13:40 PST

Duration

~10 minutes

Severity

P2 — production documentation data loss, recoverable from git

Impact

linux-eaptls-deployment-runbook.adoc (1,081 lines) zeroed in both runbooks/ and docs/asciidoc/ copies

Root Cause

cat >| redirect from an empty source file overwrote the 1,081-line target

Timeline

Time Event

~13:25

Task: add ThinkPad P16g to the EAP-TLS deployment runbook in domus-ise-linux

~13:27

Claude Code edited runbooks/linux-eaptls-deployment-runbook.adoc — added P16g to deployment status table and target workstations table

~13:28

git add attempted on runbooks/ path — git reported nothing staged. The docs/asciidoc/ copy was the one tracked, not runbooks/.

~13:29

Claude Code assumed both files had identical content and ran: cat ~/…​runbooks/linux-eaptls-deployment-runbook.adoc >| ~/…​docs/asciidoc/…​/linux-eaptls-deployment-runbook.adoc

~13:29

DATA LOSS: The runbooks/ copy was already 0 bytes (empty). The cat >| redirect wrote 0 bytes to the docs/ copy, destroying 1,081 lines of content.

~13:30

git commit succeeded — committed the deletion of 1,081 lines as "add ThinkPad P16g"

~13:30

User reported: "what happened to this page? it’s blank"

~13:32

Investigation: wc -l confirmed both copies were 0 bytes

~13:33

Recovery: git show bd6fe74:docs/asciidoc/…​/linux-eaptls-deployment-runbook.adoc recovered the full 1,081 lines from the parent commit

~13:35

P16g changes reapplied to restored file via sed (deployment status + target workstations)

~13:38

Both copies restored to 1,109 lines (1,081 original + 28 P16g additions)

~13:40

Fix committed: "fix: restore runbook + add ThinkPad P16g to deployment status"

Symptoms

  • docs/asciidoc/modules/ROOT/pages/runbooks/linux-eaptls-deployment-runbook.adoc rendered blank in Antora

  • runbooks/linux-eaptls-deployment-runbook.adoc was also 0 bytes

  • Git commit message said "add ThinkPad P16g" but git diff --stat showed 1081 deletions(-)

Investigation

Initial Triage

  • Check file sizes — both 0 bytes

  • Check git log — commit 209283a deleted 1,081 lines

  • Check parent commit bd6fe74 — docs/ copy had full content, runbooks/ copy was already empty

  • Identify root cause — cat from empty source to populated target

Root Cause Chain

  1. Claude Code edited the runbooks/ copy (which was already 0 bytes — likely a stale copy or previously emptied)

  2. The edits were written to a 0-byte file, producing a file with only the new additions (no — the Edit tool showed the edits succeeded, but the file was empty to begin with, so the edit matched on content that didn’t exist in the actual filesystem)

  3. git add runbooks/…​ showed nothing because git tracks the docs/asciidoc/ path

  4. Claude Code then ran cat runbooks/…​ >| docs/asciidoc/…​ to "sync" the files

  5. The >| (noclobber override) forced the redirect, writing 0 bytes from the empty source to the 1,081-line target

  6. No pre-condition check — the source file was not verified to have content before the destructive redirect

Diagnostic Commands Used

# Verify both files were empty
wc -l ~/atelier/_bibliotheca/domus-ise-linux/runbooks/linux-eaptls-deployment-runbook.adoc
wc -l ~/atelier/_bibliotheca/domus-ise-linux/docs/asciidoc/modules/ROOT/pages/runbooks/linux-eaptls-deployment-runbook.adoc

# Check parent commit for recovery
git -C ~/atelier/_bibliotheca/domus-ise-linux show bd6fe74:docs/asciidoc/modules/ROOT/pages/runbooks/linux-eaptls-deployment-runbook.adoc | wc -l

# Recover to temp file
git -C ~/atelier/_bibliotheca/domus-ise-linux show bd6fe74:docs/asciidoc/modules/ROOT/pages/runbooks/linux-eaptls-deployment-runbook.adoc > /tmp/runbook-restore.adoc

Root Cause

Destructive file operation without pre-condition verification.

Claude Code ran cat <source> >| <target> where: - Source (runbooks/ copy) was 0 bytes - Target (docs/asciidoc/ copy) was 1,081 lines of production content - No check was performed to verify the source had content before overwriting the target - The >| operator bypassed zsh’s noclobber safety, forcing the overwrite

Secondary factor: the runbooks/ directory contained a stale/empty copy of the file. The authoritative copy was in docs/asciidoc/. Claude Code did not verify which copy was authoritative before using one to overwrite the other.

Resolution

Immediate Fix

# Recovered from git history
git show bd6fe74:docs/asciidoc/modules/ROOT/pages/runbooks/linux-eaptls-deployment-runbook.adoc > /tmp/runbook-restore.adoc

# Reapplied P16g changes via sed
sed -i '/4+h|Security \& Compliance/i\...' /tmp/runbook-restore.adoc
sed -i '/|{ws-aw-owner}/a\...' /tmp/runbook-restore.adoc

# Wrote to both locations
cat /tmp/runbook-restore.adoc >| .../docs/asciidoc/.../linux-eaptls-deployment-runbook.adoc
cat /tmp/runbook-restore.adoc >| .../runbooks/linux-eaptls-deployment-runbook.adoc

# Committed fix
git commit -m "fix: restore runbook + add ThinkPad P16g to deployment status"

Verification

  • wc -l confirms 1,109 lines in both copies

  • Git diff shows +1,109 insertions (full restore + P16g additions)

  • Antora build confirms page renders correctly

  • PDF/DOCX exports regenerated

Impact Assessment

Systems Affected

System Status Duration

domus-ise-linux runbook (Antora page)

Blank page committed to main

~10 minutes (not pushed to origin)

domus-ise-linux runbook (standalone copy)

Zeroed

~10 minutes

Production infrastructure

No impact

N/A — documentation only, not operational systems
The destructive commit was not pushed to origin before detection. The data loss was local only. If the push had succeeded before detection, the origin copy would also have been destroyed (recoverable via git revert but with wider blast radius).

Prevention

Short-term

  • Verify source before redirect: Always wc -l or [[ -s file ]] before using a file as a source for cat >|

  • Check git diff before commit: git diff --stat showing 1081 deletions(-) for an "add P16g" commit should have been a red flag

  • Add pre-commit hook to domus-ise-linux: warn on commits deleting >100 lines

Long-term

  • Eliminate the dual-copy pattern: runbooks/ and docs/asciidoc/…​/runbooks/ should not both exist. Use a symlink or single source of truth.

  • Claude Code behavioral rule: never use cat >| to sync files without verifying both source and target sizes first

Lessons Learned

What Went Well

  • Detection was immediate — user noticed the blank page within seconds

  • Recovery was fast — git show from parent commit restored full content

  • No data was permanently lost — git history preserved the original

What Could Be Improved

  • No pre-condition check on destructive operations — cat >| should never be used without verifying the source has content

  • Dual-copy pattern is fragile — two copies of the same file in different directories invites sync errors

  • Git diff was not reviewed before committing — 1,081 deletions for a "P16g addition" commit should have been caught

  • >| (noclobber override) should be used with extreme caution — it exists to bypass a safety mechanism

Related

Metadata

Field Value

Incident ID

INC-2026-04-12-001

Author

Evan Rosado

Created

2026-04-12

Status

Resolved

RCA Required

Yes (P2 — STD-011)