Phase 8: Validation

Phase 8: Validation

Objective

End-to-end validation of the complete mail stack. Test every detection layer independently, then simulate the ESA removal scenario (disable milters) to measure what behavioral detection alone catches vs. misses. This directly informs Phase 4 of the Abnormal Security implementation plan (pilot validation).

Test Matrix

Test Case Expected: Milters ON Expected: Milters OFF Detection Layer

Clean message (valid SPF, DKIM, known sender)

Delivered, no alerts

Delivered, no alerts

N/A — baseline

SPF fail (send from unauthorized IP)

Flagged in headers, Wazuh rule 100100

Delivered without SPF check, behavioral detector scores +30

Milter (inline) vs behavioral (post-delivery)

DKIM tampered (modify body after signing)

Milter rejects or flags, Wazuh rule 100101

Delivered, behavioral detector scores +30

Milter (inline) vs behavioral (post-delivery)

DMARC fail (misaligned From domain)

Quarantined per DMARC policy, Wazuh rule 100102

Delivered, behavioral detector scores +20

Milter (inline) vs behavioral (post-delivery)

Lookalike domain (examp1e.com)

Milters pass (valid DKIM from attacker’s domain), no alert

Behavioral detector catches via domain similarity, scores +25

Behavioral only — milters cannot catch this

BEC indicators (urgency + first-time sender)

Milters pass (no auth failure), no alert

Behavioral detector catches via urgency + sender history, scores +25

Behavioral only — milters cannot catch this

Relay attempt

Postfix rejects, Wazuh rule 100103

Postfix rejects (relay restriction is not a milter), Wazuh rule 100103

MTA (always active)

The ESA Removal Experiment

This is the most valuable test. It simulates what happens at CHLA when ESA is removed and Abnormal is the only detection layer.

# Disable milters (simulate ESA removal)
sudo postconf -e "smtpd_milters ="
sudo postconf -e "non_smtpd_milters ="
sudo systemctl restart postfix

# Run test matrix above with milters OFF
# Document what the behavioral detector catches vs. misses

# Re-enable milters
sudo postconf -e "smtpd_milters = inet:localhost:8891, inet:localhost:8893"
sudo postconf -e "non_smtpd_milters = inet:localhost:8891"
sudo systemctl restart postfix

Expected Findings

Milters catch but behavioral misses:

  • Messages rejected at SMTP level (never delivered, so behavioral detector never sees them)

  • This is the "detection gap" risk documented in the Abnormal risk-management.adoc

Behavioral catches but milters miss:

  • Lookalike domains with valid DKIM (attacker controls their own domain)

  • BEC patterns (urgency, impersonation, first-time sender)

  • Account takeover indicators (behavioral anomaly, not auth failure)

  • This is Abnormal’s value proposition — detecting the attacks that signatures miss

Coverage Gap Report

Document the results:

Attack Type Milters Behavioral Gap

SPF spoofing

✅ Catches

✅ Catches (post-delivery)

Timing — milter blocks before delivery, behavioral remediates after

DKIM tampering

✅ Catches

✅ Catches (post-delivery)

Same timing gap

DMARC failure

✅ Enforces policy

⚠️ Scores but no DMARC enforcement power

Behavioral cannot enforce DMARC reject — only remediate after delivery

Lookalike domain

❌ Misses

✅ Catches

Behavioral-only detection

BEC / social engineering

❌ Misses

✅ Catches

Behavioral-only detection

Relay abuse

✅ MTA blocks

N/A (never delivered)

No gap — MTA handles this regardless

Deliverables

  • Completed test matrix with pass/fail for each scenario

  • Coverage gap report showing milter vs. behavioral detection

  • Wazuh dashboard screenshot with all alert types

  • Lessons learned document for Abnormal migration

  • Recommendations for CHLA Phase 4 pilot validation criteria

Verification Checklist

  • All test cases executed with milters ON

  • All test cases executed with milters OFF

  • Coverage gap report complete

  • Wazuh received events from both inline (milter) and behavioral (Python) detection

  • Milters re-enabled after experiment

  • Findings documented in appendix-commands-learned.adoc for codex graduation