CHLA Daily Work Log - Monday 2026-01-26 (Post-Pentest)

URGENT PERSONAL - DL EXPIRES JAN 29 (3 DAYS)

  • DMV visit TODAY - Driver’s license renewal

  • Written exam likely required - study CA DMV handbook

  • Bring: current DL, payment (~$41)

  • Passport renewal - schedule appointment (DS-82, ~$130)


Today’s Focus (Monday 2026-01-26)

Priority 1: iPSK Manager Security Hardening (CHLA)

Goal: Restrict MySQL from wildcard access to specific ISE hosts only.

Reference: PRJ-ISE-IPSK-CHLA-ANTORA/docs/asciidoc/modules/ROOT/pages/operations/security-hardening.adoc

CRITICAL FIX APPLIED: Create ipskadmin@localhost BEFORE dropping ipskadmin@% to prevent web app outage.

Execution Checklist:

  • SSH to iPSK Manager server

  • Get CHLA ISE node IPs

  • Generate passwords (openssl rand -base64 16)

  • Execute SQL (safe order confirmed)

  • Update config.php

  • Restart Apache

  • Update ISE ODBC in GUI

  • Test authentication

  • Update dsec with new credentials

Priority 2: Dr. Shahab Linux Workstation

Goal: Complete validation and ISE hardening.

Reference: DEPLOY-2026-01-26-shahab-linux-workstation.md

Field Value

MAC

b4:e9:b8:f6:c8:17

Switch

SRT-9-9300, Gi1/0/36

IP

10.238.179.128

Current Policy

Research_Onboard (TOO OPEN)

Target Policy

Linux_Research_EAP_TLS (hardened)

Tasks:

  • SSH validation (LUKS, AD, Defender, UFW, certs, 802.1X)

  • Create dACL: DACL_LINUX_RESEARCH_HARDENED

  • Create authz profile: Linux_Research_EAP_TLS

  • Add authorization rule

  • Force reauth and verify

  • Test dACL effectiveness


Carry-Over from Friday (2026-01-23)

dsec Domain Access Control Implementation

Overview

Implemented tier-based access control for the dsec secrets manager to enforce strict boundaries around client/work credentials. This prevents accidental disclosure of protected information during automation or interactive sessions.

Security Model
Domain Access Level Description

d000

Full

Home infrastructure - all tiers accessible

d001+

Restricted

Client domains - only lab tier accessible

Access Denied Warning

Unauthorized access attempts now display a legal warning banner with:

  • ASCII art “ACCESS DENIED” header (red/yellow)

  • Federal law citations (18 U.S.C. § 1030 CFAA, § 1832 Trade Secrets)

  • NDA/confidentiality agreement references

  • Resource, timestamp, and source logging

  • Audit trail entry in ~/.secrets/.metadata/audit.log

Implementation Details
  • Added check_domain_tier_allowed() function to dsec

  • Protects: show, edit, load, source commands

  • Configuration variables:

    • DSEC_HOME_DOMAIN=d000 (full access domain)

    • DSEC_ALLOWED_TIERS=lab (allowed tiers for client domains)

    • DSEC_DOMAIN_LOCK=true (enable/disable restriction)

  • Override: DSEC_DOMAIN_LOCK=false dsec show d001 prod

Directory Structure Updates

Created lab tier for both domains:

d000/lab/  (app, http, identifiers, identity, network, storage)
d001/lab/  (network)
Documentation
  • Updated dsec.adoc with Domain Access Control section

  • Updated quick-reference.adoc with access control cheat sheet

  • Updated DOMAIN_INVENTORY.yaml.age (schema v2.2)

  • Rebuilt PRJ-SECRETS PDFs


Pending Tasks

High Priority

  • Camera IPs in InfoBlox for Rodney

  • JOY workstation follow-up (Natus investigation)

  • VNC hunt with Mauricio

  • SNE-21 investigation

  • CVE-2026-20029 iTrack submission

Medium Priority

  • YouTube/GetWell review

  • Azure DevOps LFS .lfsconfig fix

  • ISE diagrams for Omer

  • dACL documentation for Omer

  • Integration architecture for Omer

Scheduling Required

  • MSCHAPv2 migration meetings

  • Victor Negri follow-up

  • Azure Legacy kick-off (Jan 26)


PRJ-ISE-HOME-LINUX: Active Directory Domain Join

Project Flow

PRJ-ISE-HOME-LINUX (modestus-p50) → PRJ-ISE-CHLA-LINUX (Dr. Shahab)
         ↓                                    ↓
   "eat your own dog food"            production deployment

Current Status

Component Status Notes

WiFi EAP-TLS

COMPLETE

Domus-Secure SSID authenticated

Wired 802.1X

COMPLETE

wpa_supplicant configured

AD Domain Join

PENDING

Blocker for next phase

SSSD/Kerberos

NOT STARTED

Depends on domain join

Prerequisites Checklist

  • DC reachable: ping dc-01.inside.domusdigitalis.dev10.50.1.50 (home-dc01) - 5ms

  • DNS SRV records: dig +short _ldap._tcp.inside.domusdigitalis.dev SRV0 100 389 home-dc01

  • Realm discovery: realm discover inside.domusdigitalis.devINSIDE.DOMUSDIGITALIS.DEV, AD detected, sssd client

  • Required packages installed: sssd realmd krb5 adcli samba (Arch) / + oddjob (Ubuntu)

Package Installation

Arch Linux (modestus-p50 - home enterprise):

sudo pacman -S sssd samba krb5
yay -S realmd adcli

Ubuntu (CHLA - Dr. Shahab production):

sudo apt install sssd sssd-tools realmd adcli krb5-user samba-common-bin oddjob oddjob-mkhomedir

Domain Join Procedure

# 1. Packages installed (see above)

# 2. Discover realm
realm discover inside.domusdigitalis.dev

# 3. Join domain (will prompt for Administrator password)
sudo realm join -U Administrator inside.domusdigitalis.dev

# 4. Verify join
realm list
id evan@inside.domusdigitalis.dev

Expected Post-Join Configuration

  • /etc/krb5.conf - Kerberos realm configuration

  • /etc/sssd/sssd.conf - SSSD domain config

  • /etc/nsswitch.conf - NSS sources updated

  • /etc/pam.d/* - PAM modules configured

Documentation Reference

  • PRJ-ISE-HOME-LINUX/docs/asciidoc/modules/ROOT/pages/04-linux-client/domain-join.adoc


Today’s Notes

SSH Backup YubiKey Fix

Issue: Backup YubiKey SSH key not authenticating to modestus-p50 despite key being in authorized_keys.

Root Cause: Stray EOF marker at the bottom of ~/.ssh/authorized_keys file corrupting the key file parsing.

Resolution:

  1. Removed the trailing EOF from authorized_keys

  2. Killed cached SSH connection: ssh -O exit modestus-p50

  3. Reconnected - backup YubiKey now authenticates successfully

Lesson Learned: When using heredocs to append keys to authorized_keys, ensure no literal EOF string gets written.


Domain Join Sequence

Order Host Status Notes

1

modestus-p50

IN PROGRESS

Test subject for domain join validation

2

razer

PENDING

Main workstation - configure after P50 succeeds


Future Requirement: Android MDM + Certificate Auth

Goal: All devices (laptops, phones, tablets) authenticate using certificate-based auth with internally issued certificates.

Device Category Auth Method Certificate Source

Linux laptops

EAP-TLS

ADCS via certbot

Windows laptops

EAP-TLS

ADCS auto-enroll

Android phones

EAP-TLS

MDM-issued certs

iOS devices

EAP-TLS

MDM-issued certs

MDM Solution Requirements:

  • Certificate enrollment via SCEP/EST from internal CA

  • WiFi profile deployment with EAP-TLS configuration

  • Device compliance checking (posture equivalent for mobile)

  • Integration with ISE for network access control

Candidate Solutions to Evaluate:

  • Microsoft Intune (Azure AD integration)

  • Workspace ONE (VMware)

  • MobileIron

  • Self-hosted: Headwind MDM, MicroMDM

Priority: After Linux domain join is validated on P50 and razer.


AD User Creation from Linux

Discovery: AD users can be created directly from domain-joined Linux machines using adcli.

Process:

  1. Create user from Linux:

sudo adcli create-user USERNAME --domain=inside.domusdigitalis.dev -U Administrator
  1. Set password (from DC):

Set-ADAccountPassword -Identity USERNAME -Reset -NewPassword (ConvertTo-SecureString "ComplexPass123!" -AsPlainText -Force)
  1. Enable account and add to groups (from DC):

Enable-ADAccount -Identity USERNAME
Add-ADGroupMember -Identity "Linux-Cert-Enrollers" -Members USERNAME
  1. Full creation with PowerShell (alternative):

New-ADUser -Name "USERNAME" -SamAccountName "USERNAME" -UserPrincipalName "USERNAME@inside.domusdigitalis.dev" -Enabled $true -AccountPassword (ConvertTo-SecureString "PASSWORD" -AsPlainText -Force) -ChangePasswordAtLogon $true
Add-ADGroupMember -Identity "Linux-Cert-Enrollers" -Members USERNAME
Get-ADUser USERNAME

Users Created:

  • gabriel@inside.domusdigitalis.dev - P50 primary user

  • evanusmodestus@inside.domusdigitalis.dev - Admin user

Credentials stored in: pass ad/inside.domusdigitalis.dev/

Security Concern: Domain-joined Linux machines with admin credentials can create AD users. Need to review delegation model and restrict adcli capabilities.


P50 Machine Certificate (EAP-TLS)

Check Certificate Command:

openssl x509 -in /etc/ssl/certs/modestus-p50-eaptls.pem -noout -subject -issuer -dates -serial

Certificate Details: | Field | Value | |——-|——-| | Subject | O=Domus Digitalis, OU=Endpoints, CN=modestus-p50.inside.domusdigitalis.dev | | Issuer | CN=HOME-ROOT-CA, DC=inside, DC=domusdigitalis, DC=dev | | Valid From | Jan 14, 2026 | | Valid Until | Jan 14, 2028 | | Serial | 270000001996A68611AFC9DCA9000000000019 |

File Locations: | File | Path | |——|——| | CA Certificate | /etc/ssl/certs/HOME-ROOT-CA.pem | | Client Certificate | /etc/ssl/certs/modestus-p50-eaptls.pem | | Private Key | /etc/ssl/private/modestus-p50-eaptls.key |

wpa_supplicant.conf:

network={
    ssid="Domus-Secure"
    key_mgmt=WPA-EAP
    eap=TLS
    identity="modestus-p50.inside.domusdigitalis.dev"
    ca_cert="/etc/ssl/certs/HOME-ROOT-CA.pem"
    client_cert="/etc/ssl/certs/modestus-p50-eaptls.pem"
    private_key="/etc/ssl/private/modestus-p50-eaptls.key"
    priority=10
}

Authentication Flow:

P50 (Machine Cert) → 802.1X → WLC → ISE → Validate cert against HOME-ROOT-CA → Grant network access
                                              ↓
                              ISE checks: CN matches, cert not revoked, in Linux-Cert-Enrollers group

Password Store Organization

Organized pass password store for infrastructure credentials:

ad/                              # Active Directory Users
├── chla.usc.edu/
│   ├── erosado
│   └── template
└── inside.domusdigitalis.dev/
    ├── evanusmodestus
    └── gabriel

infra/inside.domusdigitalis.dev/ # Infrastructure Admin Accounts
├── adcs-admin
├── dc-administrator
├── ise-admin
├── keycloak-admin
├── pfsense-admin
├── switch-admin
└── wlc-admin

svc/inside.domusdigitalis.dev/   # Service Accounts
├── certbot-svc
└── radius-shared-secrets

wifi/inside.domusdigitalis.dev/  # WiFi Config References
├── domus-secure
└── domus-iot

Usage:

  • View: pass show ad/inside.domusdigitalis.dev/gabriel

  • Edit: pass edit infra/inside.domusdigitalis.dev/ise-admin

  • Copy to clipboard: pass -c ad/chla.usc.edu/erosado


Domain Join Status Summary

What Works: | Component | Status | Test Command | |———–|——–|————–| | Domain Join | ✅ | realm list | | User Lookup (NSS) | ✅ | id gabriel@inside.domusdigitalis.dev | | Kerberos Auth | ✅ | kinit gabriel@INSIDE.DOMUSDIGITALIS.DEV | | SSSD Service | ✅ | systemctl status sssd | | Root su (bypass) | ✅ | sudo su - gabriel@inside.domusdigitalis.dev |

What Needs Fixing: | Component | Status | Issue | |———–|——–|——-| | Password Auth (su) | ❌ | PAM not calling pam_sss.so correctly | | Auto Home Directory | ❌ | pam_mkhomedir not triggering |

PAM Investigation Notes:

Files checked:

  • /etc/pam.d/su - Updated to auth include system-auth

  • /etc/pam.d/system-auth - Has auth include sssd-arch but password auth still fails

  • /etc/pam.d/sssd-arch - Contains auth sufficient pam_sss.so forward_pass

  • /usr/lib/security/pam_sss.so - Exists

Added but still failing:

sudo sed -i '/^auth include sssd-arch/a account include sssd-arch' /etc/pam.d/system-auth

Next Steps:

  1. Debug PAM ordering - pam_sss.so may need different control flag

  2. Check if [success=1 default=bad] on pam_unix.so is blocking SSSD

  3. Consider moving pam_sss.so AFTER pam_faillock preauth

  4. Test SSH password auth as alternative to su

  5. Verify pam_mkhomedir.so is in session stack

Workaround (for now):

sudo su - gabriel@inside.domusdigitalis.dev

Works because root bypasses password via pam_rootok.so.


URGENT: Mobile Device BYOD - Move from Password to Certificate Auth

Current State: Mobile devices using password-based WiFi auth (insecure)

Target State: Certificate-based EAP-TLS via ISE BYOD portal

Tomorrow’s Plan:

  1. Create Onboarding SSID

    • SSID: Domus-Onboard

    • Security: Open or simple PSK

    • Purpose: Initial device connection for BYOD redirect

  2. Configure ISE BYOD Portal

    • Work Centers → Guest Access → Portals → Create BYOD Portal

    • Portal Settings: Require AD authentication

    • Certificate Provisioning: Enable SCEP/manual

  3. Create Authorization Rules

    Rule: BYOD_Onboarding
    Condition: Wireless_MAB AND NOT (Certificate)
    Result: Redirect to BYOD Portal
    
    Rule: BYOD_Provisioned
    Condition: EAP-TLS AND Certificate_Valid
    Result: PermitAccess, VLAN=DATA
  4. Configure WLC

    • Create Domus-Onboard WLAN (open/PSK)

    • Add redirect ACL pointing to ISE

  5. Test Flow

    • Connect Android/iOS to Domus-Onboard

    • Get redirected to ISE BYOD portal

    • Authenticate with AD creds

    • Receive certificate

    • Auto-connect to Domus-Secure (EAP-TLS)

Diagrams Created:

  • byod-onboarding-flow.mmd - Full sequence diagram

  • mdm-ise-integration.mmd - Architecture options

  • work-vs-home-comparison.mmd - Intune vs Headwind vs ISE-only

Reference: PRJ-ISE-HOME/docs/asciidoc/modules/ROOT/images/diagrams/


Root Cause Analysis: Morning System Issues (2026-01-22)

Issue 1: NFS Mount Failure

Symptom:

sudo mount -t nfs nas-01.inside.domusdigitalis.dev:/volume1/borg_backups /mnt/synology
mount.nfs: failed to prepare mount: No such device

Root Cause: Kernel updated but system not rebooted. NFS kernel module not loaded.

Evidence:

uname -r
# Output: 6.18.2-arch2-1

lsmod | grep nfs
# Output: (empty - module not loaded)

Resolution:

# After reboot:
sudo modprobe nfs && echo "nfs" | sudo tee /etc/modules-load.d/nfs.conf
sudo mount -t nfs nas-01.inside.domusdigitalis.dev:/volume1/borg_backups /mnt/synology

Reference: Documented in WRKX-2026-01-21-020 “Post-Reboot TODO” section.


Issue 2: YubiKey SSH Signing Failure

Symptom:

sign_and_send_pubkey: signing failed for ED25519-SK ".../id_ed25519_sk_rk_d000" from agent: agent refused operation

Root Cause: Wrong YubiKey plugged in (secondary instead of primary).

Evidence - Key fingerprint mismatch:

Source Fingerprint YubiKey

Downloaded from plugged-in YubiKey

SHA256:HajV/...

Secondary

~/.ssh/id_ed25519_sk_rk_d000.pub

SHA256:UuOT/...

Primary

Key mapping: | Key File | Expected Fingerprint | YubiKey | |———-|———————|———| | id_ed25519_sk_rk_d000 | UuOT... | Primary | | id_ed25519_sk_rk_d000_secondary | HajV... | Secondary |

FIDO2 error confirms credential mismatch:

SSH_AUTH_SOCK= ssh -v -i ~/.ssh/id_ed25519_sk_rk_d000 gabriel@10.50.10.103 2>&1 | grep FIDO
# debug1: sk_try: fido_dev_get_assert: FIDO_ERR_NO_CREDENTIALS

FIDO2 stack verified functional:

fido2-token -L
# /dev/hidraw3: vendor=0x1050, product=0x0407 (Yubico YubiKey OTP+FIDO+CCID)

ssh-keygen -K  # Successfully downloads resident keys

Resolution:

  1. Immediate fix: Plug in Primary YubiKey

  2. Alternative: Use secondary key explicitly:

    SSH_AUTH_SOCK= ssh -i ~/.ssh/id_ed25519_sk_rk_d000_secondary gabriel@10.50.10.103
  3. System fix: Reboot to clear any kernel/FIDO2 issues:

    sudo reboot

Post-reboot verification:

lsmod | grep nfs        # Should show nfs module
ssh modestus-p50        # Should work with PRIMARY YubiKey plugged in

Carried Over from Yesterday (2026-01-21)

Quick Reference - Top Priorities

Priority Category Item Status Next Action

P0

LINUX/ISE

Xiangming & Sarah Linux Workstations

In Progress

Complete EAP-TLS testing, deploy to CHLA

P0

PENTEST

Internal Pentest Support

Day 4 (Vartan)

8a-3p

P0

PENTEST FINDING

Posture Redirect ACL Remediation

NEW

Create change request, lock down ACL

P1

AUTOMATION

Certbot/Let’s Encrypt Completion

90%

Finalize deploy hooks, test renewal

P1

ISE

Closed Mode Implementation

Pending

Design policy, test in home enterprise

P1

DEVOPS

Azure DevOps LFS Fix

NEW

Update .lfsconfig endpoint

P2

CAMERA

Static IP Reservations

IPs confirmed

Create in InfoBlox

P2

SUPPORT

JOY Natus Workstation

APIPA issue

Follow up with Willie/Edwin

P2

SECURITY

VNC Traffic Identification

Schedule needed

Schedule with Mauricio

P3

INVESTIGATION

SNE-21 Connectivity

Pending

Investigate when time permits

P3

VULNERABILITY

CVE-2026-20029 ISE XXE

Assessment done

Submit iTrack change request

ONGOING

MIGRATION

MSCHAPv2 to Cert-Based

10%

Schedule meetings with Paul/Andrew

OWED

DOCUMENTATION

Omer - ISE Diagrams/dACL/Claroty

Pending

This week

Internal Pentest Schedule (This Week)

Date Day Staff Hours Focus

01/19

Mon

Vartan

8a-3p

Day 1

01/20

Tue

Vartan

8a-3p

Day 2

01/21

Wed

Ashley

9a-2pm

Day 3

01/22

Thu

Vartan

8a-3p

Day 4

01/23

Fri

Sarah

8am-2pm

TODAY - FINAL

MSCHAPv2 Migration Status

Priority: P1-Critical | Status: In Progress (10%) | Total Devices: 6,088

Wave Device Type Count Contact Status

1

Chromebooks

1,754

Paul Tran

Reach out

2

WYSE Thin Clients

857

Andrew Rolle

Reach out

3

Windows Domain

270

Intune Team

4

macOS

331

JAMF Team

5

iOS/iPhone

1,760

Intune/JAMF

Tasks Owed to Omer Joffe

Task Format Priority

ISE policy diagrams

LaTeX

This week

dACL mapping documentation

Markdown

This week

ISE - Claroty integration architecture

Diagram

This week


AD Group Verification & Creation (home-dc01)

Verification Commands (PowerShell on DC)

# Check if OU exists
Get-ADOrganizationalUnit -Filter 'Name -eq "Linux"'

# Check existing groups
Get-ADGroup -Filter 'Name -like "Linux*"' | Select Name, GroupScope

# Check Linux-Cert-Enrollers members
Get-ADGroupMember -Identity "Linux-Cert-Enrollers"

# Check if Linux-Admins exists
Get-ADGroup -Identity "Linux-Admins" -ErrorAction SilentlyContinue

# Check users
Get-ADUser evanusmodestus
Get-ADUser gabriel

Creation Commands (If Missing)

# Create OU for Linux systems
New-ADOrganizationalUnit -Name "Linux" -Path "DC=inside,DC=domusdigitalis,DC=dev"

# Create security group for research Linux devices
New-ADGroup -Name "Linux-Research-Devices" `
    -GroupCategory Security `
    -GroupScope Global `
    -Path "OU=Linux,DC=inside,DC=domusdigitalis,DC=dev" `
    -Description "Linux research workstations - allowed domain login"

# Create group for Linux admins (gets sudo)
New-ADGroup -Name "Linux-Admins" `
    -GroupCategory Security `
    -GroupScope Global `
    -Path "OU=Linux,DC=inside,DC=domusdigitalis,DC=dev" `
    -Description "Linux administrators - sudo access"

# Add users
Add-ADGroupMember -Identity "Linux-Admins" -Members "evanusmodestus"
Add-ADGroupMember -Identity "Linux-Cert-Enrollers" -Members "evanusmodestus"

P50 Verification Commands

# Check domain membership
realm list

# Check if user resolves
id evanusmodestus@inside.domusdigitalis.dev

# Check PAM auth (currently broken)
su - evanusmodestus@inside.domusdigitalis.dev

Already Completed (from 01-21)

  • Created users: gabriel, evanusmodestus

  • Added to Linux-Cert-Enrollers

  • Domain joined P50

Still Missing

  • Linux-Research-Devices group (for login restriction)

  • Linux-Admins group (for sudo)

  • OU structure (groups may be in default location)

Unsorted notes - 2026-01-22

2026-01-22 07:58 - hyprland.conf error - RESOLVED

Issue: Config errors after reboot at lines 290-294 for blur and ignorezero fields.

Root Cause: Hyprland v0.52+ changed layerrule syntax. The old format:

layerrule = blur, waybar
layerrule = ignorezero, waybar

No longer valid. New syntax requires explicit values and match:namespace.

Resolution: Updated /home/evanusmodestus/atelier/_projects/personal/dotfiles-optimus/wm/hyprland/.config/hypr/hyprland.conf:

layerrule = blur on, ignore_alpha 1, match:namespace waybar
layerrule = blur on, ignore_alpha 1, match:namespace wofi
layerrule = blur on, ignore_alpha 1, match:namespace notifications

Reference:


Today’s Session - 2026-01-22 Morning

Completed

  • Fixed Hyprland layerrule syntax for v0.52+ compatibility

  • Reloaded Hyprland config (hyprctl reload)

  • Verified NFS/Borg backup issue documented (see Root Cause Analysis above)

  • Carried over reference material from WRKX-2026-01-21

  • Borg backup to Synology NAS completed

  • Created PRJ-RECOVERY Antora project (27 files, 14 pages, 5 Mermaid diagrams)


Borg Backup - 2026-01-22 08:39

Archive: arch-unknown-2026-01-22_083903

Metric Value

Duration

14.55 seconds

Files

291,141

Original Size

35.72 GB

Compressed

29.80 GB

Deduplicated (new data)

380.05 MB

Repository Totals: | Metric | Value | |——–|——-| | All Archives (original) | 462.97 GB | | All Archives (stored) | 41.03 GB | | Unique Chunks | 405,999 | | Total Chunks | 8,431,371 |

Verification: atelier directory confirmed present in archive.

Note: Fast backup (14s) is expected - Borg deduplication means only 380 MB of new/changed data was transferred. Previous backup data already in repo.


Post-Reboot TODO (From 01-21)

  1. Load NFS module: sudo modprobe nfs && echo "nfs" | sudo tee /etc/modules-load.d/nfs.conf

  2. Test NFS mount: sudo mount -t nfs nas-01.inside.domusdigitalis.dev:/volume1/borg_backups /mnt/synology

  3. Push to Gitea: git push gitea main

  4. Check QRadar for Mauricio’s suspicious endpoint

  5. Continue to PRJ-ISE-CHLA-LINUX for Dr. Shahab


P50 EAP-TLS Network Configuration Summary (From 01-21)

Final Network Stack (Clean)

Layer Tool Purpose Status

802.1X Auth

wpa_supplicant

Wired + WiFi EAP-TLS

enabled

DHCP

dhcpcd

IP assignment both interfaces

enabled

WiFi Driver

iwlwifi/iwlmvm

Intel hardware

loaded

iwd

(disabled)

Conflicted with wpa_supplicant

disabled

Enabled Services

systemctl enable wpa_supplicant-wired@enp0s31f6  # Wired 802.1X
systemctl enable wpa_supplicant-wifi@wlan0       # WiFi 802.1X
systemctl enable dhcpcd                          # DHCP for all interfaces
systemctl disable iwd                            # Disabled - conflicts with wpa_supplicant

Certificate Details

File Path Permissions

CA Certificate

/etc/ssl/certs/HOME-ROOT-CA.pem

0644

Client Certificate

/etc/ssl/certs/modestus-p50-eaptls.pem

0644

Private Key

/etc/ssl/private/modestus-p50-eaptls.key

0600 root:root

Identity

modestus-p50.inside.domusdigitalis.dev

Matches cert CN

Troubleshooting Commands

# Interface Recovery (when wlan0 disappears)
modprobe -r iwlmvm iwlwifi   # Unload Intel WiFi drivers
modprobe iwlwifi             # Reload drivers
ip link                      # Verify wlan0 returns

# DHCP Recovery
pkill -9 dhcpcd              # Kill all dhcpcd processes
dhcpcd -d -B wlan0           # Run foreground with debug

# Status Verification
wpa_cli -i wlan0 status      # Check auth state (should show COMPLETED)
ip addr show wlan0           # Check IP assignment
systemctl is-enabled iwd     # Confirm iwd disabled

Key Lessons for Dr. Shahab Deployment

  1. iwd and wpa_supplicant cannot coexist for WiFi - choose one

  2. wpa_supplicant is required for 802.1X - iwd doesn’t support enterprise auth well

  3. Consistent tooling - same tool (wpa_supplicant) handles both wired and WiFi EAP-TLS

  4. dhcpcd handles DHCP - separate concern from authentication

  5. Driver reload trick - modprobe -r/-i recovers interface when it disappears


NetAPI Quick Reference (From 01-21)

ISE MnT

uv run netapi ise mnt sessions
uv run netapi ise mnt sessions --details
uv run netapi ise mnt session <MAC>
uv run netapi ise mnt count
uv run netapi ise mnt version
uv run netapi ise mnt failed

ISE DataConnect (Crown Jewel)

uv run netapi ise dc test
uv run netapi ise dc stats
uv run netapi ise dc session C8:5B:76:C6:59:62  # Full session view
uv run netapi ise dc recent --hours 4 --limit 50
uv run netapi ise dc failed --hours 48

WLC

uv run netapi wlc get-info
uv run netapi wlc get-health
uv run netapi wlc get-aps
uv run netapi wlc get-clients
uv run netapi wlc get-wlans

IOS

uv run netapi ios show radius-servers
uv run netapi ios show aaa-groups
uv run netapi ios show vlans
uv run netapi ios exec "show access-session"

Quick One-Liners

# ISE MnT
declare -a c=('ise mnt sessions' 'ise mnt count' 'ise mnt version'); for i in "${c[@]}"; do echo "--- ${i} ---"; bash -c "uv run netapi ${i}"; done

# WLC
declare -a c=('wlc get-info' 'wlc get-health' 'wlc get-aps' 'wlc get-clients' 'wlc get-wlans'); for i in "${c[@]}"; do echo "--- ${i} ---"; bash -c "uv run netapi ${i}"; done

Bash Loop Cheat Sheet (From 01-21)

Pattern Use Case

for x in a b c; do cmd "$x"; done

Simple inline items

arr=(a b c); for x in "${arr[@]}"; do ...

Items with spaces

cmd --format json | jq ... | while read

Dynamic from API

bash -c "cmd \"$x\""

Complex quoting

Practical Examples

# All policy sets with full details
for ps in "Corp WIFI" "Wired Dot1X Closed" "Guest WIFI" "IoT WIFI"; do
    uv run netapi ise get-policy-set "$ps" --rules
done

# Audit all authz profiles
for profile in Domus_Secure_Profile iPSK-Auth DenyAccess PermitAccess; do
    uv run netapi ise get-authz-profile "$profile"
done

# Check multiple endpoints
for mac in "70:15:FB:F8:47:EC" "14:F6:D8:7B:31:80"; do
    uv run netapi ise get-endpoint "$mac"
done

ISE to Defender/Sentinel Integration Research (From 01-21)

ISE Data Available for SIEM Integration

Method Data Type Use Case

Syslog

Auth pass/fail, posture, profiling

Standard integration

pxGrid

Active sessions, SGT, real-time context

Advanced correlation

REST API

Endpoint inventory, policy violations

On-demand queries

High-Value Correlation Use Cases

ISE Event Defender/Sentinel Use Case

Auth failure spikes

Brute force detection

Posture fail → quarantine

Compromised device isolation

New device profiled

Shadow IT detection

SGT change

Lateral movement tracking

Session from unusual location

Impossible travel


Full Reference

For complete NetAPI command reference, advanced power scripts, and detailed P50 troubleshooting sessions, see:

  • WRKX-2026-01-21-020-chla-daily-work-log.md (2,894 lines)

Unsorted Ad-Hoc notes

2026-01-22 11:14 - assist Anthony Martinez

assess and provide advice on obsidian vault and data backup procedures from from a holistic perspective to inlclude:

2026-01-22 12:30 - Ad-Hoc: Isensix DHCP Issue (Jenny Huang)

Issue Summary

Isensix devices authenticating via ISE but not receiving IP addresses.

Field Value

Contact

Jenny Huang

MAC Address

00:1C:2C:61:90:06

NAS IP

10.134.144.109

Switch/Port

SRT-9_9300 / Gi2/0/4

VLAN

729 (CHLA-IoT)

Auth Status

MAB Success

dACL Applied

xACSACLx-IP-TEST_Medical_Temp_Monitor-68fa6c22

Diagnostic Commands

MAC Table:

SRT-9_9300#show mac address-table address 00:1C:2C:61:90:06
Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
 729    001c.2c61.9006    STATIC      Gi2/0/4

Access Session:

SRT-9_9300#show access-session mac 00:1C:2C:61:90:06 d
            Interface:  GigabitEthernet2/0/4
          MAC Address:  001c.2c61.9006
         IPv4 Address:  Unknown              ← NO IP!
               Status:  Authorized
       Current Policy:  PMAP_DefaultWiredDot1xClosedAuth_1X_MAB_V2

Server Policies:
           Vlan Group:  Vlan: 729
              ACS ACL: xACSACLx-IP-TEST_Medical_Temp_Monitor-68fa6c22

Method status list:
       Method           State
        dot1x           Stopped
          mab           Authc Success

VLAN Verification:

SRT-9_9300#show vlan id 729
VLAN Name                             Status    Ports
---- -------------------------------- --------- -------------------------------
729  CHLA-IoT                         active    Gi1/0/36, Gi1/0/38, Te1/1/1, Gi2/0/4, Te2/1/1

Root Cause: dACL Missing DHCP Permit

Current ACL (broken):

Extended IP access list xACSACLx-IP-TEST_Medical_Temp_Monitor-68fa6c22
    1 permit tcp any eq 6001 host 10.192.220.51
    2 permit tcp any eq 6001 host 10.192.220.52
    3 permit tcp any eq 6001 host 10.192.220.64
    4 permit tcp any eq 6001 host 10.192.220.65
    6 permit icmp any any
    8 deny ip any any     ← BLOCKS UDP 67/68 (DHCP)!

Resolution: Add bootpc/bootps to ISE dACL

ISE Path: Policy → Results → Authorization → Downloadable ACLs → TEST_Medical_Temp_Monitor

Corrected ACL:

permit udp any eq bootpc any eq bootps
permit tcp any eq 6001 host 10.192.220.51
permit tcp any eq 6001 host 10.192.220.52
permit tcp any eq 6001 host 10.192.220.64
permit tcp any eq 6001 host 10.192.220.65
permit icmp any any
deny ip any any

Port Reference: | Port | Name | Description | |——|——|————-| | UDP 67 | bootps | DHCP Server | | UDP 68 | bootpc | DHCP Client |

Note: Only one direction needed - dACL is ingress on client port. Client sends from port 68 to server port 67.

Post-Fix Verification

# Force reauth on switch
clear access-session mac 001c.2c61.9006

# Verify IP assigned
show access-session mac 00:1C:2C:61:90:06 d | include IPv4

Status

  • Root cause identified: dACL missing DHCP permit

  • Update dACL in ISE (Jenny/Network team)

  • Verify device gets IP after CoA

2026-01-22 14:10 Mind DLP by mind.io

  • Are there any concerns with respect to this solution and the following:

    • yaml frontmatter properties such as UUID and tags

    • git push and pull

    • encryption used for files and folders such as cryptomater, age, gocryptfs, etc.

2026-01-22 13:08 - P50 PAM/SSSD Troubleshooting Session

Issue Summary

AD user gabriel@inside.domusdigitalis.dev cannot authenticate via su despite SSSD being properly configured for domain join.

Environment Status

Component Status Notes

SSSD Service

Running

sssd_be, sssd_nss, sssd_pam, sssd_pac all active

User Resolution

Working

id gabriel@inside.domusdigitalis.dev resolves correctly

PAM Socket

Exists

/var/lib/sss/pipes/pam present

pam_sss.so

Exists

/usr/lib/security/pam_sss.so present

Troubleshooting Timeline

14:39 - Initial Hypothesis: forward_pass in sssd-arch

Theory: forward_pass requires password from previous module, but sssd-arch is first in chain.

Fix Applied:

sudo sed -i 's/pam_sss.so forward_pass/pam_sss.so/' /etc/pam.d/sssd-arch

Result: Still failing at 14:54

14:58 - Key Discovery: pam_sss.so NOT Being Called

Evidence from journalctl:

pam_unix(su-l:auth): authentication failure... user=gabriel@inside.domusdigitalis.dev

pam_unix is doing auth, not pam_sss.so. SSSD PAM log shows NO auth attempts.

15:03 - Root Cause Identified: /etc/pam.d/su Misconfiguration

Original /etc/pam.d/su (broken):

account         required        pam_unix.so    ← Doesn't know AD users!
session         required        pam_unix.so    ← Doesn't know AD users!

SSSD Auth Test - SUCCESS:

sudo sssctl user-checks gabriel@inside.domusdigitalis.dev -a auth
# Result: pam_authenticate for user [gabriel@inside.domusdigitalis.dev]: Success

This confirms SSSD CAN authenticate - the problem is PAM routing.

15:08 - Fix Attempt (WITH TYPO)

Incorrect fix applied:

account         include        pam_unix.so    ← WRONG! Should be system-auth
session         include        pam_unix.so    ← WRONG! Should be system-auth

Error: include takes a FILE name, not a module name. pam_unix.so is not a valid PAM config file.

Correct Fix Required

/etc/pam.d/su should be:

#%PAM-1.0
auth            sufficient      pam_rootok.so
auth            include         system-auth
account         include         system-auth
session         include         system-auth
password        include         system-auth

Current Status

  • SSSD service running

  • User resolves via NSS

  • SSSD can authenticate user (sssctl user-checks SUCCESS)

  • sssd-arch forward_pass removed

  • FIX /etc/pam.d/su - change include pam_unix.so to include system-auth

  • Test su - gabriel@inside.domusdigitalis.dev

Files Reference

File Purpose

/etc/pam.d/system-auth

Main PAM stack (includes sssd-arch)

/etc/pam.d/sssd-arch

SSSD PAM module config

/etc/pam.d/su

su command PAM config (NEEDS FIX)

/var/log/sssd/sssd_pam.log

SSSD PAM debug log

2026-01-22 15:23 - issue still not resovled despite the previous assessment implemented

[I] ❯ cat /etc/pam.d/su | head -10
#%PAM-1.0
auth            sufficient      pam_rootok.so
# Uncomment the following line to implicitly trust users in the "wheel" group.
#auth           sufficient      pam_wheel.so trust use_uid
# Uncomment the following line to require a user to be in the "wheel" group.
#auth           include        pam_wheel.so use_uid
auth            include        system-auth
account         include        system-auth
session         include        system-auth
password        include        system-auth



[I] ❯ su - gabriel@inside.domusdigitalis.dev
Password:
su: Authentication failure
[I]  gabriel@modestus-p50   ~   6s 272ms  ✗
  sudo tee /etc/pam.d/sssd-arch << 'EOF'
  auth sufficient pam_sss.so
  account sufficient pam_sss.so
  password sufficient pam_sss.so use_authtok
  session required pam_mkhomedir.so skel=/etc/skel/ umask=0077
  session optional pam_sss.so
  EOF

  Then test:
  su - gabriel@inside.domusdigitalis.dev
● Still showing pam_unix(su-l:auth) - pam_sss still not being called. Let me check system-auth:

● Bash(ssh modestus-p50 "cat /etc/pam.d/system-auth")
  ⎿  #%PAM-1.0
     auth include sssd-arch
     account include sssd-arch
     … +28 lines (ctrl+o to expand)

● Wait - look at the log. For su, only pam_unix shows up. But sudo DID call pam_sss. Let me check /etc/pam.d/su:

● Bash(ssh modestus-p50 "cat /etc/pam.d/su")
  ⎿  #%PAM-1.0
     auth            sufficient      pam_rootok.so
     # Uncomment the following line to implicitly trust users in the "wheel" group.
     … +7 lines (ctrl+o to expand)

● Bash(ssh modestus-p50 "cat /etc/pam.d/sssd-arch && echo '---' && sudo tail -20 /var/log/sssd/sssd_pam.log
      2>/dev/null")
  ⎿  Error: Exit code 1
       auth sufficient pam_sss.so
       account sufficient pam_sss.so
       password sufficient pam_sss.so use_authtok
       session required pam_mkhomedir.so skel=/etc/skel/ umask=0077
       session optional pam_sss.so

     ---

● Found it! sssd-arch has LEADING SPACES on each line - PAM might not parse it correctly:

    auth sufficient pam_sss.so      ← LEADING SPACES!

  Let me fix it:

● Bash(ssh modestus-p50 "cat /etc/pam.d/sssd-arch | od -c | head -5")
  ⎿  0000000           a   u   t   h       s   u   f   f   i   c   i   e   n
     0000020   t       p   a   m   _   s   s   s   .   s   o  \n           a
     0000040   c   c   o   u   n   t       s   u   f   f   i   c   i   e   n
     … +2 lines (ctrl+o to expand)

● Confirmed - 2 leading spaces on each line. PAM won't parse this. Fixing:

● Bash(ssh modestus-p50 "sed -i 's/^  //' /etc/pam.d/sssd-arch && cat /etc/pam.d/sssd-arch")
  ⎿  Error: Exit code 4
     sed: couldn't open temporary file /etc/pam.d/sed0EoyR8: Permission denied

● Need sudo. On P50 run:

  sudo tee /etc/pam.d/sssd-arch << 'EOF'
  auth sufficient pam_sss.so
  account sufficient pam_sss.so
  password sufficient pam_sss.so use_authtok
  session required pam_mkhomedir.so skel=/etc/skel/ umask=0077
  session optional pam_sss.so
  EOF

  Make sure there are NO leading spaces - copy exactly as shown.

  Then test:
  su - gabriel@inside.domusdigitalis.dev


> EOF
  auth sufficient pam_sss.so
  account sufficient pam_sss.so
  password sufficient pam_sss.so use_authtok
  session required pam_mkhomedir.so skel=/etc/skel/ umask=0077
  session optional pam_sss.so

❯   su - gabriel@inside.domusdigitalis.dev

Password:
su: Authentication failure
 gabriel@modestus-p50   ~   9s 544ms  ✗
❯

15:35 - Fix Leading Spaces in sssd-arch

Issue: heredoc was adding leading spaces to sssd-arch file, causing PAM to not parse it.

Fix Applied:

echo -e "auth sufficient pam_sss.so\naccount sufficient pam_sss.so\npassword sufficient pam_sss.so use_authtok\nsession required pam_mkhomedir.so skel=/etc/skel/ umask=0077\nsession optional pam_sss.so" | sudo tee /etc/pam.d/sssd-arch

Verification - no leading spaces:

❯ cat /etc/pam.d/sssd-arch | head -1 | od -c | head -1
0000000   a   u   t   h       s   u   f   f   i   c   i   e   n   t

Result: Still failing at 15:37


15:40 - ROOT CAUSE FOUND: su-l vs su

Discovery: su - (login shell) uses /etc/pam.d/su-l, NOT /etc/pam.d/su!

❯ ls -la /etc/pam.d/su*
-rw-r--r-- 1 root root 495 Jan 22 15:23 /etc/pam.d/su
-rw-r--r-- 1 root root 500 Dec 15 07:26 /etc/pam.d/su-l   ← THIS IS THE PROBLEM!

/etc/pam.d/su-l (broken - hardcoded pam_unix):

#%PAM-1.0
auth            sufficient      pam_rootok.so
auth            required        pam_unix.so      ← NOT using system-auth!
account         required        pam_unix.so      ← NOT using system-auth!
session         required        pam_unix.so      ← NOT using system-auth!
password        include         system-auth

Fix Required:

echo -e "#%PAM-1.0\nauth            sufficient      pam_rootok.so\nauth            include         system-auth\naccount         include         system-auth\nsession         include         system-auth\npassword        include         system-auth" | sudo tee /etc/pam.d/su-l

Status

  • Apply su-l fix

  • Test su - gabriel@inside.domusdigitalis.dev

  • Successful test conducted

❯ su - gabriel@inside.domusdigitalis.dev
Password:
[gabriel@inside.domusdigitalis.dev@modestus-p50 ~]$

Summary: P50 PAM/SSSD AD Authentication - RESOLVED

Root Cause: su - (login shell) uses /etc/pam.d/su-l, which was hardcoded to pam_unix.so instead of including system-auth.

Key Lessons Learned:

  1. su vs su-l: su uses /etc/pam.d/su, but su - uses /etc/pam.d/su-l - different files!

  2. PAM include syntax: include takes a FILE name, not a module (include system-auth NOT include pam_unix.so)

  3. forward_pass: Requires password from previous module - don’t use if pam_sss.so is first in chain

  4. Leading spaces: PAM configs must NOT have leading whitespace - use od -c to verify

  5. sssd-arch needs all 4 stacks: auth, account, password, session - missing account causes failures

  6. sssctl user-checks: Best tool to verify SSSD can authenticate independently of PAM

Files Modified:

File Change

/etc/pam.d/sssd-arch

Added account sufficient pam_sss.so, removed forward_pass

/etc/pam.d/su

Changed to include system-auth for all stacks

/etc/pam.d/su-l

Changed to include system-auth for all stacks (THE FIX!)

Arsenal Reference: ARS-SYS-003-pam-sssd-ad-authentication.md

2026-01-22 - netapi ISE Posture Condition Commands Implementation

Summary

Implemented network access condition management in netapi ise CLI, enabling programmatic creation and management of ISE posture conditions.

New CLI Commands

Command Description

netapi ise get-conditions

List all network access conditions

netapi ise get-conditions --dict Session

Filter by dictionary (posture conditions)

netapi ise get-condition <name>

Get specific condition details

netapi ise create-posture-condition

Create posture condition (Session dict)

netapi ise create-condition

Create generic condition (any dictionary)

netapi ise delete-condition

Delete condition by name

netapi ise get-dictionaries

List available ISE dictionaries

netapi ise get-dictionary <name>

Get dictionary attributes

Key Discovery: Posture Dictionary Structure

ISE stores posture conditions in the Session dictionary, NOT a “Posture” dictionary:

Dictionary: Session
Attribute:  PostureStatus
Values:     Compliant, NonCompliant, Unknown

Example Usage

# Load credentials
dsource d000 dev/network

# List posture conditions
netapi ise get-conditions --dict Session

# Create compliant check
netapi ise create-posture-condition "My_Compliant_Check" \
    --attr PostureStatus --value Compliant

# Verify creation
netapi ise get-conditions --dict Session

# Clean up
netapi ise delete-condition "My_Compliant_Check"

Files Modified

File Changes

netapi/vendors/cisco/ise/ers_client.py

Added 9 new methods for condition management

netapi/cli/ise.py

Added 7 new CLI commands

PRJ-ISE-HOME-LINUX/.../posture-conditions.adoc

Updated netapi automation section

PRJ-ISE-HOME-LINUX/.../commands.adoc

Added netapi ISE commands reference

Testing Results

✓ get-conditions - Lists all 20 conditions
✓ get-conditions --dict Session - Filters to 3 posture conditions
✓ create-posture-condition - Created successfully
✓ delete-condition - Deleted with confirmation
✓ Output formats: table, json, yaml all working

Bug Fixes During Implementation

  1. NameError: set_output_format - Used non-existent function; fixed to use global _output_format pattern

  2. Wrong dictionary name - Initially used “Posture” but ISE uses “Session” for PostureStatus

  3. Delete condition ID lookup - get_by_name response structure different from list; switched to searching list endpoint


2026-01-22 - dACL as Logical Segmentation (North-South vs East-West)

Segmentation Model

Direction Description Control Point

North ↔ South

Traffic in/out of network perimeter

Perimeter firewall

East ↔ West

Lateral movement inside network

dACL at access layer

What the 802.1X + dACL Implementation Provides

Primary Benefit: East-West Microsegmentation

                    INTERNET
                        │
                   [ Firewall ]  ← North-South control
                        │
              ┌─────────┴─────────┐
              │    Internal Net   │
              │                   │
    [Server A]  ←───────────────→  [Server B]
         ↑            ↑                ↑
         │      East-West              │
         │      movement               │
         └──────── [Linux WS] ────────┘
                      │
                   [ dACL ]  ← Your control point
                      │
                 "You can only
                  reach Server A,
                  not Server B"

Comparison to Traditional Segmentation

Method Granularity Enforcement Point

VLANs

Per broadcast domain

Switch

Firewall rules

Per subnet

Firewall

dACLs (this implementation)

Per device, per session

Switch port

Security Value

  • Two machines on the same VLAN, same switch → dACL gives them different access based on identity

  • Attacker compromises workstation → dACL limits lateral movement

  • This is identity-based microsegmentation at the access layer

  • Fills the gap most organizations leave wide open (flat internal networks)

Terminology for Documentation

“Logical segmentation via 802.1X with dynamic ACL enforcement at the access layer”

This is defense in depth - enforcement at the endpoint’s first hop, not just at the perimeter.


2026-01-22 18:11:55 - Linux testing progress

Contents:

- Privilege Model - Domain users vs Local admins vs Root
- su vs su-l Distinction - PAM config differences explained
- Sudoers Configuration - Zero-trust template
- Security Controls Validation Matrix - LUKS, UFW, ClamAV, 802.1X
- Full Validation Output - gabriel (denied) vs root (allowed)
- Architecture Diagram - PlantUML showing privilege flow
- Troubleshooting - Common PAM issues

PDFs Rebuilt:
✓ linux-802.1x-guide.pdf (1.5M) - Dark
✓ linux-802.1x-guide-light.pdf (1.5M) - Light

---

Summary for Sarah/InfoSec:

[cols="1,1"]
|===
| Control | P50 Status

| LUKS Full Disk Encryption
| ✓ All partitions

| UFW Firewall
| ✓ Active, deny incoming

| ClamAV Antivirus
| ✓ Running

| 802.1X EAP-TLS
| ✓ SUCCESS

| Domain User sudo
| ✓ DENIED
|===

Unsorted notes of the day

Linux Workstation Project - Team & Responsibilities - 2026-01-23 07:58

Core Team
Name Department Title Project Role

Sarah Clizer

IS - Information Security

Director / Deputy CISO

Approver, Project Sponsor

Evan Rosado

IS - Information Security

Information Security Engineer III

ISE Configuration, Security Validation

Ben Castillo

IS - Cloud & Hybrid Services

Systems Engineer III

Linux Workstation Configuration

Victor Negri

IS - Cloud & Hybrid Services

CloudOps Engineer III

AD / PKI / Certificate Infrastructure

Responsibility Matrix (RACI)
Phase Owner Deliverables Status

1. Linux Workstation

Ben Castillo

LUKS, Defender, UFW, SSSD, sudoers, cert install

PENDING

2. Certificate Infrastructure

Victor Negri

AD groups, cert template, machine cert issuance

PENDING

3. ISE Configuration

Evan Rosado

dACL, authz profile, policy rules

PENDING

4. Validation & Sign-off

Evan Rosado

Security controls verification

PENDING

5. Approval

Sarah Clizer

Final sign-off

PENDING

End Users & Stakeholders
Name Department Role Involvement

Dr. Shahab Asgharzadeh

Research

Researcher

Primary End User

Xiangming Ding

Research

Secondary End User

Network Team

IS - Network Infrastructure

Switch/VLAN config (COMPLETE)

Additional Contacts (TBD - confirm today)
Name Department Title Involvement

unsorted notes

Insensix devices connectivity issues - 2026-01-23 09:00

[teams message request]

GM, I've AP down in Duq 2nd Floor, IP:
IP address: 10.238.1.27
MAC address: 00:50:C2:39:F1:4B
NAS IP: 10.193.144.124
NAS Port: GigabitEthernet4/0/37
#[switch logs]

DUQUE-2-1-9300>en
Password:
DUQUE-2-1-9300#show mac add add 00:50:C2:39:F1:4B
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
DUQUE-2-1-9300#show interf
DUQUE-2-1-9300#show interfaces GigabitEthernet4/0/37 status

Port         Name               Status       Vlan       Duplex  Speed Type
Gi4/0/37     User               connected    651        a-full  a-100 10/100/1000BaseTX
DUQUE-2-1-9300#show access-sessio
DUQUE-2-1-9300#show access-session int
DUQUE-2-1-9300#show access-session interface GigabitEthernet4/0/37 de
DUQUE-2-1-9300#show access-session interface GigabitEthernet4/0/37 details
No sessions match supplied criteria.

DUQUE-2-1-9300#show run int
DUQUE-2-1-9300#show run interface GigabitEthernet4/0/37
Building configuration...

Current configuration : 467 bytes
!
interface GigabitEthernet4/0/37
 description User
 switchport access vlan 651
 switchport mode access
 switchport nonegotiate
 switchport voice vlan 424
 device-tracking attach-policy DT_POLICY
 ip flow monitor IPv4_NETFLOW input
 dot1x timeout tx-period 7
 dot1x max-reauth-req 3
 source template DefaultWiredDot1xClosedAuth
 spanning-tree portfast
 spanning-tree bpduguard enable
 service-policy input QoS_Edge_Ingress
 service-policy output QoS_Edge_Egress
end

DUQUE-2-1-9300#show vlan id 651

VLAN Name                             Status    Ports
---- -------------------------------- --------- -------------------------------
651  CHLA-Medical-Devices             active    Gi1/0/6, Gi1/0/33, Gi1/0/43, Te1/1/1
                                                Gi2/0/9, Gi2/0/28, Te2/1/1, Gi3/0/8
                                                Gi4/0/11, Gi4/0/15, Gi4/0/32, Gi4/0/33
                                                Gi4/0/37, Gi5/0/4, Gi5/0/11, Gi5/0/15
                                                Gi5/0/24

VLAN Type  SAID       MTU   Parent RingNo BridgeNo Stp  BrdgMode Trans1 Trans2
---- ----- ---------- ----- ------ ------ -------- ---- -------- ------ ------
651  enet  100651     1500  -      -      -        -    -        0      0

Remote SPAN VLAN
----------------
Disabled

Primary Secondary Type              Ports
------- --------- ----------------- ------------------------------------------

DUQUE-2-1-9300#


DUQUE-2-1-9300##[devices in wrong vlan] should be IOT
DUQUE-2-1-9300#show vlan id 751

VLAN Name                             Status    Ports
---- -------------------------------- --------- -------------------------------
751  CHLA-IoT                         active    Gi1/0/40, Te1/1/1, Gi2/0/23, Gi2/0/24
                                                Gi2/0/37, Gi2/0/48, Te2/1/1, Gi3/0/31
                                                Gi3/0/36, Gi3/0/39, Gi3/0/48, Gi4/0/17
                                                Gi5/0/25

VLAN Type  SAID       MTU   Parent RingNo BridgeNo Stp  BrdgMode Trans1 Trans2
---- ----- ---------- ----- ------ ------ -------- ---- -------- ------ ------
751  enet  100751     1500  -      -      -        -    -        0      0

Remote SPAN VLAN
----------------
Disabled

Primary Secondary Type              Ports
------- --------- ----------------- ------------------------------------------

DUQUE-2-1-9300#


# add device to connect VLAN
conf t
interface GigabitEthernet4/0/37
switchport access vlan 751
exit
exit

# validate VLAN port assignment
show vlan id 751 | inc Gi4/0/37


# check for and access session
show access-session interface GigabitEthernet4/0/37 detail

DUQUE-2-1-9300#show access-session interface GigabitEthernet4/0/37 detail
            Interface:  GigabitEthernet4/0/37
               IIF-ID:  0x1DB3CCD2
          MAC Address:  0050.c239.f14b
         IPv6 Address:  Unknown
         IPv4 Address:  10.238.1.27
            User-Name:  00-50-C2-39-F1-4B
               Status:  Authorized
               Domain:  DATA
       Oper host mode:  multi-auth
     Oper control dir:  in
      Session timeout:  N/A
  Acct update timeout:  172800s (local), Remaining: 172719s
    Common Session ID:  7C90C10A000091B9EBD6F815
      Acct Session ID:  0x0000a9dd
               Handle:  0x5500083c
       Current Policy:  PMAP_DefaultWiredDot1xClosedAuth_1X_MAB


Server Policies:
           Vlan Group:  Vlan: 751
              ACS ACL: xACSACLx-IP-TEST_Medical_Temp_Monitor-697290bb


Method status list:
       Method           State
        dot1x           Stopped
          mab           Authc Success

DUQUE-2-1-9300#

# validate device responds to ping
ping -c 2 10.238.1.27
PING 10.238.1.27 (10.238.1.27) 56(84) bytes of data.
64 bytes from 10.238.1.27: icmp_seq=1 ttl=58 time=9.84 ms
64 bytes from 10.238.1.27: icmp_seq=2 ttl=58 time=18.8 ms
--- 10.238.1.27 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 9.837/14.338/18.840/4.501 ms


## [bonus netapi]

 uv run netapi ise mnt session 00-50-C2-39-F1-4B
╭───────────────────────────── ISE Session Details ─────────────────────────────╮
│ Session: 00-50-C2-39-F1-4B  |  Status: PASSED                                 │
╰───────────────────────────────────────────────────────────────────────────────╯
                      Authentication
  Username                  00:50:C2:39:F1:4B
  Method                    mab
  Protocol                  Lookup
  Timestamp                 2026-01-23T09:12:01.168-08:00
  ISE Node                  psn-1
  Session ID                7C90C10A000091B9EBD6F815

                       Network
  Client MAC                00:50:C2:39:F1:4B
  Client IPv4               10.238.1.27
  NAD IP                    10.193.144.124
  NAD Name                  DUQUE-2-1-9300
  Port                      GigabitEthernet4/0/37
  Device Type               All Device Types#SWITCH

                 Accounting
  Session ID                0000a9dd
  Status                    Interim-Update
  RX Bytes                  128
  TX Bytes                  0

                                 Endpoint
  Profiled As               medigate_Guardian_Isensix_Temperature_Sensor

working on second device

IP address: 10.238.1.29
MAC address: 00:50:C2:39:F0:D3
NAS IP: 10.193.144.124
NAS Port: GigabitEthernet5/0/15
## [terminal session clear] clear terminal session to copy entirety of console avoiding other output
clear
## [NAS Switch]  ssh into NAS
sudo ssh erosado@10.193.144.124


## [MAC Table] check MAC table for endpoint
show mac address address 00:50:C2:39:F0:D3
## [NAS PORT] check last known port that ISE logs indicate
show interfaces GigabitEthernet5/0/15 status
## [access session] check for an access session by MAC address
show access-session mac 00:50:C2:39:F0:D3 detail
## [access session] check for an access session by interface
show access-session interface GigabitEthernet5/0/15 detail
## [nas port config] check interface port configuration
show run interface GigabitEthernet5/0/15
# [command output] notice my syntax error that i corrected

[sudo] password for evanusmodestus:
** WARNING: connection is not using a post-quantum key exchange algorithm.
** This session may be vulnerable to "store now, decrypt later" attacks.
** The server may need to be upgraded. See https://openssh.com/pq.html

*****************************************************************
* NOTICE  NOTICE            S T O P              NOTICE  NOTICE *
*                   THIS IS A PRIVATE SYSTEM.                   *
*      AUTHORIZATION IS REQUIRED TO CONNECT TO THIS DEVICE.     *
* ACTUAL OR ATTEMPTED USE, ACCESS, EXAMINATION OR CONFIGURATION *
* CHANGE BY ANY UNAUTHORIZED PERSON WILL RESULT IN CRIMINAL AND *
*     CIVIL PROSECUTION TO THE FULLEST EXTENT OF THE LAW.       *
*                  N    O    T    I    C    E                   *
*****************************************************************
(erosado@10.193.144.124) Password:
DUQUE-2-1-9300>en
Password:
DUQUE-2-1-9300### [MAC Table] check MAC table for endpoint
DUQUE-2-1-9300#show mac address address 00:50:C2:39:F0:D3
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
DUQUE-2-1-9300#$T] check last known port that ISE logs indicate
DUQUE-2-1-9300#show interfaces GigabitEthernet5/0/15 status

Port         Name               Status       Vlan       Duplex  Speed Type
Gi5/0/15     User               connected    651        a-full  a-100 10/100/1000BaseTX
DUQUE-2-1-9300#$heck for an access session by MAC address
DUQUE-2-1-9300#show access-session mac 00:50:C2:39:F0:D3 detail
No sessions match supplied criteria.

DUQUE-2-1-9300#$session] check for an access session by interface
DUQUE-2-1-9300#$-session inteface GigabitEthernet5/0/15 detail
show access-session inteface GigabitEthernet5/0/15 detail
                        ^
% Invalid input detected at '^' marker.

DUQUE-2-1-9300#$t config] check interface port configuration
DUQUE-2-1-9300#show run interface GigabitEthernet5/0/15
Building configuration...

Current configuration : 416 bytes
!
interface GigabitEthernet5/0/15
 description User
 switchport access vlan 651
 switchport mode access
 switchport nonegotiate
 switchport voice vlan 424
 device-tracking attach-policy DT_POLICY
 ip flow monitor IPv4_NETFLOW input
 source template DefaultWiredDot1xClosedAuth
 spanning-tree portfast
 spanning-tree bpduguard enable
 service-policy input QoS_Edge_Ingress
 service-policy output QoS_Edge_Egress
end

DUQUE-2-1-9300#
DUQUE-2-1-9300#
DUQUE-2-1-9300#$-session interface GigabitEthernet5/0/15 detail
No sessions match supplied criteria.

DUQUE-2-1-9300#

[VLAN] VLAN assignment is IOT

show vlan id 651

## [command output] notice incorrect vlan assigned



DUQUE-2-1-9300#show vlan id 651

VLAN Name                             Status    Ports
---- -------------------------------- --------- -------------------------------
651  CHLA-Medical-Devices             active    Gi1/0/6, Gi1/0/33
                                                Gi1/0/43, Te1/1/1
                                                Gi2/0/9, Gi2/0/28
                                                Te2/1/1, Gi3/0/8
                                                Gi4/0/11, Gi4/0/15
                                                Gi4/0/32, Gi4/0/33
                                                Gi5/0/4, Gi5/0/11
                                                Gi5/0/15, Gi5/0/24

VLAN Type  SAID       MTU   Parent RingNo BridgeNo Stp  BrdgMode Trans1 Trans2
---- ----- ---------- ----- ------ ------ -------- ---- -------- ------ ------
651  enet  100651     1500  -      -      -        -    -        0      0

Remote SPAN VLAN
----------------
Disabled

Primary Secondary Type              Ports
------- --------- ----------------- ------------------------------------------

DUQUE-2-1-9300#
DUQUE-2-1-9300#


# add device to connect VLAN
conf t
interface GigabitEthernet5/0/15
switchport access vlan 751
exit
exit
# validate VLAN port assignment
show vlan id 751 | inc Gi5/0/15
# check for and access session
show access-session interface GigabitEthernet5/0/15 detail

## [command output]


DUQUE-2-1-9300#
DUQUE-2-1-9300## add device to connect VLAN
DUQUE-2-1-9300#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
DUQUE-2-1-9300(config)#interface GigabitEthernet5/0/15
DUQUE-2-1-9300(config-if)#switchport access vlan 751
DUQUE-2-1-9300(config-if)#exit
DUQUE-2-1-9300(config)#exit
DUQUE-2-1-9300## validate VLAN port assignment
DUQUE-2-1-9300#show vlan id 751 | inc Gi5/0/15
                                                Gi4/0/37, Gi5/0/15
DUQUE-2-1-9300## check for and access session
DUQUE-2-1-9300#$-session interface GigabitEthernet5/0/15 detail
No sessions match supplied criteria.

DUQUE-2-1-9300#


## [incorrect interface template] device not assigned correct dot1x c3p3 template
## which causes a delay in processing MAB until 30 seconds after dot1x fails


DUQUE-2-1-9300#$-session interface GigabitEthernet5/0/15 detail
            Interface:  GigabitEthernet5/0/15
               IIF-ID:  0x1307B59E
          MAC Address:  0050.c239.f0d3
         IPv6 Address:  Unknown
         IPv4 Address:  Unknown
               Status:  Unauthorized
               Domain:  UNKNOWN
       Oper host mode:  multi-auth
     Oper control dir:  in
      Session timeout:  N/A
    Common Session ID:  7C90C10A000091BEEBE9D034
      Acct Session ID:  Unknown
               Handle:  0x3d0009dc
       Current Policy:  PMAP_DefaultWiredDot1xClosedAuth_1X_MAB


Server Policies:


Method status list:
       Method           State
        dot1x           Running

DUQUE-2-1-9300#


## [device session found ]

DUQUE-2-1-9300#$-session interface GigabitEthernet5/0/15 detail
            Interface:  GigabitEthernet5/0/15
               IIF-ID:  0x1307B59E
          MAC Address:  0050.c239.f0d3
         IPv6 Address:  Unknown
         IPv4 Address:  10.238.1.29
            User-Name:  00-50-C2-39-F0-D3
               Status:  Authorized
               Domain:  DATA
       Oper host mode:  multi-auth
     Oper control dir:  in
      Session timeout:  N/A
  Acct update timeout:  172800s (local), Remaining: 172701s
    Common Session ID:  7C90C10A000091BEEBE9D034
      Acct Session ID:  0x0000a9eb
               Handle:  0x3d0009dc
       Current Policy:  PMAP_DefaultWiredDot1xClosedAuth_1X_MAB


Server Policies:
           Vlan Group:  Vlan: 751
              ACS ACL: xACSACLx-IP-TEST_Medical_Temp_Monitor-697290bb


Method status list:
       Method           State
        dot1x           Stopped
          mab           Authc Success


## [test device connectivity]

ping -c 3 10.238.1.29


## [command output]

ping -c 3 10.238.1.29
PING 10.238.1.29 (10.238.1.29) 56(84) bytes of data.
64 bytes from 10.238.1.29: icmp_seq=1 ttl=58 time=13.9 ms
64 bytes from 10.238.1.29: icmp_seq=2 ttl=58 time=11.2 ms
64 bytes from 10.238.1.29: icmp_seq=3 ttl=58 time=10.1 ms


## [bonus netapi check]

 uv run netapi ise mnt session 00-50-C2-39-F0-D3
╭────────────────────────────────── ISE Session Details ──────────────────────────────────╮
│ Session: 00-50-C2-39-F0-D3  |  Status: PASSED                                           │
╰─────────────────────────────────────────────────────────────────────────────────────────╯
                      Authentication
  Username                  00:50:C2:39:F0:D3
  Method                    mab
  Protocol                  Lookup
  Timestamp                 2026-01-23T09:32:36.144-08:00
  ISE Node                  psn-1
  Session ID                7C90C10A000091BEEBE9D034

                       Network
  Client MAC                00:50:C2:39:F0:D3
  Client IPv4               10.238.1.29
  NAD IP                    10.193.144.124
  NAD Name                  DUQUE-2-1-9300
  Port                      GigabitEthernet5/0/15
  Device Type               All Device Types#SWITCH

                 Accounting
  Session ID                0000a9eb
  Status                    Interim-Update
  RX Bytes                  64
  TX Bytes                  0

                                 Endpoint
  Profiled As               medigate_Guardian_Isensix_Temperature_Sensor
 evanusmodestus  󰙅 D001/DEV/NETWORK   netapi   main   3.13  ✓

2nd AP

IP address: 10.238.1.29 MAC address: 00:50:C2:39:F0:D3 NAS IP: 10.193.144.124 NAS Port: GigabitEthernet5/0/15

please let me know, we’ve a bunch of alarms going off

good morning, let me have a look

GM


[teams message]

how is the first one looking? should be up now. Looking into the second one

it pings!

ok sorry for the delay. i’m able to ping the second one now:

ping -c 3 10.238.1.29

PING 10.238.1.29 (10.238.1.29) 56(84) bytes of data.

64 bytes from 10.238.1.29: icmp_seq=1 ttl=58 time=13.9 ms

64 bytes from 10.238.1.29: icmp_seq=2 ttl=58 time=11.2 ms

64 bytes from 10.238.1.29: icmp_seq=3 ttl=58 time=10.1 ms

thank you !

any time!!! Thank you for confirming

got 1 more 10.238.1.28

sounds good. let me look into this one for you

00:50:C2:39:F0:F7

[netapi check]

❯ uv run netapi ise mnt session 00:50:C2:39:F0:F7 ╭───────────────────────────── ISE Session Details ─────────────────────────────╮ │ Session: 00:50:C2:39:F0:F7 | Status: FAILED │ ╰───────────────────────────────────────────────────────────────────────────────╯ Authentication Username 00-50-C2-39-F0-F7 ISE Node psn-1 Session ID 7C90C10A000062EAA95411A1

                    Network
Client MAC                00:50:C2:39:F0:F7
Client IPv4               10.238.1.28
NAD IP                    10.193.144.124
NAD Name                  DUQUE-2-1-9300
Port                      GigabitEthernet4/0/11
              Accounting
Session ID                00007fa3
Status                    Stop
Duration (sec)            5393075
RX Bytes                  2097565154
TX Bytes                  0
Terminate Cause           Lost Carrier
                                 Endpoint
  Profiled As               medigate_Guardian_Isensix_Temperature_Sensor
 evanusmodestus  󰙅 D001/DEV/NETWORK   netapi   main   3.13  ✓
❯
Client MAC                00:50:C2:39:F0:F7
Client IPv4               10.238.1.28
NAD IP                    10.193.144.124
NAD Name                  DUQUE-2-1-9300
Port                      GigabitEthernet4/0/11

## [terminal session clear] clear terminal session to copy entirety of console avoiding other output
clear
## [NAS Switch]  ssh into NAS
sudo ssh erosado@10.193.144.124


## [MAC Table] check MAC table for endpoint
show mac address address 00:50:C2:39:F0:F7
## [NAS PORT] check last known port that ISE logs indicate
show interfaces GigabitEthernet4/0/11 status
## [access session] check for an access session by MAC address
show access-session mac 00:50:C2:39:F0:F7 detail
## [access session] check for an access session by interface
show access-session interface GigabitEthernet4/0/11 detail
## [nas port config] check interface port configuration
show run interface GigabitEthernet4/0/11
# [command output]

[sudo] password for evanusmodestus:
** WARNING: connection is not using a post-quantum key exchange algorithm.
** This session may be vulnerable to "store now, decrypt later" attacks.
** The server may need to be upgraded. See https://openssh.com/pq.html

*****************************************************************
* NOTICE  NOTICE            S T O P              NOTICE  NOTICE *
*                   THIS IS A PRIVATE SYSTEM.                   *
*      AUTHORIZATION IS REQUIRED TO CONNECT TO THIS DEVICE.     *
* ACTUAL OR ATTEMPTED USE, ACCESS, EXAMINATION OR CONFIGURATION *
* CHANGE BY ANY UNAUTHORIZED PERSON WILL RESULT IN CRIMINAL AND *
*     CIVIL PROSECUTION TO THE FULLEST EXTENT OF THE LAW.       *
*                  N    O    T    I    C    E                   *
*****************************************************************
(erosado@10.193.144.124) Password:
DUQUE-2-1-9300>en
Password:
DUQUE-2-1-9300### [MAC Table] check MAC table for endpoint
DUQUE-2-1-9300#show mac address address 00:50:C2:39:F0:F7
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
DUQUE-2-1-9300#$ast known port that ISE logs indicate
DUQUE-2-1-9300#show interfaces GigabitEthernet4/0/11 status

Port         Name               Status       Vlan       Duplex  Speed Type
Gi4/0/11     User               connected    651        a-full  a-100 10/100/1000BaseTX
DUQUE-2-1-9300#$heck for an access session by MAC address
DUQUE-2-1-9300#$-session mac 00:50:C2:39:F0:F7 detail
No sessions match supplied criteria.

DUQUE-2-1-9300#$heck for an access session by interface
DUQUE-2-1-9300#$nterface GigabitEthernet4/0/11 detail
No sessions match supplied criteria.

DUQUE-2-1-9300#$t config] check interface port configuration
DUQUE-2-1-9300#show run interface GigabitEthernet4/0/11
Building configuration...

Current configuration : 467 bytes
!
interface GigabitEthernet4/0/11
 description User
 switchport access vlan 651
 switchport mode access
 switchport nonegotiate
 switchport voice vlan 424
 device-tracking attach-policy DT_POLICY
 ip flow monitor IPv4_NETFLOW input
 dot1x timeout tx-period 7
 dot1x max-reauth-req 3
 source template DefaultWiredDot1xClosedAuth
 spanning-tree portfast
 spanning-tree bpduguard enable
 service-policy input QoS_Edge_Ingress
 service-policy output QoS_Edge_Egress
end

DUQUE-2-1-9300#
DUQUE-2-1-9300#


# [vlan assignment] incorrect vlan assigned

# add device to connect VLAN
conf t
interface GigabitEthernet4/0/11
switchport access vlan 751
exit
exit
# validate VLAN port assignment
show vlan id 751 | inc Gi4/0/11
# check for and access session
show access-session interface GigabitEthernet4/0/11 detail

## [command output]


DUQUE-2-1-9300## add device to connect VLAN
DUQUE-2-1-9300#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
DUQUE-2-1-9300(config)#interface GigabitEthernet4/0/11
DUQUE-2-1-9300(config-if)#switchport access vlan 751
DUQUE-2-1-9300(config-if)#exit
DUQUE-2-1-9300(config)#exit
DUQUE-2-1-9300## validate VLAN port assignment
DUQUE-2-1-9300#show vlan id 751 | inc Gi4/0/11
                                                Gi4/0/11
DUQUE-2-1-9300## check for and access session
DUQUE-2-1-9300#$nterface GigabitEthernet4/0/11 detail
No sessions match supplied criteria.

DUQUE-2-1-9300#

[command output] incorrect vlan assignment


DUQUE-2-1-9300#$nterface GigabitEthernet4/0/11 detail
            Interface:  GigabitEthernet4/0/11
               IIF-ID:  0x151A86A5
          MAC Address:  0050.c239.f0f7
         IPv6 Address:  Unknown
         IPv4 Address:  Unknown
               Status:  Unauthorized
               Domain:  UNKNOWN
       Oper host mode:  multi-auth
     Oper control dir:  in
      Session timeout:  N/A
    Common Session ID:  7C90C10A000091C5EBFF19B9
      Acct Session ID:  Unknown
               Handle:  0x3d0009ce
       Current Policy:  PMAP_DefaultWiredDot1xClosedAuth_1X_MAB


Server Policies:


Method status list:
       Method           State
        dot1x           Running

DUQUE-2-1-9300#


DUQUE-2-1-9300#$nterface GigabitEthernet4/0/11 detail
            Interface:  GigabitEthernet4/0/11
               IIF-ID:  0x151A86A5
          MAC Address:  0050.c239.f0f7
         IPv6 Address:  Unknown
         IPv4 Address:  10.238.1.28
            User-Name:  00-50-C2-39-F0-F7
               Status:  Authorized
               Domain:  DATA
       Oper host mode:  multi-auth
     Oper control dir:  in
      Session timeout:  N/A
  Acct update timeout:  172800s (local), Remaining: 172773s
    Common Session ID:  7C90C10A000091C5EBFF19B9
      Acct Session ID:  0x0000a9fb
               Handle:  0x3d0009ce
       Current Policy:  PMAP_DefaultWiredDot1xClosedAuth_1X_MAB


Server Policies:
           Vlan Group:  Vlan: 751
              ACS ACL: xACSACLx-IP-TEST_Medical_Temp_Monitor-697290bb


Method status list:
       Method           State
        dot1x           Stopped
          mab           Authc Success

DUQUE-2-1-9300#
DUQUE-2-1-9300#


## Victor update

### [teams chat] 2026-01-23 09:58

awesome! 28 pings

thanks. you beat me to it. was just about to let you know: ❯ # [ping endpoint]

❯ ping -c 3 10.238.1.28

PING 10.238.1.28 (10.238.1.28) 56(84) bytes of data.

64 bytes from 10.238.1.28: icmp_seq=1 ttl=58 time=13.5 ms

64 bytes from 10.238.1.28: icmp_seq=2 ttl=58 time=10.0 ms

64 bytes from 10.238.1.28: icmp_seq=3 ttl=58 time=10.4 ms

--- 10.238.1.28 ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 2003ms

rtt min/avg/max/mdev = 10.031/11.308/13.541/1.583 ms


[further validation ]


## [kernal (yours truely) interrupt request] LoL

## [teams chat] Victor Cancino

wow I just found a couple more, did something happen

nm, looks good

not that i know if. Was just going about my day working on a separate project for research

Just trying to let a helping hand. Everything okay?

Thanks for your assistance

Hey, you're always welcome!!


## [back to powerup validate with netapi] 2026-01-23 10:01

 uv run netapi ise mnt session 00-50-C2-39-F0-F7
╭───────────────────────────── ISE Session Details ─────────────────────────────╮
│ Session: 00-50-C2-39-F0-F7  |  Status: PASSED                                 │
╰───────────────────────────────────────────────────────────────────────────────╯
                      Authentication
  Username                  00:50:C2:39:F0:F7
  Method                    mab
  Protocol                  Lookup
  Timestamp                 2026-01-23T09:55:51.220-08:00
  ISE Node                  psn-1
  Session ID                7C90C10A000091C5EBFF19B9

                       Network
  Client MAC                00:50:C2:39:F0:F7
  Client IPv4               10.238.1.28
  NAD IP                    10.193.144.124
  NAD Name                  DUQUE-2-1-9300
  Port                      GigabitEthernet4/0/11
  Device Type               All Device Types#SWITCH

                 Accounting
  Session ID                0000a9fb
  Status                    Interim-Update
  RX Bytes                  64
  TX Bytes                  0

                                 Endpoint
  Profiled As               medigate_Guardian_Isensix_Temperature_Sensor
 evanusmodestus  󰙅 D001/DEV/NETWORK   netapi   main   3.13  ✓
❯


# [netapi gap] how can i update the following information with netapi?


DUQUE-2-1-9300#$ACLx-IP-TEST_Medical_Temp_Monitor-697290bb
Extended IP access list xACSACLx-IP-TEST_Medical_Temp_Monitor-697290bb
    1 permit tcp any eq 6001 host 10.192.220.51
    2 permit tcp any eq 6001 host 10.192.220.52
    3 permit tcp any eq 6001 host 10.192.220.64
    4 permit tcp any eq 6001 host 10.192.220.65
    6 permit tcp any host 10.192.220.51 eq 6001
    8 permit tcp any host 10.192.220.52 eq 6001
    10 permit tcp any host 10.192.220.64 eq 6001
    12 permit tcp any host 10.192.220.65 eq 6001
    14 permit udp any eq bootpc any eq bootps
    16 permit icmp any any
    18 deny ip any any
DUQUE-2-1-9300#

[CRITICAL] Microsoft Sentinel / XRadar Migration Call - Log Sources

Date: 2026-01-23 ~11:15 Context: Input for SIEM integration - ISE, FMC, Switches, WLCs


CISCO ISE - Tier 1 (IR Critical)

Log Source Security Value

RADIUS Authentication

Failed auths, brute force, credential stuffing

RADIUS Accounting

Session duration anomalies, lateral movement

Posture Assessment

Non-compliant endpoints, compromised indicators

Profiler Events

Rogue devices, MAC spoofing, device type changes

CISCO ISE - Tier 2 (Analytics)

Log Source Security Value

CoA Events

Quarantine actions, incident response triggers

Guest Portal

Unauthorized access, social engineering

Admin Audit

ISE config changes, insider threat

Policy Hit Logs

Policy bypass attempts


CISCO FMC/FTD (Firepower) - CRITICAL

Log Source Security Value

Intrusion Events

IDS/IPS alerts, exploit attempts

Malware Events

AMP detections, file disposition

Connection Events

NetFlow-like data, C2 detection

File Events

File transfers, data exfil indicators

Security Intelligence

Blocked IPs/URLs from threat feeds

SSL Decryption

Encrypted threat visibility

FMC Syslog Priority: - Priority 1: Intrusion events (Impact 1-2) - Priority 2: Malware/AMP events - Priority 3: Connection events (denied, high-risk geos) - Priority 4: File events


CISCO SWITCHES (IOS-XE/Catalyst)

Log Source Security Value

AAA Accounting

Who logged in, what commands ran

802.1X/MAB Events

Port auth success/failure

DHCP Snooping

Rogue DHCP detection

Dynamic ARP Inspection

ARP spoofing detection

Port Security

MAC flooding attacks

STP Events

Network topology attacks

Config Changes

Archive/syslog config diff


CISCO WLC (9800)

Log Source Security Value

Client Auth Events

Wireless auth success/fail

Rogue AP Detection

Unauthorized APs

Client Exclusion

Blocked clients

Mobility Events

Client roaming anomalies

RF Anomalies

Interference, jamming


KEY FIELDS FOR CORRELATION

COMMON FIELDS (All Sources):
- Timestamp (normalized to UTC)
- Source IP / MAC
- Destination IP / Port
- Username (if available)
- Device hostname
- Event severity

ISE-SPECIFIC:
- NAS-IP-Address (switch/WLC)
- NAS-Port-Id (physical port)
- Calling-Station-ID (MAC)
- Authorization Profile
- Failure Reason Code

FMC-SPECIFIC:
- Source/Dest Security Zone
- Application Protocol
- URL Category
- Intrusion Signature ID
- File SHA256

HIGH-VALUE SENTINEL DETECTION RULES

  1. ISE failed auth → FMC denied connection = compromised endpoint

  2. Same user auth from multiple MACs = credential theft

  3. Device profile change + FMC malware = compromised IoT

  4. Config change on switch → no change ticket = insider threat

  5. Rogue AP detected + new client auths = evil twin attack

  6. Spike in RADIUS failures = brute force attack

  7. FMC intrusion + ISE successful auth = post-exploitation


ISE SYSLOG MESSAGE IDS

ID Event Priority

5200

Auth succeeded

Medium

5400

Auth failed

HIGH

5405

RADIUS dropped

HIGH

5440

Endpoint abandoned EAP

Medium

86014

Guest auth failed

HIGH

FMC SYSLOG FACILITY

  • Use LOCAL4 or dedicated facility

  • Enable eStreamer for high-volume events

  • Consider Sentinel FMC connector if available


INTEGRATION RECOMMENDATIONS

  1. ISE → Syslog (TCP/TLS 6514) + pxGrid for real-time

  2. FMC → Syslog + eStreamer for full fidelity

  3. Switches → Syslog (include AAA accounting)

  4. WLC → Syslog + SNMP traps for RF events

Retention: 90 days hot, 1 year cold minimum for compliance


[ISE INCIDENT RESPONSE] - What Data Matters Most

During Active Incident - CRITICAL QUERIES

1. WHO was on the network at time X?
   → RADIUS Accounting (session start/stop times)
   → Maps MAC → IP → Username → Physical Port

2. WHERE did the compromised device connect?
   → NAS-IP + NAS-Port-Id = exact switch port
   → Can physically isolate device

3. WHAT did ISE authorize them to do?
   → Authorization Profile = VLAN + dACL + SGT
   → Shows network access scope

4. HOW did they authenticate?
   → 802.1X (creds) vs MAB (just MAC)
   → MAB = easier to spoof, less trust

5. WHEN did behavior change?
   → Profiler history = device type changes
   → Windows laptop suddenly profiled as phone = RED FLAG

ISE Data for Incident Timeline

Phase ISE Data Source What It Tells You

Initial Access

Auth logs (5200/5400)

When device first appeared

Persistence

Session accounting

How long on network

Lateral Movement

Multiple NAS-IPs same MAC

Device moved physically

Privilege Escalation

Auth profile changes

Got more access somehow

Exfiltration

Accounting (bytes TX/RX)

Data volume indicators

ISE Forensic Gold

Session ID - Links all ISE events for one connection: - Authentication attempt - Authorization applied
- Accounting start/interim/stop - CoA events - Profiler updates

pxGrid Session Directory - Real-time: - All active sessions right now - IP-to-MAC-to-User mapping - Push to SIEM for live correlation

Containment Actions FROM ISE

Action How When

Quarantine endpoint

CoA → Quarantine VLAN

Confirmed compromise

Block MAC

Add to Blacklist group

Known bad device

Force re-auth

CoA Reauthenticate

Verify current state

Kill session

CoA Disconnect

Immediate isolation

DataConnect Queries for IR

-- Find all sessions for a MAC in last 7 days
SELECT * FROM RADIUS_AUTHENTICATION
WHERE calling_station_id = '00:11:22:33:44:55'
AND timestamp > SYSDATE - 7;

-- Find auth failures preceding success (brute force)
SELECT * FROM RADIUS_AUTHENTICATION
WHERE calling_station_id = '00:11:22:33:44:55'
ORDER BY timestamp DESC;

-- All devices that hit a specific policy
SELECT calling_station_id, framed_ip_address, nas_ip_address
FROM RADIUS_AUTHENTICATION
WHERE authorization_policy = 'Suspicious_Device_Policy';

Top ISE Alerts for SOC

  1. 5400 + Failure Reason 24408 = Wrong password (brute force indicator)

  2. 5400 + Failure Reason 22056 = User not found (enumeration)

  3. Profiler change event = Device type changed (spoofing)

  4. CoA issued by admin = Someone took action (correlate with ticket)

  5. Multiple 5400 → then 5200 = Successful brute force


[CALL NOTES] Why ISE is NON-NEGOTIABLE for Sentinel

1. ISE is the ONLY source that maps IP → MAC → User → Physical Port - Without ISE: “Alert on 10.238.1.28” = who is that? - With ISE: “Alert on 10.238.1.28” = John Smith, MacBook, Building 2, Switch port Gi4/0/11

2. ISE sees devices BEFORE they do anything malicious - FMC sees traffic after access granted - ISE sees the authentication attempt FIRST - Catch bad actors at the door, not inside

3. Lateral movement detection is IMPOSSIBLE without ISE - Same MAC appearing on different switch ports = physical movement - Only ISE tracks this

4. Compromised credential detection - Same user, multiple MACs, same timeframe = STOLEN CREDS - FMC can’t tell you this

5. Rogue/spoofed device detection - Device profiled as “Windows” suddenly shows as “iPhone” = MAC SPOOFING - Only ISE profiler catches this

6. Containment capability - ISE can QUARANTINE a device in seconds via CoA - No ISE integration = manual port shutdown = SLOW

7. Healthcare/compliance (HIPAA) - Requires knowing WHO accessed WHAT and WHEN - ISE provides the WHO + WHERE - Auditors will ask for this

Key Quote: "`If we have an incident and can’t answer ‘who was on IP X at time Y’, we’ve failed. ISE is the only thing that answers that.`"


[ACTION ITEM] Establish Current Logging Baseline

They asked: What are we currently logging and from what sources?

Discovery Tasks Needed:

  1. ISE Logging Config

    # Check ISE remote logging targets
    # ISE Admin → Administration → System → Logging → Remote Logging Targets
    # Document: IP, Port, Facility, Severity levels
  2. FMC Logging Config

    # FMC → System → Configuration → Logging
    # Check: Syslog servers, eStreamer config, what event types enabled
  3. Switch Logging Config

    show logging
    show run | include logging
    # Document: syslog servers, trap levels, which VTYs logged
  4. WLC Logging Config

    show logging
    show run | include logging
    # Check SNMP trap destinations too
  5. Current SIEM/XRadar

    • What’s the current log collector IP?

    • What indexes exist?

    • What volume (EPS) currently?

Quick Commands to Run:

# ISE - check MnT for logging config (API)
# Or SSH to ISE and check /opt/CSCOcpm/logs/

# Switches - run on core switches
show logging
show run | sec logging

# WLC
show logging

# FMC - check via GUI or API
# System > Configuration > Logging

Baseline Documentation Template:

Source Destination IP Port Protocol Severity EPS (est) Currently Working?

ISE PAN

?

?

?

?

?

?

ISE PSN

?

?

?

?

?

?

FMC

?

?

?

?

?

?

Core SW1

?

?

?

?

?

?

Core SW2

?

?

?

?

?

?

WLC

?

?

?

?

?

?


QRadar Log Source Discovery

AQL queries and API methods to extract current log source inventory for Sentinel migration baseline

AQL Queries

1. All Log Sources with Details
SELECT
    logsourcename,
    logsourceid,
    logsourcetypename,
    devicetype,
    devicename,
    sourceip,
    DATEFORMAT(devicetime, 'yyyy-MM-dd HH:mm:ss') as last_event
FROM events
GROUP BY logsourcename, logsourceid, logsourcetypename, devicetype, devicename, sourceip
LAST 24 HOURS
2. Log Sources by Type with Event Counts
SELECT
    logsourcetypename,
    COUNT(*) as event_count,
    UNIQUECOUNT(logsourcename) as source_count
FROM events
GROUP BY logsourcetypename
ORDER BY event_count DESC
LAST 7 DAYS
3. Stale/Broken Log Sources (No Events in 24h)
SELECT
    logsourcename,
    logsourcetypename,
    sourceip,
    DATEFORMAT(MAX(devicetime), 'yyyy-MM-dd HH:mm:ss') as last_seen
FROM events
GROUP BY logsourcename, logsourcetypename, sourceip
HAVING last_seen < NOW() - 24*60*60*1000
LAST 30 DAYS
4. Cisco-Specific Log Sources
SELECT
    logsourcename,
    logsourcetypename,
    sourceip,
    COUNT(*) as events
FROM events
WHERE logsourcetypename ILIKE '%cisco%'
   OR logsourcetypename ILIKE '%ise%'
   OR logsourcetypename ILIKE '%firepower%'
GROUP BY logsourcename, logsourcetypename, sourceip
LAST 7 DAYS
5. Event Rate Per Log Source (EPS Baseline)
SELECT
    logsourcename,
    logsourcetypename,
    COUNT(*)/168 as avg_eps_hourly
FROM events
GROUP BY logsourcename, logsourcetypename
ORDER BY avg_eps_hourly DESC
LAST 7 DAYS
6. Top Talkers (Highest Volume Sources)
SELECT
    logsourcename,
    logsourcetypename,
    COUNT(*) as total_events,
    COUNT(*)/7/24 as avg_eps
FROM events
GROUP BY logsourcename, logsourcetypename
ORDER BY total_events DESC
LIMIT 25
LAST 7 DAYS

QRadar REST API Export

For a complete export (better than AQL for full inventory):

# Set environment
export QRADAR_IP="your-qradar-console"
export QRADAR_TOKEN="your-api-token"

# Export all log sources
curl -s -k -H "SEC: $QRADAR_TOKEN" \
  "https://$QRADAR_IP/api/config/event_sources/log_source_management/log_sources" \
  | jq -r '.[] | [.id, .name, .type_id, .enabled, .protocol_type_id, .gateway_id] | @csv' \
  > qradar_log_sources.csv

# Export log source types (for reference/join)
curl -s -k -H "SEC: $QRADAR_TOKEN" \
  "https://$QRADAR_IP/api/config/event_sources/log_source_management/log_source_types" \
  | jq -r '.[] | [.id, .name] | @csv' \
  > qradar_log_source_types.csv

# Export with full details as JSON
curl -s -k -H "SEC: $QRADAR_TOKEN" \
  "https://$QRADAR_IP/api/config/event_sources/log_source_management/log_sources?fields=id,name,type_id,enabled,gateway_id,protocol_type_id,parsing_order,average_eps,last_event_time" \
  | jq '.' > qradar_log_sources_full.json

GUI Export Method

  1. AdminData SourcesLog Sources

  2. Click Export button (top right)

  3. Select CSV format

  4. Save as qradar-log-sources-export-2026-01-23.csv

Expected Report Output Format

Log Source Name Type Source IP Status Avg EPS Last Event Migrate to Sentinel?

ISE-PAN-01

Cisco ISE

10.x.x.x

Active

150

2026-01-23

Yes - Tier 1

FMC-01

Cisco FMC

10.x.x.x

Active

500

2026-01-23

Yes - Tier 1

Core-SW-01

Cisco IOS

10.x.x.x

Active

50

2026-01-23

Yes - Tier 2

WLC-01

Cisco WLC

10.x.x.x

Active

200

2026-01-23

Yes - Tier 1

Action Items

  • Run AQL Query #1 to get current log source inventory

  • Run AQL Query #5 to establish EPS baseline

  • Export via API or GUI for complete list

  • Cross-reference with Sentinel connector availability

  • Identify gaps (sources in QRadar but no Sentinel connector)

  • Estimate ingestion costs based on EPS



Shell Configuration: AsciiDoc Terminal Preview

Added w3m terminal preview for AsciiDoc files across all shells. Enables quick previews without leaving the terminal or creating temp files.

Functions Added

Command Description

adoc-view <file>

Preview AsciiDoc in w3m (inline)

adoc-split <file>

Preview AsciiDoc in tmux split pane

Implementation

# Core pattern - pipe asciidoctor output directly to w3m
asciidoctor -o - file.adoc | w3m -T text/html

Files Modified

  • ~/.zshrc - lines 899-914

  • ~/.bashrc - lines 223-241

  • ~/.config/fish/config.fish - lines 1006-1022

Neovim Integration (manual)

-- Add to your asciidoc ftplugin/keymaps
vim.keymap.set('n', '<leader>ap', function()
  local file = vim.fn.expand('%:p')
  vim.cmd('silent !asciidoctor -o - ' .. vim.fn.shellescape(file) .. ' | w3m -T text/html')
  vim.cmd('redraw!')
end, { desc = 'Preview AsciiDoc in w3m' })

Benefits

  • No temp files created

  • Keyboard-driven (w3m is vim-like)

  • Works in SSH sessions

  • Tmux integration for side-by-side editing