Daily Worklog: 2026-02-09 (Monday)

Overview

Date: 2026-02-09 (Monday)

Location: Remote

Focus: Windows DC Migration, Documentation, ISE Troubleshooting

Sessions

Session 1: Windows Server 2025 Core Deployment

Duration: Ongoing

Accomplishments:

  • Deployed Windows Server 2025 Core VM (home-dc01) on KVM

    • Loaded VirtIO drivers (viostor + NetKVM) during installation

    • ISO: Windows Server 2025 Standard Evaluation

    • Storage: /mnt/onboard-ssd/vms/home-dc01.qcow2

  • Created pfSense DNS documentation in domus-netapi-docs

    • New page: cli/pfsense/dns/index.adoc

    • Commands: list, add, update, delete, apply

    • Fixed command syntax (all fields required for update)

  • Updated KVM host DNS record (kvm-01-lan: 192.168.1.181 → 192.168.1.192)

Session 2: ISE Rejected Endpoint Troubleshooting

Issue: Domus-IoT device not connecting to WiFi

Investigation:

netapi ise ers rejected
Table 1. Rejected Endpoint
MAC Address Reason

64:32:A8:C4:C7:19

EndPoint

Check auth history (MAC is positional, not a flag):

netapi ise dc auth-history 64:32:A8:C4:C7:19
Table 2. Auth History Result
Time Status Method Policy Set AuthZ Profile Failure Reason

2026-02-09 07:46:56

FAILED

mab

DELETE-HomeRF

DenyAccess

15039 Rejected per authorization profile

Root Cause: Device hitting DELETE-HomeRF policy set which has DenyAccess authorization profile. This is intentional policy denial, not an auth failure.

Resolution:

netapi ise release-rejected 64:32:A8:C4:C7:19

Output: ✓ Released rejected endpoint: 64:32:A8:C4:C7:19

Delete endpoint to force fresh authentication:

netapi ise delete-endpoint 64:32:A8:C4:C7:19 --force

Output: ✓ Deleted endpoint: 64:32:A8:C4:C7:19

Verify no rejected endpoints remain:

netapi ise get-rejected-endpoints

Output: No rejected endpoints

Next Steps:

  • Verify Domus-IoT SSID is broadcasting on WLC

  • Reconnect device to trigger fresh authentication

  • Add to correct identity group after re-auth

Session 3: WLC SSID Troubleshooting

Device not seeing Domus-IoT SSID despite being enabled on WLC.

Check WLANs:

netapi wlc get-wlans
Table 3. WLANs
ID Profile Name SSID Status

1

HomeRF

HomeRF

enabled

5

Domus-IoT

Domus-IoT

enabled

4

Domus-Secure

Domus-Secure

enabled

Check policy tags (WLAN-to-AP mapping):

netapi wlc policy-tags
Table 4. Policy Tags
Tag Name Description

IoT-Tag

IoT policy tag

HomeRF-Tag

TAG-DOMUS_SECURE

default-policy-tag

default policy-tag

Check APs:

netapi wlc get-aps
Table 5. Access Points
Name IP Address Model Serial Version

AP4800

10.50.10.101

AIR-AP4800-B-K9

FCW2316NH54

17.15.3.28

Analysis: Only 1 AP (AP4800). Need to verify which policy tag is assigned to AP4800 and whether Domus-IoT WLAN is mapped to that tag.

Check AP tag assignment:

netapi wlc get-ap AP4800 | grep -i tag

Result: AP4800 using default-policy-tag, but Domus-IoT WLAN not mapped to it.

Root Cause: Policy tag had stale DOMUS_IoT (underscore) mapping but WLAN is Domus-IoT (hyphen).

Fix: Recreate policy tag with correct mappings (preserving existing networks):

netapi wlc create-policy-tag default-policy-tag \
    --map HomeRF:VLAN10-Policy \
    --map Domus-Secure:POLICY-DOMUS_SECURE \
    --map Domus-IoT:POLICY-DOMUS_IoT \
    --save
Output
Creating policy tag: default-policy-tag
  HomeRF -> VLAN10-Policy
  Domus-Secure -> POLICY-DOMUS_SECURE
  Domus-IoT -> POLICY-DOMUS_IoT
  OK
Saving configuration...
  OK

Result: Domus-IoT SSID now broadcasting.

Session 4: Gopass Update

Updated ADMINISTRATIO/servers/home-dc01 with fresh credentials for Windows Server 2025 Core deployment:

  • OS: Windows Server 2025 Standard (Core)

  • Roles: AD-DS, DNS (no AD CS - PKI on Vault)

  • New Administrator password generated

  • New DSRM password generated

Session 5: Windows Server 2025 Core - home-dc01

Status: Online

  • VirtIO drivers loaded (viostor + NetKVM)

  • Computer name set: home-dc01

  • Next: Configure network (sconfig), promote to DC

Session 6: OEAP Rejected Endpoint Troubleshooting

Issue: Work OEAP (04:5F:B9:78:02:20) rejected by ISE, causing office WiFi outage.

Investigation:

Check rejected endpoints:

netapi ise get-rejected-endpoints

Check switch port status:

netapi ios exec "show access-session interface gi1/0/4 details"

Result: Status Unauthorized, MAB Stopped.

Query auth history to find what worked before:

netapi ise dc query "
SELECT
    TIMESTAMP_TIMEZONE,
    CALLING_STATION_ID,
    POLICY_SET_NAME,
    AUTHORIZATION_PROFILES,
    IDENTITY_GROUP,
    PASSED
FROM RADIUS_AUTHENTICATIONS
WHERE CALLING_STATION_ID LIKE '%04:5F:B9:78:02:20%'
ORDER BY TIMESTAMP_TIMEZONE DESC
FETCH FIRST 10 ROWS ONLY
"

Root Cause Analysis:

When Policy Set AuthZ Profile Identity Group Result

Before (working)

Corp LAN MAB

AP_Secure_VLAN10_Profile

Trusted_Access_Points

Pass

Now (failing)

Domus-Wired 802.1X

(empty)

(empty)

Fail - 22056 Subject not found

The endpoint was hitting Domus-Wired 802.1X instead of Corp LAN MAB even though it was in the correct identity group.

Verify endpoint group:

netapi ise get-endpoint 04:5F:B9:78:02:20

Result: Group = Trusted_Access_Points, Static Assignment = True (correct!)

Resolution:

Initial attempts to just release from rejection failed - endpoint kept getting re-rejected. Nuclear option required:

# Delete and immediately recreate with static group assignment
netapi ise delete-endpoint 04:5F:B9:78:02:20 --force && \
netapi ise create-endpoint 04:5F:B9:78:02:20 --group "Trusted_Access_Points" --static-group
Output
✓ Deleted endpoint: 04:5F:B9:78:02:20
✓ Created endpoint: 04:5F:B9:78:02:20
  ID: 08bf6780-05e4-11f1-8b03-5628c177fe65

Key Learning: When release-rejected alone doesn’t work, delete the endpoint entirely and recreate with --static-group flag. The && ensures create runs immediately before ISE can auto-create a new endpoint from re-auth attempts.

UPDATE - Root Cause Found:

The real issue was ISE Policy Set ordering. The OEAP was hitting Domus-Wired 802.1X instead of Domus-Wired MAB, causing error 22056 "Subject not found" (MAB lookup against wrong identity store).

Root Cause Analysis:

Issue Detail

Symptom

OEAP rejected with error 22056 "Subject not found in applicable identity store"

Wrong assumption

Endpoint group assignment issue

Actual cause

Policy set Domus-Wired 802.1X was evaluated before Domus-Wired MAB

Why it broke

Policy set ordering changed; MAB devices were hitting 802.1X policy first

Resolution:

  1. In ISE: Policy → Policy Sets

  2. Move Domus-Wired MAB above Domus-Wired 802.1X

  3. Save and bounce switch port

netapi ios config "interface GigabitEthernet1/0/4" "shutdown" && sleep 2 && netapi ios config "interface GigabitEthernet1/0/4" "no shutdown"

Verified working:

Status:  Authorized
Domain:  DATA
Vlan Group:  Vlan: 10
mab: Authc Success

Key Learnings:

  • ISE evaluates policy sets top-to-bottom; order matters

  • MAB devices must hit MAB policy set, not 802.1X

  • Error 22056 usually means wrong identity store for auth method

  • Don’t waste time on endpoint group assignment if policy set matching is wrong

Session 7: 10G Trunk Flapping - KVM Host Connectivity

Issue: Lost all connectivity to KVM host (home-dc01 VM unreachable). SSH showing "No route to host".

Symptoms:

  • Te1/0/2 (TRUNK-TO-SUPERMICRO) flapping between connected, notconnect, and err-disabled

  • Link would come up briefly then drop

  • Trunk carrying VLANs 10,20,30,40,100,999 to Supermicro E300-9D

Troubleshooting via console (picocom):

picocom -b 9600 /dev/ttyUSB0
LAB-3560CX-01#show int t1/0/2 status
Port      Name               Status       Vlan       Duplex  Speed Type
Te1/0/2   TRUNK-TO-SUPERMICR err-disabled 1            full    10G SFP-10GBase-SR

# Repeated shut/no shut cycles
LAB-3560CX-01(config-if)#shut
LAB-3560CX-01(config-if)#no shut

# Status kept cycling: notconnect → connected → err-disabled

Root Cause: Suspected SFP-10GBase-SR module incompatibility or signal issues with Supermicro Intel X710 10GbE NIC.

Resolution: After multiple port bounces, link stabilized. May need to replace SFP or check fiber/DAC cable.

Interface Config (working):

interface TenGigabitEthernet1/0/2
 description TRUNK-TO-SUPERMICRO
 switchport trunk allowed vlan 10,20,30,40,100,999
 switchport trunk native vlan 100
 switchport mode trunk
 ip arp inspection trust
 spanning-tree portfast edge trunk
 ip dhcp snooping trust

Key Learning: 10G SFP modules can be finicky with certain NICs. Keep console access ready for switch troubleshooting when remote connectivity is lost.

Session 8: Windows Server 2025 Core - SSH with YubiKey

Goal: Configure SSH on home-dc01 with YubiKey authentication.

Install OpenSSH Server (on Windows console):

Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
Start-Service sshd
Stop-Service sshd

Configure custom port (22022):

(Get-Content C:\ProgramData\ssh\sshd_config) -replace '#Port 22', 'Port 22022' | Set-Content C:\ProgramData\ssh\sshd_config

Enable password auth temporarily (to copy keys):

(Get-Content C:\ProgramData\ssh\sshd_config) -replace '#PasswordAuthentication yes', 'PasswordAuthentication yes' | Set-Content C:\ProgramData\ssh\sshd_config
Restart-Service sshd
Set-Service -Name sshd -StartupType Automatic

Firewall rule:

New-NetFirewallRule -Name "SSH-22022" -DisplayName "SSH (22022)" -Protocol TCP -LocalPort 22022 -Action Allow -Enabled True
Set-NetFirewallProfile -Profile Domain,Public,Private -Enabled True

Initial SSH (requires sudo to bypass user SSH config issues):

sudo ssh -p 22022 Administrator@10.50.1.50

Copy public keys using brace expansion + wl-copy:

# Brace expansion to match multiple keys
ls ~/.ssh/id_ed25519_{d000,sk_rk_d000,sk_rk_d000_secondary}.pub

# Copy to clipboard (Wayland)
cat ~/.ssh/id_ed25519_{d000,sk_rk_d000,sk_rk_d000_secondary}.pub | wl-copy

Add keys to Windows (PowerShell here-string):

@"
sk-ssh-ed25519@openssh.com AAAA... evanusmodestus@d000-secondary
sk-ssh-ed25519@openssh.com AAAA... evanusmodestus@d000-yubikey
ssh-ed25519 AAAA... evanusmodestus@d000
"@ | Set-Content C:\ProgramData\ssh\administrators_authorized_keys

icacls C:\ProgramData\ssh\administrators_authorized_keys /inheritance:r /grant "Administrators:F" /grant "SYSTEM:F"

SSH config (~/.ssh/config):

Host home-dc01
    HostName 10.50.1.50
    Port 22022
    User Administrator
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000_secondary
    IdentityFile ~/.ssh/id_ed25519_d000

Test connection:

ssh home-dc01

Kill multiplexed connections (ControlMaster):

ssh -O exit home-dc01

Key Learnings:

  • Windows admin SSH keys go in C:\ProgramData\ssh\administrators_authorized_keys (not user’s .ssh folder)

  • icacls permissions required: Administrators:F and SYSTEM:F

  • Brace expansion: {a,b,c} matches multiple patterns in one glob

  • wl-copy pipes output to Wayland clipboard

  • ControlMaster reuses connections - use ssh -O exit to close

  • sudo ssh bypasses user SSH config issues for initial setup

Session 9: Windows Server 2025 Core - DC Promotion

Goal: Promote home-dc01 to Domain Controller (new forest).

Install AD DS and DNS:

Install-WindowsFeature AD-Domain-Services, DNS -IncludeManagementTools

Gopass credential structure (separate entries):

# Generate passwords
gopass generate ADMINISTRATIO/servers/home-dc01/Administrator 32
gopass generate ADMINISTRATIO/servers/home-dc01/dsrm 32

# Create metadata with heredoc
gopass insert ADMINISTRATIO/servers/home-dc01/meta << 'EOF'
hostname: home-dc01
fqdn: home-dc01.inside.domusdigitalis.dev
ip: 10.50.1.50
domain: inside.domusdigitalis.dev
netbios: DOMUS
ssh_port: 22022
os: Windows Server 2025 Core
roles: AD-DS, DNS
pki: Vault (DOMUS-ROOT-CA)
created: 2026-02-09 (Monday)
EOF

# Retrieve
gopass show ADMINISTRATIO/servers/home-dc01/Administrator
gopass show -c ADMINISTRATIO/servers/home-dc01/dsrm

Promote to DC (new forest):

Install-ADDSForest -DomainName "inside.domusdigitalis.dev" -DomainNetbiosName "DOMUS" -ForestMode "WinThreshold" -DomainMode "WinThreshold" -InstallDns:$true -SafeModeAdministratorPassword (Read-Host -AsSecureString "DSRM Password") -Force:$true

Verification after reboot:

Get-Service NTDS, Netlogon, DNS | Select-Object Name, Status
Get-ADDomain | Select-Object Name, DNSRoot, NetBIOSName
Get-ADDomainController | Select-Object Name, IPv4Address, OperatingSystem
Results
Name   DNSRoot                   NetBIOSName
----   -------                   -----------
inside inside.domusdigitalis.dev DOMUS

Name      IPv4Address OperatingSystem
----      ----------- ---------------
HOME-DC01 10.50.1.50  Windows Server 2025 Standard Evaluation

Key Learnings:

  • Use separate gopass entries: /Administrator, /dsrm, /meta

  • Heredoc for multi-line gopass inserts: gopass insert path << 'EOF'

  • One-liner PowerShell works better over SSH than multi-line with backticks

  • DC promotion restarts server automatically

  • After promotion, login shows domain prefix: domus\administrator

Linux Command Reference

sed - print line range:

# Print lines 170-240
sed -n '170,240p' file.adoc

# Insert BEFORE line 220
sed -i '220i\Your text here' file.adoc

# Insert AFTER line 220
sed -i '220a\Your text here' file.adoc

# Preview before editing
sed -n '218,222p' file.adoc

Full sed reference: Aethelred-Codex/The_Codex/13_AsciiDoc/10_COMMAND_ARSENAL/11_DEVELOPMENT_LANGUAGES/2025-CMD-11-D-004-2025-DEV-022-sed-stream-editor-mastery.adoc

Learnings

  • VirtIO drivers for Windows 2025: use 2k25\amd64 folder (or 2k22 fallback)

  • netapi pfsense dns update requires ALL fields: --id, --host, --domain, --ip

  • WLC policy tag WLAN names are case-sensitive and character-sensitive (DOMUS_IoT ≠ Domus-IoT)

  • ISE rejected endpoints need release-rejected even if group assignment is correct

  • DataConnect MAC format: 04:5F:B9:78:02:20 (colons, uppercase)

  • sed -n '170,240p' prints lines 170-240 without modifying file

  • Windows 100% memory in Cockpit is normal - aggressive caching behavior

Session 10: Linux Workstation DC Migration (modestus-razer)

Goal: Update krb5.conf from old DC hostname (dc-01) to new DC (home-dc01).

Verify DC is operational:

PS C:\Users\Administrator> Get-Service DNS, Netlogon, NTDS | Select-Object Name, Status

Name      Status
----      ------
DNS       Running
Netlogon  Running
NTDS      Running
PS C:\Users\Administrator> Get-ADDomain | Select-Object Name, DNSRoot, NetBIOSName

Name   DNSRoot                   NetBIOSName
----   -------                   -----------
inside inside.domusdigitalis.dev DOMUS
PS C:\Users\Administrator> Get-ADDomainController | Select-Object Name, IPv4Address, OperatingSystem

Name      IPv4Address OperatingSystem
----      ----------- ---------------
HOME-DC01 10.50.1.50  Windows Server 2025 Standard Evaluation

Preview krb5.conf changes (never sed blindly):

$ sed -n 's/dc-01\.inside\.domusdigitalis\.dev/home-dc01.inside.domusdigitalis.dev/gp' /etc/krb5.conf

        kdc = home-dc01.inside.domusdigitalis.dev
        admin_server = home-dc01.inside.domusdigitalis.dev

Apply changes:

$ sudo sed -i 's/dc-01\.inside\.domusdigitalis\.dev/home-dc01.inside.domusdigitalis.dev/g' /etc/krb5.conf

Verify configuration:

$ cat /etc/krb5.conf

[libdefaults]
    default_realm = INSIDE.DOMUSDIGITALIS.DEV
    dns_lookup_realm = false
    dns_lookup_kdc = true

[realms]
    INSIDE.DOMUSDIGITALIS.DEV = {
        kdc = home-dc01.inside.domusdigitalis.dev
        admin_server = home-dc01.inside.domusdigitalis.dev
    }

[domain_realm]
    .inside.domusdigitalis.dev = INSIDE.DOMUSDIGITALIS.DEV
    inside.domusdigitalis.dev = INSIDE.DOMUSDIGITALIS.DEV

Refresh Kerberos and SSSD:

$ kdestroy -A
$ sudo systemctl restart sssd
$ kinit Administrator@INSIDE.DOMUSDIGITALIS.DEV
Password for Administrator@INSIDE.DOMUSDIGITALIS.DEV: <from gopass>
No output from kinit indicates success (Unix convention: silence = success).

Verification:

$ klist
$ sudo sssctl domain-status inside.domusdigitalis.dev
Expected output
Online status: Online

Active servers:
AD Domain Controller: home-dc01.inside.domusdigitalis.dev

Summary:

Item Status

DC Services (DNS, Netlogon, NTDS)

Running

Domain Controller

HOME-DC01 (10.50.1.50) - Windows Server 2025

krb5.conf updated

dc-01 → home-dc01

SSSD restarted

Complete

Kerberos ticket obtained

Administrator@INSIDE.DOMUSDIGITALIS.DEV

Documentation: ise-linux::04-linux-client/domain-join.adoc - DC Migration in domus-ise-linux

Tomorrow

  • Test 802.1X authentication against new DC

  • Verify SSSD domain status

  • Update remaining workstations (modestus-p50, modestus-aw)