Daily Worklog: 2026-02-14

Overview

Date: 2026-02-14 (Saturday)

Location: Remote

Focus: Threat Hunting Documentation, System Performance Triage, Makefile Fixes, Xianming Linux AD Auth, GSEC Certification

Carried Over from 2026-02-13

  • Practice Spanish oral presentation (Ciudad vs Campo)

  • SIEM gap analysis - review QRadar skills

  • Xianming Ding Linux researcher request

  • Continue domus-siem-ops content development

  • Update domus-push function with missing repos (done - dotfiles-optimus)

  • Fill remaining 20 domus-linux-ops stub pages

Session: Threat Hunting Documentation

Modular Examples Created

Created comprehensive threat hunting examples in domus-linux-ops/examples/hunting/:

File Content

persistence.adoc

Cron, systemd, shell rc, SSH keys audit

filesystem.adoc

Recent files, hidden files, SUID/SGID, world-writable

network.adoc

Listening ports, established connections, DNS

process.adoc

Process tree, suspicious processes, orphaned procs, lsof

users.adoc

Login activity, account anomalies

logs.adoc

Auth logs, sudo usage, bash history

rootkit.adoc

Quick checks, binary verification, kernel modules

incident-response.adoc

State capture, process freeze, IP blocking, baselines

elite.adoc

Advanced: timeline analysis, memory forensics, LOLBins, C2 detection

Index Page

Created security/hunting/index.adoc with tagged includes from examples.

Session: Runaway Process Triage (Real Incident)

Issue

System fans spinning up unexpectedly. CPU investigation revealed:

USER       PID %CPU %MEM  TIME COMMAND
user   3603830  103  0.3 2369:32 hyprlock    # 39+ hours CPU time!

Root Cause

hyprlock orphaned (PPID=1) and stuck in busy loop for 38+ hours.

$ ps -p 3603830 -o pid,ppid,stat,etime,cmd
    PID    PPID STAT     ELAPSED CMD
3603830       1 Rl    1-14:19:52 hyprlock

Additional Finding: Hyprland 59% CPU

Caused by continuous animation loop in ~/.config/hypr/hyprland.conf:

animation = borderangle, 1, 120, linear, loop

The loop flag forces constant border animation = constant CPU burn.

Resolution

# Kill orphaned hyprlock
kill 3603830

# Disable borderangle animation
sed -i 's/animation = borderangle/#animation = borderangle/' ~/.config/hypr/hyprland.conf

Documentation Added

Added to domus-linux-ops/troubleshooting/system/performance.adoc:

  • Runaway Process Triage section

  • Orphaned process detection (PPID=1)

  • Process state indicators table (R, S, D, Z, T)

  • Safe kill workflow

  • Bulk cleanup commands

Added to examples/hunting/process.adoc:

  • orphaned-procs tag

  • runaway-detection tag

Session: Kroki Auto-Stop Fix

Issue

Kroki containers running for 39 hours after builds complete. Should stop automatically.

Fix

Updated domus-docs/Makefile:

# Before
all: diagrams local

# After
all: diagrams local kroki-stop

Commit

fix(makefile): Auto-stop Kroki after default build

Session: dotfiles-optimus Updates

Changes

  • Added domus-o11y-ops to domus-push function

  • Fixed domus-push to explicitly push origin main

  • Added domus-check function - builds all repos, reports FATAL/error/warning

  • Added domus-o11y and do11y aliases

Session: Xianming Linux AD Auth Deployment Preparation

Context

Xianming Ding (Research Computing) requested Linux workstation with AD-authenticated SSH. Current Research_Onboard dACL blocks Kerberos/LDAP ports.

Work Completed

1. NetworkManager Password Flags Fix

Identified and documented the critical fix for "Secrets were required" error:

802-1x.identity-flags 0 \
802-1x.private-key-password-flags 4 \

  • identity-flags=0 - Store identity in file, not as secret

  • private-key-password-flags=4 - Key has no password (NOT_REQUIRED)

Source: Home enterprise testing documented in domus-ise-linux/04-linux-client/networkmanager-wired.adoc

2. Deployment Document Updated

Enhanced DEPLOY-2026-02-14-xianming-ding-linux-ad-auth.adoc:

  • Added Phase 0: 802.1X Supplicant Configuration

  • Documented password flags fix with examples

  • Added certificate chain creation steps

  • Connection file verification commands

This document now serves as the standard template for future Linux AD Auth deployments.

3. GSEC Certification Roadmap Created

Created PLAN-2026-02-gsec-certification.adoc:

  • SANS SEC401: Security Essentials - Network, Endpoint, and Cloud

  • 6-week study plan aligned with existing expertise

  • Gap analysis: Cloud security and SIEM operations

  • Synergies with RHCSA/RHCE and CISSP paths

Rationale: GSEC positions as security generalist with deep Linux expertise - bridges network engineering and security operations.

Deployment Strategy (Xianming)

Phase 1 (Immediate): Test SSH with current Research_Onboard (MAB) policy

  • Verify certificates issued and domain-joined

  • Apply NetworkManager config with password flags

  • Test AD connectivity to DCs

Phase 2 (After SSH confirmed): Deploy zero-trust dACL with 802.1X/EAP-TLS

  • dACL: DACL_LINUX_RESEARCH_AD_AUTH

  • AuthZ Profile: Linux_Research_AD_Auth

  • Permits AD ports, blocks RFC1918 lateral movement

Deferred: BIND DNS Architecture (Home Enterprise)

Environment Separation:

  • Home Enterprise: inside.domusdigitalis.dev - pattern validation, testing

  • CHLA Production: la.ad.chla.org - completely separate network

BIND discussion applies to home enterprise only. Never mix configurations.

Considering BIND for home enterprise DNS architecture:

Approach Use Case

AD-integrated DNS (current)

Windows DNS for inside.domusdigitalis.dev. Required for AD SRV records.

BIND as forwarder

BIND handles recursive/external, forwards inside.domusdigitalis.dev → Windows DNS

BIND as secondary

BIND holds read-only copy of AD zones (zone transfer from Windows)

pfSense Unbound (current)

Conditional forwarding: AD domain → Windows DNS, else → upstream

Current setup: pfSense Unbound forwards AD queries to Windows DNS. Works for all AD authentication scenarios.

Deferred for after Xianming deployment pattern validation completes.

Session: netapi Bulk Endpoint Commands & Static Assignment Fix

Problem: ciscoisesdk staticGroupAssignment Bug

ISE ERS API via ciscoisesdk library silently ignores staticGroupAssignment parameter on both endpoint creation and update. This allows ISE Profiler to override manual group assignments, moving devices back to "Unknown" or auto-profiled groups.

Fix: Direct HTTP Requests

Bypassed ciscoisesdk with direct requests library calls to ISE ERS API:

Method Fix

update-endpoint-group

Direct PUT to /ers/config/endpoint/{id}

create_endpoint (ERS client)

Direct POST to /ers/config/endpoint

New Commands Added

bulk-create-endpoint

Creates endpoints if they don’t exist, assigns to group with static assignment.

# From arguments
netapi ise bulk-create-endpoint DOMUS_Printers AA:BB:CC:DD:EE:03 11:22:33:44:55:66

# From file
netapi ise bulk-create-endpoint IoT_Devices --file /tmp/iot-macs.txt

# With description
netapi ise bulk-create-endpoint DOMUS_Printers AA:BB:CC:DD:EE:FF \
    --description "Network Printer - Building A"

# Update existing endpoints too
netapi ise bulk-create-endpoint Blacklist --file macs.txt --update-existing

bulk-update-endpoint-group

Moves existing endpoints to different identity group.

# From arguments
netapi ise bulk-update-endpoint-group DOMUS_Printers AA:BB:CC:DD:EE:FF

# From stdin (pipeline)
netapi ise get-endpoints --group Unknown -o json | jq -r '.[].mac' | \
    netapi ise bulk-update-endpoint-group Profiled --stdin

Validated Output

Processing 3 endpoint(s) → group 'DOMUS_Printers' (static=True)
  Description: Test IoT Device
✓ [CREATED] AA:BB:CC:DD:EE:03 → DOMUS_Printers
✓ [CREATED] 11:22:33:44:55:66 → DOMUS_Printers
✓ [CREATED] 22:33:44:55:66:77 → DOMUS_Printers

✓ Completed: 3 created

Verification shows correct group assignment:

Identity Group
  Group                   DOMUS_Printers
  Static Assignment       True

Commits

  • fix(ise): Use direct PUT for update-endpoint-group static assignment

  • feat(ise): Add bulk-update-endpoint-group command

  • feat(ise): Add bulk-create-endpoint command

  • fix(ers): Use direct POST for create_endpoint

Documentation Updated

  • domus-netapi-docs/cli/ise/ers/endpoints.adoc - Added bulk command docs

  • domus-infra-ops/runbooks/printer-eap-ttls.adoc - Updated with FreeIPA LDAP

Session: Brother Printer EAP-TTLS Configuration

Architecture Decision

Changed from ISE Internal Users to FreeIPA LDAP as external identity source for printer EAP-TTLS authentication.

Rationale: Centralized identity management via FreeIPA (to be deployed on kvm-01).

Endpoint Group Created

netapi ise create-endpoint-group "DOMUS_Printers" \
    --description "DOMUS network printers - EAP-TTLS authentication"

Printer Registered

netapi ise update-endpoint-group "90:32:4B:B8:FC:FE" "DOMUS_Printers" --static

Remaining Tasks

  • Create DACL (DACL_CORP_PRINTERS)

  • Create AuthZ Profile (AuthZ_DOMUS_Printers)

  • Deploy FreeIPA on kvm-01

  • Join FreeIPA to ISE as external LDAP identity source

  • Create printer service account in FreeIPA

  • Configure Brother printer with FreeIPA credentials

Session: Keycloak SAML Client Update via REST API

Context

After restoring ISE from ise-02 backup to ise-01, needed to update Keycloak SAML client redirect URIs from ise-02 to ise-01. Used REST API instead of GUI for reproducibility and documentation.

Keycloak Admin REST API Workflow

1. Retrieve Admin Token

KC_ADMIN_PASS="<password>"

KC_TOKEN=$(curl -s -X POST \
  "https://keycloak-01.inside.domusdigitalis.dev:8443/realms/master/protocol/openid-connect/token" \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "username=admin" \
  -d "password=$KC_ADMIN_PASS" \
  -d "grant_type=password" \
  -d "client_id=admin-cli" \
  --insecure | jq -r '.access_token')

Key Points:

  • Token endpoint is /realms/master/protocol/openid-connect/token

  • client_id=admin-cli is the built-in admin client

  • Token is valid for ~60 seconds by default

2. List SAML Clients

curl -s "https://keycloak-01.inside.domusdigitalis.dev:8443/admin/realms/domusdigitalis/clients" \
  -H "Authorization: Bearer $KC_TOKEN" \
  --insecure | jq '.[] | select(.protocol=="saml") | {id, clientId, name}'

Output:

{
  "id": "0d7b3b6b-d32f-49a0-9563-6cc8e645b59c",
  "clientId": "http://CiscoISE/a486c6ef-6c77-4bc1-bf6d-4e479b3aeae8",
  "name": "Cisco ISE Admin Portal (ise-02)"
}

3. Get Full Client Configuration

CLIENT_UUID="0d7b3b6b-d32f-49a0-9563-6cc8e645b59c"

curl -s "https://keycloak-01.inside.domusdigitalis.dev:8443/admin/realms/domusdigitalis/clients/$CLIENT_UUID" \
  -H "Authorization: Bearer $KC_TOKEN" \
  --insecure > /tmp/ise-saml-client.json

Key SAML Attributes:

Attribute Value

redirectUris

Array of allowed redirect URLs

attributes.saml_assertion_consumer_url_post

ACS URL for SAML POST binding

attributes.saml_idp_initiated_sso_url_name

URL fragment for IdP-initiated SSO

4. Update with sed and PUT

# Replace all ise-02 → ise-01
sed 's/ise-02/ise-01/g' /tmp/ise-saml-client.json > /tmp/ise-saml-client-updated.json
# Apply update via PUT
curl -s -X PUT \
  "https://keycloak-01.inside.domusdigitalis.dev:8443/admin/realms/domusdigitalis/clients/$CLIENT_UUID" \
  -H "Authorization: Bearer $KC_TOKEN" \
  -H "Content-Type: application/json" \
  -d @/tmp/ise-saml-client-updated.json \
  --insecure -w "\nHTTP_STATUS: %{http_code}\n"

Expected: HTTP_STATUS: 204 (No Content = success)

5. Verify Update

curl -s "https://keycloak-01.inside.domusdigitalis.dev:8443/admin/realms/domusdigitalis/clients/$CLIENT_UUID" \
  -H "Authorization: Bearer $KC_TOKEN" \
  --insecure | jq '{name, rootUrl, redirectUris, saml_assertion_consumer_url_post: .attributes.saml_assertion_consumer_url_post}'

Key Learnings

Concept Notes

Token retrieval

Use /realms/master/…​ even for other realm operations

Admin endpoints

All under /admin/realms/{realm}/…​

Client ID vs UUID

clientId is the SAML Entity ID; id is internal UUID for API calls

PUT requires full object

GET, modify, PUT - Keycloak doesn’t support PATCH for clients

jq for filtering

.[] | select(.protocol=="saml") filters SAML clients only

ISE SAML Note

ISE SAML IdP configuration is GUI-only - no ERS or OpenAPI endpoint exists for SAML imports. The Keycloak side can be fully automated via REST API.

Pending Tasks

  • Practice Spanish oral presentation (Ciudad vs Campo)

  • SIEM gap analysis - review QRadar skills

  • Xianming Ding Linux researcher request - deployment doc updated

  • Continue domus-siem-ops content development

  • Fill remaining 20 domus-linux-ops stub pages

  • Fix hyprland borderangle animation (comment out)

  • Push domus-linux-ops threat hunting commits

  • Xianming deployment: Test SSH under Research_Onboard

  • Xianming deployment: Apply zero-trust dACL after SSH confirmed

References