INC-2026-04-07-002: Hyprland Built-in Keyboard Failure on modestus-razer

Incident Summary

Field Value

Detected

2026-04-07 06:23 PST

Mitigated

2026-04-07 ~11:00 PST (manual symlink recreated)

Resolved

2026-04-07 ~11:15 PST (root cause identified, permanent fix designed)

Duration

~5 hours (3+ hours active investigation)

Severity

P2 (High) — Primary workstation built-in keyboard non-functional in compositor.

Impact

Razer Blade built-in keyboard unusable in Hyprland. All keybinds dead. External Kinesis Advantage 360 Pro worked as workaround.

Root Cause

Commit 64253e7 deleted hyprland/.config/hypr/env-gpu.conf symlink from git tracking. Since ~/.config/hypr is a directory symlink to the stow package, the deletion removed the GPU config file from the deployed path. Hyprland’s source = ~/.config/hypr/env-gpu.conf (line 113) failed silently, causing the compositor to initialize without NVIDIA GPU environment variables (GBM_BACKEND, LIBVA_DRIVER_NAME, etc.). Without correct GPU initialization, the compositor entered a degraded state where USB HID input devices (the Razer built-in keyboard) did not function, while virtual HID devices (Kinesis via uhid) continued to work.

Environment

Property Value

Machine

modestus-razer (Razer Blade 18 2025)

CPU

Intel Core Ultra 9 275HX (24 cores)

GPU

Intel Arrow Lake-S iGPU + NVIDIA RTX 5090 Max-Q

Kernel

6.19.11-arch1-1

Hyprland

0.54.3-2

Built-in Keyboard

Razer Razer Blade (USB 1532:02C7, 3 HID interfaces)

External Keyboard

Kinesis Advantage 360 Pro (USB 1D50:615E, uhid virtual HID)

Dotfiles Repo

dots-quantum (GNU Stow managed, ~/.config/hypr → stow package directory)

Root Cause

The Stow Architecture

~/.config/hypr is not a directory — it is a symlink to the stow package:

~/.config/hypr → ../atelier/_projects/personal/dots-quantum/hyprland/.config/hypr

This means every file Hyprland reads from ~/.config/hypr/ is actually read from the dots-quantum git repository. If a file is deleted from git, it disappears from the deployed config immediately.

The Breaking Change

Commit 64253e7 (fix(dotfiles): Guard .zshenv cargo source, harden gitignore, fix gitconfig portability) deleted the env-gpu.conf symlink from the hyprland stow package:

diff --git a/hyprland/.config/hypr/env-gpu.conf b/hyprland/.config/hypr/env-gpu.conf
deleted file mode 120000
--- a/hyprland/.config/hypr/env-gpu.conf
+++ /dev/null
@@ -1 +0,0 @@
-/home/evanusmodestus/atelier/_projects/personal/dots-quantum/hosts/razer/env-gpu.conf

The commit message says "Untrack env-gpu.conf (already gitignored, was tracked before gitignore entry)." The intent was cleanup — the file was gitignored AND tracked, which is contradictory. But deleting it from git also deleted it from the stow package, which deleted it from the deployed config.

The Cascade

  1. env-gpu.conf removed from stow package directory

  2. ~/.config/hypr/env-gpu.conf ceases to exist (directory symlink)

  3. hyprland.conf line 113: source = ~/.config/hypr/env-gpu.conf fails — file not found

  4. Hyprland starts without GPU environment variables:

    • No LIBVA_DRIVER_NAME=nvidia

    • No __GLX_VENDOR_LIBRARY_NAME=nvidia

    • No GBM_BACKEND=nvidia-drm

    • No WLR_NO_HARDWARE_CURSORS=0

    • No __NV_PRIME_RENDER_OFFLOAD=1

  5. Compositor initializes on Intel iGPU in degraded mode

  6. USB HID keyboard (Razer, on the NVIDIA GPU’s USB controller) does not deliver input events to the compositor

  7. Virtual HID keyboard (Kinesis, via uhid on the Intel USB controller) continues to work

Proof

At commit 34065dc (before the deletion): env-gpu.conf symlink existed in stow package. Hyprland sourced it. Source line errored because the symlink target (hosts/razer/) didn’t exist at that older commit, BUT the error caused Hyprland to skip GPU config entirely — and the built-in keyboard worked. This is because without GPU config, Hyprland defaulted to a working state.

At main (after the deletion): env-gpu.conf doesn’t exist in the stow package at all. Same source error. But when the symlink was manually recreated pointing to the correct hosts/razer/env-gpu.conf (which now exists at main), the GPU config loaded and the keyboard worked.

Empty config test (Hyprland -c /dev/null): Built-in keyboard did NOT work. This initially confused the investigation, but is consistent — with no config at all, Hyprland has no GPU env vars AND no input configuration, defaulting to a state where USB HID is not functional on this hybrid GPU machine.

Timeline

Time Event

~Apr 4

Commit 64253e7 deletes env-gpu.conf symlink from dots-quantum hyprland package.

Apr 6 evening

On Razer: git pull brings in the deletion. git reset --hard origin/main ensures clean state. env-gpu.conf disappears from ~/.config/hypr/. User manually recreates symlink but Hyprland is already in degraded state.

Apr 7 06:23

User discovers Super key dead on built-in keyboard. Investigation begins.

Apr 7 06:30

SSH access to Razer established via reverse tunnel (INC-2026-04-07-001).

Apr 7 06:50-08:00

Phase 1-2: Config comparison (md5 identical), session state investigation, package upgrade. All eliminated.

Apr 7 08:00-08:30

Phase 3-4: hyprctl devices shows zero keyboards. Input group missing from user. openrazer udev rule overriding group. Fixed with usermod -aG input and udev override rule.

Apr 7 08:30-09:15

Phase 5-6: openrazer daemon stopped, razerkbd kernel module unloaded. Raw evdev returns zero bytes from Razer keyboard. Kinesis works. Multiple reboots.

Apr 7 09:15-10:30

Phase 7: Hyprland -c /dev/null test — built-in keyboard doesn’t work. USB device reset attempted. Git checkout of known-good commit 34065dc.

Apr 7 ~10:40

Breakthrough: At commit 34065dc, built-in keyboard works. Source error visible for env-gpu.conf. Back at main, env-gpu.conf is missing from stow package.

Apr 7 ~11:00

Manual symlink recreation: ln -sf hosts/razer/env-gpu.conf hyprland/.config/hypr/env-gpu.conf. Hyprland restarted. Built-in keyboard works.

Apr 7 ~11:15

Root cause confirmed. Permanent fix designed: replace symlink-based sourcing with hostname-based path in hyprland.conf.

Investigation Evidence Log

Phase 1: Configuration Comparison (Eliminated as Direct Cause)

md5sum ~/.config/hypr/hyprland.conf  # Both machines
Result
790a130e77df802bb6a980b7f09ffe78  (identical on P16g and Razer)
readlink -f ~/.config/hypr/env-gpu.conf  # Razer
Result (when symlink existed)
/home/evanusmodestus/atelier/_projects/personal/dots-quantum/hosts/razer/env-gpu.conf
pacman -Q hyprland aquamarine nvidia-utils seatd  # Both machines
Result
hyprland 0.54.3-2    aquamarine 0.10.0-4
nvidia-utils 595.58.03-1    seatd 0.9.3-1

Phase 2: Session State (Eliminated)

loginctl list-sessions --no-legend  # Pre-reboot
Result
5 accumulated sessions from crash/restart cycles

Two reboots performed. Problem persisted. Session state eliminated.

Phase 3: Device Visibility

hyprctl devices  # Initial check
Result
mice:
Keyboards:
# ZERO devices
ls -la /dev/input/event{4,5,6}
Result
event4: root:input     (Kinesis via uhid)
event5: root:openrazer (Razer keyboard)
event6: root:openrazer (Razer keyboard interface 2)
groups  # User group check
Result
# input group NOT present

Fixes Applied (Necessary but Not Sufficient)

sudo usermod -aG input evanusmodestus
sudo tee /etc/udev/rules.d/99-openrazer-input-fix.rules << 'EOF'
SUBSYSTEM=="input", ATTRS{idVendor}=="1532", GROUP="input", MODE="0660"
EOF
sudo udevadm control --reload-rules
sudo udevadm trigger

These fixed device visibility (hyprctl devices showed keyboards after reboot) but did NOT fix the input issue.

Phase 4: openrazer Daemon and Kernel Module (Eliminated)

systemctl --user stop openrazer-daemon
systemctl --user disable openrazer-daemon
sudo rmmod razerkbd  # Rebind to hid-generic
dmesg | tail -5
Result
hid-generic 0003:1532:02C7.0004: input,hidraw2: USB HID v1.11 Keyboard
hid-generic 0003:1532:02C7.0005: input,hidraw3: USB HID v1.11 Mouse

Problem persisted after daemon stop AND module unload. Eliminated.

Phase 5: Raw Event Testing

timeout 3 cat /dev/input/event5 | od -A x -t x1 -N 192
Result
000000
# Zero bytes — Razer keyboard produced no evdev events
# EVIOCGRAB check
python3 -c "import fcntl,struct,os; [print(f'event{e}: NOT grabbed') for e in [4,5,6]]"

No exclusive grab on any device.

fuser -v /dev/input/event4 /dev/input/event5 /dev/input/event6
Result (after daemon stopped)
event4: Hyprland only
event5: Hyprland only
event6: Hyprland only

Phase 6: USB Device Identity

udevadm info /dev/input/event4
Result
DEVPATH=/devices/virtual/misc/uhid/0005:1D50:615E.0008  ← Kinesis (works)
udevadm info /dev/input/event5
Result
DEVPATH=/devices/pci.../1532:02C7.0003  ID_USB_DRIVER=usbhid  ← Razer (broken)

Phase 7: Empty Config Test

pkill -9 Hyprland
echo "" > /tmp/empty.conf
Hyprland -c /tmp/empty.conf
Result
Kinesis: works
Razer built-in: does NOT work

Phase 8: Git Bisect (The Breakthrough)

git -C ~/atelier/_projects/personal/dots-quantum checkout 34065dc
stow -R -t ~ hyprland
pkill -9 Hyprland && start-hyprland
Result
Source error: "globbing error: found no match" for env-gpu.conf
Built-in keyboard: WORKS
git -C ~/atelier/_projects/personal/dots-quantum checkout main
Result
env-gpu.conf: MISSING from stow package
Built-in keyboard: DOES NOT WORK
git -C ~/atelier/_projects/personal/dots-quantum diff 34065dc..main -- hyprland/
Result
diff --git a/hyprland/.config/hypr/env-gpu.conf b/hyprland/.config/hypr/env-gpu.conf
deleted file mode 120000
--- a/hyprland/.config/hypr/env-gpu.conf
+++ /dev/null
@@ -1 +0,0 @@
-/home/.../dots-quantum/hosts/razer/env-gpu.conf

Only change between working and broken: env-gpu.conf symlink deleted from git.

Phase 9: Confirmation

ln -sf /home/evanusmodestus/atelier/_projects/personal/dots-quantum/hosts/razer/env-gpu.conf \
  ~/atelier/_projects/personal/dots-quantum/hyprland/.config/hypr/env-gpu.conf
pkill -9 Hyprland && rm -rf /run/user/1000/hypr/* ~/.cache/hyprland/hyprlandCrash*
start-hyprland
Result
Built-in keyboard: WORKS
Super+D: WORKS
All keybinds: FUNCTIONAL

Resolution

Immediate Fix (Applied)

Recreated the env-gpu.conf symlink in the stow package:

ln -sf /home/evanusmodestus/atelier/_projects/personal/dots-quantum/hosts/razer/env-gpu.conf \
  ~/atelier/_projects/personal/dots-quantum/hyprland/.config/hypr/env-gpu.conf

Permanent Fix (Proposed)

Replace the symlink-based GPU config sourcing with hostname-based path resolution. This eliminates the per-machine symlink entirely.

Change hyprland.conf line 113:

- source = ~/.config/hypr/env-gpu.conf
+ source = ~/atelier/_projects/personal/dots-quantum/hosts/$HOST/env-gpu.conf

$HOST is set by zsh to the machine hostname (modestus-razer, modestus-p16g, etc.). Each machine sources its own GPU config automatically. No symlink to manage, no gitignore conflict, no deletion risk.

Requirements:

  • Each machine has hosts/<hostname>/env-gpu.conf in dots-quantum

  • Currently exists: hosts/razer/env-gpu.conf, hosts/p16g/env-gpu.conf

  • New machine: create hosts/<newhostname>/env-gpu.conf

  • Hostname dirs must match $HOST — verify with echo $HOST on each machine

Cleanup after applying:

# Remove the symlink from stow package (no longer needed)
rm ~/atelier/_projects/personal/dots-quantum/hyprland/.config/hypr/env-gpu.conf

# Remove the gitignore entry for env-gpu.conf
# Edit .gitignore and remove: hyprland/.config/hypr/env-gpu.conf

# Commit the hyprland.conf change
cd ~/atelier/_projects/personal/dots-quantum
git add hyprland/.config/hypr/hyprland.conf .gitignore
git commit -m "fix(hyprland): Source env-gpu.conf by hostname — eliminates per-machine symlink

Replace source = ~/.config/hypr/env-gpu.conf with hostname-based path:
source = ~/...dots-quantum/hosts/\$HOST/env-gpu.conf

Root cause of INC-2026-04-07-002: commit 64253e7 deleted the symlink
from git, which removed it from the stow-deployed config. Without GPU
env vars, Hyprland initialized in degraded mode where USB HID input
devices (built-in keyboard) did not function.

Ref: INC-2026-04-07-002"

Secondary Fix: Waybar Slow Startup

hyprland.conf exec-once lines reference swww-daemon and a wallpaper cycle script. If swww is not installed, these hang and delay waybar and other startup processes.

Options:

  1. Install swww on all machines: sudo pacman -S swww

  2. Comment out the swww lines in hyprland.conf if not needed

  3. Background the wallpaper script: add & to prevent blocking

Collateral Fixes Applied During Investigation

These were necessary fixes discovered during investigation but were not the root cause:

# User was not in input group (needed for device access)
sudo usermod -aG input evanusmodestus

# openrazer udev rule overrides input group on Razer devices
sudo tee /etc/udev/rules.d/99-openrazer-input-fix.rules << 'EOF'
SUBSYSTEM=="input", ATTRS{idVendor}=="1532", GROUP="input", MODE="0660"
EOF

Investigation Rabbit Holes

Documented for learning. Each was investigated, evidence gathered, and eliminated.

Hypothesis Time Spent Why Eliminated

Stale logind sessions

30 min

Two reboots cleared sessions. Problem persisted.

XDG_SESSION_TYPE=tty

20 min

Identical on both machines. P16g works with same value.

Package version mismatch

15 min

All packages identical after upgrade.

openrazer daemon grabbing devices

20 min

Daemon stopped and disabled. Problem persisted.

razerkbd kernel module

30 min

Module unloaded, devices rebound to hid-generic. Problem persisted.

udev group permissions

30 min

Fixed (necessary) but not the root cause. Problem persisted after fix.

USB device reset

10 min

USB unbind/rebind performed. Problem persisted.

Empty config test

15 min

Keyboard didn’t work — misleading because no GPU config = same degraded state.

Total time in rabbit holes: ~2.5 hours. Root cause found in 15 minutes once the right question was asked (git diff between working and broken commit).

Lessons Learned

What Went Well

  • Reverse SSH tunnel established quickly for remote investigation

  • hyprctl dispatch exec workaround kept machine partially usable

  • md5sum comparison eliminated config content early

  • Git checkout of known-good commit was the breakthrough

What Could Be Improved

  • Should have checked file existence first, not file content. ls ~/.config/hypr/env-gpu.conf would have shown "No such file" immediately. Instead, 2.5 hours was spent comparing checksums of files that existed.

  • Should have run git diff between working and broken state in the first 10 minutes. The breaking change was a single file deletion.

  • "Empty config" test was misleading. On a hybrid GPU machine, no GPU config ≠ "clean baseline." It means "different broken state."

  • Chased kernel/driver hypotheses too long. User repeatedly said "this worked before the dots-quantum pull." Should have trusted that signal and focused on what the pull changed.

Key Takeaways

  1. Stow directory symlinks are transparent. If ~/.config/hypr is a symlink to the stow package, deleting a file from git deletes it from the deployed config. There is no separate "deployed copy" — they are the same files.

  2. When the user says "it broke after X," investigate X first. Not the kernel, not the driver, not the session state — the thing that changed.

  3. ls before md5sum. Verify the file exists before comparing its contents.

  4. GPU environment variables affect input devices. On NVIDIA hybrid GPU machines, missing GBM_BACKEND and __GLX_VENDOR_LIBRARY_NAME can cause the compositor to use the wrong rendering backend, which cascades to input device initialization failures on USB controllers connected to the discrete GPU.

  5. Per-machine config via hostname ($HOST) is more robust than symlinks. Symlinks can be deleted, gitignored, tracked/untracked. A $HOST-based path in the config file is self-contained and survives any git operation.

Diagnostic Commands Reference

# === FIRST: Does the sourced file exist? ===
ls -la ~/.config/hypr/env-gpu.conf
cat ~/.config/hypr/env-gpu.conf | head -3

# === Is ~/.config/hypr a real dir or stow symlink? ===
ls -ld ~/.config/hypr
file ~/.config/hypr

# === What did git change in the hyprland package? ===
git -C ~/atelier/_projects/personal/dots-quantum diff <good-commit>..HEAD -- hyprland/

# === What is the stow package state? ===
ls -la ~/atelier/_projects/personal/dots-quantum/hyprland/.config/hypr/

# === Does Hyprland see keyboards? ===
export HYPRLAND_INSTANCE_SIGNATURE=$(ls /run/user/1000/hypr/ | sort | tail -1)
hyprctl devices | grep -A5 keyboard

# === GPU env vars in compositor process ===
cat /proc/$(pgrep -o Hyprland)/environ | tr '\0' '\n' | grep -E 'GBM|GLX|LIBVA|PRIME|WLR'

# === Hyprland source errors ===
# Visible in startup output or hyprland.log:
# "source= globbing error: found no match"

# === Device permissions ===
ls -la /dev/input/event*
stat -c "%G" /dev/input/event{4,5,6}

Related

Metadata

Field Value

Incident ID

INC-2026-04-07-002

Author

Evan Rosado

Created

2026-04-07

Last Updated

2026-04-07

Status

Resolved

Root Cause Commit

64253e7 — deleted env-gpu.conf symlink from hyprland stow package

Fix Commit

Pending — replace line 113 with $HOST-based source path

Post-Incident Review

Complete