INC-2026-03-10: Investigation

Investigation

Initial Triage

# Service status
systemctl status vault-backup.service
# Output: failed, exit code 14

# Logs
journalctl -u vault-backup.service --no-pager -n 50
# Output: "Failed to exec ssh: Permission denied (13)"

# Manual test (worked)
sudo bash -c 'tar -czf /tmp/test.tar.gz ... && rsync ...'
# Output: SUCCESS

SELinux Analysis

# Check for AVC denials
sudo ausearch -m avc --start today | grep rsync

# Output:
type=AVC msg=audit(...): avc: denied { execute_no_trans } for pid=3144
  comm="rsync" path="/usr/bin/ssh"
  scontext=system_u:system_r:rsync_t:s0
  tcontext=system_u:object_r:ssh_exec_t:s0

Findings

  1. Service runs as User=root in systemd

  2. Manual sudo runs in unconfined_t context - no SELinux restrictions

  3. systemd service runs rsync in rsync_t domain

  4. rsync_t cannot transition to ssh_exec_t by default

  5. Each denial only appears after previous one is fixed (whack-a-mole)

Root Cause

SELinux policy restriction: The rsync_t SELinux domain does not have permission to execute binaries labeled ssh_exec_t.

When rsync runs under systemd, it operates in the confined rsync_t domain. When it tries to spawn ssh for the remote transfer, SELinux blocks the execution.

Why manual worked: Running sudo bash -c '…​' from an interactive shell runs in unconfined_t, which has no restrictions.

Required permissions (5 total):

allow rsync_t ssh_exec_t:file { execute_no_trans map };
allow rsync_t ssh_home_t:dir search;
allow rsync_t ssh_home_t:file { getattr open read };
allow rsync_t systemd_conf_t:file { getattr open read };
allow rsync_t initrc_tmp_t:file open;