auditd & Linux Auditing
Quick Reference
# Service management
systemctl status auditd
systemctl start auditd
systemctl enable auditd
# View audit logs
ausearch -m USER_LOGIN -ts today
aureport --summary
# Add audit rule
auditctl -w /etc/passwd -p wa -k passwd_changes
# List current rules
auditctl -l
# Check audit status
auditctl -s
# Generate reports
aureport --login
aureport --file --summary
Understanding the Audit System
Audit Architecture
┌─────────────────────────────────────────────────────────────────┐
│ User Space │
├─────────────────────────────────────────────────────────────────┤
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │
│ │ auditd │ │ audispd │ │ ausearch/aureport │ │
│ │ (daemon) │◄──│ (dispatcher)│ │ (analysis) │ │
│ └──────▲──────┘ └─────────────┘ └─────────────────────┘ │
│ │ │
├─────────┼───────────────────────────────────────────────────────┤
│ │ Kernel Space │
│ ┌──────┴──────┐ │
│ │ auditctl │──►┌─────────────────────────────────────────┐ │
│ │ (rules) │ │ Audit Subsystem │ │
│ └─────────────┘ │ ┌─────────────────────────────────┐ │ │
│ │ │ System Calls / File Access │ │ │
│ │ │ Process Execution / Network │ │ │
│ │ └─────────────────────────────────┘ │ │
│ └─────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
Core Components
| Component | Description |
|---|---|
auditd |
Main audit daemon that writes events to |
auditctl |
Command-line tool to configure audit rules in the running kernel |
ausearch |
Search and query audit logs with filtering |
aureport |
Generate summary reports from audit logs |
audispd |
Audit event multiplexor for real-time processing |
augenrules |
Compiles rules from |
Event Types
| Event Type | Description |
|---|---|
SYSCALL |
System call events (file access, network, process) |
USER_LOGIN |
User login attempts |
USER_AUTH |
Authentication events |
USER_ACCT |
Account modifications |
USER_CMD |
Commands run by users |
ANOM_ |
Anomaly events (potential security issues) |
AVC |
SELinux Access Vector Cache denials |
CRYPTO_ |
Cryptographic events |
CONFIG_CHANGE |
Audit configuration changes |
Configuration Files
Main Configuration
#
# Audit daemon configuration
#
# Log file settings
log_file = /var/log/audit/audit.log
log_format = ENRICHED
log_group = root
# Buffer and write settings
write_logs = yes
freq = 50
flush = INCREMENTAL_ASYNC
num_logs = 5
max_log_file = 8
max_log_file_action = ROTATE
# Space monitoring
space_left = 75
space_left_action = SYSLOG
admin_space_left = 50
admin_space_left_action = SUSPEND
disk_full_action = SUSPEND
disk_error_action = SUSPEND
# Priority and network
priority_boost = 4
name_format = HOSTNAME
distribute_network = no
Key Configuration Options
| Option | Description |
|---|---|
|
|
|
Maximum size in MB before rotation |
|
|
|
Action when |
|
Action for critical low space |
|
|
|
|
Audit Rules
Rule Types
| Type | Flag | Description |
|---|---|---|
Control |
|
Configure audit system behavior |
File system |
|
Watch files/directories for access |
System call |
|
Audit specific system calls |
Exit filter |
|
Audit after syscall completion |
File System Rules
# Watch a file for writes and attribute changes
auditctl -w /etc/passwd -p wa -k passwd_changes
# Watch a directory recursively
auditctl -w /etc/ssh/ -p wa -k sshd_config
# Permission flags:
# r = read
# w = write
# x = execute
# a = attribute change
# Watch all permission types
auditctl -w /etc/shadow -p rwxa -k shadow_access
# Watch for execution of commands
auditctl -w /usr/bin/passwd -p x -k passwd_cmd
auditctl -w /usr/sbin/useradd -p x -k user_admin
auditctl -w /usr/sbin/userdel -p x -k user_admin
System Call Rules
# Basic syscall rule format:
# auditctl -a <action>,<filter> -S <syscall> -F <field>=<value> -k <keyname>
# Monitor file deletions
auditctl -a always,exit -F arch=b64 -S unlink -S unlinkat -S rename -S renameat -k file_deletion
# Monitor file permission changes
auditctl -a always,exit -F arch=b64 -S chmod -S fchmod -S fchmodat -k permission_changes
# Monitor user/group changes
auditctl -a always,exit -F arch=b64 -S setuid -S setgid -S setreuid -S setregid -k priv_escalation
# Monitor mount operations
auditctl -a always,exit -F arch=b64 -S mount -S umount2 -k mount_ops
# Monitor network connections (connect syscall)
auditctl -a always,exit -F arch=b64 -S connect -k network_connect
# Monitor process execution
auditctl -a always,exit -F arch=b64 -S execve -k process_exec
Field Filters
| Field | Description |
|---|---|
|
System architecture ( |
|
User ID, Audit (login) UID |
|
Group ID |
|
Process ID |
|
Parent process ID |
|
Effective user/group ID |
|
Set user/group ID |
|
Syscall success ( |
|
Syscall exit code |
|
Rule key for identification |
|
Full path to file |
|
Directory path |
|
Permission filter ( |
Control Rules
# Delete all rules
auditctl -D
# Set backlog buffer (number of events to buffer)
auditctl -b 8192
# Set failure mode (0=silent, 1=printk, 2=panic)
auditctl -f 1
# Set rate limit (messages per second, 0=unlimited)
auditctl -r 0
# Check status
auditctl -s
# Make rules immutable (requires reboot to change)
auditctl -e 2
Persistent Rules
Rule File Structure
# First rule - delete all existing rules
-D
# Increase buffer size
-b 8192
# Failure mode (1=printk)
-f 1
# ============================================
# File System Watches
# ============================================
# Identity files
-w /etc/passwd -p wa -k identity
-w /etc/group -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/gshadow -p wa -k identity
-w /etc/security/opasswd -p wa -k identity
# Authentication configuration
-w /etc/pam.d/ -p wa -k pam
-w /etc/nsswitch.conf -p wa -k auth_config
-w /etc/login.defs -p wa -k auth_config
# SSH configuration
-w /etc/ssh/sshd_config -p wa -k sshd_config
-w /etc/ssh/sshd_config.d/ -p wa -k sshd_config
# Sudo configuration
-w /etc/sudoers -p wa -k sudoers
-w /etc/sudoers.d/ -p wa -k sudoers
# Network configuration
-w /etc/hosts -p wa -k network_config
-w /etc/resolv.conf -p wa -k network_config
-w /etc/sysconfig/network -p wa -k network_config
# Systemd configuration
-w /etc/systemd/ -p wa -k systemd_config
-w /usr/lib/systemd/ -p wa -k systemd_config
# Cron configuration
-w /etc/crontab -p wa -k cron
-w /etc/cron.d/ -p wa -k cron
-w /etc/cron.daily/ -p wa -k cron
-w /etc/cron.hourly/ -p wa -k cron
-w /etc/cron.monthly/ -p wa -k cron
-w /etc/cron.weekly/ -p wa -k cron
-w /var/spool/cron/ -p wa -k cron
# ============================================
# Privileged Commands
# ============================================
-w /usr/bin/passwd -p x -k privileged_passwd
-w /usr/bin/sudo -p x -k privileged_sudo
-w /usr/bin/su -p x -k privileged_su
-w /usr/sbin/useradd -p x -k privileged_user
-w /usr/sbin/userdel -p x -k privileged_user
-w /usr/sbin/usermod -p x -k privileged_user
-w /usr/sbin/groupadd -p x -k privileged_group
-w /usr/sbin/groupdel -p x -k privileged_group
-w /usr/sbin/groupmod -p x -k privileged_group
# ============================================
# System Call Rules
# ============================================
# File deletion
-a always,exit -F arch=b64 -S unlink -S unlinkat -S rename -S renameat -k delete
# File permission changes
-a always,exit -F arch=b64 -S chmod -S fchmod -S fchmodat -k perm_mod
-a always,exit -F arch=b64 -S chown -S fchown -S fchownat -S lchown -k perm_mod
# Privilege escalation
-a always,exit -F arch=b64 -S setuid -S setgid -S setreuid -S setregid -k setuid
# Module loading
-a always,exit -F arch=b64 -S init_module -S finit_module -k module_load
-a always,exit -F arch=b64 -S delete_module -k module_unload
# Mount operations
-a always,exit -F arch=b64 -S mount -S umount2 -k mount
# Time changes
-a always,exit -F arch=b64 -S adjtimex -S settimeofday -k time_change
-a always,exit -F arch=b64 -S clock_settime -k time_change
-w /etc/localtime -p wa -k time_change
# ============================================
# Make configuration immutable (optional)
# Uncomment only after testing
# ============================================
# -e 2
Split Rule Files
-D
-b 8192
-f 1
-w /etc/passwd -p wa -k identity
-w /etc/group -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /usr/bin/sudo -p x -k privileged
-w /usr/bin/su -p x -k privileged
# Make immutable after all rules loaded
-e 2
Searching Audit Logs
ausearch Basics
# Search by key
ausearch -k passwd_changes
# Search by event type
ausearch -m USER_LOGIN
# Search by time
ausearch -ts today
ausearch -ts yesterday
ausearch -ts recent # Last 10 minutes
ausearch -ts "03/15/2024 09:00:00"
ausearch -ts "03/15/2024" -te "03/16/2024"
# Search by user
ausearch -ua root # By audit UID (login user)
ausearch -ui 1000 # By user ID
# Search by process
ausearch -p 12345 # By PID
ausearch -pp 1 # By parent PID (systemd children)
# Search by executable
ausearch -x /usr/bin/passwd
# Search by file
ausearch -f /etc/passwd
# Search by syscall
ausearch -sc open
ausearch -sc execve
# Search by success/failure
ausearch -sv yes # Successful operations
ausearch -sv no # Failed operations
# Search by hostname/terminal
ausearch -hn webserver01
ausearch -tm pts/0
ausearch Output Formats
# Raw format (default)
ausearch -k identity
# Interpreted (human-readable)
ausearch -k identity -i
# Text format
ausearch -k identity --format text
# CSV format
ausearch -k identity --format csv
# Specific fields only
ausearch -k identity --just-one
Combined Searches
# Failed logins today by specific user
ausearch -m USER_LOGIN -sv no -ts today -ua jsmith
# File modifications by root
ausearch -k identity -ua root -ts "24 hours ago"
# Executed commands as root
ausearch -m EXECVE -ua root -ts today -i
# Network connections by specific process
ausearch -sc connect -x /usr/bin/curl
# All events from a specific session
ausearch -se 12345
Generating Reports
aureport Summary
# Overall summary
aureport --summary
# Summary for today
aureport --summary -ts today
# Summary by time range
aureport --summary -ts "03/01/2024" -te "03/31/2024"
Specific Reports
# Login report
aureport --login
aureport --login --summary
# Authentication report
aureport --auth
aureport --auth --failed
# User report
aureport --user
# File access report
aureport --file
aureport --file --summary
# Executable report
aureport --executable
aureport --executable --summary
# System call report
aureport --syscall
# Anomaly events
aureport --anomaly
# Configuration changes
aureport --config
# Process report
aureport --process
# Terminal activity
aureport --tty
# Crypto events
aureport --crypto
# Key-based report
aureport --key
aureport --key --summary
Real-Time Monitoring
Using tail
# Follow audit log
tail -f /var/log/audit/audit.log
# Follow with grep
tail -f /var/log/audit/audit.log | grep -i passwd
# Follow with ausearch interpretation
tail -f /var/log/audit/audit.log | ausearch -i --input -
audispd Plugins
active = yes
direction = out
path = /sbin/audisp-syslog
type = always
args = LOG_INFO
format = string
active = yes
direction = out
path = builtin_af_unix
type = builtin
args = 0640 /var/run/audispd_events
format = string
Custom Event Processing
active = yes
direction = out
path = /usr/local/bin/audit-handler.sh
type = always
format = string
#!/bin/bash
# Custom audit event handler
while read line; do
# Check for critical events
if echo "$line" | grep -q "key=privileged"; then
echo "$line" >> /var/log/audit/critical.log
# Send alert (example)
# /usr/local/bin/send-alert "$line"
fi
done
Compliance Rulesets
CIS Benchmark Rules
# CIS Benchmark audit rules
# 4.1.3 Ensure events that modify date/time are collected
-a always,exit -F arch=b64 -S adjtimex -S settimeofday -k time-change
-a always,exit -F arch=b32 -S adjtimex -S settimeofday -S stime -k time-change
-a always,exit -F arch=b64 -S clock_settime -k time-change
-a always,exit -F arch=b32 -S clock_settime -k time-change
-w /etc/localtime -p wa -k time-change
# 4.1.4 Ensure events that modify user/group are collected
-w /etc/group -p wa -k identity
-w /etc/passwd -p wa -k identity
-w /etc/gshadow -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/security/opasswd -p wa -k identity
# 4.1.5 Ensure events that modify network environment are collected
-a always,exit -F arch=b64 -S sethostname -S setdomainname -k system-locale
-a always,exit -F arch=b32 -S sethostname -S setdomainname -k system-locale
-w /etc/issue -p wa -k system-locale
-w /etc/issue.net -p wa -k system-locale
-w /etc/hosts -p wa -k system-locale
-w /etc/sysconfig/network -p wa -k system-locale
# 4.1.6 Ensure events that modify MAC policy are collected
-w /etc/selinux/ -p wa -k MAC-policy
-w /usr/share/selinux/ -p wa -k MAC-policy
# 4.1.7 Ensure login and logout events are collected
-w /var/log/lastlog -p wa -k logins
-w /var/run/faillock/ -p wa -k logins
# 4.1.8 Ensure session initiation information is collected
-w /var/run/utmp -p wa -k session
-w /var/log/wtmp -p wa -k logins
-w /var/log/btmp -p wa -k logins
# 4.1.9 Ensure discretionary access control changes are collected
-a always,exit -F arch=b64 -S chmod -S fchmod -S fchmodat -k perm_mod
-a always,exit -F arch=b64 -S chown -S fchown -S fchownat -S lchown -k perm_mod
-a always,exit -F arch=b64 -S setxattr -S lsetxattr -S fsetxattr -k perm_mod
-a always,exit -F arch=b64 -S removexattr -S lremovexattr -S fremovexattr -k perm_mod
# 4.1.10 Ensure unsuccessful unauthorized access attempts are collected
-a always,exit -F arch=b64 -S creat -S open -S openat -S truncate -S ftruncate -F exit=-EACCES -k access
-a always,exit -F arch=b64 -S creat -S open -S openat -S truncate -S ftruncate -F exit=-EPERM -k access
# 4.1.11 Ensure use of privileged commands is collected
# Generate with: find / -xdev \( -perm -4000 -o -perm -2000 \) -type f
-a always,exit -F path=/usr/bin/sudo -F perm=x -k privileged
-a always,exit -F path=/usr/bin/passwd -F perm=x -k privileged
-a always,exit -F path=/usr/bin/chage -F perm=x -k privileged
-a always,exit -F path=/usr/bin/gpasswd -F perm=x -k privileged
-a always,exit -F path=/usr/bin/newgrp -F perm=x -k privileged
# 4.1.12 Ensure successful file system mounts are collected
-a always,exit -F arch=b64 -S mount -k mounts
# 4.1.13 Ensure file deletion events are collected
-a always,exit -F arch=b64 -S unlink -S unlinkat -S rename -S renameat -k delete
# 4.1.14 Ensure changes to sudoers are collected
-w /etc/sudoers -p wa -k scope
-w /etc/sudoers.d/ -p wa -k scope
# 4.1.15 Ensure system administrator command executions are collected
-a always,exit -F arch=b64 -S execve -C uid!=euid -k actions
-a always,exit -F arch=b64 -S execve -C gid!=egid -k actions
# 4.1.16 Ensure kernel module loading is collected
-w /sbin/insmod -p x -k modules
-w /sbin/rmmod -p x -k modules
-w /sbin/modprobe -p x -k modules
-a always,exit -F arch=b64 -S init_module -S delete_module -k modules
# 4.1.17 Ensure audit configuration is immutable
-e 2
PCI-DSS Rules
# PCI-DSS Requirement 10 - Track and monitor access
# 10.2.1 User access to cardholder data
-w /var/lib/mysql/cardholder/ -p rwxa -k cardholder_data
# 10.2.2 All actions by root or admin
-a always,exit -F arch=b64 -S all -F euid=0 -k root_actions
# 10.2.3 Access to audit trails
-w /var/log/audit/ -p wa -k audit_trail_access
-w /etc/audit/ -p wa -k audit_config
# 10.2.4 Invalid logical access attempts
-a always,exit -F arch=b64 -S open -F exit=-EACCES -k access_denied
-a always,exit -F arch=b64 -S open -F exit=-EPERM -k access_denied
# 10.2.5 Use of identification and authentication mechanisms
-w /etc/pam.d/ -p wa -k pam_changes
-w /var/log/secure -p wa -k auth_log
# 10.2.6 Initialization, stopping, pausing audit logs
-w /etc/audit/auditd.conf -p wa -k audit_config
-w /etc/audit/audit.rules -p wa -k audit_config
# 10.2.7 Creation and deletion of system-level objects
-a always,exit -F arch=b64 -S mknod -k system_objects
-a always,exit -F arch=b64 -S link -S symlink -k system_objects
Forensic Analysis
Incident Investigation
# Step 1: Identify the timeframe
aureport --summary -ts "2024-03-15 00:00" -te "2024-03-15 23:59"
# Step 2: Find failed authentications
ausearch -m USER_LOGIN,USER_AUTH -sv no -ts "2024-03-15" -i
# Step 3: Track user activity
ausearch -ua suspicious_user -ts "2024-03-15" -i | head -100
# Step 4: Check file modifications
ausearch -k identity -ts "2024-03-15" -i
# Step 5: Check privilege escalation
ausearch -k privileged -ts "2024-03-15" -i
# Step 6: Examine specific executable
ausearch -x /usr/bin/sudo -ts "2024-03-15" -i
# Step 7: Export for analysis
ausearch -ts "2024-03-15" --format csv > incident_2024-03-15.csv
Log Correlation
# Correlate with system logs
ausearch -ts "2024-03-15 14:00" -te "2024-03-15 15:00" -i > audit_events.txt
journalctl --since "2024-03-15 14:00" --until "2024-03-15 15:00" > journal_events.txt
# Find related events by audit session
ausearch -se 12345 -i
# Find events by process name
ausearch -x /usr/sbin/sshd -ts today -i
# Track specific user session
ausearch -ua 1000 --session 12345 -i
Analyzing Specific Events
# Decode a raw audit record
ausearch -a 12345 -i
# Parse EXECVE records (command arguments)
ausearch -sc execve -ts today -i | grep -A5 "type=EXECVE"
# Analyze PATH records
ausearch -f /etc/passwd -ts today -i | grep "type=PATH"
# Check CWD (current working directory)
ausearch -k suspicious -ts today -i | grep "type=CWD"
Performance Tuning
Buffer Configuration
# Check current backlog
auditctl -s | grep backlog
# Increase if seeing lost events
auditctl -b 16384
# Check for lost events
aureport --summary | grep lost
ausearch -m AUDIT_DAEMON -ts today | grep -i lost
Rule Optimization
# Exclude high-volume, low-value events
-a never,exit -F arch=b64 -S read -F success=1 -k exclude_reads
-a never,exit -F arch=b64 -S write -F success=1 -F path=/tmp -k exclude_tmp
# Exclude specific users (e.g., monitoring agents)
-a never,exit -F arch=b64 -S all -F auid=997 -k exclude_monitoring
# Exclude specific processes
-a never,exit -F arch=b64 -S all -F exe=/usr/bin/nagios -k exclude_nagios
# Place exclude rules FIRST (rules are processed in order)
Troubleshooting
Service Issues
# Check service status
systemctl status auditd
journalctl -u auditd -e
# Verify audit is enabled in kernel
cat /proc/sys/kernel/audit_enabled
# Check audit status
auditctl -s
# Common output:
# enabled 1 <- Audit is active
# failure 1 <- Failure mode (1=printk)
# pid 1234 <- auditd PID
# rate_limit 0 <- Rate limit (0=unlimited)
# backlog_limit 8192 <- Buffer size
# lost 0 <- Lost events (should be 0)
# backlog 0 <- Current backlog
Rule Errors
# Check for syntax errors
augenrules --check
# Verbose rule loading
augenrules --load 2>&1 | grep -i error
# Test individual rule
auditctl -w /etc/passwd -p wa -k test_rule
auditctl -l | grep test_rule
auditctl -W /etc/passwd -p wa -k test_rule # Remove test rule
# Common issues:
# - Architecture mismatch (b32 vs b64)
# - Invalid syscall names
# - Permission denied (not root)
# - Path doesn't exist
Performance Issues
# Check for lost events
auditctl -s | grep -E "(lost|backlog)"
# Monitor backlog in real-time
watch -n1 "auditctl -s | grep backlog"
# Identify high-volume rules
aureport --key --summary | head -20
# Check disk I/O from auditd
iotop -p $(pgrep auditd)
# Solutions:
# 1. Increase backlog buffer: auditctl -b 16384
# 2. Add exclude rules for noisy events
# 3. Use INCREMENTAL_ASYNC flush mode
# 4. Increase freq value in auditd.conf
Log Analysis Issues
# Verify log file permissions
ls -la /var/log/audit/
# Check log format
head -1 /var/log/audit/audit.log
# Verify ausearch can read logs
ausearch -m USER_LOGIN -ts today
# If ausearch fails, check:
# - Log file corruption
# - Permissions
# - SELinux contexts
restorecon -Rv /var/log/audit/
Integration
SIEM Integration
# Configure syslog output
cat > /etc/audit/plugins.d/syslog.conf << 'EOF'
active = yes
direction = out
path = builtin_syslog
type = builtin
args = LOG_INFO LOG_LOCAL6
format = string
EOF
# Restart audispd
systemctl restart auditd
# Forward to remote syslog in rsyslog
echo "local6.* @siem-server:514" >> /etc/rsyslog.conf
systemctl restart rsyslog
Elasticsearch/Logstash
active = yes
direction = out
path = builtin_af_unix
type = builtin
args = 0640 /var/run/audispd_events
format = string
Filebeat input configuration:
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/audit/audit.log
fields:
log_type: audit
fields_under_root: true
processors:
- decode_json_fields:
fields: ["message"]
target: "audit"
Quick Command Reference
# Service management
systemctl status auditd # Check status
systemctl restart auditd # Restart (WARNING: clears rules)
service auditd restart # Use service command on RHEL
augenrules --load # Reload rules without restart
# Rule management
auditctl -l # List current rules
auditctl -D # Delete all rules
auditctl -w /path -p wa -k key # Add file watch
auditctl -a always,exit -S syscall -k key # Add syscall rule
auditctl -s # Show status
# Searching
ausearch -k keyname # Search by key
ausearch -m USER_LOGIN # Search by message type
ausearch -ua username # Search by user
ausearch -ts today # Search from today
ausearch -f /etc/passwd # Search by file
ausearch -i # Interpret (human-readable)
# Reports
aureport --summary # Overall summary
aureport --login # Login report
aureport --auth --failed # Failed authentications
aureport --file --summary # File access summary
aureport --key # Activity by rule key
# Maintenance
augenrules --check # Check rule syntax
augenrules --load # Load rules from rules.d/
ausearch --checkpoint /tmp/checkpoint.file # Incremental searching