RCA-2026-03-16-002: SSH Config Order - Host * Overrides Specific Hosts

Executive Summary

SSH to Cisco ISE failed with "Permission denied" without prompting for password. Root cause: Host * block in SSH config set PasswordAuthentication no before the ISE-specific block was processed. SSH config uses first-match-wins for each option, so global defaults must come AFTER device-specific exceptions.

Timeline

Time Event

2026-03-16 12:30

Attempted ssh ise-02, got "Permission denied (publickey,password)"

2026-03-16 12:35

Tried various SSH options, still no password prompt

2026-03-16 12:40

Verbose output showed SSH trying keys, never attempting password

2026-03-16 12:45

Identified Host * block setting PasswordAuthentication no before ISE block

2026-03-16 12:47

Confirmed fix with explicit options on command line

Root Cause

The Problem

SSH config file structure:

Host *                          # Line ~50 - FIRST MATCH
    PasswordAuthentication no   # Sets this for ALL hosts
    PreferredAuthentications publickey
    ...

Host ise-01 ise-02              # Line ~200 - TOO LATE
    PasswordAuthentication yes  # IGNORED - already set above
    PreferredAuthentications keyboard-interactive,password

SSH Config Processing Rules

SSH config uses first match wins for each option:

  1. SSH reads config top to bottom

  2. For each option, the FIRST value found is used

  3. Later values for the same option are IGNORED

  4. Host * matches everything - if it comes first, it wins

Root Cause Statement

Device-specific SSH authentication settings were overridden by global Host * defaults because the global block appeared before the specific host blocks in the config file.

Resolution

Immediate Fix (Command Line Override)

/usr/bin/ssh -o PasswordAuthentication=yes \
    -o PreferredAuthentications=keyboard-interactive,password \
    -o PubkeyAuthentication=no \
    admin@10.50.1.21

Permanent Fix (Config Restructure)

Move all device-specific exceptions BEFORE Host *:

# ═══════════════════════════════════════════════════════════════════════════════
# EXCEPTIONS - Must come BEFORE Host *
# ═══════════════════════════════════════════════════════════════════════════════

Host ssh.dev.azure.com
    # Azure DevOps exception
    ...

Host ise-01 ise-02 ise-lab          # <-- MOVE HERE
    User admin
    PubkeyAuthentication no
    PasswordAuthentication yes
    ...

Host cisco-* switch-* router-*      # <-- MOVE HERE
    User admin
    PubkeyAuthentication no
    PasswordAuthentication yes
    ...

# ═══════════════════════════════════════════════════════════════════════════════
# GLOBAL DEFAULTS - Must come AFTER exceptions
# ═══════════════════════════════════════════════════════════════════════════════

Host *
    PasswordAuthentication no
    PreferredAuthentications publickey
    ...

# ═══════════════════════════════════════════════════════════════════════════════
# REGULAR HOSTS - These inherit from Host * but can override
# ═══════════════════════════════════════════════════════════════════════════════

Host kvm-01
    HostName 10.50.1.110
    # Inherits PasswordAuthentication no from Host *

ISE SSH Limitations

Cisco ISE does NOT support SSH certificates (Vault SSH CA).

ISE is a closed appliance with limited SSH implementation:

  • ✓ Password authentication

  • ✓ Public key authentication (limited)

  • ✗ SSH certificates

  • ✗ GSSAPI/Kerberos

  • ✗ Certificate-based authentication

For ISE, password auth is the only reliable option.

The MODEL: SSH Config Structure

# 1. EXCEPTIONS (specific overrides) - FIRST
Host legacy-device
    Ciphers +aes256-cbc
    KexAlgorithms +diffie-hellman-group1-sha1

Host cisco-ise-*
    PubkeyAuthentication no
    PasswordAuthentication yes

# 2. GLOBAL DEFAULTS - MIDDLE
Host *
    PasswordAuthentication no
    PreferredAuthentications publickey

# 3. HOST DEFINITIONS - LAST (inherit from Host *)
Host server-01
    HostName 10.0.0.1

Lessons Learned

  1. SSH config order matters - First match wins for each option

  2. Exceptions before globals - Device-specific settings must precede Host *

  3. Test with verbose - ssh -vvv shows authentication methods attempted

  4. ISE is special - Closed appliance, password-only SSH

Metadata

Field Value

RCA ID

RCA-2026-03-16-002

Author

Evan Rosado

Date Created

2026-03-16

Status

Final