CR-2026-02-25: Wazuh Credential Rotation

Change Summary

CR ID

CR-2026-02-25-002

Date

2026-02-25

Priority

P0 - Emergency

Type

Security - Credential Rotation

Status

Completed

Objective

Rotate Wazuh indexer credentials that were inadvertently exposed in Claude Code settings file.

Credential Architecture

Wazuh uses three separate credentials:

Secret Username Component Purpose

indexer-cred

admin

OpenSearch (Indexer)

Admin API access, cluster management

dashboard-cred

kibanaserver

Dashboard → Indexer

Backend auth for dashboard queries

wazuh-api-cred

wazuh-wui

Manager API

UI authentication

Only indexer-cred was exposed.

Implementation Summary

Phase 0: Pre-Rotation Validation

# Store current password
CURRENT_PASS=$(ssh k3s-master-01 "kubectl -n wazuh get secret indexer-cred -o jsonpath='{.data.password}' | base64 -d")

# Test current access
ssh k3s-master-01 "kubectl -n wazuh exec wazuh-indexer-0 -c wazuh-indexer -- \
  curl -sk -u admin:$CURRENT_PASS https://localhost:9200/_cluster/health" | jq -r '.status'

Phase 1: Generate New Credential

NEW_PASS=$(tr -dc 'A-Za-z0-9' </dev/urandom | head -c 32)
export NEW_PASS

Phase 2: Store in gopass (Source of Truth)

cat << EOF | gopass insert -m v3/domains/d000/k3s/wazuh/indexer
$NEW_PASS
---
username: admin
service: wazuh-indexer (OpenSearch)
k8s_secret: indexer-cred
k8s_namespace: wazuh
rotated: 2026-02-25
EOF

Phase 3: Update Kubernetes Secret

NEW_PASS_B64=$(echo -n "$NEW_PASS" | base64)
ssh k3s-master-01 "kubectl -n wazuh patch secret indexer-cred -p '{\"data\":{\"password\":\"$NEW_PASS_B64\"}}'"

Phase 4: Update OpenSearch Internal User Database

OpenSearch stores password hashes internally. Updating K8s secret alone does NOT change the password. Must run securityadmin.sh.

# Generate bcrypt hash locally (container's hash.sh lacks Java)
BCRYPT_HASH=$(python3 -c "import bcrypt; print(bcrypt.hashpw(b'$NEW_PASS', bcrypt.gensalt(rounds=12)).decode())")

# Update internal_users.yml via cat (sed -i fails on overlayfs)
ssh k3s-master-01 "kubectl -n wazuh exec wazuh-indexer-0 -c wazuh-indexer -- \
  cat /usr/share/wazuh-indexer/config/opensearch-security/internal_users.yml" > /tmp/internal_users.yml

# Edit with awk (handles $ in bcrypt hash correctly)
awk -v hash="$BCRYPT_HASH" '
  /^admin:/ {found=1}
  found && /hash:/ {sub(/hash:.*/, "hash: \"" hash "\""); found=0}
  {print}
' /tmp/internal_users.yml >| /tmp/internal_users.yml.new && mv /tmp/internal_users.yml.new /tmp/internal_users.yml

# Write to container /tmp (config dir is read-only)
ssh k3s-master-01 "kubectl -n wazuh exec -i wazuh-indexer-0 -c wazuh-indexer -- \
  sh -c 'cat > /tmp/internal_users.yml'" < /tmp/internal_users.yml

# Apply with securityadmin.sh
ssh k3s-master-01 "kubectl -n wazuh exec wazuh-indexer-0 -c wazuh-indexer -- \
  env JAVA_HOME=/usr/share/wazuh-indexer/jdk \
  /usr/share/wazuh-indexer/plugins/opensearch-security/tools/securityadmin.sh \
  -f /tmp/internal_users.yml -t internalusers -icl -nhnv \
  -cacert /usr/share/wazuh-indexer/config/certs/root-ca.pem \
  -cert /usr/share/wazuh-indexer/config/certs/admin.pem \
  -key /usr/share/wazuh-indexer/config/certs/admin-key.pem"

Phase 5: Restart Pods (Dependency Order)

# 1. Indexer first
ssh k3s-master-01 "kubectl -n wazuh rollout restart statefulset wazuh-indexer && \
  kubectl -n wazuh rollout status statefulset wazuh-indexer --timeout=180s"

# 2. Wait for API ready
for i in {1..30}; do
  ssh k3s-master-01 "kubectl -n wazuh exec wazuh-indexer-0 -c wazuh-indexer -- \
    curl -sk -u admin:$NEW_PASS https://localhost:9200/_cluster/health" 2>/dev/null | \
    jq -e '.status' &>/dev/null && break
  sleep 5
done

# 3. Manager master
ssh k3s-master-01 "kubectl -n wazuh rollout restart statefulset wazuh-manager-master"

# 4. Manager worker
ssh k3s-master-01 "kubectl -n wazuh rollout restart statefulset wazuh-manager-worker"

# 5. Dashboard last
ssh k3s-master-01 "kubectl -n wazuh rollout restart deployment wazuh-dashboard"

Phase 6: Post-Rotation Validation

# Verify NEW password works
ssh k3s-master-01 "kubectl -n wazuh exec wazuh-indexer-0 -c wazuh-indexer -- \
  curl -sk -u admin:$NEW_PASS https://localhost:9200/_cluster/health" | jq -r '.status'
# Expected: green

# Verify OLD password rejected
ssh k3s-master-01 "kubectl -n wazuh exec wazuh-indexer-0 -c wazuh-indexer -- \
  curl -sk -u admin:$CURRENT_PASS https://localhost:9200/_cluster/health" 2>&1 | \
  grep -qE "Unauthorized|401" && echo "✓ Old password rejected"

CLI Mastery: Container Workarounds

Problem Solution

sed -i fails on overlayfs

Use cat to read/write via stdin/stdout

kubectl cp fails (no tar)

Use cat > /tmp/file via exec

Container’s hash.sh fails (no Java)

Generate bcrypt locally: python3 -c "import bcrypt; …​"

ConfigMap/Secret mounts are read-only

Write to /tmp, point tools there

Bcrypt hash contains $ chars

Use awk -v hash="$BCRYPT_HASH" instead of sed double-quotes

which missing (securityadmin.sh)

Set JAVA_HOME explicitly: env JAVA_HOME=/usr/share/wazuh-indexer/jdk

Key Lessons

Issue Mitigation

K8s secret ≠ OpenSearch password

Must run securityadmin.sh to update internal user database

Hardcoded secrets in config files

Use environment variables: $VAR not "literal"

Flat gopass structure

Use resource-based paths: /wazuh/indexer, /wazuh/api

No pre/post validation

Always test access BEFORE and AFTER rotation