ArgoCD Deployment

Deploy ArgoCD on k3s for GitOps-based application deployment and configuration management.

Overview

ArgoCD is a declarative GitOps continuous delivery tool for Kubernetes.

Feature Description

GitOps

Git as single source of truth for deployments

Auto-Sync

Automatically deploy when Git changes

Multi-Cluster

Manage multiple k8s clusters from one ArgoCD

RBAC

Role-based access control with SSO integration

Audit Trail

Full history of all deployments and changes

Architecture

ArgoCD GitOps Flow

Prerequisites

# Verify k3s
kubectl get nodes

# Verify Helm
helm version

# Verify Gitea accessible
curl -s https://gitea.inside.domusdigitalis.dev/api/v1/version

Phase 1: Namespace and Secrets

1.1 Create ArgoCD Namespace

kubectl create namespace argocd

1.2 Create Vault TLS Secret

kubectl create secret generic vault-tls \
  --namespace argocd \
  --from-file=ca.crt=/tmp/DOMUS-CA-CHAIN.pem

1.3 Store ArgoCD Admin Password in Vault

# Generate and store password
vault kv put kv/k3s/argocd \
  admin-password="$(openssl rand -base64 24)"

# Create policy
vault policy write argocd-secrets - <<'EOF'
path "kv/data/k3s/argocd" {
  capabilities = ["read"]
}
EOF

# Create k8s auth role
vault write auth/kubernetes/role/argocd \
  bound_service_account_names=argocd-server \
  bound_service_account_namespaces=argocd \
  policies=argocd-secrets \
  ttl=1h

Phase 2: Helm Installation

2.1 Add Argo Helm Repo

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update

2.2 Create Values File

argocd-values.yaml
# Global settings
global:
  domain: argocd.inside.domusdigitalis.dev

# ArgoCD Server
server:
  # Ingress
  ingress:
    enabled: true
    ingressClassName: cilium
    hosts:
      - argocd.inside.domusdigitalis.dev
    tls:
      - secretName: argocd-tls
        hosts:
          - argocd.inside.domusdigitalis.dev
    https: true

  # Run insecure (TLS terminated at ingress)
  extraArgs:
    - --insecure

  # Resources
  resources:
    requests:
      memory: 128Mi
      cpu: 100m
    limits:
      memory: 512Mi
      cpu: 500m

# Repository Server
repoServer:
  resources:
    requests:
      memory: 128Mi
      cpu: 100m
    limits:
      memory: 512Mi
      cpu: 500m

# Application Controller
controller:
  resources:
    requests:
      memory: 256Mi
      cpu: 100m
    limits:
      memory: 1Gi
      cpu: 1000m

# Redis (for caching)
redis:
  enabled: true
  resources:
    requests:
      memory: 64Mi
      cpu: 50m
    limits:
      memory: 256Mi
      cpu: 200m

# Disable Dex (use built-in auth or Keycloak later)
dex:
  enabled: false

# Application Set Controller
applicationSet:
  enabled: true
  resources:
    requests:
      memory: 64Mi
      cpu: 50m
    limits:
      memory: 256Mi
      cpu: 200m

# Notifications Controller (for alerts)
notifications:
  enabled: true
  resources:
    requests:
      memory: 64Mi
      cpu: 50m
    limits:
      memory: 128Mi
      cpu: 100m

# Config
configs:
  # Repository credentials for Gitea
  credentialTemplates:
    gitea-creds:
      url: https://gitea.inside.domusdigitalis.dev
      username: argocd
      password: "" # Will configure with Vault later

  # Known hosts for SSH
  ssh:
    knownHosts: |
      gitea.inside.domusdigitalis.dev ssh-ed25519 AAAAC3NzaC1lZDI1NTE5...

  # ArgoCD config
  cm:
    # Timeout settings
    timeout.reconciliation: 180s

    # Resource tracking
    resource.customizations.health.argoproj.io_Application: |
      hs = {}
      hs.status = "Progressing"
      hs.message = ""
      if obj.status ~= nil then
        if obj.status.health ~= nil then
          hs.status = obj.status.health.status
          if obj.status.health.message ~= nil then
            hs.message = obj.status.health.message
          end
        end
      end
      return hs

  # RBAC
  rbac:
    policy.default: role:readonly
    policy.csv: |
      p, role:admin, applications, *, */*, allow
      p, role:admin, clusters, *, *, allow
      p, role:admin, repositories, *, *, allow
      p, role:admin, logs, *, *, allow
      p, role:admin, exec, *, */*, allow
      g, admin, role:admin

2.3 Install ArgoCD

helm install argocd argo/argo-cd \
  --namespace argocd \
  --values argocd-values.yaml \
  --version 5.53.0

2.4 Verify Installation

# Check pods
kubectl get pods -n argocd

# Expected (after 2-3 minutes):
# argocd-application-controller-0           1/1  Running
# argocd-applicationset-controller-xxx      1/1  Running
# argocd-notifications-controller-xxx       1/1  Running
# argocd-redis-xxx                          1/1  Running
# argocd-repo-server-xxx                    1/1  Running
# argocd-server-xxx                         1/1  Running

Phase 3: Initial Access

3.1 Get Initial Admin Password

kubectl get secret -n argocd argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d; echo

3.2 Secrets Management (gopass + dsec)

Credentials stored in two locations for different use cases:

System Location Use Case

gopass

v3/domains/d000/k3s/argocd

Interactive retrieval, metadata, password managers

dsec

d000/dev/app.env.age

Shell scripts, automation, eval "$(dsec source …​)"

Step 1: Get Current Password

ARGOCD_PASS=$(kubectl get secret -n argocd argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d)
echo "Password: $ARGOCD_PASS"

Step 2: Add to gopass

# Generate password and open editor for metadata
gopass generate -e v3/domains/d000/k3s/argocd 32

Add metadata below the generated password line:

---
description: "ArgoCD GitOps admin credentials"
url: "https://argocd.inside.domusdigitalis.dev"
username: "admin"
namespace: "argocd"
secret: "argocd-initial-admin-secret"
helm_release: "argocd"
gopass sync

Step 3: Add to dsec (app.env.age)

dsec edit d000 dev/app

# Add this section:
# === ArgoCD GitOps ===
K3S_ARGOCD_ADMIN_USER=admin
K3S_ARGOCD_ADMIN_PASS=<paste password>
K3S_ARGOCD_URL=https://argocd.inside.domusdigitalis.dev

Step 4: Commit and Push

# Push gopass
gopass sync

# Push dsec
cd ~/.secrets
git add environments/domains/d000/dev/app.env.age
git commit -m "feat(d000/dev): Add ArgoCD credentials"
git push origin main

Retrieve Password

# Option A: From gopass (interactive)
gopass show -c v3/domains/d000/k3s/argocd  # copies to clipboard

# Option B: From dsec (automation)
eval "$(dsec source d000 dev/app)"
echo $K3S_ARGOCD_ADMIN_PASS

# Option C: ArgoCD CLI login
argocd login argocd.inside.domusdigitalis.dev --username admin --password $(gopass show -o v3/domains/d000/k3s/argocd)

3.3 Create DNS Records

DNS records are added to BIND (authoritative DNS). See DNS Operations for full procedure.

Step 1: SSH to bind-01

ssh bind-01

Step 2: Add Forward (A) Record

sudo nsupdate -l << 'EOF'
zone inside.domusdigitalis.dev
update add argocd.inside.domusdigitalis.dev. 3600 A 10.50.1.120
send
EOF

Verify:

dig +short argocd.inside.domusdigitalis.dev @localhost

Step 3: Verify SOA Serial & Zone Transfer

dig SOA inside.domusdigitalis.dev +short | awk '{print "Serial: "$3}'
sudo rndc notify inside.domusdigitalis.dev

Step 4: Exit and Verify

exit
dig +short argocd.inside.domusdigitalis.dev

3.4 Port-Forward for Initial Access

kubectl port-forward -n argocd svc/argocd-server 8080:443

# Open browser: https://localhost:8080
# Login: admin / <password from 3.1>

3.4 Install ArgoCD CLI

# On workstation
curl -sSL -o argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
chmod +x argocd
sudo mv argocd /usr/local/bin/

# Login
argocd login localhost:8080 --username admin --password <password> --insecure

Phase 4: Configure Gitea Repository

4.1 Create ArgoCD Service Account in Gitea

  1. Gitea UI → Settings → Applications → Generate New Token

  2. Name: argocd-readonly

  3. Scopes: read:repository

  4. Save token

4.2 Add Repository to ArgoCD

argocd repo add https://gitea.inside.domusdigitalis.dev/evanusmodestus/domus-k8s-apps.git \
  --username argocd \
  --password <gitea-token>

Or via UI: Settings → Repositories → Connect Repo

4.3 Create Apps Repository Structure

Create repository domus-k8s-apps in Gitea:

ArgoCD Repo Structure
Figure 1. ArgoCD GitOps Repository Structure

Phase 5: Create First Application

5.1 Example Application Manifest

apps/hello-world/application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: hello-world
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://gitea.inside.domusdigitalis.dev/evanusmodestus/domus-k8s-apps.git
    targetRevision: main
    path: apps/hello-world
  destination:
    server: https://kubernetes.default.svc
    namespace: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

5.2 Apply Application

kubectl apply -f apps/hello-world/application.yaml

5.3 Verify Sync

argocd app get hello-world
argocd app sync hello-world

Phase 6: Keycloak SSO (Optional)

6.1 Configure OIDC

Update ArgoCD ConfigMap:

configs:
  cm:
    url: https://argocd.inside.domusdigitalis.dev
    oidc.config: |
      name: Keycloak
      issuer: https://keycloak.inside.domusdigitalis.dev/realms/domus
      clientID: argocd
      clientSecret: $oidc.keycloak.clientSecret
      requestedScopes: ["openid", "profile", "email", "groups"]

6.2 Create Keycloak Client

  1. Keycloak Admin → Clients → Create

  2. Client ID: argocd

  3. Root URL: argocd.inside.domusdigitalis.dev

  4. Valid Redirect URIs: argocd.inside.domusdigitalis.dev/auth/callback

  5. Enable Client Authentication

  6. Copy Client Secret

Troubleshooting

Application Stuck in Unknown

# Check repo connection
argocd repo list

# Test git access
argocd repo get https://gitea.inside.domusdigitalis.dev/evanusmodestus/domus-k8s-apps.git

# Check repo-server logs
kubectl logs -n argocd -l app.kubernetes.io/component=repo-server

Sync Failed

# Get detailed sync status
argocd app get <app-name> --show-operation

# Check application controller logs
kubectl logs -n argocd -l app.kubernetes.io/component=application-controller

# Force refresh
argocd app get <app-name> --refresh

Cannot Access UI

# Check server pod
kubectl get pods -n argocd -l app.kubernetes.io/component=server

# Check ingress
kubectl get ingress -n argocd

# Direct port-forward
kubectl port-forward -n argocd svc/argocd-server 8080:443

Resource Usage

| Component | Memory | CPU | |-----------|--------|-----| | Application Controller | 256Mi-1Gi | 100m-1000m | | Server | 128Mi-512Mi | 100m-500m | | Repo Server | 128Mi-512Mi | 100m-500m | | Redis | 64Mi-256Mi | 50m-200m | | ApplicationSet | 64Mi-256Mi | 50m-200m | | Notifications | 64Mi-128Mi | 50m-100m | | Total | ~700Mi-2.5Gi | ~450m-2500m |

Appendix: Vault PKI Certificate for ArgoCD

Replace the default self-signed certificate with a Vault-issued certificate for HTTPS access.

Issue Certificate from Vault

From workstation:

vault write -format=json pki_int/issue/domus-client \
  common_name="argocd.inside.domusdigitalis.dev" \
  ttl="8760h" > /tmp/argocd-cert.json

Extract Certificate Components

jq -r '.data.certificate' /tmp/argocd-cert.json > /tmp/argocd.crt
jq -r '.data.private_key' /tmp/argocd-cert.json > /tmp/argocd.key
jq -r '.data.ca_chain[]' /tmp/argocd-cert.json >> /tmp/argocd.crt

Verify Certificate

openssl x509 -in /tmp/argocd.crt -noout -subject -issuer -dates

Transfer and Create Secret

scp /tmp/argocd.crt /tmp/argocd.key k3s-master-01.inside.domusdigitalis.dev:/tmp/

On k3s-master-01:

kubectl -n argocd create secret tls argocd-tls-vault \
  --cert=/tmp/argocd.crt \
  --key=/tmp/argocd.key \
  --dry-run=client -o yaml | kubectl apply -f -

Update Helm Values

Update argocd-values.yaml to use the new certificate:

server:
  ingress:
    tls:
      - secretName: argocd-tls-vault
        hosts:
          - argocd.inside.domusdigitalis.dev

Upgrade Helm Release

helm upgrade argocd argo/argo-cd \
  --namespace argocd \
  --values argocd-values.yaml

Port-Forward with HTTPS (if not using Ingress)

cat << 'EOF' | sudo tee /etc/systemd/system/argocd-pf.service
[Unit]
Description=ArgoCD Port Forward
After=network.target k3s.service
Wants=k3s.service

[Service]
Type=simple
Environment="KUBECONFIG=/etc/rancher/k3s/k3s.yaml"
ExecStart=/usr/local/bin/kubectl port-forward -n argocd svc/argocd-server 8080:443 --address=0.0.0.0
Restart=always
RestartSec=10
User=root

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now argocd-pf

Verify HTTPS

echo | openssl s_client -connect argocd.inside.domusdigitalis.dev:8080 2>/dev/null | openssl x509 -noout -subject -issuer

See Also