MinIO S3 Storage Deployment

Deploy MinIO on k3s for S3-compatible object storage supporting backups, Terraform state, and application data.

Overview

MinIO is a high-performance, S3-compatible object storage system.

Feature Description

S3 API

Full Amazon S3 API compatibility

Buckets

Organize data into logical containers

Versioning

Keep multiple versions of objects

Lifecycle

Automatic expiration and tiering

Replication

Cross-site replication for DR

Use Cases

| Use Case | Bucket | Purpose | |----------|--------|---------| | Terraform State | terraform-state | Remote backend for IaC | | Vault Backups | vault-backups | Raft snapshots | | ISE Backups | ise-backups | Config backups | | Loki Logs | loki-chunks | Log storage backend | | Container Images | registry | OCI image cache |

Architecture

MinIO Object Storage Architecture

Prerequisites

# Verify k3s
kubectl get nodes

# Verify storage available
kubectl get pv

Phase 1: Namespace and Storage

1.1 Create MinIO Namespace

kubectl create namespace minio

1.2 Create NFS Storage

minio-storage.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: minio-pv
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nfs-minio
  nfs:
    server: 10.50.1.70
    path: /volume1/k3s/minio
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: minio-pvc
  namespace: minio
spec:
  storageClassName: nfs-minio
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
# Create directory on NAS
ssh nas-01 "mkdir -p /volume1/k3s/minio"

# Apply storage
kubectl apply -f minio-storage.yaml

1.3 Secrets Management (gopass + dsec + Vault)

Credentials stored in three locations for different use cases:

System Location Use Case

gopass

v3/domains/d000/k3s/minio

Interactive retrieval, metadata, password managers

dsec

d000/dev/app.env.age

Shell scripts, automation, eval "$(dsec source …​)"

Vault KV

kv/k3s/minio

Kubernetes pod injection via Vault Agent

Step 1: Generate Credentials

ROOT_USER="minio-admin"
ROOT_PASSWORD="$(openssl rand -base64 32)"
echo "Root User: $ROOT_USER"
echo "Root Password: $ROOT_PASSWORD"

Step 2: Add to gopass

# Generate password and open editor for metadata
gopass generate -e v3/domains/d000/k3s/minio 32

Add metadata below the generated password line:

---
description: "MinIO S3 root credentials"
url: "https://minio.inside.domusdigitalis.dev"
api_url: "https://s3.inside.domusdigitalis.dev"
username: "minio-admin"
namespace: "minio"
helm_release: "minio"
gopass sync

Step 3: Add to dsec (app.env.age)

dsec edit d000 dev/app

# Add this section:
# === MinIO S3 Storage ===
K3S_MINIO_ROOT_USER=minio-admin
K3S_MINIO_ROOT_PASS=<paste password>
K3S_MINIO_CONSOLE_URL=https://minio.inside.domusdigitalis.dev
K3S_MINIO_S3_URL=https://s3.inside.domusdigitalis.dev

Step 4: Store in Vault KV (for pod injection)

# Store in Vault
vault kv put kv/k3s/minio \
  root-user="$ROOT_USER" \
  root-password="$ROOT_PASSWORD"

# Create policy
vault policy write minio-secrets - <<'EOF'
path "kv/data/k3s/minio" {
  capabilities = ["read"]
}
EOF

# Create k8s auth role
vault write auth/kubernetes/role/minio \
  bound_service_account_names=minio \
  bound_service_account_namespaces=minio \
  policies=minio-secrets \
  ttl=1h

Step 5: Commit and Push

# Push gopass
gopass sync

# Push dsec
cd ~/.secrets
git add environments/domains/d000/dev/app.env.age
git commit -m "feat(d000/dev): Add MinIO S3 credentials"
git push origin main

Retrieve Password

# Option A: From gopass (interactive)
gopass show -c v3/domains/d000/k3s/minio  # copies to clipboard

# Option B: From dsec (automation)
eval "$(dsec source d000 dev/app)"
echo $K3S_MINIO_ROOT_PASS

# Option C: mc CLI config
mc alias set domus https://s3.inside.domusdigitalis.dev $(gopass show -o v3/domains/d000/k3s/minio | awk 'NR==1') "$(gopass show -o v3/domains/d000/k3s/minio)"

Create DNS Records

DNS records are added to BIND (authoritative DNS). See DNS Operations for full procedure.

MinIO requires TWO hostnames - minio (console UI) and s3 (S3 API endpoint).

Step 1: SSH to bind-01

ssh bind-01

Step 2: Add Forward (A) Records

sudo nsupdate -l << 'EOF'
zone inside.domusdigitalis.dev
update add minio.inside.domusdigitalis.dev. 3600 A 10.50.1.120
update add s3.inside.domusdigitalis.dev. 3600 A 10.50.1.120
send
EOF

Verify:

dig +short minio.inside.domusdigitalis.dev @localhost
dig +short s3.inside.domusdigitalis.dev @localhost

Step 3: Verify SOA Serial & Zone Transfer

dig SOA inside.domusdigitalis.dev +short | awk '{print "Serial: "$3}'
sudo rndc notify inside.domusdigitalis.dev

Step 4: Exit and Verify

exit
dig +short minio.inside.domusdigitalis.dev
dig +short s3.inside.domusdigitalis.dev

1.4 Create Vault TLS Secret

kubectl create secret generic vault-tls \
  --namespace minio \
  --from-file=ca.crt=/tmp/DOMUS-CA-CHAIN.pem

Phase 2: Deploy MinIO

2.1 Add MinIO Helm Repo

helm repo add minio https://charts.min.io/
helm repo update

2.2 Create Values File

minio-values.yaml
# Mode: standalone (single node) or distributed
mode: standalone

# Deployment
replicas: 1

# Resources
resources:
  requests:
    memory: 512Mi
    cpu: 250m
  limits:
    memory: 2Gi
    cpu: 1000m

# Root credentials (will be overridden by Vault)
rootUser: "minio-admin"
rootPassword: "changeme"

# Persistence
persistence:
  enabled: true
  existingClaim: minio-pvc

# Service
service:
  type: ClusterIP
  port: 9000

# Console
consoleService:
  type: ClusterIP
  port: 9001

# Ingress for Console
consoleIngress:
  enabled: true
  ingressClassName: cilium
  hosts:
    - minio.inside.domusdigitalis.dev
  tls:
    - secretName: wildcard-tls
      hosts:
        - minio.inside.domusdigitalis.dev

# API Ingress
ingress:
  enabled: true
  ingressClassName: cilium
  hosts:
    - s3.inside.domusdigitalis.dev
  tls:
    - secretName: wildcard-tls
      hosts:
        - s3.inside.domusdigitalis.dev

# Metrics for Prometheus
metrics:
  serviceMonitor:
    enabled: true
    namespace: monitoring
    interval: 30s

# Buckets to create on startup
buckets:
  - name: terraform-state
    policy: none
    purge: false
  - name: vault-backups
    policy: none
    purge: false
  - name: ise-backups
    policy: none
    purge: false
  - name: loki-chunks
    policy: none
    purge: false

# Service Account
serviceAccount:
  create: true
  name: minio

# Pod annotations for Vault injection
podAnnotations:
  vault.hashicorp.com/agent-inject: "true"
  vault.hashicorp.com/role: "minio"
  vault.hashicorp.com/agent-inject-secret-minio: "kv/data/k3s/minio"
  vault.hashicorp.com/agent-inject-template-minio: |
    {{- with secret "kv/data/k3s/minio" -}}
    export MINIO_ROOT_USER="{{ .Data.data.root_user }}"
    export MINIO_ROOT_PASSWORD="{{ .Data.data.root_password }}"
    {{- end }}
  vault.hashicorp.com/tls-secret: "vault-tls"
  vault.hashicorp.com/ca-cert: "/vault/tls/ca.crt"

2.3 Install MinIO

helm install minio minio/minio \
  --namespace minio \
  --values minio-values.yaml \
  --version 5.0.15

2.4 Verify Installation

# Check pods
kubectl get pods -n minio

# Check services
kubectl get svc -n minio

# Check ingress
kubectl get ingress -n minio

Phase 3: DNS Configuration

3.1 Add DNS Entries

# SSH to BIND server and add records
ssh bind-01 "sudo nsupdate -l << 'EOF'
zone inside.domusdigitalis.dev
update add minio.inside.domusdigitalis.dev. 3600 A 10.50.1.120
update add s3.inside.domusdigitalis.dev. 3600 A 10.50.1.120
send
EOF"

# Verify
dig +short minio.inside.domusdigitalis.dev
dig +short s3.inside.domusdigitalis.dev

Phase 4: Client Configuration

4.1 Install MinIO Client (mc)

# On workstation
curl -LO https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc
sudo mv mc /usr/local/bin/

# Configure alias
mc alias set domus https://s3.inside.domusdigitalis.dev minio-admin <password>

4.2 Test Connection

# List buckets
mc ls domus

# Create test file
echo "test" > /tmp/test.txt
mc cp /tmp/test.txt domus/terraform-state/

# Verify
mc ls domus/terraform-state/

# Remove test file
mc rm domus/terraform-state/test.txt

Phase 5: Configure Terraform Backend

5.1 Create Terraform User

# Create service account for Terraform
mc admin user add domus terraform-sa "$(openssl rand -base64 32)"

# Create policy
cat > /tmp/terraform-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::terraform-state",
        "arn:aws:s3:::terraform-state/*"
      ]
    }
  ]
}
EOF

mc admin policy create domus terraform-policy /tmp/terraform-policy.json
mc admin policy attach domus terraform-policy --user terraform-sa

5.2 Terraform Backend Configuration

terraform/backend.tf
terraform {
  backend "s3" {
    bucket                      = "terraform-state"
    key                         = "domus/infrastructure.tfstate"
    region                      = "us-east-1"  # Required but ignored by MinIO
    endpoint                    = "https://s3.inside.domusdigitalis.dev"
    skip_credentials_validation = true
    skip_metadata_api_check     = true
    skip_region_validation      = true
    force_path_style            = true
  }
}
# Set credentials
export AWS_ACCESS_KEY_ID="terraform-sa"
export AWS_SECRET_ACCESS_KEY="<password>"

# Initialize
terraform init

Phase 6: Configure Vault Backup

6.1 Create Vault Backup User

mc admin user add domus vault-backup "$(openssl rand -base64 32)"

cat > /tmp/vault-backup-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:GetObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::vault-backups",
        "arn:aws:s3:::vault-backups/*"
      ]
    }
  ]
}
EOF

mc admin policy create domus vault-backup-policy /tmp/vault-backup-policy.json
mc admin policy attach domus vault-backup-policy --user vault-backup

6.2 Vault Backup Script

vault-backup-to-minio.sh
#!/bin/bash
set -euo pipefail

DATE=$(date +%Y-%m-%d_%H%M%S)
SNAPSHOT_FILE="/tmp/vault-raft-${DATE}.snap"

# Take Raft snapshot
vault operator raft snapshot save "$SNAPSHOT_FILE"

# Upload to MinIO
mc cp "$SNAPSHOT_FILE" domus/vault-backups/

# Cleanup local
rm "$SNAPSHOT_FILE"

echo "Vault backup uploaded: vault-backups/vault-raft-${DATE}.snap"

Phase 7: Lifecycle Policies

7.1 Configure Retention

# Keep backups for 30 days
mc ilm rule add domus/vault-backups \
  --expire-days 30 \
  --prefix "" \
  --tags "type=backup"

# Keep Terraform state for 90 days (versioned)
mc ilm rule add domus/terraform-state \
  --noncurrent-expire-days 90 \
  --prefix ""

7.2 Enable Versioning

mc version enable domus/terraform-state
mc version enable domus/vault-backups

Troubleshooting

Connection Refused

# Check pod status
kubectl get pods -n minio

# Check service endpoints
kubectl get endpoints -n minio

# Port-forward to test directly
kubectl port-forward -n minio svc/minio 9000:9000
mc alias set local http://localhost:9000 minio-admin <password>
mc ls local

Access Denied

# Verify credentials
mc admin info domus

# Check user policies
mc admin user info domus <username>

# Check bucket policy
mc policy get domus/<bucket>

Slow Performance

# Check NFS mount
kubectl exec -n minio minio-0 -- df -h /export

# Check disk I/O
kubectl exec -n minio minio-0 -- iostat -x 1 5

# Consider using local SSD for high-performance workloads

Resource Usage

| Component | Memory | CPU | Storage | |-----------|--------|-----|---------| | MinIO | 512Mi-2Gi | 250m-1000m | 100Gi+ |

MinIO is lightweight but storage-intensive.

Appendix: Vault PKI Certificate for MinIO

Replace the default certificate with a Vault-issued certificate for HTTPS access to the MinIO Console and S3 API.

Issue Certificates from Vault

MinIO needs two certificates: one for the Console UI and one for the S3 API.

From workstation:

# Console certificate
vault write -format=json pki_int/issue/domus-client \
  common_name="minio.inside.domusdigitalis.dev" \
  ttl="8760h" > /tmp/minio-console-cert.json

# S3 API certificate
vault write -format=json pki_int/issue/domus-client \
  common_name="s3.inside.domusdigitalis.dev" \
  ttl="8760h" > /tmp/minio-s3-cert.json

Extract Certificate Components

# Console
jq -r '.data.certificate' /tmp/minio-console-cert.json > /tmp/minio-console.crt
jq -r '.data.private_key' /tmp/minio-console-cert.json > /tmp/minio-console.key
jq -r '.data.ca_chain[]' /tmp/minio-console-cert.json >> /tmp/minio-console.crt
# S3 API
jq -r '.data.certificate' /tmp/minio-s3-cert.json > /tmp/minio-s3.crt
jq -r '.data.private_key' /tmp/minio-s3-cert.json > /tmp/minio-s3.key
jq -r '.data.ca_chain[]' /tmp/minio-s3-cert.json >> /tmp/minio-s3.crt

Verify Certificates

openssl x509 -in /tmp/minio-console.crt -noout -subject -issuer -dates
openssl x509 -in /tmp/minio-s3.crt -noout -subject -issuer -dates

Transfer and Create Secrets

scp /tmp/minio-console.crt /tmp/minio-console.key /tmp/minio-s3.crt /tmp/minio-s3.key k3s-master-01.inside.domusdigitalis.dev:/tmp/

On k3s-master-01:

# Console TLS secret
kubectl -n minio create secret tls minio-console-tls-vault \
  --cert=/tmp/minio-console.crt \
  --key=/tmp/minio-console.key \
  --dry-run=client -o yaml | kubectl apply -f -

# S3 API TLS secret
kubectl -n minio create secret tls minio-s3-tls-vault \
  --cert=/tmp/minio-s3.crt \
  --key=/tmp/minio-s3.key \
  --dry-run=client -o yaml | kubectl apply -f -

Update Helm Values

Update minio-values.yaml to use the new certificates:

# Console Ingress
consoleIngress:
  enabled: true
  tls:
    - secretName: minio-console-tls-vault
      hosts:
        - minio.inside.domusdigitalis.dev

# API Ingress
ingress:
  enabled: true
  tls:
    - secretName: minio-s3-tls-vault
      hosts:
        - s3.inside.domusdigitalis.dev

Upgrade Helm Release

helm upgrade minio minio/minio \
  --namespace minio \
  --values minio-values.yaml

Port-Forward with HTTPS (if not using Ingress)

cat << 'EOF' | sudo tee /etc/systemd/system/minio-console-pf.service
[Unit]
Description=MinIO Console Port Forward
After=network.target k3s.service
Wants=k3s.service

[Service]
Type=simple
Environment="KUBECONFIG=/etc/rancher/k3s/k3s.yaml"
ExecStart=/usr/local/bin/kubectl port-forward -n minio svc/minio-console 9001:9001 --address=0.0.0.0
Restart=always
RestartSec=10
User=root

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now minio-console-pf

Verify HTTPS

echo | openssl s_client -connect minio.inside.domusdigitalis.dev:9001 2>/dev/null | openssl x509 -noout -subject -issuer

See Also