INC-2026-03-16: Z Fold 7 Termux - gopass and SSH Not Working

Incident Summary

Field Value

Incident ID

INC-2026-03-16-ZFOLD7-TERMUX

Device

Samsung Z Fold 7 (zfold7-evanusmodestus)

MAC

9C:83:06:CE:89:46

IP

10.50.10.110 (VLAN 10 - Data)

Reported

2026-03-16

Severity

Medium (productivity impact)

Status

Open - Assessment Required

Symptoms

  1. gopass not working - Unable to retrieve secrets

  2. SSH not working - Cannot SSH into Termux from workstation

Environment

  • Device: Samsung Z Fold 7

  • OS: Android with Termux

  • Network: WiFi 802.1X EAP-TLS (Domus-Secure)

  • Identity: zfold7-evanusmodestus.byod.inside.domusdigitalis.dev

Diagnostic Steps

Step 1: Start SSH Service on Termux

On the Z Fold 7 Termux app:

# Start SSH daemon
sshd

# Verify it's running
pgrep sshd

# Check what port (default 8022)
grep -i port $PREFIX/etc/ssh/sshd_config

# Show your Termux username
whoami

Step 2: Connect from Workstation

# SSH to Termux (default port 8022)
ssh -p 8022 10.50.10.110

Step 3: Diagnose gopass

# Check gopass version
gopass version

# Check GPG agent
gpg-agent --version
gpg --list-keys

# Test gopass
gopass ls

# Check for errors
gopass doctor

Step 4: Diagnose SSH Keys

# List SSH keys
ls -la ~/.ssh/

# Check SSH agent
ssh-add -l

# Test connectivity to GitHub
ssh -T git@github.com

Possible Causes

Cause Diagnostic

GPG agent not running

pgrep gpg-agent returns nothing

GPG key expired

gpg --list-keys shows expired

SSH keys missing/corrupted

ls ~/.ssh/ shows missing files

Termux storage permission

termux-setup-storage not run

Package update broke things

pkg upgrade changed configs

Resolution

Decision: Clean reinstall of Termux from F-Droid.

Runbook: Z Fold 7 Mobile Workflow Setup

Reinstall Steps

  • Uninstall Termux (and plugins)

  • Install Termux from F-Droid (NOT Play Store)

  • pkg update && pkg upgrade

  • pkg install openssh git neovim rsync stow pass gnupg fish oh-my-posh termux-api

  • passwd / sshd / termux-setup-storage / whoami

  • Phase 2: SSH key transfer and Vault cert auth

  • Phase 3-4: Git and GPG/gopass setup

  • Phase 5-6: Repos and dotfiles sync (dotfiles-optimus, domus-nvim)

  • Phase 7: Prompt setup (Starship - oh-my-posh broken on ARM64)

  • Phase 8: Clipboard for gopass (Termux:API app from F-Droid)

  • Phase 9: Set zsh as default shell

  • Phase 10: Clone productivity repos (domus-captures, domus-infra-ops)

  • Verify: gopass ls, ssh -T git@github.com

  • Remove temporary password auth from SSH config

Temporary SSH Config Change

File: ~/.ssh/config

Change: Enable password auth temporarily for initial connection (before keys transferred)

Apply Change

# Add PasswordAuthentication to fold7 entry
sed -i '/^Host fold7$/,/UserKnownHostsFile/{s/UserKnownHostsFile.*/&\n    PasswordAuthentication yes\n    PreferredAuthentications publickey,password/}' ~/.ssh/config

# Verify
grep -A 15 "^Host fold7$" ~/.ssh/config

Update User (after running whoami on phone)

# Replace old user with new (example: u0_a385 -> u0_a123)
sed -i '/^Host fold7$/,/^Host [^*]/{s/User u0_a385/User <NEW_USER>/}' ~/.ssh/config

Expected Result

 Host fold7
     HostName 10.50.10.110
     Port 8022
-    User u0_a385
+    User <whoami output>
     IdentityFile ~/.ssh/id_ed25519_vault
     CertificateFile ~/.ssh/id_ed25519_vault-cert.pub
     IdentityFile ~/.ssh/id_ed25519_sk_rk_d000_nano
     IdentityFile ~/.ssh/id_ed25519_sk_rk_d000
     IdentityFile ~/.ssh/id_ed25519_sk_rk_d000_secondary
     IdentityFile ~/.ssh/id_ed25519_d000
     StrictHostKeyChecking no
     UserKnownHostsFile ~/.ssh/known_hosts_mobile
+    PasswordAuthentication yes
+    PreferredAuthentications publickey,password

Revert After Keys Transferred

# Remove temporary password auth
sed -i '/^Host fold7$/,/^Host [^*]/{/PasswordAuthentication yes/d; /PreferredAuthentications publickey,password/d}' ~/.ssh/config
Remove PasswordAuthentication yes after keys are transferred.

Phase 2: SSH Key Transfer and Vault Cert

Generate Keypair on Phone

On phone (Termux):

ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_vault -N "" -C "zfold7-termux"

Fetch Public Key to Workstation

Use sudo to bypass hardened SSH config (too many auth failures with 6 identity files). 
# Copy public key from phone (use sudo due to SSH config)
sudo scp -o PubkeyAuthentication=no -P 8022 u0_a385@10.50.10.110:~/.ssh/id_ed25519_vault.pub /tmp/zfold7.pub

# Fix ownership (sudo created as root)
sudo chown evanusmodestus:evanusmodestus /tmp/zfold7.pub

Load Vault Credentials

dsource d000 dev/vault
vault status

Fix Vault Role: Add u0_a385 to allowed_users

Check current allowed_users (u0_a385 is NOT in list - only u0_a361 from old install):

vault read ssh/roles/domus-client -format=json | jq -r '.data.allowed_users | split(",") | .[]'

Get current allowed_users as comma-separated string for modification:

vault read ssh/roles/domus-client -format=json | jq -r '.data.allowed_users'

Update role to add u0_a385:

vault write ssh/roles/domus-client - <<'EOF'
{
  "allowed_users": "Administrator,domus\\Administrator,admin,adminerosado,ansible,evanusmodestus,gabriel,root,u0_a361,u0_a385",
  "key_type": "ca",
  "default_user": "evanusmodestus",
  "ttl": "8h",
  "allow_user_certificates": true,
  "default_extensions": {"permit-pty": "", "permit-user-rc": ""}
}
EOF

Verify u0_a385 is now in allowed_users:

vault read ssh/roles/domus-client -format=json | jq -r '.data.allowed_users | split(",") | .[] | select(test("u0_a"))'

Sign Certificate with Vault SSH CA

vault write -field=signed_key ssh/sign/domus-client \
  public_key=@/tmp/zfold7.pub \
  valid_principals="evanusmodestus,u0_a385" \
  > /tmp/zfold7-cert.pub

Verify certificate:

ssh-keygen -L -f /tmp/zfold7-cert.pub | awk '/Principals:/{p=1} p && /^ /{print; if(/^[^ ]/)p=0}'

Copy Cert Back to Phone

sudo scp -o PubkeyAuthentication=no -P 8022 /tmp/zfold7-cert.pub u0_a385@10.50.10.110:~/.ssh/id_ed25519_vault-cert.pub

Copy Other SSH Keys to Phone

# Software keys (transferable)
sudo scp -o PubkeyAuthentication=no -P 8022 \
  ~/.ssh/id_ed25519_d000 \
  ~/.ssh/id_ed25519_d000.pub \
  ~/.ssh/id_ed25519_github \
  ~/.ssh/id_ed25519_github.pub \
  ~/.ssh/id_ed25519_gitlab \
  ~/.ssh/id_ed25519_gitlab.pub \
  ~/.ssh/id_ed25519_gitea \
  ~/.ssh/id_ed25519_gitea.pub \
  ~/.ssh/id_ed25519_codeberg \
  ~/.ssh/id_ed25519_codeberg.pub \
  ~/.ssh/id_ed25519_bitbucket \
  ~/.ssh/id_ed25519_bitbucket.pub \
  ~/.ssh/config \
  u0_a385@10.50.10.110:~/.ssh/

# YubiKey keys (NOT transferable - hardware bound):
# - id_ed25519_sk_rk_d000_nano
# - id_ed25519_sk_rk_d000
# - id_ed25519_sk_rk_d000_secondary

Set Permissions on Phone

chmod 600 ~/.ssh/id_ed25519_*
chmod 644 ~/.ssh/*.pub ~/.ssh/*-cert.pub
chmod 600 ~/.ssh/config

Configure Phone authorized_keys for Cert Auth

Add workstation’s public key (fallback) and Vault CA (cert auth):

# From workstation - add d000 pubkey
cat ~/.ssh/id_ed25519_d000.pub | sudo ssh -o PubkeyAuthentication=no -p 8022 u0_a385@10.50.10.110 'cat >> ~/.ssh/authorized_keys'

# Add Vault CA for cert auth
vault read -field=public_key ssh/config/ca | awk '{print "cert-authority "$0}' | sudo ssh -o PubkeyAuthentication=no -p 8022 u0_a385@10.50.10.110 'cat >> ~/.ssh/authorized_keys'

Re-sign Workstation Cert with u0_a385 Principal

The workstation’s cert needs u0_a385 principal to authenticate to the phone:

vault write -field=signed_key ssh/sign/domus-client \
  public_key=@$HOME/.ssh/id_ed25519_vault.pub \
  valid_principals="Administrator,admin,adminerosado,ansible,domus\\Administrator,evanusmodestus,u0_a385" \
  >| ~/.ssh/id_ed25519_vault-cert.pub

# Verify u0_a385 is in principals
ssh-keygen -L -f ~/.ssh/id_ed25519_vault-cert.pub | grep -A10 Principals

Verify SSH Works with Vault Cert

# From workstation (uses id_ed25519_vault with cert)
ssh fold7

Phase 3-4: Git and GPG/gopass Setup

Create SSH Sockets Directory (Phone)

mkdir -p ~/.ssh/sockets

Verify GitHub SSH (Phone)

ssh -T git@github.com

Transfer GPG Key from Workstation

On workstation:

# Export secret key
gpg --export-secret-keys --armor 28A3183647525597 > /tmp/gpg-secret.asc

# Copy to phone
sudo scp -P 8022 /tmp/gpg-secret.asc u0_a385@10.50.10.110:~/

# Clean up
rm /tmp/gpg-secret.asc

Import GPG Key (Phone)

# Import
gpg --import ~/gpg-secret.asc

# Trust the key
gpg --edit-key 28A3183647525597
# Type: trust → 5 → y → quit

# Verify
gpg --list-secret-keys

# Clean up
rm ~/gpg-secret.asc

Clone gopass Store (Phone)

mkdir -p ~/.local/share/gopass/stores
git clone git@github.com:EvanusModestus/gopass-v3.git ~/.local/share/gopass/stores/v3

Configure GPG Agent for Termux (Phone)

mkdir -p ~/.gnupg
echo "pinentry-program /data/data/com.termux/files/usr/bin/pinentry-tty" > ~/.gnupg/gpg-agent.conf
echo "allow-loopback-pinentry" >> ~/.gnupg/gpg-agent.conf

# Restart gpg-agent
gpgconf --kill gpg-agent
gpg-agent --daemon

Configure gopass Mount (Phone)

Do NOT manually edit config file. Use gopass mounts add command. 
# Add v3 store as mount
gopass mounts add v3 ~/.local/share/gopass/stores/v3

# Verify mount
gopass mounts

# Test
gopass ls
gopass show domains/d000/identity/ssh/github

Phase 5-6: Repos and Dotfiles Sync

Clone dotfiles-optimus (Phone)

mkdir -p ~/atelier/_projects/personal
git clone git@github.com:EvanusModestus/dotfiles-optimus.git ~/atelier/_projects/personal/dotfiles-optimus

Clone domus-nvim (Phone)

git clone git@github.com:EvanusModestus/domus-nvim.git ~/.config/nvim

Build blink.cmp Fuzzy Matcher (Phone)

blink.cmp requires Rust build for performance:

pkg install rust binutils make clang
cd ~/.local/share/nvim/lazy/blink.cmp
cargo build --release

Install LSPs via Termux (Phone)

Mason doesn’t work on Termux (no Android binaries). Install LSPs directly:

# Node.js for npm-based LSPs
pkg install nodejs-lts

# TypeScript/JavaScript
npm install -g typescript typescript-language-server

# Python
npm install -g pyright

# Bash
npm install -g bash-language-server

# YAML/JSON/HTML/CSS
npm install -g yaml-language-server vscode-langservers-extracted

# Rust
pkg install rust-analyzer

# C/C++
pkg install clang

Configure Neovim for Termux (domus-nvim)

File: ~/.config/nvim/lua/domus/plugins/config/lsp/init.lua

Add at top of file:

---@diagnostic disable: undefined-global

Wrap ensure_installed to skip on Termux (around line 48):

require("mason-lspconfig").setup({
    ensure_installed = vim.fn.isdirectory("/data/data/com.termux") == 1 and {} or {
        "lua_ls", "pyright", "rust_analyzer", "ts_ls",
        -- ... rest of servers
    },

Add Termux fallback after mason-lspconfig.setup() closes (before function end):

-- Termux fallback: setup LSPs using nvim 0.11 native API
if vim.fn.isdirectory("/data/data/com.termux") == 1 then
    vim.lsp.config.lua_ls = {
        cmd = { "lua-language-server" },
        filetypes = { "lua" },
        root_markers = { ".git", ".luarc.json", "init.lua" },
        settings = { Lua = { diagnostics = { globals = { "vim" } } } },
    }
    vim.lsp.config.pyright = {
        cmd = { "pyright-langserver", "--stdio" },
        filetypes = { "python" },
        root_markers = { ".git", "pyproject.toml", "setup.py" },
    }
    vim.lsp.config.rust_analyzer = {
        cmd = { "rust-analyzer" },
        filetypes = { "rust" },
        root_markers = { ".git", "Cargo.toml" },
    }
    vim.lsp.enable({ "lua_ls", "pyright", "rust_analyzer" })
end

File: ~/.config/nvim/lua/domus/plugins/specs/lang/markdown.lua

Disable obsidian.nvim on Termux:

{
    "epwalsh/obsidian.nvim",
    cond = vim.fn.isdirectory("/data/data/com.termux") == 0, -- disable on Termux
    -- ...
},
Mason errors on startup are expected until ensure_installed is wrapped. System-installed LSPs work via native fallback.

Stow Shell and Tool Configs (Phone)

Available in base/: fish, zsh, git, vim, fzf, fd, ripgrep, gpg

cd ~/atelier/_projects/personal/dotfiles-optimus/base

# Fish shell
stow -t ~ fish

# Zsh shell
stow -t ~ zsh

# Git config
stow -t ~ git

# Vim (fallback)
stow -t ~ vim

# Search tools
stow -t ~ fzf
stow -t ~ fd
stow -t ~ ripgrep

Phase 7: Prompt Setup (Phone)

Oh-My-Posh - BROKEN on ARM64 Termux

Oh-my-posh crashes on ARM64 Termux with Go runtime panics (writeSegmentsConcurrently race condition). All themes crash, including built-in minimal themes. disable_async = true does not fix it.

Do NOT use oh-my-posh on Termux.

Solution: Use Starship Instead

Starship is Rust-based and works reliably on ARM64.

# Install starship
curl -sS https://starship.rs/install.sh | sh -s -- --bin-dir ~/.local/bin

# Remove oh-my-posh if installed
rm -f ~/.local/bin/oh-my-posh
rm -rf ~/.cache/oh-my-posh

Configure Starship in .zshrc

Comment out oh-my-posh block and add starship:

# Comment out oh-my-posh (broken on ARM64)
sed -i '/^if command -v oh-my-posh/,/^fi$/s/^/#/' ~/.zshrc

# Add starship init
echo 'eval "$(starship init zsh)"' >> ~/.zshrc

Apply a Powerline Theme

# List available presets
starship preset --list

# Apply tokyo-night (matches workstation theme)
starship preset tokyo-night -o ~/.config/starship.toml

# Or catppuccin-powerline
starship preset catppuccin-powerline -o ~/.config/starship.toml

exec zsh

Expected Result

Powerline-style prompt with git status, path, and exit code:

󰌽 u0_a385 …/dotfiles-optimus/apps   main !   20:11  ❯

Phase 8: Clipboard for gopass (Phone)

gopass requires clipboard access for gopass show -c to work.

Install Termux:API App

You need BOTH the package AND the Android app.

  1. Package (CLI tools):

    pkg install termux-api

  2. Android App (handles API calls):

    Install from F-Droid: f-droid.org/packages/com.termux.api/

Without the Android app, clipboard commands hang forever.

Test Clipboard

echo "test" | termux-clipboard-set
termux-clipboard-get

Use with gopass

# Copy password to clipboard (clears after 45s)
gopass show -c domains/d000/identity/ssh/github

# Manual workaround if -c doesn't work
gopass show domains/d000/identity/ssh/github | head -1 | termux-clipboard-set

Timeline

Time Action Notes

2026-03-16

Incident reported

gopass and SSH not working

2026-03-16

Network confirmed

Device on VLAN 10, IP 10.50.10.110

2026-03-17

Decision: Reinstall

Clean install from F-Droid, follow SETUP doc

2026-03-17

Reinstall in progress

Termux installed, pkg update running

2026-03-17

Phase 2: SSH keys

Generated keypair, fetched pubkey to workstation

2026-03-17

Vault role update needed

u0_a385 not in allowed_users (u0_a361 was old user)

2026-03-17

Vault role updated

Added u0_a385 to domus-client allowed_users

2026-03-17

SSH cert auth working

Re-signed workstation cert with u0_a385 principal, added CA to authorized_keys

2026-03-17

Phase 3-4 complete

GPG key transferred, gopass configured, gopass ls works

2026-03-17

Phase 5-6 in progress

dotfiles-optimus cloned, domus-nvim configured for Termux

2026-03-17

Neovim LSP working

blink.cmp built from source, native LSP fallback for Termux

2026-03-17 (pending)

Wrap Mason ensure_installed

Suppress Mason install errors on Termux

2026-03-18

SSH cert auth failed

Certificate expired (8-hour TTL expired at 04:19am)

2026-03-18

Root cause identified

vault-ssh-sign script has hardcoded principals with u0_a361 (old user), not u0_a385

2026-03-18

Workaround applied

Manual re-sign with correct principals restored SSH access

2026-03-18

Oh-my-posh abandoned

Go runtime panics on ARM64 Termux - all themes crash

2026-03-18

Starship installed

starship preset tokyo-night - Rust-based, stable on ARM64

2026-03-18

Clipboard fixed

Installed Termux:API app from F-Droid - gopass -c works

2026-03-18

Phase 9: zsh default

Set zsh as default shell with chsh -s zsh

2026-03-18

Phase 10: Productivity repos

Clone domus-captures, domus-infra-ops for mobile workflow

Lessons Learned

Certificate TTL and Daily Workflow

The Vault SSH CA certificate has an 8-hour TTL. This means:

  • Certificate signed at 8pm expires at 4am

  • Morning SSH attempts fail silently with "publickey" rejection

  • Always check certificate validity when SSH fails unexpectedly

Quick diagnostic:

ssh-keygen -L -f ~/.ssh/id_ed25519_vault-cert.pub | grep Valid

vault-ssh-sign Script Needs Updating

The vault-ssh-sign wrapper script hardcodes principals that don’t include the current Termux user (u0_a385).

Current principals in script:

Administrator,domus\Administrator,adminerosado,admin,ansible,evanusmodestus,gabriel,root,u0_a361

Manual workaround (until script is fixed):

vault write -field=signed_key ssh/sign/domus-client \
  public_key=@$HOME/.ssh/id_ed25519_vault.pub \
  valid_principals="evanusmodestus,u0_a385" >| ~/.ssh/id_ed25519_vault-cert.pub

Verify:

ssh-keygen -L -f ~/.ssh/id_ed25519_vault-cert.pub | grep -A5 Principals

Expected output:

        Principals:
                evanusmodestus
                u0_a385
        Critical Options: (none)
        Extensions:
                permit-pty

Common Gotcha: Tilde Expansion in Vault Commands

When using @ file syntax with Vault, ~ does NOT expand:

# WRONG - fails with "no such file or directory"
vault write ... public_key=@~/.ssh/id_ed25519_vault.pub

# CORRECT - use $HOME
vault write ... public_key=@$HOME/.ssh/id_ed25519_vault.pub

Phase 9: Set zsh as Default Shell

Bash is the default Termux shell. Change to zsh for consistency with workstation.

Change Default Shell (Phone)

# Set zsh as default
chsh -s zsh

# Verify
cat /data/data/com.termux/files/usr/etc/passwd | grep $(whoami)

Expected output:

u0_a385::10385:10385::/data/data/com.termux/files/home:/data/data/com.termux/files/usr/bin/zsh

Force zsh on Termux Start

If chsh doesn’t persist (some Termux versions), add to .bashrc:

echo 'exec zsh' >> ~/.bashrc

Verify (Close and Reopen Termux)

Terminal should show zsh prompt (starship) instead of bash $:

󰌽 u0_a385 ~   20:30  ❯

Phase 10: Mobile Productivity Repos

Clone repos to enable productive mobile workflow. Z Fold 7’s large screen makes AsciiDoc editing and studying practical.

Create Atelier Structure (Phone)

mkdir -p ~/atelier/_bibliotheca
mkdir -p ~/atelier/_projects/personal

Clone domus-captures (Phone)

Primary use: Regex training, worklogs, codex references.

git clone git@github.com:EvanusModestus/domus-captures.git ~/atelier/_bibliotheca/domus-captures

Clone domus-infra-ops (Phone)

Primary use: Runbook reference while on-call.

git clone git@github.com:EvanusModestus/domus-infra-ops.git ~/atelier/_bibliotheca/domus-infra-ops

Clone domus-netapi-docs (Phone)

Primary use: netapi CLI reference.

git clone git@github.com:EvanusModestus/domus-netapi-docs.git ~/atelier/_bibliotheca/domus-netapi-docs

Convenience Aliases

Add to ~/.zshrc or create in dotfiles:

# Quick nav aliases
alias dcap='cd ~/atelier/_bibliotheca/domus-captures'
alias dinfra='cd ~/atelier/_bibliotheca/domus-infra-ops'
alias dnetapi='cd ~/atelier/_bibliotheca/domus-netapi-docs'

# Today's worklog
alias wrklog='nvim ~/atelier/_bibliotheca/domus-captures/docs/modules/ROOT/pages/$(date +%Y/%m)/WRKLOG-$(date +%Y-%m-%d).adoc'

# Regex training
alias regex='cd ~/atelier/_bibliotheca/domus-captures/docs/modules/ROOT/pages/education/training/regex && nvim .'

Mobile Workflow Use Cases

The Z Fold 7’s large inner screen (7.6") makes terminal work practical on the go.

Regex Training on the Road

Scenario: Waiting at DMV, coffee shop, airport, walking in Medellín…​

# Jump to regex curriculum
regex

# Or open specific session
nvim ~/atelier/_bibliotheca/domus-captures/docs/modules/ROOT/pages/education/training/regex/session-03-character-classes.adoc

# Practice with grep on sample files
cd ~/atelier/_bibliotheca/domus-captures/docs/modules/ROOT/examples
grep -E 'pattern' sample.txt

Quick Reference Lookups

# jq patterns
nvim ~/atelier/_bibliotheca/domus-captures/docs/modules/ROOT/examples/codex/bash/jq-sysadmin.adoc

# awk reference
nvim ~/atelier/_bibliotheca/domus-captures/docs/modules/ROOT/examples/codex/bash/awk.adoc

# grep patterns
nvim ~/atelier/_bibliotheca/domus-captures/docs/modules/ROOT/examples/codex/bash/grep.adoc

On-Call Runbook Access

# Quick access to runbooks
cd ~/atelier/_bibliotheca/domus-infra-ops/docs/asciidoc/modules/ROOT/pages/runbooks
ls *.adoc

# View specific runbook
nvim vyos-deployment.adoc
nvim k3s-deployment.adoc

Capture Ideas While Mobile

# Open today's worklog
wrklog

# Add quick note
# (edit in nvim, save, commit later)

Sync Changes

# Pull latest before starting
cd ~/atelier/_bibliotheca/domus-captures && git pull

# Push changes when done
git add -A && git commit -m "mobile: Quick capture" && git push

Keyboard Recommendations

For serious mobile work, pair with:

  • Samsung DeX mode - Desktop-like experience on external display

  • Bluetooth keyboard - Full typing speed

  • Termux:Float - Floating terminal over other apps