Part-Time Linux SysAdmin for multiple Self-Hosted AI Infrastructure

## Job Description

We are a German AI consultancy running self-hosted AI infrastructure for multiple clients on dedicated Linux servers (mostly Hetzner, Germany).

We're looking for a reliable, part-time system administrator to help keep our servers healthy, secure, up-to-date and keep everything well documented.

### What We Run (per server)

Our standardized stack runs on

Ubuntu 24 LTS with

Docker Compose and includes:

• Traefik (reverse proxy, TLS/Let's Encrypt, security headers)
• Open WebUI (AI chat interface with RAG)
• LiteLLM (LLM API gateway/proxy)
• Ollama (local LLM inference, GPU-accelerated)
• PostgreSQL (4 separate instances: WebUI, LiteLLM, Keycloak, n8n)
• Redis (caching, WebSocket, rate limiting)
• Qdrant (vector database for RAG)
• Keycloak (SSO/authentication, optional per client)
• n8n (workflow automation)
• SearXNG (privacy-focused web search)

## Job Description

We are a German AI consultancy (Hermann Consult) running self-hosted AI infrastructure for multiple enterprise clients on dedicated Linux servers (Hetzner, Germany).

We're looking for a reliable, part-time system administrator to help keep our servers healthy, secure, and up-to-date.

### What We Run (per server)

Our standardized stack runs on

Ubuntu 24 LTS with

Docker Compose and includes:

• Traefik (reverse proxy, TLS/Let's Encrypt, security headers)
• Open WebUI (AI chat interface with RAG)
• LiteLLM (LLM API gateway/proxy)
• Ollama (local LLM inference, GPU-accelerated)
• PostgreSQL (4 separate instances: WebUI, LiteLLM, Keycloak, n8n)
• Redis (caching, WebSocket, rate limiting)
• Qdrant (vector database for RAG)
• Keycloak (SSO/authentication, optional per client)
• n8n (workflow automation)
• SearXNG (privacy-focused web search)
• Piper (text-to-speech)
• Custom MCP services (file generation, image generation)
• Maybe more in the future

Some docker containers are customized, so we need to be aware of that.

Everything is orchestrated via Docker Compose with a scripted update system (`update-`) that handles phased stop/start in dependency order.

### What You'll Do (Weekly, 3-5 hours)

1.

Health monitoring — Check that all containers are running across all servers, review disk/memory/CPU usage, confirm services are responding

2.

Updates & patches — Run OS updates (`apt update/upgrade`), coordinate Docker image updates using our existing update script, apply security patches

3.

Backup verification — Confirm PostgreSQL dumps are completing for all databases per server, verify config backups, periodically test restores

4.

Security review — Check Fail2ban logs, review SSH auth logs, verify firewall rules (UFW), review Traefik access logs for anomalies

5.

Documentation — Maintain server inventory, update runbooks, log any incidents or changes

### First-Month Setup Projects

1.

Deploy monitoring — Set up Grafana + Loki + Prometheus on our internal monitoring server. Our containers already have Promtail labels configured — we need someone to deploy the collection/visualization side and set up Slack alerting

2.

Standardize all servers — Ensure every server matches our security baseline (Fail2ban active, UFW configured, Docker log limits set, SSH hardened)

3.

Create backup automation — Write backup scripts for all PostgreSQL instances, implement offsite backup to Hetzner Storage Box

4.

Server inventory documentation — Document every server:

what it runs, which client, component versions, access details

### Requirements

Note:

AI assistance is of course allowed and even appreciated.

Must-Have:

• Linux administration (Ubuntu specifically)
• Docker, Docker Compose
• SSH, networking basics, firewall configuration (UFW)
• PostgreSQL administration (backup/restore, basic monitoring)
• Reverse proxy (Traefik)
• Bash scripting for automation
• Comfortable reading and working with YAML configurations
• Comfortable using AI tools (Claude, ChatGPT) for problem-solving — You don't need to know every tool in our stack from memory, but you need to be efficient at figuring things out with AI assistance

Strong Nice-to-Have:

• Experience with selfhosted AI tools (Open WebUI, Ollama, LiteLLM)
• Monitoring stack experience (Prometheus, Grafana, Loki)
• Kubernetes basics (we're planning migration)
• Hetzner experience
• Experience with n8n or similar workflow automation
• German language skills (not required, but helpful)

What We Currently Don't Need:

• AWS/GCP/Azure expertise (we run on dedicated servers, not cloud)
• Fullstack developers (this is sysadmin work)

### Work Arrangement

• Reporting: Weekly written status summary — what was checked, any issues found, what was done

Contract duration of more than 6 months. with 30 hours per week.

Mandatory skills:

Linux System Administration, Docker, System Administration, System Monitoring, Linux, Ubuntu, Bash