Automation

Self-healing design

The health checks are short scheduled (cron) Bash runs, not a daemon. Each run performs three steps:

1. Process state: Docker or pct reports the workload as running.
2. Service probe: an HTTP or protocol-specific check.
3. Decision: reset the counter, recover, or notify.

Consecutive failures are tracked in a RAM-backed state directory.

Control logic

Condition	Action	State change
Healthy process and probe	No restart	Failure counter resets to zero
First or second failure	Restart workload	Counter increments
Third consecutive failure	Stop automated recovery and notify	Counter remains for investigation
Workload not present locally	Skip safely	No false remediation

Design decisions

• Cron instead of a custom daemon: lower overhead, simpler inspection and fewer long-running parts.
• RAM-backed state: no unnecessary disk writes, and state clears on reboot.
• Application-layer probes: a running container is not necessarily a healthy service.
• Retry ceiling: escalate to a person instead of an endless restart loop.
• Native notification paths: platform facilities are used, so the scripts carry no credentials.

Docker health checks (Unraid)

#!/bin/bash
CONTAINERS=("service-a:8096" "service-b:3000")
MAX_RETRIES=3
STATE_DIR="/tmp/docker_monitor"
mkdir -p "$STATE_DIR"

for entry in "${CONTAINERS[@]}"; do
  name="${entry%%:*}"
  port="${entry##*:}"
  fail_file="$STATE_DIR/${name}.fails"
  [ -f "$fail_file" ] || echo 0 > "$fail_file"

  if ! docker ps -a --format '{{.Names}}' | grep -Eq "^${name}$"; then
    echo "Container not found: $name; skipping"
    continue
  fi
  status=$(docker inspect --format='{{.State.Status}}' "$name")
  http_code=$(curl -s -o /dev/null -w '%{http_code}' \
    --max-time 5 "http://127.0.0.1:${port}")

  if [ "$status" != "running" ] || [ "$http_code" = "000" ] \
     || [ "$http_code" -ge 500 ]; then
    count=$(( $(cat "$fail_file") + 1 ))
    echo "$count" > "$fail_file"
    if [ "$count" -lt "$MAX_RETRIES" ]; then
      docker restart "$name"
    else
      notify -e "Docker Failure" -s "$name" \
        -d "Health checks failed after ${MAX_RETRIES} attempts"
    fi
  else
    echo 0 > "$fail_file"
  fi
done

The Unraid routine also checks Docker health state and uses Unraid's notify facility.

Protocol-specific check

DNS services are tested with a direct lookup:

if ! dig +short +time=2 +tries=1 example.org @"$DNS_SERVER" >/dev/null; then
  reason="DNS resolution failed on port 53"
fi

LXC health checks (Proxmox)

#!/bin/bash
TARGETS=("100:192.168.X.X:8080" "101:192.168.X.X:9000")
MAX_RETRIES=3
STATE_DIR="/tmp/lxc_monitor"
mkdir -p "$STATE_DIR"

for entry in "${TARGETS[@]}"; do
  vmid=$(cut -d: -f1 <<< "$entry")
  host=$(cut -d: -f2 <<< "$entry")
  port=$(cut -d: -f3 <<< "$entry")
  fail_file="$STATE_DIR/${vmid}.fails"
  [ -f "$fail_file" ] || echo 0 > "$fail_file"

  if ! pct status "$vmid" >/dev/null 2>&1; then
    echo "LXC $vmid not present on this node; skipping"
    continue
  fi
  state=$(pct status "$vmid" | awk '{print $2}')
  code=$(curl -s -o /dev/null -w '%{http_code}' \
    --max-time 5 "http://${host}:${port}")

  if [ "$state" != "running" ] || [ "$code" = "000" ] \
     || [ "$code" -ge 500 ]; then
    count=$(( $(cat "$fail_file") + 1 ))
    echo "$count" > "$fail_file"
    if [ "$count" -lt "$MAX_RETRIES" ]; then
      [ "$state" = "running" ] && pct reboot "$vmid" || pct start "$vmid"
    else
      proxmox-notification send --severity error \
        --summary "LXC failure: $vmid" --body "Recovery limit reached"
    fi
  else
    echo 0 > "$fail_file"
  fi
done

Container IDs that are not present on the local node are skipped, so the logic is safe in multi-node operation.

Manual validation

/usr/local/bin/lxc_health_check.sh
cat /tmp/lxc_monitor/*.fails
journalctl -u cron --since "30 minutes ago"

Terraform practice

Scope

Terraform is used to provision practice VMs. Existing homelab services are not claimed to be fully managed by Terraform. This is a learning environment, not a production claim.

The practice targets Proxmox VM provisioning: provider configuration, cloud-init, addressing, storage and lifecycle.

Objectives

• Configure the Proxmox provider through variables.
• Upload cloud-init snippets as managed resources.
• Create repeatable Ubuntu VMs.
• Attach OS and application-data disks.
• Inject SSH keys without hardcoding credentials.
• Practice plan, apply, inspect and destroy.

terraform {
  required_providers {
    proxmox = {
      source  = "bpg/proxmox"
      version = "~> 0.99"
    }
  }
}
provider "proxmox" {
  endpoint  = var.proxmox_api_url
  api_token = var.proxmox_api_token
  insecure  = true # Lab-only certificate setting
}
resource "proxmox_virtual_environment_vm" "practice" {
  count     = var.vm_count
  name      = "lab-ubuntu-${count.index + 1}"
  node_name = var.target_node
  cpu { cores = 2 }
  memory { dedicated = 4096 }
  network_device { bridge = "vmbr0" }
  initialization {
    user_account {
      username = var.admin_username
      keys     = [var.ssh_public_key]
    }
  }
}