If you run a home lab, you know the drill. It starts with one server. Then a second. Before you know it, you are managing a 3-node Proxmox VE cluster, a Proxmox Backup Server (PBS), a ZimaBoard, and a sprawling fleet of 10 individual Debian servers running everything from Docker stacks to network services.
Having this much compute power at home is incredible—until patch Tuesday rolls around.
Before I discovered Ansible, my update routine was a tedious, manual chore. I had to SSH into every single instance, run apt update && apt dist-upgrade, wait for the packages to download, check if a reboot was needed, and baby-sit the terminal.
With over a dozen unique machines to look after, it easily took me 2 hours to manually go through each instance. Worse yet, because I was doing it all by hand, sometimes I would just lose track, forget a machine, and completely miss one out. That left forgotten servers sitting unpatched, out of sync, and vulnerable to security risks simply due to human error.
Then, I built an Ansible playbook.
The Game Changer: Automation
The Prerequisite: Establishing Trust
Before Ansible can wave its magic wand and run commands across your network, there is one key thing you have to set up first. Because Ansible is “agentless,” it relies entirely on SSH to communicate with your servers.
To make this seamless, you must ensure the machine you are running Ansible from has passwordless SSH access to every single client.
Before running your first playbook, you need to generate an SSH key on your Ansible control machine and copy your public key over to each client using the ssh-copy-id command:
ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected]Once your public key is copied over to all machines, Ansible can log in securely and execute commands instantly without prompting you for a password a dozen times.
Ansible completely changed how I manage my lab. Instead of manually repeating commands across a dozen different terminal windows, I now express my entire infrastructure as code in a simple inventory file.
With a single playbook, Ansible handles my entire update workflow seamlessly:
- It guarantees 100% coverage: Because every single machine—including all 10 Debian servers—is hardcoded into my inventory, I never forget or miss a server again. Ansible meticulously hits every single box on the list, no exceptions.
- It updates the Proxmox cluster safely: By configuring the script to run sequentially (
serial: 1), it updates my first Proxmox node, reboots it if a new kernel requires it, and waits for cluster quorum to return before safely moving on to the next node. My cluster never goes down, and my VMs stay online. - It handles mixed environments dynamically: The same script updates my Proxmox cluster nodes, my ZimaBoard, my PBS backup box, and my standalone Debian servers. It dynamically senses what type of server it is logging into, running strict cluster safety checks on PVE while cleanly skipping them on standard Debian boxes.
From 2 Hours to 10 Minutes
Now, when it’s time to maintain the lab, I don’t dread it. I open a terminal, type a single command, and watch the output stream in real-time:
ansible-playbook -i hosts.yml update_proxmox.ymlHere is an example of my ansible host file host.yml
proxmox_infrastructure:
hosts:
pve01.local:
ansible_host: IP.ADDRESS.GOES.HERE
pve02.local:
ansible_host: IP.ADDRESS.GOES.HERE
pve03.local:
ansible_host: IP.ADDRESS.GOES.HERE
pbs.local:
ansible_host: IP.ADDRESS.GOES.HERE
vars:
ansible_user: rootHere is an example of my ansible yaml file update_proxmox.yml
---
- name: Sequential Proxmox VE and PBS Updates
hosts: proxmox_infrastructure
gather_facts: true
become: true
# Update exactly one infrastructure host at a time.
# This is important for PVE cluster availability.
serial: 1
tasks:
# ============================================================
# IDENTIFY HOST TYPE
# ============================================================
- name: Check if this is a PVE cluster node
ansible.builtin.stat:
path: /usr/bin/pvecm
register: pvecm_binary
# ============================================================
# PVE CLUSTER SAFETY CHECK
# ============================================================
- name: Check PVE cluster quorum before updating
ansible.builtin.command: pvecm status
register: initial_cluster_status
changed_when: false
when: pvecm_binary.stat.exists
- name: Abort if PVE cluster does not have quorum
ansible.builtin.assert:
that:
- initial_cluster_status.rc == 0
- initial_cluster_status.stdout is search('(?m)^Quorate:\s+Yes\s*$')
fail_msg: |
PVE cluster does not currently have quorum.
Refusing to update this node.
pvecm status output:
{{ initial_cluster_status.stdout }}
when: pvecm_binary.stat.exists
# ============================================================
# APT PACKAGE UPDATES
# ============================================================
- name: Refresh apt package cache
ansible.builtin.apt:
update_cache: true
cache_valid_time: 3600
- name: Upgrade Proxmox packages
ansible.builtin.apt:
upgrade: dist
autoremove: true
autoclean: true
# ============================================================
# CHECK WHETHER REBOOT IS REQUIRED
# ============================================================
- name: Check if reboot is required
ansible.builtin.stat:
path: /run/reboot-required
register: reboot_required_file
# ============================================================
# REBOOT
# ============================================================
- name: Reboot host if required
ansible.builtin.reboot:
msg: "Rebooting Proxmox host following package updates"
connect_timeout: 10
reboot_timeout: 900
pre_reboot_delay: 10
post_reboot_delay: 30
test_command: "systemctl is-system-running"
when: reboot_required_file.stat.exists
# ============================================================
# POST-REBOOT PVE CLUSTER CHECK
# ============================================================
- name: Verify PVE cluster quorum after reboot
ansible.builtin.command: pvecm status
register: cluster_status
until:
- cluster_status.rc == 0
- cluster_status.stdout is search('(?m)^Quorate:\s+Yes\s*$')
retries: 30
delay: 10
changed_when: false
when:
- reboot_required_file.stat.exists
- pvecm_binary.stat.exists
- name: Display PVE cluster status after reboot
ansible.builtin.debug:
msg: |
PVE cluster is healthy after reboot.
{{ cluster_status.stdout }}
when:
- reboot_required_file.stat.exists
- pvecm_binary.stat.exists
- cluster_status is defined
# ============================================================
# PVE SERVICE HEALTH CHECKS
# ============================================================
- name: Ensure pve-cluster service is running
ansible.builtin.systemd:
name: pve-cluster
state: started
when: pvecm_binary.stat.exists
- name: Ensure corosync service is running
ansible.builtin.systemd:
name: corosync
state: started
when: pvecm_binary.stat.exists
# ============================================================
# FINAL PVE QUORUM CHECK
# ============================================================
- name: Perform final PVE quorum check
ansible.builtin.command: pvecm status
register: final_cluster_status
changed_when: false
when: pvecm_binary.stat.exists
- name: Confirm final PVE cluster quorum
ansible.builtin.assert:
that:
- final_cluster_status.rc == 0
- final_cluster_status.stdout is search('(?m)^Quorate:\s+Yes\s*$')
fail_msg: |
PVE cluster is NOT quorate after maintenance.
pvecm status output:
{{ final_cluster_status.stdout }}
success_msg: "PVE cluster quorum is healthy."
when: pvecm_binary.stat.exists
# ============================================================
# HOST MAINTENANCE COMPLETE
# ============================================================
- name: Report successful maintenance
ansible.builtin.debug:
msg: >-
{{ inventory_hostname }} has completed package maintenance
successfully. The next host may now be processed.
I know the above example is aimed at Proxmox with checks but the concept is similar but simpler for Ubuntu/Debian, see below:
---
- name: Update Ubuntu Package Repository Cache
hosts: ubuntu_servers
become: true
environment:
DISABLE_AUTO_UPDATE: "true" # Suppresses the Oh My Zsh upgrade prompt
tasks:
- name: Run apt update
ansible.builtin.apt:
update_cache: true
- name: Update apt cache and upgrade packages
ansible.builtin.apt:
update_cache: true
upgrade: dist
- name: Remove unused packages
ansible.builtin.apt:
autoremove: true
purge: true
- name: Check if a reboot is required
ansible.builtin.stat:
path: /run/reboot-required
register: reboot_required_file
- name: Reboot if required
ansible.builtin.reboot:
msg: "Reboot initiated by Ansible because Ubuntu requires it after package updates"
connect_timeout: 5
reboot_timeout: 300
pre_reboot_delay: 0
post_reboot_delay: 30
test_command: whoami
when: reboot_required_file.stat.existsWhat used to eat up an entire evening and invite mistakes now takes less than two minutes of my actual time. Ansible logs in, refreshes the apt cache, installs distribution upgrades, reboots what needs to be rebooted, confirms the cluster is healthy, and leaves me with a clean, green recap screen showing that every single server is perfectly up to date.
If your home lab is growing and manual maintenance is starting to feel like a second job where things slip through the cracks, stop SSHing into every box. Spend an hour learning Ansible—your future self will thank you for the hours of free time and peace of mind you get back.
Comments are closed