Update thousands of servers

Keeping a fleet of servers up to date is one of those tasks that sounds simple until you have a thousand of them. Updating one machine is trivial; updating a thousand, concurrently, with proper error handling, is an infrastructure challenge, whether you’re managing Linux, macOS, or Windows hosts. In this tutorial, we’ll set up CTFreak to do exactly that: deploy and execute an update script across your entire fleet on a recurring schedule, with a clear view of what succeeded and what didn’t. We’ll illustrate with Debian-based Linux servers, but the same approach applies to any OS with minimal adaptation.

What we’ll build

By the end of this tutorial, you’ll have a fully automated update pipeline that handles your entire server fleet. Here’s the plan:

1. Writing a non-interactive update script for Debian-based servers (Debian, Ubuntu, or similar)
2. Adding an SSH credential to CTFreak
3. Importing all your nodes from a single YAML file
4. Creating a project and a scheduled task that targets nodes by tag
5. Running the task and re-executing on failed nodes

Prerequisites

Before we start, make sure you have:

• A fleet of Linux servers to update (we’ll use 1,000 in this example). Each server has a hostname following the pattern serverXXXXX.local (from server00001 to server01000), runs a Debian-based distribution, and is accessible via SSH on port 22 with a shared private key and an adminuser account that has passwordless sudo privileges.
• A CTFreak instance with an administrator account. CTFreak has no external dependencies, so installation only takes a moment. The Free Edition is sufficient for everything in this tutorial.

Step 1: Writing the update script

The bash script we are going to run on each server is straightforward:

#!/bin/bash
set -euo pipefail

export DEBIAN_FRONTEND=noninteractive

sudo apt-get update -y -q

sudo apt-get upgrade -y -q \
    -o Dpkg::Options::="--force-confdef" \
    -o Dpkg::Options::="--force-confold"

sudo apt-get autoremove -y -q
sudo apt-get autoclean -q

if [[ -f /var/run/reboot-required ]]; then
    if [[ ${CTP_AUTO_REBOOT:-} == "true" ]]; then
        echo "THE SERVER IS GOING TO REBOOT"
        nohup sh -c 'sleep 5 && reboot' &>/dev/null &
        disown
    else
        echo "REBOOT REQUIRED"
    fi
fi

A few things to note:

• DEBIAN_FRONTEND=noninteractive prevents apt-get from opening interactive prompts that would block an unattended session.
• The two Dpkg::Options flags handle configuration file conflicts automatically: keep the currently installed version (--force-confold) unless the package ships a new file that didn’t previously exist (--force-confdef).
• Finally, the script checks for /var/run/reboot-required to detect whether a reboot is needed after the upgrade. If the CTP_AUTO_REBOOT environment variable is set to true, the server reboots itself automatically; otherwise it simply logs REBOOT REQUIRED so you can schedule reboots at your convenience. We will define the content of CTP_AUTO_REBOOT later via a dedicated task parameter.

If your fleet includes servers running RPM-based distributions (Rocky Linux, AlmaLinux, RHEL, or similar), replace the apt-get commands with their dnf equivalents.

Step 2: Adding the SSH credential

CTFreak needs a credential to authenticate against your servers over SSH.

Navigate to Credentials and click the New Credential button. Paste your private key, give it a recognizable name like MySSHKey, and save.

You can also use password-based authentication instead of a private key. Both options are supported when creating a credential.

Step 3: Importing nodes from a YAML file

In CTFreak, a node represents a remote server that tasks can target. Nodes are organized into node sources. You could add nodes one by one through the web interface, but with 1,000 servers the practical approach is to create an external node source backed by a YAML file.

Create the file /home/adminuser/ctfreak-nodes.yaml (it must be readable by the OS user running the CTFreak process):

- name: server0001
  tagNames:
    - deb_server
    - linux_server
  osFamily: UNIX
  connectionProtocolType: SSH
  sshConnectionProtocol:
    username: adminuser
    hostname: server0001.local
    port: 22

- name: server0002
  tagNames:
    - deb_server
    - linux_server
  osFamily: UNIX
  connectionProtocolType: SSH
  sshConnectionProtocol:
    username: adminuser
    hostname: server0002.local
    port: 22

## ...
## Complete with nodes 3 to 1000
## ...

Notice the tagNames field. Tags let you organize nodes into logical groups and, more importantly, target them from tasks. We’ll use the deb_server tag in the next step to tell CTFreak which nodes should receive the update.

Navigate to Nodes and click the New External Node Source button.

Select MySSHKey as the Credential (CTFreak will use it when connecting to these nodes via SSH), then save. The new node source, named Linux Servers in this example, will automatically resynchronize its nodes from the YAML file every two hours.

If the file was processed successfully, you should see all your nodes listed:

Step 4: Creating the project and task

Navigate to Projects and click the New Project button. Name your project Sysadmin (or any name that fits your organization), then save.

Once you are on the project page, click the New Task button, choose Bash Script as the task type, and configure the task with the following settings:

• Name: Upgrade debian servers
• Parameters: add a checkbox parameter named AUTO_REBOOT (default unchecked)
• Bash script: paste the update script from Step 1
• Node Set filter: #deb_server (targets every node carrying this tag)
• Schedule: a cron expression such as 0 4 1 * * (4:00 AM on the 1st of every month)

Save the task.

Because we used a tag-based filter rather than an explicit list of hostnames, any new node you add to the YAML file with the deb_server tag will automatically be included in future executions, with no task reconfiguration needed.

Step 5: Running the task and handling failures

Rather than waiting for the first of the month, let’s trigger a manual run. Click the Execute button.

As you can see, you can specify whether or not you want to automatically reboot servers that need it after the update (in the case of a scheduled launch, the default value AUTO_REBOOT = false will be applied).

The execution starts and CTFreak connects to the matching nodes concurrently (sequentially in the Free Edition). You can watch progress on a per-node basis.

When the execution completes, some nodes will almost certainly have failed. This is expected at scale: an SSH connection may time out, a package mirror might be temporarily unreachable, or a DNS resolution could fail. The important thing is that you have clear visibility into which nodes failed and why.

Check the Failed filter to isolate the problematic nodes and inspect their logs. Once you’ve addressed the underlying issue (or simply want to retry transient failures), click Re-execute failed nodes. CTFreak will run the same task again, but only against the nodes that failed in the previous execution. This saves time and avoids redundant work on the nodes that already succeeded.

Going further

Once the basic pipeline is running, there are several ways to extend it:

• Notifications: attach an email or Slack notifier to the task so your team is alerted immediately when an update run finishes with failures.
• Mixed-OS workflows: add RPM-based Linux nodes (Rocky Linux, AlmaLinux, RHEL) with a rpm_server tag and Windows nodes with a windows_server tag. Create a dedicated task for each OS (using Bash for Unix, PowerShell for Windows) and tie them all together in a workflow task.
• Ansible playbooks: replace the Bash or PowerShell script with an Ansible playbook to leverage built-in idempotency. Ansible modules like apt and dnf handle package state declaratively, so you don’t have to manage idempotence yourself.
• Concurrency tuning: adjust the number of workers (aka Max number of concurrent node executions) in the task configuration to control how many nodes are updated in parallel. This is useful if your network or package mirrors can’t handle a thousand simultaneous connections.

Wrapping up

With a single YAML file, one SSH credential, and a scheduled task, CTFreak gives you a repeatable, observable process for keeping your entire server fleet up to date. Failed nodes are clearly surfaced and can be retried with one click, which turns a potentially chaotic maintenance window into a routine operation.

The same pattern applies well beyond OS updates: configuration changes, log collection, security audits, or any script you need to run across many machines. Define the nodes, write the script, schedule the task, and let CTFreak handle the concurrency and error tracking.

Happy automating!