Performance optimization
If you intend to use CTFreak intensively (e.g. keep a history of more than 400,000 executions and/or launch more than 400,000 executions per day), here are some tips you can apply in priority order.
(Note that executing the same shell script on several nodes or the same sql script on several databases counts as a single execution.)
Memory usage
Generally speaking, CTFreak is quite conservative in terms of memory consumption, so optimization tips will focus on CPU & disk space usage and I/O operations.
Disable project and task execution counters
By default, the project and task lists displayed in the web interface (and returned by the API) are accompanied by execution counters.
This is valuable information, but it can be costly to retrieve when you want to keep a large execution history.
If you notice a slowdown in consulting project and task lists, go to Settings → Global Settings menu and deactivate the retrieval of these counters via the dedicated checkbox.
Reduce the execution retention period
Experience has shown that when the execution history exceeds 5 million (depending on the characteristics of the host server), this can have a significative impact on CTFreak’s performance in terms of task scheduling accuracy and CPU usage.
As a reminder, execution retention period can be set globally, but also by project or even by task.
Don’t hesitate to use it to reduce the load on CTFreak, especially for high-frequency tasks that don’t necessarily need to keep a large execution history.
Reduce disk space used by execution logs
Most task types create one or more associated log/data files for each execution.
You can significantly reduce the disk space occupied by these files by keeping them only for failed or aborted executions, or even not keeping them at all if only the execution status matters to you.
Using an NVME drive for Config folder
CTFreak uses its configuration folder to store both its embedded SQLite backend database and execution logs.
The installation prerequisites already specify that this folder should not be stored on a network drive (for performance reasons, but also to manage locks on access to the embedded database).
However, you can go one step further and store this folder on an NVME drive to prevent I/O saturation in the event of high execution frequency.
Special case of high task execution frequency on the same server
Although CTFreak was originally designed for remote task execution on multiple servers, if you need to execute more than 40,000 command or script tasks per day on a single server, it’s better to install an instance of CTFreak on that server and use local command tasks (rather than launching remote executions).
Indeed, in the case of remote execution, the number of SSH connections that the target server has to handle, combined with the network connection, may make it impossible to maintain this execution frequency.
Do without powershell
When you run a local command task under Windows, the command line is by default launched via a powershell session.
In the case of intensive execution of this type of task, the instantiation of multiple powershell sessions consumes a lot of memory & CPU. It is therefore recommended to do without powershell via the “No Shell” option.
Using PostgreSQL as backend database instead of SQLite
By default, CTFreak uses SQLite as its backend database, making it easy to install and offering excellent performance (especially if you’ve stored it on a local NVME disk) for the majority of our customers.
However, this may not be sufficient for the most demanding ones, especially if:
- you need to keep a large execution history (typically more than 1 million executions)
- you have a large number of executions status changes in a very short time span (e.g. more than 1500 executions starting or ending in one minute)
- you have a large number of executions status changes during hot backup of the embedded SQLite backend database
- you have a large number of users continuously consulting the list of running executions (more than 25 users)
In these cases, we recommend using PostgreSQL as your backend database (available with a Business Edition license), as SQLite will probably reach its performance limits.