While working on that same project that brought me to look at bash lock files I also needed to co-ordinate the running of a number of scripts away from the main thread of the originating task. The context here is that we had a number of scripts to sync yum repositories on an internal mirror that we were creating. We had implemented the bash lock files pattern from the linked post on 3 separate scripts that between them handled all of the repos necessary. There were a number of repos to sync and especially on a first launch scenario the sync would take many hours, in the region of 8 when run consecutively on a single node. As discussed in the bash lock files post we had 3 nodes in an auto-scaling group and were splitting the workload across all 3 nodes using the lock file pattern. However, a limitation of using an auto-scaling group is that there is a hard limit of 2 hours for the nodes to come up and complete their cloud-init (userdata) tasks. Even split across 3 nodes we couldn’t guarantee that all 3 nodes would sync inside the 2 hour limit. So we needed a way to trigger the scripts to be run in such a way that the cloud-init tasks would complete, even while the repo sync was running.
The approach we came up with was to use a SystemD unit file to run the scripts. This way we could trigger the service from the cloud-init tasks which would then receive a success signal for the service starting and then the cloud-init tasks could complete. The service would then run the scripts in the background and we could monitor the progress of the scripts using the journalctl command.
What is SystemD?
I’m not going to get into everything that SystemD is here, you can read all about it here, in short it’s the core of almost all modern Linux systems today (at the time of writing…) and is responsible for managing the system services and processes. If you’re like me and come from a Windows background and you’ve worked with services and services management through services.msc then you can think of SystemD as the Linux equivalent for the purposes of our discussion here.
What is a SystemD unit file?
A SystemD unit file is a configuration file that defines a service, socket, device, mount point, or other entity that SystemD can manage. The unit files are stored in /etc/systemd/system/ and can be used to start, stop, restart, enable, disable, and otherwise manage the services and processes on a Linux system. See the following documentation for more information on the use of SystemD, SystemD unit files, and SystemD service files:
Note: there are other types of unit file you will also find listed here which may be useful in other scenarios. We may even touch on some of them later in this post.
So, how do we use SystemD to run our scripts?
We can use a service file to define the service we want to run. Here’s an example of a service file that we used to run our repo sync scripts:
[Unit]
Description=Sync Yum Repositories
[Service]
Type=simple
ExecStart=/usr/local/bin/sync-repos.sh
This will would then be saved in /etc/systemd/system/ as sync-repos.service e.g. /etc/systemd/system/sync-repos.service. Obviously (hopefully), we can name the service whatever we like, but it’s good practice to name it something that makes sense and is descriptive of what the service does.
All we need to do now is use systemctl to start the service:
systemctl start sync-repos.service
If the service is intended to run on boot we can enable it:
systemctl enable sync-repos.service
And, of course, we can check the status of the service:
systemctl status sync-repos.service
What do the options in the service file mean?
Please see the docs for a full list of options, but here’s a quick rundown of the options we used in our service file:
The Description is a description of the service that will be displayed when you run systemctl status sync-repos.service.
The Type listed in the service file can be one of the following:
simpleexecforkingoneshotdbusnotifynotify-reloadidle
Details on what they all mean can here found listed under Options
The ExecStart is the command that will be run when the service is started. In this case we’re running a script that we’ve saved in /usr/local/bin/sync-repos.sh. This is effectively the approach we used for bash lock files. This gave us the benefit of being able to trigger the script to run from the cloud-init tasks and then check on the status and progress of the script using journalctl or by grepping /var/log/messages for the output of the script.
In our use case we actually opted for oneshot as the ExecType as we only wanted the script to run once and then exit. We had set the scripts to run using cron for future runs. We could have used Timer Units instead of cron however the majority of the team are familiar with cron and many other tasks already configured are using cron so we decided to stick with it for now.
Timer Units - replacing cron
So, I mentioned that we could have replaced cron… Depending on your viewpoint this might be an exciting proposition, or an act of heresy! I’m not here to get into the politics of cron vs SystemD timers or anything else; what I will say though is I’m personally a fan of building as much as possible into the system operations themselves, rather than many additional and bolt-on services. I’m also a fan of consistency and less dated and convoluted approaches to things. You’re entitled to your opinion, but for me if you command structure has some weird, non-standard syntax dating back 40 years, I’m probably going to look for something more modern.
Anyway, back to SystemD Timers! Timers are another type of unit file that can be used to trigger the execution of a service at a specific time or at a specific interval (kind of like cron). The timer unit file is saved in the same location as the service file, /etc/systemd/system/, and has the same name as the service file but with a .timer extension. Here’s an example of a timer file that would run the service every 6 hours:
[Unit]
Description=Run sync-repos.service every 6 hours
[Timer]
OnCalendar=*-*-* 0/6:00:00
Persistent=true
Unit=sync-repos.service
[Install]
WantedBy=timers.target
This file would be saved as /etc/systemd/system/sync-repos.timer.
The OnCalendar option is used to define when the service should run. In this case we’re using *-*-* 0/6:00:00 which means every 6 hours.
The Persistent option is used to ensure that if the system is down when the timer is due to run, the service will run as soon as the system is back up.
The Unit option is used to define which service the timer is triggering.
The WantedBy option is used to define which target the timer is associated with. In this case we’re using timers.target which is the target for timer units.
You can find more details here: SystemD Timer Units
Starting services when we’re using timers
If you’re using a timer to trigger the service then you’ll need to make sure that the timer is enabled and started. You can do this using the following commands:
systemctl enable sync-repos.timer
systemctl start sync-repos.timer
Note: this enables the timer, not the service. The service will be triggered by the timer when the next scheduled time is reached. If you want the service to run immediately you can start the service directly:
systemctl start sync-repos.service
If you’ve already enabled and started the timer then the service will run at the next scheduled time, as well as now.
You can get the status of the timer using:
systemctl status sync-repos.timer
And you can get the status of the service using:
systemctl status sync-repos.service
Conclusion
SystemD is a powerful tool for managing services and processes on a Linux system. It can be used to run scripts and services in the background and can be used to trigger services at specific times or intervals. It can be used to replace cron for running scripts and services at specific times or intervals and can be used to manage the execution of scripts and services in a more modern and consistent way.
So next time your project needs to run complex tasks during launch, such as AWS cloud-init, and you need to run scripts in the background, or want to set up complex schedules for running scripts, consider using SystemD to manage the execution of your scripts and services.
If this article helped inspire you please consider sharing this article with your friends and colleagues, or let me know via LinkedIn or X / Twitter. If you have any ideas for further content you might like to see please let me know too.
