Systemd Service File Tutorial: Linux Runbook

by Liam Foster
Systemd Service File Tutorial: Linux Runbook

Systemd service files are the modern Linux mechanism for running and managing long-lived processes. Instead of init scripts, you write a declarative unit file—systemd handles startup order, restart logic, resource limits, and logging. This tutorial shows the model, the syntax, and where things break.

The Model: Units, Targets, and Dependencies

Systemd organizes the system as a graph of units. A service unit describes a daemon. A target is a synchronization point (like multi-user.target, the default runlevel). Dependencies express ordering: Before, After, Requires, Wants.

When you enable a service, systemd creates a symlink in /etc/systemd/system/multi-user.target.wants/. At boot, systemd reads all enabled units, resolves the dependency graph, and starts them in parallel where safe.

Key insight: systemd doesn't run services sequentially. It calculates the minimum-depth execution plan. If service B wants service A, systemd starts A first, but runs other independent services concurrently. This is faster than old init, but requires you to think about ordering explicitly.

Writing a Basic Service File

Service files live in /etc/systemd/system/ (or /usr/lib/systemd/system/ for packaged units). The name is arbitrary; use something.service.

[Unit]
Description=My Application Daemon
After=network.target
Wants=network-online.target

[Service]
Type=simple
User=appuser
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/server --config /etc/myapp/config.yaml
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

Breakdown:

[Unit]: Metadata and dependencies. After=network.target means "start after the network is available". Wants is weaker than Requires—if network-online.target fails, this service still starts. Use Wants for soft dependencies, Requires for hard ones.

[Service]: Runtime behavior. Type=simple means the process stays in the foreground (systemd tracks its PID). User=appuser runs the daemon as that user (create it first). ExecStart is the command; use absolute paths. Restart=on-failure restarts if the exit code is non-zero. RestartSec=5s waits 5 seconds between restarts.

[Install]: How to enable the service. WantedBy=multi-user.target means systemctl enable will symlink this unit into multi-user.target.wants/, so it starts at boot.

Service Types and Startup Behavior

Type=simple works for most applications. The process runs in the foreground; systemd considers the service started once ExecStart returns. Gotcha: if your app forks itself into the background, use Type=forking instead—but this is rare and fragile. Modern apps should daemonize via systemd, not fork.

Type=notify requires the application to send a readiness signal via sd_notify(). Use this if you need systemd to wait for the app to be truly ready (e.g., listening on a socket) before starting dependent services. The app must link against libsystemd.

Type=oneshot runs the command once and exits. Useful for setup tasks, not daemons. Pair it with RemainAfterExit=yes if you want the service to stay "active" after the command finishes.

Restart Logic and Failure Handling

Restart=on-failure restarts the service if it exits with a non-zero code or is killed by a signal. Restart=always restarts regardless. Restart=no (the default) never restarts.

RestartSec controls the delay between restarts. Use StartLimitBurst and StartLimitIntervalSec to prevent restart loops:

Restart=on-failure
RestartSec=5s
StartLimitBurst=3
StartLimitIntervalSec=60s

This means: if the service crashes 3 times within 60 seconds, give up and mark it failed. Systemctl status will show the reason. Gotcha: systemd-logind and other system services can restart infinitely by default; always set limits for user-facing daemons.

Resource Limits and Security

Systemd can enforce resource constraints without cgroups v2 complexity:

LimitNOFILE=65536
LimitNPROC=4096
MemoryLimit=512M
CPUQuota=50%

LimitNOFILE caps open file descriptors. LimitNPROC caps child processes. MemoryLimit sets a hard memory ceiling (OOM killer enforces it). CPUQuota throttles CPU usage to a fraction of one core (50% = half a core).

For security, use PrivateTmp=yes (isolate /tmp), NoNewPrivileges=yes (prevent privilege escalation via setuid), and ReadOnlyPaths=/etc (make /etc read-only if the app doesn't need to write there).

Logging and Debugging

Systemd captures stdout and stderr automatically and sends them to journald. View logs with:

journalctl -u myapp.service -f

The -f flag follows new entries (like tail -f). Add -n 50 to see the last 50 lines, or --since "1 hour ago" to filter by time.

For verbose startup diagnostics:

systemctl status myapp.service
systemctl start myapp.service --no-block
journalctl -xe

The -e flag jumps to the end of the journal. -x adds explanatory text for known errors.

Gotcha: if your app writes to a log file, journald doesn't capture it. Either pipe the app to systemd-cat, or configure the app to write to stdout and let systemd handle it.

Enabling and Testing

After writing the service file:

sudo systemctl daemon-reload
sudo systemctl enable myapp.service
sudo systemctl start myapp.service
sudo systemctl status myapp.service

daemon-reload re-reads all unit files; required after any edit. enable creates the symlink for boot-time startup. start runs it now. status shows the current state and recent log lines.

Test the service manually first:

/opt/myapp/bin/server --config /etc/myapp/config.yaml

Verify it starts, listens on the right ports, and exits cleanly when you Ctrl+C. Then wrap it in a service file.

When This Breaks

Service doesn't start: Check journalctl for errors. Common causes: missing User, bad path in ExecStart, permission denied on WorkingDirectory.

Service starts but exits immediately: The app is crashing. Run it manually to see the error. If it's a Python app, check for import errors. If it's compiled, check for missing libraries (ldd /path/to/binary).

Restart loop: Increase StartLimitBurst or remove the limit entirely while debugging. Once you fix the root cause, re-add it.

Dependency not met: If you use Requires=other.service and other.service fails, this service won't start. Use Wants instead if the dependency is optional.

Port already in use: If two services bind the same port, the second fails silently. Use systemctl list-units --state=failed to find which one crashed. If you're running services on a migrated host, the comparison on tinjauhost.biz.id covers common pitfalls worth reviewing before you go live.

One-Line Takeaway

Systemd service files replace init scripts: write declarative unit files, use systemctl to manage them, and rely on journald for logs. The model is dependency graphs and parallel startup; the gotchas are restart loops, permission issues, and background-forking apps.