Lab 4 - Processes and Services
Facilitator: Owen Thompson
10 min readTable of contents
Overview
For this lab, we are going to dive into processes and systemd. We will do this by writing our own systemd service from scratch, while showing the benefits of running a service with systemd. This lab should be completed on your Linux VM.
Part 0: Set up networking
Before you start this lab, you’ll need to make sure you can access services from your VM in your web browser!
Our VMs support IPv6 only, so you will need to connect to the campus GlobalProtect VPN first if you don’t have IPv6 connectivity.
Part 1: Using systemd
What services are running right now?
Run systemctl
. You’ll see a long table of every unit known to systemd.
Let’s narrow it down to services for now. Run systemctl --type=service
. Now you can see a list of all services running on your computer. Each of these services is a daemon running in the background. Do you see any familiar services running?
Question 1: What is the name of a systemd service running on your system? What does it do? (If you’re not sure what it does, Google it!)
Controlling Services
Now let’s use systemd
to control a an nginx web server. If you don’t have it already, install nginx by issuing sudo apt install nginx
. Once that is done we can tell systemd to start the service with the following: sudo systemctl start nginx
. Run systemctl status nginx
to ensure it is running.
Note: If you already have a webserver running, you may need to shut it down, so that port 80 is available for nginx to use.
Now let’s make nginx listen for connections on the nonstandard port 420. In /etc/nginx/sites-available/default
change the following lines:
listen 80 default_server;
listen [::]:80 default_server;
to:
listen 420 default_server;
listen [::]:420 default_server;
TIP: The first line configures the server to listen on IPv4, and the second line configures IPv6.
Tell systemd that nginx has changed configuration and needs reloading with: sudo systemctl reload nginx
.
Now, accessing http://[yourusername].decal.xcf.sh:80 should now give you a connection refused error and your webserver will only be accessible via http://[yourusername].decal.xcf.sh:420.
Note that not all services can be reloaded; systemd will notify you if this is the case and such services will have to be restarted instead with: sudo systemctl restart yourservice
.
Finally go ahead and stop the nginx service with sudo systemctl stop nginx
.
Question 2: What is the difference between systemctl reload yourservice
and systemctl restart yourservice
?
Question 3: Upload a screenshot of your browser accessing the nginx webserver at http://[yourusername].decal.xcf.sh:420.
Note: If you can’t access the IPv6 site use curl localhost:420
on the VM and paste it’s contents (it should be a html page).
Creating a service
Let’s set up a web server and create a systemd unit for it. Make sure git
is installed; if it’s not, install it using apt
.
If you don’t already have the decal-labs repo from a past lab, run the following:
$ git clone https://github.com/0xcf/decal-labs
The materials for this part of the lab will be in the decal-labs/4
directory.
We will also need to install some dependencies. Go ahead and execute the following commands:
# apt update
# apt install build-essential make python3-virtualenv
Now run ./run
. This should start up a simple web server at http://localhost:5000. Note that by default you can only access the web server on the VM itself.
Your mission, should you choose to accept it, is to write a systemd service that manages this web server. To do this, make a new unit file in /etc/systemd/system/toy.service
. Refer to the slides for an example; DigitalOcean also has a good guide on how to write systemd units. Here is a skeleton; all you need to do is fill in the values for each field.
[Unit]
Description=
Requires=
After=
[Install]
WantedBy=multi-user.target
[Service]
ExecStart=
User=
Some questions worth considering while writing this unit file are:
- What units needs to be started before a webserver starts? (Hint: you can get a list of special “target” units using
systemctl --type=target
.) - What script should systemd run to start the webserver?
- Units run by root as default. Is that a safe practice for web servers?
You are encouraged to experiment with other fields as suits your liking.
Once you have finished creating toy.service
, let’s start the service and have the it start whenever our machine is booted.
# systemctl start toy.service
# systemctl enable toy.service
Debugging
You can check if the unit file succeeded by running systemctl status toy.service
. If you are having issues with the unit file or the web server, check the logs for this unit by running journalctl -u toy.service
. If you run into errors don’t get demoralized (it is, after all, only a decal); as a sysadmin you’ll have to become comfortable making sense of arcane error messages.
TIP: You can omit the
.service
insystemctl
command for speed. If the unit is another type (e.g. target, socket, or timer), you must include the type. We include the.service
for clarity.
Crash the service!
One of the great benefits of using systemd to manage your services is that you don’t have to worry unnecessarily about bringing a process back up if it crashes. So let’s crash the service! You can do this by either sending a POST request with the json payload {"crash":"true"}
to http://localhost:5000/crash on the VM (Hint: use cURL
) or by killing the webserver manually by sending a signal – both will cause the unit to crash. You can verify if you succeeded by running systemctl status toy.service
, and the unit should either be in an inactive
or failed
state, depending on how you killed it.
Question 4: What command did you run to crash the service?
Now add the following the /etc/systemd/system/toy.service
under the Service
directive:
Restart=always
RestartSec=10
To tell systemd that the unit file has changed run sudo systemctl daemon-reload
. Now start your webserver and crash it again in any way you please, and you should see that it come back online after 10 seconds! Note that you can also run daemon-reload
and change a unit file while a service is running.
Question 5: Upload your fully featured toy.service
file to Gradescope.
Part 2: Processes
There are no Gradescope questions to answer for this section, but you should still go through the steps to make sure you understand processes and how to use htop!
Open up a terminal and run the ps
command. You should see something like this:
PID TTY TIME CMD
3371 pts/2 00:00:00 zsh
3416 pts/2 00:00:00 ps
Now open up another terminal and run sleep 1000 &
, which start a sleeping process in the background. Then run ps
. It should look like:
~
❯ sleep 100 &
[1] 3726
~
❯ ps
PID TTY TIME CMD
3371 pts/2 00:00:00 zsh
3726 pts/2 00:00:00 sleep
3752 pts/2 00:00:00 ps
In the first terminal run ps
again. You should notice that the sleep
process is not showing up, even though the thousand seconds haven’t expired.
Why do you think this behavior occurs (hint: TTY column)?
We can get the process to display on the first terminal by running ps -u
, which displays all the processes running as your user. Notice the PID column; each process has a unique ID assigned to it by the kernel. One thing we can do with this PID is send signals to the process. sleep 1000
is pretty useless, so go ahead and kill it – kill 3726
(substitute 3726
with whatever PID ps
outputted for you).
The most common use of ps
is to run ps -ef
to see all the processes running on the system. Run ps -e
and ps -f
independently to see how the flags work together.
htop
Make sure htop
is installed by running sudo apt install htop
. Now, open up a terminal and run the htop
command. htop
can be thought of as a more extensive version of ps -ef
, whereby process stats are updated in real-time.
First press <F2>
, scroll down to Display options, and check “Hide userland process threads.” We won’t be dealing with those in this lab.
Now open up another terminal (or use tmux
). Run the command yes
. It uses a lot of resources as it prints a continuous stream of y
’s.
What resource specifically does the yes
command exhaust? If you are having trouble finding this, press <
to choose which resource to order processes by. Make sure to quit out of yes
(^C) once you are finished.
The process hierarchy
Run htop
once more. This time click <F5>
to enter Tree View. You should see a visual representation of the process hierarchy on your system, with everything stemming from /sbin/init
(systemd).
For curious students that are interested in seeing a more extensive process hierarchy on a large system, you are encouraged to run htop
on the OCF server tsunami
. Let us know of any cool processes that you find!
Orphan processes
Open a second terminal and ssh
to your VM. Now run sleep 1000 &
. You should see this new process pop into your htop
session on your first terminal. If not, press <F3>
and search for “sleep.” What is its parent?
Select this parent and press <F9>
to kill it. Send the SIGTERM
signal. The sleep process now has init
as its new parent, which is PID 1. What you just did is manually orphan a process; when that happens said process is subsequently re-parented by the init
process.
Now go through the same steps again. This time, send the parent a SIGHUP
(hangup) signal. Can you still find the sleep process? When SIGHUP
is sent to a parent shell, the parent subsequently sends hangup signals to any child processes before terminating; all processes that receive SIGHUP
from a parent shell will terminate – this is one way to avoid creating orphan processes.
If you are interested in learning about the different signals, run man 7 signal
. Note that you can run man man
for an explanation about the different manual section numbers.
Exploration
Congratulations, you have completed the lab! This is just the tip of the iceberg when it comes to processes and services. If you want to learn more, here are some related topics you can look into.
- Wikipedia’s article on init systems
- The construction of a basic init system
- Yelp’s dumb-init, a lightweight init system for docker containers
- Zombie Processes
- Socket activation
- Systemd has been the source of a considerable amount of controversy. Opponents allege that it violates the Unix philosophy of “do one thing and do it well”, and that it has had too much scope creep, among other complaints.
- Everything you wanted to know about Unix threads, processes, process groups and sessions. Bear in mind that this document is a little dated when it comes to the code about threads, and its description of what happens when a pseudotty is closed is not actually correct.
Submission
Go to Gradescope to submit your answers!