Lab 2 - Process management

In this laboratory you will exercise with systemd and process monitoring.

Contents:

1. Lab 2.1 Working with systemd

   • Lab objective 1: Viewing running services and configure and troubleshoot them
   • Lab objective 2: Adding a new startup service with systemd

2. Lab 2.2 Process monitoring

   • Lab objective 3: System monitoring using stress
   • Lab objective 4: Process monitoring

3. Lab 2.3 Managing user processes

   • Lab objective 5: More work with stress, top/htop and ps
   • Lab objective 6: Working with runlevels
   • Lab objective 7: More work with systemd
   • Lab objective 8: Working with foreground and background processes

Laboratory objective 1: Viewing running services and configure and troubleshoot them

Commands used: systemctl

Solution:

1. to view the status of all processes that are running on your system, just run the command:

systemctl

The list will be long, and thus will be shown in a manner similar to using the less command, page by page. In my case there were 167 (units) processes listed, while using openSUSE as a default distro.

2. check the status of a unit using the command

systemctl status [unit]
alexandru@linux-vje9:~> systemctl status apparmor.service
● apparmor.service - Load AppArmor profiles
 Loaded: loaded (/usr/lib/systemd/system/apparmor.service; enabled; vendor preset: enabled)
 Active: active (exited) since Sat 2017-09-02 12:25:55 EEST; 44min ago
 Process: 375 ExecStart=/etc/init.d/boot.apparmor start (code=exited, status=0/SUCCESS)
 Main PID: 375 (code=exited, status=0/SUCCESS)
 Tasks: 0 (limit: 512)
 CGroup: /system.slice/apparmor.service
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

3. to start or stop a service use:

systemctl start bluetooth.service
systemctl status bluetooth.service
systemctl stop bluetooth.service
systemctl restart bluetooth.service
systemctl status bluetooth.service

4. to enable or disable a service to start during boot, use the commands:

systemctl enable bluetooth.service
systemctl disable bluetooth.service

Adding or disabling a service to load during boot is basically the process of adding or removing symbolic links in the /etc/systemd/system/multi-user.target.wants directory.
Check for a systems status using the command:

systemctl is-enabled postfix.service

5. reboot or shutdown your system using systemd commands:

systemctl reboot
systemctl shutdown

Laboratory objective 2: Adding a new startup service with systemd

Commands used: touch, ls, cd, vim, chmod, systemctl

Solution:

This exercise addresses some more advanced solutions and uses commands that will be discussed later on. Nevertheless, it is of great importance as you will exercise how to work with systemd commands and services/processes.

1. create as root a file called fake2.service under /etc/systemd/system
   First let us explore the /etc/systemd/system directory, by moving inside it and do a listing there:

alexandru@linux-vje9:~ cd /etc/systemd/system/ alexandru@linux-vje9:/etc/systemd/system ls bluetooth.target.wants getty.target.wants dbus-org.bluez.service graphical.target.wants dbus-org.freedesktop.Avahi.service multi-user.target.wants dbus-org.freedesktop.ModemManager1.service network-online.target.wants dbus-org.freedesktop.nm-dispatcher.service network.service dbus-org.opensuse.Network.AUTO4.service remote-fs.target.wants dbus org.opensuse.Network.DHCP4.service sockets.target.wants dbus-org.opensuse.Network.DHCP6.service SuSEfirewall2_setup.service dbus-org.opensuse.Network.Nanny.service sysinit.target.wants default.target timers.target.wants default.target.wants xdm.service

2. now create the file fake2.service using the command (as root):

sudo touch fake2.service

3. check if the file was created by doing a listing of the directory with the ls command

4. open a text editor, like vim (which is installed by default in openSUSE) and start typing content:

[Unit]
Description=fake2 After=network.target
[Service]
ExecStart=/bin/echo I am starting the fake2 service
ExecStop=/bin/echo I am stopping the fake2 service
[Install]
WantedBy=multi-user.target

Now save the contents in vim and exit.

5. change the permissions on the file to make it executable:

chmod 755 /etc/systemd/system/fake2.service

6. now start, stop and check status of the service with systemctl

sudo systemctl start fake2.service 
sudo systemctl status fake2.service 
sudo systemctl stop fake2.service

7. to make the service run at system boot:

sudo systemctl enable fake2.service 
sudo systemctl disable fake2.service

8. reboot your system for the changes to take effect

Laboratory objective 3: System monitoring using stress

Commands used: stress-ng

Solution:

Stress is a utility program that will put stress on your system's CPU, RAM, I/O etc. You can install it in your system by looking for the stress or stress-ng into your repositories. Under openSUSE you can install the stress-ng instance:

sudo zypper in stress-ng

then you can run it

stress-ng -c 2 -i 2 -m 2 -t 10s

This will fork off 2 CPU intensive processes, each spinning on a sqrt() calculation, fork off 2 I/O intensive processes, each spinning on sync(), fork off 2 memory intensive processes, each spinning malloc() allocating 256MB by default (you can change the size with --vm-bytes 128M for example), and the processes are run for 10 seconds.

You can watch stress working by opening your system monitor.

Laboratory objective 4: Process monitoring

Commands used: ps, nice, renice, top

Solution:

1. run ps with -ef options first, and then with aux option and notice the differences

ps -ef
ps aux

The output will be LONG, so perhaps you would like to pipe this to less.

2. run ps so that only the process ID, priority, nice value and the process command line are displayed

alexandru@linux-vje9:~ ps -o pid,pri,ni,cmd
 PID PRI NI CMD
 4789 19 0 bash
 5273 19 0 ps -o pid,pri,ni,cmd

3. start a new bash session, then start another bash session with the nice command with a nice value of 10. After that, run ps like in the last exercise and notice the differences in priority and nice value, and note the process ID of the two bash sessions

alexandru@linux-vje9:~ bash
alexandru@linux-vje9:~ nice -n 10 bash
alexandru@linux-vje9:~ ps -o pid,pri,ni,cmd
 PID PRI NI CMD
 5346 19 0 bash
 5363 19 0 bash
 5380 9 10 bash
 5397 9 10 ps -o pid,pri,ni,cmd

4. change the nice value of one of the bash sessions to 15 using renice command. Observe the values.

alexandru@linux-vje9:~ renice 15 -p 5363
5363 (process ID) old priority 0, new priority 15
alexandru@linux-vje9:~ ps -o pid,pri,ni,cmd
 PID PRI NI CMD
 5346 19 0 bash
 5363 4 15 bash
 5380 9 10 bash
 5408 9 10 ps -o pid,pri,ni,cmd

5. run the top command and watch the output as it changes.

top
top - 23:35:55 up 1:26, 2 users, load average: 0.33, 0.28, 0.27
Tasks: 207 total, 1 running, 206 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.3 us, 0.7 sy, 0.0 ni, 97.8 id, 0.2 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 3928816 total, 3793096 used, 135720 free, 5200 buffers
KiB Swap: 4192252 total, 1552 used, 4190700 free. 1192920 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 
 3329 alexand+ 20 0 1643120 151776 67632 S 2.649 3.863 1:28.19 gnome-she+
 5076 alexand+ 20 0 3672540 351960 278216 S 1.656 8.958 1:37.56 VirtualBox
 1911 root 20 0 338436 95116 57904 S 0.993 2.421 1:06.75 Xorg 
 3735 alexand+ 20 0 2518836 626588 119200 S 0.993 15.95 5:07.03 firefox 
 5340 alexand+ 20 0 612128 35300 27272 S 0.993 0.898 0:01.13 gnome-ter+
 4945 alexand+ 20 0 1404536 76696 57052 S 0.662 1.952 0:06.88 VirtualBox
 3502 alexand+ 20 0 2958720 173996 44984 S 0.331 4.429 0:10.44 dropbox 
 4956 alexand+ 20 0 810256 19676 15648 S 0.331 0.501 0:05.04 VBoxSVC 
 5082 alexand+ 20 0 88316 8888 6944 S 0.331 0.226 0:00.67 VBoxXPCOM+
 5087 alexand+ 20 0 678152 17844 14936 S 0.331 0.454 0:01.89 VBoxSVC 
 1 root 20 0 185440 5988 3908 S 0.000 0.152 0:02.62 systemd 
 2 root 20 0 0 0 0 S 0.000 0.000 0:00.00 kthreadd 
 3 root 20 0 0 0 0 S 0.000 0.000 0:00.02 ksoftirqd+
 5 root 0 -20 0 0 0 S 0.000 0.000 0:00.00 kworker/0+
 7 root 20 0 0 0 0 S 0.000 0.000 0:00.72 rcu_sched 
 8 root 20 0 0 0 0 S 0.000 0.000 0:00.00 rcu_bh 
 9 root rt 0 0 0 0 S 0.000 0.000 0:00.01 migration+

Laboratory objective 5: More work with stress, top/htop and ps

For this exercise we used Debian 9.1 Stretch.

1. use the package stress as your local user

first install the stress program

sudo apt install stress

alexandru@debian:~$ su alexandru
Password:

alexandru@debian:~$ stress --cpu 2

stress: info: [11901] dispatching hogs: 2 cpu, 0 io, 0 vm, 0 hdd

2. check processes with top:

top

3. install htop and use it, as it is an advanced top application

sudo apt install htop

run htop

htop

4. use ps (process snapshot) to review the processes and focus on the stress process started as your local user:

alexandru@debian:~$ ps -A | grep stress
11901 pts/0 00:00:00 stress
11902 pts/0 00:06:35 stress
11903 pts/0 00:06:34 stress

alexandru@debian:~$ ps aux | grep stress
alexand+ 11901 0.0 0.0 7276 972 pts/0 S+ 12:20 0:00 stress --cpu 2
alexand+ 11902 99.8 0.0 7276 96 pts/0 R+ 12:20 6:56 stress --cpu 2
alexand+ 11903 99.7 0.0 7276 96 pts/0 R+ 12:20 6:55 stress --cpu 2
alexand+ 12250 0.0 0.0 12784 952 pts/1 S+ 12:27 0:00 grep stress

alexandru@debian:~$ ps -ef | grep stress
alexand+ 11901 11896 0 12:20 pts/0 00:00:00 stress --cpu 2
alexand+ 11902 11901 99 12:20 pts/0 00:07:15 stress --cpu 2
alexand+ 11903 11901 99 12:20 pts/0 00:07:15 stress --cpu 2
alexand+ 12252 11912 0 12:27 pts/1 00:00:00 grep stress

5. kill the process stress using the following commands:

alexandru@debian:~$ pgrep stress
11901
11902
11903

alexandru@debian:~$ sudo kill -9 11901
alexandru@debian:~$ pgrep stress
11902
11903

To kill all processes that are used by stress, use the command:

alexandru@debian:~$ sudo kill $(pgrep stress)
alexandru@debian:~$ pgrep stress

or you could use:

sudo pkill stresssudo killall stress

After that, you can check the first terminal session where you started stress:

alexandru@debian:~$ stress --cpu 2
stress: info: [11901] dispatching hogs: 2 cpu, 0 io, 0 vm, 0 hdd
Killed

Laboratory objective 6: Working with runlevels

Debian 9.1 was used for this exercise.

In Debian 9.1, the runlevel 5 is used as Graphical multi-user and networking runlevel. To check your runlevel, run the command:

alexandru@debian:~ sudo runlevel
[sudo] password for alexandru: 
N 5

Usually, for Debian based distros, the default runlevel is 2. Here are the main runlevel descriptions:

0 - system halt

1 - single user mode

2 - graphical multi user mode with networking

3 same as 2

4 same as 2

5 same as 2

6 system reboot

To change your runlevel you can use:

sudo rebootsudo shutdown -h now
sudo shutdown +3
sudo telinit 0

Laboratory objective 7: More work with systemd

Systemd is now used by default by all major distros, so you will not have any issue using those commands either in Debian, openSUSE, centOS or Fedora, Ubuntu or Mint.

We will now work with systemctl command options like start, stop, restart, reload, status and show, using the libvirtd service.

alexandru@debian:~ sudo systemctl status libvirtd
[sudo] password for alexandru: 
● libvirtd.service - Virtualization daemon
 Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor preset:
 Active: active (running) since Wed 2017-09-27 11:11:35 EEST; 2h 28min ago
 Docs: man:libvirtd(8)
 http://libvirt.org
 Main PID: 10405 (libvirtd)
 Tasks: 18 (limit: 4915)
 CGroup: /system.slice/libvirtd.service
 ├─10405 /usr/sbin/libvirtd
 ├─10689 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/defaul
 └─10690 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/defaul
Sep 27 11:12:11 debian dnsmasq[10689]: started, version 2.76 cachesize 150
Sep 27 11:12:11 debian dnsmasq[10689]: compile time options: IPv6 GNU-getopt DBu
Sep 27 11:12:11 debian dnsmasq-dhcp[10689]: DHCP, IP range 192.168.122.2 -- 192.
Sep 27 11:12:11 debian dnsmasq-dhcp[10689]: DHCP, sockets bound exclusively to i
Sep 27 11:12:11 debian dnsmasq[10689]: reading /etc/resolv.conf
Sep 27 11:12:11 debian dnsmasq[10689]: using nameserver 95.77.94.88#53
Sep 27 11:12:11 debian dnsmasq[10689]: using nameserver 78.96.7.88#53
Sep 27 11:12:11 debian dnsmasq[10689]: read /etc/hosts - 5 addresses
Sep 27 11:12:11 debian dnsmasq[10689]: read /var/lib/libvirt/dnsmasq/default.add
Sep 27 11:12:11 debian dnsmasq-dhcp[10689]: read /var/lib/libvirt/dnsmasq/defaul
alexandru@debian:~$ sudo systemctl stop libvirtd
alexandru@debian:~$ sudo systemctl status libvirtd
● libvirtd.service - Virtualization daemon
 Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor preset:
 Active: inactive (dead) since Wed 2017-09-27 13:40:37 EEST; 6s ago
 Docs: man:libvirtd(8)
 http://libvirt.org
 Process: 10405 ExecStart=/usr/sbin/libvirtd $libvirtd_opts (code=exited, statu
 Main PID: 10405 (code=exited, status=0/SUCCESS)
 Tasks: 2 (limit: 4915)
 CGroup: /system.slice/libvirtd.service
 ├─10689 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/defaul
 └─10690 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/defaul
Sep 27 11:12:11 debian dnsmasq-dhcp[10689]: DHCP, IP range 192.168.122.2 -- 192.
Sep 27 11:12:11 debian dnsmasq-dhcp[10689]: DHCP, sockets bound exclusively to i
Sep 27 11:12:11 debian dnsmasq[10689]: reading /etc/resolv.conf
Sep 27 11:12:11 debian dnsmasq[10689]: using nameserver 95.77.94.88#53
Sep 27 11:12:11 debian dnsmasq[10689]: using nameserver 78.96.7.88#53
Sep 27 11:12:11 debian dnsmasq[10689]: read /etc/hosts - 5 addresses
Sep 27 11:12:11 debian dnsmasq[10689]: read /var/lib/libvirt/dnsmasq/default.add
Sep 27 11:12:11 debian dnsmasq-dhcp[10689]: read /var/lib/libvirt/dnsmasq/defaul
Sep 27 13:40:37 debian systemd[1]: Stopping Virtualization daemon...
Sep 27 13:40:37 debian systemd[1]: Stopped Virtualization daemon.
alexandru@debian:~$ sudo systemctl start libvirtd
alexandru@debian:~$ sudo systemctl restart libvirtd
alexandru@debian:~$ sudo systemctl status libvirtd
● libvirtd.service - Virtualization daemon
 Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor preset:
 Active: active (running) since Wed 2017-09-27 13:41:00 EEST; 7s ago
 Docs: man:libvirtd(8)
 http://libvirt.org
 Main PID: 13481 (libvirtd)
 Tasks: 18 (limit: 4915)
 CGroup: /system.slice/libvirtd.service
 ├─10689 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/defaul
 ├─10690 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/defaul
 └─13481 /usr/sbin/libvirtd
Sep 27 13:41:00 debian systemd[1]: Starting Virtualization daemon...
Sep 27 13:41:00 debian systemd[1]: Started Virtualization daemon.
Sep 27 13:41:00 debian dnsmasq[10689]: read /etc/hosts - 5 addresses
Sep 27 13:41:00 debian dnsmasq[10689]: read /var/lib/libvirt/dnsmasq/default.add
Sep 27 13:41:00 debian dnsmasq-dhcp[10689]: read /var/lib/libvirt/dnsmasq/defaul
alexandru@debian:~$ sudo systemctl reload libvirtd
alexandru@debian:~$ sudo systemctl list libvirtd
Unknown operation list.
alexandru@debian:~$ sudo systemctl show libvirtd
Type=notify
Restart=on-failure
NotifyAccess=main
RestartUSec=100ms
TimeoutStartUSec=1min 30s
TimeoutStopUSec=1min 30s
RuntimeMaxUSec=infinity
WatchdogUSec=0
WatchdogTimestamp=Wed 2017-09-27 13:41:00 EEST
WatchdogTimestampMonotonic=14977927822
FailureAction=none
PermissionsStartOnly=no
RootDirectoryStartOnly=no
RemainAfterExit=no
GuessMainPID=yes
MainPID=13481
ControlPID=0
FileDescriptorStoreMax=0
NFileDescriptorStore=0
StatusErrno=0
Result=success
UID=4294967295
GID=4294967295
alexandru@debian:~$

Laboratory objective 8: Working with foreground and background processes

Ubuntu 16.04.3 LTS was used for this exercise.

First, we install the stress program used in the previous exercises and use it for this one.

sudo apt install stress

We will start the stress program with two cpu intensive workloads:

alexandru@desktop:~ stress --cpu 2
stress: info: [3313] dispatching hogs: 2 cpu, 0 io, 0 vm, 0 hdd

Now we will press the Ctrl+Z to pause the program and give us control of the terminal again. Hitting Ctrl+Z will give the following output:

^Z
[1]+ Stopped stress --cpu 2

Take care that this will suspend the program's activity, therefor you should think before using it, as the program you would like to stop could be running an important task. The stress program is stopped and thus send to the background, waiting... To bring it back to the foreground, you can use the "fg" command, or if you want the program to run in the background, you can just use the "bg" command. Here is an output example for the stress program:

alexandru@desktop:~ bg
[1]+ stress --cpu 2 &
alexandru@desktop:~ fg
stress --cpu 2
^Z
[1]+ Stopped stress --cpu 2

The first bg command sends the program to the background, and the fg command sends it to the foreground again. This will get control to the terminal, thus we will have to hit Ctrl+Z again, to be able to use the same terminal console.

To display the process status we will use the "ps" command. Used in its simple mode, it will show the processes that run from your current user.

alexandru@desktop:~ ps
 PID TTY TIME CMD
 2785 pts/1 00:00:00 bash
 3303 pts/1 00:00:00 bash
 3313 pts/1 00:00:00 stress
 3314 pts/1 00:00:34 stress
 3315 pts/1 00:00:34 stress
 3333 pts/1 00:00:00 ps

You will se the stress process listed, but you don't have information if it is running in the foreground/background or it is stopped. But, you know that it is stopped because you just used the Ctrl+Z command to stop it. Now, you can kill the process, by using the "kill" command, as you already know the PID. The kill command will send a signal to the application, which usually is:

1 or SIGHUP - which will terminate interactive programs and daemons and make them to re-read the configuration files

9 or SIGKILL - this will kill the process and exit without normal shutdown tasks

15 or SIGTERM - will exit the process but allows it to safely close opened files

Default is the 15 SIGTERM signal, if you don't specify any at the command line.

Remember that the kill command will only kill processes that are owned by the user that is running the command, exept, of course, for the root user.

alexandru@desktop:~ kill -s 9 3313
alexandru@desktop:~ ps
 PID TTY TIME CMD
 2785 pts/1 00:00:00 bash
 3303 pts/1 00:00:00 bash
 3338 pts/1 00:00:00 ps
[1]+ Killed stress --cpu 2

Thus, you have killed the "stress" process. Do another "fg" command and the output would be:

alexandru@desktop:~$ fg
bash: fg: current: no such job