Linux System Resource & Performance Monitoring

Linux System Resource & Performance Monitoring:

Monitoring the Hard Disk Space

Use a simple command like:

$ df -h

This results in the output:

Filesystem                Size          Used         Avail     Use%       Mounted on

/dev/sda1                 22G          5.0G          16G      24%         /

/dev/sda2                 34G           23G          9.1G     72%         /home

This shows there are two partitions (1 & 2) of the hard disk sda, which are currently at 24% and 72% utilization. The total size is shown in gigabytes (G). How much is used and balance available is shown as well. However, checking each hard disk to see the percentage used can be a big drag. It is better that the system checks the disks and informs you by email if there is a potential danger. Bash scripts may be written for this and run at specific times as a cron job.

For the GUI, there is a graphical tool called ‘Baobab’ for checking the disk usage. It shows how a disk is being used and displays the information in the form of either multicolored concentric rings or boxes.

 

Monitoring Memory Usage

RAM or memory is used to run the current application. Under Linux, there are a number of ways you can check the used memory space -- both in static and dynamic conditions.

For a static snapshot of the memory, use ‘free -m’ which results in the output:

$ free -m                                   total           used       free     shared    buffers     cached

Mem:                          1998           1896       101          0         59          605

-/+ buffers/cache:       1231            766

Swap:                          290             77         213

Here, the total amount of RAM is depicted in megabytes (MB), along with cache and swap. A somewhat more detailed output can be obtained by the command ‘vmstat’:

root@gateway [~]# vmstat
procs   -----------memory-------------       ---swap--   -----io----    --system--  -----cpu------
 r    b   swpd     free        buff  cache       si       so       bi    bo      in     cs    us  sy  id  wa  st
 1   0      0       767932        0        0        0        0       10     3       0     1      2   0   97   0   0
root@gateway [~]#

However, if a dynamic situation of what is happening to the memory is to be examined, you have to use ‘top’ or ‘htop’. Both will give you a picture of which process is using what amount of memory and the picture will be updated periodically. Both ‘top’ and ‘htop’ will also show the CPU utilization, tasks running and their PID. Whereas ‘top’ has a purely numerical display, ‘htop’ is somewhat more colorful and has a semi-graphic look. There is also a list of command menus at the bottom for set up and specific operations.

root@gateway [~]# top

top - 01:04:18 up 81 days, 11:05,  1 user,  load average: 0.08, 0.28, 0.33
Tasks:  47 total,   1 running,  45 sleeping,   0 stopped,   1 zombie
Cpu(s):  2.4%us,  0.4%sy,  0.0%ni, 96.7%id,  0.5%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1048576k total,   261740k used,   786836k free,        0k buffers
Swap:            0k total,            0k used,            0k free,        0k cached

  PID    USER       PR  NI  VIRT  RES  SHR S  %CPU   %MEM    TIME+  COMMAND                                
      1   root         15   0  10372  736  624 S   0.0       0.1        1:41.86     init                                   
 5407   root         18   0  12424  756  544 S   0.0       0.1        0:13.71    dovecot                                
 5408   root         15   0  19068 1144  892 S  0.0       0.1        0:12.09    dovecot-auth                           
 5416   dovecot   15   0  38480 2868 2008 S  0.0       0.3        0:10.80    pop3-login                             
 5417   dovecot   15   0  38468 2880 2008 S  0.0       0.3        0:49.31    pop3-login                             
 5418   dovecot   16   0  38336 2700 2020 S  0.0       0.3        0:01.15    imap-login                             
 5419   dovecot   15   0  38484 2856 2020 S  0.0       0.3        0:04.69    imap-login                             
 9745   root        18   0  71548  22m 1400 S  0.0       2.2        0:01.39    lfd                                    
11501  root        15   0  160m  67m 2824 S   0.0       6.6        1:32.51   spamd                                  
23935  firewall   18   0  15276 1180  980 S   0.0        0.1        0:00.00   imap                                   
23948  mailnull  15   0  64292 3300 2620 S   0.0       0.3        0:05.62   exim                                   
23993  root       15   0  141m  49m 2760 S   0.0       4.8         1:00.87   spamd                                  
24477  root       18   0  37480 6464 1372 S   0.0       0.6        0:04.17   queueprocd                             
24494  root       18   0  44524 8028 2200 S  0.0        0.8        1:20.86   tailwatchd                             
24526  root       19   0  92984  14m 1820 S  0.0       1.4         0:00.00   cpdavd                                 
24536  root       33  18 23892 2556  680 S   0.0       0.2         0:02.09   cpanellogd                             
24543  root       18   0  87692  11m 1400 S  0.0       1.1         0:33.87   cpsrvd-ssl                             
25952  named    22  0 349m 8052 2076 S    0.0       0.8        20:17.42   named                                  
26374  root       15  -4 12788  752  440 S    0.0       0.1         0:00.00   udevd                                  
28031  root       17   0 48696 8232 2380 S   0.0       0.8         0:00.07   leechprotect                           
28038  root       18   0 71992 2172  132 S   0.0       0.2         0:00.00   httpd                                  
28524  root       18   0 90944 3304 2584 S  0.0       0.3         0:00.01   sshd

For a graphical display of how the memory is being utilized, the Gnome System Monitor gives a detailed picture. There are other system monitors available under various window managers in Linux.

 

What is Your CPU Doing?

You may have a single, a dual core, or a quad core CPU in your system. To see what each CPU is doing or how two CPUs are sharing the load, you have to use ‘top’ or ‘htop’. These command line applications show the percentage of each CPU being utilized. You can also see process statistics, memory utilization, uptime, load average, CPU status, process counts, and memory and swap space utilization statistics.

Similar output statistics may be seen by using command line tools such as the ‘mpstat’, which is part of a group package called ‘sysstat’. You may have to install ‘sysstat’ in your system, since it may not be installed by default. Once installed, you can monitor a variety of parameters, for example compare the CPU utilization of an SMP system or multi-processor system.

Finding out if any specific process is hogging the CPU needs a little more command line instruction such as:

$ ps -eo pcpu,pid,user,args | sort -r -k1 | less

OR

$ ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10

Similar output can be obtained by using the command ‘iostat’ as root:

root@gateway [~]# iostat -xtc 5 3
Linux 2.6.18-028stab094.3 (gateway.firewall.cx)         01/11/2012

Time: 01:13:15 AM
avg-cpu:  %user   %nice   %system  %iowait  %steal   %idle
                  2.38    0.01     0.43          0.46      0.00      96.72

Time: 01:13:20 AM
avg-cpu:  %user   %nice   %system  %iowait  %steal   %idle
                  3.89    0.00     0.26          0.09      0.00      95.77

Time: 01:13:25 AM
avg-cpu:  %user   %nice   %system  %iowait  %steal   %idle
                  0.31    0.00    0.15           1.07     0.00       98.47

This will show three outputs every five seconds and show the information since the last reboot.

CPU usage under GUI is very well depicted by the Gnome System Monitor and other system monitoring applications. These are also useful for monitoring remote servers. Detailed memory maps can be accessed, signals can be sent and processes controlled remotely.

Gnome-System-Monitor

What’s Cooking?

How do you know what processes are currently running in your Linux system? There are innumerable ways of getting to see this information. The handiest applications are the old faithfuls - ‘top’ and ‘htop’. They will give a real-time image of what is going on under the hood. However, if you prefer a more static view, use ‘ps’. To see all processes try ‘ps -A’ or ‘ps -e’:

root@gateway [~]# ps -e
PID TTY          TIME CMD
    1 ?          00:01:41 init
 3201 ?        00:00:00 leechprotect
 3208 ?        00:00:00 httpd
 3360 ?        00:00:00 httpd
 3490 ?        00:00:00 httpd
 3530 ?        00:00:00 httpd
 3532 ?        00:00:00 httpd
 3533 ?        00:00:00 httpd
 3535 ?        00:00:00 httpd
 3575 ?        00:00:00 httpd
 3576 ?        00:00:00 httpd
 3631 ?        00:00:00 imap
 3694 ?        00:00:00 httpd
 3705 ?        00:00:00 httpd
 3770 ?        00:00:00 imap
 3774 pts/0    00:00:00 ps
 5407 ?        00:00:13 dovecot
 5408 ?        00:00:12 dovecot-auth
 5416 ?        00:00:10 pop3-login
 5417 ?        00:00:49 pop3-login
 5418 ?        00:00:01 imap-login
 5419 ?        00:00:04 imap-login
 9745 ?        00:00:01 lfd
11501 ?        00:01:35 spamd
23948 ?        00:00:05 exim
23993 ?        00:01:00 spamd
24477 ?        00:00:04 queueprocd
24494 ?        00:01:20 tailwatchd
24526 ?        00:00:00 cpdavd
24536 ?        00:00:02 cpanellogd
24543 ?        00:00:33 cpsrvd-ssl
25952 ?        00:20:17 named
26374 ?        00:00:00 udevd
28524 ?        00:00:00 sshd
28531 pts/0    00:00:00 bash
29834 ?        00:00:00 sshd
30426 ?        00:11:27 syslogd
30429 ?        00:00:00 klogd
30473 ?        00:00:00 xinetd
30485 ?        00:00:00 mysqld_safe
30549 ?        1-15:07:28 mysqld
32158 ?        00:06:29 httpd
32166 ?        00:12:39 pure-ftpd
32168 ?        00:07:12 pure-authd
32181 ?        00:01:06 crond
32368 ?        00:00:00 saslauthd
32373 ?        00:00:00 saslauthd
PS is an extremely powerful and versatile command, and you can learn more by ‘ps --h’:
root@gateway [~]# ps --h********* simple selection *********  ********* selection by list *********
-A all processes                                   -C by command name
-N negate selection                              -G by real group ID (supports names)
-a all w/ tty except session leaders        -U by real user ID (supports names)
-d all except session leaders                  -g by session OR by effective group name
-e all processes                                    -p by process ID
T  all processes on this terminal             -s processes in the sessions given
a  all w/ tty, including other users           -t by tty
g  OBSOLETE -- DO NOT USE                -u by effective user ID (supports names)
r  only running processes                      U  processes for specified users
x  processes w/o controlling ttys            t  by tty
*********** output format **********  *********** long options ***********
-o,o user-defined   -f full                        --Group --User --pid --cols --ppid
-j,j job control       s  signal                    --group --user --sid --rows --info
-O,O preloaded    -o  v  virtual memory  --cumulative --format --deselect
-l,l long                u  user-oriented         --sort --tty --forest --version
-F   extra full        X  registers                --heading --no-heading --context
                    ********* misc options *********
-V,V  show version        L  list format codes        f  ASCII art forest
-m,m,-L,-T,H  threads   S  children in sum         -y change -l format
-M,Z  security data       c  true command name  -c scheduling class
-w,w  wide output         n  numeric WCHAN,UID  -H process hierarchy

Linux file permissions

The first character can be any of these:

d = directory
- = regular file
l = symbolic link
s = Unix domain socket
p = named pipe
c = character device file
b = block device file

The characters are pretty easy to remember.

r = read permission
w = write permission
x = execute permission
- = no permission

As a summary, have a look at this quick reference for setting file permissions in symbolic mode:

Which user?
u	user/owner
g	group
o	other
a	all
What to do?
+	add this permission
-	remove this permission
=	set exactly this permission
Which permissions?
r	read
w	write
x	execute

How to set file permissions - numeric mode: 

4 = read (r)
2 = write (w)
1 = execute (x)
0 = no permission (-)

owner: rwx = 4 + 2 + 1 = 7

group: r-x = 4 + 0 + 1 = 5

everyone: --x = 0 + 0 + 1 = 1

Which number?
0	---
1	--x
2	-w-
3	-wx
4	r--
5	r-x
6	rw-
7	rwx

Let's start from scratch, analysing the information in the previous screenshot.

What is the "umask"

How is the file permissions are set in the first place?
When you create a new file the shell variable umask determinds the default permissions set for the new file.

We can see the default permission if we use the following command:

$ umask
022

umask is specified as a three digit octal value that represents the new files permissions. The first digit is for the owner permissions the second digit is for the grouppermissions and the third digit is for world permissions.

umask represent what is not allowed for the new file by default. If we examine our default permission for the owner we see he has zero assigned to him. This means he is allowed to do everything since nothing is forbidden. 


The Group permissions has the number 2 assigned to it. This means that the Write permission is not set by default for the group permission. How do we calculate this? Read has the value of 4, Write the value of 2, and execute has the value of 1.


The same goes with the default permission for the world. Here the owner will have Read and Write Permission to the newly created file, the group and world permissions would be Read. Why the executable bit vanished?

Why is this? where went the value 1 for the executable? By default a new file is not given an executable permission. This is done to distinguish between regular files to programs (binary files) and shell scripts.

For directories the executable permission is set since with no executable set we cannot cd into the directory.

How do you set the value of umask?

$umask 027

This will set all new files created by you to default permission of read, and write for the owner (since nothing is forbidden and this is a file). Read for the group and no permission at all for world permission. The mathematics for a file is 666-026=640.

If this was a Directory newly created by us, than the permissions for the owener were Read, Write, and Execute. Permissions for the Group were Read and Execute, and no permissions at all for the world.

Permission encoding for umask
Octal Permission
0 rwx
1 rw-
2 r-x
3 r--
4 -wx
5 -w-
6 --x
7 ---


Group permissions

When you work on a project with other system users, you may need to grant access to your team mates to modify the files on the team project directory. You need also to block access from other (world) users or restrice them in a way they could not harm your the teams work.



If the setgid is set, BSD rule apply's if the bit is not set, the System V rule apply. (We set the setgid with chmod g+s and remove it with chmod s-g as you will see later in this editorial).


The Sticky Bit

If a user has write permission on a directory he can rename and remove files on that directory even if those files not belong to him. How can we prevent this? The owner of a directory can set the directory's "sticky bit", octal value 1000, which will give the rename and remove permissions of any file in that directory to the fileowner, the directory owner, and the superuser (in this order).

If you examine your / directory using the command ls -l, 


you will notice your /tmp directory has the sticky bit on.

drwxrwxrwt 4 root root 50176 Apr 20 04:50 tmp/

This allow different users to use this directory for temporary files, when only the file owner can delete his files (the directory owner for /tmp is the root user and he can also delete files). This prevents other users from deleting temporary files created by others and harming their work.