User Guides Linux Tutorials for Performance Monitoring.

With Most of student asking Online tutorials for linux .I thought of posting some Linux tutorials for Performance Monitoring enthusiastics.

These tutorials will come handy if you are a system administrator or student trying to get rid of slow behavior of your linux machine.

So here they go.

Index:

1) How to track Application causing Memory Leak using ps command? (Skill Level : Intermediate )

2) SAR utility to monitor cpu utilization. (Skill Level : Intermediate )

3) How to track port number of running application?

*****************************************************************

1) How to track Application causing Memory Leak ?

Normally Most of applications uses some part of memory and release it after their operation is over or Application is closed.

However some applications don't just give up memory and leads to memory being utilized heavily.

For RAM consumption simulation i have used a script ( source : linux - Write a bash shell script that consumes a constant amount of RAM for a user defined time - Stack Overflow)

#!/bin/bash

echo "Provide sleep time in the form of NUMBER[SUFFIX]"

echo "   SUFFIX may be 's' for seconds (default), 'm' for minutes,"

echo "   'h' for hours, or 'd' for days."

read -p "> " delay

echo "begin allocating memory..."

for index in $(seq 1000); do

	value=$(seq -w -s '' $index $(($index + 100000)))

	eval array$index=$value

done

echo "...end allocating memory"

echo "sleeping for $delay"

sleep $delay

saved the script with name /tmp/ramload.sh

As per the script owner it had consumed 570M to 575M physical memory* for the specified time period of 5 minutes.

Ran the command.

# ./ramload.sh

Provide sleep time in the form of NUMBER[SUFFIX]

SUFFIX may be 's' for seconds (default), 'm' for minutes,

'h' for hours, or 'd' for days.

> 5m

begin allocating memory...

You can track such applications using following ps command.

#ps aux --sort rss

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND

root 2 0.0 0.0 0 0 ? S< 01:54 0:00 [migration/0]

root 3 0.0 0.0 0 0 ? SN 01:54 0:00 [ksoftirqd/0]

root 4 0.0 0.0 0 0 ? S< 01:54 0:00 [watchdog/0]

root 5 0.0 0.0 0 0 ? S< 01:54 0:00 [migration/1]

root 4732 0.0 0.4 10732 4848 tty7 Ss+ 01:55 0:01 /usr/bin/Xorg :

root 4746 0.0 0.9 24444 9784 ? SN 01:55 0:00 /usr/bin/python

gdm 4756 0.0 1.5 30488 15540 ? Ss 01:55 0:00 /usr/libexec/gd

root 8900 55.9 2.7 32188 28584 pts/1 R+ 23:58 0:25 /bin/bash ./ramload.sh

Run the command after some time.

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND

root 4731 0.0 0.3 27392 3740 ? Sl May03 0:00 /usr/libexec/gdm-rh-security-token-helper

root 4318 0.0 0.4 13256 4692 ? S May03 0:00 python ./hpssd.py

root 4732 0.0 0.4 10732 4848 tty7 Ss+ May03 0:01 /usr/bin/Xorg :0 -br -audit 0 -auth /var/gdm/:0.Xauth -nolist

root 4746 0.0 0.9 24444 9784 ? SN May03 0:00 /usr/bin/python -tt /usr/sbin/yum-updatesd

gdm 4756 0.0 1.5 30488 15540 ? Ss May03 0:00 /usr/libexec/gdmgreeter

root 8900 69.5 5.5 60624 57276 pts/1 S+ May03 0:54 /bin/bash ./ramload.sh

You can observe the last entry which is

/bin/bash ./ramload.sh

Is a reason behind memory utilization.

(

Note:

1) Only few entries are displayed from output due to its size.

2) Normally ps aux --sort pmem is used for sorting the ps output using the memory but it has some bug .

see below link for more info

[SOLVED] Unable to understand ps output.

)

From above output we can say that

./ramload.sh

process is using the max memory.You need to fire the command at regular interval to check if the process is culprit behind memory crisis.

*******************************************************************************

2) SAR utility to monitor cpu utilization.

Downloaded Sar from below url.

http://sebastien.godard.pagesperso-orange.fr/download.html

I found it little difficult to install rpm(sysstat-10.0.0-1.i586.rpm) so i moved my attention to install sysstat-10.0.0.tar.gz using make install method.

# ls -ltr

total 328

-rw-r--r-- 1 root root 330646 May 3 13:09 sysstat-10.0.0(2).tar.gz

# gunzip sysstat-10.0.0\(2\).tar.gz

# ls -ltr

total 1796

-rw-r--r-- 1 root root 1832960 May 3 13:09 sysstat-10.0.0(2).tar

# tar -xvf sysstat-10.0.0\(2\).tar

# ls -ltr

total 1800

drwxr-xr-x 8 501 501 4096 Mar 14 01:16 sysstat-10.0.0

-rw-r--r-- 1 root root 1832960 May 3 13:09 sysstat-10.0.0(2).tar

The simplest way to compile this package is:

1. `cd' to the directory containing the package's source code

# cd sysstat-10.0.0

Now type

`./configure' to configure the package for your system.

#./configure

If you're

using `csh' on an old version of System V, you might need to type

`sh ./configure' instead to prevent `csh' from trying to execute

`configure' itself.

Running `configure' takes awhile. While running, it prints some

messages telling which features it is checking for.

2. Type `make' to compile the package.

#make

3. Optionally, type `make check' to run any self-tests that come with

the package.(Not very Important)

4. Type `make install' to install the programs and any data files and

documentation.

#make install

5. You can remove the program binaries and object files from the

source code directory by typing `make clean'. To also remove the

files that `configure' created (so you can compile the package for

a different kind of computer), type `make distclean'. There is

also a `make maintainer-clean' target, but that is intended mainly

for the package's developers. If you use it, you may have to get

all sorts of other programs in order to regenerate files that came

with the distribution.

(Source of information: INSTALL file that comes bundled with sar utility)

Install various dependencies from Distribution CD.(This is only required if above installation is failed due to dependencies. )

Everything You need to know about SAR is written in below article.

Lets explore more into SAR.

Let monitor the CPU utilization for some time .Gathering a cpu stat at a point makes no sense so continuous gathering has its own advantages .

Say in below command cpu is observered 10 times in an interval of 2 seconds.

# sar 2 10

Linux 2.6.18-92.el5 (OEL-5.2-32bit) 04/06/2011 _i686_ (1 CPU)

03:45:19 PM CPU %user %nice %system %iowait %steal %idle

03:45:21 PM all 0.00 0.00 0.51 0.00 0.00 99.49

03:45:23 PM all 0.00 0.00 0.51 0.00 0.00 99.49

03:45:25 PM all 0.00 0.00 0.51 0.00 0.00 99.49

03:45:27 PM all 0.00 0.00 1.01 0.00 0.00 98.99

03:45:29 PM all 0.00 0.00 1.01 0.00 0.00 98.99

03:45:31 PM all 0.00 0.00 0.00 0.00 0.00 100.00

03:45:33 PM all 0.00 0.00 1.01 0.00 0.00 98.99

03:45:35 PM all 0.00 0.00 0.00 0.00 0.00 100.00

03:45:37 PM all 0.00 0.00 1.01 0.00 0.00 98.99

03:45:39 PM all 0.00 0.00 1.01 0.00 0.00 98.99

Average: all 0.00 0.00 0.66 0.00 0.00 99.34

The most important row from above output is the one which displays average of cpu utilization(highlighted in bold).

The Most Important column from above output is the one which displays idle cpu(Since my machine is just a sar utility testing machine ,i m not getting any cpu utilization .In short most of the time my cpu is idle).

Now its time to generate some load on CPU and then examine the output of SAR.

created a simple scipt with name cpuload.sh .

#!/bin/bash

while : ; do

true

done

and executed it in background using .

# ./cpuload.sh &

Ran above sar command one more time and examined the output.

# sar 2 10

Linux 2.6.18-92.el5 (OEL-5.2-32bit) 04/06/2011 _i686_ (1 CPU)

04:21:31 PM CPU %user %nice %system %iowait %steal %idle

04:21:33 PM all 100.00 0.00 0.00 0.00 0.00 0.00

04:21:35 PM all 99.50 0.00 0.50 0.00 0.00 0.00

04:21:37 PM all 99.49 0.00 0.51 0.00 0.00 0.00

04:21:39 PM all 99.49 0.00 0.51 0.00 0.00 0.00

04:21:41 PM all 100.00 0.00 0.00 0.00 0.00 0.00

04:21:43 PM all 98.99 0.00 1.01 0.00 0.00 0.00

04:21:45 PM all 99.49 0.00 0.51 0.00 0.00 0.00

04:21:47 PM all 99.49 0.00 0.51 0.00 0.00 0.00

04:21:49 PM all 99.49 0.00 0.51 0.00 0.00 0.00

04:21:51 PM all 99.49 0.00 0.51 0.00 0.00 0.00

Average: all 99.55 0.00 0.45 0.00 0.00 0.00

Now its time to find the culprit behind the cpu load.

Below command displays top 10 cpu utilized processes.

# ps -auxf | sort -nr -k 3 | head -10

Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.7/FAQ

[B]root 4962 87.6 0.1 4476 964 pts/1 R 16:18 4:15 \_ /bin/bash ./cpuload.sh[/B]

root 309 0.5 0.0 0 0 ? S< 15:40 0:14 \_ [kjournald]

root 4663 0.2 0.1 1948 644 ? S 15:42 0:05 \_ hald-addon-storage: polling /dev/hdc

root 4844 0.1 0.8 27388 4140 ? Sl 15:42 0:03 /usr/libexec/gdm-rh-security-token-helper

root 4224 0.1 2.0 42336 10580 ? Ssl 15:42 0:03 /usr/bin/python -E /usr/sbin/setroubleshootd

root 1 0.1 0.1 2064 620 ? Ss 15:40 0:03 init [5]

xfs 4572 0.0 0.3 3928 1700 ? Ss 15:42 0:00 xfs -droppriv -daemon

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND

smmsp 4518 0.0 0.2 8052 1484 ? Ss 15:42 0:00 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue

rpc 4211 0.0 0.1 1808 608 ? Ss 15:42 0:00 portmap

Examine the entry in bold.

Now its time to terminate the script .

# ps -ef | grep cpuload

root 4962 4875 89 16:18 pts/1 00:04:55 /bin/bash ./cpuload.sh

root 4987 4875 0 16:23 pts/1 00:00:00 grep cpuload

Kill it using process id.

# kill -9 4962

CPU is back to normal again.

# sar 2 10

Linux 2.6.18-92.el5 (OEL-5.2-32bit) 04/06/2011 _i686_ (1 CPU)

04:24:13 PM CPU %user %nice %system %iowait %steal %idle

04:24:15 PM all 0.00 0.00 0.51 0.00 0.00 99.49

04:24:17 PM all 0.00 0.00 0.00 0.00 0.00 100.00

04:24:19 PM all 0.00 0.00 0.00 0.00 0.00 100.00

04:24:21 PM all 0.00 0.00 0.50 0.00 0.00 99.50

04:24:23 PM all 0.00 0.00 0.00 0.00 0.00 100.00

04:24:25 PM all 0.00 0.00 0.50 0.00 0.00 99.50

04:24:27 PM all 0.00 0.00 0.00 0.00 0.00 100.00

04:24:29 PM all 0.00 0.00 0.00 0.00 0.00 100.00

04:24:31 PM all 0.00 0.00 0.00 0.00 0.00 100.00

04:24:33 PM all 0.00 0.00 1.01 0.00 0.00 98.99

Average: all 0.00 0.00 0.25 0.00 0.00 99.75

*************************************************************************************

3)How to track port number of running application?

How to find the port of particular application?

Question : How do I find the port which is used by a particular application ?

Answer :

Consider Tomcat application is running on port 8080.

To check whether the Tomcat is running or not.

Initially you need to understand the output presented by

netstat -ntpl .

In above case Tomcat is using java as their process so i grep with java. Your application might be using different name.

#netstat -ntpl | grep java

tcp 0 0 ::ffff:127.0.0.1:8005 :::* LISTEN 6375/java

tcp 0 0 :::8009 :::* LISTEN 6375/java

tcp 0 0 :::8080 :::* LISTEN 6375/java

You should see at least one java process and you can use ps to identify if this is Tomcat.

# ps -ef | grep 6375

root 6375 1 0 May18 pts/2 00:01:06 /usr/java/jdk1.6.0_20/bin/java -Djava.util.logging.config.file=/install/apache-tomcat-5.5.29/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/install/apache-tomcat-5.5.29/common/endorsed -classpath /install/apache-tomcat-5.5.29/bin/bootstrap.jar -Dcatalina.base=/install/apache-tomcat-5.5.29 -Dcatalina.home=/install/apache-tomcat-5.5.29 -Djava.io.tmpdir=/install/apache-tomcat-5.5.29/temp org.apache.catalina.startup.Bootstrap start

root 9222 5091 0 18:29 pts/2 00:00:00 grep 6375

for killing the process just use.

kill -9

Will post more such tutorials as time goes....

Do reply ....

 

[altruistic666]

This is very informative and exhaustive. Thanks for summing it up.