Debug and Tune Solaris in Realtime

Trying hard to figure out the exact faults or issues that pounced your
system's performance? A tough task for system administrators indeed! Sun
Microsystems has come up with a framework called 'DTrace,' in Solaris 10,
through which system admins can trace down the issues on a live system, without
taking it offline. Apart from this it also helps in understanding the behavior
of the operating system in realtime, which in turn helps system administrators
keep their systems up and healthy.

In addition, DTrace can be used by developers to get information on the list
of processes that are running, and also about creation and termination of the
processes. DTrace can also be used to create programs to dynamically monitor the
system using the DTrace 'D' programming language. As far as the working of
DTrace goes, it makes use of 'probes' which reside in the kernel of the
operating system. A probe is basically a location or an activity, or simply
putting it probes are sensors located in different parts of the kernel, which
are used by DTrace to collect necessary information as described by the system
administrator or developer, and display it. Probes originate from different
kernel modules called 'Providers'. After starting DTrace, each provider is given
an equal opportunity to publish his probes to the DTrace and then decide on the
probe to be enabled.

Getting started

Let's begin with a simple command which makes use of two probes: 'BEGIN' and
'END.'

# dtrace -n BEGIN -n END

After the command gets executed, the following result comes up:

dtrace: description 'BEGIN' matched 1 probe

dtrace: description 'END' matched 1 probe

CPU ID FUNCTION:NAME

0 1 :BEGIN

^C

1 2 :END

The 'BEGIN' probe gets fired each time you start a trace. Similarly when you
end the trace the 'END' probe gets fired. In the above scenario when we hit
'Enter' key to execute the command, the trace is started, DTrace found the two
probes and the BEGIN probe is fired. As soon as we press 'Ctrl' + C to terminate
the trace process, the END probe gets fired and DTrace displays the information
before terminating itself. Apart from the function name, which is displayed in
the result, you can also view the ID associated with each probe. You can also
view which CPU the probe has been executed on, in our case probe BEGIN has been
executed on first CPU whereas the END probe was executed on the second CPU.

The 'D'

As said earlier DTrace also provides you the option to create custom
programs with 'D' programming language. This language is almost similar to that
of 'C' programming language. In the D programming you can also make use of 'C'
pre-processors along with 'D' program by using the DTrace -C option. The D
compiler first loads up the description for the 'C' pre-processor from the
operating system before executing the D program. But if you have some 'C'
definition which does not exist in operating system or DTrace is unable to find
the description, then it is better to add the required libraries inside the C
source code during the C programming. The D program structure consists of three
parts: first the description of the probe. In D programming every clause is
started with the description of the probe. Suppose 'abc:xyz' is a probe
description, so in this 'abc' is the function which will be used to match the
related probe using DTrace and 'xyz' is the name. Second is the 'Condition',
this is taken into account when the probe mentioned is fired. And the third is
the action statement thatwill be performed. Technically the structure would look
like:

Description of Probe

/predicate/

{

Action statement

}

Let's see a simple 'Hello World' example of D programming, which is the best
to start with. For this write this simple program below in the vi-editor and
save it with extension '.d', say 'hello.d'.

BEGIN #probe description

{

trace("hello World"); #action statement

exit(0); #action statement

}

For executing the program write the following command on the terminal and you
get the result:

# dtrace —s hello.d

dtrace: script 'hello.d' matched 1 probe

CPU ID FUNCTION:NAME

0 1 :BEGIN hello world

The D programming provides you with some predefined macros which have '$' as
prefix such as '$egid' for effective user ID, '$gid' for real group ID, '$pid'
for process ID, etc. For understanding you can refer this to the arguments that
are passed to program in 'C' programming language. All the macros are converted
to some process ID or user ID except the $<0-9>+ and $target macros. Let us see
another example, this time it is little complex so as to give you a better view
of the D programming. In the following program we will have a probe description,
predicate followed by action statement. You will also see how to pass an
argument in the program. Write down the following code and save it as 'sys.d'.

syscall:::entry

/pid == $target/

{

@ = count();

}

Execute the programme by writing the following code on the terminal:

# dtrace —s sys.d -c ps

The output will be as follows:

dtrace: script 'sys.d' matched 231 PID TTY TIME CMD

3923 pts/3 0:00 dtrace

3924 pts/3 0:00 ps

3847 pts/3 0:00 sh

dtace: pid 3924 has exited

access 1

exece 1

fcntl 1

fsat 1

fxstat 1

getuid 1

ioctl 1

memcntl 1

rexit 1

getdents 2

getpid 2

Complete output of this program is omitted. The program displays all the
system calls by the 'ps' command made during execution. In the program above
'@target' is used to accept the target process, ie ps.