Red Hat Enterprise Linux 3: System Administration Guide
Prev | Chapter 43. OProfile | Next

# 43.5. Analyzing the Data

Periodically, the OProfile daemon, `oprofiled` collects the samples and writes them to the `/var/lib/oprofile/samples/` directory. Before reading the data, make sure all data has been written to this directory by executing the following command as root:

    opcontrol --dump

Each sample file name is based on the name of the executable, with a closing curly bracket (`}`) replacing each forward slash (`/`). The file name ends with a hash mark (`#`), followed by the counter number used for that sample file. For example, the following file includes the sample data for the `/sbin/syslogd` executable collected with counter 0:

    }sbin}syslogd#0

The following tools are available to profile the sample data once it has been collected:

- `op_time`

- `oprofpp`

- `op_to_source`

- `op_merge`

Use these tools, along with the binaries profiled, to generate reports that can be further analyzed.

| Warning
| The executable being profiled must be used with these tools to analyze the data. If it must change after the data is collected, backup the executable used to create the samples as well as the sample files.

Samples for each executable are written to a single sample file. Samples from each dynamically linked library are also written to a single sample file. While OProfile is running, if the executable being monitored changes and a sample file for the executable exists, the existing sample file is automatically deleted. Thus, if the existing sample file is needed, it must be backed up, along with the executable used to create it before replacing the executable with a new version. Refer to Section 43.4 Saving Data for details on how to backup the sample file.

## 43.5.1. Using `op_time`

The `op_time` tool provides an overview of all the executables being profiled.

The following is part of an example output:

    581        0.2949  0.0000 /usr/bin/oprofiled
    966        0.4904  0.0000 /usr/sbin/cupsd
    1028       0.5218  0.0000 /usr/sbin/irqbalance
    1187       0.6026  0.0000 /bin/bash
    1480       0.7513  0.0000 /usr/bin/slocate
    2039       1.0351  0.0000 /usr/lib/rpm/rpmq
    6249       3.1722  0.0000 /usr/X11R6/bin/XFree86
    8842       4.4885  0.0000 /bin/sed
    31342     15.9103  0.0000 /usr/bin/gdmgreeter
    58283     29.5865  0.0000 /no-vmlinux
    82853     42.0591  0.0000 /usr/bin/perl

Each executable is listed on its own line. The first column is the number of samples recorded for the executable. The second column is the percentage of samples relative to the total number of samples. The third column is unused, and the fourth is the name of the executable.

Refer to the `op_time` man page for a list of available command line options such as the `-r` option used to sort the output from the executable with the largest number of samples to the one with the smallest number of samples. The `-c` option is also useful for specifying a counter number.

## 43.5.2. Using `oprofpp`

To retrieve more detailed information about a specific executable, use `oprofpp`:

    oprofpp <mode> <executable>

`<executable>` must be the full path to the executable to be analyzed. `<mode>` must be one of the following:

`-l`

  List sample data by symbols. For example, the following is part of the output from running the command `oprofpp -l /usr/X11R6/bin/XFree86`:

      vma      samples  %           symbol name
      ...
      08195d10 4        3.0303      miComputeCompositeClip
      080b9180 5        3.78788     Dispatch
      080cdce0 5        3.78788     FreeResource
      080ce4a0 5        3.78788     LegalNewID
      080ce640 5        3.78788     SecurityLookupIDByClass
      080dd470 9        6.81818     WaitForSomething
      080e1360 12       9.09091     StandardReadRequestFromClient
      ...

  The first column is the starting virtual memory address (vma). The second column is the number of samples for the symbol. The third column is the percentage of samples for this symbol relative to the overall samples for the executable, and the fourth column is the symbol name.

  To sort the output from the largest number of samples to the smallest (reverse order), use `-r` in conjunction with the `-l` option.

`-s `<symbol-name>``

  List sample data specific to a symbol name. For example, the following output is from the command `oprofpp -s StandardReadRequestFromClient /usr/X11R6/bin/XFree86`:

      vma      samples  %           symbol name
      080e1360 12       100         StandardReadRequestFromClient
      080e1360 1        8.33333
      080e137f 1        8.33333
      080e13bb 1        8.33333
      080e13f4 1        8.33333
      080e13fb 1        8.33333
      080e144a 1        8.33333
      080e15aa 1        8.33333
      080e1668 1        8.33333
      080e1803 1        8.33333
      080e1873 1        8.33333
      080e190a 2        16.6667

  The first line is a summary for the symbol/executable combination.

  The first column consists of the virtual memory addresses sampled. The second column is the number of samples for the memory address. The third column is the percentage of samples for the memory address relative to the total number of samples for the symbol.

`-L`

  List sample data by symbols with more details than `-l`. For example:

      vma      samples  %           symbol name
      08083630 2        1.51515     xf86Wakeup
       08083641 1        50
       080836a1 1        50
      080b8150 1        0.757576    Ones
       080b8179 1        100
      080b8fb0 2        1.51515     FlushClientCaches
       080b8fb9 1        50
       080b8fba 1        50
      ...

  The data is the same as the `-l` option except that for each symbol, each virtual memory address used is shown. For each virtual memory address, the number of samples and percentage of samples relative to the number of samples for the symbol is displayed.

`-g` `<file-name>`

  Generate output to a file in `gprof` format. If the generated file is named `gmon.out`, `gprof` can be used to further analyze the data. Refer to the `gprof` man page for details.

Other options to further restrict the data are as follows:

`-f` `<file-name>`

  Use the specified sample file `<file-name>`. By default, the sample file in `/var/lib/oprofile/samples/` is used. Use this option to specify a sample file from a previous session.

`-i` `<file-name>`

  Use `<file-name>` as the name of the executable for which to retrieve data.

`-d`

  Demangle C++ symbol names.

`-D`

  Demangle C++ symbol names, and simplify STL library demangled names.

`--counter` `<number>`

  Gather information from a specific counter. The default counter is 0 if not specified.

`-o`

  Display the line number in the source code for each sample. When the executable was compiled, GCC's `-g` option should have been used. Otherwise, this option can not display the line numbers. None of the Red Hat Enterprise Linux executables are compiled with this option by default.

      vma      samples  %      symbol name        linear info
      0806cbb0 0        0      _start             ../sysdeps/i386/elf/start.S:47

`-e` `<symbol-name>`

  Exclude the comma-separated list of symbols from the output.

`-k`

  Display an additional column containing the shared library. This option only produces results if the `--separate=library` option to `opcontrol` is specified when configuring OProfile and if the `--dump-gprof-file` option is not used in conjunction with this option.

`-t` `<format>`

  Display the output in a specific column order. This option can not be used with `-g`.

  Use the following letters to represent the columns:

  Letter | Description
  --- | ---
  `v` | Virtual memory address
  `s` | Number of samples
  `S` | Cumulative number of samples
  `p` | Percentage of samples relative to total number of samples for the executable
  `P` | Cumulative percentage of samples relative to total number of samples for the executable
  `q` | Percentage of samples relative to all executables sampled
  `Q` | Cumulative percentage of samples relative to all executables sampled
  `n` | Symbol name
  `l` | File name of source file and line number, including full path
  `L` | Base name of the source code file name and line number
  `i` | Name of the executable, including full path
  `I` | Base name of the executable
  `d` | Details of the sample
  `h` | Display column headers

  Table 43-4. Letters for Column Order

`--session` `<name>`

  Specify the full path to the session or a directory relative to the `/var/lib/oprofile/samples/` directory.

`-p` `<path-list>`

  Specify a comma-separated list of paths in which the executables to be analyzed are located.

## 43.5.3. Using `op_to_source`

The `op_to_source` tool tries to match the samples for particular instructions to the corresponding lines in the source code. The resulting files generated should have the samples for the lines at the left. It also puts in a comment at the beginning of each function listing the total samples for the function.

For this utility to work, the executable must be compiled with GCC's `-g` option. By default, Red Hat Enterprise Linux packages are not compiled with this option.

The general syntax for `op_to_source` is as follows:

    op_to_source --source-dir <src-dir> <executable>

The directory containing the source code and the executable to be analyzed must be specified. Refer to the `op_to_source` man page for a list of additional command line options.

## 43.5.4. Using `op_merge`

If multiple sample files exist for the exact same executable or library, the sample files can be merged for easier analysis.

For example, to merge files for the library `/usr/lib/library-1.2.3.so`, execute the following command as root:

    op_merge /usr/lib/library-1.2.3.so

The resulting file is `/var/lib/oprofile/samples/}usr}lib}library-1.2.3.so`.

To limit the samples merged to a specific counter, use the `-c` option followed by the counter number.

Prev | Home | Next
Saving Data | Up | Understanding `/dev/profile/`