Last modified

This is the CPU profiler we use at Google. There are three parts to using it: linking the library into an application, running the code, and analyzing the output.

On the off-chance that you should need to understand it, the CPU profiler data file format is documented separately, here.

Linking in the Library

To install the CPU profiler into your executable, add -lprofiler to the link-time step for your executable. (It's also probably possible to add in the profiler at run-time using LD_PRELOAD, e.g. % env LD_PRELOAD="/usr/lib/libprofiler.so" <binary>, but this isn't necessarily recommended.)

This does not turn on CPU profiling; it just inserts the code. For that reason, it's practical to just always link -lprofiler into a binary while developing; that's what we do at Google. (However, since any user can turn on the profiler by setting an environment variable, it's not necessarily recommended to install profiler-linked binaries into a production, running system.)

Running the Code

There are several alternatives to actually turn on CPU profiling for a given run of an executable:

1. Define the environment variable CPUPROFILE to the filename to dump the profile to. For instance, to profile /usr/local/bin/my_binary_compiled_with_libprofiler_so:

   % env CPUPROFILE=/tmp/mybin.prof /usr/local/bin/my_binary_compiled_with_libprofiler_so

2. In your code, bracket the code you want profiled in calls to ProfilerStart() and ProfilerStop(). (These functions are declared in <google/profiler.h>.) ProfilerStart() will take the profile-filename as an argument.

In Linux 2.6 and above, profiling works correctly with threads, automatically profiling all threads. In Linux 2.4, profiling only profiles the main thread (due to a kernel bug involving itimers and threads). Profiling works correctly with sub-processes: each child process gets its own profile with its own name (generated by combining CPUPROFILE with the child's process id).

For security reasons, CPU profiling will not write to a file -- and is thus not usable -- for setuid programs.

Modifying Runtime Behavior

You can more finely control the behavior of the CPU profiler via environment variables.

Analyzing the Output

pprof is the script used to analyze a profile. It has many output modes, both textual and graphical. Some give just raw numbers, much like the -pg output of gcc, and others show the data in the form of a dependency graph.

pprof requires perl5 to be installed to run. It also requires dot to be installed for any of the graphical output routines, and gv to be installed for --gv mode (described below).

Here are some ways to call pprof. These are described in more detail below.

% pprof /bin/ls ls.prof
                       Enters "interactive" mode
% pprof --text /bin/ls ls.prof
                       Outputs one line per procedure
% pprof --gv /bin/ls ls.prof
                       Displays annotated call-graph via 'gv'
% pprof --gv --focus=Mutex /bin/ls ls.prof
                       Restricts to code paths including a .*Mutex.* entry
% pprof --gv --focus=Mutex --ignore=string /bin/ls ls.prof
                       Code paths including Mutex but not string
% pprof --list=getdir /bin/ls ls.prof
                       (Per-line) annotated source listing for getdir()
% pprof --disasm=getdir /bin/ls ls.prof
                       (Per-PC) annotated disassembly for getdir()
% pprof --text localhost:1234
                       Outputs one line per procedure for localhost:1234
% pprof --callgrind /bin/ls ls.prof
                       Outputs the call information in callgrind format

Analyzing Text Output

Text mode has lines of output that look like this:

       14   2.1%  17.2%       58   8.7% std::_Rb_tree::find

Here is how to interpret the columns:

1. Number of profiling samples in this function
2. Percentage of profiling samples in this function
3. Percentage of profiling samples in the functions printed so far
4. Number of profiling samples in this function and its callees
5. Percentage of profiling samples in this function and its callees
6. Function name

Analyzing Callgrind Output

Use kcachegrind to analyze your callgrind output:

% pprof --callgrind /bin/ls ls.prof > ls.callgrind
% kcachegrind ls.callgrind

The cost is specified in 'hits', i.e. how many times a function appears in the recorded call stack information. The 'calls' from function a to b record how many times function b was found in the stack traces directly below function a.

Tip: if you use a debug build the output will include file and line number information and kcachegrind will show an annotated source code view.

Node Information

In the various graphical modes of pprof, the output is a call graph annotated with timing information, like so:

Each node represents a procedure. The directed edges indicate caller to callee relations. Each node is formatted as follows:

Class Name
Method Name
local (percentage)
of cumulative (percentage)

The last one or two lines contains the timing information. (The profiling is done via a sampling method, where by default we take 100 samples a second. Therefor one unit of time in the output corresponds to about 10 milliseconds of execution time.) The "local" time is the time spent executing the instructions directly contained in the procedure (and in any other procedures that were inlined into the procedure). The "cumulative" time is the sum of the "local" time and the time spent in any callees. If the cumulative time is the same as the local time, it is not printed.

For instance, the timing information for test_main_thread() indicates that 155 units (about 1.55 seconds) were spent executing the code in test_main_thread() and 200 units were spent while executing test_main_thread() and its callees such as snprintf().

The size of the node is proportional to the local count. The percentage displayed in the node corresponds to the count divided by the total run time of the program (that is, the cumulative count for main()).

Edge Information

An edge from one node to another indicates a caller to callee relationship. Each edge is labelled with the time spent by the callee on behalf of the caller. E.g, the edge from test_main_thread() to snprintf() indicates that of the 200 samples in test_main_thread(), 37 are because of calls to snprintf().

Note that test_main_thread() has an edge to vsnprintf(), even though test_main_thread() doesn't call that function directly. This is because the code was compiled with -O2; the profile reflects the optimized control flow.

Meta Information

The top of the display should contain some meta information like:

      /tmp/profiler2_unittest
      Total samples: 202
      Focusing on: 202
      Dropped nodes with <= 1 abs(samples)
      Dropped edges with <= 0 samples

This section contains the name of the program, and the total samples collected during the profiling run. If the --focus option is on (see the Focus section below), the legend also contains the number of samples being shown in the focused display. Furthermore, some unimportant nodes and edges are dropped to reduce clutter. The characteristics of the dropped nodes and edges are also displayed in the legend.

Focus and Ignore

You can ask pprof to generate a display focused on a particular piece of the program. You specify a regular expression. Any portion of the call-graph that is on a path which contains at least one node matching the regular expression is preserved. The rest of the call-graph is dropped on the floor. For example, you can focus on the vsnprintf() libc call in profiler2_unittest as follows:

% pprof --gv --focus=vsnprintf /tmp/profiler2_unittest test.prof

Similarly, you can supply the --ignore option to ignore samples that match a specified regular expression. E.g., if you are interested in everything except calls to snprintf(), you can say:

% pprof --gv --ignore=snprintf /tmp/profiler2_unittest test.prof

Interactive mode

By default -- if you don't specify any flags to the contrary -- pprof runs in interactive mode. At the (pprof) prompt, you can run many of the commands described above. You can type help for a list of what commands are available in interactive mode.

pprof Options

Output Type

Reporting Granularity

By default, pprof produces one entry per procedure. However you can use one of the following options to change the granularity of the output. The --files option seems to be particularly useless, and may be removed eventually.

Controlling the Call Graph Display

Some nodes and edges are dropped to reduce clutter in the output display. The following options control this effect:

The dropped edges and nodes account for some count mismatches in the display. For example, the cumulative count for snprintf() in the first diagram above was 41. However the local count (1) and the count along the outgoing edges (12+1+20+6) add up to only 40.

Caveats

• If the program exits because of a signal, the generated profile will be incomplete, and may perhaps be completely empty.
• The displayed graph may have disconnected regions because of the edge-dropping heuristics described above.
• If the program linked in a library that was not compiled with enough symbolic information, all samples associated with the library may be charged to the last symbol found in the program before the library. This will artificially inflate the count for that symbol.
• If you run the program on one machine, and profile it on another, and the shared libraries are different on the two machines, the profiling output may be confusing: samples that fall within shared libaries may be assigned to arbitrary procedures.
• If your program forks, the children will also be profiled (since they inherit the same CPUPROFILE setting). Each process is profiled separately; to distinguish the child profiles from the parent profile and from each other, all children will have their process-id appended to the CPUPROFILE name.
• Due to a hack we make to work around a possible gcc bug, your profiles may end up named strangely if the first character of your CPUPROFILE variable has ascii value greater than 127. This should be exceedingly rare, but if you need to use such a name, just set prepend ./ to your filename: CPUPROFILE=./Ägypten.