Application Buffer-Cache Management for Performance:
Running the World's Largest MRTG

David Plonka, Archit Gupta, and Dale Carder

{plonka,archit}@cs.wisc.edu, dwcarder@doit.wisc.edu

LISA '07, November 14, 2007

Outline

Background on MRTG and RRDTool

Problem & Motivation

Investigative Tools & Methodology

Approach 1: Application-level Buffering

Approach 2: Application Advice

Scalability

Analysis: Model & Simulation

Summary and Contributions

“Questions?”

Daily, 5 min averages

Weekly, 30 min ave.

Monthly, 2 hour ave.

Yearly, 1day averages

What is MRTG?

Our Network

# MRTG daemon

readConfiguration();

do {

# 1. POLL TARGETS via SNMP

readTargets();

# 2. WRITE TARGETS to RRDs

foreach my $target (@targets) {

RRDs::update(...);

}

sleep(...); # balance of 5 mins
} while (1); # forever

What is RRDTool?

What's the Problem?

What's the Problem?

What's the Problem?

RRDCache: App-level Buffering

RRDCache Performance: Good!

Methodology & Tools

sar

System activity reports (CPU, disk, etc.)

Can visualize data with new ksar tool

fincore

A user command that exposes the cache “footprint” of a set of files; shows which file blocks are in core

fadvise

A user command that can forcibly evict a file's pages from buffer-cache

Invaluable for controlled experimentation

160K files ×

25 pages ×

4KB =

~16GB!

160K files ×

4 pages ×

4KB =

~3GB!

Simulation: Hot Pages

What's the Solution?

fadvise(RANDOM);

fadvise(RANDOM): Before & After

Analytical Model for Average RRD update Page Fault Rate

fadvise(RANDOM) Before & After

MRTG Performance: Before

MRTG Performance: After. Great!

System CPU: After

Scalability:fadvise(RANDOM)

Scalability:fadvise(RANDOM)

Scalability: RRDCache +fadvise

Scalability: RRDCache +fadvise

Contributions

A method and user tools to examine buffer-cache and readahead behaviors:

fincore

fadvise

An analytical model and simulation of page fault behavior for RRD files:

Useful to size memory and to determine system capacity

RRDTool performance optimizations:

Application-level buffering: RRDCache

Application advice: fadvise(RANDOM)

Result: approx. triples system capacity

Thanks!

Thanks to:

Hideko Mills

Robert Plankers

Kevin Kettner II

Michael Swift

Tobi Oetiker

Released rrdtool-1.2.24 yesterday (Nov 13, 2007) with our fadvise(RANDOM)patch (Patches available for 1.0.49 and 1.2.23: See “Conclusions” in paper.)

“Questions?”

2006-07-19T08:51:23