Index of /~plonka/rrdtool_fadvise

	Name	Last modified	Size	Description
	Parent Directory		-
	rrdtool-1.0.49-fadvise.patch	2007-05-16 09:41	926
	rrdtool-1.2.23-fadvise.patch	2007-11-09 15:34	866

This is a patch to rrdtool that I posted to the rrd-developers mailing list on May 16, 2007. (See message text below.)
Tobi has integrated it into the rrdtool sources, and, as of Nov 13 2007, this patch is in rrdtool-1.2.24 and in the the rrdtool beta, both of which can be found here.

If you decide to apply the patch yourself, please note the rrdtool version numbers in the patch file names.
The 1.0.49 patch does not apply correctly to rrdtool-1.2.x, so use the version that matches your rrdtool source code.
If in doubt about the patch applying correctly, apply it manually and be sure to put the call to posix_fadvise(..., POSIX_FADV_RANDOM) after the call to fopen in rrd_open.c.

You may also be interested in the related fincore and fadvise commands.

On Wed, May 16, 2007 at 09:43:50AM -0500, Dave Plonka wrote:
>
> Hi Tobi,
>
> This is one of those serendipitous occasions -
> I think we can save you perhaps lots of time:
>
> Archit Gupta, Dale Carder, and I have just comleted a research project
> on RRD performance and scalability.  I beleive we may have the largest
> single system MRTG w/RRD - over 300,000 targets and RRD files updated
> every five minutes.
>
> On Wed, May 16, 2007 at 10:00:12AM +0200, Tobias Oetiker wrote:
> >
> > I am still gathering data ... but it seems all to come down to
> >
> > * file system cache pollution through other processes
> > * and the time you give the system to deal with dirty bufffers
> > * the block queuse size may also play a role.
> > * I think we could gain quite a lot by using fadvise in rrdtool to
> >   only keep the header portion of the file in cache.
> >   will try this later today ...
>
> Over the past month, we've done a posix_fadvise RANDOM patch and
> completely evaluated the perforance impacts.  It's really good
> (obviously).  What it does is get both the readahead and page faults
> under control, in Linux at least.
>
> For others observing performance issues, the pertinent things are:
>
>  * What operating system and version are you running?
>    (to determine initial file readahead and availablity of fadvise syscall)
>
>  * What is your hardware, including physical memory?
>    to determine CPU available and max buffer-cache size
>
> * What is your update interval (rrd step) and RRD file definitions (RRAs)?
>   (e.g. typical MRTG, or output of rrdtool info.)
>
>   This determines the page fault characteristics of RRD files
>   when buffer-cache is scarce.
>
> * What version of rrdtool?
>   (for applying patch)
>
> With that we can determine if it's possible to update a givent number
> of RRD files, or how to properly (re)size it.
>
> Dave