NAME

FlowScan - a system to analyze and report on cflowd flow files


DESCRIPTION

This document is the FlowScan User Manual $Revision: 1.20 $, $Date: 2000/10/19 18:43:27 $. It describes the installation and setup of FlowScan.

FlowScan is a system which scans the flow files written by cflowd and reports on what it finds. There are two reporting modules that are included. The first, CampusIO.pm produced the graphs at:

   http://wwwstats.net.wisc.edu

which show traffic in and out at a peering point. The second, SubNetIO.pm updates RRD files for each of the subnets that you specify (so that you can produce graphs of CampusIO by subnet).

The idea behind the distinct report modules is that other users will be able to write new reports that are either derived-classes from CampusIO, or altogether new ones. For instance, one may wish to write a report module called Abuse.pm which would send email when it detected potentially abusive things going on, like Denial-of-Service attacks and various scans.

FlowScan is freely-available under the GPL, the GNU General Public License.


Use the Mailing List

Please help me to help you. It is, unfortunately, not uncommon for one to have questions or problems while installing FlowScan. Please do not email about such things to my personal email address, but instead check the FlowScan mailing list archive, and/or join the FlowScan mailing list. Information about the FlowScan mailing lists can be found at:

   http://net.doit.wisc.edu/~plonka/FlowScan/#Mailing_Lists

By reading and/or participating in the list, you will be helping me to use my time effectively so that others will benefit from questions answered and issues raised.

The mailing lists' archives are available at:

   http://net.doit.wisc.edu/~plonka/list/flowscan

and:

   http://net.doit.wisc.edu/~plonka/list/flowscan-announce


Upgrading

If you have previously installed and properly configured FlowScan-1.002 or FlowScan-1.003, you need only perform a subset of the steps described below for an initial installation.

First time FlowScan users should skip to Initial Install Requirements, below.


Software Upgrade Requirements


Configuring FlowScan when Upgrading

Just the things listed here have changed regarding the FlowScan configuration.

Upgrading CampusIO Configuration Files
There is a new <kbd>UDPServices</kbd> option for CampusIO. It is not required, but if you wish to have CampusIO maintain RRD files for UDP services of your choosing, see the UDPServices example in the CampusIO.cf file in the cf sub-directory of the FlowScan distribution. Cut-and-paste from it as necessary into your $PREFIX/bin/CampusIO.cf file.

Secondly, the Napster_subnets.boulder has changed significantly since that provided with FlowScan-1.002. If you have FlowScan configured to measure Napster traffic, replace your old Napster_subnets.boulder with the one from the newer distribution:

   $ cp cf/Napster_subnets.boulder $PREFIX/bin/Napster_subnets.boulder


Generating Graphs after Upgrading

A number of new graphs and command-line options have been added to the graphs.mf template Makefile. Be sure to try the one supplied in the current distribution as described below in Supplied Graphs. You may wish to copy it to your <kbd>graphs</kbd> sub-directory.

While it is not required, I highly recommend installing RRGrapher if you want to produce other graphs. It is referenced below in Custom Graphs.


Done Upgrading

That should be it for upgrading!


Initial Install Requirements


Hardware Requirements


Software Requirements

The packages and perl modules required by FlowScan are numerous. Their presence or absence will be detected by FlowScan's configure script but you'll save yourself some frustration by getting ahead of the game by collecting and installing them first. Below, I've attempted to present them in a reasonable order in which to obtain, build, and install them.


Configuring FlowScan Prerequisites


Choose a User to Run cflowd and FlowScan

I recommend that you create a user just for the purpose of running these utilities so that all directory permissions and created file permissions are consistent. You may find this useful especially if you have multiple network engineers accessing the flows.

I suggest that the FlowScan --prefix directory be owned by an appropriate user and group, and that the permissions allow write by other members of the group. Also, turn on the set-group-id bit on the directory so that newly created files (such as the flow files and log file) will be owned by that group as well, e.g.:

   user$ chmod g+ws $PREFIX


Configuring Your Host

The current FlowScan graphing stuff likes your machine to have the 80/tcp service to be called http. Try running this command:

   $ perl -le "print scalar(getservbyport(80, 'tcp'))"

You can continue with the next step if this command prints http. However, if it prints some other value, such as www, then I suggest you modify your /etc/services file so that the line containing 80/tcp looks something like this:

   http             80/tcp    www www-http         #World Wide Web HTTP

Be sure to leave the old name such as www as an ``alias'', like I've shown here. This will reduce the risk of breaking existing applications which may refer to the service by that name. If you decide not to modify the service name in this way, FlowScan should still work, but you'll be on your own when it comes to producing graphs.


Configuring Your Ciscos

First and foremost, to get useful flow information from your Cisco, you'll need to enable flow-switching on the appropriate ingress interfaces using this interface-level configuration statement:

   ip route-cache flow

Also, I suggest that you export from your Cisco like this:

   ip flow-export version 5 peer-as
   ip flow-export destination 10.0.0.1 2055

Of course the IP address and port are determined by your cflowd.conf. To help ensure that flows are exported in a timely fashion, I suggest you also do this if your IOS version supports it:

   ip flow-cache timeout active 1

Newer IOS versions, e.g. 12.0(9), use this syntax:

   ip flow-cache active-timeout 1

unless you've specified something such as downward-compatible-config 11.2.

Lastly, in complicated environments, choosing which particular interfaces should have ip route-cache flow enabled is somewhat difficult. For FlowScan, one usually wants it enabled for any interface that is an ingress point for traffic that is from inside to outside or vice-versa. You probably don't want flow-switching enabled for interfaces that carry policy-routed traffic, such as that being redirected transparently to a web cache. Otherwise, FlowScan could count the same traffic twice because of multiple flows being reported for what was essentially the same traffic making multiple passes through a border router. E.g. user-to-webcache, webcache-to-outside world (on behalf of that user).


Configuring cflowd

This document does not attempt to explain cflowd. There is good documentation provided with that package.

As for the tweaks necessary to get cflowd to play well with FlowScan, hopefully, an example is worth a thousand words.

My cflowd.conf file looks like this:

   OPTIONS {
     LOGFACILITY:          local6
     TCPCOLLECTPORT:       2056
     TABLESOCKFILE:        /home/whomever/cflowd/etc/cflowdtable.socket
     FLOWDIR:              /var/local/flows
     FLOWFILELEN:          1000000
     NUMFLOWFILES:         10
     MINLOGMISSED:         300
   }
   CISCOEXPORTER {
     HOST:         10.0.0.10
     ADDRESSES:    { 10.42.42.10,
                   }
     CFDATAPORT:   2055
   #  COLLECT:      { flows }
   }
   COLLECTOR {
     HOST:         127.0.0.1
     AUTH:         none
   }

And I invoke the patched cflowd like this:

   user$ cflowd -s 300 -O 0 -m /path/to/cflowd.conf

Those options cause a flow file to be ``dropped'' every 5 minutes, skipping flows with an output interface of zero unless they are multicast flows. Once you have this working, your ready to continue.


Configuring FlowScan


Configure and Install

Do not use the same --prefix value as might for other packages!

I.e. don't use /usr/local or a similar directory in which other things are installed. This prefix should be the directory where the patched cflowd has been configured to write flow files.

A good way to avoid doing something dumb here is to not run FlowScan's configure nor make as root.

   user$ ./configure --help # note --with-... options

e.g.:

   user$ ./configure --prefix=/var/local/flows
   user$ make
   user$ make -n install
   user$ make install

By the way, in the above commands, all is OK if make says ``Nothing to be done for `target'''. As long as make completes without an error, all is OK.

Subsequently in this document the ``prefix'' directory will be referred to as the ``--prefix diretory'' or using the environment variable $PREFIX. FlowScan does not require or use this environment variable, it's just a documentation convention so you know to use the directory which you passed as with --prefix.


Create the Output Directory

The OutputDir is where the .rrd files and graphs will reside. As the chosen FlowScan user do:

  $ PREFIX=/var/local/flows
  $ mkdir -p $PREFIX/graphs

Then, when you edit the .cf files below, be sure to specify this using the OutputDir directive.


FlowScan Configuration Files

The FlowScan Package ships with sample configuration files in the cf sub-directory of the distribution. During initial configuration you will copy and sometimes modify these sample files to match your network environent and your purposes.

FlowScan looks for its configuration files in its bin directory - i.e. the directory in which the flowscan perl script and FlowScan report modules are installed. I don't really like this, but that's the way it is for now. Forgive me.

FlowScan currently uses two kinds of cofiguration files:

  1. Directive-style configuration files, with the .cf extension This format should be relatively self-explanatory based on the sample files referenced below. The directives are documented in comments within those sample configuration files.

    A number of the directorives have paths to directory entries as their values. One has a choice of configuring these as either relative or absolute paths. The samples configuration files ship with relative path specifications to minimize the changes a new user must make. However, in this configuration, it is imperitive that flowscan be run in the --prefix directory if these relative paths are used.

  2. "Boulder IO" format files, with the .boulder extension I've chosen Boulder IO's ``semantic free data interchange format'' to use for related projects, and since this is the format in which our subnet definitions were available, I continued to use it.

    If you're new to ``Boulder IO'', the examples referenced below should be sufficient. Remember that lines containing just = are record seperators.

    For complete information on this format, do:

       $ perldoc Boulder # or "perldoc bolder" if that fails
    

Here's a step-by-step guide to installing, reviewing, and editing the FlowScan configuration files:


Preserving "Old" Flow Files

If you'd like to have FlowScan save your flow files, make a sub-directory named saved in the directory where flowscan has been configured to look for flow files. This has been specified with the FlowFileGlob directive in flowscan.cf and is usually the same directory that is specified using the FLOWDIR directive in your cflowd.conf.

If you do this, flowscan will move each flow file to that saved sub-directory after processing it. (Otherwise it would simply remove them.) e.g.:

   $ mkdir $PREFIX/saved
   $ touch $PREFIX/saved/.gzip_lock

The .gzip_lock file created by this command is used as a lock file to ensure that only one cron job at a time.

Be sure to set up a crontab entry as is mentioned below in Final Setup. I.e. don't complain to the author if you're saving flows and your file-system fills up ;^).


Testing FlowScan

Once you have the patched cflowd running with the -s 300 option, and it has written at least one time-stamped flow file (i.e. other than flows.current), try this:

  $ cd /dir/containing/your/time-stamped/raw/flow/files
  $ flowscan

The output should appear as something like this:

   Loading "bin/Napster_subnets.boulder" ...
   Loading "bin/local_nets.boulder" ...
   2000/03/20 17:01:04 working on file flows.20000320_16:57:22...
   2000/03/20 17:07:38 flowscan-1.013 CampusIO: Cflow::find took 394 wallclock secs (350.03 usr +  0.52 sys = 350.55 CPU) for 23610455 flow file bytes, flow hit ratio: 254413/429281
   2000/03/20 17:07:41 flowscan-1.013 CampusIO: report took  3 wallclock secs ( 0.44 usr +  0.04 sys =  0.48 CPU)
   sleep 300...

At this point, the RRD files have been created and updated as the flow files are processed. If not, you should use the diagnostic warning and error messages or the perl debugger (perl -d flowscan) to determine what is wrong.

Look at the above output carefully. It is imperative that the number of seconds that Cflow::find took not usually approach nor exceed 300. If, as in the example above, your log messages indicate that it took more than 300 seconds, FlowScan will not be able to keep up with the flows being collected on this machine (if the given flow file is representative). If the total of usr + sys CPU seconds totals more than 300 seconds, than this machine is not even capable of running FlowScan fast enough, and you'll need to run it on a faster machine (or tweak the code, rewrite in C, or mess with process priorities using nice(1), etc.)


Final Setup

Once you feel that flowscan is working correctly, you can set it (and cflowd) to start up at system boot time. Sample rc scripts for Solaris and Linux are supplied in the rc sub-directory of this distribution. You may have to edit these scripts depending on your ps(1) flavor and where various commands have been installed on your system.

Also, if you're saving your flow files, you should set up crontab entries to handle the ``old'' flows. I use one crontab entry to gzip(1) recently processed files, and another to delete the files older than a given number of hours. The ``right'' number of hours is a function of your file-system size and the rate of flows being exported/collected. See the example/crontab file.


Generating Graphs


Supplied Graphs

To generate graphs, try the graphs.mf Makefile:

  $ cp graphs.mf $PREFIX/graphs/Makefile
  $ cd $PREFIX/graphs
  $ make

This should produce the ``Campus I/O by IP Protocol'' and ``Well Known Services'' graphs in PNG files. GIF files may be produced using the filetype option mentioned below.

If this command fails to produce those graphs, it is likely that some of the requisite .rrd files are missing, i.e. they have not yet been created by FlowScan, such as http_dst.rrd. If this is the case, it is probably because you skipped the configuration of /etc/services in Configuring Your Host. Stop flowscan, rename your www_*.rrd files to http_*.rrd, modify /etc/services, and restart <kbd>flowscan</kbd>.

Alternatively, you may copy and customize the graphs.mf Makefile to remove references to the missing or misnamed .rrd files for those targets. Also, you could produce your graphs using a graphing tool such as RRGrapher mentioned below in Custom Graphs.

Note that the graphs.mf template Makefile has options to specify such things as the range of time, graph height and width, and output file type. Usage:

   make -f graphs.mf [filetype=<png|gif>] [width=x] [height=y] [ioheight=y+n] [hours=h]

as in:

   $ make -f graphs.mf filetype=gif height=400 hours=24 io_services_bits.gif


Custom Graphs

Creation of other graphs will require the use of a tool such as RRGrapher or knowledge of RRDTOOL. RRGrapher, my Graph Construction Set for RRDTOOL is available at:

   http://net.doit.wisc.edu/~plonka/RRGrapher/

For other custom graphs, if you use the supplied graphs.mf Makefile, you can use the examples there in to see how to build ``Campus I/O by Network'' and ``AS to AS'' graphs. The examples use UW-Madison network numbers, names of with which we peer and such, so it will be non-trivial for you to customize them, but at least there's an example.

Currently, RRD files for the configured ASPairs contain a : in the file name. This is apparently a no-no with RRDTOOL since, although it allows you create files with these names, it doesn't let you graphs using them because of how the API uses : to seperate arguments.

For the time being, if you want to graph AS information, you must manually create symbolic links in your graphs sub-dir. i.e.

   $ cd graphs
   $ ln -s 0:42.rrd Us2Them.rrd
   $ ln -s 42:0.rrd Them2Us.rrd

A reminder for me to fix this is in the TODO list.


Future Directions for Graphs

The current Makefile-based graphing, while coherent, is cumbersome at best. I find that the verbosity and complexity of adding new graph targets to the Makefile makes my brain hurt.

Other RRDTOOL front-ends that produce graphs should be able to work with FlowScan-generated .rrd files, so there's hope.


Copyright and Disclaimer

Note that this document is provided `as is'. The information in it is not warranted to be correct. Use it at your own risk.

   Copyright (c) 2000 Dave Plonka <plonka@doit.wisc.edu>.
   All rights reserved.

This document may be reproduced and distributed in its entirety (including this authorship, copyright, and permission notice), provided that no charge is made for the document itself.