System administrators do a lot of operating system configuration. In systems such as Unix this is done almost exclusively by editing plain text files which determine the behavior of the system. Any SA who has been tasked with duplicating all the tweaks done to a mature system knows that it's often impossible to remember all the changes one has made to a system since the base OS install. Additionally, in many environments a team of SAs share the responsibility of system maintainance. Certainly every SA in such an environment has been surprised at one time or another by an unexpected change done by a coworker. System or processes which allow the participants to track these sorts of configuration changes are not new - sometimes they are loosely referred to as "configuration management" or, usually in software developement and maintenance, simply "revision control".
A revision control system, used to track modification of such SA configuration files could provide (1) a "documentation" mechanism through which SAs explain why a change was made, (2) a "locking" mechanism so that two or more SAs don't overwrite each others changes by simultaneously editing the same file, and (3) an audit-trail or historical record of what has been done and the ability to retrieve an earlier configuration if necessary. The popular utility RCS, is the basis for the system that I will describe. (For a discussion of differences between SCCS and RCS see the sidebar RCS vs. SCCS.)
RCS, the Revision Control System, manages file revisions. It can be used to maintain all plain text files. In a Unix environment, these files often include C language source files, shell scripts, and configuration files.
RCS can enable SAs to:
Some Unix systems (e.g. HP-UX, RedHat Linux, Digital Unix) supply a version of RCS with the system. You should be sure that you are using a fairly current version such as 5.x or greater.
As of this writing, the latest version of RCS is 5.7, packaged by the name rcs-5.7.tar.gz. It is available from many sites via anonymous ftp including ftp://ftp.cs.purdue.edu/pub/RCS/ and ftp://prep.ai.mit.edu/pub/gnu/.
Installation is well documented. The only unusual bit is that the RCS author doesn't allow you to perform the build as root. The following is an example of how you might extract, build and install RCS. If your system has a nobody user, you can substitute that for user below otherwise use some other non-root account such as your own. (You should refer to the README and INSTALL files supplied in the distribution to verify that this is appropriate for your situation.)
# mkdir -p /usr/local/src # cd /usr/local/src # mv /tmp/rcs-5.7.tar.gz . # gunzip -c ./rcs-5.7.tar.gz |tar xf - # chown -R user:group rcs-5.7 # su - user ... $ cd /usr/local/src/rcs-5.7 $ rm -f config.cache $ ./configure ... $ make ... $ exit # back to root # make install clean ...At this point, the RCS commands and man pages are installed. You may also wish to obtain and install the find_revisions script, discussed in the Reporting section below. This RCS reporting tool is available at http://net.doit.wisc.edu/~plonka/find_revisions/.
There are a few prerequisites to take care of before SAs begin using RCS as root.
In applications other than system administration, RCS is most often used by "normal", non-root users. RCS uses the users login name (from the LOGNAME or USER environment variable) to uniquely identify the user when a lock is obtained (using the RCS co command) and in maintaining the RCS log information on check-in (using the ci command. Unfortunately, SAs often must do their machine configuration work as root. On most systems root is essentially an anonymous account since it doesn't identify which individual is currently operating as root. When using RCS in an environment with more than one SA, this is unacceptable.
A solution I've found for this problem is to create personal "root" accounts. That is, create multiple accounts with the user id of 0, each with a unique name identifying the SA who uses it. Since RCS uses the LOGNAME to identify the "locker", even though multiple accounts may have uid 0, RCS will be able to differentiate amongst them. If you will have multiple SAs using RCS to perform maintenance tasks on a given machine, see the sidebar titled Managing Personal root Accounts.
One caveat of which SAs should be aware when using RCS as root is that text editors as vi will sometimes allow one to force the writing of a read-only file. This can be used to side-step RCS' locking mechanism, and could result in file modifications being lost. In practice, this need not be much of a problem since vi, for instance, will warn that the file is read-only - the user must use :w! to force the write. It should be sufficient just to remind users that they still must "play along" with RCS for it to work correctly. As as often the case, those with root privilege should not abuse it to work around the intended process.
Any plain text file can be maintained using RCS. Unix configuration files have formats which allow varying amounts of freedom. For instance, shell scripts such as /etc/profile allow the introduction of comments by beginning the line with the # character. This allows one to paste a comment reminding SAs that the file is being maintained using RCS. For example:
# $Id$ # # # ### # # ###### # # ### # # # # # ### ####### ##### # # # # # # # # # # ### # # ###### # ### # THIS FILE IS BEING MAINTAINED USING RCS! # PLEASE USE the RCS co(1) and ci(1) commands to maintain it.Other files, such as /etc/passwd, have a rigid format and can not contain such reminder comments. There may be other requirements. For instance, the /etc/passwd and /etc/shadow are very special cases. These files should always be writable because the are sometimes maintained using such commands as vipw, chsh, chfn, HP-UX' sam, AIX' smit, Solaris' admintool. So, when using RCS with these files the SA would leave them locked-out at all times and periodically rcsdiff them to discover changes. When one wishes to "checkpoint" these files, he would perform check-in with ci -l. Be sure to keep these configuration file formats and other issues in mind when using RCS to maintain critical system configuration files.
Table 1 is a list of some common configuration files which usually reside on the root file-system that you would maintain in RCS.
$ su - rootyou Password: # id -u 0 # echo $LOGNAME rootyou #
# cd /etc # mkdir RCS # ci -u aliases RCS/aliases,v <-- aliases enter description, terminated with single '.' or end of file: NOTE: This is NOT the log message! >> sendmail aliases file >> ^D initial revision: 1.1 done #
# co -l aliases RCS/aliases,v --> aliases revision 1.1 (locked) done # vi aliases
# rcsdiff aliases =================================================================== RCS file: RCS/aliases,v retrieving revision 1.1 diff -r1.1 aliases 38a39,41 > > # folks who want to be notified about RCS revisions on the root file-system > root-revisions: you@your.org # ci -u aliases RCS/aliases,v <-- aliases new revision: 1.2; previous revision: 1.1 enter log message, terminated with single '.' or end of file: >> added "root-revisions" which is an e-mail alias to which the output of >> the "find_revisions" script is e-mailed as specified in root's crontab >> ^D done
# rlog aliases RCS file: RCS/aliases,v Working file: aliases head: 1.2 branch: locks: strict access list: symbolic names: keyword substitution: kv total revisions: 2; selected revisions: 2 description: ---------------------------- revision 1.2 date: YYYY/MM/DD HH:MI:SS; author: rootyou; state: Exp; lines: +3 -0 added "root-revisions" which is an e-mail alias to which the output of the "find_revisions" script is e-mailed as specified in root's crontab ---------------------------- revision 1.1 date: YYYY/MM/DD HH:MI:SS; author: rootyou; state: Exp; Initial revision ============================================================================= #
Although I was a bit verbose in clarifying some subtleties of the RCS commands, that was a basic example of an edit with RCS. Once you are more familiar with it you'll see that you usually issue just this series of commands: co -l file, vi file, ci -u file.
While learning how to use RCS you will probably make a few mistakes. Don't panic! Common mistakes include editing a file without first obtaining a lock (then finding out when attempting to write that the file is read-only), typing "garbage" characters into the RCS log message during check-in, and accidentally checking-in a revision. These problems can resolved with RCS commands. For instance, the rcs administrative command allows you to delete selected revisions within a ,v file with the -o option.
Don't resort to changing file permissions to work around RCS access control and don't attempt to edit RCS ,v files directly. The revision files can become corrupt and unusable if you edit them - treat them as a "black box". With better understanding of the RCS commands' features, you will be able to manipulate revisions through RCS' interface and avoid exacerbating the problem that you're attempting to solve.
Here's some useful RCS commands:
In my opinion, the answer is "as soon as possible". Every time you perform an edit without RCS you potentially are losing information about how a SA performs her job. This information could be used for such purposes as training new SAs or even helping you to fill in your timesheet at the end of the week. (In the Reporting section, I will show how to report on file modifications made since a specified date.)
Ideally, to build a complete revision history, start with RCS as soon as the base operating system is installed. If your system does not provide RCS, you will have to install a compiler so that you can build RCS from it's source distribution. By starting early you'll catch all those quick-and-dirty configurations and will be able to use them as a reference for configuring your next system from scratch. Another advantage to starting just after the operating system install is that the initial file revisions checked-in with RCS will be those provided by the operating system vendor. Quite often the initial configuration is useful as a reference to help determine how to fix problems that have been created by customizations.
That said, the addage "better late than never" still applies. Even if you didn't start from the initial system install, at least you will be able to track recent modifications and familiarize yourself with revision control so that you can apply it to future projects.
Now that a revision control system is in place, a reporting mechanism would be useful to periodically report on activity involving RCS-maintained files. Ideally, the report will help identify:
# find / -xdev -type f -name '*,v' -print |xargs rlog -d'>1998/07/01'While that may be sufficient in some instances, I wanted to "fine tune" the output format to show the RCS log information for revisions and to show the rcsdiff output for changes in progress. Additionally, if no changes were made I didn't want any output. The result of my modifications is a perl script called find_revisions. It is available as noted above in the Obtaining and Installing RCS section. Here is an example of the output of find_revisions showing recent changes on the root file-system:
# find_revisions -s 1998/07/01 -dx / ================================================================================ RCS file: /etc/RCS/group,v revision 1.7 date: 1998-07-07 10:55:06-05; author: rootme; state: Exp; lines: +1 -1 added Jess to the "net" group ================================================================================ RCS file: /etc/RCS/vfstab,v revision 1.3 date: 1998-08-04 10:11:52-05; author: rootme; state: Exp; lines: +6 -1 added mount of floppy This is being used to hold the tripwire database ---------------------------- revision 1.2 date: 1998-07-23 15:07:15-05; author: rootme; state: Exp; lines: +14 -2 added comment to remind folks that this file is maintained using RCS specified "nosuid" for "/export" and "/var/data" (Apr 23) ================================================================================ RCS file: /etc/RCS/defaultrouter,v revision 1.2 date: 1998-07-06 16:54:09-05; author: rootme; state: Exp; lines: +1 -1 changed the defaultrouter (May 8) now that the ATM LANE interface is up ================================================================================ RCS file: /etc/RCS/system,v revision 1.2 date: 1998-07-23 15:08:17-05; author: rootme; state: Exp; lines: +21 -0 added System V IPC configuration (May 11) ================================================================================
find_revisions has a number of options similar to those of rlog, rcsdiff and find. This is it's usage information:
$ find_revisions -? Unknown option: ? usage: find_revisions [-hx] [-n newer_than_file|-s since_date] [-u user(s)] [-d|c] [find(1)_args] -h - help (shows this usage info) mnemonic: 'h'elp -n newer_than_file - show revisions newer than - the modification time of the specified file mnemonic: 'n'ewer -s since_date - find revisions done since the date specified. The date format should be that accepted by rlog(1)'s "-d" option. e.g. "%Y/%m/%d %H:%M:%S" mnemonic: 's'ince -u user(s) - find revisions done by the user(s) specified. The argument format should be that accepted by rlog(1)'s "-w" option. mnemonic: 'u'ser(s) -d - show current rcsdiff(1) output (if any) as well. This is useful to discover changes which are in-progress, or simply have yet to be checked-in with ci(1). mnemonic: 'd'iff -c - show current rcsdiff(1) "-c" output (if any) as well. (implies "-d") mnemonic: 'c'ontext diff -x - restrict the search to the file system containing the directory specified. mnemonic: -'x'dev e.g. $ find_revisions -x / $ touch /var/tmp/find_revisions.touch $ # time passes... $ find_revisions -n /var/tmp/find_revisions.touch -x /
Below is a sample crontab entry, to run find_revisions every day at 5:30 AM. It will attempt to find RCS revisions since the previous time find_revisions was run from this crontab entry. If revisions or work-in-progress is found, the output will be e-mailed to the e-mail recipient root-revisions which we set up earlier in the sendmail aliases file. (You should create the file /var/tmp/find_revisions.touch when you initially submit this entry using crontab.)
30 5 * * * /usr/bin/touch /var/tmp/find_revisions.next && /usr/local/bin/find_revisions -n /var/tmp/find_revisions.touch -dx / >/var/tmp/find_revisions.out && /usr/xpg4/bin/mv /var/tmp/find_revisions.next /var/tmp/find_revisions.touch && test -s /var/tmp/find_revisions.out && /opt/local/bin/mutt -s "`/usr/bin/hostname`: recent revisions on / file-system" -i /var/tmp/find_revisions.out root-revisions
In assessing the practicality of the system, one should be sure that it is not prohibitively cumbersome. It's important that SAs can use such a system even amidst the "fire-fighting" that sometimes is characteristic of their work. Here are some suggestions to raise the level of awareness of RCS usage and to help remind SAs to participate in the system:
RCS is an effective tool to track file modifications. While it is often considered a programmers' tool, it is useful as a system administrators' tool as well. When used to track modifications to system configuration files, RCS provides a mechanism for SAs to cooperate in a team environment, to keep team members informed of changes, and to recall past configurations for problem solving.
Dave is a benevolent hacker and UNIX aficionado. He is employed as a Network Engineering Technology Systems Programmer in the Division of Information Technology (DoIT), at the University of Wisconsin, Madison. He can be reached at plonka@doit.wisc.edu or via http://net.doit.wisc.edu/~plonka/.
As I recall, some time ago SCCS was more popular than RCS. Today, of the freely available revision systems, it seems RCS and CVS have the lead. (For the purposes of this discussion, CVS may be thought of as a more complex system layered atop RCS. Its additional features would most likely be of little use when maintaining Unix system configuration files.) Years ago, I switched to RCS.
There are two compelling reasons for my migration from SCCS to RCS:RCS' revision file is named with a suffix (filename,v) rathern than a prefix (s.filename), such as SCCS uses. The preference for suffixes is more than cosmetic - utilities such as make(1) can rely on file-name suffixes to select appropriate build rules - so called "default (suffix) rules". So, SCCS is not easily compatible with all make(1)-based projects - a significant caveat for those who maintain source code. (Still, there are some make(1) implementations that have been enhanced to understand SCCS semantics.)
RCS stores lock information within the revision file (filename,v) rather than in a separate file (p.filename). RCS has an important advantage here in the Unix envionment, where links (symbolic or real/"hard" links) are sometimes employed to make a file appear by multiple names or paths. When locking a file with SCCS, a "p." file will be created in the working directory of the user that locked the file. This "p." lock file is only visible within that directory - a major problem if the revision file ("s.") is also linked from another directory. A user in that other directory could also lock the file in question and both users could edit the file and potentially clobber each other's changes upon "check-in".
For SA usage of a file revision system, this distinction is doubly important. Popular Unix implementations such as Solaris make heavy use of links to make its configuration files appear in more than one directory. E.g. the file containing the host database is known as /etc/hosts and /etc/inet/hosts. With RCS, you can link the revision file (hosts,v) into both the /etc and /etc/inet directories to accomodate the preferences (and experiences) of the SAs in your team.
If you ever must convert from SCCS to RCS there exists a freely available utility called sccs2rcs. I have discovered that it may require a specific minimum version of SCCS (that supplied with HP-UX is not sufficient, for instance) but otherwise it worked well for me.
The use of individual root accounts (e.g. rootjs for John Smith's root account), rather than the anonymous real root, is advantageous when SAs use RCS, and is preferable for these reasons as well:
Using vipw, if available, edit the passwd file(s) to add an entry something like this:
root:x:0:1:Super-User:/:/bin/sh rootjs:x:0:1:Root - John Smith:/:/bin/ksh
When adding additional uid 0 accounts, rememeber the 8 character limit on the length of Unix user names. It may be convenient to name the accounts rootxx where xx are the users initials.
Note that under AIX, users are added using the smit command. smit doesn't allow the adding of multiple password entries with the same user id. I work around this limitation by adding the additional rootyou accounts with other uids then subsequently editing the /etc/passwd and /etc/security/user file manually to make set the uid to 0 and set the user attributes like that of the real root account.
If your system provides a utility such as Solaris' pwck(1M), you can use it to check the passwd file integrity before logging out of your root shell.
When adding multiple "uid=0" accounts, make sure to always add the additional root accounts after the "normal" one, so that files owned by uid=0 will show root as the owner. (On most systems getpwuid(3), the function commonly used to translate a numeric user id to a user name, will perform a sequential search of the passwd file. Still, some systems, such a Digital Unix, create a hashed database of passwd file entries to efficiently perform getpwuid(3), so ls(1) might show a user name other than simply root. This is only a small annoyance though as long as you put the word root in the names of all "uid=0" accounts.) Also, it's safer to leave the real root as the first line in case the passwd file is somehow left corrupted while editing.
Make sure that the GECOS field has a unique "real" name (ie. more specific than "Super-User" or some such thing) so that if ever anyone sends e-mail using the name from the GECOS field it will go to the right place. (Many sendmails will allow you to send to the user name by the name in this field with underscores in place of spaces - eg. John_Smith. You would want this e-mail to go to John Smith's normal account... However if you had accidentally put John Smith in the GECOS field as rootjs' "real" name, e-mail to John_Smith would be delivered to rootjs instead because that entry would be encountered earlier in the passwd file.)
The - argument to su(1) causes it to invoke a login shell. This is important so that the shell startup (eg. /etc/profile, .profile) and rc (eg. /.kshrc) files will be sourced - enabling you to customize your root shell environment:
$ su - rootyou Password: #
When chaning a root account password, it's important that you use your own root account name as the argument to passwd otherwise you will end up changing the real root passwd and confusion will ensue:
# passwd rootyou New password: Re-enter new password: #
Common shell startup files, such as /etc/profile and root's .profile should be compatible with Bourne shell syntax, so that it will be compatible with all commonly used shells. (A similar rule would hold true for csh- derived shells and their respective startup and files.) Here's an example fragment for root's .profile:
# { Some systems don't set LOGNAME to the login name specified at the "login:" # or as arg to su(1). (I.e. sometimes it will be "root" even though another # uid=0 username was specified. Try to fix that here... case "$LOGNAME" in root|'') LOGNAME=`/usr/bin/logname` # set LOGNAME correctly to the real login name export LOGNAME ;; esac # } case "$LOGNAME" in you|rootyou) # rootyou uses ksh LOGNAME=rootyou export HISTFILE=/.sh_history_${LOGNAME?} ;; *) # generic root, keep syntax Bourne shell compatible HISTFILE="/.sh_history_$LOGNAME" export HISTFILE ;; esac
Lastly, under some Unixes I have noticed that the LOGNAME environment variable is not set to the real login name given at the login: prompt or as an argument to su. This problem is exhibited under RedHat Linux with pdksh as the login shell, for example. If after logging in as rootyou, you find that LOGNAME is set to just root, rather than your own personal login name, modify your shell initialization file so that it will be set appropriately. For instance, if your login shell is sh, bash, or ksh add this line to root's .profile:
LOGNAME=`/usr/bin/logname` # be sure LOGNAME is set correctly to the login name export LOGNAME
If multiple system administrators might maintain these files, it would be a good idea to maintain such files using RCS.
File | Notes |
---|---|
/.profile, /root/.profile | add reminder comment |
/etc/aliases, /etc/mail/aliases | add reminder comment |
/etc/defaultrouter | strict format |
/etc/exports, /etc/dfs/dfstab | add reminder comment |
/etc/fstab, /etc/vfstab | add reminder comment |
/etc/group | must always be locked out |
/etc/hosts, /etc/inet/hosts | add reminder comment |
/etc/inet/netmasks | add reminder comment |
/etc/inet/networks | add reminder comment |
/etc/inetd.conf | add reminder comment |
/etc/init.d/*, /etc/rc.d/init.d/* | add reminder comment |
/etc/inittab | add reminder comment |
/etc/motd | |
/etc/nsswitch.conf | add reminder comment |
/etc/passwd, /etc/shadow | must always be locked out |
/etc/profile | add reminder comment |
/etc/protocols, /etc/inet/protocols | add reminder comment |
/etc/rc*.d/*, /etc/rc.d/rc*.d/* | add reminder comment |
/etc/resolv.conf | |
/etc/services, /etc/inet/services | add reminder comment |
/etc/shells | |
/etc/syslog.conf | add reminder comment |