Joining a system administration team can be a daunting task. Often problem-solving documentation is non-existent. Also, since system administrators spend much of their working life being interrupted, experienced coworkers are often unavailable to mentor. The challenge is for these junior SAs to perform the tasks assigned and inform their "customers" while juggling tasks in their work queue. Without a tracking and communication system in place, tasks could be neglected, which perpetuates a cycle of interruptions as complaintants find the need to reintroduce their issue. All SAs, regardless of experience, would probably welcome a more predictable introduction to their tasks at hand.
SA work queues contain many "bite-sized" tasks. As such, these tasks are of a scope that they can often be described in a short synopsis by the complaintant. Junior system administrator tasks are often assigned to them based upon that administrators areas of expertise. These simple observations imply that if a sufficient synopsis of a task is supplied, a system could be devised by which the task could be routed automatically to an appropriately talented administrator. In addition, such a system could provide such features as e-mail notification of task milestones and journaling of the life-cycle of that task. An SA team member can query the database of journaled information regarding the resolution of problems to make use of past solutions logged by their peers when similar problems arise. It is just such a system that I propose.
At the core of this system is the GNU Problem Report Management System (PRMS), known as GNATS. This package, intended as a system to track software bugs, was originally authored and is currently maintained by Cygnus Support. The name "PRMS" was the original package name and is used internally by Cygnus Support and appears in some of the documentation. Cygnus, an acronym for "Cover Your GNUS", provides support and development services for free software. According to the authors, GNATS was created in response to their inability to find a suitably configurable, distributed problem report management system for their own use. Like all GNU software, GNATS is freely available and it benefits from being maintained in a robust and portable fashion. According to Cygnus Support, in their document "Keeping Track: Managing Messages With GNATS, The GNU Problem Report Management System",
[GNATS] consists of several utilities which, when used in concert, formulate and administer a database of Problem Reports grouped by site-defined problem categories. It allows a support organization to keep track of problems (hence the term Problem Report) in an organized fashion. Essentially, GNATS acts as an active archive for field-separated textual data submitted through electronic mail.
GNATS makes this easy by automatically filing incoming problem reports into appropriate places, by notifying responsible parties of the existence of the problem and (optionally) sending an acknowledgment to the submitter that the report was received, and by making these Problem Reports accessible to queries and easily editable. GNATS is a database specialized for a specific task.I would describe GNATS as a problem tracking tool of a design which is deeply rooted in the Unix paradigm: a database consisting of directories of flat ASCII files, colon-delimited-line-oriented configuration files, a sendmail-based distributed interface, and a regular-expression-fluent reporting facility. These features help to make GNATS feels like it's the Right Thing for Unix system administrators (SAs).
With GNATS, a task for the system administrator exists as a Problem Report or "PR". Junior system administrators are immediately notified via e-mail when a new task enters their work queue. Because new tasks are routed and assigned automatically, they are not left unattended in an "in-box" or incoming mail folder. Even when a manager or senior system administrator is overwhelmed or unavailable, the work is not neglected. Tasks are routed to the system administrators that will ultimately do the work according to the relationship between the category in which the problem report was submitted and the responsible party as defined by the GNATS administrator. Amongst system administrators, the obvious candidate for GNATS administrator is the senior administrator that currently assigns tasks.
GNATS keeps all interested parties informed via e-mail:
When a PR is submitted and subsequently inducted into the GNATS database, an e-mail response is sent to the complainant containing a PR number. That PR number is a unique "handle" for that PR. The system administrator to which the PR is assigned (based on the category specified by the complainant) is notified as well.
When a PR transitions from one state to another, e-mail notification is sent to the responsible system administrator and, optionally, the complainant. For instance, when a system administrator believes a problem is solved, she updates the PR state to "feedback" and a notification is sent to the original complainant giving them the opportunity to reply with feedback regarding their satisfaction with the alleged solution.
Optionally, any important change to a PR, such as reassignment to another system administrator, will notify a list of people based on the submitter or the category of that PR. Persons appearing on these notify lists can be anyone, but are usually managers of the submitter, senior system administration staff members, or the responsible system administrator's peers.
With GNATS, problem reports reside in a directory and file-based database, similar to that of InterNetNews. An audit trail is maintained within each PR. When updating PRs, the system will prompt system administrators for the reason for a given change. This response, the change itself, the responsible party's name, and a timestamp, form the audit trail. The audit trail provides documentation on the sort of problems that are encountered, how such problems are resolved, and by whom. The database of raw information is everything a senior system administrator or manager may need to summarize the activity of his or her organization. More importantly, this database can be used as a learning tool and problem solving database for fellow administrators. For example, if a new PR arrives stating that the print queue foobar is dysfunctional, a fruitful first step might be for the responsible administrator to query the database of past PRs to learn if and how this problem has been resolved previously. GNATS provides the ability to query based on any PR fields or combination of fields (such as Responsible, Category, or State) using regular expressions for patterns that occur within PRs.
$ send-pr
$ query-pr -r$LOGNAME -s open -q
$ edit-pr 999
For further information, detailed man pages are provided for each of these tools.
As of this writing, the latest official release is gnats-3.2. However, in anticipation of the upcoming gnats-4 release, a number of beta releases exist. Currently, gnats-3.100 ("three dot one hundred") beta is available. The recent beta releases contain important bug fixes to version 3.2, so I highly recommend starting there if gnats-4 is not yet available. The distribution may be obtained via anonymous ftp. See Table 1 for more information.
For my application of GNATS for SAs, the installation was straight forward. I suggest starting out with just one "site" on one Unix host, which is also the machine from which problem reports will be submitted. Excellent documentation is provided, in texinfo format, with the distribution. I found it more convenient to read the document reformatted in HTML, and was able to do so easily with the freely available texi2html translator. HTML and PostScript versions of the document are also available. See Table 1.
Once GNATS is installed, the categories, responsible and submitters files are all that must be maintained. Our categories file contained these options: novell, windows, unix, network, printers, and misc. The responsible file contains the SAs names, against which the ">Responsible:" field is validated when a PR is modified, and their respective e-mail addresses. The submitters file contains names of organizations which submit problems, and an optional list of parties to be notified.
While GNATS is serviceable as-is, I've employed these additions to either provide convenient interfaces and to provide additional features.
With GNATS, tasks no longer fall through the cracks. While objectively this is an obvious advantage, subjectively it means that GNATS is a tireless nag. As with other automatically generated e-mail, recipients of problem reports tend to dismiss them out of hand. To address this potential problem the GNATS administrator should:
Stress that complainants provide meaningful synopses. The synopsis appears in the e-mail subject and provides an opportunity for the sender to "sell" the problem to the system administrator.
Convince the recipients, the system administrators, that there is value in receiving notification of their upcoming work via e-mail. So much of what administrators do is interrupt-driven that anything that will help to alleviate this fire fighting is valuable.
GNATS can be an effective tool for Unix system administrators. While originally conceived to track software bugs, GNATS' assignment, notification, and journaling capabilities shine just as brightly when employed in the tracking of system administration tasks. Once configured, it routes tasks directly to a system administrator with the appropriate skill-set. Junior system administrators and their superiors can examine their work queue, and are automatically notified of any changes in priority or responsibility. This system returns to system administrators' the time they would have had to spend building, organizing, and reporting on their queue of work.
Dave is a benevolent hacker and Unix aficionado. He has a B.S. in Computer Science from Carroll College in Waukesha, Wisconsin (1991) and employed as Lead Systems Programmer at McHugh Freeman, (http://www.mfa.com) and can be reached at Dave.Plonka@mfa.com or via http://www.cc.edu/~plonka.