Project 1a: Sorting with Variable-Sized Records

You will write a simple sorting program. This program should be invoked as follows:

shell% ./varsort -i inputfile -o outputfile 

The above line means the users typed in the name of the sorting program ./varsort and gave it two inputs:

• an input file to sort called inputfile
• an output file called outputfile to put the sorted results

Input files are generated by a program we give you called genvar.c .

After running genvar, you will have a file that needs to be sorted. Note that this file is filled with binary data and not ASCII data, so if you try use cat or more to look at the file, you will see what looks like just a bunch of garbage characters.

The file begins with a header. This header is a single integer, R containing the number of records in this file.

After the header, the file contains a variable-sized records. Each record can be a different size and each is structured as follows:

• Key: the first four bytes of each record are an unsigned integer key. The key determines the order the records should be sorted in.
• Data size: the second four bytes are an unsigned integer indicating the size N (in integers, or words) of the following data that is associated with this key
• Data: N integers (or words) of Data for this key. This data should always stay associated with this key.

kkkk<8>DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD

Note that different records can have keys with different amounts of associated data.

Your goal: to build a sorting program called varsort that takes in one of these generated files and sorts it in ascending order (from lowest to highest) based on the 4-byte key (the remainder of the record should of course be kept with the same key). The header and the sorted keys and their associated records are written to the specified output file.

Some Details

Using genvar is easy. First you compile it as follows:

shell% gcc -o genvar genvar.c -Wall -Werror

Note: you will also need the header file sort.h to compile this program.

Then you run it:

shell% ./genvar -s 0 -n 100 -m 32 -o /tmp/outfile

There are four flags to genvar . The -s flag specifies a random number seed; this allows you to generate different files to test your sort on. The -n flag determines how many records to write to the output file. The -m flag designates the maximum amount of data that is associated with a key, measured in units of integers (note that records can have LESS than this amount of data). Finally, the -o flag determines the output file, which will be the input file for your sort.

The format of the file generated by the genvar.c program is very simple: it is in binary form, and consists of the header information and those variable-sized records as described above.

Another useful tool is dumpvar.c . This program can be used to dump the contents of a file generated by genvar or by your sorting program.

For example, if run as follows:

shell% ./dumpvar -i /tmp/file

When you create files to sort, we strongly recommend that you place those files in /tmp rather than in your home directory or course project work space. Files in /tmp are stored in the local file system for a particular machine instead of in the AFS distributed file system space; therefore, with /tmp you won't run into quota problems and accessing the local files should be relatively quick. Of course, the files you create in /tmp on one machine will not exist on a different machine.

You will probably want to the look at the source code for both of these utilities to see how to read and write to files; in particular, they could be useful for seeing how to understand the variable-sized record format.

A common header file sort.h has the detailed description. There are three different versions of the record that you might find useful in different circumstances:

• rec_nodata_t: a structure that contains no data
• rec_data_t : a structure that contains the maximum amount of data
• rec_dataptr_t : a structure that contains a pointer to the data (allocated elsewhere)

Note that you may NOT simply allocate an array of rec_data_t structures for this project because the rec_data_t structure assumes the worst-case size for the amount of data that can be associated with a key (MAX_DATA_INTS). You must dynamically allocate only the amount of memory needed for each record, which might be significantly smaller than the maximum amount. You will want to use malloc to allocate the correct amount of data for each record and set the data_ptr field in the rec_dataptr_t structure to point to this data on the heap.

Hints

In your sorting program, you should just use open() , read() , write() , and close() to access files. See the code in genvar or dumpvar for examples. Make sure you print an appropriate error message and exit correctly if an error occurs!

To sort the data, use any sort that you'd like to use. An easy way to go is to use the library routine qsort() . The qsort() routine is a bit tricky to use, so you will definitely want to look at the document for it. Typing man qsort at the command line will give you a lot of information on how to use the library sorting routine. In particular, you will want to study the example code that demonstrates how to use qsort and how to specify a comparison function. Note that the comparison function takes as arguments pointers to the objects being compared.

The routine malloc() is useful for memory allocation. Make sure to print an error message and exit cleanly if malloc fails!

Remember to write out the header (i.e., the number of records) for your sorted output. genvar won't work without it.

For efficiency, you might not want to call malloc separately for the data in each record. Maybe you can allocate the memory for all the data at once. If you want to figure out how big the input file is before reading it in, use the stat() or fstat() calls. After you've read the header and know the number of records in the file, you can calculate how much space is needed for the fixed sized portion of the records (keys, data_ints, and pointers) versus for the variable-sized data. After you've allocated enough memory, you can then carefully set up the pointer in the rec_datptr_t structure so that record refers to the correct data. Note that when you swap two records during your sort, you will now be swapping the corresponding pointers instead of all the data associated with the records -- which should be much faster for large records.

To exit, call exit() with a single argument. This argument to exit() is then available to the user to see if the program returned an error (i.e., return 1 by calling exit(1) ) or exited cleanly (i.e., returned 0 by calling exit(0) ).

If you don't know how to use these functions, use the man pages!

To do some preliminary testing of your implementation of varsort, test to see if you generate the file sample-out given the input sample-in.

In addition to testing for correct program behavior, we will also be giving points for good programming style and for careful memory management.

As programmers, we often won't be writing all our own code from scratch, and instead will be making contributions inside of an already existing project which other people are also contributing to. In these situations, other programmers will often need to be able to read and understand the code you've written. Because of this, many companies will require its programmers to adhere to a style guideline, so that the task of reading and understanding another person's code is made easier.

For grading on style, we will mostly stick to the style guidelines which Google uses for C++ (more info can be found here) with a few differences specific to C. These differences are in the config file in ~cs537-1/ta/lint/CPPLINT.cfg. To grade this, we will be running a kind of program called a "lint program", or a "linter", which just does basic style checking. The linter we will use can be found here:

~cs537-1/ta/lint/cpplint.py

We will be running it with the following options:

cpplint.py --extensions=c,h varsort.c

Another important skill to develop in C programming is good memory management. This means freeing any heap space you've allocated when you're done using it! Since memory is a limited resource, we will want to free memory when we're done with it so that it can be reused. To check that your code doesn't contain memory leaks, we will be using a tool known as valgrind. It's a simple tool which moniters every call to malloc (and other memory allocation functions) and free to make sure that all memory allocated to our program is subsequently released when we are done with it. Our tests will be running valgrind on your code in the following way:

valgrind --show-reachable=yes ./varsort -i infile -o outfile

Assumptions and Errors

32-bit integer range. You may assume that the keys are unsigned 32-bit integers.

Record size. You may assume that there are no more than USHRT_MAX data items in each record. However, most records may have many fewer data items than the max, so don't allocate this much memory for every record!

File length: May be pretty long! However, there is no need to implement a fancy two-pass sort or anything like that.

Invalid files: If the user specifies an input or output file that you cannot open (for whatever reason), the sort should EXACTLY print: Error: Cannot open file foo (with no extra spaces, and assuming the file was named foo ) and then exit.

Too few or many arguments passed to program: If the user runs varsort without enough arguments, or in some other way passes incorrect flags and such to varsort, print Usage: varsort -i inputfile -o outputfile and exit.

Important: On any error code, you should print the error to the screen using fprintf() , and send the error message to stderr (standard error) and not stdout (standard output). This is accomplished in your C code as follows:

fprintf(stderr, “whatever the error message is\n”);

Your grade will primarily depend only on the correctness of your program. However, programs that run signficantly slower than others (i.e., an order of magnitude slower) will be penalized.

History and an Optional Performance Contest

This sorting assignment derives from a yearly competition to make the fastest disk-to-disk sort in the world. See the sort home page for details. If you look closely, you will see that your professor was once -- yes, wait for it -- the fastest sorter in the world.

To continue in this tradition, we will also be holding a sorting competition. The variable-sized record sort that you need to implement for this assignment is a little trickier than the straight-forward sort, since you need to handle more than just 100-byte records. For the contest, we will average together your program's runtime on three input sets:

• 1,000,000 input keys: all small records, all less than 8 integers of data (32 bytes).
• 1,000,000 input keys: small and medium-sized records, all less than 128 integers of data (512 bytes).
• 1,000,000 input keys: small, medium, and large records, all less than 2048 integers of data (8 KB).

Whoever wins the performance contest will win a soft-cover copy of the OSTEP text book.

Read more about sorting, including perhaps the NOW-Sort paper , for some hints on how to make a sort run really fast. Or just use your common sense! Hint: you'll have to think a bit about hardware caches.

We will only measure the performance of assignments that sort correctly for all cases!

General Advice

Start small, and get things working incrementally. For example, first get a program that simply reads in the input file, one line at a time, and prints out what it reads in. Then, slowly add features and test them as you go. Don't worry about performance until you have all of the functionality working correctly.

Testing is critical. Testing your code to make sure it works is crucial. Write tests to see if your code handles all the cases you think it should. Be as comprehensive as you can be. Of course, when grading your projects, we will be. Thus, it is better if you find your bugs first, before we do.

Keep old versions around. Keep copies of older versions of your program around, as you may introduce bugs and not be able to easily undo them. A simple way to do this is to keep copies around, by explicitly making copies of the file at various points during development. For example, let's say you get a simple version of varsort.c working (say, that just reads in the file); type cp varsort.c varsort.v1.c to make a copy into the file varsort.v1.c . More sophisticated developers use version control systems like CVS (old days) or mercurial or github (modern times), but we'll not get into that here (though you can, and perhaps should!).