Project 4: Multithreaded Programming
P4 Regrade (New!)
There are a couple of announcements regarding P4 grading process.
First, we have fixed the test files; Init() calls are only made once in each test. We have rerun your code with the updated tests. Your grade has been updated in the same file.
The test files can be obtained from:
Second, some of you lost lots of points because of some tiny mistakes that can be fixed in a couple of lines of code. We are giving you another chance to fix your project (again, thanks to the grader!). Please carefully read the regrade policy below.
Regrade policy: You can only fix upto a total of 8 lines of code for all libraries. (Note that this is not 8 lines per library). Make sure that your new code will give you a better grade. If not, please do not submit any files.
Do not attempt to game this policy by putting more than one C statements in a single line. In other words, in a single line that you fix, there should only be one semi-colon (a semi-colon marks the end of a C statement).
Where to submit: Please submit all your files (as specified in the original handin section below) to your p4 regrade directory:
Again, if you are not planning to get a regrade, please do not put any file.
Final grade: Your final grade will be:
final_grade = old_grade + (new_grade - old_grade) * 75%
Questions about the project? Send them to
Final Deadline: Monday, Nov. 3rd @ Whatever time works for you.
11/2: Good values. For this project, we will pass only good values as inputs to your libraries. For example, we will always pass positive integers to the init functions, pass non-negative integers to Hash_Insert(), etc. Do not worry about malicious users. You can focus on the locking part and the main functionalities of your libraries.
10/31: Thread-safe tests. The two tests that we gave you initially (one for counter and one for hash table) only test if your libraries are thread-safe or not. They do not test the correctness of your library. In fact, those tests assume your libraries are correct in terms of manipulating your internal data structures (but they do not assume your libraries are thread-safe). For grading purpose, we will run another test that specifically targets correctness.
10/30: Grading test for counter. It is available at
10/30: Testing Hints for LRU: First, you should try to understand what testcount.c and testhash.c are really doing. Essentially, each test compares the final expected value with the value returned from your library (e.g. Counter_GetValue(), Hash_CountElements()). For LRU, you can write a similar test; create threads that randomly insert/access/remove a random number within a specified range (e.g. 1 to 1000) for thousands or even millions of times, and check the returned values so that you can keep track the expected size of your LRU. For example, if LRU_Insert() returns 0, then you should increment the expected size. If LRU_Remove() returns 0, you should decrement the expected size. When all threads have finished, you can compare the size returned from LRU_Size() and the expected size you have tracked so far in your test code. Beyond this, you can also test whether your LRU contains the expected elements; extend this test by introducing a bitmap (e.g. expectedBitmap[1..1000]). For example, if you expect element 500 to be in the LRU, then expectedBitmap should equal to true. If not, expectedBitmap should be false. When all threads have finished, you can confirm the content of your bitmap by calling LRU_Access() for each element.
10/29: Please check out the grading standards .
10/28: LRU_Init() now returns void, like the others.
10/23: No locks. You do not have to make these reader functions thread-safe: Counter_GetValue(), Hash_CountElements(), Hash_CountBucketElements(), and LRU_Size(). We only call these reader functions when all threads have finished. Hence, you should not worry about readers-writers lock because readers will be only called at the end. Simply use pthread_mutex_lock, and pthread_mutex_unlock.
There are two objectives to this assignment:
When building a multi-threaded program, one has to make sure one's data structures are multi-thread safe. In this project, we will see what this means in actuality by building three simple MT-safe libraries. The first is a simple counter that multiple threads can use. The second is a hash table, an important data structure used in many multi-threaded applications as well as in the operating system itself. Finally, the third is a data structure that tracks the recency of access to a set of objects; this structure is something like that would be needed to implement true LRU within an OS cache.
You should create three libraries that implement thread-safe data structures. The first one is a simple counter, the second one is a hash table, and the third one is our recency tracking data structure.
libcounter.so: For the first library you should implement thread-safe increment and decrement procedures. Below are the interfaces that you should implement. The code skeleton for this library has been provided in this directory:
Always read the README file first.
libhash.so: The second library you should implement is a thread-safe hash table. In your previous CS courses (e.g. CS 367), you most likely have implemented a hash table for one thread. Inside the OS, hash tables are used a lot, and often accessed by more than one thread. Hence, it is good practice to see how you should make a hash table thread-safe.
Your hash table will store integers. Below are the interfaces that you should implement. For a benchmark, a good implementation should only be around 300 lines of code. The code skeleton for this library has been provided in this directory:
Always read the README file first.
liblru.so: The final library you should build implements a data structure to track a group of positive integers (values greater than zero) in LRU fashion. The interface is as follows:
The header for the code above can be found at:
Compiling with Threads
Compiling a multi-threaded program with POSIX threads on Linux requires two things. First, you need to include the pthread header file pthread.h in your code. Second, when compiling, you need to link with the pthread library -lpthread . That's about it.
For more information, look at this tutorial , one of many available on the web.
Testing your libraries
The main focus of this project is whether your library is thread-safe or not. You should not worry too much about malicious users. In other words, you should expect us to generally use your interface appropriately and provide reasonable parameters.
However, you should still catch error codes from any system calls you make (e.g. malloc). If you catch an unexpected error, simply print an error message and exit (thus killing the process).
So far we have created two complex test codes. They can be found in ~cs537-2/public/p4/counter/ and ~cs537-2/public/p4/hash/ . Please read the corresponding README files in these directories to find out how to run the test code. To exercise threads, you must run the test code in machine with more than one processor. On Linux, you can check this by opening the cpuinfo file (e.g. "less /proc/cpuinfo") to confirm that the machine you use is a multi-processors machine.
The test code basically runs lots of threads and operations and keeps track the final expected value (e.g. the expected number of elements in the hash table). The expected value is compared against the value from your library (e.g. return value of Hash_CountElements()). If they are the same, it is a reasonable indication that your library might be thread-safe.
There are more suggestions:
We will run a bunch of tests. For each test, the expected value should be the same as the value given from your library. Details about each test will come out soon. However, they should be similar to the two test code we have provided.
We will also measure the performance of your library. In particular, to make sure you do not use one big lock for the whole hash table, and you should not just use one big list to track the LRU list (think about how slow LRU_Access() would be if you had to search the whole list to update the recency ordering of elements).
We should cover most of the stuff you need in the discussion sections. However, it is always nice to read more about the functions you will be using.
Check the manual for these calls: pthread_create, pthread_self, pthread_join, pthread_mutex_init, pthread_mutex_lock, pthread_mutex_unlock.
Read the Advanced UNIX Programming book: 11.1 intro, 11.2 thread concepts, 11.3 thread identification, 11.4 thread creation, 11.5 thread termination, 11.6 thread synchronization (up to Mutexes).
Handing in your Code
Hand in your source code and a README file. We will create a directory ~cs537-SECTION/handin/NAME/p4/, where NAME is your login name, and SECTION is the section you are in (1 or 2).
You should copy all of your source files (*.c and *.h) and a Makefile to your p4 handin directory. Do not submit any .o files. When we run your Makefile, it should minimally build all the libraries.
If your program does not work perfectly, your README file should explain what does not work and the reason (if you know).
After the deadline for this project, you will be prevented from making any changes in these directory. Remember: No late projects will be accepted!