Many computer science researchers accept the honor and burden of being program committee (PC) chair of a major conference only once. Thus, many PC chairs are performing the task for the first time. While the task is informed by being a PC member, its duties and responsibilities are considerably different. Once PC chairs have gained their experience, it is often lost.

To pass on some of what I learned serving as PC chair of the 2005 International Symposium on Computer Architecture (ISCA), I have written two short documents:

• Program Chair's Message for 2005 International Symposium on Computer Architecture (ISCA) discusses, for a wide audience, the process we used for ISCA 2005. It has more details than many PC messages. I recommend that you read this first.
• This page presents, to future PC chairs, some ideas that (appeared to) work for ISCA 2005. It is my hope that they may serve you as a good starting place.

Some Things That Worked (Reasonably Well)

Here on some things that worked well, in order of importance.

Use established software package and assign a web master. I used Grunwald's Conference Review Package (CRP) . It served well. I had one of my Ph.D. students deal with the software, including implementing minor changes. Since I knew this was a large task, I gave him credit as Web Master, an official ISCA 2005 officer. In the end, he reports that he worked 120 hours on both CRP and the conference web page.

Seek reasons to accept papers. At the PC meeting, I had the PC member who gave the highest recommendation begin the discussion by summarizing the paper and then critiquing it. Next, other PC members who read the paper spoke. Finally, we had a discussion. This set a positive tone, and, in the long run, helped focus PC members on looking to what is good (since some PC members summarized many more papers than others). I also kept mentioning 42 papers as a target, reminding members to look for papers to accept. In the end, we took 45 papers.

Keep PC meeting discussion focused on making decisions. I was ruthless about focusing on decisions. I cut people off when they rambled. I cut off discussion when it appeared consensus was reached. See "egg timer" below. We covered the top 50 papers explicitly and then moved to a model where PC members could "save" papers (to cover 30 more).

Following SIGARCH guidelines. I strictly followed Guidelines for SIGARCH Sponsored Conferences. I found it helpful to build on the wisdom of others. In particular, I fould it valuable to follow the guideline on restricting each PC member to at most two submissions. This reduced the number of PC papers that we would likely accept (reducing appearance and reality of inbreeding) and saved time at the PC meeting (since PC paper discussions take longer than others).

Being very clear on conflict of interest rules. Doing so makes the process more fair and, equally important, appear more fair. In the PC meeting, for example, I had two other researchers run the discussion for papers for which I had a conflict of interest. This was critical, since on the rest of the papers, I would often cut off discussion. If I did that for a conflicting paper, it might look like I was influencing the decision (because I would be). I found that, with reminders, authors were good at listing conflicts, but PC members were not.

Limited internet access. I had only two Internet connections. We used these to print a few key related papers. Keeping PC members from their email was critical for keeping their attention. We brought two laptops and one printer.

Getting all external reviews (2 per paper). I personally solicted non-PC reviews for most papers and, for papers where I had a conflict of interest, designated two PC members to act as PC chairs to solict reviews (indirectly, with me sending the actual email). With some nagging I got all external reviews. The reviews were less correlated with PC reviews (but were correlated with me). I believe they carried more weight at the PC meeting, since they were mostly from known people rather than junior students. This process cost me two-person weeks (but I am good at nagging).

Rebuttals. I fould rebuttals helpful. They serve as a hard pre-PC-meeting deadline for both external reviewers and PC members. They make authors feel better. They ocasionally make a difference in the PC's decision. I recommend keeping them short (e.g., 200 words) to reduce both author and PC burden.

Other recommendations.

• Asking for three PC reviews. This seemed to balance PC burden with a sufficient representation of papers at the meeting.
• Reducing review count on a few weak papers (15 papers). This somewhat reduced PC and external reviewer burden, but was not a big factor.
• Create a personalized CD for each PC member. Each CD had a directory with all papers and another with all reviews, less the reviews where the PC member had a conflict of interest. As backups, we had everything on a memory stick (less than 128 MB) and a paper copy. One could omit the paper copy in favor of two memory sticks.
• O'Hare Hilton. It worked well to have a one-day meeting near Chicago O'Hare, because PC members could leave after 6:00 PM and still reach most of the United States.
• Egg Timer. An egg timer was set to beep after five minutes of initial discussion. I did not explicitly cut off discussion, but it effectively reminded people to keep things moving.
• PostIt Notes. PostIt notes were used to display accepted, rejected, and deferred papers. "Deferred" meant "could accept depending on what else we had at the end of the day." This was a great visual way to see progress.
• Easel. The next ten or so papers to discuss were displayed on an easel. This let PC members "pipeline" refreshing their memories with discussion of papers they were less interested in. As you know, pipelining can greatly improve throughput.
• No PC Grading. I did not have PC members "grade" papers before the PC meeting, because I feared partial participation. Based on "re-scoring" experience below, I think this was the correct decision.

Things That Did NOT Work (Well)

• Re-Scoring Papers. Before the PC meeting, I encouraged PC members to read all reviews, interact with each other, and, based on this new input, optionally change their overall recommendation on a paper. Some PC members took this seriously; others did not. People are busy. I wish I knew a way to do this better. Nevertheless, even partial re-scoring helps improve the paper ranking used at the PC meeting.
• Small PC (23 members). I chose a smaller PC than recent ISCAs. This may have helped discussion some, but was quite a burden on PC members (26-ish papers each). I might increase the size by several members, but still worry about the group dynamics of 35 people.
• PC Information. PC members were not good about listing their areas of interest or conflicts of interest. You need to know or learn these by other means.
• Paper Potential. On the review form, after "Overall Recommend," I asked, At what level would you recommend this paper if you consider only its POTENTIAL (e.g., ignoring methodological flaws) and DIRECTION (e.g., Could it expose new areas of research for ISCA?)?. This question did not work well. Reviewers answered it inconsistently, and, to my surprise, sometimes lower than the overall recommendation. I asked a few reviewers about this, and they responded that this was their intention.
• Ranking papers with 2/3*(Recommendation) + 1/3*(Potential). This did not work much better than using "Recommendation" only. I had a secondary ranking that removed the single lowest recommendation and then averaged the rest. I recommend this ranking function to you, as I removed the initial effect of a single harsh reviewer (a common occurrence).

Acknowledgements

Thanks for feedback from David Albonesi, Remzi Arpaci-Dusseau, Margaret Martonosi, Kathryn McKinley and Caitlin Scopel.

References

Mark D. Hill, Program Chair's Message for 2005 International Symposium on Computer Architecture (ISCA), http://www.cs.wisc.edu/~markhill/ISCA2005PCchair_msg.pdf

Mark D. Hill, Charge to the Referees of the 2005 International Symposium on Computer Architecture (ISCA), http://www.cs.wisc.edu/~markhill/ISCA2005referee_charge.html