Charge to the Referees of the 2005 International Symposium on Computer Architecture

Mark D. Hill

Computer Sciences Department
University of Wisconsin-Madison

Filed at http: //

December 2004

The quality of ISCA depends on the quality of reviews. Please consider the following pitfalls and recommendations.

PITFALL: Seek to find all flaws in the paper, in part to show your expertise as a reviewer.
RECOMMENDATION: Look for reasons to accept a paper. Despite its flaws, does it point in new directions or expose promising insights? ISCA can benefit from flawed, insightful papers.

PITFALL: Since the review process is anonymous, it is appropriate to criticize the paper as if the authors did not have feelings.
RECOMMENDATION: The primary audience for your review is the program committee, while an important secondary audience is the authors. It is possible to assess things fairly, with civility, and offer constructive suggestions. Never attack the researchers behind the work. You can disagree with an assumption without questioning the authors's integrity.

PITFALL: Reject papers that build on recently-published new directions, but accept those that build on the established norm.
RECOMMENDATION: While truly new papers are best (and rare), consider taking papers that follow-up on recently-published promising directions. This allows the community to explore ideas that can't be fully-developed in one paper.

PITFALL: Don't really know how the reviewing process works.
RECOMMENDATION: Read Alan Jay Smith's The Task of the Referee (IEEE Computer, 4/90).

PITFALL: Advocate rejecting a paper with little comment, because it is obvious that all will agree with you. Ditto for accept.
RECOMMENDATION: Explain why you advocate a rejection or acceptance, as people will often disagree with you.

PITFALL: Advocate rejecting (almost) all papers to show about tough you are.
RECOMMENDATION: Your job is decide what is best. This is not usually accomplished by rejecting everything.

PITFALL: Advocate rejecting a paper because you seem to remember it being the same as (or similar to) unidentified prior work.
RECOMMENDATION: In this situation, the professional should reference important prior work after refreshing one's memory regarding what it contains.

The review process consists of four steps.

  1. You should read the paper and enter your review information. You can continue to return and edit your review information until the end of the review period, as long as you have not "finalized" it. You probably should not put anything in the "Comments to the Program Committee" section. This is not a place to bypass the rebuttal process, making claims that might reasonably be challenged by the authors.

  2. When you are finished with your review, "finalize" it using the first menu selection, labeled "Are you finished with your review".

  3. When all reviews are finished, authors will be able to read the two sections, questions for the rebuttal and detailed comments to the author. They will then be able to respond to those comments. That is the only information authors will see during the rebuttal phase.

  4. During program committee meeting, PC members will use all the review information and the information provided by the author to select papers for the conference.

When you review a paper, there are several things you need to tell us. This list follows the list of questions on the review form. When you change an entry on the form, you need to click the button labeled "Submit your paper review". If you leave the page without clicking that button, your review entry will not be stored.