Charge to the Referees of the 2005 International Symposium on Computer Architecture

The quality of ISCA depends on the quality of reviews. Please consider the following pitfalls and recommendations.

PITFALL: Seek to find all flaws in the paper, in part to show your expertise as a reviewer.
RECOMMENDATION: Look for reasons to accept a paper. Despite its flaws, does it point in new directions or expose promising insights? ISCA can benefit from flawed, insightful papers.

PITFALL: Since the review process is anonymous, it is appropriate to criticize the paper as if the authors did not have feelings.
RECOMMENDATION: The primary audience for your review is the program committee, while an important secondary audience is the authors. It is possible to assess things fairly, with civility, and offer constructive suggestions. Never attack the researchers behind the work. You can disagree with an assumption without questioning the authors's integrity.

PITFALL: Reject papers that build on recently-published new directions, but accept those that build on the established norm.
RECOMMENDATION: While truly new papers are best (and rare), consider taking papers that follow-up on recently-published promising directions. This allows the community to explore ideas that can't be fully-developed in one paper.

PITFALL: Don't really know how the reviewing process works.
RECOMMENDATION: Read Alan Jay Smith's The Task of the Referee (IEEE Computer, 4/90).

PITFALL: Advocate rejecting a paper with little comment, because it is obvious that all will agree with you. Ditto for accept.
RECOMMENDATION: Explain why you advocate a rejection or acceptance, as people will often disagree with you.

PITFALL: Advocate rejecting (almost) all papers to show about tough you are.
RECOMMENDATION: Your job is decide what is best. This is not usually accomplished by rejecting everything.

PITFALL: Advocate rejecting a paper because you seem to remember it being the same as (or similar to) unidentified prior work.
RECOMMENDATION: In this situation, the professional should reference important prior work after refreshing one's memory regarding what it contains.

The review process consists of four steps.

1. You should read the paper and enter your review information. You can continue to return and edit your review information until the end of the review period, as long as you have not "finalized" it. You probably should not put anything in the "Comments to the Program Committee" section. This is not a place to bypass the rebuttal process, making claims that might reasonably be challenged by the authors.

2. When you are finished with your review, "finalize" it using the first menu selection, labeled "Are you finished with your review".

3. When all reviews are finished, authors will be able to read the two sections, questions for the rebuttal and detailed comments to the author. They will then be able to respond to those comments. That is the only information authors will see during the rebuttal phase.

4. During program committee meeting, PC members will use all the review information and the information provided by the author to select papers for the conference.

When you review a paper, there are several things you need to tell us. This list follows the list of questions on the review form. When you change an entry on the form, you need to click the button labeled "Submit your paper review". If you leave the page without clicking that button, your review entry will not be stored.

•  Are you finished with this review?
  Once you "finalize" this review, you can not modify it any further. Be careful with this option.

•  Provide a short summary of the paper
  You should briefly summarize the paper: problem attacked, ideas, and methods. No opinions here.

•  Strength of Paper
  Summarize in 1 to 3 sentences the strengths of the paper. You should be able to find at least some strength (positive point) in every paper. Remember, the goal is to identify papers that have something to contribute, not to trash papers that have problems.

•  Weakness of Paper
  Summarize in 1 to 3 sentences the main weakness(es) of the paper.

•  Your qualifications to review this paper
  We realize that not all reviewers are equally qualified; accurately assessing your background will help the program committee understand your review.

•  Writing
  Is the paper readable? If there are "show stopper" issues, you should be clear what they are. Note this is a high-level question, not concerned with grammar, punctuation, and spelling.

•  Relevance to the ISCA community
  Do you think the paper would engender a lot of discussion? Remember ISCA is a very broad conference, including topics of architecture, software, and applications.

•  Experimental Methodology
  Was the experimental method reasonable? Good benchmarks? Good assumptions? Is the OS and/or multithreading modeled if appropriate?

•  Novelty of paper
  This conference focuses on novel work; it's acceptable that a novel idea have less stringent evaluation than an incremental improvement, so we need to know how novel the work is. Papers that incrementally build on novel works may also have more value that those the build on the established norm. If you think the paper is not novel, you should clearly state why and indicate related prior work in your comments to the author (or to the program committee if the work being ignored is primarily your own).

•  Overall Paper Merit
  This is the single most important decision concerning the paper - do you think it should be accepted to the conference? Consider giving high rating to innovative paper even if they have non-trivial flaws. You should answer the question if possible, though you may want to defer, for example, if you this it is a good paper, but inappropriate for ISCA.

  1) Strong Reject -

  Paper should not have been submitted to ISCA.

  2) Reject -

  Paper does not meet ISCA standards.

  3) Weak Reject -

  Paper is potentially of interest to ISCA, perhaps after some improvements.

  4) Middle of the Road -

  Paper, when revised, is a near-miss for ISCA.

  5) Weak Accept -

  Paper, when revised, compares to the LOWER-third of recent ISCA acceptances.

  6) Accept -

  Paper, when revised, compares to the MIDDLE-third of recent ISCA acceptances.

  7) Strong Accept -

  Paper, when revised, compares to the UPPER-third of recent ISCA acceptances.

•  Overall Paper Potential
  How would you revise your rating if the paper's minor subject and major methodology flaws were fixed? That is, how interesting is the potential of this paper?

•  Specific questions for the authors to address in the rebuttal
  Any major flaws in the paper must be identified here. If there are any important questions that need to be answered or you believe there are major issues with the paper, these all need to be raised here for the authors to address. Limit yourself to one, maybe two, questions, as the rebuttals are short.

•  Additional detailed comments to the author
  This is the single most important response you can provide to the author. One of the critical roles of a review process is to help authors understand any problems with their work so they can improve it. Please spend the time to indicate to the author and program committee why you've rated the paper the way you did.

•  Additional comments to the program committee
  There may be reasons that influence your rating of the paper that would reveal your identity. Please elaborate if there are any such issues; these comments will be held in confidence.