Computer Sciences Department
University of Wisconsin-Madison
Filed at http: //www.cs.wisc.edu/~markhill/ISCA2005referee_charge.html
The quality of ISCA depends on the quality of reviews. Please consider the following pitfalls and recommendations.
Seek to find all flaws in the paper, in part to
show your expertise as a reviewer.
RECOMMENDATION: Look for reasons to accept a paper. Despite its flaws, does it point in new directions or expose promising insights? ISCA can benefit from flawed, insightful papers.
Since the review process is anonymous, it is appropriate to criticize
the paper as if the authors did not have feelings.
RECOMMENDATION: The primary audience for your review is the program committee, while an important secondary audience is the authors. It is possible to assess things fairly, with civility, and offer constructive suggestions. Never attack the researchers behind the work. You can disagree with an assumption without questioning the authors's integrity.
Reject papers that build on recently-published new directions,
but accept those that build on the established norm.
RECOMMENDATION: While truly new papers are best (and rare), consider taking papers that follow-up on recently-published promising directions. This allows the community to explore ideas that can't be fully-developed in one paper.
Don't really know how the reviewing process works.
RECOMMENDATION: Read Alan Jay Smith's The Task of the Referee (IEEE Computer, 4/90).
Advocate rejecting a paper with little comment, because it is
obvious that all will agree with you. Ditto for accept.
RECOMMENDATION: Explain why you advocate a rejection or acceptance, as people will often disagree with you.
Advocate rejecting (almost) all papers to show about tough you are.
RECOMMENDATION: Your job is decide what is best. This is not usually accomplished by rejecting everything.
Advocate rejecting a paper because you seem to remember it being the
same as (or similar to) unidentified prior work.
RECOMMENDATION: In this situation, the professional should reference important prior work after refreshing one's memory regarding what it contains.
The review process consists of four steps.
You should read the paper and enter your review information. You can continue to return and edit your review information until the end of the review period, as long as you have not "finalized" it. You probably should not put anything in the "Comments to the Program Committee" section. This is not a place to bypass the rebuttal process, making claims that might reasonably be challenged by the authors.
When you are finished with your review, "finalize" it using the first menu selection, labeled "Are you finished with your review".
When all reviews are finished, authors will be able to read the two sections, questions for the rebuttal and detailed comments to the author. They will then be able to respond to those comments. That is the only information authors will see during the rebuttal phase.
During program committee meeting, PC members will use all the review information and the information provided by the author to select papers for the conference.
When you review a paper, there are several things you need to tell us. This list follows the list of questions on the review form. When you change an entry on the form, you need to click the button labeled "Submit your paper review". If you leave the page without clicking that button, your review entry will not be stored.
Are you finished with this review?
Once you "finalize" this review, you can not modify it any further. Be careful with this option.
Provide a short summary of the paper
You should briefly summarize the paper: problem attacked, ideas, and methods. No opinions here.
Strength of Paper
Summarize in 1 to 3 sentences the strengths of the paper. You should be able to find at least some strength (positive point) in every paper. Remember, the goal is to identify papers that have something to contribute, not to trash papers that have problems.
Weakness of Paper
Summarize in 1 to 3 sentences the main weakness(es) of the paper.
Your qualifications to review this paper
We realize that not all reviewers are equally qualified; accurately assessing your background will help the program committee understand your review.
Is the paper readable? If there are "show stopper" issues, you should be clear what they are. Note this is a high-level question, not concerned with grammar, punctuation, and spelling.
Relevance to the ISCA community
Do you think the paper would engender a lot of discussion? Remember ISCA is a very broad conference, including topics of architecture, software, and applications.
Was the experimental method reasonable? Good benchmarks? Good assumptions? Is the OS and/or multithreading modeled if appropriate?
Novelty of paper
This conference focuses on novel work; it's acceptable that a novel idea have less stringent evaluation than an incremental improvement, so we need to know how novel the work is. Papers that incrementally build on novel works may also have more value that those the build on the established norm. If you think the paper is not novel, you should clearly state why and indicate related prior work in your comments to the author (or to the program committee if the work being ignored is primarily your own).
Overall Paper Merit
This is the single most important decision concerning the paper - do you think it should be accepted to the conference? Consider giving high rating to innovative paper even if they have non-trivial flaws. You should answer the question if possible, though you may want to defer, for example, if you this it is a good paper, but inappropriate for ISCA.
Overall Paper Potential
How would you revise your rating if the paper's minor subject and major methodology flaws were fixed? That is, how interesting is the potential of this paper?
Specific questions for the authors to
address in the rebuttal
Any major flaws in the paper must be identified here. If there are any important questions that need to be answered or you believe there are major issues with the paper, these all need to be raised here for the authors to address. Limit yourself to one, maybe two, questions, as the rebuttals are short.
Additional detailed comments to the author
This is the single most important response you can provide to the author. One of the critical roles of a review process is to help authors understand any problems with their work so they can improve it. Please spend the time to indicate to the author and program committee why you've rated the paper the way you did.
Additional comments to the program committee
There may be reasons that influence your rating of the paper that would reveal your identity. Please elaborate if there are any such issues; these comments will be held in confidence.