Reviews for a Program Committee
John Ousterhout
Issues
- Will this advance the state of the art?
- Did you learn anything new?
- Does it provide evidence which supports/contradicts hypotheses?
- Experimental validation?
- Will the paper generate discussion at the conference?
- How readable is the paper?
(The draft can be modified, and if the ideas are very important, you may accept it anyway.)
- Is the paper relevant to a broader community?
Goals of Review
- Guide the program committe in selection process
- Help authors
(to revise paper for acceptance, to understand rejection, to improve further research and future projects)
Structure
- 1/2 to 1 page of text (2 - 4 paragraphs)
- longer reviews are generally given for better papers, shorter reviews for bad papers
- 1 paragraph executive summary
- what is the paper trying to do?
- what is potential contribution of paper?
- short summary of strengths and weaknesses (sentence or 2)
- accept/reject (hard, because you don't necessarily see the entire sample)
- several paragraphs of details (listed in order of importance)
- technical flaws?
- structure of paper?
- are key ideas brought out?
- don't want to just describe system, also need motivation and justification of approach
- presentation? (ex: undefined terms, confusing wording, unclear sections...)
- justification -- can they say why ideas are important?
comparison with other systems? For bigger conferences (SOSP, ISCA,
ASPLOS) need quantitative evidence of ideas
- grammar? (usually only point out consistent errors)