Academic Paper Review

Academic Paper Review

Overview

Produces structured, peer-review-quality analyses of academic papers. Follows

review standards used by top-tier venues (NeurIPS, ICML, ACL, Nature, IEEE) to

provide rigorous, constructive, and balanced assessments.

Covers **summary, strengths, weaknesses, methodology assessment, contribution

evaluation, literature positioning, and actionable recommendations** — all

grounded in evidence from the paper itself.

When to Use

• User provides a paper URL (arXiv, DOI, conference proceedings)
• User asks to "review", "analyze", "critique", "assess", or "summarize" a

research paper

• User wants to understand strengths and weaknesses of a study
• User requests a peer-review-style evaluation

Review Methodology

Phase 1: Paper Comprehension

Step 1.1: Identify Paper Metadata

Extract: Title, Authors, Venue/Status, Year, Domain, Paper Type (Empirical /

Theoretical / Survey / Systems / Position).

Step 1.2: Deep Reading Pass

Read the paper systematically using browse_page:

• Abstract & Introduction — Identify claimed contributions and motivation
• Related Work — Note how authors position relative to prior art
• Methodology — Understand the proposed approach in detail
• Experiments / Results — Examine datasets, baselines, metrics, outcomes
• Discussion & Limitations — Note self-identified limitations
• Conclusion — Compare concluded claims against actual evidence

Step 1.3: Key Claims Extraction

List the paper's main claims explicitly:

Claim 1: [Specific claim]
Evidence: [What evidence supports this]
Strength: [Strong / Moderate / Weak]

Phase 2: Critical Analysis

Step 2.1: Literature Context Search

Use web_search to understand the research landscape:

"[paper topic] state of the art [current year]"
"[key method name] comparison benchmark"
"[specific technique] limitations criticism"

Use browse_page on key related papers or surveys.

Step 2.2: Methodology Assessment

| Criterion | Questions to Ask | Rating |

|-----------|-----------------|--------|

| Soundness | Is the approach technically correct? | 1-5 |

| Novelty | What is genuinely new vs incremental? | 1-5 |

| Reproducibility | Are details sufficient? Code/data available? | 1-5 |

| Experimental Design | Are baselines fair? Ablations adequate? | 1-5 |

| Statistical Rigor | Results significant? Error bars? Multiple runs? | 1-5 |

| Scalability | Does it scale? Computational costs discussed? | 1-5 |

Step 2.3: Contribution Significance

| Level | Description |

|-------|-------------|

| Landmark | Fundamentally changes the field |

| Significant | Strong contribution advancing state of the art |

| Moderate | Useful contribution with some limitations |

| Marginal | Minimal advance over existing work |

| Below threshold | Does not meet publication standards |

Phase 3: Review Synthesis

Produce the final review using this template:

# Paper Review: [Paper Title]

## Paper Metadata
- **Authors**: [Author list]
- **Venue**: [Publication venue or preprint server]
- **Year**: [Year]
- **Domain**: [Research field]
- **Paper Type**: [Empirical / Theoretical / Survey / Systems / Position]

## Executive Summary
[2-3 paragraph summary of core contribution, approach, and main findings.
State overall assessment upfront.]

## Summary of Contributions
1. [First claimed contribution]
2. [Second claimed contribution]

## Strengths
### S1: [Concise strength title]
[Detailed explanation with specific references to sections, figures, tables.]

### S2: [Concise strength title]
[...]

## Weaknesses
### W1: [Concise weakness title]
[Detailed explanation. Explain impact. Suggest how to address.]

### W2: [Concise weakness title]
[...]

## Methodology Assessment
| Criterion | Rating (1-5) | Assessment |
|-----------|:---:|------------|
| Soundness | X | [Brief justification] |
| Novelty | X | [Brief justification] |
| Reproducibility | X | [Brief justification] |
| Experimental Design | X | [Brief justification] |
| Statistical Rigor | X | [Brief justification] |
| Scalability | X | [Brief justification] |

## Questions for the Authors
1. [Specific question]
2. [Question about methodology choices]

## Literature Positioning
[How does this work relate to current state of the art? Key related works cited?]

## Recommendations
**Overall Assessment**: [Accept / Weak Accept / Borderline / Weak Reject / Reject]
**Confidence**: [High / Medium / Low]
**Contribution Level**: [Landmark / Significant / Moderate / Marginal / Below threshold]

### Actionable Suggestions for Improvement
1. [Specific, constructive suggestion]
2. [Specific, constructive suggestion]

Review Principles

• Always suggest how to fix it — Don't just point out problems; propose solutions
• Give credit where due — Acknowledge genuine contributions even in flawed papers
• Be specific — Reference exact sections, equations, figures, tables
• Separate minor from major — Distinguish fatal flaws from fixable issues

Objectivity Standards

• ❌ "This paper is poorly written" (vague, unhelpful)
• ✅ "Section 3.2 introduces notation X without formal definition, making the

proof in Theorem 1 difficult to follow. Consider adding a notation table."

(specific, actionable)

Adaptation by Paper Type

| Paper Type | Focus Areas |

|------------|-------------|

| Empirical | Experimental design, baselines, statistical significance, ablations |

| Theoretical | Proof correctness, assumption reasonableness, tightness of bounds |

| Survey | Comprehensiveness, taxonomy quality, coverage of recent work |

| Systems | Architecture decisions, scalability evidence, real-world deployment |

| Position | Argument coherence, evidence for claims, impact potential |

Common Pitfalls

• ❌ Reviewing the paper you wish was written instead of the paper submitted
• ❌ Demanding additional experiments that are unreasonable in scope
• ❌ Penalizing the paper for not solving a different problem
• ❌ Being overly influenced by writing quality versus technical contribution
• ❌ Providing only a summary without critical analysis

Quality Checklist

• [ ] Paper read completely (not just abstract and introduction)
• [ ] All major claims identified and evaluated against evidence
• [ ] At least 3 strengths and 3 weaknesses with specific references
• [ ] Methodology assessment table complete with ratings and justifications
• [ ] Literature search conducted to contextualize the contribution
• [ ] Recommendations are actionable and constructive
• [ ] Review tone is professional and respectful

Output

• Output the complete review in Markdown
• Save to .poirot/outputs/review-{paper-topic}.md via write_file
• Present to user via present_files