MERC: critical analysis of research papers
Critical appraisal is the systematic evaluation of research to judge trustworthiness, value, and relevance (Young & Solomon, 2009). The MERC framework, adapted from other critical appraisal methodologies (CASP, 2018), helps you evaluate research papers in speech tech. It asks four questions of each paper:
- Methodology: are the research methods appropriate and rigorous for addressing the research question?
- Evidence: how strong is the evidence presented? Are claims supported by the data?
- Relevance: how does this work connect to your research question?
- Contribution: what does this study add to the field?
We introduce MERC in Class 4 (Friday 12 February), and you use it in Checkpoint 2, part 1. The MERC analysis worksheet, with space for your notes on each paper, is in BrightSpace. The questions on the worksheet are below.
Paper details
- Citation
- Database/Source and date
- Your RQ
Methodology
Are the research methods appropriate and rigorous for addressing the research question?
Research design & appropriateness
- Is the overall research design appropriate for the RQ?
- Are there methodological limitations that affect interpretation?
Data collection & sampling
- Are data collection methods clearly described and appropriate?
- Is the sample size adequate? Is it representative?
- Are potential confounding variables controlled?
Speech technology-specific
- Are evaluation metrics appropriate for the task?
- Is the baseline comparison fair and current?
- Are language/dialect considerations addressed?
- Do methods reflect real-world usage patterns?
Evidence
How strong is the evidence presented? Are claims supported by the data?
Statistical rigor
- Are statistical tests appropriate and properly reported?
- Are effect sizes reported, as well as p-values?
- Is statistical power adequate?
Claims vs. data
- Are claims adequately supported by evidence?
- Are alternative explanations considered?
- Do authors acknowledge limitations appropriately?
Generalizability
- Can results generalize beyond the specific sample?
- Do computational results transfer to human performance?
- Are cross-linguistic claims supported by data?
Relevance
How does this work connect to your research question?
Connection to your research
- How does this paper relate to your RQ? (direct/tangential/methodological)
- What aspects are most relevant to your investigation?
- What aspects are less relevant?
Transferable insights
- What methodological insights can you apply to your own work?
- What theoretical frameworks might inform your research?
Contribution
What does this study add to the field?
Nature of contribution
- Significant (major advancement), incremental (modest step forward), or limited? Explain.
- What is novel about the approach or findings?
Impact & limitations
- What is the potential impact on the field?
- What limitations constrain the contribution?
- What future research directions does this suggest?
Synthesis & comparative analysis
After analyzing 2+ papers, compare methodological approaches and identify patterns:
- How do different methodological approaches yield different types of insights?
- Where do studies complement or contradict each other?
- What theoretical insights emerge from considering both studies together?
References
CASP (Critical Appraisal Skills Programme). (2018). CASP checklists. Oxford: CASP UK. Retrieved from https://casp-uk.net/casp-tools-checklists/
Greenhalgh, T. (2019). How to read a paper: The basics of evidence-based medicine and healthcare (6th ed.). Hoboken, NJ: Wiley-Blackwell.
Young, J. M., & Solomon, M. J. (2009). How to critically appraise an article. Nature Clinical Practice Gastroenterology & Hepatology, 6(2), 82-91.
Prepared for Research Design (MSc Speech Technology), University of Groningen