Reviewing Research Articles

We provide a checklist that can be used when reviewing articles (yours or others) to determine whether they include accepted educational research reporting practices. This list was developed through multiple discussions with expert researchers and evaluators within the computer science education community.

When reviewing articles, there is ample opportunity to grow the number of reviewers who have an equity lens to counteract the suppression of research conducted by various researchers (Marshall & Salter, 2022; Scantlebury, 2002). Reviewers can consider the explicit and implicit power dynamics (including conflicts of interest) between and among researchers and participants. Applying an equity lens when reviewing articles includes consideration of the study’s limitations as well as its context. By critically examining academic literature, researchers can find limitations that may serve as entry points for further research–or, at a minimum, may serve as a reminder to treat the results with healthy skepticism. This practice can ensure that new publications are focused on meeting the needs of various groups of learners.

Purpose, Goals, Intent, Clarity: Do the authors…

  • Make a case for why the reader should care about the problem?
  • Provide their contact information for the activity/study organizer/instructor/designer?
  • Clearly and explicitly state the research question(s) and hypothesis?
  • Clearly state the study’s objectives, including articulating any learning outcomes?
  • Use correct language related to educational researcher?
  • Provide any definitions used that are crucial to the study?
  • Specify the research question(s) the study sought to answer?

Study Design: Do the authors…

  • Indicate the research methodology used and the rationale for that choice?
  • Use an appropriate design related to its type of study?
  • Describe the methodology in sufficient detail for another researcher to replicate the study?
  • Describe the methodological framework (quantitative, qualitative, mixed methods) in terms of educational research? (Qualitative: case studies, ethnography, longitudinal, etc.; Quantitative: (quasi) experimental designs, survey, etc.)
  • Describe any efforts to offset the novelty effect, Hawthorne effect, John Henry effect?
  • Use and rigorously apply instruments appropriate to the research question?
  • Describe and provide the instruments used within the study?
  • Fully describe the setting for the study (location, classrooms, courses, schools)?
  • Use an appropriate instrument to measure impact?
  • Consider sample size and whether it is sufficient?

Activity/Intervention: Do the authors…

  • Fully describe the intervention and/or activities?
  • Explain how the activity is suitable to the targeted participant group (age/range/experience/etc.)?
  • Describe the skill, knowledge, or disposition that was being targeted?
  • Describe the length and frequency of the intervention (hours, days, months)?
  • Describe who conducted the intervention, including qualifications?

Ethics: Do the authors…

  • Disclose their IRB approval process and methods to ensure participant privacy, confidentiality, and protection?
  • Disclose any costs/funding sources to conduct any aspect of the research/activity in order to assess possible bias?
  • Disclose whether or not participants or researchers receive monetary or gift incentives?
  • Include researchers qualifications and how researcher bias has been mitigated?
  • Declare any personal, organizational, or institutional biases?

Participants: Do the authors…

  • Include participant demographic information, including age, grade range, gender, race/ethnicity, socioeconomic status?
  • Include number of participants in the study?
  • Include recruitment process for participants (volunteer? required?)?
  • Describe sampling technique used?

Data Analysis: Do the authors…

  • Indicate the analysis methods and tools used and the rationale for those choices?
  • Describe how the data analysis methods were appropriate for the design?
  • Fully describe the analysis methods with sufficient detail for replication?
  • For quantitative frameworks, describe all statistical tests used and a rationale for non-standard measures used? Include or provide a link to the raw assessment data for others to verify/analyze? Distinguish between correlation and causality?
  • For qualitative frameworks, describe how the data was analyzed, how inter-rater reliability was maintained, and provide researcher reflexivity statement?

Results: Do the authors…

  • Provide a compelling argument (sample size, quantitative or qualitative analysis, etc.) for the significance of its results?
  • Describe the results of the study?
  • Explore the implications of the results on research, policy, and practice?
  • Describe how this research and/or results fit into the larger context of related research?
  • Consider whether the results are appropriate for the scale of the intervention?
  • Describe limitations of the study, including issues related to ability to generalize, sample size, confounding variables, whether or not participants were randomized or not representative, with any alternative hypothesis stated?
  • Include data (sample size, statistical analysis, etc.) indicating its significance?

References

Marshall, S. L., & Salter, A. O. (2022). Moving beyond the boilerplate: Reflections on equity‐centered reviewing for granting organizations. Science Education, 106(5), 1264-1282.

Scantlebury, K. (2002). A snake in the nest or in a snake’s nest: What counts as peer review for a female science educator in a chemistry department?. Research in Science Education, 32, 157-162.

Citing this Resource

McGill, M. M., & Decker, A. (2018, October). Defining Requirements for a Repository to Meet the Needs of K-12 Computer Science Educators, Researchers, and Evaluators. In 2018 IEEE Frontiers in Education Conference (FIE) (pp. 1-9). IEEE.

or

@inproceedings{mcgill2018defining,
title={Defining Requirements for a Repository to Meet the Needs of K-12 Computer Science Educators, Researchers, and Evaluators},
author={McGill, Monica M and Decker, Adrienne},
booktitle={2018 IEEE Frontiers in Education Conference (FIE)},
pages={1–9},
year={2018},
organization={IEEE}
}