Reporting of Educational Research
Provided here is a meaningful checklist that may be used to evaluate articles (yours or others) to determine whether they follow standard educational research practices. This list was developed through multiple discusses with six expert researchers and evaluators within the computer science education community.
Purpose, Goals, Intent, Clarity: Do the authors…
- Make a case for why the reader should care about the problem?
- Provide their contact information for the activity/study organizer/instructor/designer?
- Clearly and explicitly state the research question(s) and hypothesis?
- Clearly state the study’s objectives, including articulating any learning outcomes?
- Use correct language related to educational researcher?
- Provide any definitions used that are crucial to the study?
- Specify the research question(s) the study sought to answer?
Study Design: Do the authors…
- Indicate the research methodology used and the rationale for that choice?
- Use an appropriate design related to its type of study?
- Describe the methodology in sufficient detail for another researcher to replicate the study?
- Describe the methodological framework (quantitative, qualitative, mixed methods) in terms of educational research? (Qualitative: case studies, ethnography, longitudinal, etc.; Quantitative: (quasi) experimental designs, survey, etc.)
- Describe any efforts to offset the novelty effect, Hawthorne effect, John Henry effect?
- Use and rigorously apply instruments appropriate to the research question?
- Describe and provide the instruments used within the study?
- Fully describe the setting for the study (location, classrooms, courses, schools)?
- Use an appropriate instrument to measure impact?
- Consider sample size and whether it is sufficient?
Activity/Intervention: Do the authors…
- Fully describe the intervention and/or activities?
- Explain how the activity is suitable to the targeted participant group (age/range/experience/etc.)?
- Describe the skill, knowledge, or disposition that was being targeted?
- Describe the length and frequency of the intervention (hours, days, months)?
- Describe who conducted the intervention, including qualifications?
Ethics: Do the authors…
- Disclose their IRB approval process and methods to ensure participant privacy, confidentiality, and protection?
- Disclose any costs/funding sources to conduct any aspect of the research/activity in order to assess possible bias?
- Disclose whether or not participants or researchers receive monetary or gift incentives?
- Include researchers qualifications and how researcher bias has been mitigated?
- Declare any personal, organizational, or institutional biases?
Participants: Do the authors…
- Include participant demographic information, including age, grade range, gender, race/ethnicity, socioeconomic status?
- Include number of participants in the study?
- Include recruitment process for participants (volunteer? required?)?
- Describe sampling technique used?
Data Analysis: Do the authors…
- Indicate the analysis methods and tools used and the rationale for those choices?
- Describe how the data analysis methods were appropriate for the design?
- Fully describe the analysis methods with sufficient detail for replication?
- For quantitative frameworks, describe all statistical tests used and a rationale for non-standard measures used? Include or provide a link to the raw assessment data for others to verify/analyze? Distinguish between correlation and causality?
- For qualitative frameworks, describe how the data was analyzed, how inter-rater reliability was maintained, and provide researcher reflexivity statement?
Results: Do the authors…
- Provide a compelling argument (sample size, quantitative or qualitative analysis, etc.) for the significance of its results?
- Describe the results of the study?
- Explore the implications of the results on research, policy, and practice?
- Describe how this research and/or results fit into the larger context of related research?
- Consider whether the results are appropriate for the scale of the intervention?
- Describe limitations of the study, including issues related to ability to generalize, sample size, confounding variables, whether or not participants were randomized or not representative, with any alternative hypothesis stated?
- Include data (sample size, statistical analysis, etc.) indicating its significance?