Best Practices for Collecting Data and Choosing Instrumentation in Research

For effective research, it’s crucial to use high-quality, valid, and reliable methods for collecting data, transformation, and analysis (American Psychological Association, 2021; German Research Foundation, 2019). Instruments and protocols should be up-to-date, supported by literature, and provide evidence of reliability and validity (German Research Foundation, 2019; U.S. Institute for Education Sciences, 2022). Clear and understandable principles for data storage, protection, and privacy are essential for participant comprehension. Researchers should also follow the ethic of minimal burden by limiting questions to those directly related to the research questions, reducing participant workload and inconvenience (Scottish Educational Research Association, 2005), unless the study engages participants in research (e.g., action research; Stringer & Aragón, 2020).

Culturally Relevant Data Collection

Data collection methods should be culturally relevant and fit the situational context to better reflect racial and ethnic identities (Viano & Baker, 2020). Using self-reports as the main source of data can provide deeper insights. Integrating both quantitative and qualitative measures, such as open-response questions, can broaden the knowledge base and offer a more thorough understanding (Chouinard & Cousins, 2009; Garibay & Teasdale, 2019). Although qualitative self-reports are valuable, they can be challenging to analyze and aggregate, which should influence the design of the instruments.

Ensuring Validity and Reliability

Ensuring that instruments are valid and reliable in new and different contexts is essential (Decker & McGill, 2019). Validity confirms that an instrument measures what it intends to, while reliability assesses its consistency in similar contexts (Decker & McGill, 2019). Validity and reliability should consider minoritized identities and situational contexts to generate accurate inferences (Garibay & Teasdale, 2019).

Collecting Demographic Data

Demographic data collection must be handled with care, considering the wording and perception of items by different participants (Strunk & Hoover, 2019). For instance, avoiding binary gender selections and opting for options like “not listed” or “self-describe” can offer more nuanced data that accurately reflects participants’ identities. Only collecting data central to answering the research questions should be standard practice, highlighting the importance of well-formulated research questions that capture the studied populations. Allowing free-form responses, or using phrasing such as “If you had to choose one of the following options, which one most closely matches your [identity]?” with a multiple-choice option can provide more accurate self-identification (Strunk & Hoover, 2019). Factors such as country of origin, race/ethnicity, language, and prior experiences, especially with computing education, are also important to consider, as they can impact research outcomes (Viano & Baker, 2020).

Informing Participants About Data Collection

Participants must be clearly informed about the purpose of their participation, the required information, and how their data will be processed and reported (British Educational Research Association, 2018). They should also understand how their data will be retained, shared, and any potential secondary use (British Educational Research Association, 2018). Transparency in data de-identification procedures is crucial to ensure confidentiality (Natural Sciences and Engineering Research Council of Canada, 2022; Scottish Educational Research Association, 2005; Vetenskapsrådet, 2017). Additionally, participants should be able to withdraw consent for data storage and use where feasible. When providing incentives, researchers must use good judgment to avoid conflicts with participants’ values (Scottish Educational Research Association, 2005).

Data Sensitivity and Instrumentation Design

When designing instruments, norming assessments, or using secondary data, consider data sensitivity, the original purpose, and the intended audience (British Educational Research Association, 2018). Adherence to local laws and institutional best practices, including external ethics reviews, is essential for the collection, use, and protection of participant data (Scottish Educational Research Association, 2005; U.S. Institute for Education Sciences, 2022). Constructs should be equivalent across cultures and languages, which can be achieved by involving researchers or community members from similar backgrounds in the review of instrumentation before administration.

Reporting Findings to Participants and Tool Selection

Findings should be made available to participants, typically in the form of a report, whenever possible (U.S. Institute for Education Sciences, 2022). The choice of hardware and software tools for data collection must be made carefully, considering their potential unintended impacts such as accessibility, costs, and data privacy (Garibay & Teasdale, 2019). Audio and video recordings are particularly sensitive, and care must be taken to ensure recordings do not contain identifiable information. Using pseudonyms can help protect participant identity in recordings.

Data Privacy and Security

Ensuring data privacy and security is paramount. Online survey tools may collect data that could be commercialized, risking further marginalization of individuals or groups. Tools used must reflect the study’s context and culture (Garibay & Teasdale, 2019). Data storage and access must be strictly controlled to protect participant information.

Strategies for Crafting Instrumentation and Collecting Data

General

Respect participants’ diversity dimensions and lived experiences, and avoid using deficit-based framing and discriminatory language when collecting data from participants.

Check to see if instruments with evidence of validity or reliability already exist that may suit your needs before embarking on creating new instruments. When using existing instruments, however, consider the words that were used to collect diversity dimension data and whether inappropriate language or framings have been used. When using or updating any instrument, this may create a new need to collect evidence of reliability and validity.

Consider data sensitivity, original purpose, and intended audience when using secondary or tertiary data.

Consider how hardware and software tools for data collection may have unintended impacts (e.g., accessibility, costs, data privacy). 

Use good judgment when incentivising participants to partake in the study (.e.g, being too little or too much, offering incentives that might conflict with the values of the participants) (Scottish Educational Research Association, 2005).

Adhere to local laws and ethics review boards, as well as the ethic of minimal burden when appropriate to the research methodology.

Integrity and data privacy and storage

Create and follow policies and procedures for data collection, storage, use, disclosure, protection, and privacy (including allowing for participants to withdraw consent of their data where appropriate or possible and declare any incentives used during the data collection process).

Make evident how the participants’ privacy is protected, including data de-identification procedures and information regarding who will have access to the data (Natural Sciences and Engineering Research Council of Canada, 2022; Scottish Educational Research Association, 2005; Vetenskapsrådet, 2017).

Be accountable, and hold others accountable, for the ethical acquisition and use of data 

Comply with applicable statutes, regulations, practices, and ethical standards governing data collection and reporting.

Take all possible steps to protect confidentiality and anonymity of participant data.

Keep data secure by limiting who has access and ensure data protections are put into place and are followed.

Additional Resources

References

American Psychological Association. (2021). Journal Article Reporting Standards (JARS). Retrieved from https://apastyle.apa.org/jars

British Educational Research Association. 2018. Ethical Guidelines for Educational Research, fourth edition (2018) (4 ed.). British Educational Research Association. 1–48 pages.

Chouinard, J. A., & Cousins, J. B. (2009). A review and synthesis of current research on cross-cultural evaluation. American Journal of Evaluation, 30(4), 457–494.

Decker, A., & McGill, M. M. (2019, February). A topical review of evaluation instruments for computing education. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (pp. 558-564).

Garibay, C., & Teasdale, R. M. (2019). Equity and evaluation in informal STEM education. New Directions for Evaluation, 2019(161), 87–106.

German Research Foundation. 2019. Guidelines for Safeguarding Good Research Practice. https://www.htwg-konstanz.de/fileadmin/pub/forschung/Forschungsreferat/Guidelines_for_Safeguarding_Good_Researche_Practice_DFG__2019.pdf

Natural Sciences and Engineering Research Council of Canada. (2022). Guidelines on the assessment of contributions to research, training and mentoring. Retrieved from https://www.nserc-crsng.gc.ca/NSERC-CRSNG/Policies-Politiques/assessment_of_contributions-evaluation_des_contributions_eng.asp

Scottish Educational Research Association. 2005. Ethical Guidelines for Educational Research. Scottish Educational Research Association. 1–15 pages.

Stringer, E. T., & Aragón, A. O. (2020). Action research. Sage publications.

Strunk, K. K., & Hoover, P. D. (2019). Quantitative Methods for Social Justice and Equity: Theoretical and Practical Considerations. Research Methods for Social Justice and Equity in Education.

U.S. Institute for Education Sciences. 2022. Standards for Excellence in Education Research. https://ies.ed.gov/seer/index.asp

Vetenskapsrådet, S. (2017). Good research practice. Stockholm: Swedish Research Council.

Viano, S., & Baker, D. J. (2020). How administrative data collection and analysis can better reflect racial and ethnic identities. Review of Research in Education, 44(1), 301-331.