1887

Development of a Tool to Assess Interrelated Experimental Design in Introductory Biology

    Authors: Tess L. Killpack1,‡,*, Sara M. Fulmer2,‡
    VIEW AFFILIATIONS HIDE AFFILIATIONS
    Affiliations: 1: Biology Department, Salem State University, Salem, MA 01970; 2: Open Learning and Educational Support, University of Guelph, Guelph, ON N1G 2W1
    AUTHOR AND ARTICLE INFORMATION AUTHOR AND ARTICLE INFORMATION
    • Received 14 May 2018 Accepted 14 September 2018 Published 31 October 2018
    • ©2018 Author(s). Published by the American Society for Microbiology.
    • [open-access] This is an Open Access article distributed under the terms of the Creative Commons Attribution-Noncommercial-NoDerivatives 4.0 International license (https://creativecommons.org/licenses/by-nc-nd/4.0/ and https://creativecommons.org/licenses/by-nc-nd/4.0/legalcode), which grants the public the nonexclusive right to copy, distribute, or display the published work.

    • Supplemental materials available at http://asmscience.org/jmbe
    • *Corresponding author. Mailing address: Biology Department, Salem State University, 352 Lafayette Street, Salem, MA 01970. Phone: 978-542-3085. E-mail: [email protected].
    • These authors contributed equally to this work.
    Source: J. Microbiol. Biol. Educ. October 2018 vol. 19 no. 3 doi:10.1128/jmbe.v19i3.1627
MyBook is a cheap paperback edition of the original book and will be sold at uniform, low price.
  • PDF
    566.15 Kb
  • HTML
    70.63 Kb
  • XML
    80.12 Kb

    Abstract:

    Designing experiments and applying the process of science are core competencies for many introductory courses and course-based undergraduate research experiences (CUREs). However, experimental design is a complex process that challenges many introductory students. We describe the development of a tool to assess interrelated experimental design (TIED) in an introductory biology lab course. We describe the interrater reliability of the tool, its effectiveness in detecting variability and growth in experimental-design skills, and its adaptability for use in various contexts. The final tool contained five components, each with multiple criteria in the form of a checklist such that a high-quality response—in which students align the different components of their experimental design—satisfies all criteria. The tool showed excellent interrater reliability and captured the full range of introductory-student skill levels, with few students hitting the assessment ceiling or floor. The scoring tool detected growth in student skills from the beginning to the end of the semester, with significant differences between pre- and post-assessment scores for the Total Score and for the Data Collection and Observations component scores. This authentic assessment task and scoring tool provide meaningful feedback to instructors about the strengths, gaps, and growth in introductory students’ experimental-design skills and can be scored reliably by multiple instructors. The TIED can also be adapted to a number of experimental-design prompts and learning objectives, and therefore can be useful for a variety of introductory courses and CUREs.

References & Citations

1. American Association for the Advancement of Science 2011 Vision and change in undergraduate biology education: a call to action: a summary of recommendations made at a national conference organized by the American Association for the Advancement of Science July 15–17, 2009 Washington DC http://visionandchange.org/files/2011/03/Revised-Vision-and-Change-Final-Report.pdf
2. Coil D, Wenderoth MP, Cunnigham M, Dirks C 2010 Teaching the process of science: faculty perceptions and an effective methodology CBE Life Sci Educ 10.1187/cbe.10-01-0005 21123699 2995770 http://dx.doi.org/10.1187/cbe.10-01-0005
3. Jeffery E, Nomme K, Deane T, Pollock C, Birol G 2016 Investigating the role of an inquiry-based biology lab course on student attitudes and views toward science CBE Life Sci Educ 15 ar61 10.1187/cbe.14-11-0203 27856549 5132358 http://dx.doi.org/10.1187/cbe.14-11-0203
4. Shortlidge EE, Bangera G, Brownell SE 2016 Faculty perspectives on developing and teaching course-based undergraduate research experiences BioScience 66 54 62 10.1093/biosci/biv167 http://dx.doi.org/10.1093/biosci/biv167
5. Shortlidge EE, Brownell SE 2016 How to assess your CURE: a practical guide for instructors of course-based undergraduate research experiences J Microbiol Biol Educ 17 3 399 408 10.1128/jmbe.v17i3.1103 http://dx.doi.org/10.1128/jmbe.v17i3.1103
6. Dasgupta AP, Anderson TR, Pelaez N 2014 Development and validation of a rubric for diagnosing students’ experimental design knowledge and difficulties CBE Life Sci Educ 13 265 284 10.1187/cbe.13-09-0192 4041504 http://dx.doi.org/10.1187/cbe.13-09-0192
7. Brownell SE, Wenderoth MP, Theobald R, Okoroafor N, Koval M, Freeman S, Walcher-Chevillet CL, Crowe AJ 2014 How students think about experimental design: novel conceptions revealed by in-class activities BioScience 64 2 125 137 10.1093/biosci/bit016 http://dx.doi.org/10.1093/biosci/bit016
8. Deane T, Nomme K, Jeffery E, Pollock C, Birol G 2014 Development of the biological experimental design concept inventory (BEDCI) CBE Life Sci Educ 13 3 540 551 10.1187/cbe.13-11-0218 25185236 4152214 http://dx.doi.org/10.1187/cbe.13-11-0218
9. Shi J, Power JM, Klymkowsky MW 2011 Revealing student thinking about experimental design and the roles of control experiments Int J Scholarsh Teach Learn 5 1 16
10. Cooper KM, Soneral PAG, Brownell SE 2017 Define your goals before you design a CURE: a call to use backward design in planning course-based undergraduate research experiences J Microbiol Biol Educ 18 2 1 7 10.1128/jmbe.v18i2.1287 http://dx.doi.org/10.1128/jmbe.v18i2.1287
11. National Research Council 2003 BIO2010: Transforming undergraduate education for future research biologists The National Academies Press Washington, DC 10.17226/10497 http://dx.doi.org/10.17226/10497
12. Wiggins G 1998 Ensuring authentic performance 21 42 Educative assessment: designing assessments to inform and improve student performance Jossey-Bass San Francisco, CA
13. Oh DM, Kim JM, Garcia RE, Krilowicz BL 2005 Valid and reliable authentic assessment of culminating student performance in the biomedical sciences Adv Physiol Ed 29 2 83 93 10.1152/advan.00039.2004 http://dx.doi.org/10.1152/advan.00039.2004
14. Kloser MJ, Brownell SE, Chiariello NR, Fukami T 2011 Integrating teaching and research in undergraduate biology laboratory education PLOS Biol 9 11 e1001174 10.1371/journal.pbio.1001174 22110400 3216991 http://dx.doi.org/10.1371/journal.pbio.1001174
15. Dogan A, Kaya ON 2009 Poster sessions as an authentic assessment approach in an open-ended university general chemistry laboratory Procedia Soc Behavior Sci 1 1 829 833 10.1016/j.sbspro.2009.01.148 http://dx.doi.org/10.1016/j.sbspro.2009.01.148
16. Laungani R, Tanner C, Brooks TD, Clement B, Clouse M, Doyle E, Dworak S, Elder B, Marley K, Schofield B 2018 Finding some good in an invasive species: introduction and assessment of a novel CURE to improve experimental design in undergraduate biology classrooms J Microbiol Biol Educ 19 2 19.2.68 10.1128/jmbe.v19i2.1517 29983845 6022745 http://dx.doi.org/10.1128/jmbe.v19i2.1517
17. Makarevitch I, Frechette C, Wiatros N 2015 Authentic research experience and “big data” analysis in the classroom: maize response to abiotic stress CBE Life Sci Educ 14 3 ar27 10.1187/cbe.15-04-0081 26163561 4710385 http://dx.doi.org/10.1187/cbe.15-04-0081
18. Dolan EL, Collins JP 2015 We must teach more effectively: here are four ways to get started Mol Biol Cell 26 12 2151 2155 10.1091/mbc.e13-11-0675 26067566 4462934 http://dx.doi.org/10.1091/mbc.e13-11-0675
19. Jordan TC, Burnett SH, Carson S, Caruso SM, Clase K, DeJong RJ, Dennehy JJ, Denver DR, Dunbar D, Elgin SCR, Findley AM, Gissendanner CR, Golebiewska UP, Guild N, Hartzog GA, Grillo WH, Hollowell GP, Hughes LE, Johnson A, King RA, Lewis LO, Li W, Rosenzweig F, Rubin MR, Saha MS, Sandoz J, Shaffer CD, Taylor B, Temple L, Vazquez E, Ware VC, Barker LP, Bradley KW, Jacobs-Sera D, Pope WH, Russell DA, Cresawn SG, Lopatto D, Bailey CP, Hatfull GF 2014 A broadly implementable research course in phage discovery and genomics for first-year undergraduate students mBio 5 1 e01051-13 10.1128/mBio.01051-13 24496795 3950523 http://dx.doi.org/10.1128/mBio.01051-13
20. Dirks C, Cunningham M 2006 Enhancing diversity in science: is teaching science process skills the answer? CBE Life Sci Educ 5 218 226 10.1187/cbe.05-10-0121 17012213 1618688 http://dx.doi.org/10.1187/cbe.05-10-0121
21. Sirum K, Humburg J 2011 The experimental design ability test (EDAT) BioScience 37 1 8 16
22. Brookhart SM 2013 How to create and use rubrics for formative assessment and grading ASCD Alexandria, VA
23. Peat B 2006 Integrating writing and research skills: development and testing of a rubric to measure student outcomes J Public Aff Educ 12 3 295 311 10.1080/15236803.2006.12001437 http://dx.doi.org/10.1080/15236803.2006.12001437
24. Andrade HG 2000 Using rubrics to promote thinking and learning Educ Leadersh 57 13 18
25. Kishbaugh TLS, Cessna S, Horst SJ, Leaman L, Flanagan F, Neufeld DG, Siderhurst M 2012 Measuring beyond content: a rubric bank for assessing skills in authentic research assignments in the sciences Chem Educ Res Pract 13 268 276 10.1039/C2RP00023G http://dx.doi.org/10.1039/C2RP00023G
26. Science Olympiad 2017 Experimental design checklist for B/C Retrieved from https://www.soinc.org/sites/default/files/uploaded_files/ExDesChecklist17Final.pdf
27. Hallgren KA 2012 Computing interrater reliability for observational data: an overview and tutorial Tutor Quantitat Meth Psychol 8 23 34 10.20982/tqmp.08.1.p023 http://dx.doi.org/10.20982/tqmp.08.1.p023
28. Mandrekar J 2011 Measures of interrater agreement Biostat Clinic 6 1 6 7
29. Cicchetti DV 1994 Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology Psychol Assess 6 284 290 10.1037/1040-3590.6.4.284 http://dx.doi.org/10.1037/1040-3590.6.4.284

Supplemental Material

Loading

Article metrics loading...

/content/journal/jmbe/10.1128/jmbe.v19i3.1627
2018-10-31
2019-05-21

Abstract:

Designing experiments and applying the process of science are core competencies for many introductory courses and course-based undergraduate research experiences (CUREs). However, experimental design is a complex process that challenges many introductory students. We describe the development of a tool to assess interrelated experimental design (TIED) in an introductory biology lab course. We describe the interrater reliability of the tool, its effectiveness in detecting variability and growth in experimental-design skills, and its adaptability for use in various contexts. The final tool contained five components, each with multiple criteria in the form of a checklist such that a high-quality response—in which students align the different components of their experimental design—satisfies all criteria. The tool showed excellent interrater reliability and captured the full range of introductory-student skill levels, with few students hitting the assessment ceiling or floor. The scoring tool detected growth in student skills from the beginning to the end of the semester, with significant differences between pre- and post-assessment scores for the Total Score and for the Data Collection and Observations component scores. This authentic assessment task and scoring tool provide meaningful feedback to instructors about the strengths, gaps, and growth in introductory students’ experimental-design skills and can be scored reliably by multiple instructors. The TIED can also be adapted to a number of experimental-design prompts and learning objectives, and therefore can be useful for a variety of introductory courses and CUREs.

Highlighted Text: Show | Hide
Loading full text...

Full text loading...

/deliver/fulltext/jmbe/19/3/jmbe-19-98.html?itemId=/content/journal/jmbe/10.1128/jmbe.v19i3.1627&mimeType=html&fmt=ahah

Figures

Image of FIGURE 1

Click to view

FIGURE 1

TIED: A scoring tool for interrelated experimental design.

Source: J. Microbiol. Biol. Educ. October 2018 vol. 19 no. 3 doi:10.1128/jmbe.v19i3.1627
Download as Powerpoint
Image of FIGURE 2

Click to view

FIGURE 2

Histograms representing the number of students achieving each total TIED score on the pre- and post-assessment.

Source: J. Microbiol. Biol. Educ. October 2018 vol. 19 no. 3 doi:10.1128/jmbe.v19i3.1627
Download as Powerpoint

This is a required field
Please enter a valid email address
Please check the format of the address you have entered.
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error