Designing a Curriculum-Aligned Assessment of Cumulative Learning about Marine Primary Production to Improve an Undergraduate Marine Sciences Program †

FIGURE 1

Flowchart describing general assessment development process. Potential misunderstandings about marine primary production were identified from existing first-year student homework and then mapped onto a concept map describing the implemented curriculum regarding marine primary production; assessment questions were drafted to target key topics for which we had evidence of some misunderstandings. Faculty experts guided question development, and some questions were revised based on student performance and comments following a pilot deployment. See Appendix 1 for details. MPP = marine primary production.

FIGURE 2

Rasch analysis results. (a) Item map showing how each survey question varied in difficulty (y-axis) and how well it fit the Rasch model (x-axis). Question difficulty increases with vertical logit value. The vertical size of each symbol shows the standard error of the estimated difficulty. (b) Person map showing each student’s ability on the assessment (y-axis) and how well each student’s responses fit the Rasch model (x-axis). Students with more ability had larger vertical ability values. To aid readability, error estimates are not shown by symbol size. The average standard error of estimated ability was 0.65 (±0.11 SD). For both maps, the fit of the questions/students to the model is indicated by their position on the x-axis. Items/students further from 0 fit the model less well. The vertical lines at −2 and 2 infit t values indicate the acceptable range for items fitting the Rasch model.

FIGURE 3

Mean assessment scores. Mean person parameter logit values in each SMS educational level. Larger values (less negative) indicate higher performance on the assessment. Error bars show standard error. Number of students in each educational level was level 1 = 20, level 2 = 17, level 3 = 34 and level 4 = 15. The mean score at level 1 was significantly different than all other levels (Tukey’s LSD, α = 0.1). LSD = least significant difference; SMS = School of Marine Sciences.

FIGURE 4

Relationship between assessment difficulty and cognitive demand of assessment questions. The linear correlation of mean logit assessment difficulty for each question is pooled over all SMS educational levels and the cognitive demands of each question as categorized by Bloom’s Taxonomy levels (R2 = 0.37, p = 0.028). Note that the signs of item difficulty values have been inverted to emphasize increased difficulty of questions with higher Bloom’s levels. SMS = School of Marine Sciences.

FIGURE 5

Relationship between self-reported response confidence and assessment score. The graph indicates a linear relationship between categorical mean self-reported question response confidence and overall assessment score over all questions and SMS educational levels. Each symbol represents a unique student (n = 86). The least-squares regression line is plotted (Pearson correlation, R2 = 0.00, α = 0.1, p = 0.956). SMS = School of Marine Sciences.

FIGURE 6

Self-reported confidence for survey Question 1. Responses are pooled over all educational levels, with confidence in each item indicated by color. The question was: The majority of actual weight (dry biomass) gained by open-ocean photosynthetic marine algae as they grow comes from which of the following substances? The possible responses were:

1. Molecules containing carbon. (correct answer)
2. Particulate matter.
3. Dissolved forms of nitrogen.
4. Energy from the sun.

FIGURE 7

Relationship between question cognitive demand and mean student confidence. The graph shows a linear relationship between Bloom’s Taxonomy level for each question and categorical mean self-reported question response confidence pooled over all educational levels. The least-squares regression line is plotted (Pearson correlation, R2 = 0.24, α = 0.1, p = 0.089).