1887

Applying a Computer-Assisted Tool for Semantic Analysis of Writing: Uses for STEM and ELL

    Authors: Beverly L. Smith-Keiling1,*, Hye In F. Hyun1
    VIEW AFFILIATIONS HIDE AFFILIATIONS
    Affiliations: 1: University of Minnesota Medical School and College of Biological Sciences, Department of Biochemistry, Molecular Biology, and Biophysics, Minneapolis, MN 55455
    AUTHOR AND ARTICLE INFORMATION AUTHOR AND ARTICLE INFORMATION
    Source: J. Microbiol. Biol. Educ. April 2019 vol. 20 no. 1 doi:10.1128/jmbe.v20i1.1709
MyBook is a cheap paperback edition of the original book and will be sold at uniform, low price.
  • HTML
    38.36 Kb
  • XML
    45.57 Kb
  • PDF
    1.44 MB

    Abstract:

    In addition to human, close reading of student text with rubrics for assessment, educators use nonhuman, distant computer-assisted tools to help quantitatively measure otherwise qualitative keywords to prevent bias in grading and help for underlying cognitions. We apply the Linguistic Inquiry and Word Count (LIWC) software tool to analyze different forms of student writing used in STEM education and research to assess writing of native English speakers and non-native English Language Learners (ELLs), including international students. Available in several languages, LIWC measures four summary variables, , , , and , to provide outputs as raw word counts, as percentages of words used relative to the text compared with a dictionary of words in categories and sub-dictionaries, and as scores correlating these words algorithmically based on a dictionary of terms associated with underlying meanings. This tool can help measure student personal reflective writing for underlying psychosocial indicators or the cognitive and analytical process in other science writing. By selecting key variables, or creating a personal dictionary, LIWC can be used to analyze scientific writing to detect progressive development of student analytical writing from early draft to final version for different informal and formal writing styles. We share methods, examples, and the potential for using LIWC measures of cognitive processes for different measures of student writing in science courses.

References & Citations

1. Pennebaker JW, Booth RJ, Boyd RL, Francis ME 2015 Linguistic inquiry and word count: LIWC2015 Pennebaker Conglomerates (LIWC.net) Austin, TX
2. Deane P 2011 Writing assessment and cognition (ETS Research Report RR-11-14) Educational Testing Service Princeton, NJ
3. Deane P 2013 On the relation between automated essay scoring and modern views of the writing construct Assess Writing 18 1 7 24 10.1016/j.asw.2012.10.002 http://dx.doi.org/10.1016/j.asw.2012.10.002
4. Jonsson A, Svingby G 2007 The use of scoring rubrics: reliability, validity and educational consequences Educ Res Rev 2 130 144 10.1016/j.edurev.2007.05.002 http://dx.doi.org/10.1016/j.edurev.2007.05.002
5. Tausczik Y, Pennebaker J 2010 The psychological meaning of words: LIWC and computerized text analysis methods J Lang Soc Psychol 29 1 24 54 10.1177/0261927X09351676 http://dx.doi.org/10.1177/0261927X09351676
6. Francis ME, Pennebaker JW 1992 Putting stress into words: the impact of writing on physiological, absentee, and self-reported emotional well-being measures Am J Health Promot 6 4 280 287 10.4278/0890-1171-6.4.280 10146806 http://dx.doi.org/10.4278/0890-1171-6.4.280
7. Pennebaker JW, Boyd RL, Jordan K, Blackburn K 2015 The development and psychometric properties of LIWC2015 University of Texas at Austin Austin, TX
8. Smith-Keiling BL, Swanson LK, Dehnbostel JM 2018 Interventions for supporting and assessing science writing communication: cases of Asian English language learners J Microbiol Biol Educ 19 1 10.1128/jmbe.v19i1.1522 29904544 5969430 http://dx.doi.org/10.1128/jmbe.v19i1.1522
9. Tov W, Ng KL, Lin H, Qiu L 2013 Detecting well-being via computerized content analysis of brief diary entries Psych Assess 25 4 1069 10.1037/a0033007 http://dx.doi.org/10.1037/a0033007
10. Lepore SJ, Smyth JM 2002 The writing cure: how expressive writing promotes health and emotional well-being American Psychological Association Washington, DC
11. Polat B 2014 Words of experience: semantic content analysis and individual differences among successful second language learners PhD thesis Georgia State University Atlanta, GAhttp://scholarworks.gsu.edu/alesl_diss/28.
12. Pennebaker JW 2013 The secret life of pronouns: what our words say about us Bloomsbury Press New York, NY
13. Pennebaker JW, Chung CK 2007 Expressive writing, emotional upheavals and health 263 284 Friedman HS, Cohen Silver R Foundations of health psychology Oxford University Press New York, NY
14. Pennebaker JW, Chung CK 2012 Expressive writing: connections to physical and mental health 417 439 Friedman HS The Oxford handbook of health psychology Oxford University Press New York, NY
15. Allwright D, Hanks J 2009 The developing language learner: an introduction to EP Palgrave Macmillan UK Basingstoke, UK 10.1057/9780230233690 http://dx.doi.org/10.1057/9780230233690
16. Dörnyei Z 2009 Individual differences: interplay of learner characteristics and learning environment Lang Learn 59 Suppl 1 230 248 10.1111/j.1467-9922.2009.00542.x http://dx.doi.org/10.1111/j.1467-9922.2009.00542.x
17. Friginal E, Lee JJ, Polat B, Roberson A 2017 Exploring spoken English learner language using corpora 3 33 Exploring spoken English learner language using corpora: learner talk Palgrave Macmillan Cham, London 10.1007/978-3-319-59900-7_1 http://dx.doi.org/10.1007/978-3-319-59900-7_1
18. Schiffrin D, Tannen D, Hamilton HE 2008 The handbook of discourse analysis Blackwell Publishers Malden, MA, USA, and Oxford, UK

Supplemental Material

Loading

Article metrics loading...

/content/journal/jmbe/10.1128/jmbe.v20i1.1709
2019-04-26
2019-08-18

Abstract:

In addition to human, close reading of student text with rubrics for assessment, educators use nonhuman, distant computer-assisted tools to help quantitatively measure otherwise qualitative keywords to prevent bias in grading and help for underlying cognitions. We apply the Linguistic Inquiry and Word Count (LIWC) software tool to analyze different forms of student writing used in STEM education and research to assess writing of native English speakers and non-native English Language Learners (ELLs), including international students. Available in several languages, LIWC measures four summary variables, , , , and , to provide outputs as raw word counts, as percentages of words used relative to the text compared with a dictionary of words in categories and sub-dictionaries, and as scores correlating these words algorithmically based on a dictionary of terms associated with underlying meanings. This tool can help measure student personal reflective writing for underlying psychosocial indicators or the cognitive and analytical process in other science writing. By selecting key variables, or creating a personal dictionary, LIWC can be used to analyze scientific writing to detect progressive development of student analytical writing from early draft to final version for different informal and formal writing styles. We share methods, examples, and the potential for using LIWC measures of cognitive processes for different measures of student writing in science courses.

Highlighted Text: Show | Hide
Loading full text...

Full text loading...

/deliver/fulltext/jmbe/20/1/jmbe-20-18.html?itemId=/content/journal/jmbe/10.1128/jmbe.v20i1.1709&mimeType=html&fmt=ahah

Figures

Image of Untitled

Click to view

Untitled

Source: J. Microbiol. Biol. Educ. April 2019 vol. 20 no. 1 doi:10.1128/jmbe.v20i1.1709
Download as Powerpoint
Image of FIGURE 1

Click to view

FIGURE 1

Screenshot of sample Linguistic Inquiry and Word Count (LIWC) output. LIWC analysis of student Biochemical (BQA) draft questions Q (reflective) and answers A (scientific) writing. Export to Excel shows left column filenames or de-identified student numbers from a file opened in LIWC software. Top row sample categories are selected based on the desired writing style analysis. (WC), and summary variables , and are raw data or algorithmically determined and useful for many writing styles. Some variables help determine the complexity of writing: (WPS), (Sixltr). Categories are percentage scores of the number of words from the text relative to total word count, e.g., , , or other personal pronouns are used for different purposes in different styles of writing, such as determining formality in scientific writing style. Categories are algorithmically nested under summary variables, e.g., summary variable comprises , , , and others, according to the LIWC dictionary. summary variable comprises , and others. A variety of punctuation, e.g., , etc., are useful for tracking scientific writing formality. We define formality of scientific writing per our grading rubric as having these features: zero to low personal pronouns, no quotes, no contractions, no apostrophes except the four expected for two 5′ and 3′ DNA primer ends in a section. Scores matched hand-graded appropriate use of parentheses for defined abbreviations, citations, and chemical names but were not overused in layperson writing with increased definition of scientific terminology. “” (0.00) was consistent with BQA scientific answers A, whereas questions Q had allowable pronouns in the reflective style. Higher use of quotes, apostrophes, and parentheses was detected with less formal writing such as use of contractions (“Conc’t sample”), which was found in other writing samples and corrected upon later rewrite of draft (BQA1D, BQA2D, BQA3D) to final versions (BQA1F, BQA2F, BQAS3F). LIWC scores were matched to hand-graded counts by two independent raters and reviewed by an external evaluator, with >95% agreement, and two additional independent in-class graders for comparison, quantitatively assessing levels of , , and reasoning and thought ( 8 , Appendix 1 examples).

Source: J. Microbiol. Biol. Educ. April 2019 vol. 20 no. 1 doi:10.1128/jmbe.v20i1.1709
Download as Powerpoint
Image of FIGURE 2

Click to view

FIGURE 2

Visual graphical sample for comparison. Once data are graphed, detected patterns are more easily seen between the Biochemical (BQA) Question Q part, which is reflective and poses a question with more personal pronouns “”, and the Answer A part, which is formal scientific writing and has a higher score. Some progressive improvement can be detected from drafts (BQA1D, BQA2D, BQA3D) to their final versions (BQA1F, BQA2F, BQAS3F), with higher scores algorithmically detecting more formality in the scientifically written answer A section and with some detectable increases in total scores, e.g., BQA2 improved from 86 to 87, and BQA3 improved from 87 to 89. as a measure of confidence is higher in some question sections than others, demonstrating that confidence can vary per different topics in the reflectively written question Q section. While per percentage words in the dictionary as an indicator of confidence can be useful in psychosocial research studies, it is not useful in a written rubric for grading. Variables are not used for all genres, e.g., psychosocial indicator is not used in the Answer portion, which is scientific and not reflective writing. These computer-generated scores were matched with hand-grading and visual inspection by the two authors. Examples are provided in Appendix 1 .

Source: J. Microbiol. Biol. Educ. April 2019 vol. 20 no. 1 doi:10.1128/jmbe.v20i1.1709
Download as Powerpoint

This is a required field
Please enter a valid email address
Please check the format of the address you have entered.
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error