2 TWU Faculty Professional Learning

Colin Madland, Manager, Online Learning and Instructional Technology (TWU GLOBAL)

PhD Candidate, University of Victoria

Notes - https://bit.ly/twu-assessment

QR Code to access presentation notes. You can scan the QR code with your mobile phone, then send the tab to your desktop (at least in FireFox).

Presented Online for TWU Faculty Professional Learning, Thursday, March 9, 2023

I acknowledge that the land where I currently live and work remains the traditional, ancestral, and unceded land of the syilx (silks) people, whose historical stewardship of and connections to the land continue to today. I am grateful to be an uninvited guest on this land. To learn more, please visit the syilx.org.

My blind dog, Eleanor near the top of Mission Hill, overlooking syilx territory.

2.1 What is ‘assessment’?

QR Code to access Mentimeter

Pellegrino, J. W., Chudowsky, N., & Glaser, R. Knowing What Students Know: The Science and Design of Educational Assessment. National Academies Press.

“reasoning from evidence” (p. 43)

“a process of drawing reasonable inferences about what students know on the basis of evidence derived from observations of what they say, do, or make in selected situations” (p. 112)

And a quote usually ascribed to Paul Dressel at various times and in various publications…let me know if you find a verifiable source.

A grade is an inadequate report of an inaccurate judgment by a biased and variable judge of the extent to which a student has attained an undefined level of mastery of an unknown proportion of an indefinite material.

2.1.1 The Assessment Triangle

Assessment triangle showing the three components of assessment (Pellegrino, et al. 2001).

The process of assessment begins with a detailed understanding and map of the cognitive construct that is to be learned. This might be the ability to correctly calculate doses of medication, or transpose a piece of music, or write an argumentative essay. Below is an example of nursing competence from Weeks et al. (2019).

Drug dosage calculation competence from Weeks et al. (2019)

Here are two documents in use in TWU Nursing. While these aren’t constructed specifically as cognitive models of the domain, but instead as evaluation tools, the represent comprehesive models of what is required to demonstrate competency.

The second component of the assessment triangle is an instrument of some kind designed to elicit the competencies mapped in the cognitive model. The instrument can vary widely from selected-response tests, videos, podcasts, performance tests in a lab, and many more.

The final component of assessment is an inference or interpretation of the data generated by the assessment instrument (not a measurement). The accuracy of the interpretation depends on how well the instrument aligns with the cognitive model (validity) and how stable the results are across populations (reliability).

One important thing to note is that many higher ed instructors (outside of faculties of education or psychology) do not tend to have much formal preparation in educational assessment, however, higher education instructors do have extensive background and preparation for conducting research. The assessment triangle above has many parallels to research (cognitive model/literature review; instrument to gather data; interpretation/discussion and conclusions).

2.2 Thinking about your Approach to Assessment

Scenario 1

You are teaching a large enrollment course. The students will be submitting bi-weekly assignments, a midterm exam, and a culminating assignment all designed to support their learning.

  1. How would you approach grading the bi-weekly assignments?
  2. How would you use data from learners’ performance on the bi-weekly assignments?
  3. How would you respond to learners who have diverse abilities in relation to the culminating assignment?
  4. How would you deal with late assignments?

QR Code to access MentiMeter.

Scenario 2

A core assignment in your course involves students working in groups online.

  1. How would you ensure accountability and engagement with the assignment?
  2. How would you communicate feedback with the group?
  3. What factors would you consider when making grading decisions?
  4. How would you manage unexpected events that disrupt a group’s ability to complete the assignment?

QR Code to access MentiMeter.

Scenario 3

There are expectations in your department that grades should be distributed across the grading scale. However, your class averages are consistently lower than your colleagues’. Your course assessment scheme includes two term exams and one final exam.

  1. How might you incorporate assessment for learning into your assessment scheme?
  2. How might you design the exams to address the gap?
  3. What would be a fair approach to ‘catching up’ to your colleagues’ course grades?
  4. How might you analyze the exam scores to ensure reliability and validity?

QR Code to access MentiMeter.

Scenario 4

You teach a course with multiple sections taught by various instructors. Your students have complained to you that assignments are constructed and graded differently across sections.

  1. How might different purposes of assessment (…of/for/as learning) impact your response to learners?
  2. What design strategies could mitigate the perception (or reality) of inconsistent assignments across sections?
  3. What strategies could you use to ensure fairness across sections?
  4. How might you ensure that your assignments are ‘measuring’ the same things as your colleagues’ assignments?

QR Code to access MentiMeter.

Scenario 5

You discover that a student has plagiarized some of their assignment (e.g., an essay, lab report).

  1. How might your response change if the purpose of the assignment was one of assessment of/for/as learning?
  2. How might you adjust the design or deployment of the assignment to reduce plagiarism?
  3. What factors would you consider in deciding how to proceed with the student?
  4. How can you come to know what the student knows in relation to the learning outcomes?

QR Code to access MentiMeter.


2.3 A Framework for Thinking about Assessment

A Framework for Assessment as described below from DeLuca, et al. (2016)

DeLuca et al.’s model for assessment comprises four dimensions, each with three priority themes. Each of these dimensions influence the other dimensions, and all influence instructors’ approaches to assessment. The term ‘approaches to assessment’ is intended to convey the reality that assessment in formal education is tremendously complex and idiosyncratic. Assessment literacy is not a checklist of competencies, ‘best’ practices, or rules. The dimensions and their themes are listed below:

  • Assessment Purpose
    • …of learning
    • …for learning
    • …as learning
  • Assessment Processes
    • design
    • use and scoring
    • communication
  • Assessment Fairness
    • standard
    • equitable
    • balanced
  • Measurement Theory
    • consistent
    • balanced
    • contextual

References

DeLuca, C., LaPointe-McEwan, D., & Luhanga, U. (2016). Approaches to classroom assessment inventory: A new instrument to support teacher assessment literacy. Educational Assessment, 21, 248–266.

Pellegrino, J. W., Chudowsky, N., & Glaser, R. (2001). Knowing What Students Know: The Science and Design of Educational Assessment. National Academies Press.

Weeks, K. W., Coben, D., O’Neill, D., Jones, A., Weeks, A., Brown, M., & Pontin, D. (2019). Developing and integrating nursing competence through authentic technology-enhanced clinical simulation education: Pedagogies for reconceptualising the theory-practice gap. Nurse Education in Practice, 37, 29–38.