1 OTESSA22 - Assessment and Digital Technology in Higher Education

Introduction

Colin Madland, PhD Candidate, University of Victoria

Slides - https://bit.ly/otessa22-b
Find me on the web…
Twitter
Mastodon

Presented Online at OTESSA22, May 17, 2022

I acknowledge that the land where I currently live and work remains the traditional, ancestral, and unceded land of the syilx (silks) people, whose historical stewardship of and connections to the land continue to today. I am grateful to be an uninvited guest on this land. To learn more, please visit the Westbank First Nation website.

Picture of a bicycle resting against a pole along a trail in Kalamoir Park overlooking Okanagan Lake. There are many yellow Arrowleaf Balsamroot flowers on the hillside.

Figure 1. Author’s bicycle overlooking Okanagan Lake.

Hypothes.is

If you haven’t already, feel free to sign up here as we will use hypothes.is later. Also, if you have questions or comments, please annotate to your heart’s content!

Background

This review is guided by four research questions:

What are the major themes or patterns in the literature related to approaches to assessment in higher education?
What are the major themes or patterns in the literature related to the impact of technology on assessment in higher education?
What gaps exist in the literature related to approaches to assessment in technology-mediated higher education?

Scriven, 1967

Scriven, M. (1967). The methodology of evaluation. In B. O. Smith (Ed.), Perspectives of curriculum evaluation. Rand McNally

distinction between formative and summative

Bloom, 1968

Bloom, B. (1968). Learning for Mastery. Instruction and Curriculum. Regional Education Laboratory for the Carolinas and Virginia, Topical Papers and Reprints, Number 1. Evaluation Comment, 1(2), 12.

Incorporated formative and summative distinction into his ideas about mastery learning

Mislevy, 1994

Mislevy, R. J. (1994). Test theory reconcieved. ETS Research Report Series, 1994(1), i–38. https://doi.org/10/gjm236

test theory is machinery for reasoning from students’ behavior to conjectures about their competence, as framed in a particular conception of competence.”(p. 4).

Black and Wiliam, 1998

Black, P., & Wiliam, D. (1998). Assessment and Classroom Learning. Assessment in Education: Principles, Policy & Practice, 5(1), 7–74. https://doi.org/10/fpnss4

major review of the literature on formative assessment
describe formative assessment as encouraging gains in achievement that were
```
> among the largest ever reported for educational interventions (p. 61)
```

Pellegrino et al., 2001

Pellegrino, J. W., Chudowsky, N., & Glaser, R. (2001). Knowing What Students Know: The Science and Design of Educational Assessment. National Academies Press. https://doi.org/10.17226/10019

“a process of drawing reasonable inferences about what students know on the basis of evidence derived from observations of what they say, do, or make in selected situations” (p. 112)
“reasoning from evidence” (p. 43)

Assessment Triangle

Figure 2. Assessment Triangle from Pellegrino et al. (2001)

Cognition

a cognitive model of the domain

Observation

a performance task used to gather data regarding learner achievement

Interpretation

an inference or judgement of the learner’s achievement in relation to the model of the domain

Approaches to Learning

Biggs, 1993

Biggs, J. B. (1993). From Theory to Practice: A Cognitive Systems Approach. Higher Education Research & Development, 12(1), 73–85. https://doi.org/10/ccdmd9

Diagram showing three components of the 3-P Model of Teaching and Learning, described below.

Figure 3. 3-P Model of Teaching and Learning adapted from Biggs (1993)

Presage

factors that precede learning activities
- learner factors
  - prior knowledge
  - educational experience
  - affective states
  - wellness (physical & mental)
- teacher factors
  - vertical & horizontal discourses (Bernstein, 1999)
  - institutional policies
  - department norms
  - educational experiences

Process

learning focused activities
- reading, writing, discussing, building, creating, synthesizing, researching, sharing, debating, publishing…
surface approaches
- using low-level cognitive skills when high-level cognitive skills are required
deep approaches
- using high-level cognitive skills for tasks which require them

Product

learner achievement of outcomes (intended or emergent)
fed back into the system
- informs learners and instructors

Conceptions of Assessment

Brown, 1994; 1996

Brown, G. T. L. (2004). Teachers’ conceptions of assessment: Implications for policy and professional development. Assessment in Education: Principles, Policy & Practice, 11(3), 301–318. https://doi.org/10.1080/0969594042000304609

Brown, G. T. L. (2006). Teachers’ Conceptions of Assessment: Validation of an Abridged Version. Psychological Reports, 99(1), 166–170. https://doi.org/10/bf67hf

general mental structure, encompassing beliefs, meanings, concepts, propositions, rules, mental images, preferences
- improvement of teaching and learning,
- school accountability,
- student accountability, or
- treating assessment as irrelevant.

Fletcher et al., 2012

Fletcher, R. B., Meyer, L. H., Anderson, H., Johnston, P., & Rees, M. (2012). Faculty and Students Conceptions of Assessment in Higher Education. Higher Education, 64(1), 119–133. https://doi.org/10/ctccpq

instructors were more likely than learners to view assessment as consistent and trustworthy methods to understand and improve learning
learners were more likely to have negative views of assessment and viewed it as a measure of student and institutional accountability.

Earl, 2013

Earl, L. M. (2013). Assessment as learning: Using classroom assessment to maximize student learning (Second edition). Corwin Press.

Assessment OF Learning
- summative
Assessment FOR Learning
- formative
Assessment AS Learning
- metacognitive

Approaches to Assessment

Both learning and assessment are complex phenomena which are impacted by myriad factors.

Shepard (2000)

Shepard, L. A. (2000). The Role of Assessment in a Learning Culture. Educational Researcher, 29(7), 4–14. https://doi.org/10/cw9jwc

traditional assessment structures originated in behaviourist models of teaching and learning
- emphasis on culture of summative assessment
modern constructivist models of teaching and learning are less compatible with previous assessment structures, yet a culture that emphasizes summative assessment seems to persist alongside emerging models of assessment

DeLuca, 2016

DeLuca, C., LaPointe-McEwan, D., & Luhanga, U. (2016). Approaches to classroom assessment inventory: A new instrument to support teacher assessment literacy. Educational Assessment, 21, 248–266. https://doi.org/10/gfgtsg

Diagram illustrating four factors influencing teachers' approaches to assessment, described below.

Figure 4. Approaches to Classroom Assessment from DeLuca et al. (2016)

Approaches to Classroom Assessment Inventory
- designed to inventory K12 teachers’ thoughts, beliefs, actions related to assessment
  - Assessment purpose (of, for, as learning)
  - Assessment process (design, use/scoring, communication)
  - Assessment fairness (standard, equitable, differentiated)
  - Assessment theory (consistent, balanced, contextual)

Technology-Mediated Assessment in Higher Education

Contrasting with K12

There is a very large body of literature on assessment in K12 learning contexts, and a not-quite as large, but still substantial body of literature on assessment in higher education. It may be tempting to conflate the two contexts, but K12 teachers typically complete 2 full years of pedagogical training as part of their academic and practical preparation. These two years often include specific courses on assessment, learning theory, as well as domain-specific pedagogies.

On the other hand, higher education instructors (from part-time sessionals to adjuncts to tenure-track and tenured faculty) tend to engage in little academic preparation in learning theories or assessment, although they seem to absorb the signature pedagogies of their discipline.

Impact of Technology

Impact on higher education is ubiquitous (SIS, LMS/VLE, CRM, etc.)
Tends to emphasize efficiency (however ill-defined that may be)
- doing the same things with greater speed and/or reduced effort
- reinscribes mis-aligned assessment structures

Pockets of Innovation

Bearman et al. 2020

Bearman, M., Dawson, P., Ajjawi, R., Tai, J., & Boud, D. (Eds.). (2020). Re-imagining university assessment in a digital world. Springer.

cognitive offloading
artificial intelligence
- “personalized” learning; recommender systems, automated item generation, automated essay scoring
dialogic feedback
- video, audio, screencast
data & learning analytics
- process data
peer/self-assessment
micro-credentials

However…

critical to consider ethical and social impacts!
- surveillance
- equity
- algorithmic assessment

Bower, 2019

Bower, M. (2019). Technology‐mediated learning theory. British Journal of Educational Technology, 50(3), 1035–1048. https://doi.org/10.1111/bjet.12771

In technology-mediated learning contexts, agentic intentions reside with humans, and not with technology.

3 (select) premises
- technology mediates between learners and outcomes
- beliefs, knowledge, practices, and environment are mutually influential (add this to the complexity of assessment)
- role of teachers is to optimise learning through the purposeful deployment of learning technologies

Revisiting Shepard (2000)

Using hypothes.is

22 years have passed…
What has changed?
What is your experience of technology-mediated assessment in higher education?
What are your greatest challenges related to technology-mediated assessment?

Themes and Research Directions

assessment as conversation in digital environments
validity exploration of Approaches to Assessment in higher ed.
humanizing assessment, ethics

Questions? Comments?

References