OTESSA24

Colin Madland1, Valerie Irvine1, Christopher DeLuca2, and Okan Bulut3

1University of Victoria, 2Queens University, 3University of Alberta

Evolving our understanding of technology-integrated assessment: A review of the literature and development of a new framework

Acknowledgements

Our team is spread across most of the country and we each live and work on land that has been cared for by Indigenous people for millenia. We each acknowledge that our abundant lives were made possible because of the displacement of the original stewards of the land.

Background

Fall 2019

2 years into a different dissertation project

Spring 2020

Screenshot of Ross yelling ‘Pivot’ while trying to move a couch around a tight corner and up some stairs.

Summer 2021

Candidacy

The Project

  • Technology
  • Assessment

Two Papers

Madland, C., Irvine, V., DeLuca, C., & Bulut, O. (2024b). Technology-Integrated Assessment: A Literature Review. The Open/Technology in Education, Society, and Scholarship Association Journal, 4(1), 1–48. https://doi.org/10.18357/otessaj.2024.4.1.57

Madland, C., Irvine, V., DeLuca, C., & Bulut, O. (2024a). Developing the Technology-Integrated Assessment Framework. The Open/Technology in Education, Society, and Scholarship Association Journal, 4(1), 1–19. https://doi.org/10.18357/otessaj.2024.4.1.63

…

Process

Literature Review (Madland et al., 2024b)

Designing assessment in a digital world from Bearman et al. (2022)

Assessment Design in a Digital World (Bearman et al., 2022)

Digital Tools

  • Assessment Rationales
  • Level of Digital Enhancement (Puentedura, 2009)
  • Potential Harms

Digital Literacies

  • Mastery or Proficiency
  • Evaluation and Critique

Human Capabilities

  • Future Activities
  • Future Self

Global Distribution

Map of most common countries of publication.

Predict the Ranking of these Themes

QR Code

…

Actual Ranking

Number of Citations by Theme in the Literature.

Comparing Themes to Bearman et al. (2022)

Colours show different levels of alignment between Bearman et al. (2022) and the themes in the literature.

What do you notice? What do you wonder?

QR Code

…

Model Building (Madland et al., 2024a)

Missing from the Literature

  • Level of Digital Enhancement (SAMR)
  • Evaluation and Critique of Digital Tools (Digital Literacies)
  • Human Capabilities

Dropped

Missing from the Bearman et al. (2022) Framework

  • Efficiency and Workload
  • Academic Integrity
  • Community

Missing from both Literature and Framework

  • Indigenous approaches
    • Respect
    • Relevance
    • Reciprocity
    • Responsibility
    • Relationship

Maintaining

Purposes of Assessment

  • …OF Learning
  • …FOR Learning
  • …AS Learning

Modifying

  • Assessment Design

Consolidating

  • Workload + Efficiency = Technology Acceptance
  • Human Capabilities + Potential Harms = Duty of Care

UTAUT

Simplified Diagram of the Unified Theory of Acceptance and Use of Technology

Extending

  • DeLuca et al. (2016) - Measurement Theory

Indigenizing

  • Community/Relationships & Respect & Responsibility –> Duty of Care
  • Relevance & Reciprocity –> Assessment Design

4 Components

Assessment Design

Technology Acceptance

Duty of Care

Purposes of Assessment

Full Model

Alt-View

flowchart TD
classDef purpose fill:#440154,color:#fff,stroke:#440154,stroke-width:4px
classDef duty fill:#3B528B,color:#fff,stroke:#3B528B,stroke-width:4px
classDef accept fill:#21918c,color:#fff,stroke:#21918c,stroke-width:4px
classDef design fill:#5EC962,color:#000,stroke:#5EC962,stroke-width:4px
  A[Technology-Integrated Assessment] --- B(Assessment Purpose)
  A(Technology-Integrated Assessment) --- C(Duty of Care)
  A(Technology-Integrated Assessment) --- D(Technology Acceptance)
  A(Technology-Integrated Assessment) --- E(Assessment Design)
  B --- F([Assessment of Learning]) --- G([Assessment for Learning]) --- H([Assessment as Learning])
  C --- I([Bias]) --- J([Inclusion]) --- K([Relationships]) --- L([Ethical EdTech])
  D --- M([Performance Expectancy]) --- N([Effort Expectancy]) --- O([Social Influence]) --- P([Facilitating Conditions])
  E --- Q([Measurement Theory]) --- R([Academic Integrity]) --- S([Relevance]) --- T([Reciprocity])
  class B,F,G,H purpose
  class C,I,J,K,L duty
  class D,M,N,O,P accept
  class E,Q,R,S,T design

Funding and Support

This  research  was supported by the  BCcampus  Research  Fellows  Program, which provides B.C. post-secondary educators and students with funding to conduct small-scale research on teaching and learning, as well as explore evidence-based teaching practices that focus on student success and learning.

References

Bearman, M., Nieminen, J., & Ajjawi, R. (2022). Designing assessment in a digital world: An organising framework. Assessment & Evaluation in Higher Education, 48(3). https://doi.org/10.1080/02602938.2022.2069674
DeLuca, C., LaPointe-McEwan, D., & Luhanga, U. (2016). Approaches to classroom assessment inventory: A new instrument to support teacher assessment literacy. Educational Assessment, 21, 248–266. https://doi.org/gfgtsg
Madland, C., Irvine, V., DeLuca, C., & Bulut, O. (2024a). Developing the Technology-Integrated Assessment Framework. The Open/Technology in Education, Society, and Scholarship Association Journal, 4(1), 1–19. https://doi.org/10.18357/otessaj.2024.4.1.63
Madland, C., Irvine, V., DeLuca, C., & Bulut, O. (2024b). Technology-Integrated Assessment: A Literature Review. The Open/Technology in Education, Society, and Scholarship Association Journal, 4(1), 1–48.
Puentedura, R. R. (2009). Building Transformation: An Introduction to the SAMR Model. In http://hippasus.com.

Special thanks to BCcampus