What is Educational Assessment?

A key component of teaching in an educational environment is understanding how educators come to know what learners know. A teacher or peer can play the role of the more experienced ‘other’ that a learner needs to get them from the zone of not being able to complete a task, to being able to complete it with assistance in their zone of proximal development (Vygotsky 1978). However, coming to know what learners know is not really a simple task because ‘learning’ and ‘knowledge’ are not things that we can see or measure. There are many things in our world that we can measure, like the mass of an apple, or the distance between two points, or the volume of an ingredient in a recipe. Knowledge is not like that, though, because it is encoded in the myriad connections between neurons in our brains. We can’t examine a person’s brain and determine what they know.

For example, if we want to know if a learner can calculate the following:

we cannot examine the physical properties of their brain to learn anything about whether they know how to calculate that sum. Instead, we must ask them to provide the answer. If they provide the following:

we can then infer that they do not know how to calculate that sum. We can’t be certain though, because they may have mistyped, or otherwise gotten confused about the task, or distracted because they were hungry. Conversely, if they respond with

We can infer that they do know how to calculate the sum, but again, we can’t be sure. Perhaps they asked someone else, or used a calculator. The point is that we cannot truly measure a person’s knowledge because knowledge of any given thing is latent, or not directly observable, and it is always subject to error ().

Educational assessment can be framed as a trio of components that are essential to allowing instructors to learn what learners know. First, there must be some sort of cognitive model or domain that is mapped and understood. In the case above, the cognitive domain might be addition as a mathematical procedure. Second, there must be some sort of observation in the form of a performance task, like “calculate ”. Third, there is always an interpretation of the results of the performance task, the inference mentioned above.

These three components are interdependent and can be visualised as a triangle as below in Figure 1.

Mapping the Domain

In formal learning settings, the domain is mapped out in the form of learning outcomes, sometimes also called learning targets or learning objectives. It is very likely that every course you have taken in higher ed has included several learning outcomes. Hopefully you recall the outcomes for this course. The learning outcomes for a course describe exactly what you, as a learner, should know, be able to do, or have become after successful completion of the course. For example, the following is a possible learning outcome for a course:

Notice that this is not describing an activity in the course, but rather a description of some state of knowledge or being after the course. Additionally, the outcome is (indirectly) observable. I’m not entirely comfortable with advising that outcomes be measurable, as we will see in the next section, but it is important that it is observable. An observable outcome (from the perspective of the instructor) should also be an outcome that can be operationalized (from the perspective of the learner). In other words, the learner will be able to use their new knowledge or ability in their life-context.

The verbs used in learning outcomes are important determinants of how observable or operationalizable is the outcome. For example, if the outcome noted above read

A simple question on a quiz or in a conversation that simply restates the outcome as a question will allow the instructor to observe whether or not the learner can describe strategies for metacognitive learning. However, operationalizing the ability to describe these strategies is less than inspiring. On the other hand, the ability to ‘model’ these strategies is perhaps more tricky for the instructor being able to observe the learner’s ability, but it is highly relevant when operationalized in the learner’s life.

• describe = low-level cognitive skill.
• model = high level cognitive skill.

Observing Ability

The second component of the model is that of an observation of the learner’s ability, or an assessment task of some sort. Assessment tasks can take a huge variety of forms and structures, often dictated by the cognitive domain addressed in the course. Examples include essays, exams, oral presentations, videos, dramatic productions, paintings, drawings, sculptures, journals, demonstrations, peer reviews, songs, poems, lab reports, worked examples, mathematic proofs, assessment conversations, revised drafts, websites, marketing campaigns, case studies, skills demonstrations, fitness tests, agility tests, concept maps, photographs…

You can likely see from the quiz on slide 4 that someone could answer the question correctly by simply matching a few patterns between the wingding content and the wingding quiz¹, and not know anything at all about the actual content on slide 6. This means that a ‘correct response’ may not tell us anything at all about what a learner knows. So if our only method of determining the learner’s ability is to ask them questions, but the questions can be correctly answered without knowledge, then our assessment strategy is not valid.

The instrument that an instructor uses to gather data about what learners know or can do is important, and unfortunately, the range of instruments commonly used is relatively limited. Many people may be able to recall their own undergrad experience with the final grade being determined by some combination of papers, midterms, and a final exam, with the weight of each increasing throughout the semester. However, as I mentioned in the previous paragraph, there are myriad different assessment task formats.

In a sense, it both matters deeply what assessment task you choose, and matters very little what assessment task you use. The important thing is that the assessment task allows you to gather data about what a learner knows, can do, or has become. Recall that the details of what a learner will know, be able to do, or become are specified in the learning outcomes and can be summarized in the word ability.

As long as the learning task is aligned with at least one learning outcome (Biggs 1996), and can reasonably provide data about the learner’s ability

We will think about what it means to make a valid inference in the next section.

Interpreting Assessment Data

As mentioned previously, a grade is not an absolute measurement of what a learner knows, can do, or has become. It is always an interpretation, or inference based on data that the learner provides.

Inferring Location

Imagine you are giving someone directions to get from the Bus Loop on the UVic campus to the Visual Arts Building by bicycle. Your friend is late, and so you call them and ask them where they are.

• Are you past the Bus Loop?
  
• Yes.
  
• Havew you passed the fountain?
  
• Yes.
• Have you passed MacLaurin?
• No. 

Now you know that your friend is somewhere between the fountain and MacLaurin, but you don’t know specifically where. you also know that they can’t be at Visual Arts, because they haven’t yet reached MacLaurin. This is illustrated in Figure 2

Inferring Ability

This is analogous to what an instructor needs to do to assign a grade, or level of ability, to a learner. The instructor can ask a series of questions or assign a series of tasks of increasing difficulty.

• Can you do a difficulty level one task?
  
• Yes. (The learner demonstrates their ability.).
• Can you do a difficulty level two task?
  
• Yes.
  
• Can you do a difficulty level three task?
  
• No.

This is illustrated in Figure 3.

After this series of tasks, the instructor can infer that the learner is somewhere between difficulty level two and three, and since they are below difficulty level three, they can’t be at difficulty level four.

What’s Wrong with Educational Assessment?

One of the key problems with educational assessment is that higher ed instructors tend to have FAR more confidence in their assigned grades than is justified by the model described above. All we know is that any given learner is within a range of abilities, and without highly sophisticated adaptive exams like the NCLEX for Nurses, or the SAT, MCAT, LSAT, or other standardized tests in the US, that range is remarkably wide.

Research from Tom Guskey (Guskey and Brookhart 2019) suggests that the range, otherwise known as the margin of error is +/- 5-6%. That means that most higher ed faculty can really only be confident in placing your grade somewhere in a 10-12% range, like 63-75%, or 78-90%. You should be able to see that is a huge range when many instructors will quibble over 0.5% or less.

The reason for this is that a grade is the result of a simple equation shown in Equation 1.

Where

= the learner’s Observed Score

= the learner’s True Score, and

= Error

Solutions

• 5 pillars of grading
  • clear outcomes/standards
  • alignment between outcomes and tasks
  • operationalizable feedback
  • grades indicate progress
  • reassessment without penalty
• fewer categories of quality

Learning Pod Meetings

Make sure you and your pod have booked a time to meet with me (link is on our social spaces page). I have really enjoyed meeting with the groups so far, and am looking forward to more this week. It is important to get this done this week, but please let me know if that is difficult.

Assignment 2

I hear that there are some overlapping technologies that pods have identified as their choice for presentation. Please work amongst yourselves to ensure that each presentation is unique. It is best if there is little overlap in the underlying technology, rather than just the specific tool. For example, only one group should present on ‘presentation software’, so ‘Google Slides’, and ‘MS PowerPoint’ would be too similar. (I know this is counter to what I told one group last week–sorry! There is plenty of time to adjust.) If two groups can justify presenting similar technologies, I will consider your arguments if there is a genuine difference between your presentations.

Things to do this week (before our next class meeting)

• make sure you are up-to-date with your blog posts, both for Assignment 1 (Free Inquiry) and Assignment 3 (Weekly Reflections).
• If you haven’t already, book a time to meet with me as a Learning Pod. The meeting itself should take place between today and October 23.
  • We will talk about:
  • any questions or concerns you have related to the class,
  • your plans for Assignment 2, so please come prepared to give me an overview of your topic, why it is of interest, and any initial thoughts or findings.
  • Please use my booking link published on the Social Spaces page of the course website and feel free to book more than one slot in a row if you think you’ll need it.
• If you haven’t already, Read Bower (2019). Technology‐mediated learning theory. British Journal of Educational Technology, 50(3), 1035–1048.
• Read/Watch each of the posts under Education Competencies > Theories, including the linked resources.

Next Meeting

Friday, October 23, 8:30 AM at the normal Zoom link.

As always, I’m cheering you on! Feel free to reach out if you have any questions or concerns.

Things to do this week (before our next class meeting)

• make sure you are up-to-date with your blog posts, both for Assignment 1 (Free Inquiry) and Assignment 3 (Weekly Reflections).
• Book a time to meet with me as a Learning Pod. The meeting itself whould take place between today and October 23.
  • We will talk about:
  • any questions or concerns you have related to the class,
  • your plans for Assignment 2, so please come prepared to give me an overview of your topic, why it is of interest, and any initial thoughts or findings.
  • Please use my booking link published on the Social Spaces page of the course website and feel free to book more than one slot in a row if you think you’ll need it.
• Read Bower, M. (2019). Technology‐mediated learning theory. British Journal of Educational Technology, 50(3), 1035–1048.

Next Meeting

Friday, October 16, 8:30 AM at the normal Zoom link.

Please be ready for working with your learning pods. You aren’t required to, but you may want to consider meeting physically with your pod to join the meeting. We will be creating and sharing media.

FIPPA, Privacy, and Security

Data is the new oil. ~Someone on the Internet

Sites to visit

• clickclickclick.click.
• Since You Arrived.
• Privacy.net.
• Google Maps (on mobile) -> Timeline

Current Events

• Meta Muse
• Bill C-22 in Canada.
• Opposition to Bill C-22

Disconnecting

• Paris Marx on Disconnecting from US Tech.
• Stop Google from Tracking You.

Topics for Review

Here are some things we talked about in Week 1.

• RSS (This wikipedia link is a bit technical, but provides some good info.)
  • I believe that RSS is one of the grossly underappreciated technologies for learning, because subscribing to RSS feeds is a perfect way to escape the infinite scroll algorithms of social media sites.
  • Feedly is excellent, and I also use inoreader.com, which allows me to subscribe to feeds, group similar feeds together, and then publish a metafeed that other folks can subscribe to.
• Most Likely to Succeed - UVic Library login required.
  • If you have not yet watched this, please make it a priority.
• Alpha School wants AI to teach a Billion kids. Should it?
• liberating structures
  • spiral journalling
  • 1-2-4-all
• What Teachers Make - Taylor Mali
• Cooperative Learning

Things to do this week (before our next class meeting)

• Publish your first post towards completing Assignment 3. This post should be published on your blog as a new post, and should use the categories as requested on the assignment page.
  • Your post should cover items in the Topics for Review section on this post.
    
• Share a draft of your Learning Plan for the Inquiry Project with me and your learning pod. Consider the following tools:
  • Trello
    
  • MS Planner (UVic hosted)
    
  • others as you find them
    
• Read Privacy and Safety on the course site, including the links.
  
• Read Freedom of Information and Protection of Privacy (FIPPA)
  
• Read All the posts categorized Copyright on the course site.
  • Suggestion - lean on your learning pod to share the load on these posts. Each of you can pick a post or two and share your findings with your learning pod.

That’s a bunch of work for you! Please reach out if you are having challenges!

Things to do this week (before our next class meeting)

• Sign up for a WordPress site at the OpenETC
  • Contact me by Wednesday, September 16 if you have difficulties signing in to your site (e.g., you see the error message that your site is being set up and to check back in a few minutes)
• Sign up for a Learning Pod - UVic Login Required. Do not edit anyone else’s information please.
• After you have access to your own website on the OpenETC, visit our Social Spaces page and complete the two surveys linked here.
• Watch Most Likely to Succeed - UVic Library login required.
• Read Alpha School wants AI to teach a Billion kids. Should it? - Let me know if you can’t access this link.
  • Also read the discussions related to this article.

The Classroom Critical Incident Questionnaire

Please take about five minutes to respond to the questions below about this week’s class. If nothing comes to mind for any of the questions just leave the space blank. At the next class we will share the group’s responses with all of you. Thanks for taking the time to do this. What you write will help us make the class more responsive to your concerns.

• At what moment in class this week did you feel most engaged with what was happening?
  
• At what moment in class this week were you most distanced from what was happening?
  
• What action that anyone (teacher or student) took this week did you find most affirming or helpful?
  
• What action that anyone took this week did you find most puzzling or confusing?
  
• What about the class this week surprised you the most? (This could be about your own reactions to what went on, something that someone did, or anything else that occurs).

Record your responses on your blog, but please do not mention specific people by name.

Use the category #cciq.

Use the Zoom Workplace app

Using the app rather than the web is significantly easier and has the benefit of better quality streaming for audio and video.

• Go to https://zoom.com, click the ‘Meet’ menu, and choose ‘Download App’

Sign in with UVic

You might already have the Zoom Workplace app on your computer, and you might not be signed in to the UVic Zoom.

UVic will sign you out of Zoom after 12 hours of inactivity, so you will have to sign in again every time.

• Check your sign-in.
• If you are signed in to the UVic Zoom, you will see your UVic email in your profile when you click either your avatar or your initials in the top right corner of Zoom Workplace.

OPTION 1: Add an account

• click the Zoom Workplace menu and choose ‘Switch accounts’, then ‘Add account’.

OPTION 2: Sign out

• click the Zoom Workplace menu and choose ‘Sign out’ then either your personal account, or ‘Sign out of all accounts’

Sign in with UVic

After either option above, sign in with your UVic email address.

• enter your UVic email address and click ‘Next’

You may already be signed in to UVic systems, in which case, you will be signed in automatically, or you may need to enter your UVic credentials (same password you use to sign into all UVic sites).

Make sure to check your sign in each time you try to connect to our class meeting.