Throughout the end of March and moving into April, the work of crafting course content into a truly deliverable online course brought up a great number of questions focused on implementation and practicality. As we built the concepts, they moved out of the abstract and into the inquisitive, and one question that arose over and over again was essentially that of how to effectively test online students.
We discussed these options on various levels: professors described what they wanted and needed and how the course skills and knowledge ought to be measured. Instructional designers spoke with them and in their own groups about best practices and suggestible approaches. University services and vendor contracts were prodded for limitations, options, and flexibility. In order to present this information, I asked Kevin Kula and John Muir, both instructional designers within the Ohio State University’s Office of Distance Education and eLearning (ODEE), to join me in talking about how we’re approaching these questions with university faculty. Kevin, John, and I are leads on several of the online general education projects that will go live this fall at OSU.
Q: What are the primary concerns that professors and instructional designers have about online testing?
Kevin: Notably in math and statistics (and disciplines where there are problem sets), a primary concern is having additional aids during the exam. While online proctoring is eventually a direction we would like to go in, I feel that a greater amount of trial-and-error with this is needed before we decide to go this route our subject area. Another concern noted is that of students sharing exam information with someone taking the exam at a later time. While this can be true for in-person exams as much as online, it is another layer we will work through and strategize for as a group.
John: I think Kevin has hit the main issues. Assessments online are vulnerable to the same kinds of problems as assessments in the classroom–only it’s harder to monitor. There’s certainly no shortage of software and services to improve security, but I think the goal of matching a proctored in-person exam may unreachable.
Tara: Most of my experience comes from working in the humanities, but there is generally a greater sense of unease surrounding online testing, and, to some extent, that’s justified. I will say, though, that I think there’s often more anxiety than is merited. At least in the humanities, the questions are often built around thought processes and ability to apply a concept or analyze an element. Further, they may tie back to course lectures or content. Thus, I think those courses’ instructors have an easier time creating the kind of questions that can’t be searched online or plucked from the mind of a friend who took the course a year beforehand. There’s often too much contingency and critical thinking application involved.
Q: What are students’ concerns with taking tests online?
Kevin: The only thing I have heard thus far is not being able to ask the instructor or teaching assistant to clarify questions during the exam.
John: Taking a test online can be less intuitive than sitting down with a pencil and taking a paper test. The interface isn’t always great, the rules and restrictions aren’t always clear, and there’s the specter of a technical glitch. These are all challenges we can overcome if we’re generally consistent and we give students simple, clear directions.
Tara: Kevin and John both bring up good points. Generally, students aren’t that worried about being explorers with online learning when it’s purely for the sake of learning. But when it’s time for tests – this thing with which they associate a great deal of significance and often anxiety and a thing that will impact their grade – the unfamiliarity of it all and the desire for something known becomes a concern. They just want to have as much security as possible. I would add to John’s comment that making those very clear, simple instructions available beforehand can help students as well. Instructors can also add in a “live” test question and answer forum before the test via Carmen or Twitter in order to give students a chance to ask whatever they like. That can go in the direction of technical or content (think “study group”), or a mix of both.
Q: What are the options that OSU’s general education courses will be using? What are the benefits and drawbacks of them?
Kevin: Our courses will likely begin with in-person proctoring (either by the course instructors or public, proctoring locations). If using the proctoring locations, there would be a fee associated with the convenience of taking the exam at an alternate day/time. Our group is asking the right questions about expanding Ohio State’s testing facility services to meet this growing need for online education. In the future, we will explore online proctoring as a potential method for taking exams in our GE courses. However, we would first like to see its implementation and success across other departments.
John: For Math, we’ll be sticking with in-person exams scheduled along with the students in non-online sections. There still aren’t many great online alternatives for doing math and showing work with pencil and paper. This creates some limitations and makes the courses not 100% online, but it seems like the most instructionally sensible approach for now.
Tara: And the course I’m working with for the fall, English 1110, won’t be using any traditional “test” formats. Assessments will be of students’ writing, research, and analytical thinking skills. Though if a course wanted to employee a truer “test” format over those skills, I think that would be easier to do online granted that it would primarily require building effective questions or prompts.
Q: What should a professor consider when building an online assessment and choosing a means of delivering it?
John: First, I’d recommend, if possible, building an assessment that won’t require proctoring. (A good approach: Could you imagine your existing assessment as a sort of online take-home exam?) If you need to use a timed test where students can’t access materials, then take a look at some best practices for creating exams in Carmen, and talk with an instructional designer about the various online proctoring options. In any event, it will be crucial to give students clear directions in advance about the technology, the timing, and the rules.
Tara: I’ll only add that I encourage folks to inquire into the need for the test, i.e. to see if there’s another way to measure students’ knowledge and attainment that isn’t as traditional. For instance, project-based assessments can be employed in many courses, and they really provide a broad portrait of a student’s development, comprehension, and abaility. Further, I would encourage folks to consider more holistic assessment when they can take place. For instance, I have seen a traditional linguistic mechanics function exam replaced with what some might call an “oral exam” – a short series of three questions in which a situational prompt was given and a student’s verbal response was recorded. These were really effective ways to gauge how well they could apply the ideas of the course, and they really did not take significantly longer to grade. But, of course, it was more time consuming than, for instance, a Carmen quiz that will automatically score tests. Still, listening to an audio file and scoring it was much faster than reading a text response and scoring it, and it allowed the instructor to focus more on the content. But I do realize that those approaches just don’t work in all courses.
Q: What makes a “good” online assessment?
John: I think a good online assessment is the same as a good in-person assessment. It’s meaningful: students are demonstrating their mastery of your learning goals for them, and it provides them with an opportunity for feedback and reflection. Second, a good online assessment is one that takes advantage of being online instead of being imperfect or insecure because of it. If students can consult materials and classmates online, a good assessment encourages that. You may well be able to assess higher-order thinking and tasks as a result.
Tara: It doesn’t confuse students and puts the goals of the assessment first in design. I also tend to think that more assessments that take less time to complete are ideal for online classes, just because of the nature of the courses. Students don’t complete online classes in two or three major chunks per week, like in a traditional course. They may likely have five, seven, or more interactions with the course in a week. Thus, why not have assessments that function that way? Students also tend to stress a bit less when they’re frequent and short, knowing that none of the tests will be a massive chunk of their final grade. Lastly, shorter assessments can help circumnavigate some of the technical issues folks have with accounts timing out or internet connects getting interrupted.
Q: It seems like “tests” as we understand them are creations of the traditional classroom experience. As online learning evolves, what direction will assessment go in? What should the goals be?
Kevin: I think this is a wonderful question that applies to interactive and hands-on lab experiences. Online assessments could include much more than a proctored, multiple choice exam. I think two areas that we – across departments – can explore in depth are (a) online collaboration in small groups and (b) student creation of multimedia projects. With technology at our fingertips, an online setting is not synonymous with little interaction. I didn’t really address the question of assessments, but these are potential topics we are exploring across departments to push the boundaries of online learning.
John: Online courses, with content created in advance, can give instructors a game-changing opportunity to change their primary role from content deliverer to expert/guide/mentor. The hours every week that were spent delivering lectures can be shifted to providing feedback and guidance. Instructors can have students work on more authentic (ill-defined, complex, real-world) assessments, then, and have the time to help them along and give them educative feedback. Projects, research, and real work products can replace tests, to everyone’s advantage.
Tara: My answer here would probably be a reiteration of my response to the previous question to some extent, so I won’t repeat that. But I generally feel like teaching and learning have moved in this much more humane direction in the past few decades, and assessments are doing the same. When teaching, I often saw final exam week as being like a Roman arena where the students were each asked to run in with the lion and survive, and surviving that task proved that they were worth of continuing to exist in the system. Online learning itself is very much a product of looking at students’ needs and meeting them where they are, so assessments will and should continue to accomplish their goals of measuring academic success while working to make it an approachable task that students view as a fair and effective measurement that they’re capable of succeeding at, not an obstacle to overcome.