Listening to Students’ Voices

This is an article I wrote for my Faculty Association‘s newsletter. Some of it is specific to my institution, but the overall principles hold for many undergraduate instructors.

All semester long, our students listen to our voices (or watch our hands, in the case of ASL courses). And at the end of the semester, it has long been our tradition to ask students to voice their experience in the course, in the form of a course evaluation questionnaire. More than a decade ago, one of my students took the opportunity to make it clear that listening to me had not been a positive experience for them, by commenting, “She has an annoying voice.” I’ve often wondered what that student’s goal was in writing that: Did they think I should have acquired a different voice to make their experience more pleasant? Would they have learned more or better if my voice were different? Did they want my Chair to discipline me for my voice? The only plausible goal I can think of is that they wanted to hurt me. If so, they achieved some modest success – after all, I still remember their barb all these years later. But I know that my white skin and monolingual anglophone Ontario accent shield me from the hate that many colleagues receive in this medium, anonymized and typed up in a formal report issued by the employer. The bigger question, more important than what that student hoped to achieve, is what we hope to achieve: what is our goal in surveying students at the end of each course? 

What do we want to know from student surveys?

Ostensibly, the goal of these surveys is to measure Effective Teaching, or, in the case of Teaching Professors, Excellent Teaching. According to the Tenure & Promotion Policy, “A candidate for re-appointment, tenure and/or promotion must demonstrate that he or she is an effective teacher,” and, “A candidate for permanence must demonstrate that he or she is an excellent teacher”. And SPS B1 Procedures for the Assessment of Teaching opens by stipulating that “Effective teaching is a condition for […]  salary increments based on merit.” Our existing policies require the numerical data from student surveys to be considered in determining whether an instructor’s teaching is effective, which is necessary for making decisions about tenure, permanence, promotion, and merit. 

Who could argue with the goal of ensuring effective teaching? McMaster’s world-class reputation for creativity, innovation and excellence rests in no small part on the quality of our teaching, and it is clear from the immense efforts we have invested in redesigning courses for pandemic virtual learning conditions that McMaster faculty are deeply invested in providing high-quality learning experiences to our students. 

What can we know from student surveys?

The problem is, the numerical scores from student surveys have very little relationship to teaching effectiveness. A large and growing literature has repeatedly found that these surveys are strongly biased by factors such as the instructor’s race, accent, and perceived gender and by students’ grade expectations. So convincing is the evidence that a 2018 decision by arbitrator William Kaplan established the precedent that “averages [of survey responses] establish nothing relevant or useful about teaching effectiveness”, a ruling that required Ryerson University to stop using survey results for promotion or tenure decisions. And in the time since that ruling, pandemic teaching and learning conditions have introduced even more factors that affect students’ learning experiences, such as the quality of their instructor’s home internet connection or their dislike of remote proctoring.

According to our policies, we value effective and excellent teaching. According to the university’s vision statement, we value “excellence, inclusion and community”. But over the years we’ve built a system that calculates a faculty member’s annual merit and makes high-stakes decisions about tenure, permanence and promotion using a number that not only does not measure effectiveness or excellence, but actively works against promoting inclusion and community by reinforcing existing hierarchies of race and gender. The system also does real harm to equity-seeking members of our community by subjecting them to anonymous hateful comments that are irrelevant to their teaching.

If we claim to offer excellent teaching, we have a responsibility to listen to what students say about their learning. If we claim to offer excellent teaching, we have a responsibility to avoid relying on invalid, biased data as evidence.

What has MUFA done about it? 

At the time of the Kaplan ruling, MUFA began collaborating with scholars at the MacPherson Institute to develop innovative and equitable ways to observe teaching effectiveness. A 2019 report made initial recommendations, and an ongoing committee is developing an evidence base that will inform a redesigned system. 

In the absence of an unbiased, valid tool for observing effective teaching, the members of the MUFA executive thought it imperative to attempt to mitigate the biasing effects of the current surveys. To that end, we negotiated a revision to the so-called “summative question” on the surveys. Instead of asking students their opinion of the instructor’s effectiveness, that question now asks, “Overall for this course, how would you describe your learning experience?”. Furthermore, under pandemic circumstances, we negotiated an agreement that these scores should not be used at all in the assessment of merit for 2020. We hope these temporary changes go some way to reducing the biases of the student survey process, but they were only ever meant to be short-term measures.

What still needs to be done?

Excellent teaching is too complex to be tracked by a single number that gets compared across instructors. As members at a recent MUFA Council meeting pointed out, simply replacing student survey numbers with peer scores or with student focus groups is unlikely to eliminate the bias. If our goal is excellence, or even something more achievable like effectiveness, we need complex, high-quality ways of assessing progress towards that goal. It will take time to achieve this, and we’re working on it. 

Once we’ve developed a new process, we’ll need to update all our policies to make sure they’re consistent with each other. We should also build in a regular schedule for reviewing the process, to make sure we don’t inadvertently regress to overly simple, biased metrics over time. 

One MUFA Council member asked whether we had taken any steps to redress historical consequences of using biased data. The MUFA executive have not discussed this specifically with respect to student survey data, but we are continuing to investigate equity issues in our members’ compensation. The 2015 salary adjustment for women faculty was one outcome of this ongoing process. 

While all that work is going on, we can and should strive for continuous improvement of our teaching. And here is where listening to our students is vital. Students are the only ones who can provide first-hand data on their experiences of our courses. We could design the most rigorous, comprehensive courses in the world that still might not support our students’ learning. Within a semester, listening to our students might prompt us to move due dates, reweight assessments, or return to a topic that we thought we had finished. By listening to our students at the end of a semester, we can make improvements to the course for the next cohort of students, like converting a timed test to a take-home assignment, or reducing the number of low-stakes quizzes. It was listening to my students saying, “I don’t have the textbook. Is it required? Is it on reserve? It’s awfully expensive,” that led me to create an OER that is freely available to all learners everywhere. 

If we claim to offer excellent teaching, we have a responsibility to listen to what students say about their learning. If we claim to offer excellent teaching, we have a responsibility to avoid relying on invalid, biased data as evidence. MUFA is working on ways to do both of these. We welcome your contributions to this work. To get involved with the work of the MUFA Executive, please contact . 

Making undergraduate research opportunities accessible

Our department has several opportunities for undergrads to conduct original research. In the Research Practicum (Ling 3RP3), students work as lab members on a faculty member’s research project. In the Honours Thesis (4Y06), students conduct an original research project of their own, guided by a faculty supervisor. Students might also conduct a smaller-scale research project under the Independent Study (4II3) course code, though this course code also covers individual reading courses and the like. I created the Honours Thesis course and guidelines in 2009, but in recent years we learned that many students (and indeed, some faculty members!) were not aware of the process for requesting access to these courses. In 2017, I re-wrote the instructions for applying for these courses, and revised the suggested course outlines. The updated guidelines are now available on the department website.

Blended Learning for a Very Large Class

Introduction to Linguistics is a very large class — usually about 600 students in Ling 1A03 and 300-400 in Ling 1AA3. The size of the class can be overwhelming to first-year students, and the format of the class, which blends on-line components with in-person class sessions, might also be unfamiliar to them. Having a clear structure in Avenue is important to helping students stay on track in the course. In the very large classroom it’s also important to create a sense of community, to alleviate some of the sensation that students might experience of being just a face in the crowd. In the first couple of classes, I remind students that learning can work well in community and I give them a few minutes to get to know each other. In each class period there are always opportunities for students to talk to each other while analyzing data or arguing over the answer to a clicker question. I often remind them that if they don’t know the person they’re sitting next to, it’s worth taking a minute to introduce themselves before jumping into the discussion question. And I often open class with a couple of sentences about myself or my family, so that they remember that I’m a human with a life outside the classroom just as they are.

The screencast embedded below walks you through some of the on-line components of the course from the student’s point of view. The first semester that we used the blended format was Fall 2014. Each year I’ve surveyed students anonymously about their experience in the course and made changes based on their feedback. The first year, their most frequent ask was that we correct the errors in the quizzes, an entirely fair request since that first version did contain more errors than tolerable (partly because we were still learning how to use the Avenue quizzing software, and partly because we were developing materials on a short timeline). So the primary job for my TAs that next summer was to go through the quizzes very carefully and correct any errors. We also added a lot more quiz questions to each topic that summer, so there was a larger bank to randomly draw questions from.

The following year, the thing students wanted most was more practice assignments. As I gradually detached the course from its dependence on the commercial textbook, I created my own new assignments every year. I do this so I can provide them with full answer keys and commentary once the assignment is over, while not having to worry about future students getting access to the answer key. This means that I can post the previous year’s assignments and tests plus the answer keys for them, so that students get a chance to practice and check their own answers.

Here are some sample assignments and practice tests:

Sample Assignment for Ling 1A03

Sample Midterm Test + Answer Key for Ling 1A03

Sample Assignment for Ling 1AA3

Sample Midterm Test + Answer Key for Ling 1AA3

The biggest change for 2018 will be the new Open Access eBook, Essentials of Linguistics. Before September, my TAs are working to update the quiz questions and all the accompanying materials to be consistent with the notation conventions in the new book. In the Spring 2018 course, students have already expressed how pleased they are not to have to shell out $147 for a commercial textbook!

A cross-listed Level 3 course

Child Language Acquisition (Ling 3C03) is a required class for all Linguistics and CogSciL students. It is also cross-listed as Psych 3C03 and forms part of the Human Behaviour program. In recent years, I’ve taught it as an evening course with a fairly traditional lecture style, though of course my lectures are liberally dosed with discussion questions and clicker questions, and with pictures and videos of my twins during their early language development. These slides illustrate a typical evening class.

Because it’s a Level 3 course, there are fewer scaffolds for students compared to Level 1 & 2 courses. The graded components of the course are two projects, a midterm test and a final.

The goal of the PR Project is to communicate scientific findings to a non-scientific audience, a skill that will be valuable in many different careers after graduation. Students read a recent journal paper about child language, and write a one-page press release describing the findings of the paper. The challenge of the assignment is in deciding which details to include, and in describing the findings accurately but accessibly.

PR Project Guidelines

PR Project Rubric

In the Experiment Project (adapted from a similar project designed by Dr. Ann Bunger at Indiana University) students conduct a brief experiment on themselves with a short simulated verb-learning activity. The class is divided into four groups, each of which completes the activity in a slightly different condition. I tabulate the results, and the students interpret our data through the lens of a much-cited paper on syntactic bootstrapping.

Experiment Project Guidelines

Experiment Project Rubric

This year, for the first time, I made the midterm test an open-book test. My thinking was that allowing the students to refer to their notes and textbooks for factual details would allow me to ask deeper questions about research design and data interpretation in child language. But in designing deep questions, I added too many of them to the test! The students found it difficult to complete the test in the allotted time, and the scores were quite low. The week after the test, I was grateful that the students engaged in an open and honest conversation with me about how the test had gone. I acknowledged that I had made the test too long, while they revealed that almost none of them had ever encountered an open-book test before, so they didn’t really know how to prepare for it. To compensate them for making the test too long, I boosted everyone’s test score. And I promised them that I would give them access to some portion of the final exam in advance of the scheduled exam.

The last question on the final exam, which was posted to Avenue ten days before the exam, asks students to interpret some new data that we did not consider in class, and to interpret this new data in light of what they already know from class. Such a question allows them to extend their learning to a new, related situation, and having the question ahead of time allows them to actually learn something while preparing for the exam, rather than simply cramming.