Over at Uncertain Principles, Chad asks for suggestions for minor tweaks in the student evaluation system. I thought I’ll add my two cents here, and ask the readers for their opinions. Just to be clear, like many others I think the current system serves no purpose whatsoever, and is often counter-productive. In my mind, any minor tweaks will not change things much, but some re-thinking of the system can make it useful.
First, for those of you who are not currently at a university, let me remind you that every course at a north American university ends in the ancient ritual that is called student evaluations. In this, students are asked some vague questions, all of which come down to “on a scale from 1 to 5, is the instructor any good?”. We tabulate the results, and pretend that we now distinguished the good professors from the bad ones, and in the same process allowed the student-consumer have some feedback on the system. Everyone wins!
And here is the thing, I don’t think students are in any position to evaluate teaching. Any rational system of evaluating teaching a specific course will have to specify the goals of that course and measure to what degree those goals were achieved. There are many ways of doing that, but student evaluation, of any kind, is not going to cut it. Students taking the course are missing by and large the context of the course, and I think their answer to any evaluation question will be pretty much independent of the question. They will invariably answer the implicit question “did you enjoy the course?”, because that is the only data available to them.
By all means, enjoying the course should certainly be one of the goals, and student satisfaction could be measured and tabulated to decide to what degree that particular goal is achieved. I just think that student satisfaction is not the only goal of the course, and that is the only parameter we are attempting to measure at the moment. Since I believe we do need a system of evaluating teaching, pretending we already have one is counter-productive if we want to improve the quality of teaching.
All that should not be interpreted as the statement that student feedback is not useful. This could be an essential tool in improving the course and checking your teaching ideas against reality. For example, having the students answer a set of a course-specific questions, during or after the course, could very helpful for this purpose. Just forget about the 1 to 5 scales, those are just silly.
As always, comments are welcome!
Are the students’ results being used as a measure of teaching performance somewhere along the line? That would seem to be the clearest indicator of the success of the professor in doing their job.
That, in addition to the students enjoying the course, should give a reasonably complete picture.
First, let me tell you that I am in general opposed to different types of shortcuts and tests. I think any measurement on humans, where the subjects actually care about the results will be distorted. Try to measure how well students do at school using grades and quizzes, and school turns out to be all about tests and quizzes, instead of about learning. Try to evaluate how well researchers do by summing over papers times impact factor, and people will care about publishing as many papers as possible, instead of doing good science. I think one should only measure exactly the thing one is interested in.
Ok, now that I have that off my chest, I think the actual question is more complicated than I pretend it is. If not by results or questionairs, how could one evaluate how well a teacher does? What is “the real thing” one should measure, insetad of doing a test? I don’t know…. I guess one should figure out what the results are that one wants to achieve, and measure them after enough time. So if one wants teachers that produce many nobel laurates, one should measure how many of their students won the prize, using a big enough sample, and controlling for the effects of other teachers. Hmmm….
The problem is that I don’t think any other short term tests can measure what one really wants. One could have more immediate goals – maybe a programming course should prepare the students for a class, and one could measure later how well the students were indeed prepared.
That doesn’t really always work. Some cases are easier, though. If all the university wants is to make money, then maybe looking at what the students think is a good thing. Or maybe even better would be what their parents think.
But back to the subject. One could put video cameras in the class, and have evaluators evaluate how good the teaching is. I think that would probably have horrendous consequences, though maybe not quite as bad as having the evaluator come in and sit in the class (did you have that in high school?). One could have fake students especially for that purpose.
No, thinking back about my classes, I think that I (and especially Moshe), actually had a very good idea how well the professor taught. And, in retrospect, I think I know even a bit more about that. So I think that asking the students is not a bad idea, and one could also try to get opinions in hindsight. It is just that the way it is currently implemented is bad. One can’t just average what the different students say to the different questions.
I think a google-like algorithm, which could also be fed eventually by number of nobel laureates produces, or amount of money the student eventually contributed as an alumni, or how much they made per year, or whatever the university want to achieve.
Maybe if there was a 1M$ prize, someone will come up with the right algorithm…
Hiya Michael, good to hear from you. I agree with your points, and I think it is not that hard to measure exactly what you want achieved. If your class is aimed at having the students build robots, just check that the students can build them. If your class is aimed at having the students solve boundary value problems for certain class of ODEs, just check they know how to do that.
If at the end of the class a majority of students say they enjoyed the class tremendously, thought the teacher was really a cool guy, and explained himself very well, made everything make sense and encouraged students to think for themselves, etc. etc., all of that is irrelevant if they cannot build the robots or solve the equations.
And if a teacher sits in front of the class and reads Jensen aloud undecipherably, but the students study it themselves at home because it is an important class, should he count as a good teacher?
I think this comes down to a question of the role of the teacher. I have written my share of good and bad evaluations. In retrospect, I would like to take back all the bad ones. The only criteria that I have now is whether the instructor has a solid grasp of the material they are presenting and if they can answer well posed questions.
I’m not sure that any currently recognized method of evaluating the effectiveness of teaching works. The students know the immediate effect on their knowledge but may not know the overall impact of that knowledge for years. There is nothing wrong with checking to see if the students enjoyed the course. You can test to discover if they learned the appropriate “facts”. The problem as I see it is that all of these evaluations take place (as they must) years before it is possible to know the answers.
Michael, if the instructor routinely leaves the students stranded, I expect that on average they will not do very well.
Also, I don’t object to devising some sort of student evaluation that spits out a number with some information content, and then taking it into account as a small part of a more comprehensive teaching assessment. What I find counter-productive is the current system, where the number is uncorrelated with teaching effectiveness, presumably the thing you are trying to measure for. That random number is then is taken to be the only measure of teaching quality (at least in theory, in practice it is easy to ignore, because everyone knows it means nothing).
Moshe. I think you are being unfair to students. I think they are capable of more than judging their own satisfaction. Let me give you two examples from personal experience.
Firstly, a lecturer who had been teaching a course for many years showed us at the end of the course a graph in which he had done a statistical analysis of student feedback on his course over the years. The decline in grade over the years was gradual but significant. His conclusion was that it was time to teach a different course. This seems useful and brave on his behalf.
Secondly, in my undergraduate days I had a least two lecturers who were so bad that they shouldn’t have been teaching anything. One of them mumbled so that he was literally inaudible and furthermore had illegible handwriting. Consequence was that most of the students in the class stayed at home. Who else is going to tell ‘the authorities’ that this is going on day in day out? The problem in this case is that the feedback gets ignored, or it is too difficult to fire someone for being an appalling teacher and too costly to have people around who are not teaching and so in practice the feedback (which, I can guarantee was pretty negative) doesn’t have any effect and this guy continues teaching and putting people off physics (I am not exaggerating).
Hey Sean, I think most people I know are conscientious and really want to be good teachers. For those, feedback would be more useful than just a numerical result, especially for trivial flaws like not talking loudly enough. What good would it be to get (or give) horrible rating if there is no mechanism for improvement?
Also, see my comment above. Student evaluation, provided the current system eliminates all sorts of noise, could be good as a (small) part of teaching evaluation. There are so many parts of good teaching that are invisible to the students (e.g. choice of material and its organization, relevance of exercise to material in course and beyond, etc.) that making this the only yardstick for success is fundamentally unfair to us.
Dear Moshe, I used to have pretty high scores in these silly games but I agree with you.
Almost by definition, (average) students must be less qualified to judge the actual quality of the course than the instructor.
Learning and teaching is something different than entertainment, yet these evaluations always inevitably end up in the entertainment territory.
It is very clear what kind of teaching would be most “popular” i.e. earn good “grades” even in categories where it would be completely absurd. And it is very clear that this “popular” teaching is not what the universities should be trying to do.
The entertainment and softness bias can appear at many levels. Of course, there exists the lowest level in which the instructor simply demands almost nothing and makes the courses just fun with little content. But even at higher levels, there exist counterproductive temptations. In quantum mechanics, a course that would be quoting philosophers saying that quantum mechanics is nonsense would surely be popular, and so on.
It is bad if students are not treated as humans, on the other hand, it is even worse if they become the dictators of the situation.
Yeah Lubos, I also usually get good evaluations, but I know better than to take this personally (I hope I teach reasonably well, but there is no way for me to know). This probably has mostly to do with the fact that I usually teach upper level UG and graduate courses.
At least usually the system doesn’t take these evaluations all too seriously, unless they are extremely good or extremely poor, but what started this series of blog posts is the idea of financially rewarding teachers for good student evaluations, in a pretty substantial way. In such a system you’d indeed have financial disincentive to teach anything too demanding. Look at this
http://www.chron.com/disp/story.mpl/headline/metro/6205773.html
Somehow all the insane ideas end up in Texas, not sure why that is…
[...] by arjendu on February 15, 2009 About a month ago, Chad and Moshe talked about student evaluations of teaching, and I kept meaning to comment. I’ll leave out [...]