Friday, December 24, 2010

The only thing to fear was fear itself

UTSA has moved to online teaching evaluations. I thought this was a horrible idea. We used to have students write evaluations in class and so only the students who came to class would write evaluations. Under the new set up, students who don't attend either because they can't be bothered to show or because they've been failed for plagiarism are given the opportunity to write evals. Actually, they are encouraged to write evals. If they evaluate all of their courses, their names are entered into drawings for free ipads. (How about a drawing to reward attendance?)

Received my eval scores this afternoon braced myself for the worst. Students failed for academic dishonesty were going to let me have it. Someone who noticed that I wear the same outfit everyday would let me know the noticed. (Not the same token pants and shirt, but the same type. I did wear the same cardigan nearly every day.) The sleepers and the texters would explain why they had better things to do in class than pay attention.

Not to fear. Didn't happen. They didn't write evaluations. I had 1 student write an eval for phil law. 6 students in logic and another 6 for analytic phil. I think it was less than 10% response rate. They were all positive. My fears were unfounded, but it doesn't restore my confidence in the system.


Aaron said...

I suspect it may have something to do with that attendance you mentioned. When evaluations were conducted in class, the implication to the students was that they were involuntary. If you showed up that day, you had to write an evaluation. This was a more effective tactic for the university because it cast a wider net. Moving the course evaluations online may save paper and sanity (I know the coordinator of the evaluations and the personal toll that time of year took on her), but it was doomed from the beginning to lower participation from our already detached and despondent student body.

Glad to hear about your glowing reviews. If you put half the effort into teaching your other courses as you did into that Philosophy of Law class, then they were well earned.

Anonymous said...

I certainly don't want to imply that you are not an excellent instructor, but the 10% response rate--which perhaps reflects self-selection for those who liked your courses--is why this mode of student evaluation is even more worthless than other modes.

10% response in from a population of 100,000--fine. That percentage from what it seems like your class populations were at (20-50 students, eyeballing it quickly) is inherently meaningless even if there was no self-selection bias.

I'm willing to bet that student input about teaching performance could be useful, if students--often tired from work outside of school--could be motivated to provide brief discursive evaluations (say around 200-500 words). But there are a lot of reasons why that is a hypothesis not likely to be tested on any large scale.

Clayton said...

I agree with Anon. As I see it, there are three ways of collecting these things. In class, online voluntary, online mandatory (no eval, no grade). There will be some selection biases in the first two if you have a reasonable attendance policy, but I don't see much good that comes from 3 if that compels students to do evals that they won't give any thought to. That's why I like the old method.

Of course, part of what's silly about this is that there's pretty good documentation that positive evaluations don't correlate with good-making factors terribly well. (I get really good evaluations almost always, but I'm not totally self-deceived.)

Mike Almeida said...

I think anon might be missing the point of student evaluations (at least in the UT system). It's agreed that every way of doing this is bad. The results are unreliable in almost every case. I've never been moved to make substantial changes on the basis of students evals. But the point is different: it is to give students some leverage against what they pay for. It's their money (or, their parent's) and so they should have some say over what remains in the curriculum and what doesn't, who keeps working and who doesn't. The evals are supposed to make the courses and instructors face these market forces. It is the same reason they're now online: to make courses and instructors face market forces (this time from parents and taxpayers).

Anonymous said...

I'm all for accountability to students, its the form it currently takes that is the problem. Off the top of my head, I don't know of any other workplace (thought there must be some) in which customers are directly and regularly asked for "quantifiable" data regarding their evaluation the employees they happen to encounter (who are usually not the ones responsible for various policies and workplace practices) in such a way that it expected that that data will be forthcoming and will be used in decision-making.

Beyond that, student input that is based on invalid measures and insufficent or biased is *meaningless*. The use of the term *leverage* is suggestive here (though I don't mean to impute anything in particular to you). If I am a manager evaluating a subordinate's performance, I want meaningful information about that performance, as unsullied as possible by personal-political-etc. attempts to exert pressure. Given that they are often inherently meaningless, that latter function is just about the only function student evals as currently practiced can have.

A final point, the student eval system is not a market force. To the extent that universities are analogous to firms, their internal functioning is condition upon hierarchies and information-sharing and -producing activities that *supercede* the market (the price mechanism), as Coase put it. This is also why, if we see whatever system of student evaluation we employ(rightly, I think) as a primarily (putative) information producing and sharing mechanism, the current bubble-dot system may well be a *detriment* to sound decision-making. This also implies that if putting student evals online is *why* they are online, it is universities that have to face market forces--not instructors and their courses.