Readers will know that I’m not a particular fan of the WSET approach to teaching, tasting and exams (see here and here among other previous posts). Now that I’ve done the whole Diploma, I want to take a different approach. (Note: I mean ‘done’ as in I’ve gone through all the classes, exams, etc. at least once. That’s not the same as passing the exams, but I have done them.) What I’m going to do in this post, and the next, is talk about some of the ways I’d change how WSET does things, if I could.
Background
Let me start with a few qualifiers. In thinking how I’d do things differently, I’ve tried to keep in mind that WSET is a global programme. There are necessarily going to be elements that need to be taken into account because of the size and attempt to keep it uniform globally. But with that in mind, let me point out that I’ve been teaching at the university level for a quarter of a century. I’ve taught, set exams, developed courses and curricula in three different countries (US, UK and Mongolia, since you ask). This has been at various levels of student ability and interest, and resources (libraries, teaching facilities, etc.) In other words, I’m coming at this exercise not just as a WSET student, but as an educator myself. I’m going to break this discussion down into two parts: the exams / marking and the teaching, which I’ve already written about a bit.
Exams
Format
Where to start? There are all sorts of changes I’d make here. In some ways, it would be easier to say what I wouldn’t change. That would pretty much be the tasting component of the exams. Those largely make sense. If you want to say ‘yes, this person knows about wine,’ being able to talk about how it smells and tastes is a logical component of that. But I’d like to see more flexibility here than there seems to be. People don’t have the exact same sense of smell or taste, and as far as I can see, the WSET doesn’t make allowances for that. This doesn’t mean anything goes. But why should people lose points if their reaction to the alcohol in wine is slightly off from what WSET thinks it should be? Or how their mouth reacts to acidity levels? We get told by instructors that examiners do allow for some variation. But the feedback I see suggests that if this is the case, it is only so in rare circumstances.
Theory. Ideally, I think these should be dropped. Not that theory, or other things displayed through written knowledge is unimportant. Rather, I don’t think the way WSET examines this is very useful. Their approach requires that you know pretty much as much as you can about every conceivable angle of a particular topic, whether it be light wines, or spirits, or sparkling wines. This leads to data dumps as answers. WSET claims they want people to demonstrate understanding and be able to make an argument, but any feedback I’ve ever received (as well as other people I’ve talked to) suggests that providing all possible pieces of information is more important than a well-structured argument that truly displays understanding of a topic. To me, the answer to this is clear – assessed writing assignments. It is already done as half of the grade for Unit One, on business, so why not for the other units? True, people can look up information, rather than being able to spew out random figures, but more emphasis could (and should) be placed on showing you can use that information, not just regurgitate it.
Not only do I think being able to use information is important, but in my experience, the more a student has to simply regurgitate some information, the less likely they are to remember it, let alone be able to use it in a meaningful fashion later. If there is specific data they need to know for their jobs, they’ll probably either already know it, or learn it in the process. They’ll probably have forgotten it soon after the exam if that is the only reason they learned it.
Exam Marking
Here the changes are on a smaller scale, but no less important. Perhaps more so. First and foremost: get results back to students much sooner. Ten to twelve weeks simply isn’t acceptable, particularly when that means that your results come out after the deadline to register for the next round of exams, if you need to resit. This doesn’t apply to Unit 3, which is offered twice a year, or Unit 2 (which is done in class), but for the others. If you take an exam in June, you won’t know how you did until it is too late to retake the exam in November. You have to wait about eight months to be able to retake an exam, not four or so. That’s simply inexcusable.
I presume WSET’s response would be that they have so many exams to mark that quicker isn’t possible. It is. You just need to get your priorities straight. I won’t go into all the details of how exams were marked in my department when I worked at Cambridge. Suffice it to say you were expected to mark anywhere between 25 and 40 exams (sometimes more) in 48 to 72 hours. These were three hour exams, like the Unit Three theory part. However, they included a much wider range of questions students could choose from, so arguably required a broader knowledge base on the part of the examiner. Part of the marking including taking sufficiently thorough notes to be able to discuss the exam with someone else (all exams are marked twice, and a final mark agreed upon between examiners) and, if necessary, at the examiner’s meeting a few weeks later. My point is not to say we were special. We weren’t. Rather, we had a job to with a tight deadline. We all complained, but we did it. We knew when it was coming up, and planned accordingly. WSET wouldn’t have to turn around things quite so fast, but faster than ten to twelve weeks shouldn’t be much of a stretch.
Feedback
Here I’ll agree with WSET, believe it or not. As much as it would help students to get feedback on exams, that’s not really that practical. Even if you take notes as you mark, they may be thorough, but not necessarily in a form that would be useful to students. That would be quite a lot of work. But having said that – there’s no reason they can’t pass on a number grade. All you get from WSET is the category of the grade – Pass, Merit, Distinction. Again, there’s no reason not to pass on the numbers.
Passing the course
Speaking of passing on specific results, I’ve been re-reading the WSET Level 4 Diploma Specification (available here) before writing this post. It has been enlightening, but again, has things I don’t think make sense.
First: I can see the logic behind saying you need to pass each module in order to pass the Diploma. I’m not sure I agree with it though. I presume the logic is to say ‘Yes, this person knows everything they need to know about wines and spirits.’ I would point out, however, that one can quite easily graduate from university without passing everything. As long as the average of all the modules included in the calculation comes out to a passing grade, you pass. I think WSET should adopt a similar policy. Perhaps allow people to get a Diploma with one fail, if their grades are otherwise good enough. (As a disclaimer: remember, I need to resit two exams. You can argue I’m therefore biased, but under my proposal I’d still have to retake one, and would probably do both, to increase the chance of passing.)
Resits
According to the specification (p. 42) if you resit an exam, you can never get better than a Pass, no matter how well you do. Similarly, you can no longer get a Distinction overall, even if you managed to get one in all the other Units. This simply is punitive. I presume the logic is that If you’ve taken the exam once, you know what to expect, and so have an advantage. I’d disagree, particularly given two factors. First: You don’t get feedback, so if you failed the tasting, having to do it again is no advantage if you don’t know what you did right or wrong the first time around. Second: WSET prides itself on unpredictable theory questions. In other words, having answered some in no way helps you that I can see. Again, I can’t see how this is anything other than punitive. Even if WSET changed nothing else, this would be a painless change as far as I can see. (Again – full disclosure. I have two Merits, but have absolutely no expectation of achieving an overall Merit, so this discussion doesn’t really apply to my own case.)
Think of the implications of the current policy. You can have five Distinctions, and fail an exam because of a cold. Or, as happened to a friend, fail the Unit 1 essay because of a formatting error. You retake the exam, and numerically should get another Distinction. But right now, you’d get a Pass on the exam, and a Merit overall, despite numerically having six Distinctions. How is that fair? What purpose does that serve?
Significant figures
Finally, for this post. This one is going to seem a bit of a diversion, but it isn’t, really. Let’s look at the grading bands WSET uses. A Pass, for example, is listed as being 55% to 64.9%. A Merit is 65% to 74.9% (page 41). And a Distinction 75% and above. Two inter-related points should be made here. First, WSET apparently doesn’t believe in rounding up. If you get a 54.8%, you fail. Now, to be fair, I have no idea how they would actually handle this. Maybe they do round up. But in any university context I’ve worked in, the original situation wouldn’t arise. If the university rules stipulated such a thing, you’d go back and look at individual results. ‘Well, if this one exam was one point higher, that would give the student the 0.2% they need to pass. Examiners, what do you think?’ And more often than not – although not always – the examiners would say ‘Yeah, that’s fine.’ The point here comes down to the title of this subsection, significant figures. So, the digression.
What are significant figures? A good explanation can be found here. But in brief, significant figures mean you can’t get an answer that is more precise than the initial measurements. If you are measuring liquids to the nearest ml, you can’t suddenly have tenths of a millilitre when you add them together, or subtract them. You’re making data up. The same holds true for any mathematical operation – multiplication, averaging, and so forth. This, of course, applies only to real-world measurements. But that is what exam results are.
What do I mean? Imagine a result of 64.5%. Leave aside whether you’d be kind enough to round that number up to a 65%, and hence a Merit. What you are saying, according to significant figures, is that the original measurements were accurate to a tenth of a grade; for example, 64.0 and 65.0. In other words, in the world of measurements, strictly speaking, 64 is not the same as 64.0. 64.0 is precisely that – 64.0. The best you can say about 64 is it is the closest whole number. Maybe, if you were to add decimal places, that 64 should really be a 64.2 or a 63.6. The same for the 65. So calling the average of these two measurements a 64.5 is pointless. And none of this is taking into account the subjectivity of the exam marker. Even given a strict rubric to follow, anything in essay form is going to be open to interpretation and challenge.
Of course, in real life, most people don’t treat grading like this. They treat grades as abstract numbers, in which case the average of 64 and 65 is indeed 64.5. So, what’s the point of this digression? To point out that WSET’s insistence that a 54.9% is a Fail is meaningless from a technical point of view. The initial grades aren’t accurate enough to say that. I could see a case being made for grading to half a number, but not to the tenth. Here again, I’d go a step further – I’d say not only is such a distinction meaningless, it’s heartless as well.
And this isn’t a purely theoretical argument. It also relies on all the examiners being consistent. Both colleagues and I have remarked, in the run-up to the Unit 3 exams, at the wide disparity of marks. For papers that we judged to be of relatively consistent quality, the marks ranged from around 60 to 80, depending on who was doing the marking. The markers themselves are not consistent enough for decimal places to have meaning.
In my next (briefer!) post, I’ll go back and look at the instruction at the Diploma level, which, after all, is what leads up to the exams and marking.