Using & Teaching Educational Technology
Trying times (Part 1)
By Terry Freedman
Wed, 4 Jun 2008, 00:26

I've been involved in two very different forms of assessment this week, and only today I read about some new research going on. So over the next day or two I thought I would just talk about those things and, in particular:

• The two key difficulties of assessment;
• My difficulties with rubrics; and
• My problem with some newly-published research.

The two key difficulties of assessment

I had an extremely enjoyable day today. You will quite possibly think I am a masochist, because my day consisted of being locked in a room with half a dozen other people, looking at items for an on-screen test for educational ICT.

This is the test whose development I was very closely involved in whilst working at the Qualifications and Curriculum Authority (QCA) a few years ago. The idea of the test is that it tells you, or gives a very strong indication of, the level at which the student is working, or has attained at that point in time, in information and communications technology (ICT).

This test was different from all its predecessors, not because it was taken at a computer screen -- many tests do that -- but because it attempted to assess the student's ICT level by their performance in problem-solving tasks. So, think about that for a moment: we are not talking about a multiple choice test in which, if it has been written properly, there is only one correct answer per question. This is a situation in which there is not necessarily any correct answer, but one or more correct ways of approaching the problem.

For a test like this to work, there has to be a set of rules underpinning its judgements, and that's what I came up with: the rules base. This is, in effect, a vast grid of possibilities, but one which at its heart is very simple Boolean logic. Basically, the test comprises a set of rules which, if written in ordinary English, would read something like this:

"If the student does A and then does B, but does not do C", she is probably at a Level 5 in this area of the curriculum. But let's test this assumption by seeing what she does on this next task. If she does X..."

Originally, the test was intended to be a summative test, ie one that measured the student's grade at the end of what is known in England and Wales as "Key Stage 3", ie age 16. It was, therefore, to be a high-stakes assessment. And it was, perhaps inevitably, removed from the "mandatory" column to the "optional" column of the government's plans, to become, instead, a tool for formative assessment.

I think that was the right decision, because the test is so good at testing understanding in real-world problem simulations, and one that can therefore form a great basis for class discussion.

Over the past few years I have been involved in the discussions of the test items. What happens is that people write the test items, they are then translated into interactive questions, and we come along and pull them apart for hours on end.

The "we" are people from the key organisations involved, ie the QCA, the National Assessment Agency (NAA), and the National Strategies, and me. To put that another way, the participants in these discussions have been the organisation that has written the ICT curriculum for England and Wales (QCA), the organisation whose remit it is, amongst other things, to oversee the assessment of the National Curriculum (NAA), and the organisation which has taken the ICT Programme of Study and made it more concrete, in the form of a Framework and lesson plans and other resources.

The two key difficulties of assessment that become obvious in this sort of activity are:

1. Does this test item measure what it purports to measure? In other words, is it valid?
2. If so, does it succeed?

Put like that, it may sound like those are more or less the same question, so let me give an example to make my meaning clear:

The first question is about looking at what the test item requires, and seeing if that needs knowledge or understanding in a particular area in order to answer it correctly. Today, for example, we were looking at sequencing questions. If they had included examples of things like boiling water for a cup of tea, we would have said that you don't need knowledge of ICT to answer such a question. All you need is knowledge of the world and maybe some common sense.

Where a question did pass the validity test, we then had to say,

"OK, you would have to know about ICT to be able to answer this question, but are the options provided correct, does the flowchart sequence shown actually work, do you need to be highly literate in order to understand the question, would you be disadvantaged if you were a particular gender or of a particular ethnic group?"

Clearly, what is enjoyable in such discussions is that you grow in your own understanding of the subject. Also, it is very interesting to hear other people's views on particular issues. It's the discussion, the interaction, that moves things on. And sometimes there are moments of humour, like today when I realised, halfway through an impassioned  plea against a particular question, that I had successfully demolished my own argument!

One of the things I have noticed that distinguishes good ICT departments in schools from the others is that time is made available for discussion about assessment. Teachers need to have a clear and shared understanding about what, say, a Level 5 student is capable of, and they need to be able to assess their students' performance with a reasonable degree of accuracy.

They also need to be able to articulate to their students what these levels mean, and what a student must do to progress from one level to the next. They must also go beyond even this, and help students become, to use Maslow's term, self-actualised in the realm of assessment. In other words, they must be able to evaluate, accurately, their own performance.

Unfortunately, time to do all this is the one thing that teachers no longer seem to have.

Sadly, today was the last of these periodic meetings, because we have now gone through all the main areas of the ICT curriculum. In an ideal world, everyone involved would now start back at the beginning, in order to ensure a constant supply of fresh material, but, for the time being at least, that is not to be.

In my next article on this subject I will describe a rather different form of assessment I have been involved in this week, that of using a rubric as a "meta-judge" for the Horizon 2008 project.

© Terry Freedman Wed, 4 Jun 2008