I've been involved in two very different forms of assessment this week, and
only today I read about some new research going on. So over the next day or two
I thought I would just talk about those things and, in particular:
- The two key difficulties of assessment;
- My difficulties with rubrics; and
- My problem with some newly-published research.
The two key difficulties of assessment
I had an extremely enjoyable day today. You will quite possibly think I am a
masochist, because my day consisted of being locked in a room with half a dozen
other people, looking at items for an on-screen test for
educational ICT.
This is the test whose development I was very closely involved in whilst
working at the Qualifications and Curriculum
Authority (QCA) a few years ago. The idea of the test is that it tells you,
or gives a very strong indication of, the level at which the student is working,
or has attained at that point in time, in information and communications
technology (ICT).
This test was different from all its predecessors, not because it was taken
at a computer screen -- many tests do that -- but because it attempted to assess
the student's ICT level by their performance in problem-solving tasks. So, think
about that for a moment: we are not talking about a multiple choice test in
which, if it has been written properly, there is only one correct answer per
question. This is a situation in which there is not necessarily
any correct answer, but one or more correct ways of approaching
the problem.
For a test like this to work, there has to be a set of rules underpinning its
judgements, and that's what I came up with: the rules base. This is, in effect,
a vast grid of possibilities, but one which at its heart is very simple Boolean
logic. Basically, the test comprises a set of rules which, if written in
ordinary English, would read something like this:
"If the student does A and then does B, but does not do C",
she is probably at a Level 5 in this area of the curriculum. But let's test this
assumption by seeing what she does on this next task. If she does X..."
Originally, the test was intended to be a summative test, ie one that
measured the student's grade at the end of what is known in England and Wales as
"Key Stage 3", ie age 16. It was, therefore, to be a high-stakes assessment. And
it was, perhaps inevitably, removed from the "mandatory" column to the
"optional" column of the government's plans, to become, instead, a tool for
formative assessment.
I think that was the right decision, because the test is so good at testing
understanding in real-world problem simulations, and one that can therefore form
a great basis for class discussion.
Over the past few years I have been involved in the discussions of the test
items. What happens is that people write the test items, they are then
translated into interactive questions, and we come along and pull them apart for
hours on end.
The "we" are people from the key organisations involved, ie the QCA, the National Assessment Agency (NAA), and the National Strategies, and me. To put that another way, the
participants in these discussions have been the organisation that has written
the ICT curriculum for England and Wales (QCA), the organisation
whose remit it is, amongst other things, to oversee the assessment of the
National Curriculum (NAA), and the organisation which has taken the ICT
Programme of Study and made it more concrete, in the form of a Framework and lesson plans and other resources.
The two key difficulties of assessment that become obvious in this sort of
activity are:
- Does this test item measure what it purports to measure? In other words, is
it valid?
- If so, does it succeed?
Put like that, it may sound like those are more or less the same question, so
let me give an example to make my meaning clear:
The first question is about looking at what the test item requires, and
seeing if that needs knowledge or understanding in a particular area in order to
answer it correctly. Today, for example, we were looking at sequencing
questions. If they had included examples of things like boiling water for a cup
of tea, we would have said that you don't need knowledge of ICT to answer such a
question. All you need is knowledge of the world and maybe some common
sense.
Where a question did pass the validity test, we then had to
say,
"OK, you would have to know about ICT to be able to answer this question, but
are the options provided correct, does the flowchart sequence shown actually
work, do you need to be highly literate in order to understand the question,
would you be disadvantaged if you were a particular gender or of a particular
ethnic group?"
Clearly, what is enjoyable in such discussions is that you grow in your own
understanding of the subject. Also, it is very interesting to hear other
people's views on particular issues. It's the discussion, the interaction, that
moves things on. And sometimes there are moments of humour, like today when I
realised, halfway through an impassioned plea against a particular question,
that I had successfully demolished my own argument!
One of the things I have noticed that distinguishes good ICT departments in
schools from the others is that time is made available for discussion about
assessment. Teachers need to have a clear and shared understanding about what,
say, a Level 5 student is capable of, and they need to be able to assess their
students' performance with a reasonable degree of accuracy.
They also need to be able to articulate to their students what these levels
mean, and what a student must do to progress from one level to the next. They
must also go beyond even this, and help students become, to use Maslow's term, self-actualised in the realm of assessment. In
other words, they must be able to evaluate, accurately, their own
performance.
Unfortunately, time to do all this is the one thing that teachers no longer
seem to have.
Sadly, today was the last of these periodic meetings, because we have now
gone through all the main areas of the ICT curriculum. In an ideal world,
everyone involved would now start back at the beginning, in order to ensure a
constant supply of fresh material, but, for the time being at least, that is not
to be.
In my next article on this subject I will describe a rather different form of
assessment I have been involved in this week, that of using a rubric as a
"meta-judge" for the Horizon 2008
project.
Subscribe to article summaries
"educational technology"
"ICT in Education"