Guest posting to voced-coord email list
Item 5 of 8: Levels of Competence

Date: Mon, 24 Nov 1997 18:30:24 +1100
To: voced-coord@owl.qut.edu.au
From: Patrick Griffin (p.griffin@edfac.unimelb.edu.au)
Subject: Re: VocEd:Patrick Griffin (guest) - levels of competence

There have been a few comments about competent and highly competent and whether this is against the whole idea of competency based training and assessment. An interesting idea, that is or could be against the spirit of CBT. Are standards so fixed that they are immutable? How does competency based assessment fit with competency based assessment. I have a few thoughts on how some of these might go together and have a firm view that improvement in competency cannot be achieved unless we at least acknowledge that competency is never fossilised.

What are criteria and criterion-referenced assessment?

Lists of outcomes and competencies are necessary, of course to record the assessments and communicate them to other people. However, the current generalised use of checklists of achievement and performance is more in line with the corruption of criterion-referenced approaches of the 1970s and 1980s. The emphasis on checklists and atomistic approaches to every outcome and competency indicates a confusion of mastery learning with outcomes and competency assessment. The emphasis on mastery learning is made worse by compulsory use of checklists, and this can lead to a breakdown of an competency based learning. Perhaps both pre-service and in-service teacher education may need to shift to emphasise outcomes based education, the role of assessment and criterion-referenced interpretation in competency based assessment particular.

Technically speaking, criterion-referenced interpretation requires a number of levels, each representing the division between increasing levels of competence and a learning pathway overall. Checking where an assessee is in relation to a series of thresholds does several things; first, it illustrates what has been achieved so far; second, it illustrates what the assessee can do now; third, it forecasts where the learning patheay that is likely to be followed. Using competency assessment to link only training for non-competence (or 'cannot do now') is a deficit model and, while important, is of limited use; the 'can do now' information should enable us also to forecast laterally into new contexts and different learning outcomes, and give breadth to the interpretation.

A criterion describes the point at which we decide something changes from one state to another or moves from one level to another. Criterion referencing is the process of referring our evidence to this threshold in order to decide which side of it the observation best matches. Thresholds, or criteria, are however notoriously difficult to define or describe.

Some years ago, my colleague Glen Rowley, from Monash University illustrated how difficult defining the levels can be when he wrote a parody mastery learning, focusing on the question of beardedness. When do we decide that a man has a beard? How long do his whiskers have to be before we stop describing him as unshaven and start describing him as bearded? In the workplace the same kind of description is needed when a decision is required regarding the transition from 'not (yet) competent' to 'competent' or to 'highly competent'.

The following are several well-known examples of these transition points or thresholds.
Pass | fail
Not (yet) competent | Competent | Highly competent
Honors | Credit | Pass
Level 1| Level 2| Level 3| Level 4| etc. - (Australian Qualifications Framework) (National Profiles)
Clean shaven| unshaven | stubble | designer beard | bearded | full-beard

An eminent educator (Elliott Eisner), who developed the idea of the teacher as a connoisseur, described the use of criteria in the role of the connoisseur or expert judge. He argued that the use of criteria involves an exercise of judgement, the provision of reasons for the judgement and an understanding of the relevance of criteria to the learning area assessed. While this may seem difficult to achieve, competency standards provide excellent vehicles for developing appropriate forms of criterion-referenced interpretation. But there may be a need to agree that levels of competence exist in order to even use criteria and make even the simplest of judgements.

Standards and benchmarks

No discussion of criterion referencing is complete without a mention of standards. Standards for today won't be acceptable in ten years time. The threshold we use to make a distinction between 'adequate' and 'unacceptable', or between 'competent' and 'not (yet) competent' is established using the experience of a group and an understanding of what can be reasonably expected. It is a norming process, based on an understanding of the task itself - and it can change, as can the resources that may be used. 

Standards are not fixed. They are commonly adjusted as we seek better or higher quality outcomes. The very fact that we acknowledge improvement of standards means that we implicitly acknowledge the existence of the underlying continuum that allows a standard to move; standards, therefore, are defined as one threshold or a continuum. In a competitive environment, we monitor each other's performances and identify examples where the standards are higher than those we set for ourselves; this is a process of benchmarking, where we observe others' standards in order to adjust our own. A benchmark, then, is a standard or threshold to which we aspire.

When we understand how and why standards change, we will be in a better position to pre-empt these changes in developing better teaching and training. Those who argue for simple (or simplistic) standards-referenced assessment have lost sight of the relationship between criteria and standards. Standards' referencing is simply a subset of criterion referencing with one threshold on a continuum with two levels. Competency-based assessments, which are commonly restricted to a judgement of not (yet) competent/ competent, are a similar type of criterion-referenced assessment. They all refer to a criterion scale for interpretation and inference. 

If there are two levels now and we move the cut point, then we implicitly acknowledge the existence of higher levels of competent. Agreed, highly competent and competent with merit are subsets of competent. But they allow the learning pathway to emerge. They also allow individuals, enterprises, schools and industries to describe the improvement targets and show trainers, teachers, and students the learning pathway. If there can be two levels which change as standards improve, why cannot there be three or more levels.

I can't send graphics to illustrate the point so try this exercise. Draw a horizontal line and cross it with a vertical line, One side of the vertical line is not competent, the other is competent. Allow the line to move, so draw another further to the right. What do we call the section between the two vertical lines? How far can the standard be allowed to improve? Where do we set the benchmark? Are our levels the same as everyone else's? Would another person draw the cut line at another point on the horizontal line? Does each of these represent a standard?

Try having a look at the slides at the following address.

http://128.250.153.116/inaug/assess/index.htm

Patrick Griffin