Date: Thu, 17 Feb 2000 12:13:32 +1100
To: voced-coord@rite.ed.qut.edu.au
From: Shelley Gillis <s.gillis@edfac.unimelb.edu.au>
Subject: assessment decision making models

Thank you J Kerr for providing an example of how you integrate practical performance on classroom-based tasks, with completion of homework and mathematical performance to determine grades (message repeated at end).

These are what we refer to as decision making rules for multiple sources of evidence. From the information provided by J Kerr it seems as though he has adopted the conjunctive model of assessment decision making, but I may be wrong and please correct me if I am (refer below for descriptions of decision making models).

A sound competency based assessment system encourages the use of a multitude of assessment methods and tasks to gather evidence of competence. However, the use of a combination of assessment approaches yields considerable information about the competency of a student, that must be synthesised by the assessor to make an overall judgement as to whether the competency standards have been met. This assimilation process requires the assessor to evaluate the assessment information collected against pre-specified decision rules that can be internally (like J Kerr's) or externally imposed (eg Board of Studies).

A number of alternative decision making models for using multiple sources of evidence have been identified by educational researchers. Four of these are described below.

Compensatory Model
The Compensatory Model for using multiple sources of evidence can trade off weak performance on one measure with strong performance on another measure, during the deliberation process. Usually, this model assumes that there is a minimum level of performance that must be demonstrated by the student and explicit decision rules must be specified for minimal levels of performance for each task. The decision as to whether unacceptable performance on one task (eg. low ratings of performance by peer observations) can be compensated by outstanding performance on another task (eg high ratings of performance by the supervisor) will depend on a number of factors such as the critical nature of the competency to be demonstrated, the frequency of application of the competency to the workplace and the value placed upon that particular assessment task or source of information. Developers of assessment tasks and assessment policy documents will need to determine the critical nature of the competencies and to set minimum levels of performance for each source of evidence used. If using multiple sources of evidence, then again, ways in which to handle discrepancies in the information sources will need to be addressed at the system level, to ensure the credibility, reliability and acceptability of the system.

Conjunctive Model
The Conjunctive Model for using multiple evidence sources requires that the student demonstrate a minimum level of performance on each of the assessment tasks administered by the assessor. This approach is suitable when the competencies to be assessed are critical for minimal acceptable performance within the industry. Assessors who conduct assessments within high risk or emergency situations often adopt this model where a wrong assessment decision could place either the candidate or his/her peers in danger. An example of this approach is in attainment of a driving license, where the candidate must pass a written test (80% or higher), a vision test and a practical driving test prior to the issuing of the license. Again minimal acceptable levels of performance on each of the assessment tasks have to be determined and documented prior to use by the assessor.

Combined Model
This model has features of both the Compensatory Model and the Conjunctive Model and is usually applied when there are two or more decision making levels (Wheeler, 1993). An example of this is where assessments are being carried out as part of a larger assessment system. For instance, a workplace trainer may adopt the Compensatory Model for assessing an employee against a unit of competency (ie using trade-offs), but a training provider may adopt the Conjunctive Model and insist on satisfactory performance on all units of competency prior to issuing a national qualification.

Disjunctive Model
Unlike the first three models discussed above, this model does not assume that there are minimal levels of performance that must be demonstrated for each assessment task for an overall competent decision to be made. According to this model, the student needs only to demonstrate competent performance on one of the assessment tasks (for example, a peer report that indicates that all of the competencies have been demonstrated to the level of performance required in the standards). The assumption is that if the candidate can demonstrate competent performance on one assessment task (ie peer observations) there is no need for the candidate to be further assessed in that area (eg supervisor observations or portfolio etc). This however is dependent on an implicational relationship being demonstrated between tasks. That is, performance on one task implies a certain performance on another task, which have implications for transferability of the competencies. If transfer is required and can be demonstrated at the development stage of the assessment procedure(s), it should not be necessary to assess every task, ever time and in every context. Despite the cost-effective nature of this approach, the assessor's confidence with predicting the student's consistency of performance and transferability of competencies to new situations remains questionable. Issues associated with the evidence requirements, such as validity, reliability and sufficiency may emerge with the adoption of this model that relies upon satisfactory performance from one source of evidence only.

When multiple sources of evidence are used to gather evidence of competence, there must be a sound rationale for the selection of the decision-making models and the assessment procedures to be applied. Furthermore, to maximise inter-rater reliability of the assessment decisions, the decision-making rules must be clearly specified and made explicit to both assessors and students.

Some questions to be explored:

1. Can people actually identify which decision making model are being used in their organisation (ie applying the theory to practice)?
2. Are there any others?
3. How do each of these models fit within a CBA framework?

Shelley

J Kerr's message (from the USA - referred to at start):

At 10:06 15/02/00 -0700, J Kerr wrote:

I teach all competency based courses relating to residential construction. I have listed hands on projects required for each course. Students will accomplish each one correctly according to good and sufficient workmanship and appropriate building code regulations in a safe manner. My job is to make the tasks bite sized.

For example, in block masonry, one is required to cut a CMU with a powered masonry blade at a 90 deg angle. We discuss the use of safety shield and gloves, instructor demonstrates one cut and student demonstrates by aping the instructor. I would then check off the student on the task list as having completed that project. So long as the task was completed successfully and safely, it is complete. This does not make the student a skilled masonry cutter. However it does make one able to replicate that process successfully again, since I ask. "Could you do that again and make the block a required length ?" It is more objective than subjective, I think. He must have completed 80 % of the homework and related math problems successfully to keep a B average in class.

J Kerr

 top

previous message · message index · next message

First published February 17, 2000. Last modified February 24, 2000.