Date: Thu, 10 Feb 2000 10:10:13 +1100
To: voced-coord@rite.ed.qut.edu.au
From: Shelley Gillis <s.gillis@edfac.unimelb.edu.au>
Subject: Re: Enhancing validity of assessments!

Validity

Paul raised the issue as to the nature and importance of validity in CBA? There are a number of different types of validity that are considered when validating an assessment, with the most widely cited being 'face', 'content', 'criterion-related' (predictive and concurrent), 'construct' and 'consequential'. Each of these validity forms are described in
Gillis,S., & Bateman, A. (1990), "Assessing in VET: Issues of reliability and validity, NCVER: Adelaide.

(details of this publication can be found at the following URL address http://www.ncver.edu.au/cgi-bin/getlist5.pl

Below is a quick definition and examples of different types of validity.

FACE
The assessment tasks should be designed to look like they are assessing what they claim to be assessing. If assessing computing competencies, the assessment task could be designed to collect direct evidence of computing skills through practical demonstrations or simulations. Assessment of the underlying knowledge and computing skills through paper & pencil tests alone would not have face validity. It should be noted however that although there is widespread agreement that face validity is not a true form of validity in the technical sense, its importance lies within the acceptance and credibility of the assessment outcomes by the key stakeholders. Given the need for industry to accept the assessment outcomes of the VET sector, the importance of face validity in a CBA system cannot be underestimated.

CONTENT
Concerned with the extent to which the skills & knowledge demonstrated in the assessment task constitute a representative sample of the skills & knowledge to be exhibited in the competency standards. Content validity is satisifed when there is a direct match between the required knowledge and skills specified in the standards and the assessment task's capacity to collect such evidence.

CRITERION RELATED
Subdivides into concurrent and predictive validity:

· Concurrent -concerned with comparability and consistency of a candidate's assessment outcomes with other related measures of competency. Evidence of competence on one task should be consistent with evidence of competence on another related task (eg on and off the job assessments that are measuring the same unit of competency should provide consistent evidence of competence levels).

· Predictive validity - Concerned with the ability of the assessment outcomes to accurately predict the future performance of the candidate and how the candidate will be able to apply the knowledge and skills to new or other situations outside the context of the assessment event (ie transferability). Assessments should be able to predict if the candidate will be able to apply the relevant skills in knowledge in the workplace

CONSTRUCT
Concerned with the theoretical evidence of what is being assessed. Constructs are non-observable qualities such as attitudes and values, competencies and learning, which require inferences to be made by the assessor. A construct is a way of organising observations to help interpret them. Construct validity is concerned with the degree to which the evidence collected can be used to infer competence in the intended area, without being influenced by other non-related factors (such as literacy levels etc).

CONSEQUENTIAL
Concerned with the consequences of the use of the assessment information for all stakeholders (eg. hidden agendas, funding influences, maintaining pre-established relationship and continued employability of candidate/assessor). These consequences may influence the way in which assessors make the judgements of competency.

From these definitions is can be seen that validity is not simply a property of the assessment task. An assessment task that is highly valid for one use or context may be invalid for another.

Whilst the methods used to collect the evidence will impact on the reliability, the way in which assessors use and interpret the evidence collected will impact on validity. When establishing validity, the process of gathering and interpreting evidence appears to be the most crucial component of the assessment process. Similarly, the interpretation of evidence, as well as the way in which it was gathered, influences reliability. Assessors need to take into account they way in which evidence is collected, interpreted, synthesised and evaluated to make an overall valid and reliable judgement of competence.

In simple terms, validity is associated with the use and interpretation of the evidence collected, whilst reliability is concerned with precision and accuracy of the evidence and procedures used.

I hope this is helpful and encourage people to purchase the publication from NCVER as it provides much more detailed examples and guidelines.

Shelley Gillis
Research Officer
Centre for Vocational Assessment Research
Assessment Research Centre
Dept of Learning Education and Development
Faculty of Education
The University of Melbourne
Parkville VIC 3052
Ph 03 9344 8572
Fax 03 9344 8790

previous message · message index · next message

First published February 10, 2000. Last modified February 24, 2000.