Evidence-Centered Design
Evidence-centered Design was developed by Mislevy.
It is important that we see the tasks or items that we design for tests as part of a larger picture, and one approach to doing this in a systematic way is ECD, a methodology for test design and construction developed at Educational Testing Service (ETS).
ECD is a methodology for designing assessment that underscore the central role of evidentiary reasoning in assessment design. ECD is based on three premises: (1) An assessment must build around the important knowledge in the domain of interest and an understanding of how that knowledge is acquired and put to use; (2) The chain of reasoning from what participants say and do in assessments to inferences about what they know, can do, or should do next, must be based on the principles of evidentiary reasoning; (3) Purpose must be the driving force behind design decisions, which reflect constraints, resources and conditions of use.
ECD tests knowledge as scientific because it is primarily a method that leads to the test designer understanding more about relations between variables for a particular assessment context. ECD is considered to be a ‘framework’ in the sense that it is structured and formal and thus enables ‘the actual work of designing and implementing assessments in a way that makes a validity argument more explicit. It is sometimes referred to as ‘conceptual assessment framework’.
Within ECD-style test specification, there are six models or designs objects, each of which must be articulated:
- Student Model: This comprises a statement of the particular mix of knowledge, skills or abilities about which we wish to make claims as a result of the test. In other words, it is the list of constructs that are relevant to a particular testing situation, extracted from a model of communicative competence or performance. This is the highest-level model, and needs to be designed before any other models can be addressed.
- Evidence Model: Once we have selected constructs for the student model, we need to ask what evidence we need to collect in order to make inferences from performance to underlying knowledge or ability.
- Task Model: We can now see where test tasks and items fit into the picture.
- Presentation Model: Items and tasks can be presented in many different formats.
- Assembly Model: An assembly model accounts for how the student model, evidence model and task models work together.
- Delivery Model: this final model is not independent of the others, but explains how they will work together to deliver the actual test.
A preliminary first stage, Domain Analysis, involves what in performance assessment is traditionally called ‘job analysis’. Here the test developer needs to develop insight into the conceptual and organisational structure of the target domain. What follows is the most crucial stage of the process, Domain Modeling. It involves modeling three things: claims, evidence, and tasks.
Claims: This involves conceptualizing the aspects of knowledge or performance ability to which the evidence of the test will be directed and on which decisions about candidates will be based.
Evidence: It involves determining the kind of evidence that would be necessary to support the claims established instep 1.
Tasks: This involves defining in general terms the kinds of tasks in which the candidate will be required to engage so that the evidence set out in Step 2 might be sought.
All three steps precede the actual writing of specifications for test tasks; they constitute the “thinking stage” of test design. In ECD theory as outlined by Mislevy, with a clear understanding of what we wish to test and how we can describe tasks, it is possible to create highly formalized test specifications in which the production of the item is tightly controlled.
