Test Specifications
Test specifications – usually called ‘specs’ – are generative explanatory documents for the creation of test tasks. Specs tell us the nuts and bolts of how to phrase the test items, how to structure the test layout, how to locate the passage, and how to make a host of difficult choices as we prepare test materials. More importantly, they tell us the rationale behind the various choices that we make (Fulcher & Davidson, 2007).
Spec writing is an organic process. Time, debate, consensus, pilot testing and iterative re-writers cause the spec to grow and to evolve and to better represent what its development team wishes to accomplish (Fulcher & Davidson, 2007).
Spec-driven testing lends itself very well to higher-order organisational tools. Specs are, by their very nature, a form of database. Well-organised specifications which produce archetypes seem to produce trust in the entire testing system, and from such trust many difficult decisions can be made (Fulcher & Davidson, 2007).
Test specifications suggest a theoretical model of test creation. A model of spec-driven test development, or ‘spec-driven theory’, seems to have several precepts when it works at its optimal state (Fulcher & Davidson, 2007, pp. 60-61):
- Specs exist. Somehow, somewhere, the system is able to articulate the generative guiding language behind any sample test items or tasks.
- Specs evolve: They grow and change, in creative engineering-like response to changes in theory, in curriculum, in funding, and so on.
- The specs and the test they generate are not launched until ready.
- Discussion happens and that leads to transparency.
- And this is discussion to which all are welcome.
Specs are often called ‘blueprints’, and this is an apt analogy. Blueprints are used to build structures, and from blueprints many equivalent structures can be erected (Fulcher & Davidson, 2007). Specs also serve as a formal record of critical dialogue. Test specifications are not set until the final test is ready to be rolled out in actual use. That’s why the process is much messier, because all the activities feed into each other.
Test specifications should be continually evolving documents, as construct definition becomes more precise and evidence is collected to support the link between the task and the construct (Fulcher & Davidson, 2007). It is the job of the test specifications to make our claim about the links between items and responses clear. Congruence or (fit-to-spec) is the degree to which a new item fits an existing spec.
All test specifications have two components: (1) samples of the items or tasks we intend to produce, and (2) guiding language about the samples. Guiding language comprises all parts of the test spec other than the sample itself. Consider the following as some guiding language tips for a particular test specification:
- This is a four-opinion multiple-choice test question.
- The stem shall be a statement followed by a question about the statement.
- Each choice shall be plausible against real-world knowledge, and each choice shall be internally grammatical.
- The key shall be the only inference that is feasible from the statement in the stem.
