Sorry this is late. I'll call in to the meeting to present it.

-- Patrick

Title: Rationale for Test Assertions

What is a test assertion?

* A statement of behavior, action, or condition that can be measured or tested. * Each test assertion is an independent, complete, testable statement of requirements in the specification. * Assertions are derived from the specification and can be mapped to it.

* In the simplest case, an assertion may simply consist of text from within the specification.

* Assertions provide a normative foundation from which one or more test cases can be built.

Benefits of test assertions

* Improve specification quality by helping to uncover inconsistencies, ambiguities, gaps, and non-testable statements in the specification.

* If not developed by the spec authors, should be reviewed and approved by them. * Best results are achieved when assertions are developed in parallel with the spec.

* Facilitate and promote the development of conformance tests and testing tools. * Encourage the early development of conformance tests that can be used during implementation. * Simplifiy the distribution of the test development effort between different organizations while maintaining consistent test quality. * Improve confidence in the resulting tests as a measure of conformance. * Enable coverage analysis (estimating the extent to which the specification is tested).

Coverage analysis

* Partition the specification

* By feature, language elements, logical sections, or even paragraphs or pages.

* Define coverage goals for each section.

* What test-development resources are available? * How critical is it that each section is thoroughly tested?

* Consequences for incompatible implementations due to low test coverage?

* For each partition, provide a mapping between testable assertions and test coverage. * Measure or estimate the extent to which each area of the specification has been tested. * Coverage measurements can be expressed in both breadth and depth terms.

* Breadth measurements are relatively simple to derive.

* What percentage of assertions are covered by at least one test?

* Depth measurements are more subjective.

* Estimate how thoroughly each assertion is tested.

* Different coverage goals for different areas of the specification may be appropriate.

* First cover each feature at a minimal level (breadth-first). * Then focus on providing deeper coverage

* in areas that are more difficult to implement, * where incompatible implementations are more likely, * where the consequences of incompatibility will be more severe.

* Provide a coverage report with the test suite.

* At a minimum, a simple mapping of tests to areas of the specification. * Preferably include counts and averages of the number of tests associated with different areas.

* Coverage reports help users of the test suite understand its strengths and weaknesses.

Guidelines for creating test assertions

* A test assertion should be a simple, atomic statement.

* Address one feature at a time. * Keep each assertion as simple as possible.

* Multiple single-feature assertions are easier to test than complex multi-part assertions. * Simple test assertions make it easier to identify the cause of test failures and to exclude buggy tests.

* Map assertions to text within the specification. * Don't change the semantics of the specification. * Group assertions into logical sets, following the structure of the spec. * Provide a unique ID for each assertion to facilitate tool development. * Test assertions must be technology-neutral (don't assume features of the implementation). * Do not state how to test (a test assertion is not a test description). * Indicate explicit dependencies and constraints. * Describe features, values, attributes to be measured and indications of success or failure.