|Subject:||Rationale for test assertions (homework for f2f)|
|From:||Patrick Curran (Patr...@Sun.COM)|
|Date:||Aug 30, 2007 4:23:00 am|
Sorry this is late. I'll call in to the meeting to present it.
Title: Rationale for Test Assertions
What is a test assertion?
* A statement of behavior, action, or condition that can be measured or tested. * Each test assertion is an independent, complete, testable statement of requirements in the specification. * Assertions are derived from the specification and can be mapped to it.
* In the simplest case, an assertion may simply consist of text from within the specification.
* Assertions provide a normative foundation from which one or more test cases can be built.
Benefits of test assertions
* Improve specification quality by helping to uncover inconsistencies, ambiguities, gaps, and non-testable statements in the specification.
* If not developed by the spec authors, should be reviewed and approved by them. * Best results are achieved when assertions are developed in parallel with the spec.
* Facilitate and promote the development of conformance tests and testing tools. * Encourage the early development of conformance tests that can be used during implementation. * Simplifiy the distribution of the test development effort between different organizations while maintaining consistent test quality. * Improve confidence in the resulting tests as a measure of conformance. * Enable coverage analysis (estimating the extent to which the specification is tested).
* Partition the specification
* By feature, language elements, logical sections, or even paragraphs or pages.
* Define coverage goals for each section.
* What test-development resources are available? * How critical is it that each section is thoroughly tested?
* Consequences for incompatible implementations due to low test coverage?
* For each partition, provide a mapping between testable assertions and test coverage. * Measure or estimate the extent to which each area of the specification has been tested. * Coverage measurements can be expressed in both breadth and depth terms.
* Breadth measurements are relatively simple to derive.
* What percentage of assertions are covered by at least one test?
* Depth measurements are more subjective.
* Estimate how thoroughly each assertion is tested.
* Different coverage goals for different areas of the specification may be appropriate.
* First cover each feature at a minimal level (breadth-first). * Then focus on providing deeper coverage
* in areas that are more difficult to implement, * where incompatible implementations are more likely, * where the consequences of incompatibility will be more severe.
* Provide a coverage report with the test suite.
* At a minimum, a simple mapping of tests to areas of the specification. * Preferably include counts and averages of the number of tests associated with different areas.
* Coverage reports help users of the test suite understand its strengths and weaknesses.
Guidelines for creating test assertions
* A test assertion should be a simple, atomic statement.
* Address one feature at a time. * Keep each assertion as simple as possible.
* Multiple single-feature assertions are easier to test than complex multi-part assertions. * Simple test assertions make it easier to identify the cause of test failures and to exclude buggy tests.
* Map assertions to text within the specification. * Don't change the semantics of the specification. * Group assertions into logical sets, following the structure of the spec. * Provide a unique ID for each assertion to facilitate tool development. * Test assertions must be technology-neutral (don't assume features of the implementation). * Do not state how to test (a test assertion is not a test description). * Indicate explicit dependencies and constraints. * Describe features, values, attributes to be measured and indications of success or failure.