Apologies for the delay in sending this out. Please see http://www.opurt.com/tag.html if the images in this e-mail do not show up correctly. Victor and I will present this later today. Thanks, Paul

Title: Test Assertions - A Simple Approach

Test Assertions - A Simple Approach

Introduction

* Common thought at Sun is that assertions are "literal" - their text can be found within the spec

* Java specifications - JavaDoc, JLS, JVMS * Most assertion in Java Specifications are literal

* In practice, Many assertions are derived from text (or a diagram, an image, a cell within a table, an element in a formal grammar, etc) found within a specification. * Sometimes an assertion is derived from multiple locations in one specification or from multiple specifications.

Necessary assertion data:

Definition: Assertion - structured data that exists outside of a specification.

Simplest form of an assertion: * assertion ID - (should be unique within the spec and remain static) * assertion text * reference indicating the location in the specification where the text can be found or from where it can be derived.

Two assertion classifications:

* literal - Text can be found directly in the specification text

* JLS Chapter 10 Arrays. 10.7 Array Members "Every array implements the interfaces Cloneable and java.io.Serializable."

* derived - Assertion text is derived from text found within the specification or specifications. * simple - A single reference to a set of text within a specification is sufficient. * compound - More than one reference to sets of text within a specification or specifications is necessary.

At Sun, we work mostly with simple literal assertions.

Our goal:

* Define assertion markup for text within specifications

* Literal and derived

* Markup should be as simple as possible

* simplest case is where a string of text in a specification is an assertion * flexible to handle complex compound derived assertions

* Should be extendable

* Additional attributes could be added to proposed markup

Assertion markup structure

General Structure

Example of an assertion as part of an assertion list: <assertion id="123"> <text [type= "literal" | "derived"]>This is the text of the assertion.</text> <reference [type = "primary" | "secondary"]>Pointer into the spec(s)</reference> [optionally, additional <reference> tags may be provided] </assertion> * literal/derived attribute

* identifies whether the text of the assertion is literally copied from the spec or whether it has been derived from it. * Optional, default is literal.

* primary/secondary attribute

* Optional, defaulting to primary. * allows the construction of "compound" assertions where more than one reference to the spec (or even different specs) is required to create the assertion. * Assertions of this type will probably be of type derived rather than literal. * If literal, the text of the assertion will be found at the primary reference.

Example of Simplest Case

a single literal string within a specification:

<assertion id="123"> <text>This is the text of the assertion.</text> <reference>SpecID:123</reference> </assertion>

The reference into the specification should include a reference to the specification (specID) and a pointer to a location (:123) within the specification. The format of the reference into the spec has not been determined. Some possible options are: * Xpath expressions for XML and XHTML * Section, chapter, sentence, line numbers * references to HTML anchors

Specification markup structure

Sometimes it makes sense to apply markup directly to a specification so that assertion lists can be automatically generated. At the minimum, markup process involves identifying normative text. * Java Language Specification * Java Virtual Machine Specification * Java API Specifications (JavaDoc) - this is typically done with a tool (Spectrac) or on paper with a highlighter

Example:

<normative id="123" [see-also="SpecRef1 [, SpecRef2, SpecRef3...]]"> This is some normative text. </normative>

The optional see-also attribute * allows one or more additional references to other <normative> elements

* Additional references can be in same specification or external specifications.

Example - JLS:

Different format than proposed here - for illustration purpose:

Deriving assertions from marked-up specs - Simple Case

The simplest case of markup within a specification reduces to:

<normative id="123">This is some normative text.</normative>

From this it is possible to automatically generate the simple literal assertion discussed above:

<assertion id="123"> <text>This is the text of the assertion.</text> <reference>SpecID:123</reference> </assertion>

Since this is a literal assertion, the text of the assertion will be the same as the normative text identified within the specification.

More complex cases.

A specification example with secondary normative text, non-normative text, and primary normative text:

<normative id="123" type="secondary"> This is some normative text that provides some general background requirements for a range of cases. </normative> This is non-normative text..... <normative id="456" see-also="specID:123"> This is normative text that states requirements for a particular instance. </normative>

From this text the following compound literal assertion could be automatically generated:

<assertion id="456"> <text type="literal">This is normative text that states requirements for a particular instance.</text> <reference type="primary">Pointer to 456</reference> <reference type="secondary">Pointer to 123</reference> </assertion>

* A human editor might choose in this case to replace the text with something "derived" from both 123 and 456, in which case the text type would be changed to derived. * The primary attribute is optional and could be dropped

Summary

* Simple test assertion framework that allows us to cover the most common cases here at Sun * Can support compound assertions * Does the support for Compound assertions complicate the proposal?

* Would Compound assertions be used in the industry?

* Does the "see-also" attribute complicate the specification mark-up process

* Is it enough for specification writers to only identify normative text? * Test writers could manually link together normative statements to form compound assertions