|step...@systml.co.uk||Sep 21, 2007 2:19 pm|
|Dave Pawson||Sep 21, 2007 10:10 pm|
|step...@systml.co.uk||Sep 22, 2007 1:47 am|
|Dave Pawson||Sep 22, 2007 2:06 am|
|step...@systml.co.uk||Sep 22, 2007 2:46 am|
|Dave Pawson||Sep 22, 2007 5:25 am|
|step...@systml.co.uk||Sep 22, 2007 1:43 pm|
|Dave Pawson||Sep 22, 2007 10:09 pm|
|step...@systml.co.uk||Sep 23, 2007 5:37 am|
|step...@systml.co.uk||Sep 23, 2007 5:51 am|
|Dave Pawson||Sep 23, 2007 6:11 am|
|step...@systml.co.uk||Sep 23, 2007 8:22 am|
|Dave Pawson||Sep 23, 2007 9:15 am|
|Subject:||Re: [tag] TA Model|
|Date:||Sep 23, 2007 5:51:18 am|
Another example of need for IDs on not just the TA but also pre-conditions, results, etc (not a very good one I admit but just to demonstrate my reasoning)
Here it might be a test with two Schematron validation passes, say (if the spec includes two Schematron schemas for consecutive use)
The first validation with a test method might itself have a pre-condition that the XML of an XML instance item under test is valid XML but that is not tested during that type of validation. The second validation has two pre-conditions, one that the first step's result/effect was a pass and another that the first step's pre-condition was met. The desire to automate production of tests from TAs might require the use of IDs and references to those IDs in the steps of the flow of the TA.
-- Stephen Green
Partner SystML, http://www.systml.co.uk Tel: +44 (0) 117 9541606
Quoting Dave Pawson <dave...@gmail.com>:
So how about
1. introducing steps called 'Test' into the flow with IDs, 2. making such 'Test' steps multiple occurrence
.. of what Stephen?
3. adding a unique ID to each part of the step 4. allowing preconditions and postconditions to reference parts of another step
Why? IMHO a test should be independent. Dependencies on other tests goes against this. Perhaps such a test is at too low a level?
An example I'm aware of where a test depends on the pass outcome of a previous test is a Schematron validation test isn't worth performing (and therefore can't result in a pass) unless previously there was a successful pass outcome to a test of the XML syntax such as with XML Schema or RelaxNG. If the XML Schema validation test was in a previous TA then there would be a need to make that TA's test result a pre-condition which I propose could either be in the pre-condition description or be by a reference to the TA test's result or to the TA as a whole. Hence the need to have an ID in the TA as a whole but also, I think, in the Test result to allow greater precision. After all the pre-condition might be to a post-condition of the previous test - to something that isn't actually tested formally but is just found to be true (boundary clarity needed here?).
There may, however, be reason to include the XML Schema validation in the same TA as the Schematron validation (feasible but I'll take it as an unproven assumption) but to put the two validation phases into two tests in the flow in such a way that the dependency can be expressed formally. Say the second test step of the flow uses an ID from the first step to express the dependency. It could include a reference in the pre-condition of step 2 to the first step's result (which requires a unique ID on that result).
So to summarize so far, things like pre-conditions might need to reference either things like triggers, results/effects (I too prefer 'result' to 'effect'), pre- and post-conditions in tests so each need unique IDs *or* they may need to reference the final outcome of a test flow as a whole so that needs an overall unique ID which could just be the ID of the TA.
I could imagine there might be some situations where the pre-condition might need to reference something other than the entire TA or the result of a particular step/test in a multi-step/test flow. If there is an aspect of the referenced TA or step which isn't actually formally tested it could be something which is a post-condition. So the post-conditions need unique reference too to facilitate that.
Either way, I see these as all aspects of a spec (the example is part of the in-progress OASIS Codelist Methodology and plays a part in UBL's spec) and therefore to need TAs and not therefore just for the test designers. They live in the space between spec and tests in such cases. (Two-phase codelist validation is the example I worked from.)
5. adding 'Previous Test' and 'Following Test' pointers to each step
I find this confusing. Test flow / routing shouldn't be set by the test. It may be that a test could be re-used in different places?
6. adding 'Comment' outside the flow 7. adding something to state what constitutes overall pass or fail of TA 8. adding specification quote or reference. By whatever means are appropriate. 9. being explicit in the name of the part about it being an ID or reference
I don't understand this one.
Is that too complex or just complex enough? Maybe most things are optional I suggest where there is a reference such as 'Test Trigger Cross Reference' then it could be implemented as a hyperlink if the implementation is HTML (so it might not add complexity all that much).
Test Assertion ID: TA-101c Specification requirement ID: < req 101> Specification requirement quote: "Purchase Order Receipt documents must be XML documents that are valid against the XML schema po-receipt.xsd, and that must use the approved format for product identifiers." Item under test: A business endpoint. Assertion prose: "Purchase Order Receipt documents must be XML documents that are valid against the XML schema po-receipt.xsd, and that must use the approved format for product identifiers." Comment: Test TST#112 is only required when test TST#111 results in a warning Test Assertion Pass Criteria: Test Assertion results in 'pass' if no test result is 'fail' and all required tests are run.
Except for the test criteria, a good example. Pass conditions must be explicit and UUT oriented. This would might be 'Parser must not report any errors when validating the instance.
Other than that this seems to be about the right 'level' of abstraction. Says 'what' to test. Not how. Is independent of other tests. (Might add a precondition: An xml instance - PO receipt, has been received by the end point). Post condition might be, PO receipt on disk as file identified by ...PO ident number with XML extension.
I'd generalise that the 'right' level of abstraction is at, or slightly lower than, the sentence level in the specification. The usual caveats about no conjunctions etc. Unsure if its reasonable to go much beyond such a generality.
-- Dave Pawson XSLT XSL-FO FAQ. http://www.dpawson.co.uk