Dierk Konig wrote:

Dierk, Erik, et.al. Since this started with something from my thread about unit/functional testing, I'll have to throw in Tim and my $.04. Sorry about the length, but this is a philisophical discussion, not a technical one :-) True enough, unit and functional tests are indeed very different. Let's add a third term though: acceptance tests! The definition would be something like: "tests that a customer defines in order to indicate that they will accept the system as complete once they all pass". In Extreme Programming these acceptance tests are usually defined to correspond to stories (short descriptions/reminders) the customer has written about individual pieces of required functionality. With this approach, a specification of a system is a large collection of stories on index cards, conversations, and customer-defined acceptance tests instead of outdated documents that are expensive to produce and maintain. Once the development team has made all of the acceptance tests pass, the system is considered done (for the moment, anyway, until the customer adds or changes tests). These acceptance tests consistently seem to cross back and forth over the line between unit and functional tests because the customer cares about testing both the logic and the assembly of components. They are usually testing from the user's perspective, doing operations like logging in and running a couple searches, or trying to add records. Naturally, from that perspective, they want to ensure the user sees the right data, as well as that it shows up in the right place. For most people doing Exreme Programming now this takes the form of the customer defining test steps in a spreadsheet or other document, and programmers writing quite boring source code from it. Quite a waste of time, don't you think? What would be best is if we could empower them to manage the tree of test specifications and test data, and change them when their needs change with very little training. In order for that to be a real win, they need to be able to define them in an executable form, so that the developers could spend their time and the customer's money developing unit tests, functional tests, and source code for the system, not copying test specs from English (or German) into source. We originally thought that Webtest might be the tool to use in defining acceptance tests (for a web application, anyway), but it isn't. Webtest does what it set out to do quite nicely! However, it was clearly built as a _functional_ testing tool for _developers_ to use, while what is needed for Extreme Programming in the web environment is an _acceptance_ testing tool for _customers_ to use. It seems that Herr Ostermueller is also discovering that his needs are more along the lines of an acceptance testing tool, as opposed to a functional testing tool, since he wants to delegate some of the testing to non-programmers. Erik, correct me if I'm wrong, but I suspect that you want to enable these non-programmers to define tests as simple use cases that verify whatever they feel is important! Since they aren't programmers, they don't know or care about the distinction between functional and unit tests, so what they feel is important will likely include a lot of logic testing. Clearly, this is all just opinion, but I think you'll find that the distinction between "functional" and "acceptance" tests is a useful one. In the case of WebTest, this distinction helps illuminate why it doesn't fully meet our needs, and why we'd like to have access to more of HttpUnit's interface. Attempting to build an "AntHttpUnit" for acceptance testing, would, we believe, require a somewhat different outlook from the beginning, and most importantly, non-programmers helping to define the schema. So, that said, what do people think? Is this distinction (acceptance/functional) helpful to you? What would be useful in an acceptance testing tool?

Ben && Tim

Dierk König wrote: