Why acceptance tests should be signed off with formal requirements

April 16 2010

Under most models of software development of any significant scale, user sign-off of requirements is a must before the development can continue.  Even agile methodologies that involve a customer representative in the development team needs to have a set of at least significantly complete requirements before development can begin.

Why do we need firm requirements?

Requirements form the basis for generating a Tender or Quote and help generate a Statement of Work.  Detailed requirements are often elicited during contract negotiations and once established, a fixed price can be agreed (if that is the model being used). Requirements sign-off is all important in formally contracting what needs to be provided in any software project deliverable.  It is the signed requirements document that is brought out when there is confusion about items being in or out of scope, and determining if a product has been completed.

Defining 'done'

A hopefully typical contract stipulates that software is "done" when all Acceptance Tests have passed.   Although our contracts often stipulate that once all sites are Live the software has been accepted and the project is "done", though in my experience this is painful because reasons for not going Live are quite often more political than technical or caused by any software defect, and it's these situations where Acceptence Test success would greatly speed up the "Done" state of a project (and the "paid" state of the Invoice).

My company traditionally produces an "Operational Scenarios Document" (OSD) as the document indicating the requirements and to be signed off the customer.  The OSD is simply a document containing all the user stories or use cases of the system to be built.  I believe this presents a more understandable view of the system to the customer, as it's written in their language.  From the OSD Analysts produce a more engineering oriented "Software Requirements Document" (SRD), where the OSD is broken down into hundreds and thousands of numbers system requirements.

The problem with this approach is the OSD only captures user facing actions such as steps in a business process.  It is up to the SRD to capture the more fine grained requirements, such as "Patient information shall include a valid Medicare number in the format XXXX-XXXX-XXXX X".  As such, we've captured the necessary requirement, but there's nothing to formal (read "legal") to cover either us or the customer.  Generally this isn't a problem, but we've probably all experienced scope-creep and ambiguous (signed) requirements are the genesis of scope creep.

Enter signed Acceptance Tests and Acceptance Test Driven Development (TDD at a higher abstraction).  Acceptance Tests define exactly what the system has to do before the customer will consider it "Done".  Acceptance Tests don't just include functional requirements; they also test the non-functional stuff like performance, security, and useability.  If functionally is not covered by the Acceptance Tests, it's not required by the customer.  Following on the YAGNI principle.

How to define User Acceptance Tests

Now, I'm not saying that no software project should be undertaken before Acceptance Tests are supplied.  That's clearly unworkable in all but the situations where the customer team has experienced analysts, with a good understanding of the software development process, on board.  What I do think is that an analyst or developer should always be mindful of Acceptance Tests when eliciting requirements from customers during workshops.  Questions like "What are the steps you take when placing an order" are not enough.  "What state will the order be in once it has been successfully placed?", or "What happens if an order is placed and there is not enough inventory to fulfil it?" are necessary. Hopefully these are already questions that are being asked as part of the user story generation.

User Acceptance Tests versus Operation Scenarios

The differences between Acceptance Tests and Use Cases (or OSD's) are small but significant.  Where an OSD is used by the customer to accept that you understand the requirements of the software under design, Acceptance Tests are used by customer to determine if you supplied what they asked for.  Both OSD and Acceptance Test Document can be generated from the same information at the same time, so it makes sense to generate them at the same time and include both in any formal signed requirements.

How do I use User Acceptance Tests in the development process?

From a development perspective, you can use user stories/use cases/operational scenarios in tandem with acceptance tests to define the use cases to be developed in each iteration, producing software that is of business use after iteration one and the customer can hook into the feedback loop early in a project.

Critically, Acceptance Tests are generally customer supplied artefacts and as such, you as a developer have a higher level of confidence that developing a system to meet Acceptance Tests will result in a successful project.

Post a comment

comments powered by Disqus