How we build quality software

I realised after looking at what I’ve blogged so far that I haven’t written much about actual testing, my first discipline. So, today I’ll provide an insight into how we develop and test software in our department. We have adopted the stance that having a role to find defects is wasteful and effort should be made to ensure such defects are not written (more on this soon). Thus, we do not have testers. However, our developers know how to test; developing with a focus on quality throughout underpins our process.

I’ve mentioned previously about baking-in quality and not having developers throw code over a wall to testers. I also mentioned automating the execution of acceptance criteria, written to define features, in order to aid development today and form regression tests tomorrow. Today I’ll give a very brief overview of each stage (most of which may sound obvious to anyone practicing agile); I’ll go over each stage in greater detail in later posts (my promise to you).

Naturally, the following outline is within the context of our team but there are variations to practices and toolsets that we use. These depend on what we’re working on (web/desktop) to appetite for risk (safety critical/profit).

Overall: Baking quality in with XP, scrum and lean principles

My team consists of 3 developers and myself (I am considered a developer, albeit not as good as the others, but with a better fault model). Everyone on the team is concerned with not only assuring quality in what we deliver, but making it visible to ourselves and the business.

We work in an agile manner, iterating through development with extreme programming practices and Behaviour Driven Development. Facilitating our relationship with the business is Scrum and we utilise kanban principles and systems thinking to maintain a speedy throughput of high-quality work. This mixture allows us to communicate effectively, develop the correct features properly and continuously deploy our work when it is complete, thus maximising business value. I should also mention that we are fortunate enough to have our business people/customer sat across from us.

From start to end: Starting with a User Story & Acceptance Criteria

I have written about user stories, acceptance criteria and how we improved the clarity of them in the teams within our department. Our stories describe a small, releasable piece of functionality and are supported with thorough acceptance criteria, (the level of thoroughness depends on the context and risk attached). The cognitive effort put into traditional end-of-cycle testing is used upfront to obtain these criteria. By anticipating and exploring what we should deliver before we start coding, we have a development goal to aim towards and know we are done once these are met.

Automating Acceptance Criteria

Our acceptance criteria are our acceptance tests, (like a Möbius strip). We automate our acceptance tests in one of the following two ways.

If the feature relates to user interaction through the front end, e.g. pages on a website, we write the acceptance criteria in Cucumber. These are in plain text files in a given-when-then format. At the top level, Cucumber runs Watir, which manipulates the browser. We  develop with BDD to make the Cucumber acceptance tests turn green as they execute.

If the feature relates to more data-driven/domain logic then we represent the acceptance tests in Fitnesse. Fitnesse uses fixtures to interact directly with domain objects. In their simplest form, tests are written in Excel-like tables within a wiki, (which helps when collaborating with business people) and turn green when the system under test returns the expected output.

BDD and Pairing With Awesome Developers.

Practicing BDD allows us to deliver correctly working, tested software. Firstly, a failing acceptance test scenario drives development. Dropping down into the code, red-green-refactor cycles with nUnit specs help design, document and test internal domain objects. These unit test cycles are repeated until the inital failing acceptance test that we set out to pass is doing so.

When coding we pair-programme (using the pomodoro technique). Alistair Cockburn and Laurie Williams’ paper “The Costs and Benefits of Pair Programming” describes the many advantages to pair programming.

No Testers, Testing or Inspection (being lean).

Huh, no testers?! Without testers or a QA team there is no wall over which work can be thrown and the responsibility for quality absolved. We as developers are responsible for delivering quality, therefore we focus on quality. This department-wide stance is achieved by solid direction from department heads, like our Development Manager.

The inspection typically carried out end-of-cycle only yields bugs that were low severity and of no real impact to the end user. The fallacies of testing hold true, not everything can be tested and not all bugs will be found (that is, if you want to get to market), so we put the right bugs live. Following point 3 from Deming’s 14 Points: by doing more upfront (in the form of acceptance) and focussing on and building in quality, we eliminate the need for “mass inspection”.

Error logging and Customer Experience tracking tools, like TeaLeaf, provide instant feedback on any issues that do happen to creep into production.

Continuous Integration and Continuous Test Runs

An agile testing must-have, we use TeamCity to continuously run our unit tests on each check-in. We also execute our Cucumber acceptance tests on scheduled runs. The status of the builds are visible on dedicated monitors around the office as well as a nice 6′ projected screen.


The most radical part of our process is probably the lack of traditional testers. We do not, however, lack testing. We focus on and assure quality in what we develop as we develop it. As a result we have seen the quality of output improve, the rate of throughput increase and our developers thrive as we constantly deliver correct, tested software.


10 Responses to “How we build quality software”

  1. 1 Paul Carvill 17 August, 2009 at 10:48 am

    Hi Hemal,

    Who carries out your end-of-cycle ‘inspection’? And have you managed to bake any quality assurance into your client-side development i.e. JavaScript and the consistent rendering of the user interface itself?

    I’ve found the JavaScript side of things is easily handled with JSMock or JSUnit, or a homebrew solution. It also requires competent developers with a specific understanding the peculiarities of JavaScript.

    The user interface is a trickier proposition, usually relying on real people and ‘eyeballs’ to infer any errors or inconsistencies, and typically producing the majority of bugs in any piece of web software coming out of a large development team.

    I’ve found the answer is to focus on streamlining the client-side development process itself, getting developers working in that area to think and behave in a more traditionally ‘programmatic’ way, reducing the codebase and going back to the original principles of the cascade when working on cascading stylesheets.

    I wonder what your thoughts and experience are?

    • 2 Hemal 17 August, 2009 at 7:28 pm

      Hi, Paul

      We do not do any end-of-cycle inspection. WRT client-side javascript, we don’t use a great deal as we don’t have any really rich client-side functionality. Any that we do use are from libraries such as jQuery. Naturally this is tested as it is implemented in all appropriate browsers.

      In terms of UI, we have a central UI library that all web pages are built from so there isn’t a great deal changing.

      We do have a minimal set of face-saving tests which run manipulating the browser and we treat as a slide show.

      We have tests that verify deployments to other environments by checking certain canary pages report correct output and that services are running as expected. We also check for specific resources like CSS files and javascript libraries to ensure they were deployed.

  2. 3 Nathan 17 August, 2009 at 3:01 pm

    Hi Hemal,

    Really interesting post. It seems that your team are following similar practices to the team I am currently on – except you are doing them better and seem to be further along the line than we are.

    I am the only test engineer on a team of about 12 total developers, and as a result, I am very busy. I think the problem is that I am here at all. If I wasn’t here, then the developers would be forced to confront their own testing issues, whereas at the moment, I am viewed as the one who tests everything. Ultimately I hope my team will follow the approach of your team, and begin to ‘bake in’ quality using techniques such as agile acceptance testing and BDD – here’s hoping.

    Looking forward to your upcoming session at Skillsmatter.

    • 4 Steven gordon 17 August, 2009 at 5:01 pm


      Instead of just solving the developers’ testing problem for them, try pairing with a different developer each time on solving the specific testing problem that has arisen.


  3. 5 Nathan 17 August, 2009 at 5:48 pm

    Steve – Yes, that’s a good point. I do try to educate the developers in the area of Testing, and often pair-up to write FitNesse & Selenium tests. However, since the developers feel under such pressure to deliver, they put productivity of ‘product’ ahead of testing and QA activity – and being the only tester here, I also struggle for pairing time.

    Having said that, I believe the seeds have been sewn, and in the future the whole team will participate more in writing tests up front, and treating them as a roadmap for their development activities.

    Hemal – when you adopted the approaches you mention above (BDD, Cucumber, FitNesse etc.) did your team immediately adopt these practices, or did it take time to adapt? In my company, it feels like I am trying to turn an oil tanker around with my suggested new approaches.

    I assume you will be covering all of this at your skillsmatter talk though…

    • 6 Hemal 17 August, 2009 at 7:34 pm

      Leadership into organisational change like this has to come from above. We have a great management team providing sound direction and vision for how we should be doing stuff.

      I will indeed provide more of an insight on how we achieved this at the talk; to summarise though, everyone should focus on quality and don’t be afraid to take the training wheels off by completely cutting the reliance on yourself as a test engineer.

  4. 7 Justin Hunter 15 September, 2009 at 2:31 pm


    I like your post and these concepts. Given how well known Deming’s ideas are, I’m surprised I don’t hear more people articulating views like yours.

    One quick (and admittedly self-serving)thought: if you and your colleagues are interested in doing “just a little” testing, try the Hexawise test case generator. It is a tool that helps developers and/or testers develop a handful of grey box and/or black box tests that will achieve maximum coverage.

    We have both a free version and a commercial version available.

    A second “it’s a small world” thought: I notice my brother (Curious Cat Management Improvement Blog – ) beat me here. My brother and I are interested in seeing Deming’s ideas applied to improve software development and testing; our dad collaborated with Deming and he and I are both working to improve software processes. Deming used to joke that American management’s strategy of quality control was “I’ll burn the toast, you scrape it.”

    You’ve worked on fixing the toaster (which is the right thing to do); in this analogy, Hexawise would be an extra-efficient, semi-automated, burnt toast scraper (just in case any toast got burnt) that minimizes the time required to check for burnt toast.

    – Justin

  1. 1 Curious Cat Management Improvement Blog » Baking in Quality to Software Development Trackback on 15 August, 2009 at 7:34 pm
  2. 2 From bug hunting to bug prevention « Hemal, Developer in Test Trackback on 27 January, 2010 at 10:38 am
  3. 3 Summing it all up « Hemal, Developer in Test Trackback on 6 November, 2010 at 2:38 pm

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: