Introduction

No software is perfect, but some programs are better than others. I'm proud of the quality of Resolver (although there is still a lot of work ahead), and a big part of why I feel that Resolver is a well crafted program is down to the processes used to develop it. I hope this doesn't sound like marketing speak, because how software is written makes an enormous difference to the end result.

Resolver is run by a software developer. Specifically, by a software developer with experience of the financial industry. Why does this matter? Well, software is at its best when it is created by a team who care about the quality of the code. The best programmers are those who are passionately interested in code quality, and there aren't many managers who are prepared to let developers put this first.

This article explains the development processes used at Resolver, from the point of view of the humble programmer.

Agile and Extreme

The programming techniques used at Resolver are fairly mainstream 'agile' processes, in fact they follow the central orthodoxy of agile - eXtreme Programming (XP). I realise that unless you have heard of these before, it isn't necessarily obvious what this means!

The essence of agile programming is an understanding that in the process of creating software, humans matter. Although it is written to run on machines, code is created by people and for people.

Traditional software development is often called the 'waterfall' process. It usually goes something like this:

• Write a big specification of what the program ought to do
• Work out how you think the program will have to be structured to meet the requirements
• Start programming, usually working on the key structural elements first

This is sometimes, for obvious reasons, known as 'big design up front', and it suffers from various drawbacks. Specifications and requirements are often (almost always in fact) subject to change, and it is virtually impossible to specify everything about how a program should work without having tried it. Very often what you assume will be the most logical way of performing an operation (and have designed your program around) will turn out to be awkward and unnatural.

The same is true of architecture and structural decisions about how a program should work. What seems like a good idea on paper can often turn into a nightmare because there are always a myriad of minor details that you can't take into account from a theoretical overview.

With big design up front you don't usually don't have a program that does anything until you are nearing the final stages of the project. All the key elements have to be in place and wired together, including a 'user interface' so that you can actually interact with the program.

In a word, this way of developing isn't agile. It doesn't cope with change or adapting to failures in its assumptions. It is much harder to do user tests because you have nothing to test until you get towards the end.

Agile programming techniques are one of ways that the software industry is attempting to resolve these problems. The core elements of eXtreme programming include:

• Test Driven Development
• Pair Programming
• Customer Input
• Iterative Development
• Do the Simplest Thing that Could Work

Test Driven Development

This is probably the most important aspect of eXtreme programming, and makes the single largest contribution to ensuring the quality and stability of Resolver.

Testing is important for any program. In recent years automated testing has become much more prominent. There are basically three kinds of tests:

• Functional tests (also known as acceptance tests or black box tests)

  These test that the program does what it is supposed to do. They test the actual features of the program (from the 'outside'), and they are the hardest to write as you have to automate the program to simulate input in the same way users interact with it.

• Unit Tests

  These test the individual components (the units) of the software.

• Regression Tests

  When you find a bug in your software, it is very common to think that you have fixed it - only for it to reappear a few weeks or months later. Regression tests test that a bug has really gone, so that it can't 'regress'. These are really another form of unit test, but a different category.

Different companies and different developers have different testing strategies and do different amounts of testing. At Resolver we follow the practise of doing test driven development.

TDD (as it is known), means that before you write any code you write the test. Typically, adding a new feature to Resolver will follow a pattern something like this:

1. Write the Functional Test

   This tests that the feature works as expected. In fact the functional test specifies how the feature should work. At this point the test fails, because we haven't written it yet, so we go to the next step.

2. Write a Unit Test

   Adding a new feature means making changes to existing code, or adding new code. In either case, we write a unit test that tests the behaviour of the code. Obviously this fails, because we haven't written it yet, so we go to the next step.

3. Write the code

   With a failing unit test in place to tell us what needs to change, we go ahead an write the code! If we get it right, the tests will pass.

Most features will require the repetition of steps two and three several times. At every stage the development is 'driven' by the failing tests. We know when we have completed a new feature, because the functional test will pass!

TDD was created in order to assist with 'refactoring' (changing) code. If your requirements change, or the structure of your program needs to change as it grows you will often need to make big changes in key parts of your code (evolutionary architecture). Without good coverage of tests it is very hard to be certain that these changes haven't broken some part of your program in ways that you haven't considered. If you do have good tests, you can be fairly certain that you have completed your changes successfully when all the tests pass again! I can't imagine having to do a large scale refactor without tests. ouch

A secondary advantage, is that you end up with a fully automated test suite that tests every feature of your program and every unit of your code.

It turns out however, that the main advantage of TDD is neither of these two things. Do test driven development changes the way you think about programming. You think about what your code should be doing before you write it, and you are forced to write testable, well decoupled units (units of code that aren't entwined too closely with other parts of your code and are therefore easier to understand and debug).

We have an extensive test suite for Resolver. Whilst I am writing this, just a couple of weeks before Resolver is due to be released, we have about three thousand tests. For around 25 000 lines of production code we have over 75 000 lines of test code. Our functional tests automate every feature of Resolver. It is fun to see Resolver bounce around the screen, opening and closing windows, resizing and performing odd calculations under the invisible hand of our test suite. At some point I will video a small section of it and set it to music.

That isn't to say that bugs don't get through of course. Usually because of something that we just hadn't thought of. When we fix a bug we add a defect test (regression tests) to make sure that we really have killed it. Test driven development doesn't guarantee bug free software, but it does ensure a high level of code quality and make it much less likely that we will break things accidentally.

Pair Programming

The second most important facet of eXtreme programming, is pair programming. It is also one of the most controversial, which is a great shame. You can imagine the reaction of many managers - you want to halve the productivity of my development team? No way?

When I started at Resolver I had not done any pair programming before, but I was quite looking forward to it as an opportunity to learn from other programmers. I am now convinced that pair programming has massive benefits for the quality of code. When working on difficult problems, two heads are definitely better than one. This can't be overestimated.

Pair programming also has these benefits:

• With two people working on a problem, silly mistakes and poor quality code (hacks) are much less likely to get through
• Productivity tends to be higher, working as a pair you are much less likely to be distracted (or at least you have to conspire together to be distracted!)
• It is a great way to mentor junior programmers or introduce new team members to the codebase
• All the code will have been worked on by more than one person (preferably several people if you rotate pairs regularly). No more sections of the code that no-one dare touch because only one person understands it.
• It is a great way to share skills and use new tools
• It is much more fun!

Customer Input

Programmers tend to get carried away with what features they think are important. The whole point of Resolver is that it is useful to the people who use it. This is why eXtreme Programming emphasises the importance of having customer involvement in the development process.

At Resolver one of our directors acts as a customer representative. Patrick has many years experience in the financial services industry, and in fact his frustration with traditional spreadsheet solutions to the problems of this industry is part of what birthed Resolver.

He prioritises which features (or defects) are the most important ones, and is involved in specifying how features should work.

Iterative Development

One of the great difficulties with 'traditional' software design processes, is that large and expensive projects can go over budget and over time without ever producing anything that works. Central to solving this problem is the iterative development process.

The system requirements, whatever they may be, are broken down into individual features called user stories. In our iteration planning meetings the developers estimate the cost (in terms of development time) for each user story, so that our customer can prioritise them. This way the team is always working on the most important aspects of the system.

An important consequence of this is that, at every stage their should be a working deliverable - even if it doesn't yet meet all the original specifications (and specifications never change part way through a project, do they?).

This also means that the latest development version of Resolver is usually in a state where we can roll it out to customers immediately if necessary.

Do the Simplest Thing that Could Work

There are many aspects of eXtreme Programming that I haven't discussed here, like stand-up meetings, the YAGNI principle and so on. At Resolver we are fairly orthodox eXtreme, so we do follow these.

One principle worthy of mention, because I think it positively impacts quality, is DTSTTCW - Do the Simplest Thing that Could Work. It is actually very similar to YAGNI (You Aint Gonna Need It), and has the goal of reducing unnecessary complexity in software design.

Programmers find it very tempting to write not just the code they need, but the code that they think they are going to need in a few weeks or months. The result, all too often, is overengineered systems that are difficult to use and impossible to understand.

DTSTTCW means solve the problem at hand, and in the simplest and most elegant way possible. Solve today's problems, because tomorrow's problems will have changed by the time you get there. It is important however to continually evaluate decisions in the light of the whole system, have a genuinely evolutionary architecture rather than one that just accretes new code. Sometimes finding the simplest overall solution to a problem is very different to tacking on the simplest solution to the part of the code you are working on at the moment.