How to fix Novopay (and other 'Big IT' projects)

Here's one way of framing the problem: How do you get five elephants into a mini?

I said last week that there’s no way I could build a school payroll system without error. Neither could you. Together though, we could get pretty damn close.

The elephant thing is funny, but strikes at the root of the issue: we need to ignore the elephant.

Please excuse me while I flog a dead metaphor for a little while:

• We had a pretty cool elephant. It was big and hairy, occasionally left giant steaming turds around the place, and was really handy at moving logs around.
   
• Someone decided the elephant was a bit old, even though it still got the jobs done.
   
• A replacement elephant was created, but the creators thought the trunk was for squirting water, not shifting logs, so it wasn’t strong enough to do the jobs we need it to do. They then set about engineering a cybernetic augmented trunk system to strengthen the trunk.

A key issue with Novopay, and the school payroll system in general, along with many other huge black-box IT projects (yeah, I’m looking at you, $1.billion+ IRD replacement system), is the approach of replacing the system as a whole. Forget about the elephant, let’s look for a system of interconnected components that – as a whole – gets us to where we need to be. Perhaps they end up looking like an elephant, or maybe a forklift, but that’s irrelevant.

In the case of Novopay, what are the core problems that need to be solved?

Rather than tender for a complete system, we would be much better off if the Ministry of Education tendered for a “Teacher’s Payroll Calculation Component”.

This would be defined as taking a set of inputs (tenure, contract details, tax period, hours worked), and emitting outputs (gross and net pay, tax, Kiwisaver, holidays accrued). The inputs and outputs of course have to be implemented in a transparent, open manner: REST would make developer’s lives easiest, but as long as it’s well-defined and not some proprietary binary format, who cares?

ABOVE: Consultant Garth Biggs tweeted this screen grab of the mind-bending rules for paying non-teacher staff over Easter (click to zoom). It echoes a comment made by the minister in charge of Novopay, Steven Joyce - that there are 10,000 different permutations for paying education staff. The Ministry of Education's rules should have been drastically simplified before the payroll contract even went out to tender. Of course, it wasn't. "The complexity just clarifies even more for me why we need testable subcomponents," Gracewood says - CK.

Those of you immunised against pragmatism will be having convulsions right now. “But, but, where does the data come from? Where is it stored? What about security and archiving? This is only a tiny piece of the problem!”.

I’m not saying calculating a single pay run is an easy task. On the contrary it sounds incredibly complex. But by breaking this one component out separately we can do so many things:

• TEST this component against the existing systems. Throw some really hairy combinations at it and make sure the outputs are correct and identical to the existing system. Create those as automated test cases that we run daily and verify output.
   
• INTEGRATE with this component in any way we think is appropriate. Let 1,000 data entry systems bloom. Want to write a pay calculation front-end for Palm Pilot? Here’s the API and security model, go to town.
   
• SECURE this component by ensuring that only teachers and schools (and other components of the larger system) with appropriate access can get at it.
   
• IMPLEMENT this component against the existing system by delegating the single task of calculation to this component using its API. Ding! You’ve just removed a dependency on the huge project.
   
• REPLACE this component if it sucks. You know the API, you have the test cases you need to achieve: if you think you can do better, go write me a new one.
   

But wait! I hear you saying, what about storing teacher details, historical pay runs, transferring money to bank accounts? Ben! You’re lying to us. We need to spend more money on Big IT!

It’s okay, just relax and keep slicing away at that elephant.

Contract to build a “Teacher Information Storage System”. Define the fields you need to store, the security with which it needs to be stored, and the interfaces it needs to expose (surprise! one of those probably matches perfectly with the payroll calculation inputs).

The other obvious upside is you can approach each component as a best-of-breed system. Rather than using freaking* Oracle Forms for user input, you could contract the front-end data entry to a talented web development studio, safe in the knowledge that they won’t be out of their depth having to code payroll calculation.

What’s more, if one particular implementation (or implementer) really sucks (ahem Talent2), then the damage is limited to the one component they are working on, not the entire system.

You get my drift? At no stage have I said this will save money. It probably will, but even if it doesn’t the end result is a well-defined set of interconnected components, all of which could be replaced individually.

There is of course a gigantic problem with my approach: everyone working on the systems, teachers, the Ministry, all the way down to coders, would need to collaborate openly and honestly. Bummer.

Auckland software developer Ben Gracewood is practice lead at Marker Metro, founder and director of the Codemania conference, and a former principal architect at Datacom. During his time at Datacom he was not involved with the company's payroll division. He blogs at Ben Geek.

* Language has been modified. - CK