In 2016, nothing was hyped more than “artificial intelligence” and its unholy marriage to mass data. For two Australian government agencies, Centrelink and the Australian Taxation Office, it all went horribly wrong.
2016 was the year of hype, and nothing was hyped more than “artificial intelligence” and its unholy marriage to mass data. Now, in what the Australian Privacy Foundation has officially described officially as a “cluster f***," Centrelink’s Debt Recovery foray into big data and automated matching has been revealed as a complete disaster.
(Sorry for the un-family friendly language, but "clusterf*** is the term the foundation actually used.)
Centrelink delivers a range of Australian federal government payments and services for retirees, the unemployed, families, carers, parents, people with disabilities, Indigenous Australians, and people from diverse cultural and linguistic backgrounds, and provides services at times of major change. The majority of Centrelink's services are the disbursement of social security payments.
You couldn’t write this into a dystopian thriller about how the machines take over and send everyone rude tax demands. You couldn’t, because it would be boring, and stupid. But, that’s what has happened here.
It appears that Centrelink in one of those beautiful “do more with less” moments decided that big data was magic and let loose the machines to outdo the humans.
Who's ripping off the system?
The idea is simple; the system matches data between Centrelink and the Australian Taxation Office (ATO) to see who’s ripping off the system. When a discrepancy is discovered, the system automatically mails the alleged perpetrator.
But wait, it gets better.
The system has so far identified 160,000 potential fraudsters who are then mailed including threatening them with jail, imposing 10% debt recovery fees, and charging interest on money owed. Just to add to the mess, the onus of proof rests on the human, not the machine. That means that if you get one of these SkyNet generated letters, then you must disprove it.
IT and data expert Justin Warren – who has worked for IBM, ANZ, Australia Post and Telstra, among others – said Centrelink’s system appeared to rest on the “idiotic” assumption that “big data was magic.”
“It’s not. It’s a messy, complex, statistical system that is wrong a lot,” Mr Warren said. “All models are wrong, but some are useful. It’s the choice of how you deal with when the system is wrong that reveals how you view the world.”
Like all bad bureaucracies, Centrelink thinks that nothing is wrong. However, this seems to be a combination of failed PR and a head in the sand approach.
Unfair targeting
A Centrelink whistle-blower has alleged that of all the beneficiaries targeted by the system only 20 have been a valid case. Further, the system appears to target sickness beneficiaries unfairly.
The Centrelink compliance officer, who asked for anonymity, told Guardian Australia the system was error-prone but that most customers were paying debts without checking them first. The source said of the hundreds of cases they had reviewed, only about 20 (at a “generous estimate”) turned out to be genuine debts.
The worker said the system was particularly harsh on those who received Centrelink’s sickness allowance – a benefit for employees who are unable to work temporarily due to serious illness but are not paid by their employer.
Just to top it off, trying to dispute the allegations through the online portal is problematic.
'Stunned'
Frustrations with the debt recovery process have been compounded by errors with Centrelink’s online customer portal, where individuals must go to lodge a dispute. The department said the errors with its online service had affected only a small number of people and had since been resolved.
But the compliance officer said that was untrue. They said they were “stunned” when the department stated the online system was working.
“This is completely false,” the source said. “Not only do customers, especially past customers, have access issues all the time but, since the compliance system was placed online, [compliance officers] have had many access issues.
“For the past two weeks we’ve had to turn customers away because we could not access [the system] and neither could they.”
Here's the problem
Now, here’s the fascinating thing, what we see up to this point is the system gone mad, an IT issue that is out of control, a technology problem.
Of course, it isn’t.
The system was constructed this way on order of the Centrelink executive. Here’s the problem.
Executives believe big data is accurate and somehow magical. The reality is that data is often error-prone, inaccurate, incomplete, and wildly variable. This is because humans enter data and we don’t manage information well.
Enter the politicians promising a crackdown on welfare cheats putting tremendous pressure on the agency to act, fast.
When all of this first emerged, it looked like an IT story, and for good reason: the government is famous for chronic incompetence in managing IT systems; the hapless IBM is in charge of Centrelink's core systems; those systems are positively ancient and need replacing; IBM was the village idiot that ruined the 2016 Census; and people assumed that if Centrelink is creating incorrect debts it's because the IT system's a dog. - Source
So; the system was most likely rushed and a series of rules given to the machine that caused the resulting situation. It works like this;
Centrelink docks your pay if you earn over a set threshold in a fortnight, if you earn enough then you get nothing from Centrelink. However, the ATO doesn’t understand fortnights; it uses annual calculations. So, the developers (instructed by the business) were told to divide the ATO earning by 26. Yep, face palm.
When yearly / divided by 26 yielded an amount over the fortnightly threshold, the debt was created, and the beneficiary served. The beneficiary must challenge the finding sometimes going back more than six years to find records.
It’s a cautionary tale about over-reliance on data, political pressure, poor management, poor information management, and data matching hell.
Here’s hoping that we learn from it, Centrelink certainly isn't with its continued assertions that everything is ok and there is nothing to see here!
Ian Apperley is an independent cloud computing consultant and commentator based in Wellington.
