The other day I did a post about the MoD's 2018-2028 Equipment Plan, or as it's otherwise known "The common book of MoD prayers". In that post I referenced the National Audit Office's criticism of the MoD for forecasting costs using the 50th percentile. Today we're going to delve a little deeper into that, look at a highly contrived scenario involving groceries, and by the end I'll have somehow figured out how to make it all tie together in a nice neat bow. I hope.
Before we really get going, pre-warning time; a) this is probably going to be a bit boring. I want to avoid the truly dry bits about accounting, but it's not going to be a page scroller by any means, and b) this is designed to be a sort of broad look at the problem not a detailed, textbook chapter in cost estimating. c) I'll probably end up swearing at some point.
So then, cost estimates.
What the NAO takes exception to is that the MoD has a tendency to calculate various scenarios to produce a range of potential costs - which in and of itself is actually a very good thing - but then typically opts to pick a number right slap bang in the middle around the 50th percentile. In NAO accounting speak this is a P50 confidence level. Also in NAO accounting speak this is known as; "a shit idea".
The danger with a P50 confidence level is that in half of all scenarios the project ends up costing more than has been budgeted for. By comparison the Hinkley Point C project was costed to an 80% level, while the Type 26 program is priced to the equivalent of the 83rd percentile. This was due to the insistence of the contractor BAE Systems, who would be on the hook for a portion of any cost over runs compared to the agreed contract price. Even if you happen to think BAE are bastards, you cannot accuse them of being stupid.
To illustrate what we're talking about here let's imagine that you've been ordered to the shops by your better half through the medium of a note left on the kitchen work top. The note requires you to go out and buy; 4 pints of milk x2, one loaf of bread, and a newspaper. Trouble is you haven't been to the shops in a while and you're not entirely sure how much all these cost. Looking in the kitchen draw you see a bunch of loose change and now have to make a decision about how much to take with you.
Based on memory you think the 4 pinters will cost £1 each. But didn't Jim from work say it was £1.10 when he went to get milk for the staff room? And now you've started overthinking it and can vaguely recall seeing a price tag of £1.20 in relation to milk, or was that the organic stuff? The bread you're fairly certain is just a pound, but then you do buy that wishy washy 50/50 stuff that can't decide if it's wholesome or not so tries to be a bit of both. That might be £1.20. The paper is normally £1, but then it is the weekend so doesn't it go up to £1.20? Or was it £1.50?
Based on this we've established three potential price points for milk, two for the bread, and three for the paper. That's 3x2x3 which is 18 different combinations of cost. For the sake of clarity and because I had some time to kill, here are the combinations in full;
(2x)£1 + £1 + £1 = £4
(2x)£1 + £1.20 + £1 = £4.20
(2x)£1 + £1 + £1.20 = £4.20
(2x)£1 + £1.20 + £1.20 = £4.40
(2x)£1 + £1 + £1.50 = £4.50
(2x)£1 + £1.20 + £1.50 = £4.70
(2x)£1.10 + £1 + £1 = £4.20
(2x)£1.10 + £1.20 + £1 = £4.40
(2x)£1.10 + £1 + £1.20 = £4.40
(2x)£1.10 + £1.20 + £1.20 = £4.60
(2x)£1.10 + £1 + £1.50 = £4.70
(2x)£1.10 + £1.20 + £1.50 = £4.90
(2x)£1.20 + £1 + £1 = £4.40
(2x)£1.20 + £1.20 + £1 = £4.60
(2x)£1.20 + £1 + £1.20 = £4.60
(2x)£1.20 + £1.20 + £1.20 = £4.80
(2x)£1.20 + £1 + £1.50 = £4.90
(2x)£1.20 + £1.20 + £1.50 = £5.10
Now if this was the MoD they would calculate that £4.50 is about the 50th percentile and that's how much they should take. While that amount is enough to cover roughly half of all the listed scenarios, it also means by default that if any of the remaining 50% of scenarios develop then the MoD would be left begging the person behind them in the queue to borrow a few extra pence to make up the short fall.
An alternative would be to take £4.90, which represents about the 90th percentile, or what the NAO would refer to as (drum roll please)... a P90 confidence level. £4.90 in this case would be enough to cover every scenario except one. The likelyhood of a cost over run in this case is extremely low. There is however a problem with taking that much and it's the same reason that likely drives the MoD's insistence on using a P50 confidence level.
The problem with taking £4.90 in our scenario is that in all except three outcomes you end up taking too much money with you, leaving you with a bunch of unspent change. While you could spend the extra on a treat, it also means you're not working your budget to its absolute maximum potential. The risk of ultimately needing £4.90 is small and you might even be able to find 10p in your back pocket. In this case, taking £4.80 would probably be more appropriate.
Back in MoD land, the reason for costing to a P50 confidence level is known only to the MoD themselves, but I suspect it has something to do with being able to cram more stuff into the budget than they would otherwise be able to if they costed things to a realistic level. Imagine if we weren't just buying three items, but instead were doing an entire weekly shop. By using a 50th percentile approach on each item we could shove a lot more onto a £100 budget shopping list than we could if we used a much more cautious cost estimate.
Thus by underestimating the potential costs and dismissing the various risks, the MoD is not only able to make each individual item appear more affordable and thus help justify its purchase, it can also shove more shit into the equipment program which then becomes hard to cancel at a later date due to penalty clauses and the like.
At this point what we've learned is that it's actually in the MoD's interest to understate potential costs. They lose basically nothing by doing so, except another shred of their already small amount of financial credibility. Nobody really gets punished, apart maybe from frontline personnel who end up cannibalising equipment to keep it running (the NAO counted 3,230 instances of ship and submarine cannibalisation between April 2012 and March 2017 for example) and the treasury normally gets coaxed into coughing up the extra to fund a program because it's part complete anyway and the government doesn't understand the concept of sunk costs.
There really is no incentive for the MoD to cost anything in a proper and reasonable manner. Which is why the ultimate solution to the MoD's financial woes might stem from the somewhat radical step of taking the accounting process out of their hands.
The Cost Assurance and Analysis Service (CAAS), a somewhat independent body that already exists inside the MoD, is designed to serve "as a centre of excellence for pricing and costing suppport to the Acquisition community". Which begs the question why we don't just take the final costing process out of the services hands and pass it to the CAAS anyway? The military would define its requirements and do its preliminary cost analysis. This would then be passed to the CAAS who would do the final costing, without any further input from the services and to a realistic standard of confidence, before finally being passed on for approval by senior types and the start of the negotiation period.
As such the MoD would decide what it wanted and how much it was prepared to pay. It would do the early donkey work to establish the range of cost estimates. CAAS would act as the adult in the room, to make sure the MoD wasn't trying to pull the wool over anyones eyes. Then it would be up to senior officials, officers and ministers whether they still wanted to proceed or whether they sent the whole thing back to the drawing board again.
It might not solve the long term problems with the MoD's accounting, but at this stage I can't see how it would hurt it. Officers are not accountants. Nor do you simply give the kids the Argos catalogue and your credit card and let them run wild. At some point along the chain there needs to be someone who can actually add up and who has slightly more restraint than a tiger released into a field of sheep. The MoD has had more than enough opportunities to get it right. What it consistently has proven is that it can't be trusted with its own budget and now I feel the time has come to inject a bit of serious accounting skill and realism into the process.
No comments:
Post a Comment