On October 20th, Senator John McCain and Congressman Mac Thornberry discussed the National Defense Authorization Act, hosted by the Brookings Institution. Congress recently passed the act for 2016 with bipartisan support, but the President immediately vetoed the bill.
The veto is politics, in which I am neither an expert nor really all that interested in the reasons for such tactics. I’m more interested in the Department of Defense (DoD) receipt of adequate funding to protect our nation and improving this process through proposed acquisition reforms, especially IT acquisition reforms.
Who am I?
I’m a systems engineer that’s worked in the DOD IT industry for over 13 years. Currently, I work for BluestoneLogic, a small systems engineering company located in Washington, DC. We help DOD programs modernize their IT systems. I could write for days about modernizing IT systems for the DOD, but I’ll summarize it as an extraordinary challenge.
I believe we are in a golden age of IT because of the amazing things we’re able to do with it. I can FaceTime with my wife when her work sends her to the other side of the planet. But, that challenge can also be extremely frustrating because we find that our partners (the Department of Defense) are constrained by outdated acquisition policies that prevent them from having access to the innovative technology I have access to from my living room.
DOD IT Acquisition Reform
The Senator and Congressman spoke a lot about the various reforms found in the NDAA 2016, the bipartisanship it took to pass the bill, and the President’s contentious veto. They also spoke to some specific points that are relevant to DOD IT acquisition reform.
Senator McCain mentioned plans to review the Goldwater-Nichols Defense Reorganization Act and making necessary changes. I sincerely hope Congress does that, especially with respect to changes in DOD IT acquisition. Congressman Thornberry pointed out that President Reagan signed Goldwater-Nichols almost 30 years ago.
You know what else happened 30 years ago? Microsoft released Windows 1.0and the Internet boasted a whopping 2,000 hosts. I think it’s safe to say a lot has changed in the IT landscape since the passage of Goldwater-Nichols. Back then, the DOD was a leader in IT innovation. Now, we find the leaders in the private sector — such as start-ups in Silicon Valley — and the worldwide open source communities. It’s probably a good time to take serious action to reform how the DOD acquires IT systems.
From my vantage point, DOD IT acquisition is still too similar to the acquisition process for physical platforms, despite its best efforts to adopt Agile software development methods, which, by the way, were first published 14 years ago in 2001. The typical funding plan still seems to be big budget for Research, Development, Test, and Evaluation (RDT&E) to initially develop and approve the IT program for use in the field. Then a relatively smaller budget for Operations and Maintenance (O&M) to keep the IT program running for years, even decades. Once an IT system reaches this O&M state, the focus becomes keep the system running, not innovate.
That may make sense when you’re building big, physical ships, tanks, and aircraft that will be mass-produced in special factories and in service for decades. But I question that logic for IT programs, where technology evolves at a much faster pace.
The RDT&E effort for a DOD IT program needs to be continuous and concurrent with the O&M effort for that same program — where the goal to continually replace elements of the IT system under O&M with improved system elements developed under RDT&E.
This continual pursuit is how Twitter survived the Fail Whale Era. It’s how Facebook has become such an impressive force in the IT world today. They never stop innovating, and neither should DOD IT programs.
To achieve continuous innovation in IT, the DOD needs to attract IT innovators. Senator McCain mentioned engagements with Silicon Valley, trying to convince them to help the DOD. But for the most part, Silicon Valley is not interested in working with an organization that requires so many hoops to jump through just to get funded for an IT project, let alone execute it. They don’t see opportunity. They just see impediments with a low return on investment for their time and expertise.
Clay Johnson summed this up nicely in Episode #34 DMV Nation of the Reply All podcast, when he’s talking to the hosts about US government IT acquisition as a whole, not just the DOD:
“This regulatory environment is so huge and requires a real skill to understand, that the people who win the contracts are the people, often times, who understand the regulations the best. Not the people who can understand the technology the best.”
So, while the innovators in Silicon Valley continue to disrupt the IT industry, the DOD continues to award IT contracts to the same players it has for years, even decades. In my experience, the DOD has relied on big defense contractors to acquire its IT systems — Northrop Grumman, Lockheed Martin, General Dynamics, just to name a few that I’ve worked for or with in the trenches. Those are the same companies that build the ships, tanks, aircraft, missiles, etc. for the military. I think Clay Johnson is on to something.
Still, I am hopeful that the DOD can reform its IT acquisition policies to be more attractive to IT innovators. In 2014, the White House created the US Digital Services, an organization designed to recruit IT innovators for 2-year tours working for the federal government to improve its IT systems. Founded by key members of the team that salvaged the Healthcare.gov fiasco, US Digital Services proves that the government can attract IT innovators.
So the precedence has been set. Now its up to Congress, the President, and the DOD take action in order to successfully attract IT innovators like US Digital Services.
Fail Fast to Succeed
Senator McCain and Congressman Thornberry spoke about changes in the NDAA 2016 that will hold people accountable for the results of their programs. An example provided was that Service Chiefs would have to sign-off on program cost overruns, thereby accepting the responsibility for the overrun, which the DOD apparently struggles to attribute today. That’s a good step in the right direction, but I think it needs to go further. The DOD also needs to accept failure in order succeed.
Healthcare.gov is a good example of where failure can lead to success. If the initial release, called the Federally Facilitated Marketplace, had at least some moderate success, the federal government would probably still be using that system, at a cost of $7oM per year just to maintain it.
Instead, we have Healthcare.gov revamped by the US Digital Services engineers. Cost: only $4M to produce, and $4M to maintain annually. And with that, we also get the example of how the government can attract IT innovators, as well as how they can succeed in government IT acquisition.
If anything, the real failure of the Federally Facilitated Marketplace was that it did not fail fast enough! Full system test did not begin until two weeks before the launch. By that time, the US government had already spent most of the money to build the system, and yet they had no idea if it worked as an end-to-end system. Why not? I’m guessing because everyone involved was afraid of failing too early.
I’m not entirely familiar with the actual contract vehicle used, but I’m familiar with the type. It’s called an IDIQ — indefinite delivery/indefinite quantity. IDIQs often have “off-ramp” clauses, that describe how a contractor or subcontractor can be removed from the contract upon failure to meet the requirements. So, if you fail too early, you miss out on all the potential revenue than if you fail later in the process.
I do not mean to imply that anyone involved in this incident deliberately hid its failure in order to maximize its revenue. I‘m only using this as an example of how government IT acquisition discourages and penalizes failure, which I believe is detrimental in the long run. The avoidance of failure needs to change. Encourage early failure with an expectation to learn and adapt. Or have a graduated penalty for failure, such that the government discourages late failures rather than all failures.
One More Thing…
During the discussion, Senator McCain added that continuing resolution is damaging our national security. To me, the real problem is that continuing resolution has become the norm in Washington.
In IT, we have a concept similar to the continuing resolution…we call it a workaround. We use a workaround when the system isn’t behaving exactly how we want it, but we’ve found some alternative way to get by temporarily. It’s a stopgap to keep a bug or issue from bringing the system to a complete halt while we search for the root cause of the problem and find a resolution.
But workarounds are meant to be resolved by the next release. When you reuse workarounds release after release, a funny thing happens. They become part of the normal, expected behavior of the system. Until one day you have so many workarounds, you don’t even know where to begin root-cause analysis, let alone how to address them. And by then, you cannot convince leadership to fund the massive overhaul needed to get rid of the workarounds because leadership just sees it as the normal behavior of the system. So, you’re stuck with the inefficient system.
Continuing resolution is just a legislative workaround. It’s about time we stop accepting it as part of the norm and start taking action to avoid it altogether.
By Joe McCormick
Thanks to Davey Ahearn