Sometimes It Is Just Too Late To Learn From Failure
On Monday February 3rd, as fans of the Kansas City Chiefs were celebrating and the rest of us were starting another work week, the first event of the 2020 election, the Iowa Caucuses, began.
As most people know, the results of the Iowa Caucus were materially delayed because a new application developed by Shadow, Inc had technical issues that prevented data from being reported from the caucus sites. There’s been a lot written about this; I find this Associated Press article recapping what happened to be the most succinct.
“Failure is an opportunity to begin again, this time more intelligently.” Henry Ford No. Nope. Not this time. The Iowa caucuses are not a daily occurrence. They are once every four years.
This post has nothing to do with politics but everything to do with how technology can bring an operation to its knees in spectacular fashion through failure of leadership and lack of common sense. So here goes, my “Monday Morning Quarterback” take on what led to this epic failure.
- Lack of customer-centric design. There was little to no testing by the people who would be on the ground in high school gymnasiums and other public spaces. In IT in 2020, this is not just poor judgement, it’s stupidity.
- Failing to take into account the installation experience as a critical success factor. If the installation doesn’t go smoothly, an application is doomed. People were getting errors from the beginning. Given the application would be installed by a wide variety of people and devices, in varied locations, a wide array of field testing on the install experience should have been a must.
- Failing to performance test the application. Even people able to install and use the application reported delays and errors when trying to report data. There were 1700 locations (precints) needing to report. Was there any testing of this magnitude in the lead up to caucus day? In 2020, even the most junior of IT professionals know to test the sh&t out of everything.
- Lack of pragmatism about the timing and importance. File under “what the he&& were they thinking?” CIOs know that the most difficult and stressful time to deliver a new application is during a critical business cycle. If you must do it then, you need to amp up customer design work, performance testing, secure architecture, end-end user experience from installation to fulfillment exponentially. This means more time, more money, more expert resources. If you don’t have the time, talent or money to do it right – stop, do not pass Go, do not collect $100, do not proceed any further. (See also analysis on the Healthcare.gov fail .)
- Contracts matter more when you know each other. Purportedly Shadow Inc., only incorporated in September 2019, was led by people with existing connections to the Democratic party. (I’ll leave the optics of this for others to dissect.) It’s not uncommon for companies and organizations to develop partnerships with people or companies who are known to them. When that happens however, the contractual arrangement becomes even more important. The best way to respect the existing relationship is to have a contract that goes above and beyond in laying out the minutiae of what, how, and when of delivery. While I haven’t seen the contract, my CIO radar tells me that it may not have been as robust as it should have been.
- Money matters. It’s been reported that Iowa’s Democratic party spent $60,000 on this application. I’ve been in IT a long time. If anyone had said to me a few months ago “hey, the Iowa Democrats are spending $60K on this new app for the caucuses” I would have said right then & there, “that’s going to fail.” A brand new application deployed over 1700 sites that has to perform flawlessly within less than a 24 hour period in a complex environment? I’m going to just put one toe out on a solid limb and say the leaders of Shadow didn’t do themselves or Iowa any favors by providing this at a rock bottom price. Sadly, Iowa got what it paid for.
“Failure isn’t fatal, but failure to change might be.” John Wooden . To my knowledge, nobody died from this debacle. But let it be yet another wake-up call to anyone involved in a technology project – if you don’t change how you think about how to deliver technology, you will fail and drag others down with you.
I’ll be back with Part 6 of my six-part series on how to prevent IT program failure next week – after I vote in the New Hampshire primary.