Beware Technical Debt – the long-term consequences of under-investing in technology

by Rob Campbell, Mawer Investment Management, via The Art of Boring Blog

Two years ago, The Atlantic published an article ominously titled “The Coming Software Apocalypse” that opened with a frightening story:

“There were six hours during the night of April 10, 2014, when the entire population of Washington State had no 911 service. People who called for help got a busy signal. One Seattle woman dialled 911 at least 37 times while a stranger was trying to break into her house. When he finally crawled into her living room through a window, she picked up a kitchen knife. The man fled.”

The problem, as it turned out, was a single line of code embedded in a piece of software on a server 1,000 miles away in Colorado that helped to rout incoming 911 calls to the appropriate dispatcher. The bug caused the software to stop routing those calls altogether.

The 911 failure is an example of something called “technical debt.” Technical debt refers to the costs and long-term consequences of under-investing in technology, whether it be poor coding techniques, delaying routine maintenance, weak IT system architecture, outdated systems, or employing other suboptimal IT practices. There are ramifications to such kinds of technical debt, including application outages, security vulnerabilities, and higher maintenance costs down the road.

In the world of investing, we typically think of debt as a line item on a company’s balance sheet with an associated dollar value. But just as the code for my iPhone’s iOS is invisible to me as a user, technical debt is effectively invisible on a balance sheet: it’s an intangible. And over the long-term, the accumulated costs of technical debt can have a financial impact on a company and its shareholders. If they’re lucky, companies may simply have to make the requisite financial investments to reduce their technical debt load. But in other cases, technical debt has the ability to lead to the permanent impairment of a company’s reputation, customer trust, or a deterioration in its competitive advantages.

As we unpack the various types of technical debt; how they arise; what companies can do to manage their technical debt; and how technical debt assessments have influenced our portfolio holdings, we’ll also turn inward—how we think about our own technical debt at Mawer, and some of the ways we’ve had to account for it.

Three forms of technical debt

By nature, technical debt is a slow-moving risk, and it’s easy to understand how it can begin to accumulate. Having lived for some time in a condo, my wife and I woefully underestimated the effort required to maintain our lawn and garden the first summer after we purchased our house. What started as a few weeds in our garden beds literally blossomed into a full-blown invasion of undesirable plants. For the rest of the summer, we spent an absurd amount of time countering the assault—time we could have invested in more productive (or leisurely!) activities had we taken a more responsible approach to regular yard maintenance from the outset.

In the IT world, this type of technical debt—deterioration over time—is referred to as “bit rot” tech debt. A component or system slowly deteriorates due to a lack of proper maintenance, especially if only band-aid type solutions are implemented along the way (e.g., pulling out a weed without re-seeding the bare spot).

There are two other broad categories of technical debt: accidental and deliberate. Accidental tech debt occurs when an original design becomes outdated as new requirements, constraints, or possibilities arise. In other words, the original product or system may have been well-conceived for its initial purpose, and well-maintained, but new technologies or demands have made it sub-optimal or, in some cases, obsolete.

Deliberate tech debt is the most fascinating in my view, as it highlights a crucial aspect of technical debt: it’s not plausible, nor desirable, to have zero technical debt. Just because the value of a newly purchased car depreciates the second you drive it off the dealer’s lot, doesn’t mean you should buy a new car every day. Deliberate tech debt occurs when a business actively elects a sub-optimal solution that will knowingly incur technical debt. This may be due to the need to deliver something quickly (e.g., ahead of an upcoming product launch date), or due to an assessment that the status quo is sufficient and undeserving of further investment. The assumption is that any shortcomings will be addressed at a later time.

For all three types of technical debt, there’s a central trade-off that management needs to evaluate: how much should we invest today (time, resources, and dollars) to ensure the effectiveness and competitiveness of our product/platform/IT infrastructure in the future? If we save on costs today, how much more will it cost to pay down future technical debt? Debt, after all, usually needs to be paid back with interest.

As investors looking to evaluate a company’s ability to generate cash flows into the future, it’s these hidden financial impacts that interest us.

Singaporean banks

About a year ago, we decided to consolidate our positions in two Singaporean banks. We eliminated United Overseas Bank (UOB) and added to our existing position in DBS Group. The main reason: we believed that DBS’s digital strategy was putting UOB into technical debt, and that UOB management was not reacting urgently enough given the shift in the competitive landscape.

Nearly a decade ago, DBS CEO Piyush Gupta embarked on a plan to fully embrace digitization in the banking business model. The vision was that digitization would enhance efficiency, improve the customer experience, and provide more advanced analytics (big data) to better price risk and anticipate customer needs. For DBS, this would hopefully lead to lower customer acquisition and servicing costs, higher profitability per customer, higher customer satisfaction, and, ultimately, higher returns on equity for DBS shareholders.

In their book Competition Demystified, authors Bruce Greenwald and Judd Kahn argue that the most powerful barriers to entry and competitive advantages stem from the combination of customer captivity and economies of scale. Gupta’s strategy focuses squarely on these two elements.

First, with regards to customer captivity, DBS’s long history in the banking business has earned it the title of Singapore’s most valuable brand, and its digitization strategy seeks to both take advantage and enhance that customer captivity. We’ve seen some evidence that the strategy is paying off. In the retail business segment, the average digital customer generates 2x as much income and is more engaged with 16x more transactions than a traditional customer. Customer satisfaction is high, and DBS’s PayLah! app has dominant market share in mobile payments.

At the same time, while DBS already benefits from economies of scale as the biggest bank in southeast Asia, its digitization strategy seeks to further increase that competitive advantage. While it took years and considerable investment for DBS to launch its digital bank in India, it only took one year to roll out the same platform in Indonesia at a much lower cost, as 70% of the infrastructure was simply repurposed as opposed to rebuilt.

By contrast, management at UOB have been more conservative with respect to their efforts in the digital space. UOB is a more hierarchical organization, and it took them three years to formulate a digital strategy to respond to DBS’s. While UOB has since debuted their own digitization platform, and their returns on invested capital over the last couple of years have been very respectful, we believe they lost ground in marketing their technological advancements. Maybe that won’t matter—it’s possible that the advantages to digitization may be over-hyped. But by incurring deliberate technical debt through a more conservative rollout timeline/strategy, we believe this technical debt will eventually need to be settled, either through considerable reinvestment or a loss in competitive advantage.

Other investment applications of technical debt

The Singaporean banks highlight that technical debt is not simply an issue for software companies. But tech companies do provide a host of examples of how companies can benefit from tech debt related themes.

For all three types of technical debt, Microsoft is an example of a company that does a great job at managing its debt load, and this contributes to our assessment of its quality as an investment. When Microsoft releases a new product, it knows that the product may not be absolutely perfect. But Microsoft covets being first, as this puts their competitors in technical debt. Microsoft then follows up each new release with frequent updates on short cycle times. The company has developed an effective feedback loop whereby it learns about bugs or opportunities for improvement from its users, and is quick to roll out those improvements broadly, usually on what it calls “Patch Tuesday.” And due to Microsoft’s scale, it can reinvest a significant amount into engineering and developing new and better products—the type of investment that others are not able to afford. This helps to ensure that Microsoft maintains its competitive edge.

Additionally, Microsoft has developed solutions to help its customers directly address their technical debt … and in effect to take the decision-making out of their hands. Azure, Microsoft’s enterprise cloud computing platform, houses critical infrastructure such as hardware, software, servers, and storage on behalf of their customers. So long as Microsoft does their job, Azure customers don’t need to worry about bit-rot, accidental, or deliberate technical debt, as Microsoft will take care of ensuring their infrastructure is robust.

This is an area where we’ve found a number of attractive investment ideas for our clients’ portfolios. Because decisions around managing technical debt are far from black and white, customers face an asymmetry of payoffs when selecting hardware and software solutions, when upgrading, or when prioritizing where to commit capital to building solutions internally. We own a number of companies that provide IT-related advice in our global small cap portfolio that consult to corporate and government entities: Softcat, Bechtle, Atea, Kainos, and HiQ. The value they provide their customers is that they marry their knowledge of the available tools in the marketplace with their knowledge of their clients’ existing infrastructure, budget, and needs. In our view, this familiarity introduces a degree of loyalty between these companies and their customers, resulting in high customer stickiness, pricing power, and a degree of recurrence to their revenues.

What about us?

We think a lot about technical debt as it pertains to our own business, and it’s an area where we have made mistakes. For example, we use SS&C’s Advent Portfolio Exchange (APX) as our record-keeping platform for all client portfolios, positions, and performance. From an operational perspective, it is one of the most critical systems in our IT ecosystem and we have built a plethora of other systems and tools that are linked to it.

Many years ago, when SS&C released a new version of APX, we made an active decision not to upgrade. We viewed the changes between the two versions as merely incremental and not worth the effort to implement given the complexity of ensuring all our other systems remained compatible. For the first upgrade, this wasn’t an issue. Predictably, though, this snowballed as time passed and SS&C released several improved iterations of their product. Eventually, we found ourselves running a version of APX that was multiple versions behind. This led to frequent technical challenges with APX and our other systems. We had taken on deliberate technical debt and had compounded the issue by applying inadequate solutions.

Eventually, of course, we addressed and remedied the issue, but the lesson was obvious: we really shouldn’t have fallen so far behind.

On the investment side, we recognize that the tools we’re using today are likely not the tools that will give us a competitive edge into the future. While the three tenets of our investment philosophy are fixed, the tools we use need constant improvement and refinement. Recently, we’ve formalized this idea into something we call “The Lab,” which has two main areas of focus:

1. Better understand our edge/calibration culture:
   Collecting and analyzing data to better understand what parts of our investment process are adding the most value for clients—e.g. screening, various stages of due diligence, modelling work, position sizing, trading, etc. Given the long cycle times in investing, this analysis isn’t straightforward. Much of our efforts so far have gone to setting up the infrastructure that will be able to generate the right data. We believe this endeavour will result in a better understanding of our strengths and weaknesses. Not only will this allow us to address needed improvements, it can help determine those areas where we add real skill, and therefore, where we focus our efforts.
2. Process improvements:
   Developing tools—many of them quantitative—to improve the investment process. Examples include new proprietary screening tools and an automated discounted cash flow engine. The idea here is to use data/computing power to help make us better, more efficient investors.The Lab effort has been largely led by Justin Anderson, one of our Canadian equity analysts, and we now have two data engineers fully dedicated to our Research team. We believe this helps to foster a culture of innovation by allowing our investors to test new ideas, tinker often, and fail quickly.

Technical debt—one of many off-balance sheet liabilities

Taking a step back, technical debt is just one example of an intangible; something that matters but can’t be seen on a balance sheet. While asset-side intangibles—e.g., brand, reputation, expertise, R&D quality, and good management—are fairly well-established concepts in the investing industry, liability-side intangibles such as technical debt have received much less attention.

The broader concept of off-balance sheet liabilities can manifest in many other areas. Culture and employee morale can have significant long-term impacts on staff turnover, engagement, productivity, and ultimately a company’s profitability. Poor government relations can lead to unfavourable regulatory judgments. Poor customer service can result in lost customers and future expenses to reacquire such customers through an expensive marketing effort or less favourable pricing. Often, the long-term impacts of these intangibles are irreversible, and especially if they’re left unattended.

Ultimately, the effects of technical debt—and all off-balance sheet liabilities—reinforces an oft-quoted tenet: not everything that counts can be counted.

This post was originally published at Mawer Investment Management