Have We Solved “The Software Crisis”?

Most folks working in the software industry today, especially younger developers and those who didn’t experience the early decades of computing, have likely never heard of something called “The Software Crisis.” It was a term that became widespread in the late 1960s to describe the significant challenges and problems that software teams were facing at that time, as they attempted to develop increasingly complex computer programs and systems with some semblance of predictability.

The Origins of the Crisis

In the late 1960s, the term “software crisis” was coined to describe the growing challenge of writing high-quality, reliable software in a controlled and predictable manner. As computing systems became more complex and critical to businesses and society, the demand for software skyrocketed. However, software development approaches were still quite immature, leading to projects frequently going over budget, missing deadlines, and delivering buggy code.

Attempted Solutions Over the Decades

Over the subsequent decades, massive efforts were made to transform software development into a disciplined, predictable practice akin to engineering disciplines like civil engineering or medicine. Structured programming practices, modular architectures, object-oriented design, agile methodologies, automated testing, and a host of other innovations were developed and widely adopted. Software tools for managing requirements, writing code, building, testing, deploying, and monitoring evolved significantly. Note however that the real roots of the crisis were never identified, let alone tackled.

The Crisis Persists

But have we truly solved the software crisis after 60+ years? The answer is no.

While significant progress has been made, software remains a fundamentally chaotic and non-deterministic endeavour rather than an orderly engineering discipline. The core essence of software – its malleability and propensity for exponential design complexity growth – makes it more difficuly to tame in the same way physical disciplines controlling matter and energy have been tamed.

Modern software development practices can certainly produce enormously complex systems successfully when properly applied. The processes and tools allow more predictability and quality than in the past. However, software remains inherently difficult to get right due to ever-changing requirements, defects, integration challenges, security vulnerabilities, and uncontrolled technical debt accumulation. Not to mention the people side of the CKW equation.

New Challenges Emerge

Additionally, the pressures of competitive business incentivise cutting corners, resulting in quick software releases over sustainable quality. The diffusion of software across industries through open source, cloud computing, low-code/no-code platforms, etc. has created new challenges around dependency management, security, compliance, and more.

Not (Yet) an Engineering Discipline

While great strides have undoubtedly occurred, software is not yet a mature engineering discipline with rigorous, deterministic, and reliable foundations. It remains a random walk, a craft – a labour of ingenuity battling relentless forces of complexity, change, entropy and friktion.

The Perpetual Struggle

Modern software practices have provided more frameworks and guard rails for controlled delivery when stringent disciplines are followed by competent teams, but they have not resolved the existential software crisis that still plagues many organisations. For countless teams, the never-ending struggle to tame quality software from the inherent complexities, chaos of code, and clueless management remains an existential battle. Though no longer an acute emergency universally, the fundamental crisis of software’s inscrutable nature, propensity for unpredictable schedules, and having to negotiate demands from metaclueless leadership is very much an existential threat in many contexts. The dream of elevating software to a rigorous, mature engineering discipline on par with traditional fields remains elusive after decades of efforts. For many, the journey to transform writing code from crafty art to deterministic engineering while simultaneously managing unrealistic mandates from out-of-touch executives is still very much a crisis state. The quagmire of achieving true predictability amidst software’s inherent volatility and disruptive, uninformed directives from the top rages onward.