The Mysterious Case of Lost Developer Productivity

How Developers and Others Squandered Their Productivity

There is a case of mysterious loss of developer productivity. Developer productivity should have risen, but it didn’t, and I don’t know why. I have talked to many people about this, some agree and some disagree, everyone has a theory, but there is no consensus.

When I founded a startup together with friends at the end of the 90s, everything was rudimentary. We had just started to conquer the internet, which brought a huge change to software development. Before most software had one user, some client server software might have hundreds of users and the biggest systems might have thousands of concurrent users, like flight ticket reservation systems. With the internet, everyone’s software had hundreds of users, and many had millions of them. It took some time to adopt, so by the end of the 90s we still developed software the way we had before.

Since then, many things have improved, and all of them should increase developer productivity.

Developer ecosystem When we founded our startup in Berlin, it felt like we were the only tech startup (everyone was in Munich). No developers to talk to, no CTO meetups.
Programming languages There are better programming languages today. Or the ones we had are better now, like Java. We learned and adapted. In the 90s people used Perl to write web applications, some used Python with rudimentary CGI, some C and some the earliest version of Java with server side technology. Today every language is tailored towards scaling developers and developing web applications. Java had garbage collection that barely worked—I remember consultants earning tens of thousands of dollars just for tuning GC setups—today we have near perfect Java GC. Programming languages just work.
Frameworks Frameworks for internet applications were rudimentary - like Java Servlets. Since then, we have got Rails, Lavarel and Django, batteries included and more. There is much less code to write for the same functionality today compared to the 90s. We have Auth providers and billing providers as SaaS. Productivity should go up.
Knowledge In the 90s, when you wanted to learn about a framework, you did read a book (I still remember the XSLT bible because we used XML/XSLT to render our website, I know!). If you had problems, there was IRC chat and NEWS. Both with much fewer users and much fewer problem-solving capabilities. Since then, millions of blog posts have been written, and millions of questions have been answered on StackOverflow. You have a coding question? It might have taken a week back then, today it takes a day to get an answer if not one of millions of other developers has asked it before.
Faster Computers Today’s computers are much faster and still gain speed every generation. There was a huge bump in IO speed with the introduction of SSDs, and since then SSDs made jumps in every generation (the current gen is 12000mb/s - I remember owning a SCSI Seagate Barracuda with 10mb/s). This leads to faster servers, faster database servers and makes vertical scaling much faster. In 2000 people used costly Sun workstations to run websites, today cheap hardware can scale to millions of users. Fast computers and SSDs have sped up compilation by a magnitude. This should make developers more productive (For one SSD Go compilation speed comparison see https://www.octobench.com/).
Cheap and Easy Hosting (Cloud) Back in 2000 hosting was expensive and complicated. Operating your companies application took time and money. With the advent of cheap and easy cloud, it’s trivial to deploy scalable and reliable applications.
(Low effort) CI/CD Building, bundling, testing and releasing was more effort in 2000. Setting up a CI/CD pipeline in GitHub can be done in a short amount of time without maintenance costs - for most of my small projects I copy one yaml file, and I’m done.
Git Compared to no version control in the 90s, and CVS and SVN in 2000, Git is much, more productive, especially with merging and the prevention of merge conflicts.
Powerful Software Today a lot of powerful software is available, like Redis or Postgres. On one hand, these reduce costs, on the other they increase productivity by their performance, standardization. Today you don’t need to code things like Redis you would have needed to code twenty years ago (This was years even before Memcached)

So there are many factors that increased developer productivity from 2000 to now.

But from personal experience, looking into many startups, it looks like productivity has not risen. Developers do not release 10x more features per time unit. What could have eaten up all that additional productivity?

Some suspects:

Scrum meetings Today developers spend a lot of time in scrum meetings, something from 10% to 20% depending on sprint length.
Too many tools Developers might use too many different tools. Learning, maintaining, understanding and making them work together takes time.
More stakeholders Today there might be more stakeholders in companies. Whereas in the past marketing did their thing with print, today they are tightly integrated
Ever new and complex frameworks Today’s frameworks iterate fast, are complex and gain complexity adding new concepts and dropping old ones, for example, React. Developers re-learning frameworks on an ongoing basis
Over usage of software Developers use too many different technologies. Many companies use, for example, Kafka without a clear need for it.
SPA With the power of Javascript in the browser growing and the advent of REST, single page applications (SPAs) have conquered software architectures. SPAs seem to be set as a default web development model, although they might end up cutting development capacity in half.
Mobile applications Many companies maintain a mobile application on iOS and Android next to their web application. This reduces development capacity while increasing coupling and development overhead through coordination.
Complex cloud Cloud computing has many benefits. Cloud providers provide hundreds of different services for developers to choose from. Developers might use too many services and make their application needlessly complex and increase their cognitive load in a way that it reduces productivity.

And while there are suspects, they are just that, suspects. I don’t know how and where we lost productivity. Those finding it, will reign supreme.