The Technology Vortex

Straight away, I want to talk about what I call the technology vortex. Specifically, the AmaGooBookSoft technology vortex, herein referred to simply as the vortex.

AmaGooFaceSoft is a term I learned from Patrick McKenzie (better known as patio11 on the Internets). It stands for Amazon, Google, Facebook and Microsoft, the Big Four. Netflix is up there too. These companies are all unique in what they do and how they operate. But they all share a similar goal and a similar problem. The goal of serving as much of the world's population as possible and the related problem of scale.

The world and scale

The world has 7 billion people. The Big Four are targeting this world population but are nowhere near that target. A lot of people don't even have access to the internet. A lot of people who do have access to the internet use alternatives to the Big Four. My point is, getting even 1 billion people to use your product is not easy. It comes with massive technology scale issues.

Solving scaling problems has become the in-thing lately. Not because everyone has them, but because everyone would like to. These problems are good to have. They are usually an unmistakeable indicator of business growth. In the startup world, they are sometimes the only indicator that matters. I personally think that is a crazy notion, but go figure.

The technology

The Big Four have each faced a unique version of this problem in their chase for world dominance. They have faced similar versions of this problem as well. They have each invented a number of technical solutions to this problem. I am thinking Kubernetes, Hadoop, Cassandra etc. Some of these solutions are not even clearly scale problem solutions. I am thinking GraphQL, HHMV etc. They just seem like cool technology that everyone should use.

Most of these solutions clearly work. They are keeping these tech giants running after-all.

The vortex

Now, the big problem. The vortex. These solutions have been battle-tested and have come out on top. They are cool. They are very attractive. They suck in even people who do not need to use them. Think about it for a second...

None of these solutions were invented by these companies while they were young or small. They all were invented once these companies were big and had real scaling problems. By scaling problems I am not talking about a few hundred thousand users that are all located in the same timezone and have predictable traffic patterns. I am talking about millions and billions of users that are distributed across all timezones and have unpredictable traffic patterns. I am talking about being a target of malicious attacks everyday. I am talking about supporting thousands of third-party applications on your developer platform. These are problems that a lot of companies only dream about.

Companies are quick to adopt these solutions. They convince themselves they need these solutions even when they do not. They pre-scale. They develop for a billion users when they will never see a billion users. They develop for unpredictable traffic when they will never see such traffic. This is the tech vortex. They do these things just so they can use technology that is used by Google. They do it because they want to be considered peers of Facebook. They do it because they consider themselves considerable competition to Amazon.

The ugly side of the vortex is that it is easy to see things working using these solutions and think you are on top of your game. The negatives are all hard to measure and bigger than you might think. The time taken for initial setup and understanding of these solutions is usually a lot. The cognitive load on Engineers required to comfortably understand systems that run on these solutions is usually high and means longer times to develop features and fix bugs. These solutions usually need auxiliary software that you might not have the Engineers or money to support.

The cure

I think companies need to live by the following:

Do things that don't scale – Paul Graham

I also think companies should choose simple over complicated and 'scales'. They can afford to do this. The chances of seeing Google scale traffic are slim and the chances of this happening under your feet are even slimmer.

They must choose simple solutions that allow Engineers to move fast and innovate faster. When the time to use these solutions comes they will know.

Echo

Always keep in mind that Github was built on pure Ruby on Rails. Instagram was built on pure Django and using Chef to deploy to AWS instances. None of these companies died because they couldn't scale.

Twitter made design mistakes that they blamed on Ruby but even they are still around to tell their story thanks to Ruby on Rails.

Stay safe from the vortex.