My conspiracy theory about the cloud
I have a conspiracy theory. As a disclaimer, I'm just a random guy, but this is my observation. I think many businesses have a long term strategy of dumbing down consumers so that the population gradually becomes more and more dependent on their product.
It starts as a natural sales tactic. You solve a problem for people, so you want to make sure people understand that you can solve it better than they can themselves. Many services aren't competing with each other as much as they are competing against users doing it themselves. DHH, who's company makes project management software, has famously said his biggest competition isn't Microsoft Projects or some other megacorp backing a competing product, it's spreadsheets and emails.
In many ways this isn't a conspiracy, it's just a natural trajectory for most businesses. This is especially prevalent in the software business. A lot of software is a tool to get something else done. To sell any tool, you need to sell people on the fact that living without your tool is very hard. But that if they buy your tool life will be easy.
A concrete example of how effective this strategy can be is AWS. Before cloud computing, every software company that needed to host servers had to have engineers capable of linux system administration, writing and operating tools to manage things like database backups, deployments, build and test servers, and hardware knowledge. As hype for cloud computing grew, more companies bought into the idea that moving to the cloud would be more reliable and require less engineers.
In turn, those skills became less in demand, so less people started learning them, instead favoring learning the proprietary AWS tools/services. In a genius marketing move, AWS poured lots of resources into creating certification programs, and subsidized startups and students, to gain mind and market share with younger developers and companies. Eventually you hit a tipping point. Deep experts in bare metal deployments are harder and harder to come by and get more expensive. At the same time there aren't as many jobs that require those skills, and especially not entry level ones, as it becomes a more specialized job, so students and new engineers gravitate towards learning the cloud where there are more opportunities. Finally, hiring managers and leadership look at the job market and make technical decisions based on what's easiest to hire for. It becomes a self reinforcing cycle where people think hiring AWS experts is easier (because it is), so they decide to build their company around AWS, which in turn makes more jobs for AWS engineers, reinforcing the cycle.
The playbook is: get enough people hyped about the future ("cloud computing is the future, you're a dinosaur if you still host servers yourself"). Demonize the old way of doing things. It's dangerous, expensive, etc. Build proprietary services that you control. Aggressively market towards students, people early in their career, and startups. Subsidize them with low cost teasers for many years. Wait 10-15 years, and slowly ratchet up the prices as the industry gets more reliant and the cycle gets more and more reinforced. And yes, you do have to have an okay product, it can't all be smoke and mirrors.
For the naysayers who say that AWS is amazing and their success is solely due to having the best product. I agree. For niche use cases like having very spiky traffic that is only run for brief periods, it really can save money. I have no numbers to back this up, but I am confident in saying the majority of business on AWS are just using EC2, S3, and RDS, and don't meet that criteria. In other words, they need a file system, a database, and application servers, and they need a near constant/slow and steadily increasing amount of compute. For the privilege of using AWS, they then pay 5-10x more than it would cost to co locate at a regular datacenter, often with the exact same amount of staff.
Unfortunately the state of open source tooling isn't great. Using postgres as a microcosm because I know that niche the best. Self hosting postgres in a serious production environment is hard. You need to become competent with connection poolers, failover, backups, know how to connect read replicas, perform upgrades, etc. I've heard of open source projects that package all of these up like pg cloud native. But they are generally kubernetes based which is a whole other level of complexity.
I can only think of one or two engineers I've worked with in the past 5 years who would be confident self hosting a serious production database. I think if it weren't for RDS and AWS, the talent pool for those skills would be much larger, and likely the open source tooling would be much better. But they have captured enough of the market that there just aren't enough people doing it to create excellent open source tooling. At least that I'm aware of.
Once you know this pattern of purposefully trying to capture the minds of a generation of workers. You will see it in many places, because it is a very successful long term strategy. Any "student plan" service is a dead giveaway. Corporate certifications and "learning resources" are another. I don't have any good advice for how to deal with this. Hopefully you just realize that a lot of discourse you see is marketing. Think from first principles, don't be afraid of things that seem hard. And roll with the movments of the market.
Oh also AI companies are probably using this playbook to develop a whole generation of students who can't think without an AI creating a self fulfilling prophecy where AGI means dumbing down humans enough to make AI look like a super intelligence.
Woah I don't know what happened on that last part. Ok conspiracy over!