What’s the future of open source?

by Team Tacanae on Aug 08, 2021 23:34:55

Before 2005, most of the technology was open source and enterprises were consuming open source to accomplish their business goals and the trend seems to continue to grow. Take the operating system as an example. The use and popularity of Linux have been growing day by day. Another good example is the database industry where MySQL and Postgres are rapidly growing over time as opposed to Oracle or Microsoft’s SQL Server.

The open-source trend was not surprising as proprietary technology leads to lock-ins and as these enterprises grow, their reliance on these foundational pieces continues to grow over time. This makes it really difficult and costly for these enterprises to switch to new technology, locking them into the closed-source technology and leaving them vulnerable to hefty prices or punitive charges as they scale. This drove many enterprises to come together and join hands to collectively build different open-source components.

The use of open source technology freed enterprises from potential lock-ins but led to other challenges. One of the main challenges with open source was enterprises needing a dedicated team to manage the technology over time in order to customize them, to upgrade them, to extend them for better usability or observability or auditing, etc. Human resource is one of the most expensive resources and spinning a team up for every open-source software is not scalable. As a result, enterprises started abstracting the class of these technologies within their corporation by having different teams manage different use-cases. The other major advantage of this model is that as open-source technology evolves, it makes it easier for enterprises to replace individual technologies without changing the interface to the application team for the most part.

Many enterprises started embracing this new model and it still exists in many enterprises but it was still not efficient for many reasons. First, the optimization to batch different related technologies makes sense but it is still not sufficient. Look at it this way, every enterprise has to learn about operating different classes of technology at scale and have a team to maintain them. Amazon realized the same and saw the potential of externalizing these learnings by offering a managed service to other enterprises in the world, which led to the evolution of Cloud through AWS (short for Amazon Web Services). Soon after, many other providers emerged from Azure Cloud to Alibaba Cloud.

The emergence of the Cloud really allowed these enterprises to not manage these open source technologies by themselves and instead focus on their core technology by offloading some of these components to cloud providers without any lock-ins. The other advantage was that these enterprises don’t have to buy/set up data centers throughout the globe, a huge advantage to startups for not requiring upfront costs and time commitment. It’s not surprising most modern-day startups are built on top of the Cloud. Is that it? Have we reached the final nirvana state? Is the future of open-source in the hands of the Cloud?

The rise of the Cloud led to its own challenges: (a) vendor lock-in instead of software lock-in (b) control on sensitive data (c) resiliency.

We are seeing a shift by CIOs/CTOs to increasingly deploy multi-cloud and hybrid-cloud strategies as a result. In fact, there are startups like Aiven that are making it easy to adopt multi-cloud strategies for enterprises. For sensitive data, there is a constant trend by different public cloud providers to provide better integration with their on-prem datacenter or partnerships with private cloud providers.

Even with all these, there is still an imbalance of incentives where entrepreneurs and enterprises are not incentivized to open-source any new technology, just to pay for it later once hosted by the cloud providers. The other challenge is that cloud providers are not contributing back to the community. In fact, they are creating new proprietary technologies by making some of the open-source technologies cloud-native and charge more premium for it. As a result, we are seeing another paradigm shift where open source innovators are taking their offering to the cloud and offering it directly to customers as a managed offering.

Cloud trend

Typically, these offerings from open-source innovators are more cloud-native and Saas than what's offered by the cloud providers because they come from the strength of building the underlying technologies. For end-customer enterprises, they not only have the benefits of multi-cloud but also get a Saas offering directly from the innovators. As an example, AWS offers RDS MySQL or MSK but they rely on the community to make the bug fixes or security patches to MySQL and Kafka. Instead, companies like Databricks or Confluent offer a fully managed Saas offering over multi-cloud for better flexibility, resiliency, compliance, cost-effectiveness.

It’s no surprise that there has been a sudden rise in Saas providers from MongoDB to Redis Labs. Here is an interesting article from coss.media. As you can see there is a massive increase in #startups from 4 in 2018 to 27 in 2021 so far:

We would love to hear your thoughts and hear what you think.