From Agile to Serverless

If you prefer, read the same post on Medium. 

Looking back. And forth.

The microservices architecture was born as a technological answer for the iterative Agile development methodology. At the early days of microservices, many companies were doing a form of Agile (XP, Scrum, Kanban, or a broken mixture of these) during development, but the existing software architectures didn't allow an incremental way of design and deployment. As a result, features were developed in fortnight iterations but deployed every six to twelve months iteration. Microservices came as a panacea in the right time, promising to address all these challenges. Architects and developers strangled the monoliths into tens of services which enabled them to touch and change different parts of the system without breaking the rest (in theory).

Microservices on its own put light into the existing challenges of distributed systems and created new ones as well. Creating tens of new services didn't mean they are ready to deploy into production and use. The process of releasing and handing them over to Ops teams had to be improved. While some fell for the extreme of "You Build It, You Run It", others joined the DevOps movement. DevOps meant, better CI/CD pipelines, better interaction between Devs and Ops and everything it takes. But a practice without the enabling tools wasn't a leap, and burning large VMs per service didn't last for a very long. That led to containers with the Docker format which took over the IT industry overnight. Containers came as a technical solution to the pain of microservices architecture and DevOps practice. With containers, applications could be packaged and run in a format that the Devs and Ops would both understand and use. Even at the very early days, it was clear that managing tens or hundreds of containers will require automation, and Kubernetes came from the heavens and swept all the competition with a swing.

Now, we do Agile development and use microservices architectures. We have one-click build and deployment pipelines with a unified application format. We practice DevOps and manage hundreds of microservices at scale, do blue-green and canary releases with Kubernetes. This was supposed to be the time of long-lasting technological peace where we can focus on the business problems and solve them. But it was not.

From Agile to Serverless and Beyond

With Kubernetes, Devs can develop and Ops can deploy in a self-service manner. But there are still only one or two company-wide Kubernetes clusters that occasionally run out of resources. The cluster is now the center of the universe. And it is even more critical than the monolith was before as it runs the build server, and the git server, and the maven repository, the website, MongoDB, and a few other things. So a promise for a serverless platform came from the clouds. It came even before the containers reached into production. With serverless, every service would have its own cluster, a cluster where resources are limited only by the limit of the credit cards. It is a platform to which you tightly couple, and bet the existence of your company for the price of going faster. So it started all over again.

How Blockchain will Influence Open Source

This post was originally published on Opensource.com under CC BY-SA 4.0. If you prefer, read the same post on Medium.

Interactions between users and developers enabled by blockchain technology can create self-sustaining, decentralized open source.

What Satoshi Nakamoto started as Bitcoin a decade ago has found a lot of followers and turned into a movement for decentralisation. For some, blockchain technology is a religion and will have the same impact on humanity as the Internet had. For others, it is another hype and a technology suitable for Ponzi schemes only. While blockchain is still evolving and trying to find its place, one thing is for sure: it is a disruptive technology that will fundamentally transform certain industries. And my bet is, open source will be one of them.

The Open Source Model

Open source is a collaborative software development and distribution model that allows people with common interests to gather and produce something that no individual can create on their own. It allows the creation of value that is bigger than the sum of its parts. It is enabled by distributed collaboration tools (IRC, email, git, wiki, issue trackers, etc), distributed and protected by an open source licensing model and often governed by software foundations (such as Apache Software Foundation (ASF), Cloud Native Computing Foundation (CNCF), etc.).

One interesting aspect of the open source model is the lack of financial incentives in its core. There are some who believe open source work should remain detached from money and remain a free and voluntary activity driven by intrinsic motivators only (such as "common purpose" and "for the greater good”). And there are others who believe open source work should be rewarded directly or indirectly through extrinsic motivators (such as financial incentive). While the idea of open source projects prospering only through voluntary contributions is a very romantic one, in reality the majority of the open source contributions are done through paid development. Yes, we have a lot of voluntary contributions, but that is on a temporary basis from contributors that come and go, or for exceptionally popular projects while they are at their peak. Creating and sustaining open source projects that are useful for the enterprises, requires developing, documenting, testing, bug fixing for prolonged periods even when the software is no longer no longer shiny and exciting. It is a boring activity that can be best motivated through financial incentives.

Commercial Open Source

Software foundations such as ASF accept and survive on donations and other income streams suchs as sponsorships, conference fees, etc. But those funds are primarily used to run the foundations, to ensure there is a legal protection for the projects, to ensure there are enough servers to run builds, issue trackers, mailing lists, etc. 
In a similar manner, CNCF has member fees (and similar other income streams as well) used to run the foundation and provide resources for the projects. Nowadays, most software is not build on laptops, it is run and tested on hundreds of machines on the cloud, and that requires money. Creating marketing campaigns, brand designs, distributing stickers, etc. all takes money and some foundations can assist with that as well. At its core, foundations implement the right processes for interacting with users, developers, and control mechanisms for distribution of the available financial resources to the open source projects for the common good. 

If users of the open source projects can donate money and the foundations can distribute it in a fair way, what is missing then? What is missing is a direct, transparent, trusted, decentralized, automated bidirectional link for transfer of value between the open source producers and the open source consumer. Currently, the link is either:

• Unidirectional: a developer (BTW, I'm saying a developer, but think of it as any role that is involved in the production, maintenance and distribution of software) can use their brain juice and devote time to do a contribution and share that value with all open source users. But there is no reverse link.
• or Indirect: if there is a bug that affects a specific user/company, the options are:

• To have in-house developers to fix the bug and do a pull request. That is ideal, but not always possible to hire in-house developers knowledgeable about hundreds of open source projects used daily.
• To hire a freelancer specialising around that specific open source project and pay for the services. Ideally, the freelancer is also a committer for the open source project and can directly change the project code quickly. Otherwise, the fix might not make it to the project ever.
• Or to approach a company providing services around the open source project. Such companies typically employ open source committers to influence and gain credibility in the community and offer products, expertise and professional services, etc.
  

And the last one has been a successful model for sustaining many open source projects. Whether that is through services (training, consulting, workshops), support, packaging, open core, SaaS, there are companies employing hundreds of staff to work on the open source for full time. There is a long list of companies that have managed to build a successful open source business model over the years and the list is growing steadily. 

The companies backing open source projects have a very important role to play in the ecosystem. They are the catalyst between the open source projects and the users. The ones that add real value are not only packaging software nicely, but do much more. They can identify user needs, technology trends and create a full stack and even ecosystem of open source projects to address these needs. They can take a boring project and support it for years. If there is a missing piece in the stack, they can start an open source project from scratch and build a community around it. They can acquire a closed source software company and open source the projects. Here I got a little bit carried away, but yes, I'm talking about my employer Red Hat and what we do among other things.

To summarise, with the commercial open source model, the projects are officially or unofficially managed and controlled by a very few individuals or companies that monetise them and also give back to the ecosystem by ensuring the project is successful. It is a win-win-win for the open source developers, managing companies and end users. The alternative is inactive projects and expensive closed source software.

Self-sustaining, Decentralized Open Source

In order for a project to become part of a reputable foundation it has to conform to certain criteria. For example, at ASF and CNCF there are incubation and graduation processes respectively where apart from all the technical and formal requirements, a project must have a healthy number of active committer and users. And that is in the essence of forming a sustainable open source project. Having source code on Github is not the same as having an active open source project. The latter requires committers that write the code, and users using the code, and both groups enforcing each other continuously by exchanging value and forming an ecosystem where everybody benefits. Some project ecosystems might be tiny and short lived, and some may consist of multiple projects and competing service providers, with very complex interactions lasting for many years. But as long as there is an exchange of value and everybody benefits from it, the project is developed, maintained and sustaining.

If you loot at ASF Attic, you will find projects that have reached their end of life. Usually, that is the natural end of a project when it is not technologically fit for purpose any longer. Similarly in the ASF Incubator you can find tens of projects that have never graduated but got retired instead. Typically, these are projects that are not able to build a large enough community because they are very specialized or there are better alternatives available. But there are also cases where projects with high potential and superior technology cannot sustain themselves because they cannot form or maintain a functioning ecosystem for exchange of value. The open source model and the foundations do not provide a framework and mechanisms assisting developers get paid for their work, or users get their requests heard. There isn’t a common value commitment framework for either party. As a result, some projects can only sustain themselves in the context of commercial open source where a company acts as an intermediary and value adder between developers and users. That puts another constraint and the necessity of a service provider company for sustaining some open source projects. Ideally, users should be able to express their interest in a project and developers should be able to show their commitment to the project in a transparent and measurable way and form a community with common interest and intent for exchange of value.

Now, let's imagine there is a model, mechanisms and tools that enable direct interaction between the open source users and the developers. Not only code contributions through pull requests, questions over the mailing lists, GitHub stars, stickers on laptops, but also other ways to allow users influence projects destinies through a more richer, self-controlled and transparent means. That could include incentives for actions such as the following:

• Funding open source projects directly (rather than through software foundations)
• Influence the direction of projects through voting (by token holders)
• Feature requests driven by user needs
• On-time pull request merges
• Bounties for bug hunts
• Better test coverage incentives
• Up-to-date documentation rewards
• Long-term support guarantees
• Timely security fixes
• Expert assistance, support and services
• Budget for evangelism and promotion of the projects
• Budget for regular boring activities
• Fast email and chat assistance
• Full visibility of the overall project findings, etc. 
  

If you haven't guessed already, I'm talking about using blockchain and smart contracts that will allow such interactions between users and developers. Smart contracts that will give power to the hand of token holders to influence projects.

The usage of blockchain in the open source ecosystem.

The existing channels in the open source ecosystem provide ways for users to influence projects through financial commitments to the service providers or other limited means through the foundations. But the addition of blockchain based technology to the open source ecosystem could open new channels for interaction between users and developers. I'm not saying this will replace the commercial open source model. Most companies working with open source do much more and cannot be replaced by smart contracts. But smart contracts can spark a new way for bootstrapping new open source projects. A new way to give a second life to commodity projects that are a burden to maintain. A new way to motivate developers to apply boring pull requests, write documentation, get tests to pass, etc. A direct value exchange channel between users and open source developers. It can add new channels to open source projects to grow and be self-sustaining in the long term even when a company backing is not feasible. A new complementary model for self-sustaining open source projects. A win-win.

Tokenizing Open Source

There are already a number of initiatives aiming to tokenize open source. Some are focused only on an open source model, and some are generic but apply to open source development as well. Here is a list I built so far:

• Gitcoin - grow open source, one of the most promising ones in this area.
• Oscoin - cryptocurrency for open source
• Open Collective - platform for supporting open source projects.
• Fund Yourself
• Kauri - support for open source project documentation.
• Liberapay - recurrent donations platform.
• FundRequest - a decentralized marketplace for open source collaboration.
• CanYa - who recently acquired Bountysource - the world’s largest open source P2P bounty platform.
• OpenGift - a new model for open source monetization.
• Hacken - a white hat token for hackers.
• Coinlancer - decentralized job market.
• CodeFund - open source ad platform.
• IssueHunt - funding platform for open source maintainers and contributors.
• District0x 1Hive - crowdfunding and curation platform.
• District0x Fixit - github bug bounties.

The list above is varied and growing rapidly. Some of these projects will disappear, some will pivot, but a few will emerge as the SourceForge, the ASF, the Github of the future. Not necessarily replacing these platforms, but complementing them with token models and creating a richer open source ecosystem. So every project can pick its distribution model (license), governing model (foundation), and incentive model (token). In all cases, this will pump fresh blood to the open source world.

The Future is Open and Decentralised

• Software is eating the world.
• Every company is a software company.
• Open source is where innovation happens.

Given this, it is clear that open source is too big to fail, too important to be controlled by few or left to its own destiny. Open source is a shared-resource system that has value to all, and more more importantly it must be managed as such. It is only a matter of time until every company on earth will want to have a stake and a say in the open source world. Unfortunately, don't have the tools and the habits to do it yet. Such tools would allow anybody to show their appreciation or ignorance of software projects. It would create a direct and faster feedback loop between producers and consumers, between developers and users. It will foster innovation, innovation driven by user needs and expressed through token metrics.