towards blockchain solution for any micro service ecosystems
As I have spent some time playing and working with blockchain solutions, at some point I decided to try and build general purpose blockchain system myself. So, to go beyond “your first blockchain” tutorial. I wanted to do it as an experiment, for purely academic purposes and see what comes out of it. Here is the idea behind it and general overview of what came out of my time spent on analyzing, designing and building product like this…
The idea is to understand what is needed to create blockchain that can be supplementary component for your backend — a service that plays nicely with your multi / micro service backend setup and provides blockchain functionality.
If built properly this can be generally useful to anyone wanting to remove single point of failure and strengthen data security by having decentralized, replicated and secure database. Ideally, this solution should reduce throughput of the system as little as possible, but if your data handling is so special that you need blockchain, performance is small price to pay.
What is general purpose blockchain
For more proper answer we got to look at long list of resources I prepared here. Just a glance at content of given document tells you that world is still figuring out what is blockchain and what exactly this new piece of tech is able to solve for us. And if you exclude money, what are the actual use cases to optimize for?
On the other hand, what is basic set of features general purpose blockchain should have?
To answer this, we need to first look at were it may fit in your system (if it is not your core competency, product, use case and business):
Blockchain storage solution is just another storage solution with specific use case - storing transaction and sensitive data in immutable way. If you look at simplified microservice system depicted above, it is just additional store that command and event bus use to store certain information to.
Base set of features needed to be generally applicable
It should be a tool, not a star of the system
Obviously, if we are building something to be generally used as a suplement to applications, we should have something that is:
- Easy to deploy
- Highly configurable (it should adopt to needs of a system, not vice versa)
- Integration into the system should be super simple (e.g. you deploy and configure it and after that it all goes through simple RESTful API)
It should have multiple consensus algorithms or work distribution systems you can choose from
Due the fact that blockchains are distributed systems, consensus algorithms (AKA proof systems) are important part of blockchain functionality. They provide certainty around data consistency on the network, thus providing security to data storage, are responsible for data integrity and, in some cases, ensure proper work distribution.
General purpose blockchain should provide set of initial approaches to finding consensus in scalable way. Now, there is a few established consensus algorithms but using some for this case, make sense more then using others.
I suggest few from following set of algorithms:
- PoW - Proof-of-work algorithm (maybe not that needed for private industrial blockchain, but so basic that can be considered requirement)
- DPoS - Delegated proof-of-stake algorithm
- DBC - Delegated block creation
PoW - Simply put - mining machine (the one creating the lock) creates certain proof that is hard to create initially (needs some computational work), but is easy to check (like number coming after last proof and is dividable by last proof and 255).
DPoS - Generally this means that several nodes in the system choose representative and then that representative represents their interest on the network. Going deeper into implementation - network is designed is such a way that there are clusters of nodes that all connect to single mining node and that mining node is on the networks whitelist so it can broadcast newly created block. In network like this, I would recommend developing work scheduling system as well to avoid data write conflicts. Note: in fully distributed system (like crypto ledger), we would also pay attention to stuff like voting system to elect mining nodes, anonymity etc, in case of industrial blockchain it is all about use case and optimal way to achieve or maintain consensus.
DBC - this type of consensus algorithm is only to be used in private networks and is pure scheduled resource consumption - you basically broadcast transactions to all the nodes and execute every next block creation in next node on the list, or rather next healthy node on the list. Maintaining replication, stability, consensus and work distribution across the network. If done like this, other distributed computing algorithms can be used on this system, like searches and such.
It would be a plus if solution additionally provides a way to write and use new algorithm (as a plugin, or being easily added by community), tho I think each of these would be heavily embedded into the structure of a system.
Multi chain
It should be able to spawn multiple chains withe configurable structure of blocks.
In order for it to truly be general purpose we have to cover two obvious cases, first of them being the need for multiple and differently structured chains being maintained the same app.
Denormalization
Second important need to be aware of is denormalizing the data in certain form, from blockchain to some form of conventional database or reporting system.
It should offer content encryption out of the box and on by default
It would be lovely if we can just add key to encrypt and then be and then define fields to be encrypted in particular chain.
It should be easily deployable and integratabtle in devops systems
Provide Dockerfiles, manifests and other receipts out of the box, to make integration easier. Maybe also provide versioned api contracts to maintain and monitor stability.
It should have some kind of admin UI
Really thin UI layer built in something lite that just allows manipulation with small set of features. Basic features:
- Adding users
- Forking chains
- Managing encryption and keys
Basic analytics dashboard
Looking at usage, chain sizes, performance. Analyzing performance impact of chain growth etc.
Provide at least plug to send data to Google Analytics, and plan to build more.
Nice to have
- Custom code execution (aka “contracts” capability)
- Sockets integration (as an option, but possibly one of the technologies to use for scheduling control for DBC)
- CLI client
How to do it?
Not completely sure yet, but I know how to design it, and I do have some programming skills, so I have experimented with what has to be done in order for open source general purpose blockchain to make sense.
Altho done purely for academic purposes, in order to satisfy hunger for knowledge and hands on experience with thing like this, I think it has all the potential to grow further.
For this reason, I have spent some effort to add structure around the project, provide enough documentation (still some work in progress) and also add the Trello board and the license. Project can be found in the repository https://github.com/metaphorical/quantum-blockchain, I have described my idea of contribution process in readme and I want to include anyone who wants to contribute, so feel free to jump onboard if this project is interesting enough for you.
I will keep developing the project in my free time as I want to take it to version 1.0 with rounded set of features (MVP that is still undefined).