Concerto Network: Verifiable validation and aggregation compute over real-time data

[ryanwould]

New POC Proposal

Concerto Network: Verifiable validation and aggregation compute over real-time data

Core Concept and purpose statement

In this Proof of Concept (PoC), we present an upgrade plan for transforming our existing Log Store Network node into the Concerto Network node. The Log Store currently relies on the Streamr Network for peer-to-peer real-time publish/subscribe tamper-proof data transport. While its original function was to store data in a decentralised and immutable manner, we are now reevaluating the network’s architecture and functionality. Immutability is being postponed in favour of default ephemerality, and we are expanding our scope to simplify the creation of custom data protocols within a decentralised network.

The proposed upgrade will harness the Cartesi Machine to establish a verifiable and stateless compute environment within the node. This environment allows the execution of bespoke programs over incoming custom event data, delivering a decentralised and secure solution for real-time data processing that encompasses validation and aggregation. Moreover, once data undergoes processing, it is made available for real-time subscription and can be conveniently stored for future queryability.

The overarching goal of the Concerto Network is to revolutionise the processing of off-chain data in real time. It aims to provide a trusted and tamper-proof platform for creating data protocols applicable across a wide spectrum of applications. Use cases range from confirming blockchain events to multi-party compute and price feed aggregation.

Technical details: How will you use Cartesi, specifically?

In this PoC, Cartesi technology plays a pivotal role in enabling the secure and verifiable execution of bespoke programs over real-time data. Let’s break down how Cartesi is specifically used in the PoC and why it’s a suitable choice:

Network Architecture:

1. Data Publishing: The network architecture involves data publishers who create and sign events, publishing them to specific streams. Streams are identified on-chain, allowing nodes to subscribe to them.
2. Data Storage: Data received by the network is stored within each node, and consumers can query the network to retrieve aggregated data. Deduplication of query responses is the network’s default behaviour.
3. Upgrade: The PoC introduces an upgrade where data is processed immediately upon receipt. The output of this data processing is published to a Topic Stream, offering an efficient means of aggregating data.
4. Topics: Streamr Streams are a sequence of events produced by a set of publishers with appropriate permissions which are determined on-chain. A Topic is a new concept introduced for this PoC that encompasses a stream of data sourced from many streams. Topics allow for one or many streams to aggregate into a single stream of data.

System Architecture:

1. Middleware Component: Node operators’ systems include middleware, which is a Streamr Broker Node Plugin, for managing real-time data streams. Data is also stored in a time-series database.
2. Template Retrieval: Bespoke programs, also referred to as Cartesi Machine Templates, are referenced in a Topic defined on-chain. The Topic contains a parameter pointing to the Cartesi Machine Template. These templates are stored in a decentralised storage network like IPFS or Arweave.
3. VM Orchestrator: The execution of the Template is facilitated by a VM Orchestrator, responsible for loading and executing the Cartesi Machine Template in an isolated and stateless compute environment.
4. Data Processing: When an event is received by the middleware component, it is stored in the database, forwarded to the VM Orchestrator, and executed within the Cartesi Machine. The Cartesi Machine receives the event data through the Flash Drive input.
5. Throttling: The VM Orchestrator can throttle the execution of the Cartesi Machine if the Topic parameters dictate the need for aggregation over a specific time frame.
6. Result Publication: The output of the Cartesi Machine, along with the final state proof, is returned to the VM Orchestrator, and then to the middleware, where it is published to the Topic Stream. This allows consumers to subscribe and receive processed data in real time.
7. Data History Storage: The result and state proof are stored in the database allowing data consumers to query the Topic Stream for its historical record of processed events.

Technology Selection:

• VM Orchestrator: The PoC references the use of a VM Orchestrator. FirecrackerVM seems to be the best-in-class technology for this purpose as it powers AWS Lambda. The choice of the specific orchestrator may require further research to determine the most suitable technology for this use case.

Why Cartesi Makes Sense:

Cartesi Machine enables the execution of customised polyglot programs within a secure, verifiable and state-persisting compute environment. This is required to offer developer freedom in achieving verifiable compute over real-time data.

The use of Cartesi in this PoC offers a reliable and secure foundation for the execution of bespoke programs over real-time data, aligning with the project’s goals of revolutionising data processing in a decentralised and tamper-proof manner and powering a network that can enable secure data protocols from centralised environments.

Value Proposition

The PoC introduces a second network platform where the Cartesi Machine is applied for a fundamentally different purpose. This diversification enables developers to explore new horizons empowered by Cartesi’s technology.

Developers within the Cartesi ecosystem can leverage their understanding of Cartesi technology to create Cartesi Machine Templates. These templates can be deployed in conjunction with Topics to establish new data protocols, opening up a realm of monetisable opportunities.

The successful execution of this PoC expands the Cartesi ecosystem, attracting developers interested in deploying Cartesi Machines for a wide range of applications.

Estimated Duration and Funds Requested

Duration: 3 - 4 months

Funds request (USD) for the POC: $25000

Milestones and Timeline:

• (1 week) Research and Finalise VM Orchestration Engine
• (2 weeks): Create TopicManager Smart Contract
  • Develop the TopicManager Smart Contract for managing topics and Cartesi Machine Templates.
  • Implement necessary functionalities and testing.
• (3 weeks): Create VM Orchestrator Controller Module
  • Develop the VM Orchestrator Controller Module to facilitate the execution of VMs that load and execute Cartesi Machine Templates.
  • Integrate it with the Middleware Component.
• (3 weeks): Modify Middleware Component
  Modify the Middleware Component of the Node to include messaging for communication with the VM Orchestrator Controller.
• (2 weeks): Deploy VM Orchestrator and Isolated Node Operation
  • Deploy the VM Orchestrator to a Node and ensure its proper operation in an isolated environment.
  • Test the orchestration of Cartesi Machine execution.
• (3 weeks): Create VM Environment
  • Create a secure VM environment that allows for Cartesi Machine execution.
  • Ensure the environment is stateless and isolated to guarantee data integrity.
• (2 weeks): Create Cartesi Machine Template
  Develop a Cartesi Machine Template specifically for validating EVM events using Merkle-Patricia-Trie Proof verifications.

Subsequent Vision and Extensibility

The successful PoC is a cornerstone for a much larger and transformative project. In this larger endeavour, the Concerto Network will see full deployment and utilisation. The primary objective is to streamline the creation and use of bespoke inter-system data protocols. These data protocols are essential for securing the integration of off-chain data with Smart Contracts. The essential nature of DONs are a testament to this.

While existing DONs are capable of handling certain data interactions, such as price feeds, they often lack or limit programmability to customise different aspects of the network. Typically, off-chain data is centralised, which poses a security risk known as “oracle manipulation.” The larger project aims to mitigate this risk and provide a more secure and reliable solution for data integration.

Following the successful proof of concept, the immediate next steps include:

1. Deploying the Concerto Network to make it accessible for real-world use. This step is relatively straightforward due to the early stage of the project.
2. Developing user-friendly and efficient developer tools to simplify the process of deploying programs to the network.
3. Optimising the network’s infrastructure and performance to transition from a proof of concept to a production-grade system. This optimisation will ensure the network can handle real-world data processing at scale, making it a robust and reliable solution.

Reusability and Other Use Cases
The PoC includes reusability through the open source VM Orchestration tool and the VM environment for Cartesi Machine Templates. A possible use case is enabling Web2 applications to embed a SQLite database inside of stateless and serverless architecture.

As for the larger platform, bespoke data protocol use cases include:

1. Historical blockchain data can be confirmed by the network through generated proofs, or by re-requesting the event from within deployed programs. This has implications for auditing, data verification, and ensuring the integrity of historical blockchain records.
2. Cryptographic functions can be verified across the network ensuring that centralised parties do not manipulate the outcome of these functions.
3. zkRollups can be created by enabling the network to generate zkProofs of logic over data.
4. By using a centralised node to ingest confirmations on-chain for verification, the project facilitates the creation of cross-chain infrastructure. This infrastructure can enable the seamless transfer of assets and data across different blockchains. Regardless of the consensus mechanism or programming environment of different blockchains, this technology offers a common and secure ground for confirming data across blockchains.
5. The abstraction of data processing and security simplifies the creation of new Oracles. Data can be sourced by a network of independent relayers from anywhere and about anything, eliminating the need for deploying entirely new DONs. This is especially important to usher in new Oracle Data Feeds for RWAs.

Risks and Contingency Plans

Complexities in Operating the VM Orchestration Engine:

The operation of the VM Orchestration engine may introduce complexities, especially when customising it for the use case and ensuring seamless inter-communication between different components of the system.

To mitigate this risk, the project team will establish a set of conditions and milestones to verify the completion of deliverables for the PoC. Each requirement will be time-boxed to prevent over-engineering and perfectionism. This approach will help ensure that the project stays on track and meets its objectives within the allocated timeframe.

State Management in Cartesi Machine:

Understanding and managing the state within the Cartesi Machine can be challenging, and there might be edge cases where Cartesi Machine Templates yield different state outcomes.

The project team will proactively engage with the developer community to gain insights into potential edge cases and variations in state outcomes. By fostering collaboration and community involvement, any misunderstandings or uncertainties can be addressed promptly. Developer guidance will be adjusted based on community feedback and collective knowledge.

Timeline Extensions:

The timeline for the PoC might face extensions due to unforeseen complexities or issues that arise during development.

Regular progress monitoring and milestone verification will be implemented. If deviations from the timeline occur, the project team will assess the reasons for the delays and adjust the approach accordingly. This may involve reprioritising tasks, allocating additional resources, or revising project timelines to stay within the planned scope.

Resource Availability:

The availability of resources, including development skills and expertise, could impact the project’s progress and success.

If certain skills or expertise are lacking, collaboration with external experts or the developer community may be sought to fill any gaps.

Success criteria

Condition for Success: The PoC is considered successful if the system can perform an end-to-end test where it receives an Ethereum blockchain event over Streamr, stores that event, loads the event into a virtual machine (VM) that executes it within the Cartesi Machine Template. The Cartesi Machine should produce a correct result regarding the event’s verification using the current blockHash from a public RPC endpoint.

Deliverables to Demonstrate Success: The PoC will produce a set of deliverables that serve as evidence of its successful execution. These deliverables include:

1. Creation of a New Monorepo on Github: The project will establish a dedicated repository on GitHub to host the code and related materials for the PoC. This repository will serve as a central hub for the project’s documentation, source code, and resources.
2. Deployment of the VM Orchestration Engine: The PoC will demonstrate the successful deployment of the VM Orchestration engine, showcasing its ability to orchestrate the execution of Cartesi Machine Templates within an isolated and stateless virtual machine environment.
3. Docker Composition: The PoC will provide a Docker composition that includes essential components such as the middleware, VM orchestration controller, and the time-series database. This composition will illustrate the interaction between these components within the system.
4. End-to-End Test: The most critical deliverable is the successful completion of the end-to-end test mentioned in the success criteria. This test will demonstrate the system’s capability to receive, store, process, and verify an Ethereum blockchain event, using Cartesi Machine and current block hash, as specified. A video walkthrough of this test will accompany the deliverable.

Adaptation to Potential Challenges: In case Docker to VM Engine communication proves to be challenging, the PoC will adapt by considering alternative solutions, such as a Nix build or an Ubuntu bootstrapping module. The flexibility to adapt to challenges is an integral part of the success criteria.

Final report

We agree to report to the community as to why we failed, and where the failure occurred.

About Your Development team

Usher Labs

Log Store Repo: GitHub - usherlabs/logstore-mirror: Public mirror repository of the Log Store Network
CCAMP Repo: GitHub - usherlabs/ccamp: The repository for the cross chain asset management protocol

• Ryan
  Github: rsoury (Ryan Soury) · GitHub
  Dev Lead and the solutions architect driving the development of the Log Store. Ryan is capable of engineering across the entire stack, from frontend and backend to Smart Contracts. With a proven track record of successful projects, experience delivering custom software solutions for companies across Australia, and expertise in blockchain technology, Ryan leads our team with a commitment to innovation.

• Victor
  victorshevtsov (Victor Shevtsov) · GitHub
  An indispensable member of our team, specialising in DevOps, infrastructure operation, backend software development, and data engineering. His journey into the Web3 world ignited a passion for learning Rust, which he has effectively applied to Smart Contract development in a proprietary project that harnessed the capabilities of Solana and Wormhole. Victor’s expertise ensures the robustness and efficiency of our infrastructure, making him a key contributor to the project’s success.

• Alex
  Xand6r · GitHub
  Brings a wealth of experience from the Web3 startup world, including his contributions to projects like EPNS. Within our team, he takes charge of Smart Contract development across blockchains, review, and testing, in addition to his role in dApp development. His comprehensive skills and deep insights into the Web3 landscape are invaluable to our project, ensuring the reliability of our codebase.

ERC-20 Payee address

0xc6D330E5B7Deb31824B837Aa77771178bD8e6713

[hellen]

Hi @ryanwould. Thank you for submitting your proposal. I’ve reached out to you via forum DMs to schedule a call to discuss it further.

[gabrielbarros]

Hey guys! Good call today =]

We were happy to get to know your team and we liked the ambition of your project, but we are asking for a few things to keep moving forward.

The current version of your text is good and gives us a lot of context into what you are thinking; however, it’s not a Cartesi-community-focused proposal.

We suggest you keep this text in a different place (Medium, GitHub markdown…) and then link it here so people can still have the option of such context.

Here, you can drill down more into what your solution will use the Cartesi Machine for and how it impacts the overall community.

Also, Pedro thinks a demo of what you already have can add value to the proposal (he suggests that you could have mockups of missing pieces to make it easier to show).

I missed a bit of a display of how your arbitration process would work. Here’s what I mean:
So far, the Cartesi Machine functions as a way for honest, non-faulty nodes to ensure they are “on the same page” and in agreement. What happens when they are not? How can they recover?

The Cartesi Machine implementation on Compute and Rollups has a proposed arbitration process to enforce the correct result. Still, this version relies on the base layer as the source of accurate correctness.

I am excited to see what your team can build using this tech!!

[ryanwould]

Hi Gabriel,

Likewise, we appreciate the time to discuss.

These requests and points make sense.
We’re more than happy to refine this proposal.

I’ll begin to prepare an edit to this existing Topic.

Thank you

[ryanwould]

Hi @gabrielbarros,

Since our last communication, we have been dedicated to extensive research and continuous refinement of our technology platform to ensure it effectively addresses our core objective: enabling verifiable transparency in off-chain processes.

As outlined in our proposal, our approach involves harnessing replicated verifiable compute, particularly Cartesi, to manage trustless data transformation and validation. While we have successfully executed workloads through the Cartesi Machine and explored avenues to integrate ‘off-chain’ data within it, thereby enabling potential oracle mechanisms within Cartesi Machine workloads, we have also been actively researching privacy-preserving solutions. This research explores zero-knowledge technology as an alternative to Cartesi.

As of now, we are still in the research phase, carefully evaluating the best path forward. Our ongoing efforts have yielded some promising developments that could contribute positively to the Cartesi ecosystem, though they remain at a preliminary stage.

Additionally, we are keeping all ideas on standby, whether they involve utilizing the complete Cartesi Rollups tech stack or exploring other privacy-preserving methods.

In essence, this update serves to reaffirm our commitment to working with Cartesi. We are actively engaged in research and decision-making to determine the most suitable direction. Should there be any fundamental changes to our proposal for building with Cartesi, we will ensure timely communication.

We look forward to providing another update later in this quarter. Thank you for your collaboration and support!