RFP-01

Archival Nodes APIs Infrastructure

Status: Open

Category: Infrastructure

Date Created: 03/06/2024

Project Overview

Archival nodes are full nodes responsible for maintaining a comprehensive record of all transactions, blocks, and events ever recorded on the blockchain, from the genesis block. Archival nodes are a critical piece of infrastructure because they provide access to extensive historical data. Applications can query these nodes for detailed historical information, enabling complex analyses and queries.

In Dusk, archival nodes are available under a feature flag. This means that when setting up a node on Dusk, users can configure their node as an archival node. This configuration allows the node to record and store all events witnessed on the Dusk blockchain in a database.

Unlike other types of nodes, archival Nodes in Dusk are not involved in the consensus mechanism as they do not participate in block generation. Additionally, they do not perform any zero-knowledge computations or create proofs.

Instead, archival nodes listen to all events emitted in Dusk and store them in a database, enabling efficient indexing and retrieval.

These events include:

Events emitted on the Dusk blockchain (e.g. transactions)
RUES events (e.g. events emitted by a contract).

Archival nodes serve as a reliable source for historical data, as they are the only type of nodes that not only listen and broadcast events, but also store them.

Applications can also verify the legitimacy of stored events by hashing them and comparing the results against a block hash, ensuring data integrity and accuracy.

There is currently no simple way to access the data stored by archival nodes. The scope for this RFP is to leverage the existing APIs and expand on them to make data easily consumable. The project should provide comprehensive APIs that improve the accessibility and usability of the data stored in Dusk archival nodes, enabling clients to perform detailed and efficient queries.

Scope of Work

Since Dusk’s archival nodes already incorporate the logic for storing events, this RFP requests leveraging that existing logic by adding extensive API infrastructure. The solution should be compatible with the different networks provided by Dusk (e.g., mainnet, testnet, devnet…), and comprehensive documentation is expected. Dusk invites grant applicants to submit proposals providing the following elements:

Extensive APIs

Expand on and develop comprehensive APIs, to enable clients to access events stored by archival nodes. This should include both GraphQL and RUES entrypoints, providing flexible querying options for specific events, transactions, and contracts’ state.

To provide precise data retrieval, the GraphQL schema needs to allow queries by relevant attributes (e.g. event type, timestamp, emitter .. ). More specifically, the APIs should enable querying of all historical data stored by archival nodes at any block height, including both events recorded on the blockchain and events emitted by smart contracts. These APIs need to allow the GraphQL server to act as an intermediary between clients and the archival nodes’ database, so that when a query is received, the server processes it, fetches the necessary data from the archival nodes and serves it.

To facilitate these APIs it is necessary to store additional data in local storage. An example of this would be including a get_failed_txn_count function in gql, which may require adding a counter in RocksDB/SQLite to track this metric and support the API.

Failover Support and Load Balancing

Failover support procedures and load balancing network requests should be implemented, to ensure that the least overloaded archival nodes handle data provision. This can be achieved by an API gateway between the client and the archival Nodes (servers).

An API rate limiter middleware should be integrated into the current HTTP server to mitigate DoS attacks.

Real-Time Updates

Real-time data access should be available through GraphQL subscriptions, allowing clients to receive immediate notifications of new events as they are recorded by the archival nodes. These notifications should be delivered via RUES by using existing websockets.

Event Recording and Storage Limits

Archival nodes should have the ability to define limits on the types or volumes of events to be stored, to manage storage requirements and avoid the necessity of storing all events since the genesis block.

On-disk data compression supported by RocksDB, such as Snappy and LZ4, should be enabled.

Envisioned Timelines

8 weeks

Proposal Submission

Please submit applications through the Thesan grants form

Resources

Relevant repositories can be found here:

Additional Information

Vendor Qualifications

Dusk seeks applicants with the necessary experience and expertise to ensure successful implementation of the proposed solutions.

For this reason, Dusk will mainly consider applicants that have provable experience / knowledge in developing and deploying GraphQL and REST APIs in the context of blockchain technology. Applicants should rely on strong programming skills and expertise in database management. Competence in implementing failover support and load balancing mechanisms is a plus.

Evaluation Criteria

We evaluate all applications in a fair and unbiased manner. All proposals will be considered in the light of standard factors including, but not limited to:

Project approach
Cost
Timeline
Technical expertise of the individual/team
Reputation of the individual/team

Confidentiality

All proposals will be treated as confidential.

Open Source

Code should be provided with the necessary licenses for open-source use (e.g. Apache 2.0, MPL, MIT).

How to apply

Fill out the Thesan application form by taking your your time to answer all the relevant questions in detail. Please make sure to follow the general and requirements and guidelines .