This issue is intended to act as a running high-level outline and provide motivation for overall planned long-term refactors and implementation of a primitives module as time permits.
Currently, the wire module essentially acts as the fundamental building block of some data structures and other definitions for other concepts that are key to consensus. For example, block headers, transactions, blocks, and sequence lock definitions. While this has worked well over the years and is reasonable given the current architecture, it does result in the conflation of two concepts that really are distinct. Namely, that the specific details regarding the way the data structures are serialized and used in consensus is fundamentally entirely distinct from the data sharing protocol even though they are currently highly coincident.
Further, it also has led to several other fundamental consensus-related concepts such as median time sources, determination of transaction finalization, and lock times needing to be defined elsewhere since those are clearly not related to the wire protocol.
Thus, my plan is to implement a new primitives module which will become the new fundamental building block of consensus code with the following motivations and goals:
- Make the distinction between consensus primitives and data sharing much more apparent
- Consolidate some of the aforementioned related concepts
- Open the door for a wide variety of improvements at the network protocol level
- Take advantage of the major refactor to redefine the data structures in a way that significantly reduces allocations and improves cache locality
- Keep dependencies to a minimum and specifically avoid the need for deps such as leveldb
- Provide a single module for the consensus-related functionality needed by consumers that is currently spread across
wire, blockchain, blockchain/standalone, and dcrutil
- For example, block and transaction definitions, proof of work checks, and merkle root calculations
- Compute and store hashes at initialization time for lock-free access
Given that almost everything currently builds on wire either directly or indirectly, this is going to be a fairly intensive and long-running effort. Moreover, many of the individual changes will very likely look out of place and/or incomplete in isolation, so my hope is that this issue along with a high-level outline will provide insight into the end goal.
Note that some of these will very likely need to be updated as the work proceeds since unexpected things invariably come up during large development efforts such as this. They are also not in a strict order.
This issue is intended to act as a running high-level outline and provide motivation for overall planned long-term refactors and implementation of a
primitivesmodule as time permits.Currently, the
wiremodule essentially acts as the fundamental building block of some data structures and other definitions for other concepts that are key to consensus. For example, block headers, transactions, blocks, and sequence lock definitions. While this has worked well over the years and is reasonable given the current architecture, it does result in the conflation of two concepts that really are distinct. Namely, that the specific details regarding the way the data structures are serialized and used in consensus is fundamentally entirely distinct from the data sharing protocol even though they are currently highly coincident.Further, it also has led to several other fundamental consensus-related concepts such as median time sources, determination of transaction finalization, and lock times needing to be defined elsewhere since those are clearly not related to the wire protocol.
Thus, my plan is to implement a new
primitivesmodule which will become the new fundamental building block of consensus code with the following motivations and goals:wire,blockchain,blockchain/standalone, anddcrutilGiven that almost everything currently builds on
wireeither directly or indirectly, this is going to be a fairly intensive and long-running effort. Moreover, many of the individual changes will very likely look out of place and/or incomplete in isolation, so my hope is that this issue along with a high-level outline will provide insight into the end goal.Note that some of these will very likely need to be updated as the work proceeds since unexpected things invariably come up during large development efforts such as this. They are also not in a strict order.
uint256subpackage for all fixed-precision unsigned 256-bit integer needs -- Implemented by PR uint256: Introduce package. #2787blockchain/standaloneand update to use the newuint256instead of standard library big integers -- Implemented by PR primitives: Add initial proof-of-work funcs. #2788blockchain/standalone-- Implemented by PR primitives: Add inclusion proof funcs. #2827blockchain/standalone-- Implemented by PR primitives: Add subsidy calcs. #2920wire)Txincluding things such as long-term stable serialization and calculation of both witness and full hashes used in consensusBlockHeaderincluding things such as long-term stable serialization and calculation of the block hash as used in consensusBlockincluding things such as long-term stable serialization and pass through hash calculationblockchain/standalone-- Partially implemented by primitives: Add core merkle tree root calcs. #2826blockchain/standaloneblockchain/standaloneblockchain/standalonein favor of the newprimitivesmodulewireto make use of the newprimitivestypes and remove all consensus-specific definitionsMedianTimeSourcefromblockchainblockchainblockchaincode used by external consumers related to transaction checkingblockchainan internal packagetxscriptcurrently which is not desirable for the new module