High-Level Requirements

As detailed in the DA ledger model, the Digital Asset Platform provides parties with a virtual shared ledger, which contains their interaction history and the current state of their shared DAML contracts. To access the ledger, the parties must deploy (or have someone deploy for them) the so-called participant nodes. The participant nodes then expose the Ledger API, which enables the parties to request changes and get notified about the changes to the ledger. To apply the changes, the participant nodes run a synchronization protocol. We can visualize the setup as follows.

../../_images/da-ledger-architecture.svg

In general, the setup might be more complicated than shown above, as a single participant node can provide services for more than one party. Furthermore, while a party must currently use exactly one participant node, parties will be able to use more than one participant node in future versions.

In this section, we list the high-level functional requirements on the Ledger API, as well as non-functional requirements on the synchronization protocol.

Functional requirements

Functional requirements specify the constraints on and between the system’s observable outputs and inputs. A difficulty in specifying the requirements for the synchronization service is that the system and its inputs and outputs are distributed, and that the system can include Byzantine participant nodes, i.e., participants that are malicious, malfunctioning or compromised. The system does not have to give any guarantees to parties using such nodes, beyond the ability to recover from malfunction/compromise. However, the system must protect the honestly represented parties (i.e., parties all of whose participant nodes implement the synchronization service correctly) from malicious behavior. To account for this in our requirements, we exploit the fact that the conceptual purpose of the ledger synchronization service is to provide parties with a virtual shared ledger and we:

  1. use such a shared ledger and the associated properties (described in the DA ledger model) to constrain the input-output relation;
  2. express all requirements from the perspective of an honestly represented party;
  3. use the same shared ledger for all parties and requirements, guaranteeing synchronization.

We express the high-level functional requirements as user stories, always from the perspective of an honestly represented party, i.e., Ledger API user, and thus omit the role. As the observable inputs and outputs, we take the Ledger API inputs and outputs. Additionally, we assume that crashes and recoveries of participant nodes are observable. The requirements ensure that the virtual shared ledger describes a world that is compatible with the honestly represented parties’ perspectives, but it may deviate in any respect from what Byzantine nodes present to their parties. We call such parties dishonestly represented parties.

Some requirements have explicit exceptions and design limitations. Exceptions are fundamental, and cannot be improved on by further design iterations. Design limitations refer to the design of the Canton synchronization protocol and can be improved in future versions. We discuss the consequences of the most important exceptions and design limitations later in the section.

Note

The fulfillment of these requirements is conditional on the system’s assumptions (in particular, any trusted participants must behave correctly).

  • Synchronization. I want the platform to provide a virtual ledger (according to the DA ledger model) that is shared with all other parties in the system (honestly represented or not), so that I stay synchronized with my counterparties.

  • Change requests possible. I want to be able to submit change requests to the shared ledger.

  • Change request identification. I want to be able to assign an identifier to all my change requests.

  • Change request deduplication. I want the system to deduplicate my change requests with the same identifiers, if they are submitted within a time window configurable per participant, so that my applications can resend change requests in case of a restart without adding the changes to the ledger twice.

    Design limitation: currently, the deduplication is done based not solely on the identifier, but on the combination of the identifier, the requested change, and the ledger effective time. Furthermore, the deduplication time window is not configurable per participant, but is fixed in the scope of a synchronization domain.

  • Bounded decision time. I want to be able to learn within some bounded time from the submission (on the order of minutes) the decision about my change request, i.e., whether it was added to the ledger or not.

    Design limitation: If the participant node used for the submission crashes, the bound can be exceeded. This can be improved in future versions by employing multiple participant nodes.

  • Transparency. I want to get notified in a timely fashion (on the order of seconds) about the changes to my projection of the shared ledger, according to the DA ledger model, so that I stay synchronized with my counterparties.

    Design limitation: If my participant node crashes, the bound can be exceeded. This can be improved in future versions by employing multiple participant nodes.

    Design limitation: The transparency requirement can be violated if the submitter node is Byzantine. In particular, it can happen that I learn about the existence of these actions, but not about their contents (including the contracts used).

  • Integrity: ledger validity. I want the shared ledger to be valid according to the DA ledger model.

    Exception: The consistency aspect of the validity requirement on the shared ledger can be violated for contracts with no honestly represented signatories, even if I am an observer on the contract.

  • Integrity: request authenticity. I want the shared ledger to contain a record of a change with me as one of the requesters if and only if:

    1. I actually requested that exact change, i.e., I submitted the change via the command submission service, and
    2. I am notified that my change request was added to the shared ledger, unless my participant node crashes forever,

    so that, together with the ledger validity requirement, I can be sure that the ledger contains no records of:

    1. obligations imposed on me,
    2. rights taken away from me, and
    3. my counterparties removing their existing obligations

    without my explicit consent. In particular, I am the only requester of any such change. Note that this requirement implies that the change is done atomically, i.e. either it is added in its entirety, or not at all.

    Remark: As functional requirements apply only to honestly represented parties, any dishonestly represented party can be a requester of a commit on the virtual shared ledger, even if it has never submitted a command via the command submission service. However, this is possible only if no requester of the commit is honestly represented.

    Note

    The two integrity requirements come with further limitations and trust assumptions, whenever the trust-liveness trade-offs below are used.

  • Non-repudiation. I want the service to provide me with irrefutable evidence of all ledger changes that I get notified about, so that I can prove to a third party (e.g., a court) that a contract of which I am a stakeholder was created or archived at a certain point in time.

  • Finality. I want the shared ledger to be append-only, so that, once I am notified about a change to the ledger, that change cannot be removed from the ledger.

  • DAML package uploads. I want to be able to upload a new DAML package to my participant node, so that I can start using new DAML contract templates or upgraded versions of existing ones. The authority to upload packages can be limited to particular parties (e.g., a participant administrator party), or done through a separate API.

  • DAML package notification. I want to be able to get notified about new packages distributed to me by other parties in the system, so that I can inspect the contents of the package, either automatically or manually.

  • Automatic DAML package distribution. I want the system to notify my counterparties about my uploaded DAML packages the first time that I submit a change request that includes a contract that both comes from this new package and has the counterparty as a stakeholder on it.

  • DAML package vetting. I want to be able to explicitly approve (manually or automatically, e.g., based on a signature by a trusted party) every new package sent to me by another party, so that the participant node does not execute any code that has not been approved. The authority to vet packages can be limited to particular parties, or done through a separate API.

    Exception: I cannot approve a package without approving all of its dependencies first.

  • No unnecessary rejections. I want the system to add all my well-authorized and DAML-conformant change requests to the ledger, unless:

    1. they are duplicated, or
    2. they use DAML templates my counterparties’ participants have not vetted, or
    3. they conflict with other changes that are included in the shared ledger prior to or at approximately the same time as my request, or
    4. the processing capacity of my participant node or the participant nodes of my counterparties is exhausted by other change requests submitted by myself or others roughly simultaneously,

    in which case I want the decision to include the appropriate reason for rejection.

    Exception 1: This requirement may be violated whenever my participant node crashes, or if there is contention in the system (multiple conflicting requests are issued in a short period of time). The rejection reason reported in the decision in the exceptional case must differ from those reported because of other causes listed in this requirement.

    Exception 2: If my change request contains an exercise on a contract identifier, and I have not witnessed (e.g., through divulgence) any actions on a contract with this identifier in my projection of the shared ledger (according to the DA ledger model), then my change request may fail.

    Design limitation 1: My change requests can also be rejected if a participant of some counterparty (hosting a signatory or an observer) in my change request is crashed, unless some trusted participant (e.g., one run by a market operator) is a stakeholder participant on all contracts in my change request.

    Design limitation 2: My change requests can also be rejected if any of my counterparties in the change request is Byzantine, unless some trusted participant (e.g., one run by a market operator) is a stakeholder participant on all contracts in my change request.

    Design limitation 3: If the underlying sequencer queue is full for a participant, then we can get an unnecessary rejection. We assume however that the queue size is so large that it can be considered to be infinite, so this unnecessary rejection doesn’t happen in practice, and the situation would be resolved operationally before the queue fills up.

  • Seek support for notifications. I want to be able to receive notifications (about ledger changes and about the decisions on my change requests) only from a particular known offset, so that I can synchronize my application state with the set of active contracts on the shared ledger after crashes and other events, without having to read all historical changes.

    Exception: A participant can define a bound on how far in the past the seek can be requested.

  • Active contract snapshots. I want the system to provide me a way to obtain a recent (on the order of seconds) snapshot of the set of active contracts on the shared ledger, so that I can initialize my application state and synchronize with the set of active contracts on the ledger efficiently.

  • Change request processing limited to participant nods. I want only the following (and no other) functionality related to change request processing:

    1. submitting change requests
    2. receiving information about change request processing and results
    3. (possibly) vetting DAML packages

    to be exposed on the Ledger API, so that the unavailability of my or my counterparties’ applications cannot influence whether a change I previously requested through the API is included in the shared ledger, except if the request is using packages not previously vetted. Note that this inclusion may still be influenced by the availability of my counterparties’ participant nodes (as specified in the limitations on the requirement on no unnecessary rejections)

Resource limits

This section specifies upper bounds on the sizes of data structures. The system must be able to process data structures within the given size limits.

If a data structure exceeds a limit, the system must reject transactions containing the data structure. Note that it would be impossible to check violations of resource limits at compile time; therefore the DAML compiler will not emit an error or warning if a resource limit is violated.

Maximum transaction depth: 100

Definition: The maximum number of levels (except for the top-level) in a transaction tree.

Example: The following transaction has a depth of 2:

../../_images/action-structure-paint-offer.svg

Purpose: This limit is to mitigate the higher cost of implementing stack-safe algorithms on transaction trees. The limit may be relaxed in future versions.

Maximum depth of a DAML value: 100

Definition: The maximum numbers of nestings in a DAML value.

Example:

  • The value “17” has a depth of 0.
  • The value “{myField: 17}” has a depth of 1.
  • The value “[{myField: 17}]” has a depth of 2.
  • The value “[‘observer1’, ‘observer2’, …, ‘observer100’]” has a depth of 1.

Purpose:

  1. Applications interfacing the DA ledger likely have to process DAML values and likely are developed outside of DA. By limiting the depth of DAML values, application developers have to be less concerned about stack usage of their applications. So the limit effectively facilitates the development of applications.
  2. This limit allows for a readable wire format for DAML-LF values, as it is not necessary to flatten values before transmission.

Non-functional requirements

These requirements specify the characteristics of the internal system operation. In addition to the participant nodes, the implementation of the synchronization protocol may involve a set of additional operational entities. For example, this set can include a sequencer. We call a single deployment of such a set of operational entities a domain, and refer to the entities as domain entities.

As before, the requirements are expressed as user stories, with the user always being the Ledger API user. Additionally, we list specific requirements for financial market infrastructure providers. Some requirements have explicit exceptions; we discuss the consequences of these exceptions later in the section.

  • Privacy. I want the visibility of the ledger contents to be restricted according to the privacy model of DA ledgers, so that the information about any (sub)action on the ledger is provided only to participant nodes of parties privy to this action. In particular, other participant nodes must not receive any information about the action, not even in an encrypted form.

    Exception: domain entities operated by trusted third parties (such as market operators) may receive encrypted versions of any of the ledger data (but not plain text).

    Design limitation 1: Participant nodes of parties privy to an action (according to the ledger privacy model) may learn the following:

    • How deeply lies the action within a ledger commit.
    • How many sibling actions each parent action has.
    • The transaction identifiers (but not the transactions’ contents) that have created the contracts used by the action.

    Design limitation 2: Domain entities operated by trusted third parties may learn the hierarchical structure and stakeholders of all actions of the ledger (but none of the contents of the contracts, such as templates used or their arguments).

  • Transaction stream auditability. I want the system to be able to convince a third party (e.g., an auditor) that they have been presented with my complete transaction stream within a configurable time period (on the order of years), so that they can be sure that the stream represents a complete record of my ledger projection, with no omissions or additions.

    Exception: The evidence can be linear in the size of my transaction stream.

    Design limitation: The evidence need not be privacy-preserving with respect to other parties with whom I share participant nodes, and the process can be manual.

  • Service Auditability. I want the synchronization protocol implementation to store all requests and responses of all participant nodes within a configurable time period (on the order of years), so that an independent third party can manually audit the correct behavior of any individual participant and ensure that all requests and responses it sent comply with the protocol.

  • Compliance. I want the system to be compliant with international regulations.

  • Configurable trust-liveness trade-off. I want each domain to allow me to choose from a predefined (by the domain) set of trade-offs between trust and liveness for my change requests, so that my change requests get included in the ledger even if some of my participant nodes of my counterparties are offline or Byzantine, at the expense of making additional trust assumptions: on (1) the domain entities (for privacy and integrity), and/or (2) participant nodes run by counterparties in my change request that are marked as “VIP” by the domain (for integrity), and/or (3) participant nodes run by other counterparties in my change request (also for integrity).

    Exception: If the honest and online participants do not have sufficient information about the activeness of the contracts used by my change request, the request can still be rejected.

    Design limitation: The only trade-off allowed by the current design is through confirmation policies. Currently, the only fully supported policies are the full, signatory, and VIP confirmation policies. The implementation does not support the serialization of other policies. Furthermore, integrity need not hold under other policies. This corresponds to allowing only the trade-off (2) above (making additional trust assumptions on VIP participants). In this case, the VIP participants must be trusted.

    Note: If a participant is trusted, then the trust assumption extends to all parties hosted by the participant. Conversely, the system does not support to trust a participant for the actions performed on behalf of one party and distrust the same participant for the actions performed on behalf of a different party.

  • Workflow isolation. I want the system to be built such that workflows (groups of change requests serving a particular business purpose) that are independent, i.e. do not conflict with other, do not affect each other’s performance.

  • Garbage collection. I want the system to provide garbage collection capabilities, so that the required hot storage capacity for each participant node depends only on:

    1. the size of currently active contracts whose processing the node is involved in,
    2. node’s the past traffic volume within a (per-participant) configurable time window

    and does not otherwise grow unboundedly as the system continues operating. Cold storage requirements are allowed to keep growing continuously with system operation, for auditability purposes.

  • Multi-domain participant nodes. I want to be able to use multiple domains simultaneously from the same participant node.

  • Internal participant node domain. I want to be able to use an internal domain for workflows involving only local parties exclusively hosted by the participant node.

  • Connecting to domains. I want to be able to connect my participant node to a new domain at any point in time, as long as I am accepted by the domain operators.

  • Workflow transfer. I want to be able to transfer the processing of any DAML contract that I am a stakeholder of or have delegation rights on, from one domain to another domain that has been vetted as appropriate by all contract stakeholders through some procedure defined by the synchronization service, so that I can use domains with better performance, do load balancing and disaster recovery.

  • Workflow composability. I want to be able to atomically execute steps (DAML actions) in different workflows across different domains, as long as there exists a single domain to which all participants in all workflows are connected.

  • Standards compliant cryptography. I want the system to be built using configurable cryptographic primitives approved by standardization bodies such as NIST, so that I can rely on existing audits and hardware security module support for all the primitives.

  • Upgradability. I want to be able to upgrade system components, both individually and jointly, so that I can deploy fixes and improvements to the components and the protocol without stopping the system’s operation.

  • Semantic versioning. I want all interfaces, protocols and persistent data schemas to be versioned, so that version mismatches are prevented. The versioning scheme must be semantic, so that breaking changes always bump the major versions.

  • Backwards and forward protocol compatibility within a major version. I want system components supporting the same major version of the protocol to be able to communicate seamlessly.

  • Domain approved protocol versions. I want domains to specify the allowed set of protocol versions on the domain, so that old versions of the protocol can be decommissioned, and that new versions can be introduced and rolled back if operational problems are discovered.

    Design limitation: Initially, the domain can specify only a single protocol version as allowed, which can change over time.

  • Cross-version backward and forward protocol compatibility. I want new versions of system components to still support at least one previous major version of the synchronization protocol, so that entities capable of using newer versions of the protocol can still use domains that specify only old versions as allowed. Note that the requirement does not apply to completely different synchronization protocols (e.g., DAML on SQL and Canton).

  • Testability of participant node upgrades on historical data. I want to be able to test new versions of participant nodes against historical data from a time window and compare the results to those obtained from old versions, so that I can increase my certainty that the new version does not introduce unintended differences in behavior.

  • Seamless participant failover. I want the applications using the ledger API to seamlessly fail over to my other participant nodes, once one of my nodes crashes.

  • Seamless failover for domain entities. I want the implementation of all domain entities to include seamless failover capabilities, so that the system can continue operating uninterruptedly on the failure of an instance of a domain entity.

  • Backups. I want to be able to periodically backup the system state (ledger databases) so that it can be subsequently restored if required for disaster recovery purposes.

  • Site-wide disaster recovery. I want the system to be built with the ability to recover from a failure of an entire data center by moving the operations to a different data center, without loss of data.

  • Participant compromise recovery. I want to have a procedure in place that can be followed to recover from a malfunctioning or a compromised participant node, so that when the procedure is finished I obtain the same guarantees (in particular, integrity and transparency) as the honest participants on the part of the shared ledger created after the end of the recovery procedure.

  • Domain entity compromise recovery. I want to have a procedure in place that can be followed to recover a compromised domain entity, so that the system guarantees can be restored after the procedure is complete.

  • Fundamental dispute resolution. I want to have a procedure in place that allows me to limit and resolve the damage to the ledger state in the case of a fundamental dispute on the outcome of a transaction that was added to the virtual shared ledger, so that I can reconcile the set of active contracts with my counterparties in case of any disagreement over this set. Example causes of disagreement include disagreement with the state found after recovering a compromised participant, or disagreement due to a change in the regulatory environment making some existing contracts illegal.

  • Distributed recovery of participant data. I want to be able to reconstruct which of my contracts are currently active from the information that the participants of my counterparties store, so that I can recover my data in case of a catastrophic event. This assumes that the other participants are cooperating and have not suffered catastrophic failures themselves.

  • Adding parties to participants. I want to be able to start using the DA system at any point in time, by choosing to use a new or an already existing participant node.

  • Identity provider integration. I want the synchronization protocol to integrate with an identity provider service, so that I can use this service to manage the party-to-participant and participant-to-cryptographic-keys mappings.

  • Identity information updates. I want the synchronization protocol to track updates by the identity provider service, so that the parties can switch participants, and participants can roll and/or revoke keys, while ensuring continuous system operation.

  • Party migration. I want to be able to switch from using one participant node to using another participant node, without losing the data about the set of active contracts on the shared ledger that I am a stakeholder of. The new participant node need not provide me with the ledger changes prior to migration.

  • Parties using multiple participants. I want to be able to use the system through multiple participant nodes, so that I can do load balancing, and continue using the system even if one of my participant nodes crashes.

  • Read-only participants. I want to be able to configure some participants as read-only, so that I can provide a live stream of the changes to my ledger view to an auditor, without giving them the ability to submit change requests.

  • Reuse of off-the-shelf solutions. I want the system to rely on industry-standard abstractions for:

    1. messaging
    2. persistent storage
    3. identity providers
    4. metrics
    5. logging
    6. monitoring

    so that I can use off-the-shelf solutions for these purposes.

  • Metrics on communication. I want the system to provide metrics on the state of all communication links in the system, and make them available on both link endpoints.

  • Metrics on processing. I want the system to provide metrics for every major processing phase within the system.

  • Component health monitoring. I want the system to provide monitoring information for every system component, so that I am alerted when a component fails.

  • Remote debugability. I want the system to capture sufficient information such that I can debug remotely and post-mortem any issue in environments that are not within my control (OP).

  • Horizontal scalability. I want the system to be able to horizontally scale all parallelizible parts of the system, by adding processing units for these parts.

  • Large transaction support. I want the system to support large transactions such that I can guarantee atomicity of large scale workflows.

  • Resilience to erroneous behavior. I want that the system is thoroughly tested to be resilient against erroneous behavior of users and participants such that I can entrust the system to handle my business.

  • Resilience to faulty domain behavior. I want that the system is thoroughly tested to be able to detect and recover from faulty behaviour of domain components, such that occassional issues don’t break the system permanently.

Requirement Exceptions: Notes

In this section, we explain the consequences of the exceptions to the requirements.

Ledger consistency

The validity requirement on the ledger made an exception for the consistency of contracts without honestly represented signatories. We explain the exception using the paint offer example from the ledger model. Recall that the example assumed contracts of the form PaintOffer houseOwner painter obligor with the painter as the signatory, and the houseOwner as an observer (while the obligor is not a stakeholder). Additionally, assume that we extend the model with an action that allows the painter to rescind the offer. The resulting model is then:

../../_images/consistency-exception-model.svg

Assume that Alice (A) is the house owner, P the painter, and that the painter is dishonestly represented, in that he employs a malicious participant, while Alice is honestly represented. Then, the following shared ledgers are allowed, together with their projections for A, which in this case are just the list of transactions in the shared ledger.

../../_images/consistency-exception-ledger.svg
../../_images/consistency-exception-no-create.svg

That is, the dishonestly represented painter can rescind the offer twice in the shared ledger, even though the offer is not active any more by the time it is rescinded (and thus consumed) for the second time, violating the consistency criterion. Similarly, the dishonestly represented painter can rescind an offer that was never created in the first place.

However, this exception is not a problem for the stated benefits of the integrity requirement, as the resulting ledgers still ensure that honestly represented parties cannot have obligations imposed on them or rights taken away from them, and that their counterparties cannot escape their existing obligations. For instance, the example of a malicious Alice double spending her IOU:

../../_images/double-spend-with-stakeholders.svg

is still disallowed even under the exception, as long as the bank is honestly represented. If the bank was dishonestly represented, then the double spend would be possible. But the bank would not gain anything by this dishonest behavior – it would just incur more obligations.

No unnecessary rejections

This requirement made exceptions for (1) contention, and included a design limitation for (2) crashes/Byzantine behavior of participant nodes. Contention is a fundamental limitation, given the requirement for a bounded decision time. Consider a sequence \(cr_1, \ldots cr_n\) of change requests, each of which conflicts with the previous one, but otherwise have no conflicts, except for maybe \(cr_1\). Then all the odd-numbered requests should get added to the ledger exactly when \(cr_1\) is added, and the even-numbered ones exactly when \(cr_1\) is rejected. Since detecting conflicts and other forms of processing (e.g. communication, DAML interpretation) incur processing delays, deciding precisely whether \(cr_n\) gets added to the ledger takes time proportional to \(n\). By lengthening the sequence of requests, we eventually exceed any fixed bound within which we must decide on \(cr_n\).

Crashes and Byzantine behavior can inhibit liveness. To cope, the so-called VIP confirmation policy allows any trusted participant to add change requests to the ledger without the involvement of other parties. This policy can be used in settings where there is a central trusted party. Today’s financial markets are an example of such a setting.

The no-rejection guarantees can be further improved by constructing DAML models that ensure that the submitter is a stakeholder on all contracts in a transaction. That way, rejects due to Byzantine behavior of other participants can be detected by the submitter. Furthermore, if necessary, the synchronization service itself could be changed to improve its properties in a future version, by including so-called bounded timeout extensions and attestators.

Privacy

Consider a transaction where Alice buys some shares from Bob (a delivery-versus-payment transaction). The shares are registered at the share registry SR, and Alice is paying with an IOU issued to her by a bank. We depict the transaction in the first image below. Next, we show the bank’s projection of this transaction, according to the DA ledger model. Below, we demonstrate what the bank’s view obtained through the ledger synchronization protocol may look like. The bank sees that the transfer happens as a direct consequence of another action that has an additional consequence. However, the bank learns nothing else about the parent action or this other consequence. It does not learn that the parent action was on a DvP contract, that the other consequence is a transfer of shares, and that this consequence has further consequences. It learns neither the number nor the identities of the parties involved in any part of the transaction other than the IOU transfer. This illustrates the first design limitation for the privacy requirement.

At the bottom, we see that the domain entities run by a trusted third party can learn the complete structure of the transaction and the stakeholders of all actions in the transaction (second design limitations). Lastly, they also see some data about the contracts on which the actions are performed, but this data is visible only in an encrypted form. The decryption keys are never shared with the domain entities.

../../_images/privacy-leaks-dvp.svg