From f0e9e7e249db2f7ee2aa310abd1461ba72e7d269 Mon Sep 17 00:00:00 2001 From: D050513 Date: Tue, 3 Mar 2026 22:17:31 +0100 Subject: [PATCH 01/15] first draft of new event queues docs --- guides/events/event-queues-new.md | 651 ++++++++++++++++++++++++++++++ 1 file changed, 651 insertions(+) create mode 100644 guides/events/event-queues-new.md diff --git a/guides/events/event-queues-new.md b/guides/events/event-queues-new.md new file mode 100644 index 0000000000..a572f00776 --- /dev/null +++ b/guides/events/event-queues-new.md @@ -0,0 +1,651 @@ +--- +synopsis: > + Transactional Event Queues allow you to schedule events and background tasks for asynchronous, exactly-once processing with ultimate resilience. +status: released +--- + +# Transactional Event Queues + +{{ $frontmatter.synopsis }} + +[[toc]] + + + +## Motivation + +In distributed systems, things fail. A remote service may be temporarily unavailable, a network call may time out, or your process may crash right after committing a database transaction but before sending the follow-up message. +These failures can leave your system in an inconsistent state — data is committed, but dependent side effects never happen. + +_Transactional Event Queues_ solve this by persisting events and tasks in a database table **within the same transaction** as your business data. +After the transaction commits, a background process picks up the queued entries and executes them asynchronously — with retries, exactly-once guarantees, and a dead letter queue for unrecoverable failures. + +This pattern is widely known as the _Transactional Outbox_, but CAP's event queues go beyond outbound messages. +They provide a unified mechanism for four use cases: + +- **Outbox** — defer outbound calls to remote services until the transaction succeeds. +- **Inbox** — acknowledge inbound messages immediately and process them asynchronously. +- **Background Tasks** — schedule periodic or delayed tasks such as data replication. +- **Callbacks** — react to completed or failed tasks, enabling SAGA-like compensation patterns. + + + +## How It Works { #concept } + +The core principle is straightforward: + +1. Instead of executing side effects directly, you write an event message into a database table — **within the current transaction**. +2. Once the transaction commits, a task runner reads pending messages and dispatches them to the respective service. +3. If processing succeeds, the message is deleted. If it fails, the system retries with exponentially increasing delays. +4. After a configurable maximum number of attempts, the message is moved to the dead letter queue for manual intervention. + +![This graphic is explained in the accompanying text.](../../releases/2025/assets/may25/TaskController.png){width=80%} + +Because the event message and your business data share the same database transaction, you get two fundamental guarantees: + +- **No phantom events** — if the transaction rolls back, no event is ever sent. +- **No lost events** — if the transaction commits, the event is guaranteed to be processed eventually. + + + +## Use Cases + +### Outbox { #outbox } + +The outbox defers outbound calls to remote services until the main transaction succeeds. +This prevents sending requests to external systems when your transaction might still roll back. + +**Example:** When creating a travel booking, you also need to notify an external flight service. +Without the outbox, the notification could be sent even if the booking transaction fails. + +::: code-group +```js [Node.js] +const xflights = await cds.connect.to('xflights') +const qd_xflights = cds.queued(xflights) + +this.after('CREATE', 'Travels', async (travel) => { + // Persisted within the current transaction, sent after commit + await qd_xflights.send('bookFlight', { travelId: travel.ID }) +}) +``` +```java [Java] +@Autowired @Qualifier("MyCustomOutbox") +OutboxService outbox; + +@Autowired @Qualifier(CqnService.DEFAULT_NAME) +CqnService remoteFlights; + +@After(event = CqnService.EVENT_CREATE, entity = Travels_.CDS_NAME) +void notifyFlights(List travels) { + AsyncCqnService outboxedFlights = AsyncCqnService.of(remoteFlights, outbox); + travels.forEach(t -> outboxedFlights.emit("bookFlight", Map.of("travelId", t.getId()))); +} +``` +::: + +Some services are outboxed automatically, including `cds.MessagingService` and `cds.AuditLogService`. +You don't need to call `cds.queued()` or configure anything extra for these — they use the persistent queue by default. + +[Learn more about auto-outboxed services in Node.js.](../../node.js/queue#per-configuration){.learn-more} +[Learn more about the outbox in Java.](../../java/outbox){.learn-more} + + +### Inbox { #inbox } + +The inbox mirrors the outbox pattern for inbound messages. +When a message arrives from a broker like SAP Event Mesh or Apache Kafka, the messaging service immediately persists it to the database, acknowledges it to the broker, and schedules its processing. + +This brings two advantages: + +- **Quick acknowledgment** — the message broker does not have to wait for your processing to complete. This is especially important with brokers like Kafka that expect fast consumer acknowledgments. +- **Flatten the curve** — if a burst of messages arrives, they're queued in your database and processed at a controlled pace, preventing overload. + +Enable the inbox in your configuration: + +::: code-group +```json [Node.js — package.json] +{ + "cds": { + "requires": { + "messaging": { + "inboxed": true + } + } + } +} +``` +```yaml [Java — application.yaml] +cds: + messaging: + services: + - name: messaging-name + inbox: + enabled: true +``` +::: + +::: warning Inboxing moves the dead letter queue into your app +With the inbox enabled, all messages are acknowledged to the message broker regardless of whether processing succeeds. +Failures must be managed through the [dead letter queue](#dead-letter-queue). +::: + + +### Background Tasks { #background-tasks } + +Event queues are not limited to messaging. You can schedule arbitrary background tasks such as data replication, cache refresh, or garbage collection. + +**Example:** Replicate data from a remote service every 10 minutes. + +::: code-group +```js [Node.js] +const srv = await cds.connect.to('RemoteService') +await srv.schedule('replicate', { entity: 'Products' }).every('10 minutes') +``` +::: + +The `schedule` method is a shortcut for `cds.queued(srv).send()` with additional timing options: + +```js +// Execute once, as soon as possible +await srv.schedule('cleanup', { olderThan: '30d' }) + +// Execute once, after a delay +await srv.schedule('cleanup', { olderThan: '30d' }).after('1h') + +// Execute repeatedly +await srv.schedule('replicate', { entity: 'Products' }).every('10 minutes') +``` + +::: tip Real-world example: data federation +The [data federation guide](../integration/data-federation) uses `srv.schedule().every()` to implement polling-based replication, fetching incremental updates from remote services on a regular interval. +::: + + +### Callbacks (SAGA Patterns) { #callbacks } + +In distributed transactions, you often need to react when an asynchronous step completes or fails. +Event queues support this with `#succeeded` and `#failed` callback events, enabling compensation logic similar to SAGA patterns. + +**Example:** After successfully creating a flight booking via the outbox, replicate the full business object from the remote system. If the booking fails, notify the user. + +::: code-group +```js [Node.js] +const flights = await cds.connect.to('FlightService') + +// Called when the queued booking succeeds +flights.after('bookFlight/#succeeded', async (result, req) => { + console.log('Flight booked successfully:', result) + // Replicate booking details from remote +}) + +// Called when the queued booking fails after max retries +flights.after('bookFlight/#failed', async (error, req) => { + console.log('Flight booking failed:', error) + // Trigger compensation logic +}) +``` +::: + +::: tip Register on specific events +Callback handlers must be registered for the specific `#succeeded` or `#failed` events. +The `*` wildcard handler is not called for these events. +::: + + + +## How to Use { #how-to-use } + +### Queueing a Service { #cds-queued } + +Use `cds.queued(srv)` in Node.js to obtain a queued proxy of any non-database service. +All subsequent dispatches on this proxy are persisted to the event queue and processed asynchronously. + +::: code-group +```js [Node.js] +const srv = await cds.connect.to('RemoteService') +const qsrv = cds.queued(srv) + +// All operations are now queued +await qsrv.emit('someEvent', { key: 'value' }) // fire-and-forget +await qsrv.send('someRequest', { key: 'value' }) // request (result discarded) +await qsrv.run(SELECT.from('Products')) // query (result discarded) +``` +::: + +::: tip `await` is still needed +Even though processing is asynchronous, you still need to `await` because the message is written to the database within the current transaction. +::: + +In Java, use `OutboxService.outboxed(srv)` to wrap any CAP service: + +::: code-group +```java [Java] +OutboxService outbox = runtime.getServiceCatalog() + .getService(OutboxService.class, "MyCustomOutbox"); +CqnService remote = runtime.getServiceCatalog() + .getService(CqnService.class, "RemoteService"); + +// Wrap with outbox handling +AsyncCqnService queued = AsyncCqnService.of(remote, outbox); +queued.emit("someEvent", Map.of("key", "value")); +``` +::: + + +### Unqueueing a Service + +If a service is queued by configuration, you can get back the original (synchronous) service: + +::: code-group +```js [Node.js] +const srv = cds.unqueued(qsrv) // back to synchronous +``` +```java [Java] +CqnService original = outbox.unboxed(outboxedService); +``` +::: + + + +### Service API { #service-api } + +When working with event queues, you interact with the standard CAP service APIs: + +| API | Description | +|-----|-------------| +| `srv.emit(event, data)` | Emit a fire-and-forget event message | +| `srv.send(event, data)` | Send a request (return value discarded for queued services) | +| `srv.run(query)` | Run a CQL query (return value discarded for queued services) | +| `srv.schedule(event, data)` | Shortcut for `cds.queued(srv).send()` with timing options | + +The `schedule` method supports a fluent API: + +```js +await srv.schedule('task', data) // execute asap +await srv.schedule('task', data).after('1h') // execute after one hour +await srv.schedule('task', data).every('1h') // repeat every hour +``` + + +### Queueing by Configuration { #by-configuration } + +You can queue any service through configuration without changing code: + +::: code-group +```json [Node.js — package.json] +{ + "cds": { + "requires": { + "RemoteService": { + "kind": "odata", + "outboxed": true + } + } + } +} +``` +```yaml [Java — application.yaml] +cds: + outbox: + services: + MyCustomOutbox: + maxAttempts: 10 +``` +::: + + +### Auto-Outboxed Services { #auto-outboxed } + +The following services are outboxed by default — you don't need any additional configuration: + +| Service | Description | +|---------|-------------| +| `cds.MessagingService` | All messaging services (Event Mesh, Kafka, etc.) | +| `cds.AuditLogService` | Audit log events | + +This ensures that messaging and audit log events are always sent reliably and never lost due to transaction rollbacks. + +[Learn more about auto-outboxed services in Node.js.](../../node.js/queue#per-configuration){.learn-more} +[Learn more about the outbox in Java.](../../java/outbox#persistent){.learn-more} + + + +## Characteristics + +### Exactly Once { #exactly-once } + +The persistent queue guarantees exactly-once processing for database-related operations. +The system only commits database changes from event processing if the event is successfully processed, and vice versa. + +There is one active message processor per service, tenant, app instance, and message. +This prevents duplicate processing, except in the highly unlikely case of an app crash right after successful processing but before the message could be deleted from the queue. + +### No Phantom Events { #no-phantom-events } + +Because the event message is written within the same database transaction as your business data, a rollback of the transaction also removes the event message. +No event is ever dispatched for a transaction that didn't commit. + +### Guaranteed Order { #guaranteed-order } + +In Node.js, messages are processed in the order they were submitted, per service and tenant. + +In Java, the `DefaultOutboxOrdered` outbox processes entries in submission order. +The `DefaultOutboxUnordered` outbox may process entries in parallel across application instances. + +::: code-group +```yaml [Java — Configuring Order] +cds: + outbox: + services: + DefaultOutboxOrdered: + ordered: true # default + DefaultOutboxUnordered: + ordered: false # default +``` +::: + + +### Error Handling { #errors } + +When processing fails, the system retries the message with exponentially increasing delays. +After a configurable maximum number of attempts (default: 20 in Node.js, 10 in Java), the message is moved to the dead letter queue. + +Some errors are identified as _unrecoverable_ — for example, when a topic is forbidden in SAP Event Mesh. +These messages are immediately moved to the dead letter queue without further retries. + +To mark your own errors as unrecoverable in Node.js: + +```js +const error = new Error('Invalid payload') +error.unrecoverable = true +throw error +``` + +In Java, you can suppress retries by catching the error and calling `context.setCompleted()`: + +```java +@On(service = "", event = "myEvent") +void process(OutboxMessageEventContext context) { + try { + // processing logic + } catch (Exception e) { + if (isSemanticError(e)) { + context.setCompleted(); // remove from queue, no retry + } else { + throw e; // retry + } + } +} +``` + + + +## Dead Letter Queue { #dead-letter-queue } + +Messages that exceed the maximum retry count remain in the `cds.outbox.Messages` database table. +These entries form the _dead letter queue_ and require manual intervention — either to fix the underlying issue and retry, or to discard the message. + +### The Data Model + +Your database model is automatically extended with the following entity: + +```cds +namespace cds.outbox; + +entity Messages { + key ID : UUID; + timestamp : Timestamp; + target : String; + msg : LargeString; + attempts : Integer default 0; + partition : Integer default 0; + lastError : LargeString; + lastAttemptTimestamp: Timestamp @cds.on.update: $now; + status : String(23); +} +``` + + +### Managing Dead Letters + +You can expose a CDS service to manage the dead letter queue with actions to revive or delete entries. + +#### 1. Define the Service + +::: code-group +```cds [srv/outbox-dead-letter-queue-service.cds] +using from '@sap/cds/srv/outbox'; + +@requires: 'internal-user' +service OutboxDeadLetterQueueService { + + @readonly + entity DeadOutboxMessages as projection on cds.outbox.Messages + actions { + action revive(); + action delete(); + }; + +} +``` +::: + +::: warning Restrict access +The dead letter queue contains sensitive data. Ensure the service is accessible only to internal users. +::: + +#### 2. Filter for Dead Entries + +As `maxAttempts` is configurable, its value cannot be added as a static filter to the projection, but must be applied programmatically. + +::: code-group +```js [Node.js — srv/outbox-dead-letter-queue-service.js] +const cds = require('@sap/cds') + +module.exports = class OutboxDeadLetterQueueService extends cds.ApplicationService { + async init() { + this.before('READ', 'DeadOutboxMessages', function (req) { + const { maxAttempts } = cds.env.requires.outbox + req.query.where('attempts >= ', maxAttempts) + }) + await super.init() + } +} +``` +```java [Java — DeadOutboxMessagesHandler.java] +@Component +@ServiceName(OutboxDeadLetterQueueService_.CDS_NAME) +public class DeadOutboxMessagesHandler implements EventHandler { + + private final PersistenceService db; + + public DeadOutboxMessagesHandler( + @Qualifier(PersistenceService.DEFAULT_NAME) PersistenceService db) { + this.db = db; + } + + @Before(entity = DeadOutboxMessages_.CDS_NAME) + public void addDeadEntryFilter(CdsReadEventContext context) { + // Filter for entries that exceeded maxAttempts + Optional outboxFilters = createOutboxFilters(context.getCdsRuntime()); + outboxFilters.ifPresent(filter -> { + CqnSelect modified = copy(context.getCqn(), new Modifier() { + @Override + public CqnPredicate where(Predicate where) { + return filter.and(where); + } + }); + context.setCqn(modified); + }); + } +} +``` +::: + +#### 3. Implement Bound Actions + +Entries in the dead letter queue can be _revived_ by resetting the retry counter to zero, or _deleted_ permanently. + +::: code-group +```js [Node.js — srv/outbox-dead-letter-queue-service.js] +this.on('revive', 'DeadOutboxMessages', async function (req) { + await UPDATE(req.subject).set({ attempts: 0 }) +}) + +this.on('delete', 'DeadOutboxMessages', async function (req) { + await DELETE.from(req.subject) +}) +``` +```java [Java] +@On +public void reviveOutboxMessage(DeadOutboxMessagesReviveContext context) { + CqnAnalyzer analyzer = CqnAnalyzer.create(context.getModel()); + Map key = analyzer.analyze(context.getCqn()).rootKeys(); + Messages msg = Messages.create((String) key.get(Messages.ID)); + msg.setAttempts(0); + db.run(Update.entity(Messages_.class).entry(key).data(msg)); + context.setCompleted(); +} + +@On +public void deleteOutboxEntry(DeadOutboxMessagesDeleteContext context) { + CqnAnalyzer analyzer = CqnAnalyzer.create(context.getModel()); + Map key = analyzer.analyze(context.getCqn()).rootKeys(); + db.run(Delete.from(Messages_.class).byId(key.get(Messages.ID))); + context.setCompleted(); +} +``` +::: + +[Learn more about the dead letter queue in Node.js.](../../node.js/queue#managing-the-dead-letter-queue){.learn-more} +[Learn more about the dead letter queue in Java.](../../java/outbox#outbox-dead-letter-queue){.learn-more} + + + +## Deferred Principal Propagation { #principal-propagation } + +When an event is processed asynchronously, the original user context is no longer available. +CAP handles this as follows: + +- The **user ID** is stored with the queued message and re-created when the message is processed. +- **User roles and attributes** are _not_ stored. Asynchronous tasks are always processed in privileged mode. + +This means handlers for queued events must not rely on role-based authorization checks. +If you need to perform authorization in queued processing, store the necessary information in the event payload. + + + +## Configuration + +### Persistent Queue (Default) { #persistent-queue } + +The persistent queue is enabled by default. Messages are stored in a database table within the current transaction. + +::: code-group +```json [Node.js — package.json] +{ + "cds": { + "requires": { + "queue": { + "kind": "persistent-queue", + "maxAttempts": 20, + "storeLastError": true, + "timeout": "1h" + } + } + } +} +``` +```yaml [Java — application.yaml] +cds: + outbox: + services: + DefaultOutboxOrdered: + maxAttempts: 10 + ordered: true + DefaultOutboxUnordered: + maxAttempts: 10 + ordered: false +``` +::: + +Configuration options for Node.js: + +| Option | Default | Description | +|--------|---------|-------------| +| `maxAttempts` | `20` | Maximum retries before moving to dead letter queue | +| `storeLastError` | `true` | Store error information of the last failed attempt | +| `timeout` | `"1h"` | Time after which a `processing` message is considered abandoned | + +Configuration options for Java: + +| Option | Default | Description | +|--------|---------|-------------| +| `maxAttempts` | `10` | Maximum retries before the entry is considered dead | +| `ordered` | `true` | Process entries in submission order | + + +### In-Memory Queue + +For development and testing, you can use the in-memory queue. Messages are held in memory and emitted after the transaction commits. + +::: code-group +```json [Node.js — package.json] +{ + "cds": { + "requires": { + "queue": { + "kind": "in-memory-queue" + } + } + } +} +``` +::: + +::: warning No retry mechanism +With the in-memory queue, messages are lost if processing fails. There is no retry mechanism. +::: + + +### Disabling the Queue + +You can disable event queues globally: + +```json +{ + "cds": { + "requires": { + "queue": false + } + } +} +``` + +Or disable queueing for a specific service: + +```json +{ + "cds": { + "requires": { + "messaging": { + "outboxed": false + } + } + } +} +``` + + + +## Manual Processing { #flush } + +If the app crashes, another emit for the respective tenant and service is necessary to restart processing. +You can trigger it manually using the `flush` method: + +::: code-group +```js [Node.js] +const srv = await cds.connect.to('RemoteService') +cds.queued(srv).flush() +``` +::: From f7e6aaeb1419a79f6a6bbbea09dc970f8df7c086 Mon Sep 17 00:00:00 2001 From: D050513 Date: Tue, 3 Mar 2026 22:21:38 +0100 Subject: [PATCH 02/15] add to menu --- guides/events/_menu.md | 1 + 1 file changed, 1 insertion(+) diff --git a/guides/events/_menu.md b/guides/events/_menu.md index 5f12223b25..10a792f279 100644 --- a/guides/events/_menu.md +++ b/guides/events/_menu.md @@ -1,6 +1,7 @@ # [CAP-level Messaging](core-concepts) # [Event Queues](event-queues) +# [Event Queues NEW](event-queues-new) # [Messaging](messaging) # [Apache Kafka](../../../guides/events/apache-kafka) # [Advanced Event Mesh](is-aem) From 4b71456c1fa3766a13bece3eb15db341143065c3 Mon Sep 17 00:00:00 2001 From: D050513 Date: Tue, 17 Mar 2026 23:24:39 +0100 Subject: [PATCH 03/15] stash --- .../_event-queues/EventQueuesScheduling.png | Bin 0 -> 79433 bytes .../_event-queues/EventQueuesScheduling.svg | 3 + guides/events/_event-queues/architecture.md | 137 ++++++++++++++++++ 3 files changed, 140 insertions(+) create mode 100644 guides/events/_event-queues/EventQueuesScheduling.png create mode 100644 guides/events/_event-queues/EventQueuesScheduling.svg create mode 100644 guides/events/_event-queues/architecture.md diff --git a/guides/events/_event-queues/EventQueuesScheduling.png b/guides/events/_event-queues/EventQueuesScheduling.png new file mode 100644 index 0000000000000000000000000000000000000000..5cb7113672015ded5624d95d201f2ee51459bcab GIT binary patch literal 79433 zcmY&=Wmr|+^EM5K?vieh6qN4nknWUD>F#dnZV)9Uq&oyjDe3O+=G}46OkiN(J#a9 zhT8GGgR$jw4{2JX96sDXzhR)SfATmJqmF+vf{^hAF@saJL#2RW=>PAJnoM7@5qa2q z;{SZ64A<=Qq`N`)kcT&Ea~l<_W%&2&v6$e)BXkh-y2e;;hsl5M*3J%*{?6LJqT;Cj z%l5B0a`Dktue{X!yw== zifj#Mir%0Q^W|7hm(!?z#Z6dm7$={>?GFM>QEU#1AdQB_stuv-R; zjEoF4JHEZJw3?|Dq^Beu^V41ZVYqVC_Jpm9cgBto4a%U!X3zV!;QYlJv$9Gu`HSQ^}-|l=iU0tcsXi9n8U2Uw?Nyqbk>H zV;ng112OwWlC>Yac~(5d144G3Q_yil&TwTca&5j9jRWcXNktQB2@7#ejj9XfyGv@h=;qYuHIf4V*|lN z3A^9TZr>z|3z-wJxno@g{blyg1ADb7{i7U7FQ0>5H2)!aRv?!Q>LfH2(Xpqr2|r5=vyNES51M$Vfi z=Yu?qey#xB`yZ>m=m|ULor`&*2rE~9U$JTA$Q4r(I(>52N`{k~H4r*4!e<_q|P-xTCHS_F8i#OlGHatmJc-xLYu=6r>M zj9LTC8MOHxMnictsd)YPQ|HfARuE4qd9^k2r<{a0(|T+h`1z4t;!X;I-%U_%L%h1 z&F#D_{{whDLEoN;$eaP|KSlr?y97A6Lyjae`w!co@kamBdW%V^>zEuJiT*7p4bs>r zgxQ?{;56|xl?1s1v>P2It%OTuij%JflbHg(VoNA*kLE&R;UK@2ghRurLI&Ux=B5P> z5fSR5tY$sO&~oZ4IOmT74iLQYfuQv#Bbig9u`g;&Jf7F6Y($!^LeiS}@2qG%nS|OnXCN5i20%&EfrU zq4pQi{x~i(@MmpWraHxZ(Iyfns1L*+gNJ9lH1HG1M>4tV-{5b_4D$Bx>hDf`77`#K z^-B^_5S%Rx41{zyFV+z?+f$NKgK@Bvyhxl5%ao+g;p^Y|V7NgO^OaA6hEy7{QI1}r z2~)L96MEK*qNN|O`k?d!y zjEw@ty2kaNPf|(eDufd9_Q(6(bHz>u)YjJGE>)Wh2KGSTd?DFGKxY;aqeC&?7_Bk? zr936qtXXff&|OX$mOS9yV5-BvoPk-r^mEd-w<{P4681woKjb|t@MHIewS6_{_F-Wi z?T@@`LVa)LMBfJ@Pzq`)2}yG{Iv<4J-<-Yv)KgAWYcYvFNMm*roDmlcr=eVs>+Y{2 zoy?f2uj@QAT9Ykq-Ktl^iW0J+5A)O42j<#TAk!OW`v^G#N5t=sGZn}L9nZW8^xYja z><&lnFO4Prv?gYR+F$CCuH0rSHl5h_V+!XDeDuG_A~YF5+Y`eT5de8>qQFK%Kqcr3 zSd0W(TqPgOt?mH`#OJ=ehLIcQB_}6e8yruofpNw*qk-NCRQ6#^H23`Q7o^*v^5806 z(R%!1fFq`m^_6A@FC)#ljW0o3g+6FM*~<2@SbxaPlsjn(^CF{IRUmFDNF)W#h+Q21f!M_`onS`C@ibQqKA=^1(nMLrCrLC=QQ zh(+Psxwym#k4;WuIqgk}RvGq;7E0rIKHh62Xf(UBeEs@WWU)8umFR2`0uD0(7^BE! zb#-+q?6yQyvS}2%@kWY1K0XR<7rT>Pk$9{EChixzK}$YQgqN3>GoFMMDQ6m;UXziL2}nzaW77HqZZW_a18{Au zDlupTgoHuRAx%x(W_v2=bb#{;ZhO2%pq9_VoU2v32O=U8`_=j-@2>7Jj2?6pbTLFE zq?7ZV_hUzZ*B5ESVbTpuO~nI(9m?MG5&oy;&7t(|`3BahGA-(XQ37}@y1<*Wt$f+^ zcevcnUvZNI0w7BW?Q?rz>sm)w(+A(b#Jhfuihp$^4ep4a2p&6oy{khFQ&<^O%~95A6Pp=Bs4m&m6esyCzd9t2&2k`%~I%};eNI1}|^{&JWi`&I! zv>K&4v_!L(&iQDOA<0`o*8zg_+G;Eh8UTB80@G;xx9~MkJ=4=_*FSPhh}Usz0DJpk z&e&sOM*Aa`)wI(2KvOZ9AoK(EiGLYWxAjZXK+*-WgH4n4?Lio7w&k&S(H6jET4OT- zpv*vE04N1KSih7@y}g$?1-O_v_WUo`BiY^K(pZMl*Ia&BNS^ z3KVqAXkvcGuUwZ0bJR&mv7&wDI!z^ka>t<3<1RQY8h_gAi6Fsl4Q%mLa8fhXgeZVDkoLJ4FI5+DY~!IuZ%iwn&4AT*NJ zC>NOjD#Dyh6KE=;sh62dlcfAeX5|$pqo5rLLIp<##Af{JIY&&_8p3$N5Ppc?g5MPu zCzT4NX!AAgiw^XDz>B=5`qU{|c_9k#XMe(mu?+J^-O~gB3CMnz<`MfV0W80L3f%&D zR5c|n2rDljE~Cozv*&sND`V#8Bfw>5A9-1!hOlo?#MhPKmLe3teRHokzWwi5f6ZPF zSdiJ4*M`O)HLK=*M!84w<8f-#Vco~$QxrKTBR%1+6%1T5Apr^3(zzeeSno}|;s^y# zGG3BL5-n8y6Edy&4_Oq?4=3?J(_bMwxZ>uZhj2>H7H_yIt~_ol{%pZa!2>awlp*S8 zE-r%Gw&h(q`(s~fcbr&DuVgTD+5W{%`f>p`6Cyl>hygN{=ZI}Q5%8MkT|0MgG5HT0 zQRLf_-~~edaRE6K?Y9L)-ms&+vC^e9v23;@!Y|+1gw112o1r zPw&cGjtP)db$u81F=jk_cx547FLqvU61w>1w%Vl4{y0qheM{E#1a*A>H#vS*eM%4? zkS+p1>8pwwN~&AqI2z7rGDUINdfBtFMSOB&X_nTMQMttjA>7H*TBVaVH*kmqEXMf> zw0!e&AsCwei9e0q3%>vWl$jEcHk&tTy-)kSHXm~#qCTH%Rr-qiat7h!lH))v6yO%-bmcPx@BunMKDH?ORBj$YBzup_JgL_p2m%ubO&NiUo~WLPuMJ}qn)G}( zh+?;m>M6k{)Ecbr7e^`=Tbh%%{$H7ueI6%$sCb0_t%jBwa8v`S_EA=HzQuq-``)4{ zc-g=A>UiH4XK@wk&9{rF`EI<(O0fCn9!qV(%WiJ%8;`l|lYZPKC5MdMn#3uH!q{uf z?4oC#y_52PM@R!4A=`Pc`E{k+^&dcWqy(PRxGLjhbkqhF(a3$&EZDNKCAxjO_cl%} zL>PrvH0@h5zm9_hG`Na=^j9YQZ+3C;;o5)Q0$UuQsA;a#-6{VV91-6jguEQ04h21& z28Zn)I8G$TjgFli8j-AW$0Q8Scru8pGA4<32}MZllow@B@eM{F9UHYT9;s~jXP>Um zfHB0N?L3Q>#2Ou5`M8R}61pI4vHWZG9ZP3dc3Lf6kjpaoMi3|0hOU0AhaCjfB+{oAR)%agqR5t z4fZX{HiyR>=U^Xrxmbn-3#z&rJ_1KWGW+f&4E19J)R|27`g=7DD#?kzKMZ1G$X~eq zLEc`d1lUZ3vj3XJ^T7f^e=U^y_kQ(91}Q9o?Ez#m`XgmVIa5|Gu23F2)?CRjj+=qN zydp7&1|(iO+D|tb5D*l)B44+`xQcV|asB|M84Gxprt@g60`QNiU*A@Icj9Y$(0;L(23BQW*v%h~?>Qoh{?@V}iNd2Fe2Y@l3u zo-vu|87yPoUj0pUcSL~2J3{;zGNS-TXL7|06IYrH#R8Hs(qHb>=|ZUQj$lO#ECYnp;8 zymC1xj=#{S`TX57TNIS&7i~}mRodlB1+2-CgoB0Q>{droScrHU|D$RipM8)|(L5Od zx*9V2KTp3c=VT)C@tlY_y%#>g5VLn`88P(r9F>CTYjPHW2~n=s`*_BtpfG)6-ua#D z#&u2TY0-kP<`gG_ni8&g)&>#zC&_Ccm~NCjd}8%xg|+l9VIWW>xq$NCRzV@C`P4Cm zHnjYvtZa61Y2sSLXY?fY6uIJB^#s)tOc{$vFAtbXO{c{|g zl)Dw3DsV~O&ia38*1xY51OvDDoZIV^A~1!vob?YtfGa7EDVb5zoe$(r;FPqOdI;^I z0im-cwI}-ue0^fY8c9~3`o*RIfZ#0Q+DQ>4xQ+jR9{!o!?qbpFzRJfZ{B?1iZaDWU zJg`JFdcQ&TqqTY;rAJRmYrr}6tOgY=usGo{#RUB4Vp@t($9{!<{*N%V9#=hIT(U~0 zEBN{D?jlq>JG*+E+NUe7&)UK19}ip*pWKkG+AM?K<%wI}fO^b$?2LclSa%k47bwm|%<~1`Zqdt(#p*lm(kXv^xFa7R$yg3Qn9N|qqcPT!*=i=;))SGa zsqwR(5j|yi1O%LEa6i41yEfQL{mCQTHwMVzs^&r1yoFm433AZH4LssYEZ-DzJU>oY z^WzfPuKBbQaDT*0Ouq^a^YW@W;72Crzfp3Ed^BfyoOi=z@QCs=DLi6cqYowXP zxh9m4y^*~IevpB1Xz2KKrwJi}x1Tj~d8{I^dJ;MUMk{o*%$tc>OdnumUcG@zu6y*` zs?b|&d#xtIdN-2MvwQHAV_ueqJ6rfM(%m>H_Y1}Yi?;UOE*Tp=B6mzS7>MUfk!%+e z2DSd}z|+dY-Bq6KvlvnN{gNw|R!)aJ)UWy&We7-l_3j~$rO=sWWb?9^a!6H92(j18Ks}Mh1zQr?!CP0ojyU_nS2JG z7XHqYWTX@pXSj0W2S^(xy|wG#C0m1w`KDD4zxd4-E2)C7jtKIJGz_+PMfml$HRa{q zTLXhvvc0GS+rQbYuzkIDgPZ|;vA1V@3A0}?$QN*dmXHz6 zZdD(DD>+(@5x8V#Dt=A~;5luQFR%*}zTD-uG}|1>a8fvn)=6?B8S zb*E9v0ed&(+}(~$xc)97PBfjp%YGeBIezxFzD;zJ)9&)lV6u?a_3;kIqi!=C7NJz$ z$8SZWlVg5uY<|Al9SG z4s&cV$D`oYAOzvsMMCp(tir0W;R48^uwJRW2%*NJcL!zR5hok$tQgjkKkj3I>=Ftt zwpNhuta;|@PU+liiNH~(L3`pXu(U+?eTMG57JFGl(M4&#erw5AM4y(t;)sBID<*@s z0Qx%umhYzXTZ7Gs=`5xLa0++OXX*JWYg~5r9<<%TSH%1VTfL$s%5@Ztb#Yltja>Fw zOrQ&4p@=;M6gEo|UOkGjpe&}jsh^?&n&W5C@Ded8fW%DsW7W^*PBn0OD_?yL#|*Y; znY+sDnK4_$G;3v>5|_`8cvmtMU)M8MYyBRx-;`=L&mj%f5lj}BDkknfpeZBp2^7mQ z_HELo-`tF*DA&^f2WNa`GrzBdCJb+VX8d7Diw!Ugsh~=+97|YBOHv5Wr9xKCC|u}} zx`X5C+^&9G8>#_X6ypcG%U)SWt8w*Is5U0Oz!||j!NfNv>s;~+dYv7;nvBJfu`{T8 z{KWh|Fk!@nR#qa4BPhhoA15s^EvE<;sO6HN@Ni;DK92O@B}ykkfY5=WWy?(GP?>m) zz{Met8yWpu3EW(FgOwG-2AP%no1OS)!V!d;gLA%^37CfB#92`JPLPhqeQ!(OYETw| zVXNx}Hp43~4?uhU%B2%4gH^^laThCC0WgD3Pdn&7MR9LXs}nS3znbRwctgciAMV|H zed_}rMmS+w3vUugB48ppn&V7cj*q^(6)<8Nd#%_gGI6$%ISnh5-i8nTnoWF*^ppKo zZbV8~X2@`fQfor!`=+YBF%G)q7;GDxc`u5A@zc0XB#X`o3U(DphQb^4W2_|Tw_7)z zNjR%hCx5fK7ne#x4uY!^a?e$}w4eSBz42TJi;Iina6BZ~HhFsdRZL@ULK#62qrD(7 zg|S&w)ahW@lkmgU>z1O(hac)!Qbjv8KmS

GAS!*7bMqf?Kn)B=YK99$e6co1D>9 zU#ScxVDC8Quzc1OP{4<-h#^MP<7eBfrrggd{(?|lzRhC;0pn)y)a zeMvCT(A4O${DakHXO@Va5iXy35dS4kH-PF! zXcC9womKa%(bj?*gWl};y!k>+{_RvKE%irTH7n#^Fmrc0`W z3JS~_nJ}j@)q?;EZVH5~SgK{>t$h9UK}X3jSCp(Rs6cyAex;RSBi* z@BI{cJw_>EsdAj4#y<*WcaMMg#|2ac`Nw4Q1<;d~#3Z&oG7#9t$C3~WH8&?v%dpFS`L<9|q9ofR zzEY|5*#>4(2Wbp+Fw4@XL>27d8d0)?;S2M#n5yPf%C@$MeMd)`uWtdN9K)^r{51Q+ zBm=|PO>j`!4AFS9%It#p+931;ubz5yPE+6iZv$O$qYFg-fz z6nfGs7Hd47UjZH8Ad~#V`7iqnMvcWPmVsD@WL~< zPJ%=+snF77Zh(yoaBmtu`+T1GO~{!-lkkNn`$4d!_2KyP#i~&&`@7GpcUHMmEHfnw zdu>8!rI>3j&%y`kHy~ zN_eo<$_TB<7L34OQNwyDmQG^#;(O^imd8FxOKsJBiOp-24#GI8*ohjYXw@%{pN zH9!j{JwWUwK8La6x$!_boBW}~qZ$P$p3&1AVL-w{~_9FEEv)!NE$pQ@c? zyzkD}C?^i`PRaqTtwh6zi@tNzbp}@z(smomN0qch7oPTQ-U@K922Nv!4+R!>?Xv;5 z&6?w(#j+_^hVPdY;yTY2N%#soyy|Bb2jVb}bACnup*G_o+G=n$@ud5?XM@i!@#lTU zDx-nR+A{&a45!Wd{s^NvwXoyUUeceLd2S;(*h~p7nIr#+ne`!3rF=a#VxK}294*wo zsw1|co=g@NzPeQHG0-5nh~7ru?dzUVr$@^5-Vyc7@KDNtI4H_f5J6x|C_<#_9lOAU zhlcNS=yba~F;af5C5A~&jI+)!tZmlm*Vhj<5i8Jfzd8Erm+V--lsZm8rplPe-so9j zfwes*ZNlxAw9rHB22`y)WX@Iu;7)<6g7bSt8g$pz_2?VVs>qGNPxm^?6_^*Ev>!AS zuD8&lEN1-vKgQv&h8>VAd{rPt+?!Ilqv~Y^SObzl5GIon9QmE)vZ&TJ7Ud_ zDal|qG7s1@Yw#!aC3y{J=9n#OF(uGtG0EnR%;O6DYvL2w&2KYpwXigb){Kncc_;bp30yV7DfF!z7`rZDj zYX0{pQ>NH9ZB_yfdtEtc^8(!v(ixw7ZK*HjGHw9RwyAm|@Go&L2nE9ddXS>qOR{*G zKXx-mWz?^wIUV003zGK@2LSmui3t?`L}{d^%7D?2Mt;#A7hU3Gog-pq(^I$_OI^K! z!p%zSA#Km;@6Hyb79xbe?N{HOki6WU#1hV~;yI6o_v`PMIM3ojGB8+iJ>JI)=HN2e zA@<3Wxr;8DErTU$Z0t>yGTgX_fM9kw)=Mw8LgR&;9nQXjlt+*3ju(J_0%*siilRGukD*PJ;V<30lFu zwOV$}4U-n@-jAw$ zDNTl~fL23v5x_X&k=K}Oq7)vKDb_#eD>H8SW>X`@#8otZci#g?Cg2ar5DUI`+L6g< zalqgezYqM8f4dga!CQHIMVLnuMb1Vq(|tw283oV|UH&`yJ+8^N^Nb?lYPLKq)*TjK zv-t6z9-34mA2wU4^P;H;0LC$}soMx34oT)P^ub>66(C%=hlfj)2Lv%2&${sfxSYBH zAE!7qj66A*n5awS5Xv(tEVQ9Lzi`q&g!D}e5yw!Xk18q%>~uGix`f!`1FagG z)-P)Ep%fE`55@99OW_`j@;m4gkL}Tf%#CXD6Q7kqT=0Y((FD;{Z!L2ic9fWtbj3=R z`$n=A3mwhXwKT(RW!sL%@m>SP?=GLL0K!3Uv`(1BU+bvH4zP>+-!MAP)ESx=b#Sx# z_Q&Kz4%d3m&JT}E7YYEBO7_RemeA)ZIZ?^^H;>EB_uX?6g^9&o56A_4_)dKlIZ zQOWfR?dS~MselLzaKLT;GMP0g?KUY z+=e_}M+|w?e5j-hh{jygPgAgBoXdKZJGzcf#MpkGB=W2VuAI{Uu12Kzwmz3vN){(2P zn$XYWRb8%Cz75&@_bWKaKCUnhhFXOnL8gmWb%)r}0;kEAF!0fBUqeiCJ>%QWee;44 zmWQ})?;G|iQ3Xm>X`=L|Y@s#iF)OCT5{kKMeikTrBarHzAD(AchUa)&g==)~b%_Jy zlK$bqPzpyglZ?Zw_o^Ap=EG94g(Bu2udHSospVSV_=$woE#2!kC-sV1yO{k}FkbSw zN8BLD>xQKhUxm*6qCyh@ar09I4~Ko{My*q``*Y^1D11lJ2agF0ZqyjUE!zR;nEW zq7d}`_Rl+(P=k|#Nif|Sj3x^F;MW;*UUqiufW+W$Q9s+VNpVvs+9W#v0rWMOACnKP9^Lrawke9 z=FuEBaty8^T;F1mduKtSYnj2KD++X>@Ui!W;cc%ohqw(z(M;SjI{ZGzFppaiJoY$` z2&Hz76@g#6oh=IUG8`0KinzM=G?tzwJW_b^ne1AVC&ZO?qpkrnsS*O4Q_4s&Yv z0V|)~(J`xe&hA=7UYbN3(3VW!88)c)OcrBJ&!D26;u+{=MVxHWk0bX^wdC3=K@=r! zLc;To`s8{KJ4!Jtc#D*@^|RYwgHOms-#Z1N=8cA(QpUz_Q0I~InPGv$a=)Ia(7*#Y z$P$f)jL=_iHIz!^*yInjulKi@=xJ#(?~R^S8!cWyY?{(4++d}Y!ECG&+e{Kv7$+e( z%)H~aeD)jV;5MdfIDnc~IuwTjF*L#!wv>Idm@?(DO8^(ISEtx_SYsTvde zLlJ<-wRYP#(}(0snxTBW08IgXCOyLX9wSad2r3L~Rfuc!?S=eC34>;5p3JA}9{!CB z*zA##nVMu&$#Bep82rS+&c>8NntWQ=GE}B{9-ika$ck**&-?zF_frr5xBw&8)XfF& zQ>gV74L4RBUhhah)(><{RcGBG<-J7=6K=;&e?9gHe%u5}Dj(xpaeH`2LR?G{#TC-D zTQZcf^`2QK5`ltWems)NM-zC~Vo`F(C{cIm_2Vd2q^FxdAa6(LV?ZCGaD!W|=SBTZn}W#XM!#SoY%sXl2IsI4gj!pHR^|G=ikr(~l^(P7l`+ zmYO;UbbIa7)w+}fZwaE)>TJZ-b#3nWX)nC36bS0N)9>qeQmD2u{j4iK@l{D~A6{rK zJ^q(m^%N>}8=+MMj$pz0Qx4HVTcJpH|7rf-DDAcN4SfQWijeVWHM*vI_38BYw!Nwc zmlEAy<5)QCF3NOj=%qW|IS!n#WA6yJ(^4>h+9H2zrph6dGK0m`EUosubTu3I(InL! zJ&kmr+z)&ku=hocgRG?zHYC{BJ0+_*RLOm(XtZ_^!6kfkx8)AE?HcDNE+3|cDt+I| z$F0iSqY`3VrO_Rf92y$JglP)p8iIEa*00{R?+mp`4P^1bjAT$I`c7;?Tg^8hd=rln zTB@@?nOpQCvza>0bHRx3eX&8u46(iI0sg$_IqrGL^&HP?W(Y16 z@xSsUkc$Ch4BcNAq84qs#SP>%ArqOC;yVA}W6D%0}2k zkp6A|a2swhF*3(mERun2ymNcAYXU?c<0g@)O%DsNlX6GunZB4^p=lb%Pj`z2w@MV#=r5QwRF)sn0<&i!(;=U^1JeV+ywp^`YG`o8CeJ@Gs=d4m&W z&xJwy(Dy~<`{ZSWcS)7*?K#M)i_l0;l`0Y0+7R~)%Ob2eoA7VS5Rq1!g@rEtI$LcZ zM=yYENpCsvWJ>zFz0>2xI5UUY)z?=%cG4uKw6pC4@F0C%tlW7RtvxV~wg{SV<tMge= zhG9|tqp<*@i3Mt19d180)I@h|>zLZD$%V9t!wL4qm-Qzo0_I_0^%y z`{{#T+H}|LQ%P$WMtpY;^Kg>J>W&$$(+s7V?z%dTg}6Thz8=-+F^FyUD@U+-X#G>d z#j7VP@#M`R`{kb+Yy`{_>X8IoF1iWw7TWwsgmky&>Sd+ZU1CDeDg||ok`GGScy+{_ z;k|E510KOX(XO7-ekwS@5i~qgnA%|d53E$P`tdY4fI`y5I9JoWirH~=2nhjpm9-^Z zW$==3AuA9C$78T5GKC}3&(CcVGv+P*G8UiqF7@=F3fam6{<;u>0mKsA3}(HEH-1liD=bX+JppTrnDc`YRb%UEdy!6)Vu2(pn%Gcg#1$VOkga~z3`r{H z@!X^vc;5n_+2kLHPOPVw(d33v`&u=2Z<0n&HM%eQIq3t6oOU}E{437;aA{MeVcx%r z$Z05`B%{7a?I`7OI$TZ^YOq#C2FhR)H#Ov$WTsHL7|%-N1(lExnjw_S$)=N3xk>a~ z|98skxm=41;ojSmn@?lBk9vMc!4oqrobtq~*f&$k?eNW&RQ;HA z4{kIbbr#zAW*?-%4kx?KiDWb(DXdfb`IeGIfk;}yjD}Cr(Ekl78JQAZP)IR#{}?>w z`j1QmR&m6xz4uRH%igOTjQ0&e)gqyxp{S;%!f_Ix@h~L)sB;B1LWax5QSv4HAS&x> zx-I3I=;$EAP|48uj@LRxQyM*kGNgoh?}>ddJ9}O+F;To_Y{p?&Y zPGNmYV(f}k`9dqrVZh~bmK6H*@jQj&mtT@aMw9*fP!dO3x1m;IswvJ|8wC{Z=xnf? zPbAWjYsrj?aU8ie8&Y`$Q5vakfD2xK^9Eg75&czib@QoqnU#k5y`ErnMq|BMu1wNM z8PE@&x;G6CY52zA2h;KGc0l(gsoEyof9wR1y#zUUbu^r30|sViTk9vS!MHX?Lm1|cMW>4auaZ(~eeeIaoPW2r%WCt_FvEzUuNuYrbneTM2^GJ%5sd^O;RG8F|$vqr#cCffzu$sKA>>_7d3XDEc z9bewSg}gfu(Nk7s!`>UMb^GXOGmDv6crq$ST{!2|{`5E-Y#o6EZz>-a^;-445Zlgp zGae7Ww@5zzq>*X9$cZ5dc)n-f@OTP095LR(D|Uz|hFrI9^FWOMco~2>6rckILbvvW z3_^;Kh-a1##V)~byyd)9?p@=)+ z=oHOLw>`MW;ZA1g35$A_Q&A!RgKS^73Q-<~xO0f`3C?y%t?r3({Kw!~$J;M@vPaf_%&gStHemk+6w%>JfuD^?R$P>}c(Iemw=g0ZkmH*(OSgb^; z*heoJj)FxfD0h)B5T^-29t{`cp8= zS%LCzVe<7q3b;Aun^b}dH{GmpP*)BS>JxC>d2;0Kyk9`YBMM})(Lz1lDVLyr4gbJX zC^A3dM!C3Oi@q}WEsBhR>^ve|4-py>;T;-x(lzH3N-0nzz+&5JHv;;cNPa`*EgFB) z(e^^v-QCVH>+nB8W^QZ1JHI$z9U|Y0j8|2)KLwP^1R;7R6{wZc*xI5%V#NgxY)csB z)Wxqcq}BD3!KOJDHC71=t%d6_a!v!xoItdUQL?NVQPtqd+yCmmuRA)4y+1@4x|HAQ zq_dI2G1qT&OhCAV_g&H zI~9sk${s>beVQiX!J;~}#_``J0fPz35RxEF3Z-uLdY%n$DVgt~6;h3bsVaBGG6dd{ z*fgymG-G=M-H#X!zc1x$DweO3F^0BefLm9?f|p@3QJo(FIeB#-3iC+(e|iCS1xY@N z2~J^vK$xH?lGi~IB`mc1n&y1r&Szq|N9Ck?4}>%myD;?Jd?8g%6(wX#(QX5B;#FDT zWTSFq2^q!ni1uYN^4njHVw1x^0!+VvG{~&bHMj68ytwB-CDj9}_HWYh0$C2f;)>_x zC6nG`=Ty5Da1HFy^S(aMvDL&DODfCFF0Qc(iz_r*9R^5%QCy%U&g^H|s^334zMltT z1swdi$%i<%wbgu=?S4-!@b)@Xz`E!&OL+!8gto*TC>A%4Iu%09H3zo;#N=8{{s6X% z6=3`XzC4fYP1Yw<_x&p?m_YGm@kV^adcA6v#lxYN3lW`~wrbVjaBa7%Jb)iBB_0^K z?71<(@xg|%v#ZLeP&w+lFTfbWLY_pvVO>Mt)BX;FkG*$gNLTAx`&EZf@a&QX5eXnM z&~rm5>6?tIELkv@L0DE6CGk`*XGglwTe>m9!1H`kKCwB-<-@u{N@4m}vdfx`B7 zP9z1-M4KCi^ZTOy{O?x}SD}%iu@x8*o?aL5@>@UsqH_%ldpMrkS=i7eB-%saZV(5f z%loyQ@Bm$j_=SLgXCG?^0X*B{x#9V3n;;L*OA$rl6?l23Cq#C+Hd+uv+Pk2k@Am!g z3*n_0wBdg3PKvr$7E?nk%maQAk^v2bf*5$v9VUsj;R#$kE0rB=z0OB=9H#__@qngtJ<+}abS73Cmo|#?~lM3rKnclY#EFUtO zWbkg%27cgxM4DrWg0>c0`rm&>4@V-(5TKQNu8{D8C@A!4?nd9zB^U?=SEIair@x5@ zjtfsRdF}(mj?8kQJV{2&{mKS$Mrl$@p?eJP0~$dR&$lCGr_MxllJ*}m0xck1Y{p0! zm&4E;3kI~y4TQy~t85-uBM%QwB0UN+8aR}`NHh%-wemnAHMKvJ=>okNp$z&^ZaSg$ zzkXuAfq;Oh1m=)j`&fX{YIF@bJjY$kNFt{oYHD7b5Hq@I|j!>(UUG(GubD~9t~ zuy#+lG`r)ZWN*;LiQMuwnM_Y^&;HF^h6<9jPh#QhP55m+K9DUG2tg<4HK}%Ea6w7r z^|05jh8uKYhq0<)}a(QPb40t)f zDljW`4b1G1ild7HKXII4~4@ zayG(mwAf6uvOb*2T|&;A|Fx`)#r0S}HV9T_d%A*nN@}Duo>H2OhGru>y#(ob4h3Kt zv6&{$T8~R6(glj5Yz~69%qmYI_CI8|Z@xaHJxmsX?<1EN;YBcl+$Ye6EcPi0x!iyc zs~L%>dE>Su{P%qV0+Q`_{b(5#O32 zbv6rhP%xT8<0XpZk;ME&wu{Yv$>-X3_Vyiz3r+S1vu{ZifWdt5#l^+YgjOp&b{m4n z$2;5onJP%}0@F`lD835a#?Ze5X6@{?+~tBoLX2+Dca-RkS9=hExh(Mn_VtBmC=8%& z(V7&P8LgX_j7o<^;oyY&JJTUZChRcK$9UHLi`ZMxfXvN`oZFE`5AMQhsH(xm1b18P z;Z`qx76CX|A8k5q8uNoz^UT2?84aqjB9+|Fz;AFY%Jl-IPZXQ zcXfo5gy(j)3BAE?J*&LF@ZBD=P%2R<^^J)_fdraCl_1_oHb2g4?_9mDpp!K)X$H($ z4xk&17f6;U~klhKU8?v zbLR2Lf{I0d7OL=TR;E0=Pl7l|7OCeg=Mx12l6}W>!&CKJ@N8g^)$ZI(L~6+pe|<0{ zjaoi$egC(@_6}u4MEN;Gv2MAx0#YOA#{s9$gaby@S#QvRDcC6T9LzUL+;#{E5gaPn z68=R%&sT=c2keVf%3Mbj6qNP;I7m`bQnFZ+K49i|{r2R7)YKsmaM@9mu~sMkA0&#j zX0cuQ;zprh&d-tw1ImI}Eap-o6dLu<*)e~ZjMZ5Q&S%U^1j0Cv5n~^=oFsbbD-d(R zi#nvQI{+GQ#^TraiJJTknFV{Er)-*UQq*?_9r#Jqm42 zp*;IUKXoX039{m_Fc1tf0q$$+TZil86_UL$;6)3$VH6qIz_2YapO>=-!k~hy@=y8{ zTL@jax4Y|4;63$44F&VVa7GxZaIgq_Kg+W{#tS6X4*lzf-^Acp*^knc9$87f!Wna` zC>3RC&Wd^|WADo@|EjeF?C!_UMNHRAU?m0Y;6Xxu;z0d-vQ$MVg~RnYXTAq`pM@YY z=Te(b3^2L63Jj={@Z+qb1bFHI?-7B1YdHm_p`yTbz$mmc0d_zqG|a_rGam{Zn4dFF z?m6&MksM$bm>WRbLkbuQ!`tV92c);#~|_3Z2mh71SDk8CpjY{BT@^QOfIaNrv9|ImcaX4JQ@grH$kk7 ze%c)jBly7w?M>qPJ3m`F~t} zcOcc@|G&+(ue~=Ro3gpCy+_Cvk(E8m-g{>68A(JUn~<3;lCo#YmhE@0ckj>V^Zor@ z?!B+`I_G(w<1r>|qIV)fH~umP!53FhA_J9;q~5Ujcsw@Ed@et;&7VJ23m%YIk|#&t zQ5Va{l35z-Iyjj9&dzU4{AP)g4QMGi>dnRy`fN$qD5@Hg-3>W(#ZT6XUw+_!1r^g{ z=c@jfd4$*)$es)^Dc z5mRof%xqPLSOpF7pwsj7Pv3k?kCvE=T7B;JsFa>tP%bKoFfbsQcLc;wQ=5atxbQdF z4O)N1djJH%j=^QCNt4RnT@MkR#DmAiw8n;H?M1b&roOB_R|7yAmjaGEd_WyX!uLdN z2YY#a9RmjkXS}~$vp}ofR-4PX0T*nx9&jtjKUcvATNs_2Rk;cu0{!{rO|16nN?)R7 zR-1vHvIz%<*9qm4^lq*JFJt&8X zdGG#^2Zt&dFnpodOVv2@6ZQGc#}%p~$Zl z(&T_(WgC}@P{F7&dLL6v1K$1^A)@N(k6*UwKLt z54;KoA`!)M(L^Evmmid__m)EW23|Vz<^Z)J0n+lg&Q2JDhVJF%wJaYp+{EDzG-77k z{l(?wZKo=rmzS4vCeM^A)U5+zMokb}IJkFU`nO6vS6rseBnT;)x%UnhPEH9xRs@TG zxx65N=*)w`0n^6DCMITaWhE56i|EtS{S_I@en7I9Au|Udi*V~`n2B-q`s%_5+`BXm z(cH!jL0+8NQjHZmfZ8o@x)%IN#B0Z(-gX@2*SALU0epOX5!Yq8?DS935s3T&_7YJ% zO=`I-{`*eq@?n0wSoLQ9Kvrovd}(i~8zBJX@)Gde$uBNUo^GoYH2=cKTG=|ZB&l;dh<|T3@gDc>fKraE%*(7&y5@Y@;yX!-P z<~}pElZT;V;$x{|gV1VVWE9eVHZ>l79>wRf2(x&iPDVDE5%Imn8>igw=K7+h-`&%b z6n{_Yl;WO!>gkvFVDZ_3I!fI}fmY6dQxFc=D@j{ut!-^Z<>eXM0d~q!bq9cck0?gw zTb~gFp{!5GKWF6Abx4YEp7WITM)SQd!kr?ErczK)*lXg)cRrcx`qL%xHmQiE$A+yi z0IgUmt774Uj~sXnJxnkM>B$4kob5jQ6yB_$2*O2+mv@3s%?OoJ3!PcguEv^O)^MT^ zdV{4lCk5Hr*|F+-LaXh5iAowHl!Kl_7!G$4QX@Igk&D$>XQ=(k;l}7fb+JbDdrG~K zMDPt2%*aL>3hBB7wA7H&uC6XEBg^DTvr5%bqJzBs@7@a#93G<+*Mn8%K^6Td$APq5 z6_(c3p$q~cHtS~#n1|Mk-5YaQWCE#=0El}kEGQv6E;OP|aA|i`$PTHqkDQ3b+#tiU`SimKNMS@QW& zlH~OZ%{Xu@qK>yaE)Q_}$8C0=5ZeoSiME4^8s?8RwtM-b&3A=X3y8Mu0bPwKiM3r} z(<<{0y4Cs@RXi2HL0IK5lz8B@&>jyo3YI3yR4pOKpeBQfr$nJl1aAj`vQmfZya2#z zuAl3A<1L!udXPRDN<*3bXC7C)WVG`Vrxt>=zalKiLg9BozAex}fH+~9Zu zHHJyhtx4YMZ<1vrR1cD{?Q7(w5bb?8p;2=){$hgyT(E}lAi2m{Uaa%;^W_|kxi5Pl zJX73%nn^+ZMzq`+z7^7X*758xqPW7Wt@&-?><8BpAIR2NQK-@#2VA2NxOnWC1~NEA zq%EmQ?C@{o6Hcs0ANLN{Q~{!aoF$t^!bfXLy+S>C0C6WzrNGV(*0jL|Ii|tDL0K3c zF5lWN(2pHUWB0O@oAb36d)N(a%$+-96Zb&&p2SOU{mirk-tQ<)q70$5N-iQ10AnQ- zus_kxhS3D_5uJ(AOkdC_A}Vwie(a2sUO#iQvvk}sUKyL|&-GmVJsP7M2JsjXil)sd zM)+{12Amg*MQd|=o6Q#s#PhnZjdetZ-bqo=7Fq%Fa&j{3rm&Iy_gPwofb6Tr@Dq}B zk2E^jP&#EbF(|#CgIj9(b(#rd_kuN*M#jIrfM@qo;TXnNJ&GxlnXq#ZpP2lgxjAIR zVd|d-%x;L=p$QA3om&-xKx*JIl-WX!V{rd!2Q^T7!D^bZJjSA^Xn-@vplEC?si#g< zlL;2mE3;)ROMd6hR?I)66roTUP&{K#Iy5_Qkabmu)ORcWo^R?swHuCxYE~x|*(Z0w z6)fdM6CtID^>}5I@ykIVsGj&y%?(8GsjUq%wk|wJE{rAJuv_|Nl;$UX={CZ*b^=C zCLvp#{tuAod!Ye2K0j9cy?sxfugZk9sdAdoRmFET`*fFg`XS!uNc`?BXJeDfmwgEv zH8V&r?>Th0YDhw3?l~kRNGE^^m(5NwqV*F2^CB0J`0;#c0d;&dqkPnSsV}ReP_Kag zt<|prFzSkOud%QHO%S?#PhN|p8vDxfo?Pw(omfjRM}Yyngs0nOd-=F0CaP6}mAbRP z`M%rNY2uEyt{*nq?bGHj(qS~K+S9UgIN|HhE}dhKd&^1 z)kDUuqo1QH7~BJq@I50Mx|!u97xTN8P#DRJ{Oo(uIdc=6TTRupw?V(x5ug`_vc5k1 zry~u#CF`B1da*AZE_+_$o%(ErVDUuaf813?`ov?BTkY@iSdF0s%Sw@PW$Q2_MG1Ng z1lI_oysr8fO^CSG)oH_VW`0|4UrxKan2TRx!p+P`iyj5I)BI@G>EU=&CHqOzd}L=X zh_{+t&c{{#dHX5mqa%-i_MRpgs`Ns1kbgoqmIQ*G(QHk9;;HVzM)q_Y(U~#5c8;a7 zi44uOqMBo*>_elVk~y0chO$RnvlMuo#*ddP0!~*CWvnnT{JQ2TSF}1(NauvAf>bv& zVmzipTfRLw`aOoF`Pk6;^uj-)X7>&Wm#!Si%#1KB!z^@fGS?alADhU|Mw3n^R?J%N z!aaa0wy>~FL;T&MXTNj|TKwoLxb&W%eI!L~eh+vP z*7D7%G6Oz7H}^l`|KeEo@bTGA&t&ce1TsfT;a1^oJNG`$vNi`$8P zlk+f?B3celoQ2IOb&BD2ERaoF1o;t_g0wI0%tZ;or(tuf4RT&SmrrW^TalwC7Ef)% z=0Az{da4lg@tP{U+l~DAyX8yq=fnc(ST+D~9rxb$Fu3m&L)IO!vi7Z!$@KW`?vWjW z2Yp$VK{I`Hx(oo!g{p^75D0av#B6ct2=qQB<)9{yjfDQ!GtNj#sjt4Y-5IRkjp|6G zfo>7xOLqU!c(*^4F1%|{_kr>cBm%Cy1sp0FkVx6dxxectdA zE&CiDSqtq*kn~nIkU0kIML%yBX=yI$+uQ1Y*uX*MO^ffH*yx%)>%J;9cao9MYmTzA z+N;4QFdXdkD|;!=d+N}7cJ)l_{UU{!T|P_uJapQ=fifucCQmV(O;WqqOY!o(_%?!< zCWuQTOdb2>5*g44waVn_=Y21Ar;0X)hI@a?J%7J@Zm~PQxuJTMun>DPD7+QyZKiz{ ztKz3bicG5=izpGTAjpb}if?CbddB;Jjt=)5bta6qOWtfr%1}j(&V1sT9-*7rdX-#) zogetp9iy|}2N)4oe54Qh@G;WJvnIL@zdPXWjAMVl2uPg$LjCgDXYDq9f81Z+@Nvpc zcymP=d!n8EBN!T$=~|y%5lO2Qemw%S zo(kC8$K3fKoaK7{FNG>NSfyDdseUBUT+ zlloPi9|HVct&OJ|-@I!a}wkNr(i>2Z4CAyWh!?|nV^}|}vE)*_`3uXo7<$g!y zsFL9eO;bKV7mwTiD!Xh2SEk8?RYFST`;|P)v$Vz~oau&XgwgKza@X#a^SgNdf%OWU zxQ#aG%Ofy?D#-8=B?kCthO)c z6MeQMrf^eFfm5C+^V>vp<(G_vIZ6i;3|n|t@>6;sL9yeziVYmYB)^mR=lNRJ%_qmP z&FzPs=y!#$LIZ`lU>((O_}z>ELw1glapld1FU_}hs@_nDkDj1%i(XzL=h-T;9517+g=4k zMift0R+fa<%;XtI!SwjgKz$d-`t9OjmZGTpQ506c_VTe}O!xPtqNM8v-z9~JUc z7#G>5>V@eZbXGf7Jm`Q>2|J^HBP!!_xW3UE<{XR@eCR|x3@W*kvccj%K~R?qw);;P z>hSZL|0urGr+wsTOeCwQ`Dq+geCcq%jeVuw_THdgRSbs)hu|p7>v-yiJdXhU{cRlw zV^3!5ll2b)XRSgy3?d>}Hy6j|Q_k}`Ca*6Tn~LQJtPD)qzll|7F|;Sqko}KZFUehf zTu`HBWGZme&Vkw*WZtNln3!Bcr=FLRy%J(dL7NPLU%Bc0cC zyxiK^VxF8kJ3CrsDx`=phSdXKAS+C|51KAT+9e8DSXiByThu3@$q-X3_!u&mU^L^(C|8k9339(Bd4|WW%ogkVN zAC;MH8o%oJ#b2V+XBe;I)MqNRVj9f#UR8Q&OG=uO@UTmomo6kTg&%}gF{-(3r1aSt z^3ek>8wyJ3YJCqA3s04?2%aO0OfZ+oPF3vkr|-_ATt}Gd~&~;7faM2ef1-uWP zCFKBT`>ppOFZ+WDwl1E2OXeBj0&*+7{VzWyKjCWpI9wy|t95z%?xUtOK3qwOqJqaW zRWW;4pnU&rBS%Cu^ZCc#d*GaU4e^MmQ$%d14HjDb;@tdEP3c9S5YR57K@r2)CbV(} zKh)g-4KT3FLI3`~>}%M-tw#0;Z+)vt9k{<%HA$H-9LqrA~TEDRE#4 zQqCrAK{Ir`GR5I_8eg&X#_H5UDzdFq!#*I#0^5-Y#pzQ$^1-pQ#^V`6i0I1jR`RF#^ zKiJ7#K+Mp6eHePis*(zOtUR149Vj(dZx;$W;GZ&o02T1Py(QUsFu_6e91CSa7uVLp zm%1ZkPeA1z(G`mMxZ)_q_UY3&^aRAdEZ9vh0sV&!?jM)baV!=;j?*G@ z$;?-WopbirKHYO0woceYpF zxN0!_rOA!fq+lI>`DV=J%P~<($$rTSaWSKp28{U1R36rmY?nn-ZlNWfUfRcQ;c*}C zO$)M?Co@W&`=w(lT805cYYZ6=Lpdv1Q5g-0By_weQqb%isLFp|HQ8?P`~|G{pyg z9GUdO3D=MfoEFfCTF{IOXs^nWMt{Ao@u^f4hzCFe$(zN*EV+c}UKfY+S2^2Ac({fh zNKbDJy%q*yCF2F!I!As377Sbs8ov!NcQJ3G_tsDN!Ty%{N?I_P47)5)o2*53ih)?M z1xsCf_YU1{>#X$cSL4~5eyGpKD#P?@{{bac&STZc(wIue>qDCn7xo8cTwfVphjm|g z_tQ_2i)i%x$TLpvo5H1SK9b3ocqSvoVk;2N6^o4%*~R>++iu(TnJ8pUyF#l;%*2$g zZQEm4cghJe(yf1rDa@HOOlqjLSN1V?<{6K zlP*$gQo+K*K|h-=9Av4`${}gw(os9w|I}?7x4+jO$RVo0Zdl1hG7*VaHa&nNOfK}s zkVrZHjsmm2hsEw@CKq&NDQ42lUf zg|tcRLQiK2^U5-mIF;xZD{=w-zTI&`-ttCU9Q3!`#H}Q?t4&0;bepx&yje6@>$ZfD?6ERl}3b^^w0@}MnA1P*wP!SSZ3yy<# zb>1S0-E%}~vFrs&?{gIu{eI_ka*&6{)UDPmFaP>htS;8EpFjqe4Exe{Jw*{vBSCxwq)9mdPL{4qDp2P_F|chUVcw1rUFVoH z-M2>lXYc6jE}Dwn>YpO7X>2B5QB1c%!=if0r*BF@v=h0Obb(*07dPMghD#`QuPT-l zgS+u`hy2Qq!`=|dbWTVH^3&4gk`;ZF9A^u36^{KFm)0()L7JRr-l6*KFg!ix>G$Jb zXUjh$Uw&5%$_5h>+G=&EJhX#`sCFgBtwjB$AMYwH#sONe6Mlj*Ta8Fbw@hUE1U ztE;uBoA(}}t$(H=nA@m89R~H6 zfx~`ctnpjIY~pCdO;5|-*1cCOS${KLr$H(7dJXA$)6)}E%%%Xpw`_@&t_in8lxbRKxu>sn2!A@S4EkTS_rxs93*3uMuiXo}zy|gmV`!xyY^G zE*Ix8!?CB79~HK;GhavcNj-@P% z#;nl^fk=)+7flETXq^CPz^;r&L?I`)2Kz-pNf~_~NXG|(0fo}S^R3|Y*JDIP&7iz% zHh^2&+D4Ia8)4lgFj9cKx}n%xTccA`Q)`LZpZ^u~zI8^`$*YD;Ji+g7{r$yepy4w@#J8~2L%PhkbCvC06YG0QTf?8Jz zc|JQQCRwSQU%3)F=$e{Cdcu)^eOnqu#ndkLq#ibFqh9R--tvEP+d z8k}Nv?l)}ysFU}TpO)foywb{?Ho&|EyXpCd;l9W0Lsb(D##LH6V`j-;O4gm|%iaZ0p?U@_*NFQIaMB{`$ukYTyp+n<-nDyRDa$6VdKI^sI zXc3o736-fWtk>I)+Mj-qNs8O9AF_TG=8MJNw$kpH=SD#AU_rU-Ks;XqPl4!$5UAE; zoYh97wbmQ_m+pHg2Wz}a zhK016)e?TO>fi8}M@?^*PRkTOW`_yP+%yr`I5jVgWr?3Y+c(;|+QndfytwC@+2|1( zC45yaY5TQ*;(g%K&gYGf+U(XzdYJo#A6`cE$l=xj{Yp}I=AThuqFU^>HfZ_t$((vA znBy_r$-i|%z!2SFaa{rj^sn>VqQ;&pwhS6i7mIDoFb!iv+i_vAIY- zrvUn4@!ES^OQ97H%iQAXPkGF)y=l}Prb!H^zJ#{EfN@@%@a$&v2Aut?Mo)06PAEe zo&x(J!RF(6$k*@&w8-Kpr)ZsbX}ZgD)1JEn!O2EU} z-y6nPqm@J&<@qHa8}S~ds$}*<1dRxtF83M}wWX2xS3aO4X1dD|xDvX2< z7cX9`&Z7zK4aGrbcJ!ME4<@RtweQZ(iuSyMYutNm9KnubxOx{!(%*m0y45C|nY)O8 z;4zhR`^!6QES6UF$MRsx_10mAhjIa+VD^A2RP0a01PshK5jdY&(_9z;E>VNfg(_x?>{47<5J9A%27aZ1_6ZLYW*O#(#t z0jRiAmG=V((HLtllqPOvzX4vTdvcN(Xv(AhN=i{&W7Q*BlubeN@bn;>&legP+$UkM z2)fkKN{_Cs-D8y52c39JLkRax(a$4%jEQ$l1xN2P>b!_~-n=de)wzG4ibI!0;&lZL znLx3~9{&mijFuR@t8?8HNg6qvILyn1;2)9B$_k&eHI;3s;-khCgZ?{U9nx<7(Pt?= z49@30;~t)PqqnRx_ah=a#)FwEP?pQAXEgjlR?r+_p|kV=bH_GW#8I)#kIS^s($kmf zIXy=EulAqnDJ7f$)1E{OJgT4urCed>{0cFEjMz^ZYjieueWQV?_d0fm!Ki}Rmriw^ zFpAsKPr~=U@&Y-fr~vU6wXcK2Pb*KEp;-2whaX&)KEiXOczN4}#KpzOelu4-TMcCf zema%PG^1)L^BD@nj~Hc3oaqVj0k%O7*hC~_(TrH47^*WAapM9DqD1q+_!CFif=SAj zC)}7c1v60K?4`5m0B%kLJ1#!{Cjb@FdizM`>G$S)ugj^NZ@7$lV#A5w#|2eJC69k0 zH;T}BQ{El=7-bvAAjR1I_ECul&^{>m7euWDbY!gNNuCEE9UrSJDD!+oo5(=QQlAtq zs%_*eJ>Ie!RCkyhsLd-YD>Hbku|B68aC3d7^}RAlL7r{#Nc{<7+z8-|VQ0G$0AUUjBhtdAJfo(!v?55>Kxua?$60` z%@XAoY`}RY!iT@~eKY7Rsh27Y)@^=sK3LALGcpkqhcW~Q50~5BmBGfNR&cgZQk4XH z`SFO#22sWsR7gZq>87Jx`6UZV4GvR|l0_pL^n3t72}|I6Ji@je<3$`D^J;;giwhUj zHXL*mzkCjWlMTtt3PQmZ=@!9B)X=|JfQuNT=7C8cWS;R{YYJFc#LHh?kM`|rU#hWIl$YTN9g5)eejRo(P;|YL7+NV3K=)RrCK(t`_p0+#*O&)b*-$XN z9<;lKBd(#P!XqY_x~t~gR_lI|mou6f5n)2^AaJV#2v{Bx=-w2_ENmt{lO$#AvqH?> zfpZTb35lHYhrpY5mmq+B(H@IsYO~V$1YEJ++Kw}gVb<|K{uCr7*WA+bARP<1D0Krb zGq%=d0Q}fxibo?lc>Ku834hKhp_(H!y=v@UdC(E9B|(xZBQd`x<2 z-s-|iD}+A1r9;MJ?|#Nu@H zle|plg}dC`=iuQ-TcK9fCQYz~pa&1F=u}5Y?KlR(;apl@XB)QO%HenWptO)W-e@3w zxra&$_OB$&u|U)vBTw8LBTboih+rIX=Htzp$UFrtO{!x&8z?3RJ*f=OeH-0X-4eNaSb)LFnI zwBHps$7In1%w&!xn%&k^_KTI_SfAJu3Tx~jJt>tqo<;Q{&s zqIY^Ut7F+l_)yA zeq|HNY&qkf4Hg}N8z5v(=>Gjr&7MEO%Gu|vvSiBP6A=NQ3vP7A@{hvKKhQ+Uq+FNL z`Xfg{M_oP;^up6#zi`p~!U!;47}0-NKJ{nmg!##hFQ}7~lh>w(U9+hUfaP1=h@2~R zxHXi(MJ`L~BBP}2&Oqp5Gdx2a$1q}l_Tn{$qu|)w!f~A>VC{h|6D+kV35;pL`z8-= z=VwyIIfp5!r@WHR;~0yqy0x;pw<*h*b@HHDU@V2`JVFK7GqC{EmbuXLLg!%s?2!lk zhqB}`M(hKCuti7y{Q`u<_sp=sL?s{^(Wx|6*r8b88YcJbyyf8Jm$CNu;bGh(Z(#C- zr3xWpWM?k&7<_=&=)8czU0&MZaO9oJ#i5eU1u;5TB4adec#6z@%z)WF_GUf_%SY^nKHCdy z2I#W|>-o+P*5IG&5daIwyk%Wgh5noDTvhtc{z&v_BFzFRbZ%P4Nj+z0XZCKe$UT_% zn$q}xhDYuSUy@@6wQYn|1 zh7IWf(A*l!`GuXWExVs&@}xuEGJ|4{6@L+z+;XdchAJy*3_E|pME@uWH#H+>p02Qv zkgf;@>s|c8azVo=tz3i>EZ|deSoPHR+tSz&wiB$cOn!Ftc{ihBpf>K#$Sj+Sd`oFE zLb#lvIWYtCH3IKS<^ruw!>{*5#mERV!H8?LM0HiTCJ&Fgdc+w(FRjWU@ zAU&&MeKPy;IiG^@I~}pC&l3}bMs9HWFG_ob9s(Kf-@PMtNZb<3!OWKeKLp%%e$23T z)=lT0#vvBX`r%|hZgqf`w%URfw*s^ds!pMU(Sec_Kf8t<343+6#!O`}7nf}omLau{ z*jWTP7G(X;5g^Nt;JRivgI1G}e z(XZM{>kwh1Ukro|=D?NEw>z8I@6f!sqb1=+_6q5rYA4UK5r{)Vgz|fBv|r21JevDY zXOZR-QqWKj=6-Y zZbas`Gt0juT?}q(xLPnz$j>gd-N0ik*A(_dP0+wVg5g9i>##L07Cx|MvPSv?o%{U2#h^p4#C8AUWEj4^>JZeHeY(u zSy36kebgoLc}!SqD)QAUyIcv4+b05_14C1iFnDiID*u91$%mmYd+?)`Yf;=?k{BO< z5e}05^1DA@5OsMR|Xf8O6F}(Y>4%X*xNfQI+G>*G!p8eZd+YiLfx*07} zrOIFMiWw;=+^#k-eR)Mo3ovA?eaHVUegzb{E~(I((vDK6-R}(c1+Vs-66l*Z4Y%SG z5MXkb>5{?(-Ua@S1+~Exl;MU9^tpOK8sq}J&&JXR2h(jIc7Wt8q94IEiuEf34A&8(1T z!B9&?oh6m7{1F>l(ppqon?Ku0zbQygljzWFArxn^C1n;?u4+T|uT0&{0s(2}a<7}9 zaA9}1{Q@6&c%@~$OnzrxH_K$x&1=?L$oUboyDO+-)IXc6ls5!B$bnORX5bD&7sr!Sv=>TQ9 z$b)Ep0vsT*h5N(X$N1+kgU@3SCk9q$Wp67OYC}ZECZJ-I2SNv?GDEVP*)-QF3!(>U zvNN@)hGFAh23Ap-Pwo$1YYH=5{%o1mn>xuxIAp>O5Q&V0hRI+2^S^HiK}Kb;X^CiHP}0_K1GIqVio^OiUC%JY@IIvrGpirM8PHA+yyEQ0F?zv zw-LnMpDhVN0_vywJ%9I~aB~~ofPQb|5ZY8a0(&VSHn!>+kInQjx=2X{2!O`MJ=(<) zB0*$IrNoGiH-tsF`#%e2ON^Y413qaE&d>x`cy4Spc0gkPz*yu!A`_+$+*p`i?;`#A zYoVnEGluN7-afRw0{HI{mEbq!`qu>&J&{I|pos^B*KmaeDLRRNewnQ}a{gaU5i8u_ zI5Nq7Q85~C_5JiJSAqPw;l@R7#I->~=Pnd$;vS!J>d(i~qkc9s$ z@vVgRKPs3t9(gf=$rf0L-m(?D#BvdV>hh=quT#hvWqBJ-b2SV)qe6lF1FQ~#Nq(wp z!uJ}ITVX=#M1B9sG2e$9F1F6^|J??Fy0f^3-}=?&PRK^BkxyF-A5ckA+`AYW$kw%z zW0zVez)Q?S-#gKI)+(+;N+*7>)Lj;Uh2xV}bzjGr{T@6bA=obW_kX?;ad5cQOq5Kq zl6<Mefzkjo_WSl&llxb4RMF6Y!~p1i7`{a?-JWlq znicixo2bipf|5sv@A@sYWlO`)q-7U}0_7wG3BY$-CPt&j&B~o-Ls$@+yFF(vP6|Xy zkH4wxhm6=gjd}gw1qIJ(#eiG~!@4*yIZAq`Bu-)cC!$>$lBu6fznUQQ*hFMuhKfny1$He(H)h6cjuo-3Ok3M`kXvHzxRooKlrI43^`WJSe}oA)H;}v zISQ9tyW7vRN3>iuKg)0R!k1vEUB{^XAPd38c#ZiGavDdJe1s-7vv`9u7{ zz-_4)hoR+%@0%RqXbQ%blclYluSUYc`Go%*5-V=x^KWiV+9g%AXn#h~gffE?Waw78 zz9^%$%{O3OU19~(UF==^3ESx>2f@jK%3acIV7+f-9Ukq~Jl08jRM3X;lDm6r#Y=}w z0)9EXpaV{R)X1ur)KG7<42Fo?)|8Pus+Yb z#;32rQ#_q%)PbT8SJs5WXS8B$K6mtl;SjO2uX6k$#uk!fgV&lN7bbNK@N4yFYrEH| zG2BC|dkNfQ4ZHJv)zO2)(&JgCNc_jFpv;V7|B5CI052(dH+EM_wj{TI^oN|89v=DD zp|lfh49yHgZ*K4uG3m=KUxok1IvC7eGi5PB^DR)+L6JuYOfUyZLQ@14c}jZGeC#<8 z1pJs}S9DKS*m8u16i%v#8yxh|Ee;9Z zoWI$69(LBkC?J7lKHtEwI%x_U9Ovf?56IWi!QFE=ix&2MBX@f6v_uKW54DR-0-BPE zT-a%hHCw^%X!;v&H-o8^I#Qj*K=2*5)FABF|4MjLayBLa&8|LB+mHG~?~?&V>H2Fq zpraNP9!tW!f^Dw=%Zn@-yBAWcPS4-N7x=JMQkSmAn>1wfSeq#x|0U^$DvBb043hvC z(vr-sd=wTAiJd5yeP?HVZWLy_RM1Qd_x|%5?)DGYoD(Es7<_M9wQgl0f=&Q|4Sw9i>p?o(ch9YJp z%$%R3PpriQ)h4}b`_bdgaEU|4*LPbzf1g?)6XpX$y8OMMSW)9#>5F}G5^+h@D-B;{$y~OC*dGK}&U#$1 zUtv#WXf_r(K5GpE|LuvJh(waK8P%(bDtUIWXcCSs^29#o-P+(%_p1j3zZZLNt|K4Z zzYhrMNF(XovCN|G%g=M}W4U;$dr{kQD+S)|x77Tqp?v2~twY*@*wDy4Kk?WT6l}@O zob>QyYlqDZ4!m%UDV^k0HNzq>f+xFYq5A>-$jwmJ9_exN^aG=#sPND1eyMgR{k8vM>ACI{~!xQiaxr`&ug`Gb- z4U#nNwC#3aLZEK&Dy%ip(OeE4JT5LR$5FZ|DQ|pqaSuYfq5|oa@h}KezBVQqk3b9; z$P52D`3#7_kj!t=G|k1LH8HOJJCBQZ6Sea1I1AS&!FqJpB|eo6o7I zWxoW^)UV;02$htCZmzQ}@6!Tjl{h|W>11oc010d$ICpq{tnE5Pwn;$| zkGzAqc?3eZ8vm8xXIQ}kY=13w5-oNSc9LOOQNIZW#~CTZ5RSl=-j^bR3KmQD8bFcy zN0$FJUT|TfvxPxDyBepyT>F#HmT`Bzvjh(ArM=*_lt;;zCO>|K2gIvZO|B%KJ+igq zs19vwg^^J$Gtm-onUpfC0eI2v9zzs1N)0|xj|(;tzR;>mN~M9_@U{|Wkl%c1ny*L} z7_(bh5kfHHI&V$IwF~C7{q--t zx81$yo0D$G3tzA>C`=3hTPVDp&5Z3b}AP2PwL?EPqNv^Ge1C`3+~JkdUo zfgP|<7=?xXF;B|h>P}8QZqHk-A|DmD+@s?))%l$traX`Cxs}prCSu``Z^Xum4NiW?ACd5Ge$`aPl3$q!PA8iaSGZ72N%9C*$O4Kl z30@8AZ+72}rWkaHmptse-Ng2=|J6JhHxRN55!)DLR_qD^*O%Uxx2rn2;L{WCrgm@wzU2HpPp27|P; zf!|-`xsXy2Uw;O{J)3G7>o}v&CGQeNW1*a* z%OM0=8bMMYQlg4>GBXm{mG=-Nezqh(l%?R{Q@@o6lcmqnH;el{(XJs2p_s*L;1UUG zYm+Y&VGU@0P6}mYBtd6l-hbK^B69pnefw^u)613ye2* zk3GexSk2f+v+oBuQt|TAkH2gga7XBS+{N;1_QFfHu&~?CPR}cS6^zCS++>j`5aLd^ z6$fE#5Sv+@PjZ(xcf))6ou9bB^K%mAgSdG6{S30yp7erv`S&@4OCbhA_F)al$;uIM zfD``IzpZYV??qJ9lk(2yj$y&}W=@v&z7;wdPfB<+4(P&wq7&r#i|GPcpg~HBEAijY zk3{lMVuQ+N^$sbfvixHWg(&(>{#*1TJ?TVC(FC>2>N^$Jfv_qrA{9+nOcTawH2VTk z<{8$Z*r5sGXg3r@Hxis*L_X$nn~#BV3Z?*E^vZ(W2XYDW7(X^||9G^s$Yl~l1@1?H z*r~~{0lIXE$G^u4OjWagP|%SPkHJtRtPojh0pizsB&O6=D{)GPM)ZEJBrGgrX3fi8 zC8=wFge?2%pX(zC*r+bu+NkpQ5-l3umQ(_%h`$XJ=J**?dLU?PF#nVbW#@*8jIjOo zm6e9TY;A2h?aSU^i<;;+z~DlXqqo~RTQx}Wub$aoc{4jD&1yY)x1ll%c9VOQXg(KH zRXAM36qB{e*qF=p|FsjWjto`%K8Z&2*$ZZ{}gdCidMEIepsYv&g z4o5|ri#9VevxvBOEXtqX-@5^D9O(FRnGlA@LFPICxoCm@2g27>Y#}+-=s{_{pBO<# z&V@qIaJ+KQsTbsjlpgpI;Ah)X2ZKRNA@G4m$T=YV*k6*<(Io&GijN~B_H^lx~^E$XI ze9n}<)NUYbTKj2GfH;&vyPqhJX>(Y|%{BGiht)kZHRi6X{ZK}d|BJzAMKKiHnR~Aq zg@HrHMQHgE3@7B;z?iZK09}0c7K0^_iR8pQHW|zx^z`*%zI*cdBhY)0pKWr#bMpIp zY*ZAMNvjVO$RvF5{F!dMQ4o0D^*Kj)9FX6u0H!Uq$@dQrHYq1dSn7Spxko^wieWr! zZvDTlJ1YW2M`*=g-$_4Uo~EsgA&)?h;cS!hp-uQD@(Z=DyUv@WLB!3b5WSPqga zL&Q_5{{~D!7^~AT z?WGZ*;sy?_8!JPZv(>m4D7~cE;XAo6s{fgR{0&IAX714*B6k$)qrT%oC`c^J4?Vv? zEBx2v=0y$7B=RJU+iwM7U$J$PP6u^@q5I=?f*95cIRIC|>DyA-5hoM@L$7{kq8t zbL@fYPV|3sGSGcy=-i7UI{&v9036fj74qd2p#7b{9thg^&;J%ix%n1={Ey|e-vxB_ z1sU=GqwFoCs_fdRQAtT<(?~bc-7P6dNF$9jN=kQkBOo10H%NDPr$~26xA3j~Jn#F( zi80P`@Xw)RbMJetHP^gm@CtY{LlO23!T7p9L&hQx>}YVY9~?)2Sf;Oyh@YjW(Z8+U z%yzH@nS}1QNcV01F57z(T}5{29pqooF2YldTkg#u8H8I(a#7_YX9iREG4OC(cy#*M zEdmJa8$>*UzAS`U^0gXqUm-2jDh71kq7-i7@3kJY4Mn?o zTdN#oWwa3p7G01jm1sPh=EBD{CsV8xt$vzzO4}b~QEbRG(9FtMbcna9`X&YN?i69X zMb~;%92JK9Z3yZg;W#P|=Bx(zUal&C-_O&TL0F2-73(F%i|@AQ%1lnpuFvpQ@_A8G zH$ymDbD96JT_+H}Dr<|%IUhCZBsa5eI>*g;?64cOhlP$r!Fhn z1I^1)gWwnTX_U~z2N9-a$>~h=jmPnzT@+Qc4;bSkoi)ZZck0lK!767eVGVCbY+|gk zFDm*ZMVmhfZ4*dA=)TP%DU`}d>dHvxca5H0pvD>PpN|RV3pc9Kg;1But=~=~)^k=N zn)1}W`89dFfp+F8iZrX^71$v_+G*Lr=XkThy)Dz(OMR{Q?}wCtrP3^@wiT1t|2*&0 zU%?udBU-H9KiAywZ*1NOc~)kh7>RX3c{S0F%p`oXVr>E{f_t?Od6$f&1TD@N_8b&& z*PH(KjD=MB;8&NwoEdEVw**nFdykwMpBUC*ku{3-eYb-5R4;#U)^v;tGX`PX?tAES6L7St69jwAL~N~R+y zgW9<+{>6=SG()XLHT-7i@?gsjdw=#_lmj#@Z-m^yKwX){I=Ssm%Vf}Br8)#;6SFZg z_rty!2})M$l2vO12Mr3L#vrI-zm;)nK3H%b%_W>4jMsNwLl_nGWYI5n=$@7#)g8p? zURS4H2aA(|Ui0Q#&5*<&kLQTr_xADI* zo2=E1Q$=3=i+&HRh4*=aKdn17UcO|L&o$~%s`2*zH#_p1WILHnjSFP+%Zpe;-)~&L zjQC8*cmxdUA3O4@>gCcIk29OhU|&gQ<>lrI>aov!w`tjg2bhV8V*H1-SVWNx<(PSo z77t&UZ0y^>cdUiJT#rX8kmabzQ|EFN{10;|J?F1_QhL1=MT{OTcsPGK!XfOWbV|f|9LmJ23P~+JNxMaB}{bXO$LHGyxzpa(}w8qPqGG zIPGjNc9oUA_9nkW4SlaT&3%j}Q4gU==|vG_O6+?i@MkKcV5#*$Zr z1?4|iDIE6k%nAzQ2`^*LviLgT<75rw;Xcr{CJLSPV_C5XJ~@3)p^YbU+tvK}4EJYM z@KByxz3>m*?YbIiV}WyCth)y;?1S>w&J&?%C8JZFsGEh<)O>Koy=>-f)O{h<4pdH#y zdQ@=csm|BIsvV&h|yZgj(Ni^&Dn@> zBW!_Zg%wt@}AJfX{U0%mKcDhp!iiX4BNM_?mK)!AGrnW$Gv{$St*2F1>-v3PWp~ot%{Wx^%U?{qk2_&gaqZcRxlCr5`1YIq(0c^r6}NQ= z(1nKYO}vi=y0Z-sGb{OINpa zowT$#<@cUpH$pDYoN>%bB=0U&GCNLXY7(9)GUOr`gy9(;)^O-8QRz@oD~wOZx-d8x z#X$#bm=HX`|6mzB;hj3HsUA^g?cW>C@bTz4-WPbrM{wmN|=K2jy8jBuE{A|c~B&T?Yg z0L`yfBvI&+$)NcncGZrH>#RH(1QyYiVX)z_Mn#HTI*w9@4z+p(l{z#bU8`A!fQ5Mk z{v+&1R}iNo+xo+%hw~KfhakAvf%0_65+(d}2HdX>Ntt$u8w=z{dFA2U-_KB=@L$qS z_oP?WR_1+?MJrN=Z{&YlPNqx+E8bbDjLF<911q(s&kSh(Y6ZbA?=fL+AmUY`PRxA^R%E)rWLFTGmB@^e>E3%2NUdP7$IY`klcCK@oWpUkD6?Rsu~=kt z2ADuSh22ys)L8IUZl@~@u8Y@le$X$Nl<`V~-u&0&jsn|*1vwfb43OS(=bx3mF2$(b8~{3 zI#)Ux*N$J%?E(o@x@If+tOKo9&c_#(NaHK`DBE#9*{@)AauFn|7i*1Y2JeQF2)W7F z1$EH!L=R^B=SosS@>km>0q|2&0W%rPK}zHNbaoV%%&@(zsR~h{-JPG|%oq%~hn~Xf zDEc(PU_%F{WGqAt@v}<48mlSm^(rFEnTAe!}!FOLp9%OC`dCS0TynSlMdwd3C2-!b+@-3n4 zhve^jV1|RF1x?-^`H~2B#cU-qus03L3Db(TUyr8aw^9w3>c3QDXpfG|!k&?sCDwmy z{QgRB_g5Un78jIbAGe0RbIlP!t(Y1uaF-pDq?7WK^!?EX#eru$e2A1o>IsbE%Y&$5 zw;&J@q)~jt`57y97wRnHDM_aB=5oJIeT5!aBqI7!|t}~#*b1%XW`;njVUckFf;ck==>Y+{e7=+xG$)o*G!2nP{)ipTVsN+LD72UiTu23i z#^%bWl(LKseAXHDkZ{4heo?|F-HGPU=OC{}2E1vW%HSxk1Y@hW!<5MQKX zdmqUenn&iaa*s!{&4a+)2d51#8e%7i7kj^c?}f3V-EJsPf3#FYrL&&OvPoqV7m^rAX#dhc)qZ-RMO+ zXsBGFDn2D#<|(piXbLh=5-@aq4QK$^VK0~{JzZ318~-y1$-0Ga9m&!$#FDpBKKq5U z>arPnfk|9j?UU9Yiw&t@8+r|9`>ewJTpep-YOLS0(O>`(bCRY8JHpALUon()`8yQ^HrfzCx8D#!BVTyGT~uBq6c26m)uxCN!B&nq zX#r&p-3MCpp-r=&jy{POo$t)-vmod18W0B`u_bYm8ikNKEX|KsutbB^)ffZ-hQ*z^X=LYBdCfn(I2!Ujy z^b+-{_(H*Df9znvVw%wPabn;50y(@*W?k)nDUwA=(U~)f;T5$enk0p4on7JbDN z)7!xyzy+8?ekg*15+i5*Ts%=j``cC{c*qYbKvVAH)39ih`3)w1o92d|UjDyaKsB-N z@=Vxz(Dy|iRlSx4LvZSvNxiF{Z(R12>bX(_c3(n6h@~hfx@}QvEa1X_ zypv3W5*_BWF!qZUnpwVrd^31o_}B=g;a|$Olfk<+9D#-UR{6!AC|H7yKK<`Ls=&(1 zr_K_`hh#Huv%CLmcKPdpgqyop2`OFSNKr< zn&haAA1dI_Pr$0SlSm9!CdYM{I+ynC0eLB=8V_Mr2hZ;A%Z9_-E%W!Df;rIi0It7QmcA! zfAL5vIg3SQm3VBc=ZvAIARa+vkgvUtR1 zd}p7fk0bZ6&?2}zY965Srg$x%2AbmtTlsW;g5LSW+$xdT_9u5WjU(B85IuZUxUY0> zLY;t7r~Aj()44y283BEy-u=Wlq{2E;#d4clLFi{n({>(btGu1?H6fzoP4PD_u4J8U zZoSEBA!AG~nHJ$OV=s;JXuqueX@2Xr=lM;w692YHfo{RV42=`%Opubcz&;MEO>$Dj z2j%Yr#-A4&xa9o@7j+~xf4bKL>+`|+K_CrSn~y{vK5Gdf;qM(7ITgP@GXTr}`D>jp zGL3DSAV`66A|40zsgd{7pl#;tvovxBW-1C+ujS+||4- zg>C7H*I<1YTGM|*b|Z8r*&XkIgnHovD$SI3U(w{(>qyNYQwtm=REEgx=DV-UKUY(5 zEJfRUEjsbDM3~1*mAiCi=K_nR<`tG=6f)S2Sx99GMaoJa;Sr}9qiCqlhv2^OpGibH zNc=^@9!`hzq{r#~^LwykBR`0jqlDiSkWJ3i=RE}0zbR9dJ?<^}{#yR}Q`w}auX8Q) z>&~)4EUH_s{d$(EzFePW2WM|ZAACdIR!?58blk>$ejfJy$p3io-XKW?cp$cvyx-sp zuko+^VzZl?;A8Rq)`w!y&Y?j6jEPKSMPv5{k& ze<(op!R>cFW-3Ob0WCq4$85lEAt?Hm6`1G6{6IhGpY`ee=DhF(2o`IU-e;O*PJsg#?9R`v@>;| z%1?s`?`NnX7n2#P-V5X+5A*J%86l4mN%GZHVBb3ZT*F^0NmB1H2bou{i}rX2 zQG7BeX??(ia~2exa5W1oN7J^qznn1}+>Ut*>&v3ot@*>aH0W2Z+wm{jv8djpV%68T z-=GxdMcjt+iwU+V@j>GIsx zDG;86-$`q?h690e^d`s9Yiv`&F%r{$HbodVK{8!LIcrZA5>eVX^w-iMpa=C+Ym*>P zEv3@Ed<|SC&|9tB=So@fNReP*$)NfcjvU5<(zuE4?>nTY=M$T$WnphUlu)4|yHTfS z@=?LGN1EY09Iw`*kGCeu-x^|bhc=EC!O^VwiFaMj1tznt#c=%m?FxFoRfp^Z?O@Bj z{;Dx6S)Ujr&lJ<@7r&vJ|0U2nCfyN5{O_h2!-NbrL}XIttFO)kYeI#H0lI_4> zEE5MAAJN^NG20J6YzOsTMG@9ZJZakDTK?X%d5gP=)3{9$9 zSXHA>mR1>2KNTW&df=xMMXjIkK6;*>9%_9dW(KOvyyW)lFD2t;I^+$@aq$y03HB7y zAdT6mQ_=&jO#6TG$fuu8DIaF41rb$iC1Pzp>=e!NZJ)uMQ=R4v^b%I2xg&18;NKYg zU~@XlqFJl?GrFl<%ReNGtfQmGQVrXBMX!5i@5J7-hEn;gFI&NQqtcukhQxCa`TwH+ zSV&v?kVzn@W}JigAiWz zYmsLj51t<4Bi}lb#zK_6r-1rQY>rjGiZxHl=dF5|wq{5VDpPZzAoV4!=!y|gVh|M$ z9WD&gWv2M3W8lyAD`fO#&gmQeX{kHolAewieH$1)U)^t&7tOqdV&s|6Zkzs|R6}Z} zI}Mk5oSCFJBKm)_(sVzKzJI3?$#m=r#LG0|Nj)l)ZB|lPe2ea>J0ABnt4_IXjIXp* zIZu&v%l5W}1l|Q(TY+r&i2zpr7@PdzK*04U-`QNCp>MD99MHz!KG4yVjIlkWRzPkn zd72-s7(4G#g*M~#SBM}M*CS_!)2!lSq+qn``f+o^ zPePrK94GeeSDNJM_E!RTssYy}z4Ob(u!tHAegG`5*GR4RJWPEy|G9^VGRo&)`8=(1 zYn#yJMoNpUyD;lJ^6|ke`L6EtkcMcTC&K+x0C zrG~!YO@=gG<*wl3gA=O4Q^We9L*~3>?XmXEzMs>HQXR4zDWBgMk4>229{#INdbU&X zgmtLebnhKmu;dpyM>L7*N5wfH9D2tHbG!Z%VbFw-AMwZd7*;J@Oy>F9<^?P2&LlHJ z$rg{xN6CpYJvQ+*?uj8O`-KaLYmqRMM32i0L{k9kLaF=D6WCGyHc#kmcH6{WU}Uv& zB_1gN3slpGQ7m;@*ozSwAZTJc1%r8HVJkQo2V=P5&%~E&q1zjdH^>>@NLh5xxyWM( zz#3L5RifkRovZ6yX`x^BSHXJt)kM)+u8civYFEmx(CF3O$uTx@HkaBja?j}{cmkhl z)7(3xsJ~&QZ9KL7c+c`cDw*PXurq?H&7Si){#4LblqBwGzH*#{Ew&uY2$Ds&F#xXV z(8Rrg3M03$FvbyaM`9=#+KG>hT@XQK@)6zhGry5*55M z1wv3{9%_4&=X7sq9Q$<5@c|59-J*QuH04tNnnDz4pKu-# zyCbYjO4f#>azAyCMZ3BVfc}(aHxgeNGo<|Q{Z=;i@ho}+QO;b$5WznHFCK~R|Gj|{ z-?(4Ji9hTT4WlD(dLG}k2oI10A?&KNC3ml8N~ku9X7R{{o#PqHBN0?33A;K+;?r{Y zk(Fd$s7IoJUwD9rl9Q7&-hK0!(-+{^iGZeVV|gZDsjFMooQzr>&r3cYzKl;4mYm98 zaG8ya*(c{Df4#r0%+Y-OyH7~j6o!}GI^osg7>Po?fty$U%tYHTicxrragv$fuazkTmxcYd zb>=@BEBB*?YT67zSyGOD(D|}C5HDTx?h}cyhi3O}7QsJaw3ip)BFS5zWNUz^eae3q5UtYo6o(B(Bj(^J(U%$}y#8=@IWe zdhMhXjPCQ||H}n*A_V?yTM^gGkH>HMgFLH$>2na=&TCT5?n`9QXfsdtGP~ag#$HF4 zqG(7m?CRiocyHv4iMiklsEo}=6m2M6abfC=Vw%NmFH>w8WuT40l0uZ)yIt!;Tu(UT zCMhN+EDsM)0=*i&u9V@&kMKJ-R;GV|3LAL&^5mM9WVRtd2FqTqar~MPvRcRvlj8FO z8SkvUP11gMFDvnlFr(Fh&;P+c8)OllZ@g17hxckZ_N{!Lk#$%VUQn~nZc5@Ew4o?M z#z5)W8p688ZZ*~6jf_}hYWi6%}<4Euj`RHS; z)Bwj73GbSbC~a!+^+bWvo;)fqOmw*IxX@DEouO2#8WAYVBtWi5B~TNwO`M$ALW{IO zlS9G#?<#<0m=6I~MdO|QTjx3%cF5bE4 zN<~WW&2)j0o$?QP0$yHT12F!eU;v#fm!Q!(f`9|WUxn+nxUF5Ucn7z%2n{B&M1$cE z1sF?$;*a(w=Rd5TcaB8-jz|pZB^Z~`H^k_^Z9GERtWNpNBgJPMYus^-qE$eNJ(Epp zhB=6f_-&V&6V~B@zLus%!eAzY!YS>te#PbzYR!uM|4D4;K1oqxJ)$PGw%gj-$GOul zwKM_X*kAlE6?oLr@fDy4@ZRqa|F|T;v;h;kGgpYhZBeWK73mMvCPKdS^y{xEXd?xD zL2R=Lr8XKBT>cNX^&YuQQC|1kt|1ZL8Qub6gs0UCs$VBpSG{LjLw0+U)DIwpNfWe| zfpoes5Yd5@z1kZ|s9A4I4pN5TSy)&g{gZ}sWxC>=niU4V7+AnnPP_lV)dFp{%3?r^ zLBQ39l$3P%8yD(5=omBvZA0@Q$AW@_!uE7sGC3t>4gABLbnnO8=OC;?0{peuU!@fA zJaa%ugSd>v)DQIDC}O!0`*AYZkDb4{*)g?=UAz{q9$yJwiRY7E4Uz{tj))wjJl!jd z5um#<{%|RPGVqKR}#r>jD`$x_$egob54-gPyUC3YLk7 zZ~cKJA!&V7MQLqjd&GXv=uJ?2$!FbLZLqfFv`d0sY9tw+@&dM9i38S+#_24iDs=^Q z=Ju7=&jdbOs=vtH52zWBNBMH$9fpvCrU${n5s^SJ$9Sif`f~X$^lrNMb<(x$&)EMD z0y;$Z%l$x5TZb$cLLnK}-qi)S>1+g)S*$!f(l&d)P%v=|tn)6;oc^n!6ZJ!^Rs5va zV2=QDfZIVlkmOHbyv2_$EPUNh?|;X^#wG?T93}K~Fi5M=VKV5j#wB2x-fl^+?DuSD zmw((zXGe|tNaTx&%O@uAwFFn4;GLbV0nUSF9q%A^h*hH4EFP7f?k*=QOdQFL{6V?@Q|c+Ymm6lEzu zHR*>m+4=(QT6%1hz3y6?R~y?`Jf| zHAw8uQLmLe{~_#Iyt31Rq`b%#c5p^>-`sczzKM$4TB|8Kg}Ar`*OcXqQr4#r5RH8Q zIbXOlTk~K@i%q5>kRbe{Fl&W??e2yNG2qs$yzKz+##;@9$!akhM6%wnmJE6&Ehj|i z>UuuE53%MwdW6(CNFT>^r?uT)nCxVBCgKUdw%hr}Zs5$%yw)FxU&*I5k>6fYn7~gwA!XZ@4 z7+REXPsQ*8WOq0Loet_ESHPXd>$nm+CTjBFETGkWr|!!9Pd|IV{688jgdxG4A)vwH zC2^%#2aKfFyR6W3zHM)_=mTTR4%hcWYtI}tM{@ep->~$KQ&9T;e%Z{r6@psy_=o-N z6BMC-d@{Cy4+!X^BfR%3VfGUelB}vsCl0&%!?M2bs8riXCY{sp_N!|C+NtKhd(dW$ z%1TX%VA0`v4F}+TA=!^c%bNx9L{4F-!XJ@fYH?V8a8~yotMnLaM<6l1V7K7GO4R>&WBxEG2^ z?EUMkOlqmd5aW{FtS()Qim46|SHc^K)BlcSpta#jwXI5Dj8c?bKgnpM=NtOEEwWPA zX80GmWUIr~*0M@{B6NVf#T!m9s#IgPqz7KiuRnqcE&Yi;y${iC0D*X9J86|u8%;E^ ze(9;%5B(jy1vk&I563X6-})|V5=~-Yv{u<`wblm`S4?! zn_C4nFuh&f-~Hyo-5@;sh2DRSE4*V`iDlIGTKEFw=hY-tq-Sh)+fvE&GbY)EA8B{@3LujZ4VbX49tzFRuK!S3jMc1e}hD&_4QlT zlS1zAF3*%og3hlB*wLWd85veqB*G>tQz(_Re5mdZrP3bp67q>3N|COzDBRk9^g};7 z{?q&VuT!t&(1`z!hhH>|vH&Y9G1Sts|6!-+aGToJLiJt!yVB^$^dSR@QX|$O0wx7j z4==3W%+`tdGfX+`ZyEVto+VZu_?mrt@EC~v!6vT2Z?S;HWo&JQI;7V^H~5b|lMD;# z#S5wTxa%BdquuWlkFvG~>0?2Nni!+2hU`>t0XK?})|X7+`=UQvVu5aVKFCP_)kHyP zX-CeZnnHL_ojCU_meZIN5F<&2#7uw!L z5*qj%f$GyEx*Bkb#%Zu?2IPL1^wS-hz*%XcH+*xDa2GMELkT=CmCc93c7xCZ50_Kn+Z02~^V$@@<=-%T(oD`RoKNZUfjt%JE8ZL&q{r&9df) zh+?QX5r}n{=}C7jUiU6RN>@)+-lP$z6LWl<&oO12;+^t1z>82z(#u(CyH<8NCU9JK z!#h_L>H890klwSdDr4+7*WquETGJH*s-FR62Ms^5ef^1=^*^nbHOA{e=)GkzIT``{ z9`Hv+Ajwi3?G+~ys0FnbmTY}`bm!&g_j9tOs)V+5i#JzPRly!5%Lp*^Y6R_@O8(Pg zUZ#t>*un;y7R3y=fcjlL;k8B?5r{28*gA^|2p~D}wxG$vS?{TuDyvAdQw_Mp)6XLa zVxMs@p8U@ zCjQEYI?+dFpbV?bRiF+V7IBD{7ac(wM|}V}i>DjWB7RO1pz)+euoGAcP{PAN)Z_+m zh=4vz3}(G%x*ZYkMak;-@Du@7`zzUhdOgk!6d?p1trh%Zo)d1-JL7C8VG<5XX zm6ch_Ur}cfi8d4IV-4Krkm{9OaaPj0$b}v4a90KKMJbj5fW^N-MF+ zV1lo|KO8QL9^*G{-~>RM05Uaug*QbT(=PHQ=0|vcDgJkRj{qDZnooW{?cLQ8!+&~( z$GKr&AlMt5$~|}01`5n}7P^Csy)#iTL7bFnH$FFQ2waDvbEi+Y+FWrWNic5ei~;(K z3Ee|`=(b)G=);zl!HYnOs2#PQF}Xuxf5B(l2Zx>Rj7$7fPuU$&TrRBQ6xNm6_Vo1t z`*;Gb_v-4bmPX2D7<_4Yq%ihHh2`ZedgFy^n5?_<9RPkONUJ}`4wCDn)uK=$K^Pp1 zr3|ikoAuy_fNBKlHWH|VHxpt}&E9|fw*(y)6H6V0Grj;ZdZ)|MxUWIBrIeAlA`QB3 zanl6KsW|a52489lm(^D#Y*{rb9D5K+TcF=2?Dh9j!!-n?USS{fV6s?X^$`itFoo{_!&V{DdT5{wtxfXaN!`9AMk4SoPn= zL^?h7Ee`AZ3;JMb-Pvctw>vI+C+iZriZ(_fq;7KET_XxrhyvfOYnu>#ab@Wa4>~AE zWi$SdzFse`KUOuk=>ZC`ZevoK1d>i;Nk{>C_Dc_P${a&*cCE}u#5UTZ;)=tbbFz*Q`|8r zrb66-Ubm4G*t+1VXtJrfRM&t$Pkfj>9KBH;$8Go9n?)q^0GEjzRhWRXL$b~@g{}4B z|L#tCvmowT3x#M}=^8z5iA6 z@1+-)CFi{sVP9iQ8Xi#morNV%5q8GcqJJej45BA3iS)ddLk5)1@OKv7VM6Y_kN-Z_ zly=}|8?UiYS=(3xD{%d95WD-;VQ618X}F_XU!hJzl}k##AWCCnW3rGtH*mVqb`Ga- zuz?ocL2+X#{qpi2%64hX=pXMKxH{gG=0IKu>8&mk?KE5Z(w)t_k!ni?VVh)xdvFCKr`|cb>Gyh!vBnoyjL{3no=T9G{(+uvOn8l z4;bU;?-?_bvMOS-e6wzb8EP;K2<*+QTlORJdvEBB6TI5og#=jxm{8Oj7VuBl!c68V zf-Q5PhbEljW47q*_C*gP-{!;ZPNw8t8lMBJVJAF<_IYxGH16(2ATwWQUIb|1-2hF( zo%WR3?5fNRa4q3Iv?b%C^xy=OI^P}<8~S^6baV=s8`LKfWMm;NzL%f&Qh}E{a%dtQ_JOFWT4mf#9Ml(-r3_!BQO-Op zIrHf`^%p`#w-wdTELKR(n$?IM%gZ?q8<^BeRQ?a@TU?XUN%jbp^=h*93Wd=Oxt2x> z;h5$I#x!^8LLO*n;2e2c5J2T+D0++(sh*CL%kQ6LLXx4BlMQK|wQ38CETuDoqYnf7 zwD+NevS^-IW0MxY#QVm_Yrdi3Q77B$<%VnF`-K(oBA6+7m>oAbWVo-7k|t#hw#&Md z^r`T=nVR_c%@k$({n1c$c4?3lg_mfj!N9`uhxWGAwgHyMU#L$-Kw9Jlb)jw(Hyz^1 zJU7a-p8}<{ zSPfx6{QB zpBs9%81j)qe2@ZgVZ!Q5C|iE|#m${zX*+WrEoBJ0VBF&|b)I_OPrgV9<#=M&f3X+} z_naNVZy?4lPnYke`!V6<7AGvr`Pbmh#ud%{t+l(y7f(3Bd4Al)uq{QqJ-*zwFAqVg zN9F;M)ZX9`k@tLUrh*9+1S_&2r=%?8DK*3yt)Sx zC22rzF@#bod4PMNx--gay*u>gY>4IPP2NmTSaCvq5~+5hV_}L&R~6&<-%)`-ymtNM zu;BmhplRoGZRH#w}GFVj<_r8;16_rZGv+JT5!^DtWXZ-t;$8&QmG{*ZC}tfO3xrgy9k$KYPqF?|1FwTGE&yj2^SQ%eXg z(Yb_+jd%nrQvbl2Yh7|I^J~uHQ2fN->W1CxomcYOb-w~8q~ImM3ndEgCo8%hn7LQ& z3{Sg0Y+Xa}rP997SPK_8Q8%yRKEO+m$s-Z*J7F?75wSY$JI?MjEI91);(lHT{&Wv_ zZ)IU^&DeYUDlcCly)&83xc$q7t6Y5!W>c{JC+8@OElYVFPm92EHfN9@h(^E;pEe!F zK=-{?8*9VvgXKX9$EadeMGls`ChMmtX@GgkwALfAn5M=?qXkA{Y~2~-oATdv4Ilrm z^j;HB6Ly^(`0YzA$J$~neRy-k)5k*D*HLR#d2`&vfwDbtez4*`Hh>q1I>dF~>Og@M zg}8;1I~Q1q?`mb-j8i=@&{4y!u@<7(ZY$^!RcZCu(T;pDejCq6_uIP}!Y!QMLO^&$ zbd^5S27Rr;5Byi3?(p_AbxY`b7entm_MrPOHJtofEI1^iDTk=sHw#oKX~8^uq1$> z<9A$tf#ExK853^47#Bc;e8-ww+C%c2%Xv5|4hDYZ6JC&t zKQg}rSa9RNcbB&L)Yl6T&GSjQRa*!HJ<69A7dX7p6n{(4JKk94=8YF=SE0I5()3Y* z&dZ^%-=!KE9W7?z$Gyc+bUz~ELUH$?c)=n2BJsE(SV3_?FVm&&a!N5xP)OuPYuGgl z0nw@0H)=+X_wlFk(*%}ObN%&>Nx|(tw&T4u^W9?u;)W+|FtL<71Oo>8(=Q8)$zmUO zoKj!9fG%$M3V=7wKoQdtdQ*VotE3F;Q^{kpb!1)~NQ+#6M@FVwB7x@cc^>0v4go(Axh z;-sS?j#*#HXMR3=Uu7(<>Uv|?&G|WE5$<_TfnvmJXzs7Njq$~RY!pF-k;hYH6?e0- z-og#ATOPT#9d!i@xRxw7@th?x9}&6tR8Gyn1;sn!mgo^*4&wH{DK$R*#S~eY@BQ54 zsV(S7l2k1w!uyY@IXy#IvjmV_i8ec_`5TvQnv#`LJ%ZN;18?9vL!KqUZ#}4KF-^95 z-nV~Ot#(WA4EIU(=|<&2|7YVS6NP`|b3yz4Tv6ITsW*E=K&5h{UHO@oT=SnlLme(I zYHgouE6-9CTsFe)1UYBDr|ql92GD6MEcdMvTCbPgrJBuGT7~pn)(|SB5!r95HI&x( z2M7Q#ox#kQ3scBjMHi_0!jQwDE)g4qu3V&& zPnANiQ&;9NY&$AOAJx0!6|Sz5|Ek!w3%xMlq--uDg`;vUH#!YX{bE((*JfLlz6fn} z;&J1U-LI$zSNpZN#qT2z_;Mr`k*+U)b{FH#N$k9$j%A=X6z3N>>@cQ{bg_G48B)Hl zeZ70OM!s*2T;ikY4{+2h^1XwPet(G5Sf3&mZE9NwwE*21{1D-8?U>Bjd)p-4mRBD&$0 zKfj$|u-yv=cK>!wM^EopchTCul?={zqoz-p;s~@Yn=JIe7Y!HeGS1S1{~K&Rb=vX7 z&&|%D(YL&(v_~ms88AIBGIp$&wI@QU{A?z+hH{S{ZhMP_?+W6xv7R2o+nye#_?R_w zgEkaKI+kDce5i$Gdg9y3!|ks20_xl@>U>23^#ap(V_RQxKj`dDz)>&$9ASZt?NDZ1Y>Fcqb9L<$Vsd;=sW31T+2Rcbj zSLmtw!uQj!pKAlOnVhIt&^vIx=NIC(aK1Gtr`chc#W(_TU~OmAG_t{544CtIoYJ(S zfRF9)-2#X%uErOWG9(aI!X53D&%)n!|#nLN@#~XxI5NHQ6i|UX(1~mP45IPC^N= z+3p^yx&wwrw`V`t^Z*uOK%_EaLV2E&z-EJO1;y_wFQTCiuOby2t&SE3ns-+XS>2*lgRp- z$SJa&$L&RKyWbBPZabACT4-MA*0?$wg6Ni|*#!@71FW4LHzU8r@#z0D&GtLLv(r+_ zm(3yKz|XaB4MmgoH7{%#7%vtMep_W1kFhA0<&}Sxj>=?T?|IY4EZs7fzy2ULbuvs^ zeA-u}%WI*bp7Ks}4SFWN%}Ogj_B!psn`m)&7Q7gLW14deE*j7~+_pfn*YIkpO)K)K zyX4dSyr#$B%ef0~skI!rB<9ub1#F>7R4ZnraZ_-cF=QHsM>3A0jlUqI>5EqZ_=OE# zRS6=KB&xT9>NP}9ZSY~uqhSg?b8$fKAgN0YogvHB+Rl2}9Y5DI2Y#nfv&JM7*-P&H zt?AgviR6b$duef&X!TadmDYVh!C$qwIp>?}7R?+M#8E^d7E8X1nqLz?B77e%nN1LI z{%nhbb=BQ3FI=0WVoJBAlk9IbYhYQdKL0qxLK92Ts=qes{^YswA`#skM-7yysaB;L z3iF_mMC1x!xqYz92uGRIOI_n4Kq$~EB0K)%%Iwv;=fR=Tdy-1HHt{o2xyf-*k6_5N zbFVmlc0=UNcDCjU*cV;A7o|$>?hT{{cALSX?n6?_a@mZP9!4=LZ+qmYlV=SFr7zv> zzHq1e#pUZc;mqvf&;RoMk<`&4;_Xg&ypUsH97CAoO;WE%;qcu|p0bc_?ziX+dtyY| zeGkF#9md4xcc5gxwQZ~(M3RhkXD(USX?6VNmFBIWOz9ZWm`{EDSp{Ltu`qB}nB3nt zDpN3U#!?U?a99!EDT*j`Ad=8{rP-CHH;$M;Yi4r*6k(U}&C>9~uZuj4%kus-Q|vop zkv^tF)wTTEjhbF;{P?D{qF%!at+9r?agI4~cS%-XAo_Mc9j~`kvV28-wVCHh94=Fc z%Z*E_?1oTqr(E!LMjCm=0PVN(C07iArxCZYJgj{0UJQxo&u(7-Mt0q@HmcONTR~0RZ+DqJsk96NGQE zk%T;>HxA2*APAkr11A3o-0MbI{6Q)OJJ$Fl3z7AGd7BXkV)#MVfQlpjod|Rp<#^;|tBy$yvI}?q1f1>~P@>`I-XJ1a=T(~G( zcT~vf%6S*&_;LoOjl0z!2<%nBFOkO^$_%&t*GukyT4>CihzHk_L5oNan8=-g4cGw8 zZ?^^A-@M;kwKPogNOdsH!Qa;*9m6GrZ~ah!uY zXT+UG125W=JHj@`WhvShL_)sBBB_CsiV+fZFMq66uF0MJePHTOI!sc+(j9^tJ2$67F7fxlocG>@cYA82_)z?@h1UsP?>v-P+G9O9-6HHzgk>>mNH{&*+jB+j zXWP{>Rl933jePPml zdzn|Y!;S(dKf-ByI72N`E%XDP+SHHW!@mSME12cYXDD>4*2~QVhlOmP7C8lPR{d86 z8Lw{x@UeKidE9q~JA$G`oZhYlQtK2a(D?|Ys&zZ_aCaYj5uerm<`kQ4h55==NUTrhx|8)HW zLHm|49-bq1OK~$pYY^F#Pw*|{!*q9#`@?TP_>SK$<{PrGhZ9o1$;A;5*KJQdpyh`R z)CU=Z@d~_qopQbAPiq1_uUnG};Vt-d=(W0>8<)OLad^A?mqnmSvVmqjDhG_|H_8F9 zCqqiYRH3&=`>&$QOUm_^;jZ;|yF_R=#wKyy+=qOJM0g{a&x51VyXSEPBK2$E2eR&G zsGE+4BjOkhOJr_KaP#P>eSCbl-P8M9S>DlYSH6eHA0BV*bC%>FWjCkm+@V(#*;mz$ z$?Lj{aW$H8MAf*JYEyAIA+0k1*c1LPIsW;}Zv&KkzEr<^rxa7;sO7#T^ZF8*b(uMv zYA;lobTAKLH-gy*+@qh2n71Fk5znL3BF+w{~|LbwLn7-#u zgI}O*5U!1o54boQt;Aw2&`&%t-P?unj5%Na{Y0PrX=dI0JsaZ7K5B)KAwJHS!Id8^ z!KWAfYUbC=@;Y||M9rK|rHT4ES&$xWD31bq#}b`~6r5bpjTc>#8V`OIQDo+fWHMdS zG1*ki2|WGpU+~!27ed+H2q6!BM{PnWwy&iWmLFi6;~t4J^H-52?$&qs=O> zoK1N^0G0qaSdu>5j*5;RXaDo1fN?U%_KlaCa5$Y~VJ6by>EC^)JBh1T++Ks?VdnpX ztGAAd@(a6$1?etnP-&#QTN)`rq`RaWq(Qnt8irCrkw!qeB?mzT$$_D}q~kr~?|I(u zTkl%>k4wkl+~+>$y7u1J-Udz1xLA*-jXZFL2fl&EqY$Q0Qr^9hW}FKK?VYPWHWN}& z#Q|HdEclX<97nujmLtomZwpIp|1tb)=>}|PhYEJkvBz`umC3iw_>m?|33A}oiVq)S zJ!)PbL1p>jlDC!KZ9*CMm9etsy?9$e;g7+pPr+oQ7Gi)I=%e?p~vY0^N} z;R#=TVN5=D4dJZ5IM?R5XW@t?C7$tvvDgLQ`~PHRi;8-F8T19FNic4ywfci_^K@Tl z#@yLXn~+TBS)@noy5h$bOv|~JZKZr9ORb$tsUQ1$tcS+?eLkBP%7gj(=$T5EO*gDB zOJN=0aBrM8iSvL;kU6rr#|#DCUOknRq@|vEdvRzEBx-0kjk&}Ok|(2lGo8V~u^-4y zL0N>bukuzk4z!8x*3~mcoohGmMdQox?u#b80Erjfe@3#=6$T*WYKZ77o5hD$z?dI{ zI@hp5n=y^df963HHllpR3Hzv;Sq#Ko<5r0SFLT)mS+?LCcV1E)J}qoj+(*AR=cc!C zH5+|g;F$q0{bAC~YJWz!G=fBXEjd4rY?J`fweY9~?(wVH&60%EXoej={ng13;W=d* zd@CT__W@+)Y|hk{WRH5BZvGg@xuReqr9-;-v%30m%Q`cUW%mOaQ<=YR%T(gYt;$mM zO98$HtG||HE8%S0#kMCNgv9xCG^5Rb*uqaN`2%f5-*kwgCM6j~6lCsldw@9RS*y4Q zQJ4JHIwTFVl27SAr|59#Kgj&WWf|XYe`nB%ZaTUUw%Eyl5%m#kO@6~5rZ|%3J~sdU z>iGq{xmn%blH|b8dHMG(Ill-*T_2Cw?DU3qERjb15dT$xg^0K7J*O2TZ7Bya5~5$v zeRuW`XtY47I*%bXJ+lk5Kt0aNn^VPJkF7hji`pg+aFT;V$H;_)c<9g;J_mAz8J^R0 z%k6zy-up~BF)~x5Pc)>uCod?(@z&@o1(`c>1|3ocQ~aopGNY+d;SG2(qv=e+4+iroe2dn)H< zq@{op-26nX2}IT#i|8TQNtJPH&i*xtVxBaG@Ey2#i$K1*89e=xT+EwY7#}Uu9sbT7 zY#08{IzF1jvD?!r4Y&SBdLiw+RjvD?n_S|UKp02W=rD~|z~+N_sWuw~bW$e*u|$!& zNBYZr%Xn17&auJ3q)73L)kvpyANfCW?eEWx@)=CnTG%kTTD6^XO&su-okE8=M zit69EDvfXg#a{JB8&X<26<{BwR1*gL^|RKxiLcpF&u!<0Tv`g`bc#LvWeSNORH*91 zk883#SSH>#b4L%6{;Q2jkS|um+G8F!jc;fc`Hn&3o52vWoy)t5cYOJ8GEn?O52HOq zoz7y8JV)A8pST=Fs;?*RomKfRTk$>`td+o~SqiXTp>CoMWlYwzL=k-PkqLpn)4^xjYe+Z8`UxG zwrGi|&(DJIZtOV~9q2G%rLt=L&Tov6?tS+BX0ANN;#*IU#ea1?6TPNvQ#xxJn%)fb z-$t9i_)GUbFsN-Sy_O}IV|0mhfGvx`l3=t~AHS*RY(sSAgP5yF6 zJ%MUIjj5s8*fET68MxnMz> z5{~Aa$Ngl4Q1JwRUO4bY6}fI=-2$nhbL5wrGK`a+0ZJ$jdsH%ayc$YRjP%=)&L9wl z2)iLOQpk|Ao^gng>)1W^z@r?0m^!Onua9XNg?rke>&PQMRr@TcrG;>%z%EXs>Qj`z z0VFOJ^`!JWmcR8VmnIpFq;beMm$9Cg!BvAvs`mr?R_(mnMl4BcgjuwmWKx{-&bxgT zQ|pukU*lU1U%9^>at*@v$M}E=n{%A;=xcg$)#>L_Y`P_TnF{%tvVp;{GLKTlf7=$PMslH^3Fc|0)Z15U`3iJL$Su)sLGdeA{-WFJPH=)L`ONNx7}GX43y2malim>VkT05fr@CUX~rba#w| zh9>;)QPlrp0Zy?!%i06WbB?nOLuyLHwCeoh4_|f$1ukX33x;giMr_zwL)is#T?Nf^ zv$-*!<~wOpTVrqZCL|2Q6ZRWxE-Qr9nm@Bb_B9_FJ7lP&6gZ(LE{A6ux(-6wwD$f| z`1&0yw24|1uRDq)(^ZP|Dw^GF8#n*Wl^=MuFTQEiuyN>iZMkaG7aiuSLc_|*=3GkVdnWYH}N$L zjL;8_a?+_fbcmPu3G%XRNc$sUD5}3EBC<)nR+x2z#%JC<5#Sc_fawe~9zh4Ors~H- zEu1sDw|O4pJm&LMSTr}NS<|(dC=oc24Qh&3?>*$!79c6#uvwA6TJnV_AbJy!D~yIk zVD#giIg6_+okrxcZxHQ?s9?j}0{uT>#KJ)UI>bSaeFCD49{LB9<2ya?nnK|%5W&K$ zt*y1WBRBaw=hEqB8-LP%2-u)E{->MfOw%7H?9{IWZ;V`{$@B;;$b%$73h#kf+9A|L zgCas1W2@aMPDMV!_vmD7!xEn6c5$0!M#Njl`*JPnP4Mt*I8Ra<-Eb2=T`wUS5s$ zJ7Lb1Y92e;2V6R6cV%2G_ZjTFkb&k%`@70zMkux0wFjpab8hf`(V}1Tfg$<`SqFA6p zle~Q{FRr?(BK%i-&N6(&VP)Uuk>*!dZJBiI5?7fWUMs>kZ6d@;_qBLS?6WcXk}2UE z)|e>Gb>GXyxTK>f840ZQOV8g)oXd|*qVjB4VRr?}ZuW8LQTi2I>XNvC7 zL)w1N`kq{qQ-k>X%MTf8o$%{d!uy)8&Hbdf6qx%oX}*L;cPhsJ?usrb%H8XZhkmrw z`!iWMg*%LIM%{m-Ss&W;985x_`(8J|kw{BFjCxU{cj_*!*@fcU;SvDa4 zRoSgpfKE9b`2X<_Fz%=RsJM9cZx-8Hwj#PN$g@7r)(FV}{m@td>(_hW-0C*M(NqXj zcs2sKf)gU|W2U(SBW+zEdM17mCNKi5@!ZK8b2-yURL_JwR{rJu@Nau|NUCQ?`=Y;(XD`Z7hOjblH5dKlAt!1QB&tMU^Atb@W`z)x61$*OiMvqhLNlBsTfef z2E$dJ^6uLbXhuEvW`CVh!x(CF1)X>|L{P%y9(UZH8M;=HsE=MjrS1!XX*9X8>l2f} zzq(WO;bABnGBL5SJx)yvXjtTV*P&^;0M{jm0rB&&zX188Ip^xUu|P=!5bM;#@GWkw zv(a+ko?NP)|K?STIW!(%eI@Goq8C7_IZ>wZ*fmuZ>nRQwwkfiFi3j{xl<~;>eEaoH z7iyNPp`7>}C+UJix{}uBLX-v!{>RX5Df(CW^>6kvy#Z6IvyT{T_=Ffj$VH;|8$uEDB~O%k8e4v_N+<|4%lh}?+G_v zCr&&y!}q9R$EH4mjbDQDwac8IghAMT6@1Tb$fI3(ApOVJoG{bCsEU*eR@WBs**EIz z?+@)kmX898#AWdOA4X+y<5ijTx~kcNX!X(!)P)Rdk+d%Dc{55J4a)%ViZafKz4MTB z)i%H)&ouU?;7XS#2nEuL<)T*`u=}b{2^WhThnMM%t}Z87<>YDJ)$yv4913gGHn7?s z4m^(Cc#eO9`NN_s%2S>{Az|M0g!&K~ml`uHaB5`eu0$}q#PKpB6TXm+AVcaT*jy+Q z75&L#y&~$ni(*1qKiS-f`51YZu-k-~f_fsfSUxUinXw??7*ipE z5*1Ho3wX=}gLovozp!me>SsNRR2mY0N%q^%O&&&tq0nsP`_KxLymOAUtOm9_{MH~< z9wttq<^#lcusq7gfu|{3Ke|n*5i*~4UY7DhO8r$rmQ*gXyO*sDkb~D0*aw{t()NWP z--`CFq#7{3?1I%56^#CgL=QGhxN?DdzqMA2(W{OpsaYkgLpg3{+VhMIx^0#Hrr_Kb!+{33`w0I;8vrR58 zOn=)z(xsVxhJH_)hgpqcAJ}n;;ABRapd(^NK`CT2UeBx;N6I110|zukUq6{?Jig>b z*E>s%glQ9hTw-Q*l&DJaqdsC5!>{be>tk`iwiS_jZ=L)#VleEl* zyPz6>Cmy>0cX%aZJw4C;l%{3gZDctR8F=OMvL$er8TDIE6v}uOE2YqxLD6aXon}eE zbm&8WiOSct$Y+|kSY&_z?*;d>N8u<<)@&!`jtu?Jpt>b%yTI(FAaB0Ofq*=c8V zmv&;|+DLdd3p)bbBNm$k)+~MJY!^Gi&hgktlf$n(crShKXn+;J3Xs@IyD6GjJgY@iXDGSS1?mE3`9poD ze_N04-h0_5HefR&15SCiZJgC1FG0st>H}iN5CTPC=5>+8J4r5y_mu zixu79%r#q+GdPDp)O>_HubFR|#cjk-RG3+zSYy1U>X&<+2)a-R&oqZI!MQ-S+4}dH z$uh={=5K=gE^c_ANT3CDWZ>v-T^V64zc}_37k$GaS_6G)QQ7@`V?zMqQasWp z;&=S^Anv82&vIYq5mVHhZd%v_lN*-FUZT|-{wpmX%&x-=;bl5f9QLhooScs3vdS>Af| zIGZ{YLiwhoEvcAp=t%{Awvx*$gasi#HrxA5DB#n3-CJs_AAxc?T4w}>7Ig>-xrpDs z%-;m>@3-%vQBSF~6rz1oaCyHy|F0c(#1Qw>dhwIhaOS=3!7_ZaY?(eXLU4=t!spVx zvdWmM-~Y5r>WhK-L8bndDZz(&19d?q*u(dZU$jwOed@8uJ>3KxgW6C#I!f{On+wH1 zr7N}S4d|M3$VY@uO*pJly3$QFw&J6+H0i5G63@~~?vAxte__DN*GKj>Bwr;hJ3IX{Kv zQl-m7jFh7kX++$xpRBKEQ?~EEQTp`(9mb*C;!hYMHr7VFU9~tJ}n~k*#^uk=7 zAtwXPZQ8VKWz9kZ)n_;GW^R*QOHI8&wDynD1XuxRty z?aI+Jacdhf32K4kF)Dg`@{9WH<8UxjsrYc5_-ep z=F~=kct(2gW1!XewZtwTky*C9oI)`jijWZib}(Q3StqrJh&iXgsBTal@iI67;zc~x zvs6JiZ~{o(fCF6*&j=}%hLgB*7Y8!+`NfBPbanvPf!7};0ZLSsNo%SsqFXBPlsyNaHy_hsYCV^!T*Rueow z05lA-Y0xEOGcOb*p(6sB0o49*SH90Wih{Yf{bXDWQRslNhKB=kQk4FBH=s&UsLA5B z#sUbKWd>uZ(%=d20Z~nzq`j>smdEy{bDprPO_&Rt&@t(BO-kW>v{ek#m!Z`BpGE(z z2*x|&Py=f$cs6Ha9^K#DK#{Y@BFoexPmvj1w=di({WoYt6Z#kZtN-}Bm8@B{`|7o& zFptN$`KP)2!quwd52`=MMH%Wo-_u6rO_zx!15G%Lx7!E{1f_JtwgrvCF0OW}2La2j zCXR@EL6@veQbp%7{zg)73l+SZ*L3wg>_$HVYOx%~#4W`gFr)vcR%)#oCX^e3AGHhE`LB-$euCE!B~x0*HBE2(zt@zGz!&+vp<;#MYBDQ5 zi=uN-Bts5`#bk!0ZFa{3I-2cG8vYGGV$CWX>w8Iz9w%vnAywvx_P%f>{9~pGuLMg~ z%R1%8tcKL>&(lKK4KcO0?2j;Jk)Nx8+TuIWtkFZgovZ^QkYZ3*jBkN51O`}-^=7M~ zRGaD-7X+?Qf48fjgc_u~t3_k#9qwdR>&+cr8B&JwrGbA;6#Pv1k1Xtm8M(sy@36h36?_-$Edns?6X;W{CUDNsc z)aMnUE5&T+}|vENH@P?vInB3hLV; zjuOvY?wuPD^&2*<425UVHzP2G%ua2xf0zOWQK+Jrj*m8Nt2=Y#=sb*igs=rgwrINF ze5v~94jQhzvv$k<-LxC}@Da2xChKR-Q-%IMGwpCRa^nUTrr*^`#_$2_XV`gqR{c{N zmR8F>m`feLlWvl#N1iJ{ues=L=4+yQQFOr?$Vp4M6}>3p{h)ZRbl^4}4%?zDUdQQPa=4(h0+0?)*?XmMmQh4`*TzBF8gwXwM2 z^Ti!y3@WM1zM<)s4u!{ha&F?tuo(#_N*nKk#9M5#mD8a<=;;3^{BFYWsQu((T1m_O zpe&JE8OQ&M|7*}WO{uA8O7*hqlOLv4g@iCG05y)IFz>DsBm?LA1vQwm5c9s33zd3~ zh|8jQ+tC13VU|{rEb7gM)IE~FJvZD)=H2JepH;@WQHeZ>fIiEy93PbBSv`SD4VJ#h zooEHrsD|50iXtt~{%S6$Aqhtx_7h*v1IBLjhU%>LY>^SRrYyFQ=9ObY_KNc$EWRdB zxh=Yes%izaaG*+JAc76kk6FY~C2?FQn!yS_1K936L(T29X?~bTGbj}L^Wb@|R?O;n z^{?+2{l`CkZ1wh<4|T8f+hTNWj(+qg;#&vWPK3vLp+SM9`Z0E3=R|=dpJ^z3%e`jf zdrqdQpd?QMWl{f3t(}A=TI^+zSJ_6l?&q4d5w5NGej(D!JrVgvh{!EeEE4WPATw}( ztN2_|F`OhINjXp#w14l%bQ^)n1i zM}Q#g$LsF?c!>J>m4I`#lVHF_QQAippM|yNhik(BkDhvnw90T=#721INp^{A`&+x4 z1)qUo&jX0>?_Y)`&vON1?%WYw4-L|}IFX{aTA2Ujcwn4qUt+Csa`kD14C{G{Q;f&? zD#zjz7C+763IG_Th>~6Yw`raegeuPI`KX8i0d9Z@^N}9eb)QGcE3?UM<1_Qg_C(p7 zu2Acdn!5#`d|s5+>HY@qy}Y(&nz^7W%sjoYAir2r`Hy@jsE5Luq{bcMv*QIWmD?pr zqC=32bJLsKje>EvZb|`aA9uy>?O$3T&QllbXbQYew8E`eoHj>%Qy{rWbi|Gm%$+4+`kE^63>t zgVF0R3RrL;)mo(YUWw`MZfTz<_haScNfakX`#4~HV& zm5aEL6%lM?Bfk#;YO>+>HaE_;(HIffXWxY5%DpP0FkOQp1*$u?S8H}3=?U+fiM|5P zP)#AVwgWhhrvlr1;<{~nf^!rZs$E6oLVA!H4E)lh;Kn=UC90I zL9mU!E@X9^9$X3>lAgFe#laxS=BOL}F%06r-wg&qyMHO|pJ2Kd1fFFRN=>SUPtRqT zyxsbUh>H`$+IuJ+LigM=TTt@IB7_Hvg5;Sxtr|%SUL|Eby;!Mmi@EQ{laNHXKdESi zehV>EnV0w~hA5)s>O>tzoRCczEJ7(NPl6#vn@b#StWxTWExtQJxQ{ST(CHSZL+avQ zAC=~6ZWqVx_7)xhMQD65lRE#UD|AS!|NOI0=*(;Q<567yFg1`zbG<&s_XpOO`-MqE zp`*@v%*{q<55NZfQFulBxh^@X%{SYVChgz=&SUOc#QqY1tOIIvHz4DN13ftc2o1V9 zQUpVkk6)HD2(m0+e+C$!$zRN%rb`rlECL|e5P}3<*#nmD9^hlBU~0biT9Po%e9`|v z0+bkkk|eLbvsb8`1G!Fw_ulMsQ50=YFTPYfTR_j#ehkP63vMhe@HDa2{E%t#);=q) z5tBSL>q>)BIZn2J`HN-hMGG)Tz#9)BV>Sv?fmq}m^aN_O!7_Z8BU%f^5y}QlzvoY$ zW1iZXVCwO+9IW1bds@$9|6)a)%(;nJDKVMRb$e5*GI)cA`M&$L_fKVU+(8vnc#3LV z6&3$?Os%`^&on+Y)Fgp6#&22GcbD)vvkl;XY<~ot+PUJz#UJ;j8-E2720NpxU+D+aPuyR_le zid`q5`*Mg(0yZ`zh}!(ihW4lUBneg!D}#ZB;f(}DW>Mg7U6Skr)M*@#MK;aNr6mmLg34X);^vy6<*%r{OMH9z8Eo655YJFt((`8R;Ew!eY>t2KL+h?Z1*OZ@6s#@{*&r{0> zy$=Rr9%2@>7@=a*4gY#FuG5G0o&z~=&H{uGmZJAo>ets?j?Ob~$$S4~zHtJl6R<8r zKOgF?0>BNpa(h4};X2$HtEReCKe9;beF2KIW#CH(qbBh`-*0RmlWga&I>Yn@@am7b``gal+CkSVweSX=C_GqAN@H~=UK5Q``>d7aoh+{J99w* z5pa9!1Hz{SK*ZAc2cA!KB!{6!g}S~*kBFNNghmFAx0l&rrOH*p@7vB^gJQ8Q6%+qfw~v9!f~cIDDZldM+3S{L)Mxe|)0C05vT@4H2vwy~ z5yf}tA;ZL11LcMreU_Z&O=jN5 z$EosB8FpUdof(netL0%(KAb*`c`(1xkT33u0p`cNQciVdPILo99l{J4cm7V*)x`3G)a~t(I=Y?-iIXUrw9J=!Wr|XxJbITg( zDM*dY>pB6wY&SMnTL?(|$vy$yiO<_NuFY6c6|TO@0NAd5^Xpl_TN~}!(@C@E4!N)PgUN}a;O_GztPWa9eiH+A`wx5z@w2^ zaQ~a^(;)IKRpmhFsKCHU2J=ao+8Q+y9!0I?xqCIXSd#so!`V($1~JnIui1k;f1B}l zd>ZnCmK(g~9Lz6dS(mDB^0<1ZSG-o(by@`t)FC+g^%U)#BgmmZyi)?4d73wi<0k&o z684^!!gxy%b8O&XNCIwXVm!3Mt`CHk0{9j|n*wUrfeU;v3q}2wH{imx4FDdnMyU8n zyog?tv+?QnFKnR&PvV&o_?MYC!V>>@NkH2F) zt%T^psW3`pvDzDZSJr90QpY~VUec0d2fo9Z%|`OCX6RpcrG_`Hk)9-*9K>c_g__c( z%S#>|pzX)6agsQk&HBbke~%_x>?xf`K%k(&BSwlPC*tneN-Q-c<=koU8K)+W#@*6< z#KmIt6UZyy(`)9Yg3BOvs*JC}UC+q8uJ7j0^5?UCr<>*>V&aGDhkv~8gBpNM2mkf0 zkLT-FQa9C%!q@dSICR(#LV-q4EmshC_i1+#Jqew!(I*|&d7z3Y9emC=eOov|`1o;> zgYii(NWq-MAO$o{wmz@dk7D`nxkA-x{_0mljC#N(CIMVGL+mLkKdcbW6$Xu;h!Z7* zH}&S2%|2$LaTON2j|ohF$Tb0gd-sL5Djbm9*8Br!G$NrI>yEwQonI7gn{463)fp@O zF;v`)K?Ne!FUARAd89ltl(ea@ViFIGSMnwU$natdMYL64ugJgAS!MpG>>_YOVKFNw zQ?QypqoG$2M*XsVtt+hYqj=n{To|CkJ<{c?LSF*hE!6SoSr@)gVd+#LPZ869={kn; zAUlq345yFunO45$d7WWBk1FE!Ndv}>ZvM5U+7DId8D)O44)O23Zikmb%TlK73Lo{# zO3uLp6b%*=+?PlNvX}8hX|-Es!ZfumrJw+BR;#=1DQ3K7z&(zmn%7N6zl6BDDYao_ zidDG(_aHAK9?0XI^}zDTE98a!#;Sg{t&!28cxz;bCD;!9;v3mm3=;EsYMHm2DnZeV zpRLALLI;yU1_VL;<+(ujdod{b!hMA?uRJF{wM2}@hGTDUXSGbP&j#$6D-#w0R?h8q zk;Hs0MFN>e5{bS9Q@ozc<9C(&KVFQZ(&!TUyf8l#L#gd>azWGNzfxLUCIoq4=GxGK z6Gvkn_ueLq*kqRH6&QIXDL48fc{nVIol1F2< zG-~Iv2(VB<>-Jb}@c)=1F2A#;L>YU!i>cHm>>n8`^IvcJ^_Gn8>d_@tlA4?CeB_;+LtBqYr3 z6NbE`fo&bMYtL8(P#Q{$KD_zmLdzf?Z#rt>fcaV_3B5LaEjVSR>^7imI2 zofRWJy)T)kuxq<7>+U9=_Xeplf_}1;z;c=Y4kI+CoB;jyUUf1>H(mI7XGo9g%o>|@e zd)uC`wBA1nv7YmQ_HrmJYSxc-1kS(hw_|+c)T%Qjb?iPiN)`#; zYn;weumkY;1c2WVVb^3vu1FS^mj94q?pz_40R-(U>a|@6j(1@7`tc|j5Z!9MjXT5jkvt%7wrECyNoc z#%KX1Z16=wDypmA)Uof$${R3yN3MsF|u zG_2&0VeM5x{Z6{uP)&bfbc(nH7j=EDMv1UYJt~%AUZJbmbj+C#-GR_pRqNiBFsN(X zDqhFidB<|6%NlC9{U*CU~1}lDzEBx{WMv2M6IvzgoLJ zvd1h6u%QAntR@qqGRcA@taipwoC7oZf&~9Xs^5nEtjTlpEC$E^KEw_QQk((K&Xu zq92OEO@Bm|Rioy1Gx7H?k=F!$gs=Pi7w3-bxsbCN&7{Q67H;;`&h|dmqd}Y)a z2^!#FTW)D$AL0MG3k?Sf#_Ret@kGAC*%)ao?QU&GxrmHD&!iOM&QTeT7nc-*&3xfLTd0u=90Y#14uC zYKw!>iX&j8SpwWeFV-?mYph2|-?QoA0gz&W$UY`O6#=~(uf^M1oALHIY9%xn`1{M!z+$D^}YXTOyQg>ms?$r||8 zuNQJ8{<*ctMm^{}bb@DgEL^NC))7(KociP%i(Y>W_3z-HxJX?Ugp?dH_Bwe} z&&GBAA^x6CJ`ZIpFmL(;6)y0bo8MRvsgn}RNI&$et9)M6D>>MHE8=O&=3_-vh~Wy3 z4LSjb*#=hN)$Fj|^c{yL_7RS)KBWlZNp5+EO(1y?NI``kaYdII}b};_D zk}W6Ar!VgifKU> z)4KZf#mrO;T;9yhAek@se7HX-1yExN`nY+`xI-HC}5mUBsUyq>)w0o#;$BNtB?oD zNsM+@$56bHbt}@Ttu-1|1=OSBgua3REYXVAc+;K!uU_vdCXu}a94MS%U%eFQMFN+-v>np)ZV%69|U zw{|E!Tvn8w$@T$AjR|)^v0Tb#pXUNSq~*6(MtKgyK?+j0zuBF&35Hx=mj1X`bFH^D zHq*!Sdbop0d23gxsP@E(F7g5Md8J-8g!Mq;d_El5a8L-LCWF$4>Ku4JMFP%=W>-K) zs2@Xp;_~JM3P%t}=ECy`@$A4@MTdW39bmpjU|JUY2u< zCM)~D-hnSG>>9FH(|T0|C4VFLx>QCf8t6SYO1EH*CGOn7`1)8lqo;tDgo~5rYu310 zwqXrS9QS7a*kc?ju@j~yUo5Rp*CqOs)}qHYSK(=RuZpi!9f>{)YKAYNjF6%HBLqzH z6Qjml20nsCMa@4_muKCx3EjDnSA}EaCU6PxUmdM6adIYhTY)`9C>>JZ7j))iY25WT zR6z>T#8ZP;%Z^DUh`&dSiD=I&(PSTU+UEd>LzK^nT3Z`{KWb64P_8BP`GO@JV(pN6rimsO z(;3-F2}QI6%!v5t;t5y%QhWTh)y16-|SZZTT_Sk*hdJ zzq$B-3pK3vMCL!+;TlOTH$VAaaa$x8MNpCPQtcrjS5yT1VhXptt<7Ys)r>{kX}?RM zS@!^U@UIQq_yVaQT0Y{a9+Zp+JBnumFbNA6oJFC(eGdTdd?upvhK}GF%6llmQ|Xj2 zh0rh?eIi!b)!(eS;Cl^W)DZTQc~hFsT<*{(40>w{HUoHMN9>z@<-@&`**NKD;zC!M z5jBw(O3W`G8PnaI+2@f$oa8t*@Lyi5g z%s|A|1E)}LA`xW;(`kXoB+u-Kp0x#6zn25hm4M-n}TyB7$bN{Vj$+gH{7_{agRVCv$h&M6WcW75>;-Biykc< z!5GpJ@FQ1d+E=FOOVv{e2KTR4&lo|HagkyvWJU^!NJIHCG|k|7=p>vM<3&EP{L&G1 zKBULsKN1~yA)~@4#*ZqihSHj>_yCP`!~Y8UNvJ@11;{J z{VPA9H@Q*iB4uNFU1N)yU`iRVyf%W;kCkIetcy7Dz%L~FE>SQcP-{{a4Q4zYNhW5v z&FFc}vbALCVDw!Zi4tKKknWACjr6JeSgAgY!xq>{w>d+Ai7flIy^Y?Ae2$%`EaF*e zGfQ`e*rZs#(YOKGEZ(WYO2b#zMBPVJPq#*nv+$yNPo?E{21xJuP?GZ zGUg)6x_PkYTvOE3=RbyAsW#dg%}-b^(>7lz#JQ znhZFfPItrwg(>R*6Kg`ra1o3+L1lOxEKXrmnX!P{M*TO^$M4 z{n73*+ypN4M1sQC%cLALkFq+;*OV;&bE2TjkhUVplSQM-bL5|C=aFq$CZO;cGyJ%E z(u2mlq}Z0ux5o4n<1jGe-^b8dHBl84J3mO z!KNT}5^6qwDi^ZEM#3$;I{~2zPnb+*4dJEaM|QM3{_{gjdBR!dHRI?;^g(l8E;9)> z(-9rXLC!HdI532eNjPSv!irJZJSf?`I~fb3!TO26kj^F!l_oAseo9C`@l%dZLXex) z=r9{|jEH60grV!)sN?$h^4Q{TawM%U?sRh#NNlBNhzehK$L9KDk@CM3Mx|^*v>A zV86>IJm|-YCEa!Fjx{WnFJI_(9jhK@1aE2u%EgFiLhnP_qs$4Vb{7$2XG#13PIpn7 z{>LpS7Xexf*o;6zA-mTj&sFfk%MEAI85 zDdiJil#AQG1J=83S{2+_7$G+Fz2E)iXJ6k8^^eCyNWVpl#{a1Lsc>--aZQ$#cz};F zm+9g#(zG65mSug5*jxxlLP4Y;jasq+FdBo3OFtwykldbJ{HP!_nInXkRvYD_Qa>X? zGy7V0MP5n5WVjOJITq4=dRp)EU0SlcI9;B92QygvaUwJPcrdW7-1S|56cV>k#||ba z{5VaAH7|(F4zOcwiZo>b1J|ZHm`F3Sp8ib@1^K!3P>2NQ1#%ae^zwh-1*qfcyNVv} zPSQNXjnjnh^xwRRAmmz1KO84sN@v!9P!mPG_jOwrH1^+X-279&kxg%gfC*5TK&$W# zE(DBZ085CDjz^7<@f-oItMJAuh6eI8-Ay7h7wnExtrvI;+AJGD*oeG4dd5XBq4-(qDNc1Gz0CidSMrtz$qs3AlC$dX11;xgY{OUOyBA;L>n zctS@&g1qY`b414y!=^=u@e&!~3HlIUJfuW1DTio0R|x+m@9+Ks74Q=n(4Yd3=$;sW zk&$2SEnmf((1Tfr>AKaS8-{yH(4x0?nW)^TuOy9<@Q7!-CE3I}na7aPwu@9)#{occ z@?)tUD)eQ8FJLUWEeCLB?zPqB1T!f zcW2CRS}sA~Iwk-;W&9XPM7`uqfST#d~b zCD4!Hr$5u>y_*Hs6R7!u%4!u|S@=2KBc-kS-slr?^7@x8EGZ&TJOIl@7XZ-?Fnb^5 zgNla(+Sn3U&1ENWuJks73>^lhz4$x`8HIQ%@X?a3(EiBnf96P>y3!96nXzE5BV2-W zb4a37e6v)m?E?4*jJc)(T0Pffroo;VA=g9zP| zBTU~TdikATIqVDS}rV&bNF*4z##b)a$M*q*>18xBK8eeC&Z?uNi~a{>v> z0+$go&V(0x?fh2nMvUh(y~xRO(*+S;0{{`{0aI-ac0Z6o2#hS4)Et}n40_B^c#SYq zGw@F<)CN4-3r){|f?Hu$O>QRzNkAbQfoHr<#_`JH#`RUT360O9Us;R?u!6^l1Zi0? z0P!h;bgZ#jL>+_qf6BTNa45U>Z>FIclNe;OB*wnK*+Q1FW{W||DEnUCUQ5}@PL@KJ zEEz;pzTUB9$-WCkDG`+ssn?W2F-rA6&(K%j`@gQ|nrnFGoaa9Gxz9QG^1JT`q4t_t zXB2hx1yFeCsh!P>#b?XrHg0&l$Lj6UhlCkJi?599enH5}ENp~*If?jQ-HXNs+VXI7OfG>l!qR`#?TbwnRO-JieMMh) zcfw@=tsr#Tbwp{+bl`I!;2xT%7uc15_9lar${WCK*e_``$rZQyTxPxBxk(H}3qkor z@hzwbsId6D3bm{y@@tTIDyUEhYN>ZO-2(=bu_%5emb7SauzCA5HxQ~v(ZWHDV@y6F zKmq<;f&j|HbEBnzV2?%psLWm1uRjN$P*`W5cYH+dML&N@IrgTNhwAgV#_ed}O=6wr z>=q|gt81-)ZAAR!-P2yDFSP(~Xskn?F9~=z_piiuSo`ly191Gp9CJx~-|2>jBe%N9 z0%ukM03Zg=l5__YmIsJq-F=y^%su1#5BKO@1kCo~ob}3q8x_E|H77u-UK&%!b9n%q z<`}+LTeuHSRT=tPlyep*;`^myzST_N{$K1}i$$@Z2EPH2fF~UUFa67)mdkUwH?7#v z)%mW$Op&f;v5=Gj0hU#;N@Lk@Un;ynCe7g<^U26d6g36r1did_vAYgOl=4SF4J7n> zRS3u?zkW$WDr=0SZ=mKL|2&|bZT%l-0Zs6_u8Mqt?4FY$vXE^Ft{6NgFa)+7=UUCi zfGqi83sE@5GYg0$YrcaLfedSqRJWxad@gsewp&vjD1(!TjuX021}A3#5k@^7?^w8(JLeLG+wcTCXLKzv0+tF&_$QZRUe2z5*Vd(J=BrNIse& z2fEQ$3u^LgQn*&Z0MumQ-11GUCqOQoN-n+H_#aoypax9U*N50)eG)>ft?Z4Ne|^*` z^7B5yuy?}b^u9h!`}Z$rm)%tn0i4ovKr?e^9|v*zIDp3&TZ#Pi_^3a1@{rP2kG4O6 zafq%R^SjjqR(Qeq=@?N^?+C(=HL1PP-8ZFH4@8^B4_mi5 z4PIK2wcHERIit%;UvpX+0BJHo5;7Kt0xh*S-8p}7XcBVAg~x6l<0oVCEjX?RVEir~ zRQc=G%$|pfgo_Q8O$9e(s*3W3I9Q=kna>PT5ftl6P>q^ijQ-Vah? z1STl}lQz=5;K=Y8sg|o9Um7V50uPK5Em;=P={|lOmw|$BiK$yP4fZm#a%KUHy?K$^ zA<;F#YV($kD{3>OKV{d4khW2JDlZ8f@`JW>%u?Jpu_A1O&wmsOr|MQwaO$H|W>S7n3{d+W?~71K*memWPP%bL=3_SP*j_M+KEf{!$h1x$7*Q;e{f| zj*~H?uLM27?}eTq37yXDdxmQ}Sv8&v{k@uk58)!bHxOz?(!!-d0O&vV_qw?(H8G9} zHa?1?ouwka#;fXse~{tH(Fc(<;tlV&NmT)wte;y4{d-M;@zIF}D1uK%ly)aZJ*IUZ zuoBcX#OFgZkAOw#OlisMRUD$A{9JNT>rIq84$nSoe@)})<^Cd@fjFX>iW#SZbsx?W zjLJ_G4~y&YB>U!i=qj*?3a?>deIk(VIwjWWq&+q53X-9~J0X=tPrK*2hnybxdhws9 zQnY5|70z4SkcRHeM>?$VR^gteNvm;s$`&+Z1pqQ2{ZQHVg^R=Fm~>Hq4AdRQFgo34 zy?%CsGbtHVFxktWEQyr%y^WYGnGZL;EwLa4j}q!!l*tZrAP;gX80&%=0ze6R+5Y0T zKX4yyZ+C(|a2x>$VB&Q%FE(U&V*(og7*I0Dq~1J83=p_rAl|u{B9le{dO6cCy*x@H z<90G-<2d)vEFPu@hG;g-W1CDt*m^(3x;f(96Le)DE@K~9^x)_jJ!yM)P0#KU*0Bja z6o-r#jv>fcpTV)nY!GCURv81q&1%7YBWm-Q`S(zL{AshgN`Et7ByKpK%Mnq)IPbbT zx|bxZuwWL7ItGupY{#DP!Vgo}uw6+Dv>dYHKsEs>{21Yqk&KDwFJ&ay2<<_VOF6LE z-BFED1(yBl0p%FE?hZ%NvPeWRJi9szWdk~;YeJoV4Ki*en|L3*i8wBc z85D%Idk0wTX+N*XGHhyh-9V4Uhhb6U5l?WPEaLni7l-$8Y)8>qi`07W2RcfK$ual1 zt2(#c#ap|R@n;Zpi{ZuOQJXW1`dya5Qk|)AUbK3jibrsjB7l=HOVoU|uzJ_*o)KOA&;O zEVs)jaX2Ef&^$o*h8v<;d3*)@qGD;%HKuP=;J`>H*2$tH1lw|T76y(0n+{3_hdcnw z^RnaWCK!;s!72)!2%yj|*KA&_oI7EJ{CB-9idPl=0ZPpgI*y_(kUY^(tC!V9dwFCf z?TyLEXD$5qH$C;U1F^K`r|jaT;Rgd17bE295m4;IMa6{Nn_3ich8r^6doi=Aa&pZX z!1E2&3qg^)}6Y!lY(UG!vi zxaH`l{q~<~dUgVcJJscQ5Y#$}$tlE#Taii;1H~r5DeOuaW|2E@T3!Tsrtox(bK8hU zz9$hS6HllnET_;~o>S~_>)+M==Mb{P(>=!BnQhO9dhe%|$s}C@i~KMpwpGkABbKq@ z1zcn5Fqd>E^p;%HZM(!-X&X|p-f7`CP>QdPpA9CaxN&Oe^xN`n0TOkg-3sCl|P1-J}77uutoPD zN5_qZ(sOOa>QLk;x0fF5T$Dg(5_+HwJf!uG^i{xsLCh?c-KpFsBLC)8!Yn%3L+r!# zE(Mvdtvgc9v}9NyXKiO&h8(${(|`YjYadZt?*7a;Vlu=-VUMck*Uw8{+SuYpWW_u= zc#y+DSh|)RRnToJMK)eOD+Hr=*o+8*2+6YWExoc8!$O6b^V0}X?@7^mfWS;PfJgXO zdKgtXAK=4clRywMA!k(>;vJlYzpe-!N5Zr#&AW%C93dk@EPaX!lQz_|5}Qm~k<_}I zPoZN^Xc|GEI|hsE&+|v|O&Q%nLo*)F#LUXa?-W6-B<8-iaeZi-q{pUKGQ6>94(#^12~d^d4V5edG=KfCDu1yEdh8=?FrU+lO}J&F z19v;^6aW-;tX7Z)l%r5DTo}VR4u2(ecuACRNn+vz2$7`Wb#>lmU*J#wEU z7~cMFJj7RGv_IOTLjDXb^~v7KHZW%_i1Qj|2)m9kyWJ}uMV>_7RPO2yyT-@0*(Gog zK2SqUzFSZL=%lVu!NuNLug55%Brg!)nAdWBIiM~V@*4MNvhkV2`o92pF2>Oc-?li* zD%<@^{)k5mCSiMY?X?|$A};Sh)v6ZaX`4>JHKbu8>ZLCv$Y&@C3322 zhD_&(VUR$GcPqDhxAdWgO}!D1NxZIYuKeNAHBqIHuUueHr$8>&7-)a&_$N>j9`0tB z%lx??xdq}XL-)r(F26T&`BE6MJnEpRWBDV)_U(OG8@{FJArjZmvOArPgYR2}ZqY?m zTUi;aN$0pk5S$&(e|vgXJxNg%j>E&ZieB`OD(DP15N*4?ioPo+dO6_T+VE~#x;G9k zaZK;G1zRRKqKfF5dMrueWC<3)0QdR%>#hfe!=Ba3lJAx~>wf2ydr~RCf{WDF*gg$2 zAfFpO;VeJDOr<4>g@Y?R~yHR3bfs(x9k^X2~=0*M*0 zz#22ULb1NWJ+;-MGB87y^y-`^cdrt6*L|Na{(MVr&#ffGiI5kiVvgI(94P^pHbZjE z=BcMUy=|Cc6IU(qts8Al?M=-46YxZ`uKH=h-o!d5TvPCXNFy>NlZj;0xWapLW#Kuv zPou6C{TUR>pi9Sc-777;_>I??>R0>i14kVY<1(5_;ZdSCE7}E%b*OS)CZ6M4@*b`o znGvpbn`+UEP54pF1fW4~x2_oojUcPkaI;3SVIlPqwuBVn^)Q-&BdUe%i%x_ zX++zyNf2l$4&-;zo5|I*IFO1rLd=Y^zx=qEI$8waDl5(BlWt5~3`Dhm&?1Evd?U`j z()PM|j@ou0iLq(=H6WP)bVD1qKmZ$D34+e4`vx69Jy}(ko~|`L zfWaLDvSY|(tKrrd$>B>ZZ)$HD21tKl13!7UI*hFl%5 z)pP%$gXu=y(Z+A)0e|V-SubAniD-EF_Oc};gBYYZti_9kpUSNy>FdYL)mY*}lrm(S zsOcIBw)atTQl6kxFA_KGZS`J0`f|;vGFh{NClD=@lip0tf6VTRlf= z>mky)%vg^SV_i31pZo_&-Si~<T!|;B{Nd*8u zikk9Y{*ZVloYb1Rq#fk66y((qzm6t)0$^$Sa^lFmO6aw;kftfI;FeH{;Fjg9(;qJP zw7-6^!Mk9x$XJh0P~;j*t2iTAw2^M7Y6?F$T9R+_EqIaWxna{EN763TA8DQbbKqgU zw|SoDV%%ogA<~GF+>_<&*rDa%72Kv7o-QCE`=fG|PKv{7BY`Fks0&G8a%!+v+AupA zb}l&Xh)9z6LsKRuWLx{#%p_O3NAdf%^E(;Z zD#R?tMURV~GZqC|H<<~iw|C^ts1V`UA5at2CtLZC{SKlGdHk zg*eS3T2i=2F0{l-`epV z{}d^xJwBc@S#kxZw_KC*6SK9$6b9#nj|trYks7RR{&rsduYMV{4RXUQnX^JDx28)+ zejD&lzOln{dYwSZGCj3?4tNay)tIIO>yw0QgV9BBd}65gzdxB`j`x5Hcmj!kx?HmQ zulpQGxT!wa$CII~e@uJUwvg>SAZ8F(@`PcBtb<*2Q1GzB?~@H)IVW6loFTJtFfqxU zg?DkkxQZ|5B5ZE2Pk>_TO&z=MH1wZe0w0nP4OHoEI$>bian4ma8NY6eN|d<$9A?Z2 zB-6a6tPCdy)MVBHK9v9S$Dr6bkm>bCJLa89i%COCGIsiHMuvcCMJv2Mu&`5WHXA7k z#sB_$fpp(F!@V&l>bL(hC0ug$f5b{i3<#}M{@_ta+VsGB_+JjX7=h;3UJ~|!`<*%n z?$n=j?FdIOlnU{vjbLFQG=r?t+<7c;iwH*1T?TvcqtV4MtzECeSm++BKbV-9e*Bmx if3F5#SfE{U7t_iKo0%<5PZIbKlZl~)!CgJ)*#85i@+nyW literal 0 HcmV?d00001 diff --git a/guides/events/_event-queues/EventQueuesScheduling.svg b/guides/events/_event-queues/EventQueuesScheduling.svg new file mode 100644 index 0000000000..76b2c95e40 --- /dev/null +++ b/guides/events/_event-queues/EventQueuesScheduling.svg @@ -0,0 +1,3 @@ + + +
Whoever
Whoever
t1
t1
t0
t0
write event
into outbox
write event...
task for t1
at XX:XX:XX
→ "marker"
task for t1...
read
messages
read...
read
tasks
read...
delete
messages
delete...
delete
tasks
delete...
3.1
3.1
3.4
3.4
1.1
1.1
queued Service
queued Service
Service
Service
send/ emit
event
send/ emit...
t0 Task Runner
t0 Task Runner
runs on startup
+ every X mins
runs on startup...
Task Scheduler
Task Scheduler
schedule
event
scheduleevent
trigger (next) exec
(on commit)
trigger (next) exec...
send/ emit
event
send/ emit...
trigger
("flush t1")
trigger...
done
done
3.2
3.2
3.3
3.3
schedule
next exec
(if applicable)
schedule...
2.2
2.2
2.1
2.1
2.3
2.3
2.4
2.4
1.2
1.2
1.3a
1.3a
1.3b
1.3b
1.4
1.4
t1 Task Runner
t1 Task Runner
runs on startup
in non-mtx
runs on startup... \ No newline at end of file diff --git a/guides/events/_event-queues/architecture.md b/guides/events/_event-queues/architecture.md new file mode 100644 index 0000000000..c2d10f0b3d --- /dev/null +++ b/guides/events/_event-queues/architecture.md @@ -0,0 +1,137 @@ +## Overview + +![Event Queues Scheduling](./EventQueuesScheduling.png) + + + +## Approach + +The approach features three independent flows/loops that work as follows: + +### 1. Scheduling + +_Anybody_ sends/emits a request/event (hereafter simply _event_) to a service (1.1). +Because this service is _queued_, the event is intercepted and _scheduled_ for execution. +Via the additional API `srv.schedule()`, it is possible to supply `task`, `after`, and `every` arguments to make the task a _named task_ (see below) and to add delays and/or recurrence. + +The _scheduling_ described above is done by passing the event to the _task scheduler_ (1.2). +The task scheduler has three responsibilities: +1. Write the _message_ (following the outbox convention) to the tenant database (_t1_), in the same transaction if applicable, for atomicity (1.3a) +2. Write a _marker_ (see below) to the mtx database (_t0_) that captures that there is "something to do" for tenant _t1_ (1.3b) +3. Register an _on-commit_ listener that triggers execution of the scheduled task (1.4) + +Note: The task scheduler only `UPSERT`s messages and markers. + +### 2. Processing + +The _tenant task runner_ reads a configurable _chunk_ of messages from the database (2.1) and emits the respective event to the respective (_unqueued_) service (2.2). + +Each event is processed _individually_ and _in parallel_, each in its own transaction. +This must be taken into account when configuring the (default) chunk size. + +Events are also executed _exactly once_. +Two mechanisms ensure this: +1. _Application-level locking_: _Processable messages_ (see below) are `SELECT`ed `FOR UPDATE` and marked as _processing_. + - Alternatively, processable messages are `SELECT`ed `FOR UPDATE` and the lock is held for the entire duration (`legacyLocking: true`). + For migration reasons, this is still the default in cds^9, but the default will change in cds^10. +2. Messages are deleted within the same transaction in which they are processed. + - This is not possible with the legacy locking approach, because the reading transaction holds the lock on the message throughout. + +After successful processing, the message is deleted from the database (2.3). +For recurring tasks, the next execution is then scheduled via the task scheduler (2.4). + +After failed processing, the message's next attempt is scheduled via the task scheduler (2.4). +That is, the message is updated by incrementing `attempts`, setting `lastError` and `lastAttemptTimestamp`, and clearing `status`. +Scheduling the next attempt via the task scheduler is important to ensure that a respective marker is `UPSERT`ed. + +Notes: +- The task processor only `READ`s and `DELETE`s messages. +- In non-mtx scenarios, the task runner starts on app startup. + +### 3. Startup and Recovery + +The _mtx task runner_ reads a configurable _chunk_ of markers from the database (3.1) and emits the respective _flush_ event to the respective _tenant task runner_ (3.2). +A flush event resolves when all _processable messages_ have been processed (3.3). +Afterwards, all _previous markers_ (see Marker Deduplication) are deleted (3.4). + +Notes: +- It does not matter whether messages were processed successfully or not, because the next attempt is scheduled via the task scheduler, which writes a new marker. +- The mtx task runner runs on startup (only markers for "hot tenants" exist at that point) and every X minutes thereafter. +- In the future, mtx will also use the runtime's event queues implementation, so `t0` may contain markers as well as messages. + + + +## Markers + +_Markers_ contain no business data — only information about which queue of which tenant needs to be flushed at what point in time. + +### (Tenant-specific) Offset + +Because markers serve as a recovery/backup mechanism, their _timestamp_ differs from the _timestamp_ of the queued event. +Instead, a configurable _offset_ is added. + +To reduce the number of markers, they are placed on a configurable _grid_: the timestamp is determined by adding the offset to the original timestamp and then _ceiling_ the result to the next grid point. + +However, this can cause bursts of activity because task processing for multiple tenants becomes synchronized. +To avoid this, an additional _tenant-specific offset_ is added to the ceiled timestamp. +Because this offset requires no coordination, the tenant identifier (`zone id`/`app_tid`) is used as the seed of a random number generator; its first output, multiplied by the grid interval, becomes the tenant-specific offset. + +### Deduplication + +During marker selection for processing, there may be multiple "flush t1" markers with different timestamps. +However, a flush always includes all processable messages, so only a single flush is needed. +Therefore, a `SELECT DISTINCT` is used to skip logical duplicates, and after the flush, all markers with a timestamp ≤ the selected marker's timestamp are deleted. + + + +## Named Tasks + +_Named tasks_ (or _singleton tasks_) are scheduled events that: +1. Must exist only once +2. Have a non-null `task` property that allows them to be identified and addressed + +### Concurrency Issue + +There is a concurrency issue when scheduling named tasks. +Database transactions are _read-committed_ by default (on HANA and Postgres), meaning they only see committed data. +If two parallel transactions (which is common during bootstrapping) both try to schedule the same named task _for the first time_, they will not detect a conflict when `UPSERT`ing that task. + +Preventing all but the first commit would require a deferred check. +Because of the `appid` column for shared HDI containers, this would need to be a `UNIQUE INDEX` (which supports a `WHERE` clause). +Such a unique index cannot currently be created via cds. + +The alternatives are: +1. Acquire a table lock (which would require executing database-specific plain SQL, at least in Node.js), or +2. Rely on the primary key constraint of the outbox table by hashing `task` + `appid` into a deterministic `UUID` (or `String(36)`) + + + +## Messages + +### Processable Messages + +A message is _processable_ if: +1. Message timestamp + retry offset (= attempts × some exponential factor) < current time +2. Attempts < max attempts +3. Status ≠ `processing` OR the processing status has timed out + +### Schema Enhancements + +To efficiently manage markers (see Marker Deduplication), some fields currently encoded in `msg` — namely `tenant`, `queue`, and `event` — should be promoted to the top level (cf. https://github.tools.sap/cap/cds/pull/6170). + +### Migration Issue + +As with the introduction of application-level locking in cds^9, there is also a migration issue with the schema enhancement. +Old task runners may select messages written by new task schedulers, in which `tenant`, `queue`, and `event` are no longer encoded in `msg`. +(Because such old task runners are always _tenant task runners_, `tenant` is not relevant here.) +As a mitigation, `queue` and `event` must continue to be encoded in `msg` until cds^11. + + + +## TODOs + +1. The chunk size should be dynamic, based on the number of available connections. +2. The `t0` pool min should be 1 to make `UPSERT`ing a marker faster. +3. Should the message schema include a version property to avoid migration issues in the future (i.e., older runners selecting messages written by newer schedulers)? +4. Scheduled task runner runs (step 1.4) should probably be combined at some granularity. +5. Should the task runner also run every X minutes in non-mtx scenarios? From 78db55c200017d2382142ba3b9d94602c5766cad Mon Sep 17 00:00:00 2001 From: D050513 Date: Tue, 17 Mar 2026 23:37:06 +0100 Subject: [PATCH 04/15] review --- guides/events/event-queues-new.md | 85 +++++++++++++++++++++++-------- 1 file changed, 64 insertions(+), 21 deletions(-) diff --git a/guides/events/event-queues-new.md b/guides/events/event-queues-new.md index a572f00776..15d9ae546c 100644 --- a/guides/events/event-queues-new.md +++ b/guides/events/event-queues-new.md @@ -6,7 +6,18 @@ status: released # Transactional Event Queues +Persist events in the same database transaction as your business data. Process them asynchronously — with retries, ordering, and a dead letter queue. +{.subtitle} + {{ $frontmatter.synopsis }} +{.abstract} + +> [!tip] Guiding Principles +> +> 1. **Transactional** — events are written to the database within the same transaction as your business data +> 2. **Asynchronous** — a background runner dispatches events after commit, not during the request +> 3. **Resilient** — failed events are retried with exponential backoff; unrecoverable ones land in a dead letter queue +> 4. **Unified** — one mechanism covers four use cases: outbox, inbox, background tasks, and callbacks [[toc]] @@ -15,10 +26,10 @@ status: released ## Motivation In distributed systems, things fail. A remote service may be temporarily unavailable, a network call may time out, or your process may crash right after committing a database transaction but before sending the follow-up message. -These failures can leave your system in an inconsistent state — data is committed, but dependent side effects never happen. +These failures leave your system in an inconsistent state — data is committed, but dependent side effects never happen. _Transactional Event Queues_ solve this by persisting events and tasks in a database table **within the same transaction** as your business data. -After the transaction commits, a background process picks up the queued entries and executes them asynchronously — with retries, exactly-once guarantees, and a dead letter queue for unrecoverable failures. +After the transaction commits, a background runner picks up the queued entries and executes them asynchronously — with retries, exactly-once guarantees, and a dead letter queue for unrecoverable failures. This pattern is widely known as the _Transactional Outbox_, but CAP's event queues go beyond outbound messages. They provide a unified mechanism for four use cases: @@ -35,11 +46,35 @@ They provide a unified mechanism for four use cases: The core principle is straightforward: 1. Instead of executing side effects directly, you write an event message into a database table — **within the current transaction**. -2. Once the transaction commits, a task runner reads pending messages and dispatches them to the respective service. +2. Once the transaction commits, a background runner reads pending messages and dispatches them to the respective service. 3. If processing succeeds, the message is deleted. If it fails, the system retries with exponentially increasing delays. 4. After a configurable maximum number of attempts, the message is moved to the dead letter queue for manual intervention. -![This graphic is explained in the accompanying text.](../../releases/2025/assets/may25/TaskController.png){width=80%} +```mermaid +sequenceDiagram + participant H as Event Handler + participant DB as Database + participant R as Background Runner + participant S as Remote Service + + H->>DB: Write business data + H->>DB: Write event to outbox table + Note over H,DB: Both writes in the same transaction + DB-->>H: COMMIT + + loop Background processing + R->>DB: Poll for pending events + R->>S: Dispatch event + alt Success + R->>DB: Delete message + else Transient failure + R->>DB: Increment retry counter + Note over R: Retry with exponential backoff + else Max retries exceeded + R->>DB: Mark as dead letter + end + end +``` Because the event message and your business data share the same database transaction, you get two fundamental guarantees: @@ -97,9 +132,13 @@ When a message arrives from a broker like SAP Event Mesh or Apache Kafka, the me This brings two advantages: -- **Quick acknowledgment** — the message broker does not have to wait for your processing to complete. This is especially important with brokers like Kafka that expect fast consumer acknowledgments. +- **Quick acknowledgment** — the message broker does not have to wait for your processing to complete. This reduces backpressure and prevents consumer group rebalancing under load. - **Flatten the curve** — if a burst of messages arrives, they're queued in your database and processed at a controlled pace, preventing overload. +> [!note] Especially useful when brokers don't support redelivery +> Some message brokers (for example, SAP Event Mesh) do not allow retriggering delivery or correcting message payloads. +> With the inbox, failures are handled inside your app via the [dead letter queue](#dead-letter-queue), where you have full control over retry and correction. + Enable the inbox in your configuration: ::: code-group @@ -143,7 +182,11 @@ await srv.schedule('replicate', { entity: 'Products' }).every('10 minutes') ``` ::: -The `schedule` method is a shortcut for `cds.queued(srv).send()` with additional timing options: +> [!note] Node.js only +> The `srv.schedule()` API is currently available in Node.js only. +> In Java, use a `@Scheduled` annotation in combination with a queued outbox service to achieve equivalent behavior. + +The `schedule` method is a convenience shortcut that internally queues the call using `cds.queued(srv)` and adds timing options: ```js // Execute once, as soon as possible @@ -216,7 +259,7 @@ await qsrv.run(SELECT.from('Products')) // query (result discarded) Even though processing is asynchronous, you still need to `await` because the message is written to the database within the current transaction. ::: -In Java, use `OutboxService.outboxed(srv)` to wrap any CAP service: +In Java, use `AsyncCqnService.of(srv, outbox)` to wrap any CAP service with an outbox: ::: code-group ```java [Java] @@ -256,7 +299,7 @@ When working with event queues, you interact with the standard CAP service APIs: | `srv.emit(event, data)` | Emit a fire-and-forget event message | | `srv.send(event, data)` | Send a request (return value discarded for queued services) | | `srv.run(query)` | Run a CQL query (return value discarded for queued services) | -| `srv.schedule(event, data)` | Shortcut for `cds.queued(srv).send()` with timing options | +| `srv.schedule(event, data)` | Schedule a task with optional timing — Node.js only | The `schedule` method supports a fluent API: @@ -315,10 +358,10 @@ This ensures that messaging and audit log events are always sent reliably and ne ### Exactly Once { #exactly-once } The persistent queue guarantees exactly-once processing for database-related operations. -The system only commits database changes from event processing if the event is successfully processed, and vice versa. +Database changes made during event processing are only committed if — and only if — the event is successfully processed. -There is one active message processor per service, tenant, app instance, and message. -This prevents duplicate processing, except in the highly unlikely case of an app crash right after successful processing but before the message could be deleted from the queue. +To prevent duplicate processing across application instances, there is at most one active processor per service and tenant at any given time. +In the unlikely event of a process crash immediately after successful processing but before the message could be deleted, the message may be processed a second time. Handlers should therefore be idempotent where possible. ### No Phantom Events { #no-phantom-events } @@ -361,7 +404,7 @@ error.unrecoverable = true throw error ``` -In Java, you can suppress retries by catching the error and calling `context.setCompleted()`: +In Java, suppress retries by catching the error and calling `context.setCompleted()`: ```java @On(service = "", event = "myEvent") @@ -382,7 +425,7 @@ void process(OutboxMessageEventContext context) { ## Dead Letter Queue { #dead-letter-queue } -Messages that exceed the maximum retry count remain in the `cds.outbox.Messages` database table. +Messages that exceed the maximum retry count remain in the `cds.outbox.Messages` database table with their error information intact. These entries form the _dead letter queue_ and require manual intervention — either to fix the underlying issue and retry, or to discard the message. ### The Data Model @@ -524,14 +567,14 @@ public void deleteOutboxEntry(DeadOutboxMessagesDeleteContext context) { ## Deferred Principal Propagation { #principal-propagation } -When an event is processed asynchronously, the original user context is no longer available. +When an event is processed asynchronously, the original HTTP request context is no longer available. CAP handles this as follows: - The **user ID** is stored with the queued message and re-created when the message is processed. -- **User roles and attributes** are _not_ stored. Asynchronous tasks are always processed in privileged mode. +- **User roles and attributes** are _not_ stored. Asynchronous processing always runs in privileged mode. This means handlers for queued events must not rely on role-based authorization checks. -If you need to perform authorization in queued processing, store the necessary information in the event payload. +If you need to enforce authorization in queued processing, encode the necessary information in the event payload itself. @@ -575,7 +618,7 @@ Configuration options for Node.js: |--------|---------|-------------| | `maxAttempts` | `20` | Maximum retries before moving to dead letter queue | | `storeLastError` | `true` | Store error information of the last failed attempt | -| `timeout` | `"1h"` | Time after which a `processing` message is considered abandoned | +| `timeout` | `"1h"` | Time after which a `processing` message is considered abandoned and eligible for reprocessing | Configuration options for Java: @@ -604,7 +647,7 @@ For development and testing, you can use the in-memory queue. Messages are held ::: ::: warning No retry mechanism -With the in-memory queue, messages are lost if processing fails. There is no retry mechanism. +With the in-memory queue, messages are lost if processing fails. There is no retry mechanism and no dead letter queue. ::: @@ -640,12 +683,12 @@ Or disable queueing for a specific service: ## Manual Processing { #flush } -If the app crashes, another emit for the respective tenant and service is necessary to restart processing. -You can trigger it manually using the `flush` method: +After an application restart or crash, pending events in the database are not automatically picked up until a new outbox write occurs for the same service and tenant. +You can trigger reprocessing manually using the `flush` method, for example from a startup hook or admin endpoint: ::: code-group ```js [Node.js] const srv = await cds.connect.to('RemoteService') -cds.queued(srv).flush() +await cds.queued(srv).flush() ``` ::: From ceaa6dcccc082a354dba1aa0c1adade8602c588f Mon Sep 17 00:00:00 2001 From: D050513 Date: Mon, 1 Jun 2026 22:49:26 +0200 Subject: [PATCH 05/15] Event Queues --- guides/events/_menu.md | 1 - guides/events/event-queues-new.md | 694 ----------------------- guides/events/event-queues.md | 886 +++++++++++++++++++++++++++++- 3 files changed, 861 insertions(+), 720 deletions(-) delete mode 100644 guides/events/event-queues-new.md diff --git a/guides/events/_menu.md b/guides/events/_menu.md index 72a49d6dec..ac3d5b3c11 100644 --- a/guides/events/_menu.md +++ b/guides/events/_menu.md @@ -1,7 +1,6 @@ # [Core Concepts](core-concepts) # [Event Queues](event-queues) -# [Event Queues NEW](event-queues-new) # [Messaging](messaging) # [Apache Kafka](../../../guides/events/apache-kafka) # [Advanced Event Mesh](is-aem) diff --git a/guides/events/event-queues-new.md b/guides/events/event-queues-new.md deleted file mode 100644 index 15d9ae546c..0000000000 --- a/guides/events/event-queues-new.md +++ /dev/null @@ -1,694 +0,0 @@ ---- -synopsis: > - Transactional Event Queues allow you to schedule events and background tasks for asynchronous, exactly-once processing with ultimate resilience. -status: released ---- - -# Transactional Event Queues - -Persist events in the same database transaction as your business data. Process them asynchronously — with retries, ordering, and a dead letter queue. -{.subtitle} - -{{ $frontmatter.synopsis }} -{.abstract} - -> [!tip] Guiding Principles -> -> 1. **Transactional** — events are written to the database within the same transaction as your business data -> 2. **Asynchronous** — a background runner dispatches events after commit, not during the request -> 3. **Resilient** — failed events are retried with exponential backoff; unrecoverable ones land in a dead letter queue -> 4. **Unified** — one mechanism covers four use cases: outbox, inbox, background tasks, and callbacks - -[[toc]] - - - -## Motivation - -In distributed systems, things fail. A remote service may be temporarily unavailable, a network call may time out, or your process may crash right after committing a database transaction but before sending the follow-up message. -These failures leave your system in an inconsistent state — data is committed, but dependent side effects never happen. - -_Transactional Event Queues_ solve this by persisting events and tasks in a database table **within the same transaction** as your business data. -After the transaction commits, a background runner picks up the queued entries and executes them asynchronously — with retries, exactly-once guarantees, and a dead letter queue for unrecoverable failures. - -This pattern is widely known as the _Transactional Outbox_, but CAP's event queues go beyond outbound messages. -They provide a unified mechanism for four use cases: - -- **Outbox** — defer outbound calls to remote services until the transaction succeeds. -- **Inbox** — acknowledge inbound messages immediately and process them asynchronously. -- **Background Tasks** — schedule periodic or delayed tasks such as data replication. -- **Callbacks** — react to completed or failed tasks, enabling SAGA-like compensation patterns. - - - -## How It Works { #concept } - -The core principle is straightforward: - -1. Instead of executing side effects directly, you write an event message into a database table — **within the current transaction**. -2. Once the transaction commits, a background runner reads pending messages and dispatches them to the respective service. -3. If processing succeeds, the message is deleted. If it fails, the system retries with exponentially increasing delays. -4. After a configurable maximum number of attempts, the message is moved to the dead letter queue for manual intervention. - -```mermaid -sequenceDiagram - participant H as Event Handler - participant DB as Database - participant R as Background Runner - participant S as Remote Service - - H->>DB: Write business data - H->>DB: Write event to outbox table - Note over H,DB: Both writes in the same transaction - DB-->>H: COMMIT - - loop Background processing - R->>DB: Poll for pending events - R->>S: Dispatch event - alt Success - R->>DB: Delete message - else Transient failure - R->>DB: Increment retry counter - Note over R: Retry with exponential backoff - else Max retries exceeded - R->>DB: Mark as dead letter - end - end -``` - -Because the event message and your business data share the same database transaction, you get two fundamental guarantees: - -- **No phantom events** — if the transaction rolls back, no event is ever sent. -- **No lost events** — if the transaction commits, the event is guaranteed to be processed eventually. - - - -## Use Cases - -### Outbox { #outbox } - -The outbox defers outbound calls to remote services until the main transaction succeeds. -This prevents sending requests to external systems when your transaction might still roll back. - -**Example:** When creating a travel booking, you also need to notify an external flight service. -Without the outbox, the notification could be sent even if the booking transaction fails. - -::: code-group -```js [Node.js] -const xflights = await cds.connect.to('xflights') -const qd_xflights = cds.queued(xflights) - -this.after('CREATE', 'Travels', async (travel) => { - // Persisted within the current transaction, sent after commit - await qd_xflights.send('bookFlight', { travelId: travel.ID }) -}) -``` -```java [Java] -@Autowired @Qualifier("MyCustomOutbox") -OutboxService outbox; - -@Autowired @Qualifier(CqnService.DEFAULT_NAME) -CqnService remoteFlights; - -@After(event = CqnService.EVENT_CREATE, entity = Travels_.CDS_NAME) -void notifyFlights(List travels) { - AsyncCqnService outboxedFlights = AsyncCqnService.of(remoteFlights, outbox); - travels.forEach(t -> outboxedFlights.emit("bookFlight", Map.of("travelId", t.getId()))); -} -``` -::: - -Some services are outboxed automatically, including `cds.MessagingService` and `cds.AuditLogService`. -You don't need to call `cds.queued()` or configure anything extra for these — they use the persistent queue by default. - -[Learn more about auto-outboxed services in Node.js.](../../node.js/queue#per-configuration){.learn-more} -[Learn more about the outbox in Java.](../../java/outbox){.learn-more} - - -### Inbox { #inbox } - -The inbox mirrors the outbox pattern for inbound messages. -When a message arrives from a broker like SAP Event Mesh or Apache Kafka, the messaging service immediately persists it to the database, acknowledges it to the broker, and schedules its processing. - -This brings two advantages: - -- **Quick acknowledgment** — the message broker does not have to wait for your processing to complete. This reduces backpressure and prevents consumer group rebalancing under load. -- **Flatten the curve** — if a burst of messages arrives, they're queued in your database and processed at a controlled pace, preventing overload. - -> [!note] Especially useful when brokers don't support redelivery -> Some message brokers (for example, SAP Event Mesh) do not allow retriggering delivery or correcting message payloads. -> With the inbox, failures are handled inside your app via the [dead letter queue](#dead-letter-queue), where you have full control over retry and correction. - -Enable the inbox in your configuration: - -::: code-group -```json [Node.js — package.json] -{ - "cds": { - "requires": { - "messaging": { - "inboxed": true - } - } - } -} -``` -```yaml [Java — application.yaml] -cds: - messaging: - services: - - name: messaging-name - inbox: - enabled: true -``` -::: - -::: warning Inboxing moves the dead letter queue into your app -With the inbox enabled, all messages are acknowledged to the message broker regardless of whether processing succeeds. -Failures must be managed through the [dead letter queue](#dead-letter-queue). -::: - - -### Background Tasks { #background-tasks } - -Event queues are not limited to messaging. You can schedule arbitrary background tasks such as data replication, cache refresh, or garbage collection. - -**Example:** Replicate data from a remote service every 10 minutes. - -::: code-group -```js [Node.js] -const srv = await cds.connect.to('RemoteService') -await srv.schedule('replicate', { entity: 'Products' }).every('10 minutes') -``` -::: - -> [!note] Node.js only -> The `srv.schedule()` API is currently available in Node.js only. -> In Java, use a `@Scheduled` annotation in combination with a queued outbox service to achieve equivalent behavior. - -The `schedule` method is a convenience shortcut that internally queues the call using `cds.queued(srv)` and adds timing options: - -```js -// Execute once, as soon as possible -await srv.schedule('cleanup', { olderThan: '30d' }) - -// Execute once, after a delay -await srv.schedule('cleanup', { olderThan: '30d' }).after('1h') - -// Execute repeatedly -await srv.schedule('replicate', { entity: 'Products' }).every('10 minutes') -``` - -::: tip Real-world example: data federation -The [data federation guide](../integration/data-federation) uses `srv.schedule().every()` to implement polling-based replication, fetching incremental updates from remote services on a regular interval. -::: - - -### Callbacks (SAGA Patterns) { #callbacks } - -In distributed transactions, you often need to react when an asynchronous step completes or fails. -Event queues support this with `#succeeded` and `#failed` callback events, enabling compensation logic similar to SAGA patterns. - -**Example:** After successfully creating a flight booking via the outbox, replicate the full business object from the remote system. If the booking fails, notify the user. - -::: code-group -```js [Node.js] -const flights = await cds.connect.to('FlightService') - -// Called when the queued booking succeeds -flights.after('bookFlight/#succeeded', async (result, req) => { - console.log('Flight booked successfully:', result) - // Replicate booking details from remote -}) - -// Called when the queued booking fails after max retries -flights.after('bookFlight/#failed', async (error, req) => { - console.log('Flight booking failed:', error) - // Trigger compensation logic -}) -``` -::: - -::: tip Register on specific events -Callback handlers must be registered for the specific `#succeeded` or `#failed` events. -The `*` wildcard handler is not called for these events. -::: - - - -## How to Use { #how-to-use } - -### Queueing a Service { #cds-queued } - -Use `cds.queued(srv)` in Node.js to obtain a queued proxy of any non-database service. -All subsequent dispatches on this proxy are persisted to the event queue and processed asynchronously. - -::: code-group -```js [Node.js] -const srv = await cds.connect.to('RemoteService') -const qsrv = cds.queued(srv) - -// All operations are now queued -await qsrv.emit('someEvent', { key: 'value' }) // fire-and-forget -await qsrv.send('someRequest', { key: 'value' }) // request (result discarded) -await qsrv.run(SELECT.from('Products')) // query (result discarded) -``` -::: - -::: tip `await` is still needed -Even though processing is asynchronous, you still need to `await` because the message is written to the database within the current transaction. -::: - -In Java, use `AsyncCqnService.of(srv, outbox)` to wrap any CAP service with an outbox: - -::: code-group -```java [Java] -OutboxService outbox = runtime.getServiceCatalog() - .getService(OutboxService.class, "MyCustomOutbox"); -CqnService remote = runtime.getServiceCatalog() - .getService(CqnService.class, "RemoteService"); - -// Wrap with outbox handling -AsyncCqnService queued = AsyncCqnService.of(remote, outbox); -queued.emit("someEvent", Map.of("key", "value")); -``` -::: - - -### Unqueueing a Service - -If a service is queued by configuration, you can get back the original (synchronous) service: - -::: code-group -```js [Node.js] -const srv = cds.unqueued(qsrv) // back to synchronous -``` -```java [Java] -CqnService original = outbox.unboxed(outboxedService); -``` -::: - - - -### Service API { #service-api } - -When working with event queues, you interact with the standard CAP service APIs: - -| API | Description | -|-----|-------------| -| `srv.emit(event, data)` | Emit a fire-and-forget event message | -| `srv.send(event, data)` | Send a request (return value discarded for queued services) | -| `srv.run(query)` | Run a CQL query (return value discarded for queued services) | -| `srv.schedule(event, data)` | Schedule a task with optional timing — Node.js only | - -The `schedule` method supports a fluent API: - -```js -await srv.schedule('task', data) // execute asap -await srv.schedule('task', data).after('1h') // execute after one hour -await srv.schedule('task', data).every('1h') // repeat every hour -``` - - -### Queueing by Configuration { #by-configuration } - -You can queue any service through configuration without changing code: - -::: code-group -```json [Node.js — package.json] -{ - "cds": { - "requires": { - "RemoteService": { - "kind": "odata", - "outboxed": true - } - } - } -} -``` -```yaml [Java — application.yaml] -cds: - outbox: - services: - MyCustomOutbox: - maxAttempts: 10 -``` -::: - - -### Auto-Outboxed Services { #auto-outboxed } - -The following services are outboxed by default — you don't need any additional configuration: - -| Service | Description | -|---------|-------------| -| `cds.MessagingService` | All messaging services (Event Mesh, Kafka, etc.) | -| `cds.AuditLogService` | Audit log events | - -This ensures that messaging and audit log events are always sent reliably and never lost due to transaction rollbacks. - -[Learn more about auto-outboxed services in Node.js.](../../node.js/queue#per-configuration){.learn-more} -[Learn more about the outbox in Java.](../../java/outbox#persistent){.learn-more} - - - -## Characteristics - -### Exactly Once { #exactly-once } - -The persistent queue guarantees exactly-once processing for database-related operations. -Database changes made during event processing are only committed if — and only if — the event is successfully processed. - -To prevent duplicate processing across application instances, there is at most one active processor per service and tenant at any given time. -In the unlikely event of a process crash immediately after successful processing but before the message could be deleted, the message may be processed a second time. Handlers should therefore be idempotent where possible. - -### No Phantom Events { #no-phantom-events } - -Because the event message is written within the same database transaction as your business data, a rollback of the transaction also removes the event message. -No event is ever dispatched for a transaction that didn't commit. - -### Guaranteed Order { #guaranteed-order } - -In Node.js, messages are processed in the order they were submitted, per service and tenant. - -In Java, the `DefaultOutboxOrdered` outbox processes entries in submission order. -The `DefaultOutboxUnordered` outbox may process entries in parallel across application instances. - -::: code-group -```yaml [Java — Configuring Order] -cds: - outbox: - services: - DefaultOutboxOrdered: - ordered: true # default - DefaultOutboxUnordered: - ordered: false # default -``` -::: - - -### Error Handling { #errors } - -When processing fails, the system retries the message with exponentially increasing delays. -After a configurable maximum number of attempts (default: 20 in Node.js, 10 in Java), the message is moved to the dead letter queue. - -Some errors are identified as _unrecoverable_ — for example, when a topic is forbidden in SAP Event Mesh. -These messages are immediately moved to the dead letter queue without further retries. - -To mark your own errors as unrecoverable in Node.js: - -```js -const error = new Error('Invalid payload') -error.unrecoverable = true -throw error -``` - -In Java, suppress retries by catching the error and calling `context.setCompleted()`: - -```java -@On(service = "", event = "myEvent") -void process(OutboxMessageEventContext context) { - try { - // processing logic - } catch (Exception e) { - if (isSemanticError(e)) { - context.setCompleted(); // remove from queue, no retry - } else { - throw e; // retry - } - } -} -``` - - - -## Dead Letter Queue { #dead-letter-queue } - -Messages that exceed the maximum retry count remain in the `cds.outbox.Messages` database table with their error information intact. -These entries form the _dead letter queue_ and require manual intervention — either to fix the underlying issue and retry, or to discard the message. - -### The Data Model - -Your database model is automatically extended with the following entity: - -```cds -namespace cds.outbox; - -entity Messages { - key ID : UUID; - timestamp : Timestamp; - target : String; - msg : LargeString; - attempts : Integer default 0; - partition : Integer default 0; - lastError : LargeString; - lastAttemptTimestamp: Timestamp @cds.on.update: $now; - status : String(23); -} -``` - - -### Managing Dead Letters - -You can expose a CDS service to manage the dead letter queue with actions to revive or delete entries. - -#### 1. Define the Service - -::: code-group -```cds [srv/outbox-dead-letter-queue-service.cds] -using from '@sap/cds/srv/outbox'; - -@requires: 'internal-user' -service OutboxDeadLetterQueueService { - - @readonly - entity DeadOutboxMessages as projection on cds.outbox.Messages - actions { - action revive(); - action delete(); - }; - -} -``` -::: - -::: warning Restrict access -The dead letter queue contains sensitive data. Ensure the service is accessible only to internal users. -::: - -#### 2. Filter for Dead Entries - -As `maxAttempts` is configurable, its value cannot be added as a static filter to the projection, but must be applied programmatically. - -::: code-group -```js [Node.js — srv/outbox-dead-letter-queue-service.js] -const cds = require('@sap/cds') - -module.exports = class OutboxDeadLetterQueueService extends cds.ApplicationService { - async init() { - this.before('READ', 'DeadOutboxMessages', function (req) { - const { maxAttempts } = cds.env.requires.outbox - req.query.where('attempts >= ', maxAttempts) - }) - await super.init() - } -} -``` -```java [Java — DeadOutboxMessagesHandler.java] -@Component -@ServiceName(OutboxDeadLetterQueueService_.CDS_NAME) -public class DeadOutboxMessagesHandler implements EventHandler { - - private final PersistenceService db; - - public DeadOutboxMessagesHandler( - @Qualifier(PersistenceService.DEFAULT_NAME) PersistenceService db) { - this.db = db; - } - - @Before(entity = DeadOutboxMessages_.CDS_NAME) - public void addDeadEntryFilter(CdsReadEventContext context) { - // Filter for entries that exceeded maxAttempts - Optional outboxFilters = createOutboxFilters(context.getCdsRuntime()); - outboxFilters.ifPresent(filter -> { - CqnSelect modified = copy(context.getCqn(), new Modifier() { - @Override - public CqnPredicate where(Predicate where) { - return filter.and(where); - } - }); - context.setCqn(modified); - }); - } -} -``` -::: - -#### 3. Implement Bound Actions - -Entries in the dead letter queue can be _revived_ by resetting the retry counter to zero, or _deleted_ permanently. - -::: code-group -```js [Node.js — srv/outbox-dead-letter-queue-service.js] -this.on('revive', 'DeadOutboxMessages', async function (req) { - await UPDATE(req.subject).set({ attempts: 0 }) -}) - -this.on('delete', 'DeadOutboxMessages', async function (req) { - await DELETE.from(req.subject) -}) -``` -```java [Java] -@On -public void reviveOutboxMessage(DeadOutboxMessagesReviveContext context) { - CqnAnalyzer analyzer = CqnAnalyzer.create(context.getModel()); - Map key = analyzer.analyze(context.getCqn()).rootKeys(); - Messages msg = Messages.create((String) key.get(Messages.ID)); - msg.setAttempts(0); - db.run(Update.entity(Messages_.class).entry(key).data(msg)); - context.setCompleted(); -} - -@On -public void deleteOutboxEntry(DeadOutboxMessagesDeleteContext context) { - CqnAnalyzer analyzer = CqnAnalyzer.create(context.getModel()); - Map key = analyzer.analyze(context.getCqn()).rootKeys(); - db.run(Delete.from(Messages_.class).byId(key.get(Messages.ID))); - context.setCompleted(); -} -``` -::: - -[Learn more about the dead letter queue in Node.js.](../../node.js/queue#managing-the-dead-letter-queue){.learn-more} -[Learn more about the dead letter queue in Java.](../../java/outbox#outbox-dead-letter-queue){.learn-more} - - - -## Deferred Principal Propagation { #principal-propagation } - -When an event is processed asynchronously, the original HTTP request context is no longer available. -CAP handles this as follows: - -- The **user ID** is stored with the queued message and re-created when the message is processed. -- **User roles and attributes** are _not_ stored. Asynchronous processing always runs in privileged mode. - -This means handlers for queued events must not rely on role-based authorization checks. -If you need to enforce authorization in queued processing, encode the necessary information in the event payload itself. - - - -## Configuration - -### Persistent Queue (Default) { #persistent-queue } - -The persistent queue is enabled by default. Messages are stored in a database table within the current transaction. - -::: code-group -```json [Node.js — package.json] -{ - "cds": { - "requires": { - "queue": { - "kind": "persistent-queue", - "maxAttempts": 20, - "storeLastError": true, - "timeout": "1h" - } - } - } -} -``` -```yaml [Java — application.yaml] -cds: - outbox: - services: - DefaultOutboxOrdered: - maxAttempts: 10 - ordered: true - DefaultOutboxUnordered: - maxAttempts: 10 - ordered: false -``` -::: - -Configuration options for Node.js: - -| Option | Default | Description | -|--------|---------|-------------| -| `maxAttempts` | `20` | Maximum retries before moving to dead letter queue | -| `storeLastError` | `true` | Store error information of the last failed attempt | -| `timeout` | `"1h"` | Time after which a `processing` message is considered abandoned and eligible for reprocessing | - -Configuration options for Java: - -| Option | Default | Description | -|--------|---------|-------------| -| `maxAttempts` | `10` | Maximum retries before the entry is considered dead | -| `ordered` | `true` | Process entries in submission order | - - -### In-Memory Queue - -For development and testing, you can use the in-memory queue. Messages are held in memory and emitted after the transaction commits. - -::: code-group -```json [Node.js — package.json] -{ - "cds": { - "requires": { - "queue": { - "kind": "in-memory-queue" - } - } - } -} -``` -::: - -::: warning No retry mechanism -With the in-memory queue, messages are lost if processing fails. There is no retry mechanism and no dead letter queue. -::: - - -### Disabling the Queue - -You can disable event queues globally: - -```json -{ - "cds": { - "requires": { - "queue": false - } - } -} -``` - -Or disable queueing for a specific service: - -```json -{ - "cds": { - "requires": { - "messaging": { - "outboxed": false - } - } - } -} -``` - - - -## Manual Processing { #flush } - -After an application restart or crash, pending events in the database are not automatically picked up until a new outbox write occurs for the same service and tenant. -You can trigger reprocessing manually using the `flush` method, for example from a startup hook or admin endpoint: - -::: code-group -```js [Node.js] -const srv = await cds.connect.to('RemoteService') -await cds.queued(srv).flush() -``` -::: diff --git a/guides/events/event-queues.md b/guides/events/event-queues.md index 903eeba561..3fe9d0ce02 100644 --- a/guides/events/event-queues.md +++ b/guides/events/event-queues.md @@ -1,62 +1,898 @@ --- synopsis: > - Transactional Event Queues allow to schedule events and background tasks for asynchronous exactly once processing and withultimate resilience. + Transactional Event Queues allow you to schedule events and background tasks for asynchronous, resilient processing in the same transaction as your business data. status: released --- # Transactional Event Queues +Queue remote calls, inbound messages, and background tasks in the same transaction as your business data. CAP executes them later with retries, ordering, and dead letter handling. +{.subtitle} + {{ $frontmatter.synopsis }} +{.abstract} + +> [!tip] Guiding Principles +> +> 1. **Transactional** — queued work is written in the same transaction as your business data +> 2. **Asynchronous** — a background runner dispatches it after commit, not during the request +> 3. **Resilient** — failed work is retried with exponential backoff; unrecoverable entries land in a dead letter queue +> 4. **Unified** — one mechanism covers outbox, inbox, background tasks, and callbacks [[toc]] +## Before You Continue + +This guide assumes familiarity with CAP services, transactions, and remote service connections. +For broker-based scenarios such as SAP Event Mesh or Kafka, basic messaging concepts are also helpful. + +## Motivation + +Distributed side effects are hard to get right. +Your application may commit local data, but a follow-up remote call can still fail because of network errors, service outages, or a process crash. + +_Transactional Event Queues_ solve this by storing the follow-up work in the database as part of the same transaction as your business data. +After commit, a background runner executes that work asynchronously and retries failures until they succeed or become dead letters. + +This pattern is often known as the _Transactional Outbox_, but CAP's event queues go beyond outbound messages. +They provide one mechanism for four use cases: + +- **Outbox** — defer outbound calls to remote services until the transaction succeeds. +- **Inbox** — acknowledge inbound messages immediately and process them asynchronously. +- **Background Tasks** — schedule periodic or delayed tasks such as data replication. +- **Callbacks** — react to completed or failed tasks, enabling SAGA-like compensation patterns. + +Transactional Event Queues are not just a broker integration feature. +They are CAP’s general mechanism for persisting asynchronous work in the database, whether that work is an outbound message, an inbound message to be processed later, or a background task scheduled by your application. + +## When to Use Them + +Use event queues when work must happen _after_ the current transaction commits, or when that work needs durable retries and dead letter handling. +If you need an immediate response from a remote system, use a normal synchronous service call instead. + +| If you need to... | Use event queues? | Why | +|---|---|---| +| Call a remote service only after DB commit | Yes | Prevents external side effects for rolled-back transactions | +| Process inbound broker messages without blocking acknowledgment | Yes | Lets the app acknowledge early and process later | +| Schedule delayed or recurring background work | Yes | Uses the same persistence and retry mechanism | +| Need an immediate synchronous response from a remote call | No | Queued requests execute later and discard the direct return value | +| Run purely local logic inside the same request | Usually no | Direct execution is simpler when no asynchronous boundary is needed | + +```mermaid +flowchart TD + A[Need asynchronous work?] -->|No| B[Use direct service call] + A -->|Yes| C[Must only happen after DB commit?] + C -->|Yes| D[Use transactional event queue] + C -->|No| E[Need durable retries and DLQ?] + E -->|Yes| D + E -->|No| F[Simple async logic may be enough] + + D --> G[Outbox] + D --> H[Inbox] + D --> I[Background task] + D --> J[Callback / compensation] +``` + +## Quick Start + +Use a queued service when a side effect must only happen after the current transaction commits. + +```js +const flights = await cds.connect.to('FlightService') +const queuedFlights = cds.queued(flights) + +this.after('CREATE', 'Travels', async travel => { + await queuedFlights.send('bookFlight', { travelId: travel.ID }) +}) +``` + +This stores the flight booking request in the database together with the travel creation. +CAP dispatches it later in the background. +If the transaction rolls back, no booking request is sent. + +## How It Works { #concept } + +The core principle is straightforward: + +1. Instead of executing side effects directly, you write a message into a database table — **within the current transaction**. +2. Once the transaction commits, a background runner reads pending messages and dispatches them to the respective service. +3. If processing succeeds, the message is deleted. +4. If processing fails, the system retries with exponentially increasing delays. +5. After a configurable maximum number of attempts, the message is moved to the dead letter queue for manual intervention. + +```mermaid +sequenceDiagram + participant H as Event Handler + participant DB as Database + participant R as Background Runner + participant S as Remote Service + + H->>DB: Write business data + H->>DB: Write queued message + Note over H,DB: Both writes happen in the same transaction + DB-->>H: COMMIT + + loop Background processing + R->>DB: Poll for pending entries + R->>S: Dispatch work + alt Success + R->>DB: Delete message + else Transient failure + R->>DB: Increment retry counter + Note over R: Retry with exponential backoff + else Max retries exceeded + R->>DB: Mark as dead letter + end + end +``` + +Because the queued message and your business data share the same database transaction, you get two core guarantees: + +- **No phantom events** — if the transaction rolls back, no message is sent. +- **No lost events** — if the transaction commits, the queued work is persisted and processed eventually. + +There is at most one active processor per service and tenant at a given time. +That is important for understanding ordering and duplicate prevention. + +### Callback Events { #callback-events } + +When a queued message finishes processing, the system emits a callback event on the same service: + +- **`/#succeeded`** — emitted when processing completes successfully. +- **`/#failed`** — emitted when the message becomes a dead letter (after all retries are exhausted). + +Callback events let you react to outcomes — for example, to trigger compensation logic after a failure or to replicate data after a successful remote call. + +- **Queued transactionally** — callback events are written to the queue in the same transaction as the main event processing, so they only exist if processing commits. +- **Processed asynchronously** — like any queued event, they run in their own transaction afterwards. + +> [!note] Emitted by the background runner +> Callback events are emitted by the background runner, not during the original request that queued the message. + +### Single-Tenancy vs Multi-Tenancy { #tenancy } + +Event queues work in both single-tenant and multi-tenant deployments. + +[Learn more about multitenancy.](../multitenancy/){.learn-more} + +#### Single-Tenancy + +Messages are stored in a queue table that resides in the application database. A background runner starts when your application starts and processes messages continuously. When a transaction commits, processing is triggered immediately. + +#### Multi-Tenancy + +Each tenant has its own database. To coordinate across tenants, the system writes lightweight markers to a central (provider) database whenever messages are queued. A central runner periodically checks these markers and triggers processing for each tenant that has pending work. + +This ensures that: +- Pending messages are recovered after application restarts or shutdowns. +- Each tenant's messages are processed independently. + +### The Data Model { #data-model } + +Your database model is automatically extended with the following entity, used by the persistent queue: + +```cds +namespace cds.outbox; + +entity Messages { + key ID : UUID; // Unique message identifier + timestamp : Timestamp; // When the message was queued + target : String; // Target service/queue name + msg : LargeString; // Serialized event payload + attempts : Integer default 0; // Number of processing attempts + partition : Integer default 0; // Reserved, currently unused + lastError : LargeString; // Error from last failed attempt + lastAttemptTimestamp : Timestamp; // When last attempt occurred + status : String(23); // Current processing status + task : String(255); // Task name for named/singleton tasks + appid : String(255); // Application ID for shared HDI containers +} +``` + +## Direct vs Queued Calls + +A queued call changes _when_ work happens and _what the caller can expect back_. +That difference is easier to understand when seen side by side. + +```mermaid +sequenceDiagram + participant App + participant DB + participant Remote + + rect rgb(255,245,245) + Note over App,Remote: Direct call + App->>Remote: send() + Remote-->>App: result or error + App->>DB: commit business data + end + + rect rgb(245,255,245) + Note over App,Remote: Queued call + App->>DB: write business data + App->>DB: write queued message + DB-->>App: commit + App-->>App: request finished + DB-->>Remote: background dispatch later + end +``` + +> [!warning] Queued calls are asynchronous +> A queued service persists the request and returns after the message is stored, not after the remote operation finishes. +> Any return value from `send()` or `run()` is therefore not available to the caller. + +## End-to-End Example + +The following example ties together queueing, callbacks, and local state updates. +It shows a common pattern: create local business data first, then trigger remote work asynchronously. + +```js +const cds = require('@sap/cds') + +module.exports = class TravelService extends cds.ApplicationService { + async init() { + const flights = await cds.connect.to('FlightService') + const queuedFlights = cds.queued(flights) + + this.after('CREATE', 'Travels', async travel => { + await queuedFlights.send('bookFlight', { + travelId: travel.ID, + customerId: travel.customer_ID + }) + }) + + flights.after('bookFlight/#succeeded', async (_, req) => { + await UPDATE('Travels') + .set({ status: 'Booked' }) + .where({ ID: req.data.travelId }) + }) + + flights.after('bookFlight/#failed', async (err, req) => { + await UPDATE('Travels') + .set({ status: 'BookingFailed' }) + .where({ ID: req.data.travelId }) + req.warn(`Flight booking permanently failed: ${err.message}`) + }) + + await super.init() + } +} +``` + +This example highlights an important design rule: +use callbacks or persisted status updates for outcomes, not direct return values. + +## Use Cases + +### Outbox { #outbox } + +The outbox defers outbound calls to remote services until the main transaction succeeds. +This prevents sending requests to external systems when your transaction might still roll back. + +**Example:** When creating a travel booking, you also need to notify an external flight service. +Without the outbox, the notification could be sent even if the booking transaction fails. + +::: code-group +```js [Node.js] +const xflights = await cds.connect.to('xflights') +const qd_xflights = cds.queued(xflights) + +this.after('CREATE', 'Travels', async (travel) => { + // Persisted within the current transaction, sent after commit + await qd_xflights.send('bookFlight', { travelId: travel.ID }) +}) +``` +```java [Java] +@Autowired @Qualifier("FlightsOutbox") +OutboxService outbox; + +@Autowired @Qualifier(CqnService.DEFAULT_NAME) +CqnService remoteFlights; + +@After(event = CqnService.EVENT_CREATE, entity = Travels_.CDS_NAME) +void notifyFlights(List travels) { + AsyncCqnService outboxedFlights = AsyncCqnService.of(remoteFlights, outbox); + travels.forEach(t -> outboxedFlights.emit("bookFlight", Map.of("travelId", t.getId()))); +} +``` +::: + +```js +// Anti-pattern: remote side effect happens before local commit is safe +this.after('CREATE', 'Travels', async travel => { + await flights.send('bookFlight', { travelId: travel.ID }) +}) +``` + +If the surrounding transaction later fails, the external booking may already exist although the local travel record was rolled back. + +[See the *XTravels* sample for a comparable scenario.](https://github.com/capire/xtravels){.learn-more} + +Some services are outboxed automatically, including `cds.MessagingService` and `cds.AuditLogService`. +You don't need to call `cds.queued()` or configure anything extra for these — they use the persistent queue by default. + +[Learn more about auto-outboxed services in Node.js.](../../node.js/queue#per-configuration){.learn-more} +[Learn more about the outbox in Java.](../../java/outbox){.learn-more} + +### Inbox { #inbox } + +The inbox mirrors the outbox pattern for inbound messages. +When a message arrives from a broker like SAP Event Mesh or Apache Kafka, the messaging service immediately persists it to the database, acknowledges it to the broker, and schedules its processing. + +This brings two advantages: + +- **Quick acknowledgment** — the message broker does not have to wait for your processing to complete. This reduces backpressure and prevents consumer group rebalancing under load. +- **Flatten the curve** — if a burst of messages arrives, they are queued in your database and processed at a controlled pace. + +```mermaid +flowchart LR + A[Broker message arrives] --> B[Persist in app DB] + B --> C[Acknowledge broker immediately] + C --> D[Process later in app] + D --> E{Success?} + E -->|Yes| F[Done] + E -->|No| G[Retry in app] + G --> H[Dead letter queue in app] +``` + +> [!note] Especially useful when brokers don't support redelivery +> Some message brokers, for example SAP Event Mesh, do not allow retriggering delivery or correcting message payloads. +> With the inbox, failures are handled inside your app via the [dead letter queue](#dead-letter-queue), where you have full control over retry and correction. + +Enable the inbox in your configuration: + +::: code-group +```json [Node.js — package.json] +{ + "cds": { + "requires": { + "messaging": { + "inboxed": true + } + } + } +} +``` +```yaml [Java — application.yaml] +cds: + messaging: + services: + - name: messaging-name + inbox: + enabled: true +``` +::: + +::: warning Inboxing changes who owns failure handling +With inboxing enabled, the broker considers the message delivered as soon as your app stores it. +If later processing fails, recovery no longer happens in the broker; it happens in your application's retry and dead letter queue flow. +::: + +### Background Tasks { #background-tasks } + +Event queues are not limited to messaging. +You can schedule arbitrary background tasks such as data replication, cache refresh, or garbage collection. + +**Example:** Replicate data from a remote service every 10 minutes. + +::: code-group +```js [Node.js] +const srv = await cds.connect.to('RemoteService') +await srv.schedule('replicate', { entity: 'Products' }).every('10 minutes') +``` +::: + +> [!note] Node.js only +> The `srv.schedule()` API is currently available in Node.js only. +> In Java, use a `@Scheduled` annotation in combination with a queued outbox service to achieve equivalent behavior. -## Event Queues: Concept +The `schedule()` method is a convenience shortcut that internally queues the call using `cds.queued(srv)` and adds timing options: -The _Outbox Pattern_ is a reliable strategy used in distributed systems to ensure that messages or events are consistently recorded and delivered, even in the face of failures. _Event Queues_ not only apply this pattern to _outbound_ messages, but also to _inbound_ messages and to _internal_ background tasks. So, event queues can be used for four different use cases: +```js +// Execute once, as soon as possible +await srv.schedule('cleanup', { olderThan: '30d' }) -* **Outbox** → for outbound calls to remote services -* **Inbox** → for asynchronously handling inbound requests -* **Background tasks** → e.g., scheduled periodically -* **Remote Callbacks** → implementing SAGA patterns +// Execute once, after a delay +await srv.schedule('cleanup', { olderThan: '30d' }).after('1h') -The core principle remains the same: +// Execute repeatedly +await srv.schedule('replicate', { entity: 'Products' }).every('10 minutes') +``` -Instead of being sent directy to receivers, event messages are persisted in an _Event Queue_ table in the database -- **within the same transaction** as the triggering action, if applicable. +::: tip Real-world example: data federation +The [data federation guide](../integration/data-federation) uses `srv.schedule().every()` to implement polling-based replication, fetching incremental updates from remote services on a regular interval. +::: -Later on, these event messages are read from the database and actually sent to the receiving services, hence **processed asynchronously** -- with retries, if necessary, so guaranteeing **ultimate resilience**. +### Callbacks (SAGA Patterns) { #callbacks } +In distributed transactions, you often need to react when an asynchronous step completes or fails. +Event queues support this with `#succeeded` and `#failed` callback events, enabling compensation logic similar to SAGA patterns. +**Example:** After successfully creating a flight booking via the outbox, replicate the full business object from the remote system. +If the booking fails, notify the user or trigger compensation logic. +::: code-group +```js [Node.js] +const flights = await cds.connect.to('FlightService') -## Outbox { #outbox } +// Called when the queued booking succeeds +flights.after('bookFlight/#succeeded', async (result, req) => { + console.log('Flight booked successfully:', result) + // Replicate booking details from remote +}) -Regarding the _outbox_, please see the following existing documentation: -- [Transactional Outbox](../../java/outbox) in CAP Java -- [Outboxing with `cds.queued`](../../node.js/queue) in CAP Node.js +// Called when the queued booking fails after max retries +flights.after('bookFlight/#failed', async (error, req) => { + console.log('Flight booking failed:', error) + // Trigger compensation logic +}) +``` +::: +::: tip Register on specific events +Callback handlers must be registered for the specific `#succeeded` or `#failed` events. +The `*` wildcard handler is not called for these events. +::: +## How to Use { #how-to-use } -## Inbox { #inbox } +### Queueing a Service { #cds-queued } -Through the _inbox_, inbound messages can be accepted as asynchronous tasks. -That is, the messaging service persists the message to the database, acknowledges it to the message broker, and schedules its processing. +Use `cds.queued(srv)` in Node.js to obtain a queued proxy of any non-database service. +All subsequent dispatches on this proxy are persisted to the event queue and processed asynchronously. -Simply configure your messaging service for Node.js as cds.requires.messaging.inboxed = true and for CAP Java as cds.messaging.services=[{"name": "messaging-name", "inbox": {"enabled": true}}] +::: code-group +```js [Node.js] +const srv = await cds.connect.to('RemoteService') +const qsrv = cds.queued(srv) -**Inboxing moves the dead letter queue into your CAP app❗️** +// All operations are now queued +await qsrv.emit('someEvent', { key: 'value' }) // fire-and-forget +await qsrv.send('someRequest', { key: 'value' }) // request (result discarded) +await qsrv.run(SELECT.from('Products')) // query (result discarded) +``` +::: -With the inbox, all messages are acknowledged towards the message broker regardless of whether they can be processed or not. -Hence, failures need to be managed via the dead letter queue built on `cds.outbox.Messages`. +::: tip `await` is still needed +Even though processing is asynchronous, you still need to `await` because the message is written to the database within the current transaction. +::: + +In Java, use `AsyncCqnService.of(srv, outbox)` to wrap any CAP service with an outbox: + +::: code-group +```java [Java] +OutboxService outbox = runtime.getServiceCatalog() + .getService(OutboxService.class, "FlightsOutbox"); +CqnService remote = runtime.getServiceCatalog() + .getService(CqnService.class, "RemoteService"); + +// Wrap with outbox handling +AsyncCqnService queued = AsyncCqnService.of(remote, outbox); +queued.emit("someEvent", Map.of("key", "value")); +``` +::: + +### Queueing by Configuration { #by-configuration } + +You can queue any service through configuration without changing code. +That is useful when you want to switch a remote integration to durable asynchronous processing centrally. + +::: code-group +```json [Node.js — package.json] +{ + "cds": { + "requires": { + "RemoteService": { + "kind": "odata", + "outboxed": true + } + } + } +} +``` +```yaml [Java — application.yaml] +cds: + outbox: + services: + FlightsOutbox: + maxAttempts: 10 +``` +::: + +### Auto-Outboxed Services { #auto-outboxed } + +The following services are outboxed by default — you don't need any additional configuration: + +| Service | Description | +|---------|-------------| +| `cds.MessagingService` | All messaging services such as Event Mesh and Kafka | +| `cds.AuditLogService` | Audit log events | + +This ensures that messaging and audit log events are sent reliably and never lost because of transaction rollbacks. + +[Learn more about auto-outboxed services in Node.js.](../../node.js/queue#per-configuration){.learn-more} +[Learn more about the outbox in Java.](../../java/outbox#persistent){.learn-more} + +### Service API { #service-api } + +When working with event queues, you interact with the standard CAP service APIs: + +| API | Description | +|-----|-------------| +| `srv.emit(event, data)` | Emit a fire-and-forget event message | +| `srv.send(event, data)` | Send a request; for queued services the direct return value is discarded | +| `srv.run(query)` | Run a CQL query; for queued services the direct return value is discarded | +| `srv.schedule(event, data)` | Schedule a task with optional timing — Node.js only | + +The `schedule()` method supports a fluent API: + +```js +await srv.schedule('task', data) // execute asap +await srv.schedule('task', data).after('1h') // execute after one hour +await srv.schedule('task', data).every('1h') // repeat every hour +``` + +### Unqueueing a Service + +If a service is queued by configuration, you can get back the original synchronous service: + +::: code-group +```js [Node.js] +const srv = cds.unqueued(qsrv) +``` +```java [Java] +CqnService original = outbox.unboxed(outboxedService); +``` +::: + +## Guarantees + +### Transactional Persistence { #no-phantom-events } + +Because the queued message is written in the same database transaction as your business data, a rollback also removes the queued message. +No event is ever dispatched for a transaction that did not commit. + +### Eventual Processing { #exactly-once } + +The persistent queue guarantees transactional persistence and eventual processing. +For database-backed processing, CAP avoids duplicate execution under normal operation, but handlers should still be idempotent to tolerate rare crash windows or external side effects. + +Database changes made during queued processing are committed only if the event is processed successfully. + +### Ordering { #guaranteed-order } + +In Node.js, messages are processed **in parallel** by default with a chunk size of 10 for better throughput. Processing order is therefore not guaranteed. + +To enforce strict ordering, set `parallel: false` in your configuration. This processes messages one at a time and is rarely recommended for production scenarios. + +In Java, the `DefaultOutboxOrdered` outbox processes entries in submission order. +The `DefaultOutboxUnordered` outbox may process entries in parallel across application instances. + +::: code-group +```json [Node.js — package.json] +{ "cds": { "requires": { "queue": { "parallel": false } } } } +``` +```yaml [Java — application.yaml] +cds: + outbox: + services: + DefaultOutboxOrdered: + ordered: true # default + DefaultOutboxUnordered: + ordered: false # default +``` +::: + +## Operational Behavior + +### Error Handling { #errors } + +When processing fails, the system retries the message with exponentially increasing delays. +After a configurable maximum number of attempts, the message is moved to the dead letter queue. + +Some errors are identified as _unrecoverable_ — for example, when a topic is forbidden in SAP Event Mesh. +These messages are immediately moved to the dead letter queue without further retries. + +To mark your own errors as unrecoverable in Node.js: + +```js +const error = new Error('Invalid payload') +error.unrecoverable = true +throw error +``` + +In Java, suppress retries by catching the error and calling `context.setCompleted()`: + +```java +@On(service = "", event = "myEvent") +void process(OutboxMessageEventContext context) { + try { + // processing logic + } catch (Exception e) { + if (isSemanticError(e)) { + context.setCompleted(); // remove from queue, no retry + } else { + throw e; // retry + } + } +} +``` + +## Dead Letter Queue { #dead-letter-queue } + +Messages that exceed the maximum retry count remain in the `cds.outbox.Messages` database table with their error information intact. +These entries form the _dead letter queue_ and require manual intervention — either to fix the underlying issue and retry, or to discard the message. + +For troubleshooting, inspect `cds.outbox.Messages` and pay special attention to `status`, `attempts`, `lastError`, and `lastAttemptTimestamp`. +See [*The Data Model*](#data-model) for the entity structure. + +### Managing Dead Letters + +You can expose a CDS service to manage the dead letter queue with actions to revive or delete entries. + +#### 1. Define the Service + +::: code-group +```cds [srv/outbox-dead-letter-queue-service.cds] +using from '@sap/cds/srv/outbox'; + +@requires: 'internal-user' +service OutboxDeadLetterQueueService { + + @readonly + entity DeadOutboxMessages as projection on cds.outbox.Messages + actions { + action revive(); + action delete(); + }; + +} +``` +::: + +::: warning Restrict access +The dead letter queue contains sensitive data. +Ensure the service is accessible only to internal users. +::: + +#### 2. Filter for Dead Entries + +As `maxAttempts` is configurable, its value cannot be added as a static filter to the projection, but must be applied programmatically. + +::: code-group +```js [Node.js — srv/outbox-dead-letter-queue-service.js] +const cds = require('@sap/cds') + +module.exports = class OutboxDeadLetterQueueService extends cds.ApplicationService { + async init() { + this.before('READ', 'DeadOutboxMessages', function (req) { + const { maxAttempts } = cds.env.requires.outbox + req.query.where('attempts >= ', maxAttempts) + }) + await super.init() + } +} +``` +```java [Java — DeadOutboxMessagesHandler.java] +@Component +@ServiceName(OutboxDeadLetterQueueService_.CDS_NAME) +public class DeadOutboxMessagesHandler implements EventHandler { + + private final PersistenceService db; + + public DeadOutboxMessagesHandler( + @Qualifier(PersistenceService.DEFAULT_NAME) PersistenceService db) { + this.db = db; + } + + @Before(entity = DeadOutboxMessages_.CDS_NAME) + public void addDeadEntryFilter(CdsReadEventContext context) { + Optional outboxFilters = createOutboxFilters(context.getCdsRuntime()); + outboxFilters.ifPresent(filter -> { + CqnSelect modified = copy(context.getCqn(), new Modifier() { + @Override + public CqnPredicate where(Predicate where) { + return filter.and(where); + } + }); + context.setCqn(modified); + }); + } +} +``` +::: + +#### 3. Implement Bound Actions + +Entries in the dead letter queue can be _revived_ by resetting the retry counter to zero, or _deleted_ permanently. + +::: code-group +```js [Node.js — srv/outbox-dead-letter-queue-service.js] +this.on('revive', 'DeadOutboxMessages', async function (req) { + await UPDATE(req.subject).set({ attempts: 0 }) +}) + +this.on('delete', 'DeadOutboxMessages', async function (req) { + await DELETE.from(req.subject) +}) +``` +```java [Java] +@On +public void reviveOutboxMessage(DeadOutboxMessagesReviveContext context) { + CqnAnalyzer analyzer = CqnAnalyzer.create(context.getModel()); + Map key = analyzer.analyze(context.getCqn()).rootKeys(); + Messages msg = Messages.create((String) key.get(Messages.ID)); + msg.setAttempts(0); + db.run(Update.entity(Messages_.class).entry(key).data(msg)); + context.setCompleted(); +} + +@On +public void deleteOutboxEntry(DeadOutboxMessagesDeleteContext context) { + CqnAnalyzer analyzer = CqnAnalyzer.create(context.getModel()); + Map key = analyzer.analyze(context.getCqn()).rootKeys(); + db.run(Delete.from(Messages_.class).byId(key.get(Messages.ID))); + context.setCompleted(); +} +``` +::: [Learn more about the dead letter queue in Node.js.](../../node.js/queue#managing-the-dead-letter-queue){.learn-more} [Learn more about the dead letter queue in Java.](../../java/outbox#outbox-dead-letter-queue){.learn-more} -Inboxing is especially beneficial in case the message broker does not allow to trigger redelivery and/ or "fix" the message payload. +## Deferred Principal Propagation { #principal-propagation } + +When an event is processed asynchronously, the original HTTP request context is no longer available. +CAP handles this as follows: + +- The **user ID** is stored with the queued message and re-created when the message is processed. +- **User roles and attributes** are _not_ stored. Asynchronous processing always runs in privileged mode. + +This means queued handlers must not rely on request-time role checks. +If you need authorization in queued processing, encode the required information in the event payload itself or derive it from persisted business data. + +## Configuration + +### Scheduling vs Legacy Implementation { #scheduling } + +In Node.js there are two queue implementations: + +- **Scheduling-based** (recommended) — enabled by configuring `cds.requires.scheduling`. Uses markers for cross-tenant coordination, especially important for multitenancy. +- **Legacy** — runs when `scheduling` is not configured. Deprecated and to be removed in a future release. + +> [!warning] Use the scheduling-based implementation +> The legacy implementation is deprecated. Enable scheduling in your configuration: +> ```json +> { "cds": { "requires": { "scheduling": {} } } } +> ``` + +### Persistent Queue (Default) { #persistent-queue } + +The persistent queue is enabled by default. +Messages are stored in a database table within the current transaction. + +::: code-group +```json [Node.js — package.json] +{ + "cds": { + "requires": { + "scheduling": { + "markerInterval": "1h", + "flushInterval": "1h" + }, + "queue": { + "maxAttempts": 20, + "chunkSize": 10, + "parallel": true + } + } + } +} +``` +```yaml [Java — application.yaml] +cds: + outbox: + services: + DefaultOutboxOrdered: + maxAttempts: 10 + ordered: true + DefaultOutboxUnordered: + maxAttempts: 10 + ordered: false +``` +::: + +Queue options for Node.js (`cds.requires.queue`): + +| Option | Default | Description | +|--------|---------|-------------| +| `maxAttempts` | `20` | Maximum retries before a message becomes a dead letter | +| `chunkSize` | `10` | Number of messages to process per batch | +| `parallel` | `true` | Process messages in parallel; set to `false` for strict ordering | +| `storeLastError` | `true` | Store error information of the last failed attempt | +| `timeout` | `"1h"` | Time after which a `processing` message is considered abandoned | +| `legacyLocking` | `false` | Backward compatibility with `@sap/cds` v9; to be removed in a future release | + +Scheduling options for Node.js (`cds.requires.scheduling`): + +| Option | Default | Description | +|--------|---------|-------------| +| `markerInterval` | `"1h"` | Offset added to a marker timestamp before processing | +| `flushInterval` | `"1h"` | How often the central runner checks for tenants with pending work | + +Configuration options for Java: + +| Option | Default | Description | +|--------|---------|-------------| +| `maxAttempts` | `10` | Maximum retries before the entry is considered dead | +| `ordered` | `true` | Process entries in submission order | + +### In-Memory Queue + +For development and testing, you can use the in-memory queue. +Messages are held in memory and emitted after the transaction commits. + +::: code-group +```json [Node.js — package.json] +{ + "cds": { + "requires": { + "queue": { + "kind": "in-memory-queue" + } + } + } +} +``` +::: + +::: warning No retry mechanism +With the in-memory queue, messages are lost if processing fails. +There is no retry mechanism and no dead letter queue. +::: + +### Disabling the Queue + +You can disable event queues globally: + +```json +{ + "cds": { + "requires": { + "queue": false + } + } +} +``` + +Or disable queueing for a specific service: + +```json +{ + "cds": { + "requires": { + "messaging": { + "outboxed": false + } + } + } +} +``` + +## Manual Processing { #flush } +In single-tenancy, the background runner starts on application startup and processes pending messages automatically. +In multitenancy, the central runner periodically checks markers and triggers processing. +You can also trigger processing manually: -## More to Come +::: code-group +```js [Node.js] +// Flush a specific queue +const srv = await cds.connect.to('RemoteService') +await cds.flush(srv.name) -This documentation is not complete yet, or the APIs are not released for general availability. -Stay tuned to upcoming releases for further updates. +// Flush all queues +await cds.flush() +``` +::: From d4a33f05dbac83a3da36c958a7457ce5cca68f6c Mon Sep 17 00:00:00 2001 From: D050513 Date: Tue, 2 Jun 2026 00:45:45 +0200 Subject: [PATCH 06/15] some restructuring --- guides/events/event-queues.md | 822 +++++++++++++++------------------- 1 file changed, 373 insertions(+), 449 deletions(-) diff --git a/guides/events/event-queues.md b/guides/events/event-queues.md index 3fe9d0ce02..02249192ed 100644 --- a/guides/events/event-queues.md +++ b/guides/events/event-queues.md @@ -1,266 +1,65 @@ --- synopsis: > - Transactional Event Queues allow you to schedule events and background tasks for asynchronous, resilient processing in the same transaction as your business data. + Transactional Event Queues let you persist events and background tasks in the same database transaction as your business data, then process them asynchronously with retries and a dead letter queue. status: released --- # Transactional Event Queues -Queue remote calls, inbound messages, and background tasks in the same transaction as your business data. CAP executes them later with retries, ordering, and dead letter handling. -{.subtitle} - {{ $frontmatter.synopsis }} {.abstract} > [!tip] Guiding Principles > -> 1. **Transactional** — queued work is written in the same transaction as your business data -> 2. **Asynchronous** — a background runner dispatches it after commit, not during the request -> 3. **Resilient** — failed work is retried with exponential backoff; unrecoverable entries land in a dead letter queue -> 4. **Unified** — one mechanism covers outbox, inbox, background tasks, and callbacks +> 1. **Transactional** — queued work is written in the same transaction as your business data. +> 2. **Asynchronous** — a background runner dispatches it after commit, not during the request. +> 3. **Resilient** — failed work is retried with exponential backoff; unrecoverable entries land in a dead letter queue. [[toc]] -## Before You Continue - -This guide assumes familiarity with CAP services, transactions, and remote service connections. -For broker-based scenarios such as SAP Event Mesh or Kafka, basic messaging concepts are also helpful. ## Motivation Distributed side effects are hard to get right. -Your application may commit local data, but a follow-up remote call can still fail because of network errors, service outages, or a process crash. +An application may commit local data, but a follow-up remote call can still fail because of network errors, service outages, or a process crash. -_Transactional Event Queues_ solve this by storing the follow-up work in the database as part of the same transaction as your business data. +_Transactional Event Queues_ solve this by storing the follow-up work in the database as part of the **same transaction** as your business data. After commit, a background runner executes that work asynchronously and retries failures until they succeed or become dead letters. -This pattern is often known as the _Transactional Outbox_, but CAP's event queues go beyond outbound messages. -They provide one mechanism for four use cases: +```mermaid +flowchart LR + A[Request handler] -- in tx --> B[(Business data)] + A -- in tx --> C[(Queued message)] + B & C --> D{COMMIT} + D -. async .-> E[Background runner] + E --> F[Remote service] + E --> G[Background task] +``` + +This pattern is widely known as the _Transactional Outbox_, but CAP's event queues go beyond outbound messages. They cover four use cases: - **Outbox** — defer outbound calls to remote services until the transaction succeeds. - **Inbox** — acknowledge inbound messages immediately and process them asynchronously. - **Background Tasks** — schedule periodic or delayed tasks such as data replication. - **Callbacks** — react to completed or failed tasks, enabling SAGA-like compensation patterns. -Transactional Event Queues are not just a broker integration feature. -They are CAP’s general mechanism for persisting asynchronous work in the database, whether that work is an outbound message, an inbound message to be processed later, or a background task scheduled by your application. - -## When to Use Them - -Use event queues when work must happen _after_ the current transaction commits, or when that work needs durable retries and dead letter handling. -If you need an immediate response from a remote system, use a normal synchronous service call instead. - -| If you need to... | Use event queues? | Why | -|---|---|---| -| Call a remote service only after DB commit | Yes | Prevents external side effects for rolled-back transactions | -| Process inbound broker messages without blocking acknowledgment | Yes | Lets the app acknowledge early and process later | -| Schedule delayed or recurring background work | Yes | Uses the same persistence and retry mechanism | -| Need an immediate synchronous response from a remote call | No | Queued requests execute later and discard the direct return value | -| Run purely local logic inside the same request | Usually no | Direct execution is simpler when no asynchronous boundary is needed | - -```mermaid -flowchart TD - A[Need asynchronous work?] -->|No| B[Use direct service call] - A -->|Yes| C[Must only happen after DB commit?] - C -->|Yes| D[Use transactional event queue] - C -->|No| E[Need durable retries and DLQ?] - E -->|Yes| D - E -->|No| F[Simple async logic may be enough] - - D --> G[Outbox] - D --> H[Inbox] - D --> I[Background task] - D --> J[Callback / compensation] -``` ## Quick Start Use a queued service when a side effect must only happen after the current transaction commits. ```js -const flights = await cds.connect.to('FlightService') -const queuedFlights = cds.queued(flights) +const xflights = await cds.connect.to('xflights') +const qd_xflights = cds.queued(xflights) -this.after('CREATE', 'Travels', async travel => { - await queuedFlights.send('bookFlight', { travelId: travel.ID }) +this.after('CREATE', 'Travels', async (_, req) => { + await qd_xflights.send('bookFlight', { travelId: req.data.ID }) }) ``` This stores the flight booking request in the database together with the travel creation. -CAP dispatches it later in the background. -If the transaction rolls back, no booking request is sent. - -## How It Works { #concept } - -The core principle is straightforward: - -1. Instead of executing side effects directly, you write a message into a database table — **within the current transaction**. -2. Once the transaction commits, a background runner reads pending messages and dispatches them to the respective service. -3. If processing succeeds, the message is deleted. -4. If processing fails, the system retries with exponentially increasing delays. -5. After a configurable maximum number of attempts, the message is moved to the dead letter queue for manual intervention. - -```mermaid -sequenceDiagram - participant H as Event Handler - participant DB as Database - participant R as Background Runner - participant S as Remote Service - - H->>DB: Write business data - H->>DB: Write queued message - Note over H,DB: Both writes happen in the same transaction - DB-->>H: COMMIT - - loop Background processing - R->>DB: Poll for pending entries - R->>S: Dispatch work - alt Success - R->>DB: Delete message - else Transient failure - R->>DB: Increment retry counter - Note over R: Retry with exponential backoff - else Max retries exceeded - R->>DB: Mark as dead letter - end - end -``` - -Because the queued message and your business data share the same database transaction, you get two core guarantees: - -- **No phantom events** — if the transaction rolls back, no message is sent. -- **No lost events** — if the transaction commits, the queued work is persisted and processed eventually. - -There is at most one active processor per service and tenant at a given time. -That is important for understanding ordering and duplicate prevention. - -### Callback Events { #callback-events } - -When a queued message finishes processing, the system emits a callback event on the same service: - -- **`/#succeeded`** — emitted when processing completes successfully. -- **`/#failed`** — emitted when the message becomes a dead letter (after all retries are exhausted). - -Callback events let you react to outcomes — for example, to trigger compensation logic after a failure or to replicate data after a successful remote call. - -- **Queued transactionally** — callback events are written to the queue in the same transaction as the main event processing, so they only exist if processing commits. -- **Processed asynchronously** — like any queued event, they run in their own transaction afterwards. - -> [!note] Emitted by the background runner -> Callback events are emitted by the background runner, not during the original request that queued the message. - -### Single-Tenancy vs Multi-Tenancy { #tenancy } - -Event queues work in both single-tenant and multi-tenant deployments. - -[Learn more about multitenancy.](../multitenancy/){.learn-more} - -#### Single-Tenancy - -Messages are stored in a queue table that resides in the application database. A background runner starts when your application starts and processes messages continuously. When a transaction commits, processing is triggered immediately. - -#### Multi-Tenancy - -Each tenant has its own database. To coordinate across tenants, the system writes lightweight markers to a central (provider) database whenever messages are queued. A central runner periodically checks these markers and triggers processing for each tenant that has pending work. - -This ensures that: -- Pending messages are recovered after application restarts or shutdowns. -- Each tenant's messages are processed independently. - -### The Data Model { #data-model } +CAP dispatches it later in the background. If the transaction rolls back, no booking request is sent. -Your database model is automatically extended with the following entity, used by the persistent queue: - -```cds -namespace cds.outbox; - -entity Messages { - key ID : UUID; // Unique message identifier - timestamp : Timestamp; // When the message was queued - target : String; // Target service/queue name - msg : LargeString; // Serialized event payload - attempts : Integer default 0; // Number of processing attempts - partition : Integer default 0; // Reserved, currently unused - lastError : LargeString; // Error from last failed attempt - lastAttemptTimestamp : Timestamp; // When last attempt occurred - status : String(23); // Current processing status - task : String(255); // Task name for named/singleton tasks - appid : String(255); // Application ID for shared HDI containers -} -``` - -## Direct vs Queued Calls - -A queued call changes _when_ work happens and _what the caller can expect back_. -That difference is easier to understand when seen side by side. - -```mermaid -sequenceDiagram - participant App - participant DB - participant Remote - - rect rgb(255,245,245) - Note over App,Remote: Direct call - App->>Remote: send() - Remote-->>App: result or error - App->>DB: commit business data - end - - rect rgb(245,255,245) - Note over App,Remote: Queued call - App->>DB: write business data - App->>DB: write queued message - DB-->>App: commit - App-->>App: request finished - DB-->>Remote: background dispatch later - end -``` - -> [!warning] Queued calls are asynchronous -> A queued service persists the request and returns after the message is stored, not after the remote operation finishes. -> Any return value from `send()` or `run()` is therefore not available to the caller. - -## End-to-End Example - -The following example ties together queueing, callbacks, and local state updates. -It shows a common pattern: create local business data first, then trigger remote work asynchronously. - -```js -const cds = require('@sap/cds') - -module.exports = class TravelService extends cds.ApplicationService { - async init() { - const flights = await cds.connect.to('FlightService') - const queuedFlights = cds.queued(flights) - - this.after('CREATE', 'Travels', async travel => { - await queuedFlights.send('bookFlight', { - travelId: travel.ID, - customerId: travel.customer_ID - }) - }) - - flights.after('bookFlight/#succeeded', async (_, req) => { - await UPDATE('Travels') - .set({ status: 'Booked' }) - .where({ ID: req.data.travelId }) - }) - - flights.after('bookFlight/#failed', async (err, req) => { - await UPDATE('Travels') - .set({ status: 'BookingFailed' }) - .where({ ID: req.data.travelId }) - req.warn(`Flight booking permanently failed: ${err.message}`) - }) - - await super.init() - } -} -``` - -This example highlights an important design rule: -use callbacks or persisted status updates for outcomes, not direct return values. ## Use Cases @@ -269,7 +68,7 @@ use callbacks or persisted status updates for outcomes, not direct return values The outbox defers outbound calls to remote services until the main transaction succeeds. This prevents sending requests to external systems when your transaction might still roll back. -**Example:** When creating a travel booking, you also need to notify an external flight service. +**Example:** When creating a travel booking, you also want to notify an external flight service. Without the outbox, the notification could be sent even if the booking transaction fails. ::: code-group @@ -277,30 +76,30 @@ Without the outbox, the notification could be sent even if the booking transacti const xflights = await cds.connect.to('xflights') const qd_xflights = cds.queued(xflights) -this.after('CREATE', 'Travels', async (travel) => { +this.after('CREATE', 'Travels', async (_, req) => { // Persisted within the current transaction, sent after commit - await qd_xflights.send('bookFlight', { travelId: travel.ID }) + await qd_xflights.send('bookFlight', { travelId: req.data.ID }) }) ``` ```java [Java] -@Autowired @Qualifier("FlightsOutbox") +@Autowired @Qualifier("XFlightsOutbox") OutboxService outbox; @Autowired @Qualifier(CqnService.DEFAULT_NAME) -CqnService remoteFlights; +CqnService xflights; @After(event = CqnService.EVENT_CREATE, entity = Travels_.CDS_NAME) -void notifyFlights(List travels) { - AsyncCqnService outboxedFlights = AsyncCqnService.of(remoteFlights, outbox); - travels.forEach(t -> outboxedFlights.emit("bookFlight", Map.of("travelId", t.getId()))); +void notifyXFlights(List travels) { + AsyncCqnService outboxedXFlights = AsyncCqnService.of(xflights, outbox); + travels.forEach(t -> outboxedXFlights.emit("bookFlight", Map.of("travelId", t.getId()))); } ``` ::: ```js // Anti-pattern: remote side effect happens before local commit is safe -this.after('CREATE', 'Travels', async travel => { - await flights.send('bookFlight', { travelId: travel.ID }) +this.after('CREATE', 'Travels', async (_, req) => { + await xflights.send('bookFlight', { travelId: req.data.ID }) }) ``` @@ -314,29 +113,19 @@ You don't need to call `cds.queued()` or configure anything extra for these — [Learn more about auto-outboxed services in Node.js.](../../node.js/queue#per-configuration){.learn-more} [Learn more about the outbox in Java.](../../java/outbox){.learn-more} + ### Inbox { #inbox } The inbox mirrors the outbox pattern for inbound messages. -When a message arrives from a broker like SAP Event Mesh or Apache Kafka, the messaging service immediately persists it to the database, acknowledges it to the broker, and schedules its processing. +When a message arrives from a broker, the messaging service immediately persists it to the database, acknowledges it to the broker, and schedules its processing. This brings two advantages: -- **Quick acknowledgment** — the message broker does not have to wait for your processing to complete. This reduces backpressure and prevents consumer group rebalancing under load. +- **Quick acknowledgment** — the broker doesn't have to wait for your processing to complete, which keeps consumer throughput high under load. - **Flatten the curve** — if a burst of messages arrives, they are queued in your database and processed at a controlled pace. -```mermaid -flowchart LR - A[Broker message arrives] --> B[Persist in app DB] - B --> C[Acknowledge broker immediately] - C --> D[Process later in app] - D --> E{Success?} - E -->|Yes| F[Done] - E -->|No| G[Retry in app] - G --> H[Dead letter queue in app] -``` - > [!note] Especially useful when brokers don't support redelivery -> Some message brokers, for example SAP Event Mesh, do not allow retriggering delivery or correcting message payloads. +> Some message brokers do not allow retriggering delivery or correcting message payloads. > With the inbox, failures are handled inside your app via the [dead letter queue](#dead-letter-queue), where you have full control over retry and correction. Enable the inbox in your configuration: @@ -368,17 +157,18 @@ With inboxing enabled, the broker considers the message delivered as soon as you If later processing fails, recovery no longer happens in the broker; it happens in your application's retry and dead letter queue flow. ::: + ### Background Tasks { #background-tasks } Event queues are not limited to messaging. You can schedule arbitrary background tasks such as data replication, cache refresh, or garbage collection. -**Example:** Replicate data from a remote service every 10 minutes. +**Example:** Replicate airport master data from the xflights service every 10 minutes. ::: code-group ```js [Node.js] -const srv = await cds.connect.to('RemoteService') -await srv.schedule('replicate', { entity: 'Products' }).every('10 minutes') +const xflights = await cds.connect.to('xflights') +await xflights.schedule('replicate', { entity: 'Airports' }).every('10 minutes') ``` ::: @@ -386,24 +176,43 @@ await srv.schedule('replicate', { entity: 'Products' }).every('10 minutes') > The `srv.schedule()` API is currently available in Node.js only. > In Java, use a `@Scheduled` annotation in combination with a queued outbox service to achieve equivalent behavior. -The `schedule()` method is a convenience shortcut that internally queues the call using `cds.queued(srv)` and adds timing options: +The `schedule()` method is a convenience shortcut for `cds.queued(srv).send(event, data)` with optional timing: ```js // Execute once, as soon as possible -await srv.schedule('cleanup', { olderThan: '30d' }) +await xflights.schedule('cleanup', { olderThan: '30d' }) // Execute once, after a delay -await srv.schedule('cleanup', { olderThan: '30d' }).after('1h') +await xflights.schedule('cleanup', { olderThan: '30d' }).after('1h') + +// Execute repeatedly — supports time strings and cron expressions +await xflights.schedule('replicate', { entity: 'Airports' }).every('10 minutes') +await xflights.schedule('replicate', { entity: 'Airports' }).every('*/10 * * * *') +``` + +`.after()` accepts milliseconds (as a number) or a time string such as `'1s'`, `'10m'`, `'1h'`. +`.every()` accepts the same plus a five-field cron expression. + +#### Singleton Tasks { #singleton-tasks } + +Use `srv.schedule.task()` to schedule a *singleton task* — a task identified by name that exists only once: + +```js +// Replace any existing 'replicate' task with a new schedule +await xflights.schedule.task('replicate', { entity: 'Airports' }).every('10 minutes') -// Execute repeatedly -await srv.schedule('replicate', { entity: 'Products' }).every('10 minutes') +// Remove the task +await xflights.unschedule.task('replicate') ``` +The event name doubles as the task name. A subsequent call with the same name overwrites the previous schedule (tasks are upserted, not deduplicated). This is convenient for idempotent registration during application startup. + ::: tip Real-world example: data federation The [data federation guide](../integration/data-federation) uses `srv.schedule().every()` to implement polling-based replication, fetching incremental updates from remote services on a regular interval. ::: -### Callbacks (SAGA Patterns) { #callbacks } + +### Callbacks (SAGA Patterns) { #callbacks } In distributed transactions, you often need to react when an asynchronous step completes or fails. Event queues support this with `#succeeded` and `#failed` callback events, enabling compensation logic similar to SAGA patterns. @@ -413,43 +222,118 @@ If the booking fails, notify the user or trigger compensation logic. ::: code-group ```js [Node.js] -const flights = await cds.connect.to('FlightService') +const xflights = await cds.connect.to('xflights') // Called when the queued booking succeeds -flights.after('bookFlight/#succeeded', async (result, req) => { +xflights.after('bookFlight/#succeeded', async (result, req) => { console.log('Flight booked successfully:', result) // Replicate booking details from remote }) // Called when the queued booking fails after max retries -flights.after('bookFlight/#failed', async (error, req) => { +xflights.after('bookFlight/#failed', async (error, req) => { console.log('Flight booking failed:', error) // Trigger compensation logic }) ``` ::: +> [!note] Node.js only +> Callback events `#succeeded` and `#failed` are currently available in Node.js only. + ::: tip Register on specific events Callback handlers must be registered for the specific `#succeeded` or `#failed` events. The `*` wildcard handler is not called for these events. ::: + +## Guarantees + +### Transactional Persistence { #no-phantom-events } + +Because the queued message is written in the same database transaction as your business data, a rollback also removes the queued message. +No event is ever dispatched for a transaction that didn't commit. + +### Eventual Processing { #exactly-once } + +The persistent queue guarantees transactional persistence and eventual processing. +For database-backed processing, CAP avoids duplicate execution under normal operation, but handlers should still be idempotent to tolerate rare crash windows or external side effects. + +Database changes made during queued processing are committed only if the event is processed successfully. + + +## End-to-End Example + +The following example ties together queueing, callbacks, and local state updates. +It shows a common pattern: create local business data first, then trigger remote work asynchronously, then react to its outcome. + +```js +const cds = require('@sap/cds') + +module.exports = class TravelService extends cds.ApplicationService { + async init() { + const xflights = await cds.connect.to('xflights') + const qd_xflights = cds.queued(xflights) + + this.after('CREATE', 'Travels', async (_, req) => { + await qd_xflights.send('bookFlight', { + travelId: req.data.ID, + customerId: req.data.customer_ID + }) + }) + + xflights.after('bookFlight/#succeeded', async (_, req) => { + await UPDATE('Travels') + .set({ status: 'Booked' }) + .where({ ID: req.data.travelId }) + }) + + xflights.after('bookFlight/#failed', async (err, req) => { + await UPDATE('Travels') + .set({ status: 'BookingFailed' }) + .where({ ID: req.data.travelId }) + req.warn(`Flight booking permanently failed: ${err.message}`) + }) + + await super.init() + } +} +``` + +This example highlights an important design rule: +use callbacks or persisted status updates for outcomes, not direct return values. + + +## When to Use Event Queues + +Use an event queue when work must happen *after* the current transaction commits, or when that work needs durable retries and a dead letter queue. +For an immediate, synchronous response from a remote system, use a normal service call. + +### Direct vs Queued Calls + +A queued call changes _when_ work happens and _what the caller can expect back_: + +- A **direct** call returns the remote service's result (or error) and only then commits the local transaction. +- A **queued** call writes the message to the queue inside the local transaction and returns. The actual remote dispatch happens after commit, in the background. + +> [!warning] Queued calls discard the direct return value +> A queued service persists the request and returns after the message is stored, not after the remote operation finishes. +> Any return value from `send()` or `run()` is therefore not available to the caller. To act on the outcome, register a [callback handler](#callbacks) on `#succeeded` or `#failed`. + + ## How to Use { #how-to-use } ### Queueing a Service { #cds-queued } -Use `cds.queued(srv)` in Node.js to obtain a queued proxy of any non-database service. +Use `cds.queued(srv)` in Node.js to obtain a queued proxy of any service. All subsequent dispatches on this proxy are persisted to the event queue and processed asynchronously. ::: code-group ```js [Node.js] -const srv = await cds.connect.to('RemoteService') -const qsrv = cds.queued(srv) +const xflights = await cds.connect.to('xflights') +const qd_xflights = cds.queued(xflights) -// All operations are now queued -await qsrv.emit('someEvent', { key: 'value' }) // fire-and-forget -await qsrv.send('someRequest', { key: 'value' }) // request (result discarded) -await qsrv.run(SELECT.from('Products')) // query (result discarded) +await qd_xflights.send('bookFlight', { travelId: 'T-42' }) // request (result discarded) ``` ::: @@ -462,19 +346,19 @@ In Java, use `AsyncCqnService.of(srv, outbox)` to wrap any CAP service with an o ::: code-group ```java [Java] OutboxService outbox = runtime.getServiceCatalog() - .getService(OutboxService.class, "FlightsOutbox"); -CqnService remote = runtime.getServiceCatalog() - .getService(CqnService.class, "RemoteService"); + .getService(OutboxService.class, "XFlightsOutbox"); +CqnService xflights = runtime.getServiceCatalog() + .getService(CqnService.class, "xflights"); // Wrap with outbox handling -AsyncCqnService queued = AsyncCqnService.of(remote, outbox); -queued.emit("someEvent", Map.of("key", "value")); +AsyncCqnService queued = AsyncCqnService.of(xflights, outbox); +queued.emit("bookFlight", Map.of("travelId", "T-42")); ``` ::: ### Queueing by Configuration { #by-configuration } -You can queue any service through configuration without changing code. +You can queue any *outbound* service through configuration without changing code. That is useful when you want to switch a remote integration to durable asynchronous processing centrally. ::: code-group @@ -482,7 +366,7 @@ That is useful when you want to switch a remote integration to durable asynchron { "cds": { "requires": { - "RemoteService": { + "xflights": { "kind": "odata", "outboxed": true } @@ -494,7 +378,7 @@ That is useful when you want to switch a remote integration to durable asynchron cds: outbox: services: - FlightsOutbox: + XFlightsOutbox: maxAttempts: 10 ``` ::: @@ -505,7 +389,7 @@ The following services are outboxed by default — you don't need any additional | Service | Description | |---------|-------------| -| `cds.MessagingService` | All messaging services such as Event Mesh and Kafka | +| `cds.MessagingService` | All messaging services | | `cds.AuditLogService` | Audit log events | This ensures that messaging and audit log events are sent reliably and never lost because of transaction rollbacks. @@ -519,10 +403,10 @@ When working with event queues, you interact with the standard CAP service APIs: | API | Description | |-----|-------------| -| `srv.emit(event, data)` | Emit a fire-and-forget event message | | `srv.send(event, data)` | Send a request; for queued services the direct return value is discarded | -| `srv.run(query)` | Run a CQL query; for queued services the direct return value is discarded | | `srv.schedule(event, data)` | Schedule a task with optional timing — Node.js only | +| `srv.schedule.task(event, data)` | Schedule a *singleton* task identified by name — Node.js only | +| `srv.unschedule.task(name)` | Remove a previously scheduled singleton task — Node.js only | The `schedule()` method supports a fluent API: @@ -530,59 +414,233 @@ The `schedule()` method supports a fluent API: await srv.schedule('task', data) // execute asap await srv.schedule('task', data).after('1h') // execute after one hour await srv.schedule('task', data).every('1h') // repeat every hour +await srv.schedule('task', data).every('*/5 * * * *') // every 5 minutes via cron ``` +[Learn more about singleton tasks.](#singleton-tasks){.learn-more} + ### Unqueueing a Service If a service is queued by configuration, you can get back the original synchronous service: ::: code-group ```js [Node.js] -const srv = cds.unqueued(qsrv) +const xflights = cds.unqueued(qd_xflights) ``` ```java [Java] -CqnService original = outbox.unboxed(outboxedService); +CqnService xflights = outbox.unboxed(outboxedXFlights); ``` ::: -## Guarantees +### Configuration -### Transactional Persistence { #no-phantom-events } +The persistent queue is enabled by default. +Messages are stored in a database table within the current transaction. -Because the queued message is written in the same database transaction as your business data, a rollback also removes the queued message. -No event is ever dispatched for a transaction that did not commit. +::: code-group +```json [Node.js — package.json] +{ + "cds": { + "requires": { + "scheduling": {}, + "queue": { + "maxAttempts": 20, + "chunkSize": 10 + } + } + } +} +``` +```yaml [Java — application.yaml] +cds: + outbox: + services: + DefaultOutboxOrdered: + maxAttempts: 10 + ordered: true + DefaultOutboxUnordered: + maxAttempts: 10 + ordered: false +``` +::: -### Eventual Processing { #exactly-once } +::: details Queue and scheduling options for Node.js -The persistent queue guarantees transactional persistence and eventual processing. -For database-backed processing, CAP avoids duplicate execution under normal operation, but handlers should still be idempotent to tolerate rare crash windows or external side effects. +`cds.requires.queue`: -Database changes made during queued processing are committed only if the event is processed successfully. +| Option | Default | Description | +|--------|---------|-------------| +| `maxAttempts` | `20` | Maximum retries before a message becomes a dead letter | +| `chunkSize` | `10` | Number of messages to process per batch | +| `storeLastError` | `true` | Store error information of the last failed attempt | +| `timeout` | `"1h"` | Time after which a `processing` message is considered abandoned | -### Ordering { #guaranteed-order } +`cds.requires.scheduling` (multitenancy coordination): -In Node.js, messages are processed **in parallel** by default with a chunk size of 10 for better throughput. Processing order is therefore not guaranteed. +| Option | Description | +|--------|-------------| +| `markerInterval` | Grid interval for markers; CAP picks a default that spreads tenant load across the interval | +| `flushInterval` | Cadence at which the central runner checks for tenants with pending work | -To enforce strict ordering, set `parallel: false` in your configuration. This processes messages one at a time and is rarely recommended for production scenarios. +::: -In Java, the `DefaultOutboxOrdered` outbox processes entries in submission order. -The `DefaultOutboxUnordered` outbox may process entries in parallel across application instances. +::: details Configuration options for Java + +| Option | Default | Description | +|--------|---------|-------------| +| `maxAttempts` | `10` | Maximum retries before the entry is considered dead | +| `ordered` | `true` | Process entries in submission order | + +::: + +#### Disabling the Queue + +You can disable event queues globally: + +```json +{ + "cds": { + "requires": { + "queue": false + } + } +} +``` + +Or disable queueing for a specific service: + +```json +{ + "cds": { + "requires": { + "messaging": { + "outboxed": false + } + } + } +} +``` + +### Manual Processing { #flush } + +In single-tenancy, the background runner starts on application startup and processes pending messages automatically. +In multitenancy, the central runner periodically checks markers and triggers processing. + +You can also trigger processing manually: ::: code-group -```json [Node.js — package.json] -{ "cds": { "requires": { "queue": { "parallel": false } } } } +```js [Node.js] +// Flush a specific queue +const xflights = await cds.connect.to('xflights') +await cds.flush(xflights.name) + +// Flush all queues +await cds.flush() ``` -```yaml [Java — application.yaml] -cds: - outbox: - services: - DefaultOutboxOrdered: - ordered: true # default - DefaultOutboxUnordered: - ordered: false # default +::: + + +## How It Works { #concept } + +The core principle is straightforward: + +1. Instead of executing side effects directly, you write a message into a database table — **within the current transaction**. +2. Once the transaction commits, a background runner reads pending messages and dispatches them to the respective service. +3. If processing succeeds, the message is deleted. +4. If processing fails, the system retries with exponentially increasing delays. +5. After a configurable maximum number of attempts, the message is moved to the dead letter queue for manual intervention. + +```mermaid +sequenceDiagram + participant H as Handler + participant DB as Database + participant R as Runner + participant S as Service + + H->>DB: business data + queued message + Note over H,DB: same transaction + DB-->>H: COMMIT + R->>DB: poll & lock + R->>S: dispatch + alt success + R->>DB: delete + else transient failure + R->>DB: retry later + else max retries + R->>DB: dead letter + end +``` + +Because the queued message and your business data share the same database transaction, you get two core guarantees: + +- **No phantom events** — if the transaction rolls back, no message is sent. +- **No lost events** — if the transaction commits, the queued work is persisted and processed eventually. + +### Single-Tenancy vs Multi-Tenancy { #tenancy } + +Event queues work in both single-tenant and multi-tenant deployments. In both cases, processing is triggered immediately after commit; markers are an optimization plus an extra layer of resilience. + +[Learn more about multitenancy.](../multitenancy/){.learn-more} + +#### Single-Tenancy + +Messages are stored in a queue table that resides in the application database. A background runner starts when your application starts and processes messages continuously. + +#### Multi-Tenancy + +Each tenant has its own database. To avoid having a central runner periodically scan every tenant, the system writes a lightweight *marker* to a central (provider) database whenever messages are queued. On startup the central runner only triggers processing for tenants that actually have pending work, and rechecks periodically as a recovery layer in case a runner crashed before processing completed. + +CAP spreads marker timestamps across tenants so that processing doesn't synchronize into bursts — you don't need to configure that. + +::: details Architecture: scheduling, processing, recovery +Behind the scenes, event queues run three independent loops: + +1. **Scheduling** — calling `srv.send()`, `srv.emit()`, or `srv.schedule()` on a queued service writes the message to the tenant's queue table within the current transaction. In multitenancy, a *marker* is also written to the provider database, recording that this tenant has pending work. +2. **Processing** — a tenant-local task runner reads a chunk of messages, dispatches each event in its own transaction, and deletes successful messages. Failed messages are rescheduled with exponentially increasing delay; after `maxAttempts` they become dead letters. +3. **Recovery** — a central runner periodically polls the provider markers and triggers processing for any tenant with pending work. This recovers from application restarts and tenants that became "cold" without losing messages. + +*Markers* contain no business data — only the information that some queue of some tenant needs to be flushed at some point in time. +::: + +### Locking and Migration { #locking } + +CAP uses **application-level locking** to coordinate processors across application instances. When a runner picks up a message, it sets the message's `status` to `processing`; other runners skip messages in that state. After processing, the row lock is released; the message is deleted (on success) or rescheduled (on failure) in the processing transaction. + +::: warning Migrating across `@sap/cds` major versions +This guide describes the implementation in `@sap/cds` 10+. Older versions select messages differently: + +- **`@sap/cds` 8** does **not** check the `status` column at all. +- **`@sap/cds` 9** checks `status` but holds a row-level lock for the duration of processing (`legacyLocking: true` is the default in cds 9). +- **`@sap/cds` 10** uses application-level locking via `status` and releases the row lock after selection. + +A rolling upgrade from `@sap/cds` 8 directly to 10 can therefore lead to **double-processing of messages**, because cds 8 instances pick up messages that a cds 10 instance has already marked `processing`. Plan downtime, drain the queue before upgrading, or upgrade through cds 9 first. +::: + +### The Data Model { #data-model } + +The persistent queue stores its messages in this entity, automatically added to your database model: + +::: details `cds.outbox.Messages` +```cds +namespace cds.outbox; + +entity Messages { + key ID : UUID; // Unique message identifier + timestamp : Timestamp; // When the message was queued + target : String; // Target service/queue name + msg : LargeString; // Serialized event payload + attempts : Integer default 0; // Number of processing attempts + partition : Integer default 0; + lastError : LargeString; // Error from last failed attempt + lastAttemptTimestamp : Timestamp; // When last attempt occurred + status : String(23); // Current processing status + task : String(255); // Task name for named/singleton tasks + appid : String(255); // Application ID for shared HDI containers +} ``` ::: + ## Operational Behavior ### Error Handling { #errors } @@ -590,9 +648,19 @@ cds: When processing fails, the system retries the message with exponentially increasing delays. After a configurable maximum number of attempts, the message is moved to the dead letter queue. -Some errors are identified as _unrecoverable_ — for example, when a topic is forbidden in SAP Event Mesh. +Some errors are identified as _unrecoverable_ — for example, when a topic is forbidden by the broker. These messages are immediately moved to the dead letter queue without further retries. +::: details When is a message picked up next? +A pending message is *processable* when all three conditions hold: + +1. Its scheduled timestamp plus the retry backoff (`attempts × `) is in the past. +2. Its `attempts` count is less than `maxAttempts`. +3. Its `status` is not `processing`, or its `processing` status has timed out (`timeout`). + +Messages that fail criterion 2 become dead letters. Messages that fail criterion 3 are skipped on this run and become eligible again once the lock times out (recovery from a crashed runner). +::: + To mark your own errors as unrecoverable in Node.js: ```js @@ -618,6 +686,7 @@ void process(OutboxMessageEventContext context) { } ``` + ## Dead Letter Queue { #dead-letter-queue } Messages that exceed the maximum retry count remain in the `cds.outbox.Messages` database table with their error information intact. @@ -740,6 +809,7 @@ public void deleteOutboxEntry(DeadOutboxMessagesDeleteContext context) { [Learn more about the dead letter queue in Node.js.](../../node.js/queue#managing-the-dead-letter-queue){.learn-more} [Learn more about the dead letter queue in Java.](../../java/outbox#outbox-dead-letter-queue){.learn-more} + ## Deferred Principal Propagation { #principal-propagation } When an event is processed asynchronously, the original HTTP request context is no longer available. @@ -750,149 +820,3 @@ CAP handles this as follows: This means queued handlers must not rely on request-time role checks. If you need authorization in queued processing, encode the required information in the event payload itself or derive it from persisted business data. - -## Configuration - -### Scheduling vs Legacy Implementation { #scheduling } - -In Node.js there are two queue implementations: - -- **Scheduling-based** (recommended) — enabled by configuring `cds.requires.scheduling`. Uses markers for cross-tenant coordination, especially important for multitenancy. -- **Legacy** — runs when `scheduling` is not configured. Deprecated and to be removed in a future release. - -> [!warning] Use the scheduling-based implementation -> The legacy implementation is deprecated. Enable scheduling in your configuration: -> ```json -> { "cds": { "requires": { "scheduling": {} } } } -> ``` - -### Persistent Queue (Default) { #persistent-queue } - -The persistent queue is enabled by default. -Messages are stored in a database table within the current transaction. - -::: code-group -```json [Node.js — package.json] -{ - "cds": { - "requires": { - "scheduling": { - "markerInterval": "1h", - "flushInterval": "1h" - }, - "queue": { - "maxAttempts": 20, - "chunkSize": 10, - "parallel": true - } - } - } -} -``` -```yaml [Java — application.yaml] -cds: - outbox: - services: - DefaultOutboxOrdered: - maxAttempts: 10 - ordered: true - DefaultOutboxUnordered: - maxAttempts: 10 - ordered: false -``` -::: - -Queue options for Node.js (`cds.requires.queue`): - -| Option | Default | Description | -|--------|---------|-------------| -| `maxAttempts` | `20` | Maximum retries before a message becomes a dead letter | -| `chunkSize` | `10` | Number of messages to process per batch | -| `parallel` | `true` | Process messages in parallel; set to `false` for strict ordering | -| `storeLastError` | `true` | Store error information of the last failed attempt | -| `timeout` | `"1h"` | Time after which a `processing` message is considered abandoned | -| `legacyLocking` | `false` | Backward compatibility with `@sap/cds` v9; to be removed in a future release | - -Scheduling options for Node.js (`cds.requires.scheduling`): - -| Option | Default | Description | -|--------|---------|-------------| -| `markerInterval` | `"1h"` | Offset added to a marker timestamp before processing | -| `flushInterval` | `"1h"` | How often the central runner checks for tenants with pending work | - -Configuration options for Java: - -| Option | Default | Description | -|--------|---------|-------------| -| `maxAttempts` | `10` | Maximum retries before the entry is considered dead | -| `ordered` | `true` | Process entries in submission order | - -### In-Memory Queue - -For development and testing, you can use the in-memory queue. -Messages are held in memory and emitted after the transaction commits. - -::: code-group -```json [Node.js — package.json] -{ - "cds": { - "requires": { - "queue": { - "kind": "in-memory-queue" - } - } - } -} -``` -::: - -::: warning No retry mechanism -With the in-memory queue, messages are lost if processing fails. -There is no retry mechanism and no dead letter queue. -::: - -### Disabling the Queue - -You can disable event queues globally: - -```json -{ - "cds": { - "requires": { - "queue": false - } - } -} -``` - -Or disable queueing for a specific service: - -```json -{ - "cds": { - "requires": { - "messaging": { - "outboxed": false - } - } - } -} -``` - -## Manual Processing { #flush } - -In single-tenancy, the background runner starts on application startup and processes pending messages automatically. -In multitenancy, the central runner periodically checks markers and triggers processing. - -You can also trigger processing manually: - -::: code-group -```js [Node.js] -// Flush a specific queue -const srv = await cds.connect.to('RemoteService') -await cds.flush(srv.name) - -// Flush all queues -await cds.flush() -``` -::: From d56b29e469e625cc7a3fb12c10d5f424f5b853cf Mon Sep 17 00:00:00 2001 From: D050513 Date: Tue, 2 Jun 2026 10:12:13 +0200 Subject: [PATCH 07/15] New section ordering --- guides/events/event-queues.md | 218 +++++++++++++++++----------------- 1 file changed, 109 insertions(+), 109 deletions(-) diff --git a/guides/events/event-queues.md b/guides/events/event-queues.md index 02249192ed..99434a329a 100644 --- a/guides/events/event-queues.md +++ b/guides/events/event-queues.md @@ -321,6 +321,107 @@ A queued call changes _when_ work happens and _what the caller can expect back_: > Any return value from `send()` or `run()` is therefore not available to the caller. To act on the outcome, register a [callback handler](#callbacks) on `#succeeded` or `#failed`. +## How It Works { #concept } + +The core principle is straightforward: + +1. Instead of executing side effects directly, you write a message into a database table — **within the current transaction**. +2. Once the transaction commits, a background runner reads pending messages and dispatches them to the respective service. +3. If processing succeeds, the message is deleted. +4. If processing fails, the system retries with exponentially increasing delays. +5. After a configurable maximum number of attempts, the message is moved to the dead letter queue for manual intervention. + +```mermaid +sequenceDiagram + participant H as Handler + participant DB as Database + participant R as Runner + participant S as Service + + H->>DB: business data + queued message + Note over H,DB: same transaction + DB-->>H: COMMIT + R->>DB: poll & lock + R->>S: dispatch + alt success + R->>DB: delete + else transient failure + R->>DB: retry later + else max retries + R->>DB: dead letter + end +``` + +Because the queued message and your business data share the same database transaction, you get two core guarantees: + +- **No phantom events** — if the transaction rolls back, no message is sent. +- **No lost events** — if the transaction commits, the queued work is persisted and processed eventually. + +### Single-Tenancy vs Multi-Tenancy { #tenancy } + +Event queues work in both single-tenant and multi-tenant deployments. In both cases, processing is triggered immediately after commit; markers are an optimization plus an extra layer of resilience. + +[Learn more about multitenancy.](../multitenancy/){.learn-more} + +#### Single-Tenancy + +Messages are stored in a queue table that resides in the application database. A background runner starts when your application starts and processes messages continuously. + +#### Multi-Tenancy + +Each tenant has its own database. To avoid having a central runner periodically scan every tenant, the system writes a lightweight *marker* to a central (provider) database whenever messages are queued. On startup the central runner only triggers processing for tenants that actually have pending work, and rechecks periodically as a recovery layer in case a runner crashed before processing completed. + +CAP spreads marker timestamps across tenants so that processing doesn't synchronize into bursts — you don't need to configure that. + +::: details Architecture: scheduling, processing, recovery +Behind the scenes, event queues run three independent loops: + +1. **Scheduling** — calling `srv.send()`, `srv.emit()`, or `srv.schedule()` on a queued service writes the message to the tenant's queue table within the current transaction. In multitenancy, a *marker* is also written to the provider database, recording that this tenant has pending work. +2. **Processing** — a tenant-local task runner reads a chunk of messages, dispatches each event in its own transaction, and deletes successful messages. Failed messages are rescheduled with exponentially increasing delay; after `maxAttempts` they become dead letters. +3. **Recovery** — a central runner periodically polls the provider markers and triggers processing for any tenant with pending work. This recovers from application restarts and tenants that became "cold" without losing messages. + +*Markers* contain no business data — only the information that some queue of some tenant needs to be flushed at some point in time. +::: + +### Locking and Migration { #locking } + +CAP uses **application-level locking** to coordinate processors across application instances. When a runner picks up a message, it sets the message's `status` to `processing`; other runners skip messages in that state. After processing, the row lock is released; the message is deleted (on success) or rescheduled (on failure) in the processing transaction. + +::: warning Migrating across `@sap/cds` major versions +This guide describes the implementation in `@sap/cds` 10+. Older versions select messages differently: + +- **`@sap/cds` 8** does **not** check the `status` column at all. +- **`@sap/cds` 9** checks `status` but holds a row-level lock for the duration of processing (`legacyLocking: true` is the default in cds 9). +- **`@sap/cds` 10** uses application-level locking via `status` and releases the row lock after selection. + +A rolling upgrade from `@sap/cds` 8 directly to 10 can therefore lead to **double-processing of messages**, because cds 8 instances pick up messages that a cds 10 instance has already marked `processing`. Plan downtime, drain the queue before upgrading, or upgrade through cds 9 first. +::: + +### The Data Model { #data-model } + +The persistent queue stores its messages in this entity, automatically added to your database model: + +::: details `cds.outbox.Messages` +```cds +namespace cds.outbox; + +entity Messages { + key ID : UUID; // Unique message identifier + timestamp : Timestamp; // When the message was queued + target : String; // Target service/queue name + msg : LargeString; // Serialized event payload + attempts : Integer default 0; // Number of processing attempts + partition : Integer default 0; + lastError : LargeString; // Error from last failed attempt + lastAttemptTimestamp : Timestamp; // When last attempt occurred + status : String(23); // Current processing status + task : String(255); // Task name for named/singleton tasks + appid : String(255); // Application ID for shared HDI containers +} +``` +::: + + ## How to Use { #how-to-use } ### Queueing a Service { #cds-queued } @@ -540,108 +641,9 @@ await cds.flush() ::: -## How It Works { #concept } +## Working with Event Queues -The core principle is straightforward: - -1. Instead of executing side effects directly, you write a message into a database table — **within the current transaction**. -2. Once the transaction commits, a background runner reads pending messages and dispatches them to the respective service. -3. If processing succeeds, the message is deleted. -4. If processing fails, the system retries with exponentially increasing delays. -5. After a configurable maximum number of attempts, the message is moved to the dead letter queue for manual intervention. - -```mermaid -sequenceDiagram - participant H as Handler - participant DB as Database - participant R as Runner - participant S as Service - - H->>DB: business data + queued message - Note over H,DB: same transaction - DB-->>H: COMMIT - R->>DB: poll & lock - R->>S: dispatch - alt success - R->>DB: delete - else transient failure - R->>DB: retry later - else max retries - R->>DB: dead letter - end -``` - -Because the queued message and your business data share the same database transaction, you get two core guarantees: - -- **No phantom events** — if the transaction rolls back, no message is sent. -- **No lost events** — if the transaction commits, the queued work is persisted and processed eventually. - -### Single-Tenancy vs Multi-Tenancy { #tenancy } - -Event queues work in both single-tenant and multi-tenant deployments. In both cases, processing is triggered immediately after commit; markers are an optimization plus an extra layer of resilience. - -[Learn more about multitenancy.](../multitenancy/){.learn-more} - -#### Single-Tenancy - -Messages are stored in a queue table that resides in the application database. A background runner starts when your application starts and processes messages continuously. - -#### Multi-Tenancy - -Each tenant has its own database. To avoid having a central runner periodically scan every tenant, the system writes a lightweight *marker* to a central (provider) database whenever messages are queued. On startup the central runner only triggers processing for tenants that actually have pending work, and rechecks periodically as a recovery layer in case a runner crashed before processing completed. - -CAP spreads marker timestamps across tenants so that processing doesn't synchronize into bursts — you don't need to configure that. - -::: details Architecture: scheduling, processing, recovery -Behind the scenes, event queues run three independent loops: - -1. **Scheduling** — calling `srv.send()`, `srv.emit()`, or `srv.schedule()` on a queued service writes the message to the tenant's queue table within the current transaction. In multitenancy, a *marker* is also written to the provider database, recording that this tenant has pending work. -2. **Processing** — a tenant-local task runner reads a chunk of messages, dispatches each event in its own transaction, and deletes successful messages. Failed messages are rescheduled with exponentially increasing delay; after `maxAttempts` they become dead letters. -3. **Recovery** — a central runner periodically polls the provider markers and triggers processing for any tenant with pending work. This recovers from application restarts and tenants that became "cold" without losing messages. - -*Markers* contain no business data — only the information that some queue of some tenant needs to be flushed at some point in time. -::: - -### Locking and Migration { #locking } - -CAP uses **application-level locking** to coordinate processors across application instances. When a runner picks up a message, it sets the message's `status` to `processing`; other runners skip messages in that state. After processing, the row lock is released; the message is deleted (on success) or rescheduled (on failure) in the processing transaction. - -::: warning Migrating across `@sap/cds` major versions -This guide describes the implementation in `@sap/cds` 10+. Older versions select messages differently: - -- **`@sap/cds` 8** does **not** check the `status` column at all. -- **`@sap/cds` 9** checks `status` but holds a row-level lock for the duration of processing (`legacyLocking: true` is the default in cds 9). -- **`@sap/cds` 10** uses application-level locking via `status` and releases the row lock after selection. - -A rolling upgrade from `@sap/cds` 8 directly to 10 can therefore lead to **double-processing of messages**, because cds 8 instances pick up messages that a cds 10 instance has already marked `processing`. Plan downtime, drain the queue before upgrading, or upgrade through cds 9 first. -::: - -### The Data Model { #data-model } - -The persistent queue stores its messages in this entity, automatically added to your database model: - -::: details `cds.outbox.Messages` -```cds -namespace cds.outbox; - -entity Messages { - key ID : UUID; // Unique message identifier - timestamp : Timestamp; // When the message was queued - target : String; // Target service/queue name - msg : LargeString; // Serialized event payload - attempts : Integer default 0; // Number of processing attempts - partition : Integer default 0; - lastError : LargeString; // Error from last failed attempt - lastAttemptTimestamp : Timestamp; // When last attempt occurred - status : String(23); // Current processing status - task : String(255); // Task name for named/singleton tasks - appid : String(255); // Application ID for shared HDI containers -} -``` -::: - - -## Operational Behavior +This section covers what you need to know to operate an event queue in production: how errors are retried, how to manage stuck messages, and how authorization carries over from the original request. ### Error Handling { #errors } @@ -686,8 +688,7 @@ void process(OutboxMessageEventContext context) { } ``` - -## Dead Letter Queue { #dead-letter-queue } +### Dead Letter Queue { #dead-letter-queue } Messages that exceed the maximum retry count remain in the `cds.outbox.Messages` database table with their error information intact. These entries form the _dead letter queue_ and require manual intervention — either to fix the underlying issue and retry, or to discard the message. @@ -695,11 +696,11 @@ These entries form the _dead letter queue_ and require manual intervention — e For troubleshooting, inspect `cds.outbox.Messages` and pay special attention to `status`, `attempts`, `lastError`, and `lastAttemptTimestamp`. See [*The Data Model*](#data-model) for the entity structure. -### Managing Dead Letters +#### Managing Dead Letters You can expose a CDS service to manage the dead letter queue with actions to revive or delete entries. -#### 1. Define the Service +##### 1. Define the Service ::: code-group ```cds [srv/outbox-dead-letter-queue-service.cds] @@ -724,7 +725,7 @@ The dead letter queue contains sensitive data. Ensure the service is accessible only to internal users. ::: -#### 2. Filter for Dead Entries +##### 2. Filter for Dead Entries As `maxAttempts` is configurable, its value cannot be added as a static filter to the projection, but must be applied programmatically. @@ -771,7 +772,7 @@ public class DeadOutboxMessagesHandler implements EventHandler { ``` ::: -#### 3. Implement Bound Actions +##### 3. Implement Bound Actions Entries in the dead letter queue can be _revived_ by resetting the retry counter to zero, or _deleted_ permanently. @@ -809,8 +810,7 @@ public void deleteOutboxEntry(DeadOutboxMessagesDeleteContext context) { [Learn more about the dead letter queue in Node.js.](../../node.js/queue#managing-the-dead-letter-queue){.learn-more} [Learn more about the dead letter queue in Java.](../../java/outbox#outbox-dead-letter-queue){.learn-more} - -## Deferred Principal Propagation { #principal-propagation } +### Deferred Principal Propagation { #principal-propagation } When an event is processed asynchronously, the original HTTP request context is no longer available. CAP handles this as follows: From 7b0a38bdc3eaf817b4460cec0f106e907307facb Mon Sep 17 00:00:00 2001 From: D050513 Date: Tue, 2 Jun 2026 10:57:02 +0200 Subject: [PATCH 08/15] rm custom anchors --- guides/events/event-queues.md | 44 +++++++++++++++++------------------ 1 file changed, 22 insertions(+), 22 deletions(-) diff --git a/guides/events/event-queues.md b/guides/events/event-queues.md index 99434a329a..e81a38812d 100644 --- a/guides/events/event-queues.md +++ b/guides/events/event-queues.md @@ -63,7 +63,7 @@ CAP dispatches it later in the background. If the transaction rolls back, no boo ## Use Cases -### Outbox { #outbox } +### Outbox The outbox defers outbound calls to remote services until the main transaction succeeds. This prevents sending requests to external systems when your transaction might still roll back. @@ -114,7 +114,7 @@ You don't need to call `cds.queued()` or configure anything extra for these — [Learn more about the outbox in Java.](../../java/outbox){.learn-more} -### Inbox { #inbox } +### Inbox The inbox mirrors the outbox pattern for inbound messages. When a message arrives from a broker, the messaging service immediately persists it to the database, acknowledges it to the broker, and schedules its processing. @@ -158,7 +158,7 @@ If later processing fails, recovery no longer happens in the broker; it happens ::: -### Background Tasks { #background-tasks } +### Background Tasks Event queues are not limited to messaging. You can schedule arbitrary background tasks such as data replication, cache refresh, or garbage collection. @@ -193,7 +193,7 @@ await xflights.schedule('replicate', { entity: 'Airports' }).every('*/10 * * * * `.after()` accepts milliseconds (as a number) or a time string such as `'1s'`, `'10m'`, `'1h'`. `.every()` accepts the same plus a five-field cron expression. -#### Singleton Tasks { #singleton-tasks } +#### Singleton Tasks Use `srv.schedule.task()` to schedule a *singleton task* — a task identified by name that exists only once: @@ -212,7 +212,7 @@ The [data federation guide](../integration/data-federation) uses `srv.schedule() ::: -### Callbacks (SAGA Patterns) { #callbacks } +### Callbacks (SAGA Patterns) In distributed transactions, you often need to react when an asynchronous step completes or fails. Event queues support this with `#succeeded` and `#failed` callback events, enabling compensation logic similar to SAGA patterns. @@ -249,12 +249,12 @@ The `*` wildcard handler is not called for these events. ## Guarantees -### Transactional Persistence { #no-phantom-events } +### Transactional Persistence Because the queued message is written in the same database transaction as your business data, a rollback also removes the queued message. No event is ever dispatched for a transaction that didn't commit. -### Eventual Processing { #exactly-once } +### Eventual Processing The persistent queue guarantees transactional persistence and eventual processing. For database-backed processing, CAP avoids duplicate execution under normal operation, but handlers should still be idempotent to tolerate rare crash windows or external side effects. @@ -318,10 +318,10 @@ A queued call changes _when_ work happens and _what the caller can expect back_: > [!warning] Queued calls discard the direct return value > A queued service persists the request and returns after the message is stored, not after the remote operation finishes. -> Any return value from `send()` or `run()` is therefore not available to the caller. To act on the outcome, register a [callback handler](#callbacks) on `#succeeded` or `#failed`. +> Any return value from `send()` or `run()` is therefore not available to the caller. To act on the outcome, register a [callback handler](#callbacks-saga-patterns) on `#succeeded` or `#failed`. -## How It Works { #concept } +## How It Works The core principle is straightforward: @@ -357,7 +357,7 @@ Because the queued message and your business data share the same database transa - **No phantom events** — if the transaction rolls back, no message is sent. - **No lost events** — if the transaction commits, the queued work is persisted and processed eventually. -### Single-Tenancy vs Multi-Tenancy { #tenancy } +### Single-Tenancy vs Multi-Tenancy Event queues work in both single-tenant and multi-tenant deployments. In both cases, processing is triggered immediately after commit; markers are an optimization plus an extra layer of resilience. @@ -383,7 +383,7 @@ Behind the scenes, event queues run three independent loops: *Markers* contain no business data — only the information that some queue of some tenant needs to be flushed at some point in time. ::: -### Locking and Migration { #locking } +### Locking and Migration CAP uses **application-level locking** to coordinate processors across application instances. When a runner picks up a message, it sets the message's `status` to `processing`; other runners skip messages in that state. After processing, the row lock is released; the message is deleted (on success) or rescheduled (on failure) in the processing transaction. @@ -397,7 +397,7 @@ This guide describes the implementation in `@sap/cds` 10+. Older versions select A rolling upgrade from `@sap/cds` 8 directly to 10 can therefore lead to **double-processing of messages**, because cds 8 instances pick up messages that a cds 10 instance has already marked `processing`. Plan downtime, drain the queue before upgrading, or upgrade through cds 9 first. ::: -### The Data Model { #data-model } +### The Data Model The persistent queue stores its messages in this entity, automatically added to your database model: @@ -422,9 +422,9 @@ entity Messages { ::: -## How to Use { #how-to-use } +## How to Use -### Queueing a Service { #cds-queued } +### Queueing a Service Use `cds.queued(srv)` in Node.js to obtain a queued proxy of any service. All subsequent dispatches on this proxy are persisted to the event queue and processed asynchronously. @@ -457,7 +457,7 @@ queued.emit("bookFlight", Map.of("travelId", "T-42")); ``` ::: -### Queueing by Configuration { #by-configuration } +### Queueing by Configuration You can queue any *outbound* service through configuration without changing code. That is useful when you want to switch a remote integration to durable asynchronous processing centrally. @@ -484,7 +484,7 @@ cds: ``` ::: -### Auto-Outboxed Services { #auto-outboxed } +### Auto-Outboxed Services The following services are outboxed by default — you don't need any additional configuration: @@ -498,7 +498,7 @@ This ensures that messaging and audit log events are sent reliably and never los [Learn more about auto-outboxed services in Node.js.](../../node.js/queue#per-configuration){.learn-more} [Learn more about the outbox in Java.](../../java/outbox#persistent){.learn-more} -### Service API { #service-api } +### Service API When working with event queues, you interact with the standard CAP service APIs: @@ -622,7 +622,7 @@ Or disable queueing for a specific service: } ``` -### Manual Processing { #flush } +### Manual Processing In single-tenancy, the background runner starts on application startup and processes pending messages automatically. In multitenancy, the central runner periodically checks markers and triggers processing. @@ -645,7 +645,7 @@ await cds.flush() This section covers what you need to know to operate an event queue in production: how errors are retried, how to manage stuck messages, and how authorization carries over from the original request. -### Error Handling { #errors } +### Error Handling When processing fails, the system retries the message with exponentially increasing delays. After a configurable maximum number of attempts, the message is moved to the dead letter queue. @@ -688,13 +688,13 @@ void process(OutboxMessageEventContext context) { } ``` -### Dead Letter Queue { #dead-letter-queue } +### Dead Letter Queue Messages that exceed the maximum retry count remain in the `cds.outbox.Messages` database table with their error information intact. These entries form the _dead letter queue_ and require manual intervention — either to fix the underlying issue and retry, or to discard the message. For troubleshooting, inspect `cds.outbox.Messages` and pay special attention to `status`, `attempts`, `lastError`, and `lastAttemptTimestamp`. -See [*The Data Model*](#data-model) for the entity structure. +See [*The Data Model*](#the-data-model) for the entity structure. #### Managing Dead Letters @@ -810,7 +810,7 @@ public void deleteOutboxEntry(DeadOutboxMessagesDeleteContext context) { [Learn more about the dead letter queue in Node.js.](../../node.js/queue#managing-the-dead-letter-queue){.learn-more} [Learn more about the dead letter queue in Java.](../../java/outbox#outbox-dead-letter-queue){.learn-more} -### Deferred Principal Propagation { #principal-propagation } +### Deferred Principal Propagation When an event is processed asynchronously, the original HTTP request context is no longer available. CAP handles this as follows: From 42fd054b873093ef2388caf4bcb23c750d313610 Mon Sep 17 00:00:00 2001 From: D050513 Date: Tue, 2 Jun 2026 11:10:34 +0200 Subject: [PATCH 09/15] How to Use + Next Steps --- guides/events/event-queues.md | 172 +++++++++++++++++----------------- 1 file changed, 88 insertions(+), 84 deletions(-) diff --git a/guides/events/event-queues.md b/guides/events/event-queues.md index e81a38812d..c6288f8753 100644 --- a/guides/events/event-queues.md +++ b/guides/events/event-queues.md @@ -424,10 +424,13 @@ entity Messages { ## How to Use -### Queueing a Service +To get a side effect dispatched **after** your transaction commits — with the guarantees described above — you write to an event queue rather than calling the service directly. There are two ways to make a service write to an event queue: programmatically, by wrapping a service in `cds.queued()`, or declaratively, by enabling outboxing in configuration. Either way, the trigger from your code is the same — a normal `srv.send()` or `srv.schedule()` call. -Use `cds.queued(srv)` in Node.js to obtain a queued proxy of any service. -All subsequent dispatches on this proxy are persisted to the event queue and processed asynchronously. +### Programmatically + +#### Triggering a Queued Event + +Wrap a service in `cds.queued()` and dispatch normally. The call is persisted to the event queue inside your current transaction and processed asynchronously after commit. ::: code-group ```js [Node.js] @@ -436,107 +439,111 @@ const qd_xflights = cds.queued(xflights) await qd_xflights.send('bookFlight', { travelId: 'T-42' }) // request (result discarded) ``` -::: - -::: tip `await` is still needed -Even though processing is asynchronous, you still need to `await` because the message is written to the database within the current transaction. -::: - -In Java, use `AsyncCqnService.of(srv, outbox)` to wrap any CAP service with an outbox: - -::: code-group ```java [Java] OutboxService outbox = runtime.getServiceCatalog() .getService(OutboxService.class, "XFlightsOutbox"); CqnService xflights = runtime.getServiceCatalog() .getService(CqnService.class, "xflights"); -// Wrap with outbox handling AsyncCqnService queued = AsyncCqnService.of(xflights, outbox); queued.emit("bookFlight", Map.of("travelId", "T-42")); ``` ::: -### Queueing by Configuration +::: tip `await` is still needed +Even though processing is asynchronous, you still need to `await` because the message is written to the database within the current transaction. +::: -You can queue any *outbound* service through configuration without changing code. -That is useful when you want to switch a remote integration to durable asynchronous processing centrally. +To unwrap a queued service back to its synchronous original: ::: code-group -```json [Node.js — package.json] -{ - "cds": { - "requires": { - "xflights": { - "kind": "odata", - "outboxed": true - } - } - } -} +```js [Node.js] +const xflights = cds.unqueued(qd_xflights) ``` -```yaml [Java — application.yaml] -cds: - outbox: - services: - XFlightsOutbox: - maxAttempts: 10 +```java [Java] +CqnService xflights = outbox.unboxed(outboxedXFlights); ``` ::: -### Auto-Outboxed Services - -The following services are outboxed by default — you don't need any additional configuration: +##### Scheduling a Task -| Service | Description | -|---------|-------------| -| `cds.MessagingService` | All messaging services | -| `cds.AuditLogService` | Audit log events | - -This ensures that messaging and audit log events are sent reliably and never lost because of transaction rollbacks. - -[Learn more about auto-outboxed services in Node.js.](../../node.js/queue#per-configuration){.learn-more} -[Learn more about the outbox in Java.](../../java/outbox#persistent){.learn-more} - -### Service API - -When working with event queues, you interact with the standard CAP service APIs: +For delayed or recurring work, use the `schedule()` shortcut — equivalent to `cds.queued(srv).send(event, data)` plus optional timing. See [Background Tasks](#background-tasks). | API | Description | |-----|-------------| -| `srv.send(event, data)` | Send a request; for queued services the direct return value is discarded | +| `srv.send(event, data)` | Trigger a queued event; for queued services the direct return value is discarded | | `srv.schedule(event, data)` | Schedule a task with optional timing — Node.js only | | `srv.schedule.task(event, data)` | Schedule a *singleton* task identified by name — Node.js only | | `srv.unschedule.task(name)` | Remove a previously scheduled singleton task — Node.js only | -The `schedule()` method supports a fluent API: +#### Acting on the Outcome -```js -await srv.schedule('task', data) // execute asap -await srv.schedule('task', data).after('1h') // execute after one hour -await srv.schedule('task', data).every('1h') // repeat every hour -await srv.schedule('task', data).every('*/5 * * * *') // every 5 minutes via cron -``` +Because queued calls return after the *message is stored* — not after the remote operation completes — you can't use the return value of `send()` or `run()` to react to success or failure. Register a callback handler on `#succeeded` or `#failed` instead. -[Learn more about singleton tasks.](#singleton-tasks){.learn-more} +[Learn more about callbacks.](#callbacks-saga-patterns){.learn-more} -### Unqueueing a Service +#### Manual Processing -If a service is queued by configuration, you can get back the original synchronous service: +In single-tenancy, the background runner starts on application startup and processes pending messages automatically. In multitenancy, the central runner periodically checks markers and triggers processing. + +To trigger processing manually — for example, from a startup hook or admin endpoint: ::: code-group ```js [Node.js] -const xflights = cds.unqueued(qd_xflights) +// Flush a specific queue +const xflights = await cds.connect.to('xflights') +await cds.flush(xflights.name) + +// Flush all queues +await cds.flush() ``` -```java [Java] -CqnService xflights = outbox.unboxed(outboxedXFlights); +::: + +### By Configuration + +#### Auto-Outboxed Services + +The following services are outboxed by default — you don't need to wrap or configure them: + +| Service | Description | +|---------|-------------| +| `cds.MessagingService` | All messaging services | +| `cds.AuditLogService` | Audit log events | + +This ensures that messaging and audit log events are sent reliably and never lost because of transaction rollbacks. + +[Learn more about auto-outboxed services in Node.js.](../../node.js/queue#per-configuration){.learn-more} +[Learn more about the outbox in Java.](../../java/outbox#persistent){.learn-more} + +#### Outboxing a Remote Service + +You can outbox any *outbound* service through configuration without changing code. That is useful when you want to switch a remote integration to durable asynchronous processing centrally — every call from your handlers is then queued automatically. + +::: code-group +```json [Node.js — package.json] +{ + "cds": { + "requires": { + "xflights": { + "kind": "odata", + "outboxed": true + } + } + } +} +``` +```yaml [Java — application.yaml] +cds: + outbox: + services: + XFlightsOutbox: + maxAttempts: 10 ``` ::: -### Configuration +#### Configuring the Queue -The persistent queue is enabled by default. -Messages are stored in a database table within the current transaction. +The persistent queue is enabled by default. Messages are stored in a database table within the current transaction. ::: code-group ```json [Node.js — package.json] @@ -622,24 +629,6 @@ Or disable queueing for a specific service: } ``` -### Manual Processing - -In single-tenancy, the background runner starts on application startup and processes pending messages automatically. -In multitenancy, the central runner periodically checks markers and triggers processing. - -You can also trigger processing manually: - -::: code-group -```js [Node.js] -// Flush a specific queue -const xflights = await cds.connect.to('xflights') -await cds.flush(xflights.name) - -// Flush all queues -await cds.flush() -``` -::: - ## Working with Event Queues @@ -820,3 +809,18 @@ CAP handles this as follows: This means queued handlers must not rely on request-time role checks. If you need authorization in queued processing, encode the required information in the event payload itself or derive it from persisted business data. + + +## Next Steps + +For stack-specific APIs, configuration keys, and troubleshooting: + +- [Event Queues in Node.js](../../node.js/queue) — `cds.queued`, `cds.unqueued`, `cds.flush`, `srv.schedule`, queue and scheduling configuration. +- [Transactional Outbox in Java](../../java/outbox) — `OutboxService`, `AsyncCqnService`, custom outboxes, observability via OpenTelemetry. + +Most real-world event-queue use comes through messaging or remote services. From here you'll likely want to look at: + +- [Messaging](messaging) — emitting and consuming events between CAP applications and via brokers; messaging services are auto-outboxed. +- [CAP-Level Service Integration](../integration/calesi) — consuming remote services as if they were local; outboxing them centrally with `outboxed: true`. +- [CAP-Level Data Federation](../integration/data-federation) — using `srv.schedule().every()` for polling-based replication from remote services. + From 1b5cb99921957ed19c12b7db4a8d28069f78dcf5 Mon Sep 17 00:00:00 2001 From: D050513 Date: Tue, 2 Jun 2026 14:22:22 +0200 Subject: [PATCH 10/15] svg diagrams (instead of mermaid) --- .../assets/event-queues-how-it-works.drawio | 149 ++++++++++++++++++ .../event-queues-how-it-works.drawio.svg | 4 + .../assets/event-queues-motivation.drawio | 81 ++++++++++ .../assets/event-queues-motivation.drawio.svg | 4 + guides/events/event-queues.md | 31 +--- 5 files changed, 240 insertions(+), 29 deletions(-) create mode 100644 guides/events/assets/event-queues-how-it-works.drawio create mode 100644 guides/events/assets/event-queues-how-it-works.drawio.svg create mode 100644 guides/events/assets/event-queues-motivation.drawio create mode 100644 guides/events/assets/event-queues-motivation.drawio.svg diff --git a/guides/events/assets/event-queues-how-it-works.drawio b/guides/events/assets/event-queues-how-it-works.drawio new file mode 100644 index 0000000000..eb32e732b6 --- /dev/null +++ b/guides/events/assets/event-queues-how-it-works.drawio @@ -0,0 +1,149 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/guides/events/assets/event-queues-how-it-works.drawio.svg b/guides/events/assets/event-queues-how-it-works.drawio.svg new file mode 100644 index 0000000000..20d3d40cb2 --- /dev/null +++ b/guides/events/assets/event-queues-how-it-works.drawio.svg @@ -0,0 +1,4 @@ + + + +
Handler
Handler
Database
Database
Background runner
Background runner
Remote service
Remote service
same transaction
same transaction
business data
+ queued message
business data...
COMMIT
COMMIT
async
async
poll & lock
poll & lock
dispatch
dispatch
result
result
success
success
delete
delete
transient
failure
transient...
retry later
retry later
max retries
max retries
dead letter
dead letterText is not SVG - cannot display \ No newline at end of file diff --git a/guides/events/assets/event-queues-motivation.drawio b/guides/events/assets/event-queues-motivation.drawio new file mode 100644 index 0000000000..291574ddc9 --- /dev/null +++ b/guides/events/assets/event-queues-motivation.drawio @@ -0,0 +1,81 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/guides/events/assets/event-queues-motivation.drawio.svg b/guides/events/assets/event-queues-motivation.drawio.svg new file mode 100644 index 0000000000..8c88c8121e --- /dev/null +++ b/guides/events/assets/event-queues-motivation.drawio.svg @@ -0,0 +1,4 @@ + + + +
Request handler
Request handler
same transaction
same transaction
Business
data
Business...
Queued
message
Queued...
COMMIT
COMMIT
Background runner
Background runner
async
async
Remote service
Remote service
Background task
Background taskText is not SVG - cannot display \ No newline at end of file diff --git a/guides/events/event-queues.md b/guides/events/event-queues.md index c6288f8753..179a22587c 100644 --- a/guides/events/event-queues.md +++ b/guides/events/event-queues.md @@ -26,15 +26,7 @@ An application may commit local data, but a follow-up remote call can still fail _Transactional Event Queues_ solve this by storing the follow-up work in the database as part of the **same transaction** as your business data. After commit, a background runner executes that work asynchronously and retries failures until they succeed or become dead letters. -```mermaid -flowchart LR - A[Request handler] -- in tx --> B[(Business data)] - A -- in tx --> C[(Queued message)] - B & C --> D{COMMIT} - D -. async .-> E[Background runner] - E --> F[Remote service] - E --> G[Background task] -``` +![Diagram showing the request handler writing both business data and a queued message into the database within the same transaction, marked by a dashed transaction box. After the COMMIT diamond, an asynchronous arrow leads to a background runner, which in turn dispatches to either a remote service or a background task.](assets/event-queues-motivation.drawio.svg) This pattern is widely known as the _Transactional Outbox_, but CAP's event queues go beyond outbound messages. They cover four use cases: @@ -331,26 +323,7 @@ The core principle is straightforward: 4. If processing fails, the system retries with exponentially increasing delays. 5. After a configurable maximum number of attempts, the message is moved to the dead letter queue for manual intervention. -```mermaid -sequenceDiagram - participant H as Handler - participant DB as Database - participant R as Runner - participant S as Service - - H->>DB: business data + queued message - Note over H,DB: same transaction - DB-->>H: COMMIT - R->>DB: poll & lock - R->>S: dispatch - alt success - R->>DB: delete - else transient failure - R->>DB: retry later - else max retries - R->>DB: dead letter - end -``` +![Sequence diagram with four lifelines: Handler, Database, Background runner, and Remote service. Within the 'same transaction' bracket, the handler sends business data and the queued message to the database, which returns COMMIT. In the asynchronous phase, the background runner polls and locks the database, then dispatches to the remote service. The bottom 'result' bracket shows three alternative outcomes: success leads to a delete arrow back to the database, transient failure leads to a retry-later arrow, and max retries leads to a dead-letter arrow.](assets/event-queues-how-it-works.drawio.svg) Because the queued message and your business data share the same database transaction, you get two core guarantees: From 4c4ce70cf7166e3a9db5316fb2dbd382871ef04f Mon Sep 17 00:00:00 2001 From: D050513 Date: Tue, 2 Jun 2026 21:14:34 +0200 Subject: [PATCH 11/15] restructure --- guides/events/event-queues.md | 191 ++++++++++++++++------------------ 1 file changed, 92 insertions(+), 99 deletions(-) diff --git a/guides/events/event-queues.md b/guides/events/event-queues.md index 179a22587c..71dbb41706 100644 --- a/guides/events/event-queues.md +++ b/guides/events/event-queues.md @@ -241,6 +241,21 @@ The `*` wildcard handler is not called for these events. ## Guarantees +The core principle is straightforward: + +1. Instead of executing side effects directly, you write a message into a database table — **within the current transaction**. +2. Once the transaction commits, a background runner reads pending messages and dispatches them to the respective service. +3. If processing succeeds, the message is deleted. +4. If processing fails, the system retries with exponentially increasing delays. +5. After a configurable maximum number of attempts, the message is moved to the dead letter queue for manual intervention. + +![Sequence diagram with four lifelines: Handler, Database, Background runner, and Remote service. Within the 'same transaction' bracket, the handler sends business data and the queued message to the database, which returns COMMIT. In the asynchronous phase, the background runner polls and locks the database, then dispatches to the remote service. The bottom 'result' bracket shows three alternative outcomes: success leads to a delete arrow back to the database, transient failure leads to a retry-later arrow, and max retries leads to a dead-letter arrow.](assets/event-queues-how-it-works.drawio.svg) + +Because the queued message and your business data share the same database transaction, you get two core guarantees: + +- **No phantom events** — if the transaction rolls back, no message is sent. +- **No lost events** — if the transaction commits, the queued work is persisted and processed eventually. + ### Transactional Persistence Because the queued message is written in the same database transaction as your business data, a rollback also removes the queued message. @@ -296,109 +311,13 @@ This example highlights an important design rule: use callbacks or persisted status updates for outcomes, not direct return values. -## When to Use Event Queues - -Use an event queue when work must happen *after* the current transaction commits, or when that work needs durable retries and a dead letter queue. -For an immediate, synchronous response from a remote system, use a normal service call. - -### Direct vs Queued Calls - -A queued call changes _when_ work happens and _what the caller can expect back_: - -- A **direct** call returns the remote service's result (or error) and only then commits the local transaction. -- A **queued** call writes the message to the queue inside the local transaction and returns. The actual remote dispatch happens after commit, in the background. - -> [!warning] Queued calls discard the direct return value -> A queued service persists the request and returns after the message is stored, not after the remote operation finishes. -> Any return value from `send()` or `run()` is therefore not available to the caller. To act on the outcome, register a [callback handler](#callbacks-saga-patterns) on `#succeeded` or `#failed`. - - -## How It Works - -The core principle is straightforward: - -1. Instead of executing side effects directly, you write a message into a database table — **within the current transaction**. -2. Once the transaction commits, a background runner reads pending messages and dispatches them to the respective service. -3. If processing succeeds, the message is deleted. -4. If processing fails, the system retries with exponentially increasing delays. -5. After a configurable maximum number of attempts, the message is moved to the dead letter queue for manual intervention. - -![Sequence diagram with four lifelines: Handler, Database, Background runner, and Remote service. Within the 'same transaction' bracket, the handler sends business data and the queued message to the database, which returns COMMIT. In the asynchronous phase, the background runner polls and locks the database, then dispatches to the remote service. The bottom 'result' bracket shows three alternative outcomes: success leads to a delete arrow back to the database, transient failure leads to a retry-later arrow, and max retries leads to a dead-letter arrow.](assets/event-queues-how-it-works.drawio.svg) - -Because the queued message and your business data share the same database transaction, you get two core guarantees: - -- **No phantom events** — if the transaction rolls back, no message is sent. -- **No lost events** — if the transaction commits, the queued work is persisted and processed eventually. - -### Single-Tenancy vs Multi-Tenancy - -Event queues work in both single-tenant and multi-tenant deployments. In both cases, processing is triggered immediately after commit; markers are an optimization plus an extra layer of resilience. - -[Learn more about multitenancy.](../multitenancy/){.learn-more} - -#### Single-Tenancy - -Messages are stored in a queue table that resides in the application database. A background runner starts when your application starts and processes messages continuously. - -#### Multi-Tenancy - -Each tenant has its own database. To avoid having a central runner periodically scan every tenant, the system writes a lightweight *marker* to a central (provider) database whenever messages are queued. On startup the central runner only triggers processing for tenants that actually have pending work, and rechecks periodically as a recovery layer in case a runner crashed before processing completed. - -CAP spreads marker timestamps across tenants so that processing doesn't synchronize into bursts — you don't need to configure that. - -::: details Architecture: scheduling, processing, recovery -Behind the scenes, event queues run three independent loops: - -1. **Scheduling** — calling `srv.send()`, `srv.emit()`, or `srv.schedule()` on a queued service writes the message to the tenant's queue table within the current transaction. In multitenancy, a *marker* is also written to the provider database, recording that this tenant has pending work. -2. **Processing** — a tenant-local task runner reads a chunk of messages, dispatches each event in its own transaction, and deletes successful messages. Failed messages are rescheduled with exponentially increasing delay; after `maxAttempts` they become dead letters. -3. **Recovery** — a central runner periodically polls the provider markers and triggers processing for any tenant with pending work. This recovers from application restarts and tenants that became "cold" without losing messages. - -*Markers* contain no business data — only the information that some queue of some tenant needs to be flushed at some point in time. -::: - -### Locking and Migration - -CAP uses **application-level locking** to coordinate processors across application instances. When a runner picks up a message, it sets the message's `status` to `processing`; other runners skip messages in that state. After processing, the row lock is released; the message is deleted (on success) or rescheduled (on failure) in the processing transaction. - -::: warning Migrating across `@sap/cds` major versions -This guide describes the implementation in `@sap/cds` 10+. Older versions select messages differently: - -- **`@sap/cds` 8** does **not** check the `status` column at all. -- **`@sap/cds` 9** checks `status` but holds a row-level lock for the duration of processing (`legacyLocking: true` is the default in cds 9). -- **`@sap/cds` 10** uses application-level locking via `status` and releases the row lock after selection. - -A rolling upgrade from `@sap/cds` 8 directly to 10 can therefore lead to **double-processing of messages**, because cds 8 instances pick up messages that a cds 10 instance has already marked `processing`. Plan downtime, drain the queue before upgrading, or upgrade through cds 9 first. -::: - -### The Data Model - -The persistent queue stores its messages in this entity, automatically added to your database model: - -::: details `cds.outbox.Messages` -```cds -namespace cds.outbox; - -entity Messages { - key ID : UUID; // Unique message identifier - timestamp : Timestamp; // When the message was queued - target : String; // Target service/queue name - msg : LargeString; // Serialized event payload - attempts : Integer default 0; // Number of processing attempts - partition : Integer default 0; - lastError : LargeString; // Error from last failed attempt - lastAttemptTimestamp : Timestamp; // When last attempt occurred - status : String(23); // Current processing status - task : String(255); // Task name for named/singleton tasks - appid : String(255); // Application ID for shared HDI containers -} -``` -::: - - ## How to Use To get a side effect dispatched **after** your transaction commits — with the guarantees described above — you write to an event queue rather than calling the service directly. There are two ways to make a service write to an event queue: programmatically, by wrapping a service in `cds.queued()`, or declaratively, by enabling outboxing in configuration. Either way, the trigger from your code is the same — a normal `srv.send()` or `srv.schedule()` call. +> [!tip] When **not** to use event queues +> If you need an immediate, synchronous response from a remote system, use a normal service call. Queued calls execute asynchronously and discard the direct return value — for purely local logic that finishes inside the current request, an event queue adds nothing. + ### Programmatically #### Triggering a Queued Event @@ -423,6 +342,15 @@ queued.emit("bookFlight", Map.of("travelId", "T-42")); ``` ::: +A queued call changes _when_ work happens and _what the caller can expect back_: + +- A **direct** call returns the remote service's result (or error) and only then commits the local transaction. +- A **queued** call writes the message to the queue inside the local transaction and returns. The actual remote dispatch happens after commit, in the background. + +> [!warning] Queued calls discard the direct return value +> A queued service persists the request and returns after the message is stored, not after the remote operation finishes. +> Any return value from `send()` or `run()` is therefore not available to the caller. To act on the outcome, register a [callback handler](#callbacks-saga-patterns) on `#succeeded` or `#failed`. + ::: tip `await` is still needed Even though processing is asynchronous, you still need to `await` because the message is written to the database within the current transaction. ::: @@ -603,6 +531,71 @@ Or disable queueing for a specific service: ``` +## How It Works + +### The Data Model + +The persistent queue stores its messages in this entity, automatically added to your database model: + +```cds +namespace cds.outbox; + +entity Messages { + key ID : UUID; // Unique message identifier + timestamp : Timestamp; // When the message was queued + target : String; // Target service/queue name + msg : LargeString; // Serialized event payload + attempts : Integer default 0; // Number of processing attempts + partition : Integer default 0; + lastError : LargeString; // Error from last failed attempt + lastAttemptTimestamp : Timestamp; // When last attempt occurred + status : String(23); // Current processing status + task : String(255); // Task name for named/singleton tasks + appid : String(255); // Application ID for shared HDI containers +} +``` + +### Locking and Migration + +CAP uses **application-level locking** to coordinate processors across application instances. When a runner picks up a message, it sets the message's `status` to `processing`; other runners skip messages in that state. After processing, the row lock is released; the message is deleted (on success) or rescheduled (on failure) in the processing transaction. + +::: warning Migrating across `@sap/cds` major versions +This guide describes the implementation in `@sap/cds` 10+. Older versions select messages differently: + +- **`@sap/cds` 8** does **not** check the `status` column at all. +- **`@sap/cds` 9** checks `status` but holds a row-level lock for the duration of processing (`legacyLocking: true` is the default in cds 9). +- **`@sap/cds` 10** uses application-level locking via `status` and releases the row lock after selection. + +A rolling upgrade from `@sap/cds` 8 directly to 10 can therefore lead to **double-processing of messages**, because cds 8 instances pick up messages that a cds 10 instance has already marked `processing`. Plan downtime, drain the queue before upgrading, or upgrade through cds 9 first. +::: + +### Scheduling, Processing, Recovery + +Behind the scenes, event queues run three independent loops: + +1. **Scheduling** — calling `srv.send()`, `srv.emit()`, or `srv.schedule()` on a queued service writes the message to the tenant's queue table within the current transaction. In multitenancy, a *marker* is also written to the provider database, recording that this tenant has pending work. +2. **Processing** — a tenant-local task runner reads a chunk of messages, dispatches each event in its own transaction, and deletes successful messages. Failed messages are rescheduled with exponentially increasing delay; after `maxAttempts` they become dead letters. +3. **Recovery** — a central runner periodically polls the provider markers and triggers processing for any tenant with pending work. This recovers from application restarts and tenants that became "cold" without losing messages. + +*Markers* contain no business data — only the information that some queue of some tenant needs to be flushed at some point in time. + +### Single-Tenancy vs Multi-Tenancy + +Event queues work in both single-tenant and multi-tenant deployments. In both cases, processing is triggered immediately after commit; markers are an optimization plus an extra layer of resilience. + +[Learn more about multitenancy.](../multitenancy/){.learn-more} + +#### Single-Tenancy + +Messages are stored in a queue table that resides in the application database. A background runner starts when your application starts and processes messages continuously. + +#### Multi-Tenancy + +Each tenant has its own database. To avoid having a central runner periodically scan every tenant, the system writes a lightweight *marker* to a central (provider) database whenever messages are queued. On startup the central runner only triggers processing for tenants that actually have pending work, and rechecks periodically as a recovery layer in case a runner crashed before processing completed. + +CAP spreads marker timestamps across tenants so that processing doesn't synchronize into bursts — you don't need to configure that. + + ## Working with Event Queues This section covers what you need to know to operate an event queue in production: how errors are retried, how to manage stuck messages, and how authorization carries over from the original request. From d3de7a48ba847064e135b1b3037da6037c6696af Mon Sep 17 00:00:00 2001 From: D050513 Date: Tue, 2 Jun 2026 22:12:49 +0200 Subject: [PATCH 12/15] review --- guides/events/event-queues.md | 77 +++++++++++++++++++++-------------- 1 file changed, 46 insertions(+), 31 deletions(-) diff --git a/guides/events/event-queues.md b/guides/events/event-queues.md index 71dbb41706..29675d3b25 100644 --- a/guides/events/event-queues.md +++ b/guides/events/event-queues.md @@ -5,16 +5,20 @@ status: released --- # Transactional Event Queues +The *'Transactional Outbox'* Pattern, generalized {.subtitle} -{{ $frontmatter.synopsis }} +Persist events and background tasks in the same database transaction as your business data, then process them asynchronously with retries and a dead letter queue. {.abstract} -> [!tip] Guiding Principles +> [!tip] Transactional Event Queues – Guiding Principles > -> 1. **Transactional** — queued work is written in the same transaction as your business data. -> 2. **Asynchronous** — a background runner dispatches it after commit, not during the request. -> 3. **Resilient** — failed work is retried with exponential backoff; unrecoverable entries land in a dead letter queue. +> 1. Queued work is written in the same transaction as your business data → *no phantom events, no lost events* +> 2. A background runner dispatches it after commit, not during the request → *fast request handling, durable side effects* +> 3. Failed work is retried with exponential backoff; unrecoverable entries become dead letters → *ultimate resilience* +> +> => Application developers stay focused on the domain, not on failure modes. +[toc]:./ [[toc]] @@ -28,7 +32,7 @@ After commit, a background runner executes that work asynchronously and retries ![Diagram showing the request handler writing both business data and a queued message into the database within the same transaction, marked by a dashed transaction box. After the COMMIT diamond, an asynchronous arrow leads to a background runner, which in turn dispatches to either a remote service or a background task.](assets/event-queues-motivation.drawio.svg) -This pattern is widely known as the _Transactional Outbox_, but CAP's event queues go beyond outbound messages. They cover four use cases: +This pattern is widely known as the *'Transactional Outbox'*, but CAP's event queues go beyond outbound messages. They cover four use cases: - **Outbox** — defer outbound calls to remote services until the transaction succeeds. - **Inbox** — acknowledge inbound messages immediately and process them asynchronously. @@ -38,7 +42,7 @@ This pattern is widely known as the _Transactional Outbox_, but CAP's event queu ## Quick Start -Use a queued service when a side effect must only happen after the current transaction commits. +Event queues are enabled by default — there's nothing to install or activate. Use a queued service when a side effect must only happen after the current transaction commits. ```js const xflights = await cds.connect.to('xflights') @@ -49,8 +53,9 @@ this.after('CREATE', 'Travels', async (_, req) => { }) ``` -This stores the flight booking request in the database together with the travel creation. -CAP dispatches it later in the background. If the transaction rolls back, no booking request is sent. +This stores the flight booking request in the database together with the travel creation. CAP dispatches it later in the background. If the transaction rolls back, no booking request is sent. + +The `xflights` connection in the example stands in for any remote service you've configured under `cds.requires`. The complete setup, including the *XTravels* application and the *xflights* service it consumes, lives in the [@capire/xtravels](https://github.com/capire/xtravels) sample. ## Use Cases @@ -60,8 +65,7 @@ CAP dispatches it later in the background. If the transaction rolls back, no boo The outbox defers outbound calls to remote services until the main transaction succeeds. This prevents sending requests to external systems when your transaction might still roll back. -**Example:** When creating a travel booking, you also want to notify an external flight service. -Without the outbox, the notification could be sent even if the booking transaction fails. +**Example:** In the *XTravels* application, when an agent creates a `Travels` record, the application also has to notify *xflights* to book the actual flight. Without the outbox, the booking call could go out even if the local `Travels` row never commits. ::: code-group ```js [Node.js] @@ -108,7 +112,7 @@ You don't need to call `cds.queued()` or configure anything extra for these — ### Inbox -The inbox mirrors the outbox pattern for inbound messages. +The inbox mirrors the *'Outbox'* pattern for inbound messages. When a message arrives from a broker, the messaging service immediately persists it to the database, acknowledges it to the broker, and schedules its processing. This brings two advantages: @@ -144,7 +148,7 @@ cds: ``` ::: -::: warning Inboxing changes who owns failure handling +::: warning Inboxing shifts failure handling to your application With inboxing enabled, the broker considers the message delivered as soon as your app stores it. If later processing fails, recovery no longer happens in the broker; it happens in your application's retry and dead letter queue flow. ::: @@ -164,9 +168,8 @@ await xflights.schedule('replicate', { entity: 'Airports' }).every('10 minutes') ``` ::: -> [!note] Node.js only -> The `srv.schedule()` API is currently available in Node.js only. -> In Java, use a `@Scheduled` annotation in combination with a queued outbox service to achieve equivalent behavior. +> [!note] Java documentation to follow +> Java has an equivalent scheduling API; documentation is on its way. The Node.js shape on this page applies analogously. The `schedule()` method is a convenience shortcut for `cds.queued(srv).send(event, data)` with optional timing: @@ -209,8 +212,7 @@ The [data federation guide](../integration/data-federation) uses `srv.schedule() In distributed transactions, you often need to react when an asynchronous step completes or fails. Event queues support this with `#succeeded` and `#failed` callback events, enabling compensation logic similar to SAGA patterns. -**Example:** After successfully creating a flight booking via the outbox, replicate the full business object from the remote system. -If the booking fails, notify the user or trigger compensation logic. +**Example:** After successfully creating a flight booking through *xflights*, the *XTravels* application replicates the full booking back into its own database. If the booking fails, the application updates the local `Travels` row to surface the error in its UI. ::: code-group ```js [Node.js] @@ -231,7 +233,7 @@ xflights.after('bookFlight/#failed', async (error, req) => { ::: > [!note] Node.js only -> Callback events `#succeeded` and `#failed` are currently available in Node.js only. +> Callback events `#succeeded` and `#failed` are currently available in Node.js only. Java doesn't have an equivalent yet, but it's on the roadmap. ::: tip Register on specific events Callback handlers must be registered for the specific `#succeeded` or `#failed` events. @@ -322,15 +324,11 @@ To get a side effect dispatched **after** your transaction commits — with the #### Triggering a Queued Event -Wrap a service in `cds.queued()` and dispatch normally. The call is persisted to the event queue inside your current transaction and processed asynchronously after commit. +Wrap a service in `cds.queued()` (Node.js) or `AsyncCqnService.of()` (Java) and dispatch normally. The call is persisted to the event queue inside your current transaction and processed asynchronously after commit. -::: code-group -```js [Node.js] -const xflights = await cds.connect.to('xflights') -const qd_xflights = cds.queued(xflights) +For the Node.js shape, see [Quick Start](#quick-start) and the [*Outbox* use case](#outbox). In Java, you can also wrap a service at runtime through the service catalog rather than wiring through Spring: -await qd_xflights.send('bookFlight', { travelId: 'T-42' }) // request (result discarded) -``` +::: code-group ```java [Java] OutboxService outbox = runtime.getServiceCatalog() .getService(OutboxService.class, "XFlightsOutbox"); @@ -373,9 +371,11 @@ For delayed or recurring work, use the `schedule()` shortcut — equivalent to ` | API | Description | |-----|-------------| | `srv.send(event, data)` | Trigger a queued event; for queued services the direct return value is discarded | -| `srv.schedule(event, data)` | Schedule a task with optional timing — Node.js only | -| `srv.schedule.task(event, data)` | Schedule a *singleton* task identified by name — Node.js only | -| `srv.unschedule.task(name)` | Remove a previously scheduled singleton task — Node.js only | +| `srv.schedule(event, data)` | Schedule a task with optional timing | +| `srv.schedule.task(event, data)` | Schedule a *singleton* task identified by name | +| `srv.unschedule.task(name)` | Remove a previously scheduled singleton task | + +The signatures above show the Node.js shape. Java has equivalent APIs; documentation to follow. #### Acting on the Outcome @@ -400,6 +400,9 @@ await cds.flush() ``` ::: +> [!note] Node.js only +> `cds.flush()` is currently a Node.js API. You rarely need it: both stacks have built-in recovery mechanisms that pick up pending messages automatically. + ### By Configuration #### Auto-Outboxed Services @@ -535,7 +538,7 @@ Or disable queueing for a specific service: ### The Data Model -The persistent queue stores its messages in this entity, automatically added to your database model: +The persistent queue stores its messages in this entity, which CAP adds to your model on build and deploys with your other entities: ```cds namespace cds.outbox; @@ -600,6 +603,18 @@ CAP spreads marker timestamps across tenants so that processing doesn't synchron This section covers what you need to know to operate an event queue in production: how errors are retried, how to manage stuck messages, and how authorization carries over from the original request. +### Inspecting the Queue + +To see what's currently queued for a tenant, query `cds.outbox.Messages` directly. The columns most useful for triage are `status`, `attempts`, `target`, `lastError`, and `lastAttemptTimestamp`: + +```sql +SELECT ID, target, status, attempts, lastAttemptTimestamp, lastError + FROM cds_outbox_Messages + ORDER BY timestamp DESC; +``` + +For a managed view with bound *revive* and *delete* actions, expose a CDS service over the same entity — see [Dead Letter Queue](#dead-letter-queue) below. The same projection can be widened (drop the `attempts >= maxAttempts` filter) to inspect *all* pending messages, not just dead letters. + ### Error Handling When processing fails, the system retries the message with exponentially increasing delays. @@ -691,7 +706,7 @@ const cds = require('@sap/cds') module.exports = class OutboxDeadLetterQueueService extends cds.ApplicationService { async init() { this.before('READ', 'DeadOutboxMessages', function (req) { - const { maxAttempts } = cds.env.requires.outbox + const { maxAttempts } = cds.env.requires.queue req.query.where('attempts >= ', maxAttempts) }) await super.init() From e7ea8a69fd9c0e5c1eca47dd3b02683204e2a29b Mon Sep 17 00:00:00 2001 From: D050513 Date: Wed, 3 Jun 2026 15:25:52 +0200 Subject: [PATCH 13/15] add .claude/ to gitignore --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index ac2f31bf8d..dfced3185e 100644 --- a/.gitignore +++ b/.gitignore @@ -6,3 +6,4 @@ dist/ .DS_Store .idea *.iml +.claude/ From 07e001529095be81f1ebdd518da1fcec52577119 Mon Sep 17 00:00:00 2001 From: D050513 Date: Wed, 3 Jun 2026 15:41:33 +0200 Subject: [PATCH 14/15] initial Java and Node.js Event Queues docs --- guides/deploy/build.md | 2 +- guides/events/event-queues.md | 37 +- guides/security/data-protection.md | 2 +- java/_menu.md | 2 +- java/auditlog.md | 8 +- java/event-queues.md | 415 +++++++++++++++++++ java/messaging.md | 4 +- java/outbox.md | 576 -------------------------- node.js/_menu.md | 2 +- node.js/assets/dead-letter-queue-1.js | 2 +- node.js/event-queues.md | 303 ++++++++++++++ node.js/messaging.md | 4 +- node.js/queue.md | 342 --------------- redirects.md | 4 +- 14 files changed, 763 insertions(+), 940 deletions(-) create mode 100644 java/event-queues.md delete mode 100644 java/outbox.md create mode 100644 node.js/event-queues.md delete mode 100644 node.js/queue.md diff --git a/guides/deploy/build.md b/guides/deploy/build.md index f8055bbd3e..cbad2b0202 100644 --- a/guides/deploy/build.md +++ b/guides/deploy/build.md @@ -25,7 +25,7 @@ Build tasks are derived from the CDS configuration and project context. By defau - _db/_, _srv/_, _app/_ — default root folders of a CAP project - _fts/_ and its subfolders when using [feature toggles](../extensibility/feature-toggles#enable-feature-toggles) - CDS model folders and files defined by [required services](../../node.js/cds-env#services) - - Built-in examples: [persistent queue](../../node.js/queue#persistent-queue) or [MTX-related services](../multitenancy/mtxs#mtx-services-reference) + - Built-in examples: [persistent queue](../../node.js/event-queues#persistent-queue) or [MTX-related services](../multitenancy/mtxs#mtx-services-reference) - Explicit `src` folder configured in the build task diff --git a/guides/events/event-queues.md b/guides/events/event-queues.md index 29675d3b25..c075ba8c53 100644 --- a/guides/events/event-queues.md +++ b/guides/events/event-queues.md @@ -106,8 +106,8 @@ If the surrounding transaction later fails, the external booking may already exi Some services are outboxed automatically, including `cds.MessagingService` and `cds.AuditLogService`. You don't need to call `cds.queued()` or configure anything extra for these — they use the persistent queue by default. -[Learn more about auto-outboxed services in Node.js.](../../node.js/queue#per-configuration){.learn-more} -[Learn more about the outbox in Java.](../../java/outbox){.learn-more} +[Learn more about auto-outboxed services in Node.js.](../../node.js/event-queues#queueing-a-service){.learn-more} +[Learn more about the outbox in Java.](../../java/event-queues){.learn-more} ### Inbox @@ -416,8 +416,8 @@ The following services are outboxed by default — you don't need to wrap or con This ensures that messaging and audit log events are sent reliably and never lost because of transaction rollbacks. -[Learn more about auto-outboxed services in Node.js.](../../node.js/queue#per-configuration){.learn-more} -[Learn more about the outbox in Java.](../../java/outbox#persistent){.learn-more} +[Learn more about auto-outboxed services in Node.js.](../../node.js/event-queues#queueing-a-service){.learn-more} +[Learn more about the outbox in Java.](../../java/event-queues#default-outbox-services){.learn-more} #### Outboxing a Remote Service @@ -449,6 +449,9 @@ cds: The persistent queue is enabled by default. Messages are stored in a database table within the current transaction. +> [!note] Defaults differ between stacks +> Node.js enables the **persistent** queue for every queued service by default. Java enables the persistent outbox for `cds.MessagingService` and `cds.AuditLogService` only; other services use the in-memory outbox unless you opt them in via `cds.requires.outbox.kind: persistent-outbox`. The Java configuration sample below already does that. + ::: code-group ```json [Node.js — package.json] { @@ -777,8 +780,8 @@ public void deleteOutboxEntry(DeadOutboxMessagesDeleteContext context) { ``` ::: -[Learn more about the dead letter queue in Node.js.](../../node.js/queue#managing-the-dead-letter-queue){.learn-more} -[Learn more about the dead letter queue in Java.](../../java/outbox#outbox-dead-letter-queue){.learn-more} +[Learn more about the dead letter queue in Node.js.](../../node.js/event-queues#dead-letter-queue){.learn-more} +[Learn more about the dead letter queue in Java.](../../java/event-queues#dead-letter-queue){.learn-more} ### Deferred Principal Propagation @@ -792,12 +795,30 @@ This means queued handlers must not rely on request-time role checks. If you need authorization in queued processing, encode the required information in the event payload itself or derive it from persisted business data. +## Stack Differences at a Glance + +The two stacks share the concept and the data model, but their APIs and feature sets diverge in a few places. The following table summarizes the differences as of `@sap/cds` 10: + +| Feature | Node.js | Java | +|---|---|---| +| Programmatic wrap | `cds.queued(srv)` | `OutboxService.outboxed(svc)` / `AsyncCqnService.of(svc, outbox)` | +| Default for non-auto-outboxed services | persistent | in-memory | +| Custom outbox services | through configuration | dedicated API + configuration | +| `srv.schedule()` (delay / recurrence / cron) | available | equivalent API; documentation to follow | +| Singleton tasks (`srv.schedule.task` / `srv.unschedule.task`) | available | not available | +| Callback events `#succeeded` / `#failed` | available | on the roadmap | +| Manual processing trigger (`cds.flush()`) | available | not needed; both stacks recover automatically | +| Event versioning for blue/green deployments | not available | available | +| OpenTelemetry KPI metrics | not available | available | +| Shared-database isolation pattern | not applicable | available | + + ## Next Steps For stack-specific APIs, configuration keys, and troubleshooting: -- [Event Queues in Node.js](../../node.js/queue) — `cds.queued`, `cds.unqueued`, `cds.flush`, `srv.schedule`, queue and scheduling configuration. -- [Transactional Outbox in Java](../../java/outbox) — `OutboxService`, `AsyncCqnService`, custom outboxes, observability via OpenTelemetry. +- [Event Queues in Node.js](../../node.js/event-queues) — `cds.queued`, `cds.unqueued`, `cds.flush`, `srv.schedule` (incl. singleton tasks and `#succeeded` / `#failed` callbacks), queue and scheduling configuration, troubleshooting. +- [Event Queues in Java](../../java/event-queues) — `OutboxService`, `AsyncCqnService`, custom outbox services, the technical outbox API, error-handling patterns, event versioning for blue/green deployments, and OpenTelemetry observability. Most real-world event-queue use comes through messaging or remote services. From here you'll likely want to look at: diff --git a/guides/security/data-protection.md b/guides/security/data-protection.md index 1292969bbc..574b657755 100644 --- a/guides/security/data-protection.md +++ b/guides/security/data-protection.md @@ -576,7 +576,7 @@ Design your CDS services exposed to web adapters on need-to-know basis. Be espec #### CAP Service Runtime Open transactions are expensive as they bind many resources such as a database connection as well as memory buffers. -To minimize the amount of time a transaction must be kept open, the CAP runtime offers an [Outbox Service](../../java/outbox) that allows to schedule asynchronous remote calls in the business transaction. +To minimize the amount of time a transaction must be kept open, the CAP runtime offers an [Outbox Service](../../java/event-queues) that allows to schedule asynchronous remote calls in the business transaction. Hence, the request time to process a business query, which requires a remote call (such as to an audit log server or messaging broker), is minimized and independent from the response time of the remote service. ::: tip diff --git a/java/_menu.md b/java/_menu.md index 9846736087..7ed99f3648 100644 --- a/java/_menu.md +++ b/java/_menu.md @@ -20,7 +20,7 @@ # [Messaging](messaging) # [Audit Logging](auditlog) # [Change Tracking](change-tracking) -# [Transactional Outbox](outbox) +# [Event Queues](event-queues) # [Multitenancy](multitenancy) # [Security](security) # [Spring Boot Integration](spring-boot-integration) diff --git a/java/auditlog.md b/java/auditlog.md index 68c85e338d..604250c563 100644 --- a/java/auditlog.md +++ b/java/auditlog.md @@ -30,7 +30,7 @@ The following events can be emitted with the [AuditLogService](https://javadoc.i - [Configuration changes](#config-change) - [Security events](#security-event) -AuditLog events typically are bound to business transactions. In order to handle the events transactionally and also to decouple the request from outbound calls to a consumer, for example a central audit log service, the AuditLog service leverages the [outbox](./outbox) service internally which allows [deferred](#deferred) sending of events. +AuditLog events typically are bound to business transactions. In order to handle the events transactionally and also to decouple the request from outbound calls to a consumer, for example a central audit log service, the AuditLog service leverages the [outbox](./event-queues) service internally which allows [deferred](#deferred) sending of events. ### Use AuditLogService @@ -102,13 +102,13 @@ auditLogService.logSecurityEvent(action, data); ### Deferred AuditLog Events { #deferred} -Instead of processing the audit log events synchronously in the [audit log handler](#auditlog-handlers), the `AuditLogService` can store the event in the [outbox](./outbox). This is done in the *same* transaction of the business request. Hence, a cancelled business transaction will not send any audit log events that are bound to it. To gain fine-grained control, for example to isolate a specific event from the current transaction, you may refine the transaction scope. See [ChangeSetContext API](./event-handlers/changeset-contexts#defining-changeset-contexts) for more information. +Instead of processing the audit log events synchronously in the [audit log handler](#auditlog-handlers), the `AuditLogService` can store the event in the [outbox](./event-queues). This is done in the *same* transaction of the business request. Hence, a cancelled business transaction will not send any audit log events that are bound to it. To gain fine-grained control, for example to isolate a specific event from the current transaction, you may refine the transaction scope. See [ChangeSetContext API](./event-handlers/changeset-contexts#defining-changeset-contexts) for more information. As the stored events are processed asynchronously, the business request is also decoupled from the audit log handler which typically sends the events synchronously to a central audit log service. This improves resilience and performance. -By default, the outbox comes in an [in-memory](./outbox#in-memory) flavour which has the drawback that it can't guarantee that the all events are processed after the transaction has been successfully closed. +By default, the outbox comes in an [in-memory](./event-queues#persistent-vs-in-memory-outbox) flavour which has the drawback that it can't guarantee that the all events are processed after the transaction has been successfully closed. -To close this gap, a sophisticated [persistent outbox](./outbox#persistent) service can be configured. +To close this gap, a sophisticated [persistent outbox](./event-queues#default-outbox-services) service can be configured. By default, not all events are send asynchronously via (persistent) outbox. * [Security events](#security-event) are always send synchronously. diff --git a/java/event-queues.md b/java/event-queues.md new file mode 100644 index 0000000000..3afde22a0c --- /dev/null +++ b/java/event-queues.md @@ -0,0 +1,415 @@ +--- +synopsis: > + Java APIs and configuration for CAP's Transactional Event Queues — `OutboxService`, `AsyncCqnService`, custom outbox services, error handling, event versioning, and observability. +status: released +--- + +# Event Queues in Java + + +For concepts, use cases, and guarantees, see the [Transactional Event Queues](../guides/events/event-queues) guide. This page covers the Java-specific APIs and configuration on top of that. + +In Java, event queues are exposed as **outbox services**. The runtime ships two default outboxes — `DefaultOutboxOrdered` and `DefaultOutboxUnordered` — and you can register custom outbox services for advanced isolation, scaling, or shared-database scenarios. + +[[toc]] + + +## Programmatic API + +> [!warning] In-memory by default +> Custom services wrapped through this API queue messages **in memory** by default. To make them durable across application restarts, enable the persistent outbox in your configuration — see [*Persistent vs. In-Memory Outbox*](#persistent-vs-in-memory-outbox). + +### Outboxing a Service + +Wrap any CAP service with outbox handling. Events triggered on the returned wrapper are stored in the outbox first and executed asynchronously after commit. Relevant information from the `RequestContext` is stored with the event data; the user context is downgraded to a system user context. + +```java +OutboxService myCustomOutbox = ...; +CqnService remoteS4 = ...; +CqnService outboxedS4 = myCustomOutbox.outboxed(remoteS4); +``` + +If a method on the outboxed service has a return value, it returns `null` — the call is asynchronous. To make this explicit at the type level, use the variant that wraps the service with an asynchronous-suited API: + +```java +OutboxService myCustomOutbox = ...; +CqnService remoteS4 = ...; +AsyncCqnService outboxedS4 = myCustomOutbox.outboxed(remoteS4, AsyncCqnService.class); +``` + +`AsyncCqnService.of()` is a convenience for the common case: + +```java +OutboxService myCustomOutbox = ...; +CqnService remoteS4 = ...; +AsyncCqnService outboxedS4 = AsyncCqnService.of(remoteS4, myCustomOutbox); +``` + +The outboxed service is thread-safe and can be cached. Any service that implements the `Service` interface can be outboxed, and each call is asynchronously executed if the API method internally calls `Service.emit(EventContext)`. + +To recover the synchronous service from a wrapped one: + +```java +CqnService synchronous = OutboxService.unboxed(outboxedS4); +``` + +::: tip Custom asynchronous-suited APIs +When defining your own asynchronous-suited interface, it must provide the same method signatures as the interface of the outboxed service, except for the return types — those should be `void`. +::: + +::: warning Java Proxy +A service wrapped by an outbox is a [Java Proxy](https://docs.oracle.com/javase/8/docs/technotes/guides/reflection/proxy.html). It only implements the *interfaces* of the underlying object — you can't cast an outboxed service proxy back to its concrete implementation class. +::: + + +### Technical Outbox API + +The technical API outboxes custom messages for arbitrary events or processing logic. The `OutboxMessage` instance is serialized to JSON and stored in the database, so all data must be JSON-serializable. + +```java +OutboxService outboxService = runtime.getServiceCatalog() + .getService(OutboxService.class, ""); + +OutboxMessage message = OutboxMessage.create(); +message.setParams(Map.of("name", "John", "lastname", "Doe")); + +outboxService.submit("myEvent", message); +``` + +Register an `@On` handler on the outbox service to perform the processing logic when the message is published: + +```java +@On(service = "", event = "myEvent") +void processMyEvent(OutboxMessageEventContext context) { + OutboxMessage message = context.getMessage(); + Map params = message.getParams(); + String name = (String) params.get("name"); + String lastname = (String) params.get("lastname"); + + // Perform processing logic for myEvent + + context.setCompleted(); +} +``` + +The handler must complete the context after executing the processing logic. + +[Learn more about event handlers.](./event-handlers/){.learn-more} + +::: tip Customizing outbox entries +The outbox has no information about the structure or data types being serialized. If your custom messages use non-default data types — or you need extra context properties — register `@Before` and `@On` handlers to customize serialization and deserialization. *(This isn't required for CDS-model-based services.)* + +```java [srv/src/main/java/com/myapp/CustomOutboxHandler.java] +@Component +@ServiceName(value = "*", type = OutboxService.class) +public class CustomOutboxHandler implements EventHandler { + + @On + void publishedByOutbox(OutboxMessageEventContext context) { + // Restore custom values from context only + if (Boolean.FALSE.equals(context.getIsInbound())) { + return; + } + + // custom deserialization logic + Long date = (Long) context.getMessage().getParams().get("orderDate"); + context.getMessage().getParams().put("orderDate", Instant.ofEpochSecond(date)); + } + + @Before(event = "*") + void prepareOutboxMessage(OutboxMessageEventContext context) { + // prepare outbox message for storage only + if (Boolean.TRUE.equals(context.getIsInbound())) { + return; + } + + // custom serialization logic + Instant date = (Instant) context.getMessage().getParams().get("orderDate"); + context.getMessage().getParams().put("orderDate", new Long(date.getEpochSecond())); + } +} +``` + +**Don't complete the context in either of those handlers**, otherwise the next handler in the chain isn't called and processing breaks. +::: + + +### Error-Handling Patterns + +By default the outbox retries publishing a message on error until it reaches `maxAttempts`. This makes applications resilient against unavailability of external systems. + +Some errors aren't worth retrying — for example, a `400 Bad Request` from a downstream service indicates a *semantic* error that the same payload will reproduce on every attempt. Wrap the processing in a try/catch and call `context.setCompleted()` to remove the message from the queue without further retries: + +```java +@On(service = "", event = "myEvent") +void processMyEvent(OutboxMessageEventContext context) { + try { + // Perform processing logic for myEvent + } catch (Exception e) { + if (isUnrecoverableSemanticError(e)) { + // Perform application-specific counter-measures + context.setCompleted(); // indicate message deletion to outbox + } else { + throw e; // indicate error to outbox + } + } +} +``` + +If the original processing logic isn't yours and you need to wrap its error handling, use `EventContext.proceed()`: + +```java +@On(service = OutboxService.PERSISTENT_ORDERED_NAME, event = AuditLogService.DEFAULT_NAME) +void handleAuditLogProcessingErrors(OutboxMessageEventContext context) { + try { + context.proceed(); // wrap default logic + } catch (Exception e) { + if (isUnrecoverableSemanticError(e)) { + // Perform application-specific counter-measures + context.setCompleted(); + } else { + throw e; + } + } +} +``` + +[Learn more about `EventContext.proceed()`.](./event-handlers/#proceed-on){.learn-more} + + +### Scheduling + +An equivalent of the Node.js `srv.schedule()` API exists in Java; documentation is on its way. For the parity status, see [*Stack Differences at a Glance*](../guides/events/event-queues#stack-differences-at-a-glance) in the common guide. + + +## Configuration + +### Default Outbox Services + +CAP Java ships two default outbox services: + +- **`DefaultOutboxOrdered`** — used by [messaging services](messaging) by default. Processes entries in submission order. +- **`DefaultOutboxUnordered`** — used by the [AuditLog service](auditlog) by default. May process entries in parallel across application instances. + +The configuration of both can be overridden in *application.yaml*: + +::: code-group +```yaml [srv/src/main/resources/application.yaml] +cds: + outbox: + services: + DefaultOutboxOrdered: + maxAttempts: 10 + # ordered: true + DefaultOutboxUnordered: + maxAttempts: 10 + # ordered: false +``` +::: + +| Option | Default | Description | +|---|---|---| +| `maxAttempts` | `10` | Number of unsuccessful emits until the message is ignored. It still remains in the database. | +| `ordered` | `true` | Process entries in submission order. Cannot be changed for the two default outboxes. | + +The persistent outbox stores the last error in the `lastError` element of `cds.outbox.Messages`. + + +### Persistent vs. In-Memory Outbox + +CAP Java's default behavior is the **in-memory outbox** for services other than messaging and audit log: messages are kept in memory and emitted when the current transaction succeeds. To enable persistent processing across the runtime, add the `outbox` service of kind `persistent-outbox` to `cds.requires`: + +```jsonc +{ + "cds": { + "requires": { + "outbox": { + "kind": "persistent-outbox" + } + } + } +} +``` + +::: warning Schema migration required +Adding the persistent outbox enhances your CDS model. Migrate the database schema of all tenants after enabling it. +::: + +For multitenancy scenarios, apply the same configuration in the MTX sidecar service and ensure the base model in all tenants is updated. + +::: info Add the outbox to your base model +Alternatively, add `using from '@sap/cds/srv/outbox';` to your base model. With this approach, you update tenant models after deployment but don't have to update the MTX sidecar. +::: + + +### Custom Outbox Services + +Configure custom persistent outboxes in *application.yaml*: + +::: code-group +```yaml [srv/src/main/resources/application.yaml] +cds: + outbox: + services: + MyCustomOutbox: + maxAttempts: 5 + MyOtherCustomOutbox: + maxAttempts: 10 +``` +::: + +Access them either via the service catalog: + +```java +OutboxService myCustomOutbox = cdsRuntime.getServiceCatalog() + .getService(OutboxService.class, "MyCustomOutbox"); +``` + +or by Spring injection: + +```java +@Component +public class MySpringComponent { + private final OutboxService myCustomOutbox; + + public MySpringComponent(@Qualifier("MyCustomOutbox") OutboxService myCustomOutbox) { + this.myCustomOutbox = myCustomOutbox; + } +} +``` + +::: warning Removing a custom outbox +Before removing a custom outbox from the configuration, ensure no unprocessed entries remain in `cds.outbox.Messages` for it. Removing the outbox configuration does not delete the entries — they remain in the table and aren't processed anymore. +::: + + +### Outbox for Shared Databases + +CAP Java does not yet support microservices with a shared database out of the box: the two static-named default outboxes (`DefaultOutboxOrdered`, `DefaultOutboxUnordered`) would be shared across all services and introduce conflicts. + +A manual workaround uses isolated custom outboxes with service-specific names: + +#### 1. Deactivate the Default Outboxes and Create Service-Specific Ones + +```yaml +cds: + outbox: + services: + # deactivate default outboxes + DefaultOutboxUnordered.enabled: false + DefaultOutboxOrdered.enabled: false + # custom outboxes with unique names + Service1CustomOutboxOrdered: + maxAttempts: 10 + storeLastError: true + ordered: true + Service1CustomOutboxUnordered: + maxAttempts: 10 + storeLastError: true + ordered: false +``` + +#### 2. Adapt Audit Log Configuration + +The default audit-log outbox is `DefaultOutboxUnordered`. Point it at the new custom outbox: + +```yaml +cds: + auditlog: + outbox.name: Service1CustomOutboxUnordered +``` + +#### 3. Adapt Messaging Configuration + +For *each* messaging service in the application, point it at the new ordered outbox: + +```yaml +cds: + messaging: + services: + MessagingService1: + outbox.name: Service1CustomOutboxOrdered + MessagingService2: + outbox.name: Service1CustomOutboxOrdered +``` + +::: tip Important +Both deactivating the defaults *and* using unique outbox namespaces are required to achieve service isolation in a shared-DB scenario. +::: + + +### Event Versions for Blue/Green Deployments + +In blue/green scenarios, outbox collectors of an older deployment may not be able to process events emitted by a newer deployment. Configure each deployment with an *event version* so older collectors skip newer events: + +[`cds.environment.deployment.version: 2`](./developing-applications/properties#cds-environment-deployment-version) + +::: warning Ascending versions only +Configured deployment versions must increase. Messages are processed by an outbox collector only if the event version is less than or equal to the deployment version. +::: + +To automate versioning from the Maven app version, enable resource filtering in *srv/pom.xml*: + +::: code-group +```xml [srv/pom.xml] + + ... + + + src/main/resources + true + + + ... +``` +::: + +Then use the `${project.version}` placeholder: + +[`cds.environment.deployment.version: ${project.version}`](./developing-applications/properties#cds-environment-deployment-version) + +A startup log entry shows the configured version: + +```bash +2024-12-19T11:21:33.253+01:00 INFO 3420 --- [main] cds.services.impl.utils.BuildInfo : application.deployment.version: 1.0.0-SNAPSHOT +``` + +To opt a specific custom outbox out of the version check entirely, set [`cds.outbox.services.MyCustomOutbox.checkVersion: false`](./developing-applications/properties#cds-outbox-services--checkVersion). + + +## Observability via OpenTelemetry + +The transactional outbox integrates with [OpenTelemetry](./operating-applications/observability#open-telemetry) for telemetry data. In addition to the spans described in the [observability chapter](./operating-applications/observability), the outbox logs the following KPIs: + +| KPI Name | Description | KPI Type | +|--------------------------------------------|--------------------------------------------------------------------------------------------------------|----------| +| `com.sap.cds.outbox.coldEntries` | Number of entries that could not be delivered after repeated attempts and will not be retried anymore. | Gauge | +| `com.sap.cds.outbox.remainingEntries` | Number of entries pending for delivery. | Gauge | +| `com.sap.cds.outbox.maxStorageTimeSeconds` | Maximum time in seconds an entry has been residing in the outbox. | Gauge | +| `com.sap.cds.outbox.medStorageTimeSeconds` | Median time in seconds of an entry stored in the outbox. | Gauge | +| `com.sap.cds.outbox.minStorageTimeSeconds` | Minimal time in seconds an entry has been stored in the outbox. | Gauge | +| `com.sap.cds.outbox.incomingMessages` | Number of incoming messages of the outbox. | Counter | +| `com.sap.cds.outbox.outgoingMessages` | Number of outgoing messages of the outbox. | Counter | + +KPIs are logged per microservice instance (in case of horizontal scaling), outbox, and tenant. + + +## Dead Letter Queue + +The dead-letter queue lifecycle (define service → filter for dead entries → bound revive/delete actions) is the same shape across both stacks; see [*Dead Letter Queue*](../guides/events/event-queues#dead-letter-queue) in the common guide for the full flow with code in both Node.js and Java. + +::: warning Changing `maxAttempts` between deployments +You can increase `cds.outbox.services..maxAttempts` between deployments. Older entries that had reached the previous maximum will be retried automatically after the new deployment — if the dead letter queue is large, this can cause unintended load on the system. +::: + +::: tip Use paging +Avoid reading all outbox entries at once when entries with large request payloads are present. Prefer `READ` queries with paging. +::: + +--- + +Working in Node.js? See [Event Queues in Node.js](../node.js/event-queues). diff --git a/java/messaging.md b/java/messaging.md index 3d1ff977df..0f101c9107 100644 --- a/java/messaging.md +++ b/java/messaging.md @@ -85,9 +85,9 @@ As shown in the example, there are two flavors of sending messages with the mess In section [CDS-Declared Events](#cds-declared-events), we show how to declare events in CDS models and by this let CAP generate EventContext interfaces especially tailored for the defined payload, that allows type safe access to the payload. ::: tip Using an outbox -The messages are sent once the transaction is successful. Per default, an in-memory outbox is used, but there's also support for a [persistent outbox](./outbox#persistent). +The messages are sent once the transaction is successful. Per default, an in-memory outbox is used, but there's also support for a [persistent outbox](./event-queues#default-outbox-services). -You can configure a [custom outbox](./outbox#custom-outboxes) for a messaging service by setting the property +You can configure a [custom outbox](./event-queues#custom-outbox-services) for a messaging service by setting the property `cds.messaging.services..outbox.name` to the name of the custom outbox. This specifically makes sense when [using multiple channels](../guides/events/messaging#using-multiple-channels). ::: diff --git a/java/outbox.md b/java/outbox.md deleted file mode 100644 index 0cba54d407..0000000000 --- a/java/outbox.md +++ /dev/null @@ -1,576 +0,0 @@ ---- -synopsis: > - Find here information about the Outbox service in CAP Java. -status: released ---- - -# Transactional Outbox - - -{{ $frontmatter.synopsis }} - -## Concepts - -Usually the emit of messages should be delayed until the main transaction succeeded, otherwise recipients also receive messages in case of a rollback. -To solve this problem, a transactional outbox can be used to defer the emit of messages until the success of the current transaction. - -The outbox is typically not used directly, but rather through the [messaging service](../java/messaging), the [AuditLog service](../java/auditlog) or to [outbox CAP service events](#outboxing-cap-service-events). - -## In-Memory Outbox (Default) { #in-memory} - -The in-memory outbox is used per default and the messages are emitted when the current transaction is successful. Until then, messages are kept in memory. - - -## Persistent Outbox { #persistent} - -The persistent outbox requires a persistence layer to persist the messages before emitting them. Here, the to-be-emitted message is stored in a database table first. The same database transaction is used as for other operations, therefore transactional consistency is guaranteed. - -Once the transaction succeeds, the messages are read from the database table and are emitted. - -- If an emit was successful, the respective message is deleted from the database table. -- If an emit wasn't successful, there will be a retry after some (exponentially growing) waiting time. After a maximum number of attempts, the message is ignored for processing and remains in the database table. Even if the app crashes the messages can be redelivered after successful application startup. - -To enable the persistence for the outbox, you need to add the service `outbox` of kind `persistent-outbox` to the `cds.requires` section in the _package.json_ or _cdsrc.json_, which will automatically enhance your CDS model in order to support the persistent outbox. - -```jsonc -{ - // ... - "cds": { - "requires": { - "outbox": { - "kind": "persistent-outbox" - } - } - } -} -``` - -::: warning -Be aware that you need to migrate the database schemas of all tenants after you've enhanced your model with an outbox version from `@sap/cds` version 6.0.0 or later. -::: - -For a multitenancy scenario, make sure that the required configuration is also done in the MTX sidecar service. Make sure that the base model in all tenants is updated to activate the outbox. - -::: info Option: Add outbox to your base model -Alternatively, you can add `using from '@sap/cds/srv/outbox';` to your base model. In this case, you need to update the tenant models after deployment but you don't need to update MTX Sidecar. -::: - -If enabled, CAP Java provides two persistent outbox services by default: - -- `DefaultOutboxOrdered` - is used by default by [messaging services](../java/messaging) -- `DefaultOutboxUnordered` - is used by default by the [AuditLog service](../java/auditlog) - -The default configuration for both outboxes can be overridden using the `cds.outbox.services` section, for example in the _application.yaml_: -::: code-group -```yaml [srv/src/main/resources/application.yaml] -cds: - outbox: - services: - DefaultOutboxOrdered: - maxAttempts: 10 - # ordered: true - DefaultOutboxUnordered: - maxAttempts: 10 - # ordered: false -``` -::: -You have the following configuration options: -- `maxAttempts` (default `10`): The number of unsuccessful emits until the message is ignored. It still remains in the database table. -- `ordered` (default `true`): If this flag is enabled, the outbox instance processes the entries in the order they have been submitted to it. Otherwise, the outbox may process entries randomly and in parallel, by leveraging outbox processors running in multiple application instances. This option can't be changed for the default persistent outboxes. - -The persistent outbox stores the last error that occurred, when trying to emit the message of an entry. The error is stored in the element `lastError` of the entity `cds.outbox.Messages`. - -### Configuring Custom Outboxes { #custom-outboxes} - -Custom persistent outboxes can be configured using the `cds.outbox.services` section, for example in the _application.yaml_: -::: code-group -```yaml [srv/src/main/resources/application.yaml] -cds: - outbox: - services: - MyCustomOutbox: - maxAttempts: 5 - MyOtherCustomOutbox: - maxAttempts: 10 -``` -::: -Afterward you can access the outbox instances from the service catalog: - -```java -OutboxService myCustomOutbox = cdsRuntime.getServiceCatalog().getService(OutboxService.class, "MyCustomOutbox"); -OutboxService myOtherCustomOutbox = cdsRuntime.getServiceCatalog().getService(OutboxService.class, "MyOtherCustomOutbox"); -``` - -Alternatively it's possible to inject them into a Spring component: - -```java -@Component -public class MySpringComponent { - private final OutboxService myCustomOutbox; - - public MySpringComponent(@Qualifier("MyCustomOutbox") OutboxService myCustomOutbox) { - this.myCustomOutbox = myCustomOutbox; - } -} -``` - -::: warning When removing a custom outbox ... -... it must be ensured that there are no unprocessed entries left. - -Removing a custom outbox from the `cds.outbox.services` section doesn't remove the -entries from the `cds.outbox.Messages` table. The entries remain in the `cds.outbox.Messages` table and aren't -processed anymore. - -::: - -### Outbox Event Versions - -In scenarios with multiple deployment versions (blue/green), situations may arise in which the outbox collectors of the older deployment cannot process the events generated by a newer deployment. In this case, the event can get stuck in the outbox, with all the resulting problems. - -To avoid this problem, you can configure the outbox to use an event version that prevents the outbox collectors from using the newer events. For this purpose, you can set the parameter [cds.environment.deployment.version: 2](../java/developing-applications/properties#cds-environment-deployment-version). - -::: warning Ascending Versions -The configured deployment versions must be in ascending order. The messages are only processed by the outbox collector if the event version is less than or equal to the deployment version. -::: - -To make things easier, you can automate versioning by using the Maven app version. This requires you to increment the version for each new deployment. - -To do this, the Maven `resource.filtering` configuration in the `srv/pom.xml` must be activated as follows, so that the app version placeholder `${project.version}` can be used in [cds.environment.deployment.version: ${project.version}](../java/developing-applications/properties#cds-environment-deployment-version). - -::: code-group -```xml [srv/pom.xml] - - ... - - - src/main/resources - true - - - ... -``` -::: - -To be sure that the deployment version has been set correctly, you can find a log entry at startup that shows the configured version: - -```bash -2024-12-19T11:21:33.253+01:00 INFO 3420 --- [main] cds.services.impl.utils.BuildInfo : application.deployment.version: 1.0.0-SNAPSHOT -``` - -And finally, if for some reason you don't want to use a version check for a particular outbox collector, you can switch it off via the outbox configuration [cds.outbox.services.MyCustomOutbox.checkVersion: false](../java/developing-applications/properties#cds-outbox-services--checkVersion). - -### Outbox for Shared Databases - -Currently, CAP Java does not yet support microservices with shared database out of the box, as this can lead to unexpected behavior when different isolated services use the same outboxes. -Since CAP automatically creates two outboxes with a static name — **DefaultOutboxOrdered** and **DefaultOutboxUnordered** — these would be shared across all services which introduces conflicts. - -To avoid this, you can apply a manual workaround as follows: - - 1. Customize the outbox configuration and isolating them via distinct namespaces for each service. - 2. Adapt the Audit Log outbox configuration. - 3. Adapt the messaging outbox configuration per service. - - These steps are described in the following sections. - -#### Deactivate Default Outboxes - -First, deactivate the two default outboxes and create custom outboxes with configurations tailored to your needs. - -```yaml -cds: - outbox: - services: - # deactivate default outboxes - DefaultOutboxUnordered.enabled: false - DefaultOutboxOrdered.enabled: false - # custom outboxes with unique names - Service1CustomOutboxOrdered: - maxAttempts: 10 - storeLastError: true - ordered: true - Service1CustomOutboxUnordered: - maxAttempts: 10 - storeLastError: true - ordered: false - -``` - -#### Adapt Audit Log Configuration - -The **DefaultOutboxUnordered** outbox is automatically used for audit logging. Therefore, you must update the audit log configuration to point to the custom one. - -```yaml -cds: - ... - auditlog: - outbox.name: Service1CustomOutboxUnordered -``` - -#### Adapt Messaging Configuration - -Next, adapt the messaging configuration of **every** messaging service in the application so that they use the custom-defined outboxes. - -```yaml -cds: - messaging: - services: - MessagingService1: - outbox.name: Service1CustomOutboxOrdered - MessagingService2: - outbox.name: Service1CustomOutboxOrdered -``` - - -::: tip Important Note -It is crucial to **deactivate** the default outboxes, and ensure **unique outbox namespaces** in order to achieve proper isolation between services in a shared DB scenario. -::: - - -## Outboxing CAP Service Events - -Outbox services support outboxing of arbitrary CAP services. A typical use case is to outbox remote OData -service calls, but also calls to other CAP services can be decoupled from the business logic flow. - -The API `OutboxService.outboxed(Service)` is used to wrap services with outbox handling. Events triggered -on the returned wrapper are stored in the outbox first, and executed asynchronously. Relevant information from -the `RequestContext` is stored with the event data, however the user context is downgraded to a system user context. - -The following example shows you how to outbox a service: - -```java -OutboxService myCustomOutbox = ...; -CqnService remoteS4 = ...; -CqnService outboxedS4 = myCustomOutbox.outboxed(remoteS4); -``` - -If a method on the outboxed service has a return value, it will always return `null` since it's executed asynchronously. A common example for this are the `CqnService.run(...)` methods. -To improve this the API `OutboxService.outboxed(Service, Class)` can be used, which wraps a service with an asynchronous suited API while outboxing it. -This can be used together with the interface `AsyncCqnService` to outbox remote OData services: - -```java -OutboxService myCustomOutbox = ...; -CqnService remoteS4 = ...; -AsyncCqnService outboxedS4 = myCustomOutbox.outboxed(remoteS4, AsyncCqnService.class); -``` - -The method `AsyncCqnService.of()` can be used alternatively to achieve the same for CqnServices: - -```java -OutboxService myCustomOutbox = ...; -CqnService remoteS4 = ...; -AsyncCqnService outboxedS4 = AsyncCqnService.of(remoteS4, myCustomOutbox); -``` - -::: tip Custom asynchronous suited API -When defining your own custom asynchronous suited API, the interface must provide the same method signatures as the interface of the outboxed service, except for the return types which should be `void`. -::: - -The outboxed service is thread-safe and can be cached. -Any service that implements the `Service` interface can be outboxed. -Each call to the outboxed service is asynchronously executed, if the API method internally calls the method `Service.emit(EventContext)`. - -A service wrapped by an outbox can be unboxed by calling the API `OutboxService.unboxed(Service)`. Method calls to the unboxed -service are executed synchronously without storing the event in an outbox. - -::: warning Java Proxy -A service wrapped by an outbox is a [Java Proxy](https://docs.oracle.com/javase/8/docs/technotes/guides/reflection/proxy.html). Such a proxy only implements the _interfaces_ of the object that it's wrapping. This means an outboxed service proxy can't be casted to the class implementing the underlying service object. -::: - -::: tip Custom outbox for scaling -The default outbox services can be used for outboxing arbitrary CAP services. If you detect a scaling issue, -you can define custom outboxes that can be used for outboxing. -::: - -## Technical Outbox API { #technical-outbox-api } - -Outbox services provide the technical API `OutboxService.submit(String, OutboxMessage)` that can be used to outbox custom messages for an arbitrary event or processing logic. -When submitting a custom message, an `OutboxMessage` that can optionally contain parameters for the event needs to be provided. -As the `OutboxMessage` instance is serialized and stored in the database, all data provided in that message -must be serializable and deserializable to/from JSON. The following example shows the submission of a custom message to an outbox: - -```java -OutboxService outboxService = runtime.getServiceCatalog().getService(OutboxService.class, ""); - -OutboxMessage message = OutboxMessage.create(); -message.setParams(Map.of("name", "John", "lastname", "Doe")); - -outboxService.submit("myEvent", message); -``` - -A handler for the custom message must be registered on the outbox service. This handler performs the processing logic when the message is published by the outbox: - -```java -@On(service = "", event = "myEvent") -void processMyEvent(OutboxMessageEventContext context) { - OutboxMessage message = context.getMessage(); - Map params = message.getParams(); - String name = (String) param.get("name"); - String lastname = (String) param.get("lastname"); - - // Perform processing logic for myEvent - - context.setCompleted(); -} -``` - -You must ensure that the handler is completing the context, after executing the processing logic. - -[Learn more about event handlers.](./event-handlers/){.learn-more} - -::: tip Customizing Outbox Entries - -The outbox has no information regarding the structure and the data types that -shall be serialized and deserialized to and from the outbox. - -Special handling is needed to avoid serialization and deserialization errors in custom outbox handlers if custom data types are used, **or** if additional context properties are required. _Special handling isn't required for CDS model-based services._ - -```java [srv/src/main/java/com/myapp/CustomOutboxHandler.java] -@Component -@ServiceName(value = "*", type = OutboxService.class) -public class CustomOutboxHandler implements EventHandler { - - @On - void publishedByOutbox(OutboxMessageEventContext context) { - // Restore custom values from context only - if (Boolean.FALSE.equals(context.getIsInbound())) { - return; - } - - // custom deserialization logic - Long date = (Long) context.getMessage().getParams().get("orderDate"); - context.getMessage().getParams().put("orderDate", Instant.ofEpochSecond(date)); - } - - @Before(event = "*") - void prepareOutboxMessage(OutboxMessageEventContext context) { - // prepare outbox message for storage only - if (Boolean.TRUE.equals(context.getIsInbound())) { - return; - } - - // custom serialization logic - Instant date = (Instant) context.getMessage().getParams().get("orderDate"); - context.getMessage().getParams().put("orderDate", new Long(date.getEpochSecond())); - } -} -``` - -**Don't complete the context in any of those two handlers, otherwise other -handlers aren't called and functionality is broken.** - -::: - -## Handling Outbox Errors { #handling-outbox-errors } - -The outbox by default retries publishing a message, if an error occurs during processing, until the message has reached the maximum number of attempts. -This behavior makes applications resilient against unavailability of external systems, which is a typical use case for outbox message processing. - -However, there might also be situations in which it is not reasonable to retry publishing a message. -For example, when the processed message causes a semantic error - typically due to a `400 Bad request` - on the external system. -Outbox messages causing such errors should be removed from the outbox message table before reaching the maximum number of retry attempts and instead application-specific -counter-measures should be taken to correct the semantic error or ignore the message altogether. - -A simple try-catch block around the message processing can be used to handle errors: -- If an error should cause a retry, the original exception should be (re)thrown (default behavior). -- If an error should not cause a retry, the exception should be suppressed and additional steps can be performed. - -```java -@On(service = "", event = "myEvent") -void processMyEvent(OutboxMessageEventContext context) { - try { - // Perform processing logic for myEvent - } catch (Exception e) { - if (isUnrecoverableSemanticError(e)) { - // Perform application-specific counter-measures - context.setCompleted(); // indicate message deletion to outbox - } else { - throw e; // indicate error to outbox - } - } -} -``` - -In some situations, the original outbox processing logic is not implemented by you but the processing needs to be extended with additional error handling. -In that case, wrap the `EventContext.proceed()` method, which executes the underlying processing logic: - -```java -@On(service = OutboxService.PERSISTENT_ORDERED_NAME, event = AuditLogService.DEFAULT_NAME) -void handleAuditLogProcessingErrors(OutboxMessageEventContext context) { - try { - context.proceed(); // wrap default logic - } catch (Exception e) { - if (isUnrecoverableSemanticError(e)) { - // Perform application-specific counter-measures - context.setCompleted(); // indicate message deletion to outbox - } else { - throw e; // indicate error to outbox - } - } -} -``` - -[Learn more about `EventContext.proceed()`.](./event-handlers/#proceed-on){.learn-more} - -## Outbox Dead Letter Queue - -The transactional outbox tries to process each entry a specific number of times. The number of attempts is configurable per outbox by setting the configuration `cds.outbox.services..maxAttempts`. - -[Learn more about CDS Properties.](./developing-applications/properties){.learn-more} - -Once the maximum number of attempts is exceeded, the corresponding entry is not touched anymore and hence it can be regarded as dead. Dead outbox entries are not deleted automatically. They remain in the database and it's up to the application to take care of the entries. By defining a CDS service, the dead entries can be managed conveniently. Let's have a look, how you can develop a Dead Letter Queue for the transactional outbox. - -::: warning Changing configuration between deployments - -It's possible to increase the value of the configuration `cds.outbox.services..maxAttempts` in between of deployments. Older entries which have reached their max attempts in the past would be retried automatically after deployment of the new microservice version. If the dead letter queue has a large size, this leads to unintended load on the system. - -::: - - -### Define the Service - -::: code-group - -```cds [srv/outbox-dead-letter-queue-service.cds] -using from '@sap/cds/srv/outbox'; - -@requires: 'internal-user' -service OutboxDeadLetterQueueService { - - @readonly - entity DeadOutboxMessages as projection on cds.outbox.Messages - actions { - action revive(); - action delete(); - }; - -} -``` - -::: - -The `OutboxDeadLetterQueueService` provides an entity `DeadOutboxMessages` which is a projection on the outbox table `cds.outbox.Messages` that has two bound actions: - -- `revive()` sets the number of attempts to `0` such that the outbox entry is going to be processed again. -- `delete()` deletes the outbox entry from the database. - -Filters can be applied as for any other CDS defined entity, for example, to filter for a specific outbox where the outbox name is stored in the field `target` of the entity `cds.outbox.Messages`. - -::: warning `OutboxDeadLetterQueueService` for internal users only - -It is crucial to make the service `OutboxDeadLetterQueueService` accessible for internal users only as it contains sensitive data that could be exploited for malicious purposes if unauthorized changes are performed. - -[Learn more about pseudo roles](../guides/security/cap-users#pseudo-roles){.learn-more} - -::: - -### Reading Dead Entries - -Filtering the dead entries is done by adding an appropriate `where`-clause to all `READ`-queries which matches all outbox message entries that have been retried for the maximum number of times. The following code provides an example handler implementation defining this behavior for the `DeadLetterQueueService`: - -```java -@Component -@ServiceName(OutboxDeadLetterQueueService_.CDS_NAME) -public class DeadOutboxMessagesHandler implements EventHandler { - - private final PersistenceService db; - - public DeadOutboxMessagesHandler(@Qualifier(PersistenceService.DEFAULT_NAME) PersistenceService db) { - this.db = db; - } - - @Before(entity = DeadOutboxMessages_.CDS_NAME) - public void addDeadEntryFilter(CdsReadEventContext context) { - Optional outboxFilters = this.createOutboxFilters(context.getCdsRuntime()); - - if (outboxFilters.isPresent()) { - CqnSelect modifiedCqn = - copy( - context.getCqn(), - new Modifier() { - @Override - public CqnPredicate where(Predicate where) { - return outboxFilters.get().and(where); - } - }); - context.setCqn(modifiedCqn); - } - } - - private Optional createOutboxFilters(CdsRuntime runtime) { - CdsProperties.Outbox outboxConfigs = runtime.getEnvironment().getCdsProperties().getOutbox(); - - return runtime.getServiceCatalog().getServices(OutboxService.class) - .map(service -> { - OutboxServiceConfig config = outboxConfigs.getService(service.getName()); - return CQL.get(Messages.TARGET).eq(service.getName()) - .and(CQL.get(Messages.ATTEMPTS).ge(config.getMaxAttempts())); - }) - .reduce(Predicate::or); - } -} -``` - -[Learn more about event handlers.](./event-handlers/){.learn-more} - -### Implement Bound Actions - -```java -@Autowired -@Qualifier(PersistenceService.DEFAULT_NAME) -private PersistenceService db; - -@On -public void reviveOutboxMessage(DeadOutboxMessagesReviveContext context) { - CqnAnalyzer analyzer = CqnAnalyzer.create(context.getModel()); - AnalysisResult analysisResult = analyzer.analyze(context.getCqn()); - Map key = analysisResult.rootKeys(); - Messages deadOutboxMessage = Messages.create((String) key.get(Messages.ID)); - - deadOutboxMessage.setAttempts(0); - - this.db.run(Update.entity(Messages_.class).entry(key).data(deadOutboxMessage)); - context.setCompleted(); -} - -@On -public void deleteOutboxEntry(DeadOutboxMessagesDeleteContext context) { - CqnAnalyzer analyzer = CqnAnalyzer.create(context.getModel()); - AnalysisResult analysisResult = analyzer.analyze(context.getCqn()); - Map key = analysisResult.rootKeys(); - - this.db.run(Delete.from(Messages_.class).byId(key.get(Messages.ID))); - context.setCompleted(); -} -``` - -The injected `PersistenceService` instance is used to perform the operations on the `Messages` entity since the entity `DeadOutboxMessages` is read-only. Both handlers first retrieve the ID of the entry and then they perform the corresponding operation on the database. - -[Learn more about CQL statement inspection.](./working-with-cql/query-introspection#cqnanalyzer){.learn-more} - -::: tip Use paging logic -Avoid reading all outbox entries at once in case entries which have large request payloads are present. Prefer `READ`-queries with paging instead. -::: - -## Observability using Open Telemetry - -The transactional outbox integrates Open Telemetry for logging telemetry data. - -[Learn more about observability with Open Telemetry.](./operating-applications/observability#open-telemetry){.learn-more} - -The following KPIs are logged in addition to the spans described in the [observability chapter](./operating-applications/observability): - -| KPI Name | Description | KPI Type | -| ------------------------------------------ | ------------------------------------------------------------------------------------------------------ | -------- | -| `com.sap.cds.outbox.coldEntries` | Number of entries that could not be delivered after repeated attempts and will not be retried anymore. | Gauge | -| `com.sap.cds.outbox.remainingEntries` | Number of entries which are pending for delivery. | Gauge | -| `com.sap.cds.outbox.maxStorageTimeSeconds` | Maximum time in seconds an entry was residing in the outbox. | Gauge | -| `com.sap.cds.outbox.medStorageTimeSeconds` | Median time in seconds of an entry stored in the outbox." | Gauge | -| `com.sap.cds.outbox.minStorageTimeSeconds` | Minimal time in seconds an entry was stored in the outbox. | Gauge | -| `com.sap.cds.outbox.incomingMessages` | Number of incoming messages of the outbox. | Counter | -| `com.sap.cds.outbox.outgoingMessages` | Number of outgoing messages of the outbox. | Counter | - -The KPIs are logged per microservice instance (in case of horizontal scaling), outbox, and tenant. diff --git a/node.js/_menu.md b/node.js/_menu.md index bca3500453..17fe833fd4 100644 --- a/node.js/_menu.md +++ b/node.js/_menu.md @@ -39,7 +39,7 @@ ## [Class cds. Event](events#cds-event) ## [Class cds. Request](events#cds-request) ## [Error Handling](events#req-reject) - ## [Event Queues](queue) + ## [Event Queues](event-queues) # [cds. Queries](cds-ql) diff --git a/node.js/assets/dead-letter-queue-1.js b/node.js/assets/dead-letter-queue-1.js index 75becf250f..f5218040a5 100644 --- a/node.js/assets/dead-letter-queue-1.js +++ b/node.js/assets/dead-letter-queue-1.js @@ -3,7 +3,7 @@ const cds = require('@sap/cds') module.exports = class OutboxDeadLetterQueueService extends cds.ApplicationService { async init() { this.before('READ', 'DeadOutboxMessages', function (req) { - const { maxAttempts } = cds.env.requires.outbox + const { maxAttempts } = cds.env.requires.queue req.query.where('attempts >= ', maxAttempts) }) diff --git a/node.js/event-queues.md b/node.js/event-queues.md new file mode 100644 index 0000000000..18fe007514 --- /dev/null +++ b/node.js/event-queues.md @@ -0,0 +1,303 @@ +--- +synopsis: > + Node.js APIs and configuration for CAP's Transactional Event Queues — `cds.queued`, `cds.unqueued`, `srv.schedule`, `cds.flush`, callbacks, and queue configuration. +status: released +--- + +# Event Queues in Node.js + +For concepts, use cases, and guarantees, see the [Transactional Event Queues](../guides/events/event-queues) guide. This page covers the Node.js-specific APIs and configuration on top of that. + +In Node.js, you wrap a service with `cds.queued()` to queue its events, or enable queueing through configuration. The persistent queue is the default for all queued services. + +[[toc]] + + +## Programmatic API + +### Queueing a Service + +#### `cds.queued(srv)` { .method } + +```tsx +function cds.queued ( srv: Service ) => QueuedService +``` + +Wrap a non-database service in `cds.queued()` to obtain a queued proxy. All `emit` / `send` / `run` calls on the proxy are persisted in the current transaction and dispatched after commit: + +```js +const srv = await cds.connect.to('yourService') +const qd_srv = cds.queued(srv) + +await qd_srv.emit('someEvent', { some: 'message' }) // persisted, dispatched async +await qd_srv.send('someEvent', { some: 'message' }) +``` + +::: tip `await` is still needed +The persistent queue writes the message to the database within the current transaction; you still need to `await` to keep that write inside the transaction. +::: + +For backwards compatibility, `cds.outboxed(srv)` works as a synonym. + +#### `cds.unqueued(srv)` { .method } + +```tsx +function cds.unqueued ( srv: QueuedService ) => Service +``` + +Get back the original synchronous service from a queued proxy: + +```js +const srv = cds.unqueued(qd_srv) +``` + +This is useful when a service is queued through configuration and you need a synchronous call site. For backwards compatibility, `cds.unboxed(srv)` works as a synonym. + +#### Queueing through Configuration + +You can outbox any *outbound* service through configuration without changing code. The `outboxed` flag on the service config is the trigger: + +```json +{ + "requires": { + "yourService": { + "kind": "odata", + "outboxed": true + } + } +} +``` + +Some services — `cds.MessagingService` and `cds.AuditLogService` — are outboxed by default; see [*Auto-Outboxed Services*](../guides/events/event-queues#auto-outboxed-services) in the common guide. + + +### Scheduling + +`srv.schedule()` is a shortcut for `cds.queued(srv).send()` with optional timing: + +```js +await srv.schedule('someEvent', { some: 'message' }) // execute asap +await srv.schedule('someEvent', { some: 'message' }).after('1h') // delay +await srv.schedule('someEvent', { some: 'message' }).every('10 minutes') // recurrence +await srv.schedule('someEvent', { some: 'message' }).every('*/10 * * * *') // cron +``` + +`.after()` accepts milliseconds (as a number) or a time string such as `'1s'`, `'10m'`, `'1h'`. `.every()` accepts the same plus a five-field cron expression. + +#### Singleton Tasks + +A *singleton task* is identified by name and exists only once. Subsequent calls with the same name overwrite the previous schedule (tasks are upserted, not deduplicated). This is convenient for idempotent registration during application startup: + +> [!note] Node.js only +> Singleton tasks have no Java equivalent yet. See [*Stack Differences at a Glance*](../guides/events/event-queues#stack-differences-at-a-glance). + +```js +// Replace any existing 'replicate' task with a new schedule +await srv.schedule.task('replicate', { entity: 'Airports' }).every('10 minutes') + +// Remove the task +await srv.unschedule.task('replicate') +``` + +The event name doubles as the task name. + + +### Callback Events + +> [!note] Node.js only +> Callback events have no Java equivalent yet, but they're on the roadmap. See [*Stack Differences at a Glance*](../guides/events/event-queues#stack-differences-at-a-glance). + +Once a queued message has been successfully processed, the runtime emits `/#succeeded` on the same service: + +```js +srv.after('someEvent/#succeeded', (data, req) => { + // `data` is the result of the event processor + console.log('Message successfully processed:', data) +}) +``` + +Similarly, when a message becomes a dead letter (after all retries are exhausted), the runtime emits `/#failed`: + +```js +srv.after('someEvent/#failed', (data, req) => { + // `data` is the error from the event processor + console.log('Message could not be processed:', data) +}) +``` + +::: tip Register on specific events +Callback handlers must be registered for the specific `#succeeded` or `#failed` events. The `*` wildcard handler is not called for these events. +::: + + +### Manual Processing + +> [!note] Node.js only +> `cds.flush()` is a Node.js API; both stacks have built-in recovery mechanisms that pick up pending messages automatically. See [*Stack Differences at a Glance*](../guides/events/event-queues#stack-differences-at-a-glance). + +You rarely need to trigger processing manually — both single-tenant and multi-tenant runners pick up pending messages automatically. The most common use case is recovery after an application crash, where another emit for the same tenant and service would otherwise be needed to restart processing: + +```js +// Flush a specific queue +const srv = await cds.connect.to('yourService') +await cds.flush(srv.name) + +// Flush all queues +await cds.flush() +``` + + +## Configuration + +### Persistent Queue + +The persistent queue is enabled by default. Messages are stored in the `cds.outbox.Messages` table within the current transaction. + +```json +{ + "requires": { + "scheduling": {}, + "queue": { + "kind": "persistent-queue", + "maxAttempts": 20, + "chunkSize": 10, + "parallel": true, + "storeLastError": true, + "timeout": "1h" + } + } +} +``` + +::: warning `legacyLocking` and rolling upgrades +The locking mechanism changed across `@sap/cds` major versions: cds 8 doesn't check the `status` column at all, cds 9 checks it but holds row locks for the duration of processing (`legacyLocking: true` was the cds 9 default), and cds 10 uses application-level locking via `status` and releases the row lock after selection. A rolling upgrade from cds 8 directly to cds 10 can lead to **double-processing of messages** — plan downtime, drain the queue first, or upgrade through cds 9. +::: + +::: details Queue and scheduling options + +`cds.requires.queue`: + +| Option | Default | Description | +|--------|---------|-------------| +| `maxAttempts` | `20` | Maximum retries before a message becomes a dead letter | +| `chunkSize` | `10` | Number of messages to process per batch | +| `parallel` | `true` | Process messages in parallel | +| `storeLastError` | `true` | Store error information of the last failed attempt | +| `timeout` | `"1h"` | Time after which a `processing` message is considered abandoned and eligible for reprocessing | +| `legacyLocking` | `false` | Backward compatibility with `@sap/cds` v9; to be removed in a future release | + +`cds.requires.scheduling` (multitenancy coordination): + +| Option | Description | +|--------|-------------| +| `markerInterval` | Grid interval for markers; CAP picks a default that spreads tenant load across the interval | +| `flushInterval` | Cadence at which the central runner checks for tenants with pending work | + +::: + + +### In-Memory Queue + +For development and testing, the in-memory queue holds messages until the current transaction commits, then emits them — without persistence: + +```json +{ + "requires": { + "queue": { + "kind": "in-memory-queue" + } + } +} +``` + +This is similar to the following code if done manually: + +```js +cds.context.on('succeeded', () => this.emit(msg)) +``` + +::: warning No retry mechanism +With the in-memory queue, messages are lost if processing fails. There is no retry, no dead letter queue, and no recovery on application restart. +::: + + +### Disabling the Queue + +Disable event queues globally: + +```json +{ "cds": { "requires": { "queue": false } } } +``` + +Or disable queueing for a specific service — for example to make `cds.MessagingService` emit immediately: + +```json +{ + "requires": { + "messaging": { + "kind": "enterprise-messaging", + "outboxed": false + } + } +} +``` + + +## Troubleshooting + +### Inspecting `cds.outbox.Messages` + +To see what's currently queued, query `cds.outbox.Messages` directly. The columns most useful for triage are `status`, `attempts`, `target`, `lastError`, and `lastAttemptTimestamp`: + +```js +const db = await cds.connect.to('db') +const messages = await SELECT.from('cds.outbox.Messages') + .columns('ID', 'target', 'status', 'attempts', 'lastAttemptTimestamp', 'lastError') + .orderBy('timestamp desc') +``` + +For a managed view with bound *revive* and *delete* actions, see [*Dead Letter Queue*](#dead-letter-queue) below. + + +### Manually Deleting Entries + +To clear stuck messages programmatically: + +```js +const db = await cds.connect.to('db') +await DELETE.from('cds.outbox.Messages') +``` + + +### Messages Table Not Found + +If the `cds.outbox.Messages` table is missing from the database, the most common cause is insufficient model configuration in *package.json*. If you've overwritten `requires.db.model`, add the outbox model path: + +```jsonc +"requires": { + "db": { ... + "model": [..., "@sap/cds/srv/outbox"] + } +} +``` + +For projects on `@sap/cds < 6.7.0` with custom build tasks that override `options.model`, add the path there too: + +```jsonc +"build": { + "tasks": [{ ... + "options": { "model": [..., "@sap/cds/srv/outbox"] } + }] +} +``` + +Note that the model configuration isn't required for CAP projects using the standard project layout with `db`, `srv`, and `app` folders. + + +## Dead Letter Queue + +The dead-letter queue lifecycle (define service → filter for dead entries → bound revive/delete actions) is the same shape across both stacks; see [*Dead Letter Queue*](../guides/events/event-queues#dead-letter-queue) in the common guide for the full flow with code in both Node.js and Java. + +--- + +Working in Java? See [Event Queues in Java](../java/event-queues). diff --git a/node.js/messaging.md b/node.js/messaging.md index 9335ad2a4c..36503b5e67 100644 --- a/node.js/messaging.md +++ b/node.js/messaging.md @@ -265,7 +265,7 @@ this.after(['CREATE', 'UPDATE', 'DELETE'], 'Reviews', async (_, req) => { ``` ::: tip The messages are sent once the transaction is successful. -Per default, a persistent queue is used. See [Messaging - Queue](./queue) for more information. +Per default, a persistent queue is used. See [Messaging - Queue](./event-queues) for more information. ::: ## Receiving Events @@ -300,7 +300,7 @@ In general, messages don't contain user information but operate with a technical ### Inbox -You can store received messages in an inbox before they're processed. Under the hood, it uses the [task queue](./queue) for reliable asynchronous processing. +You can store received messages in an inbox before they're processed. Under the hood, it uses the [task queue](./event-queues) for reliable asynchronous processing. Enable it by setting the `inboxed` option to `true`, for example: ```js diff --git a/node.js/queue.md b/node.js/queue.md deleted file mode 100644 index 1d10ba32fe..0000000000 --- a/node.js/queue.md +++ /dev/null @@ -1,342 +0,0 @@ ---- -synopsis: > - Learn details about the task queue feature. -status: released ---- - -# Queueing with `cds.queued` - -[[toc]] - - - -## Overview - -The _task queue_ feature allows you to defer event processing. - -A common use case is the outbox pattern, where remote operations are deferred until the main transaction has been successfully committed. -This prevents accidental execution of remote calls in case the transaction is rolled back. - -Every non-database CAP service can be _queued_, meaning that event dispatching becomes _asynchronous_. - -::: tip -The _task queue_ feature can be disabled globally via cds.requires.queue = false. -::: - - -## Queueing a Service - - -### cds. queued (srv) {.method} - -```tsx -function cds.queued ( srv: Service ) => QueuedService -``` - -Programmatically, you can get the queued service as follows: - -```js -const srv = await cds.connect.to('yourService') -const qd_srv = cds.queued(srv) - -await qd_srv.emit('someEvent', { some: 'message' }) // asynchronous -await qd_srv.send('someEvent', { some: 'message' }) // asynchronous -``` - -::: tip `await` needed -You still need to `await` these operations because they're asynchronous. In case of a persistent queue, which is the default, messages are stored in the database, within the current transaction. -::: - -For backwards compatibility, `cds.outboxed(srv)` works as a synonym. - - -### cds. unqueued (srv) {.method} - -```tsx -function cds.unqueued ( srv: QueuedService ) => Service -``` - -Use this on a queued service to get back to the original service: - -```js -const srv = cds.unqueued(qd_srv) -``` - -This is useful if your service is outboxed (that is, queued) per configuration. - -For backwards compatibility, `cds.unboxed(srv)` works as a synonym. - - -### Per Configuration - -Some services are outboxed by default; these include [`cds.MessagingService`](messaging) and `cds.AuditLogService`. -You can configure the outbox behavior by specifying the `outboxed` option in your service configuration. - -```json -{ - "requires": { - "yourService": { - "kind": "odata", - "outboxed": true - } - } -} -``` - -For transactional safety, you're encouraged to use the [persistent queue](#persistent-queue), which is enabled by default. - - - -## Persistent Queue (Default) {#persistent-queue} - -The persistent queue is the default configuration. - -Using the persistent queue, the to-be-emitted message is stored in a database table within the current transaction, therefore transactional consistency is guaranteed. - -::: details You can use the following configuration options: - -```json -{ - "requires": { - "queue": { - "kind": "persistent-queue", - "maxAttempts": 20, - "storeLastError": true, - "legacyLocking": true, - "timeout": "1h" - } - } -} -``` - -The optional parameters are: - -- `maxAttempts` (default `20`): The number of unsuccessful emits until the message is considered unprocessable. The message will remain in the database table! -- `storeLastError` (default `true`): Specifies whether error information of the last failed emit is stored in the tasks table. -- `legacyLocking` (default `true`): If set to `false`, database locks are only used to set the status of the message to `processing` to prevent long-kept database locks. Although this is the recommended approach, it is incompatible with task runners still on `@sap/cds^8`. -- `timeout` (default `"1h"`): The time after which a message with `status === "processing"` is considered to be abandoned and eligable to be processed again. Only for `legacyLocking === false`. - -::: - -Once the transaction succeeds, the messages are read from the database table and dispatched. -If processing was successful, the respective message is deleted from the database table. -If processing failed, the system retries the message after exponentially increasing delays. -After a maximum number of attempts, the message is ignored for processing and remains in the database, which -therefore also acts as a dead letter queue. -See [Managing the Dead Letter Queue](#managing-the-dead-letter-queue), to learn about how to handle such messages. - -There is only one active message processor per service, tenant, app instance, and message. -This ensures that no duplicate emits happen, except in the highly unlikely case of an app crash right after successful processing but before the message could be deleted. - -::: tip Unrecoverable errors -Some errors during the emit are identified as unrecoverable, for example in [SAP Event Mesh](../guides/events/event-mesh) if the used topic is forbidden. -The respective message is then updated and the `attempts` field is set to `maxAttempts` to prevent further processing. -[Programming errors](./best-practices#error-types) crash the server instance and must be fixed. -To mark your own errors as unrecoverable, you can set `unrecoverable = true` on the error object. -::: - - -Your database model is automatically extended by the entity `cds.outbox.Messages`: - -```cds -namespace cds.outbox; - -entity Messages { - key ID : UUID; - timestamp : Timestamp; - target : String; - msg : LargeString; - attempts : Integer default 0; - partition : Integer default 0; - lastError : LargeString; - lastAttemptTimestamp : Timestamp @cds.on.update: $now; - status : String(23); -} -``` - -In your CDS model, you can refer to the entity `cds.outbox.Messages` using the path `@sap/cds/srv/outbox`, for example to expose it in a service (cf. [Managing the Dead Letter Queue](#managing-the-dead-letter-queue)). - - -### Known Limitations - -- If the app crashes, another emit for the respective tenant and service is necessary to restart the message processing. It can be triggered manually using the `flush` method. -- The service that handles the queued event must not rely on user roles and attributes, as they are not stored with the message. In other words, asynchronous task are always processed in a privileged mode. However, the user ID is stored to re-create the correct context. - - -### Managing the Dead Letter Queue - -You can manage the dead letter queue by implementing a service that exposes a read-only projection on entity `cds.outbox.Messages` as well as bound actions to either revive or delete the respective message. - -::: tip -See [Outbox Dead Letter Queue](../java/outbox#outbox-dead-letter-queue) in the CAP Java documentation for additional considerations while we work on a general outbox guide. -::: - -#### 1. Define the Service - -::: code-group -```cds [srv/outbox-dead-letter-queue-service.cds] -using from '@sap/cds/srv/outbox'; - -@requires: 'internal-user' -service OutboxDeadLetterQueueService { - - @readonly - entity DeadOutboxMessages as projection on cds.outbox.Messages - actions { - action revive(); - action delete(); - }; - -} -``` -::: - -#### 2. Filter for Dead Entries - -As `maxAttempts` is configurable, its value cannot be added as a static filter to projection `DeadOutboxMessages`, but must be considered programmatically. - -::: code-group -<<< ./assets/dead-letter-queue-1.js#snippet{5-8} [srv/outbox-dead-letter-queue-service.js] -::: - -#### 3. Implement Bound Actions - -Finally, entries in the dead letter queue can either be _revived_ by resetting the number of attempts (that is, `SET attempts = 0`) or _deleted_. - -::: code-group -<<< ./assets/dead-letter-queue-2.js#snippet{10-12,14-16} [srv/outbox-dead-letter-queue-service.js] -::: - - -### Additional APIs - -#### Task Scheduling - -You can use the `schedule` method as a shortcut for `cds.queued(srv).send()`, with optional scheduling options `after` and `every`: - -```js -await srv.schedule('someEvent', { some: 'message' }) -await srv.schedule('someEvent', { some: 'message' }).after('1h') // after one hour -await srv.schedule('someEvent', { some: 'message' }).every('1h') // every hour after each processing -``` - -#### Task Processing - -To manually trigger the message processing, for example if your server is restarted, you can use the `flush` method. - -```js -const srv = await cds.connect.to('yourService') -cds.queued(srv).flush() -``` - -#### Task Callbacks - -Once a message has been successfully processed, it triggers the `/#succeeded` handlers. - -```js -srv.after('someEvent/#succeeded', (data, req) => { - // `data` is the result of the event processor - console.log('Message successfully processed:', data) -}) -``` - -Similarly, you can use the `/#failed` event to handle failed messages (once the maximum retry count is reached). - -```js -srv.after('someEvent/#failed', (data, req) => { - // `data` is the error from the event processor - console.log('Message could not be processed:', data) -}) -``` - -::: tip Register on specific events -Event handlers have to be registered for these specific events. The `*` wildcard handler is not called for these. -::: - - - -## In-Memory Queue - -You can enable the in-memory queue globally with: - -```json -{ - "requires": { - "queue": { - "kind": "in-memory-queue" - } - } -} -``` - -Messages are emitted only after the current transaction is successfully committed. Until then, messages are only kept in memory. -This is similar to the following code if done manually: - -```js -cds.context.on('succeeded', () => this.emit(msg)) -``` - -::: warning No retry mechanism -The message is lost if the emit fails. There's no retry mechanism. -::: - - - -## Immediate Emit - -To disable deferred emitting for a particular service only, you can set the `outboxed` option of that service to `false`: - -```json -{ - "requires": { - "messaging": { - "kind": "enterprise-messaging", - "outboxed": false - } - } -} -``` - - - -## Troubleshooting - - -### Delete Entries in the Messages Table - -To manually delete entries in the table `cds.outbox.Messages`, you can either -expose it in a service, see [Managing the Dead Letter Queue](#managing-the-dead-letter-queue), or programmatically modify it using the `cds.outbox.Messages` -entity: - -```js -const db = await cds.connect.to('db') -await DELETE.from('cds.outbox.Messages') -``` - - -### Messages Table Not Found - -If the messages table is not found on the database, this can be caused by insufficient configuration data in _package.json_. - -In case you have overwritten `requires.db.model` there, make sure to add the outbox model path `@sap/cds/srv/outbox`: - -```jsonc -"requires": { - "db": { ... - "model": [..., "@sap/cds/srv/outbox"] - } -} -``` - -The following is only relevant if you're using @sap/cds version < 6.7.0 and you've configured `options.model` in custom build tasks. -Add the model path accordingly: - -```jsonc -"build": { - "tasks": [{ ... - "options": { "model": [..., "@sap/cds/srv/outbox"] } - }] -} -``` - -Note that model configuration isn't required for CAP projects using the standard project layout with `db`, `srv`, and `app` folders. In this case, you can delete the entire `model` configuration. diff --git a/redirects.md b/redirects.md index cc6ad860eb..2c7c01f270 100644 --- a/redirects.md +++ b/redirects.md @@ -88,6 +88,7 @@ - [java/indicating-errors](java/event-handlers/indicating-errors) - [java/messaging-foundation](java/messaging) - [java/observability](java/operating-applications/observability) +- [java/outbox](java/event-queues) - [java/overview](java/getting-started) - [java/persistence-services](java/cqn-services/persistence-services) - [java/provisioning-api](java/event-handlers) @@ -104,7 +105,8 @@ - [node.js/cds-dk](tools/apis/cds-import) - [node.js/middlewares](node.js/cds-serve) -- [node.js/outbox](node.js/queue) +- [node.js/outbox](node.js/event-queues) +- [node.js/queue](node.js/event-queues) - [node.js/protocols](node.js/cds-serve) - [node.js/requests](node.js/events) - [node.js/services](node.js/core-services) From aba21230a5c70b5133b9954cb452ddb5cacf0fdf Mon Sep 17 00:00:00 2001 From: D050513 Date: Wed, 3 Jun 2026 15:43:09 +0200 Subject: [PATCH 15/15] put event queues as l1 before fiori --- java/_menu.md | 2 +- node.js/_menu.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/java/_menu.md b/java/_menu.md index 7ed99f3648..624c2e5c23 100644 --- a/java/_menu.md +++ b/java/_menu.md @@ -16,11 +16,11 @@ ## [Indicating Errors](event-handlers/indicating-errors) ## [Request Contexts](event-handlers/request-contexts) ## [ChangeSet Contexts](event-handlers/changeset-contexts) +# [Event Queues](event-queues) # [Fiori Drafts](fiori-drafts) # [Messaging](messaging) # [Audit Logging](auditlog) # [Change Tracking](change-tracking) -# [Event Queues](event-queues) # [Multitenancy](multitenancy) # [Security](security) # [Spring Boot Integration](spring-boot-integration) diff --git a/node.js/_menu.md b/node.js/_menu.md index 17fe833fd4..d74179cdbf 100644 --- a/node.js/_menu.md +++ b/node.js/_menu.md @@ -39,7 +39,6 @@ ## [Class cds. Event](events#cds-event) ## [Class cds. Request](events#cds-request) ## [Error Handling](events#req-reject) - ## [Event Queues](event-queues) # [cds. Queries](cds-ql) @@ -55,6 +54,7 @@ # [cds. env](cds-env) # [cds. utils](cds-utils) +# [Event Queues](event-queues) # [Serving Fiori UIs](fiori) # [Transactions](cds-tx) # [Security](authentication)