feat(libdatadog): add apm data pipeline by rochdev · Pull Request #205 · DataDog/libdatadog-nodejs

rochdev · 2026-08-17T21:59:31Z

Summary

Adds the APM data pipeline to the universal package introduced by the previous
PR in this stack. Native and WASM expose equivalent pipeline behavior while
sharing the retry, timeout, and backoff implementation in Rust.

Details

Adds native and WASM data_pipeline bindings backed by libdatadog revision
322d9336fed24dcf78a214f488b0fd3fb9583ba6.
Keeps HTTP requests and timers host-managed and cancellable while retaining
pipeline orchestration and retries in Rust.
Uses napi-rs's async bridge for native and wasm-bindgen-futures for WASM.
Handles timeouts and cancellation during both HTTP requests and retry
backoff.
Adds lifecycle coverage for idle processes and worker teardown.

Package restructuring belongs to the previous PR; dependency and binary-size
reporting belongs to the next PR. This PR is now only the data-pipeline delta.

Validation

Native and WASM Cargo checks, release builds, and Clippy.
29 universal-package integration tests covering both backends, retries,
timeouts, cancellation, idle-process behavior, and worker teardown.
ESLint and package dry-run/content validation.

Generated by Codex.

_{Stack created with GitHub Stacks CLI • Give Feedback 💬}

github-actions · 2026-08-17T23:47:49Z

Overall package size

Self size: 52.79 MB
Deduped: 52.79 MB
No deduping: 52.79 MB

Dependency sizes

| name | version | self size | total size | |------|---------|-----------|------------|

_{🤖 This report was automatically generated by heaviest-objects-in-the-universe}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5040c623ed

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Aaalibaba42 · 2026-08-20T09:23:36Z

+}
+
+#[derive(Clone, Debug)]
+struct WasmCapabilities;


Is there a reason to not use the existing wasm implementation ? You seem to be doing the same thing all over again as crates/capabilities of this repository.

Plus, you seem to lose some of the features / introduce regressions

Is there a reason to not use the existing wasm implementation ? You seem to be doing the same thing all over again as crates/capabilities of this repository.

We cannot use libdatadog as a do-everything package. Right now the repository is in a pretty bad state and we need to split it up better, but in the meantime this was the only way to ship something usable quickly.

Plus, you seem to lose some of the features / introduce regressions

Can you enumerate them?

Actually, let me look into that further, I think I have an idea to minimally restructure the library and allow shipping things in a way that will work for all consumers.

Aaalibaba42 · 2026-08-20T09:26:08Z

+libdd-capabilities-impl = { git = "https://github.com/DataDog/libdatadog.git", rev = "4678752b4be5bec7e55112598eb1b9491ddd9cc6" }
+libdd-data-pipeline = { git = "https://github.com/DataDog/libdatadog.git", rev = "4678752b4be5bec7e55112598eb1b9491ddd9cc6" }
+libdd-shared-runtime = { git = "https://github.com/DataDog/libdatadog.git", rev = "4678752b4be5bec7e55112598eb1b9491ddd9cc6" }


It would be better to use release version of these on crates.io, or release tags of github at least. We can release and/or publish if need be.

Aaalibaba42 · 2026-08-20T09:43:24Z

You might want to add similar options as root level Cargo.toml, depending on what you wish you can have link time optimizations, optimize the artifact for size, stripping the binary etc.

In fact you could want to have it another artifact of the top level workspace, instead of making a workspace here.

You might want to add similar options as root level Cargo.toml, depending on what you wish you can have link time optimizations, optimize the artifact for size, stripping the binary etc.

For some reason the binary is small anyway, but will look into it further.

In fact you could want to have it another artifact of the top level workspace, instead of making a workspace here.

Then it would be shipped with the root package which we cannot use.

Aaalibaba42 · 2026-08-20T09:46:45Z

+# The agentless v0.4 path does not compile or evaluate user-provided patterns.
+# Use the smaller regex engine so the full engine and its Unicode tables are
+# removed by the WASM linker.
+libdd-common = { git = "https://github.com/DataDog/libdatadog.git", rev = "4678752b4be5bec7e55112598eb1b9491ddd9cc6", default-features = false, features = ["regex-lite"] }


It seems like you don't actually use libdd-common, if this is "just" to enable regex-lite, you should be able to do it directly from libdd-data-pipeline, if not it's something we should fix our side, but I'm pretty sure it's the case.

Aaalibaba42 · 2026-08-20T09:58:33Z

+        let exporter = self.take_exporter()?;
+        let result = exporter
+            .send_async(payload.as_ref())
+            .await
+            .map(|_| ())
+            .map_err(|error| Error::from_reason(format!("data-pipeline export failed: {error}")));
+        self.restore_exporter(exporter)?;


If the future is dropped in flight, the teardown happens and restore_exporter is never called, is it ? That would brick the exporter.

Aaalibaba42 · 2026-08-20T10:16:18Z

+        .as_f64()
+        .filter(|number| number.is_finite() && *number >= 0.0 && *number <= f64::from(u32::MAX))
+        .ok_or_else(|| JsValue::from_str(&format!("{key} must be an unsigned integer")))?;
+    Ok(Some(number as u32))


prefer u32::try_from

Aaalibaba42 · 2026-08-20T10:17:00Z

+        .filter(|number| number.is_finite() && *number >= 0.0 && *number <= f64::from(u32::MAX))
+        .ok_or_else(|| JsValue::from_str(&format!("{key} must be an unsigned integer")))?;


nit: iiuc, this would accept floats, even though the error says uint.

Aaalibaba42 · 2026-08-20T10:20:58Z

+
+fn optional_string(value: &JsValue, key: &str) -> Result<Option<String>, JsValue> {
+    let value = Reflect::get(value, &JsValue::from_str(key))?;
+    if value.is_undefined() {


Shouldn't null have the same behaviour here ?

Aaalibaba42 · 2026-08-20T10:30:28Z

+                ))
+            })?;
+        let body = Reflect::get(&response, &JsValue::from_str("body")).map_err(js_error)?;
+        let body = Uint8Array::new(&body).to_vec();


status is validated just above with .as_f64().ok_or_else(...), but body isn't. new Uint8Array(x) doesn't throw on bad input, a non-object is treated as a length, so body: 123 yields 123 zero bytes and body: undefined yields an empty one, so a malformed response would silently become plausible-looking data rather than an error

Aaalibaba42 · 2026-08-20T10:37:42Z

+        let _guard = SleepGuard(promise.clone());
+        let _ = JsFuture::from(promise).await;


nit: SleepGuard fires cancelSleep even on normal completion, clearTimeout on a fired timer is a no-op, so harmless, but a wasted boundary crossing per retry.

Aaalibaba42 · 2026-08-20T11:01:05Z

Also, just noticed the compression feature is not active here, if it's not a deliberate choice you might want to turn that on.

bengl

There are simply too many changes in this PR unrelated to the PR title, making it very difficult to review.

bengl · 2026-08-21T20:55:18Z

-        match res_config {
-            Ok(config) => {
-                let config_entries: Vec<ConfigEntry> = config
+        let envp = self


For readability, please keep stylistic changes to separate PRs unless the linter forces you into it.

Addressed by splitting the package foundation and library_config changes into PR #207. This PR is now only the APM data-pipeline delta, and the dependency/size-reporting changes are isolated in PR #208. The remaining library_config source edits in #207 are required by Clippy. Generated by Codex.

rochdev · 2026-08-24T18:15:56Z

The reviewability concern is addressed with a real GitHub stack: #207 contains the package foundation, this PR contains only the APM data pipeline, and #208 contains reporting/guardrails. gh stack set the bases to main → #207 → #205 → #208, so each PR now shows only its intended delta. Generated by Codex.

bengl · 2026-08-24T18:39:03Z

+    requests.get(id)?.cancel()
+  }
+
+  function sleep (id, milliseconds) {


Let's fix capabilities at some point so we don't have to do this weird sleep (TODO/non-blocker)

TODO added.

rochdev · 2026-08-25T02:12:09Z

@Aaalibaba42 The implementation was basically rewritten at this point, please have another look. The PR is significantly smaller as unrelated changes were moved to other PRs in the stack.

rochdev mentioned this pull request Aug 17, 2026

feat(apm): add data pipeline-backed agentless mode DataDog/dd-trace-js#9839

Draft

rochdev changed the title ~~feat(apm): add data pipeline package~~ feat(apm): add data pipeline package for dd-trace Aug 19, 2026

rochdev marked this pull request as ready for review August 19, 2026 20:34

rochdev requested review from a team as code owners August 19, 2026 20:34

chatgpt-codex-connector Bot reviewed Aug 19, 2026

View reviewed changes

Comment thread .github/workflows/release.yml

Comment thread .github/workflows/release.yml Outdated

Comment thread packages/apm-data-pipeline/crates/wasm/src/transport.js Outdated

Aaalibaba42 requested changes Aug 20, 2026

View reviewed changes

rochdev changed the title ~~feat(apm): add data pipeline package for dd-trace~~ feat(libdatadog): add universal native and wasm package Aug 21, 2026

bengl requested changes Aug 24, 2026

View reviewed changes

rochdev changed the title ~~feat(libdatadog): add universal native and wasm package~~ feat(libdatadog): add universal core and optional extras packages Aug 24, 2026

rochdev changed the base branch from main to rochdev/libdatadog-core August 24, 2026 18:14

rochdev force-pushed the rochdev/apm-data-pipeline branch from 717d360 to 7195850 Compare August 24, 2026 18:14

rochdev changed the title ~~feat(libdatadog): add universal core and optional extras packages~~ feat(libdatadog): add apm data pipeline Aug 24, 2026

rochdev force-pushed the rochdev/apm-data-pipeline branch from 7195850 to 11737ee Compare August 24, 2026 18:19

rochdev force-pushed the rochdev/apm-data-pipeline branch from 11737ee to bba341d Compare August 24, 2026 18:29

bengl reviewed Aug 24, 2026

View reviewed changes

rochdev force-pushed the rochdev/apm-data-pipeline branch 2 times, most recently from d546bbb to 2c86378 Compare August 25, 2026 00:19

rochdev force-pushed the rochdev/apm-data-pipeline branch from 2c86378 to ed2d9da Compare August 25, 2026 00:24

rochdev added 2 commits August 24, 2026 20:30

feat(libdatadog): add apm data pipeline

53cb16d

test(libdatadog): load pipeline wasm artifact

4e0885b

rochdev force-pushed the rochdev/apm-data-pipeline branch from ed2d9da to 4e0885b Compare August 25, 2026 00:30

docs(libdatadog): document cancellable sleep follow-up

4c2dfa4

test(libdatadog): stabilize timeout retry assertion

aeb44df

		.filter(\|number\| number.is_finite() && number >= 0.0 && number <= f64::from(u32::MAX))
		.ok_or_else(\|\| JsValue::from_str(&format!("{key} must be an unsigned integer")))?;

		let _guard = SleepGuard(promise.clone());
		let _ = JsFuture::from(promise).await;

Uh oh!

Conversation

rochdev commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Details

Validation

Uh oh!

github-actions Bot commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overall package size

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Aaalibaba42 commented Aug 20, 2026

Uh oh!

bengl left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rochdev commented Aug 24, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rochdev commented Aug 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rochdev commented Aug 17, 2026 •

edited

Loading

github-actions Bot commented Aug 17, 2026 •

edited

Loading