test(vsock): add latency test by aaron-ang · Pull Request #5603 · firecracker-microvm/firecracker

aaron-ang · 2026-01-06T02:04:15Z

Create simple ping latency test in vsock_helper host tools

Changes

Reason

Add latency measurement to the vsock performance suite by extending vsock_helper.c with ping and ping-uds commands that emit per‑request RTTs
Add latency tests in test_vsock.py that exercise guest -> host and host -> guest paths.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license. For more information on following Developer
Certificate of Origin and signing off your commits, please check
CONTRIBUTING.md.

PR Checklist

This functionality cannot be added in rust-vmm.

aaron-ang · 2026-01-06T02:29:14Z

hi @raduiliescu @Manciukic could you please take a look?

Manciukic · 2026-01-08T12:05:12Z

Hey @aaron-ang, thanks for the contribution! I only had a very quick look, but I was wondering whether we need both ping and ping-uds. Intuitively, the ping is symmetric and already measures both g2h and h2g so I would think only the ping mode is needed. Did you notice any difference in measurement between the two tests? Can you share some data/output on your reference machine to get a feel on how that looks like?

JamesC1305 · 2026-02-25T16:53:05Z

Hi @aaron-ang,

Just wanted to see if you had a chance to read our previous update? Let us know if you need anything more from us.

Thank you!

aaron-ang · 2026-02-25T17:44:52Z

Hi @JamesC1305, apologies as I have been really busy the past few weeks. I will look into this within this week.

aaron-ang · 2026-02-26T04:16:18Z

My setup: "cpu_model": "Intel(R) N100", "host_kernel": "linux-6.17", "rootfs": "ubuntu-24.04.squashfs".

I did some stats aggregation (ping-uds has slightly higher overall latency):

Direction	n	mean (µs)	median (µs)	p95 (µs)	p99 (µs)
g2h/ping	3600	633.0	375.5	2201.3	3012.1
h2g/ping-uds	3600	861.5	508.0	2610.2	3427.1

I think both should be kept.ping (g2h test) measures guest-initiated AF_VSOCK traffic to a host echo server, while ping-uds (h2g test) measures host-initiated UDS CONNECT traffic into Firecracker and then to a guest echo server.

Manciukic

Overall LGTM, just a nit to reduce boilerplate code. I need to double check this works, I will kickoff our performance CI as well so we can run these tests

Manciukic · 2026-03-04T14:43:31Z

+
+@pytest.mark.nonci
+@pytest.mark.parametrize("vcpus", [1, 2], ids=["1vcpu", "2vcpu"])
+def test_vsock_latency_g2h(uvm_plain_acpi, vcpus, metrics, bin_vsock_path):


nit: we can avoid some duped code by having the test setup in a separate fixture for all the tests in the file

@pytest.fixture def vsock_uvm(uvm_plain_acpi, request): """Fixture to initialize a microVM with vsock device.""" vcpus = request.param if hasattr(request, "param") else 1 mem_size_mib = 1024 vm = uvm_plain_acpi vm.spawn(log_level="Info", emit_metrics=True) vm.basic_config(vcpu_count=vcpus, mem_size_mib=mem_size_mib) vm.add_net_iface() vm.api.vsock.put(vsock_id="vsock0", guest_cid=3, uds_path="/" + VSOCK_UDS_PATH) vm.start() vm.pin_threads(0) return vm

then it can be used as

@pytest.mark.parametrize("vsock_uvm", [1, 2], indirect=True, ids=["1vcpu", "2vcpu"])

@aaron-ang can you use the fixture here as well?

Manciukic · 2026-03-04T14:51:01Z

Perf test dry-run: https://buildkite.com/firecracker/performance-a-b-tests/builds/833
PR tests: https://buildkite.com/firecracker/firecracker-pr/builds/16110
PR tests (optional): https://buildkite.com/firecracker/firecracker-pr-optional/builds/15364

codecov · 2026-03-04T14:55:22Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.07%. Comparing base (a481d51) to head (082c578).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #5603   +/-   ##
=======================================
  Coverage   83.07%   83.07%           
=======================================
  Files         276      276           
  Lines       29541    29541           
=======================================
  Hits        24541    24541           
  Misses       5000     5000

Flag	Coverage Δ
5.10-m5n.metal	`83.40% <ø> (+0.01%)`	⬆️
5.10-m6a.metal	`82.73% <ø> (+<0.01%)`	⬆️
5.10-m6g.metal	`79.99% <ø> (ø)`
5.10-m6i.metal	`83.40% <ø> (+<0.01%)`	⬆️
5.10-m7a.metal-48xl	`82.73% <ø> (+<0.01%)`	⬆️
5.10-m7g.metal	`79.99% <ø> (ø)`
5.10-m7i.metal-24xl	`83.38% <ø> (+0.01%)`	⬆️
5.10-m7i.metal-48xl	`83.37% <ø> (ø)`
5.10-m8g.metal-24xl	`79.99% <ø> (ø)`
5.10-m8g.metal-48xl	`79.99% <ø> (ø)`
5.10-m8i.metal-48xl	`83.37% <ø> (-0.01%)`	⬇️
5.10-m8i.metal-96xl	`83.37% <ø> (+<0.01%)`	⬆️
6.1-m5n.metal	`83.43% <ø> (+<0.01%)`	⬆️
6.1-m6a.metal	`82.76% <ø> (+<0.01%)`	⬆️
6.1-m6g.metal	`79.99% <ø> (-0.01%)`	⬇️
6.1-m6i.metal	`83.43% <ø> (+<0.01%)`	⬆️
6.1-m7a.metal-48xl	`82.75% <ø> (+<0.01%)`	⬆️
6.1-m7g.metal	`79.99% <ø> (ø)`
6.1-m7i.metal-24xl	`83.43% <ø> (+<0.01%)`	⬆️
6.1-m7i.metal-48xl	`83.44% <ø> (+<0.01%)`	⬆️
6.1-m8g.metal-24xl	`79.99% <ø> (-0.01%)`	⬇️
6.1-m8g.metal-48xl	`79.99% <ø> (ø)`
6.1-m8i.metal-48xl	`83.44% <ø> (ø)`
6.1-m8i.metal-96xl	`83.44% <ø> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Manciukic · 2026-03-11T16:05:07Z

Plotting the results, it's pretty slow compared to what we would have expected. Surprisingly some instances are much much faster. I think removing the delay would make this better.

test_vsock_latency 1 vcpu 6.1/6.1 p50 (unit=us)

aaron-ang · 2026-03-25T05:26:20Z

Plotting the results, it's pretty slow compared to what we would have expected. Surprisingly some instances are much much faster. I think removing the delay would make this better.

test_vsock_latency 1 vcpu 6.1/6.1 p50 (unit=us)

cool, thanks! I will incorporate your feedback this week

aaron-ang · 2026-03-30T02:19:42Z

@Manciukic please review

Manciukic

Just one nit but I haven't looked too deeply. I'll give it another go in our perf pipeline

Manciukic · 2026-04-01T14:42:12Z

+
+@pytest.mark.nonci
+@pytest.mark.parametrize("vcpus", [1, 2], ids=["1vcpu", "2vcpu"])
+def test_vsock_latency_g2h(uvm_plain_acpi, vcpus, metrics, bin_vsock_path):


@aaron-ang can you use the fixture here as well?

Manciukic · 2026-04-01T14:43:56Z

Perf test dry run : https://buildkite.com/firecracker/performance-a-b-tests/builds/866/

Manciukic · 2026-04-01T16:36:03Z

It looks better now! Not as good as network, but that's a separate problem (unit is microseconds)

Manciukic · 2026-04-08T14:17:51Z

@aaron-ang I kicked off CI again and there's another style issue. You can run this checks on your side before submitting running ./tools/devtool checkstyle

integration_tests/style/test_gitlint.py:21: in test_gitlint
    assert rc == 0, "Commit message violates gitlint rules: {}".format(stderr)
E   AssertionError: Commit message violates gitlint rules: Commit e273e3b9e0:
E     3: B6 Body message is missing
E     
E     Commit f371d4417d:
E     3: B6 Body message is missing
E     
E   assert 2 == 0
        _          = ''
        rc         = 2
        stderr     = ('Commit e273e3b9e0:\n'

Create simple ping latency test in vsock_helper host tools Signed-off-by: Aaron Ang <aaron.angyd@gmail.com>

aaron-ang · 2026-04-10T03:47:41Z

@aaron-ang I kicked off CI again and there's another style issue. You can run this checks on your side before submitting running ./tools/devtool checkstyle

integration_tests/style/test_gitlint.py:21: in test_gitlint
    assert rc == 0, "Commit message violates gitlint rules: {}".format(stderr)
E   AssertionError: Commit message violates gitlint rules: Commit e273e3b9e0:
E     3: B6 Body message is missing
E     
E     Commit f371d4417d:
E     3: B6 Body message is missing
E     
E   assert 2 == 0
        _          = ''
        rc         = 2
        stderr     = ('Commit e273e3b9e0:\n'

rebased with main and resolved git format error, but i'm still getting unrelated errors when running ./tools/devtool checkstyle regarding pci

Manciukic · 2026-04-10T08:52:06Z

weird, in any case it passed on the CI. Maybe there's an issue with that command on the local environment.

aaron-ang · 2026-04-15T01:28:05Z

is this good to merge?

aaron-ang force-pushed the vsock-latency-perf branch from 90ff871 to 75e4ea8 Compare January 6, 2026 02:22

JamesC1305 assigned Manciukic Jan 14, 2026

JamesC1305 added the Status: Awaiting author Indicates that an issue or pull request requires author action label Jan 14, 2026

aaron-ang force-pushed the vsock-latency-perf branch from 75e4ea8 to 1a80bd9 Compare February 26, 2026 04:04

Manciukic reviewed Mar 4, 2026

View reviewed changes

Manciukic reviewed Mar 11, 2026

View reviewed changes

Comment thread tests/integration_tests/performance/test_vsock.py Outdated

aaron-ang force-pushed the vsock-latency-perf branch from e7912cd to dd13239 Compare March 30, 2026 02:18

aaron-ang force-pushed the vsock-latency-perf branch from dd13239 to aa6f57b Compare March 30, 2026 02:25

Manciukic reviewed Apr 1, 2026

View reviewed changes

Comment thread tests/integration_tests/functional/test_vsock.py Outdated

aaron-ang force-pushed the vsock-latency-perf branch from aa6f57b to e273e3b Compare April 6, 2026 03:54

aaron-ang requested a review from Manciukic April 7, 2026 03:38

test(vsock): latency performance test

0aebe4a

Create simple ping latency test in vsock_helper host tools Signed-off-by: Aaron Ang <aaron.angyd@gmail.com>

aaron-ang force-pushed the vsock-latency-perf branch from 08f5641 to 0aebe4a Compare April 10, 2026 03:46

Merge branch 'main' into vsock-latency-perf

082c578

Manciukic added Status: Awaiting review Indicates that a pull request is ready to be reviewed and removed Status: Awaiting author Indicates that an issue or pull request requires author action labels Apr 15, 2026

Conversation

aaron-ang commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Reason

License Acceptance

PR Checklist

Uh oh!

aaron-ang commented Jan 6, 2026

Uh oh!

Manciukic commented Jan 8, 2026

Uh oh!

JamesC1305 commented Feb 25, 2026

Uh oh!

aaron-ang commented Feb 25, 2026

Uh oh!

aaron-ang commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Manciukic left a comment

Choose a reason for hiding this comment

Uh oh!

Manciukic Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

Manciukic Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

aaron-ang Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

Manciukic commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Manciukic commented Mar 11, 2026

Uh oh!

aaron-ang commented Mar 25, 2026

Uh oh!

aaron-ang commented Mar 30, 2026

Uh oh!

Manciukic left a comment

Choose a reason for hiding this comment

Uh oh!

Manciukic Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

Manciukic commented Apr 1, 2026

Uh oh!

Manciukic commented Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Manciukic commented Apr 8, 2026

Uh oh!

aaron-ang commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Manciukic commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aaron-ang commented Apr 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aaron-ang commented Jan 6, 2026 •

edited

Loading

aaron-ang commented Feb 26, 2026 •

edited

Loading

Manciukic commented Mar 4, 2026 •

edited

Loading

codecov bot commented Mar 4, 2026 •

edited

Loading

Manciukic commented Apr 1, 2026 •

edited

Loading

aaron-ang commented Apr 10, 2026 •

edited

Loading

Manciukic commented Apr 10, 2026 •

edited

Loading