Got an error from OVMS: rpc error: code = ResourceExhausted desc =

[jiekechoo]

Configuration

1. OVMS version = 2024.3-gpu
2. model server startup option:

--grpc_workers 6 # CPU count is 12

3. nireq is not set

Runing OVMS about 24 hours, when I send rpc call to OVMS, got error:

rpc error: code = ResourceExhausted desc = 

Thanks.
Jieke Choo

[atobiszei]

Hi @jiekechoo
Could you provide reproduction steps on client side?
What is the CPU usage at the time of observing this error?
What are OVMS logs at time of this error?

[jiekechoo]

Hi @jiekechoo

Could you provide reproduction steps on client side?

What is the CPU usage at the time of observing this error?

What are OVMS logs at time of this error?

@atobiszei Thanks for your support.

• CPU usage is less than 50%;
• iGPU: <20%;
• Memory: <30%;
• OVMS and the gRPC client are two containers in the same machine;
• the client has more than 30 calls in multiple threads , 2fps each call ;
• at the first 24 hours maybe 30, it worked. I found that there was no inference result for the client, got the error;

I'll try to catch more logs by DEBUG level.

[exzile]

I went looking for a resource leak that could explain ResourceExhausted appearing after long uptime while CPU/GPU/RAM stay low, and found one on the async inference path that I think is a plausible contributor here.

Root cause (modelInferAsync in src/inference_executor.hpp, used by the KServe gRPC and C-API async endpoints): the ExecutingStreamIdGuard holding an infer-request pool slot (from OVInferRequestsQueue) is std::move-d into the OV completion-callback lambda before inferRequest.start_async() is called. The slot is returned to the pool only when that lambda is destroyed — normally when the async callback fires. If start_async() throws, the function returns immediately, the callback never fires, the lambda is never destroyed, and the slot is never returned. Under sustained concurrent load, each such failure permanently loses one slot; over time the pool drains, new requests block waiting for an idle infer request, and gRPC reports ResourceExhausted even though hardware utilization is low — which matches the symptom described here. (The synchronous predict path is unaffected: it uses a stack-allocated guard with RAII release.)

Fix: draft PR #4335 — clears the request callback in the start_async() catch blocks so the guard is destroyed and the slot returned (mirrors the existing empty-callback release used after normal completion).

Caveat: this only triggers when start_async() actually throws (transient OV/device errors), so I can't claim it's definitively the cause of this report without your logs from the failure window. If you can share OVMS logs from around the time the error first appears (especially any start_async/inference error lines), that would confirm whether this is the same path. Either way it's a correct, low-risk hardening.