tune default cpu properties#4323
Conversation
There was a problem hiding this comment.
Pull request overview
This PR tunes OpenVINO CPU default property selection in Docker to better handle throughput mode detection and multi-socket hosts, and adds targeted unit tests for the new decision logic.
Changes:
- Add CPU socket-count detection and refine physical-core-per-socket detection from
/proc/cpuinfo. - Refactor
applyDefaultCpuPropertiesinto a testable overload that capsnum_streamsby physical cores across sockets and avoids settinginference_num_threadswhen the environment appears unconstrained (e.g., HT-visible cores > physical cores). - Move CPU default property application from
ModelManagerinitialization to per-model compilation forCPU, and add unit tests covering constrained/unconstrained + latency/throughput cases.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
src/test/ov_utils_test.cpp |
Adds unit tests for the new applyDefaultCpuProperties overload and its Docker/throughput/latency behaviors. |
src/systeminfo.hpp |
Declares new getSocketsCount() API to support multi-socket-aware defaults. |
src/systeminfo.cpp |
Implements getSocketsCount() and adjusts physical core detection to be per-socket. |
src/ov_utils.hpp |
Exposes a testable overload of applyDefaultCpuProperties accepting explicit system parameters. |
src/ov_utils.cpp |
Implements new CPU default property logic (streams/threads/pinning) and delegates the runtime overload to system info helpers. |
src/modelmanager.cpp |
Removes global CPU property initialization on ov::Core construction. |
src/modelinstance.cpp |
Applies CPU defaults during CPU model compilation (instead of globally at core init). |
| bool isThroughput = false; | ||
| const auto perfIt = properties.find(ov::hint::performance_mode.name()); | ||
| if (perfIt != properties.end()) { | ||
| try { | ||
| isThroughput = (perfIt->second.as<ov::hint::PerformanceMode>() == ov::hint::PerformanceMode::THROUGHPUT); |
There was a problem hiding this comment.
tests are added to ensure detection of throughput mode, socket count can be 0. It is unlikely to exceed uint16
| if (config.getTargetDevice() == "CPU") { | ||
| Status status = applyDefaultCpuProperties(pluginConfig); | ||
| if (!status.ok()) { | ||
| SPDLOG_LOGGER_ERROR(modelmanager_logger, "Failed to apply default CPU properties for model: {}; version: {}; error: {}", | ||
| getName(), getVersion(), status.string()); | ||
| return status; | ||
| } | ||
| } |
There was a problem hiding this comment.
tuning CPU properties in container is expected only in CPU device.
There was a problem hiding this comment.
What about AUTO device?
There was a problem hiding this comment.
Good point from copilot and Adrian.
There was a problem hiding this comment.
in AUTO plugin it is not possible to set those properties.
HETERO plugin is a corner case. For simplicity it will be better to keep default OV settings.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
| try { | ||
| isThroughput = (perfIt->second.as<ov::hint::PerformanceMode>() == ov::hint::PerformanceMode::THROUGHPUT); | ||
| isThroughput = (perfIt->second.as<std::string>() == "THROUGHPUT"); | ||
| } catch (...) { |
There was a problem hiding this comment.
not necessarily error, in case of unexpected problem with detecting performance mode, there will be just default properties. It is not expected to reach this state. Exception is to avoid error in such unexpected situation.
There was a problem hiding this comment.
Even in such case logging what happen is helpful. Otherwise you don't have a clue what was set unless running under debugger.
| SPDLOG_CRITICAL("Failed to initialize OpenVINO Core with CPU properties. Reason: {}", ex.what()); | ||
| throw; | ||
| } | ||
| this->ieCore = std::make_unique<ov::Core>(); |
There was a problem hiding this comment.
Will models loaded by GenAI also inherit those settings?
There was a problem hiding this comment.
no, genai is using different ov::core object
rasapala
left a comment
rasapala
left a comment
There was a problem hiding this comment.
Two copilot comments should be resolved.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
🛠 Summary
CVS-188024
Fix detection of throughput mode and correct default parameters for multi socket host
🧪 Checklist
``