Reduce IR Constant Memory Retention by barnasm1 · Pull Request #35457 · openvinotoolkit/openvino

barnasm1 · 2026-04-22T07:49:33Z

Details:

Tickets:

CVS-183693

AI Assistance:

AI assistance used: yes
If yes, summarize how AI was used and what human validation was performed (build/tests/manual checks).

…s loading

praasz · 2026-04-23T05:44:16Z

-
+        const auto separate_const_weights_loading = use_separate_const_weights_loading() && !weights;
+        if (separate_const_weights_loading) {
+            weights_provider = std::make_shared<ov::util::FileWeightsProvider>(weights_path);


The same use should be if input is stream not only for path

…ovider

…e ID and handle

praasz · 2026-05-18T09:42:46Z

+    std::filesystem::path m_weights_path;
+    size_t m_weights_size = 0;
+    size_t m_weights_source_id = 0;
+    std::shared_ptr<ov::AlignedBuffer> m_weights_source_handle;


It keeps the data source buffer + buffer in the map.
Is there no data duplication?
Will this provider will not live too long that data buffers set in Constant node will be not blocked to be removed if Constant node removed?

praasz · 2026-06-24T07:11:34Z

+    size_t m_weights_source_id = 0;
+    std::shared_ptr<ov::AlignedBuffer> m_weights_source_handle;
+    // Cache of previously loaded weights regions, keyed by (offset, size) of the region in the weights file.
+    std::map<WeightsRegionKey, std::shared_ptr<ov::AlignedBuffer>> m_loaded_weights_regions;


For futher analysis why not use Constant ID (buffer) for it?

praasz · 2026-06-24T07:14:34Z

+}
+
+std::shared_ptr<ov::AlignedBuffer> FileWeightsProvider::make_region(size_t offset, size_t size) {
+    OPENVINO_ASSERT(offset <= m_weights_size && size <= m_weights_size - offset, "Incorrect weights in bin file!");


Looks like common validation pattern, should some helper be created to return bool if offset, size are valid?

praasz · 2026-06-24T07:16:42Z

+    std::ifstream weights_stream(m_weights_path, std::ios::binary);
+    OPENVINO_ASSERT(weights_stream.is_open(), m_weights_path, " cannot be opened");


For analysis, should be opened each time and checked?
Or provider can keep it open?
The provider should be removed when frontend complete model parsing

praasz · 2026-06-24T07:17:49Z

-            if (m_weights->size() < offset + size)
+            OPENVINO_ASSERT(m_weights_provider, "Empty weights data in bin file or bin file cannot be found!");
+
+            if (m_weights_provider->size() < offset + size)


The check is done here and similar in provider are both required?

praasz · 2026-06-24T07:23:31Z

            const size_t offset = item.second.second;
-            const char* data = weights->get_ptr<char>() + offset;
+            auto buffer = weights_provider->make_region(offset, item.second.first);
+            const char* data = buffer->get_ptr<char>();


Is it required, the const accept void*, or make_region/buffer is aligned buffer const also accept such buffer

olpipi

I'd recommend to write tests with weight_provder mocks to check if they are really used as planned

olpipi · 2026-07-13T08:02:01Z

+    size_t m_weights_source_id = 0;
+    std::shared_ptr<ov::AlignedBuffer> m_weights_source_handle;
+    // Cache of previously loaded weights regions, keyed by (offset, size) of the region in the weights file.
+    std::map<WeightsRegionKey, std::shared_ptr<ov::AlignedBuffer>> m_loaded_weights_regions;


[MEDIUM] you only add new buffers into m_loaded_weights_regions but never clean unused buffers. I'd suggest to store them as weak_ptr and clean expired ones regularly.

Suggested change

std::map<WeightsRegionKey, std::shared_ptr<ov::AlignedBuffer>> m_loaded_weights_regions;

std::map<WeightsRegionKey, std::weak_ptr<ov::AlignedBuffer>> m_loaded_weights_regions;

…memory handling

…first

…e WeightsProvider for weight source management

praasz · 2026-07-30T08:21:26Z

+
+size_t get_mmap_region_threshold() {
+    const auto page_size = ov::util::get_system_page_size();
+    return page_size > 0 ? static_cast<size_t>(page_size) : 4096;


Why use some value if page size is 0? Will it be even possible?
Set threshold as 4k can be too small still a lot of overhead can be created.
Lets apply same logic as for ONNX FE where 1M is used . @t-jankowski could you check recommend threshold value?

praasz · 2026-07-30T09:06:15Z

-    if (!m_weights_path.empty())
-        model->get_rt_info()["__weights_path"] = ov::util::path_to_string(m_weights_path);
-    parse_pre_process(m_root, m_weights, model);
+    if (m_weights_provider) {


For future improvement, this check should not be required as weight is mandatory

Implement WeightsProvider and FileWeightsProvider for flexible weight…

a17e69d

…s loading

barnasm1 requested review from a team as code owners April 22, 2026 07:49

barnasm1 self-assigned this Apr 22, 2026

github-actions Bot added category: Core OpenVINO Core (aka ngraph) category: IR FE OpenVINO IR v10 / v11 FrontEnd labels Apr 22, 2026

barnasm1 added do_not_merge and removed category: Core OpenVINO Core (aka ngraph) category: IR FE OpenVINO IR v10 / v11 FrontEnd labels Apr 22, 2026

Merge branch 'master' into separate_buffer_poc

a898eb6

github-actions Bot added category: Core OpenVINO Core (aka ngraph) category: IR FE OpenVINO IR v10 / v11 FrontEnd labels Apr 22, 2026

fix win build

b1e8e47

praasz reviewed Apr 23, 2026

View reviewed changes

fix win buildweight provider + prevend double budder alocatino

8b75d6f

barnasm1 requested review from a team as code owners May 6, 2026 11:58

github-actions Bot added the category: CPU OpenVINO CPU plugin label May 6, 2026

barnasm1 added 8 commits May 6, 2026 12:05

code format

cdc4489

Merge branch 'master' into separate_buffer_poc

a58a3f4

Merge branch 'master' into separate_buffer_poc

bb4e162

do not use env flag

34c9047

weights loading logic if no enable mmap

4d26e5d

Add assertion to check if weights file can be opened in FileWeightsPr…

f64c4e3

…ovider

Add FileRegionBuffer class and enhance FileWeightsProvider with sourc…

a313c19

…e ID and handle

Merge branch 'master' into separate_buffer_poc

f1bbb4e

t-jankowski self-assigned this May 15, 2026

t-jankowski reviewed May 15, 2026

View reviewed changes

Comment thread src/frontends/ir/src/input_model.cpp Outdated

Comment thread src/frontends/ir/src/frontend.cpp Outdated

praasz reviewed May 18, 2026

View reviewed changes

praasz self-assigned this May 18, 2026

barnasm1 added 2 commits June 23, 2026 09:53

Merge branch 'master' into separate_buffer_poc

522ebf3

Fix error message formatting in weight file assertions

a77e0c0

praasz requested review from praasz and t-jankowski June 24, 2026 07:09

praasz reviewed Jun 24, 2026

View reviewed changes

barnasm1 and others added 3 commits July 3, 2026 14:03

use lazy_buffer

ab6624c

Merge branch 'master' into separate_buffer_poc

5039491

FileWeightsProvider fix CI

979b018

t-jankowski approved these changes Jul 6, 2026

View reviewed changes

Merge branch 'master' into separate_buffer_poc

d8dc9a2

olpipi reviewed Jul 13, 2026

View reviewed changes

praasz modified the milestones: 2026.3, 2026.4 Jul 14, 2026

barnasm1 added 8 commits July 27, 2026 10:25

Merge branch 'master' into separate_buffer_poc

e3af9ec

init all or none

de95e5f

avoid double check

ec779b3

Refactor XmlDeserializer to use WeightsProvider for weight management

afb3465

Refactor FileWeightsProvider to remove FileRegionBuffer and optimize …

671dc06

…memory handling

Add weights_provider source and header to xml_util CMakeLists

a9f499e

Refactor InputModelIRImpl constructor to initialize weights_provider …

1ae4bb5

…first

Refactor InputModelIRImpl to remove weights_path parameter and utiliz…

90e5ae7

…e WeightsProvider for weight source management

github-actions Bot added the category: build OpenVINO cmake script / infra label Jul 29, 2026

praasz requested review from olpipi and praasz July 30, 2026 08:13

praasz reviewed Jul 30, 2026

View reviewed changes

barnasm1 added 5 commits August 4, 2026 09:57

Merge branch 'master' into separate_buffer_poc

d8cb394

Merge branch 'master' into separate_buffer_poc

3e5501d

Refactor XmlDeserializer to use const reference for weights_provider

d93206a

Change loaded_weights_regions from shared_ptr to weak_ptr

0190e63

Refactor parse_pre_process to use shared_ptr for Constant creation

6c88331

		std::ifstream weights_stream(m_weights_path, std::ios::binary);
		OPENVINO_ASSERT(weights_stream.is_open(), m_weights_path, " cannot be opened");

	std::map<WeightsRegionKey, std::shared_ptr<ov::AlignedBuffer>> m_loaded_weights_regions;
	std::map<WeightsRegionKey, std::weak_ptr<ov::AlignedBuffer>> m_loaded_weights_regions;

Uh oh!

Conversation

barnasm1 commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Details:

Tickets:

AI Assistance:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

olpipi left a comment

Choose a reason for hiding this comment

Uh oh!

olpipi Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

barnasm1 commented Apr 22, 2026 •

edited

Loading

olpipi Jul 13, 2026 •

edited

Loading