[NPU] Weight sharing with cache: Part 1 make compiled model weights WeightContext friendly - #37427
Conversation
github-actions
Bot
added
category: build
|
|
||
| #include "compiled_model.hpp" | ||
| #include "npuw_transformations/kv_axes_position.hpp" | ||
| #include "openvino/core/weight_sharing_util.hpp" |
There was a problem hiding this comment.
not needed right now
| // TODO in case of cross-context weight sharing the current implementation has been allocating a single buffer for several weights coul be shared | ||
| // In this case we don't need to allocate the signle buffer for such weight, we can reuse it instead. | ||
| // But we following the current logic, we still have to allocate a single buffer for non-shared weights. | ||
| // Implication: we need to return multiple ZeroTensors instead of a single one. |
There was a problem hiding this comment.
using the SharedWeightAssigner we can get already several page-aligned 2GB buffers where all constant are allocated sequentially, nothing will need to be done here in that case. ZeroTensor CAN BE imported from that aligned memory without copying and reallocating.
However, we must be able to return several initInputsAllocatedTensor from that function, which must not be a problem as ones are never used in caller contexts
| }; | ||
|
|
||
| explicit SharedWeightsAssigner(Options options); | ||
| CollectResult collect_and_partition(const std::shared_ptr<ov::Model>& model); |
There was a problem hiding this comment.
the API imposes two phases here: gathering and mutating
After the gathering phase the API provides a Statistic object which metrics can be estimated in order to make a decision whether we want to proceed with weight-sharing for this particular case or not.
The mutating phase comes separately and may be carried out in the case of it is justifiable according to the desired Statistic metrics
1 participant
Details:
Tickets:
AI Assistance: