Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 36 additions & 36 deletions src/plugins/intel_npu/tools/protopipe/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,9 +54,9 @@ log_level: INFO
#### OpenVINO parameters
- `priority` - **Optional**. Model priority: _HIGH_, _MEDIUM_, _LOW_. (Default: _MEDIUM_)
- `config` - **Optional**. OpenVINO Plugin specific parameters.
- `device` - **Optional**. OpenVINO device name.
- `ip` - **Optional**. Input layer precision: _FP16_, _FP32_, _U8_, _I32_.
- `op` - **Optional**. Output layer precision: _FP16_, _FP32_, _U8_, _I32_.
- `device` - **Optional**. OpenVINO device name.
- `ip` - **Optional**. Input layer precision: _FP16_, _FP32_, _U8_, _I32_, _BOOL_.
- `op` - **Optional**. Output layer precision: _FP16_, _FP32_, _U8_, _I32_, _BOOL_.
- `il` - **Optional**. Input layer layout.
- `ol` - **Optional**. Output layer layout.
- `iml` - **Optional**. Input model layout.
Expand Down Expand Up @@ -90,7 +90,7 @@ Examples:
```

### Graph structure
There are two ways to describe the execution graph structure in Protopipe:
There are two ways to describe the execution graph structure in Protopipe:
1. Using [Dependency Graph](#dependency-graph) (preferable)
2. Using [Network Sequence](#network-sequence) (old)

Expand Down Expand Up @@ -164,7 +164,7 @@ In this case the section `connections` **can be omitted**.
```
#1: Invalid - The list must contain at least two operations to connect
- [A]
#2: Invalid - Self-loop is prohibited
#2: Invalid - Self-loop is prohibited
- [A, A]
#3: Invalid - Loop is prohibited
- [A, B, C, A]
Expand Down Expand Up @@ -218,28 +218,28 @@ graph LR;
D[Model-D.xml]
E[Model-E.xml]
F[Model-F.xml]

D --> E
D --> F

end

A --> B
B --> C
```

#### Network Sequence
There is also a way to describe the graph by using chain-like structure:
`network` - **Required**. List or list of lists of model parameters. Follow [Model Parameters](#model-parameters) for the details.
`delay_in_us` - **Optional**. Delay between models in microseconds.
There is also a way to describe the graph by using chain-like structure:
`network` - **Required**. List or list of lists of model parameters. Follow [Model Parameters](#model-parameters) for the details.
`delay_in_us` - **Optional**. Delay between models in microseconds.

```
input_stream_list:
- network:
- { name: A.xml, ip: FP16, il: NCHW, device: CPU }
- [{ name: B.xml, ip: FP16, op: FP16 }, { name: C.xml, ip: FP16, op: FP16 }]
- { name: D.xml, ip: FP16, op: FP16, config: { PEROFMRANCE_HINT: LATENCY } }
delay_in_us: 5000
delay_in_us: 5000
```

```mermaid
Expand All @@ -255,13 +255,13 @@ input_stream_list:
### Scenario parameters
The list of scenarios are specified by using `multi_inference` parameter, every scenario has the following parameters:
- `name` - **Optional**. The name of execution scenario.
- `input_stream_list` - **Required**. The list of the streams that will be run in parallel.
- `input_stream_list` - **Required**. The list of the streams that will be run in parallel.

Every stream has the following execution parameters:
- `name` - **Optional**. The name of the stream.
- `iteration_count` - **Optional**. Number of iterations to execute.
- `exec_time_in_secs` - **Optional**. Execute until timeout specified.
- `frames_interval_in_ms` - **Optional**. Execution frequency of the stream (**Default**: 0 - Unbounded)
- `name` - **Optional**. The name of the stream.
- `iteration_count` - **Optional**. Number of iterations to execute.
- `exec_time_in_secs` - **Optional**. Execute until timeout specified.
- `frames_interval_in_ms` - **Optional**. Execution frequency of the stream (**Default**: 0 - Unbounded)
- `target_fps` - **Optional**. Execution frequency of the stream. `target_fps = 1000 / frames_interval_in_ms`. `target_fps` and `frames_interval_in_ms` are mutually exclusive and cannot be provided together.
- `target_latency_in_ms` - **Optional**. When iteration isn't finished within specified interval, the next frame will be dropped from execution. (**Default**: Disabled)
- `op_desc`/`conections` or `network` - **Required**. Execution graph structure. Follow [Graph structure](#graph-structure) for the details.
Expand All @@ -276,7 +276,7 @@ Every stream has the following execution parameters:
workload_type : {initial_value: Default, change_to: [Efficient, Default], change_interval: 5, repeat: true}
```
### Config example
Consider the following scenario that consists of two parallel streams specified on `config.yaml`:
Consider the following scenario that consists of two parallel streams specified on `config.yaml`:
```
model_dir:
local: C:\workspace\models
Expand Down Expand Up @@ -315,7 +315,7 @@ multi_inference:
E-->F;
F-->G;
```

Run:
```
./protopipe -cfg config.yaml --drop_frames
Expand All @@ -329,21 +329,21 @@ stream 1: throughput: <number> FPS, latency: min: <number> ms, avg: <number> ms,
```

## How to run
Protopipe has the following `CLI` options to configure the execution behaviour:
Protopipe has the following `CLI` options to configure the execution behaviour:

`--cfg <path>` - Path to configuration file.
`--drop_frames`- **Optional**. Drop frames if they come earlier than stream is completed. E.g if `stream` works with `target_fps: 10` (~`100ms` latency) but stream iteration takes `150ms` - the next iteration will be triggered only in `50ms` if option is enabled.
`--pipeline` - **Optional**. Enables pipelined execution for all scenarios/streams.
`--niter <value>` - **Optional**. Number of iterations. If specified overwrites termination criterion specified in configuration file for all scenarios/streams.
`-t <value>` - **Optional**. Time in seconds. If specified overwrites termination criterion specified in configuration file for all scenarios/streams.
`--cfg <path>` - Path to configuration file.
`--drop_frames`- **Optional**. Drop frames if they come earlier than stream is completed. E.g if `stream` works with `target_fps: 10` (~`100ms` latency) but stream iteration takes `150ms` - the next iteration will be triggered only in `50ms` if option is enabled.
`--pipeline` - **Optional**. Enables pipelined execution for all scenarios/streams.
`--niter <value>` - **Optional**. Number of iterations. If specified overwrites termination criterion specified in configuration file for all scenarios/streams.
`-t <value>` - **Optional**. Time in seconds. If specified overwrites termination criterion specified in configuration file for all scenarios/streams.
`--mode <value>` - **Optional**. Execution mode: *performance*, *reference*, *validation*, *accuracy* (**Default**: *performance*)
`--reference_device <value>` - **Optional**. Reference device for accuracy mode comparison. (**Default**: *CPU*)
`--target_device <value>` - **Optional**. Target device for accuracy mode comparison. (**Default**: *NPU*)
`--exec_filter <value>` - **Optional**. Run only the scenarios that match provided string pattern.
`--inference_only` - **Optional**. Run only inference execution for every model excluding i/o data transfer (**Default**: true)
`--exec_filter <value>` - **Optional**. Run only the scenarios that match provided string pattern.
`--inference_only` - **Optional**. Run only inference execution for every model excluding i/o data transfer (**Default**: true)

### Filtering
Sometime it's needed to run particular set of scenarios specified in config file rather than all of them.
Sometime it's needed to run particular set of scenarios specified in config file rather than all of them.
For example consider the following config file with three scenarios specified in `scenarios.yaml`:
```
model_dir:
Expand All @@ -360,13 +360,13 @@ multi_inference:
- network:
- { name: C.xml }
```
By default all scenarios are assigned unique names according to the following `multi_inference_<number>` pattern.
E.g scenario with model `A.xml` has default name `multi_inference_0`.
Use `-exec_filter <value>` CLI option to control what scenarios from config should be executed:
By default all scenarios are assigned unique names according to the following `multi_inference_<number>` pattern.
E.g scenario with model `A.xml` has default name `multi_inference_0`.
Use `-exec_filter <value>` CLI option to control what scenarios from config should be executed:
```
./protopipe -cfg scenarios.yaml -niter 100 -exec_filter=".*[0-1]"
```
Only `multi_inference_0` and `multi_inference_1` scenarios will be executed.
Only `multi_inference_0` and `multi_inference_1` scenarios will be executed.

It's also possible to overwrite the default names in config file:
```
Expand Down Expand Up @@ -416,7 +416,7 @@ As the prerequisite for accuracy validation it's useful to have a mechanism that
Use additional parameters to configure `reference` mode:
- `input_data` - **Required**. Path that contain input data for the model, if entity under the path is empty, input data will be generated randomly and dumped into the path specified.
- `output_data` - **Required**. Path where to dump reference output data.
- `random` - **Optional**. Initializer to generate input data randomly. (Default: ` { dist: uniform, low: 0.0, high: 255, seed: -1 }`)
- `random` - **Optional**. Initializer to generate input data randomly. (Default: ` { dist: uniform, low: 0.0, high: 255, seed: -1 }`, or `{ dist: uniform, low: 0.0, high: 2.0 }` for layers with `ip: BOOL`, so only valid `0`/`1` values are generated)

Examples:
```
Expand Down Expand Up @@ -444,7 +444,7 @@ Protopipe has the dedicated `validation` mode to perform accuracy validation. Ex
- `save_validation_outputs` - **Optional**. Accepts the path where to dump actual execution outputs. (Default: disabled)
- `metric` - **Optional**. Accuracy metric to compare actual vs reference outputs. (Default: `{ name: norm, tolerance: 0.0 }`)
- `input_data` - **Required**. Path that contain input data for the model.
- `output_data` - **Required**. Path that contain **reference** data to compare with.
- `output_data` - **Required**. Path that contain **reference** data to compare with.

**Note**: If folder is provided either for **input_data** or **output_data**, it must be in the following format:
```
Expand Down Expand Up @@ -472,11 +472,11 @@ Examples:
```

### Supported metrics
1. L2 Norm: $$\text{Norm}(\mathbf{A}, \mathbf{B}) = \sqrt{\sum_{i,j} (A_{i,j} - B_{i,j})^2}$$
1. L2 Norm: $$\text{Norm}(\mathbf{A}, \mathbf{B}) = \sqrt{\sum_{i,j} (A_{i,j} - B_{i,j})^2}$$
Parameters:
- `name: norm` - **Required**. Enables L2 Norm metric.
- `tolerance` - **Required**. If value of metric is greater than **tolerance** it will be treated as **FAIL**.
3. Cosine similarity: $$\text{Cosine}(\mathbf{A}, \mathbf{B}) = \frac{\mathbf{A} \cdot \mathbf{B}}{\| \mathbf{A} \|_2 \| \mathbf{B} \|_2}$$
3. Cosine similarity: $$\text{Cosine}(\mathbf{A}, \mathbf{B}) = \frac{\mathbf{A} \cdot \mathbf{B}}{\| \mathbf{A} \|_2 \| \mathbf{B} \|_2}$$
Parameters:
- `name: cosine` - **Required**. Enables cosine similarity metric.
- `threshold` - **Required**. If value of metric is lower than **threshold** it will be treated as **FAIL**.
Expand Down Expand Up @@ -616,7 +616,7 @@ Iteration <number>:
```
### Verify the installation
**Note**: Make sure `opencv_*` libraries are visible in the environment:
- Windows:
- Windows:
```
set PATH=<path-to-opencv>\build\bin\Release\;%PATH%
```
Expand Down
3 changes: 3 additions & 0 deletions src/plugins/intel_npu/tools/protopipe/src/parser/config.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@

#include "utils/error.hpp"
#include "utils/logger.hpp"
#include "utils/utils.hpp"

#include <cstdint>
#include <filesystem>
Expand Down Expand Up @@ -99,6 +100,8 @@ static int toDepth(const std::string& prec) {
return CV_8U;
if (prec == "I32" || prec == "I64")
return CV_32S;
if (prec == "BOOL")
return utils::kBooleanDepth;
throw std::logic_error("Unsupported precision type: " + prec);
}

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
//
// Copyright (C) 2018-2026 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
Expand Down Expand Up @@ -93,12 +92,8 @@ struct InputDataVisitor {

void InputDataVisitor::operator()(std::monostate) {
// NB: No path provided - generate input random data using initializers.
const auto input_names = extractLayerNames(infer.input_layers);
const auto& initializers = opts.initializers_map.at(infer.tag);

auto default_initialzer =
opts.global_initializer ? opts.global_initializer : std::make_shared<UniformGenerator>(0.0, 255.0);
auto layer_initializers = unpackWithDefault(initializers, input_names, default_initialzer);
auto layer_initializers = resolveInitializers(infer.input_layers, initializers, opts.global_initializer);
providers = createRandomProviders(infer.input_layers, std::move(layer_initializers));
};

Expand All @@ -125,9 +120,7 @@ void InputDataVisitor::operator()(const std::string& path_str) {
// NB: Provided path doesn't exist - fall back to random data generation.
LOG_INFO() << "Input data path: " << path << " for model: " << infer.tag
<< " doesn't exist - using random data" << std::endl;
auto default_initialzer =
opts.global_initializer ? opts.global_initializer : std::make_shared<UniformGenerator>(0.0, 255.0);
auto layer_initializers = unpackWithDefault(initializers, input_names, default_initialzer);
auto layer_initializers = resolveInitializers(infer.input_layers, initializers, opts.global_initializer);
providers = createRandomProviders(infer.input_layers, std::move(layer_initializers));
}
}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
//
// Copyright (C) 2018-2026 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
Expand Down Expand Up @@ -134,6 +133,27 @@ std::vector<IDataProvider::Ptr> createRandomProviders(const LayersInfo& layers,
return providers;
}

static IRandomGenerator::Ptr defaultInitializer(int prec) {
// NB: Booleans are physically stored as U8, but only 0/1 are valid values.
if (prec == utils::kBooleanDepth) {
return std::make_shared<UniformGenerator>(0.0, 2.0);
}
return std::make_shared<UniformGenerator>(0.0, 255.0);
}

std::map<std::string, IRandomGenerator::Ptr> resolveInitializers(
const LayersInfo& layers, const LayerVariantAttr<IRandomGenerator::Ptr>& initializers,
const IRandomGenerator::Ptr& global_initializer) {
auto per_layer = unpackWithDefault(initializers, extractLayerNames(layers), IRandomGenerator::Ptr{});
for (const auto& layer : layers) {
auto& generator = per_layer.at(layer.name);
if (!generator) {
generator = global_initializer ? global_initializer : defaultInitializer(layer.prec);
}
}
return per_layer;
}

std::vector<std::filesystem::path> createDirectoryLayout(const std::filesystem::path& path,
const std::vector<std::string>& layer_names) {
std::vector<std::filesystem::path> dirs_path;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
//
// Copyright (C) 2018-2026 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
Expand Down Expand Up @@ -28,6 +27,13 @@ std::vector<IDataProvider::Ptr> createConstantProviders(LayersDataMap&& layers_d
std::vector<IDataProvider::Ptr> createRandomProviders(const LayersInfo& layers,
const std::map<std::string, IRandomGenerator::Ptr>& generators);

// NB: Resolves a random generator per input layer: explicit per-layer/global initializers
// take precedence, otherwise falls back to a precision-aware default (e.g. boolean layers
// default to {0, 1} instead of the generic U8 {0, 255} range).
std::map<std::string, IRandomGenerator::Ptr> resolveInitializers(
const LayersInfo& layers, const LayerVariantAttr<IRandomGenerator::Ptr>& initializers,
const IRandomGenerator::Ptr& global_initializer);

std::vector<std::filesystem::path> createDirectoryLayout(const std::filesystem::path& path,
const std::vector<std::string>& layer_names);
template <typename T>
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
//
// Copyright (C) 2018-2026 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
Expand All @@ -9,6 +8,7 @@
#include <openvino/openvino.hpp>

#include "utils/error.hpp"
#include "utils/utils.hpp"

#include <fstream>

Expand Down Expand Up @@ -40,6 +40,8 @@ static ov::element::Type toElementType(int cvdepth) {
return ov::element::f32;
case CV_16F:
return ov::element::f16;
case utils::kBooleanDepth:
return ov::element::boolean;
}
throw std::logic_error("Failed to convert opencv depth to ov::element::Type");
}
Expand All @@ -66,6 +68,8 @@ static int toPrecision(ov::element::Type prec) {
return CV_16F;
case ov::element::i64:
return CV_32S;
case ov::element::boolean:
return utils::kBooleanDepth;
}
throw std::logic_error("Unsupported OV precision");
}
Expand Down Expand Up @@ -128,7 +132,7 @@ static void cfgOutputPostproc(ov::preprocess::PrePostProcessor& ppp, const std::
}
}

static void cfgReshape(const std::shared_ptr<ov::Model>& model,
static void cfgReshape(const std::shared_ptr<ov::Model>& model,
const AttrMap<std::vector<size_t>> reshape_map) {
std::map<std::string, ov::PartialShape> partial_shapes;
for (const auto& [layer_name, shape] : reshape_map) {
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
//
// Copyright (C) 2018-2026 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
Expand Down Expand Up @@ -74,11 +73,8 @@ struct InputDataVisitor {
void InputDataVisitor::operator()(std::monostate) {
LOG_INFO() << "Input data path for model: " << infer.tag << " hasn't been provided. Will be generated randomly"
<< std::endl;
auto initializers = opts.initializers_map.at(infer.tag);
auto default_initialzer =
opts.global_initializer ? opts.global_initializer : std::make_shared<UniformGenerator>(0.0, 255.0);
auto per_layer_initializers =
unpackWithDefault(initializers, extractLayerNames(infer.input_layers), default_initialzer);
const auto& initializers = opts.initializers_map.at(infer.tag);
auto per_layer_initializers = resolveInitializers(infer.input_layers, initializers, opts.global_initializer);
providers = createRandomProviders(infer.input_layers, per_layer_initializers);
};

Expand All @@ -90,11 +86,8 @@ void InputDataVisitor::operator()(const std::string& path_str) {
auto layers_data = uploadData(path, infer.tag, infer.input_layers, LayersType::INPUT);
providers = createConstantProviders(std::move(layers_data), extractLayerNames(infer.input_layers));
} else {
auto initializers = opts.initializers_map.at(infer.tag);
auto default_initialzer =
opts.global_initializer ? opts.global_initializer : std::make_shared<UniformGenerator>(0.0, 255.0);
auto per_layer_initializers =
unpackWithDefault(initializers, extractLayerNames(infer.input_layers), default_initialzer);
const auto& initializers = opts.initializers_map.at(infer.tag);
auto per_layer_initializers = resolveInitializers(infer.input_layers, initializers, opts.global_initializer);
LOG_INFO() << "Input data path: " << path << " for model: " << infer.tag
<< " provided but doesn't exist - will be generated randomly" << std::endl;
providers = createRandomProviders(infer.input_layers, per_layer_initializers);
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
//
// Copyright (C) 2018-2026 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
Expand Down Expand Up @@ -81,9 +80,7 @@ void InputDataVisitor::operator()(const std::string& path_str) {
std::filesystem::create_directories(path.parent_path());
dump_path_vec = {path};
}
auto default_initialzer =
opts.global_initializer ? opts.global_initializer : std::make_shared<UniformGenerator>(0.0, 255.0);
auto layer_initializers = unpackWithDefault(initializers, input_names, default_initialzer);
auto layer_initializers = resolveInitializers(infer.input_layers, initializers, opts.global_initializer);
providers = createRandomProviders(infer.input_layers, std::move(layer_initializers));
for (uint32_t i = 0; i < infer.input_layers.size(); ++i) {
metas[i].set(Dump{dump_path_vec[i]});
Expand Down
Loading
Loading