Add model load/unload/list loaded command routing

sdk_v2/cpp/CMakeLists.txt

@@ -168,6 +168,7 @@ set(FOUNDRY_LOCAL_SOURCES
     src/inferencing/model_load_manager.cc
     src/inferencing/generative/chat/onnx_chat_generator.cc
     src/model.cc
+    src/model_command_router.cc
     src/inferencing/generative/openresponses/response_converter.cc
     src/inferencing/generative/openresponses/response_store.cc
     src/contracts/responses_json.cc

sdk_v2/cpp/docs/ModelCommandRoutingPlan.md

@@ -0,0 +1,211 @@
+# Model Command Routing (External Service Mode) — C++ Implementation Plan
+
+## Overview
+
+When the SDK is configured with `Configuration::external_service_url`, the three
+**model management commands** must be redirected to that remote Foundry Local
+service over HTTP instead of being executed against the in-process model loader:
+
+| Command     | External endpoint                          |
+|-------------|--------------------------------------------|
+| List loaded | `GET {url}/models/loaded`                  |
+| Load        | `GET {url}/models/load/{url-encoded id}`   |
+| Unload      | `GET {url}/models/unload/{url-encoded id}` |
+
+The OpenAI-compatible endpoints (`/v1/models`, `/v1/chat/completions`, …) are
+**out of scope** — callers reach those directly, not through the SDK.
+
+In the v1 SDKs this redirection was reimplemented in every language binding
+(`ModelLoadManager.cs`, `model_load_manager.py`, `modelLoadManager.ts`,
+`model_load_manager.rs`). In v2 it is handled **once** in the C++ core so every
+binding inherits it for free.
+
+**v1 reference implementation:**
+- `sdk/cs/src/Detail/ModelLoadManager.cs`
+- `sdk/python/src/detail/model_load_manager.py`
+- `sdk/js/src/detail/modelLoadManager.ts`
+- `sdk/rust/src/detail/model_load_manager.rs`
+
+---
+
+## Why a new component (not augmenting `ModelLoadManager`)
+
+The v2 `ModelLoadManager`
+([`inferencing/model_load_manager.h`](../src/inferencing/model_load_manager.h)) is
+**not** the v1 `ModelLoadManager`. Despite the shared name they have completely
+different responsibilities:
+
+- **v1 `ModelLoadManager`** — a thin *router*: three methods, each branching
+  `if (external_url) HTTP else core`.
+- **v2 `ModelLoadManager`** — the *local ORT GenAI lifecycle owner*: holds the
+  `GenAIModelInstance` map, an `IEpDetector&`, per-model session refcounts,
+  `RejectNewLoads()`, and `UnloadAll()` drain-on-shutdown.
+
+Bolting `external_service_url` + HTTP into the v2 class would conflate two
+unrelated reasons-to-change ("own in-process ORT GenAI instances" vs. "talk to a
+remote server") and its signatures fight it: `LoadModel` returns
+`LoadResult{ GenAIModelInstance* }` and `GetLoadedModel` returns a live local
+instance — neither exists in external mode. `UnloadAll`, `RejectNewLoads`, EP
+detection, and session drain are all meaningless remotely.
+
+**Decision:** introduce a higher-level façade, `ModelCommandRouter`, that owns the
+local-vs-external decision for only the three management commands. The v2
+`ModelLoadManager` stays a pure local lifecycle owner.
+
+---
+
+## Architecture
+
+```
+flModel.Load/Unload          flCatalog.GetLoadedModels
+        │                              │
+        ▼                              ▼
+   Model::Load/Unload/IsLoaded   BaseModelCatalog::GetLoadedModels
+        │                              │
+        └──────────────┬───────────────┘
+                       ▼
+              ModelCommandRouter            ← owns local-vs-external decision
+              ├── local  ──► ModelLoadManager (LoadModel / UnloadModel / GetLoadedModelIds)
+              └── external ─► http_client → {external_service_url}/models/...
+```
+
+### Router interface
+
+```cpp
+class ModelCommandRouter {
+ public:
+  ModelCommandRouter(std::optional<std::string> external_service_url,
+                     ModelLoadManager& load_manager,
+                     std::string app_name,
+                     ILogger& logger);
+
+  // Local: ModelLoadManager::LoadModel(local_path, model_id, ep)
+  // External: GET /models/load/{id}
+  void Load(std::string_view model_id, std::string_view local_path, ExecutionProvider ep);
+
+  // Local: ModelLoadManager::UnloadModel(model_id)
+  // External: GET /models/unload/{id}
+  void Unload(std::string_view model_id);
+
+  // Local: ModelLoadManager::GetLoadedModel(model_id) != nullptr
+  // External: membership test against ListLoadedModelIds()
+  bool IsLoaded(std::string_view model_id);
+
+  // Local: ModelLoadManager loaded-map keys
+  // External: GET /models/loaded  (single round-trip, JSON array of ids)
+  std::vector<std::string> ListLoadedModelIds();
+};
+```
+
+### Key design principles
+
+1. **`Model` and the catalog become mode-agnostic.** They call the router; they no
+   longer know external mode exists. `Model` swaps its `ModelLoadManager*` member for
+   a `ModelCommandRouter*`, wired in the same `Model::Create` / `Manager::CreateModel`
+   factory that wires it today.
+2. **`ModelLoadManager` gains no mode logic.** The router *holds a
+   `ModelLoadManager&`* and delegates the local branch to it. The only addition to
+   `ModelLoadManager` is one in-character method, `GetLoadedModelIds()`, that reads
+   its own loaded map.
+3. **List is a single batch call — no N-calls.**
+   `BaseModelCatalog::GetLoadedModels()` is rewritten to call
+   `router.ListLoadedModelIds()` **once** and filter `models_` by set membership,
+   instead of N per-model `IsLoaded()` calls. This also fixes a latent bug: in
+   external mode the current per-model `IsLoaded()` path always returns empty.
+4. **Local inference is untouched.** External mode means "run inference via the
+   external HTTP endpoints." The inference path fetches `GenAIModelInstance*` directly
+   in the session layer, never through `Model`/the router, so it needs no change.
+5. **Reuse existing infrastructure.**
+   [`http/http_client.h`](../src/http/http_client.h) (`HttpGetWithResponse`,
+   `DescribeFailure`) for HTTP + errors, the `UrlEncode` helper from
+   [`download/model_registry_client.cc`](../src/download/model_registry_client.cc) for
+   the `{id}` path segment, and `Configuration::app_name` for the `User-Agent`.
+
+---
+
+## Wiring (`Manager`)
+
+`Manager` owns a `std::unique_ptr<ModelCommandRouter>`, declared **after**
+`model_load_manager_` so it is destroyed first. Construction order:
+
+```
+model_load_manager_  →  model_command_router_  →  catalog_
+```
+
+The router is constructed with `config_.external_service_url`,
+`*model_load_manager_`, `config_.app_name`, and `*logger_`. It is injected into the
+catalog ctor and forwarded into each `Model` via the existing `CreateModel`
+factory (replacing the `ModelLoadManager&` argument).
+
+---
+
+## Error handling
+
+External calls use `http::HttpGetWithResponse` and throw on non-2xx / transport
+failure via `FL_THROW(FOUNDRY_LOCAL_ERROR_NETWORK, http::DescribeFailure(resp))`,
+matching v1 semantics (which threw `FoundryLocalException` on `!IsSuccessStatusCode`).
+`ListLoadedModelIds` parses the response body as a JSON array of strings; an empty
+body yields an empty list.
+
+---
+
+## Open questions — decisions
+
+1. **Load of an un-cached model in external mode.**
+   *Decision: accept the cache-only constraint.* The v2 API is model-object-centric
+   and the catalog is cache-only in external mode, so callers can only `Load()` a model
+   already present in their local cache. We ship the model-object path now and **defer**
+   any `Manager::LoadModelById(id)` escape hatch until there is a concrete need. (v1
+   took raw id strings and could load anything the remote had; that capability is
+   intentionally not reproduced yet.)
+
+2. **Session-creation guard in external mode.**
+   *Decision: add an explicit guard.* [`configuration.h`](../src/configuration.h)
+   documents that session creation is blocked in external mode, but `Session_CreateImpl`
+   currently has no explicit check — it only fails indirectly (no local instance). Add
+   an explicit `FL_THROW(FOUNDRY_LOCAL_ERROR_INVALID_USAGE, …)` guard on the
+   session-creation path when `external_service_url` is set, mirroring the existing
+   `StartWebService` guard, so the failure is clear and intentional.
+
+3. **Endpoint timeout.**
+   *Decision: drop the v1 `timeout` query param; set a deliberate client-side timeout
+   in the core instead.* Investigation of the v1 bindings showed the `timeout` query
+   param was **never implemented** — it exists only as a commented-out TODO in C#
+   (`// { "timeout", ... }`) and Python (`# "timeout": "30"`), and is absent in JS/Rust.
+   So there is no real capability to preserve. The client-side HTTP timeout was also
+   inconsistent and accidental rather than designed: Python hard-coded **10s** on all
+   three calls (effectively a latent bug — a real model load routinely exceeds 10s),
+   C# used the `HttpClient` **default 100s**, and JS/Rust were unbounded.
+
+   Moving this into the C++ core gives every binding a *single, deliberate* value for
+   the first time. Use a generous `HttpRequestOptions.timeout` for the load call
+   (target ~5 minutes — model load can be slow) and the default for list/unload. We do
+   **not** reproduce Python's 10s. A future server-side `timeout` query param can be
+   added later if the remote service ever supports one.
+
+---
+
+## Work breakdown
+
+**Implementation (`@CppCoder`)**
+- New `src/model_command_router.{h,cc}`.
+- Add `ModelLoadManager::GetLoadedModelIds()` (reads its own loaded map).
+- Swap `Model`'s member `ModelLoadManager* → ModelCommandRouter*`; update
+  `Model::Create` / `Manager::CreateModel` wiring.
+- Rewrite `BaseModelCatalog::GetLoadedModels()` to the single-batch path; thread the
+  router through the catalog ctor.
+- Construct/own the router in `Manager` with correct declaration order.
+- Add the external-mode session-creation guard (open question #2).
+
+**Tests (`@Tester`)**
+- Router unit tests covering both branches (mock/loopback the HTTP side).
+- External-mode integration test mirroring v1 `TestModelLoadManagerExternalService`:
+  start a real local web service, point a second `Manager` at it via
+  `external_service_url`, verify load / unload / list round-trip.
+- Regression test that `GetLoadedModels()` is correct in external mode.
+- Test that session creation throws in external mode.
+
+**Review (`@Reviewer`)**
+- Confirm `ModelLoadManager` gained no mode logic.
+- Confirm the router holds no inference/EP concerns.
+- Confirm error/ownership semantics match SDK conventions.

sdk_v2/cpp/src/c_api.cc

@@ -1662,6 +1662,13 @@ FL_API_STATUS_IMPL(Session_CreateImpl, const flModel* model, flSession** out_ses
     return MakeStatus(FOUNDRY_LOCAL_ERROR_INVALID_ARGUMENT, "null argument");
   }
 
+  // External mode runs inference via the remote HTTP endpoints, not through a local session.
+  // Fail fast with a clear message instead of failing indirectly when no local instance exists.
+  if (fl::Manager::Instance().GetConfiguration().external_service_url.has_value()) {
+    return MakeStatus(FOUNDRY_LOCAL_ERROR_INVALID_USAGE,
+                      "cannot create a local inference session when external_service_url is configured");
+  }
+
   auto session = fl::Session::Create(*AsImpl(model));
   *out_session = AsHandle<flSession>(session.release());
   return nullptr;

sdk_v2/cpp/src/catalog/azure_model_catalog.cc

@@ -16,12 +16,13 @@ namespace fl {
 AzureModelCatalog::AzureModelCatalog(std::vector<std::pair<std::string, std::optional<std::string>>> catalog_urls,
                                      std::string cache_dir,
                                      ModelFactory model_factory,
+                                     ModelCommandRouter& router,
                                      const IEpDetector& ep_detector,
                                      ILogger& logger,
                                      bool cache_only,
                                      std::string catalog_region,
                                      bool disable_region_fallback)
-    : BaseModelCatalog(catalog_urls.empty() ? kDefaultCatalogUrl : catalog_urls.front().first, logger),
+    : BaseModelCatalog(catalog_urls.empty() ? kDefaultCatalogUrl : catalog_urls.front().first, router, logger),
       catalog_urls_(std::move(catalog_urls)),
       cache_dir_(std::move(cache_dir)),
       model_factory_(std::move(model_factory)),

sdk_v2/cpp/src/catalog/azure_model_catalog.h

@@ -14,6 +14,8 @@
 
 namespace fl {
 
+class ModelCommandRouter;
+
 /// Azure-specific catalog. Fetches from Azure Foundry catalog API,
 /// scans local cache, merges results.
 /// Maps to C# AzureModelCatalog.
@@ -24,6 +26,7 @@ class AzureModelCatalog : public BaseModelCatalog {
   AzureModelCatalog(std::vector<std::pair<std::string, std::optional<std::string>>> catalog_urls,
                     std::string cache_dir,
                     ModelFactory model_factory,
+                    ModelCommandRouter& router,
                     const IEpDetector& ep_detector,
                     ILogger& logger,
                     bool cache_only = false,

sdk_v2/cpp/src/catalog/base_model_catalog.cc

@@ -2,13 +2,16 @@
 // Licensed under the MIT License.
 #include "catalog/base_model_catalog.h"
 
+#include "model_command_router.h"
+
 #include <foundry_local/foundry_local_c.h>
 
 #include <algorithm>
 #include <atomic>
 #include <cctype>
 #include <fmt/format.h>
 #include <map>
+#include <unordered_set>
 
 namespace fl {
 
@@ -88,8 +91,8 @@ bool CompareModelsForSort(const Model& m1, const Model& m2) {
 
 }  // anonymous namespace
 
-BaseModelCatalog::BaseModelCatalog(std::string name, ILogger& logger)
-    : name_(std::move(name)), logger_(logger) {}
+BaseModelCatalog::BaseModelCatalog(std::string name, ModelCommandRouter& router, ILogger& logger)
+    : name_(std::move(name)), router_(router), logger_(logger) {}
 BaseModelCatalog::~BaseModelCatalog() = default;
 
 void BaseModelCatalog::PopulateModels(std::vector<Model> variants) const {
@@ -318,10 +321,17 @@ std::vector<Model*> BaseModelCatalog::GetCachedModels() const {
 std::vector<Model*> BaseModelCatalog::GetLoadedModels() const {
   EnsurePopulated();
 
+  // Single batch call instead of N per-model IsLoaded() round-trips. This is also correct in
+  // external mode, where the router fetches the loaded-id set from the remote service in one
+  // request; the old per-model path always returned empty there.
+  auto loaded = router_.ListLoadedModelIds();
+  std::unordered_set<std::string> loaded_set(loaded.begin(), loaded.end());
+
   std::lock_guard<std::mutex> lock(mutex_);
   std::vector<Model*> result;
   for (auto& m : models_) {
-    if (m->IsLoaded()) {
+    // Container Id() delegates to the selected variant, preserving the prior semantics.
+    if (loaded_set.count(m->Id()) > 0) {
       result.push_back(m.get());
     }
   }

sdk_v2/cpp/src/catalog/base_model_catalog.h

@@ -14,6 +14,8 @@
 
 namespace fl {
 
+class ModelCommandRouter;
+
 /// BaseModelCatalog: holds model collection, indices, and lookup logic.
 /// Derived classes implement FetchModels() to provide models from their specific source.
 /// The base class owns caching, indexing, and thread-safe refresh.
@@ -26,7 +28,7 @@ namespace fl {
 /// Maps to C# BaseModelCatalog<TModelInfo>.
 class BaseModelCatalog : public ICatalog {
  public:
-  BaseModelCatalog(std::string name, ILogger& logger);
+  BaseModelCatalog(std::string name, ModelCommandRouter& router, ILogger& logger);
   ~BaseModelCatalog() override;
 
   const std::string& GetName() const override { return name_; }
@@ -87,6 +89,7 @@ class BaseModelCatalog : public ICatalog {
   void EnsurePopulated(bool allow_refresh = false) const;
 
   std::string name_;
+  ModelCommandRouter& router_;
   ILogger& logger_;
 };
 

sdk_v2/cpp/src/download/model_registry_client.cc

@@ -21,24 +21,6 @@ std::string BuildBaseUrl(const std::string& region) {
   return "https://" + region + ".api.azureml.ms/modelregistry/v1.0/registry/models/nonazureaccount?assetId=";
 }
 
-/// URL-encode a string (percent-encoding).
-std::string UrlEncode(const std::string& value) {
-  std::string result;
-  result.reserve(value.size() * 2);
-  for (unsigned char c : value) {
-    if ((c >= 'A' && c <= 'Z') || (c >= 'a' && c <= 'z') || (c >= '0' && c <= '9') ||
-        c == '-' || c == '_' || c == '.' || c == '~') {
-      result += static_cast<char>(c);
-    } else {
-      static const char hex[] = "0123456789ABCDEF";
-      result += '%';
-      result += hex[c >> 4];
-      result += hex[c & 0x0F];
-    }
-  }
-  return result;
-}
-
 constexpr const char* kUserAgent = "AzureAiStudio";
 
 }  // anonymous namespace

sdk_v2/cpp/src/inferencing/model_load_manager.cc

@@ -270,4 +270,16 @@ GenAIModelInstance* ModelLoadManager::GetLoadedModel(std::string_view model_id)
   return nullptr;
 }
 
+std::vector<std::string> ModelLoadManager::GetLoadedModelIds() const {
+  std::lock_guard<std::mutex> lock(mutex_);
+
+  std::vector<std::string> ids;
+  ids.reserve(loaded_models_.size());
+  for (const auto& [id, instance] : loaded_models_) {
+    ids.push_back(id);
+  }
+
+  return ids;
+}
+
 }  // namespace fl

sdk_v2/cpp/src/inferencing/model_load_manager.h

@@ -67,6 +67,10 @@ class ModelLoadManager {
   /// The returned pointer is valid until UnloadModel is called for this model_id.
   GenAIModelInstance* GetLoadedModel(std::string_view model_id);
 
+  /// Get the ids of all currently-loaded models (the keys of the loaded map).
+  /// Snapshot taken under the lock; the result is a copy, safe to use after the lock is released.
+  std::vector<std::string> GetLoadedModelIds() const;
+
   /// Reject all future LoadModel calls. Called by Manager::Shutdown().
   /// Idempotent and thread-safe.
   void RejectNewLoads();