feat(vmware-vsphere8): add the VCF Operations Real-Time Metrics source (--metrics-source=rtm)

[julienmathis]

Centreon team (internal PR)

Stacked on #6425. The base branch of this PR is fix/vmware-vsphere8-vstats-removed-in-9.1, so the diff below shows only the Real-Time Metrics work. GitHub retargets it to develop automatically once #6425 is merged. Please review #6425 first.

No JIRA ID yet — tell me the ticket and I rename both branches.

Description

#6425 restores metric collection on vSphere 9.1 through the vim25 PerformanceManager, which is the pragmatic answer: it works today, on any vCenter, with no extra infrastructure.

This PR adds the other answer — the one Broadcom officially designates as the replacement for the removed vStats API: the VCF Operations Real-Time Metrics API. It is exposed as --metrics-source=rtm.

It is deliberately shipped as a second source rather than as the answer, because it carries constraints the vim25 source does not.

Why it is not, and cannot be, the default

	vim25 (#6425)	rtm (this PR)
Served by	the vCenter itself	VCF Operations, another appliance
Prerequisite	none	Real-Time Metrics component deployed + a collection profile enabled
Environment	any vCenter	VCF or VVF only
Authentication	vCenter session	JWT bearer, three-step sequence
Credentials	the vCenter ones	its own, on its own host
Granularity	20 s	down to 2 s (ESXi Top profile)

Two consequences drive the design:

1. --metrics-source=auto never selects rtm. Silently redirecting collection to a different appliance, with different credentials, would be surprising and would fail for everyone who does not run VCF. It must be asked for explicitly.
2. Standalone vCenter customers cannot use it at all. This is a real product limitation on Broadcom's side, not something we can engineer around — hence fix(vmware-vsphere8): restore performance counters on vSphere 9.1 (vStats API removed) #6425 remaining the general-purpose answer.

Worth remembering, and it is why this PR exists now rather than later: the ESXi Top profile delivers 2-second granularity that neither vStats nor vim25 can reach. When the fleet moves to VCF 9.1+, this becomes the interesting source rather than the constrained one.

Type of change

• Patch fixing an issue (non-breaking change)
• New functionality (non-breaking change)
• Functionality enhancement or optimization (non-breaking change)
• Breaking change (patch or feature) that might cause side effects breaking part of the Software

What this PR adds

The documented API surface, implemented in full

Authentication is the three-step sequence from the Real-Time Metrics API reference:

POST /suite-api/api/auth/token/acquire      -> OpsToken
GET  /suite-api/api/integrations/services   -> service key of the entry where type = VCF_VODAP
POST /suite-api/api/auth/token/exchange     -> JWT bearer token

The JWT is cached for its announced validity (with a 60 s safety margin), so the three calls are not replayed on every check — verified below. A 401/403 on a query re-authenticates once, then gives up.

Counters are read with the Prometheus-compatible endpoints, under a configurable base path defaulting to /data-query-service:

Endpoint	Use
GET <base>/v1/query	the PromQL instant query returning the counter value
GET <base>/v1/metadata	the metric catalogue, used to validate every name before querying it

PUT /v1/vcenters/{vcId}/metrics_config exists and toggles the STANDARD / VERBOSE profiles, but is deliberately not implemented: a monitoring plugin has no business mutating the collection configuration of the system it observes. Enabling the profile stays an explicit operator action, documented below.

The same safety rules as #6425

No mode is modified. Counter ids keep their vStats naming (cpu.capacity.usage.HOST); the mapping to PromQL metric names lives entirely in the new package, so the three sources are interchangeable behind one option.

Every metric name is validated against /v1/metadata before being queried. An unknown name produces an error naming the metric, the counter and the fix, instead of an empty result.

The identifying label is discovered, not guessed. The plugin inspects the labels of the series the appliance actually returns to find which one carries the managed object id, then prints the value to pin:

Real-Time Metrics: resource ids are carried by the 'moid' label.
Pass --rtm-moid-label=moid to skip this discovery query.

Units are never converted across dimensions. Same rule as vim25: a conversion that would turn kilobytes into hertz is refused rather than reporting a wrong number.

Empty results are disambiguated. An empty PromQL result is ambiguous — an idle resource, or a selector matching nothing. One selector-less query tells them apart, so a wrong resource id or a wrong label is reported explicitly:

Metric 'vcenter_host_cpu_capacity_provisioned_megahertz' has 1 series but none with moid="host-999".
Check the resource id, or the label carrying it (--rtm-moid-label). Values seen for 'moid': host-35.

This matters: without it, the two most likely misconfigurations would look exactly like the silent failure #6425 was written to eliminate.

The two prerequisite failures get their own message. A missing component and a disabled profile are the two things a first-time user will hit:

No 'VCF_VODAP' service is registered on '<host>' (found: VCF_AUTOMATION). The Real-Time Metrics
component is most likely not deployed: install it from Build > Lifecycle > VCF Management > Add Component.

The Real-Time Metrics API of '<host>' advertises no metric. The collection profile is most likely
not enabled: activate 'Standard' and/or 'ESXi Top' for the vCenter object types in
Operate > Administration > Configurations > Policy Definition.

New options

--metrics-source=rtm
--rtm-hostname     VCF Operations appliance (required)
--rtm-username     (required)
--rtm-password     (required)
--rtm-port         default 443
--rtm-proto        default https
--rtm-auth-source  when the account is not local
--rtm-base-path    default /data-query-service
--rtm-moid-label   pin the label carrying the managed object id
--rtm-metric-map   'counter.id=metric_name', repeatable, overrides a mapping

--rtm-metric-map is the escape hatch: if the appliance publishes a metric under a name we do not know, the operator fixes it from the command line without waiting for a release. It is the direct answer to the main limitation below.

How this pull request can be tested ?

Automated tests

perl -Isrc tests/apps/vmware/vsphere8/api.t

63 assertions (38 in #6425, 25 added here):

Area	What is asserted
Options	base path default and trailing-slash stripping, port default, each of --rtm-hostname/username/password required, --rtm-metric-map parsed, malformed entry refused
Metric resolution	known counter resolves with its target unit; a metric absent from the catalogue is refused; an unmapped counter points at --rtm-metric-map; an override wins and does not leak into the package-level table
Unit conversion	megahertz→kHz, kilobytes→MB, ratio→percent, identity; cross-dimension and unknown units refused
Label escaping	quotes and backslashes escaped in the PromQL selector, undef handled

Manual end-to-end verification

No VCF Operations appliance was available, so verification used a mock implementing the documented contract (source at the end). It honours PromQL label selectors, so a wrong selector genuinely returns an empty result as a real endpoint would.

Scenario	Result
Nominal, label discovered	OK: CPU average usage is 40.00 %, used frequency is 3200000 kHz
Nominal, --rtm-moid-label=moid	identical, one query fewer
VCF_VODAP service absent	UNKNOWN: No 'VCF_VODAP' service is registered ... (found: VCF_AUTOMATION) + install path
Collection profile disabled	UNKNOWN: ... advertises no metric + policy path
Wrong password	UNKNOWN: Could not acquire a VCF Operations token ...
--rtm-metric-map to an unknown metric	UNKNOWN: Metric 'does_not_exist' ... is not advertised
Malformed --rtm-metric-map	UNKNOWN: Malformed --rtm-metric-map value 'pasdegal'
Wrong --rtm-moid-label	UNKNOWN: ... has 1 series but none with vcenter="host-35" + labels seen
Unknown resource id	UNKNOWN: ... none with moid="host-999" + values seen
Missing --rtm-hostname	refuses, rather than silently reusing the vCenter credentials

The value is identical across all three sources — 3200 MHz from RTM, 3200000 kHz from vim25, 3200000 from vStats all render as 40.00 % and 3200000 kHz. That cross-source agreement is the point of keeping the conversion layer.

JWT caching, verified

Three consecutive runs sharing a statefile directory, counting calls on token/acquire:

run 1 -> token/acquire called 1 time in total
run 2 -> token/acquire called 1 time in total
run 3 -> token/acquire called 1 time in total

One authentication for three checks: the cache works and VCF Operations is not hammered.

Non-regression on #6425

Re-run after this PR's changes to api.pm:

Scenario	Result
9.1, auto	falls back to vim25, OK ... 3200000 kHz
9.0 healthy, auto	vStats used, /sdk never called
--metrics-source=bogus	UNKNOWN: Unsupported --metrics-source 'bogus'. Expected one of: auto, vstats, vim25, rtm.

Command line

./centreon_plugins.pl --plugin=apps::vmware::vsphere8::esx::plugin --mode=cpu \
  --hostname=<vcenter> --username=<user> --password=<password> --esx-id=host-35 \
  --metrics-source=rtm \
  --rtm-hostname=<vcf-operations-fqdn> \
  --rtm-username=<ops-user> --rtm-password=<ops-password> \
  --verbose

The vCenter options are still required: inventory (resolving --esx-name to an id, listing objects) keeps going to the vCenter REST API. Only the counters come from VCF Operations.

Prerequisites on the appliance, for whoever tests this

1. Install the component — Build → Lifecycle → VCF Management → Add Component → Real-time metrics.
2. Enable a collection profile — Operate → Administration → Configurations → Policy Definition, activate vCenter / Standard and/or Host System / ESX Top. Essentials is always on and cannot be disabled; Standard and ESXi Top require the Real-Time Metrics component to be scaled from Small to Large.
3. The Swagger of the appliance is at https://<vcf-instance-services-fqdn>/data-query-service/swagger-ui/index.html, also reachable from Build → Developer Center → APIs & SDKs.

curl examples

Full authentication sequence and a first query:

OPS=$(curl -sk -X POST https://<vcf-ops>/suite-api/api/auth/token/acquire \
  -H 'Content-Type: application/json' -H 'Accept: application/json' \
  -d '{"username":"<user>","password":"<password>"}' | jq -r .token)

KEY=$(curl -sk https://<vcf-ops>/suite-api/api/integrations/services \
  -H "Authorization: OpsToken $OPS" -H 'Accept: application/json' \
  | jq -r '.services[] | select(.type=="VCF_VODAP") | .serviceKey')

JWT=$(curl -sk -X POST https://<vcf-ops>/suite-api/api/auth/token/exchange \
  -H "Authorization: OpsToken $OPS" -H 'Content-Type: application/json' \
  -d "[\"$KEY\"]" | jq -r .token)

# metric catalogue — empty means no collection profile is enabled
curl -sk https://<vcf-ops>/data-query-service/v1/metadata \
  -H "Authorization: Bearer $JWT" | jq '.data | keys | length'

# instant PromQL query
curl -sk -G https://<vcf-ops>/data-query-service/v1/query \
  -H "Authorization: Bearer $JWT" \
  --data-urlencode 'query=vcenter_host_cpu_capacity_usage_megahertz{moid="host-35"}' | jq .

The second command is the one to run first on a real appliance: it prints the real metric names, which is exactly what this PR needs validated.

Known limitations — please challenge these

1. The PromQL metric names are the weak point, and I want to be blunt about it. The endpoints, the authentication sequence and the response shapes are documented and implemented faithfully; the metric names are not published anywhere I could find, and the Swagger lives on the appliance rather than on the developer portal. The names in %counter_map follow the VCF Operations vCenter-adapter convention but are not verified against a real appliance. The design contains that risk deliberately — every name is validated against /v1/metadata before use, a mismatch is a loud error naming the fix, and --rtm-metric-map corrects any of them without a release. But expect the first run against a real VCF Operations to need a handful of --rtm-metric-map entries, and please send them back so the table can be corrected.
2. Units per metric are declared in the mapping table, not read from the appliance. Prometheus metadata carries a unit field that is empty in practice, so the unit is inferred from the metric name suffix as the convention dictates. If a name is corrected via --rtm-metric-map and its unit differs, the value would be wrong — the dimension check only catches incompatible dimensions, not a kilobytes/megabytes mix-up. Documented in the module header; a --rtm-unit-map companion option is the obvious follow-up if the field reports it is needed.
3. Only instant queries (/v1/query) are used. /v1/query_range would be needed for rate or average-over-window counters, which no current mode requires.
4. PUT /v1/vcenters/{vcId}/metrics_config is not implemented, on purpose (see above).
5. Only HostSystem and VirtualMachine resources, as in the other two sources.
6. No robot/mockoon integration test, same as fix(vmware-vsphere8): restore performance counters on vSphere 9.1 (vStats API removed) #6425. Wiring both new sources into the mockoon fixtures is the natural follow-up and I am happy to do it in either PR.

Checklist

• I have followed the coding style guidelines provided by Centreon
• I have commented my code, especially hard-to-understand areas of the PR
• I have rebased my development branch on the base branch
• In case of a new plugin, I have created the new packaging directory accordingly — n/a; no new dependency either, the source only needs JSON which is already declared
• I have implemented automated tests related to my commits
  • Data used for automated tests are anonymized
• I have reviewed all the help messages in all the .pm files I have modified
  • All sentences begin with a capital letter
  • All sentences end with a period
  • I am able to understand all the help messages
• After having created the PR, I will make sure that all the tests provided in this PR have run and passed

---

Mock VCF Operations appliance used for the end-to-end verification (standalone, click to expand)

Implements the documented contract: three-step authentication, /v1/metadata, /v1/query with real PromQL label-selector filtering. MODE is ok, nordm (VCF_VODAP not registered) or noprofile (empty catalogue).

import json, re, sys, time, urllib.parse
from http.server import BaseHTTPRequestHandler, HTTPServer

MODE = sys.argv[1] if len(sys.argv) > 1 else "ok"

METRICS = {
    "vcenter_host_cpu_capacity_usage_megahertz":       ("host-35", 3200),
    "vcenter_host_cpu_capacity_provisioned_megahertz": ("host-35", 8000),
    "vcenter_host_mem_capacity_usable_kilobytes":      ("host-35", 134217728),
    "vcenter_host_mem_consumed_vms_kilobytes":         ("host-35", 67108864),
    "vcenter_host_power_capacity_usage_watts":         ("host-35", 210),
}

class H(BaseHTTPRequestHandler):
    def log_message(self, *a): pass

    def _json(self, code, payload):
        b = json.dumps(payload).encode()
        self.send_response(code); self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(b))); self.end_headers(); self.wfile.write(b)

    def do_POST(self):
        n = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(n).decode() if n else ""
        if self.path == "/suite-api/api/auth/token/acquire":
            creds = json.loads(body)
            if creds.get("username") != "svc-centreon" or creds.get("password") != "s3cret":
                return self._json(401, {"message": "invalid credentials"})
            return self._json(200, {"token": "OPS-TOKEN-abc", "validity": 3600})
        if self.path == "/suite-api/api/auth/token/exchange":
            if self.headers.get("Authorization") != "OpsToken OPS-TOKEN-abc":
                return self._json(401, {"message": "bad ops token"})
            return self._json(200, {"token": "JWT-xyz", "expiresIn": 1800})
        self._json(404, {"message": "not found"})

    def do_GET(self):
        if self.path == "/suite-api/api/integrations/services":
            if self.headers.get("Authorization") != "OpsToken OPS-TOKEN-abc":
                return self._json(401, {"message": "bad ops token"})
            services = [{"type": "VCF_AUTOMATION", "serviceKey": "k1"}]
            if MODE != "nordm":
                services.append({"type": "VCF_VODAP", "serviceKey": "rtm-service-key"})
            return self._json(200, {"services": services})

        if not self.path.startswith("/data-query-service/"):
            return self._json(404, {"message": "not found"})
        if self.headers.get("Authorization") != "Bearer JWT-xyz":
            return self._json(401, {"status": "error", "errorType": "unauthorized", "error": "bad jwt"})

        parsed = urllib.parse.urlparse(self.path)
        if parsed.path == "/data-query-service/v1/metadata":
            if MODE == "noprofile":
                return self._json(200, {"status": "success", "data": {}})
            return self._json(200, {"status": "success",
                                    "data": {m: [{"type": "gauge", "help": m, "unit": ""}] for m in METRICS}})

        if parsed.path == "/data-query-service/v1/query":
            q = urllib.parse.parse_qs(parsed.query).get("query", [""])[0]
            name = q.split("{")[0]
            if name not in METRICS:
                return self._json(200, {"status": "success", "data": {"resultType": "vector", "result": []}})
            moid, value = METRICS[name]
            labels = {"__name__": name, "moid": moid, "vcenter": "vc01", "name": "esx01.example.tld"}
            sel = re.search(r"\{(.+)\}", q)
            if sel:
                for pair in sel.group(1).split(","):
                    m = re.match(r'\s*([A-Za-z_][A-Za-z0-9_]*)\s*=\s*"(.*)"\s*$', pair)
                    if not m or labels.get(m.group(1)) != m.group(2):
                        return self._json(200, {"status": "success",
                                                "data": {"resultType": "vector", "result": []}})
            return self._json(200, {"status": "success", "data": {"resultType": "vector", "result": [
                {"metric": labels, "value": [int(time.time()), str(value)]}]}})

        self._json(404, {"message": "not found"})

HTTPServer(("127.0.0.1", 8932), H).serve_forever()

Run it next to the vCenter 9.1 mock from #6425, then:

python3 fake_vcfops.py ok &
python3 fake_vc91.py capacity &
perl -Isrc src/centreon_plugins.pl --plugin=apps::vmware::vsphere8::esx::plugin --mode=cpu \
  --hostname=127.0.0.1 --port=8931 --proto=http --username=u --password=p --http-backend=lwp \
  --esx-id=host-35 --statefile-dir=/tmp/vsphere8-test --metrics-source=rtm \
  --rtm-hostname=127.0.0.1 --rtm-port=8932 --rtm-proto=http \
  --rtm-username=svc-centreon --rtm-password=s3cret --verbose

References

• VCF 9.1 Product Support Notes — the vStats removal this PR ultimately answers
• Real-Time Metrics API reference — authentication sequence and operation index
• Unlocking Programmable Infrastructure with VCF 9.1 — Prometheus-compatible edge APIs, collection profiles
• Real-Time Metrics policy workspace — enabling the Standard and ESXi Top profiles