perf(mobile-apps): speed up and harden Dataverse schema creation

[Shubham Agarwal (kanu-shubham)]

Summary

This PR improves Dataverse schema-provisioning performance while preserving the same generated data model, relationships, security behavior, and offline capabilities.

The optimized workflow created 8 tables and 10 lookup relationships in 4 minutes 17 seconds, compared with 5 minutes 56 seconds for the baseline.

What changed

The Dataverse creation workflow was streamlined to reduce metadata-provisioning overhead:

• Creates each table with its planned non-lookup columns included in the table request.
• Creates lookup relationships only after both endpoint tables exist.
• Preserves dependency-tier ordering.
• Avoids unsafe concurrent metadata writes, which can cause Dataverse metadata-lock failures.
• Retains collision detection, idempotency checks, solution targeting, offline availability, and change tracking.
• Does not remove or simplify any tables, columns, or relationships to achieve the improvement.

The optimization affects provisioning time only. The resulting Dataverse schema remains functionally equivalent.

Benchmark scope

Both runs used the same:

• Dataverse environment
• Default solution and publisher
• 8 new tables
• 10 one-to-many lookup relationships
• Column types, including text, Choice, Yes/No, Date/Time, Decimal, Memo, and Image
• Dependency and creation order
• Offline and change-tracking configuration

Fresh logical names were used for the optimized run to avoid idempotent skips and deleted-table tombstones.

Results

Metric	Baseline	Optimized	Improvement
Tables created	8	8	Same scope
Relationships created	10	10	Same scope
Total metadata operations	18	18	Same scope
Total duration	5m 56.1s	4m 16.8s	1m 39.2s faster
Average time per operation	19.8s	14.3s	5.5s faster
Observed performance improvement	—	—	27.9%
Throughput increase	—	—	1.39×

All 8 optimized-run tables were queried after creation and confirmed to exist in Dataverse.

Throttling-adjusted comparison

The baseline encountered one 30-second Dataverse throttling delay. The optimized run did not show a comparable delay.

After excluding that delay:

Metric	Result
Adjusted baseline	5m 26.1s
Optimized run	4m 16.8s
Adjusted saving	1m 9.2s
Adjusted improvement	21.2%

This indicates that the result is not explained only by the throttling event, although Dataverse service variability can affect individual runs.

Leadership impact

The observed improvement reduces developer waiting time during:

• Initial application provisioning
• Dataverse data-model iteration
• Automated environment setup
• Benchmark and test-environment creation
• Repeated development and CI workflows

Projected savings at the observed rate:

Provisioning runs	Approximate time saved
10	16m 32s
50	1h 22m 42s
100	2h 45m 25s

Safety and compatibility

The performance improvement does not trade away correctness:

• Dataverse metadata writes remain sequential.
• Parent tables are created before dependent tables.
• Relationships are created only after both tables are available.
• All artifacts remain targeted to the expected solution.
• Tables remain enabled for offline use and change tracking.
• Choice, lookup, image, date, and decimal metadata remain unchanged.
• No existing tables or data were deleted or modified.

Validation

• 8 of 8 tables created successfully.
• 10 of 10 lookup relationships created successfully.
• All created tables returned successful metadata queries.
• No sample records were inserted as part of the benchmark.
• No table-creation failures were observed in the optimized run.

Conclusion

The initial like-for-like benchmark shows that the optimized Dataverse provisioning workflow is approximately 28% faster, or approximately 21% faster after accounting for baseline throttling.

This is a meaningful improvement for developer productivity and automated provisioning while retaining equivalent schema output and Dataverse safety controls.

Because this comparison currently represents one run per variant, the result should be treated as an initial performance signal. Repeated runs should use the median and slowest execution times for a production performance commitment

Cut wall-clock time out of the mobile-app Dataverse workflow, and add a fail-closed reconciliation step before any metadata write.

Three independent speedups:

1. Auth short-circuit — stop spawning az account show on every Dataverse call.
2. Batched metadata reads — replace 2N per-table discovery requests with one filtered query.
3. Create-path shape — one POST per new table with all ordinary columns inline; one column snapshot per existing table.

1. Auth Short-Circuit

getAuthToken() built its tenant candidates as an array literal:

const tenantCandidates = [
  process.env.POWER_PLATFORM_TENANT_ID,
  process.env.DATAVERSE_TENANT_ID,
  await getDataverseTenantFromChallenge(resourceUrl),  // HTTPS probe
  getAzAccountTenantId(),                              // az account show
].filter(Boolean);

A JavaScript array literal evaluates every element eagerly, before .filter() runs. So the challenge probe and az account show executed on every single call, even when an env var already supplied the tenant and would have won the loop.

Measured on a warm macOS box:

Operation	Time
az account show	4.857s
az account get-access-token (warm)	0.349s
Node startup	0.107s

Candidates are now produced lazily and the loop stops at the first token. Preference order, de-duplication, and the unqualified-token fallback are unchanged — only redundant work is skipped. For reference, power-pages issues a single az account get-access-token per call and pays ~0.46s of overhead.

/add-dataverse now also exports POWER_PLATFORM_TENANT_ID once from the value resolve-environment.js already returns, so later calls short-circuit at the first candidate and skip the challenge probe entirely.

2. Batched Metadata Reads

Discovery previously issued one entity GET plus one attributes GET per planned table. Both collapse into a single documented query:

GET EntityDefinitions
  ?$select=...
  &$filter=LogicalName eq a or LogicalName eq b
  &$expand=Attributes($select=...)

This is the documented way to query multiple table definitions at once. A name present in value[] exists; a name absent does not — which is exactly the signal the reconciliation matrix needs. The expanded $select stays on base AttributeMetadata properties because one query cannot cast to a derived column type.

Two follow-on reductions fall out of that:

• Step 5a now reads the Step 4 snapshot instead of re-probing each create target. It only re-queries a name the batch did not cover, such as a rename candidate generated later.
• Step 6c verifies all created tables with one filtered query instead of one GET per table.

Tombstones and hidden collisions were never reliably visible to the pre-flight anyway — the POST-time collision rescue remains the real safety net, and is unchanged.

3. Create-Path Shape

New tables: one EntityDefinitions POST carrying the primary name plus every planned ordinary column, guarded by a five-point self-check (completeness, no inline lookup/customer/owner metadata, no deferred or server-owned columns, exactly one primary-name attribute, no duplicates). Lookups and relationships are a deliberate second pass.

Existing tables: compare against the cached snapshot, then POST only missing columns sequentially — replacing one preflight GET per planned column.

Metadata writes stay strictly sequential (Dataverse serializes them behind an exclusive lock) and no metadata $batch is introduced.

Reconciliation Before Writes

Every planned table and column is classified against live target metadata as reuse, extend, create, or block, using IsManaged, IsCustomizable, and CanCreateAttributes. A global barrier requires zero blocked items before Step 5 begins.

Missing standard or managed dependencies are block with remediation to install/import the owning solution — never recreated as a custom imitation. replace is deliberately not an automatic state, since Dataverse cannot change a column type in place.

Planning stays non-blocking: if discovery cannot run, the architect still drafts a plan, marks it Discovery skipped, flags entities Unverified, and returns DONE_WITH_CONCERNS. /add-dataverse re-queries live metadata and refuses to write what it cannot confirm.

No New Validator Script

An earlier revision added a validate-table-create-payload.js validator. It was removed: the expected-column list came from the same agent that produced the payload, so it could not independently prove plan alignment, and it compared only names — not types, formats, or lengths. The rule is now a self-check in the skill. dataverse-request.js remains the only script in that path.

Estimated Impact

For a model with ~7 planned tables (5 new, 2 extended) and 6 lookups, Dataverse calls drop from roughly 45 to 21. Combined with removing az account show from every call, per-call overhead falls from ~5.6s to ~0.5s.

Derived from the measured per-operation costs above rather than an end-to-end benchmark, that is on the order of 3–4 minutes of overhead removed per data-model run. The column-inlining change additionally removes up to C-1 writes per new table when the agent would otherwise have created a table shell.

Files

• plugins/mobile-apps/scripts/lib/validation-helpers.js — lazy tenant resolution.
• plugins/mobile-apps/scripts/tests/auth-token-resolution.test.js — new regression tests.
• plugins/mobile-apps/skills/add-dataverse/SKILL.md — batched discovery, snapshot-based pre-flight, batched verify, reconciliation matrix, write barrier, create self-check, two-pass ordering.
• plugins/mobile-apps/agents/data-model-architect.md — batched target query, target-grounded classification, non-blocking discovery failures.

Validation

• node --test plugins/mobile-apps/scripts/tests/ — 19 passed
• Regression proven: reverting getAuthToken to the eager version makes the new no az account show test fail; the fix makes it pass.
• All 8 consumer scripts parse; module exports and the async getAuthToken signature unchanged.
• validate-skill-descriptions / ensure-skill-version-check / validate-keyword-case / validate-plugin-names / validate-legacy-compatibility / validate-telemetry-ikeys — all pass.
• git diff --check — clean.

No live Dataverse environment was mutated during validation.

Compatibility and Risk

• Auth: behavior-preserving — same candidate order, same de-duplication, same fallback. Multi-tenant users still receive tenant-targeted tokens. getAuthToken remains re-callable for 401 refresh (no memoization).
• Batched reads: uses documented OData syntax. If a $filter URL would grow impractically long, the skill splits it into a few queries — still far fewer than one per table.
• Mutation: more conservative — unverified or incompatible schema now blocks instead of proceeding.
• Planning: unchanged in availability; a plan is still produced when discovery fails, just marked unverified.
• Existing-table column creation is still one POST per missing column.
• Live-environment smoke validation is still recommended for new, extend, managed-dependency, and collision cases.

Rollback

Revert the commits on this branch. No persisted runtime format changes and no repository migration.

[Copilot]

Pull request overview

This pull request hardens and speeds up the mobile-apps Dataverse schema workflow by enforcing a fail-closed reconciliation phase before any metadata writes, and by ensuring new-table creation uses the fastest supported single-POST path with a validator guard.

Changes:

• Updated /add-dataverse skill guidance to require full target reconciliation (with a global “no writes if any block” barrier) and to enforce two-pass ordering (ordinary columns first, relationships second).
• Added validate-table-create-payload.js (+ tests) to validate that new-table create payloads contain exactly the planned inline non-lookup columns before POSTing.
• Updated the data-model-architect agent instructions to classify block on unknown/unavailable target metadata and to forbid automatic replacement/guessing.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
plugins/mobile-apps/skills/add-dataverse/SKILL.md	Documents stricter reconciliation, a hard global write barrier, inline-create enforcement, and two-pass schema writes.
plugins/mobile-apps/scripts/validate-table-create-payload.js	New CLI validator to fail fast on incomplete/incorrect inline column sets for new table creates.
plugins/mobile-apps/scripts/tests/validate-table-create-payload.test.js	Unit tests for the new validator, including @file paths with spaces.
plugins/mobile-apps/agents/data-model-architect.md	Updates planning guidance to be target-grounded and fail closed with explicit Block outcomes.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

plugins/mobile-apps/agents/data-model-architect.md:194

• The Step 6 introduction links to ${PLUGIN_ROOT}/skills/add-dataverse/references/data-architecture-reference.md, but that file does not exist in the mobile-apps plugin (only skills/add-dataverse/references/dataverse-reference.md exists). This makes the tiering instructions point to a dead reference.

## Step 6 — Build Dependency Tiers

plugins/mobile-apps/agents/data-model-architect.md:33

• This bullet says the agent only classifies schema as Reuse, Extend, Create, or Block, but later in Step 5 you also introduce Unverified as a first-class decision. The hard-rules list should match the actual decision set to avoid conflicting instructions.

- **No automatic replacement.** This agent classifies schema as `Reuse`, `Extend`, `Create`, or `Block`. Replacing an existing table/column requires a separately approved migration with dependency analysis and data movement; it is outside this workflow.

plugins/mobile-apps/skills/add-dataverse/SKILL.md:368

• The self-check says to count Attributes[] and compare it to the plan's total column count. If the plan lists lookups/relationships (or deferred calculated/rollup columns) alongside ordinary columns, this comparison will incorrectly fail even when the create payload is correct. The check should explicitly compare only against the planned ordinary (non-lookup, non-calculated/rollup, non-primary-id) columns plus the single primary-name attribute.

1. **Every planned ordinary column is present** — String, Memo, Integer, BigInt, Decimal, Money, DateTime, Boolean, Choice, MultiSelect Choice, Image, and File. Count `Attributes[]` and compare with the plan's column count for this table.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

plugins/mobile-apps/agents/data-model-architect.md:155

• The definitions of Block vs Unverified are internally inconsistent: Block currently includes "target metadata is unavailable", but Unverified also includes "a non-200/404 response". This ambiguity makes it unclear whether transient metadata query failures should halt planning (Block) or produce a draft plan (Unverified) as described elsewhere in this doc.

- **Reuse** — existing table fits as-is (no schema changes needed)
- **Extend** — existing table is the right concept, all same-name columns are compatible, missing columns are custom additions, and both `IsCustomizable.Value` and `CanCreateAttributes.Value` permit extension
- **Create** — the planned item is explicitly a new custom table, the exact logical name returns 404, the publisher prefix is verified, and all required-existing dependencies are present
- **Block** — target metadata is unavailable; a standard/managed/required-existing dependency is missing; the table cannot accept attributes; or any same-name table/column is incompatible
- **Unverified** — discovery could not run for this entity (Step 1/2 failure or a non-200/404 response). Record the intended decision plus the reason; `/add-dataverse` re-checks it before any write

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

plugins/mobile-apps/scripts/tests/auth-token-resolution.test.js:64

• The fake Azure CLI shim is written only as an az file. On Windows, execFileSync('az', ...) will typically resolve via PATHEXT to az.cmd/az.exe, so this test will not reliably intercept calls (and may fail or invoke the real Azure CLI if installed). Add an az.cmd wrapper in the temp dir that forwards to the fake script so the tests run cross-platform.

function makeFakeAz(t) {
  const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'fake-az-'));
  t.after(() => fs.rmSync(dir, { recursive: true, force: true }));
  const azPath = path.join(dir, 'az');
  fs.writeFileSync(azPath, FAKE_AZ, { mode: 0o755 });
  return { dir, logPath: path.join(dir, 'az.log') };

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

plugins/mobile-apps/scripts/tests/auth-token-resolution.test.js:64

• The fake az executable is written as a Unix shebang script named az. On Windows, execFileSync('az', ...) (used by getAuthToken) typically won’t execute that file, so these tests will fail even though the production code is fine. Add an az.cmd shim in the temp dir that runs the script via the current Node binary so command resolution works cross-platform.

function makeFakeAz(t) {
  const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'fake-az-'));
  t.after(() => fs.rmSync(dir, { recursive: true, force: true }));
  const azPath = path.join(dir, 'az');
  fs.writeFileSync(azPath, FAKE_AZ, { mode: 0o755 });
  return { dir, logPath: path.join(dir, 'az.log') };
}

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

plugins/mobile-apps/scripts/tests/auth-token-resolution.test.js:64

• The auth-token-resolution tests create a fake az binary as a no-extension executable (<tmp>/az). On Windows, execFileSync('az', ...) resolves commands via PATHEXT (e.g. az.cmd / az.exe), so this fake binary won’t be discovered and the test may accidentally invoke the real Azure CLI (making the test non-deterministic / requiring auth). Add a Windows-compatible wrapper (e.g. az.cmd) in the temp dir that forwards to the Node script so the tests are cross-platform and hermetic.

  const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'fake-az-'));
  t.after(() => fs.rmSync(dir, { recursive: true, force: true }));
  const azPath = path.join(dir, 'az');
  fs.writeFileSync(azPath, FAKE_AZ, { mode: 0o755 });
  return { dir, logPath: path.join(dir, 'az.log') };

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

plugins/mobile-apps/scripts/tests/auth-token-resolution.test.js:64

• The fake az shim is written only as an az file. On Windows, execFileSync('az', ...) resolves via PATHEXT (e.g., az.cmd/az.exe) and will not execute a bare az script, so these tests will fail on win32. Write an az.cmd wrapper (and a .js payload) when process.platform === 'win32' so PATH interception works cross-platform.

function makeFakeAz(t) {
  const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'fake-az-'));
  t.after(() => fs.rmSync(dir, { recursive: true, force: true }));
  const azPath = path.join(dir, 'az');
  fs.writeFileSync(azPath, FAKE_AZ, { mode: 0o755 });
  return { dir, logPath: path.join(dir, 'az.log') };
}

[Ameya Apte (ameyaapte1)]

Is block state necesary, what happens if user hits this block scenario?

[Ameya Apte (ameyaapte1)]

block state can be removed?

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

plugins/mobile-apps/skills/add-dataverse/SKILL.md:215

• The reconciliation matrix currently treats a failed metadata read as create/"Proceed". If discovery cannot confirm the target state, creating is not a safe default; it should be deferred/unverified so the workflow doesn't mutate schema based on unknown state.

| Batched query failed (non-2xx) after retry and per-table split | `create` | unknown | Proceed; Step 5a's POST-time collision rescue resolves it by extend or rename. |

plugins/mobile-apps/skills/add-dataverse/SKILL.md:48

• Step 2 only calls out reuse/extend/create/adapt/defer decisions, but the data-model-architect instructions in this PR introduce block/unverified outcomes. If the plan contains those, /add-dataverse needs to carry them forward explicitly (and treat them as non-mutating) to avoid silently proceeding with an unresolved state.

**If present:** read the `## Data Model` section. Extract:
- The target reconciliation table (`reuse` / `extend` / `create` / `adapt` / `defer` decisions and evidence)
- The Mermaid ER diagram (informational)
- The "Creation Order" tier list

Carry forward any `adapt` (auto-renamed) and `defer` (out-of-scope this run) decisions with their recorded reasons, and apply the alias map to every name you use. A data-modelling conflict never halts this skill — it resolves to `adapt` or `defer` and is reported in Step 9.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

plugins/mobile-apps/skills/add-dataverse/SKILL.md:215

• Step 4 says any name still unreadable after retry+split is classified defer (and the PR description emphasizes fail-closed reconciliation), but the matrix row for a batched-query failure currently sets the table decision to create and proceeds. That contradicts the preceding guidance and can lead to unverified metadata writes when discovery is unavailable.

| Batched query failed (non-2xx) after retry and per-table split | `create` | unknown | Proceed; Step 5a's POST-time collision rescue resolves it by extend or rename. |

plugins/mobile-apps/agents/data-model-architect.md:150

• This decision list omits Defer, but the surrounding text and examples use Defer for missing standard/managed/required-existing dependencies. As written, readers have to infer whether Defer is a valid state or whether Block/Unverified should be used instead, which can cause inconsistent plans.

- **Reuse** — existing table fits as-is (no schema changes needed)
- **Extend** — existing table is the right concept, all same-name columns are compatible, missing columns are custom additions, and both `IsCustomizable.Value` and `CanCreateAttributes.Value` permit extension
- **Create** — the planned item is explicitly a new custom table, the exact logical name returns 404, the publisher prefix is verified, and all required-existing dependencies are present
- **Block** — target metadata is unavailable; a standard/managed/required-existing dependency is missing; the table cannot accept attributes; or any same-name table/column is incompatible
- **Unverified** — discovery could not run for this entity (Step 1/2 failure or a non-200/404 response). Record the intended decision plus the reason; `/add-dataverse` re-checks it before any write

[Ameya Apte (ameyaapte1)]

Do we need so many states, Reuse, Extend, Create sounds enough