Make indirect write job/temp BigQuery options optional with clear defaults

[fqshopify]

Context

The indirect BigQuery sink currently requires users to configure the BigQuery project used to submit load/copy jobs:

'write.indirect.bigquery-job-project' = '<bq-job-project>'

Was raised in #319 (comment) whether this should be optional, and whether we can default it to the current/destination project when not provided.

Proposal

Consider making write.indirect.bigquery-job-project optional. If omitted, the connector would submit BigQuery load/copy jobs in the destination/current project, i.e. the project from the standard BigQuery connection options / Table API project.

Open questions

If we add a default for write.indirect.bigquery-job-project, we should decide whether related temporary BigQuery options should also get defaults:

1. Should write.indirect.bigquery-job-project default to the destination project?
2. Should write.indirect.temp-bigquery-project also default to the destination project?
3. Should write.indirect.temp-bigquery-dataset default to the destination dataset?

I’m not strongly opinionated, but I have some concerns around defaulting the temp dataset to the destination dataset:

• It would mix connector-managed temporary tables with user/application tables.
• The docs currently recommend configuring tableExpirationMs on the temp dataset as a safety net for best-effort cleanup; users may not want that policy on their destination dataset.
• It may obscure the IAM/resource-boundary distinction between:
  • the project where BigQuery jobs run,
  • the project/dataset where temp tables are created,
  • and the final destination table.

Possible options

Option A: Only default bigquery-job-project

• If omitted, use the destination project for BigQuery load/copy jobs.
• Keep temp-bigquery-project and temp-bigquery-dataset required.
• Lowest-risk change; reduces one required option without changing temp-table placement.

Option B: Default bigquery-job-project and temp-bigquery-project

• If omitted, both default to the destination project.
• Keep temp-bigquery-dataset required.
• Still keeps temp tables in an explicitly chosen dataset.

Option C: Default all three

• bigquery-job-project defaults to destination project.
• temp-bigquery-project defaults to destination project.
• temp-bigquery-dataset defaults to destination dataset.
• Most convenient, but potentially surprising because temp tables may be created alongside final/user tables.

Option D: No defaults

With defaults, users can be unaware of the full implications of their decisions from a storage/compute-separation standpoint leading to surprises later. Leaving these as required arguments is a forcing function for them to understand/acknowledge the implications to costs and quotas.

Acceptance criteria

• Decide which defaulting behavior we want.
• Update DataStream API config builder behavior.
• Update Table/SQL API option handling.
• Update validation to allow omitted values where defaults exist.
• Add tests covering omitted/defaulted values.
• Update README/docs and examples to describe the defaults clearly.