Skip to content

feat: add users migration task - #1716

Merged
davidgamez merged 10 commits into
mainfrom
migrate_users
Jun 3, 2026
Merged

davidgamez merged 10 commits into
mainfrom
migrate_users

Conversation

@davidgamez

@davidgamez davidgamez commented Jun 1, 2026 •

Copy link
Copy Markdown
Member

Summary:

This PR introduces the user migration from Firebase+Datastore to SQL(isolated schema). The migration insert-only, existing rows are never modified. Brevo is the source of truth for is_registered_to_receive_api_announcements.

Expected behavior:

Users are migrated using the task, except guest users.

Testing tips:

internal team: This can be tested in DEV using retool.

From our AI friend

This pull request introduces support for synchronizing user subscription status with the Brevo (Sendinblue) API announcements mailing list, and adds a new task for migrating Firebase users into the users.app_user PostgreSQL table. It also updates workflow and configuration files to support the new environment variables and dependencies required for this integration.

Brevo integration and user migration:

  • Added a new shared helper brevo.py to encapsulate Brevo API logic, including checking a user's subscription status for a specific list (BrevoSubscriptionStatus) and handling API errors robustly.
  • Introduced the migrate_firebase_users task in the tasks executor, which migrates Firebase Auth users into the users database, synchronizes their API announcements subscription status from Brevo, and documents its usage and parameters in the README. [1] [2] [3]

Workflow and environment configuration:

  • Updated GitHub Actions workflows (api-deployer.yml, api-dev.yml, api-qa.yml, api-prod.yml) to add the BREVO_API_ANNOUNCEMENTS_LIST_ID input/variable, ensuring it is available in deployment environments and passed to backend scripts. [1] [2] [3] [4] [5] [6]
  • Updated function_config.json for the tasks executor to include BREVO_API_KEY, BREVO_API_ANNOUNCEMENTS_LIST_ID, and USERS_DATABASE_URL as environment variables, and to include the new helper and database folders. [1] [2]

Dependency management:

  • Added the Brevo (Sendinblue) Python SDK (sib-api-v3-sdk), Firebase Admin SDK, and Google Cloud Datastore to the requirements for both the main API and the tasks executor. [1] [2]

Other improvements:

  • Ensured that the __wrapped__ attribute is preserved in the users_database.py wrapper function for better compatibility with introspection tools.
  • Minor documentation formatting and whitespace improvements in the OpenAPI YAML spec. [1] [2] [3] [4]
    Please make sure these boxes are checked before submitting your pull request - thanks!
  • Run the unit tests with ./scripts/api-tests.sh to make sure you didn't break anything
  • Add or update any needed documentation to the repo
  • Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
  • Linked all relevant issues
  • Include screenshot(s) showing how this pull request works and fixes the issue(s)

@davidgamez davidgamez linked an issue Jun 3, 2026 that may be closed by this pull request
@davidgamez
davidgamez marked this pull request as ready for review June 3, 2026 13:19
@cka-y
cka-y requested a review from Copilot June 3, 2026 15:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds an insert-only migration task to copy non-guest Firebase Auth users into the isolated users.app_user PostgreSQL schema, including optional enrichment from Datastore profile fields and subscription status from Brevo. It also updates infra/workflows/config to provide the required environment variables and permissions, and adds a migrated_at column to track which rows were created by the migration.

Changes:

  • Added migrate_firebase_users task (plus unit tests) to insert missing users into users.app_user, pulling profile fields from Datastore and subscription status from Brevo.
  • Extended users schema with migrated_at and wired the migration into tasks executor runtime configuration (env vars, dependencies, IAM).
  • Added Brevo helper utilities and deployment/config plumbing for BREVO_API_ANNOUNCEMENTS_LIST_ID.

Reviewed changes

Copilot reviewed 22 out of 23 changed files in this pull request and generated 8 comments.

Show a summary per file
File Description
scripts/init-local-folder.sh Adds user-service OpenAPI stub generation to local init script.
liquibase/changes_user/feat_1684.sql Adds migrated_at column to app_user for migration tracking.
liquibase/changelog_user.xml Includes the new Liquibase change set for migrated_at.
infra/vars.tfvars.rename_me Adds brevo_api_announcements_list_id template variable.
infra/vars.tf Declares Terraform variable for Brevo announcements list id.
infra/main.tf Passes Brevo list id variable into functions-python module.
infra/functions-python/vars.tf Declares Brevo list id variable for functions module.
infra/functions-python/main.tf Exposes BREVO_API_ANNOUNCEMENTS_LIST_ID to tasks executor; adds IAM roles for Firebase/Datastore access.
functions-python/tasks_executor/tests/tasks/users/test_migrate_firebase_users.py Unit tests covering insert-only behavior and Brevo/Datastore handling.
functions-python/tasks_executor/src/tasks/users/migrate_firebase_users.py Implements the migration task and handler + parameter parsing.
functions-python/tasks_executor/src/tasks/users/init.py Package marker for users tasks.
functions-python/tasks_executor/src/main.py Registers the new task in the tasks executor registry.
functions-python/tasks_executor/requirements.txt Adds Firebase Admin, Datastore, and Brevo SDK dependencies.
functions-python/tasks_executor/README.md Documents migrate_firebase_users task usage and env vars.
functions-python/tasks_executor/function_config.json Adds required env/secret vars and includes users_database_gen folder.
functions-python/helpers/requirements.txt Adds Brevo SDK dependency to shared helpers.
docs/OperationsAPI.yaml Whitespace-only OpenAPI YAML formatting changes.
api/src/shared/database/users_database.py Preserves __wrapped__ for the users DB session decorator wrapper.
api/src/shared/common/brevo.py Adds shared Brevo helper for contact subscription status.
.github/workflows/api-qa.yml Injects BREVO_API_ANNOUNCEMENTS_LIST_ID into QA workflow env.
.github/workflows/api-prod.yml Injects BREVO_API_ANNOUNCEMENTS_LIST_ID into prod workflow env.
.github/workflows/api-dev.yml Injects BREVO_API_ANNOUNCEMENTS_LIST_ID into dev workflow env.
.github/workflows/api-deployer.yml Adds deployer input and variable replacement wiring for BREVO_API_ANNOUNCEMENTS_LIST_ID.

Comment thread scripts/init-local-folder.sh Outdated
Comment thread functions-python/tasks_executor/src/main.py Outdated
Comment thread functions-python/tasks_executor/src/tasks/users/migrate_firebase_users.py Outdated
Comment thread functions-python/tasks_executor/src/tasks/users/migrate_firebase_users.py Outdated
Comment thread functions-python/tasks_executor/README.md Outdated
Comment thread api/src/shared/common/brevo.py
Comment thread api/src/shared/common/brevo.py Outdated

@cka-y cka-y left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM added a few nitpicks and questions for my understanding

Comment thread api/src/shared/common/brevo.py
Comment thread docs/OperationsAPI.yaml Outdated




Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] reverse changes in this file as it adds noise to the PR

registration_completed_at=_parse_datastore_timestamp(
doc_data.get("registrationCompletionTime")
),
migrated_at=now,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit]

Suggested change
migrated_at=now,
migrated_at=datetime.now(timezone.utc),

The variable now is not used elsewhere and is overwritten for each user insertion

id=user_record.uid,
email=user_record.email,
email_verified=user_record.email_verified,
created_at=created_at,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit]

Suggested change
created_at=created_at,
created_at=_ms_to_datetime(
user_record.user_metadata.creation_timestamp
if user_record.user_metadata
else None
),

Or move the created_at variable closer to the usage because we compute it (although inexpensive) even if we don't use it (in case of a dry_run for example)

Comment thread functions-python/helpers/requirements.txt Outdated
@davidgamez
davidgamez merged commit 1cc927c into main Jun 3, 2026
28 checks passed
@davidgamez
davidgamez deleted the migrate_users branch June 3, 2026 17:54
@cka-y cka-y mentioned this pull request Jun 10, 2026
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[user-service] migrate_firebase_users task in tasks_executor

3 participants