Repository navigation
feat: add users migration task - #1716
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds an insert-only migration task to copy non-guest Firebase Auth users into the isolated users.app_user PostgreSQL schema, including optional enrichment from Datastore profile fields and subscription status from Brevo. It also updates infra/workflows/config to provide the required environment variables and permissions, and adds a migrated_at column to track which rows were created by the migration.
Changes:
- Added
migrate_firebase_userstask (plus unit tests) to insert missing users intousers.app_user, pulling profile fields from Datastore and subscription status from Brevo. - Extended users schema with
migrated_atand wired the migration into tasks executor runtime configuration (env vars, dependencies, IAM). - Added Brevo helper utilities and deployment/config plumbing for
BREVO_API_ANNOUNCEMENTS_LIST_ID.
Reviewed changes
Copilot reviewed 22 out of 23 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/init-local-folder.sh | Adds user-service OpenAPI stub generation to local init script. |
| liquibase/changes_user/feat_1684.sql | Adds migrated_at column to app_user for migration tracking. |
| liquibase/changelog_user.xml | Includes the new Liquibase change set for migrated_at. |
| infra/vars.tfvars.rename_me | Adds brevo_api_announcements_list_id template variable. |
| infra/vars.tf | Declares Terraform variable for Brevo announcements list id. |
| infra/main.tf | Passes Brevo list id variable into functions-python module. |
| infra/functions-python/vars.tf | Declares Brevo list id variable for functions module. |
| infra/functions-python/main.tf | Exposes BREVO_API_ANNOUNCEMENTS_LIST_ID to tasks executor; adds IAM roles for Firebase/Datastore access. |
| functions-python/tasks_executor/tests/tasks/users/test_migrate_firebase_users.py | Unit tests covering insert-only behavior and Brevo/Datastore handling. |
| functions-python/tasks_executor/src/tasks/users/migrate_firebase_users.py | Implements the migration task and handler + parameter parsing. |
| functions-python/tasks_executor/src/tasks/users/init.py | Package marker for users tasks. |
| functions-python/tasks_executor/src/main.py | Registers the new task in the tasks executor registry. |
| functions-python/tasks_executor/requirements.txt | Adds Firebase Admin, Datastore, and Brevo SDK dependencies. |
| functions-python/tasks_executor/README.md | Documents migrate_firebase_users task usage and env vars. |
| functions-python/tasks_executor/function_config.json | Adds required env/secret vars and includes users_database_gen folder. |
| functions-python/helpers/requirements.txt | Adds Brevo SDK dependency to shared helpers. |
| docs/OperationsAPI.yaml | Whitespace-only OpenAPI YAML formatting changes. |
| api/src/shared/database/users_database.py | Preserves __wrapped__ for the users DB session decorator wrapper. |
| api/src/shared/common/brevo.py | Adds shared Brevo helper for contact subscription status. |
| .github/workflows/api-qa.yml | Injects BREVO_API_ANNOUNCEMENTS_LIST_ID into QA workflow env. |
| .github/workflows/api-prod.yml | Injects BREVO_API_ANNOUNCEMENTS_LIST_ID into prod workflow env. |
| .github/workflows/api-dev.yml | Injects BREVO_API_ANNOUNCEMENTS_LIST_ID into dev workflow env. |
| .github/workflows/api-deployer.yml | Adds deployer input and variable replacement wiring for BREVO_API_ANNOUNCEMENTS_LIST_ID. |
cka-y
left a comment
There was a problem hiding this comment.
LGTM added a few nitpicks and questions for my understanding
|
|
||
|
|
||
|
|
||
|
|
There was a problem hiding this comment.
[nitpick] reverse changes in this file as it adds noise to the PR
| registration_completed_at=_parse_datastore_timestamp( | ||
| doc_data.get("registrationCompletionTime") | ||
| ), | ||
| migrated_at=now, |
There was a problem hiding this comment.
[nit]
| migrated_at=now, | |
| migrated_at=datetime.now(timezone.utc), |
The variable now is not used elsewhere and is overwritten for each user insertion
| id=user_record.uid, | ||
| email=user_record.email, | ||
| email_verified=user_record.email_verified, | ||
| created_at=created_at, |
There was a problem hiding this comment.
[nit]
| created_at=created_at, | |
| created_at=_ms_to_datetime( | |
| user_record.user_metadata.creation_timestamp | |
| if user_record.user_metadata | |
| else None | |
| ), |
Or move the created_at variable closer to the usage because we compute it (although inexpensive) even if we don't use it (in case of a dry_run for example)
Summary:
This PR introduces the user migration from Firebase+Datastore to SQL(isolated schema). The migration insert-only, existing rows are never modified. Brevo is the source of truth for
is_registered_to_receive_api_announcements.Expected behavior:
Users are migrated using the task, except guest users.
Testing tips:
internal team: This can be tested in DEV using retool.
From our AI friend
This pull request introduces support for synchronizing user subscription status with the Brevo (Sendinblue) API announcements mailing list, and adds a new task for migrating Firebase users into the
users.app_userPostgreSQL table. It also updates workflow and configuration files to support the new environment variables and dependencies required for this integration.Brevo integration and user migration:
brevo.pyto encapsulate Brevo API logic, including checking a user's subscription status for a specific list (BrevoSubscriptionStatus) and handling API errors robustly.migrate_firebase_userstask in the tasks executor, which migrates Firebase Auth users into the users database, synchronizes their API announcements subscription status from Brevo, and documents its usage and parameters in the README. [1] [2] [3]Workflow and environment configuration:
api-deployer.yml,api-dev.yml,api-qa.yml,api-prod.yml) to add theBREVO_API_ANNOUNCEMENTS_LIST_IDinput/variable, ensuring it is available in deployment environments and passed to backend scripts. [1] [2] [3] [4] [5] [6]function_config.jsonfor the tasks executor to includeBREVO_API_KEY,BREVO_API_ANNOUNCEMENTS_LIST_ID, andUSERS_DATABASE_URLas environment variables, and to include the new helper and database folders. [1] [2]Dependency management:
sib-api-v3-sdk), Firebase Admin SDK, and Google Cloud Datastore to the requirements for both the main API and the tasks executor. [1] [2]Other improvements:
__wrapped__attribute is preserved in theusers_database.pywrapper function for better compatibility with introspection tools.Please make sure these boxes are checked before submitting your pull request - thanks!
./scripts/api-tests.shto make sure you didn't break anything