Repository navigation
feat(doqa): bump to 4.3.1-box with new queue topology - #110
Merged
Merged
Conversation
Vendor box release 4_3_1 (configs_4_3_1.zip): - Image pins: backend 4.3.1-box; frontend, autotest-parser, statistic, notify, telegram-bot, llm 4.3.0-box - autotest-result-parser removed upstream; the chart drops the autotest-result-parser Deployment and the autotestResultParser values section - New backend-image workers: autotests-worker (declares the eight RabbitMQ queues, runs AUTOTESTS_WORKER_PROCESSES parallel autotests:work consumers) and single-replica llm-bulk-queue (queue:work rabbitmq --queue=llm-bulk --timeout=3660) - queue command now consumes default,search,requirements,autotests-domain,outbound-webhooks,import - New ConfigMap keys: MINIO_USE_PATH_STYLE=true, QUEUE_BATCHING_CONNECTION=pgsql, AUTOTEST_PARSER_SERVICE_URL computed from the autotest-parser Service name - New generated secret INTERNAL_SYSTEM_KEY in the app Secret, wired into backend/queue/cron/workers via backendSecretEnv and explicitly into autotest-parser, which also gains explicit MINIO_* credentials - NetworkPolicy: result-parser rules replaced by the new workers; autotest-parser and autotests-worker allowed to reach MinIO Chart version 0.6.3 -> 0.7.0 (autotestResultParser removal is breaking-ish for user values).
Review feedback on the 4.3.1 bump: - allow-nginx-to-autotest-parser renamed to allow-app-to-autotest-parser; backend (AUTOTEST_PARSER_SERVICE_URL), queue, and autotests-worker can now reach autotest-parser:8000 alongside nginx - allow-app-to-llm: llm-bulk-queue added (it consumes llm-bulk jobs that call the LLM API) - allow-app-to-websocket / -notification / -statistic: autotests-worker and llm-bulk-queue added alongside queue, which carried this traffic on 0.6.x - minio ingress rule: llm-bulk-queue added (receives MINIO_* env via backendSecretEnv) - scripts/audit-doqa-images.sh: drop the removed doqa-autotest-result-parser repository - README: note that autotests-worker redeclares the 8 queues with durable/no-args params (PRECONDITION_FAILED only for hand-redeclared legacy queues)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Vendor 4_3_1 changes (configs_4_2_2 → configs_4_3_1)
Structural update, not a tag bump. nginx configs and the Dockerfile are unchanged.
Image pins (
.env.install):doqa-autotest-result-parserremoved upstream (no image pin, no compose service)New
.env.installvariables:MINIO_USE_PATH_STYLE=true,QUEUE_BATCHING_CONNECTION=pgsql→<release>-envConfigMapAUTOTEST_PARSER_SERVICE_URL→ ConfigMap, computed from the autotest-parser Service name (same pattern asPUSHER_APP_HOST)INTERNAL_SYSTEM_KEY=GENERATE_SECRET_KEY→ new generated keyinternal-system-keyin the<release>-app-secretsSecret (lookup-stable like the others). Vendor's env_file delivers it to backend/queue/cron and the new workers — the chart reaches all of them viadoqa.backendSecretEnv— plus an explicit env entry on autotest-parser, matching vendor's explicit list.Compose topology:
doqa_queuenow consumesdefault,search,requirements,autotests-domain,outbound-webhooks,import(llm-bulk moved out)doqa_autotests→templates/autotests-worker.yaml: declares the 8 RabbitMQ queues, then runsautotestsWorker.workerProcesses(default 3, vendorAUTOTESTS_WORKER_PROCESSES) parallelautotests:workconsumers withautotestsWorker.prefetch(default 10, vendorAUTOTESTS_PREFETCH); replicas default 3 (vendor scales withQUEUE_WORKERS)doqa_llm_bulk_queue→templates/llm-bulk-queue.yaml: single replica,queue:work rabbitmq --queue=llm-bulk --sleep=3 --tries=1 --timeout=3660doqa_autotest_parsergains explicitMINIO_BUCKET/ENDPOINT/KEY/REGION/SECRET(MinIO creds from the minio Secret) andINTERNAL_SYSTEM_KEYwait-for-rabbitmqinitContainer the chart already runs for queue/migrations (disable viaautotestsWorker.waitForRabbitmq/llmBulkQueue.waitForRabbitmq). autotest-parser keeps relying on restart policy, as before.Breaking / upgrade notes
autotestResultParservalues section and theautotest-result-parserDeployment are gone. RemoveautotestResultParserfrom your values files. (values.schema.jsondoes not setadditionalProperties: false, so leftover keys still validate — they are simply ignored.)secrets.create=falseor an externalsecrets.app, the Secret must now also containinternal-system-key(render fails otherwise with arequirederror / pods CrashLoop on the missing key). Chart-managed secrets generate it automatically on upgrade.Image contract audit (
scripts/audit-doqa-images.sh)/bin/sh -c "nginx -g 'daemon off;'") to exec form (nginx -g 'daemon off;'). The chart does not override the frontend command, so this is compatible. OKTG_PROXY=with empty default; vendor.env.installdoes not set it, so the chart does not wire it. OKValidation
scripts/audit-doqa-images.sh— reviewed above; benign diffs onlyhelm template test-doqa charts/doqa -f charts/doqa/ci/test-values.yaml— renders, 16 Deployments; also rendered withnetworkPolicy.enabled=true+telegramBot.enabled=trueand withci/external-values.yamlscripts/check-helm-docs.sh,scripts/helm-lint-all.sh,scripts/helm-template-all.sh— passpre-commit run --all-files— all hooks pass (incl. doqa portability)ct lint --config .github/ct.yaml --target-branch main— pass (version increment ok, both ci values files lint clean)xray-health-exporteruntouched.