Skip to content

Consolidate broker CI and audit fixes #1122

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mpelteshki wants to merge 10 commits into from
Closed

Consolidate broker CI and audit fixes #1122

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
79ad0ed
Redact Coinbase private key example
mpelteshki Jul 10, 2026
fc8c36b
Add Alpaca live broker strategy CI
mpelteshki Jul 10, 2026
d562b2f
Add broker live strategy coverage
mpelteshki Jul 10, 2026
7122d23
Harden broker live strategy CI
mpelteshki Jul 10, 2026
eba56f5
Harden broker data and CI behavior
mpelteshki Jul 10, 2026
53d232b
Allow one native bar of strict lag
mpelteshki Jul 10, 2026
8e4fcea
Harden live broker workflow isolation
mpelteshki Jul 10, 2026
77871c6
Preserve native bar validation performance
mpelteshki Jul 10, 2026
8e07ae7
Preserve sparse option history semantics
mpelteshki Jul 10, 2026
230b903
Separate native bar lag tolerances
mpelteshki Jul 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
116 changes: 116 additions & 0 deletions .github/workflows/alpaca-live-broker.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
name: Alpaca Live Broker

on:
workflow_dispatch:
push:
branches:
- dev
paths:
- "lumibot/**"
- "tests/test_alpaca_live_broker_apitest.py"
- "tests/test_agent_runtime_remote_mcp.py"
- "tests/test_agent_runtime_mcp_transports.py"
- "tests/test_agent_runtime_provider_keys.py"
- "tests/test_agent_runtime_errors.py"
- "tests/test_agent_tool_permissions.py"
- "tests/test_agent_alpaca_news_builtin.py"
- "tests/test_agent_alpaca_news_live_apitest.py"
- "tests/backtest/test_agent_runtime_backtest.py"
- "tests/backtest/test_ai_committee_builtin_tools_backtest.py"
- "requirements*.txt"
- "pyproject.toml"
- "setup.py"
- ".github/workflows/alpaca-live-broker.yml"
pull_request:
branches: [dev, main]
paths:
- "lumibot/**"
- "tests/test_alpaca_live_broker_apitest.py"
- "tests/test_agent_runtime_remote_mcp.py"
- "tests/test_agent_runtime_mcp_transports.py"
- "tests/test_agent_runtime_provider_keys.py"
- "tests/test_agent_runtime_errors.py"
- "tests/test_agent_tool_permissions.py"
- "tests/test_agent_alpaca_news_builtin.py"
- "tests/test_agent_alpaca_news_live_apitest.py"
- "tests/backtest/test_agent_runtime_backtest.py"
- "tests/backtest/test_ai_committee_builtin_tools_backtest.py"
- "requirements*.txt"
- "pyproject.toml"
- "setup.py"
- ".github/workflows/alpaca-live-broker.yml"

permissions:
contents: read

env:
AIOHTTP_NO_EXTENSIONS: 1
BACKTESTING_DATA_SOURCE: none
BACKTESTING_SHOW_PROGRESS_BAR: "false"
ALPACA_TEST_API_KEY: ${{ secrets.ALPACA_TEST_API_KEY }}
ALPACA_TEST_API_SECRET: ${{ secrets.ALPACA_TEST_API_SECRET }}
ALPACA_NEWS_API_KEY: ${{ secrets.ALPACA_NEWS_API_KEY || secrets.ALPACA_TEST_API_KEY }}
ALPACA_NEWS_API_SECRET: ${{ secrets.ALPACA_NEWS_API_SECRET || secrets.ALPACA_TEST_API_SECRET }}
Comment on lines +52 to +53

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep Alpaca news credentials as a pair

These independent fallbacks can combine ALPACA_NEWS_API_KEY with ALPACA_TEST_API_SECRET (or the reverse) when only one optional news secret is configured. The verification step only sees non-empty environment variables, so the live news test proceeds with a mismatched key/secret pair and fails authentication instead of falling back to the complete paper pair or reporting the missing news pair.

Useful? React with 👍 / 👎.


jobs:
alpaca-live-broker:
name: Alpaca live broker strategy
runs-on: blacksmith-2vcpu-ubuntu-2404
timeout-minutes: 55
if: >-
github.actor != 'dependabot[bot]' &&
(github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository)
concurrency:
group: alpaca-live-broker-${{ github.repository }}
cancel-in-progress: false
Comment on lines +63 to +65

@coderabbitai coderabbitai Bot Jul 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift

Cross-workflow paper-account concurrency risk.

Both alpaca-live-broker.yml (concurrency group alpaca-live-broker-*) and cicd.yaml's broker-live-strategy-tests job (concurrency group broker-live-strategy-tests-*-unit-tests) can trigger on PRs to dev/main. They use independent concurrency groups, so both can run simultaneously against the same Alpaca paper account, potentially interfering with each other's order lifecycle tests.

Consider a shared concurrency group (e.g., keyed on github.repository alone) across both workflows to serialize all paper-account access.

Also applies to: 284-286

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/alpaca-live-broker.yml around lines 63 - 65, Use a shared
concurrency group keyed only by github.repository for alpaca-live-broker.yml and
the broker-live-strategy-tests job in cicd.yaml, replacing their independent
group names so all paper-account tests serialize across workflows while
preserving cancel-in-progress: false.


steps:
- uses: actions/checkout@v4
with:
persist-credentials: false

- name: Set up Python 3.10
uses: actions/setup-python@v5
with:
python-version: "3.10"
cache: pip

- name: Install dependencies
run: |
set -euo pipefail
echo "Set AIOHTTP_NO_EXTENSIONS=$AIOHTTP_NO_EXTENSIONS so that aiohttp doesn't try to install C extensions"
python -m pip install --upgrade pip
pip install requests
pip install -r requirements_dev.txt

- name: Verify Alpaca paper credentials are available
run: |
set -euo pipefail
if [ -z "${ALPACA_TEST_API_KEY}" ] || [ -z "${ALPACA_TEST_API_SECRET}" ]; then
echo "::error::ALPACA_TEST_API_KEY and ALPACA_TEST_API_SECRET repository secrets are required for Alpaca live broker CI."
exit 1
fi
if [ -z "${ALPACA_NEWS_API_KEY}" ] || [ -z "${ALPACA_NEWS_API_SECRET}" ]; then
echo "::error::ALPACA_NEWS_API_KEY and ALPACA_NEWS_API_SECRET, or fallback ALPACA_TEST_API_KEY and ALPACA_TEST_API_SECRET, are required for Alpaca news live apitests."
exit 1
fi

- name: Run agent MCP and built-in action coverage
run: |
set -euo pipefail
timeout 1500 python -m pytest -q \
tests/test_agent_runtime_remote_mcp.py \
tests/test_agent_runtime_mcp_transports.py \
tests/test_agent_runtime_provider_keys.py \
tests/test_agent_runtime_errors.py \
tests/test_agent_tool_permissions.py \
tests/test_agent_alpaca_news_builtin.py \
tests/test_agent_alpaca_news_live_apitest.py \
tests/backtest/test_agent_runtime_backtest.py \
tests/backtest/test_ai_committee_builtin_tools_backtest.py

- name: Run real Alpaca broker strategy apitests
run: |
set -euo pipefail
timeout 1200 python -m pytest -q -m apitest --tb=short \
tests/test_alpaca_live_broker_apitest.py
99 changes: 84 additions & 15 deletions .github/workflows/cicd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,14 @@ name: LumiBot CI/CD

on:
# Full LumiBot CI is intentionally expensive: lint + 6 unit shards + 4 backtest
# shards. Run it when explicitly requested, for release/CI tags, or for main PRs.
# shards. Run it when explicitly requested, for release/CI tags, or for PRs to
# either long-lived branch.
push:
tags:
- "v*.*.*"
- "ci-*"
pull_request:
branches: [main]
branches: [main, dev]
paths:
- "lumibot/**"
- "tests/**"
Expand Down Expand Up @@ -54,10 +55,10 @@ jobs:
timeout-minutes: 15
environment: unit-tests
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
Comment thread
coderabbitai[bot] marked this conversation as resolved.

- name: Set up Python 3.10
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: "3.10"
cache: pip
Expand Down Expand Up @@ -99,10 +100,10 @@ jobs:
shard: [0, 1, 2, 3, 4, 5]

steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4

- name: Set up Python 3.10
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: "3.10"
cache: pip
Expand All @@ -123,7 +124,7 @@ jobs:

# CI target: keep PR checks fast and deterministic.
# Downloader + apitests are run separately / opt-in.
PYTEST_MARKERS='not apitest and not downloader'
PYTEST_MARKERS='not apitest and not downloader and not broker_strategy_live'
export PYTEST_MARKERS

echo "Shard ${SHARD_INDEX}/${SHARD_TOTAL} markers=${PYTEST_MARKERS}"
Expand Down Expand Up @@ -174,8 +175,10 @@ jobs:
)
PY

echo "Running $(wc -l shard_files.txt | awk '{print $1}') files"
timeout 1500 python -m pytest -m "${PYTEST_MARKERS}" --tb=short -q --durations=30 -x $(cat shard_files.txt)
mapfile -t shard_files < shard_files.txt
((${#shard_files[@]} > 0)) || { echo "No unit-test files selected"; exit 1; }
echo "Running ${#shard_files[@]} files"
timeout 1500 python -m pytest -m "${PYTEST_MARKERS}" --tb=short -q --durations=30 -x "${shard_files[@]}"

backtest-tests:
name: Backtest Tests (shard ${{ matrix.shard }}/4)
Expand All @@ -189,10 +192,10 @@ jobs:
shard: [0, 1, 2, 3]

steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4

- name: Set up Python 3.10
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: "3.10"
cache: pip
Expand Down Expand Up @@ -223,7 +226,7 @@ jobs:
run: |
set -euo pipefail

PYTEST_MARKERS='not apitest and not downloader'
PYTEST_MARKERS='not apitest and not downloader and not broker_strategy_live'
export PYTEST_MARKERS

echo "Shard ${SHARD_INDEX}/${SHARD_TOTAL} markers=${PYTEST_MARKERS}"
Expand Down Expand Up @@ -264,20 +267,78 @@ jobs:
print(f"selected_nodeids={len(selected)} total_nodeids={len(nodeids)}")
PY

echo "Running $(wc -l shard_nodeids.txt | awk '{print $1}') nodeids"
timeout 1500 python -m pytest -m "${PYTEST_MARKERS}" --tb=short -q --durations=30 -x $(cat shard_nodeids.txt)
mapfile -t shard_nodeids < shard_nodeids.txt
((${#shard_nodeids[@]} > 0)) || { echo "No backtest nodeids selected"; exit 1; }
echo "Running ${#shard_nodeids[@]} nodeids"
timeout 1500 python -m pytest -m "${PYTEST_MARKERS}" --tb=short -q --durations=30 -x "${shard_nodeids[@]}"

broker-live-strategy-tests:
name: Broker Live Strategy Tests
runs-on: ubuntu-latest
timeout-minutes: 20
environment: unit-tests
needs: lint
if: >-
github.event_name != 'pull_request' ||
(github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')
concurrency:
group: broker-live-strategy-tests-${{ github.repository }}-unit-tests

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Serialize all Alpaca paper-account jobs together

For internal PRs touching lumibot/** or tests/**, this CI workflow and .github/workflows/alpaca-live-broker.yml can run at the same time, but their concurrency groups are different while both use ALPACA_TEST_* and submit/cancel real AAPL paper orders. That lets the two live suites race on the shared Alpaca paper account and potentially observe or cancel each other's orders; use one shared concurrency group for all Alpaca live-paper jobs.

Useful? React with 👍 / 👎.

cancel-in-progress: false

steps:
- uses: actions/checkout@v4

- name: Set up Python 3.10
uses: actions/setup-python@v5
with:
python-version: "3.10"
cache: pip

- name: Install dependencies
run: |
echo "Set AIOHTTP_NO_EXTENSIONS=$AIOHTTP_NO_EXTENSIONS so that aiohttp doesn't try to install C extensions"
python -m pip install --upgrade pip
pip install requests
pip install -r requirements_dev.txt

- name: Validate broker paper secrets
run: |
set -euo pipefail
python - <<'PY'
import os

required = [
"ALPACA_TEST_API_KEY",
"ALPACA_TEST_API_SECRET",
"TRADIER_TEST_ACCESS_TOKEN",
"TRADIER_TEST_ACCOUNT_NUMBER",
]
missing = [name for name in required if not os.environ.get(name)]
if missing:
raise SystemExit("Missing broker live strategy test secrets: " + ", ".join(missing))

print("Broker live strategy test secrets present.")
PY

- name: Run broker live strategy tests
run: |
set -euo pipefail
timeout 900 python -m pytest -q --tb=short -x \
-m "broker_strategy_live" \
tests/test_broker_live_strategy_run_apitest.py

LintAndTest:
name: LintAndTest
runs-on: ubuntu-latest
if: always()
needs: [lint, unit-tests, backtest-tests]
needs: [lint, unit-tests, backtest-tests, broker-live-strategy-tests]
steps:
- name: Check results
run: |
echo "lint: ${{ needs.lint.result }}"
echo "unit-tests: ${{ needs.unit-tests.result }}"
echo "backtest-tests: ${{ needs.backtest-tests.result }}"
echo "broker-live-strategy-tests: ${{ needs.broker-live-strategy-tests.result }}"

if [ "${{ needs.lint.result }}" != "success" ]; then
exit 1
Expand All @@ -288,3 +349,11 @@ jobs:
if [ "${{ needs.backtest-tests.result }}" != "success" ]; then
exit 1
fi
if [ "${{ needs.broker-live-strategy-tests.result }}" = "skipped" ] && \
[ "${{ github.event_name }}" = "pull_request" ] && \
{ [ "${{ github.event.pull_request.head.repo.full_name }}" != "${{ github.repository }}" ] || \
[ "${{ github.actor }}" = "dependabot[bot]" ]; }; then
echo "broker-live-strategy-tests: intentionally skipped for untrusted pull request"
elif [ "${{ needs.broker-live-strategy-tests.result }}" != "success" ]; then
exit 1
fi
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated
70 changes: 70 additions & 0 deletions docs/ALPACA_LIVE_BROKER_CI.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# Alpaca Live Broker CI

One-line description: Real Alpaca paper-account CI for live broker and agent-tool smoke coverage.

Last Updated: 2026-07-03

Status: Active

Audience: Developers, AI Agents

## Overview

`.github/workflows/alpaca-live-broker.yml` runs opt-in live paper API coverage for Alpaca. It is separate from the normal `not apitest` suite because it contacts Alpaca, reads broker/data endpoints, and submits then cancels real paper orders.

The workflow runs for internal PRs/pushes and manual dispatches when relevant broker, agent, dependency, or workflow files change. Fork PRs are skipped because repository secrets are unavailable.

## Credentials

Repository secrets required:

- `ALPACA_TEST_API_KEY`
- `ALPACA_TEST_API_SECRET`

Optional repository secrets:

- `ALPACA_NEWS_API_KEY`
- `ALPACA_NEWS_API_SECRET`

The live news apitest uses `ALPACA_NEWS_API_KEY` / `ALPACA_NEWS_API_SECRET` when present and falls back to `ALPACA_TEST_API_KEY` / `ALPACA_TEST_API_SECRET` for CI/local smoke runs. The built-in `alpaca_news` tool intentionally does not read generic `ALPACA_API_KEY` values unless it is bound to an active Alpaca broker. Do not commit real key values.

## Test Coverage

`tests/test_alpaca_live_broker_apitest.py` is marked `apitest` and `alpaca`.

- `_require_alpaca()` validates paper credentials, authenticates through the real Alpaca API, fails if the paper account is trading-blocked, and configures the broker test market as `24/7` so scheduled one-shot tests run outside NYSE hours.
- `_LiveOrderDataStrategy` calls `run_live(run_once=True)`, reads AAPL last price and daily bars, submits a non-marketable AAPL limit buy, cancels it, waits for terminal cancel state, and verifies order retrieval.
- `_LiveOptionsChainStrategy` calls `run_live(run_once=True)`, reads SPY price, pulls SPY option chains through the broker data source, selects an expiration, and resolves a valid call contract.

`tests/test_agent_alpaca_news_live_apitest.py` uses Alpaca news credentials, falling back to the paper test credentials, to verify the built-in Alpaca news agent tool against real historical news pagination and full-content reads.

The workflow also runs local-only agent runtime, MCP transport, permission, provider-key, and backtest tool coverage before the live broker apitests.

## Concurrency

The live broker job has a repository-wide concurrency group. This prevents overlapping runs from using the same shared Alpaca paper account at once and avoids one run canceling or observing another run's test orders.

## Local Run

Use paper credentials only:

```bash
export ALPACA_TEST_API_KEY="..."
export ALPACA_TEST_API_SECRET="..."

python -m pytest -q \
tests/test_agent_runtime_remote_mcp.py \
tests/test_agent_runtime_mcp_transports.py \
tests/test_agent_runtime_provider_keys.py \
tests/test_agent_runtime_errors.py \
tests/test_agent_tool_permissions.py \
tests/test_agent_alpaca_news_builtin.py \
tests/test_agent_alpaca_news_live_apitest.py \
tests/backtest/test_agent_runtime_backtest.py \
tests/backtest/test_ai_committee_builtin_tools_backtest.py

python -m pytest -q -m apitest --tb=short \
tests/test_alpaca_live_broker_apitest.py
```

If credentials are missing, `alpaca`-marked apitests skip with the missing Alpaca variables instead of requiring unrelated Polygon or ThetaData credentials.
Loading
Loading