diff --git a/docs/AI_AGENT_BUILTIN_TOOLS.md b/docs/AI_AGENT_BUILTIN_TOOLS.md index cdc62f57a..29fe6de39 100644 --- a/docs/AI_AGENT_BUILTIN_TOOLS.md +++ b/docs/AI_AGENT_BUILTIN_TOOLS.md @@ -58,6 +58,15 @@ FRED macro tools: Built-in FRED tools require `FRED_API_KEY` so Lumibot can request official FRED/ALFRED observations with `realtime_start` and `realtime_end`. Lumibot does not use public CSV fallbacks for macro data, because those endpoints can contain revised values and are not a safe default for historical simulations. +FXMacroData tools: + +- `list_fxmacrodata_indicators` +- `get_fxmacrodata_series` +- `get_fxmacrodata_latest` +- `get_fxmacrodata_snapshot` + +FXMacroData tools fetch FX-focused macro announcement rows from FXMacroData. USD announcement data is public. Set `FXMD_API_KEY` or `FXMACRODATA_API_KEY` for non-USD and paid endpoint access. Lumibot sends the key as an `X-API-Key` header rather than adding it to request URLs. + Memory tools: - `remember` diff --git a/docs/AI_TRADING_AGENTS.md b/docs/AI_TRADING_AGENTS.md index 6dfec062e..7e62db68d 100644 --- a/docs/AI_TRADING_AGENTS.md +++ b/docs/AI_TRADING_AGENTS.md @@ -80,7 +80,7 @@ if __name__ == "__main__": ) ``` -The agent gets Lumibot's built-in `get_fred_series` tool automatically when `FRED_API_KEY` is configured. During backtests the tool defaults `as_of` to the strategy datetime and uses FRED/ALFRED realtime parameters, so the agent does not see future macro revisions. +The agent gets Lumibot's built-in macro tools automatically. `get_fred_series` is exposed when `FRED_API_KEY` is configured and uses FRED/ALFRED realtime parameters during backtests. FXMacroData tools such as `get_fxmacrodata_series` are also available for FX-focused macro announcement rows; set `FXMD_API_KEY` or `FXMACRODATA_API_KEY` for non-USD access. No local MCP server scripts. No npm installs. No explicit built-in tool lists. Lumibot includes all built-in tools by default, and you can add custom `@agent_tool` functions when you need proprietary APIs or special research logic. diff --git a/docs/ENV_VARS.md b/docs/ENV_VARS.md index 94e568b04..52cb0bc18 100644 --- a/docs/ENV_VARS.md +++ b/docs/ENV_VARS.md @@ -363,6 +363,24 @@ Notes: - Purpose: Override the local FRED macro data cache. - Default: `~/.lumibot/cache/fred`. +### `FXMD_API_KEY` +- Purpose: Optional FXMacroData API key for non-USD and paid macro announcement endpoints. +- Default: unset. +- Notes: Lumibot sends this key as an `X-API-Key` header. `FXMACRODATA_API_KEY` is also accepted. USD announcement data can be fetched without a key. + +### `FXMACRODATA_API_KEY` +- Purpose: Alternate environment variable for the FXMacroData API key. +- Default: unset. +- Notes: Used when `FXMD_API_KEY` is not set. + +### `LUMIBOT_FXMACRODATA_API_BASE_URL` +- Purpose: Override the FXMacroData API base URL. +- Default: `https://api.fxmacrodata.com/v1`. + +### `LUMIBOT_FXMACRODATA_CACHE_DIR` +- Purpose: Override the local FXMacroData macro release cache. +- Default: `~/.lumibot/cache/fxmacrodata`. + ### `LUMIBOT_MEMORY_DIR` - Purpose: Override the local SQLite agent memory root. - Default: `.lumibot/memory` under the current working directory. diff --git a/docs/FRED_MACRO_DATA.md b/docs/FRED_MACRO_DATA.md index ab62f429a..edda6acdf 100644 --- a/docs/FRED_MACRO_DATA.md +++ b/docs/FRED_MACRO_DATA.md @@ -4,7 +4,7 @@ Lumibot includes native Federal Reserve Economic Data (FRED) macro tools for str Use macro data for interest rates, inflation, employment, growth, liquidity, credit spreads, and market-risk context. -## Strategy API +## FRED Strategy API ```python self.macro.list_series() @@ -13,9 +13,9 @@ self.macro.get_latest("UNRATE") self.macro.get_snapshot(["FEDFUNDS", "DGS10", "CPIAUCSL", "UNRATE"]) ``` -## Agent Tools +## FRED Agent Tools -Agents receive these built-ins automatically: +Agents receive these FRED built-ins automatically: - `list_fred_series` - `get_fred_series` @@ -24,9 +24,11 @@ Agents receive these built-ins automatically: You do not need to manually attach these tools. They are included with the rest of the built-in agent tool surface. -## API Key Behavior +## FRED API Key Behavior -`FRED_API_KEY` is required for the official FRED/ALFRED API path and for strict point-in-time macro backtests. +`FRED_API_KEY` is required for the official FRED/ALFRED API path and for FRED macro data fetches. + +This FRED credential is not used by FXMacroData. FXMacroData access is described separately below. Lumibot uses the official FRED/ALFRED API and passes `realtime_start` and `realtime_end` based on the strategy datetime. This is the strict point-in-time path for macro backtests. @@ -53,3 +55,37 @@ export LUMIBOT_FRED_CACHE_DIR=/path/to/cache ``` Backtests should fetch each series once and reuse the local cache instead of hitting FRED on every trading iteration. + +## FXMacroData Macro Releases + +Lumibot also includes an FXMacroData provider for FX-focused macro announcement rows: + +```python +self.macro.fxmacrodata.list_indicators() +self.macro.fxmacrodata.get_series("eur", "inflation") +self.macro.fxmacrodata.get_latest("jpy", "policy_rate") +self.macro.fxmacrodata.get_snapshot("gbp", ["inflation", "policy_rate", "unemployment"]) +``` + +FXMacroData agents receive these read-only built-ins automatically: + +- `list_fxmacrodata_indicators` +- `get_fxmacrodata_series` +- `get_fxmacrodata_latest` +- `get_fxmacrodata_snapshot` + +USD announcement data is public. Set `FXMD_API_KEY` or `FXMACRODATA_API_KEY` for non-USD and paid endpoint access. Lumibot sends the key as an `X-API-Key` header, not as an `api_key` query parameter. + +In a backtest, `as_of` defaults to `self.get_datetime()`. Lumibot filters release rows by `announcement_datetime` so the strategy does not see macro releases after the simulated datetime. + +In backtests, FXMacroData responses are cached under: + +```text +~/.lumibot/cache/fxmacrodata +``` + +Override with: + +```bash +export LUMIBOT_FXMACRODATA_CACHE_DIR=/path/to/cache +``` diff --git a/docsrc/agents.rst b/docsrc/agents.rst index cb6e61743..c5f018986 100644 --- a/docsrc/agents.rst +++ b/docsrc/agents.rst @@ -88,7 +88,7 @@ Here is a complete AI trading agent strategy that uses Lumibot's built-in FRED m benchmark_asset="SPY", ) -That is the entire strategy file. No local MCP server scripts, no npm installs, and no explicit built-in tool lists. LumiBot includes built-in tools by default, including ``get_fred_series`` when ``FRED_API_KEY`` is configured. +That is the entire strategy file. No local MCP server scripts, no npm installs, and no explicit built-in tool lists. LumiBot includes built-in tools by default, including ``get_fred_series`` when ``FRED_API_KEY`` is configured and FXMacroData tools such as ``get_fxmacrodata_series`` for FX-focused macro announcement rows. How ``@agent_tool`` Works ------------------------- diff --git a/docsrc/agents_builtin_tools.rst b/docsrc/agents_builtin_tools.rst index 359896f88..64c07a85b 100644 --- a/docsrc/agents_builtin_tools.rst +++ b/docsrc/agents_builtin_tools.rst @@ -136,6 +136,20 @@ using realtime parameters. LumiBot's built-in FRED tools do not use public CSV fallbacks; macro tool output should either come from the official API or fail clearly. +FXMacroData +----------- + +FXMacroData tools expose FX-focused macro announcement rows to agents: + +- ``list_fxmacrodata_indicators`` +- ``get_fxmacrodata_series`` +- ``get_fxmacrodata_latest`` +- ``get_fxmacrodata_snapshot`` + +USD announcement data is public. Set ``FXMD_API_KEY`` or +``FXMACRODATA_API_KEY`` for non-USD and paid endpoint access. LumiBot sends the +key as an ``X-API-Key`` header rather than adding it to request URLs. + News ---- diff --git a/docsrc/environment_variables.rst b/docsrc/environment_variables.rst index 83a4b271a..31c819b67 100644 --- a/docsrc/environment_variables.rst +++ b/docsrc/environment_variables.rst @@ -909,6 +909,35 @@ LUMIBOT_FRED_CACHE_DIR - Purpose: Override local FRED macro data cache. - Default: ``~/.lumibot/cache/fred``. +FXMD_API_KEY +^^^^^^^^^^^^ + +- Purpose: Optional FXMacroData API key for non-USD and paid macro announcement + endpoints. +- Default: unset. +- Notes: LumiBot sends this key as an ``X-API-Key`` header. + ``FXMACRODATA_API_KEY`` is also accepted. USD announcement data can be fetched + without a key. + +FXMACRODATA_API_KEY +^^^^^^^^^^^^^^^^^^^ + +- Purpose: Alternate environment variable for the FXMacroData API key. +- Default: unset. +- Notes: Used when ``FXMD_API_KEY`` is not set. + +LUMIBOT_FXMACRODATA_API_BASE_URL +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +- Purpose: Override the FXMacroData API base URL. +- Default: ``https://api.fxmacrodata.com/v1``. + +LUMIBOT_FXMACRODATA_CACHE_DIR +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +- Purpose: Override local FXMacroData macro release cache. +- Default: ``~/.lumibot/cache/fxmacrodata``. + LUMIBOT_MEMORY_DIR ^^^^^^^^^^^^^^^^^^ diff --git a/docsrc/macro_data.rst b/docsrc/macro_data.rst index 3b6f55027..b32f1a3c7 100644 --- a/docsrc/macro_data.rst +++ b/docsrc/macro_data.rst @@ -8,8 +8,8 @@ employment, growth, liquidity, credit spreads, and market-risk context. .. image:: ../docs/assets/ai_committee/docs_fred_macro_data.png :alt: FRED macro data tools and point-in-time behavior in Lumibot -Strategy API ------------- +FRED Strategy API +----------------- .. code-block:: python @@ -18,10 +18,10 @@ Strategy API self.macro.get_latest("UNRATE") self.macro.get_snapshot(["FEDFUNDS", "DGS10", "CPIAUCSL", "UNRATE"]) -Agent Tools ------------ +FRED Agent Tools +---------------- -Agents receive these built-ins automatically: +Agents receive these FRED built-ins automatically: - ``list_fred_series`` - ``get_fred_series`` @@ -31,13 +31,18 @@ Agents receive these built-ins automatically: These tools are available to read-only research agents and trading-enabled portfolio agents. They do not submit, cancel, or modify orders. -API Key Behavior ----------------- +FRED API Key Behavior +--------------------- ``FRED_API_KEY`` is required for the official FRED/ALFRED API path and for -all macro data fetches. LumiBot uses the official API path instead of public -CSV fallbacks so tool output has a clear provenance and backtests can request -point-in-time vintage observations. +FRED macro data fetches. + +This FRED credential is not used by FXMacroData. FXMacroData access is +described separately below. + +LumiBot uses the official API path instead of public CSV fallbacks so tool +output has a clear provenance and backtests can request point-in-time vintage +observations. With a key, LumiBot passes ``realtime_start`` and ``realtime_end`` based on the strategy datetime so the backtest sees the vintage observations that were @@ -60,3 +65,35 @@ Cache FRED data is cached under ``~/.lumibot/cache/fred`` by default. Override this with ``LUMIBOT_FRED_CACHE_DIR``. + +FXMacroData Macro Releases +========================== + +LumiBot also includes an FXMacroData provider for FX-focused macro announcement +rows: + +.. code-block:: python + + self.macro.fxmacrodata.list_indicators() + self.macro.fxmacrodata.get_series("eur", "inflation") + self.macro.fxmacrodata.get_latest("jpy", "policy_rate") + self.macro.fxmacrodata.get_snapshot("gbp", ["inflation", "policy_rate", "unemployment"]) + +FXMacroData agents receive these read-only built-ins automatically: + +- ``list_fxmacrodata_indicators`` +- ``get_fxmacrodata_series`` +- ``get_fxmacrodata_latest`` +- ``get_fxmacrodata_snapshot`` + +USD announcement data is public. Set ``FXMD_API_KEY`` or +``FXMACRODATA_API_KEY`` for non-USD and paid endpoint access. LumiBot sends the +key as an ``X-API-Key`` header, not as an ``api_key`` query parameter. + +In a backtest, ``as_of`` defaults to ``self.get_datetime()``. LumiBot filters +release rows by ``announcement_datetime`` so the strategy does not see macro +releases after the simulated datetime. + +In backtests, FXMacroData responses are cached under +``~/.lumibot/cache/fxmacrodata`` by default. Override this with +``LUMIBOT_FXMACRODATA_CACHE_DIR``. diff --git a/lumibot/components/agents/builtins.py b/lumibot/components/agents/builtins.py index 4da6fc5e5..fa979f86c 100644 --- a/lumibot/components/agents/builtins.py +++ b/lumibot/components/agents/builtins.py @@ -1199,6 +1199,97 @@ def get_fred_snapshot(series_ids: list[str] | str, as_of: str | None = None) -> ) +def _fxmacrodata_client(strategy: Any) -> Any: + macro = getattr(strategy, "macro", None) + client = getattr(macro, "fxmacrodata", None) or getattr(macro, "fxmd", None) + if client is None: + raise ValueError("FXMacroData is not configured on strategy.macro.") + return client + + +def _bind_list_fxmacrodata_indicators(strategy: Any, manager: Any) -> BoundTool: + def list_fxmacrodata_indicators(category: str | None = None) -> dict[str, Any]: + return _fxmacrodata_client(strategy).list_indicators(category=category) + + return BoundTool( + name="list_fxmacrodata_indicators", + description=( + "List common FXMacroData macro announcement indicators for FX-focused event-risk research. " + "Use this before requesting inflation, policy-rate, labor, growth, trade, or sentiment series." + ), + function=list_fxmacrodata_indicators, + source="builtin", + metadata={"kind": "macro", "cache_scope": "strategy_day"}, + ) + + +def _bind_get_fxmacrodata_series(strategy: Any, manager: Any) -> BoundTool: + def get_fxmacrodata_series( + currency: str, + indicator: str, + start: str | None = None, + end: str | None = None, + as_of: str | None = None, + limit: int | None = None, + ) -> dict[str, Any]: + return _fxmacrodata_client(strategy).get_series( + currency, + indicator, + start=start, + end=end, + as_of=as_of, + limit=limit, + ) + + return BoundTool( + name="get_fxmacrodata_series", + description=( + "Get an FXMacroData macro announcement series for a currency and indicator. " + "In backtests, as_of defaults to the strategy datetime and release rows are gated by announcement time. " + "USD is public; set FXMD_API_KEY or FXMACRODATA_API_KEY for non-USD and paid endpoint access." + ), + function=get_fxmacrodata_series, + source="builtin", + metadata={"kind": "macro", "cache_scope": "strategy_day"}, + ) + + +def _bind_get_fxmacrodata_latest(strategy: Any, manager: Any) -> BoundTool: + def get_fxmacrodata_latest(currency: str, indicator: str, as_of: str | None = None) -> dict[str, Any]: + return _fxmacrodata_client(strategy).get_latest(currency, indicator, as_of=as_of) + + return BoundTool( + name="get_fxmacrodata_latest", + description=( + "Get the latest FXMacroData release row for a currency and indicator available as of the strategy datetime " + "or explicit as_of timestamp." + ), + function=get_fxmacrodata_latest, + source="builtin", + metadata={"kind": "macro", "cache_scope": "strategy_day"}, + ) + + +def _bind_get_fxmacrodata_snapshot(strategy: Any, manager: Any) -> BoundTool: + def get_fxmacrodata_snapshot( + currency: str, + indicators: list[str] | str, + as_of: str | None = None, + ) -> dict[str, Any]: + return _fxmacrodata_client(strategy).get_snapshot(currency, indicators, as_of=as_of) + + return BoundTool( + name="get_fxmacrodata_snapshot", + description=( + "Get latest FXMacroData release rows for several indicators in one currency. " + "Pass indicators as a list or comma-separated string such as inflation,policy_rate,unemployment." + ), + function=get_fxmacrodata_snapshot, + source="builtin", + metadata={"kind": "macro", "cache_scope": "strategy_day"}, + ) + + def _bind_notify_user(strategy: Any, manager: Any) -> BoundTool: def notify_user(title: str, message: str, severity: str = "info", enabled: bool | None = None) -> dict[str, Any]: results = strategy.notify(title, message, severity=severity, enabled=enabled) @@ -1554,6 +1645,34 @@ def get_fred_latest(self) -> ToolDefinition: def get_fred_snapshot(self) -> ToolDefinition: return ToolDefinition(name="get_fred_snapshot", description="Get a multi-series FRED macro snapshot.", binder=_bind_get_fred_snapshot) + def list_fxmacrodata_indicators(self) -> ToolDefinition: + return ToolDefinition( + name="list_fxmacrodata_indicators", + description="List common FXMacroData announcement indicators.", + binder=_bind_list_fxmacrodata_indicators, + ) + + def get_fxmacrodata_series(self) -> ToolDefinition: + return ToolDefinition( + name="get_fxmacrodata_series", + description="Get an FXMacroData macro announcement series.", + binder=_bind_get_fxmacrodata_series, + ) + + def get_fxmacrodata_latest(self) -> ToolDefinition: + return ToolDefinition( + name="get_fxmacrodata_latest", + description="Get the latest FXMacroData macro release row.", + binder=_bind_get_fxmacrodata_latest, + ) + + def get_fxmacrodata_snapshot(self) -> ToolDefinition: + return ToolDefinition( + name="get_fxmacrodata_snapshot", + description="Get a multi-indicator FXMacroData macro snapshot.", + binder=_bind_get_fxmacrodata_snapshot, + ) + class _NotificationTools: def notify_user(self) -> ToolDefinition: @@ -1662,6 +1781,10 @@ def all(self) -> list[ToolDefinition]: self.macro.get_fred_series(), self.macro.get_fred_latest(), self.macro.get_fred_snapshot(), + self.macro.list_fxmacrodata_indicators(), + self.macro.get_fxmacrodata_series(), + self.macro.get_fxmacrodata_latest(), + self.macro.get_fxmacrodata_snapshot(), self.notifications.notify_user(), self.memory.remember(), self.memory.search(), diff --git a/lumibot/components/agents/manager.py b/lumibot/components/agents/manager.py index 0d7a9f2a3..d8a175fd0 100644 --- a/lumibot/components/agents/manager.py +++ b/lumibot/components/agents/manager.py @@ -1193,6 +1193,10 @@ def _derive_warnings(self, result: AgentRunResult, runtime_context: dict[str, An "get_fred_series", "get_fred_latest", "get_fred_snapshot", + "list_fxmacrodata_indicators", + "get_fxmacrodata_series", + "get_fxmacrodata_latest", + "get_fxmacrodata_snapshot", } for name in tool_names ) diff --git a/lumibot/components/agents/runtime.py b/lumibot/components/agents/runtime.py index 9c4a2be74..59552e46f 100644 --- a/lumibot/components/agents/runtime.py +++ b/lumibot/components/agents/runtime.py @@ -1160,6 +1160,13 @@ def _build_user_text(self, request: RuntimeRequest) -> str: fred_tools = sorted(name for name in tool_names if name.startswith("get_fred_") or name == "list_fred_series") if fred_tools: required_categories.append(" or ".join(fred_tools)) + fxmacrodata_tools = sorted( + name + for name in tool_names + if name.startswith("get_fxmacrodata_") or name == "list_fxmacrodata_indicators" + ) + if fxmacrodata_tools: + required_categories.append(" or ".join(fxmacrodata_tools)) required_categories.extend( [ "get_income_statement, get_balance_sheet, get_cash_flow, or get_company_facts", diff --git a/lumibot/macro/__init__.py b/lumibot/macro/__init__.py index 2db558606..7d4e80fa1 100644 --- a/lumibot/macro/__init__.py +++ b/lumibot/macro/__init__.py @@ -1,5 +1,7 @@ """Per-strategy macroeconomic data helpers.""" from .fred import FREDMacroData +from .fxmacrodata import FXMacroData +from .macro_data import MacroData -__all__ = ["FREDMacroData"] +__all__ = ["FREDMacroData", "FXMacroData", "MacroData"] diff --git a/lumibot/macro/fxmacrodata.py b/lumibot/macro/fxmacrodata.py new file mode 100644 index 000000000..e17878bd4 --- /dev/null +++ b/lumibot/macro/fxmacrodata.py @@ -0,0 +1,372 @@ +import hashlib +import json +import os +import re +import time +from datetime import date, datetime, timezone +from pathlib import Path +from typing import Any + +import requests + +FXMACRODATA_API_BASE_URL = "https://api.fxmacrodata.com/v1" + + +CURATED_FXMACRODATA_INDICATORS: dict[str, dict[str, str]] = { + "policy_rate": {"category": "rates", "name": "Policy Rate"}, + "inflation": {"category": "inflation", "name": "Inflation"}, + "unemployment": {"category": "labor", "name": "Unemployment Rate"}, + "non_farm_payrolls": {"category": "labor", "name": "US Non-Farm Payrolls"}, + "gdp_growth": {"category": "growth", "name": "GDP Growth"}, + "retail_sales": {"category": "demand", "name": "Retail Sales"}, + "trade_balance": {"category": "trade", "name": "Trade Balance"}, + "current_account": {"category": "trade", "name": "Current Account"}, + "business_confidence": {"category": "sentiment", "name": "Business Confidence"}, + "consumer_confidence": {"category": "sentiment", "name": "Consumer Confidence"}, +} + + +def _parse_dt(value: Any) -> datetime | None: + if isinstance(value, datetime): + parsed = value + elif isinstance(value, date): + parsed = datetime(value.year, value.month, value.day, tzinfo=timezone.utc) + else: + text = str(value or "").strip() + if not text: + return None + text = text.replace("Z", "+00:00") + try: + parsed = datetime.fromisoformat(text) + except ValueError: + try: + parsed = datetime.strptime(text[:10], "%Y-%m-%d") + except ValueError: + return None + if parsed.tzinfo is None: + parsed = parsed.replace(tzinfo=timezone.utc) + return parsed.astimezone(timezone.utc) + + +def _as_of_datetime(value: Any) -> datetime: + parsed = _parse_dt(value) + if parsed is not None: + return parsed + return datetime.now(timezone.utc) + + +def _date_text(value: Any | None) -> str | None: + parsed = _parse_dt(value) + if parsed is None: + return None + return parsed.date().isoformat() + + +def _safe_float(value: Any) -> float | None: + if value is None: + return None + text = str(value).strip() + if not text or text.lower() in {"nan", "none", "null", "."}: + return None + try: + return float(text) + except Exception: + return None + + +def _first_present(row: dict[str, Any], keys: tuple[str, ...]) -> Any: + for key in keys: + value = row.get(key) + if value is not None and str(value).strip() != "": + return value + return None + + +class FXMacroData: + """FXMacroData macro-release client with local caching and strategy-date gating. + + USD data can be fetched without credentials. Non-USD and paid endpoint + access require ``FXMD_API_KEY`` or ``FXMACRODATA_API_KEY``. Credentials are + sent in the ``X-API-Key`` header so keys do not appear in request URLs. + """ + + def __init__( + self, + strategy: Any | None = None, + *, + cache_dir: str | os.PathLike[str] | None = None, + api_key: str | None = None, + base_url: str | None = None, + min_request_interval_seconds: float = 0.2, + ) -> None: + self.strategy = strategy + self.cache_dir = Path( + cache_dir + or os.environ.get("LUMIBOT_FXMACRODATA_CACHE_DIR") + or Path.home() / ".lumibot" / "cache" / "fxmacrodata" + ) + self.cache_dir.mkdir(parents=True, exist_ok=True) + self.api_key = ( + api_key + or os.environ.get("FXMD_API_KEY") + or os.environ.get("FXMACRODATA_API_KEY") + ) + self.base_url = ( + base_url + or os.environ.get("LUMIBOT_FXMACRODATA_API_BASE_URL") + or FXMACRODATA_API_BASE_URL + ).rstrip("/") + self.min_request_interval_seconds = max(float(min_request_interval_seconds), 0.0) + self._last_request_at = 0.0 + + def _strategy_as_of(self) -> datetime: + if self.strategy is not None and hasattr(self.strategy, "get_datetime"): + try: + return _as_of_datetime(self.strategy.get_datetime()) + except Exception: + pass + return datetime.now(timezone.utc) + + def _effective_as_of_datetime(self, as_of: Any | None) -> datetime: + return _as_of_datetime(as_of) if as_of is not None else self._strategy_as_of() + + def _cache_path(self, *parts: str) -> Path: + safe = [re.sub(r"[^A-Za-z0-9_.=-]+", "_", str(part)).strip("_") for part in parts] + return self.cache_dir.joinpath(*safe) + + def _rate_limit(self) -> None: + elapsed = time.monotonic() - self._last_request_at + if elapsed < self.min_request_interval_seconds: + time.sleep(self.min_request_interval_seconds - elapsed) + self._last_request_at = time.monotonic() + + def _headers(self) -> dict[str, str]: + if self.api_key: + return {"X-API-Key": self.api_key} + return {} + + def _use_cache(self) -> bool: + is_backtesting = getattr(self.strategy, "is_backtesting", False) + if callable(is_backtesting): + is_backtesting = is_backtesting() + return bool(is_backtesting) + + def _write_cache(self, cache_path: Path, payload: dict[str, Any]) -> None: + cache_path.parent.mkdir(parents=True, exist_ok=True) + temp_path = cache_path.with_name(f"{cache_path.name}.{os.getpid()}.{time.time_ns()}.tmp") + try: + temp_path.write_text(json.dumps(payload, indent=2, sort_keys=True), encoding="utf-8") + os.replace(temp_path, cache_path) + finally: + if temp_path.exists(): + temp_path.unlink() + + def _require_key_for_currency(self, currency: str) -> None: + if currency.lower() != "usd" and not self.api_key: + raise ValueError( + "FXMD_API_KEY or FXMACRODATA_API_KEY is required for non-USD FXMacroData requests. " + "USD announcement data can be fetched without credentials." + ) + + def _get_json(self, path: str, params: dict[str, Any], cache_path: Path) -> dict[str, Any]: + use_cache = self._use_cache() + if use_cache and cache_path.exists(): + return json.loads(cache_path.read_text(encoding="utf-8")) + self._rate_limit() + response = requests.get( + f"{self.base_url}{path}", + params={key: value for key, value in params.items() if value is not None}, + headers=self._headers() or None, + timeout=30, + ) + response.raise_for_status() + payload = response.json() + if use_cache: + self._write_cache(cache_path, payload) + return payload + + def list_indicators(self, category: str | None = None) -> dict[str, Any]: + """Return curated FXMacroData indicators, optionally filtered by category.""" + rows = [] + wanted_category = str(category).strip().lower() if category else None + for indicator, metadata in CURATED_FXMACRODATA_INDICATORS.items(): + if wanted_category and metadata["category"] != wanted_category: + continue + rows.append({"indicator": indicator, **metadata}) + categories = sorted( + {metadata["category"] for metadata in CURATED_FXMACRODATA_INDICATORS.values()} + ) + return { + "source": "fxmacrodata", + "indicators": rows, + "categories": categories, + "notes": ( + "These are common FXMacroData announcement indicators. " + "USD announcement data is public; " + "set FXMD_API_KEY or FXMACRODATA_API_KEY for non-USD and paid endpoint access." + ), + } + + def get_series( + self, + currency: str, + indicator: str, + *, + start: Any | None = None, + end: Any | None = None, + as_of: Any | None = None, + limit: int | None = None, + ) -> dict[str, Any]: + """Fetch a point-in-time-safe FXMacroData announcement series.""" + currency_code = str(currency or "").strip().lower() + indicator_slug = str(indicator or "").strip().lower() + if not currency_code: + raise ValueError("currency is required.") + if not indicator_slug: + raise ValueError("indicator is required.") + self._require_key_for_currency(currency_code) + + as_of_dt = self._effective_as_of_datetime(as_of) + start_text = _date_text(start) + end_text = _date_text(end) + if end_text is None or date.fromisoformat(end_text) > as_of_dt.date(): + end_text = as_of_dt.date().isoformat() + + params: dict[str, Any] = {"start_date": start_text, "end_date": end_text} + if limit is not None: + params["limit"] = max(int(limit), 1) + cache_key = json.dumps( + { + "authenticated": bool(self.api_key), + "currency": currency_code, + "indicator": indicator_slug, + "params": {key: value for key, value in params.items() if value is not None}, + }, + sort_keys=True, + ) + payload = self._get_json( + f"/announcements/{currency_code}/{indicator_slug}", + params, + self._cache_path( + "api", + currency_code, + indicator_slug, + f"{hashlib.sha256(cache_key.encode()).hexdigest()}.json", + ), + ) + observations = self._normalize_observations( + payload, + currency_code, + indicator_slug, + as_of_dt, + ) + if limit is not None: + observations = observations[-max(int(limit), 1):] + return { + "source": "fxmacrodata_api", + "currency": currency_code, + "indicator": indicator_slug, + "as_of": as_of_dt.isoformat(), + "point_in_time_safe": True, + "observations": observations, + } + + def get_latest( + self, + currency: str, + indicator: str, + *, + as_of: Any | None = None, + ) -> dict[str, Any]: + """Return the latest FXMacroData observation for an indicator.""" + payload = self.get_series(currency, indicator, as_of=as_of, limit=20) + observations = payload.get("observations", []) + latest = observations[-1] if observations else None + return {**payload, "latest": latest, "observations": observations[-10:]} + + def get_snapshot( + self, + currency: str, + indicators: list[str] | tuple[str, ...] | str, + *, + as_of: Any | None = None, + ) -> dict[str, Any]: + """Return latest values for several FXMacroData indicators.""" + if isinstance(indicators, str): + requested = [part.strip() for part in indicators.split(",") if part.strip()] + else: + requested = [str(part).strip() for part in indicators if str(part).strip()] + as_of_dt = self._effective_as_of_datetime(as_of) + values = {} + errors = {} + for indicator in requested: + key = indicator.lower() + try: + values[key] = self.get_latest(currency, key, as_of=as_of_dt)["latest"] + except (RuntimeError, ValueError, requests.RequestException, OSError) as exc: + errors[key] = str(exc) + return { + "source": "fxmacrodata", + "currency": str(currency or "").strip().lower(), + "as_of": as_of_dt.isoformat(), + "values": values, + "errors": errors, + } + + def _normalize_observations( + self, + payload: dict[str, Any], + currency: str, + indicator: str, + as_of_dt: datetime, + ) -> list[dict[str, Any]]: + rows = payload.get("data") + if not isinstance(rows, list): + rows = payload.get("observations") + if not isinstance(rows, list): + rows = payload.get("results") + if not isinstance(rows, list): + rows = [] + + observations = [] + for row in rows: + if not isinstance(row, dict): + continue + announcement_dt = _parse_dt( + _first_present( + row, + ( + "announcement_datetime", + "announcement_datetime_utc", + "release_datetime", + "published_at", + ), + ) + ) + row_date = _date_text( + _first_present(row, ("date", "release_date", "observation_date", "period")) + ) + comparison_dt = announcement_dt or _parse_dt(row_date) + if comparison_dt is None: + continue + if comparison_dt > as_of_dt: + continue + value = _safe_float(_first_present(row, ("value", "val", "actual", "latest_value"))) + normalized = { + "date": row_date, + "value": value, + "announcement_datetime": ( + announcement_dt.isoformat() if announcement_dt is not None else None + ), + "currency": str(row.get("currency") or currency).lower(), + "indicator": str(row.get("indicator") or indicator).lower(), + } + for key in ("forecast", "previous", "revision", "unit", "source", "event_name"): + if key in row: + normalized[key] = row.get(key) + observations.append(normalized) + + observations.sort( + key=lambda row: (row.get("announcement_datetime") or row.get("date") or "") + ) + return observations diff --git a/lumibot/macro/macro_data.py b/lumibot/macro/macro_data.py new file mode 100644 index 000000000..336aec234 --- /dev/null +++ b/lumibot/macro/macro_data.py @@ -0,0 +1,41 @@ +import os +from typing import Any + +from .fred import FREDMacroData +from .fxmacrodata import FXMacroData + + +class MacroData(FREDMacroData): + """Strategy macro-data container. + + ``MacroData`` preserves the historical ``self.macro`` FRED methods while + exposing additional providers under named attributes. + """ + + def __init__( + self, + strategy: Any | None = None, + *, + cache_dir: str | os.PathLike[str] | None = None, + api_key: str | None = None, + min_request_interval_seconds: float = 0.2, + fxmacrodata: FXMacroData | None = None, + fxmacrodata_api_key: str | None = None, + fxmacrodata_base_url: str | None = None, + fxmacrodata_cache_dir: str | os.PathLike[str] | None = None, + ) -> None: + super().__init__( + strategy, + cache_dir=cache_dir, + api_key=api_key, + min_request_interval_seconds=min_request_interval_seconds, + ) + self.fred = self + self.fxmacrodata = fxmacrodata or FXMacroData( + strategy, + api_key=fxmacrodata_api_key, + base_url=fxmacrodata_base_url, + cache_dir=fxmacrodata_cache_dir, + min_request_interval_seconds=min_request_interval_seconds, + ) + self.fxmd = self.fxmacrodata diff --git a/lumibot/strategies/_strategy.py b/lumibot/strategies/_strategy.py index ee1166cf3..e8a394176 100644 --- a/lumibot/strategies/_strategy.py +++ b/lumibot/strategies/_strategy.py @@ -975,7 +975,7 @@ def __init__( self.agents = _lazy_strategy_component("lumibot.components.agents", "AgentManager", self) self.indicators = _lazy_strategy_component("lumibot.indicators", "Indicators", self) self.fundamentals = _lazy_strategy_component("lumibot.fundamentals", "SECFundamentals", self) - self.macro = _lazy_strategy_component("lumibot.macro", "FREDMacroData", self) + self.macro = _lazy_strategy_component("lumibot.macro", "MacroData", self) self.notifications = _lazy_strategy_component( "lumibot.components.notifications", "NotificationManager", diff --git a/tests/test_agent_tool_permissions.py b/tests/test_agent_tool_permissions.py index 5cbc8ed5e..41d19c6a6 100644 --- a/tests/test_agent_tool_permissions.py +++ b/tests/test_agent_tool_permissions.py @@ -145,6 +145,10 @@ def test_agent_allow_trading_false_removes_only_mutating_order_tools(monkeypatch assert "account_positions" in tool_names assert "get_income_statement" in tool_names assert "get_indicator" in tool_names + assert "list_fxmacrodata_indicators" in tool_names + assert "get_fxmacrodata_series" in tool_names + assert "get_fxmacrodata_latest" in tool_names + assert "get_fxmacrodata_snapshot" in tool_names assert "list_fred_series" not in tool_names assert "get_fred_series" not in tool_names assert "get_fred_latest" not in tool_names diff --git a/tests/test_fxmacrodata_macro.py b/tests/test_fxmacrodata_macro.py new file mode 100644 index 000000000..276e90a73 --- /dev/null +++ b/tests/test_fxmacrodata_macro.py @@ -0,0 +1,201 @@ +"""Tests for FXMacroData macro data support.""" + +# pylint: disable=missing-class-docstring,missing-function-docstring + +from datetime import datetime, timezone + +import pytest + +from lumibot.macro import FXMacroData, MacroData + + +class _Response: + def __init__(self, *, payload=None): + self._payload = payload + + def raise_for_status(self): + return None + + def json(self): + return self._payload + + +class _Strategy: + def get_datetime(self): + return datetime(2025, 1, 15, 12, 0, tzinfo=timezone.utc) + + +class _BacktestingStrategy(_Strategy): + is_backtesting = True + + +def _payload(): + return { + "data": [ + { + "date": "2024-12-01", + "val": "2.4", + "announcement_datetime": "2025-01-15T11:30:00Z", + "currency": "eur", + "indicator": "inflation", + "forecast": "2.5", + }, + { + "date": "2025-01-01", + "val": "2.6", + "announcement_datetime": "2025-01-15T12:30:00Z", + "currency": "eur", + "indicator": "inflation", + }, + ] + } + + +@pytest.fixture +def fxmacrodata_factory(monkeypatch, tmp_path): + """Create FXMacroData instances with isolated environment and network state.""" + + def _build(strategy, fake_get, *, api_key=None): + monkeypatch.delenv("FXMD_API_KEY", raising=False) + monkeypatch.delenv("FXMACRODATA_API_KEY", raising=False) + if api_key is not None: + monkeypatch.setenv("FXMD_API_KEY", api_key) + monkeypatch.setattr("lumibot.macro.fxmacrodata.requests.get", fake_get) + return FXMacroData(strategy, cache_dir=tmp_path, min_request_interval_seconds=0) + + return _build + + +def test_fxmacrodata_uses_x_api_key_header_and_filters_future_rows(fxmacrodata_factory): + calls = [] + + def fake_get(url, **kwargs): + calls.append((url, kwargs)) + return _Response(payload=_payload()) + + fxmd = fxmacrodata_factory(_Strategy(), fake_get, api_key="test-fxmd-key") + + result = fxmd.get_series("eur", "inflation", start="2024-01-01") + + assert result["source"] == "fxmacrodata_api" + assert result["point_in_time_safe"] is True + assert [row["date"] for row in result["observations"]] == ["2024-12-01"] + + url, kwargs = calls[0] + assert url == "https://api.fxmacrodata.com/v1/announcements/eur/inflation" + assert kwargs["headers"] == {"X-API-Key": "test-fxmd-key"} + assert "api_key" not in kwargs["params"] + assert kwargs["params"]["start_date"] == "2024-01-01" + assert kwargs["params"]["end_date"] == "2025-01-15" + + +def test_fxmacrodata_drops_rows_without_parseable_dates(fxmacrodata_factory): + def fake_get(url, **kwargs): + return _Response(payload={"data": [{"val": "9.9"}, {"date": "2025-01-01", "val": "3.0"}]}) + + fxmd = fxmacrodata_factory(_Strategy(), fake_get) + + result = fxmd.get_series("usd", "inflation") + + assert [row["date"] for row in result["observations"]] == ["2025-01-01"] + assert result["observations"][0]["value"] == 3.0 + + +def test_fxmacrodata_live_requests_bypass_disk_cache(fxmacrodata_factory, tmp_path): + calls = [] + payloads = [ + {"data": [{"date": "2025-01-01", "val": "3.0"}]}, + {"data": [{"date": "2025-01-01", "val": "4.0"}]}, + ] + + def fake_get(url, **kwargs): + calls.append((url, kwargs)) + return _Response(payload=payloads[len(calls) - 1]) + + fxmd = fxmacrodata_factory(_Strategy(), fake_get) + + first = fxmd.get_latest("usd", "inflation") + second = fxmd.get_latest("usd", "inflation") + + assert first["latest"]["value"] == 3.0 + assert second["latest"]["value"] == 4.0 + assert len(calls) == 2 + assert not list(tmp_path.rglob("*.json")) + + +def test_fxmacrodata_backtests_use_disk_cache(fxmacrodata_factory, tmp_path): + calls = [] + + def fake_get(url, **kwargs): + calls.append((url, kwargs)) + return _Response(payload={"data": [{"date": "2025-01-01", "val": str(len(calls))}]}) + + fxmd = fxmacrodata_factory(_BacktestingStrategy(), fake_get) + + first = fxmd.get_latest("usd", "inflation") + second = fxmd.get_latest("usd", "inflation") + + assert first["latest"]["value"] == 1.0 + assert second["latest"]["value"] == 1.0 + assert len(calls) == 1 + assert len(list(tmp_path.rglob("*.json"))) == 1 + assert not list(tmp_path.rglob("*.tmp")) + + +def test_fxmacrodata_usd_requests_do_not_require_api_key(fxmacrodata_factory): + calls = [] + + def fake_get(url, **kwargs): + calls.append((url, kwargs)) + return _Response(payload={"data": [{"date": "2025-01-01", "val": "3.0"}]}) + + fxmd = fxmacrodata_factory(_Strategy(), fake_get) + + result = fxmd.get_latest("usd", "inflation") + + assert result["latest"]["value"] == 3.0 + assert calls[0][1]["headers"] is None + + +def test_fxmacrodata_non_usd_requires_api_key(monkeypatch, tmp_path): + monkeypatch.delenv("FXMD_API_KEY", raising=False) + monkeypatch.delenv("FXMACRODATA_API_KEY", raising=False) + fxmd = FXMacroData(_Strategy(), cache_dir=tmp_path, min_request_interval_seconds=0) + + try: + fxmd.get_series("jpy", "policy_rate") + except ValueError as exc: + assert "FXMD_API_KEY or FXMACRODATA_API_KEY is required" in str(exc) + else: + raise AssertionError("non-USD FXMacroData requests should require an API key") + + catalog = fxmd.list_indicators(category="rates") + assert any(row["indicator"] == "policy_rate" for row in catalog["indicators"]) + + +def test_fxmacrodata_snapshot_reports_per_indicator_errors(fxmacrodata_factory): + def fake_get(url, **_kwargs): + if url.endswith("/policy_rate"): + raise RuntimeError("upstream unavailable") + return _Response(payload={"data": [{"date": "2025-01-01", "val": "3.0"}]}) + + fxmd = fxmacrodata_factory(_Strategy(), fake_get, api_key="test-fxmd-key") + + result = fxmd.get_snapshot("eur", ["inflation", "policy_rate"]) + + assert result["values"]["inflation"]["value"] == 3.0 + assert "policy_rate" in result["errors"] + + +def test_macro_data_preserves_fred_methods_and_adds_fxmacrodata(tmp_path): + macro = MacroData( + _Strategy(), + cache_dir=tmp_path / "fred", + fxmacrodata_cache_dir=tmp_path / "fxmacrodata", + min_request_interval_seconds=0, + ) + + assert macro.fred is macro + assert macro.list_series(category="rates")["series"] + assert macro.fxmacrodata.list_indicators(category="rates")["indicators"] + assert macro.fxmd is macro.fxmacrodata