feat: getTransfersByAddress method

CONTRIBUTING.md

@@ -176,12 +176,13 @@ Validation scenarios compare Superbank RPC responses against a reference endpoin
 REFERENCE_RPC_URL=https://api.mainnet-beta.solana.com scripts/test/run-k6.sh
 ```
 
-`getTransactionsForAddress` is a Superbank-specific method, so its validation scenario uses
-`TFA_REFERENCE_RPC_URL` instead of the standard Solana reference endpoint:
+`getTransactionsForAddress` and `getTransfersByAddress` are Superbank-specific methods, so its validation scenarios use
+`TFA_REFERENCE_RPC_URL` and `TBA_REFERENCE_RPC_URL` instead of the standard Solana reference endpoint:
 
 ```bash
 REFERENCE_RPC_URL=https://api.mainnet-beta.solana.com \
 TFA_REFERENCE_RPC_URL=http://localhost:8898 \
+TBA_REFERENCE_RPC_URL=http://localhost:8898 \
 scripts/test/run-k6.sh
 ```
 

README.md

@@ -85,8 +85,8 @@ ClickHouse.
 
 For single-node ClickHouse (local dev), apply the schemas under `ddl/local/` in this order.
 `transactions.sql` must be applied before the materialized-view schemas (`gsfa*.sql`,
-`signatures.sql`, and `token_owner_activity.sql`) because those views read from the transactions
-table.
+`signatures.sql`, `token_owner_activity.sql`, and `transfers.sql`) because those views
+read from the transactions table.
 
 ```bash
 cat ddl/local/transactions.sql | docker exec -i clickhouse clickhouse-client --multiquery
@@ -97,8 +97,13 @@ cat ddl/local/gsfa.sql | docker exec -i clickhouse clickhouse-client --multiquer
 cat ddl/local/signatures.sql | docker exec -i clickhouse clickhouse-client --multiquery
 # Optional: required only for `tokenAccounts` filters in `getTransactionsForAddress`.
 cat ddl/local/token_owner_activity.sql | docker exec -i clickhouse clickhouse-client --multiquery
+cat ddl/local/transfers.sql | docker exec -i clickhouse clickhouse-client --multiquery
 ```
 
+`transfers.sql` only indexes transaction rows inserted after the materialized view is created.
+Do not advertise `getTransfersByAddress` as historical until verified backfill coverage has been
+recorded; see [transfer endpoint operations](docs/get-transfers-by-address-operations.md).
+
 If you use `gsfa_hot.sql` and want hot addresses excluded from the main GSFA table, apply
 `ddl/local/gsfa_nohot.sql` instead of `ddl/local/gsfa.sql`, then apply `ddl/local/gsfa_hot.sql`.
 

Tiltfile

@@ -117,6 +117,8 @@ data:
 {signatures}
   token_owner_activity.sql: |
 {token_owner_activity}
+  transfers.sql: |
+{transfers}
 """.format(
     namespace=namespace,
     transactions=_indent_block(read_file("ddl/local/transactions.sql"), 4),
@@ -126,6 +128,7 @@ data:
     gsfa_hot=_indent_block(read_file("ddl/local/gsfa_hot.sql"), 4),
     signatures=_indent_block(read_file("ddl/local/signatures.sql"), 4),
     token_owner_activity=_indent_block(read_file("ddl/local/token_owner_activity.sql"), 4),
+    transfers=_indent_block(read_file("ddl/local/transfers.sql"), 4),
 )
 )
 
@@ -297,7 +300,8 @@ for f in \\
   ddl/local/gsfa.sql \\
   ddl/local/gsfa_hot.sql \\
   ddl/local/signatures.sql \\
-  ddl/local/token_owner_activity.sql
+  ddl/local/token_owner_activity.sql \\
+  ddl/local/transfers.sql
 do
   echo "[apply-clickhouse-schema] Applying $f..."
   cat "$f" | kubectl -n "$ns" exec -i "$pod" -c clickhouse -- clickhouse-client --user "$ch_user" --password "$ch_password" --multiquery
@@ -313,6 +317,7 @@ echo "[apply-clickhouse-schema] Done."
         "ddl/local/gsfa_hot.sql",
         "ddl/local/signatures.sql",
         "ddl/local/token_owner_activity.sql",
+        "ddl/local/transfers.sql",
     ],
     resource_deps=["clickhouse"],
     trigger_mode=TRIGGER_MODE_MANUAL,

crates/superbank-rpc/README.md

@@ -23,6 +23,7 @@ writer that matches the same schemas).
 - `getInflationReward`
 - `getEpochSchedule`
 - `getTransactionsForAddress` (custom)
+- `getTransfersByAddress` (custom)
 
 Notes:
 - JSON-RPC batch envelopes are supported. Batch execution is bounded by
@@ -87,6 +88,21 @@ Notes:
   these aliases cannot be combined with same-side slot filters (`lt`/`lte` for `beforeSlot`,
   `gt`/`gte` for `untilSlot`).
   Token account filters require the token-owner activity table (see below).
+- `getTransfersByAddress` returns successful indexed SOL/SPL transfers with
+  top-level `with`, `direction=in|out`, `mint`, `solMode=merged|separate`, `sortOrder=asc|desc`,
+  `limit`, `paginationToken` (`slot:transactionIdx:instructionIdx:innerInstructionIdx:type`),
+  `commitment`, `minContextSlot`, and `filters.amount|slot|blockTime`. Amounts are raw integer
+  strings (lamports for SOL, token base units for SPL), with `uiAmount` derived from decimals.
+  `filters.amount` accepts an unsigned JSON integer or a base-10 string; decimal and floating
+  values are rejected so values above JavaScript's safe-integer range remain exact. `feeAmount` is
+  a raw Token-2022 `TransferCheckedWithFee` fee when known, otherwise `null`.
+  The index is derived from pre/post balance deltas, so the opposite user account is nullable when
+  no counterparty can be inferred. This method requires the transfers table (see below).
+  Creating the materialized view only indexes future inserts. Do not describe the endpoint as
+  historically complete until an operator has completed and recorded verified backfill coverage;
+  see [`docs/get-transfers-by-address-operations.md`](../../docs/get-transfers-by-address-operations.md).
+  Response `paginationToken` is non-null only when another matching page exists after all filters
+  are applied; the token points to the last row returned by the current page.
 
 ## ClickHouse schemas
 
@@ -104,10 +120,12 @@ Required files in the chosen set:
 Optional:
 - `gsfa_hot.sql` when using hot-address routing.
 - `token_owner_activity.sql` when using token-owner filters in `getTransactionsForAddress`.
+- `transfers.sql` when using `getTransfersByAddress`.
 
 Apply `transactions.sql` before the materialized-view schemas (`gsfa*.sql`, `signatures.sql`, and
-`token_owner_activity.sql`) because those views read from the transactions table. If you use
-`gsfa_hot.sql`, apply `gsfa_nohot.sql` instead of `gsfa.sql`, then apply `gsfa_hot.sql`.
+`token_owner_activity.sql`, `transfers.sql`) because those views read from the
+transactions table. If you use `gsfa_hot.sql`, apply `gsfa_nohot.sql` instead of `gsfa.sql`, then
+apply `gsfa_hot.sql`.
 
 For the Agave 4.2 rollout, apply transaction-column and materialized-view DDL first, deploy
 `superbank-rpc` next (disk-cache schema 3 intentionally rebuilds existing caches), and only then
@@ -414,6 +432,7 @@ CLI flags and environment variables (see `crates/superbank-rpc/src/config.rs`):
 | `--clickhouse-gsfa-hot-local-table` | `CLICKHOUSE_GSFA_HOT_LOCAL_TABLE` | `default.gsfa_hot_local` | Shard-direct only. Local hot table queried by hot-address fanout. |
 | `--clickhouse-signatures-local-table` | `CLICKHOUSE_SIGNATURES_LOCAL_TABLE` | — | Shard-direct only. |
 | `--clickhouse-token-owner-activity-local-table` | `CLICKHOUSE_TOKEN_OWNER_ACTIVITY_LOCAL_TABLE` | — | Shard-direct only. |
+| `--clickhouse-transfers-local-table` | `CLICKHOUSE_TRANSFERS_LOCAL_TABLE` | — | Shard-direct only. |
 | `--clickhouse-transactions-local-table` | `CLICKHOUSE_TRANSACTIONS_LOCAL_TABLE` | — | Shard-direct only. |
 | `--clickhouse-blocks-metadata-local-table` | `CLICKHOUSE_BLOCKS_METADATA_LOCAL_TABLE` | — | Shard-direct only. |
 | `--clickhouse-shard-http-port` | `CLICKHOUSE_SHARD_HTTP_PORT` | — | Shard-direct only. |
@@ -429,6 +448,8 @@ Table selection (environment variables, read at startup):
 | `CLICKHOUSE_GSFA_HOT_TABLE` | `default.gsfa_hot` | — |
 | `CLICKHOUSE_SIGNATURE_STATUSES_TABLE` | `default.signatures` | — |
 | `CLICKHOUSE_TOKEN_OWNER_ACTIVITY_TABLE` | `default.token_owner_activity` | — |
+| `CLICKHOUSE_TRANSFERS_TABLE` | `default.transfers` | Transfers ledger used by `getTransfersByAddress`. |
+| `CLICKHOUSE_TRANSFERS_BY_ADDRESS_TABLE` | — | Legacy alias for `CLICKHOUSE_TRANSFERS_TABLE`. |
 
 Shard routing:
 When `CLICKHOUSE_SCOPE=distributed`, superbank-rpc sends every ClickHouse query through `CLICKHOUSE_URL`. It does not read `CLICKHOUSE_TOPOLOGY_CONFIG`, discover `system.clusters`, connect to shard endpoints, query local tables, or validate local schemas. Explicit shard-local settings are ignored with a startup warning.
@@ -493,6 +514,7 @@ Additional env flags:
 
 - Scope: `superbank-rpc` applies `use_query_condition_cache=1` only on selected historical address-filtered reads:
   - `getTransactionsForAddress`
+  - `getTransfersByAddress`
   - the transactions-table fallback path for `getSignaturesForAddress`
 - Point lookups and slot-range reads do not opt in.
 - This setting is enabled separately from the query-result cache via:

crates/superbank-rpc/src/clickhouse/client.rs

@@ -27,6 +27,7 @@ use super::constants::DEFAULT_BUCKET_MODULUS;
 use super::gsfa::GsfaShardRouter;
 use super::queries::{
     GSFA_REQUIRED_COLUMNS, SIGNATURES_REQUIRED_COLUMNS, TOKEN_OWNER_REQUIRED_COLUMNS,
+    TRANSFERS_REQUIRED_COLUMNS,
 };
 use super::sharding::{
     BucketColumn, ClusterRow, DescribeTableRow, RoutingPolicy, RoutingScope, RoutingTransport,
@@ -416,11 +417,14 @@ pub struct ClickHouseClient {
     pub(crate) gsfa_hot_pubkeys: HashSet<Pubkey>,
     pub(crate) signature_statuses_table: String,
     pub(crate) token_owner_activity_table: String,
+    pub(crate) transfers_table: String,
     pub(crate) signatures_local_table: Option<String>,
     pub(crate) token_owner_activity_local_table: Option<String>,
+    pub(crate) transfers_local_table: Option<String>,
     pub(crate) transactions_local_table: Option<String>,
     pub(crate) blocks_metadata_local_table: Option<String>,
     pub(crate) token_owner_activity_available: bool,
+    pub(crate) transfers_available: bool,
     pub(crate) bucket_moduli: BucketModuli,
     pub(crate) allow_query_settings: bool,
     pub(crate) query_cache: QueryCacheConfig,
@@ -666,6 +670,9 @@ impl ClickHouseClient {
             .unwrap_or_else(|_| "default.signatures".to_string());
         let token_owner_activity_table = std::env::var("CLICKHOUSE_TOKEN_OWNER_ACTIVITY_TABLE")
             .unwrap_or_else(|_| "default.token_owner_activity".to_string());
+        let transfers_table = std::env::var("CLICKHOUSE_TRANSFERS_TABLE")
+            .or_else(|_| std::env::var("CLICKHOUSE_TRANSFERS_BY_ADDRESS_TABLE"))
+            .unwrap_or_else(|_| "default.transfers".to_string());
 
         let signatures_local_table = shard_routing.as_ref().and_then(|config| {
             config
@@ -679,6 +686,12 @@ impl ClickHouseClient {
                 .clone()
                 .or_else(|| derive_local_table_name(&token_owner_activity_table, None))
         });
+        let transfers_local_table = shard_routing.as_ref().and_then(|config| {
+            config
+                .transfers_local_table
+                .clone()
+                .or_else(|| derive_local_table_name(&transfers_table, None))
+        });
         let transactions_local_table = shard_routing.as_ref().and_then(|config| {
             config
                 .transactions_local_table
@@ -709,6 +722,9 @@ impl ClickHouseClient {
             if config.token_owner_activity_local_table.is_none() {
                 config.token_owner_activity_local_table = token_owner_activity_local_table.clone();
             }
+            if config.transfers_local_table.is_none() {
+                config.transfers_local_table = transfers_local_table.clone();
+            }
             if config.transactions_local_table.is_none() {
                 config.transactions_local_table = transactions_local_table.clone();
             }
@@ -734,11 +750,14 @@ impl ClickHouseClient {
             gsfa_hot_pubkeys: HashSet::new(),
             signature_statuses_table,
             token_owner_activity_table,
+            transfers_table,
             signatures_local_table,
             token_owner_activity_local_table,
+            transfers_local_table,
             transactions_local_table,
             blocks_metadata_local_table,
             token_owner_activity_available: true,
+            transfers_available: true,
             bucket_moduli: BucketModuli::default(),
             allow_query_settings: !std::env::var("CLICKHOUSE_DISABLE_QUERY_SETTINGS")
                 .map(|value| env_truthy(&value))
@@ -942,6 +961,14 @@ impl ClickHouseClient {
         self.allow_query_settings
     }
 
+    pub(crate) async fn with_timeout<T>(
+        &self,
+        operation: &'static str,
+        fut: impl std::future::Future<Output = ProcessingResult<T>>,
+    ) -> ProcessingResult<T> {
+        self.with_http_query_timeout(operation, fut).await
+    }
+
     pub(crate) async fn with_http_query_timeout<T>(
         &self,
         operation: &'static str,
@@ -951,9 +978,6 @@ impl ClickHouseClient {
             .await
     }
 
-    /// [`Self::with_http_query_timeout`] with an explicit deadline, for operations whose
-    /// budget differs from the interactive query timeout (e.g. disk-cache
-    /// backfill range scans).
     pub(crate) async fn with_http_query_timeout_duration<T>(
         &self,
         operation: &'static str,
@@ -1443,6 +1467,68 @@ impl ClickHouseClient {
             }
         }
 
+        let transfers_table = &self.transfers_table;
+        match self.startup_table_check {
+            ClickHouseStartupTableCheck::Count => {
+                match self
+                    .with_timeout("startup_transfers_count", async {
+                        self.client
+                            .query(&format!("SELECT COUNT(*) FROM {}", transfers_table))
+                            .fetch_one::<u64>()
+                            .await
+                            .map_err(|e| ProcessingError::database(e.to_string(), e))
+                    })
+                    .await
+                {
+                    Ok(count) => {
+                        self.transfers_available = true;
+                        tracing::info!(
+                            "📊 Database initialized - {} table: {} rows",
+                            transfers_table,
+                            count
+                        );
+                    }
+                    Err(e) => {
+                        self.transfers_available = false;
+                        tracing::warn!(
+                            "Transfers-by-address table '{}' unavailable; getTransfersByAddress disabled. Error: {}",
+                            transfers_table,
+                            e
+                        );
+                    }
+                }
+            }
+            ClickHouseStartupTableCheck::Exists => {
+                match self
+                    .with_timeout("startup_transfers_exists", async {
+                        self.client
+                            .query(&format!("SELECT count() FROM {} WHERE 0", transfers_table))
+                            .fetch_one::<u64>()
+                            .await
+                            .map(|_| ())
+                            .map_err(|e| ProcessingError::database(e.to_string(), e))
+                    })
+                    .await
+                {
+                    Ok(()) => {
+                        self.transfers_available = true;
+                        tracing::info!(
+                            "📊 Database initialized - {} table accessible",
+                            transfers_table
+                        );
+                    }
+                    Err(e) => {
+                        self.transfers_available = false;
+                        tracing::warn!(
+                            "Transfers-by-address table '{}' unavailable; getTransfersByAddress disabled. Error: {}",
+                            transfers_table,
+                            e
+                        );
+                    }
+                }
+            }
+        }
+
         let blocks_metadata_table = &self.blocks_metadata_table;
         match self.startup_table_check {
             ClickHouseStartupTableCheck::Count => {
@@ -1634,6 +1720,22 @@ impl ClickHouseClient {
                         self.token_owner_activity_local_table = None;
                     }
 
+                    if let Some(local_table) = config.transfers_local_table.clone()
+                        && let Err(e) = validate_table_schema_on_shards(
+                            topology.as_ref(),
+                            &local_table,
+                            &TRANSFERS_REQUIRED_COLUMNS,
+                            self.query_timeout,
+                        )
+                        .await
+                    {
+                        tracing::warn!(
+                            "Transfers shard routing disabled; local table validation failed: {}",
+                            e
+                        );
+                        self.transfers_local_table = None;
+                    }
+
                     if hot_routing_configured {
                         validate_table_schema_on_shards(
                             topology.as_ref(),
@@ -2157,6 +2259,7 @@ mod tests {
                     token_owner_activity_local_table: Some(
                         "default.token_owner_activity_local".to_string(),
                     ),
+                    transfers_local_table: Some("default.transfers_local".to_string()),
                     transactions_local_table: Some("default.transactions_local".to_string()),
                     blocks_metadata_local_table: Some("default.blocks_metadata_local".to_string()),
                 }),
@@ -2198,6 +2301,7 @@ mod tests {
                     gsfa_local_table: None,
                     signatures_local_table: None,
                     token_owner_activity_local_table: None,
+                    transfers_local_table: None,
                     transactions_local_table: None,
                     blocks_metadata_local_table: None,
                 }),
@@ -2457,6 +2561,7 @@ nodes:
             gsfa_local_table: None,
             signatures_local_table: None,
             token_owner_activity_local_table: None,
+            transfers_local_table: None,
             transactions_local_table: None,
             blocks_metadata_local_table: None,
         };

crates/superbank-rpc/src/clickhouse/mod.rs

@@ -13,17 +13,19 @@ mod rows;
 mod sharding;
 mod signatures;
 mod transactions;
+mod transfers;
 mod types;
 mod util;
 
 pub use client::{ClickHouseClient, ClickHouseClientOptions, InflationRewardQueryLimits};
 #[allow(unused_imports)]
 pub use types::TransactionsForAddressRecord;
 pub use types::{
-    BlockMetadataRecord, NumericFilter, PaginationToken, QueryTimings, SignatureFilter,
-    SignatureRecord, SignatureStatusRecord, SortOrder, StoredAccountsTransactionRecord,
+    BlockMetadataRecord, NumericFilter, PaginationToken, QueryTimings, RawAmount, SignatureFilter,
+    SignatureRecord, SignatureStatusRecord, SolMode, SortOrder, StoredAccountsTransactionRecord,
     StoredBlockPayload, StoredBlockRecord, StoredTransactionRecord, TokenAccountsFilter,
-    TransactionStatusFilter, TransactionsForAddressQuery,
+    TokenTransferTypes, TransactionStatusFilter, TransactionsForAddressQuery,
+    TransferDirectionFilter, TransferPositionFilter, TransferRecord, TransfersByAddressQuery,
 };
 
 pub(crate) use types::{