Skip to content

Latest commit

 

History

History
460 lines (320 loc) · 18.7 KB

streaming.md

File metadata and controls

460 lines (320 loc) · 18.7 KB

Streaming

This lib supports streaming for v1 and v2 API. The v1.1 streaming endpoints are no longer available on most X API plans, so their methods are deprecated: use the v2 ones.

Using streaming

For both V1 and V2 APIs, streaming methods returns a TweetStream object.

Each event of TweetStream is stored into a TypeScript enum.

import { ETwitterStreamEvent, TwitterApi } from 'twitter-api-v2';

const client = new TwitterApi(); // (create a Bearer OAuth2 client)

// Create the stream without connecting it, to set its listeners and connection options first
const stream = client.v2.searchStream({ autoConnect: false });

stream.on(
  // Emitted when Node.js {response} emits a 'error' event (contains its payload).
  ETwitterStreamEvent.ConnectionError,
  err => console.log('Connection error!', err),
);

stream.on(
  // Emitted on .close(), or when the connection is lost and won't be retried.
  ETwitterStreamEvent.ConnectionClosed,
  () => console.log('Connection has been closed.'),
);

stream.on(
  // Emitted when a Twitter payload (a tweet or not, given the endpoint).
  ETwitterStreamEvent.Data,
  eventData => console.log('Twitter has sent something:', eventData),
);

stream.on(
  // Emitted when a Twitter sent a signal to maintain connection active
  ETwitterStreamEvent.DataKeepAlive,
  () => console.log('Twitter has a keep-alive packet.'),
);

// Connect. With autoReconnect, a failed first connection is retried too,
// and the stream reconnects by itself when the connection is lost
await stream.connect({ autoReconnect: true, autoReconnectRetries: 'unlimited' });

// Be sure to close the stream when you don't want to consume data from it anymore:
// X allows a single connection to the filtered stream per app
process.once('SIGTERM', () => stream.close());

See Keep a stream running for how reconnections work.

Iterate with for await

You can also iterate over the tweets with an async iterator:

const stream = client.v2.searchStream({ autoConnect: false });
await stream.connect({ autoReconnect: true, autoReconnectRetries: 'unlimited' });

for await (const { data } of stream) {
  console.log('This is my tweet:', data);
}
// Reached once stream.close() has been called
  • The loop gets every tweet in order, including the ones received while its body was running.
  • With autoReconnect, it goes on across reconnections. It ends when stream.close() or stream.destroy() is called, and throws an error if the reconnection attempts run out (ReconnectLimitExceeded).
  • Without autoReconnect, it throws the connection error when the connection is lost, after yielding the tweets already received.
  • It also throws when X sends a payload that can't be parsed (TweetParseError). The stream stays connected: start a new loop, or close it.
  • Leaving the loop (break, return or an exception) doesn't close the stream. Call stream.close() when you're done with it.
  • On a stream that is already closed, the loop ends right away.

Note: Use streaming without auto-connection feature

You can create streams that doesn't connect immediately. This leads to the advantage of that endpoint wrappers will directly returns a TweetStream object, not wrapped in a Promise.

Give autoConnect: false in object parameters to disable auto-connect.

Call .connect() to start stream.

// Not needed to await this!
const stream = client.v2.sampleStream({ autoConnect: false });

// Assign yor event handlers
// Emitted on Tweet
stream.on(ETwitterStreamEvent.Data, console.log);
// Emitted only on initial connection success
stream.on(ETwitterStreamEvent.Connected, () => console.log('Stream is started.'));

// Start stream!
await stream.connect({ autoReconnect: true, autoReconnectRetries: 'unlimited' });

Keep a stream running

The first connection

await client.v2.searchStream() connects right away, with autoReconnect off. If X refuses this first connection, the promise rejects and nothing retries it. Setting stream.autoReconnect = true afterwards only covers the connections lost later.

To retry the first connection too, create the stream with autoConnect: false, then call connect({ autoReconnect: true }), like in Using streaming.

Reconnections and retries

With autoReconnect, the stream reconnects when X closes the connection, on a network error, and when nothing has been received for keepAliveTimeout milliseconds (2 minutes by default, while X sends a keep-alive every 20 seconds).

  • The first reconnection attempt is made right away. After the n-th failed attempt, the stream waits nextRetryTimeout(n, error) milliseconds before the next one. The default delays are 5s, 15s, 30s, 1min, 1min30, 2min, 3min, 5min, 10min, then 15min.
  • After autoReconnectRetries failed attempts (5 by default), the stream gives up: it emits ReconnectLimitExceeded, then ConnectionClosed. Pass autoReconnectRetries: 'unlimited' to connect() to never give up, which is usually what a long-running bot wants.
  • Once a reconnection succeeds, the count starts over for the next lost connection.
  • close() and destroy() also stop a reconnection that is running or scheduled.

X recommends exponential backoff, starting at 5 seconds for HTTP errors and at 1 minute for 429 errors (X docs). nextRetryTimeout gets the error, so you can follow this advice:

import { ApiResponseError } from 'twitter-api-v2';

await stream.connect({
  autoReconnect: true,
  autoReconnectRetries: 'unlimited',
  nextRetryTimeout: (tryOccurrence, error) => {
    // Starts at 1 minute for a 429 error and at 5 seconds otherwise, doubles each time, up to 15 minutes
    const firstDelay = error instanceof ApiResponseError && error.code === 429 ? 60 * 1000 : 5 * 1000;
    return Math.min(firstDelay * 2 ** (tryOccurrence - 1), 15 * 60 * 1000);
  },
});

One connection per app

On pay-per-use plans, X allows one connection to the filtered stream per app (X docs). Another connection is refused with a 429 error, whose data.connection_issue is 'TooManyConnections'. That happens when an older connection is still open: another instance of your app, a stream you didn't close, or the connection of a process that crashed, which X keeps open for a while.

  • Close the stream (close() or destroy()) when you stop using it, f.e. on shutdown.
  • Create one stream and let it reconnect, instead of creating a new stream after each disconnection.
  • await client.v2.terminateAllStreamConnections(), with your Bearer OAuth2 client, closes the open connections of your app (see Streaming connections).

Bots that reply

Since February 2026, X only lets apps on self-serve plans reply to a post when its author has summoned the app's account, by mentioning it or quoting one of its posts. Other replies are rejected (X changelog). So a bot that replies to the tweets of its stream should use a rule that matches those posts, like @yourbot, rather than reply to every tweet that matches a keyword.

Specific API v1.1 implementations

Deprecated: these endpoints are no longer available on most X API plans. Use the v2 search stream instead.

API v1.1 streaming-related endpoints works only with classic OAuth 1.0a authentication.

Filter endpoint

Method: v1.filterStream.

Endpoint: statuses/filter.json.

Reference: https://developer.x.com/en/docs/twitter-api/v1/tweets/filter-realtime/api-reference/post-statuses-filter

Level: Read-only.

const client = ...; // (create a OAuth 1.0a client)

const streamFilter = await client.v1.filterStream({
  // See FilterStreamParams interface.
  track: 'JavaScript',
  follow: [1842984n, '1850485928354'],
});

// Event data will be tweets of v1 API.

Sample endpoint

Method: v1.sampleStream.

Endpoint: statuses/sample.json.

Reference: https://developer.x.com/en/docs/twitter-api/v1/tweets/sample-realtime/api-reference/get-statuses-sample

Level: Read-only.

const client = ...; // (create a OAuth 1.0a client)

const stream = await client.v1.sampleStream();

// Event data will be tweets of v1 API.

Specific API v2 implementations

API v2 streaming-related endpoints works only with Bearer OAuth2 authentication.

Search endpoint

Method: v2.searchStream.

Endpoint: tweets/search/stream.

Reference: https://docs.x.com/x-api/stream/stream-filtered-posts

Level: Read-only.

const client = ...; // (create a Bearer OAuth2 client)

const stream = await client.v2.searchStream();

// Event data will be tweets of v2 API.

Search endpoint - Get applied rules

Method: v2.streamRules.

Endpoint: tweets/search/stream/rules (GET).

Reference: https://docs.x.com/x-api/stream/get-stream-rules

Level: Read-only.

Returns: StreamingV2GetRulesResult.

You can limit the number of rules returned per page with max_results (min 1, max 1000, default 1000) and paginate using pagination_token.

const client = ...; // (create a Bearer OAuth2 client)

// Retrieve up to 500 rules per page
const rules = await client.v2.streamRules({ max_results: 500 });

// Fetch next page if a next_token is present
if (rules.meta.next_token) {
  const nextPage = await client.v2.streamRules({
    pagination_token: rules.meta.next_token,
    max_results: 500,
  });
}

// Log every rule ID (X leaves out `data` when there are no rules)
console.log((rules.data ?? []).map(rule => rule.id));

Search endpoint - Add or delete rules

Method: v2.updateStreamRules.

Endpoint: tweets/search/stream/rules (POST).

Reference: https://docs.x.com/x-api/stream/update-stream-rules

Level: Read-write.

Takes: StreamingV2UpdateRulesParams. Returns: StreamingV2UpdateRulesResult.

const client = ...; // (create a Bearer OAuth2 client)

// Add rules
const addedRules = await client.v2.updateStreamRules({
  add: [
    { value: 'JavaScript', tag: 'js' },
    { value: 'TypeScript', tag: 'ts' },
  ],
});

// Delete rules
const deleteRules = await client.v2.updateStreamRules({
  delete: {
    ids: ['281646', '1534843'],
  },
});

// Delete all rules
await client.v2.updateStreamRules({}, { delete_all: true });

Search endpoint - Count rules

Method: v2.streamRuleCounts.

Endpoint: tweets/search/stream/rules/counts (GET).

Reference: https://docs.x.com/x-api/stream/get-stream-rule-counts

Level: App-only (Bearer auth).

Returns: StreamingV2RuleCountsResult.

const client = ...; // (create a Bearer OAuth2 client)

const { data: counts } = await client.v2.streamRuleCounts();
console.log(`${counts.project_rules_count} rules, ${counts.cap_per_project} at most`);

Sample endpoint

Only available on X Enterprise plans (X docs).

Method: v2.sampleStream.

Endpoint: tweets/sample/stream.

Reference: https://docs.x.com/x-api/stream/stream-sampled-posts

Level: Read-only.

const client = ...; // (create a Bearer OAuth2 client)

const stream = await client.v2.sampleStream();

// Event data will be tweets of v2 API.

Sample 10% endpoint

Streams approximately 10% of the public tweet volume in real time.

Only available on X Enterprise plans (X docs).

Method: v2.sample10Stream().

Endpoint: tweets/sample10/stream.

Reference: https://docs.x.com/x-api/stream/stream-10-sampled-posts

Level: Read-only.

Arguments: options?: Partial<Tweetv2FieldsParams> & { autoConnect?: boolean }.

Returns: TweetStream<TweetV2SingleResult>.

const stream = await client.v2.sample10Stream();
for await (const { data } of stream) {
  console.log(data.id);
}
// or create a non-auto-connected stream:
const stream = client.v2.sample10Stream({ autoConnect: false });
stream.on(ETwitterStreamEvent.Data, console.log);
await stream.connect({ autoReconnect: true });

Streaming connections

List the streaming connections of your app (active, past or both), and close them. Useful when a stream fails to connect because a previous connection is still open.

Methods: v2.streamConnections, v2.terminateAllStreamConnections and v2.terminateStreamConnections.

Endpoints: connections (GET), connections/all (DELETE) and connections/:endpoint_id (DELETE).

Reference: https://docs.x.com/x-api/connections/get-connection-history

Level: App-only (Bearer auth).

Takes: StreamingV2ConnectionsParams (for streamConnections) or a TStreamingV2ConnectionEndpoint, like 'filtered_stream' (for terminateStreamConnections). Returns: StreamingV2ConnectionsResult or StreamingV2TerminateConnectionsResult.

const client = ...; // (create a Bearer OAuth2 client)

const { data: connections } = await client.v2.streamConnections({ status: 'active' });

// Close the connections to the filtered stream, or every connection of the app
await client.v2.terminateStreamConnections('filtered_stream');
await client.v2.terminateAllStreamConnections();

Make a custom request

If you know endpoint and parameters (or you don't want them to be parsed), you can make raw requests using shortcuts by HTTP methods:

  • getStream()
  • postStream() or using raw request handler:
  • sendStream()

NOTE: Be careful to select the good API prefix for version 1.1. 1.1 does not use the same URL for classic endpoints and streaming endpoints. You can access quicky to an instance with the streaming prefix using v1.stream.

// For v1 (deprecated)
const streamFilter = await client.v1.stream.getStream('statuses/filter.json', { track: 'JavaScript,TypeScript' });
// For v2
const sampleFilterv2 = await client.v2.getStream('tweets/sample/stream');

TweetStream reference

Methods / properties

  • .autoReconnect: boolean / defaults false / Set this to true to enable reconnect feature.
  • .autoReconnectRetries: number / default 5 / If autoReconnect is true, number of failed reconnection attempts before giving up. Each try is spaced by return of .nextRetryTimeout() call milliseconds. Infinity never gives up (autoReconnectRetries: 'unlimited' in .connect()). See Reconnections and retries.
  • .keepAliveTimeoutMs: number / default 120000 (2 minutes) / Defined whenever connection should be automatically closed if nothing is received from Twitter during this time (it should not happen in any situation, because Twitter sends keep-alive packets). Can be set to Infinity to disable this feature.
  • .nextRetryTimeout: TStreamConnectRetryFn / Override this function that takes a tryOccurrence (starting from 1) and the error of the failed attempt, and returns the number of milliseconds to wait before trying to reconnect to Twitter.
  • .close(): Emits ConnectionClosed event and terminates connection. Also stops a reconnection that is running or scheduled, and ends the for await loops on the stream.
  • .destroy(): Same as close(), but unbind all registered event listeners before.
  • .clone(): Promise<TweetStream>: Returns a new TweetStream with the same request parameters, with the same event listeners bound.
  • .reconnect(): Promise<void>: Tries to make a new request to Twitter with the same original parameters. If successful, continue streaming with new response. It also reopens a closed stream.
  • .connect(params?: IConnectTweetStreamParams): Promise<TweetStream>: Connect the stream. Only if autoConnect has been set to false when stream is created. Params: autoReconnect, autoReconnectRetries (a number or 'unlimited'), keepAliveTimeout (milliseconds or 'disable') and nextRetryTimeout.

Events

All events are part of enum ETwitterStreamEvent exported by the package.

  • .ConnectionError: Emitted with the err parameter given by request.on('error') or response.on('error') handlers.
  • .ConnectionClosed: Emitted when .close() is called, or when the connection is lost and won't be retried (autoReconnect is off, or the retries ran out).
  • .ConnectionLost: When nothing is received from Twitter during .keepAliveTimeoutMs milliseconds, emit this event and start either close or reconnection process.
  • .ReconnectAttempt: Emitted before a reconnect attempt is made (payload: attempt number).
  • .Connected: Emitted when the initial connection attempt succeeds. (only using manual .connect() after creating a stream with autoConnect: false)
  • .ConnectError: Emitted when the initial connection has failed. Event data is 0, to match logic of .ReconnectError.
  • .Reconnected: Emitted when a reconnection attempt succeeds.
  • .ReconnectError: Emitted when a auto-reconnect try attempt has failed. Event data is a number representing the number of times the request has been re-made (starts from 0).
  • .ReconnectLimitExceeded: Emitted when .autoReconnectRetries limit exceeds.
  • .DataKeepAlive: Emitted when Twitter sends a \r\n to maintain connection open.
  • .Data: Emitted with stream data, when Twitter sends something.
  • .DataError: Emitted when Twitter sends a JSON error payload.
  • .TweetParseError: When the thing sent by Twitter cannot be JSON-parsed. Contains the parse error.
  • .Error: Emitted when a .ConnectionError, .TweetParseError, .DataError, .ConnectError or .ReconnectError occurs. Contains { type, error: any, message }, where type is the event that occurred.