Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 47 additions & 4 deletions .github/workflows/build-binary.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@ name: Build Binaries
# downloads its artifact and runs `node scripts/smoke-binary.ts`, so it needs
# neither `npm ci` nor Bun.
#
# Successful main builds publish the smoke-tested Linux x64 binary to the
# dedicated nibrun-latest prerelease. Pull requests keep binaries as short-lived
# workflow artifacts only.
#
# The build job's `npm run build:binary` runs `npm run build:web`, which fetches
# https://oomol.com/en/apps/catalog.json (the provider icon map) at build time,
# so an outage there fails the build job. This is the first PR-gated path
Expand All @@ -40,13 +44,14 @@ on:
pull_request:
workflow_dispatch:

# Cancel superseded runs for the same ref (e.g. new push to an open PR).
# Cancel superseded PR runs. Main builds update a rolling release asset, so one
# must finish before the next main build starts.
concurrency:
group: build-binary-${{ github.ref }}
cancel-in-progress: true
cancel-in-progress: ${{ github.event_name == 'pull_request' }}

# Least privilege: these jobs only read the repository; the binaries are
# workflow artifacts, nothing is published.
# Least privilege by default. The nibrun release job grants itself write access
# only while updating its dedicated prerelease after a main build.
permissions:
contents: read

Expand Down Expand Up @@ -121,6 +126,9 @@ jobs:
- name: Smoke test linux-x64 (SQLite)
run: node scripts/smoke-binary.ts dist/open-connector-linux-x64

- name: Smoke test linux-x64 (nibrun)
run: node scripts/smoke-binary.ts dist/open-connector-linux-x64 --nibrun

# PostgreSQL mode: the binary's own `migrate` subcommand applies the embedded
# postgresql/ migrations, then the server must pass its schema check.
- name: Migrate PostgreSQL with linux-x64
Expand Down Expand Up @@ -245,3 +253,38 @@ jobs:

- name: Smoke test ${{ matrix.target }}
run: node scripts/smoke-binary.ts dist/${{ matrix.file }}

publish-nibrun:
name: Publish nibrun binary
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
needs: [build, smoke]
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- name: Download linux-x64 binary
uses: actions/download-artifact@v8
with:
name: binary-linux-x64
path: dist

- name: Publish rolling prerelease
env:
GH_REPO: ${{ github.repository }}
GH_TOKEN: ${{ github.token }}
run: |
if gh release view nibrun-latest >/dev/null 2>&1; then
gh release upload nibrun-latest dist/open-connector-linux-x64 --clobber
gh api --method PATCH "repos/${GH_REPO}/git/refs/tags/nibrun-latest" \
--field sha="$GITHUB_SHA" \
--field force=true >/dev/null
gh release edit nibrun-latest \
--notes "Built from ${GITHUB_SHA}. This asset is replaced after every successful build on main."
else
gh release create nibrun-latest dist/open-connector-linux-x64 \
--target "$GITHUB_SHA" \
--title "nibrun deployment" \
--prerelease \
--latest=false \
--notes "Built from ${GITHUB_SHA}. This asset is replaced after every successful build on main."
fi
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ OpenConnector is an open-source connector gateway for AI agents and an alternati
Connect user app accounts once, then expose a shared catalog of 1,000+ providers and 10,000+
prebuilt Actions to agents and applications.

[![Deploy on nibrun](https://nibrun.com/button.svg)](https://app.nibrun.com/deploy?name=open-connector&binary=https%3A%2F%2Fgithub.com%2Foomol-lab%2Fopen-connector%2Freleases%2Fdownload%2Fnibrun-latest%2Fopen-connector-linux-x64&port=3000&env=OOMOL_CONNECT_ENCRYPTION_KEY&env=OOMOL_CONNECT_ADMIN_TOKEN&env=OOMOL_CONNECT_RUNTIME_TOKEN)

<table>
<tr>
<td width="33.33%" align="center"><img src="assets/deployment-options/oomol.svg" alt="OOMOL" width="140"></td>
Expand Down
33 changes: 33 additions & 0 deletions docs/single-binary.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,39 @@ PORT=3000 \
With SQLite, migrations are applied automatically when the database opens, exactly as with
`npm start`.

## Deploy on nibrun

The README's **Deploy on nibrun** button creates an app from the latest successful `main` build of
the Linux x64 binary. It asks for the three secrets a public deployment should use:

- `OOMOL_CONNECT_ENCRYPTION_KEY`
- `OOMOL_CONNECT_ADMIN_TOKEN`
- `OOMOL_CONNECT_RUNTIME_TOKEN`

Generate long random values, save them in a password manager, and keep the encryption key stable
across redeployments. Losing that key makes encrypted credentials in the persistent database
unreadable.

OpenConnector automatically follows nibrun's runtime contract: it listens on the assigned `PORT`
at `0.0.0.0`, stores SQLite and transit-file data under `NIBRUN_DATA_DIR`, and uses
`https://$NIBRUN_HOSTNAME` as its public origin. Explicit `HOST`, `OOMOL_CONNECT_DATA_DIR`, and
`OOMOL_CONNECT_ORIGIN` values still take precedence.

To deploy with the CLI instead, install and authenticate `nib`, then run:

```bash
nib run \
https://github.com/oomol-lab/open-connector/releases/download/nibrun-latest/open-connector-linux-x64 \
--name open-connector \
--port 3000 \
--env OOMOL_CONNECT_ENCRYPTION_KEY="$OOMOL_CONNECT_ENCRYPTION_KEY" \
--env OOMOL_CONNECT_ADMIN_TOKEN="$OOMOL_CONNECT_ADMIN_TOKEN" \
--env OOMOL_CONNECT_RUNTIME_TOKEN="$OOMOL_CONNECT_RUNTIME_TOKEN"
```

For later deployments, replace `--name open-connector` with `--app <app-slug>` so nibrun updates
the existing app and preserves its environment and data.

### PostgreSQL Migrations

PostgreSQL migrations are explicit. The binary has a `migrate` subcommand that applies the embedded
Expand Down
67 changes: 57 additions & 10 deletions scripts/smoke-binary.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,9 @@ import { setTimeout as sleep } from "node:timers/promises";
// Start a built single-file executable against a fresh data directory and check that its embedded catalog,
// web console, migrations and shutdown path work.
//
// Usage: `node scripts/smoke-binary.ts <path-to-binary>`. Set OOMOL_CONNECT_DATABASE_URL to run the PostgreSQL
// mode; every other OOMOL_CONNECT_* variable is removed from the server's environment. Uses only Node built-ins so
// the smoke runners need no `npm ci`.
// Usage: `node scripts/smoke-binary.ts <path-to-binary> [--nibrun]`. Set OOMOL_CONNECT_DATABASE_URL to run the
// PostgreSQL mode; every other OOMOL_CONNECT_* and NIBRUN_* variable is removed from the server's environment. Uses
// only Node built-ins so the smoke runners need no `npm ci`.

interface ProcessExit {
code: number | null;
Expand All @@ -25,12 +25,14 @@ interface ServerProcessOptions {
dataDir: string;
port: number;
databaseUrl: string | undefined;
nibrun: boolean;
}

const healthTimeoutMs = 60_000;
const healthPollIntervalMs = 500;
const requestTimeoutMs = 10_000;
const shutdownTimeoutMs = 10_000;
const nibrunHostname = "open-connector-smoke.nibrun.app";

/** The server binary with its captured output and exit state. */
class ServerProcess {
Expand Down Expand Up @@ -107,14 +109,15 @@ class ServerProcess {
}

const binaryPath = await resolveBinaryPath(process.argv[2]);
const nibrun = parseNibrunFlag(process.argv.slice(3));
const databaseUrl = process.env.OOMOL_CONNECT_DATABASE_URL?.trim() || undefined;
const mode = databaseUrl ? "postgresql" : "sqlite";
const startedAt = performance.now();
// Probe the port before creating the data directory so a probe failure cannot leak the temp directory.
const port = await findFreePort();
const dataDir = await mkdtemp(join(tmpdir(), "open-connector-smoke-"));
const baseUrl = `http://127.0.0.1:${port}`;
const server = new ServerProcess({ binaryPath, dataDir, port, databaseUrl });
const server = new ServerProcess({ binaryPath, dataDir, port, databaseUrl, nibrun });

try {
await waitForHealth(server, baseUrl);
Expand All @@ -123,11 +126,14 @@ try {
await checkConsoleAssets(baseUrl, indexHtml);
await checkProviders(baseUrl);
await checkApps(baseUrl);
if (nibrun) {
await checkNibrunRuntime(server, baseUrl, port);
}
await checkDatabaseBackend(server, dataDir, databaseUrl);
await checkGracefulShutdown(server);
await removeDataDir(dataDir);
console.log(
`PASS ${binaryPath} mode=${mode} startup=${formatMs(readyAt - startedAt)} total=${formatMs(performance.now() - startedAt)}`,
`PASS ${binaryPath} mode=${mode}${nibrun ? " deployment=nibrun" : ""} startup=${formatMs(readyAt - startedAt)} total=${formatMs(performance.now() - startedAt)}`,
);
} catch (error) {
console.error(`FAIL ${binaryPath}: ${error instanceof Error ? error.message : String(error)}`);
Expand All @@ -141,7 +147,7 @@ try {
/** Resolve the binary before spawning: spawn resolves relative paths against the child's cwd, which is the data dir. */
async function resolveBinaryPath(argument: string | undefined): Promise<string> {
if (!argument) {
console.error("Usage: node scripts/smoke-binary.ts <path-to-binary>");
console.error("Usage: node scripts/smoke-binary.ts <path-to-binary> [--nibrun]");
process.exit(1);
}

Expand All @@ -156,18 +162,37 @@ async function resolveBinaryPath(argument: string | undefined): Promise<string>
return resolved;
}

/** process.env without OOMOL_CONNECT_* so the caller's shell cannot leak configuration into the server under test. */
function parseNibrunFlag(arguments_: string[]): boolean {
if (arguments_.length === 0) {
return false;
}
if (arguments_.length === 1 && arguments_[0] === "--nibrun") {
return true;
}

console.error("Usage: node scripts/smoke-binary.ts <path-to-binary> [--nibrun]");
process.exit(1);
}

/** Remove app and platform configuration so the caller's shell cannot leak it into the server under test. */
function buildServerEnvironment(options: ServerProcessOptions): NodeJS.ProcessEnv {
const env: NodeJS.ProcessEnv = {};
for (const [key, value] of Object.entries(process.env)) {
if (!key.toUpperCase().startsWith("OOMOL_CONNECT_")) {
const upperKey = key.toUpperCase();
if (!upperKey.startsWith("OOMOL_CONNECT_") && !upperKey.startsWith("NIBRUN_")) {
env[key] = value;
}
}

env.PORT = String(options.port);
env.HOST = "127.0.0.1";
env.OOMOL_CONNECT_DATA_DIR = options.dataDir;
if (options.nibrun) {
env.NIBRUN_HTTP_PORT = String(options.port);
env.NIBRUN_HOSTNAME = nibrunHostname;
env.NIBRUN_DATA_DIR = options.dataDir;
} else {
env.HOST = "127.0.0.1";
env.OOMOL_CONNECT_DATA_DIR = options.dataDir;
}
if (options.databaseUrl) {
env.OOMOL_CONNECT_DATABASE_URL = options.databaseUrl;
}
Expand Down Expand Up @@ -280,6 +305,28 @@ async function checkApps(baseUrl: string): Promise<void> {
assert(Array.isArray(data), "/v1/apps data is not an array");
}

async function checkNibrunRuntime(server: ServerProcess, baseUrl: string, port: number): Promise<void> {
assert(
server.stdout.includes(`"url":"http://0.0.0.0:${port}"`),
"server did not bind to 0.0.0.0 under NIBRUN_HOSTNAME",
);

const form = new FormData();
form.set("file", new File(["nibrun smoke"], "smoke.txt", { type: "text/plain" }));
const response = await fetch(`${baseUrl}/api/files`, {
method: "POST",
body: form,
signal: AbortSignal.timeout(requestTimeoutMs),
});
const body: unknown = await response.json();
assert(response.status === 200, `POST /api/files returned ${response.status}`);
assert(isRecord(body), "POST /api/files did not return a JSON object");
assert(
typeof body.downloadUrl === "string" && body.downloadUrl.startsWith(`https://${nibrunHostname}/api/files/`),
`POST /api/files returned unexpected downloadUrl ${JSON.stringify(body.downloadUrl)}`,
);
}

/** Return `data` of a `{ success: true, data }` envelope. */
async function fetchEnvelopeData(baseUrl: string, path: string): Promise<unknown> {
const response = await fetch(`${baseUrl}${path}`, { signal: AbortSignal.timeout(requestTimeoutMs) });
Expand Down
8 changes: 5 additions & 3 deletions src/server/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,12 @@ import {
} from "./storage/node-runtime-database.ts";
import { DEFAULT_RUN_LIMIT } from "./storage/runtime-store.ts";

const nibrunHostname = optionalEnv("NIBRUN_HOSTNAME");
const port = Number(process.env.PORT ?? 3000);
const hostname = process.env.HOST ?? "127.0.0.1";
const publicOrigin = process.env.OOMOL_CONNECT_ORIGIN ?? `http://localhost:${port}`;
const dataDir = process.env.OOMOL_CONNECT_DATA_DIR ?? join(process.cwd(), "data");
const hostname = process.env.HOST ?? (nibrunHostname ? "0.0.0.0" : "127.0.0.1");
const publicOrigin =
process.env.OOMOL_CONNECT_ORIGIN ?? (nibrunHostname ? `https://${nibrunHostname}` : `http://localhost:${port}`);
const dataDir = process.env.OOMOL_CONNECT_DATA_DIR ?? optionalEnv("NIBRUN_DATA_DIR") ?? join(process.cwd(), "data");
const transitFileTtlSeconds = readPositiveIntegerEnv("OOMOL_CONNECT_TRANSIT_FILE_TTL_SECONDS", 86_400);
const transitFileMaxBytes = readPositiveIntegerEnv("OOMOL_CONNECT_TRANSIT_FILE_MAX_BYTES", 100 * 1024 * 1024);
const runLimit = readPositiveIntegerEnv("OOMOL_CONNECT_RUN_LIMIT", DEFAULT_RUN_LIMIT);
Expand Down