Skip to content
Open
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
The diff you're trying to view is too large. We only load the first 3000 changed files.
9 changes: 9 additions & 0 deletions .github/workflows/deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,20 @@ jobs:
ruby-version: '2.7'
bundler-cache: true

- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'

- name: Install npm dependencies (Biga + Sourdough for import_file)
run: npm ci

- name: Deploy
env:
AWS_REGION: us-east-1
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is not set, and I don't think we should set it. if we need github auth, we can use the builtin github actions permission model, but I don't think it's necessary here, and it wouldn't have any special permissions to the repositories that host packages.

@ellenfogarty-dbt ellenfogarty-dbt Apr 20, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As above, I'll remove this now too

run: |
bundle exec middleman build
aws s3 cp --recursive build/ s3://com.getdbt.hub/
Expand Down
2 changes: 2 additions & 0 deletions .npmrc
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
@dbt-labs:registry=https://npm.pkg.github.com
//npm.pkg.github.com/:_authToken=${GITHUB_TOKEN}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need the github token to be able to pull in biga as well?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure on this, I think I have updated this now to use the builtin github actions for this.

47 changes: 47 additions & 0 deletions config.rb
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
require 'open-uri'
require 'json'
require 'time'

class Middleman::Util::EnhancedHash
# Mila: Looked into Hashie::Mash's source code where this class is
Expand Down Expand Up @@ -151,6 +153,45 @@ def combine_packages(packages)

set :package_index, combine_packages(@app.data.packages)

# Fetches repository metadata from the GitHub API.
# Returns an empty hash on any error so callers can treat missing fields as absent.
def fetch_github_repo(org, repo, token = nil)
url = "https://api.github.com/repos/#{org}/#{repo}"
options = {
'Accept' => 'application/vnd.github+json',
'X-GitHub-Api-Version' => '2022-11-28',
'User-Agent' => 'hub.getdbt.com-middleman-build'
}
options['Authorization'] = "Bearer #{token}" if token
JSON.parse(URI.open(url, options).read)
rescue => e
warn "[hub] GitHub API fetch failed for #{org}/#{repo}: #{e.message}"
{}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like the idea that the site will render differently based on whether the github api was up or down during the time that we built it. IMO this should fail hard!

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've removed the 'rescue' block here. Claude suggested adding in timeouts so these have been added in as follows:
open_timeout: 5,
read_timeout: 5

end

# Converts a GitHub ISO-8601 timestamp (e.g. "2025-12-11T14:23:45Z") to
# a human-readable label like "Updated Dec 2025".
def format_pushed_at(iso_string)
return nil if iso_string.nil? || iso_string.empty?
Time.parse(iso_string).strftime('Updated %b %Y')
rescue ArgumentError
nil
end

# Fetch live description + last-push date for each featured package at build time.
# Falls back gracefully to the values in data/featured.json if the API is unavailable.
github_token = ENV['GITHUB_TOKEN']

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this won't work in CI/CD, we don't make a github token available to the build process. shouldn't these be public repositories anyway? why do we need a github token?

@ellenfogarty-dbt ellenfogarty-dbt Apr 20, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks - I'll remove the github token now

featured_live = {}
@app.data.featured.each do |feat|
repo = fetch_github_repo(feat['org'], feat['package'], github_token)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this slow down the build process significantly?

@ellenfogarty-dbt ellenfogarty-dbt Apr 20, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TBH I'm not sure on this one. My knowledge is pretty limited on this. I've ran it past Claude and gotten the following response - keen to know your thoughts:

The more meaningful risk is reliability, not speed — if GitHub's API is slow or down, each failed request waits for a timeout before moving on. The rescue block in fetch_github_repo does catch errors and fall back gracefully, but depending on what timeout URI.open applies by default (Ruby doesn't set one), a hung connection could stall the build noticeably.

You could add an explicit read timeout to make the fallback snappy:

def fetch_github_repo(org, repo)
url = "https://api.github.com/repos/#{org}/#{repo}"
options = {
'Accept' => 'application/vnd.github+json',
'X-GitHub-Api-Version' => '2022-11-28',
'User-Agent' => 'hub.getdbt.com-middleman-build',
open_timeout: 5,
read_timeout: 5
}
JSON.parse(URI.open(url, options).read)
rescue => e
warn "[hub] GitHub API fetch failed for #{org}/#{repo}: #{e.message}"
{}
end

live = {
'description' => repo['description'],
'updated' => format_pushed_at(repo['pushed_at']),
}.compact
featured_live["#{feat['org']}/#{feat['package']}"] = live
end
set :featured_live, featured_live

after_configuration do

proxy "/api/v1/packages.json",
Expand Down Expand Up @@ -220,6 +261,12 @@ def combine_packages(packages)
ignore '/api/v1/package.template.json.erb'
ignore '/api/v1/raw.json.erb'

# Vendor CSS from npm (Biga tokens must load before Sourdough). Requires `npm install`.
import_file File.expand_path('node_modules/@dbt-labs/biga/tokens/tokens.css', root),
'/stylesheets/vendor/biga-tokens.css'
import_file File.expand_path('node_modules/@dbt-labs/sourdough/dist/sourdough.css', root),
'/stylesheets/vendor/sourdough.css'

activate :livereload

configure :development do
Expand Down
36 changes: 30 additions & 6 deletions data/featured.json
Original file line number Diff line number Diff line change
@@ -1,26 +1,50 @@
[
{
"org": "dbt-labs",
"package": "dbt_utils"
"package": "dbt_utils",
"tile": {
"description": "Utility functions for dbt projects",
"updated": "Updated Dec 2025"
}
},
{
"org": "dbt-labs",
"package": "codegen"
"package": "codegen",
"tile": {
"description": "Macros that generate dbt code",
"updated": "Updated Nov 2025"
}
},
{
"org": "dbt-labs",
"package": "dbt_project_evaluator"
"package": "dbt_project_evaluator",
"tile": {
"description": "This package contains macros and models to find DAG issues automatically",
"updated": "Updated Feb 2026"
}
},
{
"org": "dbt-labs",
"package": "audit_helper"
"package": "audit_helper",
"tile": {
"description": "Useful macros when performing data audits",
"updated": "Updated Mar 2026"
}
},
{
"org": "dbt-labs",
"package": "dbt_external_tables"
"package": "dbt_external_tables",
"tile": {
"description": "dbt macros to stage external source",
"updated": "Updated Mar 2026"
}
},
{
"org": "metaplane",
"package": "dbt_expectations"
"package": "dbt_expectations",
"tile": {
"description": "Great Expectations tests for dbt models",
"updated": "Updated Dec 2025"
}
}
]
1 change: 1 addition & 0 deletions node_modules/.bin/ansi-to-html

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions node_modules/.bin/cssesc

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions node_modules/.bin/csv2json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions node_modules/.bin/csv2tsv

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions node_modules/.bin/dsv2dsv

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions node_modules/.bin/dsv2json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions node_modules/.bin/jiti

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions node_modules/.bin/jsesc

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions node_modules/.bin/json2csv

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions node_modules/.bin/json2dsv

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions node_modules/.bin/json2tsv

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions node_modules/.bin/loose-envify

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions node_modules/.bin/mini-svg-data-uri

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions node_modules/.bin/nanoid

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions node_modules/.bin/parser

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions node_modules/.bin/resolve

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions node_modules/.bin/sucrase

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions node_modules/.bin/sucrase-node

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions node_modules/.bin/tailwind

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions node_modules/.bin/tailwindcss

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions node_modules/.bin/tsv2csv

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions node_modules/.bin/tsv2json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading