Skip to content

chore: added s3 flag - #1061

Open
saksham-datazip wants to merge 25 commits into
stagingfrom
feat/added-s3-command
Open

chore: added s3 flag#1061
saksham-datazip wants to merge 25 commits into
stagingfrom
feat/added-s3-command

Conversation

@saksham-datazip

@saksham-datazip saksham-datazip commented Aug 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Description

This PR makes NFS optional for the UI/Helm workflow by adding S3-backed configuration handling for CLI commands.

When a command receives an s3:// path, the CLI now:

  • Detects the S3 URI.
  • Reads S3 credentials and configuration from environment variables.
  • Downloads the referenced configuration files to a local temporary directory.
  • Rewrites the command to use the downloaded local paths.
  • Executes the command normally without changing connector behavior.
  • Uploads generated or updated files (such as streams.json, state.json, stats.json, and difference_streams.json) back to the corresponding S3 job folder after successful execution.

This allows the UI and Helm deployment to use S3 as the shared storage backend instead of requiring NFS while keeping the connector execution flow unchanged.

Fixes # (issue)

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

  • Scenario A
  • Scenario B

Screenshots or Recordings

N/A

Documentation

  • Documentation Link: [link to README, olake.io/docs, or olake-docs]
  • N/A (bug fix, refactor, or test changes only)

Related PR's (If Any):

N/A

vikaxsh
vikaxsh reviewed Aug 5, 2026
View reviewed changes
Comment thread protocol/discover.go Outdated
},
RunE: func(cmd *cobra.Command, _ []string) error {
RunE: func(cmd *cobra.Command, _ []string) (err error) {
defer func() {

@vikaxsh vikaxsh Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what if we wrap this in a common fn and eliminate 3x defer

Comment thread protocol/s3.go Outdated
Comment on lines +21 to +33
if err := resolveS3PathFlag(ctx, &configPath); err != nil {
return err
}
if err := resolveS3PathFlag(ctx, &destinationConfigPath); err != nil {
return err
}
if err := resolveS3PathFlag(ctx, &streamsPath); err != nil {
return err
}
if err := resolveS3PathFlag(ctx, &statePath); err != nil {
return err
}
if err := resolveS3PathFlag(ctx, &differencePath); err != nil {

@vikaxsh vikaxsh Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use a loop here

@saksham-datazip saksham-datazip Aug 31, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed

Comment thread protocol/s3.go Outdated
local string
name string
}{
{filepath.Join(configFolder, "streams.json"), "streams.json"},

@vikaxsh vikaxsh Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wee can use constants.StreamsPath instead of rebuilding

Comment thread protocol/s3.go Outdated
@@ -0,0 +1,101 @@
package protocol

@vikaxsh vikaxsh Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we move these to utils? Since these aren't commands, it would be better to keep the protocol layer thin and shift as much of this logic as possible into utils

Comment thread utils/s3_utils.go Outdated
Comment on lines +37 to +41
accessKey := envFirst("OLAKE_S3_ACCESS_KEY_ID", "AWS_ACCESS_KEY_ID")
secretKey := envFirst("OLAKE_S3_SECRET_ACCESS_KEY", "AWS_SECRET_ACCESS_KEY")
if accessKey != "" && secretKey != "" {
sessionToken := envFirst("OLAKE_S3_SESSION_TOKEN", "AWS_SESSION_TOKEN")
configOpts = append(configOpts, config.WithCredentialsProvider(

@vikaxsh vikaxsh Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for using s3 a remote fs we should follow strict protocol for env vars

Comment thread utils/s3_utils.go Outdated
}

// newS3Client creates a new S3 client.
func newS3Client(ctx context.Context) (*s3.Client, error) {

@vikaxsh vikaxsh Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we avoid creating the S3 client separately for upload and resolve? We can create it once and reuse it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vikaxsh vikaxsh vikaxsh left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@saksham-datazip @vikaxsh