Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@
which could cause problems with GitOps tools (e.g. ArgoCD) reporting a diff in the custom resources.
See [our internal issue](https://github.com/stackabletech/hdfs-operator/issues/626) and [the fix](https://github.com/kube-rs/kube/pull/2042) for details ([#840]).
- Task logs are served from `BASE_LOG_FOLDER` instead of the `task` handler's `base_log_folder` at the Vector agent log directory ([#834]).
- Avoid Python import race conditions by pre-cloning the git repo not only for Celery-based stacklets, but also for Kubernetes executor-based setups ([#844]).

[#814]: https://github.com/stackabletech/airflow-operator/pull/814
[#821]: https://github.com/stackabletech/airflow-operator/pull/821
Expand All @@ -40,6 +41,7 @@
[#834]: https://github.com/stackabletech/airflow-operator/pull/834
[#835]: https://github.com/stackabletech/airflow-operator/pull/835
[#840]: https://github.com/stackabletech/airflow-operator/pull/840
[#844]: https://github.com/stackabletech/airflow-operator/pull/844

## [26.7.0] - 2026-07-21

Expand Down
12 changes: 4 additions & 8 deletions rust/operator-binary/src/controller/build/resource/executor.rs
Original file line number Diff line number Diff line change
Expand Up @@ -143,14 +143,10 @@ pub fn build_executor_template_config_map(
.add_volume_mount(&*LOG_VOLUME_NAME, STACKABLE_LOG_DIR)
.context(AddVolumeMountSnafu)?;

add_git_sync_resources(
&mut pb,
&mut airflow_container,
git_sync_resources,
false,
true,
Comment thread
adwk67 marked this conversation as resolved.
)
.context(PodSnafu)?;
// We don't need a git-sync sidecar, an initial clone via the init-container is sufficient for
// Kubernetes executors, as they are short-lived.
add_git_sync_resources(&mut pb, &mut airflow_container, git_sync_resources, false)
.context(PodSnafu)?;

cluster
.metadata_database_connection_details()
Expand Down
18 changes: 13 additions & 5 deletions rust/operator-binary/src/controller/build/resource/pod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -85,23 +85,31 @@ pub(crate) fn add_authentication_volumes_and_volume_mounts(
Ok(())
}

/// Adds the needed git-sync init-container and (optionally) sidecar.
///
/// If the DAG is modularized we may encounter a timing issue whereby the main process
/// has started *before* all modules referenced by the DAG have been fetched by gitsync
/// and registered. This will result in ModuleNotFoundError errors. This can be avoided
/// by running a one-off git-sync process in an init-container so that all DAG
/// dependencies are fully loaded. The sidecar git-sync is then used for regular updates.
///
/// For that reason, we always add a init-container that clones the repo initially. All Pods (except
/// the Kubernetes operators) additionally use a sidecar to keep the git contents up-to-date.
Comment thread
sweb marked this conversation as resolved.
Outdated
pub(crate) fn add_git_sync_resources(
pb: &mut PodBuilder,
cb: &mut ContainerBuilder,
git_sync_resources: &git_sync::v1alpha2::GitSyncResources,
add_sidecar_containers: bool,
Comment thread
sweb marked this conversation as resolved.
Outdated
add_init_containers: bool,
) -> Result<()> {
if add_sidecar_containers {
for container in git_sync_resources.git_sync_containers.iter().cloned() {
pb.add_container(container);
}
}
if add_init_containers {
for container in git_sync_resources.git_sync_init_containers.iter().cloned() {
pb.add_init_container(container);
}
for container in git_sync_resources.git_sync_init_containers.iter().cloned() {
pb.add_init_container(container);
}

pb.add_volumes(git_sync_resources.git_content_volumes.to_owned())
.context(AddVolumeSnafu)?;
pb.add_volumes(git_sync_resources.git_ssh_volumes.to_owned())
Expand Down
17 changes: 3 additions & 14 deletions rust/operator-binary/src/controller/build/resource/statefulset.rs
Original file line number Diff line number Diff line change
Expand Up @@ -253,20 +253,9 @@ pub fn build_server_rolegroup_statefulset(
.context(AddVolumeMountSnafu)?;
}

// If the DAG is modularized we may encounter a timing issue whereby the celery worker
// has started *before* all modules referenced by the DAG have been fetched by gitsync
// and registered. This will result in ModuleNotFoundError errors. This can be avoided
// by running a one-off git-sync process in an init-container so that all DAG
// dependencies are fully loaded. The sidecar git-sync is then used for regular updates.
let use_git_sync_init_containers = matches!(executor, AirflowExecutor::CeleryExecutors { .. });
add_git_sync_resources(
&mut pb,
&mut airflow_container,
git_sync_resources,
true,
use_git_sync_init_containers,
)
.context(PodSnafu)?;
// We need a git-sync sidecar to keep the git contents up-to-date
add_git_sync_resources(&mut pb, &mut airflow_container, git_sync_resources, true)
.context(PodSnafu)?;

validated_cluster
.metadata_database_connection_details()
Expand Down
Loading