From 0fe27dedce8d979093f9b2a2b858e53559464771 Mon Sep 17 00:00:00 2001 From: Helena Edelson Date: Thu, 18 Jul 2019 16:40:36 -0700 Subject: [PATCH 1/7] Expand on rolling update recommendations, best practices #27192 --- .../src/main/paradox/additional/deploy.md | 2 +- .../paradox/additional/rolling-deploys.md | 39 -------- .../paradox/additional/rolling-updates.md | 93 +++++++++++++++++++ 3 files changed, 94 insertions(+), 40 deletions(-) delete mode 100644 akka-docs/src/main/paradox/additional/rolling-deploys.md create mode 100644 akka-docs/src/main/paradox/additional/rolling-updates.md diff --git a/akka-docs/src/main/paradox/additional/deploy.md b/akka-docs/src/main/paradox/additional/deploy.md index aadfbdbbb80..b001ff6b581 100644 --- a/akka-docs/src/main/paradox/additional/deploy.md +++ b/akka-docs/src/main/paradox/additional/deploy.md @@ -6,7 +6,7 @@ * [Packaging](packaging.md) * [Deploying](deploying.md) -* [Rolling Deploys](rolling-deploys.md) +* [Rolling Updates](rolling-updates.md) @@@ diff --git a/akka-docs/src/main/paradox/additional/rolling-deploys.md b/akka-docs/src/main/paradox/additional/rolling-deploys.md deleted file mode 100644 index 60a9dcbd275..00000000000 --- a/akka-docs/src/main/paradox/additional/rolling-deploys.md +++ /dev/null @@ -1,39 +0,0 @@ -# Rolling Deploys - -## Rolling Updates - -A rolling update is the process of replacing one version of the system with another without downtime. -The changes can be new code, changed dependencies such as new Akka version, or modified configuration. - -In Akka, rolling updates are typically used for a stateful Akka Cluster where you can't run two separate clusters in -parallel during the update, for example in blue green deployments. - -### Compatibility of remote message protocols and serialization - -There are two parts of Akka that need careful consideration when performing an rolling update. - -1. Compatibility of remote message protocols. Old nodes may send messages to new nodes and vice versa. -1. Serialization format of persisted events and snapshots. New nodes must be able to read old data, and - during the update old nodes must be able to read data stored by new nodes. - -There are many more application specific aspects. It's important to have a strategy for serialization -format that can be evolved and you can find advice in -@ref:[Persistence - Schema Evolution](../persistence-schema-evolution.md), which also applies to -remote messages when deploying with rolling updates. - - -For rolling updates related to Akka dependency version upgrades and the migration guides, please see -@ref:[Rolling Updates and Akka versions](../project/rolling-update.md) - - -### Akka Cluster Configuration Compatibility Check - -Relevant information on rolling updates and enforcing [Akka Cluster configuration compatibility checks](cluster-usage.md#configuration-compatibility-check) -on joining nodes. - -### Akka Cluster Shutdown order - -It's best to keep the node that has been running the longest until last. That is because -Cluster Singletons (including Cluster Sharding coordinators) are running on the oldest nodes. It's more -efficient to avoid moving the singletons more than necessary since those typically have to recover -their state and it might introduce unnecessary delays in for example access to new sharded actors. diff --git a/akka-docs/src/main/paradox/additional/rolling-updates.md b/akka-docs/src/main/paradox/additional/rolling-updates.md new file mode 100644 index 00000000000..3af84cad43c --- /dev/null +++ b/akka-docs/src/main/paradox/additional/rolling-updates.md @@ -0,0 +1,93 @@ +# Rolling Updates + +A rolling update is the process of replacing one version of the system with another without downtime. +The changes can be new code, changed dependencies such as new Akka version, or modified configuration. + +In Akka, rolling updates are typically used for a stateful Akka Cluster where you can't run two separate clusters in +parallel during the update, for example in blue green deployments. + +For rolling updates related to Akka dependency version upgrades and the migration guides, please see +@ref:[Rolling Updates and Akka versions](../project/rolling-update.md) + +#### This document covers: +* [Serialization Compatibility](#serialization-compatibility) +* [Cluster Sharding](#cluster-sharding) +* [Migrating Untyped to Typed](#migrating-untyped-to-typed) +* [Cluster Shutdown](#cluster-shutdown) +* [Cluster Configuration Compatibility Check](#cluster-configuration-compatibility-check) + +## Serialization Compatibility + +There are two parts of Akka that need careful consideration when performing an rolling update. + +1. Compatibility of remote message protocols. Old nodes may send messages to new nodes and vice versa. +1. Serialization format of persisted events and snapshots. New nodes must be able to read old data, and + during the update old nodes must be able to read data stored by new nodes. + +There are many more application specific aspects. It's important to have a strategy for serialization +format that can be evolved and you can find advice in +@ref:[Persistence - Schema Evolution](../persistence-schema-evolution.md), which also applies to +remote messages when deploying with rolling updates. + +### Retiring a serializer without downtime + +One approach is carried out in two phases: + +1. Deploy a version of your application that still supports deserializing messages in the old form +(your deserializer is still in "akka.actor.serializers"-section of your configuration, but it no longer serializes new messages +(it is removed from “akka.actor.serialization-bindings”-section). See @ref:[Serialization Configuration](../serialization.md#configuration) for details. +1. After this upgrade has succeeded and there are no longer systems sending messages serialized with the decommissioned serializer, +you can upgrade to a version that also no longer contains this serializer in `akka.actor.serializers`. + +## Cluster Sharding + +During a rolling upgrade, sharded entities receiving traffic may be moved during @ref:[shard rebalancing](../cluster-sharding.md#shard-rebalancing), +to an old or new node in the cluster, based on the pluggable allocation strategy and settings. +When an old node is stopped the shards that were running on it may be allocated to one of the +other old nodes remaining in the cluster. + +It is recommended to leave the oldest node until last, because the shard coordinator +is running on that node. See @ref:[Cluster Shutdown](#cluster-shutdown) below for more detail. + +There are some cases when @ref:[a full cluster restart is needed](../cluster-sharding.md#rolling-upgrades). + +## Migrating Untyped to Typed + +It is recommended with a two step approach: + +* Deploy with the new nodes set to `akka.cluster.configuration-compatibility-check.enforce-on-join = off` +and ensure all nodes are in this state +* Deploy again and with the new nodes set to `akka.cluster.configuration-compatibility-check.enforce-on-join = on`. + +The configuration from existing nodes should pass the @ref:[Cluster Configuration Compatibility Checks](#cluster-configuration-compatibility-check). +Find out more about coexisting and @ref:[untyped to typed](../typed/coexisting.md#untyped-to-typed). + +### With Cluster Sharding and Persistence + +Rolling upgrades where shards on old nodes are running untyped persistent actors +and new ones are running typed persistent behaviors have been tested successfully by the team and users. +Samples coming soon. + +## Cluster Shutdown + +@ref:[Coordinated Shutdown](../actors.md#coordinated-shutdown) will automatically run when the cluster node sees itself as Exiting. +@ref:[Graceful shutdown](../cluster-sharding.md#graceful-shutdown) of Cluster Singletons and Cluster Sharding similarly happen automatically. + +But in case of network failures during rolling restarts and updates it +may still be necessary to set the node’s status to Down in order to complete the removal. + +Find out more about +* @ref:[Cluster Downing](../cluster-usage.md#downing) and providers +* [Cluster Bootstrap](https://doc.akka.io/docs/akka-management/current/bootstrap/index.md#rolling-updates) and Rolling updates + +### Cluster Shutdown Order + +If possible keep the oldest node until last because +Cluster Singletons (including Cluster Sharding coordinators) are running on them. It's more +efficient to avoid moving the singletons more than necessary since those typically have to recover +their state and it might introduce unnecessary delays, for example in access to new sharded actors. + +## Cluster Configuration Compatibility Checks + +Relevant information on rolling updates and enforcing @ref:[Akka Cluster configuration compatibility checks](../cluster-usage.md#configuration-compatibility-check) +on joining nodes. From 7db54ca419f336f307c83cb0f55f553e5df39a7f Mon Sep 17 00:00:00 2001 From: Helena Edelson Date: Mon, 22 Jul 2019 15:43:37 -0700 Subject: [PATCH 2/7] Review feedback. --- akka-docs/src/main/paradox/.htaccess | 2 + .../paradox/additional/rolling-updates.md | 49 +++++++++---------- 2 files changed, 25 insertions(+), 26 deletions(-) diff --git a/akka-docs/src/main/paradox/.htaccess b/akka-docs/src/main/paradox/.htaccess index e6c4ebf1abf..48eed5d6ee3 100644 --- a/akka-docs/src/main/paradox/.htaccess +++ b/akka-docs/src/main/paradox/.htaccess @@ -25,3 +25,5 @@ RedirectMatch 301 ^(.*)/stream/operators/Source-or-Flow/UnzipWith\.html$ $1/stre RedirectMatch 301 ^(.*)/stream/operators/Source-or-Flow/apply\.html$ $1/stream/operators/Source/from.html RedirectMatch 301 ^(.*[^d])/guide/(.*) $1/typed/guide/$2 + +RedirectMatch 301 ^(.*)/additional/rolling-deploys\.html$ $1/additional/rolling-updates.html diff --git a/akka-docs/src/main/paradox/additional/rolling-updates.md b/akka-docs/src/main/paradox/additional/rolling-updates.md index 3af84cad43c..0dc6468467b 100644 --- a/akka-docs/src/main/paradox/additional/rolling-updates.md +++ b/akka-docs/src/main/paradox/additional/rolling-updates.md @@ -12,6 +12,7 @@ For rolling updates related to Akka dependency version upgrades and the migratio #### This document covers: * [Serialization Compatibility](#serialization-compatibility) * [Cluster Sharding](#cluster-sharding) +* [Cluster Singleton](#cluster-singleton) * [Migrating Untyped to Typed](#migrating-untyped-to-typed) * [Cluster Shutdown](#cluster-shutdown) * [Cluster Configuration Compatibility Check](#cluster-configuration-compatibility-check) @@ -24,33 +25,35 @@ There are two parts of Akka that need careful consideration when performing an r 1. Serialization format of persisted events and snapshots. New nodes must be able to read old data, and during the update old nodes must be able to read data stored by new nodes. -There are many more application specific aspects. It's important to have a strategy for serialization -format that can be evolved and you can find advice in +There are many more application specific aspects for serialization changes during rolling upgrades to consider. +For example, whether to allow dropped messages or tear down the TCP connection when the manifest is unknown. + +* When some message loss during a rolling upgrade is acceptable versus a full shutdown and restart, assuming the application recovers afterwards + - If a `java.io.NotSerializableException` is thrown in `fromBinary` this is treated as a transient problem, the issue logged and the message is dropped + - If other exceptions are thrown it can be an indication of corrupt bytes from the underlying transport, and the connection is broken +* For more zero-impact rolling upgrades, it is important to consider a strategy for serialization format that can be evolved. You can find advice in @ref:[Persistence - Schema Evolution](../persistence-schema-evolution.md), which also applies to remote messages when deploying with rolling updates. -### Retiring a serializer without downtime - -One approach is carried out in two phases: - -1. Deploy a version of your application that still supports deserializing messages in the old form -(your deserializer is still in "akka.actor.serializers"-section of your configuration, but it no longer serializes new messages -(it is removed from “akka.actor.serialization-bindings”-section). See @ref:[Serialization Configuration](../serialization.md#configuration) for details. -1. After this upgrade has succeeded and there are no longer systems sending messages serialized with the decommissioned serializer, -you can upgrade to a version that also no longer contains this serializer in `akka.actor.serializers`. +One approach to retiring a serializer without downtime is carried out in @ref:[two rolling upgrade steps to switch to the new serializer](../serialization.md#rolling-upgrades). ## Cluster Sharding During a rolling upgrade, sharded entities receiving traffic may be moved during @ref:[shard rebalancing](../cluster-sharding.md#shard-rebalancing), to an old or new node in the cluster, based on the pluggable allocation strategy and settings. When an old node is stopped the shards that were running on it may be allocated to one of the -other old nodes remaining in the cluster. - -It is recommended to leave the oldest node until last, because the shard coordinator -is running on that node. See @ref:[Cluster Shutdown](#cluster-shutdown) below for more detail. +other old nodes remaining in the cluster. See @ref[ClusterSingleton](#cluster-singleton) for a useful `ShardCoordinator` optimization. There are some cases when @ref:[a full cluster restart is needed](../cluster-sharding.md#rolling-upgrades). +## Cluster Singleton + +It's more efficient to avoid moving a `ClusterSingleton` more than necessary because they typically have to recover their state +and it might introduce unnecessary delays. + +An optional optimization is to leave the oldest node running a `ClusterSingleton` until last +to avoid it having to move more than once. + ## Migrating Untyped to Typed It is recommended with a two step approach: @@ -70,22 +73,16 @@ Samples coming soon. ## Cluster Shutdown -@ref:[Coordinated Shutdown](../actors.md#coordinated-shutdown) will automatically run when the cluster node sees itself as Exiting. +@ref:[Coordinated Shutdown](../actors.md#coordinated-shutdown) will automatically run on SIGTERM when the cluster node sees itself as Exiting. +Thus running shutdown tasks in a JVM shutdown hook is not recommended. @ref:[Graceful shutdown](../cluster-sharding.md#graceful-shutdown) of Cluster Singletons and Cluster Sharding similarly happen automatically. -But in case of network failures during rolling restarts and updates it -may still be necessary to set the node’s status to Down in order to complete the removal. +In case of network failures it may still be necessary to set the node’s status to Down in order to complete the removal. Find out more about * @ref:[Cluster Downing](../cluster-usage.md#downing) and providers -* [Cluster Bootstrap](https://doc.akka.io/docs/akka-management/current/bootstrap/index.md#rolling-updates) and Rolling updates - -### Cluster Shutdown Order - -If possible keep the oldest node until last because -Cluster Singletons (including Cluster Sharding coordinators) are running on them. It's more -efficient to avoid moving the singletons more than necessary since those typically have to recover -their state and it might introduce unnecessary delays, for example in access to new sharded actors. +* [Cluster Bootstrap](https://doc.akka.io/docs/akka-management/current/bootstrap/index.html#rolling-updates) and Rolling updates +* [Split Brain Resolver](https://doc.akka.io/docs/akka-enhancements/current/split-brain-resolver.html) ## Cluster Configuration Compatibility Checks From 103a4e84ce611142e19180df9fc0db48626ed3e6 Mon Sep 17 00:00:00 2001 From: Helena Edelson Date: Wed, 24 Jul 2019 05:58:35 -0700 Subject: [PATCH 3/7] Round two PR feedback changes. --- .../paradox/additional/rolling-updates.md | 35 ++++++++----------- 1 file changed, 14 insertions(+), 21 deletions(-) diff --git a/akka-docs/src/main/paradox/additional/rolling-updates.md b/akka-docs/src/main/paradox/additional/rolling-updates.md index 0dc6468467b..b160a8f3bdc 100644 --- a/akka-docs/src/main/paradox/additional/rolling-updates.md +++ b/akka-docs/src/main/paradox/additional/rolling-updates.md @@ -9,14 +9,6 @@ parallel during the update, for example in blue green deployments. For rolling updates related to Akka dependency version upgrades and the migration guides, please see @ref:[Rolling Updates and Akka versions](../project/rolling-update.md) -#### This document covers: -* [Serialization Compatibility](#serialization-compatibility) -* [Cluster Sharding](#cluster-sharding) -* [Cluster Singleton](#cluster-singleton) -* [Migrating Untyped to Typed](#migrating-untyped-to-typed) -* [Cluster Shutdown](#cluster-shutdown) -* [Cluster Configuration Compatibility Check](#cluster-configuration-compatibility-check) - ## Serialization Compatibility There are two parts of Akka that need careful consideration when performing an rolling update. @@ -31,7 +23,7 @@ For example, whether to allow dropped messages or tear down the TCP connection w * When some message loss during a rolling upgrade is acceptable versus a full shutdown and restart, assuming the application recovers afterwards - If a `java.io.NotSerializableException` is thrown in `fromBinary` this is treated as a transient problem, the issue logged and the message is dropped - If other exceptions are thrown it can be an indication of corrupt bytes from the underlying transport, and the connection is broken -* For more zero-impact rolling upgrades, it is important to consider a strategy for serialization format that can be evolved. You can find advice in +* For more zero-impact rolling upgrades, it is important to consider a strategy for serialization that can be evolved. You can find advice in @ref:[Persistence - Schema Evolution](../persistence-schema-evolution.md), which also applies to remote messages when deploying with rolling updates. @@ -41,10 +33,10 @@ One approach to retiring a serializer without downtime is carried out in @ref:[t During a rolling upgrade, sharded entities receiving traffic may be moved during @ref:[shard rebalancing](../cluster-sharding.md#shard-rebalancing), to an old or new node in the cluster, based on the pluggable allocation strategy and settings. -When an old node is stopped the shards that were running on it may be allocated to one of the +When an old node is stopped the shards that were running on it are moved to one of the other old nodes remaining in the cluster. See @ref[ClusterSingleton](#cluster-singleton) for a useful `ShardCoordinator` optimization. -There are some cases when @ref:[a full cluster restart is needed](../cluster-sharding.md#rolling-upgrades). +Some changes to sharding configuration require @ref:[a full cluster restart](../cluster-sharding.md#rolling-upgrades). ## Cluster Singleton @@ -62,7 +54,6 @@ It is recommended with a two step approach: and ensure all nodes are in this state * Deploy again and with the new nodes set to `akka.cluster.configuration-compatibility-check.enforce-on-join = on`. -The configuration from existing nodes should pass the @ref:[Cluster Configuration Compatibility Checks](#cluster-configuration-compatibility-check). Find out more about coexisting and @ref:[untyped to typed](../typed/coexisting.md#untyped-to-typed). ### With Cluster Sharding and Persistence @@ -74,17 +65,19 @@ Samples coming soon. ## Cluster Shutdown @ref:[Coordinated Shutdown](../actors.md#coordinated-shutdown) will automatically run on SIGTERM when the cluster node sees itself as Exiting. -Thus running shutdown tasks in a JVM shutdown hook is not recommended. +Thus running shutdown tasks in a JVM shutdown hook is not recommended over Coordinated Shutdown. @ref:[Graceful shutdown](../cluster-sharding.md#graceful-shutdown) of Cluster Singletons and Cluster Sharding similarly happen automatically. -In case of network failures it may still be necessary to set the node’s status to Down in order to complete the removal. - -Find out more about -* @ref:[Cluster Downing](../cluster-usage.md#downing) and providers -* [Cluster Bootstrap](https://doc.akka.io/docs/akka-management/current/bootstrap/index.html#rolling-updates) and Rolling updates -* [Split Brain Resolver](https://doc.akka.io/docs/akka-enhancements/current/split-brain-resolver.html) +In case of network failures it may still be necessary to set the node's status to Down in order to complete the removal. +@ref:[Cluster Downing](../cluster-usage.md#downing) details downing nodes and downing providers, and how using +[Split Brain Resolver](https://doc.akka.io/docs/akka-enhancements/current/split-brain-resolver.html) can help resolve +split brain during network partitions. +Additionally, [Cluster Bootstrap](https://doc.akka.io/docs/akka-management/current/bootstrap/index.html#rolling-updates) +can be leveraged during rolling updates for joining and downings. + ## Cluster Configuration Compatibility Checks -Relevant information on rolling updates and enforcing @ref:[Akka Cluster configuration compatibility checks](../cluster-usage.md#configuration-compatibility-check) -on joining nodes. +During rolling updates the configuration from existing nodes should pass the Cluster configuration compatibility checks. +Find out more about enforcing these checks on joining nodes and optionally adding custom checks in the +@ref:[Akka Cluster configuration compatibility checks](../cluster-usage.md#configuration-compatibility-check) documentation. From d6d5de5558e3b58aead735f986d42f27a87a5129 Mon Sep 17 00:00:00 2001 From: Helena Edelson Date: Wed, 24 Jul 2019 06:13:20 -0700 Subject: [PATCH 4/7] Further review suggestions. --- .../paradox/additional/rolling-updates.md | 37 +++++++------------ 1 file changed, 13 insertions(+), 24 deletions(-) diff --git a/akka-docs/src/main/paradox/additional/rolling-updates.md b/akka-docs/src/main/paradox/additional/rolling-updates.md index b160a8f3bdc..13107b5cf9b 100644 --- a/akka-docs/src/main/paradox/additional/rolling-updates.md +++ b/akka-docs/src/main/paradox/additional/rolling-updates.md @@ -40,27 +40,10 @@ Some changes to sharding configuration require @ref:[a full cluster restart](../ ## Cluster Singleton -It's more efficient to avoid moving a `ClusterSingleton` more than necessary because they typically have to recover their state -and it might introduce unnecessary delays. - -An optional optimization is to leave the oldest node running a `ClusterSingleton` until last -to avoid it having to move more than once. - -## Migrating Untyped to Typed - -It is recommended with a two step approach: - -* Deploy with the new nodes set to `akka.cluster.configuration-compatibility-check.enforce-on-join = off` -and ensure all nodes are in this state -* Deploy again and with the new nodes set to `akka.cluster.configuration-compatibility-check.enforce-on-join = on`. - -Find out more about coexisting and @ref:[untyped to typed](../typed/coexisting.md#untyped-to-typed). - -### With Cluster Sharding and Persistence - -Rolling upgrades where shards on old nodes are running untyped persistent actors -and new ones are running typed persistent behaviors have been tested successfully by the team and users. -Samples coming soon. +Cluster singletons are always running on the oldest node. To avoid moving cluster singletons more than necessary during a rolling upgrade, +it is recommended to upgrade the oldest node last. This way cluster singletons are only moved once during a full rolling upgrade. +Otherwise, in the worst case cluster singletons may be migrated from node to node which requires coordination and initialization +overhead several times. ## Cluster Shutdown @@ -74,10 +57,16 @@ In case of network failures it may still be necessary to set the node's status t split brain during network partitions. Additionally, [Cluster Bootstrap](https://doc.akka.io/docs/akka-management/current/bootstrap/index.html#rolling-updates) -can be leveraged during rolling updates for joining and downings. +can be leveraged during rolling updates for joining and downing nodes in the cluster. ## Cluster Configuration Compatibility Checks During rolling updates the configuration from existing nodes should pass the Cluster configuration compatibility checks. -Find out more about enforcing these checks on joining nodes and optionally adding custom checks in the -@ref:[Akka Cluster configuration compatibility checks](../cluster-usage.md#configuration-compatibility-check) documentation. +For example, when migrating from Classic to Typed Clusters, a two step approach is possible: + +* Deploy with the new nodes set to `akka.cluster.configuration-compatibility-check.enforce-on-join = off` +and ensure all nodes are in this state +* Deploy again and with the new nodes set to `akka.cluster.configuration-compatibility-check.enforce-on-join = on`. + +Full documentation about enforcing these checks on joining nodes and optionally adding custom checks are in +@ref:[Akka Cluster configuration compatibility checks](../cluster-usage.md#configuration-compatibility-check). From 139bf89858dcc7ef5ca257cf3b358b60d3f5a8ad Mon Sep 17 00:00:00 2001 From: Helena Edelson Date: Wed, 24 Jul 2019 10:55:46 -0700 Subject: [PATCH 5/7] Clarified Cluster Configuration Compatibility Checks migration example. --- .../src/main/paradox/additional/rolling-updates.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/akka-docs/src/main/paradox/additional/rolling-updates.md b/akka-docs/src/main/paradox/additional/rolling-updates.md index 13107b5cf9b..8decbee6502 100644 --- a/akka-docs/src/main/paradox/additional/rolling-updates.md +++ b/akka-docs/src/main/paradox/additional/rolling-updates.md @@ -34,7 +34,8 @@ One approach to retiring a serializer without downtime is carried out in @ref:[t During a rolling upgrade, sharded entities receiving traffic may be moved during @ref:[shard rebalancing](../cluster-sharding.md#shard-rebalancing), to an old or new node in the cluster, based on the pluggable allocation strategy and settings. When an old node is stopped the shards that were running on it are moved to one of the -other old nodes remaining in the cluster. See @ref[ClusterSingleton](#cluster-singleton) for a useful `ShardCoordinator` optimization. +other old nodes remaining in the cluster. The `ShardCoordinator` is itself a cluster singleton. +To minimize downtime of the shard coordinator, see the strategies about @ref[ClusterSingleton](#cluster-singleton) rolling upgrades below. Some changes to sharding configuration require @ref:[a full cluster restart](../cluster-sharding.md#rolling-upgrades). @@ -62,11 +63,12 @@ can be leveraged during rolling updates for joining and downing nodes in the clu ## Cluster Configuration Compatibility Checks During rolling updates the configuration from existing nodes should pass the Cluster configuration compatibility checks. -For example, when migrating from Classic to Typed Clusters, a two step approach is possible: +For example, it is possible to migrate Cluster Sharding from Classic to Typed Actors in a rolling update using a two step approach +as of Akka version `2.5.23`: * Deploy with the new nodes set to `akka.cluster.configuration-compatibility-check.enforce-on-join = off` and ensure all nodes are in this state * Deploy again and with the new nodes set to `akka.cluster.configuration-compatibility-check.enforce-on-join = on`. - -Full documentation about enforcing these checks on joining nodes and optionally adding custom checks are in + +Full documentation about enforcing these checks on joining nodes and optionally adding custom checks can be found in @ref:[Akka Cluster configuration compatibility checks](../cluster-usage.md#configuration-compatibility-check). From a42aae3188c3940fd0602eebd754954db288ae40 Mon Sep 17 00:00:00 2001 From: Helena Edelson Date: Wed, 24 Jul 2019 11:56:27 -0700 Subject: [PATCH 6/7] Further clarification on cluster shutdown during rolling updates. --- .../paradox/additional/rolling-updates.md | 25 +++++++++++-------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/akka-docs/src/main/paradox/additional/rolling-updates.md b/akka-docs/src/main/paradox/additional/rolling-updates.md index 8decbee6502..b4541e407c2 100644 --- a/akka-docs/src/main/paradox/additional/rolling-updates.md +++ b/akka-docs/src/main/paradox/additional/rolling-updates.md @@ -18,7 +18,7 @@ There are two parts of Akka that need careful consideration when performing an r during the update old nodes must be able to read data stored by new nodes. There are many more application specific aspects for serialization changes during rolling upgrades to consider. -For example, whether to allow dropped messages or tear down the TCP connection when the manifest is unknown. +For example based on the use case and requirements, whether to allow dropped messages or tear down the TCP connection when the manifest is unknown. * When some message loss during a rolling upgrade is acceptable versus a full shutdown and restart, assuming the application recovers afterwards - If a `java.io.NotSerializableException` is thrown in `fromBinary` this is treated as a transient problem, the issue logged and the message is dropped @@ -48,18 +48,21 @@ overhead several times. ## Cluster Shutdown -@ref:[Coordinated Shutdown](../actors.md#coordinated-shutdown) will automatically run on SIGTERM when the cluster node sees itself as Exiting. -Thus running shutdown tasks in a JVM shutdown hook is not recommended over Coordinated Shutdown. +### Graceful shutdown + +For rolling updates it is best to leave the Cluster gracefully via @ref:[Coordinated Shutdown](../actors.md#coordinated-shutdown), +which will run automatically on SIGTERM, when the Cluster node sees itself as `Exiting`. +Environments such as Kubernetes send a SIGTERM, however if the JVM is wrapped with a script ensure that it forwards the signal. @ref:[Graceful shutdown](../cluster-sharding.md#graceful-shutdown) of Cluster Singletons and Cluster Sharding similarly happen automatically. - -In case of network failures it may still be necessary to set the node's status to Down in order to complete the removal. -@ref:[Cluster Downing](../cluster-usage.md#downing) details downing nodes and downing providers, and how using -[Split Brain Resolver](https://doc.akka.io/docs/akka-enhancements/current/split-brain-resolver.html) can help resolve -split brain during network partitions. -Additionally, [Cluster Bootstrap](https://doc.akka.io/docs/akka-management/current/bootstrap/index.html#rolling-updates) -can be leveraged during rolling updates for joining and downing nodes in the cluster. - +### Ungraceful shutdown + +In case of network failures it may still be necessary to set the node's status to Down in order to complete the removal. +@ref:[Cluster Downing](../cluster-usage.md#downing) details downing nodes and downing providers. +[Split Brain Resolver](https://doc.akka.io/docs/akka-enhancements/current/split-brain-resolver.html) can be used to ensure +the cluster continues to function during network partitions and node failures. For example +if there is an unreachability problem Split Brain Resolver would make a decision based on the configured downing strategy. + ## Cluster Configuration Compatibility Checks During rolling updates the configuration from existing nodes should pass the Cluster configuration compatibility checks. From 74761476ad273cb6256ab12cbb299ae7aa36a9f0 Mon Sep 17 00:00:00 2001 From: Helena Edelson Date: Wed, 24 Jul 2019 12:01:41 -0700 Subject: [PATCH 7/7] Clarified and cleaned up rolling upgrade serialization compatibility options. --- .../src/main/paradox/additional/rolling-updates.md | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/akka-docs/src/main/paradox/additional/rolling-updates.md b/akka-docs/src/main/paradox/additional/rolling-updates.md index b4541e407c2..7e269f38bd7 100644 --- a/akka-docs/src/main/paradox/additional/rolling-updates.md +++ b/akka-docs/src/main/paradox/additional/rolling-updates.md @@ -19,15 +19,14 @@ There are two parts of Akka that need careful consideration when performing an r There are many more application specific aspects for serialization changes during rolling upgrades to consider. For example based on the use case and requirements, whether to allow dropped messages or tear down the TCP connection when the manifest is unknown. +When some message loss during a rolling upgrade is acceptable versus a full shutdown and restart, assuming the application recovers afterwards +* If a `java.io.NotSerializableException` is thrown in `fromBinary` this is treated as a transient problem, the issue logged and the message is dropped +* If other exceptions are thrown it can be an indication of corrupt bytes from the underlying transport, and the connection is broken -* When some message loss during a rolling upgrade is acceptable versus a full shutdown and restart, assuming the application recovers afterwards - - If a `java.io.NotSerializableException` is thrown in `fromBinary` this is treated as a transient problem, the issue logged and the message is dropped - - If other exceptions are thrown it can be an indication of corrupt bytes from the underlying transport, and the connection is broken -* For more zero-impact rolling upgrades, it is important to consider a strategy for serialization that can be evolved. You can find advice in -@ref:[Persistence - Schema Evolution](../persistence-schema-evolution.md), which also applies to -remote messages when deploying with rolling updates. +For more zero-impact rolling upgrades, it is important to consider a strategy for serialization that can be evolved. +One approach to retiring a serializer without downtime is described in @ref:[two rolling upgrade steps to switch to the new serializer](../serialization.md#rolling-upgrades). +Additionally you can find advice on @ref:[Persistence - Schema Evolution](../persistence-schema-evolution.md) which also applies to remote messages when deploying with rolling updates. -One approach to retiring a serializer without downtime is carried out in @ref:[two rolling upgrade steps to switch to the new serializer](../serialization.md#rolling-upgrades). ## Cluster Sharding