Consul has a couple configuration options that allow operators to configure the behaviour when the agent is asked to shut down. Understanding these is important for predictable cluster operation.

A Consul cluster must maintain quorum at all times for continued availability. What this means is that at least n+1/2 number of consul agents running in -server mode¹ must be healthy in a given cluster to continue serving raft requests. Agents running in non-server mode can usually leave and join the cluster as they please, affecting only registered services and health checks for the node they represent.

All this is fine if a cluster is healthy, but not when performing a consul version upgrade or swapping existing nodes with new ones. In such cases, ensuring that old servers and clients leave the cluster in a way that results in the minimum amount of disruption is critical.

A Graceful “leave”

Performing a graceful “leave” causes the agent to explicitly report to the other nodes that it is planning to shut down. This prevents other nodes from marking the agent as “failed”, when they’re unable to reach it. In comparison, ungraceful leaves are functionally indistinguishable from real failures. The agent is marked as “failed”, as soon as it is unreachable by the cluster.

There are additional implications of “leaving” a cluster depending on whether the agent is running as a client or a server:

• When running as a -server, a graceful leave also decrements of the total number of servers needed for a raft quorum. This ensures that a server leaving gracefully will never cause the cluster to become unavailable and stop processing writes. This is different from an ungraceful leave or failure where the server going down will not cause a decrement, and can also make the cluster lose quorum if enough servers become unhealthy.

• In non-server mode, a graceful leave is not too different from an ungraceful one. The agent is marked as “left” as soon as it is unreachable, instead of the “failed” state.

Determinism over the leave behaviour is vital when operating a cluster. To configure this, Consul allows specifying behaviour for the various kill signals with two configuration options.

Config: leave_on_terminate

leave_on_terminate configures leave behaviour for the TERM or “terminate” signal.

leave_on_terminate	TERM signal behaviour
true	gracefully leave the cluster before shutting down
false	shut down without a graceful leave
-server default	same as false
client default	same as true

Config: skip_leave_on_interrupt

skip_leave_on_interrupt configures behaviour for the INT or “interrupt” signal. This one has negated naming² so we need to be careful when configuring this one.

skip_leave_on_interrupt	INT signal behaviour
true	shut down without a graceful leave
false	gracefully leave the cluster before shutting down
-server default	same as true
client default	same as false

Understanding the Defaults

The nature of the defaults can be explained by understanding what would happen if both the configuration variables are not specified:

• On servers, sending either a TERM or INT will cause the agent to shutdown without a graceful leave.
  • Thus simply shutting down servers may cause the cluster to lose quorum if not enough healthy servers remain.
  • The recommended way to prevent this is to perform “rolling” restarts.
• On clients, sending either a TERM or INT will always result in a graceful leave.
  • This is not too surprising, as clients are expected to be swapped somewhat frequently.

Tweaking the Defaults

Usually, leave_on_terminate does not need changing.

For servers, skip_leave_on_interrupt has some edge cases to keep in mind. Setting it to true is vital when bringing in new servers with distinct node IDs from the existing ones. A gracefully leaving server ensures that the cluster will not lose quorum, as long as there are enough remaining healthy servers.

Kubernetes Concerns

When running consul on Kubernetes, the defaults should just work for most use cases. Both skip_leave_on_interrupt and leave_on_terminate cannot be configured using the consul command line arguments and thus require more complicated setups in immutable environments.

systemd Configuration

When using systemd to manage the consul agent process, the KillSignal parameter can be used to terminate the consul process. If not specified, systemd will default to TERM. This can be handy when the configuration for the consul process cannot be changed, for some reason.

Conclusion

Ensuring no outages when upgrading consul versions or removing servers is super important, especially when usually there is only a single consul cluster per datacenter. While the defaults are good, the naming is very confusing which can cause confusion when interacting with this the first time. Configuring them explicitly with proper documentation is often a good idea for posterity.

---

Footnotes: