Upgrading TSB

Managing TSB Upgrades - a quick overview

• Upgrade your Management Plane minor release at least once per year, to stay within the (N-2) support policy
• Upgrade your Control Plane versions to stay within the N-2 compatibility range of your Management Plane version

Staying up-to-date with the latest releases in your selected minor version:

• ensures compatibility and avoids complications such as the OpenCensus/OTEL change and the need to upgrade to later calendar versions at each stage
• provides access to the latest features and security updates.

This document explains the potential complications that you should be aware of before upgrading, and the recommended practices.

TSB Management Planes, Control Plane and Data Plane compatibility

Control Planes: TSB follows an 'N-2' policy for compatibility and stability between Management Plane and Control Plane. This assures that the latest 1.N.x release of the TSB Management Plane (MP) is fully compatible with the latest Control Plane (CP) versions released up to two versions prior 1.(N-2).x.

Data Planes: TSB follows an 'M-1' policy for compatibility between Control Plane and Data Plane. When a Control Plane is upgraded, the Data Plane (Gateways and Sidecar proxies) are not always automatically updated. Gateway upgrades may be held back, and sidecars require explicit pod reloads. The TSB Control Plane version 1.M.x supports Data Plane components from one previous minor version 1.(M-1).x.

Good Upgrade Practice

It is good practice and recommended to:

• Maintain your chosen Management Plane release at the latest point release for your chosen minor release
• Ensure that Control Plane clusters are maintained at the latest point release, and do not fall more than 2 minor versions (1.(N-2)) behind the management plane
• Restart application workloads as necessary

What Upgrade Paths are supported?

TSB Management Plane and Control Plane instances can be upgraded in-place from releases up to 2 minor versions behind (N-2). For example, a 1.10.x installation can be upgraded to 1.12.x, but should not be directly upgraded to 1.13.x. If you wish to upgrade by more than 2 minor versions, you should first upgrade to an intermediate release.

There are two important caveats to bear in mind:

• Always upgrade the Management Plane first. The Control Plane version must not exceed the Management Plane version. For example, a control plane 1.13.1 is not supported with a Management Plane 1.13.0.
• Always upgrade to a later calendar release. Tetrate make point releases for each supported version, which include current CVE fixes and necessary dependency updates. For example, a recent 1.11.4 release may contain fixes not present in an earlier 1.12.0 release. To enforce good security hygiene, you cannot upgrade to an earlier release, even if it has a later minor version number.

Good Upgrade Practice

It is always safe to upgrade to the latest point release in your chosen minor release. Doing so will avoid complications from the 'later calendar release' requirement.

Planning your Upgrades

Always upgrade the Management Plane first. When planning to upgrade to a version, e.g. 1.13.1, upgrade the Management Plane first, and then upgrade your control plane clusters to this version.

Always upgrade to the latest point release in your chosen minor version. For example, if you are running TSB version 1.12.x, always upgrade to the latest point release in that version to ensure that you meet the 'later calendar release' requirement.

Upgrade Notes by Version

OpenCensus and OTEL support in TSB 1.14.0, 1.13.1, 1.12.7, 1.11.5

Prior to these releases, TSB relied on OpenCensus to transmit metrics from Control Plane clusters to Management Plane clusters. OpenCensus has been sunsetted, and from these releases onwards, Control Plane clusters will use OTEL to transmit metrics to the Management Plane. To ease migration, the TSB Management Plane will support both OpenCensus and OTEL to meet the N-2 support commitment:

Release	Management Plane	Control Plane
1.16.x	OTEL	OTEL
1.15.x	OTEL and OpenCensus	OTEL
1.14.x	OTEL and OpenCensus	OTEL
1.13.1+	OTEL and OpenCensus	OTEL
1.13.0	OpenCensus	OpenCensus
1.12.7+	OTEL and OpenCensus	OTEL
1.12.0-6	OpenCensus	OpenCensus
1.11.5+	OTEL and OpenCensus	OTEL
1.11.0-4	OpenCensus	OpenCensus

As a general rule, an older Control Plane version (using OpenCensus) will function correctly with a newer Management Plane instance, provided the (N-2) compatibility is met.

OpenCensus support in the Management Plane may be disabled in two situations:

• New Management Plane installations will only enable OTEL support. Matching control plane installations will use OTEL, so no compatibility issues should arise
• An administrator can explicitly disable the legacy OpenCensus support in the MP, using the components.collectors.enableOpencensusBackwardsCompatibility: false flag in the Management Plane custom resource. They may wish to do this once all CP instances have been verified to use OTEL

Disable WASM extension mounting before upgrading to TSB 1.14.x

Starting with TSB 1.14.0 (and TSB 1.12.10 in the 1.12.x line), the mountInternalWasmExtensions setting in the ControlPlane CR defaults to false. In addition, starting with TSB 1.14.1, the control plane stops registering the wasmfetcher sidecar injection template altogether, because the Istio version shipped with these releases enables Kubernetes native sidecars, which are incompatible with the mounting mechanism.

If a Control Plane cluster is upgraded to TSB 1.14.1 or later while this setting is still enabled, TSB-managed gateway Deployments keep requesting the wasmfetcher injection template (via the inject.istio.io/templates: gateway,wasmfetcher pod annotation) while the upgraded Istio control plane no longer defines that template. Gateway pods that are already running are not affected, but any pod that needs to be recreated after the upgrade — due to a scale-up, pod deletion, node drain, or cluster upgrade — is rejected by the sidecar injection webhook:

admission webhook "object.sidecar-injector.istio.io" denied the request:
failed to run injection template: requested template "wasmfetcher" not found

and the gateway is left without replacement pods.

Upgrading to TSB 1.14.0 does not remove the template yet, but the next patch upgrade will — so perform the steps below before any upgrade into the 1.14.x line.

You are affected if the setting is enabled OR unset

This setting defaulted to true on TSB 1.13.x and earlier, and on TSB 1.12.9 and earlier. Your cluster is affected both when spec.components.istio.mountInternalWasmExtensions: true is explicitly set in the ControlPlane CR, and when the field is absent from a CR created on a release where it defaulted to true.

Before upgrading a Control Plane cluster to 1.14.x, perform the following steps on the current (pre-upgrade) version:

1. Explicitly disable the setting in the ControlPlane CR:

   spec:
     components:
       istio:
         mountInternalWasmExtensions: false

2. Wait for the TSB-managed gateway Deployments (ingress, egress and Tier-1 gateways) to be re-rendered and rolled out. Verify that the wasmfetcher template is no longer requested — the following must print gateway only, without wasmfetcher:

   kubectl get deploy <gateway-name> -n <gateway-namespace> \
     -o jsonpath='{.spec.template.metadata.annotations.inject\.istio\.io/templates}'

3. Proceed with the upgrade.

Disabling the setting does not disable WASM-based features such as WAF: the extensions are fetched through the wasmfetcher service at load time instead of being pre-mounted into the proxy pods. No changes to WasmExtension resources are required.

If you have already upgraded and gateway pods are failing with the error above, set mountInternalWasmExtensions: false as shown in step 1 and verify per step 2 that the annotation is removed from the gateway Deployments, which unblocks pod creation. If the annotation is not removed after the change, contact Tetrate support.

TSB 1.14.3 upgrade and rollback notes

Upgrading to 1.14.3: align IsolationBoundary revision versions

If your ControlPlane CR pins an Istio version in its IsolationBoundary revisions (spec.components.xcp.isolationBoundaries[].revisions[].istio.tsbVersion), update any revision still set to 1.14.0, 1.14.1, or 1.14.2 to 1.14.3 as part of the upgrade. This is due to an incompatible API change introduced in upstream Istio in 1.14.3. In-place upgrade within patch versions is safe.

spec:
  components:
    xcp:
      isolationBoundaries:
      - name: global
        revisions:
        - name: default
          istio:
            tsbVersion: 1.14.3   # update from 1.14.0 / 1.14.1 / 1.14.2

Revisions that do not set tsbVersion follow the current release automatically and need no change.

Rolling back the control plane from 1.14.3

A control plane rollback moves to an older set of APIs, and a few resources written by 1.14.3 carry newer values that older releases do not recognize yet. These are known issues with simple workarounds - apply the ones relevant to your rollback target.

Gateway API CRDs block xcp-operator-edge

1.14.3 advances the stored version of the experimental Gateway API CRDs (backendtlspolicies.gateway.networking.k8s.io, tlsroutes.gateway.networking.k8s.io). After the rollback, xcp-operator-edge can fail to reconcile these CRDs because the newer stored version is no longer listed in the older CRD spec (... must remain in spec.versions until a storage migration ...). Tell the edge operator to skip re-creating these CRDs by setting DISABLE_K8S_GATEWAY_API_CRD_CREATION=true on the xcp component, then restart it:

spec:
  components:
    xcp:
      kubeSpec:
        deployment:
          env:
          - name: DISABLE_K8S_GATEWAY_API_CRD_CREATION
            value: "true"

kubectl rollout restart deploy/xcp-operator-edge -n istio-system

GatewayDeployment status phase not recognized (target older than 1.12.14)

When rolling back from 1.14.3 to a release older than 1.12.14 (that is, 1.12.13 or earlier), the older control-plane operator may not recognize a newer GatewayDeployment status phase written by 1.14.3 (RECONCILIATION_DIRTY / RECONCILIATION_PAUSED) and stops reconciling with an error such as unknown value "RECONCILIATION_DIRTY" for enum ... GatewayPhase. Releases 1.12.14 and later already understand these values, so this step only applies to older targets.

Reset the affected GatewayDeployments to a phase the older operator understands, then restart the control-plane operator:

kubectl get gatewaydeployments.install.xcp.tetrate.io -A -o json \
  | jq -r '
      .items[]
      | select(.status.phase == "RECONCILIATION_DIRTY" or .status.phase == "RECONCILIATION_PAUSED")
      | [ .metadata.namespace, .metadata.name,
          (if (.status.workload.deployment.isReady and .status.workload.service.isReady)
           then "READY" else "PENDING" end) ]
      | @tsv' \
  | while IFS=$'\t' read -r ns name phase; do
      kubectl patch gatewaydeployment -n "$ns" "$name" \
        --subresource=status --type=merge -p "{\"status\":{\"phase\":\"$phase\"}}"
    done

kubectl rollout restart deploy/tsb-operator-control-plane -n istio-system