v2.28.6 Armory Continuous Deployment Release (Spinnaker™ v1.28.6)

2023/04/28 Release Notes

Note: If you’re experiencing production issues after upgrading Spinnaker, rollback to a previous working version and please report issues to http://go.armory.io/support.

Required Armory Operator version

To install, upgrade, or configure Armory 2.28.6, use Armory Operator 1.70 or later.

Security

Armory scans the codebase as we develop and release software. Contact your Armory account representative for information about CVE scans for this release.

Breaking changes

Orca requires RDBMS configured with UTF-8 encoding

This release includes a change from MySQL JDBC drivers to AWS drivers. We have seen this cause issues when the database is NOT in a utf8mb4 format.

Update kubectl to 1.20

Impact

With 2.28 of Spinnaker, we’ve updated the kubectl binary to a 1.20 release. You may have potential caching issues as a result due to certain resources in Kubernetes being removed and/or no longer supported. Look for failures in your log files and exclude resources that don’t match your target cluster. For example, adding “PodPreset” to the “omitKinds” on your Kubernetes account configs would cause Spinnaker to skip trying to cache resources that no longer be able to be cached in newer kubernetes releases.

Introduced in: Armory CD 2.28.0

Java 11.0.11+, TLS 1.1 communication failure

This is an issue between Java 11.0.11 and TLSv1.1. Only installations using TLSv1.1 will encounter communication failures between services when those services upgrade to Java 11.0.11+.

TLSv1.1 was deprecated in March of 2020 and reached end-of-life in March of 2021. You should no longer be using TLSv1.1 for secure communication.

Oracle released Java 11.0.11 in April of 2021. Java 11.0.11 dropped support for TLSv1.1. See the Java release notes for details.

Impact

Any services running under Java 11.0.11+ and using TLSv1.1 will encounter a communication failure. For example, you will see a communication failure between an Armory CD service running under Java 11.0.1 and MySQL 5.7 if the MySQL driver is using TLSv1.1.

The version of Java depends on the version used by the Docker container’s OS. Most Armory CD services are using Alpine 3.11 or 3.12, which does not use Java 11.0.11. However, Alpine 3.11 is end-of-life in November of 2021, and 3.12 is end-of-life in May of 2022. There is no guarantee that Java 11.0.11+ won’t be added to those container images by some other manner. You should modify your TLSv1.1 environment now so you don’t encounter communication failures.

Fix

Choose the option that best fits your environment.

1. Disable TLSv1.1 and enable TLSv1.2 (preferred):

   See Knowledge Base articles Disabling TLS 1.1 in Spinnaker and Specifying the Protocols to be used and How to fix TLS error “Reason: extension (5) should not be presented in certificate_request”.

2. Add a query parameter to the MySQL JDBC URIs:

   ?enabledTLSProtocols=TLSv1.2
   

   Note that this only fixes communication between Armory CD and MySQL.

   See MySQL communication failure when using TSL1.1 for more information.

Kubernetes version for deployment targets

Armory CD 2.26 no longer supports Kubernetes deployment targets prior to version 1.16.

Impact

Any Kubernetes deployment target must run version 1.16 or higher. If you try to deploy to clusters older than 1.16, you may see errors like the following in the UI:

Additionally, errors like the following appear in the Clouddriver logs:

2021-05-04 21:17:16.032 WARN 1 --- [0.0-7002-exec-9] c.n.s.c.k.c.ManifestController : Failed to read manifest

com.netflix.spinnaker.clouddriver.kubernetes.op.handler.UnsupportedVersionException: No replicaSet is supported at api version extensions/v1beta1
at com.netflix.spinnaker.clouddriver.kubernetes.op.handler.KubernetesReplicaSetHandler.status(KubernetesReplicaSetHandler.java:98) ~[clouddriver-kubernetes.jar:na]

2021-05-05 14:29:09.653 WARN 1 --- [utionAction-538] c.n.s.c.k.c.a.KubernetesCachingAgent : kubernetes/KubernetesCoreCachingAgent[1/1]: Failure adding relationships for service

com.netflix.spinnaker.clouddriver.kubernetes.op.handler.UnsupportedVersionException: No replicaSet is supported at api version extensions/v1beta1
at com.netflix.spinnaker.clouddriver.kubernetes.op.handler.KubernetesReplicaSetHandler.getPodTemplateLabels(KubernetesReplicaSetHandler.java:167)

Workaround

If you are affected by this change, perform the following tasks to update your applications:

• Upgrade the Kubernetes clusters that you are trying to deploy to. They must run version 1.16 or higher.
• If you have manifest files using deprecated APIs, update them to use newer APIs. For more information on which APIs are deprecated in each Kubernetes version and how to migrate, see the Kubernetes Deprecated API Migration Guide.

Introduced in: Armory CD 2.26.0

Kubernetes infrastructure in the UI

Starting in 2.26, the UI has been updated to more closely follow immutable infrastructure principles.

When you navigate to the Infrastructure tab in the UI for an application that has the Kubernetes provider configured, actions that change the Kubernetes infrastructure (such as Create or Delete), including Clusters, Load Balancers, and Firewalls, are no longer available.

Impact

Users do not see these actions in the UI by default. You must configure the UI to display them if you want your users to be able to perform them through the UI.

Workaround

Whether or not these actions are available in the UI is controlled by the following property in settings-local.yml:

window.spinnakerSettings.kubernetesAdHocInfraWritesEnabled = <boolean>;

This setting does not completely prevent users from modifying Kubernetes infrastructure through Armory CD. To do so, you must use the Policy Engine and write policies using the spinnaker.http.authz package.

If you use the Policy Engine to control which user roles can see the UI actions and be able to use them, you must set this property to true. Setting the value to false hides the buttons for all users regardless of whether you grant specific users access to the buttons through the Policy Engine.

This property affects Kubernetes infrastructure only. The behavior is slightly different depending on if the application has only the Kubernetes provider configured or Kubernetes and other providers, such as AWS.

If the application only has the Kubernetes provider configured, the following applies:

• When set to true, this property causes the UI to function as it did in previous releases. This allows people to manually create and delete Kubernetes infrastructure from the UI.
• When set to false, this property causes the actions to be unavailable to users. This prevents users from manually creating and deleting Kubernetes infrastructure from the UI. The users can still view the infrastructure but cannot make changes through the UI.

If the application includes Kubernetes and other providers, the following applies:

• When set to true, this property causes the UI to function as it did in previous releases. This allows people to manually create and delete Kubernetes infrastructure from the UI. Users can continue to select whether they want to create Kubernetes or other infrastructure in the UI.
• When set to false, this property causes Kubernetes to be unavailable as an option when trying to modify infrastructure from the UI. Users can still make changes to infrastructure for the application from cloud providers, such as AWS, but not Kubernetes.

Introduced in: Armory CD 2.26.0

Git Artifact Constraint Trigger

When setting an artifact constraint to limit when a pipeline executes on a git web hook, it’s likely the trigger will break with an error message The following required artifacts could not be bound: ‘[ArtifactKey(type=docker/image, …’ See https://github.com/spinnaker/spinnaker/issues/6757

Halyard deprecation

Halyard is no longer supported for installing Armory Continuous Deployment 2.27.0 and later. Use the Operator. For more information, see Halyard Deprecation.

Plugin Compatibility

Due to changes in the underlying services, older versions of some plugins may not work with Armory CD 2.28.x or later.

The following table lists the plugins and their required minimum version:

Breaking changes are kept in this list for 3 minor versions from when the change is introduced. For example, a breaking change introduced in 2.21.0 appears in the list up to and including the 2.24.x releases. It would not appear on 2.25.x release notes.

Known issues

MySQL Permission Repository error in 2.28.1-2.28.4

Armory is investigating an issue with FIAT when using MySQL as the backend permission repository. When it is enabled, unexpected authorization errors appear in logs across Clouddriver/Orca/Front50 for users and Service Accounts. Admins appear unaffected.

The issue does not exist for 2.28.0 and lower and has since been resolved as of 2.28.5. Environments using the default Redis backend do not encounter these errors. For more information about the changes that resolve this fix, please visit the following PR

Workaround: Use Redis backend, 2.28.0, 2.28.5+

Affected versions: Armory CD 2.28.1-2.28.4

Application Attributes section displays “This Application has not been configured”

There is a known issue that relates to the Application Attributes section under the Config menu. An application that was already created and configured in Spinnaker displays the message, “This application has not been configured.” While the information is missing, there is no functional impact.

Affected versions: Armory CD 2.28.0

Google App Engine account authentication

Spinnaker 1.28 introduced a new API to allow adding accounts to Spinnaker. This change required all existing providers to register themselves with the credentials repo system. Several cloud providers were missed as part of the migration, including Google and AppEngine providers. It’s possible other non-supported providers may also be broken. The result of this is that users cannot see any Google or App Engine accounts listed in the UI, and non-admin users get an Access Denied error when trying to run pipelines as no permissions are granted. Admins can still deploy to these accounts since they bypass any permissions restrictions.

Workaround:

2.28.X or later: Replace the Clouddriver image in 2.28 with the Armory Clouddriver 2.27 image

1.28.X or later: Replace the Clouddriver image in 1.28 with the OSS Clouddriver 1.27 image

SpEL expressions and artifact binding

There is an issue where it appears that SpEL expressions are not being evaluated properly in artifact declarations (such as container images) for events such as the Deploy Manifest stage. What is actually happening is that an artifact binding is overriding the image value.

Workaround:

2.27.x or later: Disable artifact binding by adding the following parameter to the stage JSON: enableArtifactBinding: false.

2.26.x or later: Change the artifact binding behavior in spec.spinnakerConfig.profiles.clouddriver (Operator) or clouddriver-local.yml (Halyard) to the following, which causes artifacts to only bind the version when the tag is missing:

kubernetes:
  artifact-binding:
    docker-image: match-name-only

This setting only binds the version when the tag is missing, such as image: nginx without a version number.

Affected versions: Armory CD 2.26.x and later

Pipelines-as-Code GitHub comments

There is a known issue where Pipelines-as-Code can generate hundreds of comments in a GitHub Pull Request (PR) when updates are made, such as when a module that is used by multiple dinghyfiles gets changed. These comments may prevent the GitHub UI from loading or related API calls may lead to rate limiting.

Affected versions: Armory CD 2.26.x and later

Workaround:

You can either manually resolve the comments so that you can merge any PRs or turn the notifications that Pipelines-as-Code sends to GitHub.

For information about about how to disable this functionality, see GitHub Notifications.

Secrets do not work with Spring Cloud Config

If you enable Spring Cloud Config all the properties (e.g. Docker) using Secrets are not resolved when Spring Cloud tries to refresh.

Affected versions:

• 2.26.x and later

Known Affected providers in Clouddriver:

• Kubernetes
• Cloudfoundry
• Docker

Workaround:

Do not use secrets for properties that are annotated with @RefreshScope.

Blue/Green(Red/Black) in the non-default namespace for Kubernetes fails

Version 2.28.6 introduced a bug which affects the Blue/Green(Red/Black) deployment strategy in Kubernetes.

In the afterStage during a deployment that uses the Blue/Green (Red/Black) rollout strategy, Spinnaker tries to select the rollout strategy and switch the traffic to the newly deployed rollout strategy.

However, when Orca sends the API request to Clouddriver for the rollout strategy details, Orca doesn’t include the namespace, which results in failure of the task with Manifest not found.

Early access

Dynamic Rollback Timeout

To make the dynamic timeout available, you need to enable the feature flag in Orca and Deck.

On the Orca side, the feature flag overrides the default value rollback timeout - 5 min - with a UI input from the user. You must add this block to the orca.yml file if you want to enable the dynamic rollback timeout feature.

{
  "rollback:"
  "timeout:"
    "enabled: true"
}

On the Deck side, the feature flag enhances the Rollback Cluster stage UI with timeout input.

window.spinnakerSettings.feature.dynamicRollbackTimeout = true;

The default is used if there is no value set in the UI.

Pipelines as Code multi-branch enhancement

Now you can configure Pipelines as Code to pull Dinghy files from multiple branches on the same repo. Cut out the tedious task of managing multiple repos; have a single repo for Spinnaker application pipelines. See Multiple branches for how to enable and configure this feature.

Enhanced BitBucket Server pull request handling

Trigger Spinnaker pipelines natively when pull requests are opened in BitBucket with newly added events including PR opened, deleted, and declined. See Triggering pipelines with Bitbucket Server in the Spinnaker docs for details.

Terraform template fix

Armory fixed an issue with SpEL expression failures appearing while using Terraformer to serialize data from a Terraform Plan execution. With this feature flag fix enabled, you will be able to use the Terraform template file provider. Please open a support ticket if you need this fix.

Automatically Cancel Jenkins Jobs

You now have the ability to cancel triggered Jenkins jobs when a Spinnaker pipeline is canceled, giving you more control over your full Jenkins workflow. Learn more about Jenkins + Spinnaker in this documentation.

Fixed Issues

Clouddriver

Fixed an issue where customers are unable to use mixed instances and other advanced launch template features. (https://github.com/spinnaker/spinnaker/issues/6755)

Deck

Fixed an issue where the UI crashes when running pipeline(s) with many stages.

Spinnaker Community Contributions

There have also been numerous enhancements, fixes, and features across all of Spinnaker’s other services. See the Spinnaker v1.28.6 changelog for details.

Detailed updates

Bill Of Materials (BOM)

Here’s the BOM for this version.

artifactSources:
  dockerRegistry: docker.io/armory
dependencies:
  redis:
    commit: null
    version: 2:2.8.4-2
services:
  clouddriver:
    commit: 95037f6d709c2dd03318b78c42c426106ee16a01
    version: 2.28.6
  deck:
    commit: e24db15a97545fd5dda3bdb88a70f640bc5a1104
    version: 2.28.6
  dinghy:
    commit: 912007004f7720b418cd133301c7fb20207e1f2f
    version: 2.28.6
  echo:
    commit: a602d9d5def0815cb52bdf6d695ca69cbf0abe3b
    version: 2.28.6
  fiat:
    commit: 45039aa7952cc409329faa30b8443667566459c1
    version: 2.28.6
  front50:
    commit: 3292cf2715a9e52bb4690601d4fd877407505ced
    version: 2.28.6
  gate:
    commit: c8058f4362f3f4ad108fa146d628a162445c7579
    version: 2.28.6
  igor:
    commit: 60964526194a1273a03a7ade1f8939751e337735
    version: 2.28.6
  kayenta:
    commit: 22fe5d47baacce77917c9026cefdedf91c64a956
    version: 2.28.6
  monitoring-daemon:
    commit: null
    version: 2.26.0
  monitoring-third-party:
    commit: null
    version: 2.26.0
  orca:
    commit: 06352546281e21597eb59d3e993e842b9259f142
    version: 2.28.6
  rosco:
    commit: 27d4a2b4a1d5f099b68471303d4fd14af156d46d
    version: 2.28.6
  terraformer:
    commit: ea9b0255b7d446bcbf0f0d4e03fc8699b7508431
    version: 2.28.6
timestamp: "2023-04-21 21:46:22"
version: 2.28.6

Armory

Armory Igor - 2.28.5…2.28.6

• chore(cd): update base service version to igor:2023.03.02.19.40.25.release-1.28.x (#419)
• chore(cd): update base service version to igor:2023.03.02.21.16.28.release-1.28.x (#420)

Armory Front50 - 2.28.5…2.28.6

• chore(cd): update base service version to front50:2023.03.27.19.08.11.release-1.28.x (#509)
• chore(cd): update armory-commons version to 3.11.5 (#534)

Armory Gate - 2.28.5…2.28.6

• chore(cd): update base service version to gate:2023.03.02.19.38.56.release-1.28.x (#523)
• chore(cd): update base service version to gate:2023.03.02.21.17.09.release-1.28.x (#524)
• chore(cd): update base service version to gate:2023.03.27.19.10.16.release-1.28.x (#530)
• chore(cd): update armory-commons version to 3.11.5 (#556)

Armory Orca - 2.28.5…2.28.6

• chore(cd): update base orca version to 2023.03.02.19.46.59.release-1.28.x (#597)
• chore(cd): update base orca version to 2023.03.02.21.24.49.release-1.28.x (#598)
• chore(cd): update base orca version to 2023.03.28.15.32.53.release-1.28.x (#608)
• chore(cd): update armory-commons version to 3.11.5 (#632)

Armory Deck - 2.28.5…2.28.6

• chore(alpine): Upgrade alpine version (backport #1302) (#1303)
• chore(build): only run security scans on PR merge (backport #1319) (#1327)
• chore(cd): update base deck version to 2023.0.0-20230410180145.release-1.28.x (#1330)
• chore(cd): update base deck version to 2023.0.0-20230410180145.release-1.28.x (#1332)
• chore(cd): update base deck version to 2023.0.0-20230420193214.release-1.28.x (#1333)

Terraformer™ - 2.28.5…2.28.6

• chore(alpine): Update alpine version (backport #497) (#499)

Armory Rosco - 2.28.5…2.28.6

• chore(cd): update base service version to rosco:2023.03.02.19.37.08.release-1.28.x (#500)
• chore(cd): update armory-commons version to 3.11.5 (#526)
• chore(cd): update armory-commons version to 3.11.6 (#529)

Armory Clouddriver - 2.28.5…2.28.6

• chore(cd): update base service version to clouddriver:2023.03.02.19.57.01.release-1.28.x (#812)
• chore(cd): update base service version to clouddriver:2023.03.03.02.32.42.release-1.28.x (#813)
• chore(cd): update base service version to clouddriver:2023.03.21.20.21.00.release-1.28.x (#821)
• chore(cd): update base service version to clouddriver:2023.03.23.14.57.28.release-1.28.x (#826)
• chore(cd): update base service version to clouddriver:2023.03.27.19.24.21.release-1.28.x (#829)
• chore(cd): update armory-commons version to 3.11.5 (#862)
• Bumped aws-cli to 1.22 for FIPS compliance (#854) (#863)
• chore(cd): update armory-commons version to 3.11.6 (#868)

Armory Kayenta - 2.28.5…2.28.6

• chore(cd): update base service version to kayenta:2023.03.03.04.50.48.release-1.28.x (#393)
• chore(cd): update base service version to kayenta:2023.03.28.19.09.35.release-1.28.x (#398)
• chore(cd): update armory-commons version to 3.11.5 (#423)
• chore(cd): update armory-commons version to 3.11.6 (#430)

Armory Echo - 2.28.5…2.28.6

• chore(cd): update base service version to echo:2023.03.02.19.38.24.release-1.28.x (#542)
• chore(cd): update base service version to echo:2023.03.02.21.17.15.release-1.28.x (#543)
• chore(cd): update base service version to echo:2023.03.27.19.10.41.release-1.28.x (#550)
• chore(cd): update armory-commons version to 3.11.5 (#576)
• chore(cd): update armory-commons version to 3.11.6 (#579)

Dinghy™ - 2.28.5…2.28.6

• chore(dependencies): bumped oss dinghy version to 20230201103309-a73a68c80965 (#480)
• chore(alpine): Upgrade alpine version (backport #481) (#482)

Armory Fiat - 2.28.5…2.28.6

• chore(cd): update base service version to fiat:2023.03.02.19.37.20.release-1.28.x (#441)
• chore(cd): update base service version to fiat:2023.03.16.17.55.44.release-1.28.x (#446)

Spinnaker

Spinnaker Igor - 1.28.6

• chore(dependencies): Autobump korkVersion (#1093)
• chore(dependencies): Autobump fiatVersion (#1094)

Spinnaker Front50 - 1.28.6

• chore(dependencies): Autobump korkVersion (#1214)
• chore(dependencies): Autobump fiatVersion (#1215)
• fix(sql): Populating lastModified field for pipelines when loading objects. (#1220) (#1223)
• chore(dependencies): Autobump fiatVersion (#1227)

Spinnaker Gate - 1.28.6

• chore(dependencies): Autobump korkVersion (#1624)
• chore(dependencies): Autobump fiatVersion (#1625)
• chore(dependencies): Autobump fiatVersion (#1632)

Spinnaker Orca - 1.28.6

• chore(dependencies): Autobump korkVersion (#4405)
• chore(dependencies): Autobump fiatVersion (#4406)
• chore(dependencies): Autobump fiatVersion (#4425)
• Fix/blue green deploy (backport #4414) (#4419)

Spinnaker Deck - 1.28.6

• fix: UI crashes when running pipeline(s) with many stages. (backport #9960) (#9973)
• fix(aws): Fixing bugs related to clone CX when instance types are incompatible with image/region (backport #9901) (#9975)

Spinnaker Rosco - 1.28.6

• chore(dependencies): Autobump korkVersion (#959)

Spinnaker Clouddriver - 1.28.6

• chore(dependencies): Autobump korkVersion (#5897)
• chore(dependencies): Autobump fiatVersion (#5898)
• fix(core): Renamed a query parameter for template tags (#5906) (#5908)
• chore(aws): Update AWS IAM Authenticator version (#5910)
• chore(dependencies): Autobump fiatVersion (#5916)

Spinnaker Kayenta - 1.28.6

• chore(dependencies): Autobump orcaVersion (#936)
• chore(dependencies): Autobump orcaVersion (#943)

Spinnaker Echo - 1.28.6

• chore(dependencies): Autobump korkVersion (#1261)
• chore(dependencies): Autobump fiatVersion (#1262)
• chore(dependencies): Autobump fiatVersion (#1270)

Spinnaker Fiat - 1.28.6

• chore(dependencies): Autobump korkVersion (#1026)
• fix(logs): Redacted secret data in logs. (#1029) (#1030)