v2.27.1 Armory Continuous Deployment Release LTS (Spinnaker™ v1.27.0)

2021/11/18 Release Notes

Note: If you’re experiencing production issues after upgrading Spinnaker, rollback to a previous working version and please report issues to http://go.armory.io/support.

Required Operator version

To install, upgrade, or configure Armory 2.27.1, use the following Operator version:

• Armory Operator 1.4.0 or later

Security

Armory scans the codebase as we develop and release software. Contact your Armory account representative for information about CVE scans for this release.

Breaking changes

Breaking changes are kept in this list for 3 minor versions from when the change is introduced. For example, a breaking change introduced in 2.21.0 appears in the list up to and including the 2.24.x releases. It would not appear on 2.25.x release notes.

Pipelines-as-Code Slack notifications stop working

Impact

After upgrading to 2.27.x, your Pipelines-as-Code Slack notifications may stop working even though they were working previously.

Hotfix

See the Dinghy Slack Notifications not working KB article for the Hotfix.

Introduced in: Armory CD 2.27.0

Java 11.0.11+, TLS 1.1 communication failure

This is an issue between Java 11.0.11 and TLSv1.1. Only installations using TLSv1.1 will encounter communication failures between services when those services upgrade to Java 11.0.11+.

TLSv1.1 was deprecated in March of 2020 and reached end-of-life in March of 2021. You should no longer be using TLSv1.1 for secure communication.

Oracle released Java 11.0.11 in April of 2021. Java 11.0.11 dropped support for TLSv1.1. See the Java release notes for details.

Impact

Any services running under Java 11.0.11+ and using TLSv1.1 will encounter a communication failure. For example, you will see a communication failure between an Armory CD service running under Java 11.0.1 and MySQL 5.7 if the MySQL driver is using TLSv1.1.

The version of Java depends on the version used by the Docker container’s OS. Most Armory CD services are using Alpine 3.11 or 3.12, which does not use Java 11.0.11. However, Alpine 3.11 is end-of-life in November of 2021, and 3.12 is end-of-life in May of 2022. There is no guarantee that Java 11.0.11+ won’t be added to those container images by some other manner. You should modify your TLSv1.1 environment now so you don’t encounter communication failures.

Fix

Choose the option that best fits your environment.

1. Disable TLSv1.1 and enable TLSv1.2 (preferred):

   See Knowledge Base articles Disabling TLS 1.1 in Spinnaker and Specifying the Protocols to be used and How to fix TLS error “Reason: extension (5) should not be presented in certificate_request”.

2. Add a query parameter to the MySQL JDBC URIs:

   ?enabledTLSProtocols=TLSv1.2
   

   Note that this only fixes communication between Armory CD and MySQL.

   See MySQL communication failure when using TSL1.1 for more information.

Kubernetes version for deployment targets

Armory CD 2.26 no longer supports Kubernetes deployment targets prior to version 1.16.

Impact

Any Kubernetes deployment target must run version 1.16 or higher. If you try to deploy to clusters older than 1.16, you may see errors like the following in the UI:

Additionally, errors like the following appear in the Clouddriver logs:

2021-05-04 21:17:16.032 WARN 1 --- [0.0-7002-exec-9] c.n.s.c.k.c.ManifestController : Failed to read manifest

com.netflix.spinnaker.clouddriver.kubernetes.op.handler.UnsupportedVersionException: No replicaSet is supported at api version extensions/v1beta1
at com.netflix.spinnaker.clouddriver.kubernetes.op.handler.KubernetesReplicaSetHandler.status(KubernetesReplicaSetHandler.java:98) ~[clouddriver-kubernetes.jar:na]

2021-05-05 14:29:09.653 WARN 1 --- [utionAction-538] c.n.s.c.k.c.a.KubernetesCachingAgent : kubernetes/KubernetesCoreCachingAgent[1/1]: Failure adding relationships for service

com.netflix.spinnaker.clouddriver.kubernetes.op.handler.UnsupportedVersionException: No replicaSet is supported at api version extensions/v1beta1
at com.netflix.spinnaker.clouddriver.kubernetes.op.handler.KubernetesReplicaSetHandler.getPodTemplateLabels(KubernetesReplicaSetHandler.java:167)

Workaround

If you are affected by this change, perform the following tasks to update your applications:

• Upgrade the Kubernetes clusters that you are trying to deploy to. They must run version 1.16 or higher.
• If you have manifest files using deprecated APIs, update them to use newer APIs. For more information on which APIs are deprecated in each Kubernetes version and how to migrate, see the Kubernetes Deprecated API Migration Guide.

Introduced in: Armory CD 2.26.0

Kubernetes infrastructure in the UI

Starting in 2.26, the UI has been updated to more closely follow immutable infrastructure principles.

When you navigate to the Infrastructure tab in the UI for an application that has the Kubernetes provider configured, actions that change the Kubernetes infrastructure (such as Create or Delete), including Clusters, Load Balancers, and Firewalls, are no longer available.

Impact

Users do not see these actions in the UI by default. You must configure the UI to display them if you want your users to be able to perform them through the UI.

Workaround

Whether or not these actions are available in the UI is controlled by the following property in settings-local.yml:

window.spinnakerSettings.kubernetesAdHocInfraWritesEnabled = <boolean>;

This setting does not completely prevent users from modifying Kubernetes infrastructure through Armory CD. To do so, you must use the Policy Engine and write policies using the spinnaker.http.authz package.

If you use the Policy Engine to control which user roles can see the UI actions and be able to use them, you must set this property to true. Setting the value to false hides the buttons for all users regardless of whether you grant specific users access to the buttons through the Policy Engine.

This property affects Kubernetes infrastructure only. The behavior is slightly different depending on if the application has only the Kubernetes provider configured or Kubernetes and other providers, such as AWS.

If the application only has the Kubernetes provider configured, the following applies:

• When set to true, this property causes the UI to function as it did in previous releases. This allows people to manually create and delete Kubernetes infrastructure from the UI.
• When set to false, this property causes the actions to be unavailable to users. This prevents users from manually creating and deleting Kubernetes infrastructure from the UI. The users can still view the infrastructure but cannot make changes through the UI.

If the application includes Kubernetes and other providers, the following applies:

• When set to true, this property causes the UI to function as it did in previous releases. This allows people to manually create and delete Kubernetes infrastructure from the UI. Users can continue to select whether they want to create Kubernetes or other infrastructure in the UI.
• When set to false, this property causes Kubernetes to be unavailable as an option when trying to modify infrastructure from the UI. Users can still make changes to infrastructure for the application from cloud providers, such as AWS, but not Kubernetes.

Introduced in: Armory CD 2.26.0

Halyard deprecation

Halyard is no longer supported for installing Armory Continuous Deployment 2.27.0 and later. Use the Operator. For more information, see Halyard Deprecation.

Plugin compatibility

Due to changes in the underlying services, older versions of some plugins may not work with Armory Continuous Deployment 2.27.x or later.

The following table lists the plugins and their required minimum version:

Known issues

Bake failures

The Packer version included with Rosco disregards package overrides that use the -var-file= option. This may cause bakes to fail.

Affected versions: 2.22.2 and later

SpEL expressions and artifact binding

There is an issue where it appears that SpEL expressions are not being evaluated properly in artifact declarations (such as container images) for events such as the Deploy Manifest stage. What is actually happening is that an artifact binding is overriding the image value.

Workaround:

2.27.x or later: Disable artifact binding by adding the following parameter to the stage JSON: enableArtifactBinding: false.

2.26.x or later: Change the artifact binding behavior in spec.spinnakerConfig.profiles.clouddriver (Operator) or clouddriver-local.yml (Halyard) to the following, which causes artifacts to only bind the version when the tag is missing:

kubernetes:
  artifact-binding:
    docker-image: match-name-only

This setting only binds the version when the tag is missing, such as image: nginx without a version number.

Affected versions: 2.26.x and later

Pipelines-as-Code GitHub comments

There is a known issue where Pipelines-as-Code can generate hundreds of comments in a GitHub Pull Request (PR) when updates are made, such as when a module that is used by multiple dinghyfiles gets changed. These comments may prevent the GitHub UI from loading or related API calls may lead to rate limiting.

Affected versions: 2.26.x and later

Workaround:

You can either manually resolve the comments so that you can merge any PRs or turn the notifications that Pipelines-as-Code sends to GitHub.

For information about about how to disable this functionality, see GitHub Notifications.

Highlighted updates

Deployment targets

General fixes

• Fixed an NPE related to the kubesvcCredentialsLoader.
• Fixed an issue where new namespaces failed to get created as part of a Deploy Manifest Stage. This issue occurred because of a validation problem.
• The Clouddriver service is now more resilient when starting. Previously, the service failed to start if an account that gets added has permission errors.
• Fixed an issue where a stage never completes if the manifest getting deployed is of the kind CSIDriver.

AWS Lambda

Note that these updates also require v1.0.9 of the AWS Lambda plugin.

• Fixed an issue in the UI where a stack trace gets displayed when you try to view functions.
• Fixed an issue where the UI did not show functions for an application if there are no configured clusters. Functions now appear instead of a 404 error.
• Caching behavior and performance have been improved. The changes include fixes for the following issues:
  • The Lambda API returns request conflicts (HTTP status 409).
  • Event Source Mapping of ARNs fails after initially succeeding. This occured during the Lambda Event Configuration Task.
  • Underscores (_) in environment variable names caused validation errors.
  • An exception related to event configs occurred intermittently during the Lambda Event Configuration Task.
  • Lambda function creation using the Deploy Lambda stage failed causing subsequent runs of the pipeline to encounter an error that states the function already exists.
  • The Lambda Cache Refresh Task did not refresh the cache. This led to issues where downstream tasks referenced older versions.
  • A permission issue caused the Infrastructure view in the UI (Deck) to not display Lambda functions.

Cloud Foundry

• Improved the resiliency of the Cloud Foundry provider. Invalid permissions for a caching agent, such as if permissions are missing for one region, no longer cause all deployments to that account to fail.

• Improved the caching behavior for the provider. Previously, the cache on a dedicated caching pod (such as when cache sharding or HA is enabled) may not have been updated if a different pod performed an operation that modifies the cache. This led to situations where the Cloud Foundry provider attempts actions for Server Groups that no longer existed. You can configure this behavior with the following properties:

  • expireAfterWrite: the amount of time (in seconds) to wait before expiring the cache after a write operation
  • expireAfterAccess: the amount of time (in seconds) to wait before expiring the cache after a access operation

• Improved error handling when a caching agent has insufficient permissions. A RuntimeException no longer occurs.

• Added an Unbind Service Stage to the Cloud Foundry provider. Services must be unbound before they can be deleted. Use this stage prior to a Destroy Service stage. Alternatively, you can unbind all services before they are deleted in the Destroy Service stage by selecting the checkbox in the stage to do this.

• Improved error handling when a Deploy Service stage fails

• The Cloud Foundry provider now supports the following manifest attributes:

  • Processes Health
  • Timeout
  • Random route

• Fixed an issue where attributes that involved health-check-type parameters were not respected. Armory Continuous Deployment (Spinnaker) now allows single parameters, such as timeout, to be set. Note that Armory Continuous Deployment does not perform validation on parameters, so you may encounter runtime exceptions for Cloud Foundry if you provide invalid parameters.

Instance registration

When you log in to the UI, you are prompted to register your Armory Continuous Deployment (Spinnaker) instance. When you register an instance, Armory provides you with a client ID and client secret that you add to your Operator manifest.

Registration is required for certain features to work.

Note that registration does not automatically turn on Armory Diagnostics. This means that registration does not send information about your apps and pipelines to Armory. If you are sending diagnostic information to Armory, registering your deployment ensures that Armory can know which logs are yours, improving Armory’s ability to provide support.

For more information, see Instance Registration.

Plugin compatibility

Due to changes in the underlying services, older versions of some plugins may not work with Armory Continuous Deployment 2.27.x or later.

The following table lists the plugins and their required minimum version:

Detailed updates

Bill Of Materials (BOM)

Here’s the BOM for this version.

version: 2.27.1
timestamp: "2021-11-04 21:25:35"
services:
    clouddriver:
        commit: a9bd461d8e5e862925b4c04f77774da97e2ecd73
        version: 2.27.1
    deck:
        commit: 8912ef4f4a4f171384497b787aee0e83847ffd5c
        version: 2.27.1
    dinghy:
        commit: 71f2ed003fe6b75d8e4f43e800725f2ff3a8a1fe
        version: 2.27.1
    echo:
        commit: 5ec4a67ff921c2bdefc776dda03a0780ff853bcf
        version: 2.27.1
    fiat:
        commit: f23a1b97346816afc4e8e85dfc3ac137282af64a
        version: 2.27.1
    front50:
        commit: f6339ea78bf6edc39250289b1a9e5545d53bc94f
        version: 2.27.1
    gate:
        commit: 68ccfd60091751f192cebde89572fe555b914ea5
        version: 2.27.1
    igor:
        commit: 9f4db42f060f6fb45aad4c038525d71528a2f9f5
        version: 2.27.1
    kayenta:
        commit: 1cdf69a42c359a1f12077b6b1cba5606ac3e5daf
        version: 2.27.1
    monitoring-daemon:
        version: 2.26.0
    monitoring-third-party:
        version: 2.26.0
    orca:
        commit: 522655a252c5a1a97f7745fe622ba06bccb99a8c
        version: 2.27.1
    rosco:
        commit: ac6fe57054e435c6058911c4caa177cba5fa64b3
        version: 2.27.1
    terraformer:
        commit: 5e69c32279c6516047eaf6de261d3632095677aa
        version: 2.27.1
dependencies:
    redis:
        version: 2:2.8.4-2
artifactSources:
    dockerRegistry: docker.io/armory

Armory

Armory Orca - 2.27.0…2.27.1

• [create-pull-request] automated change (#335) (#367)
• [create-pull-request] automated change (#371) (#372)
• Revert “[create-pull-request] automated change (#371) (#372)” (#373)

Armory Deck - 2.27.0…2.27.1

Armory Echo - 2.27.0…2.27.1

Armory Front50 - 2.27.0…2.27.1

• chore(build): remove platform build (#311)
• chore(cd): update base service version to front50:2021.10.28.01.40.53.release-1.27.x (#324)

Armory Igor - 2.27.0…2.27.1

Terraformer™ - 2.27.0…2.27.1

• Updating Terraform Versions 1.0.8 1.0.9 (#444) (#445)

Armory Rosco - 2.27.0…2.27.1

• [create-pull-request] automated change (#286) (#309)

Armory Clouddriver - 2.27.0…2.27.1

• [create-pull-request] automated change (#364) (#457)
• chore(cd): update base service version to clouddriver:2021.10.12.23.25.53.release-1.27.x (#462)
• chore(cd): update base service version to clouddriver:2021.10.12.23.52.07.release-1.27.x (#463)
• chore(cd): update base service version to clouddriver:2021.10.18.21.53.46.release-1.27.x (#466)
• chore(cd): update base service version to clouddriver:2021.10.19.23.47.12.release-1.27.x (#467)
• chore(cd): update base service version to clouddriver:2021.11.01.21.19.09.release-1.27.x (#476)
• chore(cd): update base service version to clouddriver:2021.11.02.19.30.35.release-1.27.x (#478)
• chore(cd): update base service version to clouddriver:2021.11.03.20.45.14.release-1.27.x (#480)

Dinghy™ - 2.27.0…2.27.1

Armory Kayenta - 2.27.0…2.27.1

• [create-pull-request] automated change (#268) (#280)

Armory Gate - 2.27.0…2.27.1

• [create-pull-request] automated change (#305) (#328)
• Rebuild artifacts (#336)
• chore(release): Bump armory header version (#337) (#338)

Armory Fiat - 2.27.0…2.27.1

• [create-pull-request] automated change (#232) (#256)
• [create-pull-request] automated change (#261) (#262)