Armory Diagnostics

Turning on Armory Diagnostics

When working with our support teams, you might be asked about enabling Armory Diagnostics. This sends the log and event data from your system to Armory so that we can remotely investigate what might be going on with your system, resulting in a faster turnaround on solutions.

Here’s the Halyard command to enable this feature:

hal armory diagnostics enable
hal armory diagnostics edit --logging-enabled=true

And similarly, to disable it:

hal armory diagnostics disable
hal armory diagnostics edit --logging-enabled=false

You need to to run hal deploy apply after enabling or disabling the diagnostics.

What Gets Sent

You’re probably wondering exactly what gets sent to Armory when you’re sharing your log output with us.

There are actually two streams of output that we will receive from an enabled system. The first is the easiest, it’s a copy of what gets logged in the pod (the same output you get from running kubectl logs -f <pod_id>). This tends to be status messages, but we will also capture any software failures that might have gotten caught. This often gives us the best insight into why something is going wrong.

The second stream is actually the set of events being passed internally from Spinnaker microservice to microservice. These are JSON payloads that are then tagged on our side with your unique customer ID (so we can isolate your instances), a timestamp, and then two primary keys, “details” and “content”.

The Details key maps some very common elements for the event:

  • application – The name of the Spinnaker application
  • created – Timestamp of when the event was created
  • requestHeaders – HTTP request headers for the internal request
  • source – The microservice that generated the event (ex. ‘clouddriver’)
  • type – An identifier as to what kind of event this was (ex. `orca:pipeline:starting’)

The Content section is variable, and depends on what kind of event is being tracked. Here are some of the more common events:

  • Web access events contain your basic web event data, the request headers, URL, User Agent, encoding directives, etc.

  • Build events pass along the name of the build job (from Jenkins, Travis, etc), the success/failure, duration of the job.

  • Webhook events (like commits from Github) will contain whatever the webhook has sent along in its payload. For most SCMs, this is just the name of the organization, the repo name, a hash representing the commit, and a timestamp. If you’ve set up any other webhook events that you using to trigger Spinnaker, take care with what data is being sent.

  • Pipeline events will contain all the same data you would see in Spinnaker if you click on “Source” under a pipeline’s stage. This includes the name of the application, pipeline, the build information, the trigger information, and the status and results of all the stages included in that pipeline (including stages that have not been executed yet).

Other events may be the result of performing operations within Spinnaker, such as creating a new load balancer, or resizing a server group.

A Note on Private Data Exposure Risks

The data passed out of Spinnaker is based on payloads controlled by the open source Spinnaker community (except for those proprietary microservices provided by Armory specifically). Armory is not aware of any cases where private data (passwords, source code, etc.) would be passed out of Spinnaker as part of an event payload during normal operation. Armory is not responsible for secrets a customer has entered into an otherwise unprotected field being transmitted (for example, entering a password into a “Notes” field of a pipeline stage).