- Prerequisites
- Overview
- Installation
- Creating and configuring the necessary AWS IAM roles for your Kubernetes cluster
- Installing the Armory Spinnaker Custom Resource Definitions (CRDs) into your Kubernetes cluster
- Installing the Armory Spinnaker Operator
- Creating a SpinnakerService Custom Resource
- Exposing your Spinnaker instance
- Set up an Ingress for
spin-deck
andspin-gate
- Configure Spinnaker to be aware of its endpoints
- Configuring TLS certificates
- Next steps
Spinnaker Operator is a Kubernetes Operator that makes it easier to install, deploy, and upgrade Armory Spinnaker.
The AWS Container Marketplace offering for Spinnaker Operator will install the Armory Spinnaker Operator in an EKS cluster.
Spinnaker can be installed in any namespace in your EKS cluster; this document assumes that Spinnaker will be installed in the spinnaker
namespace.
This document is intended only for users of the Armory AWS Container Marketplace offering that have purchased the offering; it will not work if you have not subscribed to the Armory Container Marketplace offering.
Please contact Armory (hello@armory.io) if you’re interested in an AWS Marketplace Private Offer.
Prerequisites
This document requires the following:
- An EKS cluster (Kubernetes 1.16 or above) configured with IAM roles for service accounts
- An ingress controller for your EKS cluster (this document assumes the EKS cluster is using the NGINX Ingress Controller)
cluster-admin
access on the EKS cluster- An AWS S3 bucket to store Spinnaker application and pipeline configuration
Overview
This document covers the following high-level steps:
- Creating and configuring the necessary AWS IAM roles for your Kubernetes cluster
- Installing the Armory Spinnaker Custom Resource Definitions (CRDs) into your Kubernetes cluster
- Installing the Armory Spinnaker Operator
- Creating a SpinnakerService Custom Resource
- Exposing your Spinnaker instance
Installation
Creating and configuring the necessary AWS IAM roles for your Kubernetes cluster
AWS IAM permissions are granted to Spinnaker via the use of AWS’s IAM roles for Kubernetes Service Accounts. This feature must be enabled at a cluster level. You need to create three IAM roles:
- An IAM role to be used by Operator (
spinnaker-operator
ServiceAccount inspinnaker-operator
Namespace) that will have these permissions:aws-marketplace:RegisterUsage
s3:*
on your AWS Bucket
- An IAM role to be used by the Spinnaker Front50 service (
front50
ServiceAccount inspinnaker
Namespace), that will have these permissions:s3:*
on your AWS Bucket
- An IAM role to be used by the Spinnaker Clouddriver service (
clouddriver
ServiceAccount inspinnaker
Namespace). It does not require any explicit permissions for now; if you want Spinnaker to deploy AWS resources (AWS EC2, AWS ECS, AWS Lambda, or other AWS EKS clusters), permissions for these can be added later.- AWS permissions are not needed to deploy to the EKS cluster where Spinnaker is installed.
If you do not already have an AWS S3 bucket created, create one with these settings:
- Versioning turned on (“Keep all versions of an object in the same bucket”)
- Default encryption turned on
- All public access blocked
You will end up with these three IAM roles:
arn:aws:iam::AWS_ACCOUNT_ID:role/eks-spinnaker-operator
granted to the Kubernetes Service Accountsystem:serviceaccount:spinnaker-operator:spinnaker-operator
arn:aws:iam::AWS_ACCOUNT_ID:role/eks-spinnaker-front50
granted to the Kubernetes Service Accountsystem:serviceaccount:spinnaker:front50
arn:aws:iam::AWS_ACCOUNT_ID:role/eks-spinnaker-clouddriver
granted to the Kubernetes Service Accountsystem:serviceaccount:spinnaker:clouddriver
IAM Role for Operator Pod
Create an IAM role to be used by the Operator pod (call it eks-spinnaker-operator
, and configure it for use by EC2; the trust relationships will be replaced later)
Grant it the AWS managed policy AWSMarketplaceMeteringRegisterUsage
Grant it an inline policy granting permissions on your S3 bucket (replace BUCKET_NAME
with the name of your bucket):
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::BUCKET_NAME",
"arn:aws:s3:::BUCKET_NAME/*"
]
}
]
}
For example:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::my-spinnaker-bucket",
"arn:aws:s3:::my-spinnaker-bucket/*"
]
}
]
}
Create this trust relationship on the IAM role; with these fields replaced:
- replace
AWS_ACCOUNT_ID
with your AWS account ID - replace
OIDC_PROVIDER
with the “OpenID Connect provider URL” for your Kubernetes cluster (with thehttps://
removed)
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::AWS_ACCOUNT_ID:oidc-provider/OIDC_PROVIDER"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"OIDC_PROVIDER:sub": "system:serviceaccount:spinnaker-operator:spinnaker-operator"
}
}
}
]
}
For example:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::111222333444:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/AAAABBBBCCCCDDDDEEEEFFFF00001111"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.us-east-1.amazonaws.com/id/AAAABBBBCCCCDDDDEEEEFFFF00001111:sub": "system:serviceaccount:spinnaker-operator:spinnaker-operator"
}
}
}
]
}
IAM Role for Front50 Pod
Create an IAM role to be used by the Operator pod (call it eks-spinnaker-front50
, and configure it for use by EC2; the trust relationships will be replaced later)
Grant it an inline policy granting permissions on your S3 bucket (replace BUCKET_NAME
with the name of your bucket):
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::BUCKET_NAME",
"arn:aws:s3:::BUCKET_NAME/*"
]
}
]
}
For example:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::my-spinnaker-bucket",
"arn:aws:s3:::my-spinnaker-bucket/*"
]
}
]
}
Create this trust relationship on the IAM role; with these fields replaced:
- replace
AWS_ACCOUNT_ID
with your AWS account ID - replace
OIDC_PROVIDER
with the “OpenID Connect provider URL” for your Kubernetes cluster (with thehttps://
removed)
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::AWS_ACCOUNT_ID:oidc-provider/OIDC_PROVIDER"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"OIDC_PROVIDER:sub": "system:serviceaccount:spinnaker:front50"
}
}
}
]
}
For example:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::111222333444:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/AAAABBBBCCCCDDDDEEEEFFFF00001111"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.us-east-1.amazonaws.com/id/AAAABBBBCCCCDDDDEEEEFFFF00001111:sub": "system:serviceaccount:spinnaker:front50"
}
}
}
]
}
IAM Role for Clouddriver Pod
Create an IAM role to be used by the Operator pod (call it eks-spinnaker-clouddriver
, and configure it for use by EC2; the trust relationships will be replaced later)
It will not need explicit AWS permissions (for now).
Create this trust relationship on the IAM role; with these fields replaced:
- replace
AWS_ACCOUNT_ID
with your AWS account ID - replace
OIDC_PROVIDER
with the “OpenID Connect provider URL” for your Kubernetes cluster (with thehttps://
removed)
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::AWS_ACCOUNT_ID:oidc-provider/OIDC_PROVIDER"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"OIDC_PROVIDER:sub": "system:serviceaccount:spinnaker:clouddriver"
}
}
}
]
}
For example:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::111222333444:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/AAAABBBBCCCCDDDDEEEEFFFF00001111"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.us-east-1.amazonaws.com/id/AAAABBBBCCCCDDDDEEEEFFFF00001111:sub": "system:serviceaccount:spinnaker:clouddriver"
}
}
}
]
}
Installing the Armory Spinnaker Custom Resource Definitions (CRDs) into your Kubernetes cluster
Download the Kubernetes manifest for Operator, and install them into your Kubernetes cluster:
mkdir -p spinnaker-operator && cd spinnaker-operator
bash -c 'curl -L https://github.com/armory/marketplace/releases/latest/download/marketplace.tgz | tar -xz'
# Install or update CRDs cluster wide
kubectl apply -f manifests/crds/
Installing the Armory Spinnaker Operator
The manifest for the Armory Spinnaker Operator has to be updated with this:
AWS_ACCOUNT_ID
(in the ServiceAccount annotation) has to be updated with your account ID, so the ServiceAccount can accesss your AWS IAM roles
export AWS_ACCOUNT_ID=111122223333
sed -i.bak "s|AWS_ACCOUNT_ID|${AWS_ACCOUNT_ID}|g" manifests/operator/ServiceAccount.yaml
rm manifests/operator/ServiceAccount.yaml.bak
# Install the operator
kubectl apply -f manifests/operator
It may take a little bit of time for the Operator to come up. You can monitor its status by running this command:
kubectl -n spinnaker-operator get pod -owide
You’re looking for the Deployment to be completely up (READY of 2/2
and STATUS of Running
)
Creating a SpinnakerService Custom Resource
The manifest for the SpinnakerService object needs to be updated with these:
AWS_ACCOUNT_ID
(in both ServiceAccount annotations) has to be updated with your account ID, so the ServiceAccount can accesss your AWS IAM rolesBUCKET_NAME
(in the SpinnakerService) needs to be updated with the name of your AWS S3 Bucket
export AWS_ACCOUNT_ID=111122223333
export BUCKET_NAME=my-spinnaker-bucket
sed -i.bak "s|AWS_ACCOUNT_ID|${AWS_ACCOUNT_ID}|g" manifests/spinnaker/ServiceAccount-clouddriver.yaml
sed -i.bak "s|AWS_ACCOUNT_ID|${AWS_ACCOUNT_ID}|g" manifests/spinnaker/ServiceAccount-front50.yaml
rm manifests/spinnaker/ServiceAccount-clouddriver.yaml.bak
rm manifests/spinnaker/ServiceAccount-front50.yaml.bak
sed -i.bak "s|BUCKET_NAME|${BUCKET_NAME}|g" manifests/spinnaker/SpinnakerService.yaml
rm manifests/spinnaker/SpinnakerService.yaml.bak
# Install the operator
kubectl apply -f manifests/spinnaker
If everything is configured properly, the Spinnaker Operator should see the SpinnakerService custom resource, and start creating Kubernetes Deployments, ServiceAccounts, and Secrets in the spinnaker
Namespace. You can monitor this with the following:
kubectl -n spinnaker get all -owide
Exposing your Spinnaker instance
Once your Spinnaker instance is up and running, you need to configure it so that it is accessible. There are two main parts to this:
- Expose the
spin-deck
andspin-gate
services so that they can be reached by your end users (and client services) - Configure Spinnaker so that it knows about the endpoints it is exposed on
Given a domain name (or IP address) (such as spinnaker.domain.com or 55.55.55.55), you should be able to:
- Reach the
spin-deck
service at the root of the domain (http://spinnaker.domain.com
orhttp://55.55.55.55
) - Reach the
spin-gate
service at the root of the domain (http://spinnaker.domain.com/api/v1
orhttp://55.55.55.55/api/v1
)
You can use either http or https, as long as you use the same for both. Additionally, you have to configure Spinnaker to be aware of its endpoints.
This section assumes the following:
- The NGINX Ingress Controller has been installed in the EKS cluster
- You can set up a DNS CNAME Record pointing at the AWS Load Balancer in front of your NGINX Ingress Controller
Set up an Ingress for spin-deck
and spin-gate
First, determine a DNS name that you can use for Spinnaker, and set up a CNAME pointing that DNS name at your AWS Load Balancer. For example:
- NGINX Ingress Controller has created an NLB at
abcd1234abcd1234abcd1234abcd1234-1234567812345678.elb.us-east-1.amazonaws.com
- Desired domain name for Spinnaker is
spinnaker.domain.com
- Create a CNAME DNS Record pointing
spinnaker.domain.com
atabcd1234abcd1234abcd1234abcd1234-1234567812345678.elb.us-east-1.amazonaws.com
(you may also use an ALIAS Record in Route 53)
Then, create a Kubernetes Ingress to expose spin-deck
and spin-gate
. Create a file called spin-ingress.yml
with the following content:
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: spin-ingress
namespace: spinnaker
labels:
app: spin
cluster: spin-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
-
host: spinnaker.domain.com # Make sure to update this field
http:
paths:
- backend:
serviceName: spin-deck
servicePort: 9000
path: /
- backend:
serviceName: spin-gate
servicePort: 8084
path: /api/v1
Make sure the host field is updated with the correct DNS record
Apply the ingress file you just created:
kubectl -n spinnaker apply -f spin-ingress.yml
Configure Spinnaker to be aware of its endpoints
Update the spec.spinnakerConfig.config.security section of manifests/spinnaker/SpinnakerService.yaml
:
spec:
spinnakerConfig:
config:
# ... more configuration
security:
uiSecurity:
overrideBaseUrl: http://spinnaker.domain.com # Replace this with the IP address or DNS that points to our nginx ingress instance
apiSecurity:
overrideBaseUrl: http://spinnaker.domain.com/api/v1 # Replace this with the IP address or DNS that points to our nginx ingress instance
# ... more configuration
Make sure to specify http or https according to your environment
Apply the changes:
kubectl apply -f manifests/spinnaker/SpinnakerService.yaml
If you receive an error here, delete and recreate the SpinnakserService
Configuring TLS certificates
Configuring TLS certificates for ingresses is environment-specific. In general, you want to do the following:
- Add certificate(s) so that our ingress controller can use them
- Configure the ingress(es) so that NGINX (or the load balancer in front of NGINX, or your alternative ingress controller) terminates TLS using the certificate(s)
- Update Spinnaker to be aware of the new TLS endpoints, by replacing
http
byhttps
to override the base URLs in the previous section.
Next steps
Now that Spinnaker is running, here are potential next steps:
- Configuring certificates to secure our cluster (see this section for notes on this)
- Configuring authentication/authorization (see the Open Source Spinnaker documentation)
- Adding external Kubernetes accounts to deploy applications to (see Creating and Adding a Kubernetes Account to Spinnaker (Deployment Target))
- Adding AWS accounts to deploy applications to (see the Open Source Spinnaker documentation)