Using AWS CodeBuild with Jenkins and Capistrano

10 min readSep 23, 2017

https://commons.wikimedia.org/wiki/File%3AContainer_City3.jpg

As usual when I hit my head against the wall trying to get something done, I hasten to write down what I did in the form of a blog post. This time it’s about running AWS CodeBuild via the AWS Console, and also via Jenkins (either as a free-style project or as a pipeline, both using the AWS CodeBuild Jenkins plugin.)

There are many moving parts to getting all of this right. My big-picture goal here is to use AWS CodeBuild to do the heavy lifting of building code from GitHub via capistrano, then publishing a tar.gz archive of that processed code to S3, for further consumption by AWS CodeDeploy. I am not going to talk about CodeDeploy here though. I had a similar process going by using a Jenkins worker instance. The beauty of this approach is that I don’t need that, since AWS CodeBuild uses an ephemeral Docker container within its infrastructure.

Capistrano setup

Why use Capistrano? Because I already have a set of rake files that I used successfully with Capistrano in the usual approach of running the Capistrano tasks via ssh on the target node that I want to deploy the code to. In the case of AWS CodeBuild, I am running Capistrano locally on a Docker container launched by AWS CodeBuild, against the source code checked out on that container. After Capistrano finishes its processing of the code, AWS CodeBuild will tar and gz the processed code and upload it as an artifact to S3.

Here are some interesting snippets regarding the Capistrano side of things. First of all, I have a Dockerfile which builds a Docker image containing Capistrano and other utilities needed during the run of the Capistrano tasks:

FROM ubuntu:14.04# disable interactive functions
ARG DEBIAN_FRONTEND=noninteractive
ARG BUILD_KEY=builder.opensshRUN apt-get update && \
 apt-get install -y — allow-unauthenticated git-core curl zlib1g-dev build-essential libssl-dev libreadline-dev libyaml-dev libsqlite3-dev \
 sqlite3 libxml2-dev libxslt1-dev libcurl4-openssl-dev python-software-properties libffi-dev php5 php5-mysqlRUN git clone https://github.com/rbenv/rbenv.git ~/.rbenv && \
 echo ‘export PATH=”$HOME/.rbenv/bin:$PATH”’ >> ~/.bashrc && \
 echo ‘eval “$(rbenv init -)”’ >> ~/.bashrc && \
 exec $SHELL && \
 git clone https://github.com/rbenv/ruby-build.git ~/.rbenv/plugins/ruby-build && \
 echo ‘export PATH=”$HOME/.rbenv/plugins/ruby-build/bin:$PATH”’ >> ~/.bashrc && \
 exec $SHELL && \
 git clone https://github.com/rbenv/rbenv-gem-rehash.git ~/.rbenv/plugins/rbenv-gem-rehash && \
 ~/.rbenv/bin/rbenv install 2.2.3 && \
 ~/.rbenv/bin/rbenv global 2.2.3 && \
 /root/.rbenv/versions/2.2.3/bin/gem install capistrano -v “=3.4.0” && \
 ln -s /root/.rbenv/versions/2.2.3/bin/cap /usr/local/bin/cap && \
 /root/.rbenv/versions/2.2.3/bin/gem install capistrano-locally# Install rvm and compass gem for SASS image compilation
RUN curl https://raw.githubusercontent.com/rvm/rvm/master/binscripts/rvm-installer -o /tmp/rvm-installer.sh && \
 chmod 755 /tmp/rvm-installer.sh && \
 gpg — keyserver hkp://keys.gnupg.net — recv-keys D39DC0E3 && \
 /tmp/rvm-installer.sh stable — path /root/.rvm — auto-dotfiles — user-install && \
 /root/.rvm/bin/rvm get stable && \
 /root/.rvm/bin/rvm reload && \
 /root/.rvm/bin/rvm autolibs 3RUN /root/.rvm/bin/rvm install ruby-2.2.2 && \
 /root/.rvm/bin/rvm alias create default ruby-2.2.2 && \
 /root/.rvm/wrappers/ruby-2.2.2/gem install bundler && \
 /root/.rvm/wrappers/ruby-2.2.2/gem install compassCOPY var/capistrano /var/capistrano
COPY scripts/compile_sass.sh /root/compile_sass.shWORKDIR /var/capistrano/CMD [“cap”,”-T”]

I have a separate Jenkins job which builds this image and pushes it to Amazon ECR, tagging it with the name of the project I am building the code for (I’ll call it MYPROJECT in what follows).

AWS CodeBuild project setup

Next, I’ll talk about how to configure AWS CodeBuild so that you can run a build via the AWS Console. When you create a new project in the AWS CodeBuild screen of the console, you need to specify the following parameters:

Under “Configure your project”

Project name: MYPROJECT

Under “Source: What to build”

Source provider: I chose GitHub and I was prompted to authorize AWS CodeBuild to connect to my GitHub repositories
Repository: I chose “Use a repository in my account” (aka private repo)
Choose a repository: I selected the GitHub repo for MYPROJECT

Under “Environment: How to build”

Environment image: I chose “Specify a Docker image”
Environment type: I chose “Linux”
Custom image type: I chose “Amazon ECR”
Amazon ECR repository: I chose “capistrano”
Amazon ECR image: I chose the image tagged “MYPROJECT” in the capistrano ECR repository
Build specification: I chose “Use the buildspec.yml in the source code root directory” (I’ll describe this file in detail further below)

Under “Artifacts: Where to put the artifacts from this build project”

Type: I chose “Amazon S3”
Name: MYPROJECT
Path: I left it blank
Namespace type: None
Bucket name: codepipeline-us-west-2-myorganization

Under “Service role”

I chose “Create a service role in your account”
I left the role name as the suggested name “codebuild-MYPROJECT-service-role”

That’s it, at this point you can create the CodeBuild project.

AWS CodeBuild IAM role configuration

One important note about the codebuild-MYPROJECT-service-role IAM role. In order for AWS CodeBuild to pull images from ECR, I had to modify the permissions of this role and add the following snippet in the Statement portion of the role’s policy (per this article):


### BEGIN ADDING STATEMENT HERE ###{  “Action”: [    “ecr:GetRepositoryPolicy”,    “ecr:SetRepositoryPolicy”  ],  “Resource”: “*”,  “Effect”: “Allow”},### END ADDING STATEMENT HERE ###

For reference, here is the policy associated with the codebuild-MYPROJECT-service-role role:

{
 “Version”: “2012–10–17”,
 “Statement”: [
     {
         “Action”: [
             “ecr:GetRepositoryPolicy”,
             “ecr:SetRepositoryPolicy”
         ],
         “Resource”: “*”,
         “Effect”: “Allow”
     },
     {
         “Effect”: “Allow”,
         “Resource”: [
             “arn:aws:logs:us-west-2:MYAWSID:log-group:/aws/codebuild/MYPROJECT”,
             “arn:aws:logs:us-west-2:MYAWSID:log-group:/aws/codebuild/MYPROJECT:*”
         ],
         “Action”: [
             “logs:CreateLogGroup”,
             “logs:CreateLogStream”,
             “logs:PutLogEvents”
         ]
     },
     {
         “Effect”: “Allow”,
         “Resource”: [
             “arn:aws:s3:::codepipeline-us-west-2-*”
         ],
         “Action”: [
             “s3:PutObject”,
             “s3:GetObject”,
             “s3:GetObjectVersion”
         ]
     },
     {
         “Effect”: “Allow”,
         “Action”: [
             “ssm:GetParameters”
         ],
         “Resource”: “arn:aws:ssm:us-west-2:MYAWSID:parameter/CodeBuild/*”
     }
  ]
}

Note that other than the ECR statement I added, the policy also allows this role access to CodeWatch logs generated by AWS CodeBuild, to S3 buckets whose names start with codepipeline-us-west-2-(which is why I named the S3 bucket in the CodeBuild project definition as codepipeline-us-west-2-myorganization) and also to EC2 Parameter Store variables whose names start with /CodeBuild/. I will talk about this very handy Parameter Store feature when I discuss the buildspec.yml file.

We are done with the IAM role modifications. Next, I followed the instructions in the article I mentioned and did the following:

Open the Amazon ECS console at https://console.aws.amazon.com/ecs/.
Choose Repositories.
In the list of repository names, choose the name of the repository you created or selected. (NB: capistrano in my case)
Choose the Permissions tab, choose Add, and then create a statement.
For Sid, type an identifier (for example, CodeBuildAccess).
For Effect, leave Allow selected because you want to allow access to AWS CodeBuild.
For Principal, type codebuild.amazonaws.com. Leave Everybody cleared because you want to allow access to AWS CodeBuild only.
Skip the All IAM entities list.
For Action, select Pull only actions.
All of the pull-only actions (ecr:DownloadUrlForLayer, ecr:BatchGetImage, and ecr:BatchCheckLayerAvailability) will be selected.
Choose Save all.

This policy will be displayed in Policy document:

{
 “Version”: “2008–10–17”,
 “Statement”: [
   {
      “Sid”: “CodeBuildAccess”,
      “Effect”: “Allow”,
      “Principal”: {
          “Service”: “codebuild.amazonaws.com”
      },
      “Action”: [
          “ecr:GetDownloadUrlForLayer”,
          “ecr:BatchGetImage”,
          “ecr:BatchCheckLayerAvailability”
      ]
    }
 ]
}

The buildspec.yml file

At this point, AWS CodeBuild is configured properly and all that is left is to come up with a proper buildspec.yml file which needs to be dropped in the root folder of the GitHub repo associated with AWS CodeBuild.

buildspec.ymlis a file which tells AWS CodeBuild what commands to run inside the Docker container that you specified when you chose the image to run (in my case, the image tagged MYPROJECT in the capistrano ECR repo). There is very good documentation on the syntax and composition of this file. Here is how the file looks in my case:

version: 0.2env:
  variables:
    PROJECT: “MYPROJECT”
    STAGE: “local”
    TASK: “deploy”
    APPLICATION: “myapp”
    BRANCH: “develop”
    DEPLOY_TO: “/var/www/MYPROJECT.com”
    KEEP_RELEASES: “5”
    TMP_DIR: “/tmp”
  parameter-store:
    GITHUB_TOKEN: “/CodeBuild/MYPROJECT_github_token”phases:
  build:
    commands:
      — cd /var/capistrano
      — export REPO_URL=”https://${GITHUB_TOKEN}@github.com/myorganization/MYPROJECT.git"; cap $STAGE $TASK --trace
  post_build:
    commands:
      — cd $DEPLOY_TO
      — sed -i -e ‘s/[()]//g’ revisions.log
      — (cat revisions.log | awk ‘{print $2}’) > git_branch.log
      — (cat revisions.log | awk ‘{print $4}’) > git_commit.log
      — (cat revisions.log | awk ‘{print $8}’) > cap_release.log
      — cat git_branch.log; cat git_commit.log; cat cap_release.log
      — cd /root
      — tar -czf /root/${PROJECT}-source.tar.gz -C ${DEPLOY_TO} .
artifacts:
  files:
     — /root/${PROJECT}-source.tar.gz
  discard-paths: yes

A lot of stuff to take in here. First the env section. It is used to declare environment variables which then will be available to the phases of the build process (there is good documentation on env variables). All the env variables can be overridden when an AWS CodeBuild project is being built. Most of the ones I declared above are used by capistrano. Their values are retrieved in capistrano in config/deploy.rb like this:

set :application, ENV[‘APPLICATION’]
set :deploy_to, ENV[‘DEPLOY_TO’]
set :branch, ENV[‘BRANCH’] || ‘*/master’
set :keep_releases, ENV[‘KEEP_RELEASES’].to_i || 5
set :tmp_dir, ENV[‘TMP_DIR’]

Then the variables application, deploy_to, branch, etc are used in capistrano’s task files.

Note the use of the parameter-store section under env in buildspec.yml. The AWS CodeBuild documentation strongly recommends not to expose secrets such as passwords, or in my case GitHub tokens, as plain-text variable values in buildspec.yml. Instead, they recommend creating EC2 Systems Manager Parameter Store variables that can then be referenced in the parameter-store section. In terms of naming the Parameter Store variables, it is recommended that their names start with the name of the service that uses them. That way, you can restrict that service to only those variables it needs. In this case, the /CodeBuild/MYPROJECT_github_token variable starts with /CodeBuild/, and the IAM role codebuild-MYPROJECT-service-rolehas access only to Parameter Store variables starting with /CodeBuild/:

    {
         “Effect”: “Allow”,
         “Action”: [
             “ssm:GetParameters”
         ],
         “Resource”: “arn:aws:ssm:us-west-2:MYAWSID:parameter/CodeBuild/*”
     }

I wasn’t that familiar with the EC2 System Manager Parameter Store but I think I will get a lot of mileage out of it for managing secrets for my applications, especially when combined with AWS CodeBuild and AWS CodeDeploy. See also this article from Segment on their use of the Parameter Store.

Going back to the buildspec.yml file, I’ll talk about the phases section. You can somewhat arbitrarily split your build steps into 4 phases: install, pre_build, build and post_build. See again the build specification reference documentation.

In the build step in my file, I call cap, but first I change directories to the root of the Capistrano configuration, which is /var/capistrano in my case (that’s the way it is set up in the Dockerfile which builds the Capistrano image.) Initially, I was not doing this simple cd /var/capistrano and this cost me lots of hair pulling. I didn’t understand why the cap $deploy $task command was failing with this message:

23:04:32 [Container] 2017/09/20 23:04:32 Running command cap $STAGE $TASK --trace23:04:34 ** Invoke ensure_stage (first_time)23:04:34 ** Execute ensure_stage23:04:34 Stage not set, please call something such as `cap production deploy`, where production is a stage you have defined.

At first I thought that the $STAGE and $TASK variables are not correctly interpolated and that’s why cap was complaining. It turns out it was just the current directory not being the right one. Sigh.

Another thing to note is that before I call cap, I set the REPO_URL environment variable used by Capistrano by interpolating the GITHUB_TOKEN variable (containing the secret retrieved from EC2 Parameter Store) in the GitHub URL of the repository I want to check out in Capistrano.

The post_build phase begins by running some sed and awk commands in the DEPLOY_TO directory. They are used to retrieve the current release directory name, the git branch and the git commit of this particular build. Then the directory structure resulting from the cap run is tar-ed and gzipped in the /root directory of the container running the build process.

Finally, in the artifacts section of the build, the tar.gz file from the post_build step is uploaded to the location specified in the CodeBuild project MYPROJECT’s parameters — in my case the S3 bucket codepipeline-us-west-2-myorganization.

Jenkins setup

I’ll show how to integrate AWS CodeBuild in Jenkins using 2 methods. Before that, you need to install the Jenkins AWS CodeBuild Plugin. Then you need to create an IAM user and associate the AWSCodeBuildDeveloperAccess policy to it. Next, add the AWS Access Key and Secret Access Key for this user to Jenkins as Jenkins credentials of type “CodeBuild Credentials” and not “AWS Credentials “— this is important, otherwise things will not work and you’ll get a message like this when you try to run the Jenkins project:

Invalid Jenkins credentials ID. Credentials must be of type CodeBuildCredentials.

Method 1: Run AWS CodeBuild for MYPROJECT as a free-style Jenkins project

I created a new free-style Jenkins project and added a build step of type “AWS CodeBuild”. I also added a string parameter called BRANCH for this project, with a default value of develop for the Git branch I want to build. For the build step, I specified Jenkins CodeBuild credentials created above, then I chose “Use Project source” and in the Environment Variables Override section I specified: [ { BRANCH, $BRANCH } ]

This way, when I run this Jenkins project, I can specify a different BRANCH variable than the one defined in buildspec.yml, so I can build a specific revision of the code in the GitHub repo for MYPROJECT.

Method 2: Run AWS CodeBuild for MYPROJECT as a Jenkins pipeline

I created a Jenkins project of type Pipeline (parameterized as in method 1 with a parameter called BRANCH) and added this as the Pipeline script definition:

node {
   stage(“Run AWS CodeBuild for branch $BRANCH”) {
     awsCodeBuild credentialsId: ‘jenkins-codebuild-credentials’, 
                  credentialsType: ‘jenkins’,
                  sourceControlType: ‘project’,
                  envParameters: ‘’, 
                  envVariables: ‘[ { BRANCH, $BRANCH } ]’, 
                  projectName: ‘MYPROJECT’, 
                  region: ‘us-west-2’
   }
}

Note that I specified the credentialsId of the CodeBuild credentials created above (jenkins-codebuild-credentials in my case), the credentialsType (jenkins), the sourceControlType (project), the CodeBuild projectName (MYPROJECT), the AWS region (us-west-2) and the envVariables which will override the definitions in buildspec.yml. The syntax for envVariables is the same as in method 1: a list of key/value pairs of variable names and their values.

Now when you run the Jenkins project with either method 1 or 2, the AWS CodeBuild Jenkins plugin will invoke a run of the AWS CodeBuild project MYPROJECT and you’ll see the output of that run in the Jenkins console output. The AWS CodeBuild plugin also includes a link to a CloudWatch log stream where you can also see the results of the AWS CodeBuild run.

My next step is to integrate AWS CodeDeploy in this pipeline. There is some good documentation on it and I’ll write down my findings in another blog post.