Android Continuous Integration: Installing Jenkins on an Amazon Ec2 Instance

The end goal: I want to check my Android code into Bitbucket or Github and minutes later, my app is live in the Google Play Store.

Let’s summarise: Check in code; app is live; profit.

We will do this using an Amazon Ubuntu 14 EC2 Small Instance (basically a Unix computer in the cloud), Nginx, Jenkins and the Android SDK. And I’m assuming that you use Bitbucket or Github to host your android project(s).

That’s some serious hours you have saved and can instead spend on Minecraft building a replica Starship Enterprise.

Why host your CI Server in the Cloud?

Despite the numerous choices available for running your continuous integration on a hosted platform, I’ve always prefered having a physical computer, with maybe an oldskool KVM switch to give me complete control over the build server.

But having a physical computer seems so 1990s, and unless you have complete control over your IT network, you can’t take full advantage of the benefits of cloud computing. For a CI server this means that things like webhooks, auto-scaling, slave servers and redundancy are manual tasks… and having to poll your git repo for changes just makes me feel like a noob.

But is it really worth putting your build server into the cloud? Truth. I’m not certain yet. I haven’t had Jenkins running on EC2 long enough to say for sure, but my gut feel is that yes it is.

I gave myself a budget of US$10 per month. It seems like a reasonable figure, given the flexibility of having my own custom made system to play with, not to mention not having to actually buy another computer. After 2 days of fairly intensive setup and test builds, I finally got it all up and running. Amazon tells me that my projected cost for the month is US$5 , that’s without any scheduling or any sort of automation of the instance — so it’s been at least a worthwhile experiment!

It is pretty cool that Amazon allows you to set up billing alarms that notify you when you reach a predefined threshold. It also goes so far as to predict your monthly total costs and finally gives you a detailed breakdown of them. How much you pay depends mostly on your server’s uptime, how much data you transmit and how much storage space you consume. Minimise your usage of these three components and you save money.

Costs aside, there are definitely benefits to having your continuous integration server (or any server for that matter) in the cloud:

• Scalable — when you hit a memory/cpu/network/hard disk limit. You can actually magically grow your server on demand. Like Jack and the beanstalk!
• Redundancy and stability — it never goes down. Like. Never.
• Publicly accessible — no more flushing Darren-from-IT’s head down the toilet in his lunch break to get a port forwarded to your physical machine so that you can use webhooks and access the build server from the beach in Thailand.
• Move your servers around the globe — distributed team? No problem, the server can be mirrored to different locations around the globe. From Timbuktu to Saigon, Berlin to Mexico. Peace man.
• Maintenance — do kids these days even know how to partition hard drives, create disk images and setup RAID arrays? Didn’t think so.

How?

Here are a list of the things you should be at least a little familiar with already, but don’t be scared if you think you’re not an expert, neither am I, I’m just good at copying and pasting from StackOverflow.

• Jenkins
• Git
• Java
• nginx
• webhooks
• Google Play Publishing
• unix command line… see below

All you ever need

Here are a list of the main unix commands that I used, if they don’t scare you, read on.

sudo apt-get update
sudo apt-get install nginx
sudo /etc/init.d/nginx status
sudo vim /etc/nginx/nginx.conf
tail -f /var/log/nginx/error.log
sudo /etc/init.d/jenkins start
sudo cat /var/lib/jenkins/secrets/initialAdminPassword
sudo apt-get install git
sudo apt-get install openjdk-7-jdk
sudo apt-get install lib32stdc++6 lib32z1
cd ~
wget http://dl.google.com/android/android-sdk_r24.4.1-linux.tgz
tar -xvzf android-sdk_r24.4.1-linux.tgz
export ANDROID_HOME=~/android-sdk-linux
export PATH=$ANDROID_HOME/tools:$PATH
export PATH=$ANDROID_HOME/platform-tools:$PATH
sudo su jenkins
ssh-keygen -t rsa -b 4096 -C “myemail@gmail.com”
cat /var/lib/jenkins/.ssh/id_rsa.pub
android list sdk — all — extended
android update sdk — filter [names or id’s, id’s dont always work, use the — all and then the extended name for a particular package] — no-ui

Not too bad right? Good let’s jump straight in

Setting up an Ec2 instance

This can be daunting at first because the Amazon Console looks like the cockpit of a jumbo. And in many ways it is. Just one false move and your instance will crash harder than the 747 in Die Hard 2. This actually happened to me (the ec2 instance crashed that is, I have never crash landed in a 747… but don’t you ever fantasise about a mid air collision? I digress…) after I finally got Jenkins all up and working but needed to move to a larger instance because I ran out of memory. I shut down the instance but it was part of some Auto Scaling group and so Amazon started terminating instances and starting up new ones … basically the technical equivalent of dropping an Uzi on the ground in a server room. Big mistake. I went back to the drawing board, and this time I wrote down exactly what I did for all you lovely boys and girls.

Start with a micro instance, use Ubuntu 14 — Jenkins, nginx and Android all support Ubuntu. I had to upgrade to a small instance because one of my builds required more physical memory than was available. I would recommend trying it with a test subject before doing it with your real instance, just incase it gets blown away like mine did.

Once you have an Amazon instance running, I have only two tips for you. Both can be done from in the Amazon console itself:

Create an Elastic IP. When you restart or upgrade or migrate or fart, amazon will change your instance’s public IP and DNS name. This means that anything that has to communicate with your server also has to be notified of the new address (ie ssh, github, bitbucket, all your browser links and Jenkins itself). Elastic IP is basically an IP that you can then point to any instance — a bit like a reverse proxy that is really easy to manage.

Backup your volume when you reach installation checkpoints. You can do this while the machine is live. Maybe it costs you a few cents in storage. Worth it. Did I mention that I lost a full day’s work when mine was blown away?

SSH like a boss

If you are using git you hopefully know something about SSH. SSH is used by git, but git keeps it at arm’s length from most users. Most git users don’t really get too up close and personal with SSH, but now you will be using it to remote into your EC2 instance.

You now have two SSH keys, an id_rsa pair and Amazon has given you a pem file. Wtf? It’s time to start automating your terminal life. If you are just using the regular terminal, do yourself a favour and install iTerm and ohmysh to automatically become the coolest kid in the open space office. And if you’re using Windows… just leave.

So now you have the Mustang of the Terminals but wait there’s more, if you really want to impress Gloria from Accounting, make a casual remark about your SSH config file as she walks past. No more typing in the whole hostname every time you want to push to git or remote into amazon... no sirreee configure ssh to automatically choose the right key. I can’t tell you how many seconds of my life that has saved me — not to mention looks of admiration from… well nobody ever.

Nginx … because using port 8080 is for noobs

Jenkins by default will run on port 8080 and port 80 is reserved for processes running as root. And running Jenkins as root is very naughty.

So, if we want to use something professional like

http://www.iseewithmylittleci.com 

instead of

http://www.suchadev.com:8080

we need to use a reverse proxy to route all requests on port 80 to port 8080.

Ja ja ja, the guys with the beards are all shaking their heads because you can use apache or iptables, but it’s the new millennium so we use nginx because it’s super easy to setup. And there are plenty of how to setup nginx articles out there that are actually in plain English and not Klingon. Follow the guide and you will have it running in no time. Here’s the basic idea:

sudo apt-get update
sudo apt-get install nginx

Here are some cool commands to impress your friends with:

sudo /etc/init.d/nginx status
sudo vim /etc/nginx/nginx.conf

If this is your first time using a command line text editor. Here is a good starters guide to vim, and a cool cheat sheet to impress the ladies. You will need to make the following change to the nginx.conf file:

http {
    ##
    # Basic Settings
    ##
    server {
        listen 80;
        server_name 42.58.16.179; //your Elastic IP!
        location / {
            proxy_pass http://127.0.0.1:8080;
        }
    }
    ....

and restart nginx for the settings to take affect. Tail the log to make sure you haven’t ballsed up the config file, it’s easier to do now you are using old skool command line editors.

tail -f /var/log/nginx/error.log

OK nginx is is purring away, reversing the shit out of your incoming requests. Now it’s time to install monsieur Jenkins himself.

Note that Jenkins will complain about your reverse proxy not being configured correctly because it needs to rewrite the response headers… there is a massive config file in the Jenkins documentation that apparently explains this but … well it’s in Klingon. Feel free to send me the important parts of that config file if you figure it out.

Installing Jenkins on Ubuntu: RTFM

Copy and paste this

wget -q -O - https://jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
sudo sh -c 'echo deb http://pkg.jenkins-ci.org/debian binary/ > /etc/apt/sources.list.d/jenkins.list'
sudo apt-get update
sudo apt-get install jenkins

Taken verbatim from the Jenkins website:

Installing Jenkins on Ubuntu

It doesn’t really get any easier than that. Once you have it installed, check that it’s started using:

sudo /etc/init.d/jenkins start

Use your browser to see if you are up and running:

http://www.iseewithmylittleci.com

and if that doesn’t work try going directly to port 8080 instead

http://www.suchadev.com:8080

if the 8080 one works and the 80 url doesn’t, nginx isn’t configured/running properly. If the 8080 address doesn’t work, try substituting the elastic IP with the current DNS hostname of your instance in the console. If all else fails, try

wget 127.0.0.1:8080

This will see if it’s even running locally on the instance itself.

If Jenkins does work you should be asked for the admin password, this command will help you there:

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Create your first Job

The basic strategy is to create and build your project and fix the errors as they present themselves. I will assume that you are familiar with Jenkins and know what Jenkins Plugins you require to run your job. I will focus more on the problems that come up.

The first thing you do is setup the git repo and get an error message when you press build, that git isn’t even installed. Once you have git installed, Java will not work properly. Then the Android SDK will give you problems, then you will need to setup environment variables. And once the build is finally working, you should setup webhooks and notifications to further automate your life.

But first things first:

Installing git

sudo apt-get install git

Jenkins also needs it’s own ssh key so that it can access your git repo.
Switch to the jenkins user and generate ssh keys:

sudo su jenkins
ssh-keygen -t rsa -b 4096 -C “myemail@gmail.com”
Generating public/private rsa key pair.
Enter file in which to save the key (/var/lib/jenkins/.ssh/id_rsa):

Copy the public ssh key and add it to github/bitbucket:

cat /var/lib/jenkins/.ssh/id_rsa.pub

You should now be able to at least retrieve the git repo when you press Build Now.

Installing the Android SDK

Installing the SDK involves downloading the latest SDK and unzipping it into a folder on the server, it doesn’t really matter where, but I prefer to put it into the home directory.

Note: you don’t want android studio, command line tools only.

cd ~
wget http://dl.google.com/android/android-sdk_r24.4.1-linux.tgz
tar -xvzf android-sdk_r24.4.1-linux.tgz

Note: replace the wget url with the latest sdk (you can find the link here http://developer.android.com/sdk/index.html)

You will now have a directory in your home directory with the following name:

/home/user/ubuntu/android-sdk-linux

and it will contain all the files you need to get started. Note down the location of the sdk — you will need that later.

You will also need these old skool libraries — don’t ask why, just do it.

sudo apt-get install lib32stdc++6 lib32z1

Now you have the SDK, you need to update it. Having worked on Android and installed Android Studio, you will of course be familiar with the Android SDK Manager:

Android SDK Manager

What you may not be aware of is that this graphical user interface is also available (feature complete) at the command line. Which is handy because your Amazon EC2 instance sure doesn’t have a GUI!

First of all, add these lines to your .bashrc file. It makes life easier.

export ANDROID_HOME=~/android-sdk-linux
export PATH=$ANDROID_HOME/tools:$PATH
export PATH=$ANDROID_HOME/platform-tools:$PATH

You will also need to add the ANDROID_HOME environment variable to Jenkins global variables, use the root path which is something like:

/home/user/ubuntu/android-sdk-linux

Which components of the Android SDK you actually choose to install is completely dependant on your project. Here are some handy commands to get started:

android list sdk — all — extended
android update sdk — filter 1,3,5 — no-ui

Install Java

sudo apt-get install openjdk-7-jdk

Github Webhooks

If you are using Github to host your project, install the plugin and rtfm:
Manage Jenkins — Configuration — Github — add github server.

• Leave the default server address there, unless you are running an enterprise account from Github.
• Credentials — generate a personal access token from your settings page on github’s website and put it here.
• Leave manage hooks ticked

Go back to your build job’s config:

• General — tick github project and put the url in there
• Build triggers — tick build when pushed to github

Push some test commits to your repo to check if it’s working.

Bitbucket Webhooks

Not as straightforward as Github, but still really easy.

From your repo’s settings on the Bitbucket website:

Repo settings - enable webhooks

• url is [your jenkins url]/bitbucket-webhooks/
  (trailing slash is important)

Make sure the Bitbucket Plugin is installed on your Jenkins machine

• Build triggers — tick build when pushed to Bitbucket

Push some test commits to your repo to check if it’s working.

Google Play Publisher

The hard part here is just getting the credentials right, but the plugin’s documentation is very clear about how to do that. Make sure you are the owner of the account and everything else will fall into place.

Once you have navigated the console you will get a json key file. Give this key to Jenkins and you can now upload to Google Play.

Finito

You should now be able to push to git, github/bitbucket will then tell your new jenkins server, jenkins will kickoff the build, if it succeeds, it will upload the app to the Play Store. Voila!

There are a million hiccups that can and will stop you along the way. Feel free to get in touch if you have a drama and I will add it into this guide. Also if you have some cool advice for me, especially around automating my ec2 instances I would love to hear it.

…but if you’re upset about my comments about beards and Windows … eat my shorts man.