Eugene Teo

Downgrading Vagrant with Brew

Sun, 27 Nov 2016 00:00:00 +0800

Update Jan 16, 2017 This issue has been addressed in vagrant-installers with commit e1867522. It’s not always a good idea to downgrade a software, so do make sure you know what you are doing.

I was trying to provision the spark-in-action VM with Vagrant. When I issued the vagrant box add command to download the VM, I got the following error message:

$ vagrant box add spark-in-action-box.json
==> box: Loading metadata for box 'spark-in-action-box.json'
    box: URL: file:///Users/eugeneteo/path/to/first-edition/spark-in-action-box.json
An error occurred while downloading the remote file. The error
message, if any, is reproduced below. Please fix this error and try
again.

If you look at this bug report, it looks like there is an issue with the version of libcurl.4.dylib in Vagrant 1.8.7:

$ /opt/vagrant/embedded/bin/curl -V
dyld: Library not loaded: @rpath/libcurl.4.dylib
  Referenced from: /opt/vagrant/embedded/bin/curl
  Reason: Incompatible library version: curl requires version 9.0.0 or later, but libcurl.4.dylib provides version 7.0.0
Trace/BPT trap: 5

Fortunately, it is easy to workaround this problem.

First, run brew cask remove to uninstall Vagrant. Next, look through Vagrant’s Cask file commits, find the version of Vagrant you want to downgrade from, and run brew cask install.

$ brew cask install https://raw.githubusercontent.com/caskroom/homebrew-cask/60531a2812005dd5f17dc92f3ce7419af3c5d019/Casks/vagrant.rb
==> Downloading https://raw.githubusercontent.com/caskroom/homebrew-cask/60531a2812005dd5f17dc92f3ce
######################################################################## 100.0%
==> Downloading https://releases.hashicorp.com/vagrant/1.8.6/vagrant_1.8.6.dmg
######################################################################## 100.0%
==> Verifying checksum for Cask vagrant
==> Running installer for vagrant; your password may be necessary.
==> Package installers may write to any location; options such as --appdir are ignored.
Password:
==> installer: Package name is Vagrant
==> installer: Installing at base path /
==> installer: The install was successful.
🍺  vagrant was successfully installed!

Issuing the vagrant box add command should work now!

$ vagrant box add spark-in-action-box.json
==> box: Loading metadata for box 'spark-in-action-box.json'
    box: URL: file:///Users/eugeneteo/path/to/first-edition/spark-in-action-box.json
==> box: Adding box 'manning/spark-in-action' (v1.0.0) for provider: virtualbox
    box: Downloading: https://s3-us-west-2.amazonaws.com/manning-code/spark-in-action.box
[...]

Hopefully, this issue gets resolved soon, and I will upgrade Vagrant back to the current version.

Replacing Travis CI with Docker

Sat, 26 Nov 2016 00:00:00 +0800

Since my paid Travis CI subscription has expired and I don’t intend to renew it, I wanted to see if I could use Docker containers to build and deploy my blog instead. It turns out to be an easy exercise, and I can even reuse my cibuild script to do exactly the same workflow as I did with Travis CI.

I used the official Jekyll and Ryan’s alpine-surge-bash Docker images. As of writing, Ryan has over 50K+ pulls, and it’s actively maintained. And even if the alpine-surge-bash image is no longer supported, I can easily build one from his Dockerfile.

Here’re the steps I took:

$ cd source # my Jekyll blog
$ docker run --rm -it -v `pwd`:/srv/jekyll -p 4000:4000 jekyll/jekyll bundle exec jekyll s --force_polling
$ docker run --rm -it -v `pwd`:/srv/jekyll jekyll/jekyll bundle exec sh /srv/jekyll/script/cibuild
$ docker run -it --rm andthensome/alpine-surge-bash surge token
$ docker run --rm -it -v `pwd`/_site/:/srv -e SURGE_LOGIN=[email] \
    -e SURGE_TOKEN=[token] andthensome/alpine-surge-bash surge -p /srv/

Did you do it differently? Share with me how you did it!

Using Travis CI to build and deploy sites

Tue, 10 May 2016 00:00:00 +0800

I finally got around the time to use Travis CI for my Jekyll blog. Travis CI is a continuous integration service for projects hosted on GitHub.

For those who have used Jekyll, you would know that each time you make changes to your blog, you have to regenerate the static pages, perhaps test them, and then deploy the changes to your hosting provider, which in my case is Surge. Now, Travis CI does all that for me automatically whenever I push an update to GitHub!

Here’re what I did:

Enable Travis builds for your GitHub repository. Enable Build only if .travis.yml is present in the settings page
Create SURGE_LOGIN and SURGE_TOKEN variables with the surge token command
Create a build and test script, and commit it to your repository. For some reasons, Travis CI had issues with a few of my external links, so I disabled the external link checker with the --disable-external option. Make sure you fix all the issues that htmlproofer has found until you see the HTML-Proofer finished successfully message

#!/usr/bin/env bash
set -e # halt script on error

bundle exec jekyll build
bundle exec htmlproofer ./_site --disable-external
# bundle exec htmlproofer ./_site

Update your Gemfile so that Travis can automatically install all the Ruby dependencies with bundle install. For example,

$ grep -v ^# Gemfile
source 'https://rubygems.org'

gem 'jemoji'
gem 'jekyll-redirect-from'
gem 'jekyll-paginate'
gem 'kramdown'
gem 'rouge'
gem 'html-proofer'
gem 'minitest'
gem 'html-pipeline'

Commit the .travis.yml configuration file

language: ruby
rvm:
- 2.3.0 # ruby 2.3.0p0 (2015-12-25 revision 53290) [x86_64-darwin15]

before_script:
 - chmod +x ./script/cibuild
 - npm install --global surge

script: ./script/cibuild

branches:
  only:
  - source

env:
  global:
  - NOKOGIRI_USE_SYSTEM_LIBRARIES=true # speeds up installation of html-proofer

sudo: false # route your build to the container-based infrastructure for a faster build

after_success:
   surge -p ./_site/

When Travis CI detects a push to GitHub, it will set up a build environment in a clean VM, make ./script/cibuild an executable, install surge, run the script only on the source branch, and if the build is successful, deploy the blog to Surge.

That’s all you have to do. Hope you find these steps useful.

Amazon EMR, Apache Spark, and Apache Zeppelin

Sat, 12 Mar 2016 00:00:00 +0800

I wrote a couple of guest posts about getting started with Apache Spark and Apache Zeppelin on Amazon EMR. These articles assumed that an IAM user has been created with AmazonElasticMapReduceFullAccess managed policy attached to it. In this blog post, I will show you how to set up this IAM user for EMR.

To begin using the AWS CLI, you need to create an administrative IAM user and configure it to use its access keys. After that, we can then create the IAM user for EMR. We will call this user emr.

$ aws iam create-user --user-name emr
{
    "User": {
        "UserName": "emr",
        "Path": "/",
        "CreateDate": "2016-03-11T14:32:08.782Z",
        "UserId": "AIDAID746LKJUJI5SGLOC",
        "Arn": "arn:aws:iam::548698731369:user/emr"
    }
}

To attach a policy to user emr, we can call the aws iam attach-user-policy command with the policy ARN.

$ aws iam attach-user-policy --policy-arn arn:aws:iam::aws:policy/AmazonElasticMapReduceFullAccess --user-name emr

We also want to grant the user with the iam:GetUser action. This way, we can verify that we are executing the AWS CLI commands as user emr.

$ cat iam-get-user.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
	"iam:GetUser"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:iam::548698731369:user/emr"
    }
  ]
}
$ aws iam create-policy --policy-name make-emr-get-user --policy-document file://iam-get-user.json
{
    "Policy": {
        "PolicyName": "make-emr-get-user",
        "CreateDate": "2016-03-11T15:05:22.936Z",
        "AttachmentCount": 0,
        "IsAttachable": true,
        "PolicyId": "ANPAJ3I5BYXIMET5E5UVC",
        "DefaultVersionId": "v1",
        "Path": "/",
        "Arn": "arn:aws:iam::548698731369:policy/make-emr-get-user",
        "UpdateDate": "2016-03-11T15:05:22.936Z"
    }
}
$ aws iam attach-user-policy --user-name emr --policy-arn arn:aws:iam::548698731369:policy/make-emr-get-user

Once this is done, we can go ahead and create an access key for user emr.

$ aws iam create-access-key --user-name emr
{
    "AccessKey": {
        "UserName": "emr",
        "Status": "Active",
        "CreateDate": "2016-03-11T14:37:03.144Z",
        "SecretAccessKey": "ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890abcd",
        "AccessKeyId": "ABCDEFGHIJKLMNOPQRST"
    }
}

We will export two environment variables. After that, run aws iam get-user and it should show you information about user emr.

$ export AWS_SECRET_ACCESS_KEY=ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890abcd
$ export AWS_ACCESS_KEY_ID=ABCDEFGHIJKLMNOPQRST
$ aws iam get-user
{
    "User": {
        "UserName": "emr",
        "Path": "/",
        "CreateDate": "2016-03-11T14:32:08Z",
        "UserId": "AIDAID746LKJUJI5SGLOC",
        "Arn": "arn:aws:iam::548698731369:user/emr"
    }
}

If you see this error, that means that you did not grant the user with the iam:GetUser action.

$ aws iam get-user

A client error (AccessDenied) occurred when calling the GetUser operation: User: arn:aws:iam::548698731369:user/emr is not authorized to perform: iam:GetUser on resource: arn:aws:iam::548698731369:user/emr

After performing all these steps, you can now follow along with my articles to create your own EMR cluster. Have fun!

Hello World

Sat, 02 Jan 2016 00:00:00 +0800

What better way to start the year than beginning a new blog?

I plan to use this space to build my portfolio of data science and data engineering projects. I also intend to write about my technology and business endeavours as I tackle them. More importantly, I want to get into a writing habit. I do occasionally write, but I should write more.

Join me as I embark on a new journey to reinvent myself to become a better technology leader.

To infinity and beyond!