Alexandre Guimarães Malucelli

AWS CloudWatch Logs for Chef

Sun, 21 May 2017 00:00:00 -0300

Last year I released an unofficial Chef Cookbook on Chef Supermarket that install and configure AWS CloudWatch Logs Agent.

AWS CloudWatch Logs is great service, it allows you to monitor your log files in centralized way in a nearly real-time through AWS Console, and also allows you to search in logs for specific phrases, values, or patterns.

In order to help the deploy of CloudWatch Logs configurations, I started developing this cookbook when we at fluig Identity, chose AWS CloudWatch Logs as the solution for centralize all the logs that our applications generate (today is about 250GB per month).

Below I’m going to show you how the cookbook works and how you can use it in your Chef.

Usage

First add this cookbook to your base recipe to install the agent in all of your servers (you can use berks to download directly from Chef Supermarket):

cookbook 'aws-cloudwatchlogs', '~> 1.1.6'

You need to configure the following node attributes by an environment or role, so the recipe can configure the agent by your specific configuration. All attributes are described at CloudWatch Logs official documentation, so please take a visit to understand and adapt it to your need.

default_attributes(
   'aws_cwlogs' => {
      'region' => 'your_aws_region',
      'aws_access_key_id' => 'your_aws_access_key',
      'aws_secret_access_key' => 'your_aws_secret_key',
      'log' => {
         'syslog' => {
            'datetime_format' => '%b %d %H:%M:%S',
            'file' => '/var/log/syslog',
            'buffer_duration' => '5000',
            'log_stream_name' => '{instance_id}',
            'initial_position' => 'start_of_file',
            'log_group_name' => '/var/log/syslog'
         }
      }
   }
)

Note: If you don’t specify AWS credentials, the CloudWatch Logs Agent will attempt to use the AWS IAM Role assigned to the instance.

You can also opt to configure by declaring it in another cookbook at a higher precedence level.

default['aws_cwlogs']['region'] = 'your_aws_region'
default['aws_cwlogs']['aws_access_key_id'] = 'your_aws_access_key'
default['aws_cwlogs']['aws_secret_access_key'] = 'your_aws_secret_key'
default['aws_cwlogs']['log']['syslog'] = {
   'datetime_format' => '%b %d %H:%M:%S',
   'file' => '/var/log/syslog',
   'buffer_duration' => '5000',
   'log_stream_name' => '{instance_id}',
   'initial_position' => 'start_of_file',
   'log_group_name' => '/var/log/syslog'
}

Once you defined those attributes, you will need to reference aws_cwlogs resource in one of your recipes, by doing:

include_recipe 'aws-cloudwatchlogs'

aws_cwlogs 'syslog' do
  log node['aws_cwlogs']['log']['syslog']
end

This will create a unique log configuration file that will be stored in etc/config directory of your CloudWatch Logs Agent.

If you have more that one log attribute, each log referred will be created in its own configuration file. This avoid us to create invalid configurations in servers that don’t have the log specified.

For a complete documentation of the cookbook, please take a look at the project on GitHub.

Example

Those attributes used before will generate the CloudWatch Logs configuration below.

[syslog]
datetime_format = %b %d %H:%M:%S
file = /var/log/syslog
buffer_duration = 5000
log_stream_name = {instance_id}
initial_position = start_of_file
log_group_name = /var/log/syslog

Remember: You can configure as many logs as you need with the log attribute.

What’s next?

Currently, this cookbook was develop and tested only in Ubuntu, but, of course, pull requests are always welcome. If you have any questions, please feel free to create a New Issue on the project.

PPTP VPN on macOS Sierra

Tue, 16 May 2017 00:00:00 -0300

Apple stopped the support for it’s builtin PPTP VPN client on macOS Sierra, but they kept their libraries, so it’s still possible to create a PPTP VPN connection over command line without using any 3rd party clients, like FlowVPN or Shimo.

I know that PPTP VPN has become outdated and is less secure than other protocols, but in a corporate world you sometimes don’t have a choice.

The following procedure will show how you can create a PPTP VPN on macOS Sierra. It’s based on my own configuration (MPPE-128), so you might have to adapt it to your need.

The procedure is quite simple, you first need to create a file in /etc/ppp/peers with a name that represent your domain, or company, so in my case I will call it vpn.example.com.

$ sudo touch /etc/ppp/peers/vpn.example.com

This file will contain the configuration that pppd daemon will reference and try to connect.

plugin PPTP.ppp
noauth
# logfile /tmp/ppp.log
remoteaddress "vpn.example.com"
user "username"
password "mYS3cUr3P@ssW0rd!"
redialcount 1
redialtimer 5
idle 1800
# mru 1368
# mtu 1368
receive-all
novj 0:0
ipcp-accept-local
ipcp-accept-remote
# noauth
refuse-eap
refuse-pap
refuse-chap-md5
hide-password
mppe-stateless
mppe-128
# require-mppe-128
looplocal
nodetach
# ms-dns 8.8.8.8
usepeerdns
# ipparam gwvpn
defaultroute
debug

Now you only need start the pppd daemon with the following command and that’s it.

$ sudo pppd call vpn.example.com

Note: This configuration will also change your default route to the VPN.

Testing Amazon SES SMTP with OpenSSL

Sun, 14 May 2017 00:00:00 -0300

Over the last few months, we are using Amazon Simple Email Service (SES) as our default mail service at fluig Identity. AWS SES is just like any other SMTP service, it also requires a username and password for authentication, but as SES is a AWS service, those credentials are based on IAM credentials, so Access Key ID in this case will be our username, and Secret Access Key, using a HMAC-SHA256 algorithm, will be our password.

This tutorial will show you how you can simulate a communication with AWS SES SMTP interface through OpenSSL, where you can troubleshoot IAM problems before setting them up in your application.

For this example, AKIAIOSFODNN7EXAMPLE will be our Access Key ID, wJalrXUtnFEMI/K7MDENG/bPxRfiCYzEXAMPLEKEY our Secret Access Key and noreply@malucelli.net the e-mail address registered and verified in AWS SES.

To create a password with HMAC-SHA256 algorithm, the first thing we need to do is to encode our Secret Access Key. You can use the Python function below to encode a string with HMAC-SHA256.

#!/usr/bin/env python

import base64, hmac, hashlib, sys

print base64.b64encode("{0}{1}".format('\x02', (hmac.new(sys.argv[1].encode('utf-8'), 'SendRawEmail', digestmod=hashlib.sha256)).digest()))

Now you can simply call the function by passing the Secret Access Key as a parameter, that you will get your password encoded in HMAC-SHA256.

$ python encode.py "wJalrXUtnFEMI/K7MDENG/bPxRfiCYzEXAMPLEKEY"
AlRfBgIG4YXDUfEVX5UTIZKnYjvlGay7eQtqp1ifwd7Z

To communicate with a AWS SES SMTP interface, both username and password need to be encoded in base64, as you can see below.

$ echo -n "AKIAIOSFODNN7EXAMPLE" | base64
QUtJQUlPU0ZPRE5ON0VYQU1QTEU=

$ echo -n "AlRfBgIG4YXDUfEVX5UTIZKnYjvlGay7eQtqp1ifwd7Z" | base64
QWxSZkJnSUc0WVhEVWZFVlg1VVRJWktuWWp2bEdheTdlUXRxcDFpZndkN1o=

Also the communication needs to be done using Transport Layer Security (TLS), so we will use openssl rather than telnet.

In the example below, we will open a SMTP connection, authenticate using our IAM credential encoded and send a simple message to myself.

$ openssl s_client -crlf -quiet -starttls smtp -connect email-smtp.us-east-1.amazonaws.com:587

HELO malucelli.net
AUTH LOGIN
QUtJQUlPU0ZPRE5ON0VYQU1QTEU=
QWxSZkJnSUc0WVhEVWZFVlg1VVRJWktuWWp2bEdheTdlUXRxcDFpZndkN1o=
MAIL FROM: <noreply@malucelli.net>
RCPT TO: <alexandre@malucelli.net>
DATA
Hi, this is a example mail.
.
QUIT

This saved me time while we were implementing AWS SES, where I could test IAM credentials before setting them up in our applications. I hope this help you as well.

My Personal Dotfiles

Tue, 20 Oct 2015 00:00:00 -0200

The name “dotfiles” are derived from Unix-like configuration files that start with a dot (like .bash_profile, .vimrc, etc), and are common to serve as the files that you can use to customize your system and your apps.

Today already exists a large community around this, with a big number of repositories ready to go.

I have some of my friends that already did and shared their own dotfiles, so I decided to share mine as well!

In the beginning, I did a few research and I tried some dotfiles, but none pleased me as the @mathiasbynens/dotfiles, which ended as the base of my dotfiles.

I created a repository on GitHub, with all necessary steps and files that are included in this configuration. It covers Terminal, Git, Bash, Sublime Text and Vim configurations and preferences. I hope you like it!

Fell free to use and customize. If you wanna contribute with anything, just let me know.

The final result looks like this:

Terminal

Vim

Missing xcrun on OS X El Capitan

Thu, 01 Oct 2015 00:00:00 -0300

Today I updated my MacBook to the latest version of OS X, El Capitan.

The upgrade ran successfully and everything seemed great, except when I open the Terminal and realized that some of the commands that I use daily, like git or brew, started giving me the error below:

xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun

I found after a few googling that this error is origined by the lack of Xcode Command Line Tools.

The wired part is that I had the Command Line Tools installed in Yosemit. I guess that El Capitan remove it by default.

I only had to reinstall Xcode Command Line Tools with the command below, to fix the problem.

$ xcode-select --install

Ciao Mondo

Mon, 18 May 2015 00:00:00 -0300

Hey! If you don’t know me, first take a look at my profile!

This blog begins with the urge to write again, to share some of my knowledge, learnings, thoughts, projects and moments of my life that worth a post.

I had a personal blog years ago, but at the time, it was not my priority and I was not committed to the project, so I always had an excuse to abandon the blog.

The Challenge

Write is a challenge to me, it demands time and a lot of concentration, two things that will be difficult to me at beginning. Actually, I’m very excited with this project, because it will help me to take a time to put my thoughts together.

As you can see, I’m also writing in english. This time I chose only english because of the technical content that I’ll produce.

I tend to write about:

Articles about DevOps - tools, cookbooks, culture
Projects I’m working on
Things that don’t fill in 140 characters

Blogging Platform

I gave a chance to Jekyll and hosted the blog at GitHub Pages. I already knew some blogs that were built with this setup, and the performance of those sites was something that shined my eyes.

I’ll not blog about “how to setup jekyll with github pages” because there are a thousands over the internet.

In short: I’m enjoying so far, is pure static content with markdown language, you should try it!