Digital Sanctum

How to Install OpenStack Heat

2013-05-01 00:00:00 +0000

If you'd like to get up and running and hacking on OpenStack Heat this guide is for you. In this post I'll describe how to get up and running quickly with the goal of creating a fully functional environment that's easy to hack on (not for production use).

What is OpenStack?

OpenStack is open source software for building public and private clouds.

The OpenStack Open Source Cloud Mission: to produce the ubiquitous Open Source Cloud Computing platform that will meet the needs of public and private clouds regardless of size, by being simple to implement and massively scalable.

What is Heat?

Heat is a service to orchestrate multiple composite cloud applications using the AWS CloudFormation template format, through both an OpenStack-native ReST API and a CloudFormation-compatible Query API.

Install devstack

As a convenience, we'll use devstack which is basically a shell script maintained by the OpenStack developer community to build a complete OpenStack environment.

First, if you haven't done so already create a fresh install of a recent Ubuntu or Fedora linux distro on a machine with a decent amount of resources. I recommend at least 8GB of RAM and 250GB of disk space. I've tested with Ubuntu 12.04 and Fedora 18 so ymmv.

Second, ssh to the machine and install devstack by running the following commands:

sudo apt-get update
sudo apt-get install git
git clone git://github.com/openstack-dev/devstack.git (I simply clone this to ~/devstack)
cd devstack
./stack.sh
you'll be prompted several times for a password. I recommend just making all the passwords the same for the sake of simplicity.

After several minutes, the script will eventually stop and you'll be presented with output similar to the following:

Horizon is now available at http://172.16.8.33/
Keystone is serving at http://172.16.8.33:5000/v2.0/
Examples on using novaclient command line is in exercise.sh
The default users are: admin and demo
The password: password
This is your host ip: 172.16.8.33
stack.sh completed in 514 seconds.

Enabling Heat

Heat is included in devstack but it must be enabled by adding the following to ~/devstack/localrc:

ENABLED_SERVICES+=,heat,h-api,h-api-cfn,h-api-cw,h-eng
IMAGE_URLS+=",http://fedorapeople.org/groups/heat/prebuilt-jeos-images/F16-x86_64-cfntools.qcow2,http://fedorapeople.org/groups/heat/prebuilt-jeos-images/F16-i386-cfntools.qcow2"

Now, restart OpenStack with ./unstack.sh, then ./stack.sh

After several minutes (depending on your internet connection speed and the number of images specified above), you should have a working OpenStack/Heat installation.

How to Create ISO Storage on XenServer

2013-04-30 00:00:00 +0000

This post walks you through how to create local volume on a XenServer host to serve as ISO repo.

1. ssh to the XenServer host

2. Check free local disk space:

df -h

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1              4127440   1976852   1940924  51% /
none                    381120         4    381116   1% /dev/shm
/opt/xensource/packages/iso/XenCenter.iso
                         52408     52408         0 100% /var/xen/xc-install

3. Check available physical disk space:

[root@Host1 ~]# pvs
  PV         VG                                                 Fmt  Attr PSize   PFree  
  /dev/sda3  VG_XenStorage-43851e54-d3bf-0602-5e84-93629cfe7851 lvm2 a-   457.75G 457.75G

4. Check volume groups:

[root@Host1 ~]# vgs
  VG                                                 #PV #LV #SN Attr   VSize   VFree  
  VG_XenStorage-43851e54-d3bf-0602-5e84-93629cfe7851   1   1   0 wz--n- 457.75G 457.75G

5. Create the new LV (Logical Volume) in the available VG:

lvcreate -L 10G -n iso-images VG_XenStorage-43851e54-d3bf-0602-5e84-93629cfe7851

6. list:

lvscan

7. make a filesystem on this new LV:

mkfs.ext3 /dev/VG_XenStorage-43851e54-d3bf-0602-5e84-93629cfe7851/iso-images

8. create a mount point:

mkdir /mnt/iso-images

9. make the LV in the VG known to the kernel:

vgchange -a y

10. create the repository:

xe sr-create name-label=iso-images type=iso device-config:location=/mnt/iso-images/ device-config:legacy_mode=true content-type=iso

11. confirm new repository listed in XenServer:

xe sr-list

12. mount the new FS:

mount /dev/VG_XenStorage-43851e54-d3bf-0602-5e84-93629cfe7851/iso-images /mnt/iso-images/

13. [OPTIONAL] edit /etc/fstab to mount at boot

After following these steps, you can now use something like wget to download ISO images to the /mnt/iso-images mount and have them available to XenServer.

OpenStack Summit 2013

2013-04-23 00:00:00 +0000

Last week I attended OpenStack Summit along with almost 3,000 other geeks in my favorite city, Portland, Oregon. I've been to a few conferences over the last several years but this one was especially good. Not because of the host city, the venue or the parties (although they were good too) but because of the OpenStack community.

The NSA and OpenStack

Where else can you hear about how the NSA leverages OpenStack for their internal cloud to run "mission" apps:

Design Sessions

There were also many sessions in which the audience was integral in helping to shape the future of OpenStack by participating in design sessions. The design sessions were great for both newcomers wanting to know more about how to hack on OpenStack as well as veterans who were able to present problems and ask and answer questions. While these sessions occurred, etherpad was actively used to present issues and design questions as well as document decisions and solutions. An example is this one which supported the "How to run multiple heat engines and scaling" session. This was nice because your time wasn't monopolized by trying to take notes while trying to follow and participate in the lively discussions.

State of the Stack

One of my favorite sessions, "State of the Stack", was presented by Randy Bias, CTO of Cloudscaling in which he gave a great overview of the history and momentum of OpenStack followed by some opinionated thoughts on each major component of the OpenStack architecture. Randy also had some great slides on who the players are in the OpenStack ecosystem and what their motivations are. In closing, he claims that OpenStack is the clear winner among competitors such as Cloudstack, Eucalyptus, and vCloud. Perhaps the best illustration of this session is this slide:

Curvature and Donabe

Perhaps the shiniest, "ooh and ahh" session of the conference was the demo of a project by a group of University of Kent undergrad students slash Cisco engineers. Curvature, which is "an interactive visual orchestration tool for applications on OpenStack", was demoed first and illustrated a deviation from the tab and grid ui elements of Horizon, to a fluidy, floating representation of an application topology.

"Users draw their desired application topology on a canvas using a toolset of Quantum L2/L3 components and virtual machine images. This topology can then be deployed onto a running OpenStack environment at the click of a button – with Curvature handling all of the orchestration necessary for provisioning the workload, i.e. the Quantum networks and Nova VMs."

Donabe is a "recursive container service" which is loosely coupled with Curvature and can be run indendently. Although Donabe seems like a nice companion to Curvature, it seems to have a lot of overlap with the official OpenStack orchestration project, Heat.

While the source for the project is to be open sourced in the near future, they mention the backend was built using Ruby on Rails and the front end used SVG. It should be interesting to see what happens after it's released into the wild and perhaps integrated with Heat.

Common Themes

Perhaps one the the best outcomes of a conference like this is that it allows you to identify common themes across all of the OpenStack components. Naturally, one of these themes in a cloud architure is "how to scale". I saw this theme repeated in at least three different areas: Heat, Nova and Ceilometer.

In Heat design discussions, there was talk about how best to horizontally scale the "heat-engine" component as well as discussion about concurrent resource scheduling.

In Nova/Cinder/Networking, or perhaps in OpenStack as a whole, there seemed to be a need for an overall "task/workflow system". While different projects have taken different approaches based on needs at the time, there's increased potential to leverage a common service/library to be used by all. Several sessions related to "orchestration" underscores the momentum.

The newest project, Ceilometer, has the extraordinary challenge of orchestrating data measurements and aggregation in OpenStack and easily sharing that data to suit a variety of use cases including customer billing. Among the Ceilometer sessions, which spanned all four days of the summit, in "Feedback from Ceilometer users" sought to learn and improve from users' pains and successes.

Summary

To me, the summit is a testament to how well the OpenStack community...is a community...and works well together to solve common problems and create new features at a break-neck three month development cycle. Compare this with the six months (or longer) cycle of older, but not necessarily more mature competitors such as Cloudstack (which is currently a month past their last deadline). As a relative newcomer to OpenStack, I'm thrilled to have had the opportunity to attend the summit and look forward to being a part of the community and contributing to it's success.

How to Copy File Contents From a Remote Machine to Local Clipboard

2013-03-27 00:00:00 +0000

I use ssh to gain access to several remote servers and have always wondered how to copy the contents from remote files to my local clipboard. I use a mac as my desktop so ymmv:

Assuming you're logged into a remote machine,

cat /path/to/remote/file | ssh YOUR_DESKTOP_IP pbcopy

Note that this assumes that your desktop is reachable from the remote machine and that your desktop allows remote access.

How to Setup a PXE Server on Ubuntu

2013-03-22 00:00:00 +0000

This post shows you how to setup a PXE (Preboot execution environment) server on Ubuntu. It assumes a fresh install of Ubuntu 12.04 and at least two machines. One acting as the PXE server and another as the client with a PXE-supported firmware. The following will be installed on the server:

tftp-hpa - (trivial file transfer protocol) is used in conjunction with dhcp for the automated transfer of bootstrap/configuration files.
dhcp - used to locate the appropriate boot server
nginx - serves the kickstart file and install files for one or more client(s)

tftp-hpa setup

install:

sudo apt-get update
sudo apt-get -y install tftpd-hpa
sudo /etc/init.d/openbsd-inetd stop
sudo update-rc.d -f openbsd-inetd remove
sudo sed -i s/no/yes/ /etc/default/tftpd-hpa
sudo /etc/init.d/tftpd-hpa start

add the following to /etc/default/tftpd-hpa:

RUN_DAEMON="yes"
OPTIONS="-l -s /var/lib/tftpboot"

now restart tftpd-hpa

sudo /etc/init.d/tftpd-hpa restart

dhcp setup

install dhcp server:

sudo apt-get -y install dhcp3-server

edit /etc/dhcp/dhcpd.conf and add the following:

subnet 192.168.1.0 netmask 255.255.255.0 {
  range 192.168.1.221 192.168.1.241;
  filename "pxelinux.0";
}

For the purposes of this tutorial, the following are used:

IP address of the PXE server: 192.168.1.6
IP range that the DHCP server will use to assign to client(s): 192.168.1.221 - 192.168.1.241

start the dhcp server:

sudo service isc-dhcp-server start

inetd setup

install:

sudo apt-get install inetutils-inetd

edit /etc/inetd.conf and add the following:

tftp    dgram   udp    wait    root    /usr/sbin/in.tftpd /usr/sbin/in.tftpd -s /var/lib/tftpboot

You may need to change udp to udp4 above if you're using IPv6.

copy install files

Now it's time to copy the files from the Ubuntu install disk. In my case, I'm using a thumb drive so I install the usbmount package in order to have the thumb drive automatically mounted with I plug it in:

sudo apt-get install usbmount

At this point, I plug in the thumb drive and it should be available at /media/usb.

sudo cp -r /media/usb/install/netboot/* /var/lib/tftpboot/

nginx

install:

sudo apt-get install nginx

I had an issue with installs until I commented the following line in the nginx configuration:

location / {
    # First attempt to serve request as file, then
    # as directory, then fall back to index.html
    # comment this --> try_files $uri $uri/ /index.html;
    # Uncomment to enable naxsi on this location
    # include /etc/nginx/naxsi.rules
}

By commenting the try_files $uri $url/ /index.html; line above, nginx will return a 404 instead of falling back to /index.html.

The default document root will be: /usr/share/nginx/www.

Create a directory for ubuntu and copy the install files from the thumb drive:

mkdir -p /usr/share/nginx/www/ubuntu
cp -r /media/usb/* /usr/share/nginx/www/ubuntu/

customizing via kickstart

create /usr/share/nginx/www/ks.cfg with the following contents:

install
url --url http://192.168.1.6/ubuntu/

The above is a minimal kickstart configuration and you can augment it to fully automate installs.

create/edit /var/lib/tftpboot/pxelinux.cfg/default and add the following:

default linux
prompt 0
timeout 60

label linux
    menu default
    menu label Linux
    kernel ubuntu-installer/amd64/linux
    append ks=http://192.168.1.6/ks.cfg vga=normal initrd=ubuntu-installer/amd64/initrd.gz ramdisk_size=16432 root=/dev/rd/0 rw  --

You should now be able to start a client machine up, hit something like F12 to boot via PXE and be on your way!

References

How to Setup VNC Between a Mac and Ubuntu

2013-03-13 00:00:00 +0000

This tutorial instructs you how to quickly VNC into a headless Ubuntu server. It's assumed that you have a server running somewhere and for whatever reason you need access to a graphical environment or browser instead of just SSH. Of course, all of the normal security precautions apply here. Tested on Ubuntu 12.04.1 server (64-bit).

On the Server

SSH into the server and run the following:

$ sudo apt-get update && sudo apt-get install vnc4server

Now start up vncserver temporarily so it generates files it needs to run (this will only happen on the initial run)

$ vncserver

You will get prompted for a password that will be used to connect via VNC. Note that anything that's over 8 characters will be truncated so I suggest using something shorter (between 6 and 8 characters).

When the setup is complete, shutdown the vncserver so we can make some changes:

$ vncserver -kill :1

Now you're ready to perform one of the following steps depending on your needs:

1. Minimal Gnome desktop:

$ sudo apt-get install --no-install-recommends gnome-core gnome-session-fallback

Edit the generated ~/.vnc/xstartup:

#!/bin/sh

# Uncomment the following two lines for normal desktop:
unset SESSION_MANAGER
# exec /etc/X11/xinit/xinitrc
gnome-session –session=gnome-classic &


[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
xsetroot -solid grey
vncconfig -iconic &
#x-terminal-emulator -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" &
#x-window-manager &

2. Minimal Ubuntu desktop:

$ sudo apt-get install --no-install-recommends ubuntu-desktop

Edit the generated ~/.vnc/xstartup:

#!/bin/sh

# Uncomment the following two lines for normal desktop:
unset SESSION_MANAGER
. /etc/X11/xinit/xinitrc

[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
xsetroot -solid grey
vncconfig -iconic &
#x-terminal-emulator -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" &
#x-window-manager &

3. Firefox only (the most light weight solution if you just need access to a browser):

$ sudo apt-get install firefox

Edit the generated ~/.vnc/xstartup:

#!/bin/sh

# Uncomment the following two lines for normal desktop:
unset SESSION_MANAGER
. /etc/X11/xinit/xinitrc

[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
xsetroot -solid grey
vncconfig -iconic &
#x-terminal-emulator -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" &
#x-window-manager &

Once you've installed one

On the Mac

Open the "Screen Sharing" app via command line:

$ open vnc://your_ip:your_vnc_display_number

For example:

$ open vnc://192.168.0.99:5901

You can also accomplish the above by opening Finder > Go > Connect to server...

References

http://www.ubuntugeek.com/how-to-install-gui-on-ubuntu-12-04-precise-server.html

How to allow sudo without a password

2013-01-03 00:00:00 +0000

Suppose you're setting up an automated task or something similar and it would be nice not to worry about entering a password for sudo.

Here's how to allow sudo without a password for a specific user (tested on Ubuntu 11.10):

sudo visudo
add the following line at the end of the file: someuser ALL=NOPASSWD: ALL and exit
now test with something like sudo ls and you should not get prompted for a password

For completeness, here's what the sudoers file should look like (Ubuntu 11.10) when your done with the edit above:

# This file MUST be edited with the 'visudo' command as root.
#
# Please consider adding local content in /etc/sudoers.d/ instead of
# directly modifying this file.
#
# See the man page for details on how to write a sudoers file.
#

Defaults

    env_reset

# Host alias specification

# User alias specification

# Cmnd alias specification

# User privilege specification
root    ALL=(ALL:ALL) ALL

# Members of the admin group may gain root privileges
%admin ALL=(ALL) ALL

# Allow members of group sudo to execute any command
%sudo   ALL=(ALL:ALL) ALL

shane ALL=NOPASSWD: ALL

#includedir /etc/sudoers.d

Killing Multiple Instances of a Process on Mac OS X

2013-01-01 00:00:00 +0000

In this post, I'll outline a couple ways to kill multiple running instances of a process on OS X.

A handy program to have is pidof, which is a utility that returns the process identifier (PID) of a running process or processes.

Since it's not available by default on Mac, you can use homebrew: brew install pidof

Now, you can easily find the PIDs for a process, for example to see the PIDs associated with all running instances of nginx, run pidof nginx.

To kill all running instances of nginx use kill $(pidof nginx)

Sometimes using pidof won't work because the processes don't have an accurate name associated with them. An example is if you're running a program with a runtime such as java.

In this case you can first find the processes with something like ps aux |grep tomcat

You can exclude the grep process and pipe them to awk and kill them:

kill `ps aux | grep tomcat | grep -v grep | awk '{print $2}'`

kill `ps aux | grep '[t]omcat' | awk '{print $2}'`

How to Install Java 7 on Mountain Lion

2012-12-20 00:00:00 +0000

To start, download and install the latest release of Java 7 from Oracle.com here.

After the install is complete you should have it here: /Library/Java/JavaVirtualMachines/jdk1.7.0_10.jdk

Since Apple has done away with the Java Preferences application which allowed you to easily switch between versions of Java, this is now a manual process.

Assuming you're using the default Java version you first move it out of the way:

sudo mv /System/Library/Java/JavaVirtualMachines/1.6.0.jdk /System/Library/Java/JavaVirtualMachines/1.6.0.jdk-orig

Now create a symlink from the old 1.6 jdk path to the new 1.7 path:

sudo ln -s /Library/Java/JavaVirtualMachines/jdk1.7.0_10.jdk /System/Library/Java/JavaVirtualMachines/1.6.0.jdk

and verify with java -version:

java version "1.7.0_10"
Java(TM) SE Runtime Environment (build 1.7.0_10-b18)
Java HotSpot(TM) 64-Bit Server VM (build 23.6-b04, mixed mode)

Top 10 Java Companies and Projects to Follow on GitHub

2012-12-02 00:00:00 +0000

Java is the fourth most popular language on Github so naturally here's my hand-picked top 10 companies and projects to follow: