spf13.com - IT, Web 2.0 and Social Media ramblings of Steve Francia

Transitions

Steve Francia — Mon, 08 Mar 2010 20:39:40 +0000

I couldn’t be more excited to announce that I have accepted a position at The Open Sky Project. I am leading the architecture, development and technology.

It’s rare in life that one has the opportunity to do what they love to do and be paid to do it. Even rarer is to do something great with people you love working with. At OpenSky I have found this and more. There is an energy and excitement at OpenSky; come spend 10 minutes in our office and you will feel it. The team is passionate about what they are doing and the passion is growing. OpenSky has assembled the strongest team I have ever seen at an early stage startup. Each member of the management team has excelled in their particular domain and exhibits youthful passion for doing it. This level of deep experience allows us to focus on building the business, which is hard enough without the distraction of trying to learn how to do your job as well.

The more time I spend at OpenSky, the more impressed I am. There isn’t a weak link in the chain, and when you combine this with a great idea, good things are bound to happen. I couldn’t be more excited to be a part of this team working together to build the next great e-commerce revolution.

We are at a really good place right now. All startups begin with an idea, turn it into an experiment, and if that goes well, build a business around it. We are at the exciting stage where the experiment was not only successful but exceeded all possible expectations. Now we get to go full speed ahead building a great business and all the technology and systems to power it.

We have the challenge and opportunity to build a modern scalable e-commerce system using the best open source software, libraries and technologies.

I am looking for the best and brightest to come and join our great team of all star engineers. If you’re interested give me a shout out on twitter @spf13 or check out our jobs page.

If you want to find out more about OpenSky, come check out our blog!

VIM Crash Course

Steve Francia — Wed, 03 Mar 2010 17:33:54 +0000

Image via Wikipedia

Most developers know the basics of VIM, enough to edit a conf file, but most stay there, unaware of the power and beauty of vim. One of my developers has expressed desire to abandon the bloated GUI ways of eclipse and discover VIM. I have been using VIM for such a long time I forgot how difficult that transition is. Here are some resources and approaches to help you learn vim. The easiest place to start is by using a tutor. Did you know that Vim comes with it’s own tutor? Simply type vimtutor in your shell (outside of vim). If you prefer the gui version, type gvimtutor. On MS-Windows you can find it in the Program/Vim menu.

The tutor is a 30 minute interactive tutorial that will walk you through the basics of editing with vim.

Once you have completed the tutor it’s critical that you understand how to move around (the tutor covers these). Practice Practice Practice.

h j k l  ...  e b w f t * / ? % $ ^

These are your bread and butter and most everything that follows builds on these.

Vim also comes with a very comprehensive user manual. Launch vim and type “:help user-manual” (without quotes) in command mode

Most important is that the first couple days trying to use vim instead of a gui based editor will be frustrating and hard. Stick with it. Within a week or two you things with flip on you and you will begin to be frustrated that every program doesn’t work like vim, and be shocked how often you go to the mouse or arrow keys.

Unix Jobs Management

Steve Francia — Tue, 02 Feb 2010 19:58:57 +0000

Image via Wikipedia

Every self respecting linux, mac os X or *nix user should have a solid handle on managing jobs in unix. The following will explain how to run tasks in the background, bring tasks to the foreground, background already running tasks and keeping a task running while logged out.

Run a task in the background

All you need to to is follow a command with the ‘&’ character. Pretty simple. What this does is start the command and background it. It will keep running and when it is finished it will present the results to the foreground.

$ cp -av stuff /mnt/backup &amp;

[1] 21394

The response indicates that the job is now backgrounded and the job id is 1 and the process id is 21394.

A backgrounded process is still attached to your session. This means that if you logout it will also stop, though it will warn you that a process is running in the background when you ‘exit’.

Listing jobs (processes running in the background)

In this example I ran ‘sleep 100 &’. It is also in the background. To see backgrounded jobs, simply type jobs.

$ jobs

[1]-  Running                 cp -av stuff /mnt/backup &amp;
[2]+  Stopped                 sleep 100 &amp;

Bring a backgrounded app to the foreground

I want to see how my backup is going so I’ll bring it back to the foreground to view it. To do this you need to know the job id which is the number in the [] above.

There are couple ways to do this… The first works everywhere, where the latter is specific to bash (though it may also exist in your preferred shell).

$ %1

$ fg 1

Suspending a task.

Satisfied that the backup is going well, you now want to backgroud it again. type -z to suspend it. This will temporarily stop the process from running and bring you to a shell prompt again. From there just ‘bg‘ it.

hit <ctrl>-z

[1]+  Stopped               cp -av stuff /mnt/backup &amp;
$

$ bg 1
[1]+ cp -av stuff /mnt/backup &amp;

Running a process and logging out

There are three different approaches to doing this. Screen, nohup and disown. Screen seems like the most versile, though you technically aren’t logging out, but you can disconnect completely without worrying that it will stop. Both nohup and screen require you to do something prior to or during starting the process so neither can work if the process is already going. For these circumstances their is disown.

Screen

Screen is a versitile tool and a must use for any system admin. I could easily devote a full blog post on screen, in fact I did. Be more productive using gnu screen

nohup

nohup is a POSIX command to ignore the HUP (hangup) signal, enabling the command to keep running after the user who issues the command has logged out. The HUP (hangup) signal is by convention the way a terminal warns depending processes of logout.

nohup is most often used to run commands in the background as daemons. Output that would normally go to the terminal goes to a file called nohup.out if it has not already been redirected. This command is very helpful when there is a need to run numerous batch jobs which are inter-dependent.

If you redirect the streams you can avoid filling up your filesystem with nohup.out files as follows…

$ nohup tar czf /backup/home.tgz . &gt; /dev/null 2&gt;&amp;1 &amp;

Some shells (bash) provide a similar function called disown which can be run after the command is already running. If you are using bash, there really isn’t a reason to use nohup, use disown instead.

disown

Disown is easy enough to use, background a job (bg 1 for example) then type disown.

    $ bg 1
    $ disown

Alternatives

Use ‘setsid’ which will run a program in a new session.

It is also possible to use “dislocate” for this.

Under Debian, it is possible to use /sbin/start-stop-daemon to daemonise a process.

Another way to avoid the process being bound to a terminal is to have the at daemon run it, as for example with echo command | at now.

Warning

Once a job is disowned (or run using nohup) you can no longer bring it to the foreground and you no longer own it. It will not be listed in jobs. Only do this if you won’t need to access it again. Your only option to stop it is to kill it.

Notes

bg defaults to the last suspended job, so it isn’t necessary to provide the job id.

human readable du sorted by size

Steve Francia — Sat, 23 Jan 2010 15:34:46 +0000

Image via Wikipedia

du is the *nix command for disk usage. It tells you how much space everything in the given directory is taking up. GNU du introduced a handy option -h making it human readable, or showing sizes using K, M, G rather than bytes. Unfortunately this makes it not sortable numerically. Here’s how to sort du by size and keep it as human readable. Insert the following function into your .profile or .bash_profile file.

 function duf {                                                                                                                                                                
     du -k "$@" | sort -n | while read size fname; do for unit in k M G T P E Z Y; do if [ $size -lt 1024 ]; then echo -e "${size}${unit}\t${fname}"; break; fi; size=$((size/1024)); done; done
 }

By writing this as a function, it enables you to pass along parameters to the newly created duf command just as you would du.

For convenience, I also create the following aliases which you can also place in your .profile or .bash_profile file.

alias du1='duf --max-depth=1'
alias du2='duf --max-depth=2'
alias du0='duf --max-depth=0'

Once you have added these lines, remember to source .profile to use it without logging out.

Benchmarking Cloudfront (and S3)

Steve Francia — Tue, 19 Jan 2010 20:20:53 +0000

Image via CrunchBase

Amazon has done it again bringing another computing service to the masses. This time it’s the Content Delivery Network or CDN. Cloudfront is a direct competitor to other popular CDNs such as Akamai. While Akamai requires a fairly substantial amount of traffic to become a customer, Cloudfront doesn’t. It follows all of Amazons, pay for what you use mentality. This means that everyone can benefit from incorporating Cloudfront into their blog, site, store, etc..

For purposes of experimentation, I decided to track how much of a difference Amazons Cloudfront made over simple s3 public hosting vs hosting locally on my webhost. I decided to use stevefrancia.com for this experiment since it is a very small and simple site with a single html file a couple css and js files and a handful of images. No server side processing or other significant variables, perfect for benchmarking. The next series of images shows the improvement as I moved towards all files (except the primary html file) on cloudfront.

Note, the scale of the following images isn’t the same from one to the other, please note the x-axis scale for each one

Base Case

Google Hosted jQuery

Resources on S3

Resources on Cloudfront

Summary

Conclusion

Cloudfront is a significant improvement over both my web host as well as serving files directly from s3. Each case the single HTML file was served from my web host. When you remove that from the totals (assuming an average of .5 s for the index.html file) it shows an even larger improvement. From just under 4 seconds in our base case, to around .4 seconds on cloudfront. Using cloudfront is easy and inexpensive. CloudFront integration is a no brainer. It works well, easily integrates and is inexpensive. Most people should use it to host all of their resources.

Of course this is a very small scale test and your mileage will vary, pingdom is a good resource for simple tests like this as it produces the graphs seen above and is easy to use.

Using the right keys

Steve Francia — Tue, 19 Jan 2010 16:27:16 +0000

Image by HeyThereSpaceman. via Flickr

Today I was visiting a friends office and like many offices in NYC they have a shared bathroom in the hall for the entire floor. In this building it had five buttons on the door that when pressed in the correct order unlocked the door. A simple password. In our office we have a similarly shared bathroom, but instead of a password, we have a physical key required to unlock the door.

A password for the bathroom was just the right amount of security. It prevented just any stranger from easily accessing the bathroom, but wasn’t inconvenient and easy to provide access to. My friend simply told me the password and I as a visitor had all I needed to enter. It was the right amount of security for the right purpose.

Our physical key certainly provides a higher level of security, but with additional inconvenience. Physical keys hang next to our door which is easy enough, but what happens when someone took the key into the bathroom, or home with them. The solution is for everyone to make their own copy which is excessive and unnecessary for a bathroom.

If our expensive merchandise was stored in a room, would we use a password to guard it. Never, wouldn’t think twice about it, we would have multiple security measures in place (and do at Portero Luxury). Keys, codes and surveillance.

In our increasingly electronic world how many times are you using passwords for sensitive data when a key (or key + alarm + password + …) is appropriate. Or even worse, how many places share the same password. Is your email and bank password the same? If someone had access to your email address & password from a single site how much could they access? what damage could they do?

For those that claim they choose insecure over inconvenient I say to you why choose. You can have both secure and convenient. Excellent tools exist to fix this problem, whether it is server (SSH keys) or browser 1 password (mac) or KeePass (win).

SOAP vs. REST

Steve Francia — Fri, 15 Jan 2010 04:46:50 +0000

Image via Wikipedia

Someone asked me a question today “Why would anyone choose SOAP (Simple Object Access Protocol) instead of REST (Representational State Transfer)?” My response: “The general rule of thumb I’ve always heard is ‘Unless you have a definitive reason to use SOAP use REST’”. He asked “what’s one reason?” I thought about it for a minute and honestly answered that I haven’t ever come across a reason. My background is building great internet companies.

While he seemed satisfied, I wasn’t very happy with that answer, I did some homework and here’s my summary on the difference between SOAP and REST and why anyone would choose SOAP. Clearly both have value, as always the challenge is to know when to use each one (spoiler: luckily the answer is almost always REST).

I’m clearly boiling down a somewhat so please don’t flame me for simplifying things, but feel free to provide any corrections you feel are necessary.

Definitions

REST

RESTs sweet spot is when you are exposing a public API over the internet to handle CRUD operations on data. REST is focused on accessing named resources through a single consistent interface.

SOAP

SOAP brings it’s own protocol and focuses on exposing pieces of application logic (not data) as services. SOAP exposes operations. SOAP is focused on accessing named operations, each implement some business logic through different interfaces.

Though SOAP is commonly referred to as “web services” this is a misnomer. SOAP has very little if anything to do with the Web. REST provides true “Web services” based on URIs and HTTP.

By way of illustration here are few calls and their appropriate home with commentary.

getUser(User);

This is a rest operation as you are accessing a resource (data).

switchCategory(User, OldCategory, NewCategory)

This is a SOAP operation as you are performing an operation.

Yes, either could be done in either SOAP or REST. The purpose is to illustrate the conceptual difference.

Why REST?

Here are a few reasons why REST is almost always the right answer.

Since REST uses standard HTTP it is much simpler in just about ever way. Creating clients, developing APIs, the documentation is much easier to understand and there aren’t very many things that REST doesn’t do easier/better than SOAP.

REST permits many different data formats where as SOAP only permits XML. While this may seem like it adds complexity to REST because you need to handle multiple formats, in my experience it has actually been quite beneficial. JSON usually is a better fit for data and parses much faster. REST allows better support for browser clients due to it’s support for JSON.

REST has better performance and scalability. REST reads can be cached, SOAP based reads cannot be cached.

It’s a bad argument (by authority), but it’s worth mentioning that Yahoo uses REST for all their services including Flickr and del.ici.ous. Amazon and Ebay provide both though Amazon’s internal usage is nearly all REST source. Google used to provide only SOAP for all their services, but in 2006 they deprecated in favor of REST source.

Why SOAP?

Here are a few reasons you may want to use SOAP.

WS-Security

While SOAP supports SSL (just like REST) it also supports WS-Security which adds some enterprise security features. Supports identity through intermediaries, not just point to point (SSL). It also provides a standard implementation of data integrity and data privacy. Calling it “Enterprise” isn’t to say it’s more secure, it simply supports some security tools that typical internet services have no need for, in fact they are really only needed in a few “enterprise” scenarios.

WS-AtomicTransaction

Need ACID Transactions over a service, you’re going to need SOAP. While REST supports transactions, it isn’t as comprehensive and isn’t ACID compliant. Fortunately ACID transactions almost never make sense over the internet. REST is limited by HTTP itself which can’t provide two-phase commit across distributed transactional resources, but SOAP can. Internet apps generally don’t need this level of transactional reliability, enterprise apps sometimes do.

WS-ReliableMessaging

Rest doesn’t have a standard messaging system and expects clients to deal with communication failures by retrying. SOAP has successful/retry logic built in and provides end-to-end reliability even through SOAP intermediaries.

Summary

In Summary, SOAP is clearly useful, and important. For instance, if I was writing an iPhone application to interface with my bank I would definitely need to use SOAP. All three features above are required for banking transactions. For example, if I was transferring money from one account to the other, I would need to be certain that it completed. Retrying it could be catastrophic if it succeed the first time, but the response failed.

External Resources

My Online Business Card (vCard)

Steve Francia — Tue, 12 Jan 2010 21:27:14 +0000

I wanted to polish up my javascript coding so I decided that the best way to do that was with a project that I’ve been wanted to do for a while anyway, my own identity site, or my online business card, or my online vcard. The idea was inspired by Tim Van Damme’s website.

To accomplish this I wrote a jquery plugin to handle the navigation, animation etc.. The site itself is rather simple, a single html page and a few images. The markup is written in such a way that it works perfectly (minus the hiding and animation) even when javascript isn’t present.

I’ll write a separate post on how I wrote the jquery plugin and would be happy if anyone else could use it. I just ask that if used that people provide proper attribution both in the code and with a link back to stevefrancia.com.

Please enjoy my new site and feel free to provide suggestions in the comments below or via the contact form.

Vim Plugins: snipMate

Steve Francia — Tue, 29 Dec 2009 05:42:23 +0000

Image via Wikipedia

Today I discovered snipMate, and I feel like I have finally found the holy grail. SnipMate plays well with supertab and autocompletion. It provides pretty much all the beloved snippet features of TextMate, it even has a nearly identical syntax to TextMate. If you’re not using the snipMate plugin for vim yet you need to be.

Why use it

It’s super easy to use
It has tons of snippets
It’s pretty well compatible with TextMate snippets for easy portability
Dynamic variables, for all the times you use the same string multiple times
It’s really easy to define your own snippets
It’s better than anything else out there, trust me I’ve tried them all

Sometimes a video works better to explain things.. check out this video

snipMate.vim Introductory Screencast from Michael Sanders on Vimeo.

Installing

There are a couple of ways to obtain it either download the zip from the vim site: snipMate or grab it from the authors git repo.
Extract the zip file or tarball to ~/.vim (on Unix/Linux) or ~\vimfiles (on Windows).
As good measure, from inside vim, run :helptags ~/.vim/doc (on Unix/Linux) or :helptags ~/vimfiles/doc (on Windows) to rebuild the tags file
Restart Vim

Using snipMate

snippet is a piece of often-typed text that you can insert into document using a trigger word followed by a <tab>. For instance, in a C file using the default installation of snipMate.vim, if you type "for<tab>" in insert mode, it will expand a typical for loop in C:

for (i = 0; i < count; i++) { }

To go to the next item in the loop, simply over to it; if there is repeated code, such as the "i" variable in this example, you can simply start typing once it’s highlighted and all the matches specified in the snippet will be updated. To go in reverse, use <shift-tab>.

Beyond Installation

Grabbing Scrooloose’s snippets repository from github

Martin Grenfell (Scrooloose) maintains a nice repository of snippets. Though it is a bit ruby heavy, it has dozens of languages including a nice jquery section, and frankly quite a bit more than the stock snippets. I recommend installing his snippets and either fork it or customize it with your own. Here’s how to install the repo (*nix instructions)

cd ~/.vim
mv snippets snippets.orig
git clone git://github.com/scrooloose/snipmate-snippets.git snippets

Git will take care of the installation for you. If you don’t have git, I really recommend it, and I have a guide on how to install it locally. One thing you’ll notice about this repository is that it’s much more structured that the stock snippets. Rather than each language containing a snippet file, each language contains a directory full of snippets with each snippet is in it’s own file. For situations where you want multiple snippets under the same keyword, create a subdirectory and place the snippets inside it. It works quite well, though it seems either approach would be sufficient for personal use.

Creating your own snippets

While there are a bunch of stock snippets, it didn’t take me very long to realize it didn’t have everything I wanted. Fortunately it’s super easy to create your own snippets.

The following is largely lifted from the help file. I strongly recommend reading it (:help snipMate).

Snippets are by default looked for any ‘snippets’ directory in your ‘runtimepath’. Typically, it is located at ‘~/.vim/snippets/’ on *nix or ‘$HOME\vimfiles\snippets' on Windows. (To change that location or add another one, change the g:snippets_dir variable in your |.vimrc| to your preferred directory.)

Snippets can be defined in two ways. They can be in their own file, named after their trigger in ‘snippets/[filetype]/[trigger].snippet’, or they can be defined together in a ‘snippets/[filetype].snippets’ file.

Syntax

The syntax for snippets in *.snippets files is the following: snippet trigger expanded text more expanded text Note that the first hard tab after the snippet trigger is required, and not expanded in the actual snippet. The syntax for *.snippet files is the same, only without the trigger declaration and starting indentation.

tabstops

By default, the cursor is placed at the end of a snippet. To specify where the cursor is to be placed next, use "${#}", where the # is the number of the tab stop. E.g., to place the cursor first on the id of a <div> tag, and then allow the user to press <tab> to go to the middle of it:

snippet div

${2}

placeholders

Placeholder text can be supplied using "${#:text}", where # is the number of the tab stop. This text then can be copied throughout the snippet using "$#", given # is the same number as used before. So, to make a C for loop:

snippet for for (${2:i}; $2 < ${1:count}; $1++) { ${4} }

This will cause "count" to first be selected and change if the user starts typing. When <tab> is pressed, the "i" in ${2}’s position will be selected; all $2 variables will default to "i" and automatically be updated if the user starts typing. NOTE: "$#" syntax is used only for variables, not for tab stops as in TextMate.

Variables within variables are also possible. For instance:

snippet opt ${2:$1}

Will, as usual, cause "option" to first be selected and update all the $1 variables if the user starts typing. Since one of these variables is inside of ${2}, this text will then be used as a placeholder for the next tab stop, allowing the user to change it if he wishes.

To copy a value throughout a snippet without supplying default text, simply use the "${#:}" construct without the text; e.g.:

snippet foo ${1:}bar$1

7 security practices you need to follow

Steve Francia — Tue, 22 Dec 2009 14:52:29 +0000

Some of this may seem like a broken record, yet every single time you hear about a bank losing millions of customer data, or a company having a security breach they consistently have failed to implement and enforce the most basic security practices. Here are 7 simple security practices that you cannot afford to not follow.

1. Secure pass phrases

Throw away the notion of a password. Pass phrases consisting of multiple words and symbols are considerably more secure and easy to remember. Most people use the same password for everything and it’s almost always a word or a word+#. A good pass phrase can be something like "Mary.had.@.little.Lamb". It’s really easy to remember and nearly impossible to guess or brute force. Using a password management system like One Password for Mac is also a good idea.

2. Educate your users

All too often policies are put into place but lacking training necessary to ensure that policies are understood and followed. Quarterly web casts can go a long way to ensuring that the most critical policies are complied with without adding unnecessary disruptions to the business.

3. Perform regulatory compliance self audits

Hackers are attempting to break into your systems right now. If you aren’t holding self audits than the only ones auditing your systems are the hackers. It’s critical that this practice takes place routinely to ensure that your data and systems are as safe as possible. It only takes one small mistake to make you vulnerable. Routine system checking means there is a chance you will find it before someone else does.

4. Access provided on a needs to have basis

Yes this is inconvenient. Yes you need to do it. Firewalls, ACL, fine grained permissions. Properly setup roles. In every system you do use, in every level. This also means that root/admin permission should always be behind sudo and never logged in.

5. Encrypt all sensitive data

All sensitive data should be encrypted. Hard drives should be encrypted; Tunnels should be encrypted using SSH, VPN or SSL; Wireless networks and even wireless keyboards. Passwords should be stored via a one way encryption like md5.

6. Never take data off site (flash drives, laptops, etc)

It seems every time you hear about a bank or government agency losing millions of critical identity information a portable drive or laptop is involved. While encryption is part of the solution here, it could entirely be avoided if sensitive data is not permitted off site. Data should reside on servers sitting behind properly established ACL and not be available to be copied or transferred onto a laptop or portable drive.

7. Use common sense

Lastly, the most important security principle is to use common sense. If it seems wrong, it probably is. Common sense is your best defense. Use it wisely.