SEP Bits and Bytes

echo

2012-11-01T00:00:00-07:00

Shell (~bash)

Today, an easy one from the list of commands you should know: echo.

Close your eyes for a moment and speculate about what echo does. You’re right!

According to the man page, echo “writes arguments to the standard output”.

> echo Hello, world.
Hello, world.

> echo "Hello, world."
Hello, world.

> echo "Hello, world." "Foo"
Hello, world. Foo

By default, echo adds a newline at the end of its output. For writing things to the screen, this is likely exactly what you want. (Yay!) However, if you’re planning to pipe echo’s output to some other commands input, you might not want the newline.

Enter -n:

> echo mySecretPassw0rd | shasum 
0f7262d74d2d48c30121c7e80a5bda5d22968a32  -

> echo -n mySecretPassw0rd | shasum
de72bcbdefa256e34fac7fb0abe08b43aa54df49  -

That’s it…echo only has the one option. It does one thing…and it does it well. It’s handy in scripts and in the middle of one-liners. It’s less handy by itself.

cut

2012-06-29T00:00:00-07:00

Shell (~bash)

Recently, when proofreading my xargs post, a coworker asked “Cut? What does that do?” So, back into the trenches to investigate one of the super-cool, fundamental building blocks of really great command line concoctions.

% echo "1,red,grumpy\n2,orange,sleepy\n3,yellow,doc"
1,red,grumpy
2,orange,sleepy
3,yellow,doc

% echo "1,red,grumpy\n2,orange,sleepy\n3,yellow,doc" | cut -d, -f2
red
orange
yellow

According to the man page, cut “cuts out selected portions of each line of a file”. As usual, instead of an input file, we can use the output of some other command piped in through stdin. the -d, option above tells cut that the lines to be processed are comma-delimited. The -f2 specifies that we want the second “field” in each line returned.

To conserve some typing, let’s set up a file to experiment on:

% echo "1,red,grumpy,monday\n2,orange,sleepy,tuesday\n3,yellow,doc,wednesday" \
>> test.txt

Now, we can use cut on the file rather than stdin:

% cut -d, -f2 test.txt
red
orange
yellow

We can also do multiple fields in several ways:

% cut -d, -f2,4 test.txt
red,monday
orange,tuesday
yellow,wednesday

% cut -d, -f2-4 test.txt
red,grumpy,monday
orange,sleepy,tuesday
yellow,doc,wednesday

% cut -d, -f2- test.txt
red,grumpy,monday
orange,sleepy,tuesday
yellow,doc,wednesday

% cut -d, -f-3 test.txt
1,red,grumpy
2,orange,sleepy
3,yellow,doc

One important caveat…cut won’t reorder fields:

% cut -d, -f2,4 test.txt
red,monday
orange,tuesday
yellow,wednesday

% cut -d, -f4,2 test.txt
red,monday
orange,tuesday
yellow,wednesday

Reordering fields takes something a little more sophisticated; like awk or a while loop. (More on awk and while later.)

cut will, however, happily work on bytes or characters rather than delimited fields if you have data that’s already columnar. Look at the output of ps, for example:

% ps x
PID   TT  STAT      TIME COMMAND
251   ??  Ss     0:00.95 /sbin/launchd
254   ??  S      0:01.30 /usr/libexec/UserEventAgent -l Aqua
256   ??  S      0:06.10 /usr/sbin/distnoted agent
259   ??  S      0:02.46 /System/Library/Frameworks/ApplicationServices.frame
261   ??  S      0:00.01 /usr/sbin/pboard

cut can condense and pick out just the pieces you want to see:

% ps x | cut -c-6,18-40 
PID      TIME COMMAND
251   0:00.97 /sbin/launchd
254   0:01.30 /usr/libexec/
256   0:06.13 /usr/sbin/dis
259   0:02.47 /System/Libra
261   0:00.01 /usr/sbin/pbo

So, go experiment with cut. It does one pretty straightfoward thing…and it does it really well.

follow your tail

2012-06-26T07:00:00-07:00

Shell (~bash) - OS X, Windows, Linux

Riding the coat-tails of yesterday’s head and tail post… is another tail trick.

Just like Toucan Sam never said:

Follow Your Tail

Not Toucan Sam

Anyways, tail -f (the f is for follow) follows a file as it is updated.

For example, say your application writes to a log file, and you’d like to watch the log as it gets appended to, without having to constantly hit refresh in your editor. Just pop open a terminal (or bash) and tail -f /path/to/log.txt and watch the log entries start rolling in!

Pretty slick if you ask me. I’ll even do this for unit test runs or builds that I redirect their output to a file. This way the test or build runs faster since it doesn’t have to output to the screen, but I can still watch the output in another terminal or tab window.

head to tail

2012-06-25T07:00:00-07:00

Shell (~bash) - OS X, Windows, Linux

Use head or tail to dump the beginning, or end of a file, respectively, to STDOUT.

The default number of lines to show for both head and tail is 10.

For example, say you’ve got a log file and you’d like to see the first five entries, to make sure it’s from the session you’re trying to debug:

 head -5 /path/to/log.txt

 > 2012-06-07 05:45:23 Application Started
 > 2012-06-07 05:45:25 Database Service Started
 > 2012-06-07 05:45:25 Login Service Started
 > 2012-06-07 05:45:26 HTTP Server Started
 > 2012-06-07 05:45:26 Waiting for HTTP Client Connections

Now, to look at the last 50, so we can see everything that led up to the event we care about:

tail -50 /path/to/log.txt

 > -snip-
 > 2012-06-07 05:55:16 log log log... log log
 > 2012-06-07 05:55:19 more log stuff
 > 2012-06-07 05:55:19 KABOOM!

I use head and tail quite a bit to look at the beginnings and ends of files (particularly long XML files) so I can make the right data is being sent at the beginning and end of the file.

clipboard

2012-06-21T07:00:00-07:00

Shell/cmd - OS X, Windows

Place text or program output onto the clipboard from your shell!

For OS X (the pb stands for pasteboard):

echo "hello world" | pbcopy

For Windows:

echo "hello world" | clip

It works the same for both platforms with input redirection as well. The following drops the contents of a file onto the clipboard:

pbcopy < src/LICENSE.txt

You’ll need to snag the Microsoft(tm) clip utility here: http://www.petri.co.il/software/clip.zip.

Another example? Sure! Glad you asked. Want a list of all the files in the current directory that start with the letter ‘A’ and put that on the clipboard?

ls | grep '^A' | pbcopy

This is way easier than having to go pick up the mouse and highlight + copy.

I’m sure there are ways to do this in Linux too, I’d be glad for you to tell me. Have I mentioned that not only are contributions accepted, they’re encouraged? (this is me encouraging you to contribute)

Happy clipboarding and contributing!

last dir

2012-06-20T10:00:00-07:00

Shell (~bash) - OS X, Windows, Linux

Change to your previous directory using the special directory marker -.

> pwd
>> /Users/jcfuller/Downloads
> cd /Users/jcfuller/dev/github
> pwd
>> /Users/jcfuller/dev/github
> cd -
> pwd
>> /Users/jcfuller/Downloads

I use this quite a bit when I need to run a command from a specific directory, but then want to get right back to where I was.

cmd visual history

2012-06-19T09:00:00-07:00

cmd.exe - Windows

Did you know you can use the up/down arrows in cmd (or a Visual Studio Command Prompt to scroll back through your history of commands? Of course you knew that… nothing new there.

Did you know you could press F7 and see a visual version of your history?

If you’re in Windows, and use cmd.exe, this looks really helpful.

I Lied: Redirection

2012-06-18T09:00:00-07:00

Shell - OS X, Windows, Linux

I lied. Twice. Well, probably more than that, but specifically about input and output.

These are really instances of input or output redirection.

We already covered that < is input redirection (I previously called this piping in from a file).

We also covered output redirection:

> - redirect output to a file
>> - redirect output and append to a file
2> - redirect standard error to a file

Sometimes I’d like both STDOUT and STDERR to end up in the same file… check this out!

grep * > outfile 2>&1

The > outfile portion is just like from before. The 2>&1 portion says take STDERR (2) and point it at STDOUT (1).

Redirecting to a black hole:

Redirect to /dev/null to send output into a black hole.

I use this when I’m running a program with lots of output that I don’t care about (outputting to the screen is slow!). Redirect it to /dev/null and it will be much faster.

xargs

2012-06-15T00:00:00-07:00

Shell (~bash)

So, you want to sling command line like a boss, eh? I’ve heard it said and believe it more and more each day that the core things you should learn on the command line are find, xargs, grep, sed, cut, while, and echo. (add wc, sort, and history to taste)

Of these, the one I see ignored most often is xargs. So, let’s check it out:

>xargs
1
2
3
4
^D
1 2 3 4

Technically, xargs “reads space, tab, newline and end-of-file delimited strings from the standard input and executes a command with the strings as arguments”…and the default command is echo.

So, in the above, we invoke xargs, it starts listening to stdin, we provide a list of numbers separated by newlines, ^D signals the end of the input, and xargs turns our numbers into a space-separated list that gets passed to echo. Voila!

We could also separate things with tabs or spaces. We can even mix them:

>xargs
1
2 3
4	5
^D
1 2 3 4 5

In practice, xargs usually takes a list of things generated by some other command (like find) and iterates a command over them. What if we wanted to delete all the files larger than 100MB in the current directory tree?

>find . -size 100M | xargs rm

(If you’re really worried that this will hiccup on filenames with a space in them, you can use find -print0 and xargs -0 instead. Those create/consume null delimited lists instead. In practice, I’ve never had that be a problem.)

What if we wanted to clean up our project a little bit?

>find . -name *.o | xargs rm -rf

…and so on…

CAUTION: It’s really easy to combine find, xargs, and rm in clever ways that will do terrible things to your system. Measure twice, rm once.

windows shell

2012-06-14T00:00:00-07:00

Shell (~bash) - Windows

Every post so far has been a shell trick, and they all say they work on Windows. What am I smoking?

There are several ways to get a decent (though still not first class) shell on Windows. You owe it to yourself to use at least one of them.

git-bash

Install yourself some msysgit, or better yet, github for windows or even git extensions.

Any of these will get you a pretty full-featured bash. The latter two also come with a pretty nice git gui.

cygwin

Cygwin: Get that linux feeling on Windows.

A layer on top of Windows that can act as a linux API.

If you’re a linux geek and you’re stuck on Windows, check this thing out. You can buy me beer later. You’re welcome.

gow

Gnu-On-Windows.

The (not-so) new kid on the block.

A lightweight version of cygwin. Really, just a bunch of the gnu binaries (think ls, find, grep, gzip, etc.) in a simple installer.

Yes, there are others, and oh yeah, powershell, but I’m not going to talk about those. If you’d like to, just let me know.

output pipe

2012-06-13T10:00:00-07:00

Shell (~bash) - OS X, Windows, Linux

As mentioned last time, many command line tools consume and/or produce text.

Even more useful than piping things into a program from a file, is piping things out of a program; either into another program, or into a file.

Take the previous example of gzipping an xml document.

gzip -c < thexmlfile.xml

Looking at the output of that, is going to be pretty unfriendly (actually, gzip tries to protect you from doing this to yourself, because it will likely start doing all sorts of wacky stuff to your terminal). What to do?

Output it to a file:

gzip -c < thexmlfile.xml > thexmlfile.gz

Now you’ve got a gzipped version of that xml file.

The > operator is the key there. That operator takes anything that is output from a program on stdout and puts it into the specified file. If that file exists already, it’ll be overwritten. I use this constantly when doing builds from the command line because I always want to be able to go back and search through the build output for any errors or warnings that happened.

Use >> if you’d like to append instead of overwrite.

What about stderr? Try using 2>.

input-pipe

2012-06-11T00:00:00-07:00

Shell (~bash) - OS X, Windows, Linux

Disclaimer: These pipes are to be used for tobacco only.

Most of the programs I use at the command line consume and/or produce text. If you want to feed a program text that accepts input on stdin, I’ll either pipe it in from the previous program, or pipe it in directly from file.

For example, if I’m gzipping up an xml file:

gzip -c < thexmlfile.xml

This dumps the contents of thexmlfile.xml directly into the gzip program. Alternatively, sometimes I need to feed gzip the output of some other program:

cat thexmlfile.xml | gzip -c

This is functionally equivalent, but the cat part could really be any program that generates output, possibly an xml cleaning tool like xmllint. The | passes any output from the previous command as input to the next command.

touch

2012-06-08T00:00:00-07:00

Shell (~bash) - Windows/OS X/Linux

Occasionally (okay, at least once a day) I find myself needing to create an empty file. Sometimes it’s to make sure I have write permissions in a particular directory, other times I want to create the file and then edit it in vim.

It’s easy enough to right click in the folder and click create new file, but if you’re already in the shell (and why aren’t you?), just try this command.

touch path/to/the/file/you/want/to/create

The touch command creates an empty file. If the file already exists, it update the modified timestamp in the filesystem for that file.

clear/cls

2012-06-07T00:00:00-07:00

Shell (~bash) - OS X, Windows, Linux

I live in the shell as much as possible. Sometimes it’s hard to see what’s going on because there is so much junk in my scrollback (btw, you ought to up your scrollback to as large as possible, you never know when you need to scroll up through a week’s worth of work to see something you did last Monday).

Anyways, sometimes I just want to clear the screen. Wouldn’t you know it…

clear

That’s for bash-ish shells. If you’re in Windows (in cmd.exe) it’s:

cls

(It stands for clear screen)

Open Explorer/Finder Here

2012-06-06T10:00:00-07:00

Shell (~bash) - Windows/OS X

Even though I love my shell, sometimes I want the gui. Quickly open up a Windows Explorer or Finder window pointed to your current working directory with the explorer or open command (respectively).

Windows

explorer .

OS X

open .

Of course, you can do many more things with both the open and explorer commands, but I use this one constantly.

history | grep

2012-06-05T00:00:00-07:00

Shell (~bash) - OS X, Windows, Linux

Pipe history into grep to search your past commands in a shell

> history | grep unicorn
> ... <snip>
> 976  bundle exec unicorn -c config/unicorn/production.rb -E production -D
> ... </snip>

cat

2012-06-04T00:00:00-07:00

Shell (~bash) - OS X, Windows, Linux

Use cat to dump the contents of a file to the screen

cat .gitignore

At least 10 times a day I’m in a shell and want to see what is in a small text file. Maybe it’s a log file, or configuration file. cat is the easiest way to get it onto the screen.

What the..?

2012-05-31T00:00:00-07:00

The intent of this blog is to be a place to share short practical snippets of things you do every day that you’d like to share with other folks.

I’m envisioning things like shell/command prompt tricks, or a cool C# snippet.

If you have something that you’d like to share or contribute, shoot me an email, stop by my desk, tweet at me @jon_fuller, send me a pull request, or anything else you can think of.

Here’s an example of what a typical post will look like:

Short description maybe including some code or a snippet.

usage of the actual command

Some additional context on usage and how or why you use it every day. This could be a one sentence testimonial, or a paragraph if it takes some explaining as to what’s going on. Anyways, please contribute!