Basics: By default, fields are separated by any number of [...]]]> [author's note: personally, I use awk a bunch in MySQL DBA work, for tasks like scrubbing data from a production export for use in qa/dev, but usually have to resort to Perl for really complex stuff, but now I know how to do .]

Basics:
By default, fields are separated by any number of spaces. The -F option to awk changes the separator on commandline.
Print the first field, fields are separated by a colon.
awk -F: ‘{print $1}’ /etc/passwd

Print the first and fifth field:
awk -F: ‘{$print $1,$5}’ /etc/passwd

Can pattern match and use files, so you can replace:
grep foo /etc/passwd | awk -F: ‘{print $1,$5}’
with:
awk -F: ‘/foo/ {print $1,$5}’ /etc/passwd

NF = built in variable (no $) used to mean “field number”
This will print the first and last fields of lines where the first field matches “foo”
awk -F: ‘$1 ~/foo/ {print $1,$NF}’ /etc/passwd

NF = number of fields, ie, “7″
$NF = value of last field, ie “/bin/bash”
(similarly, NR is record number)

Awk makes assumptions about input, variables, and processing that you’d otherwise have to code yourself.

- “main loop” of input processing is done for you
- awk initializes variables for you, to 0
- input is viewed by awk as ‘records’ which are splittable into ‘fields’

This all makes a lot of operations very concise in awk, many things can be done w/ a one-liner that would otherwise require several lines of code.

awk key points:
- splits text into fields
- default delimiter is “any number of spaces”
- reference fields
- $0 is entire line
- create filters using ‘addresses’ which can be regexps (similar to sed)
- Turing-complete language
- has if, while, for, do-while, etc
- built-in math like exp, log, rand, sin, cos
- built-in string sub, split, index, toupper/lower

Patterns and actions
Pattern is first, then action(s)
Actions are enclosed in {}

only a pattern, no action:
‘length>42′
but, the default action is to print the whole line, so this will actually do something — print lines where the length of the line is > 42. strings are just arrays in awk

only action, no pattern:
{print $2,$1}
do this to all lines of input

NR % 3 == 0
print every third line (pattern is %NR mod 3)

{print $1, $NF, $(NF-1)}
print the first field, last field, and 2nd to last field

built-in variables
NF, NR we’ve done
FS = field separator (can be regexp)
OFMT = output format for numbers (default %.6g)

Patterns
- used to filter lines processed by awk
- can be regexp
/^root/ is the pattern in the following
awk -F:’/^root/ {print $1,$NF}’ /etc/passwd

- Patterns can use fields and relational operators
To print 1st, 4th and last field if value of 4th field >10:
awk -F: ‘$4 > 10 {print $1, $4, $NF}’ /etc/passwd

awk -F: ‘$0 !~ /^#/ && $4 > 10 {print $1, $4, $NF}’ /etc/passwd

Range patterns
sed-like addressing : you can have start and end addresses
awk ‘NR==1,NR==3′
prints only first three lines of the file
You can use regular expressions in range patterns:
awk -F:’/^root/,/^daemon/ {print $1,$NF}’ /etc/passwd
start printing at the line that starts with “root”, the last line that is processed is the line starting with “daemon”

Range pattern “gotcha” – can’t mix a range with other patterns:
To do “start at non-commented line where value of $4 is less than 10, end at the first line where value of $4 is greater than 10″

This does not work!
awk -F: ‘$0 !~ /^#/ $4 <= 10, $4 > 10′ /etc/passwd

This is how to do it, {next} is an action that skips:
awk -F: ‘$0 ~ /^#/ {next} $4 <= 10, $4 > 10 {print $1, $4′ /etc/passwd

Basic Aggregation
awk -F: ‘$3 > 100 {x+=1; print x}’ /etc/passwd
This gives a line of output as each matching line is processed. This gives a running total of x.

awk -F: ‘$3 > 100 {x+=1} END {print x}’ /etc/passwd
This processes the “{print x}” action only after the entire file has been processed. This gives only the final value of x.

Arrays:
Support for regular arrays
Technically multi-dimensional arrays are not supported, but array indexes are not supported, so you can make your own associative arrays.

Example:
awk -F: ‘{x[$1] = $2*($4 – $3)} END {for(key in x) {print key, x[key]}}’ stocks.txt

The part before the END creates the associative array, the part after the END prints the array.

Extreme data munging:
awk -f: ‘{x[$1]=($2′($4 – $3))} END {for(z in x) {print z, x[z]}}’ stocks.txt

ABC,100,12.14,19.12
FOO,100,24.01,17.45

output
BAR 271.5
ABC 698


For the line “ABC,100,12.14,19.12″
the function becomes

x[ABC] = 100 * (19.12 - 12.14) = 698


Aggregate across multiple variables:
awk -F, ‘{x[$1] = $2*($4 – $3); y+=x[$1]} END {for(z in x) {print z, x[z]}} {print “Net:”y}}’ stocks.txt

Note that y is a running *sum* (not a running count like before).

Now, the above is hard to read, this is much easier.


#!/usr/bin/awk -f

BEGIN { FS="," }
{ x[$1] = $2*($4 - $3)
y+=x[$1]
}
END {
for(z in x) {
print z, x[z]
}
} # end for loop
{
print "Net:"y
} # end END block


This was liveblogged, so please point out any issues, as they may be typos on my part….

This blog entry was provided by Sheeri Cabral, and originally published online at http://www.pythian.com/news/12221/liveblogging-senior-skills-grok-awk/

- Low WTF per minute factor - Passes the 6-month test (if you write python code, going back in 6 months, you pretty much know what you were trying to do) - Small Shift/no-Shift ratio (ie, you use the “Shift” key a lot in Perl because you use $ % ( ) [...]]]> Why Python?

- Low WTF per minute factor
- Passes the 6-month test (if you write python code, going back in 6 months, you pretty much know what you were trying to do)
- Small Shift/no-Shift ratio (ie, you use the “Shift” key a lot in Perl because you use $ % ( ) { } etc, so you can tell what something is by context, not by $ or %)
- It’s hard to make a mess
- Objects if you need them, ignore them if you don’t.

Basics
Here’s a sample interpreter session. The >>> is the python prompt, and the … is the second/subsequent line prompt:

>>> x='hello, world!';
>>> x.upper()
'HELLO, WORLD!'
>>> def swapper(mystr):
... return mystr.swapcase()
File "", line 2
return mystr.swapcase()
^
IndentationError: expected an indented block


You need to put a space on the second line because whitespace ‘tabbing’ is enforced in Python:

>>> def swapper(mystr):
... return mystr.swapcase()
...
>>> swapper(x)
'HELLO, WORLD!'
>>> x
'hello, world!'


Substrings
partition is how to get substrings based on a separator:

>>> def parts(mystr, sep=','):
... return mystr.partition(sep)
...
>>> parts(x, ',')
('hello', ',', ' world!')


You can replace text, too, using replace.

>>> def personalize(greeting, name='Brian'):
... """Replaces 'world' with a given name"""
... return greeting.replace('world', name)
...
>>> personalize(x, 'Brian')
'hello, Brian!'


By the way, the stuff in the triple quotes is automatic documentation. A double underscore, also called a “dunder”, is to print the stuff in the triple quotes:

>>> print personalize.__doc__
Replaces 'world' with a given name


Loop over a list of functions and do that function to some data:

>>> funclist=[swapper, personalize, parts]
>>> for func in funclist:
... func(x)
...
'HELLO, WORLD!'
'hello, Brian!'
('hello', ',', ' world!')


Lists

>>> v=range(1,10)
>>> v
[1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> v[1]
2
>>> v[5]
6
>>> v[-1]
9
>>> v[-3]
7


List slicing with “:”
>>> v[:2]
[1, 2]
>>> v[4:]
[5, 6, 7, 8, 9]
>>> v[4:9]
[5, 6, 7, 8, 9]
Note that there’s no error returned even though there’s no field 9. If you did v[9], you’d get an error:
>>> v[9]
Traceback (most recent call last):
File ““, line 1, in
IndexError: list index out of range

Python uses pointers (or pointer-like things) so v[1:-1] does not print the first and last values:

>>> v[1:-1]
[2, 3, 4, 5, 6, 7, 8]


The full array syntax is [start:end:index increment]:

>>> v[::2]
[1, 3, 5, 7, 9]
>>> v[::-1]
[9, 8, 7, 6, 5, 4, 3, 2, 1]
>>> v[1:-1:4]
[2, 6]
>>> v[::3]
[1, 4, 7]


Make an array of numbers with range

>>> l=range(10)
>>> l
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]


Make a list from another list

>>> [pow(num,2) for num in l]
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]


append appends to the end of a list

>>> l.append( [pow(num,2) for num in l])
>>> l
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]]
>>> l.pop()
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]


extend takes a sequence and puts it at the end of the array.

>>> l.extend([pow(num,2) for num in l])
>>> l
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 4, 9, 16, 25, 36, 49, 64, 81]


A list can be made of a transformation, an iteration and optional filter:
[ i*i for i in mylist if i % 2 == 0]
transformation is i*i
iteration is for i in mylist
optional filter is if i % 2 == 0

>>> L=range(1,6)
>>> L
[1, 2, 3, 4, 5]
>>> [ i*i for i in L if i % 2 == 0]
[4, 16]


Tuples
Tuples are immutable lists, and they use () instead of []
A tuple always has 2 elements, so a one-item tuple is defined as
x=(1,)

Dictionaries aka associative arrays/hashes:


>>> d = {'user':'jonesy', 'room':'1178'}
>>> d
{'user': 'jonesy', 'room': '1178'}
>>> d['user']
'jonesy'
>>> d.keys()
['user', 'room']
>>> d.values()
['jonesy', '1178']
>>> d.items()
[('user', 'jonesy'), ('room', '1178')]
>>> d.items()[0]
('user', 'jonesy')
>>> d.items()[0][1]
'jonesy'
>>> d.items()[0][1].swapcase()
'JONESY'

There is no order to dictionaries, so don’t rely on it.

Quotes and string formatting
- You can use single and double quotes inside each other
- Inside triple quotes, you can use single and double quotes
- Variables are not recognized in strings, uses printf-style string formatting:

>>> word='World'
>>> punc='!'
>>> print "Hello, %s%s" % (word, punc)
Hello, World!


Braces, semicolons, indents
- Use indents instead of braces
- End-of-line instead of semicolons

if x == y:
print "x == y"
for k,v in mydict.iteritems():
if v is None:
continue
print "v has a value: %s" % v


This seems like it might be problematic because of long blocks of code, but apparently code blocks don’t get that long. You can also use folds in vim [now I need to look up what folds in vim are].

You can’t assign a value in a conditional statement’s expression — because you can’t use an = sign. This is on purpose, it avoids bugs resulting from typing if x=y instead of if x==y.

The construct has no place in production code anyway, since you give up catching any exceptions.

Python modules for sysadmins:
- sys
- os
- urlib/urlib2
- time, datetime (and calendar)
- fileinput
- stat
- filecmp
- glob (to use wildcards)
- shutil
- gzip
- tarfile
- hashlib, md5, crypt
- logging
- curses
- smtplib and email
- cmd

The Zen of Python
To get this, type ‘python’ in a unix environment, then type ‘import this’ at the commandline. I did this on my Windows laptop running Cygwin:

cabral@pythianbos2 ~
$ python
Python 2.5.2 (r252:60911, Dec 2 2008, 09:26:14)
[GCC 3.4.4 (cygming special, gdc 0.12, using dmd 0.125)] on cygwin
Type “help”, “copyright”, “credits” or “license” for more information.
>>> import this
The Zen of Python, by Tim Peters


Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!


This was liveblogged, please let me know any issues, as they may be typos….

One point to keep in mind: The goal is not equality for equality’s sake. The goal is to have a diverse range of experience to make your company/project/whatever [...]]]> I am moderating and liveblogging the Professional IT Community Conference panel called Tech Women Rule! Creative Solutions for being a (or working with a) female technologist.

One point to keep in mind: The goal is not equality for equality’s sake. The goal is to have a diverse range of experience to make your company/project/whatever the best it could be.

That being said, these issues are not just around women; they are about anyone who is “different”, whether it’s race, ethnicity, gender, sexual orientation, cultural.

So what are some of the solutions?

0) Better align expectations with reality. Are you expecting more from someone who is one gender than another? If a woman makes a mistake is it worse because she has to prove herself? Is it worse because she is representative of her gender? If she does something good is the achievement elevated more because of her gender? Either is bad.

1) Respect people for who they are. Everyone deserves respect; if someone is not at your technical level, they still deserve respect.

If someone says something that is completely wrong from a technical perspective, do not assume that they have no idea what they are talking about. It could be that they are the exact case in which that technical scenario is appropriate for them. If they are correct, your attitude will be refreshing and you might learn something. If they are indeed wrong, ask them about a scenario in which their thinking falls apart, or otherwise guide them through learning why what they are saying is wrong.

2) Be nice. Don’t condescend.

3) Be helpful. “RTFM, n00b!” is not helpful, and certainly does not follow rule #2.

4) Don’t do #1-3 for women only. Don’t treat women nicely because they’re women, and be a jerk to men because they’re men. Being helpful is good for anyone, not just women.

5) Cooperate, do not compete. Whether you are co-workers, working together on a software project, or just in a conversation, the game of “one-upping” another is a lot less useful than working together.

6) When hiring or when in an interview, concentrate on skills, not knowledge. “Skills” refers to attributes such as their ability to listen, how judgmental they are about a legacy system, whether they are open to new ideas, whether they disdain anything that is not cutting edge, and even technical skills such as thinking patterns, research patterns, algorithms, etc.

If someone says “I don’t know” in an interview, ask them “how would you go about figuring it out?” If someone says “I think it’s x and y” ask “how would you confirm/test that?” If a backup failed, do they start the backup over or do they try to figure out why it failed?

Are they thorough? Do they follow through? It is a lot easier to teach knowledge than it is to teach something like “debugging skills”.

7) Specifically encourage people to speak up. Train yourself to NOTICE when folks are not speaking up, and ask them if they have any suggestions or ideas.

8) If you are running an IT conference, specifically ask qualified women you know to speak, not about “women in IT”. If you hear of an IT conference, tell specific women you know that you think they would be a great speaker. Get women to speak at local user groups to get practice in a less intimidating space.

Resources:
Read HOWTO Encourage Women in Linux

You Just Don’t Understand
The Male Mind at Work
How to succeed in business without a penis: a guide for working women (this is a humor book)

Join and/or send messages to Systers, the world’s largest e-mail list of women in computing and technology fields.

This blog entry was provided by Sheeri Cabral, and originally published online at http://www.pythian.com/news/12359/liveblogging-tech-women-rule/

Ways to learn: Audio Visual Kinetic (doing it)

Everyone learns differently, but most people learn with some combination of all these three.

However, you can also learn by training [that's the truth, I learned a LOT by writing the book, even things [...]]]> Liveblog of the Professional IT Community Conference session Mentoring: It’s for everyone

Ways to learn:
Audio
Visual
Kinetic (doing it)

Everyone learns differently, but most people learn with some combination of all these three.

However, you can also learn by training [that's the truth, I learned a LOT by writing the book, even things I knew, I ended up needing to research more].

Ways to train:
Explanation
Observation
Demonstration
Questioning (Socratic Method)

What is a mentor?
noun: experienced and trusted adviser. It’s not just someone who teaches, it’s someone who advises.
experienced person in a company, college or schools who trains and counsels new employees or students.

verb: to advise or train (someone, esp. a younger colleague).

A mentorship is a safe place to ask questions.

A mentor is a trainer, but a trainer who also is a professional advisor.

Finding a mentor
Someone…..
- you respect/admire
- works with similar technology
- has a compatible personality
- you have a good rapport with

Being a mentor
- Teach technical skills
- Provide advanced technical/design guidance
- Model and teach professional skills
- Be interested and invested in the [student's] career

This blog entry was provided by Sheeri Cabral, and originally published online at http://www.pythian.com/news/12367/liveblogging-mentoring-its-for-everyone/.

But your boss will probably let you attend a 2-day conference if it is inexpensive, has many of the nationally recognized speakers you would normally attend, and since it [...]]]> Don’t be afraid. It’s just New Jersey!

The economy sucks. Training budgets are non-existent. That national conference you usually attend? Not going to happen.

But your boss will probably let you attend a 2-day conference if it is inexpensive, has many of the nationally recognized speakers you would normally attend, and since it is a Friday/Saturday conference you’ll only miss one day of work. Heck, point out that staying nearby is more "green" than flying to a distant conference.

We know you don’t love New Jersey. But we promise that except for a 3-block walk from the train station to the hotel, you can stay indoors the entire time and pretend you are still in the city. We promise! (There are some amazing restaurants within 1 block of the hotel. It’s a shame that the registration fee includes all meals.)

Hotel info:

The event is at the Hyatt Regency New Brunswick, Two Albany Street, New Brunswick, New Jersey, USA 08901. (Special room rates available.)

Driving directions

Directions and GPS advice.

Take the train from NYC!

The conference is 3 blocks from the New Brunswick train station. Just take NJ Transit from the New York Penn Station.

Going to the conference:

1. From New York Penn Station take the North East Corridor line to New Brunswick.
   • Traveling on Thursday, May 6th
   • Traveling on Friday, May 7th (training: arrive by 8:15, conference-only: arrive by 4pm)
   • Traveling on Saturday, May 8th (please arrive by 8:30am)
2. Map of walking to the Hotel.
3. After arriving at the New Brunswick train station, go to Albany Street. It is parallel to the train tracks, on the other side (if you came from NYC). You should be walking down hill to get there.
4. Cross over Albany Street to the east-bound side.
5. Turn LEFT and walk along Albany Street’s east-bound side. You should be walking with traffic.
6. The Hyatt is on the RIGHT. Look for signs. The main entrance is on Neilson St.
7. If walk under a train bridge, you went the wrong way. If you walk over a bridge, you went too far.

Going home to NYC:

1. Train schedule: Returning Saturday, May 8th (conference ends by 6:30pm, 7pm at the latest)

Take the train from Philly!

It is much faster to drive. New Brunswick is 1 hour away. However, there are two ways to get to New Brunswick by train.

Option 1: SEPTA to Trenton then NJ Transit to new Brunswick:

1. Take SEPTA to Trenton (R7).
2. Take NJ Transit "Northeast Corridor" towards New York Penn Station. Get off at the New Brunswick station.
   • Traveling on Thursday, May 6th
   • Traveling on Friday, May 7th (training: arrive by 8:15, conference-only: arrive by 4pm)
   • Traveling on Saturday, May 8th (please arrive by 8:30am)

Option 2: Amtrak from Philly to New Brunswick:

This is faster and you don’t have to change trains. However it is much more expensive.

Seats on Amtrak are now pre-reserved for specific trains (you used to be able to buy a ticket and use it any train in the next few months). Buy your ticket ahead of time on www.amtrak.com before they sell out.

Take Amtrak from Philadelphia – 30th Street Station, PA (PHL) to New Brunswick, NJ (NBK)

Walking from the train station to the hotel:

1. Map of walking to the Hotel.
2. After arriving at the New Brunswick train station, go to Albany Street. It is parallel to the train tracks, on the same side of the tracks (if you came from Phily). You should be walking slightly down hill to get there.
3. Cross over Albany Street to the east-bound side.
4. Turn LEFT and walk along Albany Street’s east-bound side. You should be walking with traffic.
5. The Hyatt is on the RIGHT. Look for signs. The main entrance is on Neilson St.
6. If walk under a train bridge, you went the wrong way. If you walk over a bridge, you went too far.

Going home to Philly:

1. Train schedule: Returning Saturday, May 8th (conference ends by 6:30pm, 7pm at the latest)

Registration now starts as low as $249 for the Friday night/Saturday package.

See internationally known speakers and trainers without hardly any travel!

Come to a regional conference and meet your local [...]]]> Good news!  The early-bird discount is being extended 7 days!   Save $75 if you register for training between now and Monday.  http://picconf.org

Registration now starts as low as $249 for the Friday night/Saturday package.

See internationally known speakers and trainers without hardly any travel!

Come to a regional conference and meet your local peers!
        http://lopsanj.org/events/picc10/overview

Originally the discount registration would have ended tonight at midnight. However, due to the full schedule only recently being available, we’ve decided to add 1 week to the early-bird discount (Monday, 3/22/2010).  Save $75 now!  We can’t extend it a second time!

http://lopsanj.org/events/picc10/registration

Convention scheduled for May 7th and 8th at the Hyatt Regency New Brunswick

Princeton, NJ, January 25, 2010 – Despite the down economy, things are heating up for New Jersey’s fledgling IT community. Professional computer and network system administrators from startups, [...]]]> Press Release

‘Two days of speakers, panels, training, and “unconference”. For technical people, by technical people.’

LOPSA-New Jersey Professional IT Community Conference (PICC) Fri/Sat May 7-8, 2010, Hyatt Regency, in sunny New Brunswick, NJ.

http://www.picconf.org

LOPSA [...]]]> LOPSA-NJ is excited to announce a conference for the IT and system administration community in New Jersey and surrounding area!

‘Two days of speakers, panels, training, and “unconference”.
For technical people, by technical people.’

LOPSA-New Jersey Professional IT Community Conference (PICC)
Fri/Sat May 7-8, 2010, Hyatt Regency, in sunny New Brunswick, NJ.

http://www.picconf.org

LOPSA = the League of Professional System Administrators
NJ = New Jersey
PICC = Professional IT Community Conferencee

The long version:
PICC10 is a gathering of professionals from the diverse IT (computer and network administration) community in New Jersey to learn, share ideas, and network. The conference includes invited speakers and keynotes, training by top-notch experts that is relevant, useful, and recession-friendly; plus an “unconference” track where attendees propose and host their own topics during the event. We expect attendance of 100 to 150 IT professionals from mid- to large-sized companies and academia from New Jersey/New York/Pennsylvania. We go by many titles but everyone is invited: system administrators, network administrators, network engineers, Windows, Linux, Unix, DBAs, and so on. YOU are invited!

Awesome! Keep me informed:

• Twitters: http://www.twitter.com/picconf
• Facebook: http://tinyurl.com/fb-picc
• Emails: http://lopsanj.org/mailman/listinfo/picc2010-announce

Read the full “Call for Participation”:

• http://lopsanj.org/events/picc10/cfp
• (Yes, we need you to propose conference topics)

Sponsors needed! Reach a motivated, highly-targeted demographic:

• http://lopsanj.org/events/picc10/sponsors

See you there!

LOPSA is vendor-independant and non-profit. This conference is too.