1. Background

Classification is an everyday task, it is about selecting one out of several outcomes based on their features, e.g

• In recycling of garbage you select the bin based on the material, e.g. plastic, metal or organic.
• When purchasing you select the store from based e.g. on its reputation, prior experience, service, inventory and prices

Computational Classification – Supervised Machine Learning – is quite similar, but requires (relatively) well-formed input data combined with classification algorithms.

1.1 Examples of classification problems

• Finance/Insurance
  • Classify investment opportunities as good or not e.g. based on industry/company metrics, portfolio diversity and currency risk.
  • Classify credit card transactions as valid or invalid based e.g. location of transaction and credit card holder, date, amount, purchased item or service, history of transactions and similar transactions
• Biology/Medicine
  • Classification of proteins into structural or functional classes
  • Diagnostic classification, e.g. cancer tumours based on images
• Internet
  Document Classification and Ranking
  Malware classification, email/tweet/web spam classification
• Production Systems (e.g. in energy or petrochemical industries)
  Classify and detect situations (e.g. sweet spots or risk situations) based on realtime and historic data from sensors

1.2 Classification Algorithms

Classification algorithms comes in various types (e.g. linear, nonlinear, discriminative etc), see my prior postings Pragmatic Classification: The Very Basics and Pragmatic Classification of Classifiers.

Key takeaways about classifiers:

1. There is no silver bullet classifier algorithm or feature extraction method.
2. Classification algorithms is that they tend to be computationally hard to train, this encourages using a parallel approach, in this case with Hadoop/Mapreduce.

2. Parallel Classification for Hadoop Streaming

The classifier described belongs to a familiy of classifiers which have in common that they can mathematically be described as Tikhonov Regularization with a Square loss function, this family includes Proximal SVM, Ridge Regression, Shrinkage Regression and Regularized Least-Squares Classification. (note: If you replace the Square Loss function with a Hinge-Loss function you get Support Vector Machine classification). The implemented classifier – proximal SVM – is from the paper Incremental Support Vector Machine Classification, referred to as the paper below.

2.1 training data

The classifier assumes numerical training data, where each class is either -1.0 og +1.0 (negative or positive class), and features are represented as vectors of positive floating point numbers. In the algorithm below are:

D - a matrix of training classes, e.g. [[-1.0, 1.0, 1.0, .. ]]
A - a matrix with feature vectors, e.g. [[2.9, 3.3, 11.1, 2.4], .. ]
e - a vector filled with ones, e.g [1.0, 1.0, .., 1.0]
E = [A -e]
mu = scalar constant # used to tune classifier
D - a diagonal matrix with -1.0 or +1.0 values (depending on the class)

2.2 the classifier algorithm

Training the classifier can be done with right side of the equation (13) from paper

(omega, gamma) = (I/mu + E.T*E).I*(E.T*D*e)

Classification of an incoming feature vector x can then be done by calculating:

x.T*omega - gamma

which returns a number, and the sign of the number corresponds to the class, i.e. positive or negative.

2. Parallelization of the classifier with Hadoop Streaming and Python

Expression (16) in the paper has a nice property, it supports increments (and decrements), in the example there are 2 increments (and 2 decrements), but by induction there can be as many as you want:

(omega, gamma) = (I/mu + E_.T*E_1 + .. + E_i.T*E_i).I*
                 (E_1.T*D_1*e + .. + E_i.T*D_i*e)

where

E.T*E = E_1.T*E_1 + .. + E_i.T*E_i

and

E.T*De = E_1.T*D_1*e + .. + E_i.T*D_i*e

This means that we can parallelize the calculation of E.T*E and E.T*De, by having Hadoop mappers calculate each of the elements of the sums in as in the Python map() code below (sent to reducers as tuples)

2.3 – the mapper

def map(key, value):
   # input key= class for one training example, e.g. "-1.0"
   classes = [float(item) for item in key.split(",")]   # e.g. [-1.0]
   D = numpy.diag(classes)

   # input value = feature vector for one training example, e.g. "3.0, 7.0, 2.0"
   featurematrix = [float(item) for item in value.split(",")]
   A = numpy.matrix(featurematrix)

   # create matrix E and vector e
   e = numpy.matrix(numpy.ones(len(A)).reshape(len(A),1))
   E = numpy.matrix(numpy.append(A,-e,axis=1)) 

   # create a tuple with the values to be used by reducer
   # and encode it with base64 to avoid potential trouble with '\t' and '\n' used
   # as default separators in Hadoop Streaming
   producedvalue = base64.b64encode(pickle.dumps( (E.T*E, E.T*D*e) )    

   # note: a single constant key "producedkey" sends to only one reducer
   # somewhat "atypical" due to low degree of parallism on reducer side
   print "producedkey\t%s" % (producedvalue)

2.4 – the Reducer

def reduce(key, values, mu=0.1):
  sumETE = None
  sumETDe = None

  # key isn't used, so ignoring it with _ (underscore).
  for _, value in values:
    # unpickle values
    ETE, ETDe = pickle.loads(base64.b64decode(value))
    if sumETE == None:
      # create the I/mu with correct dimensions
      sumETE = numpy.matrix(numpy.eye(ETE.shape[1])/mu)
    sumETE += ETE

    if sumETDe == None:
      # create sumETDe with correct dimensions
      sumETDe = ETDe
    else:
      sumETDe += ETDe

    # note: omega = result[:-1] and gamma = result[-1]
    # but printing entire vector as output
    result = sumETE.I*sumETDe
    print "%s\t%s" % (key, str(result.tolist()))

2.5 – Mapper and Reducer Utility Code

Code used to run map() and reduce() methods, inspired by iterator/generator approach from this mapreduce tutorial.

def read_input(file, separator="\t"):
    for line in file:
        yield line.rstrip().split(separator)

def run_mapper(map, separator="\t"):
    data = read_input(sys.stdin,separator)
    for (key,value) in data:
        map(key,value)

def run_reducer(reduce,separator="\t"):
    data = read_input(sys.stdin, separator)
    for key, values in groupby(data, itemgetter(0)):
        reduce(key, values)

3. Finished?

Assume your running time goes through the roof even with the above parallel approach, what to do?

3.1 Mapper Increment Size really makes a difference!

Since there is only 1 reducer in the presented implementation, it is useful to let mappers do most of the job. The size of the (increment) matrices – E.T*E and E.T*D*e given as input to the reducer is independent of number of training data, but dependent on the number of classification features. The workload on the reducer is also dependent on the number of matrices received by the mappes (i.e. increment size), e.g. if you have a 1000 mappers having one billion examples with 100 features each, the reducer would need to do a sum of one trillion 101×101 matrices and one trillion 101×1 vectors if the mapper sent one matrix pair per training example, but if each mapper only sent one pair of E.T*E and E.T*D*e representing all the mappers billion training examples the reducer would only need to summarize 1000 matrix pairs.

3.2 Avoid stressing the reducer

Add more (intermediate) reducers (combiners) that calculates partial sums of matrices. In the case of many small increments (and correspondingly many matrices) it can be useful to add an intermediate step that (in parallel) calculates sums of E.T*E and E.T*D*e before sending the sums to the final reducer, this means that the final reducer gets fewer matrices to summarize before calculating the final answer, see figure below.

3.3 Parallelize (or replace) the matrix inversion in the reduction step

If someone comes along with a training data set with a very high feature-dimension (e.g. recommender systems, bioinformatics or text classification), the matrix inversion in the reducer can become a real bottleneck since such algorithms typically are O(n^3) (and lower bound of Omega(n^2 lg n)), where n is the number of features. A solution to this can be to use or develop hadoop/mapreduce-based parallel matrix inversion, e.g. Apache Hama, or don’t invert the matrix...

3.4 Feature Dimensionality Reduction

Another approach when having training data with high feature-dimension could be to reduce feature-dimensionality, for more info check out Latent Semantic Indexing (and Analysis), Singular Value Decomposition or t-Distributed Stochastic Neighbor Embedding

3.5 Reduce IO between mappers and reducers with compression

Twitter presented using LZO compression (on the Cloudera blog) to speed up Hadoop. Inspired by this one could in the case of high feature dimension, i.e. large E.T*E and E.T*D*e matrices, compress the output in the mapper and decompress in the reducer by replacing base64encoding/decoding and pickling above with:

producedvalue = base64.b64encode(lzo.compress(pickle.dumps( (E.T*E, E.T*D*e) ), level=1)

and

ETE, ETDe = pickle.loads(lzo.decompress(base64.b64decode(value)))

3.6 Do more work with approximately the same computing resources

The D matrix above represents binary classification with a value of +1 or -1 representing each class. It is quite common to have classification problems with more than 2 classes. Supporting multiple classes is usually done by training by several classifiers, either 1-against-all (1 classifier trained per class) or 1-against-1 (1 classifier trained per unique pair of classes), and the run a tournament of them against each other and pick the most confident. In the case of 1-against-all classification the mapper could probably send multiple E.T*D_c*e – with one D_c per class and keep the same E.T*E, the reducer would then need to calculate (I/mu + E.TE).I once and independently multiply with several E.T*D_c*e sums to create a set of (omega,gamma) classifiers. For 1-against-1 classification it becomes somewhat more complicated, because it involves creating several E matrices since in the 1-against-1 case only the rows in E where the 2 classes competing occur are relevant.

4. Code

(Early) Python code of the algorithm presented above can be found at http://code.google.com/p/snabler/ (open source with Apache Licence). Please let me know if you want to contribute to the project, e.g. from mapreduce and hadoop algorithms in academic papers.

5. More resources about machine learning with Hadoop/Mapreduce?

• Apache Mahout – active project that implements (in Java) several machine learning algorithms (also unsupervised machine learning, i.e. clustering)
• Good paper about machine learning algorithms with mapreduce – http://www.cs.stanford.edu/people/ang//papers/nips06-mapreducemulticore.pdf

More details: http://aws.amazon.com/solutions/case-studies/atbrox/

Contact us if you need help with Hadoop/Elastic Mapreduce.

How to combine Elastic Mapreduce with other AWS Services

This posting shows how to load boto in an Elastic Mapreduce mapper and gives a simple example how to use simpledb from the same mapper. For accessing other AWS services, e.g. SQS from Elastic Mapreduce check out the Boto documentation (it is quite easy when the boto + emr integration is in place).

Other tools used (prerequisites):

• s3cmd – to upload/download files to S3
• Elastic Mapreduce Ruby Client – to launch Elastic Mapreduce jobs

Step 1 – getting and preparing the Boto library

wget http://boto.googlecode.com/files/boto-1.8d.tar.gz
# note: using virtualenv can be useful if you want to
# keep your local Python installation clean
tar -zxvf boto-1.8d.tar.gz ; cd boto-1.8d ; python setup.py install
cd /usr/local/lib/python2.6/dist-packages/boto-1.8d-py2.6.egg
zip -r boto.mod boto

Step 2 – mapper that loads boto.mod and uses it to access SimpleDB

# this was tested by adding code underneath to the mapper
# s3://elasticmapreduce/samples/wordcount/wordSplitter.py

# get boto library
sys.path.append(".")
import zipimport
importer = zipimport.zipimporter('boto.mod')
boto = importer.load_module('boto')

# access simpledb
sdb = boto.connect_sdb("YourAWSKey", "YourSecretAWSKey")
sdb_domain = boto.create_domain("mymapreducedomain") # or get_domain()
# ..
# write words to simpledb
  for word in pattern.findall(line):
      item = sdb_domain.create_item(word)
      item["reversedword"] = word[::-1]
      item.save()
      # ...

Step 3 – json config file – bototest.json – for Elastic Mapreduce Ruby Client

[
  {
	"Name": "Step 1: testing boto with elastic mapreduce",
        "ActionOnFailure": "<action_on_failure>",
        "HadoopJarStep": {
		"Jar": "/home/hadoop/contrib/streaming/hadoop-0.18-streaming.jar",
          	"Args": [
            	"-input", "s3n://elasticmapreduce/samples/wordcount/input",
            	"-output", "s3n://yours3bucket/result",
            	"-mapper", "s3://yours3bucket/botoWordSplitter.py",
            	"-cacheFile", "s3n://yours3bucket/boto.mod#boto.mod",
          	]
        }
  }
]

Step 4 – Copy necessary files to s3

s3cmd put boto.mod s3://yours3bucket
s3cmd put botoWordSplitter.py s3://yours3bucket

Step 5 – And run your Elastic Mapreduce job

 elastic-mapreduce --create \
                   --stream \
                   --json bototest.json \
                   --param "<action_on_failure>=TERMINATE_JOB_FLOW"

Conclusion
This showed how to dynamically load boto and use it to access one other AWS service – SimpleDB – from Elastic Mapreduce. Boto supports most AWS services, so the same integration approach should work also for other AWS services, e.g. SQS (Queuing Service), RDS (MySQL Service) and EC2, check out the Boto API documentation or Controlling the Cloud with Python for details.

Note: a very similar integration approach should work for most Python libraries, also those that use/wrap C/C++ code (e.g. machine learning libraries such as PyML and others), but then it might be needed to do step 1 on Debian AMIs similar to what Elastic Mapreduce is using, check out a previous posting for more info about such AMIs.

80legs

80legs is a company specializing in the crawling and preprocessing part, where you can upload your seed urls (where to start crawling), configure your crawl job (depth, domain restrictions etc.) and also run existing or custom analysis code (upload java jar-files) on the fetched pages. When you upload seed files 80legs does some filtering before starting to crawl (e.g. if you have seed urls which are not well-formed), and also handles domain throttling and robots.txt (and perhaps other things).

Computational model: Since you can run custom code per page it can be seen as a mapper part of a MapReduce (Hadoop) job (one map() call per page); for reduce-type processing (over several pages) you need to move your data elsewhere (e.g. EC2 in the cloud). Side note: another domain with “reduce-less” mapreduce is quantum computing, check out Michael Nilsen’s Quantum Computing for Everyone.

Testing 80legs

Note: We have only tried with the built-in functionality and no custom code so far.

1) URL extraction

Job description: We used a seed of approximately 1,000 URLs and crawled and analyzed ~2.8 million pages within those domains. The regexp configuration was used (we only provided the URL matching regexp).

Result: Approximately 1 billion URLs were found, and results came in 106 zip-files (each ~14MB packed and ~100MB unpacked) in addition to zip files of the URLs that where crawled.

Note: Based on a few smaller similar jobs it looks like the parallelism of 80legs is somewhat dependent of the number of domains in the crawl and perhaps also on their ordering. In case you have a set of URLs where each domain has more than one URL it can be useful to randomize your seed URL file before uploading and running the crawl job, e.g. by using rl or coreutil’s shuf.

2) Fetching pages

Job description: We built a set of URLs – ~80k URLs that we wanted to fetch as html (using their sample application called 80App Get Raw HTML) for further processing. The URLs were split into 4 jobs of ~20k URLs each.

Result: Each job took roughly one hour (they all ran in parallel so the total time spent was 1 hour). We ended up with 5 zip files per job, each zip file having ~25MB of data (100MB unpacked), i.e. ~4*5*100MB = 2GB raw html when unpacked for all jobs.

Conclusion

80legs is an interesting service that has already proved useful for us, and we will continue to use it in combination with AWS and EC2. Custom code needs to be built (e.g. related to ajax crawling).

(May 2000 – A few thoughts about the future of Internet Information Retrieval)

Unstructured Search for SimpleDB

1. Support unstructured search with very low maintenance
2. Combine structured and unstructured search
3. Figure out the feasibility of unstructured search on top of SimpleDB

The Structure of SimpleDB

SimpleDB is roughly a persistent hashtable of hashtables, where each row (a named item in the outer hashtable)  has another hashtable with up to 256 key-value pairs (called attributes). The attributes can be 1024 bytes each, so 256 kilobyte totally in the values per row (note: twice that amount if you store data also as part of the keys + 1024 bytes in the item name). Check out Wikipedia for detailed SimpleDB storage characteristics.

Inverted files

Inverted files is a common way of representing indices for unstructured search. In their basic form they (logically) contain a word with a list of pages or files the word occurs on. When a query comes one looks up in the inverted file and finds pages or files where the words in the query occur. (note: if you are curious about inverted file representation check out the survey - Inverted files for text search engines)

One way of representing inverted files on SimpleDB is to map the inverted file on top of the attributes, i.e. have one SimpleDB domain with one word (term), and let the attributes store the list of URLs containing that word. Since each URL contains many words, it can be useful to have a separate SimpleDB domain containing a mapping from hash of URL to URL and use the hash URL in the inverted file (keeps the inverted file smaller). In the draft code we created 250 key-value attributes where each key was a string from “0″ to “249″ and each corresponding value contained hash of URLs (and positions of term) joined with two different string separators. If too little space per item – e.g. for stop words – one could “wrap” the inverted file entry with adding the same term combined with an incremental postfix (note: if that also gave too little space one could also wrap on simpledb domains).

Preliminary query latency results

Warning: Data sets used were  NLTK’s inaugural collection, so far from the biggest.

Conclusion: the results from 1000 fetches of inverted file entries are relatively stable clustered around 0.020s (20 milliseconds), which are promising enough to pursue further (but still early to decide given only tests on small data sets so far). Balancing with using e.g. memcached could be also be explored, in order to get average fetch time even lower.

Preliminary Python code including timing results (this was run on an Fedora large EC2 node somewhere in a US east coast data center).

1. Start a small EC2 node with AMI similar to the one Elastic Mapreduce is using (Debian Lenny Linux)
• note: We used Alestic’s ami-ff46a796
2. Skim quickly through the Shedskin tutorial
3. Log into the EC2 node and install the Shedskin Python compiler
• Shedskin requires a few libraries: 1) the Boehm-Demers-Weiser garbage collector for C++, 2) PCRE – Perl Compatible Regular Expressions. The Shedskin tutorial for detailed install instructions.
• note: The Alestic Debian AMI is fairly slim, so we had to add some more software make Shedskin work, i.e. GDB
4. Write your Python mapper or reducer program and compile it into C++ with Shedskin
• E.g. the commandpython ss.py mapper.py – would generate C++ files mapper.hpp and mapper.cpp, a Makefile and an annotated Python file mapper.ss.py.
5. Optionally update the C++ code generated by Shedskin to use other C or C++ libraries
• note: with Fortran-to-C you can probably integrate your Python code with existing Fortran code (e.g. numerical/high performance computing libraries). Similar for Cobol (e.g. in financial industry) with OpenCobol (compiling Cobol into C). Please let us know if you try or need help with help that.
6. Add -static as the first CCFLAGS parameter in the generated Makefile to make it a static executable
7. Compile the C++ code into a binary with make and check that you don’t get a dynamic executable with ldd (you want a static executable)
8. Run strip on the binary to make it smaller
9. Upload your (ready) binary to a chosen location in Amazon S3
   • e.g. via commandline with S3CMD, with a UI using S3Fox or Cloudberry S3 Explorer or programmatically with Boto.
10. Read Elastic Mapreduce Documentation on how to use the binary to run Elastic Mapreduce jobs.
    • note: Peter Skomoroch has written a good tutorial for Elastic Mapreduce

Note: if you skip the shedskin-related steps this approach would also work if you are looking for how to use C or C++ mappers or reducers with Elastic Mapreduce.

Note: this approach should probably work also with Cloudera’s distribution for Hadoop.

• Hadoop World NYC (Hilary Mason)
• The View from HadoopWorld (Stephen O’Grady)
• Post Hadoop World Thoughts (Deepak Singh)
• Hadoop World, NYC 2009 (Dan Milstein)
• Hadoop World Impressions (Steve Laniel)

—

Location: Roosevelt Hotel, NYC

1235 Joe Cunningham – Visa – Large scale transaction analysis
– responsible for Visa Technology Strategy and Innovation
been playing with Hadoop for 9 months
probably many in audience learning and starting out with Hadoop

Agenda:
1) VisaNet overview
2) Value-added information products
3) Hadoop@Visa – research results

About Visa:
- 60 Billion market cap
- well-known card products, and also behind the scene information products
- Visa brand has high trust
- For a card-holder a Visa-card means global acceptance
- For a shopowner, if you get a Visa payment aproval you will be payed

VisaNet
VisaNet is the largest, most advanced payment network in the world
characteristics:
28M locations,
130M authorizations/day,
1500 endpoints,
Processes transactions faster than 1s
1.4M ATMs,
Processes in 175 currencies,
Less than 2s unavailability per year (!)
- according to my calculations six 9s (0.999999366)
16300 financial institutions

Visa Processing Architecture
Security/Access Services -> Message|File|Web
VisaNet Services Integration -> Authorization|Clearing&Settlement
Dispute handling, Risk, Information
Scoring every transaction (used for issuer to approve/decline transaction)

Value added Info products
- Info services
Client: Portfolio Analysis, Visa Incentive Network
Accountholder: transaction alerts, accoutnt updater, tailored rewards
- Risk management services
Account monitoring
Authentication
Encyption

Hadoop@Visa
Run a pipeline of prototypes in lab facility in SF
Any technology taken into Visa needs to match scalability and reliability requirements

Research Lab Setup
- VM System:
Custom Analytic Stacks
Encryption Processing
Relational Database
- Hadoop Systems
Management Stack
Hadoop #1 ~40TB / 42 nodes (2 years of raw transaction data)
Hadoop #2 ~300TB / 28 nodes

Risk Product Use Case
Create critical data model elements, such as keys and transaction statistics, which feed our real-time risk-scoring systems
Input: Transactions – Merchant Category, Country/Zip
Output: Key & Statistics – MCCZIP Key – stats related to account, trans. type, approval, fraud, IP address etc.
Research Sample: 500M distinct accounts, 100M transactions per day, 200 bytes per transaction, 2 years – 73B transaction (36TB)
Processing time from 1 month to 13 minutes! (note: ~3000 times faster)
(Generate synthetic transactions used to test the model)

Financial Enterprise Fit
- key questions under research:
- what will the Hadoop Solution Stack(s) look like?
- File system, Transaction Sample System, Relational Back-end (integration path), Analytics Processing
- Internal vs external cloud
- How do I get data into a cloud in a secure way.
- How does HSM and security integration work in Hadoop
- What are the missing pieces?

Why Hadoop@Visa?
- analyze volumes of data with response that are not possible today
- requirement: need to fit with existing solutions

Cross Data Center Log Processing – Stu Hood, Rackspace

(Email and apps division, work on search team)

Agenda
Use Case Backgound
- “Rackapps” – Hybrid Mail Hosting, 40% use a mix of exchange and rackspace mail

Use Case: Log Types

Use Case: Querying
- was the mail delivered?
- spam – why was it (not) marked as spam
- access – who checked/failed to check mail?
more advanced questions:
- which delivery routes have the highest latency?
- which are the spammiest IP?
- Where in the world do customers log in from
Elsewhere:
- billing

Previous Solutions
- 1999-2006 – go to where log files are generated, querying with grep
- 2006-2007 / bulk load to MySQL – worked for a year

Hadoop Solution
- V3 – lucene indexes in Hadoop
- 2007- present
- store 7 days uncompressed
- queries take seconds
- long term queries with mapreduce (6M avail for MR queries)
- all 3 datacenters

Alternatives considered:
- Splunk – good for realtime, but not great for archiving
- Data warehouse package – not realtime, but fantastic for longterm analysis
- Partioned MySQL – half-baked solution
=> Hadoop hit the sweet spot

Hadoop Implementation
SW
- collect data using syslog-ng (considering Scribe)
- storage: deposits into Hadoop (scribe will remove that)
HW
- 2-4 collector machines per datacenters
- hundreds of source machines
20 solr nodes

Implementation: Indexing/Querying
- indexing – uniqe processing code for schema
- querying
- “realtime”
- sharded lucene/solr instances merge-index chunk from Hadoop
- using Solr-API
- raw logs
- using Hadoop Streaming and unix grep
- Mapreduce

Implementation: Timeframe
- development – 1.5 people in 3 months
- deployments – using clouderas distribution
- roadblocks – bumped into job-size limits

Have run close to 1 million jobs on our cluster, and it has not gone down (except for other reasons such as maintenance)

Advantages – storage
- all storage in one place
Raw logs: 3 days, in HDFS
Indexes: 7 days
Archived Indexes: 6 months

Advantages – analysis
- Java Mapreduce API
- Apache Pig
- ideal for one-off queries
- Hadoop Streaming

Pig Example – whitehouse.gov mail spoofing

Advantages – Scalability, Cost, Community
- scalability – easy to add nodes
- cost – only hardware
- community – cloudera has been a benefit, deployment is trivial

Data Processing for Financial Services – Peter Krey and Sin Lee, JP Morgan Chase

Innovation & Shared Services, Firmwide Engineering & Architecture

note: certain constraints what can be shared due to regulations

JPMorgen Chase + Open Source
- QPD (AMQP) – top level apache project
- Tyger – Apache + Tomcat + Spring

Hadoop in the Enterprise – Economics Driven
- attractive: economics
- Many big lessons from Web 2.0 community
- Potential for Large Capex and Opex “Dislocation”
- reduce consumption of enterprise premium resources
- grid computing economics brought to data intensive computing
- stagnant data innovation
- Enabling & potentially disruptive platform
- many historical similarities
- java, linux, tomcat, web/internet
- minis to client/server, client/server to web, solaris to linux, ..
- Key question: what can be built on top of Hadoop?
Back to economics driven – very cost-effective

Hadoop in the Enterprise – Choice Driven
- Overuse of relational database containers
- institutional “Muscle memory” – not too much else to choose from
- increasingly large percentage of static data stored in proprietary transactional DBs
- Over-Normalized Schemas: still Makes sense with cheap compute&storage?

- Enterprise Storage “Prisoners”
- Captive to the economics & technology of “a few” vendors
- Developers need more choice
- Too much proprietary, single-source data infrastructure
- increasing need for minimal/no systems + storage admins

Hadoop in the Enterprise – Other Drivers
- Growing developer interest in “Reduced RDBMS” Data technologies
- open source, distributed, non-relational databases
- growing influence of web 2.0 technologies & thinking of enterprise
- hadoop, cassandra, hbase, hive, couchdb, hadoopDB, .. , others
- memcached for caching

FSI Industry Drivers
- Increased regularity oversight + reporting = More data needed over longer period of time
- triple data amounts from 2007 to 2009
- growing need for less expensive data repository/store
- increased need to support “one off” analysis on large data

Active POC Pipeline
- Growing stream of real projects to gauge hadoop “goodness of fit”
- broad spectrum of use cases
- driven by need to impact/dislocate OPEX+CAPEX
- looking for orders of magnitude
- evaluated on metric based performance, functional and economic measures
- avoid the “data falling on the floor phenomena”
- tools are really really important, keep tools and programming models simple

Hadoop Positiong
- Latency x Storage amount curve,

Cost comparisons
- SAN vs Hadoop HDFS cost comparison (GB/month)
- Hadoop much cheaper

Hadoop Additions and Must Haves:
- Improves SQL Front-End Tool Interoperability
- Improved Security & ACL enforcement – Kerberos Integration
- Grow Developer Programming Model Skill Sets
- Improve Relational Container Integration & Interop for Data Archival
- Management & Monitoring Tools
- Improved Developer & Debugging Tools
- Reduce Latency via integration with open source data caching
- memcached – others
- Invitation to FSI or Enterprise roundtable

Protein Alignment – Paul Brown, Booz Allen

Biological information
- Body – Cells – Chromosomes – Gene – DNA/RNA

Bioinformatics – The Pain
- too much data

So What? Querying a database of sequences for similar sequences
- one-to-many comparison
- 58000 proteins in PDB
- Protein alignment frequently used in the development of medicines
- Looking for a certain sequence across species, helps indicate function
Implementation in Hadoop
- distribute database sequence accross each node
- send query seq. inside Mapreduce (or dist.cache)
- scales well
- existing algorithms port easily

So What? Comparing sequences in bulk
- many-to-many
- DNA hybridiation (reconstruction)
Ran on AWS
Hadoop:
- if whole dataset fit into one computer
- Used distributed cache, assign each node a piece of the list
- But if the does not fit on one computer….
- pre-join all possible pairs with one MapReduce

So What? Analyzing really big sequences
- one big sequence to many small sequences
- scanning dna for structure
- population genetics
- hadoop implementatoin

Demonstration Implementation: Smith-Waterman Alignment
- one of the more computationally intensive matching and aligmnent techniques
- big matrix – (sequences to compare on row and column and calculations within)

Amazon implementation
- 250 machines
- E2
- run in 10 minutes for a single sequence. Runs in 24hrs for NxN comparison
- cost $40/hr

==> very cool 3D video of amazon ec2 nodes
- failing job due to 10% of nodes stuck on something (e.g. very long sequences)

Real-time Business Intelligence, Bradford Stephens

Topics
- Scalability and BI
- Costs and Abilities
- Search as BI

Tools: Zookeeper, Hbase, Katta (dist.search on Hadoop) and Bobo (faceted search for lucene)
- http://sourceforge.net/projects/bobo-browse/
- http://sourceforge.net/projects/katta/develop

100TB structured and unstructed data – Oracle 100M$, Hadoop and Katta 250K$

Building data cubes in real time (with faceted search)

Real-time Mapreduce on HBase
Search/BI as a platform – “google my datawarehouse”

Counting, Clustering and other data tricks, Derek Gottfried, New York Times

back in 2007 – would like to try as many EC2 instances as possible

Problem
- freeing up historical archives of NYTimes.com (1851-1922)
(in the public domain)

Currently:
- 2009 – web analytics
3 big data buckets:
1) registration/demographics
2) articles 1851-today
- a lot of metadata about each article
- unique data, extract people, places, .. to each article => high precision search
3) usage data/web logs
- biggest piece – piles up

How do we merge the 3 datasets?

Using EC2 – 20 machines
Hadoop 0.20.0
12 TB of data
Straight MR in Java
(mostly java + postprocessing in python)

combining weblog data with demographic data, e.g. twitter clicks backs by age group

09:11 – Christophe Bisciglia (Cloudera)

Announcement about BOFs HBASE and UI Birds of a Feather
Hadoop history overview
happenings during the last year: Hive, Pig, Sqoop (data import) ++
yesterday: Vertica announced mapreduce support for their database system
Walkthrough of Clouderas distribution for Hadoop
ANNOUNCEMENT: deploy clouderas dist. for hadoop on softlayer and rackspace

09:23 – Jeff Hammerbacher (Cloudera)
started his career at bear sterns
- Cloudera is a software company with Apache Hadoop at the core
There is a lot more sw to be built:
1) collection,
2) processing
3) report and analysis

The Apache Hadoop community is the center of innovation for Big Data
- Yahoo pushing env. on scalability
- Large clusters for academic research (yahoo, hp and intels open cirrus)
- nsf, ibm and google’s clue
- sigmod best paper award: Pig team from Yahoo
- worldwide – Hadoop world beijing

Cloudera Desktop
4 applications running on this desktop (inside the browser)
1) HDFS Web Interface
- file browser
2) Hadoop Mapreduce Web Interface (can potentially debug)
- Job Browser (Cluster Detail)
3) Cluster Health
- pulls in all kinds of metrics from a hadoop cluster
4) Job Designer
- makes it easier to use for non-tech users
note: available for free (can be locally modified), but not redistribute
window manager based on MooTools

Cloudera Desktop API
- building a reusable API for dev. dekstop appl
- would like to capture innovation of ecosystem in a single interface
- desktop-api-s

0940 – Peter Sirota (Amazon, general manager Amazon Elastic Mapreduce – EMR)

motivation: large scale data processing has a lot of MUCK, wanted to fix that.

Use cases for EMR:
- data mining (log processing, clicks analysis)
- bioinformatics (genome analysis)
- financial simulation (monte carlo)
- file processing (resize jpegs, ocr) – a bit unexpected
- web indexing

Customer feedback:
Pros: easy to use and reliable
Challenges: require fluency in mapreduce, and hard to debug

New features:
support for Apache Pig (batch and interactive mode), August 2009
support for Apache Hive 0.4 (batch and interactive mode), TODAY
- extended language to support S3
- specify off-instance-metadata store
- optimized data writes to S3
- reference resources on S3

ANNOUNCEMENT TODAY – Karmashpere Studio for Hadoop – Netbeans IDE
- deploy hadoop jobs to EMR
- monitor progress of EMR job flows
- amazon S3 file browser

ANNOUNCEMENT TODAY – Support for Cloudera’s Hadoop distribution
- can specify Cloudera’s distribution (and get support from Cloudera)
- in private beta

0951 – Amazon EMR case – eHarmony – Carlos – Will present Use case for matchmaking system
data: 20 million users, 320 item questionaire => big data
results: 2% of US marriages
Using Amazon, S3 and Elastic Mapreduce
Interesting with HIVE to do analysis

0958 – Amazon EMR IDE Support – Karmasphere IDE for Hadoop
works with all versions of Hadoop
tighly integrated with EMR (e.g. monitoring and files)

1005 – Eric Baldeschwieler – Yahoo
Largest contributor, tester and user of Hadoop
Hadoop is driving 2% of marriages in the US!
4 tiers of Hadoop clusters:
1) dev. testing and QA (10% of HW)
- continuous integration and testing
2) proof of concepts and ad-hoc work (10% of HW)
- run the latest version, currently 0.20
3) science and research (60% of HW)
- runs more stable versions, currently 0.20
4) production (20% of HW)
- the most stable version of Hadoop, currently 0.18.3

Yahoo has more than 25000 nodes with Hadoop (4000 nodes per cluster), 82 Petabytes of data.

Why Hadoop@Yahoo?
- 500M users, billions of “transactions”/day, Many petabytes of data
- analysis and data processing key to our business
- need to do this cost effectively
=> Hadoop provides solution to this

Previous job: chief architect for web search at Yahoo
Yahoo frontpage example (use of Hadoop):
- content optimization, search index, ads optimization, spam filters, rss feeds,

Webmap 2008-2009
- 70 hours runtime => 73 hours runtime
- 300TB shuffling => 490TB shuffling
- 200TB output -> 280TB (+55% HW, but more analysis

Sort benchmark 2008-2009
- 1 terabyte 209 seconds => 62 seconds on 1500 nodes
- 1 petabyte sorted – 16.25 hours, 3700 nodes

Hadoop has Impact on productivity
- research questions answered in days, not months
- moved from research to prod easily

Major factors:
- don’t need to find new HW to experiment
- can work with all your data
- prod. and research on same framework
- no need for R&D to do IT, clusters just work

Search Assist (index for search suggest)
3 years of log-data, 20 steps of mapreduce
before hadoop: 26 days runtime (SMP box), C++, 2-3 weeks dev.time
after hadoop: 20 minutes runtime, python, 2-3 days dev.time

Current Yahoo Development
Hadoop:
- simplifies porting effort (between hadoop versions), freeze APIs, Avro
- GridMix3, Mumak simulator – for performance tuning
- quality engineering
Pig
- Pig – SQL and Metadata, Zebra – column-oriented storage access layer, Multi-query, lots of other optimizations
Oozie

1035 Rod Smith, IBM

Customer Scenarios
- BBC Digital Democracy project
- Thomson Reuters
- IBM Emerging Technology Projects: M2 (renamed later to M42)
- insight engine for ad-hoc business insights running ontop of Hadoop and Pig
- macro-support (e.g. extract patent information)
- collections (probably renamed to worksheets later)
- visualization (tag cloud)
- example 1: evaluate companies with patent information(1.4 million patents)
- using American Express as case study
- counting patent citations
- example 2: patents in litigation
- quote: “in god we trust, everybody else bring data”

1104 Ashish Thusoo – Facebook – Hive datawarehousing system

Hadoop
Pros: superior in availability/scalability/manageability, open system, scalable cost
Cons: programmability and metadata, mapreduce hard to program (users know sql/bash/python/perl), need to publish in well-known schemas
=> solution: Hive

Hive: Open and Extensible
- query your own formats and types with serializer/deserializer
- extend SQL functionality through user defined functions
- do any non-SQL TRANSFORM operator (e.g. embed Python)

Hive: Smart Execution Plans for Performance
- Hash-based Aggregations
- Map-Side Joins
- Predicate Pushdown
- Partition Pruning
- ++

Interoperability
- JDBC and ODBC interfaces available
- integrations with some traditional SQL tools (e.g. Microstrategy for reports within Facebook) with some minor modifications
- ++

Hive Information
- subproject of Hadoop

— Date Warehousing @ Hadoop –

Data Flow Architecture at Facebook
web server logs -> Scribe -> filers (Hadoop clusters)
to save cost: Scribe/Hadoop integration
Federated MySQL also connected to the Production Hive/Hadoop Cluster
Connected to Oracle BAC and also replicated to an AdHoc Hive cluster

Showed a Picture of Yahoo cluster/datacenter

Dimensions:
4800 cores, 5.5 PB,

Statistics per day:
- 4TB compr.data/day
- 135TB scanned per day
- 7500 Hive jobs/day
- 80K compute hours per day

Hive Simplifies Hadoop:
- New engineers go through a Hive training session
- 200 people/moth use it

Applications:
- reporting (daily/weekly aggregations of impression/click counts)
- measures of user engagement
- microstragy dashboards

Ad hoc analysis
- how many group admins broken down by state/country

Machine learning (assembling training data)
- ad optimization
- e.g. user engagement as function of user attributes

Facebook Hive contributions
- Hive, HDFS features, Scheduler work
- Talks by Dhruba Borhthakur and Zheng Shao in the dev.track

Q from audience: relation to Cassandra?
A: Cassandra serving live traffic,

Q from audience: when to use Pig or Hive?
A: Hive has more SQL support, but Pig also gets more of that. Hive is
very intuitive. If you want interoperability (e.g. microstrategy)
advantages with using Hive. Pig has some nice primitatives and
supports more unstructured data model

Disclaimer: this is work in progress (look for updates)

Input Data – Academic Papers
Scholar has 981 papers citing the original Mapreduce paper from 2004 – a citation amount that is approximately 10 thousand pages (~ size of a typical encyclopedia)

What types of papers cite the mapreduce paper?

1. Algorithmic papers
2. General cloud overview papers
3. Cloud infrastructure papers
4. Future work sections in papers (e.g. “we plan to implement this with Hadoop”)

=> Looked at category 1 papers and skipped the rest

Who wrote the papers?

Search/Internet companies/organizations: eBay, Google, Microsoft, Wikipedia, Yahoo and Yandex.
IT companies: Hewlett Packard and Intel
Universities: Carnegie Mellon Univ., TU Dresden, Univ. of Pennsylvania, Univ. of Central Florida, National Univ. of Ireland, Univ. of Missouri, Univ. of Arizona, Univ. of Glasgow, Berkeley Univ. and National Tsing Hua Univ., Univ. of California, Poznan Univ.

Which areas do the papers cover?

Machine Translation
Grammar based statistical MT on Hadoop (2009)
Large Language Models in Machine Translation (2008)

Information/Entity Extraction and Tagging
Web-Scale Distributional Similarity and Entity Set Expansion (2009)
The infinite HMM for unsupervised PoS tagging (2009)

Classification
Scaling Up Classifiers to Cloud Computers (2008)

Ads Analysis
Large-Scale Behavioral Targeting (2009)
Search Advertising using Web Relevance Feedback (2008)
Predicting Ads’ ClickThrough Rate with Decision Rules (2008)

Search Query Analysis
BBM: Bayesian Browsing Model from Petabyte-scale Data (2009)
AIDE: Ad-hoc Intents Detection Engine over Query Logs (2009)

Indexing & parsing
On Single-Pass Indexing with MapReduce (2009)
A Data Parallel Algorithm for XML DOM Parsing (2009)
Semantic Sitemaps: Efficient and Flexible Access to Datasets on the Semantic Web (2008)

Spam & Malware Detection
Characterizing Botnets from Email Spam Records (2008)
- Clustering of emails into spam campaign
- Finding probability that 2 spam messages are sent form same machine
- Estime likelihood of botnets based on common senders in spam campaigns
The Ghost In The Browser Analysis of Web-based Malware (2007)

Image and Video Processing
MapReduce Optimization Using Regulated Dynamic Prioritization (2009)
- Video Stream Re-Rendering
Map-Reduce Meets Wider Varieties of Applications (2008)
- Location detection in images

Networking
Reducible Complexity in DNS

Simulation
Map-Reduce Meets Wider Varieties of Applications (2008)
- Simulation of earthquakes (geology)

Statistics
Fast Parallel Outlier Detection for Categorical Datasets using Mapreduce (2009)
MapReduce Optimization Using Regulated Dynamic Prioritization (2009)
- Digg.com story recommendations
Calculating the Jaccard Similarity Coefficient with Map Reduce for Entity Pairs in Wikipedia (2008)
- Measuring Wikipedia Editor similarity
Map-Reduce Meets Wider Varieties of Applications (2008)
- Netflix video recommendation
Large-scale Parallel Collaborative Filtering for the Netflix Prize (2008)

Graphs
DOULION: Counting Triangles in Massive Graphs with a Coin (2009)
Fast counting of triangles in real-world networks: proofs, algorithms and observations (2008)

Conclusion
On the papers looked at most of them are focused on IT-related areas, there is lots of unwritten in academia about mapreduce and hadoop applied for algorithms in other business and technology areas.

Opportunity for following up this posting can be to: 1) in more detail describe the algorithms (e.g. input/output formats), 2) try to classify them by patterns (e.g. with similar code structure), 3) offer the opportunity to simulate them in the browser (on toy-sized data sets) and 4) provide links to Hadoop implementations of them.

In short: virtualenv lets you switch seamlessly between isolated Python environments, Fabric automates remote deployment, while pip takes care of installing required packages and dependencies. If you have ever had to wrestle with more than one development project at the same time, then virtualenv is one of those tools that, once mastered, you can’t see yourself living without. Fabric and pip are somewhat immature, but still highly useful in their present shapes. It is likely that you will end up learning them anyway. Best of all, these three tools play very nicely together.

Except on Cygwin.

Here at Atbrox, we spend quite a lot of our time on Windows platforms. While Cygwin adds a fair amount of unix functionality to Windows, configuring certain applications can be difficult. This article describes the steps we go through to get an operational virtualenv, Fabric and pip setup on Windows Vista. It also gives you a brief taster of how virtualenv and Fabric works.

Step 1 – Install Cygwin: If you haven’t already, Cygwin can be installed from this page. Click the “View” button once to get a full list of available packages. Make sure to include at least the following packages (the numbers in the parentheses indicate the versions used at the time of writing):

• python (2.5.2-1)
• python-paramiko (1.7.4-1)
• python-crypto (2.0.1-1)
• gcc (3.4.4-999)
• wget (1.11.4-3)
• openssh (5.1p1-10)

Now would also be a good time to install other common packages such as vim, git, etc.—but you can always go back and install them at a later time.

Note that we are using Cygwin Python rather than the standard Windows Python. I had nothing but trouble trying to get Windows Python to play nicely along with virtualenv and Fabric, so this is a compromise. The downside is that you are stuck with a rather dated and somewhat buggy version of Python. If someone manages to get this setup working with Windows Python, then let me know!

Step 2 – Get paramiko working: The python-paramiko and python-crypto packages are required to get Fabric deployment over SSH working properly. If you are lucky, paramiko should work out of the box. If you don’t get the following error message when importing paramiko then skip the rest of this step:

$ python
Python 2.5.2 (r252:60911, Dec  2 2008, 09:26:14)
[GCC 3.4.4 (cygming special, gdc 0.12, using dmd 0.125)] on cygwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import paramiko
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "__init__.py", line 69, in <module>
 File "transport.py", line 32, in <module>
 File "util.py", line 31, in <module>
 File "common.py", line 101, in <module>
 File "rng.py", line 69, in __init__
 File "randpool.py", line 87, in __init__
 File "randpool.py", line 120, in _randomize
IOError: [Errno 0] Error

According to the discussion here, this appears to be a lingering Cygwin bug. The workaround is to change line 120 in /usr/lib/python2.5/site-packages/Crypto/Util/randpool.py from

if num!=2 : raise IOError, (num, msg)

to

if num!=2 and num!=0 : raise IOError, (num, msg)

Paramiko should now import without any complaints.

Step 3 – Install setuptools: Setuptools are required for installing the rest of the required Python packages. Instructions for Cygwin are found on the setuptools pages—but just enter the following and you’ll be all set:

$ wget http://pypi.python.org/packages/2.5/s/setuptools/setuptools-0.6c9-py2.5.egg
$ sh setuptools-0.6c9-py2.5.egg

Step 4 – Install pip, virtualenv and virtualenvwrapper: We haven’t said anything about virtualenvwrapper so far. This extension to virtualenv streamlines working with multiple environments and is well recommended:

$ easy_install pip
$ easy_install virtualenv
$ easy_install virtualenvwrapper
$ mkdir ~/.virtualenvs

That last line creates a working directory for your virtual Python environments. When e.g. working with an environment named myenv, all packages will be installed in ~/.virtualenvs/myenv.

I find it useful to create and activate a default environment called sandbox. This helps prevent package installations to the default Python site-packages. It’s a good strategy in general to avoid polluting the main package directory so that almost all package installations are per project and virtual environment. Run the following commands to create the sandbox environment:

$ export WORKON_HOME=$HOME/.virtualenvs
$ export PIP_VIRTUALENV_BASE=$WORKON_HOME
$ source /usr/bin/virtualenvwrapper_bashrc
$ mkvirtualenv sandbox

mkvirtualenv is a virtualenvwrapper command that creates the given environment. If you get an IOError: [Errno 2] No such file or directory: '/usr/local/bin/python2.5' you will have to add a symbolic link to the Python executable:

$ ln -s /usr/bin/python2.5.exe /usr/bin/python2.5

Note that whenever you execute a shell command, the bash prompt will remind you of the active environment:

$ echo "foo"
foo
(sandbox)

To make the sandbox activation permanent, append the following lines to your ~/.bashrc:

export WORKON_HOME=$HOME/.virtualenvs
export PIP_VIRTUALENV_BASE=$WORKON_HOME
source /usr/bin/virtualenvwrapper_bashrc
workon sandbox

The workon is another virtualenvwrapper extension that switches you to the given environment. To get a full list of available environments, type workon without an argument. Other useful commands are deactivate to step out of the currently active environment, and rmvirtualenv to delete an environment. Refer to the virtualenvwrapper documentation for the whole story.

As a sanity check, try exiting and restarting the Cygwin shell. If you have paid attention so far, you should now automatically end up in the sandbox environment.

Step 5 – Install Fabric: From this point and on, all installed packages, including Fabric, will end up in a virtual environment. Fabric is undergoing a major rewrite right now, so given that its interface is quite unstable it is preferable to have a per-project installation anyway.

First we create a test environment named myproject:

$ mkvirtualenv myproject

We have to make some modifications to the Fabric source code, so we can’t use pip for installing it. Make sure to use version 0.9 or higher, as version 0.1 is already quite outdated:

$ mkdir ~/tmp
$ cd ~/tmp
$ wget http://git.fabfile.org/cgit.cgi/fabric/snapshot/fabric-0.9b1.tar.gz
$ tar xzf fabric-0.9b1.tar.gz
$ cd fabric-0.9b1

Fabric is run using the fab command, but if we try to install it as is, the following error might show up:

$ fab
Traceback (most recent call last):
 File "/home/brox/.virtualenvs/myproject/bin/fab", line 8, in <module>
   load_entry_point('Fabric==0.1.1', 'console_scripts', 'fab')()
 File "/home/brox/.virtualenvs/myproject/lib/python2.5/site-packages/setuptools
-0.6c9-py2.5.egg/pkg_resources.py", line 277, in load_entry_point
 File "/home/brox/.virtualenvs/myproject/lib/python2.5/site-packages/setuptools
-0.6c9-py2.5.egg/pkg_resources.py", line 2180, in load_entry_point
 File "/home/brox/.virtualenvs/myproject/lib/python2.5/site-packages/setuptools
-0.6c9-py2.5.egg/pkg_resources.py", line 1913, in load
 File "/home/brox/.virtualenvs/myproject/lib/python2.5/site-packages/fabric.py"
, line 53, in <module>
   import win32api
ImportError: No module named win32api

At the time of writing there is a small bug in Fabric that is likely to be fixed in the near future. For now you have to manually modify a file in fabric/state.py before you install. Change the line that says

win32 = sys.platform in ['win32', 'cygwin']

to

win32 = sys.platform in ['win32']

This is just to tell Fabric that Cygwin isn’t really Windows and that the win32api module therefore isn’t available. Having made the necessary change, do a regular installation from source:

$ python setup.py install

The following error message about paramiko not being found might pop up; just ignore it:

local packages or download links found for paramiko==1.7.4
error: Could not find suitable distribution for Requirement.parse('paramiko==1.7.4')

And that’s it! You should now have a fully functional virtualenv/Fabric/pip setup. To verify that Fabric works, create a file called fabfile.py:

from fabric.api import local, run

def local_test():
    local('echo "foo"')

def remote_test():
    run('uname -s')

This file, of course, only scratches the surface of what you can do with Fabric—refer to the latest documentation for more information.

To test the fabfile, type the following:

$ fab local_test
[localhost] run: echo "foo"

Done.

The biggest issue is that of getting Fabric to play along with your SSH installation so that you can deploy on remote servers. (You did install the openssh package, right?). Try the following command, substituting test@atbrox.com with one of your own accounts:

$ fab remote_test
No hosts found. Please specify (single) host string for connection: test@atbrox.com
[test@atbrox.com] run: uname -s
Password:
[test@atbrox.com] out: Linux

Done.
Disconnecting from test@atbrox.com... done.

The next step would be to set up password-less logins, but that is a different story.

Afterthoughts: While Cygwin is a lifesaver, it has some quirks and annoyances that may or may not be an issue depending on your system configuration. For instance, on my setup the following error tends to show up randomly when using Fabric for remote deployment:

sem_init: Resource temporarily unavailable
Traceback (most recent call last):
 File "build/bdist.cygwin-1.5.25-i686/egg/fabric/main.py", line 454, in main
 File "/cygdrive/c/Users/brox/workspace/quote_finder/fabfile.py", line 187, in
deploy
   _prepare_host_global()
 File "/cygdrive/c/Users/brox/workspace/quote_finder/fabfile.py", line 137, in
_prepare_host_global
   if not exists(u'/usr/bin/virtualenvwrapper_bashrc'):
 File "build/bdist.cygwin-1.5.25-i686/egg/fabric/contrib/files.py", line 32, in
exists
 File "/usr/lib/python2.5/contextlib.py", line 33, in __exit__
   self.gen.throw(type, value, traceback)
 File "/usr/lib/python2.5/contextlib.py", line 118, in nested
   yield vars
 File "build/bdist.cygwin-1.5.25-i686/egg/fabric/contrib/files.py", line 32, in
exists
 File "build/bdist.cygwin-1.5.25-i686/egg/fabric/network.py", line 371, in host
_prompting_wrapper
 File "build/bdist.cygwin-1.5.25-i686/egg/fabric/operations.py", line 422, in r
un
 File "channel.py", line 297, in recv_exit_status
 File "/usr/lib/python2.5/threading.py", line 368, in wait
   self.__cond.wait(timeout)
 File "/usr/lib/python2.5/threading.py", line 210, in wait
   waiter = _allocate_lock()
thread.error: can't allocate lock

This is a known problem that is not likely to go away anytime soon, due to an inherent race condition in Cygwin’s implementation of sem_init. Still, having a functional virtualenv/Fabric/pip environment on Windows is all in all pretty convenient.

There is a slew of useful articles out there if you need more information on the tools described in this article. These are my current favorites:

• Tools of the Modern Python Hacker: Virtualenv, Fabric and Pip (note that most of the Fabric articles out there use an outdated version of the Fabric API, so have a look at the latest documentation as well.)
• A Primer on virtualenv
• Django Deployment with virtualenv and pip
• Using pip Requirements