Railize - Another blog on Ruby on Rails

Hadoop - An OpenSource System Modelled after Google File System

2010-09-17T18:10:00.032+05:30

Hadoop

We can't start talking about Hadoop immediately unless we know some buzz words that are around for a while... and they are Parallel Processing / computing, Concurrent Computing, Distributed processing / computing, Google File Systems, Map Reduce, BigTable, etc...
If we've to discuss about everything, then one post isn't enough...

So i've decided to write a series of posts which will start from the basics & finally coverup Hadoop / HBase / HyperTable with installation instructions, samples...

I always get stuck when i've a mammoth task... lets break into small... shall we?
and am lazy to write, so here is the quick run thru of the buzz words...(wiki is always there for reference)
Parallel Processing / Computing

Good old math !!!, if one person takes 10 days to do one job, how long it'll take if 2 persons with same caliber are working?

Parallel processing is the simultaneous processing of the same task on two or more microprocessors in order to obtain faster results. The computer resources can include a single computer with multiple processors, or a number of computers connected by a network, or a combination of both. The processors access data through shared memory. Some supercomputer parallel processing systems have hundreds of thousands of microprocessors.

With the help of parallel processing, a number of computations can be performed at once, bringing down the time required to complete a project. Parallel processing is particularly useful in projects that require complex computations, such as weather modeling and digital special effects. Let's take a real-life example to understand the efficacy of parallel processing.

Eventually parallelism is the future of computing (Rather should i say that it has already started). A successful implementation of parallel computing involves two things apart from having a Strong Distributed processing system:

Tasks should be structured in such a manner that they can be executed at the same time

The sequence of tasks which must be executed one after the other should be maintained

Check out Parallel Computing article @ wiki for more information...

Concurrent Computing

Its again not much different than the Parallel processing...

Its a collection of processes which can interact with each other and run in parallel !!!

Biggest problem in writing Concurrent programs is to make sure that the sequence & the interaction between them are happening properly. These Concurrent programs can be executed in a single machine with single core machine, yet, it'll perform the same stuff but it'll take time. Or they can be ran in a machine with multiple core processors or in different machines across the network to get the parallelism.

Many programs or methods or threads (performing a specific task)... think of Concurrent computing as a Kernel core, which performs several things simultaneously & Seamlessly (though it is running in a single machine).

Several other things are playing some vital role like shared memory / Message passing (for communication between concurrent threads / processes / pieces of code)

Surely, the application throughput & high responsiveness are the core needs of opting for Concurrent Computing. I would strongly suggest you to keep the parallel processing / concurrent computing options as the last ones. Try everything you can do better your piece of code to meet your performance expectations. When you think / know that you can't do it with traditional way of doing it... start exploring into parallelism.

Distributed Processing / Computing

Distributed Processing / computing is again the same thing only difference is that, there will be more than one computers connected thru network to achieve a goal.

The program which will run on multiple computers is known as Parallel program !!! (it took so long to some to this state )

Please go thru the wiki page of Distributed Computing to get more information on this..

Google File System (GFS)

Google File System (GFS) is designed to meet the rapidly growing demands of data processing needs at Google

. (without which i dont think it is possible to crawl the internet, index them, do the keyword / content analysis, do the analytics, Searching, and knowing who is searching for what !!!, advertising and even then get the response in (Search for Google File System in Google)

About 57,000,000 results (0.13 seconds) !!!

It is not much different than its predecessors (Distributed file systems) in terms of performance, scalability, reliability and of course data redundancy to achieve the availability.

To get an app processing around 800 to 1000 requests per second, we are breaking out heads on the wall (at-least me)... think about database designing, caching, indexing, memory leaks, architectural constraints, horizontal / vertical scaling, Code optimizations, Server tweaking, phew... How the hell they get this kinda load getting processed and still it works (at-least if we know how to search in google, it gives us what we want :) )

They had (yes they had) thousands of storage systems and connectivity between those systems and what not, to connect and retrive data from those systems. problem happens when one of the machine is down and can't self-recover!!!

Each system should be capable enough to store retrive, process multi-GB to few TB of data! Data was written into files and stored, updating the records, Random writes were few of the problems which led to design GFS. And ofcourse, the system that are going to be written should be scalable enough interms of processing, I/O, etc.

Enough of Story, lets have a look at the architecture.

Here is how they do it... Google File System, Big Table, Map Reducer

Hope you already have idea about Master - Slave, In memory data structures, Meta-data, data replication, GC (Garbage Collection), balancing and fault tolerance.

GFS has a Single Master and several chunk servers which will be accessed by several clients. Generally these machines / servers are inexpensive commodity hardwares... based on the capacity of the system, the chunk server & the client can run on the same system.

Extract From : gfs-sosp2003.pdf

Master will take care of assigning an

immutable and globally unique 64 bit chunk handle at the time of chunk creation. Chunk servers

store chunks on local disks as Linux files and read or write chunk data specified by a chunk handle and byte range. For reliability, each chunk is replicated on multi- ple chunk servers. By default, GFS stores three replicas, though users can designate different replication levels for different regions of the file namespace.

The master will maintain all file system metadata. This includes the namespace, access control information, the mapping from files to chunks, and the current locations of chunks. It also controls system-wide activities such as chunk lease management, garbage collection of orphaned chunks, and chunk migration between chunkservers. The master periodically communicates with each chunk server in HeartBeat messages to give it instructions and collect its state.

GFS client code linked into each application implements the file system API and communicates with the master and chunkservers to read or write data on behalf of the application. Clients interact with the master for metadata opera- tions, but all data-bearing communication goes directly to the chunk servers.

Neither the client nor the chunkserver caches file data. Client caches offer little benefit because most applications stream through huge files or have working sets too large to be cached. Not having them simplifies the client and the overall system by eliminating cache coherence issues. (Clients do cache metadata, however.) Chunkservers need not cache file data because chunks are stored as local files and so Linux’s buffer cache already keeps frequently accessed data in memory.

Image Credit: gfs-sosp2003.pdf

Some of the challenges of GFS is to have Concurrent Append to the files, Reading the large set of streaming data, random reads and processing huge data / storing huge data in sequential manner !!

In Memory Data structure was required to achieve faster garbage collection and ofcourse the load balancing. Master will take care of scanning the state of each chunk server periodically and take care of Garbage Collection and Load balancing.

And literally, these smaller chunks reduces the amount of metadata required to be stored.

Activities of a Master includes Namespace management & locking, Replica management & replacements, rebalancing, Garbage collection, stale replica deduction and management

Interesting enough, GFS promises a few things...

Fast recovery - but how? using checkpoints.
Chunk replications - to achieve high availability
Master replications for reliability

Enough about GFS now... but wait, do you need more info... please go thru the gfs-sosp2003.pdf

Big Table

BigTable is a distributed storage system for managing structured data and it is meant to scale to a very large size (call it zeta bytes of data across several thousands of commodity servers). It is a DBMS system which didnot follow the principles of a traditional DBMS concepts, it has millions of columns (it can have different number of columns for different records).

BigTable uses the GFS distributed File System to store the data files. BigTable is designed in keeping mind that even the hundreds of thousands of servers might not be sufficient and it should be easy to scale horizontally by just adding more servers when required.

Many of the Google products (as per bigtable-osdi06.pdf document more than sixty products & projects) including Analytics, Indexing, Earth are using BigTable. Having said that, these products require different sized data, latency, high performance, etc. And BigTable offers them all (after all it is meant for that)

BigTable is a sparse, distributed, persistent multi-dimensional sorted map. It has a row-key, column-key and a timestamp.

Trust me, it is very hard to think in terms of Column - oriented database after used to the traditional DBMS systems. Need to understand the Row Key, Column Families, TimeStamps and flush out your DBMS knowledge and start designing the DBMS for the column-oriented Database.

(row:string, column:string, time:int64) → string

Figure 3. Image Credit: bigtable-osdi06.pdf

Rows in a BigTable are uniquely identified by the row key. BigTable maintains data in lexicographic order by the row key. The Row range is dynamically partitioned. Each Range is called a tablet (used for the load balancing). Tables are split into multiple tablets - segments of the table. Each tablet will be around 200MB approximately. When the size grows beyond the specified limit, compression takes place.

There are multiple level hierarchy analogous to that of a B+ tree to store the Tablet information itself !!!

First level is known as the Chubby which contains the location of the root tablet. Root tablet contains the location of all tablets in a special METADATA tablet. and in-turn METADATA tablet contains the location of a set of user tables. (though the Root Tablet is the first tablet in the METADATA tablets, it is treated specially)

Column keys are grouped in to sets known as column families and it forms the basic unit of access control. A column family has to be created before the data is stored in to the family.

Column key is named like ColumnFamily:Qualifier.

Each column key in a anchor family (as shown in the Figure 3) represents a single anchor. Qualifier is the name of the referring site. Cell contents is the link text.

TimeStamps are used to versioning the data in a table in BigTable. Each cell in BigTable can contain multiple versions of the data which is distinguished by the timestamp.

BigTable also provides an API to create / delete / lookup for values in the BigTable.

For more information on how BigTable is implemented or how it works refer bigtable-osdi06.pdf

Map Reduce

MapReduce is a programming model for processing and generating large datasets in an distributed processing environment.

The underlying architecture like GFS / Hadoop are capable enough to make these kind of programs (MapReduce) run in a parallel mode on a large number of computers (known as clusters). GFS / Hadoop takes care of the splitting the input data and creating as many mappers and reducer steps required (configuring the number mappers & reducers is also possible), schedules the program executions, handle machine failures, inter-machine communication. Terms Map & Reduce came from the primitives present in Lisp (and other functional languages).

Input for the MapReduce programs are a set of key/value pairs and it produces a set of output key/value pairs.

Map: a piece of program written by the user, which takes an input pair and produces intermediate key/value pair output. The Library takes care of splitting the huge data file into smaller chunks and sending it to different machines and receiving the intermediate outputs, combining them with other intermediate outputs of the same Key. Master node will take care of sending the data to worker nodes (until all the data is processed).

Reduce: another piece of program written by the user, which takes the intermediate key and a set of values for that key. Merges them together to form a smaller set of values. Aggregating all these output values (combined ones) will form the final output of the problem for the given data.

Advantage of the MapReduce is that it allows distributed processing of mappers and reducers (ofcourse unless there is no dependency between the mappers, it can run in parallel). MapReduce library (provided by GFS / Hadoop) will take care of the situation when a processing node (map or reduce) fails, It automatically reschedules the map / reduce task)

A sample pseudo-code for map-reducer.

Lets count the number of occurrences of each word in a large collection of documents.
Map function emits each word and a number (which is count).
Reducer simply receives the word and adds the count it received from the Map to the total count for each word
In case of Map output, key is the word, count is the value.

For Reducer, it simply gets the key, value and keep adding the value to associated to the key.
map (k1, v1) -> list (k2, v2)
reduce(k2, list(v2)) -> list (v2)

The figure above shows the execution flow of a typical map reduce program (in terms of GFS)

Step1: MR Library splits the huge data file into multiple chunks of 16 to 64 MB of data. and starts as many copies of programs as required on the cluster
Step2: One of the Copy acts as the Master. Rest are workers. Master has the information about which machine is idle and which is getting processed.
Step3: Get the input content which is allocated to the worker.
Step4: store the output into the local disk
Step5: Master notifies the reducer about the output of map is available
Step6: Reducer iterates over the sorted intermediate data and produces a output which is appended to the final output file.
Step7: Master wakesup the user program when all the map/reduce programs are completed.

The Master node:

Master node keeps track of each map & reducers, tasks assigned to them, Intermediate outputs, state of these tasks assigned to each map/reduce instances (forks) and the status of the worker machines.

The good thing about these MR libraries is that, even if the Master task dies, it has created several checkpoints for a given job... next time, it wont start a-fresh, instead, start from the last checkpoint.

phew... We've seen enough to get the grounds up for the Hadoop. But we are gonna wait for sometime to get the 2nd article of Hadoop Series.

Until then, if you need more information... do checkout the following links or post a comment.

http://en.wikipedia.org/wiki/Distributed_computing
http://en.wikipedia.org/wiki/GoogleFS
http://en.wikipedia.org/wiki/Concurrent_computing
http://en.wikipedia.org/wiki/Parallel_computing
http://en.wikipedia.org/wiki/Hadoop
http://en.wikipedia.org/wiki/MapReduce
http://en.wikipedia.org/wiki/BigTable
http://en.wikipedia.org/wiki/HBase
http://en.wikipedia.org/wiki/Google_File_System
http://labs.google.com/papers/gfs.html
http://storagemojo.com/google-file-system-eval-part-i/
http://storagemojo.com/google-file-system-eval-part-ii/
http://www.baselinemag.com/c/a/Infrastructure/How-Google-Works-1/
http://queue.acm.org/detail.cfm?id=1594206

Agile & Scrum

2010-08-12T12:04:00.011+05:30

What is Agile?

Agile is a project management methodology built around the idea that projects should be completed in smaller chunks to achieve greater results. Agile projects are projects that focus on self-organizing and iterative techniques to achieve success. Agile projects build problem solving into the project management process.

Agile software development refers to a group of software development methodologies based on iterative development, where requirements and solutions evolve through collaboration between self-organizing cross-functional teams.

Agile methods generally promote a disciplined project management process that encourages frequent inspection and adaptation, a leadership philosophy that encourages teamwork, self-organization and accountability, a set of engineering best practices intended to allow for rapid delivery of high-quality software, and a business approach that aligns development with customer needs and company goals.

Some of the principles behind the Agile Manifesto are:

Customer satisfaction by rapid, continuous delivery of useful software
Working software is delivered frequently (weeks rather than months)
Working software is the principal measure of progress
Even late changes in requirements are welcomed
Close, daily cooperation between business people and developers
Face-to-face conversation is the best form of communication (co-location)
Projects are built around motivated individuals, who should be trusted
Continuous attention to technical excellence and good design
Simplicity
Self-organizing teams
Regular adaptation to changing circumstances

A Typical Agile (Iterative) Development Process will look like....

In Agile methodology, the development team, working closely with the stakeholders / Customers / Clients / Product Owners, crafts an application solution to some set of identified and prioritized functions. Most of the rigor can be derived in the deployment aspect of the process, using continuous testing and integration of the application components, as shown in Left.

Moving along the spectrum, a unified process methodology breaks up the development effort into specific disciplines (design, code, test, and so on), as shown in simplified form in right. Input from and feedback to stakeholders is more stylized and formal, as well as less immediate and continuous. This type of method is applicable when the requirements are well understood, with little or no reusable design and implementation assets available.

Benefits of Agile software development

Software that works!
Collaborative approach ensuring customer satisfaction
Ability to make changes throughout the project
Shorter and frequent software deliveries
Better communication between client and the development team
‘Sustainable’ development practices that allow developers to work at a sustainable rate, rather than attempting to achieve possibly unrealistic budgets or deadlines.
Robust software using the highest quality code, design and architecture.
Rapidly evolving teams that helps them stay agile.
Self-directing teams that work together as a whole.
Streamlined software that is flexible, changeable and maintainable.

Summary of Agile
The following are key points with agile development:

Agile methodology is adaptive rather than predictive
Agile methodology is people-oriented rather than process-oriented.
Agile development is adaptive, visible, quick, coded and tested all the time.

Scrum Methodology !!

Let’s start with a big question: What is it? Scrum is a framework, a value-system, and a process — and explaining it from each of those perspectives may help different audiences.

Scrum is a framework

Scrum is an iterative, incremental framework for project management and a specialized version of agile project management methodologies. Scrum projects are characterized by a product backlog - a list of tasks that come up while the sprint (a highly focused period where tasks are completed, generally running 30 days) is underway. The idea behind Scrum is to create a streamlined project management process that produces a quality end product.

Scrum is a framework for managing projects or, more generally, work. It is iterative and incremental, which means that it asks a team to work for a short period of time (a “sprint” or “iteration”) and then demonstrate real stuff (a product increment) that matters to the end-product at the end of each sprint. Goals for each subsequent sprint are evaluated, negotiated, and committed to based on the stuff that was created at the end of the previous sprint. There’s still an infinite amount of work to be done, but Scrum divides that work into manageable chunks (usually about two-weeks in length).

Scrum is a Value-System

Scrum is also a value system that asks teams to work together to accomplish a common goal, focusing on the output of the team rather than the input of the individuals. It values communication, openness, transparency, self-organization, and the worth of employees as individuals and professionals. Scrum can and does help people feel good about their jobs and their contributions to their teams. It is also in tune with reality, assessing progress and evolving business conditions to dictate a project’s direction.

Scrum is a Process

Scrum is also a process that invites the application of those values by asking that teams generally organize themselves into three roles, participate in four regular meetings, and produce and maintain three artifacts. This simple process can be scaled to any size software team (or non-software team, for that matter).

What Are the Basics of Scrum Methodology?

Scrum Methodology depends on the execution of highly defined roles and processes. Scrum begins with the Scrum Content and the Scrum Master who leads and initiates the scrum project.

Scrum content includes the following items:

Three specific roles - the Scrum Master (or leader of the project), the Product Owner (the person responsible for maximization of the work's value – basically a Client) and the Team (those who carry out the Scrum work and develop and test the product).
Time Boxes - Time boxes include the release planning meeting (where qualities of the software to be developed are discussed and agreed upon), the Sprint development meeting (where the work to be completed during the sprint is discussed and agreed upon), the Sprint (the aforementioned focused work time period), the daily scrum (a brief meeting held at the beginning of each day for the purpose of progress checking), the sprint review (a brief meeting held at the end of each day to recap the work and progress made that day) and the sprint retrospective (occurring at the end of a sprint, no more than three hours, to discuss the successes and failures of the sprint).
Artifacts - There are four artifacts involved in a Scrum project. The Scrum artifacts are: the product backlog (a to-do list that is constantly being prioritized when new tasks are found and added to it), the sprint backlog (the to-do list for the duration of one sprint, during the sprint no new objects are added to this list and the goal during the sprint is to turn this sprint into a workable product), Release burn-down (takes the product backlog and projects its completion over time during the project duration), Sprint burn-down (measures the sprint backlog items and tracks them across the remaining duration of a sprint).
Rules - Various rules govern sprint projects. Rules are what tie together the Sprint roles with the sprint time boxes and artifacts. Rules help to ensure that the project's Scrum methodology is implemented correctly.

Scrum Time Boxes

Release Planning Meeting - The release planning meeting, where the plans for the product are decided upon and goals are set. Scrum Alliance calls the release planning meeting an optional time box, meaning if Scrum Masters are secure in the plans and goals for a product release, they can skip this initial meeting.

Sprint Development Meeting - the sprint development meeting is slotted to eight hours for an average sprint lasting one month. During the sprint development meeting, held only once a sprint iteration has been scheduled, the top priority items from the product backlog are presented to the team. Also during this meeting, the methods for completing these items are discussed.

The Sprint - The sprint is a highly focused period of time where the team works towards creating a workable version of the product being completed. While a sprint is underway, no changes can be made that will affect the outcome or the product being produced in the sprint.

The Daily Scrum - The daily scrum is a fifteen minute meeting that allows each sprint team member to present what each has been working on, what will be completed, and any problems encountered during the process.

The Sprint Review - The sprint review is held at the end of a sprint. It is time blocked for 4 hours and serves as a forum to discuss what work was completed and what work had not been completed.

The Sprint Retrospective - The sprint retrospective is held after a sprint review, and before the next sprint planning session. The purpose of this meeting is to review how the previous sprint went in terms of the mechanics (people, tools, organization, etc.).

The Scrum time boxes formulate the foundation for Scrum methodology.

What are Scrum Processes?

Scrum processes develop the project from the planning phases, through the sprint, to the next sprint, until the project is completed. Processes take the time boxes, the Scrum roles, and work with the artifacts to complete a scrum project and produce a quality product.

Scrum Artifacts

Before you can understand the Scrum process, it is important to first understand Scrum artifacts. Scrum artifacts are lists and schedules created in order to facilitate the completion of work during a Sprint.

Product Backlog - The product backlog is a prioritized to do list. Anytime a task comes up during the duration of the project, the task is added to the product backlog and the list is re-prioritized. The only items on the product backlog that are detailed are the items that are of the highest priorities. This is done in order to ensure that extra and unnecessary work is not undertaken.

Release burn-down - The release burn-down maps out in graphic form the relationship of the items on the product backlog to the time left in the Scrum project. During release planning, the initial release burn-down chart is created, thereafter, it is updated as the product backlog changes.

Sprint Backlog - During the Sprint development session, the items that will appear on the sprint backlog for completion are determined. The sprint backlog does not change during the Sprint. Instead, team members complete the items on the backlog during the sprint duration - if additional items come up, then those items are placed on the product backlog for a future sprint. If the sprint becomes impossible to complete unless one of these new issues is completed, then the sprint is canceled and the team goes back to the sprint development meeting stage, generating a new sprint backlog in the process.

Sprint burn-down - The sprint burn-down, like the release burn-down, is a scheduled chart, but this chart demonstrates the relationship between items on the sprint backlog and the time left in the sprint. The sprint burn-down demonstrates all the work left in the sprint.

Applying Scrum Processes to a Project

Because Scrum involves rigid rules for its implementation, the process for applying Scrum to a project is fairly straight forward. Imagine you are creating a video game program for your employer. You will have a product release planning session to discuss the requirements for the game. Once the requirements have been determined, then you will begin to assemble the product backlog. The first sprint development meeting will be conducted, during which the initial sprint backlog will be derived from the product backlog. A sprint burn down chart is created. The sprint begins and your coworkers and team members begin completing the work for the sprint. Fifteen to Thirty days (depending on the Sprint length) go by, many issues and tasks have been identified, and you have a sprint review meeting to discuss what has been completed for the video game and what needs to be completed in the next sprint. This cycle will repeat until the product reaches completion and achieves its first release.

A Scrum Environment - What's That?

Scrum methodologies include artifacts, time boxes, roles and rules, and the Scrum processes involve the utilization of these various methods in a project. The Scrum environment is the workplace situation in which scrum is being implemented. Scrum environments are committed to the utilization of Scrum Masters, Product Owners, and team members. The Scrum environment is one where this particular type of agile project management occurs seamlessly.

Kinds of Scrum Environments

Scrum environments may vary in type and in size. There are multi-project environments where Scrum Masters have to juggle multiple teams, product backlogs, sprint backlogs, and more. There are also fixed-date environments where the product being created is dependent upon a particular deadline. Scrum can be used in a variety of ways, and it doesn't have to be only used in software development. Scrum can be a valuable tool for projects in academia, publishing and more. While Scrum is meant for teams, the idea behind scrums - planning what will be done in a specific time period and then carrying out that plan without adding anything new to the product back log - could be a great way to help projects with stakeholders who often change scope demands.

Benefits and Disadvantages of Scrum Environments

Despite its downfalls, Scrum can be a boon to a company's project management process. Scrum is fast-paced and produces results in a short amount of time - leading to saved money on the part of the company. Because the requirements for scrum are so rigid, and because scrum methodologies feature the daily scrum meeting, productivity and product quality are increased. Finally, operating in a Scrum environment is beneficial to companies because issues are identified before they become major problems that deteriorate a product's viability.

What Does "Scrum" Stand For?

This is one of the most common questions asked about Scrum. The truth is, however, that "Scrum" doesn't stand for anything, it is not an acronym. Instead, are you familiar with Rugby at all? If not, you won't recognize the term. "Scrum" refers to a move in Rugby where a team packs together and they all act together to get the ball from one end of the field to another end of the field. In fact, if you are using the word "Scrum" properly, only the first letter of the word should be capitalized, the rest is in lower case.

What if I Have an Issue That Needs To Be Solved...Now?

Finally, if during the course of a Sprint, you come across an issue that needs to be dealt with immediately, and that issue completely destroys any ability to make further progress on the Sprint backlog, then the Sprint needs to be canceled. Canceling a Sprint should only be done as a last-ditch effort, and should only be done if progress on the product is completely untenable.

Why Scrum?

Scrum methodology includes dedicated people or teams, who are pigs or chickens, and the Scrum Master. While it sounds strange, if you look at the idea behind Scrum, they all make sense. According to Wikipedia, the roles of chickens and pigs was based on an old joke:

"A pig and a chicken are walking down a road. The chicken looks at the pig and says, "Hey why don't we open a restaurant?" The pig looks back at the chicken and says, "Good idea, what do you want to call it? The chicken thinks about it and says, "Why don't we call it 'Ham and Eggs'?" "I don't think so," says the pig, "I'd be committed but you'd only be involved."

So, the pigs build or manage a project using Scrum, while the chickens have an interest but aren't responsible for the project whether it succeeds or fails. Chickens do have a role; however. They provide important feedback to the project. A project that will produce a key chain with a client's logo, will use pigs, the development team, and chickens, the vendors who will be responsible for selling that key chain. Chicken roles are designed to help the pigs stay on track.=

Defining Scrum Roles

Each scrum role has a specific job in the project in order for it to succeed. A scrum role list might look like this:

Pig Roles:

Client - The client will be the owner of the finished product. It is their idea the Scrum Master and team will use to develop the product. The client's role in Scrum is to ensure the project's team is utilizing the correct or best methods for the project.
Scrum Master - The Scrum Master oversees the project, but doesn't manage the team. Instead, the Scrum Master is a sort of cushion between the client and the team. The Scrum Master is responsible for setting the rules of the project, but not the delivery of the project.
Team - The team is made up of team members with the appropriate skill sets to complete the project and perform the actual work. They are responsible for delivering the product and each team member intertwines throughout the duration of the project toward its goal.

Chicken Roles:

Users - The users are considered the people or places that will utilize the projects outcome or product. In our key chain example, users would actually "use" the product.
Stakeholders - The stakeholders, or vendors who sell the key chain have an interest in its development but only offer overall feedback at project meetings.
Managers - The managers develop how the key chain will be offered, advertised, and the environment they will use to facilitate its offering.

Working Together

If the people and teams involved in the pig roles don't or can't deliver, they must begin again or lose the project. Those involved in chicken roles, who offer feedback, suggestions, and provide an arena for the product's outcome don't lose if the project fails. They await a new pig outcome in the Scrum process.

For more information on how to identify Scrum roles throughout projects, read, Scrum, An Agile Management Process. This guide offers useful tips for project managers on how Scrum works and ensures everyone involved knows their specific role throughout the Scrum process.

The Scrum Team

In a Scrum methodology, the most visible members of the Scrum team are:

Scrum Master
Product Owner
Team

You may have noticed in one of the articles on receiving certification as a Scrum Master that there are other roles involved as well in the Scrum environment. These roles also play into Scrum environments, but will only be treated briefly.

The Scrum Master

If you're like many, you may think the term "Scrum Master" conjures a picture of a person with a wand. Well, the Scrum Master isn't too off from this visualization. The Scrum Master coaches the Scrum team in its endeavors and ensures that the Scrum methodology is properly implemented during the project. The Scrum Master also instructs the team in productivity principals and organizes the Scrum environment. Finally, the Scrum Master helps to remove any obstacles a team may face during a Sprint. In this way, the Scrum Master is both a leader and a follower with regards to a Scrum project environment.

The Product Owner

While the Scrum Master ultimately serves as a facilitator for the Scrum project environment, the Product Owner is the sole person responsible for maintaining the Product Backlog and checking on the quality of the team's work. Because of the Product Owner's role, everyone on the team knows what is expected of them, what items have been given the highest priority, and what should be worked on during the Sprint. The Product Owner is also responsible for ensuring that the only work being done is the work that's been listed on the Sprint Backlog. The Product owner is also the only person who has the ability to add items to the Product Backlog.

The Team

Each Scrum Team member is a developer that is responsible for turning action items on the Product Backlog into functional pieces of a shippable product. Teams are made up of people with specialties - quality control, programing, business analysis, etc. And any given team is only made up of 5-9 people. This way each person can communicate with every other person - and the team size is a manageable one so that Daily Scrum meetings are kept to the requisite 15 minutes or less. A Scrum Team is self organizing - meaning that they determine how to proceed with the items on the Sprint backlog.

What is the Product Backlog?

While by now, you should know that the product backlog is a place where all to-do items are prioritized in waiting for transfer to the Sprint Backlog, you may not know what components need to be represented on a Product Backlog you are using in your Scrum project. Here is a list of components that should be found on your Product Backlog:

An identification number for each item, or "Story" on the list.
A Priority of each story on the list
Stories - Sprint stories are the descriptions of the work that must be completed during the Sprint. For example, a story may be as simple as "Finish creating database."
An estimated number of hours for each story's completion.
Who the story has been assigned to.
The number of the Sprint that the story has been assigned to

What About These Project Stories?

Project stories, defined by the product owner, will follow Bill Wake's INVEST model where "INVEST" stands for:

Independent
Negotiable
Valuable
Estimable
Small
Testable

Each of the stories in a project should follow the above six criteria before they are put onto the Product backlog. Stories should be independent, meaning that they should never overlap with another story and stories should be negotiable, meaning that the story when created will be a collaboration. Stories also need to be valuable to the product customer (not just to those creating the software). You should be able to estimate how long the story will take to implement and how large the story will be. Stories will be small - each story should only involve a couple of weeks work for one person. Finally, and perhaps most important, the story should be testable. If not, then there could be a huge problem in implementing the story during the Sprint.

Creating the Task Plan from the Product Backlog

In addition to stories needing to follow the INVEST model, tasks should always follow the SMART model, meaning that each task is:

Specific
Measurable
Achievable
Relevant
Time-Boxed

By creating your task plan following the SMART model, you will ensure that your team members will not be lost in a sea of busywork. Instead, they will be working on tasks and stories that lead to results and profit for your product and the company.

Creating the Product Backlog

In an Excel Worksheet, or in a Scrum project management tool such as Scrum Desk, You will want to track your Product Backlog. If you are using Excel or another spreadsheet program, from right to left, list the following headings:

Priority
Estimate
Sprint
User Type
Story
Story Type

You can then break your Excel worksheet into sections with "Very High Priority,"High Priority," "Medium-High Priority" and "Medium Priority." Finally, remember, if an item has a low priority, you may not want to include it on your list.

What hasn't been covered is a guide to some of the best practices involved in utilizing the Scrum project management methodology. Best practices are the most effective means by which to execute some task. Below you will find a list of Scrum best practices.

1. Holding Effective Daily Scrum Meetings

In order to have a successful Scrum project, you will need to hold effective daily scrum meetings. Effective daily scrum meetings exhibit the following features:

Members stand rather than sit - this helps to keep the meeting short.
There are five goals of the daily scrum: to recommit to the project as a team, to communicate status of the project, to identify any obstacles the team has come across, to set the day's direction, and to help build the cohesiveness of the team.
All project stakeholders should attend the daily scrum (possibly).
Keep all summaries short - each member should focus only on the following three items: what was done yesterday, what will be done today, what issues may cause problems for progress.
The meeting should be held in the same place and at the same time every day.
The meeting should be fifteen minutes or less.

By adhering to the above practices during your daily scrum meeting, you can increase the efficiency of your team and improve the quality of your project by avoiding tedious and time-wasting meetings.

2. The Sprint and Product Backlogs

It is important to keep your Sprint Backlog and your Product Backlogs separate, organized, and prioritized. Remember that the Sprint Backlog contains the work you will be working on during the Sprint and nothing else. If other issues come up during a Sprint, then they should be documented on the Product Backlog. Here are some tips for dealing effectively with your Sprint and Product

Backlogs:

If you're not already using Excel, look into using it to create your Backlog. This way you can prioritize and re-prioritize your tasks quickly and easily.
For that matter, you may wish to use one of the many project management software options to manage your backlogs.
Make sure you do not confuse your Sprint Backlog with your Product Backlog. If you wind up putting items on your Sprint Backlog that should be on your Product Backlog, you will wind up putting out fires.
Re-prioritize your Product Backlog each time you add to it. It will save you trouble in the long-run.
Do not add more to your Sprint Backlog than can be reasonably completed during your Sprint. You will only frustrate yourself and your team.
Give each item on your backlog an ID number - this way you can keep track of tasks and stories no matter what you call them.
Don't forget to include an estimate of how long it will take to complete the item in your backlog
Your Sprint Backlog should center around one, unifying, goal.
Make sure to indicate in your Sprint backlog whether someone is currently working on the item (and who it is) and the status of that item.
Be as specific as possible with each part of your Sprint Backlog story.

By maintaining your Backlogs, you can save time and effort - and you can help maximize productivity by working on only the items that matter to your project.

3. Sprints & Sprint Planning Meetings

Sprints are the focused time period where team members work on the items from the Sprint Backlog. Most sprints run about 15 to 30 days. A successful sprint depends upon a successful sprint planning meeting. Here are some tips on sprint planning and sprint best practices:

Make sure you come up with a clear goal for your sprint during the planning session. This will help focus your sprint efforts.
Come up with a list of highly committed team members for the sprint.
Don't overlook the importance of keeping a Sprint burn-down chart for your Sprint backlog. A sprint burn-down, if you recall from the article on Scrum methodology, represents in graphic form what is left to do of the sprint backlog work.
Keep the sprint time to fifteen to thirty days, the average for a sprint. Any longer and you are trying to accomplish too much. Any shorter and you may not have time to complete a demo.
Only begin a sprint planning session once your product backlog has enough organization and detail. If you start a sprint planning session when you lack adequate detail, then your project may suffer from scope creep.
Once you select the sprint backlog activities, realize that you must commit to them completely.
The sprint planning meeting should take between four to eight hours depending upon the length of the sprint.

4. Teams

Finally, Scrum teams can make or break your Scrum project. You should choose a team, not only for each member's previous successes, but also for the ability of the members to cooperate and communicate with one another. Here are some more best Scrum team practices:

Most Scrum teams consist of between six and ten people. However, with careful scaling, your Scrum project may consist of one hundred or more people.
Spend some time on team-building strategies. By dedicating time to creating a team that works well together, you can help to ensure that your Scrum project will succeed.
Teams in Scrum must be able to self-organize. If a team contains one or two procrastinators, it may be hard for the team to move forward on projects.
Facilitate communication among team members by utilizing handy project collaboration software. This ensures that everyone in the team will be on the same page, know what's going on, and know what needs to be worked on next. It also allows for the testers to have open access to new lines of code in software projects.
Most teams will realize that what seemed like "not that much work" is actually a lot of work once a Sprint starts. Be sure that your team members do not assign themselves more than can be reasonably completed with accuracy during a Sprint.
If a team member is absent, the team's work could be seriously hindered, especially if that individual was working on something that has other task dependencies. If a team member knows she will be absent ahead of time, this is the ideal situation since the team members can pick up the slack. If the team member does not know, then this could create a problem. One way to handle absences is to plan six hour days. Another is to only plan for a certain percentage of the team to show up. Whatever you do, make sure you take this contingency into account when planning.

A Quick Scrum Tutorial

This quick tutorial will introduce you and your company to utilizing the Scrum project management process.

Overview of Scrum

Scrum is perhaps one of the simpler project management methods to learn and implement. In this tutorial, I will show you how to speak the language of Scrum and utilize this valuable management tool for your company

Basic Scrum Vocabulary

Before being able to implement Scrum, it is important to be familiar with some key words in the the Scrum vocabulary.

Sprint- a 30-day focused effort moving the team toward fixed goals.

Product Backlog- a constantly prioritized to-do list.

Sprint Backlog- a list of the highest prioritized items from the product backlog.

Scrum Master- the coach for the product management team and works to ensure the realization of the goals of the sprint.

Product Owner- represents the customer and is responsible for prioritizing the backlog.

Scrum Team- a group of 5-9 people who self-organize and have joint responsibility for the completed tasks.

The Scrum Process

Now that basic vocabulary related to Scrum has been defined I will detail the Scrum process step-by-step.

Creating a backlog – in this step, the Product Owner and Scrum Team meet in order to discuss the priority and items on the Product Backlog. The Product Owner must be able to form the product vision. The Product Backlog then, is a prioritized list of what is required for the project and is ranked with regard to importance.
Once the product manager creates the backlog, then there will be a Sprint Planning Meeting. During the first phase of the meeting, the Product Manager describes to the team the goals of the project and explains the Product Backlog. During the second phase, the Scrum Team will select the items to be completed during the sprint from those with highest priority on the Product Backlog.
Once the items to be worked on have been selected, a potential Sprint schedule is constructed – taking into account the availability of the team members to devote their time to the project. The items in the Product Backlog are assigned and broken down into individual tasks. Once this occurs, this document is the Sprint Backlog.
The Sprint begins and lasts from 15-30 days. During the Sprint, no other tasks are added to the backlog.
Daily Scrum begins when the sprint begins. The Daily Scrum is a 15 minute stand-up meeting where each member of the team gives a very brief report to everyone else – what they accomplished since the last Daily Scrum, what they hope to accomplish, and issues that have come up. Here, the Scrum Master will make note of issues and attempt to resolve them – after the meeting.
Sprint Review – once the Sprint ends, everyone gets together in a meeting to share what he or she accomplished during the sprint.
The process begins again with a new list of prioritized tasks on the Product Backlog.

While this is a very brief overview of the Scrum process, there is enough information here in order to begin implementing Scrum in your own company. You can find a valuable resource for some of the more intricate areas of Scrum at Scrum Alliance. ScrumDesk is valuable software available online to assist you in utilizing the Scrum process. It is available free for up to five users, and for a per month fee based on the size of the groups larger than five.

Implications Of OpenID (Google Tech Talk)

2010-08-02T12:02:00.001+05:30

Presentation on OpenID by Simon Willison

Implications Of OpenID (Google Tech Talk)

View more presentations from Simon Willison.

NatSent - National Sentiment

2010-07-25T03:46:00.006+05:30

About NatSent

NatSent envision a more democratic world, a world where democracy is even more accessible and its benefits continually apparent.

NatSent stands for "the National Sentiment". NatSent is an innovative web platform that leverages the efficiencies of the Internet to empower the voices of individuals. Through NatSent, we hope to enable individuals to identify, follow, frame, and influence the issues that they care about. Our goal is to decrease the transaction costs between voters and politicians, consumers and businesses, citizens and thought leaders. NatSent wants individuals to say and ultimately get some form of what they want without having to go through a lot of red tape.

NatSent is designed to be like a personalized, digital newspaper where users are both the editors and the writers. What makes the front page depends on the users. How an issue is framed also depends on the users. And when NatSent doesn't cater to a specific user's needs, he can proactively make a difference by posting, responding, and voting on NatSent.

NatSent was founded by two, Generation Y, former roommates at Rice University in 2010. Houston, Texas is hot as hell in the summer but it's home.

Rails Server Deployment - Ruby, Rails, ImageMagick, MySQL, Nginx Installation and configuration

2010-07-25T03:11:00.002+05:30

Ruby Rails Server Deployment Instructions

This isn't totally new, Its there everywhere. Just thought of keeping it a compiled note.
(this is scraped from my notes)
I've used a Ubuntu 9.10 Server to install...

Install Ruby 1.9.1, Rails 2.3.5, ImageMagick, MySql 5.1
----------------------------------------------------
Install Gcc :
apt-get install gcc

Install Zlib (optional) :
wget http://zlib.net/zlib-1.2.5.tar.gz
tar zxvf zlib-1.2.5.tar.gz
cd zlib-1.2.5
./configure
make
sudo make install

Install Openssl(optional)
wget http://mirrors.isc.org/pub/openssl/source/openssl-0.9.8g.tar.gz
tar zxvf openssl-0.9.8g.tar.gz
cd openssl-0.9.8g
./config
make
make install

Ruby 1.9.1
apt-get install libopenssl-ruby1.9.1 libssl-dev

wget ftp://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.1-p376.tar.gz
tar zxvf ruby-1.9.1-p376.tar.gz
cd ruby-1.9.1-p376

For Including Openssl and Zlib
------------------------------
vi ruby-1.9.1-p376/ext/Setup
#openssl (uncomment it)
#zlib (uncomment it)

Installation
------------
./configure
make
sudo make install

Verify Ruby Installation
----------------------
whereis ruby

ruby -v
gem -v

Install ImageMagick
apt-get install imagemagick

NOTE: If any problem occurs with the above step... then run this one also.
apt-get install graphicsmagick-imagemagick-compat

Verify ImageMagick Installation
-------------------------------
whereis identify
whereis convert
convert logo: logo.gif

Install RMagick

apt-get install libmagick9-dev
sudo gem install rmagick

Install Rails and other required gems
gem update --system
gem install rails -v=2.3.5

sudo gem install mysql --source http://gems.rubyinstaller.org

Install MySQL Server 5.1
sudo apt-get install mysql-server-5.1
sudo apt-get install libmysqlclient16-dev

Verify MySQL Installation
mysql --version
mysql Ver 14.14 Distrib 5.1.37, for debian-linux-gnu (x86_64) using EditLine wrapper
whereis mysql
ps -ef | grep mysql
mysql -uroot -p

Create User in MySQL and set Privileges
CREATE USER 'monty'@'localhost' IDENTIFIED BY 'some_pass';
GRANT ALL PRIVILEGES ON . TO 'monty'@'localhost' WITH GRANT OPTION;

Memcached, Nginx and Mongrel Cluster

http://articles.slicehost.com/2009/3/11/ubuntu-intrepid-nginx-rails-and-mongrels

Install nginx
sudo apt-get install apache2-utils
sudo aptitude install nginx
sudo /etc/init.d/nginx start

Verify nginx Installation

Install Mongrel and Mongrel Cluster
sudo gem install mongrel -v 1.2.0.pre2 --pre --source http://rubygems.org
sudo gem install gem_plugin
sudo gem install fastthread cgi_multipart_eof_fix daemons

sudo gem install mongrel_cluster

mongrel_rails cluster::configure -e production -p 8000 -N 2 -c /home/demo/public_html/testapp -a 127.0.0.1
mongrel_rails cluster::start
mongrel_rails cluster::restart
mongrel_rails cluster::stop

Scrape proper content from wikipedia

2010-07-25T02:43:00.000+05:30

Scrape the 1st paragragh & Image from a Wikipedia entry

Sometime we would need a automated way of getting the description or possibly the image too to be available automatically for the keywords that we use in our projects...

One good place to look for this content is Wikipedia. But...

When you search for Company Apple in Wiki... what we get is probably not the correct one.

And we might not be in a position or we cant expect our users to type the full information about the keyword like 'Apple Inc'.

The solution is to use google search combined with Wiki.

Here is the code for getting the description & Image from wiki (Hope there is a wikipage for the keyword we search for.... unless and until it is really crazyyyyyyyyy)

require 'hpricot'

require 'open-uri'

def fetch_description(query_item)

page_title, uri_title = get_wiki_name(query_item)

return get_wiki_description(page_title, uri_title)

end

def upload_photo(wiki_photo)

begin

base_uri = URI.parse(wiki_photo)

uploaded_data = open(base_uri)

def uploaded_data.original_filename; base_uri.path.split('/').last; end

return uploaded_data.original_filename.blank? ? nil : uploaded_data

rescue

return nil

end

#Method to fetch wiki page and strip first two
Tags

def get_wiki_description(page_title, uri_title)

url = uri_title

final_content = ""

if url.size > 10

buffer = Hpricot(open(url, "UserAgent" => "reader"+rand(10000).to_s).read)

#Capture first two paragraphs of text

content = buffer.search("//div[@id='content']").search("//div[@id='bodyContent']").search("//p")[0..2]

#Remove the extra spaces and strip html tags fromt the fetched content

content.each do |c|

final_content+=c.inner_html.gsub(/<\/?[^>]*>/, '').gsub(/&#\d+;/,'').gsub(/\([^\)]+\)/,'').gsub(/\[[^\]]+\]/,'').gsub(/ +/,' ')+"\n"

end

return final_content

end

#Method to get the link for wikipedia from google search results

def get_wiki_name(query_item)

search_keywords = query_item.strip.gsub(/\s+/,'+')

url = "http://www.google.com/search?q=#{search_keywords}+site%3Aen.wikipedia.org"

begin

doc = Hpricot(open(url, "UserAgent" => "reader"+rand(10000).to_s).read)

result = doc.search("//div[@id='ires']").search("//li[@class='g']").first.search("//a").first unless doc

rescue

return '',''

end

if result

return result.inner_html.gsub(/<\/?[^>]*>/,"").gsub(/./,""),result.attributes["href"]

else

return '',''

end

wiki_description, wiki_photo = fetch_description("Apple")

upload_photo(wiki_photo)

Note: After all of this done... Please make sure to give credits to Wikipedia :)

Railize - Another blog on Ruby on Rails

Hadoop - An OpenSource System Modelled after Google File System

Agile & Scrum

What is Agile?

Benefits of Agile software development

Summary of Agile The following are key points with agile development:

Scrum Methodology !!

Scrum is a framework

Scrum is a Value-System

Scrum is a Process

What Are the Basics of Scrum Methodology?

Scrum Time Boxes

What are Scrum Processes?

Scrum Artifacts

Before you can understand the Scrum process, it is important to first understand Scrum artifacts. Scrum artifacts are lists and schedules created in order to facilitate the completion of work during a Sprint.

Applying Scrum Processes to a Project

A Scrum Environment - What's That?

Benefits and Disadvantages of Scrum Environments

What Does "Scrum" Stand For?

What if I Have an Issue That Needs To Be Solved...Now?

Why Scrum?

Defining Scrum Roles

Working Together

The Scrum Team

The Scrum Master

The Product Owner

The Team

What is the Product Backlog?

What About These Project Stories?

Creating the Task Plan from the Product Backlog

Creating the Product Backlog

1. Holding Effective Daily Scrum Meetings

2. The Sprint and Product Backlogs

3. Sprints & Sprint Planning Meetings

4. Teams

Overview of Scrum

Basic Scrum Vocabulary

The Scrum Process

Implications Of OpenID (Google Tech Talk)

NatSent - National Sentiment

About NatSent

Rails Server Deployment - Ruby, Rails, ImageMagick, MySQL, Nginx Installation and configuration

Scrape proper content from wikipedia

Scrape the 1st paragragh & Image from a Wikipedia entry

Summary of Agile
The following are key points with agile development: