Developer Techno

Different Jobs Available for People Who Love Microsoft

2012-05-26T14:20:00.000+08:00

Are you a Microsoft fan? Do you want to combine your passion for Microsoft products with computer-related skills in a fulfilling career? If so, you are in luck because there are many career avenues for Microsoft lovers. With the proper training and savvy networking, you can secure one of the many different jobs available for people who love Microsoft.

Education and Training

In pursuing a Microsoft-related career, it is important that you obtain an adequate level of education and training. The best way to accomplish is with a formal Microsoft training or certificate program. You can choose among many specialized programs for whatever type Microsoft job that you want to pursue. For more information on Microsoft-related education and training, check out some online Microsoft Certifications.

Microsoft Jobs

There are a wide range of jobs for people who love Microsoft. These include:

Artist/Graphic Designer – Accounting for several factors, such as color, form, and light, artists and graphic designers concentrate on the visual and aesthetic aspects of Microsoft games, applications, and displays.
Software Development Engineers – Software engineers create codes and computing languages that helping transform computing ideas into practical Microsoft features and applications. Collaborating with other Microsoft professionals such as test engineers and program managers, software developers help create quality Microsoft products at the cutting edge of technology.
Test Engineers – Test engineers test newly developed Microsoft products. Pushing the Microsoft products to the limit, test engineers help ensure that they are of a high standard of quality and are also capable of meeting customer's expectations.
Program Managers – Program managers oversee and work with multiple Microsoft professionals in ensuring that computing ideas gets carried out and result in a quality Microsoft products. Program managers are involved in all phases of product creation, from understanding and anticipating the needs of the customer, brainstorming ideas, creating and testing the product, and developing a marketing plan.
Audio Designer – Audio designers produce the voice aspects, sound, audio effects, and music for Microsoft games and applications.
Content Publishers – Content publishers perform several tasks such as the conception, design, development, production, editing, and publishing of material for Web audiences, Microsoft games and applications, customer support, and Microsoft hard-copy literature.
Game Designers – Game designers create and design Microsoft games, while considering such factors as user experience, aesthetics, usability, software compatibility, level of gaming difficulty, audio and visual landscape, emotional elements, and how real or "lifelike" the game appears to the game user.
User Experience – Microsoft professionals in the user experience specialization take existing or newly developed Microsoft products and develop and facilitate interactions with computing users. By getting feedback from users, professional in the user experience specialization can help make changes for Microsoft products before they hit the market or fine-tune existing Microsoft products.

Additional Microsoft Jobs – Additional jobs for lovers of Microsoft products include a build engineer, international product and localization engineer, and product planner. Also, there are many electronics stores and computer outlet chains in which an individual can sell, provide customer service, or repair Microsoft products.

Defining a private cloud

2012-05-24T01:09:00.000+08:00

There’s confusion — as well as passionate debate — over the definition of a private cloud. When we say private cloud, we mean a highly virtualized cloud data center located inside your company’s firewall. It may also be a private space dedicated to your company within a cloud vendor data center designed to handle your company’s workloads.

The characteristics of the private cloud are as follows:
✓ Allows IT to provision services and compute capability to internal users in a self-service manner
✓ Automates management tasks and lets you bill business units for the services they consume
✓ Provides a well-managed environment
✓ Optimizes the use of computing resources such as servers
✓ Supports specific workloads
✓ Provides self-service based provisioning of hardware and software resources

You might think this sounds a lot like a public cloud! A private cloud exhibits the key characteristics of a public cloud, including elasticity, scalability, and self-service provisioning. (Please refer to Chapter 1 for detailed information on cloud characteristics.) The major difference is control over the environment. In a private cloud, you (or a trusted partner) control the service management.

It might help to think of the public cloud as the Internet and the private cloud as the intranet.

If private and public clouds are so similar, why would you develop a private cloud instead of ordering capacity on demand from an Infrastructure as a Service provider or using Software as a Service? Here are several good reasons companies are using a private rather than a public cloud:

✓ Your organization has a huge, well-run data center with a lot of spare capacity. It would be more expensive to use a public cloud even if you have to add new software to transform that data center into a cloud.

✓ Your organization offers IT services to a large ecosystem of partners as part of your core business. Therefore, a private cloud could be a revenue source.

✓ Your company’s data is its lifeblood. You feel that to keep control you must keep your information behind your own firewall.

✓ You need to keep your data center running in accordance with rules of governance and compliance.

✓ You have critical performance requirements, meaning you need 99.9999 percent availability. Therefore, a private cloud may be your only option. This higher level of service is more expensive, but is a business requirement.

Some early adopters of private cloud technology have experienced server use rates of up to 90 percent. This is a real breakthrough, particularly in challenging economic times.

Source of Information : cloud computing for dummies 2010 retail ebook distribution

Talking to Your Cloud Vendor about Data

2012-05-21T01:05:00.000+08:00

You’re thinking about using some of the data services in the cloud. Before you sign the contract, remember that data (especially your company’s data) is a precious asset and you need to treat it as such.

In addition to issues surrounding security and privacy of your data that we cover earlier in the chapter, we recommend asking your potential vendor about the following topics:

✓ Data integrity: What controls do you have to ensure the integrity of my data? For example, are there controls to make sure that all data input to any system or application is complete, accurate, and reasonable? What about any processing controls to make sure that data processing is accurate? And, there also need to be output controls in place to ensure that any output from any system, application, or process can be verified and trusted. This dovetails with the next bullet about any specific compliance issues that your particular industry might have.

✓ Compliance: You are probably aware of any compliance issues particular to your industry. Obviously, you need to make sure that your provider can comply with these regulations.

✓ Loss of data: What provisions are in the contract if the provider does something to your data (loses it because of improper backup and recovery procedures, for instance)? If the contract says that your monthly fee is simply waived, you need to ask some more questions.

✓ Business continuity plans: What happens if your cloud vendor’s data center goes down? What business continuity plans does your provider have in place: How long will it take the provider to get your data back up and running? For example, a SaaS vendor might tell you that they back up data every day, but it might take several days to get the backup onto systems in another facility. Does this meet your business imperatives?

✓ Uptime: Your provider might tell you that you will be able to access your data 99.999 percent of the time — however, read the contract. Does this uptime include scheduled maintenance?

✓ Data storage costs: Pay-as-you-go and no-capital-purchase options sound great, but read the fine print. For example, how much will it cost to move your data into the cloud? What about other hidden integration costs? How much will it cost to store your data? You should do your own calculations so you’re not caught off guard. Find out how the provider charges for data storage. Some providers offer a tiered pricing structure. Others offer pricing based on server capacity.

✓ Contract termination: How will data be returned if the contract is terminated? If you’re using a SaaS provider and it has created data for you too, will any of that get turned over to you? You need to ask yourself if this is an issue. Some companies just want the data destroyed. Understand how your provider would destroy your data to make sure that it isn’t floating around in the cloud.

✓ Data ownership: Who owns your data after it goes into the cloud? Some service providers might want to take your data, merge it with other data, and do some analysis.

✓ Switching vendors: If you create applications with one cloud vendor and then decide to move to another vendor, how difficult will it be to move your data? In other words, how interoperable are the services? Some of these vendors may have proprietary APIs and it might be costly to switch. You need to know this before you enter into an agreement.

Source of Information : cloud computing for dummies 2010 retail ebook distribution

Databases and data stores in the cloud

2012-05-18T01:01:00.000+08:00

Given the scale of some of these applications, it isn’t surprising that new database technologies are being developed to support this kind of computing.

Some database experts believe that relational database models may have difficulty processing data across large numbers of servers — in other words, when the data is distributed across multiple machines. Performance can be slow when you’re executing complex queries that involve a join across a distributed environment. Additionally, in an old-style database cluster, data must either be replicated across the boxes in the cluster or partitioned between them. According to other database experts, this makes it hard to provision servers on demand.

In response, some large cloud providers have developed their own databases. Here’s a sample listing:

✓ Google Bigtable: This hybrid is sort of like one big table. Because tables can be large, they’re split at row boundaries into tablets, which might be 100 megabytes or so. MapReduce is often used for generating and modifying data stored in Bigtable. Bigtable is also the data storage vehicle behind Google’s App Engine (a platform for developing applications).

✓ Amazon SimpleDB: This Web service is for indexing and querying data. It’s used with two other Amazon products to store, process, and query data sets in the cloud. Amazon likens the database to a spreadsheet in that it has columns and rows with attributes and items stored in each. Unlike a spreadsheet, however, each cell can have multiple values and each item can have its own set of associated attributes. Amazon then automatically indexes the data.

✓ Cloud-based SQL: Microsoft has introduced a cloud-based SQL relational database called SQL Database (SDS). SDS provides data storage by using a relational model in the cloud and access to that data from cloud and client applications. It runs on the Microsoft Azure services platform. The Azure platform is an Internet-scale cloud-services platform hosted in Microsoft data centers; the platform provides an operating system and a set of developer services.

Numerous open-source databases are also being developed:
✓ MongoDB (schema-free, document-oriented data store written in C++)
✓ CouchDB (Apache open-source database)
✓ LucidDB (Java/C++ open-source data warehouse)

Source of Information : cloud computing for dummies 2010 retail ebook distribution

Large-scale data processing in Cloud

2012-05-15T00:58:00.000+08:00

The lure of cloud computing is its elasticity: You can add as much capacity as you need to process and analyze your data. The data might be processed on clusters of computers. This means that the analysis is occurring across machines.

Companies are considering this approach to help them manage their supply chains and inventory control. Or, consider the case of a company processing product data, from across the country, to determine when to change a price or introduce a promotion. This data might come from the point-of-sale (POS) systems across multiple stores in multiple states. POS systems generate a lot of data, and the company might need to add computing capacity to meet demand.

This model is large-scale, distributed computing and a number of frameworks are emerging to support this model, including

✓ MapReduce, a software framework introduced by Google to support distributed computing on large sets of data. It is designed to take advantage of cloud resources. This computing is done across large numbers of computers, called clusters. Each cluster is referred to as a node. MapReduce can deal with both structured and unstructured data. Users specify a map function that processes a key/value pair to generate a set of intermediate pairs and a reduction function that merges these pairs.

✓ Apache Hadoop, an open-source distributed computing platform written in Java and inspired by MapReduce. It creates a computer pool, each with a Hadoop file system. It then uses a hash algorithm to cluster data elements that are similar. Hadoop can create a map function of organized key/value pairs that can be output to a table, to memory, or to a temporary file to be analyzed. Three copies of the data exist so that nothing gets lost.

Source of Information : cloud computing for dummies 2010 retail ebook distribution

COMPARING NOSQL PRODUCTS - Access and Interface Availability

2012-05-12T01:51:00.000+08:00

MongoDB has the notion of drivers. Drivers for most mainstream libraries are available for interfacing and interacting with MongoDB. CouchDB uses web-standard ways of interaction and so you can connect to it using any programming language that supports the web idiom of communication. Wrappers for some languages make communication to CouchDB work like drivers for MongoDB, though CouchDB always has the RESTful HTTP interface available.

Redis, Membase, Riak, HBase, Hypertable, Cassandra, and Voldemort have support for language bindings to connect from most mainstream languages. Many of these wrappers use languageindependent services layers like Thrift or serialization mechanisms like Avro under the hood. So it becomes important to understand the performance characteristics of the various serialization formats.

One good benchmark that provides insight into the performance characteristics of serialization formats on the JVM is the jvm-serializers project at https://github.com/eishay/jvmserializers/wiki/. The performance measures via the efforts of this project relate to a number of data formats. The formats covered are as follows:

protobuf 2.3.0 — Google data interchange format. http://code.google.com/p/protobuf/

thrift 0.4.0 — Open sourced by Facebook. Commonly used by a few NoSQL products, especially HBase, Hypertable, and Cassandra. http://incubator.apache.org/thrift/

avro 1.3.2 — An Apache project. Replacing Thrift in some NoSQL products. http://avro.apache.org/

kryo 1.03 — Object graph serialization framework for Java. http://code.google.com/p/kryo/

hessian 4.0.3 — Binary web services protocol. http://hessian.caucho.com/

sbinary 0.3.1-SNAPSHOT — Describing binary format for scala types. https://github.com/harrah/sbinary

google-gson 1.6 — Library to convert Java objects to JSON. http://code.google.com/p/google-gson/

jackson 1.7.1 — Java JSON-processor. http://jackson.codehaus.org/

javolution 5.5.1 — Java for real-time and embedded systems. http://javolution.org/

protostuff 1.0.0.M7 — Serialization that leverages protobuf. http://code.google.com/p/protostuff/

woodstox 4.0.7 — High-performance XML processor. http://woodstox.codehaus.org/

aalto 0.9.5 — Aalto XML processor. www.cowtowncoder.com/hatchery/aalto/index.html

fast-infoset 1.2.6 — Open-source implementation of Fast infoset for binary XML. http://fi.java.net/

xstream 1.3.1 — Library to serialize XML and back. http://xstream.codehaus.org/

The performance runs are on a JVM but the results may be as relevant to other platforms as well. The results show that protobuf, protostuff, kryo, and the manual process are among the most efficient for serialization and de-serialization. Kyro and Avro are among the formats that are most efficient in terms of serialized size and compressed size.

Having gained a view into the performance of formats, the next section segues into benchmarks of NoSQL products themselves.

Source of Information : NoSQL

COMPARING NOSQL PRODUCTS - Querying Support

2012-05-09T01:47:00.000+08:00

Storage is one part of the puzzle. The other is querying the stored data. Easily and effectively querying data is almost mandatory for any database to be considered seriously. It can be especially important when building the operational data store for applications with which people are interacting. An RDBMS thrives on SQL support, which makes accessing and querying data easy. Standardized syntax and semantics make it an attractive choice. The fi rst chapter in this book talks about the quest for a SQL-like query language in the world of NoSQL.

Among document databases, MongoDB provides the best querying capabilities. Best is a relative term, and developers argue about what they consider superior to alternatives, but I base my judgment on three factors: similarity to SQL, an easy syntax, and an easy learning curve. CouchDB’s querying capabilities are equally powerful and rather more straightforward once you understand the concepts of views and design documents. However, the concept of views as CouchDB defines it is new and can pose initial challenges to developers.

For key/value pairs and in-memory stores, nothing is more feature-rich than Redis as far as querying capabilities go. Redis has one of the most exhaustive sets of methods available for querying the data structures it stores. To add icing to the cake, it is all nicely documented. Read about the access methods at http://redis.io/commands.

Column-family stores like HBase have little to offer as far as rich querying capabilities go. However, an associated project called Hive makes it possible to query HBase using SQL-like syntax and semantics. Hypertable defines a query language called HQL and also supports Hive.

Bringing Hive into the mix raises the question of manipulating data for operational usage versus accessing it for batch processing and business intelligence. Hive is not an interactive tool in a way SQL is to RDBMS. Hive resembles SQL in syntax but is really a way to abstract MapReduce-style manipulations. Hive allows you to use SQL like predicate-driven syntax instead of map and reduce function definitions to carry batch data manipulation operations on the data set.

Source of Information : NoSQL

COMPARING NOSQL PRODUCTS - Data Modeling

2012-05-05T01:44:00.000+08:00

RDBMS offers a consistent way of modeling data. Relational algebra underlies the data model. The theory is well established and implementation is standardized. Therefore, consistent ways of modeling and normalizing data is well understood and documented. In the NoSQL world there is no such standardized and well-defined data model. This is because all NoSQL products do not intend to solve the same problem or have the same architecture. If you need an RDBMS-centric data model for storage and querying and cannot under any circumstances step outside those definitions, just don’t use NoSQL. If, however, you are happy with SQL-type querying but can accommodate non-relational storage models, you have a few NoSQL options to choose from.

Document databases, like MongoDB, provide a gradual adoption path from formal RDBMS models to lose document-centric models. MongoDB supports SQL-like querying, rudimentary relational references, and database objects that draw a lot of inspiration from the standard table- and column based model. If relaxed schema is your primary reason for using NoSQL, then MongoDB is a great option for getting started with NoSQL.

MongoDB is used by many web-centric businesses. Foursquare is perhaps its most celebrated user. Shutterfl y, bit.ly, etsy, and sourceforge are a few other users that add feathers to MongoDB’s cap. In many of these use cases MongoDB is preferred because it supports a flexible data model and offers speedy reads and writes. Web applications often evolve rapidly and it often gets cumbersome for developers to continuously change underlying RDBMS models, especially when the changes are frequent and at times drastic. Added to the schema change challenges are the issues relating to data migration. MongoDB has good support for web framework integration. Rails, one of the most popular web application frameworks, can be used effectively with MongoDB. The data from Rails applications can be persisted via an object mapper. Therefore, MongoDB can easily be used in place of an RDBMS. Read about Rails 3 integration at www.mongodb.org/display/DOCS/Rails+3+-+Getting+Started.

For Java web developers, Spring offers fi rst-class support for MongoDB via its Spring Data project. Read more about the Spring Data Document release that supports MongoDB at www.springsource.org/node/3032. Spring Data project, in fact, adds support for a number of NoSQL products, and not just MongoDB. It integrates Spring with Redis, Riak, CouchDB, Neo4j, and Hadoop. Get more details online at the Spring Data project homepage, which is www.springsource.org/spring-data. MongoDB acts like a persistent cache, where data is kept in memory and flushed to disk as required. Therefore, MongoDB could also be thought of as an intermediate option between an RDBMS and an in-memory store or a flat file structure. Many web applications like real-time analytics, comments system, ratings storage, content management software, user data system, and event logging applications benefit from the fluid schema that MongoDB offers. Added to that, such applications enjoy MongoDB’s RDBMS-like querying capabilities and its ability to segregate data into collections that resemble tables.

Apache CouchDB is a document database alternative to MongoDB. Apache CouchDB is now available as Couchbase server, with the primary creators of CouchDB having recently merged their company, CouchOne, with Membase, Inc. Couchbase offers a packaged version of Apache CouchDB with GeoCouch and support in the form of Couchbase Server.

Couchbase Server epitomizes adherence to web standards. Couchbase’s primary interface to the data store is through RESTful HTTP interactions and is more web-centric than any database has ever been. Couchbase includes a web server as a part of the data store. It is built on top of Erlang OTP. This means you could effectively create an entire application using Couchbase. Future versions of Couchbase will be adding access to the data store through the Memcached protocol, gaining from Membase’s ability to manage speed and throughput with a working set. Couchbase also plans to scale up, growing from Membase’s elastic capabilities to seamlessly span across more nodes. Although Couchbase is very powerful and feature-rich, it has a very small footprint. Its nimble nature makes it appropriate for installation on a smartphone or an embedded device. Read more about mobile Couchbase at www.couchbase.com/products-and-services/mobile couchbase.

Couchbase models support REST-style data management. A database in CouchDB can contain JSON format documents, with additional metadata or supporting artifacts as attachments. All operations on data — create, retrieve, update, and delete — are performed via RESTful HTTP requests. Long-running complex queries across replicated Couchbase servers leverage MapReduce.

Not Just a Map
In typical in-memory databases and caches, the most well-known data structure is a map or a hash. A map stores key/value pairs and allows for fast and easy access to data. In-memory NoSQL stores provide filesystem-backed persistence of in-memory data. This means that stored data survives a system reboot. Many NoSQL in-memory databases support data structures beyond just maps, making using them for more than simple cache data extremely attractive. At the most basic level, Berkeley DB stores pairs of binary key/value pairs. The underlying store itself does not attach any metadata to the stored key/value pairs. Layers on top of basic storage, like the persistence API or the object wrappers, allow persistence of higher-level abstractions to a Berkeley DB store.

Membase, on the other hand, supports the Memcached protocol, both text and binary, and adds features around distributed replica management and consistent hashing on top of the basic key/value store. Membase also adds the ability to grow and shrink the number of servers as part of a cluster without interrupting client access. Redis takes a slightly different approach. It supports most popular data structures out of the box. In fact, it is defined as a “data structure” server. Redis supports lists, sets, sorted sets, and strings in addition to maps. Redis has even added transactionlike capabilities to specify atomicity across a number of discrete operations.

If your use case gains from using a file-backed in-memory NoSQL product, consider the supported data models to make a choice on the best fi t. In many cases, a key/value storage is enough, but if you need more than that look at Berkeley DB, Membase, and Redis. If you need a powerful and stable distributed key/value store to support large user and activity load, you are not likely to go wrong with Membase.

What about HBase and Hypertable?
In the previous section on scalability, I gave my entire vote in favor of the column-family stores. When it comes to supporting the rich data models, though, these are generally not the most favorable choices. The upfront choice of row-keys for lookup and only column-family-centric model metadata support is usually considered inadequate. With a powerful abstraction layer on top of column-family stores, a lot becomes possible.

Google started the column-family store revolution. Google also created the data modeling
abstraction on top of its column-family store for its very popular app engine. The GAE data modeling support provides rich data modeling using Python and Java. With the DataNucleus JDO and JPA support, you can use the popular object modeling abstractions in Java to persist data to HBase and Hypertable. You can also draw inspiration from the non-relational support in Django that works well on the app engine.

Source of Information : NoSQL

COMPARING NOSQL PRODUCTS - Transactional Integrity and Consistency

2012-05-01T01:38:00.000+08:00

Transactional integrity is relevant only when data is modified, updated, created, and deleted. Therefore, the question of transactional integrity is not pertinent in pure data warehousing and mining contexts. This means that batch-centric Hadoop-based analytics on warehoused data is also not subject to transactional requirements.

Many data sets like web traffic log files, social networking status updates (including tweets or buzz), advertisement click-through imprints, road-traffic data, stock market tick data, and game scores are primarily, if not completely, written once and read multiple times. Data sets that are written once and read multiple times have limited or no transactional requirements.

Some data sets are updated and deleted, but often these modifications are limited to a single item and not a range within the data set. Sometimes, updates are frequent and involve a range operation. If range operations are common and integrity of updates is required, an RDBMS is the best choice. If atomicity at an individual item level is sufficient, then column-family databases, document databases, and a few distributed key/value stores can guarantee that. If a system needs transactional integrity but could accommodate a window of inconsistency, eventual consistency is a possibility. HBase and Hypertable offer row-level atomic updates and consistent state with the help of Paxos. MongoDB offers document-level atomic updates. All NoSQL databases that follow a master-slave replication model implicitly support transactional integrity.

Source of Information : NoSQL

COMPARING NOSQL PRODUCTS - Scalability

2012-04-28T01:34:00.000+08:00

Although all NoSQL databases promise horizontal scalability they don’t rise up equally to the challenge. The Bigtable clones — HBase and Hypertable — stand in front and in-memory stores, like Membase or Redis, and document databases, like MongoDB and Couchbase Server, lag behind. This difference is amplified as the data size becomes very large, especially if it grows over a few petabytes.

Bigtable and its clones promote the storage of large individual data points and large collections of data. The Bigtable model supports a large number of columns and an immensely large number of rows. The data can be sparse where many columns have no value. The Bigtable model, of course, does not waste space and simply doesn’t store cells that have no value.

Google led the column-family-centric data store revolution to store the large and ever growing web index its crawlers brought home. The Web has been growing in unbounded ways for the past several years. Google needed a store to grow with the expanding index. Therefore, Bigtable and its clones were built to scale out, limited only by the hardware available to spin off newer nodes in the cluster. Over the past few years, Google has successfully used the Bigtable model to store and retrieve a variety of data that is also very large in volume.

The HBase wiki lists a number of users on its Powered By page (http://wiki.apache.org/hadoop/Hbase/PoweredBy). Some users listed clearly testify to HBase’s capability to scale. Meetup (www.meetup.com) is a popular site that facilitates user groups and interest groups to organize local events and meetings. Meetup has grown from a small, unknown site in 2001 to 8 million members in 100 countries, 65,000+ organizers, 80,000+ meetup groups, and 50,000 meetups each week (http://online.wsj.com/article/SB10001424052748704170404575624733792905708.html). Meetup is an HBase user. All group activity is directly written to HBase and is indexed per member. A member’s custom feed is directly served from HBase.

Facebook is another big user of HBase. Facebook messaging is built on HBase. Facebook was the number one destination site on the Internet in 2010. It has grown to more than 500 million active users (www.facebook.com/press/info.php?statistics) and is the largest software application in terms of the number of users. Facebook messaging is a robust infrastructure that integrates chat, SMS, and e-mail. Hundreds of billions of messages are sent every month through this messaging infrastructure. The engineering team at Facebook shared a few notes on using HBase for their messaging infrastructure. Read the notes online at www.facebook.com/notes/facebook-engineering/the-underlyingtechnology-of-messages/454991608919.

HBase has some inherent advantages when it comes to scaling systems. HBase supports auto load balancing, failover, compression, and multiple shards per server. HBase works well with the Hadoop distributed fi lesystem (a.k.a. HDFS, which is a massively scalable distributed fi lesystem). You know from earlier chapters that HDFS replicates and automatically re-balances to easily accommodate large files that span multiple servers. Facebook chose HBase to leverage many of these features. HBase is a necessity for handling the number of messages and users they serve. The Facebook engineering notes also mention that the messages in their infrastructure are short, volatile, and temporal and are rarely accessed later. HBase, and in general Bigtable clones, are particularly suitable when ad-hoc querying of data is not important. You know that HBase supports the querying of data sets but is a weak replacement to an RBDMS as far as its querying capabilities are concerned. Infrastructures like Google App Engine (GAE) successfully expose a data modeling API, with advanced querying capabilities, on top of the Bigtable.

So it seems clear that column-family-centric NoSQL databases are a good choice if extreme scalability is a requirement. However, such databases may not be the best choice for all types of systems, especially those that involve real-time transaction processing. An RDBMS often makes a better choice than any NoSQL flavor if transactional integrity is very important. Eventually consistent NoSQL options, like Cassandra or Riak, may be workable if weaker consistency is acceptable. Amazon has demonstrated that massively scalable e-commerce operations may be a use case for eventually consistent data stores, but examples beyond Amazon where such models apply well are hard to find. Databases like Cassandra follow the Amazon Dynamo paradigm and support eventual consistency. Cassandra promises incredibly fast read and write speeds. Cassandra also supports Bigtable-like column-family-centric data modeling. Amazon Dynamo also inspired Riak. Riak supports a document store abstraction in addition to being an eventually consistent store. Both Cassandra and Riak scale well in horizontal clusters but if scalability is of paramount importance, my vote goes in favor of HBase or Hypertable over the eventually consistent stores. Perhaps places where eventually consistent stores fare better than sorted ordered column-family stores is where write throughput and latency is important. Therefore, if both horizontal scalability and high write throughput are required, possibly consider Cassandra or Riak. Even in these cases, consider a hybrid approach where you can logically partition the data write process from the access and analytics and use two separate databases for each of the tasks.

If scalability implies large data becoming available at an incredibly fast pace, for example stock market tick data or advertisement click tracking data, then column-family stores alone may not provide a complete solution. It’s prudent to store the massively growing data in these stores and manipulate them using MapReduce operations for batch querying and data mining, but you may need something more nimble for fast writes and real-time manipulation. Nothing is faster than manipulating the data in memory and so leveraging NoSQL options that keep data in memory and flush it to disk when it fi lls the available capacity are probably good choices. Both MongoDB and Redis follow this strategy. Currently, MongoDB uses mmap and Redis implements a custom mapping from memory to disk. However, both MongoDB and Redis, have actively been re-engineering their memory mapping feature and things will continue to evolve. Using MongoDB or Redis with HBase or Hypertable makes a good choice for a system that needs fast real-time data manipulation and a store for extensive data mining. Memcached and Membase can be used in place of MongoDB or Redis. Memcached and Membase act as a layer of fast and efficient cache, and therefore supplement well on top of column-family stores. Membase has been used effectively with Hadoop-based systems for such
use cases. With the merger of Membase and CouchDB, a well integrated NoSQL product with both fast cache-centric features and distributed scalable storage-centric features is likely to emerge.

Although scalability is very important if your data requirements grow to the size of Google’s or Facebook’s, not all applications become that large. Scalable systems are probably relevant for cases much smaller than these widespread systems but sometimes an attempt to make things scalable can become an exercise in over-engineering. You certainly want to avoid unnecessary complexity.

In many systems, data integrity and transactional consistency are more important than any other requirements. Is NoSQL an option for such systems?

Source of Information : NoSQL

Correlating SEO/Paid Search and Site Search

2012-04-25T00:44:00.000+08:00

SEO, paid search, and site search each offer a unique opportunity to understand the mind-set and the interconnected thoughts of the people who are looking for your content. The one downside to this data is that it is drawn from people who have already found you, not from people who have not found you yet. That is, in some way or another, these people have had at least one interaction with your brand.

SEO provides insights into the relevancy of your content to a specific set of terms. There really shouldn’t be many surprises when looking at what words drive people to your website if you are in a clear and defined field or if your site has a clear and defined focus. If your website is about cars, you should expect to see a great deal of car-related terms driving traffic to the site. In this case, what you are looking for is not so much themes as language and use of words, and pairings between search terms. Looking at two- or three-word search terms can give you a better idea of associations people make when looking for your type of content. This can give you insight into language use, and you may discover geographical variations as well. If your site is an exception to the rule of one thematic element—for example, if your business is involved in many disciplines or your site hosts blogs that cover a variety of topics—there may not be a simple theme that you can expect to see driving traffic to your site. In this case, you will need to look at theme clustering, or thematic grouping. You can figure out how to do this through your analytics program.

Paid search provides a testing ground for terms whose relevancy to your market or segment you are uncertain of. Perhaps they are terms that you think are thematically relevant, but you don’t have any traffic to validate this. You can use paid search to test terms and thematic elements and to validate the quality and type of traffic you get. You can also get a sampling of interaction data based on the words and terms that drive people to your site. Looking at this, you will be able to formulate more informed strategies for both your SEO and site search campaigns.

Site search allows you to see what people are looking for on your site. They know your brand, but do they have their own terms for describing your products? Looking at the terms people search for is a good way of determining whether your product naming conventions are in sync with how your customers think about your products. Site search
is a rich source of data to gather and analyze in order to further improve your SEO and paid search campaigns. Conversely, SEO can give you some ideas of where your Site Search is lacking, based on empty site search results. The three search sources can all provide feedback to each other, in both directions.

Tools you will need in this chapter:
• Clickstream tracking package (Google Analytics, Adobe SiteCatalyst, etc.)
• Spreadsheet program (Excel or something similar)
• AdWords or some other paid search tool
• A/B or multivariate testing tool (Google Website Optimizer, Test & Target, etc.)

Source of Information : MASTERING SEARCH ANALYTICS MEASURING SEO, SEM AND SITE SEARCH

The Problem of Only Capturing ROI

2012-04-20T00:35:00.000+08:00

ROI is a key measurement and the first thing you should look at, but how do you measure ease of use? How do you know if your online experience is a positive one for most of your customers? ROI won’t tell you these things. If you plan on growing ROI, you need to understand the factors that will help you move the ROI needle. ROI is a dollar value; further analytics are needed to understand how to optimize your ROI values. These typically come in the form of measuring the user experience. Further, not all sites derive their revenue directly from sales. Facebook, for example, generates revenue through advertising, as does Google, and both have worked to create a great user experience, striving to maximize this user experience and increase profit. There is a reason why Google is the most-used engine while Overture is a thing of the past. It’s why Facebook now rules, while MySpace is circling the drain. Connecting with users, and building experiences and communities, has helped these sites grow bigger than any competitor around them.

The goal of your analytics should be to improve the overall experience for your customers. A happy customer is a good customer. In some cases, a happy customer may simply be one who finds support quickly. There isn’t direct ROI in this case; instead, you’re looking at cost savings or customer satisfaction, both of which are part of the lifetime value of a customer. If you can show that improving self-support services on your website reduces costs, you have an ROI figure you can use. And even if it doesn’t reduce costs on the support side, it may improve long-term customer loyalty. In these cases, you may need to estimate what the long-term value of a customer is. You may not be able to get an exact ROI, but you will be able to estimate the value of improving your customer experience.

Beyond ROI, you should also be looking to use your website to foster communication and dialog with your customers. People who find your site may have come with a specific intent and found that you don’t offer what they are looking for. Running voiceof-the-customer surveys on your site can help you get feedback on what you can do to improve the site experience. Dialog may not have any direct ROI attached to it in this case, but you will get an idea of what people want, and that can drive improved ROI. You may get some ideas you can use to improve your customers’ experience. You may also find that what your customers want is not what you think they need.

ROI is a value and it helps you to know if you will be profitable or not, but it can’t tell you if customers are satisfied with the service you are providing. Ideally, customers will provide feedback to help you improve the overall experience.

As we look at ROI, keep in mind that to improve it you will need to look at many other metrics and analytics, related not only to increasing traffic to your site but also to improving the user experience once people have arrived. You need to start thinking beyond delivering users, and start to think about outcomes for users. Your job is to look at this data to figure out how to maximize profit. I see this being achieved by delivering more of the right people to the right web pages, improving what people currently do when they get to your pages, and making sure your site is as findable as possible by as many people as possible.

Source of Information : MASTERING SEARCH ANALYTICS MEASURING SEO, SEM AND SITE SEARCH

Presenting ROI Data

2012-04-17T00:31:00.000+08:00

Businesspeople understand dollars much better than they will ever understand inbound links, HTML tags, duplicate content issues, or any number of other issues you may feel are important. If you are lucky, you will be supported by people who trust you. Often, you will need to make a business case. When presenting your ROI data, you must make clear whether you are using direct or estimated values. Setting expectations clearly builds trust. Further, once you have an ROI value, you can begin to do some really neat stuff with this data.

The best search strategists focus on conversion. Having identified your key revenue points, you have also discovered what your top-line KPIs should be. These are the points you will want to optimize and improve upon with the traffic your search programs bring in.

When we talk about actionable analytics, optimizing against data points is a big one. Successful search marketers will think like salespeople and try to move people through to success events (e.g., sales). If you identify three or four different conversion points, and you know the different values of each of them, you can look at each point, start to build up ideas on how to improve them, and begin to make informed decisions based on revenue impacts.

By estimating the extent to which you can improve each point, you can begin to figure out where to attack first to improve the bottom line. For example, if today’s revenue sits at $1,000/month, based on 10 users converting each month, you can project that doubling the number of conversion will also double the revenue each month, thus effectively doubling your investment. You can now present a current ROI and a projected ROI that will result from improving a conversion point or increasing traffic (that is, assuming that traffic converts at the same rate as before). By projecting estimates as to what you can do with your search program, you can create a better story about the opportunity that exists with the right investment. Most of this book will be dedicated to how to more accurately estimate the ROI and make decisions based on the impact it will have to your bottom line.

These improvements will come from improving your external search rankings, keywords, and targets, as well as what happens when users get to your site. While you do not have to be a user experience expert, you do need to understand how people move through your site, and what the goals of your site are.

Source of Information : MASTERING SEARCH ANALYTICS MEASURING SEO, SEM AND SITE SEARCH

Capturing Estimated ROI

2012-04-13T00:29:00.000+08:00

Estimated revenue or revenue influence is calculated by “guessing” how much revenue an action will generate. There are several online models where you do not have a direct line to impact ROI. You do, however, have influence. The following list shows direct and indirect actions that can impact ROI. Understanding attribution models such as direct influence versus indirect influence will enable you to have a much deeper understanding of your impact on ROI. Direct influences are measurable and attributed to sales through very clear channels (such as coupons, or referral IDs). Some actions may fall into both categories, depending on whether you apply a cost to them online:

• Offline sales (indirect or direct)
• Lead generation such as email opt-ins (indirect or direct)
• Relationship management such as PR and influential marketing (indirect)
• Purchases (direct)
• Downloading software (indirect or direct)
• Downloading a white paper (indirect or direct)
• Using a specific tool on your website (indirect or direct)
• Sharing content through social media (indirect or direct)
• Clicks on an ad (indirect or direct)
• Referrals to other sites (indirect or direct)

Using the example of selling products offline, you may need to make an estimate as to how many of your users will actually make a purchase offline. To arrive at this estimate, you can partner with a company such as ComScore or Compete to get industry analysis based on user panels, or you can gather data yourself by conducting surveys on your site (there are any number of online survey options, such as Survey Monkey). Surveys are a good way to find out about your users’ buying intent. You can trigger surveys after specific actions, such as using a “help me choose” tool or an interactive demo, and you can ask the respondents targeted questions such as whether they intend to purchase a product, if they found the demo useful, how much they are planning to spend, in what time frame they plan on purchasing a product, and if they have purchased anything recently. Surveys are a very useful tool when used well.

The feedback you get from survey questions such as “Do you plan on making a purchase?,” “How much do you plan to spend?,” and “When do you plan on making the purchase?” will allow you to create some estimates. Based on sampling data, you can derive the average expected spend, and how close your customers are to making a purchase. This latter point allows you to identify whether a particular action is more likely to be taken by people closer to or farther from making a purchase. Using this data, you can also see if there is a variance in average estimated revenue based on how close the customers are to making a purchase. The closer to purchase a customer is, the more likely it is that she has an idea of her budget; however, this number is typically on the low side, as the customer is shopping around and looking for the best deal. This is actually a good thing, as it’s better to use conservative estimates when setting expectations. It’s usually better to overperform than underperform.

In the case of email opt-ins or other lead-generation activities, you can establish a value per lead or name captured. Dividing the total number of leads by the average revenue generated by the leads will provide you with an average cost per acquisition.

It can be trickier to attach a monetary value to PR and marketing efforts, but it is still possible to establish an estimated impact to revenue. In very simple terms, advertising spend is typically based on the number of people it can influence—meaning that for each set of eyeballs there is a positive value or potential influence. If you understand the value of the eyeballs, you can create an online strategy that has an ROI comparable to that of your offline strategies. Consistently benchmarking across all routes to market allows for informed decision making.

Your estimated ROI will be captured as the revenue estimated through surveys and panel data to be based on an action on your website that influenced a sale that was not trackable directly by your metrics suite. The ROI formula remains the same.

Source of Information : MASTERING SEARCH ANALYTICS MEASURING SEO, SEM AND SITE SEARCH

Capturing Actual ROI

2012-04-09T00:24:00.000+08:00

Online you can use clickstream data to track the impact certain web pages have on online sales. Capturing actual monetary values allows you to be much more specific about the return of an action. Discovering where sales happen, be they on or off your site, enables tracking of sales, providing instant insight into revenue. Further, capturing more granular data, such as information on products that are added to or removed from users’ carts but never purchased, allows you to track opportunities you may be missing out on by not improving your checkout process or other sections of your site. Understanding what roadblocks are preventing people from making purchases is critical to finding ways to enable more purchases to happen. That said, actual revenue is a great number to have access to, and most clickstream tools, such as Adobe SiteCatalyst and Google Analytics, have the ability to capture this data. Spending the time to set this up can result in valuable insights that can target revenue specifically. Your actual ROI will be captured as the revenue spent directly online or directly measurable to an action on your website. The ROI formula remains the same.

Source of Information : MASTERING SEARCH ANALYTICS MEASURING SEO, SEM AND SITE SEARCH

Changed Roles --> Database Administrators

2012-04-05T22:47:00.000+08:00

Data professionals, whether they go by the title of database administrators, database engineers, or something else, can be among the most resistant to adopting Scrum. Much of what the preceding section said about programmers will also be true about database administrators. Additionally, data professionals will be faced with learning how to do incrementally what has traditionally been viewed as a part of a project's up-front work.

Standard advice in database design has been to do a complete analysis of the system's needs, create a logical or conceptual database design, and then map the concepts to the constraints of a real-world database during physical database design. Success at this series of steps is predicated on a full and accurate analysis up front. The traditional data professional's view was best summed up to me by a fellow traveler on a plane from Chicago to Sacramento. He was a vice president of database development for a relatively large healthcare company. His view on the world was "applications change; data is forever."

This type of thinking leads to an intense focus on doing a complete analysis up front. This is nice in theory, but while we're taking the time to do that complete analysis, the world is continuing to evolve. Users' needs are changing. Competitors are releasing their products. Databases need to evolve to support the evolving applications built on them.

Much of a DBAs day-to-day work will not change significantly, but how the DBA approaches and schedules that work will change dramatically.

Source of Information : Pearson - Succeeding with Agile Software Development Using Scrum 2010

Changed Roles -> Programmers

2012-04-02T22:42:00.000+08:00

What do programmers do on a Scrum team? They program. They test. They analyze. They design. They do anything necessary to help the team complete the work committed to for a sprint. Although it is OK to have specialists on a Scrum team, specialists need to be willing to work outside their specialties whenever needed for the greater good of the team. There are exceptions. A game development project may, for example, benefit from specialists in artificial intelligence programming. Because of the highly specialized nature of their product, these specialists may do nothing outside their specialty. The majority of programmers on a Scrum team, however, should be willing to contribute in any number of ways to optimize the throughput of the overall team. This means they will test when necessary, sometimes program in a non preferred language, and so on.

One of the most striking changes for programmers on a Scrum team is that they can no longer sit in their cubicles and wait to be told exactly what to program. They need to become active participants in understanding product requirements. Surprisingly, there are many people who simply want to be told what to work on. I've heard this expressed as "if they tell me what to work on and I do it, then I can't be fired." Programmers on a Scrum team—like all others on the team—are expected to share in the responsibility for the overall success of the product. When this responsibility is fully felt, it is easier to do the things that go beyond one's normal job description.

Programmers will also be expected to talk to customers and users. The amount of this can be adjusted up or down based on the programmer, the organization, the strengths of other team members, and the nature of the project. Programmers do not need to develop the personalities of gregarious, glad-handing salespeople. But they do need to be comfortable occasionally talking to a user or customer, even if it's just over the phone.

Similarly, programmers can expect to spend more time interacting with their coworkers. A programmer may not be allowed to come in at 11 and clamp headphones on until quietly leaving at 7. Instead, programmers may be expected to sit in a group space, engage in discussions, help others with problems, and participate in pair programming.

These changes can be quite unsettling for the many programmers (including myself) who got into this field because we thought we could sit alone in our cubicles all day. Prior to my first programming job, I worked in a six-foot by four-foot totally enclosed dark room developing photos all day. I would pop out for regularly scheduled breaks and lunch; otherwise I was alone in the dark all day and loved it. Moving to the lighted world of cubicles was a big change. Moving from quiet cubicles to an energetic, talkative culture is an even bigger change. Programmers on a Scrum team will be expected to make this transition. Fortunately, though, the change isn't that hard for most of us. We may like to be alone, but we find participating in structured conversations (as in the meetings and decision-making discussions on a Scrum project) much easier than unstructured conversations as at a cocktail party.

Beyond the communication and interaction changes, programmers will almost certainly experience changes in how they do their work. The team may not choose to adopt all of these practices at first (or ever in some cases), but I suggest all be considered and tried.

Source of Information : Pearson - Succeeding with Agile Software Development Using Scrum 2010

Changed Roles --> Functional Managers

2012-03-31T22:39:00.000+08:00

Functional managers, such as development managers, QA directors, and so on, who are used to working in a matrixed manner will continue to work that way on Scrum projects. A typical functional manager will likely experience some diminution in power after the transition, but this will depend greatly on how the role was defined in the organization prior to transitioning.

Functional managers usually retain the job of assigning individuals to projects. They will be expected to continue to make these decisions based on the competing needs of all projects, project locations, developmental needs and career aspirations of individuals, and so on. In some organizations, functional managers are accustomed to going beyond assigning individuals to projects and have been involved in the assignment of tasks to individuals within their groups. They will no longer do this after transitioning to Scrum. Individual selection of work is a fundamental aspect of how the members of a team self-organize and must be left to the team.

The Leadership Role of the Functional Manager
Functional managers have always been leaders. Broad leadership trends over the years have affected individual style. While I was growing up, for example, my father managed Sears stores. This was back in the era when Sears was the world's largest retailer. My father's management style was very much top-down. He would establish goals, quotas, and other measurements; communicate them to store employees; and then measure each employee against those targets. This was also an era when prevailing wisdom was that a good manager could manage anything. My father should presumably have been able to take his experience managing a retail store and manage a bank or manufacturing operation with equal skill.

In an organization using Scrum, functional managers should operate in the
top-right quadrant, where they combine a deep understanding of the work with
a bottom-up style. A functional manager is responsible for providing guidance and coaching to members of the group. ScrumMasters and product owners also
provide guidance and coaching, but their views are limited to a single project or
product. A functional manager will have a broader perspective, including the ability
to establish cross-project standards and set expectations for quality, maintainability,
reusability, and many of the other -ilities or nonfunctional requirements.

Functional managers also retain responsibility for developing the people in their groups. Securing the budget and time to send them to conferences, challenging them with appropriate projects, and encouraging them to join or form communities of practice are all part of the functional manager's role.

Personnel Responsibilities
In most organizations, functional managers will retain responsibility for writing periodic reviews of the personnel in their departments. Although the functional manager has hopefully always incorporated input from each employee's coworkers and customers into the review, the need to do so is greater in a Scrum environment because the employee will likely be working less closely with the functional manager on a day-to-day basis.

In many organizations, functional managers also retain responsibility for making hiring and firing decisions. Neither the ScrumMaster nor the product owner has this level of authority over individuals on the product development teams.

After the organization adopts Scrum, most functional managers find themselves with more time available than they had before. This time is most often used to stay in closer touch with their direct reports, to know more about each project the group's employees are working on (by attending various sprint reviews and so on), and to pay more attention to cross-project standards and future directions.

Source of Information : Pearson - Succeeding with Agile Software Development Using Scrum 2010

Changed Roles --> Project Managers

2012-03-27T22:06:00.000+08:00

On a project using a sequential development process, the project manager has the difficult job of ensuring that the product a customer wants is the one that is developed. To do this, the project manager must try to manage everything about the project, including scope, cost, quality, personnel, communication, risk, procurement, and more. Some of these responsibilities really belong to others. Scope control, for example, rightfully belongs with the customer. No one else is in the position to make the necessary trade-off decisions that will arise during product development, as priorities, team velocity, and market conditions shift. Prioritization is not a static, one-time, all-at-the-start activity that can be managed by a project manager.Yet time and again, sequential projects demand that project managers make educated guesses to deliver the right product.

On Scrum projects we acknowledge the untenable role of the project manager and eliminate it. Eliminating the role, though, does not mean we can do away with the work and responsibilities. As you might guess, since self-organizing teams are a core tenet of Scrum, a great deal of the responsibility previously shouldered by the project manager is transferred to the Scrum team. For example, without a project manager to assign tasks to individuals, team members assume the responsibility of selecting tasks themselves. Other responsibilities shift to the ScrumMaster or product owner.

Former project managers often assume one of the roles that have taken on some part of their past responsibilities—the project manager becomes either a ScrumMaster, product owner, or team member, depending on experience, skills, knowledge, and interests.

Some people became project managers because they considered it the next step in a desirable career path, yet they don't enjoy project management. These individuals miss the technical challenge of working as a programmer, tester, database engineer, designer, analyst, architect, or so on. Many of these individuals will take advantage of the elimination of the project manager role to return to work they found more satisfying.

Other project managers have used their roles to become knowledgeable about the business and its customers. A project manager in this situation will leverage that knowledge into a role as a product owner. This can be an excellent fit, especially for the project manager who is having a hard time completely relinquishing the ability to tell the team what to do. As part of their role, product owners are allowed to tell the team a bit of the "what to do" as long as they stay largely away from telling them how to do it. This can satisfy a former project manager whose nature makes it hard to stop occasionally directing the team. If a project manager can overcome the old habits of directing the team and making decisions for it, it is likely such a project manager can become a good ScrumMaster. This is the most common new role for project managers in organizations adopting Scrum. The new role will likely be difficult at first for the former project manager as she learns to bite her tongue and let the team learn how to work through its own issues and make decisions. Often, new ScrumMasters are put in the challenging position of coaching teams at something that they are not yet good at themselves—being agile. The best strategies for a ScrumMaster in this situation include the following:

• Stick as close as possible to doing Scrum by the book. Initially follow the advice of this or another Scrum book closely. Or engage an on-site trainer or coach and follow her advice to the letter. Only begin customizing the process after you have real, hands-on experience with it.

• Talk to other ScrumMasters as much as possible. If there are multiple ScrumMasters in your organization, form a community of practice with 7 5 . SEE ALSO the other ScrumMasters and share good and bad experiences. Look to . . Communities of learn by extracting lessons from commonalities among these experiences.

• Learn as much as you can as quickly as you can. Read books, articles, blogs, and websites. Look into local agile interest groups and attend their meetings. Try to attend one or more of the major agile or Scrum conferences.

Doris Ford, a software engineering manager with Motorola, was a classically trained project manager and a Project Management Professional (PMP). However, despite having a traditional background in project management, Doris's approach has always been about supporting and enabling her teams. Because of that, she was able to easily move from project manager to ScrumMaster. She writes of how her job has changed with Scrum.

Over the years in managing agile development I have learned not to sweat the task details. As a traditional project manager, I always needed to stay on top of who was doing which tasks, what were their dependencies, and would they be done on time. I spent countless number of hours just asking these questions to get the answers in attempt to meet the scope/schedule/budget/quality constraints and reporting upwards on the progress (sometimes using earned value). In an agile environment I had to learn to trust the team members that they would identify and do the tasks necessary to complete the scope for each sprint. It was hard letting go at first, but I quickly learned that the team could do this. I now spend the majority of my time supporting the team members by addressing impediments that they raise and keeping external noise from diverting their focus.

Why the Title Change?
If it's possible for a project manager to become a team's ScrumMaster or product owner, why do we need to change the person's title? Let's consider the term ScrumMaster. Years ago, when I first started running Scrum projects, the term ScrumMaster didn't exist, and it never dawned on me to call the role anything but project manager. This worked well enough. But, I was hiring new individuals into these roles; I was clear with these new hires about my expectations for how they'd interact with the team. I avoided domineering, command-and-control-style individuals. Also, these new project managers reported to me, which allowed me a lot of influence over how they interacted with their teams. Calling them project managers worked fine.

As our company continued to succeed and grow, we began to acquire other companies. In those companies I would inherit project managers who sometimes did have very traditional mindsets about the role of the project manager. I was confronted with helping them shift that mindset to one more compatible with agile development. I found this much harder than just hiring project managers with a collaborative approach suitable to self-organizing teams.

Years later in a discussion with Ken Schwaber, he helped me understand why transitioning existing project managers had been more difficult than I had anticipated. Schwaber informed me that by allowing the project managers to retain their titles, I was allowing them to think that the changes were less allencompassing than they were. He invented the word ScrumMaster in 1997 in part because it would remind everyone that this was not just the project manager role with a few additional responsibilities removed or added. Schwaber told me that "the vocabulary of Scrum is a vocabulary of change. The words are often intentionally ugly—burndown, backlog, ScrumMaster—because they remind us that change is occurring."

Although I recommend it, you do not necessarily need to banish the title project manager. If you or your organization is enamored of it, continue to use it. But be mindful of Ken Schwaber's advice and my experience that using the old words will slow or prevent the adoption of the new approach. Retaining an old title discourages thinking in the new way. Further, if people are unwilling to relinquish something as insignificant as a j ob title, they will probably also be unwilling to make the far harder changes necessary to adopt Scrum.

Source of Information : Pearson - Succeeding with Agile Software Development Using Scrum 2010

Changed Roles -> Analysts

2012-03-23T17:12:00.000+08:00

With an intimate knowledge of the product and strong communication skills, some analysts will tend to shift into product owner roles. This is especially common on large projects that make use of a hierarchy of product owners. Someone with product manager on her business card, for example, may act as the chief product owner for the overall product, spending most of her time looking outward at users and the market. An individual with analyst on her business card, on the other hand, may act as product owner for the various teams, working with the chief product owner to translate her vision into product backlogs for her teams.

Many teams find that having an analyst on the team continues to be very beneficial, although the ways in which the analyst works will change. On traditionally managed projects, the analyst's mission seemed to be to get as far ahead of the team as possible. On a Scrum project, just-in-time analysis becomes the goal. The analyst's new aim is to stay as slightly ahead of the team as possible while still being able to provide useful information to the team about current and near-term features.

Analysts can be instrumental in achieving the goal of shifting the emphasis from writing about requirements to talking about them. Because analysts are not working as far ahead of the team as they may be used to, they need to become more comfortable sharing information with the team more informally, rather than through a large document. As much information as possible should be shared through verbal discussion, but analysts will still need to document some requirements, especially when working on a distributed team. Often though, what the analyst writes will be less formal—more often a wiki than a document with a signature page.

On traditional projects, analysts often become intermediaries through whom other team members and the product owner communicate. On a Scrum project the analyst should become more a facilitator of team-product owner discussion than an intermediary. Team members and product owners need to talk. Rather than be the conduit for all conversation, the good agile analyst focuses on making sure those conversations are as productive as possible given the time constraints the team or product owner may be under. This may mean that the analyst steers the product owner and team toward talking about one user story rather than another because that is where there is more risk of going astray. Or it may mean that the analyst conveys a top-level understanding of a new feature to the team before bringing the team and product owner together to discuss the details.

On a traditional project, an analyst may say to the team,"I've talked to our key stakeholder, understand what he wants, and have written this document describing it in detail." By contrast, on a Scrum project, the same analyst should say, "I've spoken to our product owner and have a feeling for what he's after. I wrote these six user stories to give you a start, and I've got a bunch of additional questions to ask the product owner. But I want to make sure that I bring along a couple of you when we have those discussions."

With all this talk of analysts looking ahead, it can be tempting to think that analysts work a sprint ahead of the team. They don't. Gregory Topp, an analyst with Farm Credit Services of America, describes how using Scrum has allowed him to concentrate on the current sprint: "Before Scrum, I had to focus on requirements that were not going to be developed for several weeks, if not months. Now, I focus on the current sprint (two weeks for us), so more time can be spent on user story details, development, and testing." An analyst's first priority is to achieve the goals of the current sprint. An analyst on a Scrum team will assist in testing, will answer questions (or track down answers to questions) about features being developed, will participate fully in all regular sprint meetings, and so on.

However, it is quite possible that these activities will not fully consume the analyst's time. Time that is not needed to complete the work of the current sprint can be used to look ahead. However, being a part of the team on this sprint and spending some time looking ahead is not the same as working a sprint ahead of the team. Topp explains how jumping too far ahead actually put him behind: "I tried working ahead a sprint or two, defining user story details. I found that this caused the current sprint to suffer. I also found that many times the details of a user story changed by the time the team actually started working on the story."

A common question is whether the effort analysts spend looking ahead should be included on the sprint backlog. My recommendation is to include on the sprint backlog any specific analysis tasks that can be identified during sprint planning. For example, suppose the team is working on an application that approves or rejects loan applications. If the product owner and team agree that the next sprint will include work on calculating the applicant's credit score, then preliminary analysis tasks related to that should be identified, estimated, and included in the sprint backlog. On the other hand, if the next sprint's work is unknown, no specific tasks related to the next sprint should be included on the sprint backlog. Overall, many analysts enjoy the change to Scrum even though they relinquish the role of sole interpreter of customer desires. Two years after adopting Scrum, Topp commented on how his relationship with others on the team had changed.

Because we are all on the same team and all work on the same user stories at the same time, the team seems to have more unity. Before using Scrum, it seemed each function (analyst, programmer, tester, DBA) was done in a silo. There was more fingerpointing when in that mode. Now using Scrum, the team is all focused on a small set of stories. The finger-pointing has been eliminated with an "as a team" mindset.

Source of Information : Pearson - Succeeding with Agile Software Development Using Scrum 2010

Overcoming Common Problems of a Good Product Owner

2012-03-20T16:33:00.000+08:00

There are many potential pitfalls when selecting the initial product owner. Some of the most common early-stage problems and what you can do to address them include

The product owner delegates decision making but then overrules the decision maker. To fit the new duties of the product owner into their schedules, some
product owners delegate decisions about specific parts of the product. Other
product owners enlist a business analyst to be a "feature owner" over some part
of the system. This can work well because the product owner has more time to
dedicate to areas that are not as easy to delegate.

Problems arise when the product owner says that decision-making authority has been delegated but then continues to approve or sometimes reverses decisions. Before delegating, product owners should be sure they are really willing to delegate without later second-guessing. Because of the pressure of the short, timeboxed sprints, Scrum teams often move much more quickly than they did before transitioning. It is inevitable that some decisions that the product owner delegates will turn out to be wrong, and these should be revisited. What we want to avoid, however, are situations where the product owner says,"Get your answers from Dave; he owns this part of the system," and then consistently overrules Dave's answers.

My advice to a new and overloaded product owner is to free some time by delegating just beyond the point at which you're comfortable. You may be pleasantly surprised and find no important decisions to reverse. But you'll occasionally find some decisions you would have made differently. Often the best thing to do in this case is the same thing we're all taught when learning to drive: If the car starts to slide, steer into the slide. Rather than pull against the decision (assuming it is not a horrendous one), allow that decision to persist through the end of the sprint; then decide if it should be changed. When the cost of reversing a decision is compared against all the other valuable work on the product backlog, you may find that the decision wasn't so bad after all.

The product owner pushes the team too hard. Product owners are often under pressure to deliver financial results to the company; more features delivered sooner is one way for them to achieve it. As I've said, I have no objection to a product owner who announces at the start of a project, "We need to build a product that is smaller, better-performing, and cheaper than our competitor's, and we need to do it in three months less than we spent on the last product." As long as a challenging goal like that is accompanied by appropriate freedom in how the goal is achieved, the team will do its best. The problems arise when the team is kept under constant and changing pressure from sprint to sprint. One difficult goal of "do this amazing thing in 6 months" is in many ways less stressful for the team than 13 successive two-week sprints of "I need more, more, more!" If you have product owners who are pushing teams this way, the ScrumMasters should first push back and then work with the product owners to set longer-term goals for the teams while ensuring teams have commensurate degrees of freedom in how those goals are achieved.

The product owner wants to cut quality. Cutting quality is an oh-so-tempting decision when trying to deliver a challenging set of features by a difficult date. It can lead to the short-term appearance of having met the objectives established at the start of the project. Eventually, however, the cost of having cut quality becomes apparent as more post-release bugs than usual are reported, the team's velocity decreases, and customers clamor for the product to behave as they thought it would.

Ken Schwaber has called quality a "corporate asset" (2006). As such, no one except the chief executive has the authority to sacrifice quality in exchange for achieving a short-term goal such as making a release date. A decision to cut quality may be the appropriate one; I can't tell you otherwise without knowing the full context of a situation. But, that decision is one that needs to be made sufficiently high up in the organization and with such openness that no one is surprised by the negative impacts that will almost certainly follow

Selecting product owners who understand this is sometimes challenging in organizations that are consistently focused on this quarter's numbers. Pushing back against attempts to reduce quality is the job of the ScrumMaster. The ScrumMaster does not need to prevail in these early disagreements. The ScrumMaster does, however, have to succeed in making the decision visible.

Time is on the ScrumMaster's side. If the ScrumMaster successfully raises the visibility of decisions to cut quality, he should eventually be able to win later arguments against reducing quality. "Remember on the Gouda project how I told you that cutting quality on version one would hurt us during version two?" the ScrumMaster can say. "Well, here are the graphs of velocity on the two projects. Note that version two had a lower velocity even though we added two experienced people. That was because we left bugs behind during version one (here's a graph showing that) and because the team didn't feel it had the time to do a good job of keeping its code clean. We even skipped the automated unit tests on a few modules. Here's a comparison of the number of defects found during the six months following release, broken down by whether the module had automated unit tests. It's up to you if we do this again on this project, but I think you know my opinion."

Our product owner is in a different city than the development team. With more projects being developed by remote teams, this is an increasingly common situation. Both the team and product owner in this situation should assume some of the burden of over communicating with the other. I have worked with many remote product owners, and it can work very successfully as long as the product owner does the following:
• Remains engaged in the project
• Establishes a rapport with the team
• Performs all usual duties of the role
• Is available to the team for phone calls for at least some part of the day, even if it is after the usual workday for the product owner
• Responds by e-mail or phone when not available in person

Source of Information : Pearson - Succeeding with Agile Software Development Using Scrum 2010

Attributes of a Good Product Owner

2012-03-16T16:19:00.000+08:00

As when describing what to look for in selecting or hiring a good ScrumMaster, I've culled the long list of desirable product owner traits down to five must-have attributes.

Available. By far the most frequent complaint I hear from teams about their product owners is that they are unavailable when needed. When a fast-moving team needs an answer to a question, waiting three days for an answer is completely disruptive to the rhythm it has established. By being available to the team, a product owner demonstrates commitment to the project. The best product owners demonstrate their commitment by doing whatever is necessary to build the best product possible. On some projects this includes doing things like assisting in test planning, performing manual tests, and being actively engaged with other team members.

Business -savvy. It is essential that the product owner understand the business. As the decision maker regarding what is in or out of the product, the product owner must have a deep understanding of the business, market conditions, customers, and users. Usually this type of understanding is built over years of working in the domain, perhaps as a past user of the type of product being developed. This is why many successful product owners come from product manager, marketing, or business analyst roles.

Communicative. Product owners must be good communicators and must be able to work well with a diverse set of stakeholders. Product owners routinely interact with users, customers, management within the organization, partners, and, naturally, others on the team. Skilled product owners will be able to deliver the same information to each of these different audiences while at the same time tailoring their message to best match the audience. A good product owner must also listen to users, customers, and perhaps most important the team. Especially as team members learn more about the product and market (as they should over time, especially on a Scrum project), they will be able to offer valuable suggestions about the product. Additionally, all teams will have much to say to the product owner about the technical risks and challenges of the project. Although it is true that the product owner prioritizes all work for the team, the wise product owner will listen to her team when it recommends some adjustments in those priorities based on technical factors.

Decisive. Another common complaint teams make about their product owners is their lack of decisiveness. When team members go to the product owner with an issue, they want a resolution. Scrum puts a lot of pressure on teams to produce functionality as quickly as possible. Teams are frustrated when a product owner responds to a question with, "Let me call a meeting or convene a task force to work on that." A good team will understand that this is sometimes necessary, but teams are very perceptive at knowing when a product owner is actually just trying to avoid making a hard decision. Just as bad as a product owner who won't make a decision is the product owner who makes the same decision over and over but with different answers. A good product owner will not reverse prior decisions without a good reason.

Empowered. A good product owner must be someone empowered with the authority to make decisions and one who is held accountable for those decisions. The product owner must be sufficiently high up in the organization to be given this level of responsibility. If a product owner is consistently overruled by others in the organization, team members will learn to go to those others with their important questions.

Source of Information : Pearson - Succeeding with Agile Software Development Using Scrum 2010

ROI—The Universal Metric

2012-03-13T00:23:00.000+08:00

There is one language that is understood universally across companies, and that is the language of finance. Every company is working to maximize its potential value and each one will have a different strategy for achieving this, but at the end of the day, a company can only be considered successful if it can afford to keep the lights on.

Every company has to pay taxes and every company has bills to pay. Some companies may believe in producing products that they want to use, while other companies may strive to provide the best customer experience and service possible—whatever their goals are, companies can only meet them by generating revenue.

ROI provides insights into the overall cost of an investment, taking into account everything from the marketing budget to the costs of equipment and software. There are other variations of ROI where the investment is defined differently. One of the more common variations you may see is return on ad spend (ROAS).

Here is the best way to think of the two. Suppose you are selling lemonade and you spend $25 on flyers that you put up around your neighborhood. Further suppose it costs another $25 to make that lemonade. The money you spend on flyers would be your ad spend, so you would need to make at least $25 to get a positive ROAS, while you would need to make $50 to get a positive ROI. To make this even more interesting, suppose you paid someone $25 to sit at your lemonade stand to sell the lemonade. That is another $25 that may not come out of your ad spend. Now you need to make $75 to have a positive ROI, but still only $25 for a positive ROAS.

ROI and ROAS are calculated as follows:
ROI = ((Revenue - Cost) / Cost) × 100.
ROAS = ((Revenue - Advertising Cost) / Advertising Cost) × 100
Both ROI and ROAS are expressed as percentages, hence the multiplication by 100.

For simplicity’s sake, I will use the term ROI to talk about measuring the value of an investment. Understand, however, that there are several ways to measure this, and your
own organization may have a specific way as well, depending on how budgets are split and defined.

As part of your measurement model, it is best practice to establish baselines. One of the first baselines you should establish is your ROI value. To do this, you will likely run through several stages of your analytics program. Bruce Clay uses the following framework (http://www.bruceclay.com/web_analytics.htm) that I think works very well:

Determine Needs → Identify Goals → Define Metrics → Collect Data → Record
Baseline → Test Improvement Strategies → Implement Improvements → Measure
Results → Repeat Process Periodically

we talked about understanding your business needs and goals. At this point, we are starting to define our metrics. The first metric we want to define is how to capture an ROI value. We understand how to calculate this, but now we must capture and collect it.

Because ROI is tied to monetary value, we must identify actions that generate measurable revenue. There are two approaches: actual and estimated. Actual ROI is measured when you have direct access to purchase data and the referring lead. For example, offline you can use coupons to measure the direct impact of the source of those coupons.

Source of Information : MASTERING SEARCH ANALYTICS MEASURING SEO, SEM AND SITE SEARCH

Who Are You Optimizing For?

2012-03-10T22:44:00.000+08:00

First, let me come right out and say it: why do SEO and SEM people always have to prove which is more important? Why can’t we just all be search people? A well-defined search program should utilize both SEO and SEM tactics to provide maximum coverage and exposure to the right person at the right time, to maximize your revenue. I do not believe that SEO and SEM should be optimized separately from each other; in fact, there should be open sharing and examination of your overall search strategy. With that said, each practice has its own needs and methods that may be unique to it. Still, remember that the goal is to maximize your revenue by investing smartly. This means that you should invest in traffic that will convert at the maximum value for you.

Search optimization—particularly SEO—is traditionally thought of as improving the rankings of pages. Therefore, pages are optimized for search engines. My personal opinion on this is that if this is your only objective, you are doomed to fail. The simple fact is that the engines are changing every day. There are no “rules,” just best practices that have been adopted because they show positive results in rankings and can have positive results for the end user. To me, it’s not how much traffic you get that counts, but what you do with the traffic.

Your goals in improving search results should include positive impacts to your customers, and ultimately, positive impacts to your revenue streams. Optimizing entry pages for SEO is about improving the flow of traffic into your site to achieve a positive outcome for both your customers and your business. Users arriving from search should have an even better customer experience than those who enter through your home page. Search analytics are just as much about what happens on your site as what drives people to your site.

On top of all this, the online world has brought us an overwhelming multitude of information points. SEO in particular moves at such a rapid pace of change that keeping up with changes in the algorithms is practically impossible. Eric Schmidt, former CEO of Google, claims Google uses over 200 ranking factors to establish what shows up on every organic search result (http://searchengineland.com/schmidt-listing-googles-200-ranking-factors-would-revealbusiness-secrets-51065). On top of this, there are over 500 tweaks made to the algorithm every year—more than one change per day. Optimizing to the engines is not a game you can win, but optimizing to people and their behaviors is.

Relevancy can also have an impact on paid search. Google AdWords (the largest of the paid search options) measures relevancy through Quality Score, a metric that takes into account the click-through rate (CTR) of the keyword, the historical CTR of all ads and keywords in your account, the CTR of the display URL in the ad group, the quality of your landing page, the relevance of the keyword both in the ad group and to the search query, and other factors. The better your Quality Score is during each search, the less you will pay per click. It should also be noted that the Quality Score used to determine the cost per click is generated for every search and is not a direct reflection of the Quality Score you see in AdWords (http://adwords.google.com/support/aw/bin/answer.py?hl=en
&answer=21388).

Beyond this, the real power of paid search is that you have full control over the user experience: everything from what copy and text the users will see to what pages they will be directed to. You can control the time of day results will be displayed, and you can even target specific device types (mobile or desktop) or geographies (particular cities or countries). The amount of direct and immediate control you have over your paid search campaigns means a greater opportunity to optimize and improve results. Site search can vary from site to site. How effective is your site search? How frequently is your site search used? There are a great deal of data points specific to search; the challenge is figuring out what points need to be used to answer specific sets of questions. Data only has value if it enables someone to do something.

For those of you used to practicing website measurement through Google Analytics or other clickstream tools, there will be some familiarity, though search analytics also use off-site factors, as well as user experience (UX) and information architecture (IA) factors. Some are paid and some are free, but the most important things to consider will be which tools enable you to make insights that help you meet your business needs, and which ones you feel most comfortable using. Sometimes the free options can be just as good as an enterprise-level paid option, and whenever possible.

Search analytics requires a bit of psychology; because we are dealing with words and people, we are given partial insights into our users’ thoughts. Think of a search box as a word association test. People provide a word or a group of words describing or identifying what they are looking for. The engine’s job is to interpret the user’s intent and match that word or words to the page or pages it thinks will best serve the user.

Further, a great deal of data aggregation is carried out to identify patterns that groups of searchers follow. At times you may need to make some assumptions. When you find this to be the case, I strongly urge you to use surveys to help eliminate this guesswork. Qualitative data can go a long way. For example, you can ask people on the page with the highest abandonment rate, “What are you looking for?”

When you need to make an educated guess, it’s important to remember it is just a guess. It can act as a starting place, but it’s only ever a hypothesis. Be prepared to follow a different avenue if it turns out that you are wrong. Like a good detective, you should be able to use your analytics to eventually answer questions or support theories you may have, but until you have supporting data, your hypothesis will only ever be an unproven guess. Also, because people change, you will never be able to stop measuring your site if you plan on improving sales and the user experience.

Source of Information : MASTERING SEARCH ANALYTICS MEASURING SEO, SEM AND SITE SEARCH

How Is Search Data Different from Clickstream Data?

2012-03-07T21:57:00.000+08:00

Search analytics introduces some qualitative data from the search term coupled with many quantitative data points in the form of click-through rates, traffic volume, conversion rates, and more.

You can get some insight into users’ intent and decision-making processes by looking at groups of search terms as qualitative data points. For example, if a user comes to your site from three different search terms, looking at those terms may show some of the decision making that has occurred. For instance, if the search pattern looks like “ACME widgets”→“ACME widget reviews”→“ACME widget sale”, we can see that the user started out looking for a product, was then influenced by the reviews, and then looked for a discounted price. You can further amplify this qualitative data with site surveys. By bringing together qualitative data and quantitative data, we can start to get a better idea of the intent of our users, and optimize their experience.

Qualitative data measures behavior and the reasons driving that behavior. For example, surveys and questionnaires can provide qualitative data; in our case, this may also come from search parameter patterns over repeat sessions.

Quantitative data is numerical data. Examples of this are the number of visits to a website, or the number of people who purchase a product.

Clickstream data measures the actions users take on your website by tracking what and where they click.

You can get better ideas of “intent patterns” through search when you merge your external search data with your on-site search data. Perhaps many users come to your site through branded terms, meaning words that are specific to only your brand. An example of this would be a search for “iPhone”: that is specific to Apple, and there is a brand association there. When users show up from a branded term, what secondary searches do they perform on your site?

Have you enabled your clickstream analytics to capture the referring search terms and the associated site search terms? Are you recognizing that many users who come to your site on branded terms are looking for a specific product? Or perhaps there are a great number of support searches. If you have paid for the click-through SEM, maximize the value by learning what users are looking for and develop landing pages to bring this needed information closer to the user. In the case of SEO, you cannot always dictate what page will rank, but you can improve your site’s general navigation to include links to these deeper pages. Recognizing your customers’ needs and providing content that helps them will in the long run also help improve your business.

Search also gives you insight into the language of your customers. The goal of keyword research is to understand the language landscape of the search engines. What are the search volumes like, and how competitive are other sites on these terms? This is data that can be fed to other marketing channels. Why not apply this keyword research to your email deployments or your in-store flyers, bringing online and offline insights together to create subject lines and in-store banners with the language customers use?

When we talk about branded versus nonbranded words, I see all too often a confusion with internal versus external language. Internally you may want to call your product “the super best product ever!” while users may simply call it “widget.” Unless you have the dollars and branding resources to get people to change their language, you may need to recognize that it is easier to get people to think about your product by creating at least a small word association to the word “widget.” Ignoring the elephant in the room and calling your product anything but may result in people not having that “ah ha!” moment and realizing that your widget is really also “the super best product ever.” Search is as much about measuring word use and linguistic needs as it is about measuring clicks, inbound links, and other data points.

Beyond these factors, SEO has unique challenges in that all the major search engines operate as black boxes. The search engines do not let anyone know the recipe for their secret sauce, or in this case the algorithm that makes them run. To better understand these algorithms, SEO specialists have had to try to reverse engineer them. It’s also important to understand that each engine runs different algorithms—for example, Bing runs a different algorithm than Google. Each engine’s algorithm is proprietary to that engine; in some cases, other engines may lease these algorithms (as Yahoo! now leases Bing’s algorithm), but each major algorithm will have its own quirks and issues to test against.

The best way to reverse engineer something such as a search algorithm is to look at data points, examining them to try to determine which return positive feedback and which return negative feedback. Analytics help take a lot of the guesswork out of the SEO’s job.

Search analytics is not just about measuring traffic delivered, but also about landing page optimization (LPO) and conversion rate optimization (CRO). LPO is focused on retaining and moving people through your site; it acknowledges that not everyone comes to your site through the home page. You typically start by optimizing your highvolume entry pages and work down from there. CRO is focused on moving people through a funnel to a goal. This conversion may occur over several visits as part of the overall life cycle and decision-making process of purchasing a product. CRO takes into account the stages of this process and the needs of a user to help that user make a decision as quickly as possible.

A search strategy is not concerned simply with delivering traffic to a site; it must take into consideration the handoff of that traffic, as well as the pathing of that traffic to each goal and objective. Site search also helps facilitate the measurement of on-site navigation issues and needs. Think about moving beyond measuring traffic volume, and measuring business objectives and goals. Think about measuring to improve those objectives and goals.

Source of Information : MASTERING SEARCH ANALYTICS MEASURING SEO, SEM AND SITE SEARCH