Arnon's shared items in Google Reader

Inconsistent data, poor performance, or SOA – pick one

2011-09-18T16:52:17Z

One of the things that surprises some developers that I talk to is that you don’t always get consistency even with end-to-end synchronous communication and a single database. This goes beyond things like isolation levels that some developers are aware of and is particularly significant in multi-user collaborative domains.

The problem

Let’s start with an image to describe the scenario:

Image 1. 3 transactions working in parallel on 3 entities

The main issue we have here is that the values transaction 2 gets for A and B are those from T0 – before either transaction 1 or 3 completed. The reason this is an issue is that these old values (usually together with some message data) are used to calculate what the new state of C should be.

Traditional optimistic concurrency techniques won’t detect any problem if we don’t touch A or B in transaction 2.

In short, systems today are causing inconsistency.

Some solutions

1. Don’t have transactions which operate on multiple entities (which probably isn’t possible for some of your most important business logic).

2. Turn on multi-version concurrency control – this is called snapshot isolation in MS Sql Server.

Yes, you need to turn it on. It’s off by default.

The good news is that this will stop the writing of inconsistent data to your database.
The bad news is that it will probably cause your system many more exceptions when going to persist.

For those of you who are using transaction messaging with automatic retrying, this will end up as “just” a performance problem (unless you follow the recommendations below). For those of you who are using regular web/wcf services (over tcp/http), you’re “cross cutting” exception management will likely end up discarding all the data submitted in those requests (but since that’s what you’re doing when you run into deadlocks this shouldn’t be news to you).

The solution to the performance issues

Eventual consistency.

Funny isn’t it – all those people who were afraid of eventual consistency got inconsistency instead.

Also, it’s not enough to just have eventual consistency (like between the command and query sides of CQRS). You need to drastically decrease the size of your entities. And the best way of doing that is to partition those entities across multiple business services (also known in DDD lingo as Bounded Contexts) each with its own database.

This is yet another reason why I say that CQRS shouldn’t be the top level architectural breakdown. Very useful within a given business service, yes – though sometimes as small as just some sagas.

Next steps

It may seem unusual that the title of this post implies that SOA is the solution, yet the content clearly states that traditional HTTP-based web services are a problem. Even REST wouldn’t change matters as it doesn’t influence how transactions are managed against a database.

The SOA solution I’m talking about here is the one I’ve spent the last several years blogging about. It’s a different style of SOA which has services stretch up to contain parts of the UI as well as down to contain parts of the database, resulting in a composite UI and multiple databases. This is a drastically different approach than much of the literature on the topic – especially Thomas Erl’s books.

Unfortunately there isn’t a book out there with all of this in it (that I’ve found), and I’m afraid that with my schedule (and family) writing a book is pretty much out of the question. Let’s face it – I’m barely finding time to blog.

The one thing I’m trying to do more of is provide training on these topics. I’ve just finished a course in London, doing another this week in Aarhus Denmark, and another next month in San Francisco (which is now sold out). The next openings this year will be in Stockholm, London; Sydney Australia and Austin Texas will be coming in January of next year. I’ll be coming over to the US more next year so if you missed San Francisco, keep an eye out.

I wish there was more I could do, but I’m only one guy.

Hmm, maybe it’s time to change that.

The Three Ages of Google - Batch, Warehouse, Instant

2011-08-29T16:22:46Z

The world has changed. And some things that should not have been forgotten, were lost. I found these words from the Lord of the Rings echoing in my head as I listened to a fascinating presentation by Luiz André Barroso, Distinguished Engineer at Google, concerning Google's legendary past, golden present, and apocryphal future. His talk, Warehouse-Scale Computing: Entering the Teenage Decade, was given at the Federated Computing Research Conference. Luiz clearly knows his stuff and was early at Google, so he has a deep and penetrating perspective on the technology. There's much to learn from, think about, and build.

Lord of the Rings applies at two levels. At the change level, Middle Earth went through three ages. While listening to Luiz talk, it seems so has Google: Batch (indexes calculated every month), Warehouse (the datacenter is the computer), and Instant (make it all real-time). At the "what was forgot" level, in the Instant Age section of the talk, a common theme was the challenge of making low latency systems on top of commodity systems. These are issues very common in the real-time area and it struck me that these were the things that should not have been forgotten.

What is completely new, however, is the combining of Warehouse + Instant, and that's where the opportunities and the future is to be found- the Fourth Age.

The First Age - The Age of Batch

Visualized: the Apple mothership

2011-08-14T00:23:00Z

We've already caught a glimpse of Apple's proposed ~~science fiction museum~~ new campus in Cupertino, but screen grabs just don't do this behemoth justice. A recently released set of renderings of Apple Campus 2, as it's known to the city of Cupertino, however, give it that proper otherworldly glow. According to the accompanying proposal, the building will take up a measly 2.8 million square feet, contain a 1,000-seat auditorium and research facilities totaling 300,000 square feet. Really? Is that all? If architectural renderings are your thing, hit the source link for some building-plan booty.

Gallery: The Apple mothership

Visualized: the Apple mothership originally appeared on Engadget on Sat, 13 Aug 2011 20:23:00 EDT. Please see our terms for use of feeds.

Permalink TechCrunch | Cupertino.org | Email this | Comments

A Detailed Guide to Oozie

2011-08-03T19:05:17Z

Boris Lublinsky and Michael Segel series of articles about Oozie, the Hadoop workflow framework, published on InfoQ:

Introduction to Oozie

Oozie workflow is a collection of actions (i.e. Hadoop Map/Reduce jobs, Pig jobs) arranged in a control dependency DAG (Direct Acyclic Graph), specifying a sequence of actions execution. This graph is specified in hPDL (a XML Process Definition Language).
Oozie by Example

In this article we will describe a more complex Oozie example, which will allow us to discuss more Oozie features and demonstrate how to use them. The workflow which we are describing here implements vehicle GPS probe data ingestion. Probes data is delivered to a specific HDFS directory hourly in a form of file, containing all probes for this hour. Probes ingestion is done daily for all 24 files for this day. If the amount of files is 24, an ingestion process should start. […]
Extending Oozie

In this article we will show how to leverage Oozie extensibility to implement custom orchestration language extensions.

These should be enough not only to give you on overview of Oozie but also a good start to using it when complex workflows are needed by your Hadoop MapReduce jobs.

Original title and link: A Detailed Guide to Oozie (NoSQL database©myNoSQL)

Presentation: Testing for the Unexpected

2011-07-29T15:00:00Z

Ulf Wiger discusses the importance of automated testing along with some lessons learned at Ericsson, including using randomized and extensive testing, aiming to achieve system robustness. By Ulf Wiger

What is Enterprise Search ?

2011-07-27T09:56:00Z

Searching for something of interest on the internet has become a norm for billions of
people around the world. Internet Search has revolutionized the way business
transformation happens. Similarly Enterprise Search has revolutionized the way data is
handled and analyzed in corporate and e-commerce world. The one single point that
differentiates Enterprise Search from internet Search is the importance of organizing and
categorization of data, which is most of the time over looked in internet search.

• Enterprise search is a process of organizing and categorizing information either
structured or unstructured inside a single corporate websites/networks to search
with utmost user friendliness and meeting to-the-point business objectives as
motivation.

• Enterprise search is here to increase business value to the randomly distributed
data across the corporate website/ networks. It’s a foundation for new generation
business intelligence solution. It provides contextual and logical insight into all
enterprise data with high precision, increased scalability and friendly usability.

• Cutting across structured and unstructured information, Enterprise search provides
relevant data to ordinary workers, research scholars to CEO's which increases the
productivity and quality of work accomplished using the result data set.

• Traffic and revenue increase in significant proportions is the hall mark of enterprise
search.

Every Enterprise Search Software (ESS) product would have to support following
functionality:

• Ability to crawl and fetch the Enterprise content and perform indexing,
summarization, etc on it for faster and effective access.

• Ability to perform free text-based search on the indexed Enterprise content and
serve matching set of records to the Enterprise users.

• Ability to ensure that user access to the Enterprise content records is based on
access rights granted to the Enterprise user on the Enterprise content.

• Ability to report changes to the content records (additions, modifications,
deletions) on the data sources.

• Ability to report changes to the access rights information in terms of grant and
revocation to users and security groups.

• Ability to perform interactions with the connected data sources in a secure and
encrypted manner.

Real Time Analytics for Big Data: An Alternative Approach

2011-07-14T15:26:26Z

Lately, we've been talking to various clients about realtime analytics, and with convenient timing Todd Hoff wrote up how Facebook's realtime analytics system was designed and implemented (See previous review on that regard here).

They had some assumptions in design that centered around the reliability of in-memory systems and database neutrality that affected what they did: for memory, that transactional memory was unreliable, and for the database, that HBase was the only targeted data store.

What if those assumptions are changed? We can see reliable transactional memory in the field, as a requirement for any in-memory data grid, and certainly there are more databases than HBase; given database and platform neutrality, and reliable transactional memory, how could you build a realtime analytics system?

Joseph Ottinger and I discussed this, and this is what we came up with.

A Summary of History

To understand what a new design might look like, it’s often useful to consider a previous design. This is a very short summary of Facebook’s realtime analytics system.

First, it’s based on a system of key/value pairs, where the key might be a URL and the value is a counter. Thus, there’s a requirement for atomic, transactional updates to a very simple piece of data. The difficulties come from scale, not from the focus of the system.

The process flow is fairly simple:

A user creates an event by performing some action on the website. This generates an AJAX request, sent to a service.
Scribe is used to write the events into logs, stored on HDFS.
PTail is used to consolidate the HDFS logs.
Puma takes the consolidated logs from PTail and stores them into HBase in groupings that represent roughly 1.5 seconds’ worth of events.
HBase serves as the long-term repository for analytics data.

There are some questions around how PTail and Puma serve as scaling agents, and some of the notes around their use are still limited in scale – for example, one of the concerns is that an in-memory hash table will fill up, which sounds like fairly serious limitation to have to keep in mind.

A Potential for Improvement

There are lots of areas in which you can see potential improvements, if the assumptions are changed. As a contrast to Facebook's working system:

We can simplify the design. If memory can be seen as transactional - and it can - we can use them without transforming them as they proceed along our analytics workflow. This makes our design and implementation much simpler to implement and test, and performance improves as well.
We can strengthen the design. With a polling semantic, such systems are brittle, relying on systems that pull data in order to generate realtime analytics data. We should be able to reduce the fragility of the system, even while making it faster.
We can strengthen the implementation. With batching subsystems, there are limits shouldn’t exist. For example, one concern in Facebook's implementation is the use of an in-memory hash table that stores intermediate data; the in-memory aspect isn’t a concern until you realize that the batch sizes are chosen partially to make sure that this hash table doesn’t overflow available space.
We can allow deployments to change databases based on their requirements. There's nothing wrong with HBase, but it's got specific characteristics that aren't appropriate for all enterprises. We can design a system which you’d be able to deploy on various and flexible platforms, and we can migrate the underlying long-term data store to a different database if needed.
We can consolidate the analytics system so that management is easier and unified. While there are system management standards like SNMP that allow management events to be presented in the same way no matter the source, having so many different pieces means that managing the system requires an encompassing understanding, which makes maintenance and scaling more difficult.

What we want to do, then, is create a general model for an application that can accomplish the same goals as Facebook’s realtime analytics system, while leveraging the capabilities that in-memory data grids offer where available, potentially offering improvement in the areas of scalability, manageability, latency, platform neutrality, and simplicity, all while increasing ease of data access.

That sounds like quite a tall order, but it’s doable.

The key is to remember that at heart, realtime analytics represent an events system. Facebook’s entire architecture is designed to funnel events through various channels, such that they can safely and sequentially manage event updates.

Therefore, they receive a massive set of events that “look like” marbles, which they line up in single file; they then sort the marbles by color, you might say, and for each color they create a bundle of sticks; the sticks are lit on fire, and when the heat goes up past a certain temperature, steam is generated, which turns a turbine.

It’s a real-life Rube Goldberg machine, which is admirable in that it works, but much of it is still unnecessary if the assumptions about memory ("unreliable") and database ("HBase is the only target that counts") are changed. Looking at the analogy from the previous paragraph, there’s no need to change a marble into anything. The marble is enough.

A Plan for Implementation

Our design for implementation is built around putting data and messaging together. A data grid is a perfect mechanism for this, as long as it provides some basic features: transactional operations, push and pull semantics, and data partitioning.

A data grid does provide those basic features, or else it's not really much of a data grid; it'd be more of a cache otherwise.

With a data grid, then, the events come in as individual messages. When the user chooses an operation on the web site, an asynchronous operation would write the event, just as Facebook does today. However, instead of filtering and batching the events into various forms, the events are dispatched to waiting processes that perform many transactional updates in parallel.

There’s a danger that those updates might be slower than the generated events, if each event is processed sequentially. That said, this isn’t as much a problem as one might think; if data partitioning is used, then event handlers can receive partitioned events, which localizes updates and speeds them up dramatically.

In fact, you can still use batching to process events as a group; since the events would be partitioned coming in, the batch process would still be updating local data very quickly, which would be faster than individual event processing, even while retaining simplicity.

With this design, there is no overflow condition, because a system that’s designed to scale in and out as most data grids are will repartition to maintain even usage. If a data grid can’t provide this feature intrinsically, of course some management will be necessary, but finding data grids with this feature isn’t very difficult.

One other advantage of data grids is in write-through support. With write-through, updates to the data grid are written asynchronously to a backend data store – which could be HBase (as used by Facebook), Cassandra, a relational database such as MySQL, or any other data medium you choose for long-term storage, should you need that.

The memory system and the database - the external data store - work together. The in-memory solution is ideal for the realtime aspects, the events that affect now. The external data storage solution is designed to handle long-term data, for which speed is not as much of an issue.

A Discussion of Strengths

The key concept here is that event handling is the lever that can move the realtime analytics mountain. By providing a simple, scalable publisher/subscriber model, you simplify design; by using a platform that supports data partitioning, transactional updates, and write through capabilities, you gain scalability.

The data grid’s flexible query API means that events can literally react when data is available.

For a call center, for example, you want to immediately identify signals that show that the caller should be handled differently; imagine an ecommerce site that was able to determine immediately if a user was losing interest, and thus could respond appropriately, before the customer moves on.

With external processes and a long funnel for data, immediate-response capabilities are very difficult to implement, not just because of latency but because the data transformations tend to homogenize the data, instead of allowing rich expressions and flexible event types.

The data grid also has much richer support in terms of client applications. Instead of applications going through an API that focuses on a specific phase of the data’s life (for example, an API focused on HBase), you can focus on a generic API that can capture events at any point in their lifecycle, and from anywhere. An external monitoring process, then, can have the same immediate, partition-aware access to data that the integrated message-handling system does; adding features and analysis is just a matter of connecting a client to the data grid.

Here we have a quick demo that shows much of this in motion. We have a market analysis application, deployed into GigaSpaces XAP via our new Cloud deployment too, Cloudify; it uses an event-driven system to display realtime data, with a write-through to Cassandra on the back-end. The design is very simple, and demonstrates the principles we've discussed here - and can scale up and down depending on demand.

Final words

As real-time-analytics gets into mainstream application design, it becomes important to draw a blueprint on how to build this sort of system without re-inventing the wheel.

Todd Hoff (HighScalability) and Alex Himel (Facebook) provided a fairly detailed description on their solution and even more importantly they even shared the rationales that made them do things in certain ways.

One main difference in assumptions that lead to the different implementation strategies are in reliable memory for event processing, and in the use of passive data storage.

Another difference is that we had to to think of the solution as an easily cloneable solution and therefore a lot of attention was put on the simplicity of the runtime, packaging and management of the solution.

Yet another difference is that we couldn’t decide on a specific database as there isn’t a "one size fits all" solution – for certain customers, SQL would still be preferred choice and the fact that we can buffer the write to the database gives them more headroom while still allowing them to scale on writes.

I hope that this would lead to constructive dialogue on the various tradeoffs which will serve the entire industry...

References

Video: Realtime Analytics for Big Data: A Facebook Case Study

Martin Fowler: Bliki: CQRS

2011-07-14T13:33:00Z

CQRS stands for Command Query Responsibility Segregation. It's a pattern that I first heard described by Greg Young. At its heart is a simple notion that you can use a different model to update information than the model you use to read information. This simple notion leads to some profound consequences for the design of information systems.

The mainstream approach people use for interacting with an information system is to treat it as a CRUD datastore. By this I mean that we have mental model of some record structure where we can create new records, read records, update existing records, and delete records when we're done with them. In the simplest case, our interactions are all about storing and retrieving these records.

As our needs become more sophisticated we steadily move away from that model. We may want to look at the information in a different way to the record store, perhaps collapsing multiple records into one, or forming virtual records by combining information for different places. On the update side we may find validation rules that only allow certain combinations of data to be stored, or may even infer data to be stored that's different from that we provide.

As this occurs we begin to see multiple representations of information. When users interact with the information they use various presentations of this information, each of which is a different representation. Developers typically build their own conceptual model which they use to manipulate the core elements of the model. If you're using a Domain Model, then this is usually the conceptual representation of the domain. You typically also make the persistent storage as close to the conceptual model as you can.

This structure of multiple layers of representation can get quite complicated, but when people do this they still resolve it down to a single conceptual representation which acts as a conceptual integration point between all the presentations.

The change that CQRS introduces is to split that conceptual model into separate models for update and display, which it refers to as Command and Query respectively following the vocabulary of CommandQuerySeparation. The rationale is that for many problems, particularly in more complicated domains, having the same conceptual model for commands and queries leads to a more complex model that does neither well.

By separate models we most commonly mean different object models, probably running in different logical processes, perhaps on separate hardware. A web example would see a user looking at a web page that's rendered using the query model. If they initiate a change that change is routed to the separate command model for processing, the resulting change is communicated to the query model to render the updated state.

There's room for considerable variation here. The in-memory models may share the same database, in which case the database acts as the communication between the two models. However they may also use separate databases, effectively making the query-side's database by a real-time ReportingDatabase. In this case there needs to be some communication mechanism between the two models or their databases.

The two models might not be separate object models, it could be that the same objects have different interfaces for their command side and their query side, rather like views in relational databases. But usually when I hear of CQRS, they are clearly separate models.

CQRS naturally fits with some other architectural patterns.

As we move away from a single representation that we interact with via CRUD, we can easily move to a task-based UI.
Interacting with the command-model naturally falls into commands or events, which meshes well with Event Sourcing.
Having separate models raises questions about how hard to keep those models consistent, which raises the likelihood of using eventual consistency.
For many domains, much of the logic is needed when you're updating, so it may make sense to use EagerReadDerivation to simplify your query-side models.
CQRS is suited to complex domains, the kind that also benefit from Domain-Driven Design.

When to use it

Like any pattern, CQRS is useful in some places, but not in others. Many systems do fit a CRUD mental model, and so should be done in that style. CQRS is a significant mental leap for all concerned, so shouldn't be tackled unless the benefit is worth the jump.

In particular CQRS should only be used on specific portions of a system (a Bounded Context in DDD lingo) and not the the system as a whole. In this way of thinking, each Bounded Context needs its own decisions on how it should be modeled.

So far I see benefits in two directions. Firstly is handling complexity - a complex domain may be easier to tackle by using CQRS. I do have to hedge this, usually there's enough overlap between the command and query sides that sharing a model is easier. Each domain has different characteristics.

The other main benefit is in handling high performance applications. CQRS allows you to separate the load from reads and writes allowing you to scale each independently. If your application sees a big disparity between reads and writes this is very handy. Even without that, you can apply different optimization strategies to the two sides. An example of this is using different database access techniques for read and update.

If your domain isn't suited to CQRS, but you have demanding queries that add complexity or performance problems, remember that you can still use a ReportingDatabase. CQRS uses a separate model for all queries. With a reporting database you still use your main system for most queries, but offload the more demanding ones to the reporting database.

It's also true that we haven't seen enough uses of CQRS in the field yet to be confident that we understand its pros and cons. So while CQRS is a pattern I'd certainly want in my toolbox, I wouldn't keep it at the top.

Martin Fowler: The LMAX Architecture

2011-07-12T13:43:00Z

When I was at QCon London last year, there was much buzz around a talk about a new retail trading system. The thing that got people’s attention was its approach to achieving its high performance needs - reaching 6 million TPS. It does this by running all its business logic on a single JVM thread, getting high speed without all the complications of concurrent programming. This article describes the architecture they use to pull this off.

Interop Bindings for WCF released in codeplex

2011-06-14T01:06:40Z

I am happy to announce that one of the projects in which Microsoft and Tellago have been collaborating together during the last few months was released as part of wcf.codeplex.com. The primary aim of this “WCF Interop Bindings” project is to simplify the work needed for making WCF services interoperable with other service stacks like Oracle WebLogic, IBM Websphere, Metro or Apache Axis.

The main goal behind web services has always been to enable communication between heterogeneous systems. However, we all know that archiving such level of interoperability in the context of web services and WS-* is really complicated and requires good knowledge of the different message stacks to generate compatible messages at the wire level.

For that reason, the WCF team came up with this interesting idea of providing a set of pre-defined bindings with a limited set of settings that you can use out of the box for supporting the most common interop scenarios with the mentioned stacks using protocols like WS-Security, WS-SecureConversation or WS-ReliableMessaging.

Bindings provided out of the box in the current bits

The current bits provides a set of custom binding implementations for Oracle WebLogic, IBM Websphere, Metro and Apache Axis. The name of those bindings basically matches the message stack you want to use. For example, the following configuration illustrates how a service can be configured to be compatible at wire level with Oracle WebLogic using Username over certificates as the security scenario.

<configuration>

  <system.serviceModel>

    <services>

      <service name="Microsoft.ServiceModel.Interop.Samples.HelloWorldService" behaviorConfiguration="helloWorld">

        <endpoint address="Username_WebLogic" binding="webLogicBinding" bindingConfiguration="helloWorld_UsernameOverCertificate"

contract="Microsoft.ServiceModel.Interop.Samples.IHelloWorld"/>

        <endpoint address="Mex" binding="mexHttpBinding" contract="IMetadataExchange"/>

      </service>

    </services>

    <bindings>

      <webLogicBinding>

        <binding name="helloWorld_UsernameOverCertificate">

          <security mode="UserNameOverCertificate"/>

        </binding>

      </webLogicBinding>

    </bindings>

    <behaviors>

      <serviceBehaviors>

        <behavior name="helloWorld">

          <serviceMetadata httpGetEnabled="true"/>

          <serviceDebug httpHelpPageEnabled="true"/>

          <serviceCredentials>

            <serviceCertificate

              storeLocation="LocalMachine"

              storeName="TrustedPeople"

              x509FindType="FindBySubjectDistinguishedName"

              findValue="CN=HelloWorldService"/>

          </serviceCredentials>

        </behavior>

      </serviceBehaviors>

    </behaviors>

    <extensions>

      <bindingExtensions>

        <add name="webLogicBinding" type="Microsoft.ServiceModel.Interop.WebLogic.Configuration.WebLogicBindingCollectionElement,

 Microsoft.ServiceModel.Interop, Version=1.0.0.0, Culture=neutral, PublicKeyToken=4fc38efee625237e"/>

      </bindingExtensions>

    </extensions>

  </system.serviceModel>

</configuration>

Visual Studio 2010 Integration

The binding implementations are not the only thing you get out of the box with the current bits. The WCF team also wanted to provide developers with a rich experience for configuring and creating the services with one of the pre-defined interop scenarios. Having that in mind, this project also integrates in Visual Studio 2010 by providing a set of project templates for creating WCF services for an specific scenario and a configuration wizard for generating the binding configuration.

As you can see in the image above, two new project templates are automatically registered under the "WCF" category that you can use to create a new service configured with one of the provided interop bindings. You can use the "WCF Interoperable Service Library" template for creating a new WCF service as part of the service library, or "WCF Interoperable Service Application" for creating a new WCF service as part of a Web Application. The templates will automatically launch a configuration wizard for selecting the desired interop scenario as it is shown in the images bellow.

In the wizard you can pick one of the existing interop scenarios, and specific settings for those scenarios like Security or Reliable Messaging to give an example. At the end of wizard, the configuration file associated to the WCF project will be automatically updated with the chosen binding and settings.

Enjoy!!

The trouble with APIs

2011-06-09T13:45:20Z

I posted a couple of thoughts on Twitter and a few comments elsewhere recently as a results of a discussion on George Reese's blog and William's followup. I thought I'd repost these thoughts here with annotation as it seems to be evolving an interesting discussion.

1. REST APIs are a case of good intentions run amok. The point of REST was to *eliminate* specific APIs. Not cause their proliferation.

2. When someone says you shouldn't document a REST API, they're saying you shouldn't have an API. Not that you shouldn't document.

I mean by this, Network APIs. I also mean APIs that aren't re-implementable across sites. The point is that the uniform interface isn't just HTTP - it's a small number of evolving media types, and the goal is to enable sharing of information at a global scale. Your pet JSON format may not be the way to do this.

3. That said, RESTians saying you shouldn't have an API are somewhat vacuous because they offer no good alternative. Custom media types ain't.

4. Thus, IMO, until we have a workable generic media type or two for the 80% case of "REST APIs", we're just growing a frustrating mess.

In other words, I completely recognize that crafting a media type is a bit heady, and even still a custom media type for generic use is quite hard. With a REST API at least you get the benefits of HTTP GET (to some degree). It does the job, if all you want is a trendy vaguely hopeful way of building something for the long term without getting too bogged down in philosophy. I get it.

What I would like to see is a generic media type (above & beyond a "format" like JSON) to cover the 80/20 cases. This is seemingly contrary to where many RESTheads are at - they prefer often prefer opinionated media types, media types that "say something".

I'm actually asking for that too, I think -- say something about the programming model that breaks people's habits away from RPCs. Find a programming model that doesn't orient all "write-side" communication in terms of a flat list of actions, and into something that can be extended & shared across the web via hyperlinks and URIs. Hierarchical statecharts is my stab at this. There may be others.

5. From William's blog (with a couple tweaks)....

I've sort of half given up on REST advocacy because it's pretty impossible these days. People want to ship APIs that will be valid for the next next 6 months, not build systems that will evolve over a decade or more.

Jan Algermissen's comment about how when you've documented an API, you're not really doing REST, is actually spot on, fundamentally, but is met with ridicule. I can completely understand how vacuous the comment sounds if you just want to ship an API nownownow, are being short-term practically minded, or are just playing buzzword bingo with architectural styles. But if we actually want to leverage the benefits of the style, we'd work on the core issue of having a generic media type or two for these sorts of use cases.

If one is thinking of "how to methods map to my URI structures", we've basically lost the plot as to why we have HTTP, URIs and media types in the first place. The goal is to *eliminate* the API as we normally think of it. (It's not about a library of calls over a network).

Five points. The consequences of REST changes some rules:

1. The assumption can't be that the end user of your information service is going to be a programmer. Building an API bakes that assumption in how you expose your information or controls to manipulate it. The assumption rather should be that "this can be any agent of any sort". If your goal really is "I want a programmer to learn my API and program against this", then there's really no reason to use REST other than that it's trendy (and HTTP GET).

2. The assumption can't be that you will be the only site implementing your interface. The whole point of the Web is sharing information - publishing an API bakes in the assumption that YOU, the publisher, are in control. It is the consumer, on the other hand - the agent itself, that's in control of how/what the information is consumed or manipulated.

3. The semantics of the interface that need to be documented are the media type that you use to describe your data and links. The structure of your URIs really shouldn't be a part of said semantics.

4. But doing this in a custom media type implies #2, that you're the only person implementing your interface. Of course, every media type has to start somewhere, but it really doesn't help matters unless you understand that others may be *implementing* your interface, not just *talking* to it. Facebook's Open Graph, for example, is an example of a custom overlay media type that takes great pains to be generic in either direction.

5. All of the above is really heady stuff that is above most developer's pay grades, so it won't ever happen. The best we can hope for is some person/organization to step up with a generic media type or two + guidelines that help describe and document a Web of Data interface for accessibility by programmers. Call it a description language, if you'd like. Some RESTians don't like the idea of a description language. I think it's an essential 80/20 piece if developers are going to actually *use* the architecture the way it was meant to be used.

Currently (as I mentioned in my WS-REST keynote) I think some kind of hierarchical lifecycle/state machine might be the right abstraction for such an interface. I'd like to demonstrate this if I ever got past my day job sucking up most of my time. (But I may have an opportunity to work on this there soon.)

SATURN Conference 2011 Presentations are Available

2011-06-07T15:19:00Z

The SATURN Conference 2011 presentations are available for download.

Links to the downloads and descriptions of the presentations are available here.

The Secret Sauce

2011-05-29T18:50:00Z

Last time, I was talking about what I consider to be the general lack of a crisis in software development. And it got me thinking - if there is no crisis in software development, no inherent flaws in our tools or our methods, then there must somehow be a way to convey the appropriate use of these tools and methods in order that everybody could get it right every time.

Ah ha, I thought. A book. A book entitled "The Secret Sauce". A book that will make ... [more ->]

Zynga's Z Cloud - Scale Fast or Fail Fast by Merging Private and Public Clouds

2011-05-19T16:06:44Z

Release early and often. A/B testing. Creating a landing page and buying ads on AdSense. All are ways of providing quick feedback in order to validate an idea. If you are like Zynga, with 250 million active users a month, how do you cost effectively prove out a game that could flop or get 90 million users (like CityVille) in an instant?

Zynga handles this problem inlle an innovative way, by inverting the typical cloud burst scenario that has excess traffic flowing from a datacenter to a cloud, to having a game start in the cloud and then moving to the datacenter once the game has proved popular enough to keep.

This process is nicely described by Charles Babcock in Lessons From FarmVille: How Zynga Uses The Cloud, in an interview with Allan Leinwand, CTO of infrastructure engineering at Zynga.

When paired down to its essence, Zynga's strategy goes something like this:

Links for 2009-12-16 [del.icio.us]

2009-12-17T00:00:00-08:00

The Common Principles Behind the NOSQL Alternatives

Links for 2009-12-13 [del.icio.us]

2009-12-14T00:00:00-08:00

Diagnostics in Windows Azure

2009-12-08T13:11:38Z

Microsoft introduced a set of classes supporting diagnostics monitoring in the November 2009 Azure Services SDK. The classes are composed into two namespaces:

The Diagnostics namespace comprises classes for the configuration and collection of diagnostic information. The Diagnostics.Management namespace comprises classes supporting local and remote management of the collection of diagnostics information and its transfer to persistent storage in Azure Tables and Blobs. This post is focused on the Diagnostics namespace and a subsequent post will consider the Diagnostics.Management namespace.

MSDN has a good section on Implementing Windows Azure Diagnostics for Hosted Services that includes the following sections:

Matthew Kerner gave an excellent presentation on Windows Azure Monitoring, Logging, and Management APIs at PDC 2009 – he has posted his samples here. The November 2009 Update of the Windows Azure Platform Training Kit has a good hands-on-lab Windows Azure: Deploying and Monitoring Applications in Windows Azure with Example 3 being Monitoring Applications in Windows Azure. Sumit Mehrotra, from the Azure team, has a couple of posts – here and here – looking at Azure Diagnostics.

UPDATE 12/1/2010: This post describes some configuration changes to Azure Diagnostics implemented in the Azure SDK v1.3 release.

UPDATE 5/6/2011. I uploaded a Windows Azure Diagnostics sample to the MSDN Code Samples Gallery that demonstrates the features described in this post.

Diagnostics Information

Azure Diagnostics captures the following information:

Directories – file-based data buffers (including IIS Logs, failed request logs and crash dumps)
Diagnostic infrastructure logs – logs the diagnostics infrastructure itself
Logs – Windows Azure logs
Performance counters – Windows Performance Counters
Windows Event Logs

Note that developers can create and configure their own user-defined file-based data buffers.

The diagnostic information is captured in data buffers in the local storage of each instance. Diagnostic infrastructure logs, IIS Logs, (Windows Azure) Logs are stored by default – the others must be initiated programmatically. Once captured, diagnostic information can be transferred either by schedule or on-demand to persistent Azure Storage. Directories (file-based data buffers) are stored in Blobs and all other diagnostics information is is stored in Tables.

The data buffers containing the diagnostics information are configured through several data-buffer configuration classes. Diagnostics monitoring is configured through the DiagnosticMonitor and DiagnosticMonitorConfiguration classes.

Data Buffer Configuration

The data buffers are configured through a simple set of classes which persist the configuration in an Azure blob named wad-control-container. The configuration for each instance of each role is persisted in an individual blob in the container named with a combination of the deployment Id, role name and instance Id of the current role instance. For example:

deployment(301)/WebRole1/deployment(301).NovemberStorageClient.WebRole1.0

The diagnostics configuration classes can be invoked either from the running instance of the Azure Service or through, for example, a remote desktop application interacting with the configuration through the diagnostics management and diagnostics configuration classes. Every Azure instance hosts a MonAgentHost.exe executable to manage the Azure diagnostics configuration and management process.

DiagnosticDataBufferConfiguration

DiagnosticDataBufferConfiguration is an abstract base class for the various diagnostics configuration classes representing data buffers. It is declared:

public abstract class DiagnosticDataBufferConfiguration {
    // Constructors
    protected DiagnosticDataBufferConfiguration();

// Properties
    public Int32 BufferQuotaInMB { get; set; }
    public TimeSpan ScheduledTransferPeriod { get; set; }
}

The DiagnosticsDataBufferConfiguration properties are exposed through its derived classes each of which represents a data buffer containing a specific type of diagnostics information. These derived classes are:

BufferQuotaInMB specifies the maximum quota for an data buffer. The maximum overall quota is specified by DiagnosticMonitorConfiguration.OverallQuotaInMB. By default the overall quota is 4GB and there is no maximum quota specified for the individual buffers. (When an individual quota is reached I believe the policy is to delete the oldest records.) ScheduledTransferPeriod is the TimeSpan in minutes between scheduled transfers of the data buffer to Azure Storage.

BasicLogsBufferConfiguration

The BasicLogsBufferConfiguration class is derived from DiagnosticDataBufferConfiguration and supports configuration of (basic windows) logs. It is declared:

public class BasicLogsBufferConfiguration : DiagnosticDataBufferConfiguration {
    // Constructors
    public BasicLogsBufferConfiguration();

// Properties
    public LogLevel ScheduledTransferLogLevelFilter { get; set; }
}

The BasicLogsBufferConfiguration adds a ScheduledTransferLogLevelFilter to the BufferQuotaInMB and ScheduledTransferPeriod of the base DiagnosticDataBufferConfiguration class. ScheduledTransferLogLevelFilter specifies a filter level to limit the log information persisted to Azure Tables. LogLevel is the following enumeration:

public enum LogLevel {
    Undefined,
    Critical,
    Error,
    Warning,
    Information,
    Verbose
}

Note that when a filter level, here or another data buffer class, is set to a value from this enumeration it implicitly includes any levels earlier in the enumeration. Undefined has the same effect as Verbose.

The Logs are persisted in an Azure Table named:

WadLogsTable

DirectoriesBufferConfiguration

The DirectoriesBufferConfiguration class is derived from DiagnosticsDataBufferConfiguration and supports configuration of file-based data buffers such as IIS Logs, failed request logs, crash dumps, and user-defined diagnostics. DirectoriesBufferConfiguration is declared:

public class DirectoriesBufferConfiguration : DiagnosticDataBufferConfiguration {
    // Constructors
    public DirectoriesBufferConfiguration();

// Properties
    public IList<DirectoryConfiguration> DataSources { get; }
}

The DirectoriesBufferConfiguration adds a list of DataSources to the BufferQuotaInMB and ScheduledTransferPeriod of the base DiagnosticDataBufferConfiguration class. The DataSources are instances of the DirectoryConfiguration class declared:

public class DirectoryConfiguration {
    // Constructors
    public DirectoryConfiguration();

// Properties
    public String Container { get; set; }
    public Int32 DirectoryQuotaInMB { get; set; }
    public String Path { get; set; }
}

Each directory has its own quota, DirectoryQuotaInMB, which is 1GB by default. Path specifies the absolute path of the directory containing the file-based data buffer such as the Crash Dumps (and eventually the IIS Logs). that contains the log Container names the Azure Blob container into which the diagnostic log will be transferred for persistence. The Azure provided file-based data buffers are transferred into the following containers by default:

wad-iis-failedreqlogfiles
wad-iis-logfiles
wad-crash-dumps

Note that the IIS Failed Request Logs require that IIS be configured to capture the information. The feature is not turned on by default for performance reasons. The additional configuration comprises an additional section in the web.config file and is documented here.

Note that although this section implies that IIS Logs are configured like other Directories this does not appear to be true. Currently, the scheduled transfer of IIS Logs is configured through BasicLogsBufferConfiguration rather than DirectoriesBufferConfiguration. Although not confirming this, Matthew Kerner’s presentation states: Coming Soon: IIS Logs generated in role’s local data directory. I suspect that once this change happens IIS Logs will be configured through DirectoriesBufferConfiguration rather than BasicLogsBufferConfiguration.

UPDATE 3/25/2010: IIS Logs are now configured through DirectoriesBufferConfiguration.

PerformanceCountersBufferConfiguration

The PerformanceCountersBufferConfiguration class is derived from DiagnosticsDataBufferConfiguration and supports configuration of performance counters data buffers. PerformanceCountersBufferConfiguration is declared:

public class PerformanceCountersBufferConfiguration : DiagnosticDataBufferConfiguration {
    // Constructors
    public PerformanceCountersBufferConfiguration();

// Properties
    public IList<PerformanceCounterConfiguration> DataSources { get; }
}

The PerformanceCountersBufferConfiguration adds a list of DataSources to the BufferQuotaInMB and ScheduledTransferPeriod of the base DiagnosticDataBufferConfiguration class. The DataSources are instances of the PerformanceCounterConfiguration class declared:

public class PerformanceCounterConfiguration {
    // Constructors
    public PerformanceCounterConfiguration();

// Properties
    public String CounterSpecifier { get; set; }
    public TimeSpan SampleRate { get; set; }
}

CounterSpecifier specifies a Windows Performance Counter to be captured. This specification is in special syntax defined for Windows Performance Counters. An example of this syntax is:

@”Processor(*)% Processor Time”

This captures the % of processor time for all processors. Note that the names of performance counters on a development machine can be found by typing the following at a command prompt:

typeperf.exe /q

SampleRate is a TimeSpan specifying when the performance counter is measured. The SampleRate is rounded to the nearest second.

The Windows Performance Counters are persisted in an Azure Table named:

WadPerformanceCountersTable

Note that each measurement of a performance counter is stored as one entity in the table.

WindowsEventLogsBufferConfiguration

The WindowsEventLogsBufferConfiguration class is derived from DiagnosticsDataBufferConfiguration and supports configuration of Windows Event Log data buffers. WindowsEventLogsBufferConfiguration is declared:

public class WindowsEventLogsBufferConfiguration : DiagnosticDataBufferConfiguration {
    // Constructors
    public WindowsEventLogsBufferConfiguration();

// Properties
    public IList<String> DataSources { get; }
    public LogLevel ScheduledTransferLogLevelFilter { get; set; }
}

The WindowsEventLogsBufferConfiguration adds a list of DataSources and a ScheduledTransferLogLevelFilter to the BufferQuotaInMB and ScheduledTransferPeriod of the base DiagnosticDataBufferConfiguration class. ScheduledTransferLogLevelFilter specifies a filter level taken to limit the log information persisted to Azure Tables. The filter level comes from the LogLevel enumeration. The DataSources are Strings in a special syntax defined for specifying how to consume events from Windows Event Logs. This syntax allows all events in an Event Log to be retrieved or only a filtered list of events.

The Windows Event Logs are persisted in an Azure Table named:

WadWindowsEventLogsTable

Crash Dumps

There is one type of diagnostics data that is not configured through the normal data buffer configuration technique described here – crash dumps. These are initiated through the CrashDumps class declared:

public static class CrashDumps {
    // Methods
    public static void EnableCollection(Boolean enableFullDumps);
    public static void EnableCollectionToDirectory(String directory, Boolean enableFullDumps);
}

EnableCollection() enables either full or mini crash dumps. This data is placed in the CrashDumps directory in the same location as the other file-based Directories. EnableCollectionToDirectory() allows the specification of an alternative directory in local storage.

Note that ASP.Net handles almost all exceptions so that it is almost impossible to create a crash dump in a web role.

Diagnostics Monitoring

Diagnostics monitoring is supported by two classes: DiagnosticMonitor which controls the diagnostics monitoring process; and DiagnosticMonitorConfiguration which associates the data buffer configuration with the diagnostics monitoring process.

DiagnosticMonitorConfiguration

DiagnosticMonitorConfiguration associates the data buffer configuration with the diagnostics monitoring process. DiagnosticMonitorConfiguration is declared:

public class DiagnosticMonitorConfiguration {
    // Constructors
    public DiagnosticMonitorConfiguration();

// Properties
    public TimeSpan ConfigurationChangePollInterval { get; set; }
    public BasicLogsBufferConfiguration DiagnosticInfrastructureLogs { get; set; }
    public DirectoriesBufferConfiguration Directories { get; set; }
    public BasicLogsBufferConfiguration Logs { get; set; }
    public Int32 OverallQuotaInMB { get; set; }
    public PerformanceCountersBufferConfiguration PerformanceCounters { get; set; }
    public WindowsEventLogsBufferConfiguration WindowsEventLog { get; set; }
}

DiagnosticInfrastructureLogs, Directories, Logs, PerformanceCounters and WindowsEventLog provide access to an instance of each of the five Azure Diagnostics data buffers. ConfigurationChangePollInterval specifies the TimeSpan dictating when the diagnostics monitor polls for configuration changes. OverallQuotaInMB specifies the total amount of local storage allocated to all Azure Diagnostics data buffers.

DiagnosticMonitor

DiagnosticMonitor controls the Azure Diagnostics process by providing methods to start and stop diagnostics monitoring and to transfer the diagnostics logs to persistent Azure Storage. DiagnosticMonitor is declared:

public class DiagnosticMonitor {
    // Properties
    public static Boolean AllowInsecureRemoteConnections { get; set; }
    public String LocalDataDirectory { get; }

// Methods
    public static DiagnosticMonitorConfiguration GetDefaultInitialConfiguration();
    public void Shutdown();
    public static DiagnosticMonitor Start(String diagnosticsStorageAccountConfigurationSettingName,
       DiagnosticMonitorConfiguration initialConfiguration);
    public static DiagnosticMonitor Start(CloudStorageAccount storageAccount,
       DiagnosticMonitorConfiguration initialConfiguration);
    public static DiagnosticMonitor Start(String diagnosticsStorageAccountConfigurationSettingName);
    public void UpdateStorageAccount(CloudStorageAccount storageAccount);
}

A DiagnosticMonitor object is created using one of the static Start() methods parameterized by various combinations of CloudStorageAccount, DiagnosticMonitorConfiguration and storage account configuration setting name. The Start() methods actually starts the Monitoring Agent Host, MonAgentHost.exe, which provides for out-of-service diagnostics monitoring and management. Either a diagnosticsStorageAccountConfigurationSettingName String or a CloudStorageAccount must be supplied to provide the authentication required to access Azure Storage both to save the diagnostics configuration information and to persist the Azure Diagnostics data buffers. The Start() methods can be invoked either with a supplied initial DiagnosticMonitorConfiguration or using a default initial DiagnosticMonitorConfiguration. A common pattern is to retrieve a copy of the default initial configuration using the static GetDefaultInitialConfiguration() method, modify this configuration, and then use the modified configuration when Start() is invoked.

The basic example is:

DiagnosticMonitor diagnosticMonitor = DiagnosticMonitor.Start(“DiagnosticsConnectionString”);

This starts the Azure Diagnostics monitoring agent host and uses the Azure Storage configuration information identified by the name, DiagnosticsConnectionString, in the Azure Service configuration file. This name is merely a convention and is not mandatory.

A more sophisticated example is:

protected void StartDiagnosticMonitor()
{
    DiagnosticMonitorConfiguration diagnosticMonitorConfiguration =
       DiagnosticMonitor.GetDefaultInitialConfiguration();
    diagnosticMonitorConfiguration.Directories.ScheduledTransferPeriod = TimeSpan.FromMinutes(5.0);
    diagnosticMonitorConfiguration.Logs.ScheduledTransferPeriod = TimeSpan.FromMinutes(1.0);
    diagnosticMonitorConfiguration.WindowsEventLog.ScheduledTransferPeriod = TimeSpan.FromMinutes(5.0);
    diagnosticMonitorConfiguration.PerformanceCounters.ScheduledTransferPeriod = TimeSpan.FromMinutes(1.0);

diagnosticMonitorConfiguration.WindowsEventLog.DataSources.Add(“Application!*”);
    diagnosticMonitorConfiguration.WindowsEventLog.DataSources.Add(“System!*”);

PerformanceCounterConfiguration performanceCounterConfiguration = new PerformanceCounterConfiguration();
    performanceCounterConfiguration.CounterSpecifier = @”Processor(*)% Processor Time”;
    performanceCounterConfiguration.SampleRate = System.TimeSpan.FromSeconds(1.0);
    diagnosticMonitorConfiguration.PerformanceCounters.DataSources.Add(performanceCounterConfiguration);

CloudStorageAccount cloudStorageAccount = CloudStorageAccount.DevelopmentStorageAccount;
    DiagnosticMonitor diagnosticMonitor = DiagnosticMonitor.Start(
       cloudStorageAccount, diagnosticMonitorConfiguration);
}

This example retrieves the DiagnosticMonitor default initial configuration. It then specifies in this configuration various transfer periods to persistent Azure Storage for Directories, Logs, Windows Event Logs. and Windows Performance Counters. It adds Event Log data sources for the Windows Application and System Event Logs to the configuration. Then it configures the collection of a Windows Performance Counter for the %processor time for all processors with data collection taking place every second. Finally, it retrieves the built-in DevelopmentStorageAccount and starts the Monitoring Agent Host by invoking the DiagnosticMonitor Start() method.

Note. I got an error in the Start() method when I tried to use a CloudStorageAccount created as follows:

CloudStorageAccount cloudStorageAccount = CloudStorageAccount.FromConfigurationSetting(“DiagnosticsConnectionString”);

This happened even though the Azure Service configuration file had the following:

<Setting name=”DiagnosticsConnectionString” value=”UseDevelopmentStorage=true” />

and CloudStorageAccount.SetConfigurationSettingPublisher() had been invoked.

This problem could easily be due to user error.

The AllowInsecureRemoteConnections property specifies whether or not the Monitoring Agent Host can communicate with Azure Storage using http instead of https. LocalDataDirectory gets the full path in local storage where the Azure Diagnostics data files and configuration are stored.

The Shutdown() method stops the Monitoring Agent Host and stops the collection of diagnostics data. UpdateStorageAccount() allows new CloudStorageAccount information to be supplied to the Monitoring Agent Host.

Tracing to the Log

Azure has a special trace listener, DiagnosticMonitorTraceListener, that writes Trace information to the Azure log files. An Azure role is configured to use the DiagnosticMonitorTraceListener by placing the following in the web.config or app.config file as appropriate:

<system.diagnostics>
    <trace>
        <listeners>
            <add type=”Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorTraceListener,
                 Microsoft.WindowsAzure.Diagnostics, Version=1.0.0.0, Culture=neutral,
                 PublicKeyToken=31bf3856ad364e35″ name=”AzureDiagnostics”>
                <filter type=”" />
            </add>
        </listeners>
    </trace>
</system.diagnostics>

Tracing is then performed by using static methods in the System.Diagnostics.Trace class. For example, TraceInformation() writes an informational trace to the Azure logs while TraceWarning() writes a warning trace to the event logs. These log levels can later be used to filter messages during analysis of the trace output. The Trace class also provides basic Write() and WriteIf() methods, the latter writing a trace depending on the Boolean value of a parameter. For example, the following writes the instance Id of the current instance as an informational trace to the Azure logs:

protected void WriteTraceInformation()
{
String roleInstance = RoleEnvironment.CurrentRoleInstance.Id;
System.Diagnostics.Trace.TraceInformation(“Role instance: ” + roleInstance);
}

UPDATE 10/13/2010 – A post on the Dot Net Solutions blog warns that Trace.WriteLine() issues only Verbose messages.

Accessing Logs

The Azure Diagnostics logs are accessible once they have been persisted to Azure Storage. They can then be viewed using an application like Azure Storage Explorer or Cloud Storage Studio. They can also be downloaded from Azure Storage to a local system for further analysis.

UPDATE 3/25/2010

Confirmed that IIS Logs are now configured through DirectoriesBufferConfiguration.

Technorati Tags: Azure,Azure Diagnostics,Azure Storage

Time Has Changed

2011-05-04T21:02:43Z

See TechCrunch: "comScore: Facebook Now Serves One Third Of Online Ads In U.S."

Captcha On mensa.org

2011-05-01T20:03:50Z

What I don’t like about WIF’s Claims-based Authorization

2011-04-30T06:45:09Z

In my last post I wrote about what I like about WIF’s proposed approach to authorization – I also said that I definitely would build upon that infrastructure for my own systems. But implementing such a system is a little harder as it could be. Here’s why (and that’s purely my perspective):

First of all WIF’s authorization comes in two “modes”

Per-request authorization.
When an ASP.NET/WCF request comes in, the registered authorization manager gets called. For SOAP the SOAP action gets passed in. For HTTP requests (ASP.NET, WCF REST) the URL and verb.
Imperative authorization
This happens when you explicitly call the claims authorization API from within your code. There you have full control over the values for action and resource.

In ASP.NET per-request authorization is optional (depends on if you have added the ClaimsAuthorizationHttpModule). In WCF you always get the per-request checks as soon as you register the authorization manager in configuration.

I personally prefer the imperative authorization because first of all I don’t believe in URL based authorization. Especially in the times of MVC and routing tables, URLs can be easily changed – but then you also have to adjust your authorization logic every time. Also – you typically need more knowledge than a simple “if user x is allowed to invoke operation x”.

One problem I have is, both the per-request calls as well as the standard WIF imperative authorization APIs wrap actions and resources in the same claim type. This makes it hard to distinguish between the two authorization modes in your authorization manager. But you typically need that feature to structure your authorization policy evaluation in a clean way.

The second problem (which is somehow related to the first one) is the standard API for interacting with the claims authorization manager. The API comes as an attribute (ClaimsPrincipalPermissionAttribute) as well as a class to use programmatically (ClaimsPrincipalPermission). Both only allow to pass in simple strings (which results in the wrapping with standard claim types mentioned earlier). Both throw a SecurityException when the check fails.

The attribute is a code access permission attribute (like PrincipalPermission). That means it will always be invoked regardless how you call the code. This may be exactly what you want, or not. In a unit testing situation (like an MVC controller) you typically want to test the logic in the function – not the security check.

The good news is, the WIF API is flexible enough that you can build your own infrastructure around their core. For my own projects I implemented the following extensions:

A way to invoke the registered claims authorization manager with more overloads, e.g. with different claim types or a complete AuthorizationContext.
A new CAS attribute (with the same calling semantics as the built-in one) with custom claim types.
A MVC authorization attribute with custom claim types.
A way to use branching – as opposed to catching a SecurityException.

I will post the code for these various extensions here – so stay tuned.

Use More VMs On Your Laptop

2011-04-15T17:14:00Z

Developers grossly underutilize virtualization on their development platforms

Stop Blaming the Customers - the Fault is on Amazon Web Services

2011-04-25T23:00:00Z

Almost as galling as the Amazon Web Services outage itself is a the litany of blog posts, such as this one and this one, that place the blame not on AWS for having a long failure and not communicating with its customers about it, but on AWS customers for not being better prepared for an outage. It's a tendency that displays a "blame the customer" mentality I've been seeing a lot lately. To understand why it's wrong one has to understand what actually happened and what claims AWS made about it services.

We covered the differences between availability zones and availability regions, and AWS' lack of communication, in our previous coverage. Now that the dust has settled, it's worth looking back at what happened. This timeline by Eric Kidd explains the series of events, and the various options different customers had. RightScale provides another good summary. What can we learn?

Sponsor

What Amazon Claims

Here's what AWS claims about Availability Zones:

Each availability zone runs on its own physically distinct, independent infrastructure, and is engineered to be highly reliable. Common points of failures like generators and cooling equipment are not shared across Availability Zones. Additionally, they are physically separate, such that even extremely uncommon disasters such as fires, tornados or flooding would only affect a single Availability Zone.

In other words, AWS claimed that by putting your data in different availability zones, within one region, was redundant. As far as AWS' customers were concerned, they didn't have a single point of failure.

Amazon Relational Database Service customers have the option of paying double the regular cost of the service for a multi-zone service: "When you run your DB Instance as a Multi-AZ deployment for enhanced data durability and availability, Amazon RDS provisions and maintains a standby in a different Availability Zone for automatic failover in the event of a scheduled or unplanned outage."

What Happened vs. What Was Supposed to Happen

The mass outage was due to Elastic Block Storage (EBS) service problems in a single Availability Zone. EBS instances can only live in one Availability Zone, but users should have been able to use their snapshots to create a new EBS instance in another availability zone. RDS depends on EBS, but RDS customers paying for multi-zone service should have had their databases failed-over to another zone automatically.

However, the "control plane" for creating new EBS instances suffered congestion, preventing any ability to failover either manually or automatically. The current assumption is that it was overloaded by customers who's initial EBS instance failed. Kidd calls this a "bank run."

The important thing here is that there actually was, unbeknownst to AWS customers, a single point of failure across zones: the control plane. This made AWS unable to fulfill its own failover promises. In fact, RDS customers ended up in worse shape than many others - it took over 14 hours to get many of their databases moved over, longer than those that were able to failover manually.

Multi-Region Multi-Vendor Deployments

So why not place applications in multiple regions, just to be safe? It's not that simple. First, AWS charges more for transfers between regions. But more importantly, it's technologically more complex. Amazon Machine Instances (AMIs) can't just been moved from one region to another. Justin Santa Barbara writes "The different regions have different features available, different AMI ids, I think reserved instances can't be moved between datacenters - in reality failover between regions is not realistic."

Barbara writes that it may actually be easier to failover to an entirely separate cloud than to use regions as failover. I'm not sure that's the case, but regional failover is certainly complicated. And based on the claims made about Availability Zones, would have seemed unnecessary before last week. After all, if each data center in the availability region is a discrete entity insulated from the failures of each other data center, then why would it be necessary add yet another data center in another region? Especially if doing so adds great expense?

Chris M Evans recommends using multiple cloud providers. To his credit, he recommended this even before the AWS outage (one of the things that bothers me about the blame the customer crowd is that their wisdom about what customers should have done comes entirely after the fact). Again, however, this adds additional complexity - and with that additional complexity, additional costs and additional risks. To many customers it seemed natural to just live with having multiple Availability Zones instead of multiple providers.

Even BigDoor CEO Keith Smith, concluded his widely cited piece on Amazon's failure to communicate with customers by writing:

We can spend cycles designing and building technical belts and suspenders that will help us avoid a massive failure like this in the future, or we can continue to rely on a single huge partner and also continue our break-neck pace of iteration and product development.
I can't tell you today which option we will choose. But I'm sure it will be the question on the mind of many startups across the country.

George Reese of enStratus wrote for O'Reilly Media: "In short, if your systems failed in the Amazon cloud this week, it wasn't Amazon's fault. You either deemed an outage of this nature an acceptable risk or you failed to design for Amazon's cloud computing model."

That misses the point. Accepting a certain amount of downtime is one thing, accepting 14 hours of downtime when you've already paid extra for redundancy is another.

Yes, customers accept a certain amount of risk, but that doesn't make it their fault when Amazon screws up.

Why Didn't Some Sites, Like SmugMug and Twilio, Go Down?

What about the companies that had the good fortune to avoid outages? Aren't they evidence that it's the customers' fault for not setting things up right? Not really. Both Twilio and SmugMug boast about their "design for failure" but the important thing is that neither company relied on EBS. Had these companies been dependent on EBS, they likely would have suffered a similar fate.

What About Netflix?

What about Netflix? Netflix, as documented by Adrian Cockcroft, does use EBS.

Kidd writes about Netflix:

Run in 3 AZs, at no more than 60% capacity in each. This is the approach taken by Netflix, which sailed through this outage without no known downtime. If a single AZ fails, then the remaining two zones will be at 90% capacity. And because the extra capacity is running at all times, Netflix doesn't need to launch new instances in the middle of a "bank run."

It's not clear how much Netflix uses EBS, but Cockcroft gave a presentation saying Netflix avoids it. This tweet indicates that Netflix is more reliant on S3, SimpleDB and Apache Cassandra than on EBS, but Cockcroft did note that the company was having EBS trouble during the outage.

It's also worth noting that Cockcroft tweeted that Netflix only runs out of one region.

It's The Customers' Fault Because They Shouldn't Have Been Using EBS in the First Place

I love this argument - that it's customers' fault for using EBS in the first place. Mashery co-founder Clay Loveless makes this case.

AWS has been offering the EBS service since 2008. It's not considered a "beta" product. Why shouldn't customers be able to rely on it? True it's had issues over the years, leading some companies to decide not to use it. But AWS has happily taken money from customers for years now. If it's a product that isn't ready for product, AWS should have said so. (Unfortunately for customers, the EBS outage won't count towards their SLAs.)

What the "they shouldn't have used EBS" argument comes down to is: customers are stupid for trusting AWS to provide the service promised. It's saying that customers that paid for multi-zone RDS replication should have expected 14+ hours of downtime. If AWS itself were to tell its customers "You should have known better than to trust our service," we would be up in arms - wouldn't we?

I keep seeing similar arguments. "We shouldn't blame Dropbox for lying about its encryption, we should blame customers for trusting Dropbox." "We shouldn't blame Apple for not giving users control over their location logs, we should blame customers for expecting privacy." I'm sick of it.

It might in fact be true that we can't expect vendors to provide customers' what they promise. But that is squarely on the shoulders of vendors, not the customers. And I'm sick of "savvy" pundits putting down customers and excusing failure and bad behavior on the part of companies.

Yes, things happen. AWS is run by humans, and humans make mistakes. AWS deserves some forgiveness . But let's not forget who messed up.

How to Fix the Problem

In the short term, I suspect many customers will move away from using EBS and RDS. In the medium term, infrastructure-as-a-service providers need to come up with a standard system for sharing instances across clouds, whether that's OpenStack, Cloud Foundry, Eucalyptus or something else. Customers shouldn't have to choose between trusting only one provider or committing to a complex and potentially unreliable multi-vendor solution. The days of vendor lock-in must come to end.

Meanwhile, bloggers, analysts, journalists and other opinion-makers need to put the blame back where it belongs: on service providers that don't live-up to their promises.

(Lead image by Ian)

Disclosure: Mashery is a ReadWriteWeb sponsor.

Discuss

Links for 2009-11-28 [del.icio.us]

2009-11-29T00:00:00-08:00

SpecFlow - workflow

Links for 2009-11-27 [del.icio.us]

2009-11-28T00:00:00-08:00

Links for 2009-11-26 [del.icio.us]

2009-11-27T00:00:00-08:00

Thinktecture.IdentityModel v0.6

Links for 2009-11-12 [del.icio.us]

2009-11-13T00:00:00-08:00

Transparency Models: A Tale of Two Levels

Links for 2009-11-08 [del.icio.us]

2009-11-09T00:00:00-08:00

Exploding Software-Engineering Myths - Microsoft Research
More Isnt Always Better, Write Test Code First, Proving the Utility of Assertions, Organizational Structure Does Mattera Lot. Geographical Distance Doesnt Matter Much.