Oran Dennison

The Quest for Raw Feeds in the Live Framework

2009-04-17T17:22:00.001-08:00

Ever since I started playing with Live Framework, I’ve wanted to import Atom feeds from external sources such as blogs, Twitter, and various Google Data APIs. Surprisingly, as we will see later, this is not easy.

I am most interested in annotating Twitter feeds and synchronizing those annotations between multiple devices, apps, and users. Why would I want to do this? Because this can address a number of limitations with Twitter and existing Twitter clients:

Read/unread status isn’t shared between apps and devices
Groups and saved searches aren’t synchronized
No third option between public and protected accounts
Favorites are public (no private or semi-private favorites)
Availability issues due to fail whales, being offline, etc.
You must be online to tweet, favorite, and follow
You don’t own your data, Twitter does
No good path for migrating from centralized tweets (twitter.com) to a decentralized, federated model
Twitter’s crossdomain.xml doesn’t support direct access by Silverlight and Flash apps

That last point on crossdomain.xml is particularly frustrating since you would think that between Silverlight 3 Out-of-Browser and Live Framework it would be straight-forward to build a reasonable competitor to TweetDeck, Twhirl, or Seesmic Desktop (Adobe AIR’s killer apps), but that is not the case.

Atom in the Live Framework

Before discussing the issues with importing Atom feeds, it is useful to consider how the Live Framework uses Atom. As you may know, Atom is the Live Framework’s native infoset. These Atom feeds can be accessed using other representations such as RSS, JSON, and POX, but the abstract infoset is fundamentally Atom, supplemented by AtomPub for CRUD and FeedSync for sync.

Live Framework then layers a resource model on top of Atom. This resource model adds schemas for data such as News, Contacts, MeshDevices, MeshObjects, and more. These schemas can be discovered by using the OPTIONS HTTP verb or by reading the Resource Model documentation. As you can see from the documentation, all schemas are based on the abstract Resource schema which includes a number of general-purpose Atom properties as well as Mesh-specific properties such as Triggers.

What’s wrong with a little schema?

The resource type closest to a raw Atom entry is the DataEntryResource. Conveniently, it supports arbitrary element and attribute extensions, so it looks like you ought to be able to shove any arbitrary Atom entry into a DataEntry.

It turns out that while DataEntry’s schema provides a home for every possible Atom data element, in practice some of the elements are reserved or have special behavior. Further, this behavior is inconsistent between the local LOE and the cloud LOE, although the inconsistency can be used to hack around some of the limitations, highlighting the power of the back door endpoints used by the Live Framework Client.

Reserved elements

The Live Framework doesn’t let you store your own arbitrary data in the following elements of an Atom entry:

id
published
updated
content

If you attempt to provide your own values for these elements, they will be overwritten with auto-generated values by the cloud Live Operating Environment.

The <id> element will always be set to a random GUID such as:

The <published> and <updated> elements will always be set to the cloud LOE’s DateTime.UtcNow. Actually, this isn’t true for <published> on the local LOE but we’ll get to that later.

Forcing <published> and <updated> to DateTime.UtcNow causes problems when bulk importing entries. All of the entries will share the same time, making sorting impossible. This is particularly problematic because currently you can’t sort on custom elements, so even if you store the original <published> and <updated> elements under different names, you’re out of luck when it comes to sorting and filtering.

The <content> element is used as a grab bag for all sorts of LiveFX-specific content:

<content> is also where you will see Triggers if any have been attached to the resource.

Clobbering the <content> element is a big problem for importing blog feeds because this is where the blog post body content lives.

On a side note, it appears that the Live Framework only supports <title> and <subtitle> elements where type=“text”, but I could be wrong.

Additional elements

Live Framework preserves the original <author> element, but adds a second <author> element with the <name>, <uri>, and <email> of the LiveID user account that imported the entry. The Live Framework author element appears before any external author elements. It is perfectly valid for Atom entries to have more than one author, but this is something to watch out for if your app assumes only one author per entry or if author information has app-level significance.

I haven’t tried importing entries with pre-existing <category> or <link> elements, but I assume they would successfully import and be supplemented by additional LiveFX-specific <category> and <link> elements.

Local LOE differences

After learning how the cloud LOE behaves, I tried importing Atom feeds from Twitter into the local LOE. Unlike the cloud LOE, the local LOE preserved the original <published> element. However, the <updated> element was again set to DateTime.UtcNow, just like on the cloud LOE. At least I can now use LiveFX’s support for sorting, filtering, and paging imported Twitter feeds by the <published> date.

After successfully preserving the <published> element by using the local LOE, I checked the same feed on the cloud LOE after it synchronized. Amazingly, the cloud LOE now showed the correct original <published> date! “Aha,” I thought, “I should be able to make the cloud LOE accept the original <published> date by talking to it using FeedSync instead of AtomPub.”

Unfortunately, accessing the cloud LOE DataFeed’s Sync feed using FeedSync did not preserve the original <published> date. Very interesting! Then how did the local LOE successfully sync the original <published> dates to the cloud LOE?

The parallel universe of Windows Live Core

“This is your last chance. After this, there is no turning back. You take the blue pill - the story ends, you wake up in your bed and believe whatever you want to believe. You take the red pill - you stay in Wonderland and I show you how deep the rabbit-hole goes.” – Morpheus, The Matrix

Ok, perhaps I’m being overly dramatic. :-) But seriously, what you know as MeshObjects, ApplicationInstances, and so on are actually CoreObjects living in the grittier, undocumented world of the Windows Live Core (WLC). The local and cloud LOEs are like the Matrix, hiding the WLC from you and letting you believe that the shiny world of Live Framework is all there is.

There are three main endpoints in WLC:

accounts.developer.mesh-ctp.com
storage.developer.mesh-ctp.com
enclosure.developer.mesh-ctp.com

Accounts is responsible for managing the 3 types of identity in the mesh: users, devices, and apps. Accounts is also responsible for ApplicationClaims (mapping apps to users) and DeviceClaims (mapping devices to users). Storage is where CoreObjects live. Enclosure is where CoreObject media resources live.

The Live Framework Client synchronizes with the cloud using storage.developer.mesh-ctp.com, not user-ctp.windows.net, and instead of hitting the DataFeed’s /Sync URI, it hits the /Sse URI. SSE is Simple Sharing Extensions, the old name for FeedSync.

storage.developer.mesh-ctp.com’s FeedSync implementation is more tolerant of arbitrary data than user-ctp.windows.net. By using this back door, the Live Framework Client is able to preserve our original <published> date.

If you want to learn more about Windows Live Core, you can use Fiddler to inspect the Live Framework Client’s communication. You can also use Reflector to check out the Microsoft.Live.Core.Resources namespace in Microsoft.MeshOperatingEnvironment.Runtime.Client.WlcProxies.dll located in C:\Users\[username]\AppData\Local\Microsoft\Live Framework Client\Bin\Moe2\.

DataFeed vs. DataEntries feed

So far I have focused on the issues when importing individual Atom entries within a feed. There are also issues when importing metadata for the feed itself. But first it is useful to discuss the differences between a DataFeed and its DataEntries feed.

In the picture above, green=entry and orange=feed. The most obvious thing to note here is that DataFeed is an entry that links to an Entries feed containing multiple DataEntries.

The Entries feed’s <title> is read-only and is always “DataEntries”. <id> and the rest of the elements listed above are auto-generated by the Live Operating Environment. The DataEntries feed isn’t extensible, so you can’t add links to things such as the original feed’s self link or alternate link.

This means that if you want to preserve any feed metadata from your original feed, you must store it somewhere else. The DataFeed entry is a likely candidate.

You can store the original feed’s title in the DataFeed’s title. You can also add any links from the original feed that don’t conflict with LOE-managed links such as self link. If you need to store the original feed’s self link, you can do this by renaming its rel and title attributes.

It should now be quite clear that importing external feeds is like putting a square peg in a round hole. Even if you find a way to do it, you won’t be able to take existing Twitter clients and easily tweak them to use Live Framework’s imported Twitter feeds.

Other considerations for apps

I noticed a few other interesting things as I was exploring. The local LOE correctly lists imported tweets in reverse chronological order (the order in which they were imported). When these tweets are synchronized with the cloud LOE, they are listed in chronological order (the exact opposite of the local LOE).

You can use Resource Scripts to import external feeds, as I demonstrated in the code sample for this post. With the new Loop statement, this becomes even easier. However, Resource Scripts can only read external feeds when the script is executed locally, not in the cloud.

Whether you’re importing one item at a time or using Resource Scripts for bulk imports, it is clearly preferable to import using the local LOE. But what if you want to use delegated auth to import data using a 3rd-party website? Unless you figure out how to hack WLC (which I doubt supports delegated auth), you’re out of luck when it comes to the <published> date because you don’t have a local LOE.

Hope for the future: Federated Storage

I don’t know much about Federated Storage Services since I didn’t have access to the pre-CTP release that supposedly exposed them, but I believe the idea is to allow third parties to create proxies to their services, similar to how Contacts and Profiles from Hotmail’s Address Book Clearing House (ABCH) are currently exposed. You can see hints of Federated Services if you paste the following link into the Live Framework Resource Browser:

https://user-ctp.windows.net/V0.1/Mesh/SomeRandomUri

I would imagine that this would provide a better option for “importing” 3rd-party data into the mesh. You would probably have much greater control over the URIs, titles, and more. Furthermore, this data wouldn’t need to be stored in Microsoft’s datacenters. This is especially important if you want to “import” feeds with sizeable enclosures such as pictures and video.

One possible downside of Federated Storage is that I imagine you won’t be able to arbitrarily annotate data in external feeds, reducing their potential in data mashup scenarios. This limitation already manifests itself in the inability to persist Delete triggers for Contacts and Profiles.

Another request for “Yahoo Pipes for AtomPub”

At the end of this blog post, I proposed using Live Framework to enable something like Yahoo Pipes for AtomPub. This would enable you to import and mash up arbitrary feeds, perhaps supplemented by a visual Resource Script designer. I believe support for raw Atom feeds is crucial for this scenario. Support for raw RSS feeds may also be desirable.

Feature request

If you want to easily build Live Framework apps that import feeds from external sources, please vote on this feature request on Connect. Thanks!

Live Mesh and Live Framework Presentation

2009-04-08T01:05:00.001-08:00

Yesterday I gave a presentation on Live Mesh and the Live Framework for Software Professionals of Alaska.

High-quality WMV download (78.6 MB)

Slides (5.6MB pptx)

What did I talk about?

As you can see by Yertle the Meshified Turtle, I had some fun with the presentation.

Besides putting my own spin on the typical introductory material, I gave some guidance on things that tend to trip up developers coming from a background of relational databases, file systems, and app-owned data (36:42 onwards).

The last part of my presentation is the most interesting part (to me anyway). This is where I speculate wildly about what the future might hold. There are some good scoops in there that most people probably don’t know about. I might get in trouble for talking about them, but I assure you that every scoop or bit of speculation has a publicly available source to back it up. I’ve got them right here in my notes. Check it out starting at 42:12.

More info

As I mentioned in the presentation, the Live Framework Forum is the best one stop shop for more resources to get you started. There are several sticky threads at the top that contain everything you need.

There were two things I intended to demo but forgot. First is mobile web access to Live Mesh at http://m.mesh.com. It works in desktop browsers too, so check it out. Make your browser window tiny and pretend it’s a phone. ;-)

The second thing I forgot to demo was the Live Framework Resource Browser. Fortunately, there is already a video showing it in action. Real clicky-clicky browsing action starts around the 7-minute mark.

Afterward several people said they wanted to see what it’s like to develop a Mesh-Enabled Web App. Check out this nice 11-minute video showing how to build a Silverlight MEWA.

Since we’re on a roll with videos, I’d also recommend you check out the very well done Live Mesh ad I mentioned in the “futures” part of my presentation. If you have the bandwidth, go to the source and click the HD button. It’s beautiful.

A note on presenting

I thought I would be embarrassed to listen to a recording of myself, but it turned out better than I expected. One thing I learned is that Dan was right when he held up the “Talk louder” sign. Thanks, Dan. Sometimes I fade away at the end of sentences. I blame it on my time in Finland (their sentences all run downhill, even questions). Something to work on for next time.

Listening to myself in the third person without visual cues, I noticed several times when my dry humor was so deadpan, you might be left wondering whether it was intentional. So if you have “was that intentional?” moments while listening, the answer is probably “yes.” There was probably an equally subtle smile on my face as I said it. ;-)

RESTful UDP: a Live Framework Feature Request

2009-03-12T23:51:00.001-08:00

Yes, I admit, “RESTful UDP” sounds unnatural, maybe even unethical. I also admit that I’m getting ahead of myself, talking about design before requirements. So what motivates me to ask for such a feature?

I want real-time P2P messaging between users, apps, and devices. Notifications get us most of the way there, but they aren’t enough. Before I discuss the issues with notifications, let’s talk about scenarios that motivate this.

Scenarios

Imagine walking up to a large-screen Mesh-enabled device such as an XBox, Microsoft Surface, or public kiosk. You pair your Mesh-enabled smartphone and project its apps and data onto the big screen, with your smartphone acting as the data entry device.

Expanding on this scenario, imagine a game that takes advantage of the smartphone’s accelerometer and camera, turning your phone into a high-powered Wiimote with the entire touch screen used for control surface. You might want to attach a wrist strap…

Imagine the cool apps you could write if a group of people shares real-time GPS data from smartphones and carputers.

This feature would be useful for more than just extending the capabilities of smartphones. You could remotely control media playback, chat with people, push real-time financial data, and build a variety of interesting distributed apps that are designed to run in real-time across a mesh of devices, aggregating specialized device capabilities into a single composite experience.

Cross-platform support exponentially increases the possibilities and relevance of the mesh. You can imagine special-purpose devices whose entire reason for existence is to be plugged into the mesh to supplement apps and user experiences. This is true even without real-time messaging, but this capability is crucial for enabling the most seamless composite device experiences.

Who else wants this?

Back in December, Kevin Hoffman kicked off an extended discussion on this feature request. Here’s a brief excerpt:

Low Latency: Other actions that people take within the application need to happen quickly. I need very low latency between when the action takes place and when the other client(s) are notified about the action. Think of these as instant messages, though with a domain-specific purpose. Some can be directed at an individual MEWAs, others can be broadcasts. I do not currently have a solution for the low latency.

I know that Silverlight applications cannot receive push messages because of their highly restricted network Sandbox. However, I'm wondering if they would be able to create an HTTP WCF service via .NET Services and host the proxy in the cloud that would allow near-real-time HTTP message posting between MEWAs... Is this possible?

Strages has a big wish list in the Mesh forum that includes:

as mobile phones and laptops will get close and closer together.. there will get a time that you just plug your phone to a local monitor and keyboard.. or even better bring your own flexible seized monitor etc etc.. you get it

John Macintyre’s PDC session had a demo of remotely controlling a Media Center. Sync delays in the demo made for a not so seamless experience.

Warning: detailed discussion ahead

I could probably end the feature request here. Please take everything that follows as optional food for thought. If you like what you’ve heard so far and just want to vote for this, please visit this feature request on the Live Framework forum and vote it up. If you have comments, please post them on the forum rather than here.

Why UDP?

First, I don’t intend UDP to be taken literally (well maybe I do, but that’s an implementation detail). Specifically, I’m interested in the following UDP characteristics:

One-way messaging
Low latency
Stateless (lossy, no sessions, unordered, etc.)
Multicasting

It shouldn’t be necessary to expose the concept of P2P session initiation, even if that is ultimately an implementation detail.

Why REST?

I do intend REST to be taken literally. I’m interested in the following REST characteristics:

URI-addressable resources
Hyperlinks to other resources
Arbitrary user content
Transport-agnostic (yes, I think that’s RESTful)
POST and GET (PUT and DELETE don’t make sense here)

Links to arbitrary resources are an important tool for keeping message size small.

Being transport-agnostic is important, for performance and for constrained network environments. Of course HTTP should be supported, but it should also be possible to use TCP, UDP, the Messenger Relay, or even use SMS like Mesh4x does. Just like with Notifications, there should be no need for senders and receivers to use the same transport or representation.

Making this feature available to plain old DHTML Mesh apps would be pretty amazing.

Meshisms

I would also like support for some Mesh-specific features:

AtomPub feed-based model
Multiple representations (ATOM, POX, JSON, Binary XML)
Expansions
“LINQ to REST”
Send messages from triggers (a fine-grained alternative to subscriptions)
Local LOE support

Why aren’t Notifications enough?

In my Notifications post I covered some of the issues that make Notifications less than ideal for real-time messaging, even when paired with Activities.

Three round-trips per notification cycle (can be reduced to two using expansions)
Watermarks are destructive
You can’t directly publish your own notifications
Notification queues are per-user and designed for single-threaded use (one per client)
Each app does its own polling rather than sharing one connection, unlike iPhone’s push notifications
Notifications are one-way from cloud to client (no client-to-cloud or P2P)
Notification polls don’t chain from local LOE to cloud LOE
While Notifications are usually instantaneous, they can become backlogged and delayed by several minutes
The local LOE doesn’t currently implement Activities
There is no way for multiple clients to efficiently receive only unseen Activities in a single round trip

A proposed solution

There may be better solutions, but I visualize this problem being solved by a REST front-end to a “transport-agnostic UDP” messaging system. Under the hood, the system prefers direct UDP communication but can use Messenger Relay or HTTP if necessary. The front-end seen by developers is an AtomPub interface very much like the Notifications interface. When using HTTP, push messaging is enabled by parking requests for up to 30 seconds if no messages are waiting.

It is preferable to program against the local LOE. Polling the local LOE for messages causes the local LOE to poll the cloud LOE, and if any participating devices are reachable, they are also polled if another push mechanism can’t be established. It is important to be able to establish local P2P connections even if the cloud LOE isn’t reachable. Care is needed to avoid round-robin message loops.

Messages would auto-generate a short MaxAge upon receipt by each LOE, similar to Activities. It is probably most efficient to simply let Messages expire instead of explicitly deleting them.

In order to support multiple recipients of the same message, clients should be able to poll the queue with a nondestructive query string watermark. An alternative might be to let each client create its own destructive NotificationQueue that receives full copies from the main message queue, but this assumes the 3-round-trips issue is solved.

This can be a soft state service with no need for additional infrastructure recovery features.

If it is necessary to impose the UDP equivalent of Twitter’s 140-character limit, that’s fine. 65,507 bytes seems like plenty. Users can always just send links to large resources. If expansions are supported, the receiver can optionally inline the large data on demand without necessarily transmitting it across the network if it already exists in the local LOE.

Addressability

Messages should be able to target one or more users and devices, and scope a broadcast by MeshObject/AppInstance. This could be solved in a variety of ways.

You could scope all messages by MeshObject (MeshObjects/{id}/MessageQueue), with the option to subscope them further with links to Members and Mappings. This is the option I prefer. It doesn’t require directly addressing users and devices, it doesn’t require any changes to enable the “cloud device” to be addressed, and it should provide decent partitioning. A downside is that it requires separate requests for separate MeshObjects. However, if you program against a local LOE which multiplexes everything under the hood, this shouldn’t be a problem.

Or you could have one queue to rule them all at Mesh/MessageQueue. Messages would have one or more links to users, devices, and/or MeshObjects. Subscribers would poll the queue with the option of filtering on these links by query string.

Or you could add /MessageQueue to various scoping contexts such as Devices/{id} and {id}/Profiles. This would require exposing the cloud LOE as a device entry.

I’m not sure if devices or mappings from other users are currently discoverable. This would need to be addressed.

Device connectivity

I suspect there may already be a way to access existing device connectivity information, but if not, it would be very helpful to see which devices (including the cloud device) are reachable for real-time messaging.

Issues with just exposing Mesh’s P2P support

At the PDC I heard talk of exposing Mesh’s peer-to-peer channel to enable developers to establish connections between devices for streaming data. Although this is a great solution for some scenarios (and I would like to have such a feature), it’s not so great for other scenarios.

Streaming in this context is a reliable, sessionful feature. In order to get the lowest latency possible, I want unreliable, sessionless communication.

You can optimize for latency or for throughput, but not both. Streaming is optimized for high throughput. I want low latency (UDP, or disable TCP’s Nagle algorithm).

Bonus points: ad-hoc device discovery

This is a tangentially related issue that can be addressed independently. Although apps can be written to select from a known list for pairing with other users and devices, I would love to have ad-hoc user/device pairing.

There are many, many ways to accomplish this. I would like to use Microsoft Tag for pairing devices from unknown users and meshes. One device displays a tag containing its mesh address, and the other device scans the tag. The tag could be a printed tag, or displayed on a screen. The tag exchange could be two-way for greater security. For public kiosks, the tag could be changed every minute or generated on demand.

There are a whole host of additional issues raised by ad-hoc device pairing, but it sure sounds like the future to me. Someone’s going to solve this problem in a generic way, please don’t let Apple get there first. ;-)

Conclusion

Hopefully the additional detail is a helpful starting point for discussion. I am quite open to alternate solutions that can provide real-time P2P messaging. As I noted earlier, if you like this idea, please visit this feature request on the Live Framework forum and vote it up. If you have any comments, please post them on the forum thread. Thanks!

Exploring Live Framework Notifications

2009-03-08T14:31:00.001-08:00

In order to tune in to the data that matters to you, Live Framework lets you receive near-real-time notifications of changes by subscribing to objects and feeds. Although the SDK documentation is fairly sparse at the moment, Viraj Mody wrote an excellent blog post describing how the notification system works, and John Macintyre also gave a great PDC session on notifications. Being the curious sort, I still had many questions, so I dug in further and this blog post and ResourceClient library are the result.

Overview

Notifications are exposed in the high-level programming model through the ChangeNotificationReceived event. Under the hood, this is made possible by queues, subscriptions, and notifications working together. These three building blocks can be accessed through the resource-oriented programming model, enabling even more interesting scenarios. Even if you have no desire to program at this level, understanding how notifications work can help you take fuller advantage of the high-level model.

Slide 9 from John Macintyre’s presentation shows how the pieces tie together.

The typical usage pattern looks like this:

Client creates a queue
Client subscribes its queue to one or more resources or feeds
Client polls the queue’s Notifications feed
A resource or feed changes
Subscription Service posts a notification to each of the queues that are subscribed to the resource or feed
Client’s poll returns with one or more notifications
Client takes the watermark of the most recent notification and posts it to the queue
Client takes some action based on the notifications such as requesting the newest version of the changed resource or feed
Client polls the queue’s Notifications feed…

After the initial queue creation, a minimum of three round-trips are necessary to complete each cycle. Later we will see a trick for doing this in two round-trips.

Although the client has to poll the queue, this is actually more of a push than a pull. The HTTP request stays “parked” at the server until a notification arrives or a timeout expires (just like relay binding’s HTTP “parked requests” in ~~BizTalk Services~~ .NET Service Bus). This means that when a notification arrives, the server can immediately push it to the client on the HTTP response of the parked request, saving half a round-trip of latency. Jeremy Mazner has more to say on the subject in this Channel9 thread.

So far I have described the behavior of the cloud LOE. We will cover the client LOE’s slightly different behavior later.

Recovery from failures

In John’s diagram above, the Queue Service and Subscription Service are both in-memory soft state services. This means it is possible for a Queue Service instance to go down resulting in queue loss (including all its notifications). A Subscription Service instance can also go down, resulting in loss of subscriptions. In both cases, the client will receive a specific error notification the next time it tries to poll the queue.

In the case of queue loss, it is the responsibility of the client to create a new queue and resubscribe to each resource. In the case of subscription loss, the client is told “resources from these addresses lost subscriptions” and it just needs to resubscribe to each one using its existing queue.

As Viraj notes in his blog post,

A short summary of the solution is that in cases where one or several Queue and/or PubSub Servers go down, the system is able to detect exactly what happened and take remedial action to restore state in the cloud in cooperation with clients (because clients were the original source for all the transient data that was resident on those servers before they lost state).

What can you subscribe to?

From the high-level object model, only the following types are subscribable via ChangeNotificationReceived:

MeshObject
MeshDevice
LiveItemCollection (feeds)
- Mesh.Devices
- Mesh.MeshObjects
- Mesh.News
- MeshObject.DataFeeds
- MeshObject.Mappings
- MeshObject.Members
- MeshObject.News
- These may not work yet:
  - Contact.Profiles
  - LOE.Contacts
  - LOE.Profiles
  - Member.Profiles

If you use the resource-oriented programming model, you can subscribe to all of the resources behind the high-level objects as well as:

ApplicationResource
ApplicationInstanceResource

In addition to the high-level feeds, you can also subscribe to feeds for:

MeshObject.Activities
Applications
InstalledApplications

One quirk is that MeshObject isn’t subscribable from the local LOE, although you can still subscribe to the local MeshObjects feed.

It is useful to consider the MeshObject->DataFeed->DataEntry hierarchy in terms of what is and isn’t subscribable.

Mesh/MeshObjects/Subscriptions
Mesh/MeshObjects/{id}/Subscriptions
Mesh/MeshObjects/{id}/DataFeeds/Subscriptions
- this is a “feed of feeds”
~~Mesh/MeshObjects/{id}/DataFeeds/{id}/Subscriptions~~
- this doesn’t exist
Mesh/MeshObjects/{id}/DataFeeds/{id}/Entries/Subscriptions
~~Mesh/MeshObjects/{id}/DataFeeds/{id}/Entries/{id}/Subscriptions~~
- this doesn’t exist

What notifications do you receive?

You would expect all resources and feeds to notify you when resources are created, updated, or deleted, but that isn’t the case. Some resources and feeds don’t notify you when entries are updated, some aren’t subscribable, and some are read-only. This varies between the cloud LOE and the local LOE and between various objects. Here’s an incomplete listing of which notification triggers work based on my experimentation:

	Local	Cloud
MeshObject	Not subscribable	Update, Delete
MeshDevice	Not subscribable	Nothing? at least not Update
MeshObjects feed	Create, Update, Delete	Create, Delete (no Update)
Devices feed	Can’t update locally	Update, maybe others
DataFeeds	Create, Update, Delete	Create, Update, Delete
DataEntries	Create, Update, Delete (with double notifications for each)	Create, Update, Delete

I’m not sure why, but a subscription to DataEntries creates two notifications for each change on the local LOE.

How subscriptions work

As you have seen, you can subscribe to any resource or feed that has a Subscriptions URL (ex: Mesh/MeshObjects/{id}/Subscriptions). This Subscriptions feed only supports HTTP POST (no GET). Individual subscriptions only support PUT (no DELETE).

Subscriptions have the following interesting properties:

NotificationQueueLink
ExpirationDuration
ResourceEntityTag

NotificationQueueLink is the only thing you need to include when you create your subscription. This link should point to the SelfLink of the NotificationQueue you have created, not its NotificationsLink.

ExpirationDuration is expressed in seconds and is assigned a random number between 2700 and 4500 (45 min to 75 min) from the cloud LOE and a fixed value of 3600 (60 min) from the local LOE. The internal class NotificationManager that is used to implement ChangeNotificationReceived has a hard-coded subscriptionRenewalInterval of 60 minutes, so it seems there’s a chance of cloud subscriptions being in an unrenewed state for up to 15 minutes, but I could be wrong.

ResourceEntityTag is the ETag of the resource when the subscription was created. I believe a notification is sent each time the resource’s ETag changes.

Note that the subscription doesn’t have a link to the resource or feed you subscribed to. It is important for you to maintain your own copy of the URI to the resource you subscribed to. First, if you want to imitate ChangeNotificationReceived and execute different event handlers for different subscriptions, you will need this URI to demux from notifications to event handlers. Second, you will need this URI to create new subscriptions when you receive an AllSubscriptionsLost notification.

Clients should track their subscriptions for a variety of reasons:

You can’t GET the subscription feed
You can’t GET a subscription via its SelfLink
For subscription renewal
To recover from queue or subscription loss

How notification queues work

Notification queues are created by posting to Mesh/NotificationQueues. If using AtomPub, you can simply post an empty entry:

You can’t GET Mesh/NotificationQueues to see which queues exist, so you will need to maintain a reference to the queue you get back.

Queues are intended to only have a single consumer (one queue per client), and that consumer’s queue usage should effectively be single-threaded.

There are a few important properties on queues:

ExpirationDuration
Watermark
SelfLink
NotificationsLink

ExpirationDuration is expressed in seconds and is 300 (5 min) from the cloud LOE and 600 (10 min) from the local LOE. ExpirationDuration is the duration after which the notification queue expires if its Notifications feed isn’t polled.

Watermark is used together with PUT to remove all notifications in the queue with watermarks less than or equal to the watermark you sent. Updating a watermark is destructive. You can’t PUT an earlier watermark to roll back the queue, and that’s fine with me.

SelfLink is the URI you use for a new subscription’s NotificationQueueLink. You also PUT to the SelfLink URI when updating the queue’s watermark. SelfLink is PUT only. You can’t GET or DELETE a queue.

NotificationsLink is the feed where new notifications appear. This feed is not your typical feed. As I mentioned earlier, polling the NotificationsLink “parks” the HTTP request on the server until a notification appears or a timeout expires. This timeout varies between 25 and 30 seconds for the cloud LOE and is hopefully low enough so that intermediary proxies don’t prematurely close the connection. Requests will return immediately if the queue already contains notifications. Unfortunately, requests to the NotificationsLink on the local LOE always return immediately whether or not the queue is empty. This is a significant problem not only because the programming model is inconsistent, but because it means that in order to get the same low latency as the cloud LOE you have to hammer the snot out of the local LOE, pegging the CPU in the process. Hopefully this gets fixed. The SDK sidesteps this issue by only polling every 5 seconds, but that loses much of the benefit of push notifications.

The notifications feed behaves strangely when it contains more than one notification. Sometimes it displays all of the notifications (up to 10), and sometimes it only displays the first one. For example, you can poll the queue and see 4 notifications. You can poll it again and perhaps only see one. Polling a third time might show all 4 again. At least it always displays notifications in order (oldest first). This means you shouldn’t count on seeing more than one notification at once, but be aware that it is possible. Simply act on all of the notifications you receive, PUT the watermark of the last one, and poll the queue again. Using this technique you will eventually see all of the notifications in the queue, even if it appears there is only one or if the queue appears to be clipping the results at 10 notifications.

Queues are semi-private. They can’t be seen by other user accounts and can’t be shared. Since you can’t enumerate queues with GET and their URLs are randomly generated, the only way another app or client within the same user account can see your queue is if you choose to share the queue’s URI, although I can’t think of any good reason to do that other than for testing.

Queue loss

Queues can be lost either by failing to poll them within their ExpirationDuration or by the unexpected death of a queue manager in the LOE. You will find out the queue is gone when you poll the queue’s notifications feed and receive an AllSubscriptionsLost notification.

You might be wondering, if the queue is gone, then how can I poll its notifications feed? It turns out that if a queue doesn’t exist at a particular URI, a new “dead” queue will be automatically created at that URI. You can see this by connecting to the cloud LOE with the LiveFX Resource Browser and visiting the following URI:

Mesh/NotificationQueues/1234-1234-1234/Notifications

Note that this only works on the cloud LOE. The local LOE will return an error. Apparently you shouldn’t expect to lose a queue on the local LOE.

A weird bit of trivia is that you can create two different queues with identical URIs in two different user accounts. I’m guessing this is because queue managers are allocated per-user.

Another weird bit of trivia is that the AllSubscriptionsLost notification has a watermark that increments each time a subscription attempts (and fails) to post a notification to the queue. This is one reason why the name AllSubscriptions lost is misleading and ought to be renamed to something like QueueLost. The subscriptions are most certainly still alive.

How notifications work

You can’t create notifications directly. In other words, you can’t POST a new notification to a queue’s Notifications feed. Notifications are only created by subscriptions (and queue or subscription loss).

Notifications have the following interesting properties:

NotificationType
ResourceLink
Watermark

NotificationType can be one of the following:

ResourceChanged
SubscriptionLost
AllSubscriptionsLost
System

ResourceChanged is what you will see as the result of subscribing to a resource. SubscriptionLost is received when a Subscription Service unexpectedly dies on the cloud LOE. SubscriptionLost is not received when a subscription expires due to its ExpirationDuration reaching zero. As mentioned earlier, AllSubscriptionsLost would probably be better named QueueLost. I’m not sure if System is used anywhere in the public API. Perhaps it’s used for device connectivity.

ResourceLink points to the feed or object that changed, or in the case or SubscriptionLost, I believe it points to the resource whose subscription was lost.

Watermark is a counter string that increases with each new entry. On the cloud LOE these look like “1.248.0”, “2.248.0”, “3.248.0”. On the client LOE they are simply “1”, “2”, “3”.

Notification SelfLinks are incrementing integers on the cloud LOE and GUID-like strings on the local LOE. You can’t GET a notification using its SelfLink. You can only see notifications by polling the notifications feed. You also can’t do queries on notification feeds. If you try to use something like $skip or $top, you will get an AllSubscriptionsLost notification and every notification in the queue gets discarded without you ever seeing them. However, the subscriptions still work and the queue continues to function normally afterward (other than the data loss).

Notifications and expansions

A cool trick for saving a round-trip is to use expansions to return changed resources and feeds inline in the notification results. A not-so-cool potential side-effect is that if the notifications feed returns more than one notification for the same resource or feed, the expansion will result in duplicate expanded data, wasting bandwidth.

If you could combine watermark updates with the next poll request, you could get the poll-watermark-update cycle from 3 round-trips down to just 1 round-trip.

Notifications and activities

When you combine notifications, expansions, activities, and the cloud LOE, this enables near-real-time messaging between clients. By subscribing to an Activities feed, clients can poll the Notifications feed using $expand and receive complete activity entries as soon as someone posts a new activity. The latency in this scenario is half a round-trip to post the activity plus half a round-trip to receive the activity through your parked HTTP request to the notifications feed. Since notifications, subscriptions, and activities all use in-memory stores, this should have quite good performance. There are a number of issues with this technique that I’ll cover in a future blog post, but it is promising for near-real-time communications.

Sync bypasses update notifications

Sync appears to bypass notification of updated feed entries. Specifically, if I subscribe to the MeshObjects feed on the local LOE, I am notified when sync from the cloud causes a MeshObject to be added or removed from the feed, but I am not notified if sync causes a MeshObject to be updated. I haven’t experimented with other feed types, but the same issue might exist with feeds such as DataEntries.

Trigger support

You can use Create and Update triggers on subscriptions and queues. Delete triggers aren’t persisted and therefore won’t work. You might use this to POST a queue and create subscriptions for it in its PostCreateTrigger, saving a round trip. Of course these subscriptions wouldn’t be tracked and managed by the SDK. You could also create a MeshObject and add a subscription to it in the MeshObject’s PostCreateTrigger.

Updating resources and feeds through Resource Scripts or triggers doesn’t bypass notifications.

Client vs. Cloud

The client LOE waits for data to sync to it from the cloud before notifying you of any changes. This can take a while, so if you want quick notifications, be sure to subscribe to the cloud LOE, not the local LOE. It would sure be nice if local subscriptions caused the local LOE to subscribe to the same resource in the cloud (if connected), taking advantage of push notifications to achieve the same latency for local subscriptions as if you were subscribed directly to the cloud LOE.

It is probably stating the obvious at this point, but I think it’s worth noting that queues are one-way from cloud to client.

Notification delays

Although notifications are normally posted to queues immediately, it is possible for notifications to be delayed if there is a large backlog generated by lots of rapid updates to a subscribed resource or feed. You might see a few updates trickle in, then 15 seconds later another batch of updates appears, and so on, for several minutes.

Device connectivity

Supposedly device connectivity uses the subscription and notification services under the hood as a signal channel for P2P session establishment. It may be possible to see this in action and even participate in the process, but I haven’t explored this. There must be a reason you can subscribe directly to individual MeshDevices, but I haven’t seen anything interesting pop up yet. Check out George Moore’s description of P2P notifications and file sync for a fascinating scenario that I’m not sure is possible with the current CTP.

Other transports

Supposedly there is a TCP transport for receiving push notifications, but it doesn’t appear to be used or available at the moment. This could be useful for chaining subscriptions from the local LOE to the cloud LOE (or other clients) while preserving an HTTP programming experience for developers.

The high-level programming model

If you have managed to read this far, you should now have a healthy appreciation for the services provided by the high-level programming model. At this level, all we see is the ChangeNotificationReceived event on feeds, MeshObject, and MeshDevice. You simply subscribe to this event on the appropriate object and your event handler will be called when entries are created (on feeds only of course), updated, and deleted.

Here are some of the gory details it takes care of for you:

Queue creation
Queue polling
Updating the watermark
Subscription creation
Subscription renewal
Recovering from queue loss
Recovering from subscription loss
Demuxing from notifications to event handlers

Let’s dig into that last one a bit deeper. If you use Reflector to examine how ChangeNotificationReceived works, you will see that each object subscribes to receive any and all notifications that arrive on the queue. When these notifications are received, each object iterates through all notification entries, checking to see if the notification’s ResourceLink matches its own SelfLink. If there is a match, the object raises the ChangeNotificationReceived event and stops iterating the notification list, effectively filtering out multiple notifications for itself that might have been received in a single response. It appears that if the client’s LiveOperatingEnvironment is configured with AutoLoadRelationships, the object will then be reloaded, after the event is raised, meaning that if you examine the object in your event handler, it may not contain the latest changes. This is reported in the forums here and here and is supposed to be fixed in the next release. Those threads also mention a performance issue with notifications when using the Silverlight library that will be fixed in the next release.

Programming at the Resource level

As part of my explorations, I wrote a library called ResourceClient that makes it easier to work directly with Resources such as queues, subscriptions, and notifications. Here’s a brief example of the syntax it enables.

using (new ResourceClientContext(username, password))

    var queue = Uris.Cloud.NotificationQueues.Post(new NotificationQueueResource());

    queue.StartAutoPoll((notifications, context) =>

        if (notifications.Entries.Count() > 0)

            Console.WriteLine(notifications.ToAtomString());

});

    Uris.Cloud.MeshObjects.Subscribe(queue);

    var mo = Uris.Cloud.MeshObjects.Post(new MeshObjectResource("My Object"));

    mo.DataFeedsLink.Subscribe(queue);

    var feed = mo.DataFeedsLink.Post(new DataFeedResource("My Feed"));

This code writes to the console Atom-formatted notifications for the new MeshObject and the new DataFeed. I will cover the details of this library in another blog post. For now I will just mention that in addition to being a generic resource-oriented API, it contains notification-specific helpers for automatic queue polling, manual polling, sending watermarks, subscribing, and dispatching to different event handlers for each subscription.

MeshNotificationPlayground sample app

I have written a little WPF app that demonstrates queues, subscriptions, notifications, and activities in action.

You can create a queue, poll it, copy the queue’s URL to the clipboard for pasting into Resource Browser, select a notification and send its watermark, select a MeshObject and subscribe to its Activities feed, and create new Activities for the selected object.

Here are some things to try:

Poll an empty queue and see that the request returns after 25 to 30 seconds
Poll the queue with the WPF app and Resource Browser at the same time, notice that they both wait, then cause a new notification and see that both requests return immediately
Subscribe to multiple Activities feeds, create activities in each of them, and notice the resulting notifications have different ResourceLinks
Send a watermark that isn’t the last watermark and see that the queue only empties up to the watermark you sent
Wait 5 minutes and poll the queue to see an AllSubscriptionsLost notification
After receiving AllSubscriptionsLost, create more activities for subscribed feeds and see that AllSubscriptionsLost’s watermark increases each time
Click Refresh Activities List and watch MaxAge count down for each activity. Notice the random MaxAges. When a maxAge reaches zero, the activity disappears. See that this causes a new notification.
Create more than 10 notifications and see that no more than 10 are returned. Send the latest watermark, poll again, and see that the remaining notifications appear.
Poll the queue repeatedly when it has multiple notifications and see that sometimes only 1 notification appears.
Click Create Activity many times in a row and see that notifications for that Activities feed continue to trickle in several minutes later.

You can download the code here. It includes and uses the ResourceClient library.

Comparison to iPhone’s Push Notifications

If for some reason your eyes haven’t glazed over yet, check out Viraj’s analysis of the iPhone Push Notification Service. If you read between the lines, this is a fascinating compare-and-contrast to Live Mesh’s notification solution, with Live Mesh being better in many ways. It hints that Microsoft might imitate Apple and use notifications to mine usage metrics for apps.

Apple’s solution is slightly more efficient in that it creates a single channel from the cloud to each device, whereas Live Framework currently requires each app to establish its own channel. This could be solved by having apps subscribe to the local LOE using local queues which would then transparently chain the subscriptions and queues through a single channel to the cloud LOE.

Conclusion

Hopefully this helps clarify how you should expect notifications to behave in your apps, as well as providing ideas for more creative uses of the building block features of queues, notifications, and subscriptions. I plan to follow up with more details on my ResourceClient library, and write up a feature request that combines the best parts of notifications and activities to enable better real-time communication between clients.

Exploring Live Framework Activities

2009-02-21T23:44:00.001-09:00

Live Framework has an interesting undocumented feature called Activities. You can think of Activities as a more general-purpose, transient alternative to News in the Mesh. Currently the only resource for learning more about Activities is John Macintyre’s PDC session Live Services: Notifications, Awareness, and Communications. Some of the information in this post is from John’s presentation, but much of it comes from my own exploration.

What are Activities good for?

Unlike News, Activities aren’t necessarily meant to be displayed or used in a predefined way. Live Mesh uses Activities to track transient state such as which users are online, which users are currently in a folder, and which users are currently using a particular app (more details on this later). You can use Activities in your own apps to build features such as chat, remote control of apps, and near-real-time transmission of small messages (when used with notifications).

Where can you use Activities?

Only cloud LOE MeshObjects expose Activities feeds. There is no Activities link from client LOE MeshObjects. Technically ApplicationInstance also has an Activities feed, but in practice the ApplicationInstance Activities feed points to the Activities feed of the app’s MeshObject.

An interesting side note is that ApplicationInstance is essentially the same resource as its MeshObject. It shares the same entry id and contains a copy of all the MeshObject’s elements, plus a few more app-specific elements. I assume these entries map to a single CoreObject under the hood.

What do Activities look like?

An Activity has the following interesting data elements:

MemberLink (Uri)
ActivityTime (DateTimeOffset)
MaxAge (short int)
Type (string)
UserData (serialized object)

MemberLink is created for you automatically and points to the parent MeshObject’s Member entry corresponding to the user who created the Activity. You can’t create a MemberLink that points to a different user’s Member entry.

ActivityTime is optional and is separate from the entry’s published and updated times. You can specify any ActivityTime you want in the past or in the future, although you would typically set it to something like DateTimeOffset.UtcNow.

MaxAge is the number of seconds until the Activity expires. MaxAge is automatically generated for you.

Type can contain anything you want. Later we’ll see an example of how Live Mesh encodes user presence in Type.

UserData is the same GetUserData/SetUserData extensibility point you find on MeshObject and DataEntry.

The Title of an Activity is always blank, even if you try to supply one.

How do Activities behave?

The most interesting Activity behavior is the automatic expiration based on MaxAge. When an Activity is created, a random MaxAge between 600 and 900 (between 10 and 15 minutes) is assigned. This randomness is intended to smooth out the server load in the datacenter. The same technique is used for Subscriptions. See slide 11 of John’s presentation for an illustration of what happened before they started using random TTLs:

Once an Activity has been created, its MaxAge value counts down to zero. You can see the countdown happen as you poll the Activity or its feed. When MaxAge reaches zero, the Activity entry automatically disappears from the feed. The Activity actually hangs around for a few seconds while MaxAge is zero, and during this time the MaxAge element is missing from the entry.

You can update an Activity entry which will reset its MaxAge to a new random value between 10 and 15 minutes. Later we will see how Live Mesh user presence uses this technique.

You can also delete an Activity immediately without waiting for MaxAge to expire if you so choose.

Activity feeds are subscribable, so you can register to receive notifications as entries are added or removed. Entries that are removed due to MaxAge reaching zero also trigger a notification. I’m not sure if you will receive a notification if Activity state is lost due to unexpected server state loss (remember, Activities are in-memory only).

Activity feeds are not extensible, meaning you can’t use ElementExtensions and AttributeExtensions. As mentioned earlier they do support UserData which is hopefully sufficient for most scenarios.

Activities support Create and Update triggers but not Delete triggers. This is the same partial trigger support exhibited by Contacts.

Earlier I mentioned you can’t create a MemberLink that points to a different user’s Member entry. It turns out that you also can’t delete an Activity created by another user, even though you can delete your own Activities in the same feed. That’s right, in this case it is possible to have different permissions on each item within a single feed! So much for MeshObject being the most granular unit of permissioning… ;-)

The OPTIONS verb and $metadata do not work for Activities.

What is the programming model?

Today, Activities are only exposed through the resource model as MeshObjectActivityResource. There is only an ActivitiesLink property on MeshObject, not an Activities collection, and there is no LiveItem-based Activity object.

Actually, that’s only true for the .NET and Silverlight SDKs. The JavaScript SDK has a MeshObjectActivity class and MeshObject has an Activities property, but I haven’t used the JavaScript SDK.

For now, using Activities from .NET or Silverlight requires using AtomPubClient, Resource Scripts, or raw HTTP.

I’m guessing Activities aren’t yet exposed through the high-level programming model because of the additional complexity of MaxAge expiration, automatically handling MaxAge resets, and the possibility of wrapping Activities in a scenario-specific programming model for user presence.

Behind the scenes

The Mesh is composed of a variety of services spread across many servers in the data center. Some of these services use reliable state stores and others use soft state stores.

Reliable state includes the Accounts store (accounts.developer.mesh-ctp.com), user-data structured storage (storage.developer.mesh-ctp.com), and user-data blob storage (enclosure.developer.mesh-ctp.com).

Soft state includes storage for device presence, notification queues, subscriptions, activities, and dictionary state for the Live Desktop. You can see most of these on slide 21 from Abolade’s PDC session:

The design of the Activity Service is similar to the notification and subscription services. This means the Activity Service uses only in-memory tables for high performance, so state loss is a possibility. Therefore you shouldn’t use Activities for any state that you can’t afford to lose. On the bright side, the high performance in-memory design of Activities fits together quite nicely with notifications and subscriptions.

~~The client LOE doesn’t appear to implement any of the soft state stores at this time. This is probably why client LOE MeshObjects don’t have an Activities link.~~ Update: The client LOE implements Notifications and Subscriptions, although they behave slightly differently from the cloud LOE. However, the client LOE still doesn't implement Activities.

How does Live Mesh use Activities?

One way Live Mesh uses Activities is to track which users are currently in a Live Folder. Here’s an example of such an activity:

<entry>
  <id>...</id>
  <title type="text">
  </title>
  <published>2009-02-21T08:27:15Z</published>
  <updated>2009-02-21T08:27:15Z</updated>
  <link rel="LiveFX/Member" title="LiveFX/Member" href="Mesh/MeshObjects/.../Members/..." />
  <link rel="self" title="self" href="Mesh/MeshObjects/.../Activities/...-..." />
  <link rel="edit" title="edit" href="Mesh/MeshObjects/.../Activities/...-..." />
  <category term="Activity" label="Activity" scheme="http://user.windows.net/Resource" />
  <content type="application/xml">
    <ActivityContent>
      <UserDataBuffer>
      </UserDataBuffer>
      <ActivityTime>2009-02-21T08:27:13Z</ActivityTime>
      <MaxAge>539</MaxAge>
      <Type>UserActivity:Type[Presence];LiveFolderId[::{4ba12b8a-865f-4f52-99a4-12901be64d54}];</Type>
    </ActivityContent>
  </content>
</entry>

The important pieces of information are the link to the user who is in the folder, the ActivityTime when they opened the folder, and UserActivity encoded in the Type field. LiveFolderId is the entry id of the folder’s MeshObject. Type[Presence] implies that other user activity types are probably tracked using the same format.

If you open a folder on the Live Desktop and watch the corresponding Activity entry, you will see that the Live Desktop does a PUT to reset the MaxAge when 15 seconds are left. All of the other entry details such as ActivityTime remain unchanged.

If you close the folder, the Activity entry is deleted immediately without waiting for MaxAge to expire.

It turns out that anything that has a Mesh companion bar tracks user presence activity. This means that Mesh-Enabled Web Applications also get this same behavior for free. In this case the UserActivity’s LiveFolderId is the entry id of the app’s MeshObject.

The Activity’s Member link is used by the Mesh companion bar to display an orange box next to users who are currently in the folder or app.

In this case you can see that I am playing Collaborative Crossword all by myself. Apparently Ray doesn’t have time for me anymore. ;-)

There is nothing stopping you from creating an Activity that makes another user believe you are in a folder or app when you’re not, although you would have to periodically update the Activity to continue to appear present. So if you must be present to spoof your presence, how much are you really spoofing? Hmm… It may also be possible to delete Activities to hide the fact that you’re in a folder or app, although I assume the Activity will eventually be recreated.

You would think that user presence should be determined by the combination of the Member link and the specially formatted Type field’s Presence and LiveFolderId, but it turns out that all you need to do to appear present is create any Activity whatsoever in a MeshObject’s Activities feed.

Source Code

I have source code that demonstrates much of what I’ve described, but it is part of a larger project that also demonstrates notifications and subscriptions, so I will publish the source with my notifications and subscriptions blog post. If you were surprised by how much there is to know about activities, just wait until you see notifications and subscriptions!

Mesh4Linux

2009-02-16T14:33:00.001-09:00

Last week my eyes lit up when I saw the following tweet from Miguel de Icaza, leader of the Mono project.

migueldeicaza

Miguel de Icaza

RT: @bradyanderson:is trying to authenticate my Linux Live Operating Environment using Windows Live Delegated authentication (Mesh4Linux)

4:21 PM Feb 11th from web

I immediately tried to find out more, but this is the only mention of “Mesh4Linux” I could find on the web. Brady is a distinguished engineer at Novell who works on the Mono project. I checked out his tweets, and he’s been talking about building a Live Mesh / Live Framework implementation for Linux ever since PDC.

bradyanderson

is attending the "What I learned building My first live mesh app" session at #pdc2008

11:35 AM Oct 28th, 2008 from twitterrific

is excited to start hacking on a Live Operating Environment for the Linux Desktop. Opens the door for some amazing applications

2:04 PM Oct 29th, 2008 from twitterrific

created his first MeshObject on Linux! Now DataFeed, DataEntry, Membership.... bows his head and goes away quietly

1:02 PM Dec 10th, 2008 from twitterrific

just implemented DataFeed creation and Mesh/MeshObject enumeration

4:06 PM Dec 11th, 2008 from twitterrific

"is hoping this is the first twitter message pushed from the sync framework (2)"

2:18 PM Jan 27th from web

is researching Differential Synchronization algorithms

10:04 AM Jan 28th from twitterrific

is catching up on the latest Live Mesh developments. I can't seem to find the January tools update :-(

8:11 AM Feb 11th from web

is trying to authenticate my Linux Live Operating Environment using Windows Live Delegated authentication - getting closer.

2:55 PM Feb 11th from web

the CTP versions of Live Mesh and Azure Services are so slow they're barely usable. *screams profanities*

3:10 PM Feb 11th from web

I’m guessing Brady is the guy Ori Amiga is referring to in the following quote from this PDC session (15:50 onwards):

The whole point I wanted to make, it's just plain good old HTTP, and if you can talk that, every device, programming language, stack is welcome to the party.

Some guy walked up to me after the stage yesterday, if you're here I'd love to keep chatting with you, said man I want to write Live Operating Environment for Linux. Can I do that? I was like, hell yeah, we'll hire you, come write it even in-house if you want to.

But really the idea is the Mesh will never be, I can't imagine we'll be successful in making people's lives better if we only stick to a Microsoft stack. That makes no sense. let's say, I'd admit, most of my devices at home and my receivers, my TVs, all the media stuff we have, the car, they don't run Windows, and that's ok, there's nothing wrong with that. It's great that my Windows devices are gonna behave really well in the Mesh, but I'd love for everything else that's sort of net connected to behave that way as well.

It is worth noting that Ori Amiga, a Principal Group Program Manager on the Live Mesh team, has built multiple carputers, one using Linux and another using Live Mesh.

I pinged Brady and Miguel for details on Mesh4Linux but haven’t heard back yet. There appears to be no connection to Mesh4x, another open source project with many similarities to Live Mesh, including FeedSync support.

Mesh4Linux is in the early stages at this point, but I’m already dreaming of the possibilities it will enable not just on Linux desktops but on the iPhone, Google Android phones, and embedded devices such as carputers. Go Brady!

2/28/09 Update:

Scott Hanselman used kyte to live stream Miguel’s Mono on iPhone session at the Alt.NET conference in Seattle. Via the online comments, I asked about Mesh4Linux (as well as Miguel’s ~~communist C#~~ Turkish flag t-shirt). Scott got a chance to ask Miguel about Mesh4Linux at the end (50:08) right after Miguel mentioned the benefits of sync for disconnected scenarios:

SH: What about Mesh4Linux?

MdI: There is no Mesh4Linux as far as I know. I know there is an engineer at Novell who wants Mesh4Linux.

SH: So it’s a dream, not a project.

MdI: Yes.

Exploring Live Framework Triggers

2009-01-15T16:53:00.001-09:00

The Live Framework has the ability to add triggers to resources. There is some documentation on triggers here and here (pgs. 14-15), but after reading it I was left with more questions than answers. So I took a deep dive exploring the nooks and crannies of triggers and this blog post is the result.

Overview of triggers

Triggers are scripts that can be executed before and after resources are created, updated, and deleted. The scripts are written using Resource Scripts (AKA MeshScripts), a tiny DSL for working with AtomPub and FeedSync in Live Mesh. Think of it as the T-SQL of Live Mesh. MeshScripts be used as sprocs as well as triggers, but I’ll be focusing on triggers in this post. See my previous posts for examples of sproc-style usage.

There are six triggers that can be attached to each resource:

PreCreateTrigger
PostCreateTrigger
PreUpdateTrigger
PostUpdateTrigger
PreDeleteTrigger
PostDeleteTrigger

The Create triggers run before and after each HTTP POST of a resource, the Update triggers run before and after each HTTP PUT of a resource, and the Delete triggers run before and after each HTTP DELETE of a resource. This enables you to pack quite a bit of custom business logic inside a single call to the server.

Trigger parameters

The resource that you’re creating, updating, or deleting is accessible from inside each trigger as a script parameter. For Create and Update triggers, the parameter is the actual resource sent from the client to the server in the POST or PUT request. For Delete triggers, the parameter is the server’s version of the resource being deleted since a resource isn’t sent from the client to the server for delete requests (the client simply specifies the URL of the resource to delete).

Three steps are necessary to use a script parameter:

Define the parameter
Bind to the parameter from one or more statements
Add the parameter to the script’s root statement

Here’s what this looks like using the syntax I created in my helper library. I’ve bolded the three steps.

var param =

    S.ResourceParameter<MeshObjectResource>();

mo.Resource.Triggers.PostCreateTrigger =

    S.Sequence(

        S.CreateResource(news)

            .Bind(s => s.CollectionUrl,

                param, p => p.NewsFeedLink)

            .Bind(s => s.Request.Title,

                param, p => p.Title)

    .AddParameters(param)

    .Compile();

The script snippet above adds a news entry to the news feed of the MeshObject you are creating (after it has been created, of course). You can see this code in the context of a working sample in the download at the end of this post. The sample also shows the equivalent “classic” syntax for the same trigger script.

Parameters are optional. If you don’t need to access the original resource from your trigger script then you can safely omit all three steps and simply create a trigger script without any parameters.

There is only one actual resource parameter per script. If you add more than one to the script, they are all treated as the same parameter. This makes sense since all resource parameters are named “$Resource” under the hood.

There is another kind of script parameter called the ConstantParameter that lets you specify a name for the parameter, thus letting you to have more than one of them per script, but I have been unable to get ConstantParameters to work so we’ll ignore them for now. I’m guessing they are used for looping statements which aren’t available in the current CTP.

Create/Update triggers

Create and Update triggers share many similarities, so I will cover them together.

Create and Update triggers are a one-shot deal. You must attach new Create or Update triggers each time you Add() or Update() the resource. Only the triggers appropriate for the HTTP verb are used. So for POST, the Create triggers are executed but the Update triggers are silently tossed, and for PUT, the Update triggers are executed and the Create triggers are tossed. By “tossed” I mean they aren’t executed, and the trigger is set to null in the response you get back.

In case it’s not clear, Create and Update triggers are not persisted on the server. They only exist for the duration of the HTTP request/response.

Unlike sproc-style MeshScripts, the trigger script’s Source property becomes null after the script has executed. At first I thought this was a bug, but then I realized that this was necessary so that if you then proceeded to call Update() on the item it wouldn’t re-run the same trigger again.

Just like sproc-style scripts, Create and Update triggers return the results of script execution in the Result property of the trigger script which you can inspect for details. Use them immediately or lose them because they won’t stick around for subsequent requests.

Original vs. updated values

The script parameter for Update triggers contains the updated resource being PUT by the client. If you need access to the original value that will be replaced by the PUT, you can access it in the PreUpdateTrigger using the following code, replacing MeshObjectResource with the appropriate resource type:

originalValue = S.ReadResource<MeshObjectResource>()

    .Bind(s => s.EntryUrl, param, p => p.SelfLink)

You can then bind to originalValue in subsequent statements. Note that “param” in the sample is the trigger script’s resource parameter.

Delete triggers

Only Delete triggers have a non-null Source property after a POST or a PUT. This is because only Delete triggers are persisted along with the resource on the server. Delete triggers can be added to a resource using either POST or PUT. Since Delete triggers are round-tripped (the Source doesn’t become null in the response), you don’t need to remember to re-add them on subsequent updates, unlike Update triggers. However, they are re-persisted each time you do an update. This means that you can remove a Delete trigger by setting it to null and calling Update().

Delete triggers are executed when you perform an HTTP DELETE on the URL of a resource that already has a Delete trigger added to it by a previous operation. Since no actual resource is posted or returned by the DELETE operation, there is no way to examine the script results or learn about errors.

How triggers deal with errors

They don’t. :-) To be more precise, errors are simply ignored. They don’t cancel the POST/PUT/DELETE operation. Similar to sproc-style scripts, no script Result is returned to the client if an error occurs. Unlike sproc-style scripts, the error is not returned to the client.

Transactions

While we’re on the subject of sproc-style scripts, it should be noted that sproc-style scripts are not transactional, and trigger-style scripts aren’t transactional either. Sure, they may execute within the scope of a single HTTP request/response “transaction” but there is no rollback on failure. Future releases are expected to include compensation/undo support.

Comparison to SQL triggers

Various databases support statement-level triggers and row-level triggers. Statement-level triggers are executed once for a batch of rows resulting from a single statement, while row-level triggers are executed once for each row. Statement-level triggers and row-level triggers attached to each table in the database.

While Live Framework triggers can inspect data “per row,” the triggers are actually attached to each “row,” not to each “table.” And as you already know, only Delete triggers actually remain attached to the “row.”

This means that it isn’t possible to put triggers on feeds (the equivalent of tables) that fire when entries are added, updated, or removed from the feed.

And as I explain in the next section, you can’t currently modify the incoming data before it is added or updated, unlike with SQL triggers.

Parameters are read-only (I think…)

At first I was under the impression that the incoming POST/PUT data exposed in the parameter to the PreCreate and PreUpdate triggers could be modified and the modified values would be passed along to the actual POST or PUT operation. I made this assumption based on the following quote from page 15 of this document:

"The output of the PreCreateTrigger can be data-bound to the actual POST request entity and the data is propagated dynamically in the request pipeline. Similarly, the response entity of the POST operation can be data bound to the PostCreateTrigger. A similar binding can be done using the PreUpdateTrigger to the request entity of the PUT operation and the response of the PUT operation and the PostUpdateTrigger. Note that such a model to flow the data dynamically between the PostDeleteTrigger script and the response entity is not applicable to the DELETE operation since we do not return response entity in the DELETE operation."

This sounds promising, but unfortunately I have been unable to find a way to update the script parameter.

The problem is that I can’t find a way to bind to the resource parameter. The resource parameter is exposed as a StatementParameter, not as a Statement. All of the Bind() methods that take a StatementParameter have the parameter on the right-hand-side. This means that you can assign from a resource parameter, but you can’t assign to it.

So I tried binding to “Parameters[0].Value” on the root statement of the script, but that didn’t work. Then I tried binding to the parameter using its secret “$Resource” name, but that didn’t work either.

Perhaps someone forgot to add the appropriate Bind() overload, or perhaps there’s another way to get at the parameter that I’m not thinking of. But until this is sorted out, parameters are read-only, at least on my box.

Once parameters can be modified, it will be interesting to see if you can completely replace the parameter (even set it to null?), or only update properties on it. It will also be interesting to see if you can delete the resource in the PostCreate trigger and return a completely different resource to the client. This could be a useful technique for creating singleton Mesh objects.

Triggers and the local LOE

Triggers don’t work at all if you’re connecting to the local client LOE. If you add triggers to a resource and then Add() or Update() it, the resource comes back with all its triggers set to null. This makes sense because the ability to execute scripts inside the client LOE is expected to be added in a later release.

But not even the Delete triggers are persisted and propagated up to the server. It turns out that Delete triggers also don’t propagate from the server down to the client. This made me nervous, wondering what will happen if I update a client-side resource that has a server-side Delete trigger. Will the absence of a client-side trigger clobber the server-side trigger? Thankfully the server properly merges the client-side update with the server-side resource’s Delete triggers. Must be some FeedSync magic.

Then I tried deleting a resource on the client that had server-side Delete triggers. The resource was successfully removed on the server, but the server-side triggers failed to execute! So synchronization bypasses triggers.

Speculation regarding client script execution

Once client script execution is added in a future release, how will this probably change the situation?

Create/Update triggers will run on the client if you connect via ConnectLocal().

Assuming synchronization of Delete triggers is fixed, you will be able to add Delete triggers on either the client or the server. If you delete the resource via Connect(), the trigger will run on the server. If you delete via ConnectLocal(), the trigger will run on the client.

But what if you want a trigger to always run on the server? Perhaps the trigger accesses external resources that you are unable to access while the client is offline. Or perhaps the trigger accesses resources that aren’t synced to the client such as Contacts, Profiles, or MeshObjects that aren’t mapped to that particular device. Perhaps there could be a client-side queue of pending triggers that are synchronized up to the server?

Creating triggers inside of scripts

Officially, you can’t add triggers to resources from inside of scripts. If you try, you will get the following error message: “Trigger can not be associated with a resource which is being modified using meshscripts.” Hey, look! They said MeshScripts! Personally, I think that’s a far better name than Live Framework Resource Scripts, as you can tell from the titles of my previous blog posts. :-)

Anyway, it is possible to add Delete triggers to resources from inside of a script. The trick is that you must copy them from a pre-existing resource, like so:

S.Sequence(

    originalCollection = S.ReadResourceCollection<MeshObjectResource>(ScriptHelper.MeshObjectsUrl)

    .WithQuery<MeshObjectResource, MeshObject>(

        q => q.Where(o => o.Resource.Title.StartsWith("Original"))),

    S.CreateResource(ScriptHelper.MeshObjectsUrl,

        new MeshObjectResource("I have delete triggers"))

    .Bind(s => s.Request.Triggers.PreDeleteTrigger,

        originalCollection, c => c.Response.Entries[0].Triggers.PreDeleteTrigger)

    .Bind(s => s.Request.Triggers.PostDeleteTrigger,

        originalCollection, c => c.Response.Entries[0].Triggers.PostDeleteTrigger)

).Compile().RunAtServer();

Technically, you can use this technique to add Create and Update triggers too. This can be verified by inspecting the the script result and seeing that the resource was returned with Create and Update triggers containing the Source script that you specified. However, these triggers don’t run. Why not?

Scripts bypass trigger execution

Just as synchronization bypasses trigger execution, scripts also bypass trigger execution. This is why our Create and Update triggers were added but didn’t run.

What happens if we use a script to delete a resource with Delete triggers on the server? The script deletes the resource without running its triggers.

Consequences of bypassing triggers

If you choose to use Delete triggers, you must be careful to do all of your Delete operations through direct HTTP DELETE calls to the server. Don’t use ConnectLocal(), and don’t use MeshScripts to delete resources.

This loophole could be useful in “oops” situations where you don’t want the triggers to run.

The bigger issue here is that you can’t reliably enforce server-side business logic. I spoke with Abolade about this after his PDC session and he mentioned that perhaps the content screening hook points (used to block enclosures containing viruses and other inappropriate content) could be exposed to users for running custom business logic that is capable of rejecting content. This could also be used to implement table-style triggers that are guaranteed to always run. At first I thought this would be cool to have, but now I’m starting to think that such a server-centric feature isn’t an appropriate fit with the design philosophy of Mesh. I may elaborate why in a future post.

Triggers on non-Mesh objects

Currently the root ServiceDocument at https://user-ctp.windows.net/ exposes Profiles and Contacts in addition to Mesh. I think these are known as Federated Storage Services, but I’m not sure. Contacts map out of the Mesh to your actual Hotmail contacts. Anyway, you access /Profiles and /Contacts using the same resource-based programming model as the rest of /Mesh. Anything that is a Resource can have triggers, so what happens if we add triggers to a Contact?

I added a new Contact containing Create and Delete triggers. The Create triggers worked, but the Delete triggers weren’t persisted and therefore didn’t run when I deleted the Contact.

I’m guessing there’s a service integration layer that translates back and forth between Mesh’s resource-based programming model and external services. The Contacts service probably doesn’t have a place to store arbitrary data such as triggers, so they get lost in translation. But the Create and Update triggers can still run because they don’t need to be persisted anywhere, so they can live entirely in the world of Mesh’s resource-oriented request/response pipeline that wraps the calls to the Contacts service. Hmm, maybe there are benefits to not having to persist triggers… But it would also be nice to have a consistent programming model for Create, Update, and Delete.

Summary of limitations

There are a number of limitations scattered throughout this blog post, so here’s a more concise list:

Create and Update triggers aren’t persisted
No row-level/statement-level triggers on feeds
Trigger parameters are read-only (I think)
Can’t add triggers from scripts
Synchronization bypasses triggers
Scripts bypass triggers
Delete triggers don’t work on non-Mesh objects
Local LOE doesn’t support triggers
Triggers can’t reliably enforce business logic

Download

You can download the sample code here. The samples use my Fluent MeshScripts library with a few minor updates.

While writing this I discovered and fixed a bug in my library’s expression-to-string code when it encounters expressions such as “c => c.Response.Entries[0].Triggers.PreDeleteTrigger”. I also created an AddParameters overload that takes an SResourceParameter<TResource>.

The code includes examples of using all the trigger types, creating a resource with triggers from a script, bypassing delete triggers with a script, triggers on Contacts, and the “can’t add triggers from meshscripts” error.

Wrap-up

Besides providing some detailed documentation and code samples for Live Framework triggers, hopefully this post has helped you think about scenarios where you might want to use them, as well as provided some pointers on when to avoid them or use them with care. I also hope this can be used to improve the usability and functionality of this powerful feature of the Live Framework.

Update: it appears that triggers don't work on DataFeeds and DataEntries. See Raviraj's post in this forum thread for details.

MeshScript Ideas for the Future

2009-01-07T16:32:00.001-09:00

In my last post where I added LiveItem syntax to MeshScripts, I said I had some more ideas for MeshScripts. Some of these ideas are very small, some are very big, and some are in between. The reason I’m listing them here is that there’s no way I could get to even a fraction of them (I’d like to move on and explore other areas of Live Mesh), so hopefully they spark your imagination.

Enhancements for existing MeshScripts

These enhancements could be applied to MeshScripts without taking a dependency on the library I wrote.

Chunking and chaining

There is an upper limit on the number of statements the server will process in a single script. It would be nice to implement automatic script chunking and chaining based on a configurable statement batch size. For cross-batch bindings, outputs from one batch can be fed into inputs for the next batch. Special care needs to be taken at CompoundStatement boundaries, especially with ConditionalStatement. This could also be used to implement a progress indicator for large jobs while still preserving most of the performance benefits of batching.

Automatic parallelization option

I’m guessing most people are going to write their scripts using SequenceStatement as their CompoundStatement of choice. It would be cool to have the option of automatically transforming and optimizing scripts by wrapping sections in InterleaveStatements where possible based on analysis of binding dependencies and URLs. This ought to be a feature the user explicitly opts into.

Enhancements to Fluent MeshScripts library

These enhancements are specific to my Fluent MeshScripts library.

Use control flow MeshScript features

Note that ScriptContext’s record-replay model doesn’t need any “programmatic” script control flow features such as ConditionalStatement or the coming-soon LoopStatement. Perhaps there’s an opportunity to add If/Else logic to the SLiveItem syntax. This would require an expression tree visitor to touch up references to script statements since currently you must hard-code the statement ID/Name in the condition.

A switch statement could be added that uses multiple ConditionalStatements under the hood.

A similar sub-ScriptContext scoping solution could be used for generating InterleaveStatement sections.

Until we get a real LoopStatement, a fake Loop statement could be created that unwraps the loop a specified number of times. This could be used for scenarios such as copying the last 10 items from a Twitter feed into a Mesh feed.

Miscellaneous enhancements

Create a helper method that enables single-round-trip Live Folder creation that returns LiveItems.

I haven’t investigated expansions yet, but it seems like there’s an opportunity to take advantage of them in MeshScripts, preferably with helper methods to simplify common scenarios, whatever those might be.

Taking MeshScripts in bold new directions

Here are some wild and crazy ideas that probably won’t come to pass, but wouldn’t they be cool?

Add batching to LiveItems

It would be nice if LiveItems had the option of operating either in batched mode (like my SLiveItem implementation) or real-time mode (as they currently do). Perhaps this could be enabled through a new Batched property on LiveItemAccessOptions. This means LiveItems would be able to speak both AtomPub/FeedSync and MeshScripts.

Escaping the MEWA sandbox

This next one is more of a “see if it’s possible” item than a new thing to implement, but it could open up some interesting new scenarios that would be ripe for additional library development. It would be interesting to see if MEWAs can use MeshScripts to escape the MEWA sandbox, either by calling them as sprocs or as triggers. Of course if this is possible, it’s likely it will be quickly disabled, but who knows, perhaps it’s acceptable. I tried running a simple ReadCollection script from a Silverlight MEWA and got an exception trying to deserialize the result (no public default constructor), so I haven’t pursued it further.

Yahoo Pipes for AtomPub

Yes, building Yahoo Pipes for AtomPub and FeedSync would involve much more than just MeshScripts, but I think MeshScripts could play an important role in its implementation, especially with the forthcoming visual script designer.

This idea first came to me as I was experimenting with pulling in external Atom feeds using MeshScripts (it’s also possible using LiveItems). Most feeds had formatting that broke the script, but a few external Atom feeds magically worked. I thought, wouldn’t it be nice if there were a MeshAtomTidy service that touched up external feeds with the appropriate data to ensure they load nicely into the Mesh?

That would be great for read-only feeds. Wouldn’t it be even nicer if you could map LiveID credentials to external credentials and access other AtomPub APIs such as Google Calendar, Google Spreadsheets, Picasa, and more? It would also be cool if you could write a little bit of glue code or script to wrap an AtomPub API around non-AtomPub APIs such as Twitter. Or even better, just select from a list of pre-existing AtomPub wrappers for popular services.

Next, I’d like to enable automatic synchronization between Mesh feeds and external feeds. It’s not very exciting to send tweets while you’re offline, but offline access to the Google APIs is more compelling. There may be a need for additional transformations and business logic in between which is where the full-featured suite of Yahoo Pipes modules comes in handy.

Popfly integration

The closest Microsoft equivalent to Yahoo Pipes is Popfly. It has a similar set of modules and a drag-and-drop design experience. Perhaps there is an opportunity to integrate Popfly mashups with Mesh feeds. It would also be cool if you could package Popfly games as MEWAs that can run on your desktop or on your phone and maybe even sell them through a Live Mesh App Store, but that’s probably enough crazy talk for one blog post. :-)

MeshScript Queries, LiveItems, and Magic

2009-01-07T16:27:00.001-09:00

I’ve continued to extend my Fluent Resource Scripts library in several interesting new ways. I’ve added:

Strongly typed LINQ queries
Turning script results into LiveItems (MeshObject, DataFeed, DataEntry, etc.)
LiveItem syntax for scripts

I’ve had to resort to more significant hacks to implement these, and I don’t feel that these features are as solid or as complete as the features in my original Fluent MeshScripts post. But I think these ones are far more interesting, so I hope you will look past the rough edges and imagine the potential if these features were done properly. The LiveItem syntax for scripts is especially cool, if I do say so myself.

Queries

Both LiveQuery and ResourceQuery let you generate query strings from strongly-typed LINQ queries. ResourceQuery is broken, which is unfortunate because its only generic parameter is of type Resource, which also happens to be the only generic parameter for most script statements. This would have allowed us to implicitly pass along the statement’s generic parameter to our helper method without having to write any generic angle brackets.

So we have to use LiveQuery instead, which takes a generic parameter of type LiveItem (the non-generic LiveItem, not LiveItem<TResource>). This makes it so that instead of calling my helper method like this:

S.ReadResourceCollection<MeshObjectResource>(ScriptHelper.MeshObjectsUrl)

    .WithQuery(
   q => q.Where(o => o.Resource.Title.StartsWith("my")))

You must instead call it like this, specifying both the Resource type and the LiveItem type:

S.ReadResourceCollection<MeshObjectResource>(ScriptHelper.MeshObjectsUrl)

    .WithQuery<MeshObjectResource, MeshObject>(
   q => q.Where(o => o.Resource.Title.StartsWith("my")))

Oh well, it’s still useful, and once ResourceQuery is fixed we can switch to the shorter syntax.

I should note that these queries are immediately turned into query strings under the hood. In other words, they are evaluated at script design-time, not at script runtime. I have put in a feature request to support query generation at runtime.

Turning script results into LiveItems

So you’ve written a resource script, you run it, and you get some results back. Then you think, “I’d like to do further work with the results I’ve gotten back.” You dig into the individual statements in the script’s Result property, cast them to the appropriate statement type so you can then dig into the Resource property and examine the actual data that was returned.

But what if instead of working with results MeshScript-style, you wanted to work with them LiveItem-style, without having to do a bunch of tedious digging and casting?

DataFeed outFeed = null;

using (new ScriptContext(loe))

    S.Sequence(

        ..., // create moStatement

        S.CreateResource(scriptFeed)

            .AtUrl(moStatement, mo => mo.Response.DataFeedsLink)

            .SaveResult(ref outFeed)

    ).Compile().Run();

DataEntry de = new DataEntry("new entry");

outFeed.DataEntries.Add(ref de);

I’m not quite sure what programming idiom to compare SaveResult() to since the result isn’t actually saved until the script is run. The closest thing I can think of is a future.

SaveResult() creates a new LiveItem of the appropriate type, and after the script is run, this LiveItem is filled in with all of the necessary response information to give you a “live” LiveItem that you can start partying on right away without requiring another server round-trip to flesh it out. This required a fair bit of reflection magic because by default LiveItems are essentially “DeadItems” until they’re associated with a LOE and Reloaded.

Notice I’ve added the use of a ScriptContext to enable the generated LiveItems to be automatically associated with an existing LiveOperatingEnvironment.

SaveResult() has been implemented for CreateResourceStatement, ReadResourceStatement, and ReadResourceCollectionStatement. I have also added Statement extension methods ToMeshObject(), ToDataFeed(), and ToDataEntry() if you prefer to work that way.

When you combine ReadResourceCollection with SaveResult() and the query support above, you can now batch multiple LiveItem queries.

LiveItem syntax for MeshScripts

So we’re starting to get a decent bridge between the world of MeshScripts and the world of LiveItems. But man, that MeshScript syntax is still pretty nasty, even with the fluent stuff I’ve added. Wouldn’t it be nice if you could write a MeshScript the same way you write LiveItem code?

using (new ScriptContext(loe))

    var mo = new SMeshObject("original title");

    var feed = new SDataFeed("first feed");

    var feed2 = new SDataFeed("second feed");

    var feed3 = new SDataFeed("third feed");

    mo.Resource.Title = "script-generated title";

    mo.Resource.Type = "LiveMeshFolder";

    feed.Resource.Type = "LiveMeshFiles";

    feed2.Resource.Title = feed.Resource.Title;

    feed3.Resource = feed.Resource;

    mo.DataFeeds.Add(feed);

    mo.DataFeeds.Add(feed2);

    mo.DataFeeds.Add(feed3);

    var entry = new SDataEntry("my entry");

    feed.DataEntries.Add(entry);

The only syntactic differences in the code above are the S-prefixes on the various SLiveItems, the absence of ref parameters, and there are no explicit calls to loe.Mesh.MeshObjects.Add() for the SMeshObject, although that extra syntactic hoop could easily be enabled.

Yes, that code results in just one round-trip to the server. “You’re kidding,” you say? “Where’s the man behind the curtain?” Let’s see that again, annotated with comments.

using (new ScriptContext(loe))

    // CreateResourceStatements

    var mo = new SMeshObject("original title");

    var feed = new SDataFeed("first feed");

    var feed2 = new SDataFeed("second feed");

    var feed3 = new SDataFeed("third feed");

    // set properties at runtime using ExpressionBindings

    mo.Resource.Title = "script-generated title";

    mo.Resource.Type = "LiveMeshFolder";

    feed.Resource.Type = "LiveMeshFiles";

    // bind sFeed's Response.Title to sFeed2's Request.Title

    feed2.Resource.Title = feed.Resource.Title;

    // bind sFeed's Response to sFeed3's Request

    feed3.Resource = feed.Resource;

    // bind DataFeedsLinks to CollectionUrls

    mo.DataFeeds.Add(feed);

    mo.DataFeeds.Add(feed2);

    mo.DataFeeds.Add(feed3);

    // CreateResourceStatement

    var entry = new SDataEntry("my entry");

    // bind DataEntriesLink to CollectionUrl

    feed.DataEntries.Add(entry);

} // automatically run the script

In ORM terms, the ScriptContext now functions as a UnitOfWork that automatically saves all items created or modified within its scope. You can think of it as a script recorder that replays what it has recorded when it’s done. In addition to the LOE parameter, ScriptContext also takes an optional RunLocality parameter that determines whether the script is executed by the client or by the server. It defaults to running at the server.

The CreateResourceStatements were surprisingly straight-forward to implement. So were the Add() methods.

The property getters and setters required a bit more magic. String-based property getters all return a magic string that specifies the binding. String-based property setters generate a PropertyBinding if they are passed a magic string, otherwise they generate a constant ExpressionBinding with the string they are given. SResource-based setters generate a PropertyBinding. The magic string approach could also be used for Uri-based properties, although I didn’t implement those in this prototype.

At first I generated script statements in the order that the SLiveItems were created, but later I added dependency-tracking so that property assignments and Add() calls re-order the statements if necessary.

Another advantage of late SLiveItem statement execution is that you won’t run into NullReferenceExceptions if you Add() SLiveItems to other SLiveItems in the “wrong” order. That forum thread was actually part of the inspiration for imitating LiveItem syntax.

In that same forum thread I also detailed how LiveItems lead a dual life as a request and as a response. A similar duality exists with SLiveItems. When assigning properties, the object on the left-hand-side sets properties on its request, while the object on the right-hand-side gets properties on its response.

Each SLiveItem has a Result property that returns a “live” LiveItem once the script has been run by explicitly calling ScriptContext.Current.CreateScript().Run(). This uses the SaveResult() technique described earlier.

Each SLiveItem exposes a WrappedStatement property that lets you make changes to the underlying Statement if you need that escape hatch into script-land to touch things like the request resource.

Using SLiveItem syntax with triggers

With these tools in our bag, we are now ready to make triggers more accessible.

First, I created a new type, SResourceParameter<TResource> and a factory method, S.ResourceParameter<TResource>(). This enables a more strongly-typed Bind() call than what I had in V1.

Then I updated the Bind() methods to tell the ScriptContext about any ParameterBindings so they can be automatically added to the auto-generated root script statement.

Finally, I decided to try out yet another syntax for Bind() that looks like Set().EqualTo(). It exists as a method on SLiveItem instead of as an extension method for Statements and I only implemented it for Resource ParameterBindings, but it could just as easily be applied to all of the other binding types.

This lets us write:

using (new ScriptContext(loe))

    var originalObject = new MeshObject("Original object");

    var triggerParam = S.ResourceParameter<MeshObjectResource>();

    var triggerCreatedObject = new SMeshObject("trigger-created object");

    triggerCreatedObject.Set(s => s.Request.Title).EqualTo(triggerParam, p => p.Title);

    originalObject.Resource.Triggers.PostCreateTrigger = ScriptContext.Current.CreateScript();

    loe.Mesh.MeshObjects.Add(ref originalObject);

The Set().EqualTo() syntax isn’t as nice as the LiveItem property getter/setter syntax, but Set().EqualTo() was quick to write and I didn’t feel like taking more time to write the wrapper that would enable plain old property support. I’m sure it could be done.

What remains to be done?

The LiveItem syntax turned out to be quite a bit more work than I was expecting, and it’s still nowhere near being done. In fact it may have some major flaws that necessitate a rewrite, I’m not quite sure yet. One major issue is that it currently only supports CreateResourceStatements. Adding support for other statement types could have a huge ripple effect. The use of magic strings is another design decision that may cause headaches down the road, but so far I’m getting away with it, and the alternatives aren’t nearly as nice to use.

I started out without any generics in the SLiveItem and SResource base classes but later introduced the rather insane explosion of generics to consolidate repetitive functionality from the derived classes. It may be desirable to first undo the base class generics to create some mental breathing room and then add more functionality.

SLiveItem wrappers need to be created for Contact, Mapping, Member, Device, News, and Profile. This may require creating additional wrapper classes for the helper classes they depend on such as NewsItemContext.

As mentioned in the previous section, Uri properties need magic string support, and ParameterBindings need LiveItem-style property assignment support.

Once the ResourceQuery bug is fixed, WithQuery should use it instead of LiveQuery. If the runtime query feature is implemented, this should also be supported.

SLiveItem needs to implement Update() and CreateQuery().

The current implementation freely reorders statements. It also assumes that when you’re doing property assignments, everything on the left-hand-side is a request and everything on the right-hand-side is a response. If you set the same property more than once, only the last set sticks, and getting the same property at different points always returns the same value. This may result in unexpected behavior if the user is depending on the same property having different values at different points in the script. This could be dealt with by generating more than one statement per SLiveItem and/or by generating AssignStatements.

Per the comments in the code, my usage of ETags when generating LiveItems from script results may not be correct. There are also a number of other TODOs in the code comments.

Last but not least, this needs unit tests. More interfaces such as IScriptContext probably need to be created to enhance testability.

I’m sure there’s more I’m forgetting. I told you, it’s a lot of work! :-)

Download

You can download V2 of the Fluent MeshScripts library here. I have enhanced the console app with additional examples that demonstrate most of the new features in this post.

To run the sample you will need to change the username and password. If the project references to Microsoft.LiveFX.Client.dll, Microsoft.LiveFX.ResourceModel.dll, and Microsoft.Web.dll are broken, you will need to remove and recreate them in both projects.

Wrap-up

I was going to discuss future directions for this library and for MeshScripts in general but this post has gone on for long enough so I’ll save that for my next post.

Hopefully my enhancements help you visualize more possibilities for MeshScripts. At the very least, they should make MeshScripts much easier for you to write and work with.

As always, I’d love to hear any feedback.

Fluent LiveFX Resource Scripts

2008-12-20T23:22:00.001-09:00

After writing my Resource Script demo post, I’ve been digging deeper into Live Framework Resource Scripts. Along the way I’ve written a helper library to make them easier to work with. My enhancements focus primarily on keeping your scripts strongly typed and enabling a more concise fluent interface syntax. These enhancements let Intellisense help you out quite a bit more, resulting in greater discoverability and productivity.

I must warn you that the following discussion won’t make much sense unless you’re already somewhat familiar with Resource Scripts. I apologize and promise to follow up in future posts with material that’s more suitable as an introduction, using my library of course. :-)

Strongly typed bindings

If you’ve played with Resource Scripts at all, you’ve almost certainly run into Bindings. These creatures consume magic strings such as “EntryUrl”, “CollectionUrl”, “Request.Title”, “Response.SelfLink”, “Response.DataFeedsLink”, and “Response.DataEntriesLink” to name a few. Yuck! To make matters worse, the types of Request and Response are usually (but not always!) generic parameters to a statement, meaning that their available sub-properties will vary based on the generic type. Also, some statement types don’t have Request, and others don’t have either Request or Response. It would sure be nice if I don’t have to consult MSDN documentation or Reflector each time I write a binding statement. Bindings are strongly-typed at runtime, so why not at design-time too?

Then this post popped up in Google Reader and reminded me that I can generate those icky dirty strings from nice shiny expression trees, just like LINQ to SQL generates SQL from strongly-typed C# statements. So my Bindings can go from this:

Statement.CreateResource("feedStatement", null, dataFeed,

    Statement.Bind("CollectionUrl",

        "folderStatement", "Response.DataFeedsLink"));

to this:

S.CreateResource(dataFeed)

    .Bind(df => df.CollectionUrl,

        folderStatement, fs => fs.Response.DataFeedsLink);

Due to the way I’ve defined the generic parameters on the lambda expressions, the types of the source property and the target property have to match. If they don’t, you get immediate red squiggly feedback in Visual Studio. I’m not sure if that’s a Visual Studio thing or a Resharper thing, but that’s how it works on my box. At the very least you will find out at compile-time instead of at runtime.

Besides the lambda expressions and the ability to call Bind() on the statement after it has been created, it’s worth noting that the Statement.Name “folderStatement” string has been replaced with a reference to the source Statement itself. No more remembering statement names (until we get to ConditionalStatements that is…).

Simpler statement construction

So what was that “S” thing in the previous example? That’s my static utility class that offers methods equivalent to most of the static factory methods on the Statement class. Methods in “S” have the same names, but they typically have fewer parameters, resulting in a more concise syntax when you’re data binding. They also give you the option of using strings instead of Uri objects.

Yes I know, it’s not fair that my utility class gets the short, easy to type name while “Statement” makes you type twice as much before Intellisense kicks in and wastes a bunch of horizontal space. So put “using S = Microsoft.LiveFX.ResourceModel.Scripting.Statement;” at the top of your code if that makes you feel better. :-)

You may also have noticed that I didn’t supply a name for the statement. All of the factory methods in “S” automatically generate a random statement name so that every statement is inherently bindable. If you need a well-known name for inspecting script results or for use in a ConditionalStatement, you can use the NameStatement() extension method:

S.CreateResource(dataFeed).NameStatement("myName")

NameStatement() also checks for valid statement names so you find out at design-time rather than at runtime that statements can’t start with a number and can’t contain spaces.

Functional statement construction

Once you start writing utility methods to generate groups of statements that you string together into a script, you quickly run into the issue that you’re always having to write little bits of shim code to repackage your statements into a single Statement[] before feeding them into your CompoundStatement of choice (Sequence, Interleave, or Conditional). Wouldn’t it be nice if you could throw anything you wanted into a CompoundStatement and it would all be taken care of, similar to the XElement constructor in LINQ to XML?

If you’re not familiar with the XElement constructor, it looks like this:

public XElement(XName name, params object[] content)

It’s a bit loosey-goosey with the object[] parameter, but according to the documentation it allows you to pass in objects that are (or can be converted to) XML nodes, as well as IEnumerable<T> of such objects. Null content is silently ignored. Anything else results in an exception at runtime.

When this style is applied to CompoundStatement construction, it enables the following code:

S.Sequence(

    readMeshObjects,

    conditionallyCreateFolders,

    createAnotherFolder,

    createFiles)

That doesn’t look very interesting without the type declarations, but imagine the first two parameters are different types of Statements, the third parameter is a custom object that implements IEnumerable<Statement>, and the last parameter is a Statement array. You can see this code in action in the IfElseSample in the download.

Binding to URLs and Requests

One of the most common uses of bindings is to perform CRUD operations on a URL that comes from the result of a previous statement. Specifying the target URL in the binding can become quite repetitive. The property name for the target URL also varies by statement type. Sometimes it’s EntryUrl, sometimes it’s CollectionUrl, and sometimes it’s MediaResourceUrl.

I address this with the AtUrl() extension method which eliminates the need to specify the target property and lets you write:

S.CreateResource(dataFeed)

    .AtUrl(folderStatement, fs => fs.Response.DataFeedsLink);

A similar WithRequest() extension method exists for binding to the Request property on CreateResource, UpdateResource, and SynchronizeResourceCollection.

Conditional statements

The ConditionalStatement is worth an entire blog post. Until then, here’s an example of the syntax I’ve enabled:

S.If(statement =>

     (((ReadResourceCollectionStatement<MeshObjectResource>)

       statement.FindStatement("ReadObjects")).Response.Entries

       .Where(mo => mo.Title == "My Folder").Count() == 0))

    .Then(ScriptHelper.CreateFolder("Folder didn't exist"))

    .Else(ScriptHelper.CreateFolder("Folder DID exist")

I should note that ConditionalStatement already exposed the ability to use lambda expressions. All I did was enable the If().Then().Else() syntax. The Else() is optional.

Strongly typed statement groups

Notice the CreateFolder() helper method in the previous example? Originally this method returned a Statement[] containing two statements. The first statement created the MeshObject that represents the folder and the second statement created a DataFeed at the DataFeedsLink of the folder. This Statement[] was sufficient for creating a folder with a given name, but if I wanted to do something interesting with it such as put files in the folder or use a binding to change its title, it quickly became a pain to grab the appropriate entry from the array and cast it to the correct type.

So I created a helper class named CreateFolderStatementPair that exposes strongly typed properties named FolderStatement and FilesFeedStatement. This lets you write:

S.CreateResource<DataEntryResource>()

    .AtUrl(folder.FilesFeedStatement, f => f.Response.DataEntriesLink)

and

folder.FolderStatement.Bind(mo => mo.Request.Title, "new title");

CreateFolderStatementPair inherits from an abstract class named StatementGroup which implements IEnumerable<Statement> and also has an implicit operator conversion to Statement[]. Implementing IEnumerable<Statement> means you can pass a StatementGroup into S.Sequence(), S.Interleave(), and the Then()/Else() methods. The implicit conversion to Statement[] means you can pass a StatementGroup into methods that expect a Statement[] such as the original Statement.Sequence() method. You can use StatementGroup to create your own strongly typed group of statements that play well with bindings and with the S.*/Statement.* factory methods.

Miscellaneous helpers

Besides CreateFolder(), The ScriptHelper static class has a few other useful methods. CreateMedia() takes a CreateResourceStatement<DataFeedResource> and an external media URL and does a CreateMedia at the MediaResourcesLink of the CreateResourceStatement. There is also a CreateMedias() method that takes multiple media URLs. ScriptHelper has a few other convenience properties and methods, but nothing noteworthy.

There are a few other extension methods I haven’t mentioned yet.

ToSequence() and ToInterleave() turn a Statement array into a SequenceStatement or an InterleaveStatement respectively.

AddBindings() and AddParameters() let you add bindings and parameters to statements after they have been created.

FindStatement<TStatement>() recursively finds the first statement of the specified type in an IEnumerable<Statement>.

A Compile() method has been added to all statement types, not just CompoundStatements. Run() and RunAtServer() with an implicit Compile() have also been added to any Statement type.

Run() and RunAtServer() no longer require any parameters if you first call ScriptHelper.SetCredential(username, password).

Download

You can download the code here. The solution contains a console app that demonstrates some of the features with a few sample scripts.

To run the sample you will of course need to change the username and password. If the project references to Microsoft.LiveFX.Client.dll, Microsoft.LiveFX.ResourceModel.dll, and Microsoft.Web.dll are broken, you will need to remove and recreate them in both projects.

Conclusion

If you’re wondering why Resource Scripts didn’t have these features already, you need to remember that Resource Scripts were designed to be written using a visual designer tool similar to the Windows Workflow designer. The team was also under intense pressure to make the CTP available in time for PDC.

Think of this library as an experiment to see what a more code-centric API might look like and whether it could co-exist with a visual designer tool. Who knows, maybe some of the concepts such as strongly-typed StatementGroups might find their way into such a visual designer.

Hopefully this library enables and encourages more people to play with Resource Scripts. If you have any feedback, I’d love to hear it. Have fun scripting your Mesh!

L1v3 M35H L337 H4x0rZ

2008-11-06T18:33:00.001-09:00

In case you aren’t already persuaded that the Live Mesh team are a bunch of L337 H4x0rZ, check out the ids of entries in the Profiles feed.

id	title
G3N3RaL	GeneralProfile
480u7Y0U	AboutYouProfile
k0n74C7Inf0	ContactProfile
wORK1nfo	WorkProfile
1n7eRE572	InterestsProfile

I believe this also demonstrates their far-reaching commitment to open web standards and the new generation of social apps. Or perhaps the hidden message is “so easy, even script kiddies can hack it!”

You can see this for yourself by firing up the Live Framework Resource Browser (LivefxResourceBrowser.exe from the SDK tools) and drilling down into Cloud LOE > Profiles.

On a slightly related note, as I was digging around with the Resource Browser I discovered that the following two URL styles appear to be interchangeable.

https://user-ctp.windows.net/V0.1/cid-1234567890123456789/Profiles

https://user-ctp.windows.net/V0.1/email-abc@live.com/Profiles

I’m not sure if the email identifier format is stable enough to bank on, but it’s convenient for typing or tweaking URLs by hand. Does anyone know of other equivalent identifier types in Mesh resources?

Live Mesh Resource Script Demo

2008-11-05T12:51:00.001-09:00

In the Live Framework Programming Model Architecture and Insights session, Ori Amiga (standing in for Dharma Shukla, previously a WF architect) demos a Live Mesh resource script that runs in the cloud. The script creates a folder on the Live Mesh desktop and downloads two images from external resources, placing them in the newly created folder.

I couldn’t find this sample on the web, so I recreated it from the session video. You can download the demo project here.

You may need to touch up the references to Microsoft.LiveFX.Client.dll, Microsoft.LiveFX.ResourceModel.dll, and Microsoft.Web.dll since they live under C:\Program Files (x86)\ on my 64-bit box and are probably under C:\Program Files\ if you’re running 32-bit.

At first, my demo threw an error trying to run the following line:

script.RunAtServer(creds);

After investigating with Reflector, I discovered that RunAtServer() hard-codes a default script URL of https://user.windows.net/V0.1/Script/ which needs to be changed to https://user-ctp.windows.net/V0.1/Script/ . You can override this either by calling an overload of RunAtServer() that takes a URI, or by creating an App.config file and adding the following line in the <appSettings> section.

I chose to use the appSettings solution since that is what Ori must have used in the demo.

I really would prefer ScriptUrl to be exposed as a property on ResourceScript<> that appSettings/ScriptUrl maps into rather than having to specify the URL either in config or in each method call. My philosophy is that you should always be able to do in code what you can do in config.

I’m looking forward to playing more with Live Mesh resource scripts, documented here. Right now they feel a bit convoluted to create programmatically, but they appear to be designed to put a friendlier layer on top such as an Oslo DSL or a “resource workflow designer”.

Dissecting Live Mesh App Packages

2008-11-04T23:03:00.001-09:00

After bundling Flash inside a Mesh app, I took a closer look at what Visual Studio is doing behind the scenes. The Mesh-enabled Web Application template creates a project with a .meshproj extension. A .meshproj file has several important properties. <OutputName> is the prefix used to name the resulting zip file. <DebuggerCodeType> is set to either JavaScript or Silverlight, depending on whether you create an HTML/JavaScript or a Silverlight Mesh app. <ApplicationUri> is the Application Self-Link that you are supposed to copy-and-paste from the Developer Portal after you upload the zip file, as instructed below:

A .meshproj file also imports $(MSBuildExtensionsPath)\Microsoft\Live Framework\v1.0\Microsoft.LiveFramework.targets which first ensures that your project has an index.html and then zips up the output directory, naming the zip file using the OutputName you specified.

You don’t need to use Visual Studio to do this. You can easily create your own Mesh app package by hand. At a minimum, your zip file must contain:

index.html
Manifest.xml
Logo.png

Index.html is the entry point for your app. Logo.png is the icon that will be displayed on your desktop and should be a 32-bit 256 x 256 png. Manifest.xml is your app manifest. A detailed description of the manifest configuration options is documented here.

I believe you can bundle anything you want in the zip file, although Microsoft supposedly runs an antivirus scan on the contents, and there may be additional checks for inappropriate content. Anything in the zip file gets downloaded to your computer when the app is installed on your local desktop. This is why my Flash app was able to run offline.

You might be able to use this to “install” an XBAP application that can run offline. To make it cross-platform, you could bundle the XBAP together with a “down-level” Silverlight version and choose which one to display based on what the client supports. If download size is a concern, it might be possible to put the executables in a DataFeed instead of in the app zip file and selectively sync only the version you want to display, but I haven’t dug into DataFeeds enough yet to see if this kind of per-client sync filtering is possible. Of course you would be working against the built-in versioning management if you did this (updates should only occur when the user closes and re-opens the app).

Ok, so uploading a zip file sounds nice and simple, right? Then why does Visual Studio want me to copy-and-paste the Application Self-Link URI? It turns out that if you use Visual Studio you only upload the zip file once per app. Once you’ve uploaded the zip and told Visual Studio about the Self-Link URI, Visual Studio will use that URI for subsequent deployments to upload the individual files directly.

If you watch Visual Studio using Fiddler (you’ll need to configure HTTPS support) you will see it query the Mesh for the resource feeds of your app, do HTTP DELETEs for each resource that was inside your zip file, and then do a bunch of POSTs to upload each item in your project. That seems a bit risky. What if Visual Studio dies before reposting all the resources it deleted? It seems like updating an app by manually uploading a zip file is a safer, slightly more atomic operation. It’s no big deal right now, but once real production apps are being upgraded, something more robust would be nice. I’m guessing we will see more explicit versioning, giving the user the choice of whether or not to upgrade. If such a feature is added, the direct app resource update trick might be useful for bypassing an explicit upgrade prompt.

The next time your Live Mesh client (MOE.exe) talks to the cloud, it will download the new versions of the files into your local app cache (AppData\Local\Microsoft\Live Framework Client\Bin\Moe2\MR\). For some reason I was unable to pinpoint the download traffic with Fiddler, so I can’t say for certain whether individual files are downloaded or if they are zipped up first. It appears older versions of files aren’t removed. This is probably to support the explicit user upgrade scenario in the future, but it seems like they could still be doing more cleanup.

I’m really curious why Visual Studio updates individual app resources rather than following the documented workflow of uploading a zip file with the updates. Anyone know?

Update: I posed this question in the comments on Danny Thorpe’s blog and he responded:

On your second question, the reason we upload files individually instead of uploading the zip file is because the REST API we’re uploading to doesn’t handle zip files. The dev portal that you manually upload your zip file to unzips the file and uploads the individual bits to the production storage. The Live Services REST APIs that the VS tools use to upload files goes (as far as I know) straight into the production storage.

In a nutshell, the dev portal that you see in your web browser is just a front end to the actual cloud service. VS doesn’t upload to the dev portal UI, it uploads to the cloud itself.

Keep in mind that the long side trip of manual steps that you currently have to go through to get a new app created and uploaded to the cloud will all be going away as soon as the cloud APIs to create and provision a new application are implemented.

He also explains the debugging versioning scheme in the comments, and I suggest you go read it for more great details.

One other related insight from Danny comes from this forum thread:

Our goal for the VS tools is to do all development against the local LOE and let the local LOE deal with sync'ing things back to the cloud. All the parts needed to do that aren't ready yet, so for the PDC CTP we redirected the VS tools to upload and debug mesh apps in the cloud.

This makes the current chattiness (and the “glue” dialog box) much more acceptable to me since the end goal is to use the local REST API rather than the cloud API.

Update 2: Danny has posted a thorough response to this post. There’s lots of great information there, so I won’t quote it all here. One “aha” moment for me was the concept of separate debug application resources. He also confirms that the “glue” dialog will be going away soon. Go read it for details. Thanks, Danny!

Live Mesh + Flash == Adobe AIR

2008-10-31T17:23:00.001-08:00

Yes, that’s an Adobe Flash app running as a Live Mesh app, and it was easy. Feel free to install the app and try it out for yourself.

First I snagged a pre-existing .swf file since I am not a Flash developer. Then I created a new Visual Studio project using the Mesh-enabled Web Application template that comes with the Live Framework Tools for Visual Studio. I added the .swf file to the project with the default build action of Content and copied-and-pasted the object embed tag into the body of index.html. Then I ran through the usual Ctrl-F5 steps to upload and deploy the resulting zip package and boom, it just worked! I was able to use the app in the browser in my Live Desktop, and an icon for the app magically appeared on my Windows desktop that let me run the app offline, “outside the browser” (I believe MeshAppHost.exe actually hosts a chromeless IE browser control). I’m guessing you would also get the same desktop experience using the Mac Tech Preview.

If I were an actual Flash developer, I would take it to the next step and call the Mesh APIs using the Microsoft.LiveFramework.js library. That should “just work” as ActionScript, right?

If Flash for Windows Mobile appears before Silverlight for Windows Mobile, this could make for a very interesting deployment option when combined with the Live Mesh client for Windows Mobile once it supports Mesh apps. Three days ago Amit Chopra announced that a public CTP of Silverlight 2 for Mobile will be available in Q1 of 2009. I’m guessing this will coincide with MIX09 which starts March 18.

The skeptics might say hosting Flash in a Live Mesh app is an unsupported hack that Microsoft will quickly disable, but I don’t think so. David Chappell’s whitepaper Introducing the Azure Services Platform specifically states:

“A mesh-enabled Web application must be implemented using a multi-platform technology, such as Microsoft Silverlight, DHTML, or Adobe Flash. These technologies are supported on all of the operating systems that can run the Live Framework: Windows Vista/XP, Macintosh OS X, and Windows Mobile 6.”

I think this is a very cool option that highlights the fact that Microsoft designed Mesh to be an open platform with the broadest possible reach.

Update:

A whitepaper just published by a Program Manager and an Architect on the Live Framework team contains the following quote that confirms Flash support:

What application types are supported by the Live Framework?

The Live Framework supports client side applications of all types including the following application
types on Windows to interact with Client or Cloud versions of Live Operating Environment:

Browser based apps (Javascript, Flash and Silverlight) on IE, Firefox and Safari

Managed desktop applications written using WPF, WinForms, or other languages like Python,
Ruby, or Perl. All you need is an HTTP Client stack in your programming environment of choice.

Traditional native Win32 applications (all you need is WinInet/IXmlHttpRequest and MSXML)

Additionally, on the server side, you can use PHP, WCF, ASP.Net or any other server-side language or
technology to interact with the cloud version of the Live Operating Environment.

LINQ to NHibernate in LINQPad

2008-04-18T10:39:00.002-08:00

LINQPad is like Query Analyzer for LINQ queries. Out of the box it does LINQ to SQL, LINQ to Objects, and LINQ to XML. Wouldn't it be nice if it did LINQ to NHibernate as well? Here's how. The setup process is a bit tedious, but you only need to do it once.

Get a working copy of LINQ to NHibernate

If you haven't done so already, use Subversion to check out LINQ to NHibernate from https://nhcontrib.svn.sourceforge.net/svnroot/nhcontrib/trunk/src/NHibernate.Linq and build it. If you run NHibernate.Linq.Tests.exe, the MbUnit AutoRunner will run through all the tests and display an HTML report of the test results. I get 134 passing tests and 31 failing tests which is to be expected since not all of the LINQ features have been implemented yet, but if you check the commit logs you will see this is actively being worked on. Note that you will need a standard Northwind database (get it here if you don't already have it), you will need to create a new database named Test, and you may need to modify the connection strings in App.config to match your setup.

Add assembly references

Once you've got LINQ to NHibernate working, open LINQPad and press F4 to bring up Advanced Query Properties. Add references to NHibernate.dll, NHibernate.Linq.dll, and the assemblies containing your entities and your data context. In this example, those would be Northwind.Entities.dll and NHibernate.Linq.Tests.exe respectively. Note that when you click Browse to add an assembly reference, you will need to enter *.* in the File Name textbox and press enter to change the file type filter from *.dll to all file types so you can add the Tests.exe reference.

Import namespaces

While you're in the Advanced Query Properties dialog, go to the Additional Namespace Imports tab and enter namespace imports for NHibernate.Cfg and NHibernate.Linq.Tests.Entities. You may want to click the "Set as default for new queries" button in the lower left so you don't need to set up these assembly and namespace references the next time you start LINQPad. Alternatively, when you save a LINQPad .linq query file it will save these references and reload them the next time you open the .linq query file. This can be handy for switching between different databases and data contexts.

Resolve connection string config issues

If you provide NHibernate with connection strings from App.config, LINQPad will not be able to automatically pick these up so you will need to tweak your hibernate.cfg.xml file to contain the actual connection string instead of a named connection string reference. This involves renaming "connection.connection_string_name" to "connection.connection_string" and changing the value to the connection string found in your App.config. If you don't want to mess up your real hibernate.cfg.xml file, make a copy and modify the copy.

Bootstrap the NHibernate data context

In LINQPad, change the query type to "C# Statement(s)" and paste the following code, modifying the path and name of your hibernate.cfg.xml file as necessary:

var cfg = new Configuration().Configure(@"C:\NHibernate.Linq\NHibernate.Linq.Tests\bin\Debug\hibernate.cfg.xml");
var factory = cfg.BuildSessionFactory();
var db = new NorthwindContext(factory.OpenSession());

var q =
    from c in db.Customers
    where c.City == "London"
    orderby c.CustomerID
    select new { c.CustomerID, c.CompanyName };
q.Dump();

Press F5 or Ctrl+E to run this and you should see the following result:

The SQL generated by NHibernate is displayed above the results due to the show_sql=true line in hibernate.cfg.xml. Be sure to save this "query template" as a LINQPad .linq file so you don't have to go through this process again!

WCF Message Size Tester

2007-12-25T15:13:00.001-09:00

Recently on the WCF Forum someone suggested that WCF wasn't able to send messages larger than 2 GB, so I wrote up a little example to disprove this theory. I was able to send a 500 GB message with no problems, and I'm guessing that if I let it run long enough, there will be no problem sending up to 8 million terabytes, the maximum size of a WCF message (Int64.MaxValue).

The main "trick" to sending messages larger than 2 GB is to change the binding's TransferMode from its default of TransferMode.Buffered to TransferMode.Streamed. Besides enabling you to send messages with minimal in-memory overhead, this also allows you to set MaxReceivedMessageSize larger than 2 GB without WCF throwing an exception when it tries to build the channel stack.

It's worth noting that MaxReceivedMessageSize is enforced by WCF regardless of whether the TransferMode is Buffered or Streamed. The same is true for SendTimeout and ReceiveTimeout, so these still need to be adjusted accordingly.

Stream-based messaging in WCF is typically accomplished by using service operations that take a Stream as the only parameter, return a Stream, or both. So what kind of Stream should I ask WCF to send for me? I wanted my example to run as fast as possible, which means I didn't want it to be touching the disk in order to send or receive messages. And I don't have enough memory or address space to buffer a 500 GB MemoryStream. So I created a little hack stream implementation whose Read method leaves the byte buffer untouched and simply returns the appropriate number of bytes "read". This results in sending 500 GB of zeros. :-)

Download the example code here.

Rhino Mocks Quick Reference

2007-12-11T15:51:00.001-09:00

Last week I gave a presentation on mock objects for Software Professionals of Alaska. Once the PowerPointy hand-waving was out of the way, I illustrated the concepts with code examples using Rhino Mocks. I thought I was well-versed in Rhino Mocks before I started, but in the process of preparing the presentation I realized there was a lot that I didn't know or had forgotten. So I distilled all the main facts and features down into a 3-page quick reference of tables and example usage. You might argue that 3 pages makes it a "slow reference," but regardless I still think it's useful to see all the basics stripped down to the bare minimum. Much of the content came from the Rhino Mocks Documentation Wiki, so look there first if you need more details on a particular feature.

So without further ado, I present the Rhino Mocks 3.3 Quick Reference. It includes:

the 3 different record/replay syntax styles
4 types of mocks, 7 if you include MultiMocks
expectations and setup
methods
properties
events
exceptions
delegates
custom behavior
assert messages
ordered and unordered
repetition
the full set of argument constraints

Feedback is quite welcome. Enjoy!

Google Street View in Anchorage

2007-09-26T09:46:00.001-08:00

Yesterday as Garret and I were on our way to the Software Professionals of Alaska meeting where Dan and I presented on MbUnit, we spotted one of these on 36th Avenue! So I grabbed my cell phone and snapped a few shots. Too bad the camera was covered due to the rain, otherwise we might be famous! The Starbucks-sipping driver frowned at us as we enthusiastically pointed, waved, and took pictures.

Generic NHibernate Enum String Mapping

2006-12-19T18:06:00.000-09:00

Today one of my coworkers mentioned all the little NHibernate enum mapping turd classes that were accumulating in our data access layer. See this post for more details on how it works and why you would want to do this. I wondered aloud if generics could be used to eliminate this waste of disk space. A few minutes later, this is what we came up with.

public class GenericEnumMapper<TEnum> : EnumStringType
{
    public GenericEnumMapper() : base(typeof(TEnum))
    {
    }
}

To use this class in your NHibernate mapping file, just use the following lovely .NET 2.0 generics syntax in your "type" attribute:

MyNamespaceB.GenericEnumMapper`1[[MyNamespaceA.MyEnum,
 MyAssemblyA]], MyAssemblyB

No, that ` is not an apostrophe, it's a backtick. It lives on the same key as ~ on my keyboard.

Here it is in a sample hbm.xml mapping file:

<?xml version="1.0" encoding="utf-8" ?>
<hibernate-mapping xmlns="urn:nhibernate-mapping-2.0">
  <class name="MyNamespaceA.UserCredential, MyAssemblyA" 
table="UserCredential" lazy="false">
    <id name="Id" column="ID" type="Guid">
      <generator class="guid.comb" />
    </id>
    <property name="UserId" />
    <property name="CredentialType" column="CredentialTypeID" 
type="MyNamespaceB.GenericEnumMapper`1[[MyNamespaceA.CredentialType,
MyAssemblyA]], MyAssemblyB"/>
    ...
  </class>
</hibernate-mapping>

Copy, paste, and replace CredentialType with your own enum type, and you're good to go!

Debugging with Reflector and Visual Studio

2006-10-26T15:08:00.000-08:00

Mike Taulty recently posted about how he debugs using Visual Studio, Reflector, and SOS. I've been doing a similar style of debugging lately that I think is a bit more efficient, and here's how I do it. You may want to first take a look at what Mike's doing to put this in context.

First I use Jamie Cansdale's awesome TestDriven.NET which has the very cool Go To Reflector feature.

You can also Go To Reflector from any frame in a call stack:

One cool thing about this is that your Reflector instance stays open the whole time, and as you bounce around your call stack using Go To Reflector, the Reflector instance synchronizes to the new location that you specified using Go To Reflector.

You can also set breakpoints on code from within Reflector, which is a bit friendlier than setting breakpoints from the Disassembly window. Yes, you can set breakpoints on assemblies that you don't even have the source for.

Instead of dropping into SOS mode to look at local variables and the value of this, I simply pull up the Locals debug window. The Locals window shows values from the current stack frame as you double-click each frame in the Call Stack pane.

Sometimes the Autos window will show additional relevant variables that the Locals window doesn't and vice versa, so you may want to check both if you don't find what you're looking for.

Unfortunately I haven't found a way to get Visual Studio to do the equivalent of SOS's !dumpheap or !dumpobj for browsing and searching the heap, so you'll still need to drop down to SOS/WinDbg if you want to find all in-memory instances of particular objects or if you're analyzing a dump file. Supposedly you can load WinDbg/AdPlus dumps in Visual Studio, but I haven't gotten that to work very well with dumps of managed code. If anyone knows the secret to getting Visual Studio to analyze dumps of managed code in the same level of detail as it analyzes managed processes, I'd love to find out.

For more commentary on Mike's original goal of being able to debug the .NET Framework using its actual source code, see this post by Andrew Stopford about using Rotor to debug with source, and check out the comments regarding Deblector.

WCF Service Dependency Injection

2006-10-24T20:45:00.000-08:00

Whoops, I accidentally got some link love from Phil Haack, but my blog has been dormant for 8 months! Time to start posting all those half-finished Windows Live Writer drafts! Here goes...

Say you've implemented a WCF service that uses a business logic layer which in turn uses a data access layer. Rather than hard-code these dependencies into your service's constructor, you would like to use dependency injection to enable unit testing of each layer using a mock object framework such as Rhino Mocks.

For our purposes we'll do constructor injection using Spring.NET, although I've also heard good things about the Castle containers, and ObjectBuilder from Microsoft patterns & practices looks promising.

So we have a few simple classes and interfaces that we'd like to automagically hook up:

    [ServiceContract]
    public interface IServiceContract
    {
        [OperationContract]
        ...
    }

    public class ServiceLayer : IServiceContract
    {
        IBusinessLogic _businessLogic;
        public ServiceLayer(IBusinessLogic businessLogic)
        {
            _businessLogic = businessLogic;
        }
        ...
    }

    public interface IBusinessLogic
    {
        ...
    }

    public class BusinessLogic : IBusinessLogic
    {
        IDataAccess _dataAccess;
        public BusinessLogic(IDataAccess dataAccess)
        {
            _dataAccess = dataAccess;
        }
        ...
    }

    public interface IDataAccess
    {
        ...
    }

    public class DataAccess : IDataAccess
    {
        ...
    }

We'd like to accomplish the equivalent of the following code whenever our service is created:

    return new ServiceLayer(new BusinessLogic(new DataAccess()));

What does the Spring.NET equivalent look like?

    IApplicationContext ctx = ContextRegistry.GetContext();
    return (ServiceLayer)ctx.GetObject("ServiceLayer");

How does Spring.NET know how to wire up the dependencies? It uses reflection and config hints to determine who depends on whom:

    <configSections>
        <sectionGroup name="spring">
            <section name="context" type="Spring.Context.Support.ContextHandler, Spring.Core"/>
            <section name="objects" type="Spring.Context.Support.DefaultSectionHandler, Spring.Core" />
        </sectionGroup>
    </configSections>
    <spring>
        <context>
            <resource uri="config://spring/objects"/>
        </context>
        <objects xmlns="http://www.springframework.net">
            <object name="DataAccess" type="NamespaceA.DataAccess, AssemblyA" />
            <object name="BusinessLogic" type="NamespaceB.BusinessLogic, AssemblyB" autowire="constructor" />
            <object name="ServiceLayer" type="NamespaceC.ServiceLayer, AssemblyC" autowire="constructor" />
        </objects>
    </spring>

That's great, but how do we get WCF to do this for us at the right place at the right time? By using WCF's "behavior injection" functionality of course!

We'll need to implement an IInstanceProvider that allows us to serve up instances of our service each time WCF needs a new instance.

    public class DependencyInjectionInstanceProvider : IInstanceProvider
    {
        private Type _serviceType;

        public DependencyInjectionInstanceProvider(Type serviceType)
        {
            _serviceType = serviceType;
        }

        public object GetInstance(InstanceContext instanceContext)
        {
            return GetInstance(instanceContext, null);
        }

        public object GetInstance(InstanceContext instanceContext, Message message)
        {
            object result = null;

            IApplicationContext context = ContextRegistry.GetContext();
            string[] objectNames = context.GetObjectNamesForType(_serviceType);
            if (objectNames.Length != 1)
            {
                throw new YourOwnException(
                    string.Format(
                    CultureInfo.InvariantCulture,
                    "There must exist exactly one <object> definition for the {0} service in the Spring configuration",
                    _serviceType.Name)
                    );
            }
            return context.GetObject(objectNames[0]);
        }

        public void ReleaseInstance(System.ServiceModel.InstanceContext instanceContext, object instance) { }
    }

Then we'll need an IServiceBehavior that plugs in our IInstanceProvider at the right place at the right time. Remember, WCF could choose to activate our service per call, per private session, per shared session, or as a singleton.

    public class DependencyInjectionServiceBehavior : IServiceBehavior
    {
        public void ApplyDispatchBehavior(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase)
        {
            foreach (ChannelDispatcherBase cdb in serviceHostBase.ChannelDispatchers)
            {
                ChannelDispatcher cd = cdb as ChannelDispatcher;
                if (cd != null)
                {
                    foreach (EndpointDispatcher ed in cd.Endpoints)
                    {
                        ed.DispatchRuntime.InstanceProvider = 
                            new DependencyInjectionInstanceProvider(serviceDescription.ServiceType);
                    }
                }
            }
        }

        public void AddBindingParameters(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase,  
                Collection<ServiceEndpoint> endpoints, BindingParameterCollection bindingParameters) {}
        public void Validate(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase) {}
    }

Now we want to actually use this behavior. You could implement classes to apply this behavior using custom attributes, via config, or programmatically via a custom ServiceHost. I chose to go the ServiceHost route.

    public class MyServiceHost : ServiceHost
    {
        public MyServiceHost() : base() { }
        public MyServiceHost(Type serviceType, params Uri[] baseAddresses) : base(serviceType, baseAddresses) { }

        protected override void OnOpening()
        {
            this.Description.Behaviors.Add(new DependencyInjectionServiceBehavior());
            base.OnOpening();
        }
    }

And of course a custom ServiceHost needs a custom ServiceHostFactory if you want IIS to host it (you can use your custom ServiceHost directly when self-hosting):

    public class MyServiceHostFactory : ServiceHostFactory
    {
        protected override ServiceHost CreateServiceHost(Type serviceType, Uri[] baseAddresses)
        {
            return new MyServiceHost(serviceType, baseAddresses);
        }
    }

This factory can then be referenced from your .svc files:

<%@ ServiceHost
Service="NamespaceC.ServiceLayer, AssemblyC"
Factory="NamespaceD.MyServiceHostFactory, AssemblyD"
%>

Steve Maine has a great series of posts with more details on how service activation and hosting work:

FogBugz browser search integration

2006-02-17T17:07:00.000-09:00

FogBugz 4.0 has a handy new search URL that lets you enter a bug number to go directly to a bug, or search text to find all open cases matching your query. Wouldn't it be nice to hook this into your browser's search box? Fortunately, IE7 uses A9's OpenSearch for its search providers, and Firefox uses Mycroft.

Creating the OpenSearch provider

Create an XML file with the following text:

<?xml version="1.0" encoding="UTF-8" ?>
<OpenSearchDescription xmlns="http://a9.com/-/spec/opensearch/1.1/">
<ShortName>FogBugz</ShortName>

<Description>Search FogBugz</Description>
<Url type="text/html" template="http://[your-ip-goes-here]/fogbugz/default.asp?pre=preMultiSearch&pg=pgList&pgBack=pgSearch&search=2&searchFor={searchTerms}" />
</OpenSearchDescription>

Replace [your-ip-goes-here] with the appropriate address.

Put the XML file on your web server and put the following link somewhere that's publicly accessible:

<a href="#" onClick="window.external.AddSearchProvider("http://[url-to-your-directory]/FogBugzOpenSearch.xml");">Add FogBugz search provider to IE7</a>

Replace [url-to-your-directory] with the appropriate URL.

Now, when you click on the link from IE7, you'll be prompted to add a new search provider. This will make a FogBugz option available in the dropdown list of your search box:

Creating the Firefox provider

Create a file named FogBugzSearch.src in a public directory on your web server with the following text:

<search
version="7.1"
name="FogBugz"
description="Search FogBugz"
action="http://[your-ip-goes-here]/fogbugz/default.asp" searchForm="http://[your-ip-goes-here]/fogbugz/" method="GET"
>
<input name="pre=preMultiSearch&pg=pgList&pgBack=pgSearch&search=2&searchFor" user=""> </search>
<browser

update="http://[url-to-your-directory]/FogBugzSearch.src"    
updateIcon="http://[url-to-your-directory]/FogBugzSearch.png" 
        updateCheckDays="360"

Fill in the blanks with the appropriate addresses and URLs.

Put the following image in the same directory:

Now to create the link. Put the following in your head element:

<script type="text/javascript">
function addEngine(name,ext,cat,type)
{
if ((typeof window.sidebar == "object") && (typeof window.sidebar.addSearchEngine == "function")) {


     window.sidebar.addSearchEngine(
       "http://[url-to-your-directory]"+name+".src",
       "http://[url-to-your-directory]"+name+"."+ext, name, cat );

}
}
</script>

and the following link in your page:

<a href="javascript:addEngine('FogBugzSearch','png','General','0')">Add FogBugz search provider to Firefox</a>

Now, when you click on the link from Firefox, you'll be prompted to add a new search provider. This will make a FogBugz option available in the dropdown list of your search box:

Private offices: what do they really compensate for?

2005-10-29T17:43:00.000-08:00

Recently someone on the team forwarded a link to Joel's mini-essay Multitasking in the Workplace, a small rant against the problem of coworker interruptions with private offices as the solution. In case you haven't read many Joel articles yet, he really likes private offices.

I think Joel's focus on private offices is a result of his belief that superstar programmers create the best software, described in Hitting the High Notes (and other places). Superstars are most productive if they don't have to work directly with each other. In the High Notes article, Joel says, "A single good programmer working on a single task has no coordination or communication overhead. Five programmers working on the same task must coordinate and communicate." As if communication is an evil that should be avoided if at all possible, especially where it involves his superstar programmers (perhaps he's on to something).

Notice the diagram at the top of the article:

Best Working Conditions

→

Best Programmers

→

Best Software

→

Profit!

Where is Best Team in this picture?

A number of people have commented on this, filling in the team side of the equation:

Scott Berkun discusses the relationship between teams and stars

Eric Sink replies with a choir analogy

The Five Dysfunctions of a Team says the fifth dysfunction, inattention to results, happens when team members put their individual needs (ego, career development, recognition) above the success of the team. This is a big risk when you've got a bunch of superstars working in isolated silos.

I think the most telling statement in Multitasking in the Workplace is "She does mention that open-spacers do frequently decide to 'interrupt themselves' to participate in another conversation that they overheard, something which is probably net beneficial for the team's productivity but which drives me crazy." This implies that he's not going to encourage behavior that boosts the net productivity of the group! How can individual productivity possibly matter more than the net productivity of the group? I don't get it.

This isn't the first time Joel has heard stuff like this. Several years ago he asked Tom DeMarco what he thought of pair programming contrasted with private offices. Tom replied that "groups of people who work in the same style on the same kinds of tasks will not distract each other." It's worth noting that Tom is the author of the excellent Peopleware, the book Joel often refers to when speaking of private offices.

I do agree that removing distractions is important for productivity. I'm just not sure I agree with Joel that those distractions are always necessarily my coworkers.

From observing our team's development cycles, I've found it most helpful to be "distracted" by on-topic conversations when we're in our analysis and design phases, and least helpful when we're in the coding phase. Joel's right… when your head is full of all the variables necessary for writing code, it's very expensive to re-load after an interruption. I guess my "it depends" answer means I also disagree with Joel that a developer is always working on code. Of course Joel is 45% more likely to say programmer than developer...

I think it's up to each person to determine when they need to be in "flow" mode and put on headphones, hang a do-not-disturb sign, or disappear for a while when they need uninterrupted focus. Telecommuting can be awesome for getting lots done when you know exactly what needs to be done and can do it all on your own.

I must admit that I have been strongly on both sides of the fence on this issue at various points in the past, so I reserve the right to change my mind depending on the circumstances. In the here and now, the benefits of sacrificing my personal productivity for the productivity of the team are quite clear.

That's enough for now… I really do enjoy Joel's articles, even the ones I disagree with.

Fixing my VS2005 RTM install

2005-10-29T12:10:00.000-08:00

After downloading Visual Studio 2005 Professional Edition RTM from MSDN I debated whether or not to install it since I would have to go without all the cool WinFX goodies like Avalon, Linq, and Windows Workflow Foundation. In the end I broke down and decided to move to the RTM.

In addition to the 24-step pre-RTM uninstall instructions you will need to uninstall a bunch of other things like the Linq Visual Studio extensions (pull up the local documentation HTML file and click the link that pulls up the folder with the Linq VS uninstall bat file), Linq, Visual Studio 2005 Extensions for Windows Workflow Foundation, Windows Workflow Foundation, WinFS, WinFX, and I think I'm forgetting some more of them. Make sure you pass through your entire Add/Remove Programs list, because some of them aren't grouped under Microsoft.

In my case, I overlooked Visual Studio 2005 Extensions for Windows Workflow Foundation until after VS 2005 Beta 2 was uninstalled. Oops! But it appeared to uninstall just fine once I discovered that.

So I went through the Visual Studio 2005 Professional Edition RTM install and everything went smoothly until right towards the end when a dialog popped up telling me that Visual Studio 2005 encountered an error, and would I like to send it to Microsoft? This error occured while the installer was on the "Executing action:HxMerge_VSCC_v80" step. I continued with the installation, and it appeared to complete successfully.

But when I tried to start Visual Studio 2005, the splash screen appeared and then the devenv process went away! I noticed the splash screen showed Microsoft Workflow Tools on the list of loaded modules... I Googled the HxMerge installer step above and found someone else with the same problem with an earlier version. So I tried starting devenv with the /safemode switch, and sure enough it started! I went to Tools -> Options and saw a configuration section for the Windows Workflow Designer, which threw an error when I tried to expand it.

So, I installed Windows Workflow Foundation and Visual Studio 2005 Extensions for Windows Workflow Foundation on top of my VS2005 RTM installation, uninstalled both of the WWF items in Add/Remove Programs, and now VS2005 is happy again!

And no, VS2005 RTM doesn't work with the WWF designer. It reports an error and gives you an option to not load the module in the future.

Hopefully RTM-friendly versions of Avalon, WWF, and all the other WinFX goodies show up soon! If an RTM-friendly Linq update is possible without destabilizing the C# compiler, that would also be cool.

Hello world!

2005-10-28T00:52:00.000-08:00

That is all.