Archive for March, 2010

SAX-like Xml parsing

For those of you who don’t know or can’t remember, SAX (Simple API for XML) is a technology for reading Xml in an event based way. Instead of loading your Xml into a great big DOM and looping thru it or plucking out nodes via XPath or LINQ to XML, SAX allows the Xml parser to notify your application as new nodes were encountered. SAX (like XmlReader, is a forward-only Xml reader) and efficient for parsing large and streaming chunks of Xml.

Reactive extensions fits well with the SAX way of thinking because it is designed to “push” information to it’s consumer instead of making you “pull” information from it. At the same time, Reactive LINQ allows you to select on observable objects in a very natural feeling pull-like language. Therefore, it occurred to me: why not combine XmlReader and Reactive extensions to build a SAX-like Xml reader that you could use via Reactive LINQ!?

An observable XmlReader would allow you to subscribe to it and would iterate over your Xml document for you, notifying you when each node was read. The programmer using this could easily write reactive LINQ expressions to select on specific nodes in the Xml resulting in code that would look and feel much like LINQ to XML but with all the performance benefits of XmlReader.

For example, imagine you wanted to find all the values in nodes of a certain name. You could write something like this…

    1 XmlReader reader = XmlReader.Create(“TreeOfLife.xml”);

    2 IObservable<XmlReader> RxReader = reader.ToObservable();

    3 IObservable<string> NameFinder =

    4     from nodeReader in RxReader

    5     where nodeReader.NodeType == XmlNodeType.Element && nodeReader.Name == “NAME”

    6     select nodeReader.ReadElementContentAsString();

    7 NameFinder.Subscribe((item) => names.Add(item));

I’ll dissect:

  1. Create an XmlReader
  2. Use my newly created extension method to turn that XmlReader into an IObservable.
  3. Construct a new IObservable
  4. Select all the XmlReaders in the IObservable (one for each node)
  5. Filter for node type and element name
  6. Select their string values
  7. Actually initiate the XmlReader to iterate and notify you when a new name is available.

Now let’s look at the code in my extension method. It’s surprisingly simple!

public static IObservable<XmlReader> ToObservable(this XmlReader reader)



    Observable.CreateWithDisposable<XmlReader>(observer =>




            while (reader.Read())




        catch (Exception e)





        return reader;



All I am doing is looping on the XmlReader and passing the XmlReader itself (at it’s current state) to the observer. Violla!

Canceling the operation

Suppose you want to cancel the operation mid subscribe: Because I create the observer via Observable.CreateWithDisposable and return the XmlReader itself as my Disposable object. This allows me to cancel at any time by simply calling:

IDisposable processor = NameFinder.Subscribe((item) => names.Add(item));


More complex selections

Suppose you want to get only the child nodes within a parent node:

IObservable<string> NameFinder =

    from r1 in RxReader

    where r1.NodeType == XmlNodeType.Element && r1.GetAttribute(“ID”) == “76937”

    from r2 in r1.ReadSubtree().ToObservable()

    where r2.NodeType == XmlNodeType.Element && r2.Name == “NAME”

    select r2.ReadElementContentAsString();

I simply combine reactive LINQ statements and create a new reactive XmlReader to iterate a sub node using XmlReader.ReadSubtree().ToObservable().

Here’s a demo of the code above using the tree of life Xml for a Danaus butterfly genus.

Here’s the source code for the project.


Read Full Post »

In my last post I demonstrated how to use reactive extensions in Silverlight to easily build responsive UIs. In this post I am going to expand on that and demonstrate how to use reactive extensions (Rx) from within a view model by asynchronously loading an RSS feed into your model and populating the UI while the bytes are still coming in. This is about as fast and responsive as it gets and as you will see, by using Rx the code is concise and relatively easy on the eyes.

Note: All of this can be done without reactive extensions. Nothing here is new from the end users point of view. But as programmers, we should always be looking for ways to create more readable code in fewer lines to accomplish tasks even if we already know how to do them in other ways. Rx has the advantage of enabling you to easily write multi-threaded code without creating Thread objects and avoid using local variables or constructing user state objects to maintain state between events. Enough talk, time to make my case with code.

Note: If you want to get directly to the Rx stuff, skip down to the DataClient section.

First, let’s set up the foundation of a demonstration app that reads an RSS feed and displays the results to the user. Regardless of whether you’re going to use Rx or not, you’ll need to create the following:

1. The model. We clearly need a model to hold each RSS item. I’m going to use XmlSerializer to deserialize the objects from Xml, so I’ll add a bunch of XmlSerializer attributes to my class to help map the RSS Xml.


public class RssItem



    public string Title { get; set; }


    public string Link { get; set; }


    public string PublishDate { get; set; }


    public string Description { get; set; }


    public string PlainTextDescription


        get { return StripHtml(Description); }



Note: Because RSS item descriptions often contain HTML, I created a read-only property that strips out the html tags from the description. Thanks to John Papa for providing the StripHtml algorithm in his book.

2. A routine to turn a stream into RSS item objects. There are different ways to accomplish this (such as using LINQ to XML) but we need a method that allows us to read the bytes as they become available. Therefore, I’ll use a combination of XmlSerializer and XmlReader in order to support a stream based approach and avoid waiting for the entire Xml document to be delivered before we can start processing it.

public static IEnumerable<RssItem> GetRssItems(XmlReader reader)


    XmlSerializer rssItemSerializer = new XmlSerializer(typeof(RssItem));

    while (reader.GoToElement(“item”))

        yield return rssItemSerializer.Deserialize(reader) as RssItem;


Note: I’ve created an extension method (.GoToElement) for the XmlReader class in order to make using XmlReader easier. Download the source to see this method.

3. A routine to create and return a WebRequest object.

private static WebRequest GetWebRequest(Uri Uri) {

    var Result = (HttpWebRequest)WebRequest.Create(Uri);

    Result.AllowReadStreamBuffering = false;

    return Result;


Note: I also set AllowReadStreamBuffering to false in order to prevent the response from being buffered, thus allowing us to start using the data as it is being downloaded.

Also Note: I’m using a WebRequest object instead of a WebClient object for 3 reasons: 1) the response callback occurs on a background thread 2) Silverlight only allows us to process unbuffered data on a background thread 3) Rx has an extremely simple way to turn async patterned calls into observable objects (more on this below).

4. The view. We will create our view the exact same way regardless of whether we use Rx or not. This is important because it allows a designer to build the view without caring about how we go about getting our data. This is a typical and simple M-V-VM view that merely binds a property from the view model to a control on the view. The important thing to take away here is that using Rx in this example does not affect our view in any way, shape or form.

    <local:ViewModel />
<ScrollViewer VerticalScrollBarVisibility="Auto"> 
    <ItemsControl ItemsSource="{Binding RssItems}"> 
                    <HyperlinkButton Content="{Binding Title}" FontSize="16" NavigateUri="{Binding Link}" Margin="3" /> 
                    <StackPanel Orientation="Horizontal" Margin="3"> 
                        <TextBlock Text="Posted: " Foreground="Gray" /> 
                        <TextBlock Text="{Binding PublishDate}" Foreground="Gray" /> 
                    <TextBlock Text="{Binding PlainTextDescription}" TextWrapping="Wrap" Margin="3" /> 

The view simply shows our RSS items in a templated ItemsControl and wraps the whole thing in a ScrollViewer.

5. The ViewModel’s public API. The ViewModel needs to provide RssItem objects, so all we need is a property that can be bound to the view:

private ObservableCollection<RssItem> rssItems;

private ObservableCollection<RssItem> rssItems;

public IEnumerable<RssItem> RssItems


    get { return rssItems; }


So far everything above is probably identical to how you would have built the app regardless of whether or not you were going to use Rx! This is all standard stuff but I wanted to list everything involved in order to stress just how much of your code and coding practices do NOT need to change in order to take advantage of Rx. Now we have most of the major pieces of our demonstration app all nice and isolated and ready to be used with or without Rx.

The only things left are the DataClient (the thing that actually goes and gets the RSS Xml and populates your model) and the code in the ViewModel to call that DataClient.

The DataClient: Rx Magic time!

public static IObservable<RssItem> GetRssItems(Uri Uri)



        (from request in Observable.Return(GetWebRequest(Uri))

        from response in Observable.FromAsyncPattern<WebResponse>(request.BeginGetResponse, request.EndGetResponse)()

        from item in Mapper.GetRssItems(XmlReader.Create(response.GetResponseStream())).ToObservable()

        select item).ObserveOnDispatcher();


That’s it! We just need one great big expression to return something that our ViewModel can use. No events, no lambda, no long routines.

The purpose of this function is super simple: to return an IObservable<RssItem> whose job is to “push” fully populated RSS item objects out to the consumer one at a time as soon as they are available. Later I’ll show you how to consume an IObservable object.

The basic mechanics of this function is to combine IObservable objects using a Reactive LINQ expression and return one super IObservable object that performs its duties on .Subscribe and pushes out results one at a time back to the consumer.

Let’s dissect:

First we have:


This calls GetWebRequest (described earlier) to get a standard WebRequest and puts it into an IObservable object that does one simple thing: it “pushs” that single WebRequest object to it’s consumer.

By using Reactive LINQ notation:

from request in Observable.Return(GetWebRequest(Uri))

we can chain together other operations that use that WebRequest to create their own IObservable objects.

This might be tough to get your head around but will become clearer as we keep going.

Next, we take that request and use it in our next expression:

Observable.FromAsyncPattern<WebResponse>(request.BeginGetResponse, request.EndGetResponse)()

This feature of Rx is awesome! Here we call the FromAsyncPattern method and pass in references to 2 methods that already exist on the WebRequest object. FromAsyncPattern wraps this in an IObservable object that does a few things… It calls BeginGetResponse when someone subscribes to it, it handles the async callback, calls EndGetResponse, and finally returns the result via a “push” to the subscriber.

In a nutshell, FromAsyncPattern hides all the plumbing of using the async pattern and wraps it in a nice IObservable object. One thing to note: The call out to the web does NOT get initiated until someone subscribes to the resulting IObservable. This gives the consumer the ability to control the timing.

Quick recap on our giant expression: At this point, we now have  a new IObservable object that works like this…

When someone subscribes to the resulting IObservable, it goes and subscribes to the first IObservable in our Reactive LINQ statement, which immediately “pushes” notification out that tells us that we have a web request. That push notification is handled by the statement in the Reactive LINQ statement which uses that request object to initiate and handle the asynchronous call to the server. When it is done, it “pushes” the response on to the subscriber (the subscriber being the next thing in our big ‘ol Reactive LINQ chain). Which is…


The first thing you may notice is that there’s no Rx in this expression. All it is doing is creating an XmlReader from the response stream and passing it into the function that creates models from an XmlReader. At this point we just have an IEnumerable<RssItem> on which we call…


ToObservable is an extension method that extends IEnumerable and turns it into an IObservable. This allows a consumer to get notified when the next item in the collection is available instead of making us loop through the collection manually. We also need an IObservable because we’re chaining together IObservable expressions. You can’t just switch to IEnumerables in the middle of your LINQ expression. For example. A traditional LINQ expression with IEnumerable might look like:

from purchases in customers

from order in purchases

select order;

but you can’t start with an IObservable and switch to an IEnumerable or vice versa…

Finally, we want to select the item that ultimately will be pushed back to the consumer and wrap our IObservable in a new one that performs all “push” operations on the dispatcher. This is important because WebRequest returns all its data on a background thread and so without it, push notifications will occur on that same thread and throw an exception when we try to modify the UI.

select item).ObserveOnDispatcher();

Note: We could also have built the guts of this procedure using more traditional methods like handling events by creating our own class that implemented the IObservable interface. However, I think it’s better to re-use methods like FromAsyncPattern to do as much of the leg-work for us as possible.

Back to the ViewModel:

public ViewModel()


    rssItems = new ObservableCollection<RssItem>();

    IObservable<RssItem> client = DataClient.GetRssItems(new Uri(http://pheedo.msnbc.msn.com/id/3032091/device/rss/io&#8221;));

    client.Subscribe(item => rssItems.Add(item));


Done! All we had to do is get the IObservable object from the DataClient, subscribe to it and for each item returned, and add that item to the ObservableCollection bound to the view.

And remember, because this is running on a background thread and not buffering the response from the server, we can easily be spitting back model objects and showing them in the UI as more bytes are still being downloaded and processed.

More Rx magic!

OK, so we have an ultra responsive UI that shows data as soon as physically possible and we have a super slim DataClient that leverages Rx to avoid writing our own code to call and handle asynchronous webrequests. But what else can we do with Rx in the context of this app?

A: “a ton! “ but this post is already getting long so I’ll get you going with a couple quick examples and let your imagination take over.

Imagine you wanted to combine two different RSS feeds:

public ViewModel()


    rssItems = new ObservableCollection<RssItem>();

    var TopNewClient = DataClient.GetRssItems(new Uri(http://pheedo.msnbc.msn.com/id/3032091/device/rss/io&#8221;));

    var BusinessNewsClient = DataClient.GetRssItems(new Uri(http://pheedo.msnbc.msn.com/id/3032221/device/rss/&#8221;));

    var client = Observable.Merge(TopNewClient, BusinessNewsClient);

    client.Subscribe(item => rssItems.Add(item));


Or how about you wanted to expose an IsBusy state to the client:

< toolkit:BusyIndicator IsBusy=”{Binding IsBusy}” />

 public ViewModel()


    rssItems = new ObservableCollection<RssItem>();

    IObservable<RssItem> client = DataClient.GetRssItems(new Uri(http://pheedo.msnbc.msn.com/id/3032091/device/rss/io&#8221;));

    client = client.Finally(() => IsBusy = false);

    isBusy = true;

    client.Subscribe(item => rssItems.Add(item));



bool isBusy;

public bool IsBusy


    get { return isBusy; }

    private set


        isBusy = value;

        if (PropertyChanged != null)

            PropertyChanged(this, new PropertyChangedEventArgs(“IsBusy”));



Check out the vast library of Rx extension methods available to use. Getting an IObservable and pumping the results into an ObservableCollection is just the beginning of what you can do with a data client that returns an IObservable. Have fun and start using Rx!

See demo (Note: I threw in a very minor delay during deserialization of each item to exagerate the effect).

Download source

Read Full Post »

If you haven’t heard of Reactive extensions (Rx) for .NET, it’s time to get familar. Rx is extremely cool and once you understand it, it will change the way you write client-side code forever. Instead of providing an overview of Rx, I’m going to urge readers to check out Pencho Popadiyn’s blog post, where he does a much better job of explaining it that I probably could.

Instead, I’m going to demonstrate one practical use for Rx extensions here in my 2 part blog. Yes, 2 parts! That’s how cool I think Rx is. 🙂 And keep in mind that I’m only scratching the surface on the ways Rx can be used. Enough fanfair, let’s get started by building a super simple Silverlight app that updates a ListBox with and without Rx and you be the judge of which is better.

Let’s work backwards by looking at the end result first and then going over how to create it.

Click here to run

Download source code here

Now, here’s how it works…

Step 1: Download and install Rx for Silverlight (link about one page down on the right hand side).

Step 2: In your Silverlight 3 project, Add a reference to the 3 Rx assemblies: System.CoreEx, System.Observable, & System.Reactive.

Note: The 3 files only add ~70K to your .xap file.

Step 3: Create a function that returns a model. For this example, we’ll just have an IEnumerable<String> as our model and I’ll toss in a delay between each item so we can exagerate the difference between Rx and non-Rx.

public static IEnumerable<string> GetModel()
    for (int i = 0; i < 5; ++i)
        yield return "Item " + i;

Step 4: The standard way to put that model into our ListBox would be to assign it (the IEnumerable) to ListBox.ItemsSource. However, for the sake of an apples to apples comparison, I’ll manually loop thru my model and insert each item into an ObservableCollection that is bound to the ListBox.

//MyList.ItemsSource = GetModel();
var items = new ObservableCollection<string>();
MyList.ItemsSource = items;
foreach (string item in GetModel())

Step 5: Rx magic time! First, I’ll write similar code as in Step 4 but using Rx…

var items = new ObservableCollection<string>();
MyList.ItemsSource = items;
IObservable<string> observable = GetModel().ToObservable();
observable.Subscribe(item => items.Add(item));

Instead of iterating through the IEnumerable, we create an Observable object from our IEnumerable using the .ToObservable extension method. Then, we subscribe to that observer which tells it to iterate over the collection and run our lamba expression for each item. Bottom line: same result.

Step 6: More Rx magic. Let’s use the power of multi-threading and get our model to be created on a background thread, while we add our items to our observable collection (in order to avoid an invalid cross thread exception). We just have to change the last two lines a little bit..

IObservable<string> observable = GetModel()
observable.ObserveOnDispatcher().Subscribe(item => items.Add(item));

Passing in NewThread to .ToObservable() causes the GetModel function to run on a new thread and .ObserveOnDispatcher() causes our Subscription callbacks to run through the dispatcher (therefore allowing us to update the UI). The end result is a UI left totally responsive while the model is manufactured on it’s own thread and each new model item appears as soon as possible in the UI.

This is certainly possible to do by running GetModel() on a new thread and calling Dispatcher.BeginInvoke for each item. Afterall, this is what Rx is doing internally for us, but I personally think the Rx code is much more readable and writeable.

Next to come: Combining M-V-VM, Rx, and unbuffered REST webservices for clean, easy, and hyper responsive applications that need to communicate with the server!

Read Full Post »

Silverlight provides a variety of brushes that can be used to paint just about any area in your app: from text to shapes to borders. The most common brushes are for painting color and images: SolidColorBrush, LinearGradientBrush, RadialGradiantBrush, and ImageBrush.

However, today I paired the less common VideoBrush with an unusual visual element. Click below to see a demonstration of a fun and novel use of the VideoBrush in Silverlight:

Video Jigsaw Puzzle


Note: PuzzleTouch also now allows you to create your own video puzzles using .wmv or .mp4 files. All work is done 100% on the client! To create your own video puzzle, go to the Online Jigsaw Puzzles by PuzzleTouch homepage, choose “Custom Puzzle”, and have fun!

Read Full Post »