Can you implement publish-subscribe load balancing with MSMQ?

Surprisingly, prior to Windows Server 2008 the answer was no! (Well, technically the capability is in Vista too, but who runs Vista in their datacenter?)

It’s a common scenario. I have work that I want to share among several load-balanced, redundant servers. Only one server can be working on an item at a time. If a server crashes while working on an item, then another server should pick it up instead.

What’s the solution? Publish the work items to a queue and have each of the servers subscribe. The servers receive the work items transactionally, so that if the work fails or the server crashes it is aborted back to the queue. But while one server has the work item locked, other servers will not receive it. They will get work items further down in the queue.

The problem is that MSMQ 3.0 and prior cannot receive messages transactionally from a remote queue. This feature only works with local queues. Instead of a publish-subscribe model, you have to use a push model. Push the message to the machine that you want to process it.

John Breakwell posted your options. You can upgrade to Windows Server 2008, implement a dispatcher (turning your pub-sub into a push), or peek-then-receive. The last is not really an option, because it violates the requirement that only one server is working on an item at a time.

Here’s my solution
I propose a fourth option. Don’t use MSMQ. Instead, write a journal table in SQL Server. Getting the locking right is a bit tricky, but it’s doable. (Hint: you will make use of UPDLOCK.)

Honestly, this is a primary use case for a queuing technology. No, this is the primary use case. How could you even release version 1.0 without support for pub-sub? I’ve gone my entire career assuming that this feature was there, only to find out in integration testing that it is lacking.

I learned a couple of CSS positioning rules that are not immediately apparent from reading W3Schools. Web designers are familiar with these rules, but this information needs to trickle down to us brute-force developers.

The position CSS style
Here are some rules I learned from Aucustic Cuckoo.

The position: relative and position: absolute styles don’t work the way you might expect as a developer. We expect that absolute means to set the position from the top-left corner of the window, and that relative means to set the position from the container. Both of these assumptions are wrong.

If you don’t set a position CSS style on your element, the browser lays it out based on the surrounding elements and reserves space for it. This is called static positioning.

The position: relative setting positions the element based on its static position. The element still takes up space as if it were in its static position. On its own, it’s almost useless.

The position: absolute setting positions the element based on its containing block. The containing block is a parent element that is not statically positioned. Most of the time, that means you are placing elements in window coordinates.

To get the most out of these settings, you have to use them together. Set the position of a parent div to relative, but don’t touch any of its other positioning properties. This will cause the parent to act as if it were statically positioned, but qualify as a containing block. Then you can set the position of child elements to absolute to position them relative to the parent. Yeah, it blew my mind, too.

The right and bottom CSS styles
Here’s one I picked up from watching a video on css-tricks.com. Chris doesn’t explicitly call this out, so I had to pause the video to read his CSS.

I usually move elements by setting their X and Y coordinates. To me, X maps to left and Y maps to top. It never occurred to me that an element has right and bottom properties, too.

If you set the right property and not the left, then the right side of the child element will be positioned relative to the right side of the parent. So if you want to right-align an image inside a div, set the div position to relative, the image position to absolute, and the image right to 0. Same thing for the bottom.

The float and clear styles
And finally, here’s one from BarelyFitz.

The float style is usually used to wrap text around an image. But web designers have figured out how to use this property for columns. If you float your main content to the left or right, then your side bar will sit nicely next to it. But be sure that the floating element has greater height, or the other elements will wrap around it at the bottom. That’s why the main content is usually floating, and not the sidebar.

To put more content after all of the floating elements, set the clear property to both. This could be used to put a footer on a page, but is not as easy as nesting divs. Put the side bar and the content in one div, and the footer in another.

This is a case of RTFM. The real solution to the problem of creating an object via WMI is the ManagementCreate attribute. This attribute decorates a static method that takes key parameters (themselves decorated with ManagementName) and returns a new management entity.

The mental block that I was running into was the proxy. I was equating a .NET object with a WMI object. I assumed that creation of a .NET object via "new" was equivalent to the creation of a WMI entity via "path … create". Not the case.

The .NET objects are proxies for the WMI entities. The are created and destroyed with each request. A call to the create method should cause the enumerate method to include the new entity, as well as return the new proxy object. However, the .NET object returned from the create method will be different from the one returned from the create method. These are merely proxies to the same entity.

Problem solved. Moving on.

I am attempting to create a Windows Service in C# that exposes management interfaces through WMI. I’m having trouble creating new objects from the WMI command line. See the problem description.

In working through the referenced article, I learned a bit more about managed MWI providers. At the half-way point, you have a working coupled WMI provider written in C#. You should be able to enumerate service objects with the following command:

C:\>wmic /NAMESPACE:\\root\test PATH win32_servicehost get /value

I tried that, and sure enough I get a list of service hosts and the services that they are running. These objects are created in code by the EnumerateServiceHosts method.

My problem requires that I create a new object. I have to allow operations to specify its properties and issue it commands. So I tried do create an instance of a Win32_ServiceHost with an ID not yet in the list:

wmic:root\cli>PATH win32_servicehost create id=821
Create instance of 'win32_servicehost' class (Y/N)?y
ERROR:
Code = 0x80041024
Description = Provider is not capable of the attempted operation
Facility = WMI

Not surprising that this would fail, since the code returns null if the process ID is not found. But if I try to create an object using a process ID already in the list, this is what happens:

wmic:root\cli>PATH win32_servicehost create ID=820
Create instance of ‘win32_servicehost’ class (Y/N)?y
ERROR:
Code = 0×80041019
Description = Object or property already exists
Facility = WMI

So I changed the example code to allow me to create process hosts with IDs that are not in use. Now every ID I use gives me "Object or property already exists". But if I query for an ID that does not exist, it tells me "No Instance(s) Available".

wmic:root\cli>PATH win32_servicehost create id=820
Create instance of ‘win32_servicehost’ class (Y/N)?y
ERROR:
Code = 0×80041019
Description = Object or property already exists
Facility = WMI

wmic:root\cli>PATH win32_servicehost create id=821
Create instance of ‘win32_servicehost’ class (Y/N)?y
ERROR:
Code = 0×80041019
Description = Object or property already exists
Facility = WMI

wmic:root\cli>PATH win32_servicehost where ID=820
ID   Services
820  {"SCardSvr"}

wmic:root\cli>PATH win32_servicehost where ID=821
No Instance(s) Available.

Here’s my solution

I think I’m approaching the problem from the wrong angle. Create doesn’t mean what I think it means. WMI is set up to access objects that already exist.

Instead of creating instances of objects from WMI, I’m going to change my service so that I can return objects created in code. The operations team can issue commands to these existing objects to cause new objects to be created — in code. Based on what works in the example, this should solve the problem. I’ll post when it’s working.

Current Status
Solved

Problem
Using the .NET WMI provider extensions 2.0, I cannot create an instance of my custom provider class.

Windows Management Interface (WMI) is a way to expose management and configuration components from your application to scripting and operations tools, like SMS and MOM. In the past, creating a WMI provider required the use of COM. The WMI provider extensions for .NET created the ability to expose read-only management objects. In .NET 3.5, this capability was enhanced to allow for writing properties and calling methods.

Using these extensions, I created a management class for my application. I am having trouble creating instances of this class from the WMI command line, or any of the other WMI tools that I’ve tried.

Here’s the class:

using System.Management.Instrumentation;

namespace ConsoleDecoupled
{
    [ManagementEntity]
    public class Activity
    {
        private string _name;

        [ManagementBind]
        public Activity(string Name)
        {
            _name = Name;
        }

        [ManagementKey]
        public string Name
        {
            get { return _name; }
        }
    }
}

Here’s the program that publishes the class to WMI:

using System;
using System.Management.Instrumentation;

namespace ConsoleDecoupled
{
    class Program
    {
        static void Main(string[] args)
        {
            //InstrumentationManager.RegisterType(typeof(Activity));

            Activity instance = new Activity("foo");
            InstrumentationManager.Publish(instance);

            Console.WriteLine("Press enter to exit");
            Console.ReadLine();

            InstrumentationManager.Revoke(instance);
            //InstrumentationManager.UnregisterType(typeof(Activity));
        }
    }
}

And here’s what wmic does when I try to create an instance:

wmic:root\cli>path Activity create Name="fee"
Create instance of 'Activity' class (Y/N)?y
ERROR:
Code = 0x80041002
Description = Not found
Facility = WMI

I know that the class is found, because I can query for the instance that I created programmatically:

wmic:root\cli>path Activity where Name="foo"
Name
foo

My next plan of attack is to go through this article on Writing coupled WMI providers, and seeing if I run into similar problems.

“New” is the new “goto”. As mock object testing frameworks and inversion of control containers are becoming more commonplace, we are encouraged to avoid having objects directly create other objects.

Why is “new” a bad word?
When a line of code includes the “new” keyword, it must also include the name of a concrete class. This line of code will appear within a different class (unless you are writing a singleton, in which case you have even bigger problems). You have just created a compile-time dependency between two classes.

The dependency inversion principle states that we should depend upon abstractions (i.e. interfaces and abstract base classes), rather than concrete classes. The “new” keyword immediately violates this principle because it forces us to depend upon a concrete class. The name after the “new” cannot be an interface or abstract base class, but a concrete packet of implementation.

Without dependency inversion, it is difficult or impossible to change units of behavior. If a class uses “new” to get access to a service component, you must change that code to replace it with a different service. If the change is a design-time decision, then recompiling the code just means more testing is required. But if the change is a run-time decision, then recompiling the code is not an option.

Without dependency inversion, it is difficult or impossible to mock objects for testing. If the code that you are testing is invoking “new”, then the mock framework cannot intercept that and replace it with a test harness. The dummy object is there to observe the calls that leak out of the object that you are testing, and make sure that they are the ones that the test expects to induce. If those calls go into a concrete class, then the scope of the test is bigger than intended, and visibility is lost.

Building blocks
I once had a manager who compared good software component design to Legos. A complete solution can be built from interchangeable pieces. If you are using “new”, your components are more like a puzzle pieces than building blocks. You create a solution out of pieces, but those pieces fit together in only one way.

As a rule of thumb, critically examine every “new” in your software. See if you can find a way to push it up through dependency injection. If you can, push it all the way out of your code and into an IoC container. The fewer “new”s, the better.

Here’s an obscure error that I fought for several hours today:

Jul 2, 2008 5:42:42 PM org.apache.catalina.core.ApplicationDispatcher invoke
SEVERE: Servlet.service() for servlet jsp threw exception
java.lang.NullPointerException
at org.apache.struts.taglib.TagUtils.retrieveMessageResources(TagUtils.java:1175)
at org.apache.struts.taglib.TagUtils.message(TagUtils.java:1038)
at org.apache.struts.taglib.TagUtils.message(TagUtils.java:1013)
at org.apache.struts.taglib.bean.WriteTag.retrieveFormatString(WriteTag.java:254)
at org.apache.struts.taglib.bean.WriteTag.formatValue(WriteTag.java:360)
at org.apache.struts.taglib.bean.WriteTag.doStartTag(WriteTag.java:232)
…. (the rest is unimportant)

This was thrown by the following line in a JSP:

<bean:write name=”report” property=”exceptionReportId” />

The problem is that my method getExceptionReportId() returns an int. The struts bean tag library expects all properties to be Strings. So by changing my method to return a String and using Integer.toString, I made it happy.

This presents a real separation-of-concerns problem. A bean should be a dumb data container. It should give you back exactly what you put in, and it should preserve your data types. The restriction that all properties are strings puts formatting into the bean, and therefore makes it specific to the JSP page.

Is there a tag library that separates formatting concerns from beans?

[UPDATE: You can format non-String property values using the bean:write format attribute. For a date, for example, <bean:write name=”report” property=”reportDate” format=”MM/dd/yyyy”/>. Furthermore, to fix the NullPointerException above, I just added an empty format attribute, as in <bean:write name=”report” property=”exceptionReportId” format=”" />.]

I’ve started a new project using Spring MVC. In doing so, I’ve had to invert my thinking.

Spring is an inversion of control container, which means that you don’t code dependencies of one class directly upon another. Instead you put all of your dependencies into one configuration file and keep your code as loosely coupled as possible. This one configuration file creates a graph of objects, each with references to the others. Since the configuration file specifies the classes and references, the code for one class doesn’t need to know the names of other classes.

Why is this important?
The dependency inversion principle tells us that it is better to depend upon something abstract than something concrete. This helps us to change how something is done without breaking the things that need it to be done. If your code depends upon an interface, you can change the thing that implements that interface without changing your code.

There are other reasons which have come to light since Robert Martin published his paper back in 1996. Dependency inversion makes unit testing easier, because you can replace the components that a unit calls with mock objects. Dependency inversion makes distributed computing easier, because you can replace business objects with proxies that call business logic on remote servers. In general, dependency inversion is a good goal.

How does Spring do it?
So if your class depends upon an interface rather than a concrete class, how does it get a reference to an object that implements that interface? It can’t use “new” to create an instance, because “new” needs a concrete class name. To invert dependency, you have to move all of your “new”s into one place. That place is an inversion of control (IoC) container.

Spring reads an XML file that contains a bunch of object descriptions. You can think of each of these as a call to “new”. Each one specifies a class name, an instance name, and a set of properties. These are write-once properties (what I like to call definitive), and should be initialized and never changed. These properties can include references to other object instances, thus forming a graph.

MVC
Spring MVC combines the dependency inversion principle with the model-view-controller pattern to create a pretty compelling web framework. The controllers and URL mappings are all configured through the IoC container. The URL mapper has a reference to each of the controllers, so it knows how to delegate the handling of a request. Because dependency has been inverted, the URL mapper doesn’t know about the concrete classes that are the controllers, it only knows about an interface. So you can use the out-of-the-box URL mapper with your own custom controllers.

But it just so happens that Spring has a quaint mechanism for database access. In your XML file, you can configure a data source by providing a driver class name and a connection string to an instance of “org.apache.commons.dbcp.BasicDataSource”.  Then you can use this data source to execute queries using “org.springframework.jdbc.core.simple.SimpleJdbcTemplate”. I wanted to use this technique from my data access layer. However, the XML file that defines the object graph is way up in my web layer. How can I push that object graph, or at least the data source, down through the business logic layer and into the data access layer?

The epiphany
That’s when the full realization of dependency inversion hit me. I was thinking about the web layer depending upon the business logic layer, which then depended upon the data access layer. This is not the Spring way. Instead, the web, business logic, and data access layers are all independent. The IoC container depends upon them all. The individual components within these layers only depend upon interfaces.

So the one XML file declares a data source. It then declares a data access component and gives it a reference to the data source. Next comes a business logic component and with a reference to the data access component. Then, the controller with a reference to the business logic component. And finally, the URL mapper comes last with a reference to the controller. As more URLs, controllers, and components are added, this chain widens into a graph.

You can’t pick and choose which pieces of your application use dependency inversion. Please don’t try. Once you start down the Spring path, all dependencies are inverted. The graph that’s defined at the highest layer of your application delves deep into the lowest, and touches all layers in between. Consider dependency inversion for your next project, and think carefully about the consequences.

This is not linq for Java, but it does bear a small resemblance. Download the source code and unit test. [Update: One unit test requires the Java tuple library.]

If you’ve ever needed to return an Iterable<T> from a Java method, you know that it can be challenging. If you already have exactly the collection that you want to iterate, then all is well. If not, you have two choices.

Option 1, you could construct a new ArrayList<T>, populate it, and return it. This option is less memory efficient than it could be. But it is by far the easiest way to go. You get to write your iteration code using “for”.

Or option 2, you could implement Iterable<T> with an anonymous inner class. Instead of using “for”, you’ll have to manage the state of your iterator yourself. Basically, you have to turn the looping code inside out and put the initialization, testing, and stepping code in separate methods. However, this saves on memory, as you don’t have to store the results in an intermediate collection.

On a number of occasions, I’ve taken option 2. Having rewritten the same ugly code several times, I decided to generalize it.

Inspiration from C#
Having spent some time in the .NET world, I’m familiar with two ways of solving this problem.

First, C# provides the “yield” keyword for turning a loop inside out. You write the code as per option 1, but instead of adding an object to a collection, you “yield return” it. The compiler turns the code inside out and makes it work like option 2. Yael has an implementation of yield return in java, if you like this approach.

Second, C# now has built-in syntax for querying any data source, including collections of objects. This syntax is called linq, or language integrated query. I didn’t want to implement all of the linq functionality in java, but I did use it as a source of inspiration.

From
Linq is like upside-down SQL. The first thing that you write is the “from” clause. This is important for type safety and intellisense. Borrowing from this convention, you start a java query with:

Query
    .from(collection)

This returns a Query<T>, which gives you the rest of the methods.

Where
A where clause filters the collection on-the-fly. The iterator that is produced only lands on the items that satisfy the where condition. You specify the condition by anonymously implementing the Predicate interface, as in:

Query
    .from(collection)
    .where(new Predicate<T>() {
        public boolean where(T row) {
            return /* Condition based on row */;
        }
    })

Join
A join traverses a nested collection. For each object of the first collection, you obtain an iterator over the second. The result is the same as two nested for loops. You return the inner collection using the Relation interface, as in:

Query
    .from(collection)
    .join(new Relation<PARENT, CHILD> () {
        public Iterable<CHILD> join(PARENT parent) {
            return parent.getChildren();
        }
    })

This produces an iterator of Join<PARENT, CHILD>.

Select
The last clause in a query is the select. Now that you’ve built up and narrowed down your collection of objects, the select clause gives you a chance to transform them into the data type that you want. You provide an anonymous implementation of the Selector interface:

Query
    .from(collection)
    .select(new Selector<T, RESULT>() {
        public RESULT select(T row) {
            return row.getProperty();
        }
    })

If you want to get the objects themselves, you can simply use “.selectRow()” instead.

Put it all together
You can combine these clauses to create the query you need. It has to start with a “from” and end with a “select”, but you can have any combination of “join” and “where” clauses in between. Here’s an example from the unit test:

// Iterate through the order items. Pull out order numbers where a widget is ordered.
Iterator<Integer> orderNumbers = Query
    .from(customers)
    .join(new Relation<Customer, Order>() {
        public Iterable<Order> join(Customer parent) {
            return parent.getOrders();
        }
    })
    .join(new Relation<Join<Customer,Order>, Item>() {
        public Iterable<Item> join(Join<Customer, Order> parent) {
            return parent.getSecond().getItems();
        }
    })
    .where(new Predicate<Join<Join<Customer,Order>,Item>>() {
        public boolean where(Join<Join<Customer, Order>, Item> row) {
            return row.getSecond().getPart().getName().equals(“widget”);
        }
    })
    .select(new Selector<Join<Join<Customer,Order>,Item>, Integer>() {
        public Integer select(Join<Join<Customer, Order>, Item> row) {
            return row.getFirst().getSecond().getNumber();
        }
    })
    .iterator();

Now you can generate iterators on-the-fly without wasting memory or writing really ugly code. Under the covers, it’s just doing what a for loop would do. But with a query, you can return the resulting collection to a caller without turning your code inside-out.

The first Dallas TechFest was this weekend. There were sessions on .NET, Ruby, REST, Groovy, Flex, AIR, and many other languages, tools, and platforms. It was a chance for members of different communities to get together and cross-train each other. I made a point of stepping outside my comfort zone, rather than sticking with the .NET track.

But the two sessions that I attended within the .NET track were hosted by Carl Franklin and Richard Campbell of .NET Rocks. I couldn’t miss the chance to see them. The first was a recording of a .NET Rocks episode on community. The second was Richard on web application performance and scalability. If you ever get a chance to hear Richard speak, you must go.

After his talk, I asked Richard to take a look at Update Controls. He was very gracious and gave me at least a half an hour of his time over lunch to show him the demo. He understood the benefits of the library immediately, and gave me some pointers for getting the idea across more effectively. Finally, he told me that I need to complete the database story, which, as it happens, is in the works.

It was a thrill to finally meet Carl and Richard, the guys who inspired me to launch this site. And it was enlightening to see what’s been going on in other parts of the industry. Who knows. Maybe I’ll get a chance to present next year.