Using Command Pattern to capture language and intent for services


This is a follow up to my previous post http://rogeralsing.com/2013/12/01/why-mapping-dtos-to-entities-using-automapper-and-entityframework-is-horrible/

That post was about both the technical and conceptual problems faced when using DataTransferObjects(DTOs) as the base of your service design.

In this post I will try to show you another way to model services that does not suffer from those problems on neither plane.

First, let’s talk about language.
I want to capture language and intent when modelling services, that is, I want to see the same vocabulary that the domain experts on my team use in my code.

Let’s re-use the same scenario as we used in my previous post:

//entities
class Order
    int Id //auto inc id
    string DeliveryAddress
    ICollection Details

class Detail
    int Id //auto inc id
    decimal Quantity
    string ProductId

Maybe our domain experts have told us that we need to be able to do the following:

Place new orders.
Edit an existing order.
Add a product to an order.
Change the quantity of a product in an order.
Remove a product from an order.
Set a delivery address for an order.
Ship an order.
Cancel an order.

If you are used to think in terms of CRUD, you might look at this and think:
Well, “placing an order” is the same as “CreateOrder”.
And “Editing an existing order” is the same as “UpdateOrder”.
And “Cancel an order” is the same as “DeleteOrder”
And I’ll solve the rest just by passing my DTO, let’s put a DeliveryAddress and a nullable ShippingDate property on order and just copy details from the DTO to the order.

OK, you will probably get away with it (for now), but the code will look like something in the previous post liked at the top here.
And most importantly, you will lose semantics and intention.

e.g. Does “order.ShippingDate = DateTime.Now” mean that the order is shipped now? that it is up for shipping?
What if we reset the order.ShippingDate to null? does that mean that the order have been returned to us?
Is that even a valid state transition?
Your code might be small’ish, except for the weird change tracking replication you have in your update service.

Wouldn’t it be nice if we instead could capture valid state transitions and the domain experts language at the same time?
What if we do something like this:

public int PlaceNewOrder(){
    using (UoW.Begin())
    {
       var order = new Order();
       UoW.Add(order);
       UoW.Commit();
       return order.Id;
    }
}

public void AddProduct(int orderId,string productId,double quantity)
{
    using (UoW.Begin())
    {
       var order = UoW.Find(o => o.Id == orderId);
       if (order.IsCancelled)
          throw new Exception("Can not add products to cancelled orders");

       var detail = new Detail
       {
           ProductId = productId,
           Quantity = quantity,
       }
       order.Details.Add(detail);
       UoW.Commit();
    }
}

public void ShipOrder(int orderId,DateTime shippingDate)
{
  ...
}

This is of-course a very naive example, but still, we managed to capture the language of the domain experts, and even if this will make our code more verbose, it captures explicit state transitions in a very clean way.
Someone new to the project can open up this code and they will be able to figure out what is going on in each individual method.
e.g. you will most likely not misunderstand the intention of a method called “ShipOrder” which takes an argument called ShippingDate.

So far so good, but from a technical point of view, this code is horrible, we can’t expose a service interface that is this fine-grained over the wire.
This is where command pattern comes in.

Lets redesign our code to something along the lines of:

public static class OrderService
{
    public static void Process(IEnumerable<command></command> commands)
    {
        var ctx = new MyContext();
        Order order = null;
        foreach (var command in commands)
            order = command.Execute(ctx, order);
        ctx.SaveChanges();
    }
}

public abstract class Command
{
    public abstract Order Execute(MyContext container,Order order);
}

public class CreateNewOrder : Command
{
    public override Order Execute(MyContext container, Order order)
    {
        order = new Order();
        container.OrderSet.Add(order);
        return order;
    }
}

public class EditExistingOrder : Command
{
    public int OrderId { get; set; }
    public override Order Execute(MyContext container, Order order)
    {
        order = container.OrderSet.Single(o => o.Id == OrderId);
        return order;
    }
}

public class SetDeliveryAddress : Command
{
    public string DeliveryAddress { get; set; }

    public override Order Execute(MyContext container, Order order)
    {
        order.Address = DeliveryAddress;
        return order;
    }
}

public class RemoveProduct : Command
{
    public string ProductId { get; set; }

    public override Order Execute(MyContext container, Order order)
    {
        order.Details.Remove(d => d.Name == ProductId);
        return order;
    }
}

public class AddProduct : Command
{
    public string ProductId { get; set; }
    public decimal Quantity { get; set; }

    public override Order Execute(MyContext container, Order order)
    {
        var detail = new Detail
        {
            Name = ProductId,
            Quantity = Quantity,
        };
        order.Details.Add(detail);
        return order;
    }
}
...etc

So, what happened here?
We have created a single operation, Execute, that receives a sequence of “commands”.
Each command is modeled as a subclass of “Command”.
This enables us to send a sequence of commands and process them all in the same transaction.

We have captured language, intentions, and each command can easily be tested.
And as a bonus, it doesn’t suffer from the weird mapping problems that the DTO approach suffers from.

Another nice aspect of this is that if we want to edit a few details of a large object, e.g. we want to change quantity of a single product in an order of 100 order details, we can send a single command.

So to sum it up, pros:

  • Captured domain language
  • Explicit state transitions
  • Network effective
  • Testable
  • Easy to implement, no odd change tracking problems like using the DTO approach
  • Extensible, we can easily add new commands to the existing pipeline w/o adding more service operations

Cons:

  • Verbose
  • Puts somewhat more responsibility on the client

The code in this example can of-course be improved, we are using an anemic domain model right now, but the essence of the command pattern can be seen here.

I will post a more “correct” implementation in my next post

That’s all for now

Entity Framework to REST – EfREST


I’ve been doing a fair amout of rest API coding lately, and what struck me during this time was that it was horribly hard to actually create a clean custom REST API using .NET.
Sure, there is WebApi and OData and alot of other stuff.
Odata almost gets me where I want to be, but it does have it’s limitations.

I like to design my services according to the guidelines that http://stormpath.com has: http://www.stormpath.com/blog/designing-rest-json-apis
I really do like their approach to how services should be designed.

So I decided to give it a try if I could accomplish this kind of design using standard tools; WebApi and EF5:

The goal was to be able to:

  1. Select all data needed for a given resource in one go, a single database call.
  2. Transform my Entity Model to resources any way I want, not expose my entities directly to the world
  3. Include a resource path (href) for each resource.
  4. Support ?fields=…  to select a subset of fields il the resource
  5. Support ?expand=… to expand navigational properties in the resource
  6. Re-use select projections for multiple resources

1) This is really important IMO, I’m not a big fan of ripple loading where a single entity fetch results in a ton of database calls.
This can be accomplished using entity framework using projections (or .Include()).

var result = model.Users.Select (u => new {
        id = u.Id,
        name = u.Name,
        items = u.Items.Select (i => new {
          id = i.Id,
          name = i.Name,
        })
    }).ToList();

This would select users together with their “items” whatever that might be in a single go.
And this is the way I normally use EF for selecting specific subsets of data.

2) Transforming my data into resources is also something I find important. I do not want to expose entities straight off to the wire. I might have calculated properties and data from external resources included in my result.

Luckily, transformations can be solved the same way as the above example, using projections. so this approach solves both point 1) and 2).

3) I don’t like to include Id’s as properties in my resources, I want to expose a “href” for the resource, ofcourse, this href may include the ID of the entity behind the resource, but the data exposed will be a href to where the actual resource can be found.

This makes it possible to get/post/put/delete directly to the href of the resource w/o manually building an URL from ID’s and strings in the client.

To solve this I like to do something along the lines of this:

var result = model.Users.Select (u => new {
        href = "/api/v1/users/" + u.Id,
        name = u.Name,
        items = u.Items.Select (i => new {
          href = "/api/v1/users/" + u.Id + "/items/" + i.Id,
          id = i.Id,
          name = i.Name,
        })
    }).ToList();

This is however not possible with Entity Framework.
WHAT?!
No, Entity Framework can’t translate the above code to SQL.
You have to resort to SqlFunctions and weird casts to make it work. making the code unreadable.
This could however be fixed with some Linq Expression Tree magic.
I rolled my own ExpressionTreeVisitor and simply patched the above code into the code that EF needs in order to work.

4 and 5)
Some clients may want to exclude certain fields in order to lower request times and memory consumption for e.g. mobile devices.
This is somewhat tricky using Entity Framework since we are in a strongly typed environment here.
How do you make a Linq request against an anonymous type return only a subset of what is specified in the query?
-More Linq expression magic and some Reflection.Emit, I rewrite the above query according to the ?fields/expand arguments passed into the service, selecting a new type which contains only those fields.

6) Re-use of projections in Entity Framework is a no-go, you simply can not do it more than at the base level.
e.g.
You can do var res = model.Users.Select(someProjectionExpression);
But you can not do this for subqueries:

Expression<Func<Item,object>> itemProjection = i => new {
          id = i.Id,
          name = i.Name,
        };

Expression<Func<User,object>> userProjection = u => new {
        id = u.Id,
        name = u.Name,
        items = u.Items.Select (itemProjection)
    };

This will give a compile error if User.Items is of type ICollection of T since we are trying to pass an Expression.
This was also solved using the ExpressionVisitor, I simply expand the Linq Expression in the Select call.

Well..  that’s pretty much it, I have implemented the above features in a small framework called EfRest, which can be found here: https://github.com/rogeralsing/EfRest

Here is a real world example using the framework in WebApi:

//projection for "Site" resource
public Expression<Func<Site, object>> SiteSelector()
{
    return s => new
    {
        href = sitesUrl + "/" + s.Id,
        name = s.Domain,
        sitemapUrl = s.SiteMapUrl,
        pages = new
        {
            href = sitesUrl + "/" + s.Id + "/pages",
            items = s
                    .WebPages
                    .ExpandSelect(PageSelector())
        }
    };
}

//projection for "Page" resource
public Expression<Func<WebPage, object>> PageSelector()
{
    return p => new
    {
        href = sitesUrl + "/" + p.Site.Id + "/pages/" + p.Id,
        fetchDate = p.FetchDate,
        statusCode = p.StatusCode,
        url = p.Url,
        botVisits = new
        {
            href = sitesUrl + "/" + p.Site.Id + "/pages/" + p.Id + "/botvisits",
            items = p
                .BotVisits
                .ExpandSelect(BotSelector())
        }
    };
}

//projection for "BotVisit" resource
public  Expression<Func<BotVisit, object>> BotSelector()
{
    return b => new
    {
        name = b.BotName,
        visitDate = b.VisitDate,
    };
}

[Authorize]
[HttpGet("api/v1/users/me/sites/{siteId}")]
public HttpResponseMessage MeSitesGet(int siteId, string fields = null,string expand = null)
{
    using (var model = new ModelContainer())
    {
        var res = model
            .Sites
            .AsResource(fields, expand) //apply what fields and expands to include
            .Where(s => s.UserId == WebSecurity.CurrentUserId && s.Id == siteId)
            .OrderBy(s => s.Domain)
            .ExpandSelect(SiteSelector()) //note ExpandSelect instead of Select
            .FirstOrDefault();

        return Request.CreateResponse(HttpStatusCode.OK, res);
    }
}

The above sample is used on our Javascript SEO tool RankJS.com

Enjoy!

O/R Mapping and domain query optimizations


One of the cons of O/R mapping is that the abstraction is a bit too high.
You write object-oriented code and often forget about eventual performance problems.

Take this (somewhat naive) example:

class Customer
{
   ...
  public double GetOrderTotal()
   {
       var total = ( from order in this.Orders
                        from detail in order.Details
                        select detail.Quantity * detail.ItemPrice)
                       .Sum();

       return total;
   }
}

For a given customer, we iterate over all the orders and all the details in those orders and calculate the sum of quantity multiplied with itemprice.
So far so good.

This will work fine as long as you have all the data in memory and the dataset is not too large, so chances are that you will not notice any problems with this code in your unit tests.

But what happens if the data resides in the database and we have 1000 orders with 1000 details each?
Now we are in deep s##t, for this code to work, we need to materialize at least 1 (cust) + 1000 (orders) * 1000 (details) entities.
The DB needs to find those 1 000 001 rows , the network needs to push them from the DB server to the App server and the App server needs to materialize all of it.
Even worse, what if you have lazy load enabled and aren’t loading this data using eager load?
Then you will hit the DB 1 000 001 times… GL with that! :-)

So clearly, we can not do this in memory, neither with lazy load nor eager load.

But what are the alternatives?
Make an ad hoc sql query?
In that case, what happens to your unit tests?

Maybe we want to keep this code, but we want to execute it in the database instead.

This is possible if we stop beeing anal about “pure POCO” or “no infrastructure in your entities”

Using an unit of work container such as https://github.com/rogeralsing/Precio.Infrastructure

We can then rewrite the above code slightly:

class Customer
{
   ...
  public double GetOrderTotal()
   {
  var total = ( from customer in UoW.Query<Customer>() //query the current UoW
                        where customer.Id == this.Id //find the persistent record of "this"
                        from order in customer.Orders
                        from detail in order.Details
                        select detail.Quantity * detail.ItemPrice)
                       .Sum();

       return total;
   }
}

This code will run the query inside the DB if the current UoW is a persistent UoW.
If we use the same code in our unit tests and use an in mem UoW instance, this code will still work, if our customer is present in the in mem UoW that is..

So the above modification will reduce the number materialized entities from 1 000 001 to 1 (we materialize a double in this case)

I don’t know about you , but I’d rather clutter my domain logic slightly and get a million times better performance than stay true to POCO and suffer from a broken app.

UoW / NWorkspace with Linq support


I have blogged about this for quite a while now.
Now I’ve finally cleaned up the code and published it at github:https://github.com/rogeralsing/Precio.Infrastructure

This is a small framework for UoW/Workspace support in .NET with Linq support.

The framework contains a Unit of Work implementation and providers for Entity Framework 4, NHibernate and MongoDB(using NoRM).
There is also a small incomplete Blog sample project included.

Linq to SqlXML


I’m hacking along on my Document DB emulator ontop of Sql Server XML columns.

I have some decent Linq support in place now.
The following query:

var query = from order in orders
            //must have status shipped
            where order.Status >= OrderStatus.Shipped      
            //must contain foo or bar products
            where order.OrderDetails.Any(d => d.ProductNo == "Foo" || d.ProductNo == "Bar")
            //must have an order total > 100
            where order.OrderDetails.Sum(d => d.ItemPrice * d.Quantity) > 100 
            select order;

will yield the following Sql + XQuery to the Sql Server:

select *
from documents
where CollectionName = 'Order'  and 
--must have an order total > 100
(documentdata.exist('/object/state[(
     fn:sum( 
          for $A in OrderDetails[1]/object/state 
                return ($A/ItemPrice[1] * $A/Quantity[1])) > xs:decimal(100))]') = 1) and 
--must contain foo or bar products
(documentdata.exist('/object/state[OrderDetails[1]/object/state[((ProductNo[1] = "Foo") or 
 (ProductNo[1] = "Bar"))]]') = 1) and 
--must have status shipped
(documentdata.exist('/object/state[(Status[1] >= xs:int(2))]') = 1)

Building a Document DB ontop of Sql Server


I’ve started to build a Document DB emulator ontop of Sql Server XML columns.
Sql Server XML columns can store schema free xml documents, pretty much like RavenDB or MongoDB stores schema free Json/Bson documents.

XML Columns can be indexed and queried using XPath queries.

So I decided to build an abstraction layer ontop of this in order to achieve similair ease of use.
I’ve built a serializer/deserializer that deals with my own XML structure for documents (state + metadata) and also an early Linq provider for querying.

Executing the following code:

var ctx = new DocumentContext("main");
var customers = ctx.GetCollection<Customer>().AsQueryable();

var query = from customer in customers
            where customer.Address.City == "abc" && customer.Name == "Acme Inc5"
            orderby customer.Name
            select customer;

var result = query.ToList();
foreach (var item in result)
{
    Console.WriteLine(item.Name);
    Console.WriteLine(item.Address.City);
}

Will yield the following SQL + XPath query:

select *

from documents

where CollectionName = 'Customer' and
   ((documentdata.exist('/object/state/Address/object/state/City/text()[. = "abc"]') = 1) and
    (documentdata.exist('/object/state/Name/text()[. = "Acme Inc5"]') = 1))

order by documentdata.value('((/object/state/Name)[1])','nvarchar(MAX)')

The result of the query will be returned to the client and then deserialized into the correct .NET type.