Hotswap and Supervision – Pigeon – Akka Actors for .NET


This is a follow up on my previous post; http://rogeralsing.com/2014/01/01/pigeon-akka-actors-for-net/

The code for this project can be found here: https://github.com/rogeralsing/Pigeon

And again, before I begin, please do note that Pigeon is just my hobby project of cloning some of the Akka Actor API.

Akka is in no way affiliated with me or my project.

I’ve done some refactoring to mimic Akka’s API more closely.
e.g. I’ve removed the need for IMessage on messages, you can now pass any object as an actor message.

I’ve also added some more nifty features from Akka - Hotswap and Supervision.

Hotswap is the ability to change the message receiver, this is useful if you have some sort of state machine, e.g. if you receive one message and you want to handle messages differently once that message is received.

Like this:

public class GreetingActor : UntypedActor
{
    protected override void OnReceive(IMessage message)
    {
        Pattern.Match(message)
            .With<Greet>(m => {
                Console.WriteLine("Hello {0}", m.Who);
                //this could also be a lambda
                Become(OtherReceive);
            });
    }

    void OtherReceive(IMessage message)
    {
        Pattern.Match(message)
            .With<Greet>(m => {
                Console.WriteLine("You already said hello!");
                //Unbecome() to revert to old behavior
            });
    }
}

In the example above, the actor will first output “Hello {who}” if a Greet message is passed to it.
But if we pass another greet message to it, it will output “You already said hello”.
This is ofcourse a contrived example, but it is very useful for larger state machines.
e.g. Lets say we write a game and each player/bot has its own actor.
We could then make it respond differently if you are stunned, dead, or trapped by some spell or whatever.

You can read more about this feature in the Akka documentation here: http://doc.akka.io/docs/akka/snapshot/java/untyped-actors.html#HotSwap

The next feature is Supervision, supervision lets a parent actor monitor it’s children and decide what to do if they start failing.
e.g. you can configure it to restart, restart or escalate if an exception of a given type occurs too many times in a given timespan.

Like this:

public class MyActor : UntypedActor
{
    private ActorRef logger = Context.ActorOf<LogActor>();

    // if any child, e.g. the logger above. throws an exception
    // apply the rules below
    // e.g. Restart the child if 10 exceptions occur in 30 seconds or less
    protected override SupervisorStrategy SupervisorStrategy()
    {
        return new OneForOneStrategy(
            maxNumberOfRetries: 10,
            duration: TimeSpan.FromSeconds(30),
            decider: x =>
            {
                if (x is ArithmeticException)
                    return Directive.Resume;
                if (x is NotSupportedException)
                    return Directive.Stop;

                return Directive.Restart;
            });
    }

    ...
}

The “logger” above is a child actor of “MyActor”, this means that MyActor is the parent and supervisor for logger.
If logger starts throwing exceptions, those will be forwarded to the supervisor strategy, and depending on configuration, the parent can resolve the issue by restarting or stopping the child etc.
You can read about this in the Akka documentation: http://doc.akka.io/docs/akka/snapshot/general/supervision.html

About these ads

Pigeon – Akka Actors for .NET


I’m working on a port of the Java/Scala framework Akka, or rather the Actor Model part of it.
As of now, this is only a bit more than a weekend hack, so don’t expect too much.
But I already have a couple of nice features in:

Git hub repository goes here: https://github.com/rogeralsing/Pigeon

Write your first actor:

public class Greet : IMessage
{
    public string Who { get; set; }
}

public class GreetingActor : UntypedActor
{
    protected override void OnReceive(IMessage message)
    {
        message.Match()
            .With(m => Console.WriteLine("Hello {0}", m.Who));
    }
}

Usage:

var system = new ActorSystem();
var greeter = system.ActorOf("greeter");
greeter.Tell(new Greet { Who = "Roger" }, ActorRef.NoSender);

Remoting support:

//Server Program.CS
var system = ActorSystemSignalR.Create("myserver", "http://localhost:8080);
var greeter = system.ActorOf("greeter");
Console.ReadLine();

//Client Program.CS
var system = new ActorSystem();
var greeter = system.ActorSelection("http://localhost:8080/greeter");
//pass a message to the remote actor
greeter.Tell(new Greet { Who = "Roger" }, ActorRef.NoSender);

There are still alot of things that needs to be implemented/fixed.
Actor paths don’t work correctly at the momeent for example.
Supervision and other features are missing etc.

One nice feature I managed to squeeze in was “Futures”.
Futures in Akka is pretty much Task of T in .NET.
However, Futures in Akka is synchronous and you need to (bussy) wait for them to complete.
[edit]
As Alexey Romanov pointed out in the comments, this is not true.
You can get async futures using either onComplete callbacks,
or using Scala async support https://github.com/scala/async and use async/await.

In Pigeon however, we have Async/Await support.

You can async await responses from other actors like this:


//inside an actor
var result = await Ask(someActor, messageToAsk);
result.Match()
    .With(m => {
         Console.WriteLine("got awaited result {0}",m);
    })
    .Default(_ => Console.WriteLine("Unknown message"));

The nice thing is that the await continuation will execute in the actor context, so there is no threading issues inside the actor here.

Hope you like it :)

Using Command Pattern to capture language and intent for services


This is a follow up to my previous post http://rogeralsing.com/2013/12/01/why-mapping-dtos-to-entities-using-automapper-and-entityframework-is-horrible/

That post was about both the technical and conceptual problems faced when using DataTransferObjects(DTOs) as the base of your service design.

In this post I will try to show you another way to model services that does not suffer from those problems on neither plane.

First, let’s talk about language.
I want to capture language and intent when modelling services, that is, I want to see the same vocabulary that the domain experts on my team use in my code.

Let’s re-use the same scenario as we used in my previous post:

//entities
class Order
    int Id //auto inc id
    string DeliveryAddress
    ICollection Details

class Detail
    int Id //auto inc id
    decimal Quantity
    string ProductId

Maybe our domain experts have told us that we need to be able to do the following:

Place new orders.
Edit an existing order.
Add a product to an order.
Change the quantity of a product in an order.
Remove a product from an order.
Set a delivery address for an order.
Ship an order.
Cancel an order.

If you are used to think in terms of CRUD, you might look at this and think:
Well, “placing an order” is the same as “CreateOrder”.
And “Editing an existing order” is the same as “UpdateOrder”.
And “Cancel an order” is the same as “DeleteOrder”
And I’ll solve the rest just by passing my DTO, let’s put a DeliveryAddress and a nullable ShippingDate property on order and just copy details from the DTO to the order.

OK, you will probably get away with it (for now), but the code will look like something in the previous post liked at the top here.
And most importantly, you will lose semantics and intention.

e.g. Does “order.ShippingDate = DateTime.Now” mean that the order is shipped now? that it is up for shipping?
What if we reset the order.ShippingDate to null? does that mean that the order have been returned to us?
Is that even a valid state transition?
Your code might be small’ish, except for the weird change tracking replication you have in your update service.

Wouldn’t it be nice if we instead could capture valid state transitions and the domain experts language at the same time?
What if we do something like this:

public int PlaceNewOrder(){
    using (UoW.Begin())
    {
       var order = new Order();
       UoW.Add(order);
       UoW.Commit();
       return order.Id;
    }
}

public void AddProduct(int orderId,string productId,double quantity)
{
    using (UoW.Begin())
    {
       var order = UoW.Find(o => o.Id == orderId);
       if (order.IsCancelled)
          throw new Exception("Can not add products to cancelled orders");

       var detail = new Detail
       {
           ProductId = productId,
           Quantity = quantity,
       }
       order.Details.Add(detail);
       UoW.Commit();
    }
}

public void ShipOrder(int orderId,DateTime shippingDate)
{
  ...
}

This is of-course a very naive example, but still, we managed to capture the language of the domain experts, and even if this will make our code more verbose, it captures explicit state transitions in a very clean way.
Someone new to the project can open up this code and they will be able to figure out what is going on in each individual method.
e.g. you will most likely not misunderstand the intention of a method called “ShipOrder” which takes an argument called ShippingDate.

So far so good, but from a technical point of view, this code is horrible, we can’t expose a service interface that is this fine-grained over the wire.
This is where command pattern comes in.

Lets redesign our code to something along the lines of:

public static class OrderService
{
    public static void Process(IEnumerable<command></command> commands)
    {
        var ctx = new MyContext();
        Order order = null;
        foreach (var command in commands)
            order = command.Execute(ctx, order);
        ctx.SaveChanges();
    }
}

public abstract class Command
{
    public abstract Order Execute(MyContext container,Order order);
}

public class CreateNewOrder : Command
{
    public override Order Execute(MyContext container, Order order)
    {
        order = new Order();
        container.OrderSet.Add(order);
        return order;
    }
}

public class EditExistingOrder : Command
{
    public int OrderId { get; set; }
    public override Order Execute(MyContext container, Order order)
    {
        order = container.OrderSet.Single(o => o.Id == OrderId);
        return order;
    }
}

public class SetDeliveryAddress : Command
{
    public string DeliveryAddress { get; set; }

    public override Order Execute(MyContext container, Order order)
    {
        order.Address = DeliveryAddress;
        return order;
    }
}

public class RemoveProduct : Command
{
    public string ProductId { get; set; }

    public override Order Execute(MyContext container, Order order)
    {
        order.Details.Remove(d => d.Name == ProductId);
        return order;
    }
}

public class AddProduct : Command
{
    public string ProductId { get; set; }
    public decimal Quantity { get; set; }

    public override Order Execute(MyContext container, Order order)
    {
        var detail = new Detail
        {
            Name = ProductId,
            Quantity = Quantity,
        };
        order.Details.Add(detail);
        return order;
    }
}
...etc

So, what happened here?
We have created a single operation, Execute, that receives a sequence of “commands”.
Each command is modeled as a subclass of “Command”.
This enables us to send a sequence of commands and process them all in the same transaction.

We have captured language, intentions, and each command can easily be tested.
And as a bonus, it doesn’t suffer from the weird mapping problems that the DTO approach suffers from.

Another nice aspect of this is that if we want to edit a few details of a large object, e.g. we want to change quantity of a single product in an order of 100 order details, we can send a single command.

So to sum it up, pros:

  • Captured domain language
  • Explicit state transitions
  • Network effective
  • Testable
  • Easy to implement, no odd change tracking problems like using the DTO approach
  • Extensible, we can easily add new commands to the existing pipeline w/o adding more service operations

Cons:

  • Verbose
  • Puts somewhat more responsibility on the client

The code in this example can of-course be improved, we are using an anemic domain model right now, but the essence of the command pattern can be seen here.

I will post a more “correct” implementation in my next post

That’s all for now

Why mapping DTOs to Entities using AutoMapper and EntityFramework is horrible


One of the most common architectures for web apps right now is based on passing DataTransferObjects(DTOs) to and from CRUD services that updates your business/domain entities using tools like AutoMapper and EntityFramework.

I will try to explain why this is a truly horrible approach.

Horrible from a technical perspective

Firstly, lets set up a scenario that we can reason about here:
The scenario here is an Order service with an Order entity and an OrderDTO.

Example code (pseudo code):

 //entities
 class Order
     int Id //auto inc id
     List<Detail> Details

 class Detail
     int Id //auto inc id
     decimal Quantity
     string ProductId

 //DTO's
 class OrderDTO
     int Id
     List<DetailDTO> DetailsDTO

 class DetailDTO
     int Id
     decimal Quantity
     string ProductId

I’m pretty sure you have seen something similar to this before.

Lets also assume there is some sort of service that updates orders:

    public void CreateOrUpdateOrder(OrderDTO orderDTO)
    {
       ...
    }

So what do we need to do in order to make this work?
Maybe you do something like this:

    public void CreateOrUpdateOrder(OrderDTO orderDTO)
    {
        //this code would probably reside somewhere else, 
        //in some app startup or config
        Mapper.CreateMap<OrderDTO,Order>();
        Mapper.CreateMap<DetailDTO,Detail>();

        //real code starts here
        var context = ... //get some DbContext
        var order = Mapper.Map<OrderDTO,Order>(orderDTO);
        context.OrderSet.Add(order); 
        context.SaveChanges();
    }      

Looks fair enough?

This will work fine for a new order, the above code will map from DTOs to new entities and then add the new entities to the dbcontext and save the changes.
All is fine and dandy.

But what happens if we pass an existing order?
We will get into trouble because of the context.OrderSet.Add, existing orders will become new orders in the DB because we are always “adding”.
So what can we do about this?

Maybe we can do it like this?

public void CreateOrUpdateOrder(OrderDTO orderDTO)
{
   //this code would probably reside somewhere else, 
   //in some app startup or config
   Mapper.CreateMap<OrderDTO,Order>();
   Mapper.CreateMap<DetailDTO,Detail>();

   //real code starts here
   var context = ... //get some DbContext

   //try to fetch an existing order with the same Id as the DTO
   var order = context.OrderSet.FirstOrDefault(o => o.Id == orderDTO.Id);
   if (order == null)
   {
       //if no order was found, create a new and add it to the context
       order = new Order();
       context.OrderSet.Add(order);
   }
        
   //Map the DTO to our _existing_ or new order
   Mapper.Map<OrderDTO,Order>(orderDTO,order); 
   context.OrderSet.Add(order); 
   context.SaveChanges();
}      

Ah, that’s better, right?
Now we will not add new orders every time we try to update an existing one.
So, whats the problem now?

We still create new entities every time for each detail, AutoMapper knows nothing about EF and ID’s so it will simply recreate every object in the .Details collection.
So even if we can create and update the order entity, we are still messed up when dealing with the details.

Lets try again:

public void CreateOrUpdateOrder(OrderDTO orderDTO)
{
   var context = ... //get some DbContext
   
   //let automapper be able to find or create the entity based on Id
   Mapper.CreateMap<OrderDTO,Order>()
   .ConstructUsing((OrderDTO orderDTO) =>
   {
       if (orderDTO.Id == 0)
       {
           var order = new Order();
           context.OrderSet.Add(order);
           return order;
       }
       return context
              .OrderSet.Include("Details")
              .First(o => o.Id == orderDTO.Id);
   });

  //let automapper be able to find or create the entity based on Id
  Mapper.CreateMap<DetailDTO, Detail>()
  .ConstructUsing((DetailDTO detailDTO) =>
  {
      if (detailDTO.Id == 0)
      {
          var detail = new Detail();
          context.DetailSet.Add(detail);
          return detail;
      }
      return context
            .DetailSet.First(d => d.Id == detailDTO.Id);
   });    
 
   //AutoMapper will now know when to create a 
   //new entity and when to fetch one from the DbContext       
   Mapper.Map<OrderDTO,Order>(orderDTO); 

   context.SaveChanges();
}      

OK, now we are on the right track, we can update existing entities and create new entities when needed.
All of the magic is in the AutoMapper config.
Nice, right?
This will work fine for every Create or Update scenario.

Well, err.. it won’t handle deleted details. if a detain have been removed from the orderDTO detail collection, then we will have detail entities that are unbound in entity framework.
We have to delete those otherwise we will have a foreign key exception when calling SaveChanges.

OK, so lets see what we can do about this:

public void CreateOrUpdateOrder(OrderDTO orderDTO)
{
   var context = ... //get some DbContext
     
   Mapper.CreateMap<OrderDTO,Order>()
   .ConstructUsing((OrderDTO orderDTO) =>
   {
       if (orderDTO.Id == 0)
       {
           var order = new Order();
           context.OrderSet.Add(order);
           return order;
       }
       return context
              .OrderSet.Include("Details")
              .First(o => o.Id == orderDTO.Id);
       })
       //find out what details no longer exist in the 
       //DTO and delete the corresponding entities 
       //from the dbcontext
       .BeforeMap((dto, o) =>
       {
          o
           .Details
           .Where(d => !dto.Details.Any(ddto => ddto.Id == d.Id))
           .ToList()
           .ForEach(deleted => context.DetailSet.Remove(deleted));
       });

       Mapper.CreateMap<DetailDTO, Detail>()
      .ConstructUsing((DetailDTO detailDTO) =>
      {
         if (detailDTO.Id == 0)
         {
            var detail = new Detail();
            context.DetailSet.Add(detail);
            return detail;
         }
         return context.DetailSet.First(d => d.Id == detailDTO.Id);
      });    
 
     Mapper.Map<OrderDTO,Order>(orderDTO); 

     context.SaveChanges();
}      

Ah, sweet, now we can handle add, update and deletes for the order details.
This code will work fine in your unit tests..

But..

It will bite you in production.
Why?
Because we are binding the dbcontext using closures in the static AutoMapper mappings.
This code is not thread safe, if two requests arrive at the server at the same time, it will break, becuase both will have mappings bound to one of the active DbContexts.

AutoMapper’s static Mapper class is not threadsafe, we have to modify our code even further.

public void CreateOrUpdateOrder(OrderDTO orderDTO)
{
   var config = new ...

   //create an instanced mapper instead of the static API
   var mapper = new AutoMapper.MapperEngine(config);
   var context = ... //get some DbContext
     
   config.CreateMap<OrderDTO,Order>()
   .ConstructUsing((OrderDTO orderDTO) =>
   {
      if (orderDTO.Id == 0)
      {
          var order = new Order();
          context.OrderSet.Add(order);
          return order;
      }
      return context.OrderSet.Include("Details").First(o => o.Id == orderDTO.Id);
   })
   //find out what details no longer exist in the DTO and delete the corresponding entities 
   //from the dbcontext
   .BeforeMap((dto, o) =>
   {
     o
     .Details
     .Where(d => !dto.Details.Any(ddto => ddto.Id == d.Id)).ToList()
     .ForEach(deleted => context.DetailSet.Remove(deleted));
   });

   config.CreateMap<DetailDTO, Detail>()
   .ConstructUsing((DetailDTO detailDTO) =>
   {
       if (detailDTO.Id == 0)
       {
            var detail = new Detail();
            context.DetailSet.Add(detail);
            return detail;
       }
       return context.DetailSet.First(d => d.Id == detailDTO.Id);
   });    
 
   mapper.Map<OrderDTO,Order>(orderDTO); 

   context.SaveChanges();
}      

Now then?
Yeah, we solved it.
This will work in threaded environments and it does solve the add,modify and delete issues.

However, our code is now so bloated with AutoMapper configs that it is hard to tell what is going on here.
Even if this solves the use case here, I find this a horrible design, give this so called “simple design” to a rookie that is going to maintain this code for years to come and he will not have any idea what to do with it.

Pros of this design:

  • Looks simple on paper
  • Easy to implement on read side and client side

Cons:

  • Bloody horrible to implement on the write side, and gets even worse the larger the DTO is
  • Relies on magic names if using AutoMapper
  • Network ineffective if dealing with large DTOs
  • You lose semantics and intentions, there is no way to know WHY a change happened

But do note that this can be a valid design, IF, you use a document database or similar.
That is, if you don’t have a relational storage where you have to analyze what changes was made to the server data.
e.g. in a Rest API using MongoDB or such, but that has little to do with the key points in this article, which is auto mapping to ORMapper entities.

One of the biggest pain points in this specific case is actually AutoMapper, I think AutoMapper is a great tool, just not for this specific usecase.
If we remove automapper from the equation here. we could end up with some code that looks like this:

public void CreateOrUpdateOrder(OrderDTO orderDTO)
{
    var ctx = ... //get some DbContext
    var order = ctx.OrderSet.FirstOrDefault(o => o.Id == orderDTO.Id);
    if (order == null)
    {
        order = new Order();
        ctx.OrderSet.Add(order);
    }

    //Map properties
    order.Address = orderDTO.Address;            

    //remove deleted details
    order.Details
    .Where(d => !orderDTO.Details.Any(detailDTO => detailDTO.Id == d.Id))
    .Each(deleted => ctx.DetailSet.Remove(deleted));

    //update or add details
    orderDTO.Details.Each(detailDTO =>
    {
        var detail = order.Details.FirstOrDefault(d => d.Id == detailDTO.Id);
        if (detail == null)
        {
            detail = new Detail();
            order.Details.Add(detail);
        }
        detail.Name = detailDTO.Name;
        detail.Quantity = detailDTO.Quantity;
    });

   context.SaveChanges();
}      

IMO, this is cleaner, a rookie should not have much problem understanding what is going on, possibly that the “Remove deleted details” part may cause some confusion.
But still, more readable than the first AutoMapper approach (And refactor friendly).

If I saw the above code in a real project, I wouldn’t be too bothered, it’s not a disaster, but do remember that it will get worse the larger and more complicated the DTO is.
Just imagine that the details have references to product entities, we soon have a big ball of “ripple load” where each detail iteration causes a load on product.
If you continue down this route, you will have a really hard to maintain code base and an angry DB breathing up your butt because the code is touching persisted objects in every iteration..

Conclusion:
AutoMapper is a great tool, but it is not built for mapping DTOs to O/R-Mapped entities. it lacks knowledge about things like dirty tracking, identity maps and orphan entities.
EntityFramework is what it is, many will bash it for being a naive mapper, I think it does it’s job in most cases even if it cant handle all the fancy mappings that NHibernate can.
The problem here is that most people knows little to nothing about how an ORM really works behind the scenes, and simply believes that you can slap a changed object graph onto the Context/Manager/Session/UoW and the mapper should magically figure out what the heck you did to that object. sorry but that’s not how it works.
If it did, it would have to do crap loads of queries against the database to compare the new state to the old state.
If you decide to use an advanced tool like an ORM, make sure you know how they work, ORM’s are insanely complex and they try to hide that complexity from you by simulating that you are working with a normal object graph, this is the true danger of ORMs, they make explicit concepts like a db-query implicit. you can’t see what is going on, you just touch objects and properties.

Horrible from a conceptual perspective

The design described in this post prevents you from doing Domain Driven Design, it is 100% impossible to use DDD if you are patching state directly into your entities from an DTO.
You are simply throwing state around w/o any safety net, your entities can go into all sorts of invalid states.

e.g. if you receive a “TraingleDTO” with all its angles set to 0 , oh well, just slap those numbers straight into your Triangle entity.. a total sum for all angles of a triangle that equals 0 is valid, right?

When I do domain modelling I want to capture the language and intentions of the domain experts.
If the domain experts say that you can add a product to an order, I want to see that somewhere in my code.
But more on that in the next post.

Well, this is all for now, I hope I managed to explain some of the pitfalls of this architecture.

In my next post I will be explaining why Command pattern and semantics matter.

DDD – Prentity?


This post is actually a question:

Is there any word for an entity/object that not yet have any value to the domain at hand?
e.g. an entity like a document that can be edited until some point in time where it gets processed and the result is then used in some domain action.

Lets say I download a job application in PDF format and save on my computer, I can edit the data in the application over and over and re-save and the state changes/transitions are not interesting to anyone.
The objects state only become interesting once it enters the domain it belongs to.

Another example could be a shopping cart, OK, I know that there might be actual domain logic attached to a shopping cart in many cases.
But if we look at it in real life with a real physical shopping cart, nobody cares if I add or remove items to my cart until I go to check out.

Or maybe a draft email, I can edit and change it in any way I want and it is not until I press “Send” that the contents/state of the object becomes important, even if the object is persistent. (I can re-open my draft and continue so it is persisted)

The reason I got to think about this is because this is actually a case where it will be OK to map from DTO to entity straight of w/o any implications.

Many people (including me) in the DDD community don’t like to map from DTO to entity because you lose semantics and intention.
But I was thinking about this today and in that kind of case, that might be a completely valid approach.
This is a special kind of object that does not need the same design considerations as a “normal” domain entity would have.
That is the key here, it doesn’t have any domain value and therefore the design can be lighter/simpler than we otherwise need/should use.

That is, before it gets any value to the domain at hand. you simply map it from DTO to entity in order to make it persistent and  it doesn’t matter if you lose intention here since it does not yet hold any value to anyone, you just save it so you can continue edit it later.
And at some point in time, pass it into the domain.
(Am I repeating myself now?)

Agree? disagree?

[Edit]
My friend Henric @fat_coder suggested that this might be a command.
Maybe it actually is?

Could it be that a command could be persistent and editable before it is sent of to a command handler?

Interesting thought..
I do like the idea, it could be a ApplyForJobCommand in the job scenario, and a CheckoutCommand in the shopping cart scenario.