Wednesday, May 23, 2012

Running total. Here and now

Sql Server 2012 is upon us, it's 2012 already. But for some people, their here and now is still Sql Server 2008


Read first how not to do(set-based) running total at http://sqlblog.com/blogs/adam_machanic/archive/2006/07/12/running-sums-redux.aspx


Given Sql Server 2008 windowing capability limitation, most would give their new-fangled CTE skills a shot by writing running total query in a recursive manner:

with T AS
(
 select ROW_NUMBER() over(order by OrderID) as rn, * from test
)
,R(Rn, OrderId, Qty, RunningTotal) as
(
 select Rn, OrderID, Qty, Qty
 from t 
 where rn = 1
 
 union all
 
 select t.Rn, t.OrderId, t.Qty, p.RunningTotal + t.Qty
 from t t
 join r p on t.rn = p.rn + 1
 
)
select R.OrderId, R.Qty, R.RunningTotal from r
option(maxrecursion 0)


All is fine and dandy, except when that query is ran on a production database, that query will not scale, that query took a good 9 seconds on a 5,000 rows table. (DDL at the bottom of this post)

Now let's try another approach, let's think for a while we are back in time, say Sql Server 2000. What would your DBA grandpa would do to facilitate that running total report?

create function TestRunningTotal()
returns @ReturnTable table(
    OrderId int, Qty int, RunningTotal int
)
as begin

    insert into @ReturnTable(OrderID, Qty, RunningTotal)
    select OrderID, Qty, 0 from Test
    order by OrderID;

    declare @RunningTotal int = 0;

    update @ReturnTable set 
           RunningTotal = @RunningTotal, 
           @RunningTotal = @RunningTotal + Qty;

    return;
end;

And that query took 0 second.


And go back further in time, say Sql Server 7, what would he do? He would follow Adam Machanic's approach: http://sqlblog.com/blogs/adam_machanic/archive/2006/07/12/running-sums-redux.aspx. Cursor is one of the rarity cases where running total is a very good choice.


The following is every developer's heaven, works on Postgresql 8.4, Sql Server 2012 and Oracle 9(or 8.1.7 ?):

select OrderID, Qty, sum(Qty) over(order by OrderID) as RunningTotal from test

Now back to regular programming(Sql Server 2008).

Some called that kind of update that relies on physical sort a quirky update. If you feel uneasy with quirky update, you might want to put a guard statement to prevent wrong update order.

create function TestRunningTotalGuarded()
returns @ReturnTable table(
    OrderId int, Qty int, RunningTotal int not null, RN int identity(1,1) not null
)
as begin

    insert into @ReturnTable(OrderID, Qty, RunningTotal)
    select OrderID, Qty, 0 from Test
    order by OrderID;
    
    declare @RunningTotal int = 0;
    
    declare @RN_check INT = 0;
    
    update @ReturnTable set 
            @RN_check = @RN_check + 1,
            @RunningTotal = (case when RN = @RN_check then @RunningTotal + Qty else 1/0 end),
            RunningTotal = @RunningTotal;

    return;
end;


If UPDATE really update rows in unpredictable order, the @RN_Check will not be equal to RN(identity order) anymore, the code will raise a divide-by-zero error then.



DDL

create table Test(
 OrderID int primary key,
 Qty int not null
);


declare @i int = 1;

while @i <= 5000 begin
 insert into Test(OrderID, Qty) values (@i * 2,rand() * 10); 
 set @i = @i + 1;
end;


Running total example results:
OrderId     Qty         RunningTotal
----------- ----------- ------------
2           4           4
4           8           12
6           4           16
8           5           21
10          3           24
12          8           32
14          2           34
16          9           43
18          1           44
20          2           46
22          0           46
24          2           48
26          6           54
28          2           56
30          8           64
32          6           70
34          0           70
36          4           74
38          2           76
40          5           81
42          4           85



UPDATE: May 29, 2012

Use cursor, quirky update is well.. quirky. Until further notice, please use something predictable, like cursor. http://www.ienablemuch.com/2012/05/recursive-cte-is-evil-and-cursor-is.html

UPDATE: May 29, 2012 8:27 PM

Just by putting a clustered primary key on the table variable, it makes the updating of rows in order. Check the test(looped 100 times) on the bottom of this article, it has no divide-by-zero(guard statement) error anymore: http://www.ienablemuch.com/2012/05/recursive-cte-is-evil-and-cursor-is.html

Until further notice, I would say quirky update is not really quirky.

Thursday, May 10, 2012

Generics object-orientation. Untyped generic is the key to generic's OOPness

Suppose you have these classes:


class Animal {
}

class Dog : Animal {
}

class Plant {

}


We knew that...
...
{
 // these works:
 MakeSound(new Animal());
 MakeSound(new Dog());
 
 // and this doesn't:
 MakeSound(new Plant());   
}


public static void MakeSound(Animal a) {
}





Then suppose we have this existing code:

public static void AddAnimal(IList<Animal> aList) {
 foreach(Animal a in aList) {
 }
 
 aList.Add(new Animal());
}


And we want that function to be instantly accessible to all Animal's derived type. That is, we want the IList<Dog> be accepted on that function too.


That is not possible, and if that could be possible, it will be dangerous, which we shall discover later on. So this will fail:

IList<Dog> dogs = new List<Dog>();
AddAnimal(dogs);

Produces this compile-time error:


cannot convert `System.Collections.Generic.IList<Dog>' expression to type `System.Collections.Generic.IList<Animal>'


For an AddAnimal to accept other types, we follow this pattern:

public static void AddAnimal<T>(IList<T> aList) where T : new() {
 foreach(Animal a in aList) {
 }
 
 aList.Add(new T());
}  

Using that function, the IList<Dog>'s Dog can be slotted on untyped T, hence the compiler allowing us to pass the dogs of type IList<T> to that function. You need to put new() on function declaration if you intend to create an object out of T. So this will work now:


IList<Dog> dogs = new List<Dog>();
AddAnimal(dogs);

And you could do this as well:


IList<Plant> plants = new List<Plant>();
AddAnimal(plants);

Oops! Any discerning object-oriented programmers worth his salt, could quickly discern that the above-code is not object-oriented, plant did not derive from Animal, AddAnimal should accept Animal only. To do that, simply put a constraint on the accepted types on the generic's parameter. We just put a where T : BaseType where the BaseType here is the Animal class

public static void AddAnimal<T>(IList<T> aList) where T : Animal, new() {
  foreach(Animal a in aList) {
  }
  
  aList.Add(new T());
}  

This will not work anymore:

IList<Plant> plants = new List<Plant>();
AddAnimal(plants);

Its compilation error:
Plant' cannot be used as type parameter `T' in the generic type or method `TestGenCompat.MainClass.AddAnimal<T>(System.Collections.Generic.IList<T>)'. There is no implicit reference conversion from `Plant' to `Animal'


To recap, these should work:

IList<Animal> anims = new List<Animal>();
AddAnimal(anims);

IList<Dog> dogs = new List<Dog>();
AddAnimal(dogs);

Now let's explore again the old code, I mentioned that it's dangerous if it's possible to pass dogs to this method:

public static void AddAnimal(IList<Animal> aList) {
 foreach(Animal a in aList) {
 }
 
 aList.Add(new Animal());
}



What will happen if they allowed passing derived types to that method? Let's simulate if that is allowed in the first place.

public static void AddAnimal<T>(IList<T> xList) where T : Animal, new() {
 IList<Animal> aList = (IList<Animal>) xList;  

 foreach(Animal a in aList) {
 }
 
 aList.Add(new Animal());
}

But alas, C#'s generic carries the type it is genericizing. Though our casting of IList<T> to IList<Animal> is allowed, during runtime it is checked if the passed variable's type signature matches the type we are casting to. So if we pass an instance of IList<Dog>, that would result to casting error during runtime.


So to simulate the inherent danger if a given language allows us to merely use the untyped generic, let's look at other languages, let's choose choose Java.

First we already knew that this is not valid and can be caught during compile-time, same with C# :

List<Dog> dogs = new ArrayList<Dog>();
List<Animal> anims = (List<Animal>)dogs;

Now let's turn to Java's method that is constrained on Animal type. Then we try to cast it:

public static <T extends Animal> void addAnimal(List<T> aList)
  throws InstantiationException, IllegalAccessException
{
 // On Java, not exactly equal generic types can't be caught during runtime.
 // C# can
 List<Animal> list = (List<Animal>) aList;

 for(Animal x : list) {
 }
 
 list.add(new Animal());
}


Now let's iterate the list after we passed it to that function:

{
 List<Dog> dogs = new ArrayList<Dog>();
 addAnimal(dogs);
 addAnimal(dogs);
 System.out.println("See " + dogs.size());

 for(Animal x : dogs ) {
  System.out.println(x);
 }
}

That code prints 2. The problem is in the for loop.

Exception in thread "main" java.lang.ClassCastException: Animal cannot be cast to Dog

Though the content of the dogs collection are two Animals, and is compatible to Animal x. The for loop don't even reach that part(Animal x) of the loop. The mere act of extracting an object from dogs' iterator is actually doing these steps:


Dog d = dogs.get(0); 
Animal x = d; 

The second line is perfectly fine. However, the first line has the problem, or rather the object in the collection is the root cause of the problem, if the Animal was not possible to be added in dogs collections, we will not be receiving any casting exception, as all dogs' elements are Dog.


So while a Dog Is-An Animal:

Dog x = new Dog();
Animal y = x;

An Animal Is-Not-A Dog, hence this would result to casting exception:

Animal a = new Animal(); // think of this as dogs.get(0)
Dog b = a; // casting exception
Animal x = b; // no error

With type erasure, this code:

public static <T extends Animal> void addAnimal(List<T> aList)
  throws InstantiationException, IllegalAccessException
{
 // Not exactly equal generic can't be caught during runtime
 List<Animal> list = (List<Animal>) aList;
}

Is actually compiled to JVM like this:

public static void addAnimal(List aList) {
   List list = aList;
   
   list.add(new Animal());
}


So that's it, in Java it's not entirely feasible during runtime that adding an Animal to a List<Dog> type can be prevented. And the consequence is, when we ultimately needed to unbox the object out of that list to its proper type, it will cause a casting exception. C# generics can prevent that scenario, as its generics carry the type; Java's generics erases the type, its generics merely shift the burden of castings away from the programmer. Behind the scenes(in JVM level), Java generics are untyped objects and are merely cast back when accessing the object.


So there goes the rationale of not allowing OOP on typed generics on function. And it requires type erasure on generic's parameter, of which C# is not designed to be.


To summarize, untyped generics coupled with type constraining (via where T : typehere) is the only way to achieve OOP nirvana on generics

Sunday, April 29, 2012

ASP.NET MVC Editor Templates

EditorTemplates reminds me of college days using assembly language. Copying bytes using MOV and LOOP instruction could get the job done, but not knowing the simpler way(REP MOVSB) to do this solved problem makes your code not as readable or maintainable as it could possibly be. Seeing many loops in code and deciphering their intent is counter-productive.


As much as we want to believe in this axiom "If At First You Don't Succeed, Remove All Evidence You Ever Tried", there's something to be said for knowing how a bad code looks like. With this in mind, this is not what to do in an ASP.NET MVC view:


~/Views/Home/Index.cshtml
@model SoQna.ViewModels.QnaViewModel

@using (Html.BeginForm("SubmitAnswers", "Home"))
{
    int i = 0;
    foreach (SoQna.ViewModels.AnswerToQuestion answer in Model.Answers)
    {
        @: Question #@(answer.ToQuestionId) <br />

        @Html.Hidden("Answers[" + i + "].ToQuestionId", answer.ToQuestionId)
        @Html.DisplayFor("Answers[" + i + "].QuestionText", answer.QuestionText)

        <p />

        @Html.TextArea("Answers[" + i + "].AnswerText", answer.AnswerText)

        <hr />
        
        ++i;
    }

    <input type="submit" value="Done" />
}

Sure that code is very model-centric and capable of being directly usable by our controller code...


// POST /Home/SubmitAnswers

[HttpPost]
public ViewResult SubmitAnswers(SoQna.ViewModels.QnaViewModel a)
{
    foreach (SoQna.ViewModels.AnswerToQuestion answer in a.Answers)
    {
        answer.QuestionText = _qnaRepo.Single(x => x.QuestionId == answer.ToQuestionId).QuestionText;
    }
    return View(a);
}


...but the problem with the ~/Views/Home/Index.cshtml view above is we cannot use strongly-typed model on html helpers. As much as possible, with strongly-typed framework such as ASP.NET MVC, we should not use magic strings in our code. We should let strong typing take over the reins in our ASP.NET MVC app. With this in mind, we shall do this instead on ~/Views/Home/Index.cshtml:

@model SoQna.ViewModels.QnaViewModel

@using (Html.BeginForm("SubmitAnswers", "Home" ))
{    
    @Html.EditorFor(x => x.Answers) 
    <input type="submit" value="Done" />
}

Now you might ask, where's the loop? How does it know how to display the corresponding HTML for our object's properties?

On first question, Html.EditorFor does the loop for us if the property is an IEnumerable one. On second question, that's where we will use the EditorTemplates. When you want to use a pre-defined view(editor template) for a given model or view-model, you place that view in this folder ~/Views/Shared/EditorTemplates, but if you intend your pre-defined view for a given model/view-model be a controller-specific one, placed them in their specific controller folder, e.g. ~/Views/XXX/EditorTemplates where XXX is your controller name. ASP.NET MVC will first look for editor templates specific to your controller; if it cannot find one, it will look in ~/Views/Shared/EditorTemplates folder


To make our first editor template, please create an EditorTemplates folder on ~/Views/Home, given Home is your controller name. Then add an MVC 3 Partial Page (Razor) item to your ~/Views/Home/EditorTemplates folder, you do this by doing a right-click on EditorTemplates and selecting Add > New Item

Name your editor template Razor page after your model type or view-model type.

This is how our ~/Views/Home/EditorTemplates/AnswerToQuestion.cshtml shall look like:

@model SoQna.ViewModels.AnswerToQuestion

Question #@(Model.ToQuestionId) <br />
@Html.HiddenFor(x => x.ToQuestionId)
@Html.DisplayFor(x => x.QuestionText)

<p />

@Html.TextAreaFor(x => x.AnswerText)

<hr />

Here's a sample output:



Download: http://code.google.com/p/aspnet-mvc-demo-editor-templates/downloads/list

SVN: http://code.google.com/p/aspnet-mvc-demo-editor-templates/source/checkout

Friday, April 27, 2012

Multiple dispatch in C#

This code...

using System;


public class MainClass {

    public static void Main() {
        Asset[] xx = { new Asset(), new House(), new Asset(), new House() };

        foreach(Asset x in xx) {
            Foo(x);
        }
    }

        
    public static void Foo(Asset a) {
        Console.WriteLine("Asset");
    }

    public static void Foo(House h) {
        Console.WriteLine("House");
    }

}


public class Asset {
}

public class House : Asset {
}



...outputs:

Asset
Asset
Asset
Asset


If you want an output of Asset, House, Asset, House, i.e. you want to use the overloaded method that matches the object type(House) not by object's reference type(on this expression, Asset is the reference type: foreach(Asset x in xx), we are using Asset as a reference for some of the House object in array of Asset object), there are multiple approach to solve the problem, one is to design your class polymorphism by using virtual and override; another is to amend the Foo(Asset a) code.
Another way is to use dynamic of C# 4, this is covered on later part of this post

Let's try with this one:
public static void Foo(Asset a) {
    if (a.GetType() == typeof(Asset))
        Console.WriteLine("Asset");
    else if (a.GetType() == typeof(House))
        Foo((House) a);
}

That's too much for asking on the developer of Foo(Asset a). What if there's another class that derives Asset? Say Car, that would entail adding another else if (a.GetType() == typeof(Car)), and what's more difficult is you cannot know in advanced what new classes will derive from Asset. And that approach is an antithesis of http://www.antiifcampaign.com/

You can call the runtime-matched method by using reflection for those purposes:

using System;


public class MainClass {

    public static void Main() {
        Asset[] xx = { new Asset(), new House(), new Asset(), new House(), new Car() };

        foreach(Asset x in xx) {
            Foo(x);
        }
    }

        
    public static void Foo(Asset a) {
 
        if (a.GetType() == typeof(Asset))
            Console.WriteLine("Asset");
        else {
            Type t = typeof(MainClass);
            t.InvokeMember("Foo", System.Reflection.BindingFlags.InvokeMethod, null, 
                t, new object[] { a } ); 
        }

    }

    public static void Foo(House h) {
        Console.WriteLine("House");
    }

    public static void Foo(Car c) {
        Console.WriteLine("Car");
    }

}


public class Asset {
}

public class House : Asset {
}

public class Car : Asset {
}


Outputs:
Asset
House
Asset
House
Car


If you are using C# 4, you can use dynamic to avoid resorting to reflection, and code-wise it has less friction(if you don't treat casting to dynamic as friction) on your code, i.e. you don't need to modify the Foo(Asset a), dynamic feature is worth considering if Foo helpers comes in binary form, i.e. no source code.

using System;


public class MainClass {

    public static void Main() {
        Asset[] xx = { new Asset(), new House(), new Asset(), new House(), new Car() };

        foreach(Asset x in xx) {
            Foo((dynamic)x);
        }
    }

        
    public static void Foo(Asset a) {
        Console.WriteLine("Asset");
    }

    public static void Foo(House h) {
        Console.WriteLine("House");
    }

    public static void Foo(Car c) {
        Console.WriteLine("Car");
    }

}


public class Asset {
}

public class House : Asset {
}

public class Car : Asset {
}

Outputs:
Asset
House
Asset
House
Car


Another good thing with multiple dispatch via dynamic as compared to reflection, dynamic multiple dispatch is similar with polymorphism that happens on recommended class polymorphism(i.e. one that uses virtual & override), i.e. if there's no available overloaded static method for a given derived class, dynamic will automatically find a method that best matches the given derived class, and what best matches the derived class than its base class? Nice, isn't it? :-)

So in our code example, if we remove the public static void Foo(Car c) from the code above, the Car type will be resolved by dynamic to invoke Foo Asset overload instead, the output is:

Asset
House
Asset
House
Asset


Whereas if you use static method invocation using reflection, you don't have that kind of luxury, the output for reflection approach when public static void Foo(Car c) is not available:

Asset
House
Asset
House


Notice the absence of 5th output? Your code will just fail silently if you just have the kind of reflection code above.

If you want to use the Foo Asset overload for Car type on absence of Foo Car overload, you have to find a way how reflection can invoke the base type overload given the absence of actual type overload, and perhaps a bit of google-fu or stackoverflow-fu could help you find a solution for this problem. Please advise me if you find out how ツ

Monday, April 23, 2012

OUTER APPLY Walkthrough

Given a task to display the nearest immediate elder brother's birthdate of a given Person, we might come up with subquery.

create table Member
(
 Firstname varchar(20) not null, 
 Lastname varchar(20) not null,
 BirthDate date not null unique
);

insert into Member(Firstname,Lastname,Birthdate) values
('John','Lennon','Oct 9, 1940'),
('Paul','McCartney','June 8, 1942'),
('George','Harrison','February 25, 1943'),
('Ringo','Starr','July 7, 1940');



Assuming that no persons share the same birthdate, this is how we might code it in subquery:


select m.*,
  
 ElderBirthDate = 
  (select top 1 x.BirthDate 
  from Member x 
  where x.BirthDate < m.BirthDate 
  order by x.BirthDate desc)
  
from Member m
order by m.BirthDate;


Now, some nasty users requested for more information, she wanted to see that elder person's Firstname too; as a good developer you are, of course you will comply. Here's your new query:

select m.*,

 ElderBirthDate = 
  (select top 1 x.BirthDate 
  from Member x 
  where x.BirthDate < m.BirthDate 
  order by x.BirthDate desc),
 ElderFirstname = 
  (select top 1 x.Firstname 
  from Member x 
  where x.BirthDate < m.BirthDate 
  order by x.BirthDate desc)
  
from Member m
order by m.BirthDate

Output:
Firstname            Lastname             BirthDate  ElderBirthDate ElderFirstname
-------------------- -------------------- ---------- -------------- --------------------
Ringo                Starr                1940-07-07 NULL           NULL
John                 Lennon               1940-10-09 1940-07-07     Ringo
Paul                 McCartney            1942-06-08 1940-10-09     John
George               Harrison             1943-02-25 1942-06-08     Paul

(4 row(s) affected)


Then a day after, she requested to add the Lastname, she deemed that it is nice to have that information on the report too. Things are getting hairy isn't it? We are violating DRY principle, if we are using subquery approach.


Enter OUTER APPLY, this neat technology is ought to be present in all RDBMS, unfortunately(if you expect that it is available on all RDBMS at the time of this writing) this is available on SQL Server only:

select m.*, elder.*
from Member m
outer apply
(
 select top 1 ElderBirthDate = x.BirthDate
 from Member x 
 where x.BirthDate < m.BirthDate 
 order by x.BirthDate desc
) as elder
order by m.BirthDate

Compared to subquery, at first glance it doesn't seem to add much in terms of value; but where it shines is it can pick up all the columns on the matched condition. Now back to the requested new column on report by our dear user, we can neatly add those column(s) if we are using OUTER APPLY instead:

select m.*, elder.*
from Member m
outer apply
(
 select top 1 ElderBirthDate = x.BirthDate, ElderFirstname = x.Firstname
 from Member x 
 where x.BirthDate < m.BirthDate 
 order by x.BirthDate desc
) as elder
order by m.BirthDate

Firstname            Lastname             BirthDate  ElderBirthDate ElderFirstname
-------------------- -------------------- ---------- -------------- --------------------
Ringo                Starr                1940-07-07 NULL           NULL
John                 Lennon               1940-10-09 1940-07-07     Ringo
Paul                 McCartney            1942-06-08 1940-10-09     John
George               Harrison             1943-02-25 1942-06-08     Paul

(4 row(s) affected)


Not only there is less friction on modifying our query based on user's requests when we uses OUTER APPLY, our OUTER APPLY query scales nicely too:



Now there's a new request in town to display the person's two immediate elder brothers; if we are using subquery, we might cringe at the thought of rewriting our query just to facilitate such whimsical requirement. But hey, we are using OUTER APPLY, you can laugh in triumph rather than quivering in pain, just modify the TOP 1 to TOP 2 to support that requirement. Convenient isn't it?

select m.*, elder.*
from Member m
outer apply
(
 select top 2 ElderBirthDate = x.BirthDate, ElderFirstname = x.Firstname
 from Member x 
 where x.BirthDate < m.BirthDate 
 order by x.BirthDate desc
) as elder
order by m.BirthDate, elder.ElderBirthDate desc


Output:
Firstname            Lastname             BirthDate  ElderBirthDate ElderFirstname
-------------------- -------------------- ---------- -------------- --------------------
Ringo                Starr                1940-07-07 NULL           NULL
John                 Lennon               1940-10-09 1940-07-07     Ringo
Paul                 McCartney            1942-06-08 1940-10-09     John
Paul                 McCartney            1942-06-08 1940-07-07     Ringo
George               Harrison             1943-02-25 1942-06-08     Paul
George               Harrison             1943-02-25 1940-10-09     John

(6 row(s) affected)


Live test: http://www.sqlfiddle.com/#!3/19a63/1