The Joy of JSON

So back to JSON. For those of you that don't know JSON stands for JavaScript Object Notation and is lightweight text based data interchange format. Rather than quote other people verbatim you can find thorough explanations of JSON here:

• Introducing JSON
• JSON in Javascript

As mentioned in my previous post on Ajax I came upon JSON quite by accident and was actually using it for some time without having any idea.

But let's pull back a bit. Let's start with the JavaScript Object Literal. Some years ago I came upon this article by Christan Heilmann about the JavaScript Object Literal which had been published all the way back in 2006: Show love to the JavaScript Object Literal

Now when I read this it was a revelation to me. I hadn't really used JavaScript objects a great deal at this point (yes I am one of those people that started using JavaScript without actually learning the thing) and when I had used them is was through the var obj = new Object() pattern (as that's the only approach I knew).

So it was wonderful to discover that instead of the needlessly verbose:

var myCar = new Object();
myCar.wheels = 4;
myCar.colour = "blue";

I could simply use the much more concise object literal syntax to declare an object instead:

var myCar = { wheels: 4, colour: "blue" };

Lovely. Henceforth I adopted this approach in my code as I'm generally a believer that brevity is best.

It was sometime later that I happened upon JSON (when I started looking into jqGrid). Basically I was looking to pass complex data structures backward and forward to the server and, as far as I knew, there was no way to achieve this simply in JavaScript. I was expecting that I would have to manually serialise and deserialise (yes dammit I will use the English spellings!) objects when ever I wanted to do this sort of thing.

However, I was reading the the fantastic Dave Ward's Encosia blog which on this occasion was talking about the troubles of UpdatePanels (a subject close to my heart by the way) and more interestingly the use of PageMethods in ASP.NET. This is what he said that made me prick up my ears:

"Page methods allow ASP.NET AJAX pages to directly execute a page’s static methods, using JSON (JavaScript Object Notation). JSON is basically a minimalistic version of SOAP, which is perfectly suited for light weight communication between client and server."

JSON is a lightweight SOAP eh? I've used SOAP. I wonder if I could use this....

To my complete surprise, and may I say delight, I discovered that a wonderful fellow called Douglas Crockford, he of JavaScript, The Good Parts fame had quietly come up with JSON some time ago. JSON, from my perspective, turned out to be a simple way to turn an object into a string and then from a string back into an object. So simple that it consists of 2 methods on a JSON object:

• JSON.stringify(myObject) - take an object and make me a JSON string. (and by the way isn't "stringify" just the loveliest method name ever?)
• JSON.parse(myJSONString) - take a JSON string and make me an object

Let me illustrate the above method names using the myCar example from earlier:

var myCar = { wheels: 4, colour: "blue" };
// myCar is an object

var myCarJSON = JSON.stringify(myCar);
//myCarJSON will look like this: '{"wheels":4,"colour":"blue"}'

var anotherCarMadeFromMyJSON = JSON.parse(myCarJSON); 
//anotherCarMadeFromMyJSON will be a brand new "car" object

I've also demonstrated this using the Chrome Console:

(click on the image to see it a little larger)

Crockford initially invented/discovered JSON himself and wrote a little helper library which provided a JSON object to be used by all and sundry. This can be found here: JSON on GitHub

Because JSON was so clearly wonderful, glorious and useful it ended up becoming a part of the EcmaScript 5 spec (in fact it's worth reading the brilliant John Resig's blog post on this). This has lead to JSON being offered natively in browsers for quite some time. However, for those of us (and I am one alas) still supporting IE 6 and the like we still have Crockfords JSON2.js to fall back on.

WCF Transport Windows authentication using NetTcpBinding in an Intranet environment

Update

Since I wrote this initial post I've taken thinks on a bit further. Take a look at this post to see what I mean: http://icanmakethiswork.blogspot.com/2012/03/wcf-moving-from-config-to-code-simple.html
I know I said I'd write about JSON this time. I will get to that but not this time. This time WCF authentication quirks.

I've been working on a project that uses .NET Remoting to have a single central point to which web applications and Windows services can call into. This is used in an intranet environment and all the websites and Windows services were hosted on the same single server along with our .NET Remoting Windows service. (They could quite easily have been on different servers but there was no need in this case.)

It was decided to "embrace the new" by migrating this .NET Remoting project over to WCF. The plan wasn't to do anything revolutionary, just to move from one approach to the other as easily as possible. I found the following useful article on MSDN:

http://msdn.microsoft.com/en-us/library/aa730857%28v=vs.80%29.aspx

This particular article was helpful and following the steps enclosed I was quickly up and running with a basic WCF service hosted in a Windows service. It was at this point I started thinking about security. The existing .NET Remoting approach had no security in place. This wasn't ideal but also probably wasn't the worry you might think. It was hosted in an intranet environment and hence not so exposed to the rigours of the Wild Wild Web. However, since I was looking at WCF I thought it would be a good opportunity to get some basic security in place. This generally pleases auditors. I opted to use Windows Transport authentication as this seemed pretty appropriate for an intranet environment. The idea being that we'd authenticate with Windows for an account in our domain.

After headbutting Windows for some time I managed to get a successful client call going from the website running on my development machine to the (separate) development server that was hosting our WCF Window service using Transport Windows authentication.

However, when deploying the website to the development server I discovered we would experience the following error when the website attempted to call the WCF service (on the same server).

Event Type:            Failure Audit
Event Source:         Security
Event Category:     Logon/Logoff 
Event ID: 537
Date:                       15/02/2012
Time:                       16:32:04
User:                       NT AUTHORITY\SYSTEM
Computer:               MINE999
Description:
Logon Failure:
               Reason:                   An error occurred during logon
               Logon Type:            3
               Logon Process:         ^
               Authentication Package:         NTLM
               Status code:            0xC000006D

Not terribly helpful. At the end of the day it seemed we were suffering from a security "feature" introduced by Microsoft to prevent services calling services on the same box with a fully qualified name. An explanation of this can be found here:

http://developers.de/blogs/damir_dobric/archive/2009/08/28/authentication-problems-by-using-of-ntlm.aspx

Using method 1 in the enclosed link I initially worked round this by amending the registry and rebooting the server:

http://support.microsoft.com/kb/887993

This was not a fantastic solution. Fortunately I subsequently found a better one but since the resources on the web are *ATROCIOUS* on this point I thought I should take the time to note down the full explanation since otherwise it'll be lost in the mists of time. Here we go:

The equivalent security to the previous .NET Remoting solution in WCF was to use this config setting on client and service:

<security mode="None" />

As I've said, this is an intranet environment and so having this "none" security setting in place is made less worrying by the fact that the network itself is secured. But obviously this is not ideal and unlikely to be audit compliant. To use Windows security you need this netTcpBinding config setting on client and service:

<security mode="Transport">
  <transport clientCredentialType="Windows" />
</security>

To call the service with this setting in place you will need to be an authenticated Windows user. (Or at the very least impersonating one - but you knew that.)

NOW FOR THE MOST IMPORTANT BIT.....

The endpoint addresses *must* be "localhost" for both client and service when both are deployed to the same server. If this is not the case then you will suffer from the aforementioned security "feature" which will provide you with unhelpful "the server has rejected the client credentials" messages and *nothing* else.

OK FINISHED - MOVE ALONG NOW... NOTHING MORE TO SEE HERE

With WCF Windows Transport authentication in place you can interrogate the calling user id within the service methods by simply evaluating ServiceSecurityContext.Current.PrimaryIdentity.Name (which will be something like "myDomain\myUserName"). So we you wanted to, we could have a simple step which evaluated if the calling user is on the "approved" / "authorised" list. I'm sure this could be made more sophisticated by using groups etc I guess - though I haven't investigated it further as yet. In fact, I suspect Microsoft may have something even more sophisticated still available for use which I'm unaware of - if anyone knows a simple explanation of this then please do let me know!

In closing, I do think Microsoft could work on providing more helpful error messages than "the server has rejected the client credentials". Going by what I read as I researched this error many people seem to have struggled much as I did before eventually bailing out and ended up chancing it by turning security off in their applications. Clearly it is not desirable to have people so confused by errors that they give up and settle for a less secure solution.

A Potted History of using Ajax (on the Microsoft Stack of Love)

This post originally started out as an explanation of JSON. However as I wrote this quickly got abandoned in favour of writing about how I came to use JSON in the first place - which was through the use of Ajax. Having written a goodly amount I've now decided to move the actual JSON stuff into another post since I think Ajax is probably worth thinking about by itself rather than as an aside.

So let me start at the beginning and explain how I came to use Ajax in the first place (this may take some time so please bear with me). In late 2004 I first started working on a project which I was to remain involved with (on and off) for a very long time indeed. The project was part financial reporting system and part sales incentivisation tool; it was used internally in the investment bank in which I was working. The project had been in existence for a number of years and had a web front end which at that point would been built in a combination of HTML, JavaScript, classic ASP and with a Visual Basic 6.0 back end. One of the reasons I had been brought on to the project was to help ".Net-ify" the thing and migrate it to ASP.NET and C#.

I digress. The interesting thing about this app was that there were actually some quite advanced things being done with it (despite the classic ASP / VB). The users could enter trades into the system which represented actual trades that had been entered into a trading system elsewhere in the organisation. These trades would be assigned a reporting value which would be based on their various attributes. (Stay with me people this will get more interesting I *promise*.)

The calculation of the reporting value was quite an in depth process and needed to be performed server-side. However, the users had decreed that it wasn't acceptable to do a full postback to the server to perform this calculation; they wanted it done "on-the-fly". Now if you asked me at the time I'd have said "can't be done". Fortunately the other people working on the project then weren't nearly so defeatist. Instead they went away and found Microsoft's webservice.htc library. For those of you that don't know this was a JavaScript library that Microsoft came up with to enable the access of Web Services on the client. Given that it was designed to work with IE 5 I suspect it was created between 1999-2001 (but I'm not certain about that).

Now it came as a revelation to me but this was a JavaScript library that talked to our web services through the medium of XML. In short it was my first encounter with anything remotely Ajax-y. It was exciting! However, the possibilities of what we could do didn't actually become apparent to me for some years. It's worth saying that the way we were using webservice.htc was exceedingly simplistic and rather than investigating further I took the limited ways we were using it as indications of the limitations of Ajax and / or webservice.htc. So for a long time I thought the following:

• The only way to pass multiple arguments to a web service was to package up arguments into a single string with delimiters which you could split and unpackage as your first step on the server.
• The only valid return type was a single string. And so if you wanted to return a number of numeric values (as we did) the only way to do this was to package up return values into a very long string with delimiters in and (you guessed it!) split and unpackage as your first step on the client.
• The only thing that you could (or would want to) send back and forth between client and server was XML

So to recap, I'm now aware that it's possible for JavaScript to interact with the server through the use of web services. It's possible, but ugly, not that quick and requires an awful lot of manual serialization / deserialization operations. It's clearly powerful but not much fun at all. And that's where I left it for a number of years. Let's fade to black...

It's now 2007 and Microsoft have released ASP.NET Ajax, the details of which are well explained in this article (which I have only recently discovered). Now I'm always interested in "the new" and so I was naturally interested in this.

Just to be completely upfront about this I should confess that when I first discovered ASP.NET Ajax I didn't clock the power of it at all. Initially I just switched over from using webservice.htc to ASP.NET Ajax. This alone gave us a *massive* performance improvement (I know it was massive since we actually received a "well done" email from our users which is testament to the difference it was making to their experience of the system). But we were still performing our manual serialisation / deserialisation of values on the client and the server. ie. Using Ajax was now much faster but still not too much fun.

Let's jump forward in time again to around 2010 to the point in time when I was discovering jQuery and that JavaScript wasn't actually evil.

It's not unusual for me to play around with "what if" scenarios in my code, just to see what might might be possible. Sometimes I discover things. So it was with JSON. We had a web service in the system that allowed us to look up a counterparty (ie a bank account) with an identifier. Once we looked it up we packaged up the counterparty details (eg name, location etc) into a big long string with delimiters and sent it back to client. One day I decided to change the return type on the web service from a string to the actual counterparty class. So we went from something like this:

[WebService(Namespace = "http://tempuri.org/")]
[WebServiceBinding(ConformsTo = WsiProfiles.BasicProfile1_1)]
[ScriptService]
public class CounterpartyWebService : System.Web.Services.WebService
{
  [WebMethod]
  public string GetCounterparty(string parameters)
  {
    string[] aParameters = parameters.Split("|");
    int counterpartyId = int.Parse(aParameters[0]);
    bool includeLocation = (aParameters[1] == "1");
    Counterparty counterparty = _counterpartyDb
      .GetCounterparty(counterpartyId);

    string returnValue = counterparty.Id + 
                       "|" + counterparty.Name + 
                       (includeLocation
                         ? "|" + counterparty.Location 
                         : "");

    return returnValue;
  }
}

To something like this:

[WebMethod]
public Counterparty GetCounterparty(string parameters)
{
  string[] aParameters = parameters.Split("|");
  int counterpartyId = int.Parse(aParameters[0]);
  bool includeLocation = (aParameters[1] == "1");
  Counterparty counterparty = _counterpartyDb
    .GetCounterparty(counterpartyId);

  return counterparty;
}

I genuinely expected that this was just going to break. It didn't. Suddenly on the client I'm sat there with a full blown object that looks just like the object I had on the server. WHAT BLACK MAGIC COULD THIS BE??????????

Certain that I'd discovered witchcraft I decided to try something else. What would happen if I changed the signature on the method so it received individual parameters and passed my individual parameters to the web service instead of packaging them up into a string? I tried this:

[WebMethod]
public Counterparty GetCounterparty(int counterpartyId,
                                    bool includeLocation)
{
  Counterparty counterparty = _counterpartyDb
    .GetCounterparty(counterpartyId);

  return counterparty;
}

And it worked! IT WORKED!!!!!!!!!!!!!!!!!!!!! (And yes I know I wasn't actually using the includeLocation parameter - but the point was it was being passed to the server and I could have used it if I'd wanted to.) I couldn't believe it. For years I'd been using Ajax and without any idea of the power available to me. The ignorance! The stupidity of the man! To my complete surprise it turned out that:

• Ajax could be quick! ASP.NET Ajax was lightening fast when compared to webservice.htc
• You could send multiple arguments to a web service without all that packaging nonsense
• You could return complex objects without the need for packaging it all up yourself.

Essentially the source of all this goodness was the magic of JSON. I wouldn't really come to comprehend this until I moved away from using the ASP.NET Ajax client libraries in favour of using the jQuery.ajax functionality. (Yes, having mostly rattled on about using webservice.htc and ASP.NET Ajax I should clarify that I have now forsaken both for jQuery as I find it more powerful and more configurable - but it's the journey that counts I guess!)

It's abysmal that I didn't discover the power of Ajax sooner but the difference this discovery made to me was immense. Approaches that I would have dismissed or shied away from previously because of the amount of "plumbing" involved now became easy. This massively contributed to my programmer joy!

Next time I promise I'll aim to actually get onto JSON.