Monitoring MongoDB with Application Insights

At LivingLens we have the somewhat dubious blessing of using MongoDB as our database. Historically we used to use LogEntries for our logging and it worked pretty nicely but these days we’re using Application Insights. One of the great features of application insights is the Application Map which generates a visual representation of your application and its dependencies. In the screenshot below you can see a browser application talking to a server which in turn is talking to a SQL Server database.

That map is auto generated and you can click through into any part to filter and drill down to the details of the data. you can see that the database is identified as an external SQL dependency and  I wanted the same for Mongo.

  1. To log all MongoDB calls to Application Insights
  2. To categorise that logged data as an external database dependency

Fortunately Application Insights provides the TrackDependency method to ‘track the response times and success rates of calls to an external piece of code’. So by extending the logger we’re using I can easily log dependency data.

Even more fortunately the MongoDB driver exposes a ClusterConfigrator which allows you to subscribe to events when you are creating a MongoClient instance. This is a little more involved than simply creating a MongoClient by passing it a connection string but it’s still pretty painless.

And now when I look at the Application Map I can see and drill down into the details of every success and failure event raised by Mongo.

Sweet!

 

Firing up WordPress as a service with a built in database on Azure

This here blog’s been living happily on a VM in the cloud for years with a nother VM hosting MySQL for it. They both required very little maintenance but I wanted them to require less so I figured why not fire up a simple hosted solution in Azure instead. So here goes let’s see how long that takes.

First I logged into the Azure portal, click the New button and search for WordPress which gives me a few choices from different publishers. You can see that I have a few options including hardened version from Cognosys, a Docker container with MySQL included or a Multi-Tier install from the folks at Bitnami.

I’m just going to choose the standard install from WordPress and click then click Create.

This takes me to the form you see above where I can choose the name of my app the resource group (for organising my resources inside Azure) and a choice of Database Provider. Previously from here I’d have to specify a cloud host or an instance I’ve setup separately but I’ve chosen the new MySQL in App (Preview) option. The description for that comes with a pretty heavy caveat:

MySQL In App (Preview) runs a local MySQL instance with your app and shares resources from the App Service plan. Note that apps using MySQL In App are not intended for production environments, and they will not scale beyond a single instance.

But this blog is pretty low traffic and I like to live dangerously so I’m going to try it out and see how it performs. I click Create once more and Azure starts the deploying and since I’m British I make a cup of tea. A couple of minutes later I receive a ‘Deployment succeeded’ and i can see that my shiny new WordPress instance has been deployed.

And when I navigate to it’s URL I hit the famous five-minute WordPress installation process.

 

A quick look at Smart Unit Tests

Looking at the latest Visual Studio 2015 CTP release I noticed the new Smart Unit Tests feature. This is an evolution of Microsoft Research’s Pex project and to quote Microsoft:

Smart Unit Tests explores your .NET code to generate test data and a suite of unit tests

That sounds pretty amazing so I fired VS 2015 to have a look. I grabbed a  piece of sample code that I wrote a while back for an interview, you can find it on github if you want to follow along:

RomanNumeralGenerator generates the roman numeral string equivalent of an integer, it throws an exception if the integer is above 3999 or below 1. To generate the test for this code I just right click inside the class file and select Smart Unit Tests.

SmartUnitTestsScreenshot1

This fires up the Smart Unit Tests Exploration generates a set of tests and lets you view details about them.

SmartUnitTestsScreenshot2

This is actually really neat, tests have been created that trigger the exceptions as well as two tests that confirm correct strings are returned. Technically this class now has 100% code coverage, although really we would need to write a much more in depth test suite to cover all the potential bugs, and if I select one of the tests I can see what test code was generated.

SmartUnitTestsScreenshot3

I can also explore the events that generated the tests, modify how those tests are generated and finally select any or all of the tests and save them.

SmartUnitTestsScreenshot4

This generates a new unit test project, MSTest only at the moment, and adds it to your solution.

My first impressions

This seems like a great way to backport tests onto existing code but I hope that people don’t think that now there’s no longer any need to write tests because Visual Studio will write them for you. Smart Unit Tests are going to be a great tool for refactoring currently untested code, creating baseline tests so you can change with more confidence. I’m looking forward to trying out using them on a more complicated code base next.

Accessing an IIS Express website from another device

IIS Express is great, no need to install IIS and mess around setting up sites it’s there ready for you with your Visual Studio installation. Unfortunately out of the box your IIS Express sites are only accesible from the local machine, fortunately with only a few steps you can easily give external devices access.

For this example I have a web application named Sample.Website configured to run on port 51502 and the IP address of my local machine is 192.168.2.123, you will replace those values with your own name/IP/port combination.

Add a new site binding

First we find the site config in C:\Users\[YourName]\Documents\IISExpress\config\applicationhost.config with the name Sample.Website, it should look something like:

Add a new binding  below the existing binding to localhost:

So now the site config should look something like:

Register the URL

Run the following from an administrative command prompt to allow access to the url:

Open your firewall port

Finally run the following from an administrative command prompt to open your firewall on the necessary port:

Browse the site from another device

That’s it you should now be able to access the site via it’s IP/port combination, happy browsing 🙂

If you’re on a domain you could of course also use your machine name instead of IP address.

Defeating deadlocks with READ_COMMITTED_SNAPSHOT isolation

I was recently asked by a client to look into an issue they were having with a WCF web service. The application was generating a large number of errors, filling a 5MB log file every 5 minutes, and the performance of the underlying database was so bad that a simple query such as:

would take up to a minute and a half to return. Checking the error logs I could see a huge number of exceptions like:

First things first I made sure that the server wasn’t lacking any physical resources but memory was sitting at less than 50% usage, processors less than 10% and disk less than 5%. So I fired up SQL Profiler to let me trace exactly which queries were causing the deadlocks and which queries were being sacrificed. If I didn’t have direct access to the server I could have used Trace Flags to log details of the deadlock for me with trace flag 1222 but fortunately I didn’t have to jump through that extra hoop and I could point  SQL Profiler directly at the server and select the ‘Locks:Deadlock graph’ event:

Trace properties

I could also turn on ‘Lock:Deadlock’ and ‘Lock:Deadlock chain’ but the deadlock graph is great because it gives you visual representation of the deadlock and you can also see the queries causing the deadlocks so I just had to wait for a minute and up popped the following:

Deadlock graph

As you can see the deadlocks were caused by a page lock in this case and I could simply hover my mouse over either transaction (the two ellipses) to see the specific queries that were involved in the deadlock. The victim in this case was a SELECT and the winner an INSERT which were both trying to access the same page of a specific table, each one trying to lock the page because NHibernate was configured to use a ReadComitted IsolationLevel. Now normally that’s exactly the isolation level I would recommend but the problem for this application was that it was doing a large number of read transactions and they were getting blocked by some rather slow write transactions.  This was having a knock on effect on other reads/writes and eventually we ended up with locks all over the place and the deadlocks and horrible performance I started this post with.

OK so at this point we can modify the code to use a less strict isolation level for reads and redeploy it and hooray our issue will be fixed. But there’s a couple of reasons I didn’t want to do that, firstly the web service code was fiendishly complicated and had no test coverage (I should point out at this point that I didn’t write it)  so any changes to the code were inherently risky. Secondly the magic of READ_COMMITTED_SNAPSHOT means that modifying the code is unnecessary. So what is ‘Snapshot Isolation’ I hear you cry, well the MSDN definition is:

Specifies that data read by any statement in a transaction will be the transactionally consistent version of the data that existed at the start of the transaction. The transaction can only recognize data modifications that were committed before the start of the transaction. Data modifications made by other transactions after the start of the current transaction are not visible to statements executing in the current transaction. The effect is as if the statements in a transaction get a snapshot of the committed data as it existed at the start of the transaction.

Now you have probably realised that this isolation level has potential drawbacks, see this question on stackexchange for an excellent summary of them, but in this case the benefits far outweighed any potential risk and the application was capable of handling those risks anyway. By enabling READ_COMMITTED_SNAPSHOT at the database level we can make sure that although the  application’s connection to the database is specifying ReadCommitted IsolationLevel it is achieved using a snapshot rather than a lock, we do that with the following SQL:

N.B. being in single user mode stops the query from being suspended and never completing

After making this one change there has been a dramatic increase in the performance, there are no more deadlock exceptions thrown and simple queries are back to completing in less than a second. Obviously READ_COMMITTED_SNAPSHOT is not a panacea and should be used with caution and with an awareness of it’s possible risks but in the right situation it can be an incredibly powerful tool to have in your belt.

 

Awesome links of the week – Part 2

ASP.NET SignalR

WebSockets let you have bi-directional communication  between the browser and the server. SignalR lets you use WebSockets with ASP.NET and falls back  gracefully if they’re not available as well as providing some other nifty RPC functionality.

http://typesetinthefuture.com/moon/

Taking a second look at free fonts

Smashing magazine are re-evaluating free fonts and seem to have nice things to say about some of them. They’ve also, very helpfully, selected some of the best available and there’s even a few more linked in the comments.

http://www.smashingmagazine.com/2014/03/12/taking-a-second-look-at-free-fonts/

Unreal Engine 4 running in Firefox

Epic and Mozilla have put together a demo video of the upcoming Unreal Engine 4 running a demo inside Firefox with no plugins.

https://blog.mozilla.org/blog/2014/03/12/mozilla-and-epic-preview-unreal-engine-4-running-in-firefox/

WTF, HTML and CSS?

A great insight into some of the voodoo problems you’ll come across working with HTML and CSS clearly explained and with concise solutions.

http://wtfhtmlcss.com/

Awesome links of the week – Part 1

Ionic

Like bootstrap for building HTML 5 apps http://ionicframework.com/ check out the getting started video then dive into the great quality docs.

TypeLITE

Do you want to automatically generate TypeScript Interfaces from your C# classes. So did I and I was going to have to write a T4 template to do it, fortunately someone far more awesome than me has already done just that, open sourced it,  created a NuGet package and put together a great web site.

A re-introduction to JavaScript

A re-introduction to JavaScript (JS Tutorial) from the lovely folks over at the Mozilla Developer Network is the kind of article I wish I’d read when I first started working with JavaScript. It covers everything you need to know if you are already, or are planning to start, programming in JavaScript. Read it, you won’t regret it.

Erik Johansson

Erik Johansson is a photographer who creates rather wonderful Photoshop montages.

 

 

JavaScript Documentation

I use JavaScript on a pretty much daily basis so as I haven’t blogged about anything recently I thought I’d help promote the great (as in comprehensive) JavaScript docs that the Mozilla foundation provide. Right now this is being pushed via the http://promotejs.com domain which will hopefully make it more likely that anyone searching for JS docs will find good quality, in depth docs easily.

The campaign aims to point people to the Mozilla Developer Centre for JavaScript but I’d also always recommend Douglas Crockford’s JavaScript. JavaScript’s a much maligned and misunderstood languages so hopefully between the two you’ll appreciate some of it’s many benefits. Enjoy 🙂

JavaScript JS Documentation: JS Array every, JavaScript Array every, JS Array .every, JavaScript Array .every

Seven essential Visual Studio 2010 keyboard shortcuts

Microsoft recently released some reference posters for the default keybindings in Visual Studio 2010 in pdf format. I’m constantly amazed by how few of these most developers seem to know so I thought I’d list my favourites. I use C# but the majority should work in VB and all but Ctrl + Comma will work in VS 2008.

Ctrl + Full Stop (.)

Displays the available options on the Smart Tag menu. This is by far my favourite, Smart Tags allow you to rename properties/methods/classes throughout your solution, add a required using statement or even create a new class/property/method.

Ctrl + Comma

Displays the NavigateTo window, which gives you search-as-you-type support for files, types and members. Scott Gu has a great blog post on just how useful this is.

F12/Shift F12

F12 will go to the definition of the currently selected symbol. Shift F12 will find all references for the currently selected symbol.

Ctrl + K, C/Ctrl + K, U

Ctrl + K, C comments out all currently selected lines of text or the current line if no text is selected. Ctrl + K, U uncomments all currently selected lines of text or the current line if no text is selected. This works in .js, .cs, .aspx, .config and .xml files.

Ctrl + K, D

Formats the current document according to the indentation and code formatting settings specified on the Formatting pane under Tools | Options | Text Editor | C#. Instantly tidy up a poorly formatted code file!

Ctrl + M, O

Collapses all declarations down to an outline to give you a quick high-level overview of your code file.

Ctrl + M, M

Toggles the currently selected region, method, class or property between collapsed and expanded view.

Enable Service Broker taking forever

Today I had to enable Service Broker in SQL 2008 because when using a SqlCacheDependency I was getting the error:

The SQL Server Service Broker for the current database is not enabled, and as a result query notifications are not supported.  Please enable the Service Broker for this database if you wish to use notifications.

This should be pretty easy just using the following command:

The problem I had was that it was taking forever for this to execute. The reason it turns out is that other processes were stopping the script from acquiring an exclusive lock. The solution is pretty simple, a script to kill all other processes. Be aware this will kill all processes if they’re important that could be a very bad thing!

And Service Broker’s enabled, time for tea.