Amazon EC2: Automatic EBS volume snapshots using C#

Amazon EC2 lets you create snapshots of the EBS volumes using the Amazon Console or the EC2 Java-based command-line tools. The Console works fine until you have to manage a decent amount of volumes or need frequent snapshots. As for the Java-based command-line tools, well, they are Java-based...

Another alternative is to use the .NET API so you can customize what volumes are backed up and how often, the retention criteria, and anything you want while using your favorite programming language. You have to start by downloading the AWS dll from Amazon. Click on Download the DLL and Samples.

The following program takes a snapshot of all the volumes that are in use in your account. The snapshot name matches the name of the source volume:

using System;
using System.Configuration;
using System.Collections.Generic;
using System.Net;
using System.Linq;
using Amazon;
using Amazon.EC2;
using Amazon.EC2.Model;

namespace AWSBackup {
    class Program {
        static void Main (string [] args)
        {
            // The following line is not needed in .NET.
            // In Mono, it is not needed if you trust the AWS certificate chain
            // See http://stackoverflow.com/questions/3285489/mono-problems-with-cert-and-mozroots
            ServicePointManager.ServerCertificateValidationCallback = (a, b, c, d) => { return true; };

            var volumes = GetVolumes ();
            foreach (var v in volumes) {
                if (String.IsNullOrEmpty (v.VolumeId) || String.CompareOrdinal (v.Status, "in-use") != 0)
                    continue;
                var name = v.Tag.FirstOrDefault (t => t.Key == "Name").Value;
                var snapshot = CreateSnapshot (v, name);
                Console.WriteLine ("Snapshot of {0} initiated at {1}", name, snapshot.StartTime);
            }
        }

        static IEnumerable<Volume> GetVolumes ()
        {
            AmazonEC2 ec2 = GetEC2Client ();
            var request = new DescribeVolumesRequest ();
            var response = ec2.DescribeVolumes (request);
            return response.DescribeVolumesResult.Volume;
        }

        static Snapshot CreateSnapshot (Volume v, string name)
        {
            AmazonEC2 ec2 = GetEC2Client ();
            var request = new CreateSnapshotRequest ()
                            .WithVolumeId (v.VolumeId)
                            .WithDescription (name);
            var response = ec2.CreateSnapshot (request);
            var snapshot = response.CreateSnapshotResult.Snapshot;

            if (!String.IsNullOrEmpty (name)) {
                ec2.CreateTags (new CreateTagsRequest ()
                                .WithResourceId (snapshot.SnapshotId)
                                .WithTag (new Tag { Key = "Name", Value = name })
                );
            }
            return snapshot;
        }

        static AmazonEC2 GetEC2Client ()
        {
            var settings = ConfigurationManager.AppSettings;
            return AWSClientFactory.CreateAmazonEC2Client (settings ["AWSAccessKey"], settings ["AWSSecretKey"]);
        }
    }
}

...And that's it! Instead of a shell script that executes several programs per volume, we have a single program that runs on both MS.NET and Mono. Add it to your crontab or the Windows Task Scheduler and you get automatic backups for your volumes. Automatic deletion of old backups is left as an exercise for the reader.

C# source, application configuration file, and Makefile can be downloaded from here: ec2-backup.zip.

Introducing Mono.Dns

The internal implementation of System.Net.Dns in Mono ends up calling getaddrinfo () or some other similar function in libc. This is perfectly fine for most applications. However, sophisticated applications that perform more than just a few DNS queries, like crawlers, aggregators, and log processors, are taking a big performance hit when running under Mono. One of the issues is that the libc implementation is not optimized to be used from multiple threads. Barring an external library dependency, there is not much that can be done to address that problem. Another issue is that the implementation of the asynchronous methods in System.Net.Dns are based on asynchronous delegates invoking the synchronous version of the methods. In other words, calling, for instance, Dns.BeginGetHostAddresses () will use a threadpool thread to block on the call to Dns.GetHostAddresses (). Mono.Dns is here to address these problems and more.

The Mono.Dns assembly provides a set of classes that can help you implement a DNS client or server. There is also an implementation of a simple asynchronous DNS resolver with an API inspired by the System.Net.Socket.Socket.*Async() set of methods. The source code is available in the Mono.Dns GitHub repository. There is an example of use in test/resolver.cs.

As an example of the performance advantage, grab the plainolddns.cs file, compile it, and download this file. Make sure you pull the latest Mono from the master branch and run the program like this:

$ mono plainolddns.exe $(cat google-domains.txt)
...A BUNCH OF NAMES and IP ADDRESSES
...
00:00:18.4342605

And then run it again, but this time set the MONO_DNS environment variable:

$ MONO_DNS=1 mono plainolddns.exe $(cat google-domains.txt)
...THE SAME BUNCH OF NAMES and IP ADDRESSES
...
00:00:00.6138950

Yep, about 30 times faster!

A partial copy of Mono.Dns is now integrated in the master branch of Mono and will be enabled when you set the environment variable MONO_DNS. By default, the old code will be used for DNS resolution and, once the variable is set, the managed resolver implementation will be used for any internal name resolution operation (WebRequest,...).

Caveat: if you are using /etc/nsswitch.conf, setting the variable is not an option for you.

ASP.NET Razor in Mono

After Marek Habersack's work on getting ASP.NET MVC 3 working on Mono, we just needed a couple of fixes to have Razor in Mono.

This is how the photo gallery sample I got from WebMatrix looks in Mono:

pager

You will need Mono master/a2ed4f6 if you want to try Razor out. Publish your Razor web site and remove Microsoft.Web.Infrastructure.dll from the bin directory, Mono will load its own from the GAC.

Update: You can get a zip file with the sample from here. Unzip it, 'cd webdeploy', run 'xsp4 --applications /test:.' and browse http://127.0.0.1:8080/test/nodb.cshtml.

Exceptions in thread pool threads

When .NET 2.0 came out back in January 2006 there was a significant change in how the .NET runtime treats exception in thread pool threads: they terminate the process if they are not handled. There are a couple of exceptions to the rule, the most common one being Thread.Abort() called on the thread.

Until now Mono was not compatible in this regard and instead we behaved like the .NET 1.x runtime, i.e., exceptions in thread pool threads were silently ignored. Finally, as some folks already running Mono from master have noticed, Mono behaves like .NET 2.0. The first release incorporating this change will be Mono 2.8.

Code samples

  1. An exception in a thread pool thread causes this code to terminate the process when executed

         ThreadPool.QueueUserWorkItem ((a) => { throw new Exception ("From the threadpoool"); });
  2. The program is also terminated if the callback of an asynchronous delegate invocation throws.

         void Method ()
         {
              WaitCallback wcb = new WaitCallback ((a) => {
                   Console.WriteLine ("Here");
              });
              wcb.BeginInvoke (wcb, OnCBFinished, null);
              ...
         }

         static void OnCBFinished (object arg)
         {
              throw new Exception ("Whatever");
         }

  3. Surprise! If the BeginInvoke() call does not have a callback, the exception is ignored as in the good old 1.x days

         WaitCallback wcb = new WaitCallback ((a) => { 
              throw new Exception ("From the threadpoool");
         });     
         wcb.BeginInvoke (wcb, nullnull);

    Update: yes, if EndInvoke() is called, the exception will be raised in the caller thread. The surprise is the difference between this case and (2), since EndInvoke() is not called in (2).

Effects of this change to the Mono runtime

We have fixed the classes in Mono that were not handling exceptions thrown in the thread pool. If you find one that we've missed, please, let me know.

If your application uses the thread pool explicitly or through Delegate.BeginInvoke(), make sure you are handling the exceptions that might happen there. Look for occurences of (1) and (2) above, but ignore (3). If there are too many to count, you can start by applying the workaround described later or by adding an unhandled exception handler to the root application domain:

        static int Main ()
        {
                AppDomain.CurrentDomain.UnhandledException += OnUnhandledException;
                ...
                return 0;
        }

        static void OnUnhandledException (object sender, UnhandledExceptionEventArgs e)
        {
                if (Thread.CurrentThread.IsThreadPoolThread)
                        return// Ignore exceptions in thread pool threads
                Console.Error.WriteLine (e.ExceptionObject);
                Environment.Exit (1);
        }

Workaround: legacy compatibility

If you are an end user stuck with an old application (or a lazy developer), adding the following legacyUnhandledExceptionPolicy to your application configuration file will make exceptions in the thread pool behave like in .NET 1.x:

     <configuration>
          <runtime>
               <legacyUnhandledExceptionPolicy enabled="1"/>
          </runtime>
     </configuration>

How to get a Samsung Captivate with AT&T for $0 or $25

  1. Go to Samsung.com.
  2. Hover the mouse over Mobile at the top of the screen and click Cell Phones
  3. Then select AT&T cell phones and click on the Samsung Captivate. On the next page click on the samsung captivate again.
  4. Open the Where to Buy options (top right) and select Find Online Retailers. This will take you to the letstalk.com Samsung store
  5. Select the type of plan and then Add To Cart (you might have to Add To Cart twice).
  6. If it's a new contract or new line, the price will be $0. If it's an upgrade, the price will be $50. If you get the $50 price, enter the following coupon code before checking out: 25rafpcver1 and you will get $25 off.

And that's all!

Note that if you're upgrading a family plan, your phone number has to be the main number for the account, otherwise, go to Amazon Wireless and you can get it for $175.

Microbenchmark: Mono's new generational GC vs Boehm GC

The Mono Generational GC (a.k.a. SGEN) has been in the works for quite some time now. Given the no-hassle way to use it these days, I decided to compare the performance of both garbage collectors.

The test itself is very simple. I used xsp listening on port 8080 and hit empty.ashx, which is an IHttpHandler that serves a 0-length web page. In both cases, Boehm and SGEN, I warmed up the server by hitting it once and then ran the following command:

ab -c 20 -n 100000 -k http://127.0.0.1:8080/1.1/handlers/empty.ashx

This will fetch 100k empty pages with 20 concurrent requests over keep-alive connections. The results at the end of each run follows:

Boehm SGEN
Time (s) 29.194 24.687
Memory in use (MB, RSS) 80 61
Requests per second 3425.36 4050.70
Average CPU use (%) 82 93

Over 600 req/s more and using 19MB less memory! Amazing! Kudos to the guys doing all the work, specially Mark.

Gnome-keyring-sharp 1.0.2 released

There is a new gnome-keyring-sharp available for download.

The only changes from 1.0.1 are related to a breaking change in gnome-keyring: the socket interface is no longer there as of GNOME 2.30+.

Thanks a lot to Chris Halse Rogers at Canonical for doing all the work.

msnbot/2.0b and robots.txt

Looks like the new MSN spider is having some trouble following the rules. While they fix it, this can be blocked in Apache by doing:
 SetEnvIfNoCase user-agent "^msnbot/2.0b" banned_by_now
 ...
 <Directory "/srv/www/htdocs/">
    Order allow,deny
    Allow from all
    Deny from env=banned_by_now
 </Directory>

Precompiled ASP.NET web sites in Mono

I just added support for running precompiled non-updatable ASP.NET web sites in mono. The benefit from this, apart from the well-known faster initial request and not shipping source code, is that now you will be able to run you ASP.NET applications written in VB.NET in mono without crossing your fingers and expecting our VB.NET compiler to not give you any problem.

The support for precompiled web sites will be in the upcoming Mono 2.4 release and is already in SVN HEAD for those of you not afraid to compile Mono from sources.

Given a web project, you can generate the precompiled version either form Visual Studio or from the command line, like:

c:\test\>aspnet_compiler -f -p c:\test\BlogEngine.Web -v /blog c:\test\output

This command will precompile BlogEngine.net and leave all the generated files in c:\test\output. With the above commands, there are no source files in the output folder at all. Now, to run this on Mono, all you need to do is copy the output folder to your linux machine and configure Apache/mod_mono to serve that folder at /blog/. If you just want to do a quick test, assuming you copied c:\test\output to /tmp/output:

$ cd /tmp/output
$ xsp2 --applications /blog:.

...and then go to http://localhost:8080/blog/ with your browser.

For this application in particular, the time taken to display the first page is about 2s on my laptop, while the non-precompiled site takes about 18s.

Not all browsers are created equal

Somebody reported that the nice little pager that I posted a few days ago didn't work properly on IE if you use the zoom-in/zoom-out feature (hold the Ctrl key and move the mouse wheel). Parts of the text are hidden behind the input textbox. Probably a bug in IE...?

I couldn't figure out how to make it work in IE with CSS, so the new code uses a table and different cells for each part of the pager. The updated code is here.

.NET library to access the iPhone filesystem

Peter Bartok has started the project manzana, which provides a .NET library to access the iPhone filesystem. There's also a Windows Forms program that lets you browse, copy,...

Nice GridView pager without images

The predefined options for the GridView , which can be set through the PagerSettings.Mode property are:

  • NextPrevious
  • NextPreviousFirstLast
  • Numeric
  • NumericFirstLast
Additionally, you can set the text (FirstPageText, NextPageText,...) or image URL (FirstPageImageUrl, NextPageImageUrl, ...) to be displayed for each of those LinkButtons to be shown in the pager.

I wanted to avoid using images and tried to use the Unicode characters for 'left-pointing pointer' (\u25c4 ◄) and 'right-pointing pointer' (\u25ba ►). Also, to make it nicer, I wanted to have a TextBox in which the user can type the page he wants to go to

In the end, I decided to create a class implementing ITemplate, set the GridView's PagerTemplate to this class and add the controls I needed manually. The final result looks like this:

pager
Pager with no images.

To use this ITemplate in a GridView, you can do something like:

     protected void Page_Init (object sender, EventArgs e) {
          myGridView.PagerTemplate = new FancyPagerTemplate (myGridView);
     }

...and you're all set. The events generated by a change in the TextBox or a click on the LinkButtons and the TextBox width will be taken care of.

You can download the source code for the ITemplate from FancyPagerTemplate.cs. Feel free to customize it to your needs.

New job: Venafi, Inc.

It's been a long time since I started working for Ximian and the Mono project back in March of 2002.

I remember how a few months after starting, Miguel asked me if I knew anything about ASP.NET and I said "I don't even know what that is". He wanted me to start working on what later became XSP, the ASP.NET server. Once the server was working on Windows under the MS runtime, my job was to make it work on Mono, which involved a lot of work in System.Web, System.Configuration, the threadpool...

At that time only Dick, Paolo, Dietmar, Miguel and I were payed to work on Mono. Later on Lluis, Jackson, Martin, Sebastien, Atsushi, Duncan, Ben, Mike, Raja, Toshok, Peter, Jordi, Ravi,.. The best team EVER! Not only that, but also a whole bunch of good hackers spread all over the world were contributing (and still do) to the Mono project. They all have made the last 5 years of my life a wonderful experience.

In September I received an offer from Venafi, Inc. and I started working for them a couple of weeks ago.

I am going to miss the Mono project and all the people involved in it. I wish them and Novell the best.

Fire in a building nearby the office

I was having lunch in a pub across the street and the place was completely packed. While eating, I overheard a guy saying that there was lot of black thick smoke and they were told to evacuate and go home. Walking back to the office I saw a lot of police cars and fire trucks. The Longfellow bridge and part of Broadway street were closed to the traffic.

You can read more on the incident at boston.com.

Reporter
Reporting in front of the affected building

If any of you is without sin...

Three surprises today. One is that I found a post in slashdot that reveals interesting information with regards to all the patent FUD spread lately, the other one is that Nat wrote it. And last, but not least:

Another point I want to make. Open Source Risk Management is a company that makes its money by selling insurance on Linux IP infringement. So if you're worried that Linux infringes someone's IP, you buy their products. Two years ago OSRM went off and funded a study by Dan Ravicher -- whose PubPat is in my view a great organization -- that looked at Linux to determine whether it actually violates anyone's software patents. Then in August of 2004 (a few months after Bruce Perens joined their board, I might add), OSRM published a study stating that Linux infringes 283 patents, 27 from Microsoft. You can read about it here: http://news.com.com/Group+Linux+potentially+infringes+283+patents/2100-7344_3-5291403.html

And yet he was one of the first to throw the stone.

BTW, he recently stepped down from the board of directors of OSRM.

Update: the link http://slashdot.org/comments.pl?sid=208652&cid=17014168

This is a personal web page. Things said here do not represent the position of my employer.