Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

An Agile Developer's workflow when SCRUM is used

If you are planning to start SCRUM at your company, you might need to train developers and QA to get into the mindset of an Agile developer. SCRUM is only successful when the developers and QA get into the habit of following the principles of SCRUM by heart. So, sometimes you need to offer training or do trial sprints to give some room to your developers how to get used to the working fashion of SCRUM. Giving them a handy workflow diagram that shows how they should work helps soothe the steep learning curve required for non-super star developers. I made such a workflow while I was teaching SCRUM at my friend's company. The following diagram was printed and hung over the desk of each and every developer to help them grasp the culture of SCRUM quickly:

We use Flyspray for issue tracking, so you will see the mention of it frequently.

You will see the step to "Update Sprint backlog with remaining hours" is missing. This is done kinda verbally and scrum master (sometimes same person who is the product owner) keeps track of it.

Hope you find this useful.

ASP.NET website Continuous Integration+Deployment using CruiseControl.NET, Subversion, MSBuild and Robocopy

You can setup continuous integration and automated deployment for your web application using CruiseControl.NET, Subversion, MSBuild and Robocopy. I will show you how you can automatically build the entire solution, email build report to developers and QA, deploy latest code in IIS all using CruiseControl.NET every N minutes.

First get the following:

• CruiseControl.NET
• Subversion (install the command line tools and add the Subversion bin path to PATH environment variable)
• Robocopy (Windows Vista/2008 has it built-in, here's the link for Windows 2003)
• Install .NET Framework. You need it for MSBuild.

You will learn how I have configured Continuous Integration and Deployment for my open source AJAX Portal project www.Dropthings.com. The code is hosted at CodePlex. When some developer makes a commit, CruiseControl downloads the latest code, builds the entire solution, emails build report and then deploys the latest web site to IIS 6.0.

After installing CruiseControl.NET, go to Programs -> Cruise Control -> CruiseControl.NET Config.

Now keep copying and pasting the following XML blocks and make sure you understand each block and make necessary changes:

   1: <cruisecontrol>

   2:     <project name="Dropthings" queue="DropthingsQueue" queuePriority="1">

   3:         <!-- 

   4:         Path to the trunk folder where the full solution starts from. This is where

   5:         subversion checkout and incremental update is performed 

   6:         -->

   7:         <workingDirectory>d:\cc\dropthings\code\trunk\</workingDirectory>

   8:         <!-- Some path where CCNet writes its logs and stuffs. It can be outside the log folder -->

   9:         <artifactDirectory>d:\cc\dropthings\artifact\</artifactDirectory>

  10:         <category>Dropthings</category>

  11:         <!-- CCNet installs a web dashboard. Enter the URL of that dashboard here -->

  12:         <webURL>http://localhost/ccnet/</webURL>

  13:         <modificationDelaySeconds>60</modificationDelaySeconds>

  14:         <labeller type="defaultlabeller">

  15:             <prefix>0.1.</prefix>

  16:             <incrementOnFailure>true</incrementOnFailure>

  17:             <labelFormat>000</labelFormat>

  18:         </labeller>

  19:         <state type="state" directory="State" />

First change the working directory. It needs to be the path of the folder where you will have the solution downloaded. I generally create folder structure like this:

• D:\CC - Root for all CC.NET enabled projects
  • \ProjectName - Root project folder
    • \Code - Code folder where code is downloaded from subversion
    • \Artifact - CC.NET generates a lot of stuff. All goes here.

Next comes the Subversion integration block:

   1: <sourcecontrol type="svn">

   2:     <!-- Subversion trunk repository to keep checking for latest code -->

   3:     <trunkUrl>http://localhost:8081/tfs02.codeplex.com/dropthings/trunk</trunkUrl>

   4:     <workingDirectory></workingDirectory>

   5:     <username>***** SUBVERSION USER NAME *****</username>

   6:     <password>***** SUBVERSION PATH *****</password>

   7: </sourcecontrol>

Here specify the subversion location where you want to download code to the working folder. You should download the entire solution because you will be building the entire solution using MSBuild soon.

I left <workingDirectory> empty. This means whatever is specified earlier in the <workingDirectory> is used. Otherwise you can put some relative folder path here or any absolute folder.

Now we start building the tasks that CC.NET executes - Build, Email, and Deploy.

   1: <tasks>

   2:     <artifactcleanup   cleanUpMethod="KeepLastXBuilds"   cleanUpValue="5" />

   3:     <modificationWriter>

   4:         <filename>mods.xml</filename>

   5:         <path></path>

   6:     </modificationWriter>

   7:  

   8:     <!-- MSBuild task to build a .msbuild file that basically builds a .sln file -->

   9:     <msbuild>

  10:         <executable>C:\windows\Microsoft.NET\Framework64\v3.5\MSBuild.exe</executable>

  11:         <workingDirectory></workingDirectory>

  12:         <projectFile>Dropthings.msbuild</projectFile>

  13:         <targets>Build</targets>

  14:         <timeout>300</timeout>

  15:         <logger>C:\Program Files (x86)\CruiseControl.NET\server\ThoughtWorks.CruiseControl.MsBuild.dll</logger>

  16:     </msbuild>

This block first says, keep artifacts for last 5 build and remove olders. Artifacts are like build reports, logs etc. You can increase the value for longer history.

Then the most important <msbuild> task. The executable path is to the MSBuild.exe. I am using .NET 3.5 Framework 64bit edition. You might have .NET 2.0 and 32bit version. So, set the right path here for the MSbuild.exe.

<projectFile> maps to a MSBuild file. It's a skeleton MSBuild file which basically says build this Visual Studio solution file. Here's how the msbuild file looks like:

<Project DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

  <Target Name="Build"> 

    <!-- Rebuild entire solution -->

    <MSBuild Projects="Dropthings.sln" Targets="Rebuild" />

  </Target>

</Project>

The Dropthings.msbuild and Dropthings.sln file exists in the same trunk folder. This file says - build Dropthings.sln and do a rebuild.

Now you got the build done. Next is to deploy it. You will be using robocopy to copy files from the code folder to a destination folder which is mapped in IIS to a website. Robocopy will do a synchronization of the directories. It will add new files, overwrite old files and removes files from destination folder which no longer exists in the source folder.

Before you can deploy, you need to stop the website or restart IIS. Otherwise some files may be in use and you will not be able to delete or overwrite the files. Here's how to stop IIS using the iisreset command line tool:

<!-- 

Stop IIS before copying over the latest web project files so that there's no write lock and IIS does not

start restarting the site half way through

-->

<exec>

    <executable>iisreset</executable>

    <buildArgs>/stop</buildArgs>

</exec>

If you do not want to stop the entire IIS, instead just stop a website and recycle an application pool, you can use the iisweb.vbs script for stopping a website and iisapp.vbs script for recycling application pool. Here's an example:

<exec>

    <executable>iisweb</executable>

    <buildArgs>/stop "Dropthings"</buildArgs>

</exec>

<exec>

    <executable>iisapp</executable>

    <buildArgs> /a "Dropthings" /r</buildArgs>

</exec>

You need to first register cscript as the default script runtime. In order to do this, go to command line and enter iisweb. It will tell you that it cannot use wscript to run this script and it needs to make cscript default. Let it make cscript as default.

Now the time to do the deployment of latest web site files. The following task launches robocopy to do the deployment:

<!--

Sync the web project folder with the deployment folder. The deployment folder is where IIS

is mapped to serve the site. The deployment folder is at the buildArgs node. The robocopy 

utility does an exact sync, adding new files, updating old files, deleting files that no longer

exist.

-->

<exec>

    <!--<executable>C:\Program Files (x86)\Windows Resource Kits\Tools\robocopy.exe</executable>-->

    <executable>robocopy.exe</executable>

    <baseDirectory>Dropthings\</baseDirectory>

    <buildArgs>.\ d:\cc\Dropthings\Deploy *.* /E /XA:H /PURGE /XO /XD ".svn" /NDL /NC /NS /NP</buildArgs>

    <buildTimeoutSeconds>60</buildTimeoutSeconds>

    <successExitCodes>1,0</successExitCodes>

</exec>

First you need to correct the robocopy.exe path. For Windows Vista/Windows 2008, keep it as it is. For Windows 2003, you need to specify the full path. You also need to remove the (x86) from the path if you have 32bit OS.

Next is the <baseDirectory>. This is relative to the working directory. It's the path of the website folder. Dropthings website folder is located under the Dropthings folder under trunk. So, I have specified Dropthings\ as the subfolder where the website files are located. You need to specify your project's website folder's relative path here form the <workingDirectory>.

Next change the path in the <buildArgs> node. First one is the source ".\" which you keep as it is. It means copy files from the baseDirectory. Next is the absolute path to the deployment folder where the web site is mapped in IIS. You can use both relative or absolute path here. While using relative path, just keep in mind the robocopy is running from the <workingDirectory>\<baseDirectory> folder.

After the path keep the *.* and the remaining flags intact. The flags mean:

• Copy all subdirectories /E
• Copy hidded files /XA:H
• Do not copy old files /XO
• Exclude .svn directory while copying files /XD ".svn"
• Do not show list of files and directories being copie /NDL, /NC, /NP

After the deployment, you need to turn IIS back on or start the website that you stopped:

<!-- Turn IIS back on -->

<exec>

    <executable>iisreset</executable>

    <buildArgs>/start</buildArgs>

</exec>

<!--

<exec>

    <executable>iisweb</executable>

    <buildArgs>/start "Dropthings"</buildArgs>

</exec>            

-->

Now we got the build and deployment done. Next is to email a nice report to developers and QA. If build succeeds, email both developers and QA so that they can check out the latest build. But if build fails, email only developers.

<publishers>

    <rss/>

    <xmllogger />

    <statistics />

    <!-- Email build report to development and QA team -->

    <email from="admin@yourcompany.com" mailhost="localhost" mailport="25" includeDetails="TRUE"

           mailhostUsername="" mailhostPassword="" useSSL="FALSE">

        <users>

            <user name="Developer1" group="devs" address="dev1@yourcompany.com"/>

            <user name="Developer2" group="devs" address="dev2@yourcompany.com"/>

            <user name="Developer3" group="devs" address="dev3@yourcompany.com"/>

            <user name="QA1" group="qa" address="qa1@yourcompany.com"/>

            <user name="QA2" group="qa" address="qa2@yourcompany.com"/>

            <user name="QA3" group="qa" address="qa3@yourcompany.com"/>                    

        </users>

        <groups>

            <group name="devs" notification="Always"/>

            <group name="qa" notification="Success"/>

        </groups>

        <converters>

            <!--<regexConverter find="$" replace="@dropthings.com" />-->

        </converters>

        <modifierNotificationTypes>

            <NotificationType>Always</NotificationType>

        </modifierNotificationTypes>

    </email>

    <modificationHistory  onlyLogWhenChangesFound="true" />

</publishers>

First you need to change the <email> tab where you specify the from address, mail server name, and optionally a user account for the email address that you need to use to send out emails.

Then edit the <users> node and put your developers and QA.

That's it! You got the configuration file done. Next step is to launch the CruiseControl.NET from Programs -> CruiseControl.NET -> CruiseControl.NET. It will launch a process that will execute the tasks according to the configuration. On Windows Vista, you will have to run it with Administrative privilege.

There's also a Windows Service that gets installed. It's named CruiseControl.NET. You can start the service as well on a server and go to sleep. It will do continuous integration and automated deployment for you.

There's also a web based Dashboard that you can use to force a build or stop a build or see detail build reports.

You can create multiple projects. You can have one project to build trunk code only, but do no deployment. Then you can create another project to build, deploy some branch that's ready for production. You can create another project to build and deploy on QA server and so on.

Here's the full configuration file that you can use as your baseline.

Using multiple broadband connections without using any special router or software

I have two broadband connections. One cheap connection, which I mostly use for browsing and downloading. Another very expensive connection that I use for voice chat, remote desktop connection etc. Now, using these two connections at the same time required two computers before. But I figured out a way to use both connections at the same time using the same computer. Here's how:

Connect the cheap internet connection that is used mostly for non-critical purpose like downloading, browsing to a wireless router.

Connect the expensive connection that is used for network latency sensitive work like Voice Conference, Remote Desktop directly via LAN.

When you want to establish a critical connection like starting voice conf app (Skype) or remote desktop client, momentarily disconnect the wireless. This will make your LAN connection the only available internet. So, all the new connections will be established over the LAN. Now you can start Skype and initiate a voice conference or use Remote Desktop client and connect to a computer. The connection will be established over LAN.

Now turn on wireless. Wireless will now become the first preference for Windows to go to internet. So, now you can start Outlook, browser etc and they will be using the wireless internet connection. During this time, Skype and Terminal Client is still connected over the LAN connection. As they use persisted connection, they keep using the LAN connection and do not switch to the wireless.

This way you get to use two broadband connections simultaneously.

Here you see I have data transfer going on through two different connection. The bottom one is the LAN which is maintaining a continuous voice data stream. The upper one is the wireless connection that sometimes consumes bandwidth when I browse.

Using Sysinternal's TCPView, I can see some connection is going through LAN and some through Belkin router. The selected ones - the terminal client and the MSN Messenger is using LAN where the Internet Explorer and Outlook is working over Wireless connection.

Best practices for creating websites in IIS 6.0

Every time I create an IIS website, I do some steps, which I consider as best practice for creating any IIS website for better performance, maintainability, and scalability. Here' re the things I do:

Create a separate application pool for each web application

I always create separate app pool for each web app because I can select different schedule for app pool recycle. Some heavy traffic websites have long recycle schedule where low traffic websites have short recycle schedule to save memory. Moreover, I can choose different number of processes served by the app pool. Applications that are made for web garden mode can benefit from multiple process where applications that use in-process session, in memory cache needs to have single process serving the app pool. Hosting all my application under the DefaultAppPool does not give me the flexibility to control these per site.

The more app pool you create, the more ASP.NET threads you make available to your application. Each w3wp.exe has it's own thread pool. So, if some application is congesting particular w3wp.exe process, other applications can run happily on their separate w3wp.exe instance, running under separate app pool. Each app pool hosts its own w3wp.exe instance.

So, my rule of thumb: Always create new app pool for new web applications and name the app pool based on the site's domain name or some internal name that makes sense. For example, if you are creating a new website alzabir.com, name the app pool alzabir.com to easily identify it.

Another best practice: Disable the DefaultAppPool so that you don't mistakenly keep adding sites to DefaultAppPool.

First you create a new application pool. Then you create a new Website or Virtual Directory, go to Properties -> Home Directory tab -> Select the new app pool.

Customize Website properties for performance, scalability and maintainability

First you map the right host headers to your website. In order to do this, go to WebSite tab and click on "Advanced" button. Add mapping for both domain.com and www.domain.com. Most of the time, people forget to map the domain.com. Thus many visitors skip typing the www prefix and get no page served.

Next turn on some log entries:

These are very handy for analysis. If you want to measure your bandwidth consumption for specific sites, you need the Bytes Sent. If you want to measure the execution time of different pages and find out the slow running pages, you need Time Taken. If you want to measure unique and returning visitors, you need the Cookie. If you need to know who is sending you most traffic - search engines or some websites, you need the Referer. Once these entries are turned on, you can use variety of Log Analysis tools to do the analysis. For example, open source AWStats.

But if you are using Google Analytics or something else, you should have these turned off, especially the Cookie and Referer because they take quite some space on the log. If you are using ASP.NET Forms Authentication, the gigantic cookie coming with every request will produce gigabytes of logs per week if you have a medium traffic website.

This is kinda no brainer. I add Default.aspx as the default content page so that, when visitors hit the site without any .aspx page name, e.g. alzabir.com, they get the default.aspx served.

Things I do here:

• Turn on Content Expiration. This makes static files remain in browser cache for 30 days and browser serves the files from its own cache instead of hitting the server. As a result, when your users revisit, they don't download all the static files like images, javascripts, css files again and again. This one setting significantly improves your site's performance.
• Remove the X-Powered-By: ASP.NET header. You really don't need it unless you want to attach Visual Studio Remote Debugger to your IIS. Otherwise, it's just sending 21 bytes on every response.
• Add "From" header and set the server name. I do this on each webserver and specify different names on each box. It's handy to see from which servers requests are being served. When you are trying to troubleshoot load balancing issues, it comes handy to see if a particular server is sending requests.

I set the 404 handler to some ASPX so that I can show some custom error message. There's a 404.aspx which shows some nice friendly message and suggests some other pages that user can visit. However, another reason to use this custom mapping is to serve extensionless URL from IIS. Read this blog post for details.

Make sure to set ASP.NET 2.0 for your ASP.NET 2.0, 3.0 and 3.5 websites.

Finally, you must, I repeat you "MUST" turn on IIS 6.0 gzip compression. This turns on the Volkswagen V8 engine that is built into IIS to make your site screaming fast.

Create ASP.NET MVC Controllers under Namespace and specific URL

When you have a lot of controllers, you need to organize them under Namespaces. Also, it is better to put controllers under subfolders in order to organize them properly and have some meaningful URL for them like /Admin/User where Admin is the subfolder and User is the controller. For example, you might have a lot of controllers that are APIs exposed by your web app, not regular pages. So, you might want to put them under /API/ folder. You also want to make sure no one can access those controllers from the root url. For example, no one must call /User/GetUserList instead they must call /API/User/GetUserList

ASP.NET MVC default routing and Controller Factory is very greedy, it ignores the subfolders inside the "Controllers" folder. There's a DefaultControllerFactory class in ASP.NET MVC which traverses all controllers under the "Controller" folder and creates a cache using just the class name as the key. So, it ignores any namespace or any subfolder where you have put the controller. So, I created a derivative of the default controller factory and created a new factory that checks if the requested controller belongs to any specific namespace and whether that controller must be inside a specific subfolder. You can map /Admin folder to respond to all controllers that are under then YourWebApp.Admin namespace. Similarly, you can map /API folder to respond to all controllers that are under the YourWebApp.API namespace. None of these controllers can be accessed outside the specified URL. Here's the factory that does it:

This controller checks the namespace of the controller being returned by ASP.NET MVC's default implementation and ensures the requested URL is the right url where controllers from the namespace can be accessed. So, this means if the controller matches MvcWebAPI.API.UserController, it ensures the URL being requested must be /MvcWebAPI/API/User. It will return null if the URL was something else like /MvcWebAPI/User or /MvcWebAPI/SomeotherFolder/User.

Here's how you use it form Global.asax:

You create a mapping for a Namespace and the subfolder it must belong to. Then you register the new Controller Factory as the default controller factory.

Now the second catch is, the default Route for the {controller}/{action}/{id} won't work for you. You need to create a specific router that starts with the subfolder name. For example, API/{controller}/{action}

Here' two things to notice:

• The use of PathStartWith routing constraint, which I will explain soon
• The last parameter which tells the route to include the API namespace for this route. Otherwise it can't find the controllers in the API namespace

So, the PathStartsWith routing constraint ensures this route gets hit only when the requested URL is under the /API/ folder. For any other URL, it returns false and thus the routing handler skips this route.

It just does a comparison on the AbsolutePath of the current request URL to ensure the URL starts with the specified match.

Similarly, we need to tell the Default route to ignore all paths with /API. Here's how to do it: