Debugging at APT - Part 3: Fiddler

Fiddler has been around for a while, but I was only introduced to it a few months ago. Earlier this year, we were able to budget some time toward improving our software's performance, rather than continuing to add new features as quickly as we had been. Fiddler was especially useful for telling us just how much work our web servers (and our clients' browsers) were doing, and where some of our bottlenecks and improvement opportunities were.

In my first post I mentioned that we have some pretty complex pages throughout our software. Before optimizing them, we were making hundreds of HTTP requests per page (including the now-infamous 85 iframes), and we were applying the same super-strict "no cache" rules to our static files that we applied to our dynamically-generated content. As a result, even our simplest pages required well over 100K of HTTP traffic, and some of the most complex pages were closer to 2MB. Fiddler's biggest strength for us was seeing how each of the various caching directives you can send in HTTP headers affects the browser and network traffic.

Coincidentally, right after I learned about Fiddler, I was asked to help troubleshoot a performance concern raised by one of our newer clients. They were sporadically and unpredictably experiencing "This page cannot be displayed (cannot find server or DNS error)" messages. It was inconsistent but seemed to happen primarily during the client's core business hours, and none of our other clients were experiencing comparable symptoms, so we figured it could be a network problem on their end. A combination of Fiddler (on one client user's computer) and Wireshark (on our web server) showed us a pretty clear pattern of dropped and resubmitted requests at the HTTP level. At the same time, our own internal logging showed an extraordinarily high variance in round-trip performance.

It turned out that the client's internal firewall was sporadically dropping some outgoing requests and incoming responses during times of peak traffic on their general-purpose network. The solution was to establish a VPN tunnel, using a separate infrastructure that they already had in place for their vendors and business partners. In hindsight, we might have reached the same conclusion even more quickly if we had run Fiddler on both sides and compared the traces.

Debugging at APT - Part 2: IE Developer Toolbar

As I said last time, we don't officially support Firefox yet because our clients use it so rarely. So we were relieved when Microsoft finally started to catch up by releasing the IE Developer Toolbar. We're often frustrated by an inexplicable error (either an actual JS error, or just ugly/broken HTML/CSS) that vexes us in IE but works perfectly fine in Firefox. For the JavaScript errors, we've had the Visual Studio Script Debugger available to us for a while (once we jumped through the necessary hoops to enable it), but sometimes the problem is that some of our JS code just isn't being loaded or executed properly. (Never mind that Script Debugger often shows JS exceptions in the middle of completely unrelated HTML markup.) Since we get some of our JS from asynchronous AJAX calls, it's something that View Source wouldn't be able to tell us. Once again, "select element by click" (equivalent to "Inspect" in Firebug) has been a great friend.

I also like the syntax-highlighted View Source, although I usually end up using something like Notepad++ to view source from a browser. But even better, Developer Toolbar's View Source includes the list of CSS declarations that apply to the portion of the page I've selected.



As you can see above and below, I have selected one element buried pretty deep within one of our pages, and I can easily use "Element Source with Style" to see not only its entire innerHTML but also all the CSS declarations, where they came from, and all parent tags' IDs and styles.



Next time I'll talk about Fiddler, also owned by Microsoft, which helps us debug "over the wire."

Javascript Basics: The second and third functions you should write and use all the time

After an embarrassingly long delay, I've decided to put two functions I use all the time in this post to get us somewhat back on track. I also plan on including some other posts about our experimentation with the Yahoo! User Interface (YUI) library and the jQuery library I mentioned last time.

But anyways, back to this post's topic, javascript functions you should write for yourself and be using all the time. This weeks functions are actually pretty simple, but I've talked to and helped a lot of people who weren't using them or had written them so that they're not quite as generic as they should be.

2. show()

One of the main uses of javascript is to interact with the document object model (DOM) created by the browser. At the most basic level, the ability to take static HTML and manipulate not only the properties of those HTML elements, but also remove those elements as well as create new elements really gives javascript a tremendous amount of power.

Quick digression: Of course, this isn't the only thing javascript does for us. Javascript also plays a key role in asynchronous javascript and XML (AJAX), which has revolutionized the way a lot of users interact with web pages. Some would probably argue that AJAX is even more important than the ability to create dynamic HTML (DHTML). I think they're both critical features, and which you think is more important probably will depend a lot on the web site or application you're developing.

Regardless of whether manipulating the DOM is the most important feature of javascript, everyone can admit it's extremely powerful. One of the most frequent manipulations that developers use javascript for is showing and hiding HTML elements. On pretty much every web page there is some extra or more detailed information that you might not want to display to the user immediately, but would like to reveal when they click a link or check a checkbox. Showing the user only the information they need to know at any particular time can greatly increase the usability of a site.

Here is a simple function that will automatically display an element based on its ID:

function show(id) {
    $(id).style.display = "";
}

Notice that this function builds off of the $() function from the first post that will get an HTML element based on its ID. It gets a reference to that element, accesses the style attribute of that element, and finally sets the display property of the style attribute to the empty string. At this point you might have a couple questions worth addressing.

First, it's worth pointing out that the style attribute of an element is not a simple value as it is for an element's ID or class. In this case, the style attribute is an object because there are a lot of styles that might apply to a particular element such as the font size, positioning on the page, or background color. All of these style properties could be stored as one big string, but it's pretty easy to see how that would be a nightmare for developers (editing an element's "font-size" wouldn't be very easy).

The other two questions worth answering are why did we choose to change the display property and why did we set it to the empty string? These are both good questions. There are two style properties that control whether an item can be seen or not, display and visibility. The visibility property can have a value of "hidden". It might seem like this would do what we want, but what the visibility property actually does is changes whether an element is visible without removing the space it takes up from the page. This means if a I have a paragraph element whose visibility I set to hidden, I won't be able to see that paragraph element, but I will see a big white space where that element should be. This is rarely what you want, but sometimes it is. Usually what you want is for that element to take up space on the page when it is visible or displayed, but be removed from the flow or layout of the page when it is not being displayed. This is exactly what the display property does. When the display property for an element is set to "none", then that element is completely removed from the flow of the page so there is no unnecessary white space.

So the only question remaining is why did we set the display property to the empty string in order to get our element to appear. If anything, an empty string might indicate that something shouldn't be displayed. In this case, as is the case with a lot of the style properties, it is set to the default value for that property. It just so happens that the default value for the display property is whatever display value for that element will display it properly. This means for block elements like <div> or <p> elements, the display value will be set to "block". For inline elements such as <strong> or <span> elements, it will be "inline". Certain elements, such as elements, even have different display types in different browsers. In Internet Explorer, a <tr> is displayed properly with the value of "block", but needs to be set to "table-row" in Firefox. This is where setting the display value to the empty string and allowing the browser to then apply the default really shines.

So at this point it should be pretty clear why this function was written this way (but if it isn't, let me know in the comments). It's nice to have all of these somewhat complex choices encapsulated in a single easy to understand function that anyone can use, not to mention there's a lot less typing to do.

So what is the other function for this post?

3. hide()

I don't think this one needs much explanation, so here it is:

function hide(id) {
    $(id).style.display = "none";
}

I don't think there's anything I need to say about this function. The only slightly interesting thing to note here is that, unlike displaying elements, if you don't want display elements, setting their display value to "none" always does the trick.

Cool, so that's it for this post. Add these functions to wherever you put the $() function -- hopefully an external javascript file that you include on pages you need javascript on. Then use them in your javascript and see how much shorter and easier to understand your javascript becomes!

Debugging at APT - Part 1: Firebug

Like most developers (using any language or platform), folks at APT crave robust and powerful debugging tools. Fortunately, ColdFusion 8 introduced some fantastic new debugging and monitoring tools, on top of the robust logging and request debugging that has been available for a long time. Unfortunately, our architecture seems to prevent us from using most of those tools. (But we'll have plenty of architecture discussions later.) More generally, our system is not "pure" ColdFusion -- we also use Java directly, as well as SQL, JavaScript, CSS, .NET, VBScript, Ant... you name it -- so we often wonder which debugging tools are best for which jobs.

I recently hosted an internal "Wednesday Noon Session" on debugging at APT. (Every three weeks or so, one of our engineers puts together a presentation that focuses on one aspect of development at APT.) I learned a lot by preparing my session, so I'm going to post a series of articles on the tools we use and how we use them. I'll start today with Firebug.

Firebug

For the client side, as well as some server-side debugging, Firebug is one of our best friends. We don’t officially support Firefox for our clients, because their IT departments pretty much all mandate IE6 and prohibit the installation of any other browser. In fact, 95% of our product logins come from IE6, and almost all of the rest are from IE7. Once in a while we even see someone try to use IE6 on Win2K, which typically is missing the past five years worth of bug fixes and security patches. (Yes, even in mid-2008!)

But we’re working toward full Firefox support for a number of reasons, one of which is debugging capability. We have all sorts of complex pages in our software. If you’re not familiar with them, simply finding an element on the page, let alone debugging a script, can be daunting. Thank goodness for the "Inspect" button.


We make heavy use of the Script tab to evaluate JS expressions on-the-fly and step through functions, as well as the Net tab to see what's going on with our AJAX requests (like sorting, searching and pagination within each component). We also used Yahoo!'s YSlow extension to discover that our menus were rendering upwards of 85 iframe tags on each page. (Ouch! Well, it turns out that all of the iframes were there to work around a well-known IE6 bug. But we've since gotten smarter about the "hidden iframe hack" and we've also switched to using YUI menus, which are also smarter about using iframes and about rendering efficiency in general.)

Next time: Firebug's cousin and competitor, the IE Developer Toolbar.

Javascript Basics: Five functions you should write and use all the time

If you're new to Javascript, you've either already found out or will very shortly that there are some tasks that you need to do all the time. It can get pretty tedious having to remember what the code is for those tasks, or even worse, knowing what the code is, but still having to type it out each time you want to use it.

In this article, I'm going to walk through the first of five functions I use frequently when writing Javascript code that will make not only make writing code easier because the code is more concise, but also because in a lot of cases it's easier to understand. Certainly this series won’t cover all of the shortcut functions you may want to write, but if you're new to Javascript this should get you going so that soon you can write your own. So let's get started:

1. $() or id()

If you've written Javascript for more than 15 minutes, you've probably used the “document.getElementById” function. If you've written Javascript for more than 30 minutes, you've probably made the mistake of capitalizing the last "d" and had a Javascript error thrown. Hopefuly the error message from the browser made it obvious what was wrong, but in some browsers deciphering the error message can be an art in itself. Because it is easy to misspell and requires typing out 24 characters to accomplish one of the most basic and frequent tasks in Javascript, it doesn’t take long to get frustrated with the “document.getElementById” function.

If you've ever used the popular jQuery library, you know you can get a reference to an HTML element by writing:

var referenceToElement = $("elementID");

The folks who wrote jQuery felt your pain in writing out "document.getElementById" each time, and made a shortcut function whose name was just the dollar sign. Because the dollar sign is a valid character for use in a Javascript function name but it's not a character that is often used, they chose to create a function that was only one character long to effectively act as an alias for the "document.getElementById" function.

This was such a great idea, we’ve copied it for ourselves at APT and included it in our own Javascript library by writing the following function: (Note: If you use jQuery already, there’s no need to write this function yourself since you already have it!)

$ = function(id) {
   // If "id" is a string, return the value of "document.getElementById",
   // otherwise, just return whatever was passed in.
   return (typeof id == "string") ? document.getElementById(id) : id;
}

Armed with this code, we can now replace all instances of "document.getElementById" with the much shorter "$".

Another side benefit of using this function that hasn’t been mentioned yet is that every time we get a reference to an element, we've reduced the size of our code by over 93% (well, not exactly, but you get the idea). If you add the space saved across a large Javascript file by making this replacement, the result is usually pretty significant.

You'll also notice from our implementation, that when we call our “$’ function, we’re not just directly calling the “document.getElementById” function. Instead we built a little bit of extra functionality into our “$” function.

Notice that if you pass in an object that is not a string, our function won't throw an error, it will just pass back the same object that was passed in. This can be useful in functions where you want your code to be flexible for whoever is using it so that they can pass in either a string that is the ID of an HTML element or a reference to the HTML element itself. At the top of your code you can pass that object through our “$” function, and rest assured that what is returned is a reference to a HTML element.

function myCoolFunction(eitherAnIdOrAnElementReference) {
    var definitelyAnElementReference = $(eitherAnIdOrAnElementReference);
    // Do something to that element...
}

As a final point, some people don't like using the dollar sign as the function name. You can feel free to name this function whatever you want. For instance, you could name it "id" instead. I prefer using "$" because I can imagine writing code where I name a variable "id" and accidentally overwrite the function I created, which could then lead to all sorts of funky errors. If I use "$", the chances of that happening are greatly reduced.

Alright, so one down. I'll post the next two functions tomorrow which are related to showing and hiding HTML elements.

VBScript Syntax Checking

This post describes how to use VBScript to check the syntax of a given block of VBScript code.

The Motivation:

At APT, one software suite we use to run automated GUI tests of our web-based software is Mercury QuickTest Professional. QuickTest runs tests written in VBScript that click through a prescribed path in our software to make sure we don't throw any errors.

Rather than store hundreds of static .vbs files containing the definition of tests, we store our test definitions in a SQL database and dynamically generate the corresponding .vbs files on demand. We do this for a few reasons:

1. It makes test maintenance easier (you maintain only the structure of the test, not the code's syntax)
2. It enables users who are unfamiliar with VBS to author and maintain tests
3. By introducing separation between the test's definition and execution, we can change how we execute our test without changing our test's definition (for example, migrating test execution to another platform such as Selenium, which can execute different languages)

The Problem:

One of the challenges of dynamically generating code is making sure that what you generate is syntactically valid. Making the assumption that your generated code will be perfect is generally a bad idea, because:

• Your code-generation code may contain errors
• As you modify your test definition model, the code-generation code can fall out of sync, resulting in invalid generated code
• You'll likely want to support user-entered "custom" sections of code, which you can't assume will be syntactically valid

The Solution:

After we generate the VBS code for the requested test, we check its syntax prior to executing it by calling the function defined below:

Public Function checkVBSSyntax(vbsCode)
    ' Save any pre-existing error so we can revert to it
    oldErr = Err
    Err.Number = 0

    On Error Resume Next
    ' Execute the vbs code to see if it throws an error
    ExecuteGlobal vbsCode
    hasError = Err.Number <> 0

    On Error Goto 0

    ' Revert to the pre-existing error
    Err = oldErr

    checkVBSSyntax = (Not hasError)
End Function

What we're doing here is simply executing the questionable code in the global scope to see if it throws an error. If it throws an error, we assert that it must be due to a syntax error.

A couple of notes:

• This function assumes that the questionable code contains only Function and Sub definitions; that is, no code exists outside of a Function or Sub declaration. If it does contain code outside of a Function or Sub declaration, that code will actually be executed (not just checked for its syntax). If you have such code, wrap it in a Public Sub main() ... End Sub statement.
  


• This has the side effect of defining all the Functions / Subs in your questionable code in global scope; that is, after calling this function, you'll be able to call any functions defined in your questionable code, as if you included it as a function library. As a result, you'll want to be careful that Function / Sub names in the questionable code don't collide with Function / Sub names in your syntax checking code.
  

Fault-Tolerant ColdFusion Error Reporting

ColdFusion’s default reporting mechanism for uncaught errors is to dump the error information to the console. This has several disadvantages, two of them being that the console would have to be monitored and whoever is monitoring it is limited to the dumped information. To address this issue, ColdFusion Administrator has a setting for a Site-wide Error Handler. This is a custom template to execute when an error is encountered, and allows for sophisticated post-processing of the error.

At APT, we take advantage of this feature and have a very robust Site-Wide Error Handler. Among other things, it retrieves additional information about the user and setup, retrieves information about the product’s state, diagnoses the error, logs the error to a database, and sends an e-mail to relevant engineering and delivery team members. But there is a catch: if an error is encountered within the Site-wide Error Handler it stops processing and dumps that error to the console. Should this happen, we are right back where we started.

The solution is simple: make sure your Site-wide Error Handler never fails. However, if you have a complicated template, this is easier said than done. In this situation, we must make the Site-wide Error Handler fault-tolerant.

How do we achieve fault-tolerance? The foundation is liberal use of try-catch blocks within your Site-wide Error Handler. While it may at first appear sloppy and clutter up the code, it is critical for isolating the effects of errors, making sure the maximum amount of processing takes place, and ensuring the most information possible makes it back to you.

Look at this simple example template:

<cftry>
    Retrieve user information
    <cfcatch> Handle exception </cfcatch>
</cftry>
<cftry>
    Retrieve information about the product’s state
    <cfcatch> Handle exception </cfcatch>
</cftry>
<cftry>
    Send an e-mail with error and additional
     retrieved information
    <cfcatch> Handle exception </cfcatch>
</cftry>

If retrieving user information fails, we will still be sent an e-mail containing the error information and information about the product’s state. This is exactly the isolation we are going for. Subcomponents within in the try-catch blocks can be wrapped in their own try-catch blocks as well, creating an even more granular and fault-tolerant template.

When errors with your Site-wide Error Hander are caught, be sure you attempt to report information on them instead of letting them get swallowed up silently. In the above example, the information about the error encountered while retrieving user information should be included in the final e-mail sent. This feedback is critical in debugging and improving the Site-wide Error Handler—you will be glad you have it. However, be sure to wrap this reporting in its own try-catch in case there is an error with reporting the error. The overall theme is: the more paranoid the better!

By treating your Site-wide Error Handler as a series of granular tasks contained within try-catch blocks, the effect of failures will be limited while the maximum amount of error information safely makes its way back to you. This will be speed up the time it takes to identify, diagnose, and fix bugs and thus improve the quality of your software. <cftry> and <cfcatch> tags are essentially free, but engineering time is not.