Formatting source code and other text for blogger

The biggest nemesis of this blog is that I regularly include everything from source code to log files in here, which really do not fit well into Blogger without some help. Today I got fed up with this enough to look for better ways than what I had been doing.

My HTML skills are still mired in cutting-edge 1995 design, I lost touch somewhere around CSS, so my earlier blog entries used this bit of HTML to insert text I didn't want the blogger formatting to touch as the quickest hack I found that worked:

<div style="padding: 4px; overflow: auto; width: 400px; height: 100px; font-size: 12px; text-align: left;"><pre>
Some text goes here
</pre></div>

That looks the way things formatted that way will look, except with only the inner scroll bar, and getting that posted turned quite self-referential.

Two things were painful about this. The first is that I had to include this boilerplate formatting stuff every time, which required lots of cut and paste. The second is that I had to manually adjust the height every time, and the heights didn't match between the preview and the actual post. I think I did that on purpose at one point, so that I could display a long bit of source code without having to show the whole thing. In general, this is a bad idea though, and you instead want to use "width: 100%" and leave out the height altogether.

What are the other options? Well, you could turn that formatting into a proper "pre" style entry which cuts down on the work there considerably, and is much easier to update across the whole blog. Then you just wrap things with the pre/code combo and you're off, which is a bit easier to deal with. There's an example of this at Blogger Source Code Formatter that even includes a GreaseMonkey script to help automate wrapping the text with what you need. Another example of adjusting there is at How to show HTML/java codes in blogger.

You probably want to save a copy of everything before you tinker and track your changes; the instructions at Can I edit the HTML of my blog's layout? covers this. I put my template into a software version control tool so I can track change I make and merge them into future templates; I'm kind of paranoid though so don't presume you have to do that. I settled on the "Simple II" theme from Jason Sutter as being the one most amenable as a base for a programming oriented blog, as it provides the most horizontal space for writing wide lines. I'd suggest considering a switch to that one before you customize your template, then tweak from there.

The main problem left to consider here, particularly when pasting source code, is that you need to escape HTML characters. I found two examples of "web services" that do that for you, including producing a useful header, that are minimally useful. I like the UI and output of Format My Source Code For Blogging better than Source Code Formatter for Blogger, but both are completely usable, and the latter includes the notion that you might want to limit the height on long samples. I think in most cases you'd want to combine using one of them with the approach of saving the style information into your template advocated by the GreaseMonkey-based site, just using the code and its wrapper from these tools in a typical case rather than using a one-off style every time. If you do that, you can just wrap things in a simple pre/code entries and possibly use something as simple as Quick Escape just to fix the worst things to be concerned about.

Here's what I got from the simpler tool I mentioned first:

<pre style="font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; color: #000000; background-color: #eee;font-size: 12px;border: 1px dashed #999999;line-height: 14px;padding: 5px; overflow: auto; width: 100%"><code>
Some text goes here
</code></pre>

That's a bit more reasonable to work with, looks better (I favor simple over fancy but like something to make the code stand apart), and it easy to dump into my template for easy use (and changes) in the future.

After considering all the samples available, here's the config I ended up dumping into my own Blogger HTML template, after switching themes. This goes right before "]]></b:skin>" in the template:

pre
{
font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace;
background:#efefef;
color: #000000;
font-size:100%;
line-height: 100%;
overflow: auto; 
width: 100%;
padding: 5px;
border: 1px solid #999999;
}

code
{
color: #000000;
font-size:100%;
text-align:left;
margin:0;
padding:0;
}

That's a bit better to my eye, the dashes looked bad. Code is easier to follow too.

Now, what if you want real syntax highlighting for source code? Here the industrial strength solution is SyntaxHighlighter. There's a decent intro to using that approach at Getting code formatting with syntax highlighting to work on blogger. The one part I'm not comfortable with there is linking directly to the style sheets and Javascript code to the trunk of the SyntaxHighlighter repo. That's asking for your page to break when the destination moves (which has already happened) or someone checks a bad change into trunk. And that's not even considering the security nightmare if someone hostile takes over that location (less likely when it was on Google Code, I'm not quite as confident in the ability of
alexgorbatchev.com to avoid hijacking). You really should try to find a place you have better control over to host known stable copies of that code at instead.

I may publish a more polished version of what I end up settling on at some point, wanted to document what I found initially before I forgot the details.

Module API documentation in Python

Sometimes I fondly reminisce about the days when all of the code I worked on was in one programming language. Nowadays, it's a mix of C (mainly related to the PostgreSQL code base), Java (my employer's middleware and lot of my personal code), and Python (systems programming, general utilities, and QA test code). Python is the most recent of those to be added to the mix, and it's proven to have its own unique code documentation challenges, some of which have clarified how to deal with the other languages in the process.

First I should label my expectations here. I'm not a big fan of dynamic typing to begin with, and I'd at least like to document what type each parameter all of the code I intend to be reusable expects, even if those restrictions aren't enforced at compile time. Both C and Java require specifying types for every parameter, and Java includes its Javadoc mechanism for labeling the parameters with their intended purpose and function. That's all I really want: feed in a bit of source code that includes some markup for what all the parameters mean, along with general text commentary; get HTML/PDF output that documents the API presented by that code.

One thing I've found very disappointing about Python is that that its development community seems to actively reject the idea of good parameter documentation directly in the source code. The closest thing I've seen is the PEP for Function Annotations, which are so barebones I wouldn't consider them a help even if they were more mainstream (they're not yet). All we really get for in-code documentation are the Docstring Conventions and pydoc, which don't provide any standard way to label parameters in a way more complicated browsing or analysis tools can utilize.

The first tool I considered for this purpose is Epydoc. This understands Javadoc formatted docstring and ReST, which are two standards I already code documentation using. This includes its own somewhat odd variable docstring syntax, which I didn't find very useful. A similar tool that knows much more about subclassing is pydoctor, whose introduction mentions a bunch of other projects in this area neither I nor them were impressed by.

Another Python specific tool here is pythondoc. My first problem with that project are that it seems kind of dead. Ultimately, my bigger concern is that I'd like to use Python docstrings as much as possible, just with additional markup inside them. pythondoc seems to prefer # formatted comments which aren't really acceptable here.

I keep circling back to Javadoc markup as the only reasonable one here. Ultimately, if I'm using Javadoc format, with nothing Python specific, I have to ask myself why I should adopt a one-off tool such as Epydoc, if instead I can get one that supports the other languages I use and provides a wider feature set. To see the perils of that approach, check out the train wreck answer to the FAQ how to print Javadoc to PDF. What a disaster. To work around that Javadoc limitation, I'd already started moving toward using Doxygen, which I know works great on the C code I browse most via the PostgreSQL code base. (Arguing the merits of doxygen vs. javadoc just in a Java context is a popular topic; see Javadoc or Doxygen? and Doxygen Versus Javadoc for two examples)

A quick check of the full Comparison of documentation generators page didn't give other tools that looked like they would help here. At this point I started to settle on a tentative approach that would unify my work with one tool to use: doxygen + Javadoc formatted parameters in a docstring I could live with. One problem: if you use the Python standard docstring approach, doxygen's Python support won't allow any special commands in there. That's pretty much useless.

Luckily I'm not the first person to make that leap: doxypy is a filter that takes regular Python code with the usual docstring format in, producing an intermediate file in the format doxygen wants to work with. But where's the examples of how it works to get people started?

Luckily, like all good software the authors eat their own dogfood, and the filter itself is a Python program documented so that doxypy can process it. Here's a simple example of a method call from inside it:

def makeTransition(self, input):
   """ Makes a transition based on the given input.

   @param    input    input to parse by the FSM
   """

In this case FSM means "finite-state machine" and not my deity of choice.

Something this simple was all I was looking for, and the only open point here is that Javadoc format presumes one can divine the type from the declaration; that's not so clear here.

Following symlinks in Python

Today's Python trivia question: you have the path of a symbolic link. How do you get the full destination that link points to? If your answer is "use os.readlink", well it's not quite that easy. I'm not alone in finding the docs here confusing when they say: "the result may be either an absolute or relative pathname" and then only tell you how to interpret the result if it's relative. This guy wonders the same thing I did, which is how to know whether the returned value is a relative or absolute path?

I found a clue as to the way to handle both cases in the PathModule code, which is that you use os.path.isabs on the result to figure out what you got back. That module is a lot of baggage to pull in if you just want to correct this one issue, though, so here's a simpler function that knows how to handle both cases:

def readlinkabs(l):
    """
    Return an absolute path for the destination 
    of a symlink
    """
    assert (os.path.islink(l))
    p = os.readlink(l)
    if os.path.isabs(p):
        return p
    return os.path.join(os.path.dirname(l), p)

I hope my search engine cred bubbles me up so someone else trying to look this up like I did doesn't have to bother reinventing this particular tiny wheel.

Upgrading Flex from source RPM to compile PostgreSQL from CVS

This week I'm working on reviewing a patch that's part of the current PostgreSQL CommitFest, the periodic points where all outstanding patches are looked at and committed if ready. The patch I'm looking at requires some performance testing, and all my servers I'd do that on are running CentOS 5, the popular RedHat Enterprise clone. There's a fun surprise waiting for anyone else who tries this: as of last week, you can't build PostgreSQL from the development CVS or GIT repositories anymore on that platform without updating the Flex package from the default one.

The problem started when I ran "make" to build everything and saw this:

make[3]: Entering directory `/home/gsmith/pgproject/pgsql.tpgbench/src/backend/bootstrap'
***
ERROR: `flex' is missing on your system. It is needed to create the
file `bootscanner.c'. You can either get flex from a GNU mirror site
or download an official distribution of PostgreSQL, which contains
pre-packaged flex output.
***

I knew I had flex on my system so this was kind of confusing. Usually there's more detail about dependency failures in the config.log file, and sure enough it had the details:

configure:6793: WARNING:
*** The installed version of Flex, /usr/bin/lex, is too old to use with PostgreSQL.
*** Flex version 2.5.31 or later is required, but this is /usr/bin/lex version 2.5.4.
configure:6807: result: no
configure:6815: WARNING:
*** Without Flex you will not be able to build PostgreSQL from CVS nor
*** change any of the scanner definition files.  You can obtain Flex from
*** a GNU mirror site.  (If you are using the official distribution of
*** PostgreSQL then you do not need to worry about this because the Flex
*** output is pre-generated.)

A quick check shows this change was made to allow some more powerful scanning capabilities in the PostgreSQL language parser, followed by inserting that warning.

Now, while it's possible to just build quickly from source and install directly over top of the existing flex in this case, I don't like to do that on production systems (or even development ones). It's not a good idea to mix the RPM packages on the system with stuff installed that way. It's better if you can create a new flex RPM package based on newer source code and upgrade to that one, then everything stays managed consistently with RPM.

You can locate SRPMs with a newer version of flex, the one released with Fedora 9, from the following locations:

• kernel.org


• stanford.edu


• usc.edu

Those were the first I found with a new enough version number that they should work. You could easily try to substitute the flex SRPM that comes with Fedora 10 or Fedora 11 instead, haven't tested that here myself yet to say how that goes; probably fine.

I used the kernel.org link, downloaded and installed like this:

$ wget ftp://mirrors.kernel.org/fedora/releases/9/Fedora/source/SRPMS/flex-2.5.35-1.fc9.src.rpm
$ sudo rpm -i flex-2.5.35-1.fc9.src.rpm 

This dumps the source into /usr/src/redhat:

$ cd /usr/src/redhat/
$ ls SOURCES SPECS
SOURCES:
flex-2.5.35.tar.bz2

SPECS:
flex.spec

You can then build like this:

$ cd /usr/src/redhat/SPECS/
$ sudo rpmbuild -bb flex.spec 

That produces the new RPM we want:

$ ls -l /usr/src/redhat/RPMS/x86_64/
-rw-r--r-- 1 root root 322619 Jul 22 17:14 flex-2.5.35-1.x86_64.rpm
-rw-r--r-- 1 root root 283675 Jul 22 17:14 flex-debuginfo-2.5.35-1.x86_64.rpm

Now install it:

$ cd /usr/src/redhat/RPMS/x86_64/
$ ls
flex-2.5.35-1.x86_64.rpm  flex-debuginfo-2.5.35-1.x86_64.rpm
$ rpm -qa flex
flex-2.5.4a-41.fc6
$ sudo rpm -Uvh flex-2.5.35-1.x86_64.rpm 
Preparing...                ########################################### [100%]
   1:flex                   ########################################### [100%]
$ rpm -qa flex
flex-2.5.35-1

And then cleanup:

$ cd /usr/src/redhat/SPECS/
$ sudo rpmbuild --clean --rmsource flex.spec
Executing(--clean): /bin/sh -e /var/tmp/rpm-tmp.99970
+ umask 022
+ cd /usr/src/redhat/BUILD
+ rm -rf flex-2.5.35
+ exit 0

After that, I had to run "configure" again to pick up the change, ran "make again", and now my build against CVS checkout of the future PostgreSQL 8.5 in progress compiles without any problems.

Note that normally, when you substitute a package like this you need to be careful you keep up with security patches to it because you're not going to get them automatically anymore. Since flex is a pretty low-level tool used only for developing software, I'm not too concerned about the security implication of my running a custom version here.

Python logging TypeError messages

I've been writing small Python programs for about two years now. There are a few things that slipped my notice until really recently though, and having good application logging instead of using "print" is on that list. Lost some time today figuring out that "global" goes inside functions where that global is used, rather than as a modifier when creating the variable; Understanding 'global' in Python cleared that up for me. While trying to sort that out, I started converting my program's mess of print statements into something that used the logging facility instead. There's an unexpected and quite messy surprise waiting there, a simple to make mistake that the logging facility completely botches handling in a friendly way.

Let's say we start with a simple program that logs the value of a variable at a point:

#!/usr/bin/env python
x=1
print "x =",x

We decide to switch that over to use logging instead, so that we can adjust whether that bit of detail is logged or not better. You might do that (wrongly!) like this:

#!/usr/bin/env python
import logging
x=1
logging.warning("x =",x)

I'd guess this is a pretty common amateur mistake. If you pass more than one argument into a logging statement, the module presumes the first one is a format string and the rest fill in parameters in that string. This gives you the following cryptic error when it uses the "%" facility to format that string with the variable given:

Traceback (most recent call last):
File "/usr/lib/python2.6/logging/__init__.py", line 760, in emit
msg = self.format(record)
File "/usr/lib/python2.6/logging/__init__.py", line 644, in format
return fmt.format(record)
File "/usr/lib/python2.6/logging/__init__.py", line 432, in format
record.message = record.getMessage()
File "/usr/lib/python2.6/logging/__init__.py", line 302, in getMessage
msg = msg % self.args
TypeError: not all arguments converted during string formatting

OK, so that's my bad. But notice what's missing from there? There's not a hint of an idea where in my code the error is at! All of the error messages refer to things within the logging __init__.py module. What to do?

This topic came up recently on the Python list, with a couple of messages about how to hack up the logging module to present more information. That works, but isn't particularly elegant. Buried in there was a clever way around this problem though: override the error handler. Here's what that looks like patched into our bad code example:

#!/usr/bin/env python
import logging

def handleError(self, record):
  raise
logging.Handler.handleError = handleError

x=1
logging.warning("x =",x)

Now when this is run, the error traceback you get starts by showing you the bad line in your code:

Traceback (most recent call last):
File "./logtest.py", line 9, in 
logging.warning("x =",x)
File "/usr/lib/python2.6/logging/__init__.py", line 1441, in warning
root.warning(*((msg,)+args), **kwargs)
...

Note that just inserting an error handler like this isn't what you want to deploy, because an error in logging should introduce an application error. It's really more for making sure you get all your log messages correct in the first place. Make sure you read and following through understanding the comments about how handleError works later in the message thread before leaving this error dumping code in your app permanently.

I broke this down into a trivial case for demonstration, in my actual problem I had 600 lines of code filled with logging calls and only one of them was bad. Patching the error handler took me right to the bad one, and it was obvious how to fix it once I was staring at it.

Here's a correct snippet of code below, including some initialization bits I borrowed from the excellent Manage concurrent threads tutorial that makes the log messages more appropriate for threads. The reason I needed logging and global variables here were to cope with some program-wide thread locking issues with a single shared resource, and something like this is what's actually going into my app now that the log messages are all formatted properly and I could punt the error handler hack out of the program. But you can bet I'll remember that for the next time I run into the cryptic logging TypeError.

#!/usr/bin/env python
import logging
logging.basicConfig(
  level=logging.DEBUG,
  format='(%(threadName)-10s) %(message)s')          
x=1
logging.warning("x = %s",x)

Note that there are all sorts of additional format strings like "%(threadname)" available, including timestamps and line numbers, and making changes to the base format lets you adjust that for the whole program. If you find yourself starting to put function names into your log messages or similarly repetitive work, you should consider using those substitution formatters instead.

Bottom-up PostgreSQL benchmarking and PGCon2009

Last week I got a lot of positive feedback from my PGCon presentation in Ottawa about how to benchmark systems at a low-level when the intended application is to run a database. There were three main topics I was trying to cover in that:

1. Why you should always run your own hardware benchmarks on every piece of hardware you can
   
2. Examples of the simplest benchmarks I've found to be accurate
3. How do organize your tests and your vendor interactions to support performance measurement as a purchasing requirement

There was one slide missing from the set I presented. I've uploaded a version of the slides that fixes that (along with a typo in the sysbench seeks slide) to my home page. For those who missed it, a couple of people have put their notes from the talk as part of PGCon coverage on the PG wiki, and video of many talks from the conference is already available from FOSSLC.

Also available on my web page now is a presentation I did last month at PG East 2009. Titled "Using and Abusing pgbench", that talk also has 3 things it tries to convey:

1. How does pgbench and its internal scripting language work? (Most people aren't even aware there is such a scripting language available)
   
2. What should you do in order to get good results from the built-in pgbench tests?
3. How can you use pgbench as a test harness for writing your own tests?

The hardware benchmarking presentation ends where the pgbench ones starts, with a bit of overlap. That's intentional--I always consider pgbench tests to be something you should do only after confirming all of your hardware does the right thing, top to bottom. A perfect example just came out recently: even someone who's done as much benchmarking work as Joshua Drake can end up measuring the wrong thing, because he skipped the step I suggest for confirming expected commit rate before moving onto higher-level pgbench tests. Since not many people saw the pgbench talk at PG East I'm hoping to repeat that one in the near future to a larger audience.

As part of putting that presentation together, I did more work on a toolchain I've been using for a couple of years now (since I was working on 8.3 development) I've named pgbench-tools. The current 0.4 release posted to my home page is the first to benefit from having some users, which has gotten me an enormous amount of feedback toward making the program bug-free and more usable. Thanks in particular to Robert Treat and Jignesh Shah for their contributions. I think it's finally mature enough that it might be useful for others who want to automate running large numbers of pgbench tests too.

Documentation is still minimal, but I have written some (and what's there is accurate, both of which put me ahead of a lot of open-source projects I guess). There is an into README in the tar file and the presentation tries to give some examples of usage too. When I get more time I'm putting the source code into the PostgreSQL git repository (the repo is already there, I just haven't pushed to it yet), where it will be easier for other people to work with and on. There's a growing need in the PG community for regression testing of performance results, and at the yearly PGCon Developer Meeting I volunteered to see if an improved version of this pgbench-tools package might be useful in that role. I hope the ideas in my presentations and the suggested practice demonstrated by these tools turns out to be helpful to others.

The approach taken in pgbench-tools, that you should parse results from pgbench, save them to a database, and then graph the lot of them using SQL to summarize as needed, is only partially mine. I stole the first rev of the graphing code and several other ideas from the work Mark Wong and others did on the dbt2 program (here's an intro to using dbt2). Now that I've got something useful for my purposes and am free from conferences for a while, I'm hoping to spend some time investigating how to integrate the unique things I'm doing with some of the tools he's already written. The biggest thing the dbt tests have that I haven't provided for pgbench yet is a framework for measuring I/O and similar statistics during the test run. Given that the PostgreSQL development process already has a heavy requirement on Perl, I really should fall into line and adopt that myself too--despite my strong personal preference for Python in this role.

Copying Virtual Box snapshots

I've really become comfortable nesting all of my Windows installs inside of Virtual Box lately (my main systems run both RedHat and Ubuntu Linux). Just being able to shuttle that image around to wherever I happen to be working is one big help. And the value of working with VMs was just reinforced this week when I learned that my recently installed XP Service Pack 3 introduced an incompatibility with the version of Microsoft SQL Server 2005: Express I needed to install to extract some client data. I just rolled back to the snapshot I took just before I installed that service pack (I am that paranoid), and then it installed fine. While I've been known to do a backup before such an operation even on a real hard disk, that's painful; VM snapshots are so trivial I can take them far more often.

This left me with a dilemma though: my XP install now has this bloated SQL Server mess installed on it that I can't delete until that project is done, but I need to do some real work that I want to keep beyond when I blow that image away. The snapshot tool in the VirtualBox GUI is pretty coarse: the only thing it will do is destructively revert a snapshot. What I want to do is fork the previous snapshot into another machine.

I found some clues for image import/export; it seemed easy. As usual, it wasn't at all.

Background: each disk image in VirtualBox gets a unique UUID. This is why you can't just copy the underlying disk image files to somewhere else--the UUID will still be the same when you import it and it won't work. The export tool "clonevdi" takes care of that for you, but you'll need all the relevant UUIDs for that to do anything useful. I'll show how you can get this info below; when I worked through it the first time I got an unpleasant surprise once I tried using the clone utility I want to talk about first:

$ VBoxManage clonevdi b534a21d-a24a-44b2-35ae-66502938a0b9 image.vdi
VirtualBox Command Line Management Interface Version 1.6.4
(C) 2005-2008 Sun Microsystems, Inc.
All rights reserved.

[!] FAILED calling hardDisk->CloneToImage(Bstr(argv[1]), vdiOut.asOutParam(), progress.asOutParam()) at line 3314!
[!] Primary RC  = NS_ERROR_FAILURE (0x80004005) - Operation failed
[!] Full error info present: true , basic error info present: true
[!] Result Code = NS_ERROR_FAILURE (0x80004005) - Operation failed
[!] Text        = Cloning differencing VDI images is not yet supported ('/d2/virtualbox/Machines/XP Pro/Snapshots/{b534a21d-a24a-44b2-35ae-66502938a0b9}.vdi')
[!] Component   = HardDisk, Interface: IHardDisk, {fd443ec1-000f-4f5b-9282-d72760a66916}
[!] Callee      = IHardDisk, {fd443ec1-000f-4f5b-9282-d72760a66916}

How fun is that? Differencing image, no copies for you!

After some poking around with the forum uber-thread covering copies, it sounded to me like anybody who makes any sort of snapshot is just screwed here. If it's not a plain old disk image, too bad.

As it would take far too long to recreate what's in this VM, I was plenty motivated to find a workaround. Here's how I eventually managed to extract those images:

• With Virtual Box not running, backup the entire .VirtualBox directory.
  
• Start the main image tool and navigate to the snapshot list for the relevant image.
  
• If the one you want is in the history, rather than the current one you want, start at the bottom and blow away any change sets below that one; right-click on them and choose "Revert to current snapshot" if you just want to get rid of changes since then, or "Discard current snapshot and state" if the one you want is actually below either of them.
  
• Once you've gotten to where the image you want is current, select each of the snapshots above it and right-click for "Discard snapshot" to merge their differences into the image below. You'll see the encouraging "Preserving changes to normal hard disk" message here.
  
• Only the one image you want left? It's description should read like this: "IDE Primary Master: [Normal, 10.00GB]" Now you're set to use the command line tools! Exit the GUI, create the image as shown below, save that file somewhere else (it's created in the same VDI directory all the other images live in), then you can restore your original config to get everything back.
  

Here's how the command-line tools worked once I'd done the above to slim down to only the one image I wanted as available. First I take a look at all the UUIDs available:

$ VBoxManage list vms | grep UUID
UUID:            aeb41dd3-d9f5-44fb-b1a2-a32e84f79e64
Primary master:  /d2/virtualbox/VDI/XP Pro.vdi (UUID: 328bcae8-ddf8-4121-139c-f7d0566526f4)

That first UUID is for the whole VM (including the associated config files), that we can ignore. What I want to do then is copy the current running copy, the one labeled "Primary master", to a new image file. First I confirm I can access the right one via the command line tools:

$ VBoxManage showvdiinfo 328bcae8-ddf8-4121-139c-f7d0566526f4

That shows what I expected, so now I can carefully edit that working line via the old up arrow, changing that to the copy command instead. This time it works:

$ VBoxManage clonevdi 328bcae8-ddf8-4121-139c-f7d0566526f4 imagename.vdi

Now, how to actually use one of these images? You need a configuration XML file in the "Machines" directory that matches the one associated with this image copy. Make sure to save that matching file from the VDI/[Machine]/[Machine].xml directory before you do anything drastic (like restore the original configuration with all the snapshots); we'll need it later.

Once I made all the VDI images and had their matching config files, I put back the original .VirtualBox directory, then copied the new images into its VDI directory. Setting up a new VM to use those copies went like this:

• Create a new snapshot with the correct type. When you get to "Virtual Hard Disk", select "Existing". Click on "Add".
  
• You'll see everything listed in your VDI directory. Select the one associated with the snapshot you made and finish making this entry. Exit VirtualBox.
  
• Now what you want to do is copy the long Machine... line from the new Machines/[Machine]/[Machine].xml file you just created to somewhere else (a text editor perhaps), along with the HardDiskAttachment... one that refers to your relocated snapshot. There are more details about this part at Cloning a complete VM.
  
• Overwrite the new machine XML file with the original one associated with your VM, replacing just those two lines with the ones you saved. Basically, you need the new machine UUID and hard disk UUID, everything else should be the same as your original configuration to make sure this machine clones the original as closely as possible.
  

After going through all that, I had everything: the original VM with all its respective snapshots were still there. I had a copy of the install with SQL Server I could tinker with separately, while continuing my regular work in the original VM. Quite an unexpectedly long diversion, but now that I understand what does and doesn't work here I'll make sure to structure my images and snapshots accordingly.

Since I realize snapshots are a lot less useful than I originally thought because of these limitations, it strikes me I might even switch to using the more portable VMDK image files instead of the native VDI format. Snapshot compatibility was the only reason I didn't do that in the first place (can't use them with VMDK). I think I can do that by creating a new virtual disk in VMDK format, attaching that as the secondary master, using a boot CD image to dd the VDI one to the VMDK one, then detaching those disk images and making a new machine based on the VMDK-formatted one. Easy as can be, right? I got the idea from How To Resize a VirtualBox Virtual Disk. But not right now; I've had enough of a VM disk manipulation workout already today.