Showing posts with label virtualbox. Show all posts
Showing posts with label virtualbox. Show all posts

Monday, December 8, 2008

Copying Virtual Box snapshots

I've really become comfortable nesting all of my Windows installs inside of Virtual Box lately (my main systems run both RedHat and Ubuntu Linux). Just being able to shuttle that image around to wherever I happen to be working is one big help. And the value of working with VMs was just reinforced this week when I learned that my recently installed XP Service Pack 3 introduced an incompatibility with the version of Microsoft SQL Server 2005: Express I needed to install to extract some client data. I just rolled back to the snapshot I took just before I installed that service pack (I am that paranoid), and then it installed fine. While I've been known to do a backup before such an operation even on a real hard disk, that's painful; VM snapshots are so trivial I can take them far more often.

This left me with a dilemma though: my XP install now has this bloated SQL Server mess installed on it that I can't delete until that project is done, but I need to do some real work that I want to keep beyond when I blow that image away. The snapshot tool in the VirtualBox GUI is pretty coarse: the only thing it will do is destructively revert a snapshot. What I want to do is fork the previous snapshot into another machine.

I found some clues for image import/export; it seemed easy. As usual, it wasn't at all.

Background: each disk image in VirtualBox gets a unique UUID. This is why you can't just copy the underlying disk image files to somewhere else--the UUID will still be the same when you import it and it won't work. The export tool "clonevdi" takes care of that for you, but you'll need all the relevant UUIDs for that to do anything useful. I'll show how you can get this info below; when I worked through it the first time I got an unpleasant surprise once I tried using the clone utility I want to talk about first:


$ VBoxManage clonevdi b534a21d-a24a-44b2-35ae-66502938a0b9 image.vdi
VirtualBox Command Line Management Interface Version 1.6.4
(C) 2005-2008 Sun Microsystems, Inc.
All rights reserved.

[!] FAILED calling hardDisk->CloneToImage(Bstr(argv[1]), vdiOut.asOutParam(), progress.asOutParam()) at line 3314!
[!] Primary RC = NS_ERROR_FAILURE (0x80004005) - Operation failed
[!] Full error info present: true , basic error info present: true
[!] Result Code = NS_ERROR_FAILURE (0x80004005) - Operation failed
[!] Text = Cloning differencing VDI images is not yet supported ('/d2/virtualbox/Machines/XP Pro/Snapshots/{b534a21d-a24a-44b2-35ae-66502938a0b9}.vdi')
[!] Component = HardDisk, Interface: IHardDisk, {fd443ec1-000f-4f5b-9282-d72760a66916}
[!] Callee = IHardDisk, {fd443ec1-000f-4f5b-9282-d72760a66916}


How fun is that? Differencing image, no copies for you!

After some poking around with the forum uber-thread covering copies, it sounded to me like anybody who makes any sort of snapshot is just screwed here. If it's not a plain old disk image, too bad.

As it would take far too long to recreate what's in this VM, I was plenty motivated to find a workaround. Here's how I eventually managed to extract those images:


  • With Virtual Box not running, backup the entire .VirtualBox directory.
  • Start the main image tool and navigate to the snapshot list for the relevant image.
  • If the one you want is in the history, rather than the current one you want, start at the bottom and blow away any change sets below that one; right-click on them and choose "Revert to current snapshot" if you just want to get rid of changes since then, or "Discard current snapshot and state" if the one you want is actually below either of them.
  • Once you've gotten to where the image you want is current, select each of the snapshots above it and right-click for "Discard snapshot" to merge their differences into the image below. You'll see the encouraging "Preserving changes to normal hard disk" message here.
  • Only the one image you want left? It's description should read like this: "IDE Primary Master: [Normal, 10.00GB]" Now you're set to use the command line tools! Exit the GUI, create the image as shown below, save that file somewhere else (it's created in the same VDI directory all the other images live in), then you can restore your original config to get everything back.


Here's how the command-line tools worked once I'd done the above to slim down to only the one image I wanted as available. First I take a look at all the UUIDs available:


$ VBoxManage list vms | grep UUID
UUID: aeb41dd3-d9f5-44fb-b1a2-a32e84f79e64
Primary master: /d2/virtualbox/VDI/XP Pro.vdi (UUID: 328bcae8-ddf8-4121-139c-f7d0566526f4)


That first UUID is for the whole VM (including the associated config files), that we can ignore. What I want to do then is copy the current running copy, the one labeled "Primary master", to a new image file. First I confirm I can access the right one via the command line tools:


$ VBoxManage showvdiinfo 328bcae8-ddf8-4121-139c-f7d0566526f4


That shows what I expected, so now I can carefully edit that working line via the old up arrow, changing that to the copy command instead. This time it works:


$ VBoxManage clonevdi 328bcae8-ddf8-4121-139c-f7d0566526f4 imagename.vdi


Now, how to actually use one of these images? You need a configuration XML file in the "Machines" directory that matches the one associated with this image copy. Make sure to save that matching file from the VDI/[Machine]/[Machine].xml directory before you do anything drastic (like restore the original configuration with all the snapshots); we'll need it later.

Once I made all the VDI images and had their matching config files, I put back the original .VirtualBox directory, then copied the new images into its VDI directory. Setting up a new VM to use those copies went like this:


  • Create a new snapshot with the correct type. When you get to "Virtual Hard Disk", select "Existing". Click on "Add".
  • You'll see everything listed in your VDI directory. Select the one associated with the snapshot you made and finish making this entry. Exit VirtualBox.
  • Now what you want to do is copy the long Machine... line from the new Machines/[Machine]/[Machine].xml file you just created to somewhere else (a text editor perhaps), along with the HardDiskAttachment... one that refers to your relocated snapshot. There are more details about this part at Cloning a complete VM.
  • Overwrite the new machine XML file with the original one associated with your VM, replacing just those two lines with the ones you saved. Basically, you need the new machine UUID and hard disk UUID, everything else should be the same as your original configuration to make sure this machine clones the original as closely as possible.


After going through all that, I had everything: the original VM with all its respective snapshots were still there. I had a copy of the install with SQL Server I could tinker with separately, while continuing my regular work in the original VM. Quite an unexpectedly long diversion, but now that I understand what does and doesn't work here I'll make sure to structure my images and snapshots accordingly.

Since I realize snapshots are a lot less useful than I originally thought because of these limitations, it strikes me I might even switch to using the more portable VMDK image files instead of the native VDI format. Snapshot compatibility was the only reason I didn't do that in the first place (can't use them with VMDK). I think I can do that by creating a new virtual disk in VMDK format, attaching that as the secondary master, using a boot CD image to dd the VDI one to the VMDK one, then detaching those disk images and making a new machine based on the VMDK-formatted one. Easy as can be, right? I got the idea from How To Resize a VirtualBox Virtual Disk. But not right now; I've had enough of a VM disk manipulation workout already today.

Monday, August 4, 2008

Virtualbox and custom kernels

I've been using Virtualbox successfully for a few months now. On my laptop, with Linux as the host operating system, the virtual Windows XP install runs faster than the real XP install on another partition, even though I've only given it 512MB of RAM to work with. Very interesting program.

But on my main desktop system I run a custom kernel, and the same Virtualbox install didn't work there. The kernel module it loads just failed. You can get that to reinstall using

/etc/init.d/vboxdrv setup

Or at least you're supposed to be able to. The error message it gave me in /var/log/vbox-install.log suggested I needed to set KERN_DIR to point to the root of my kernel source; did that. No help, and now there's nothing useful in that log file. The module was there, it just didn't work. Manually doing "modprobe vboxdrv" said it was missing symbols.

To track this down, I started by reading the source for the vboxdrv init script. Under the hood, it runs /usr/share/virtualbox/src/build_in_tmp to recompile the module. You can run this by hand to see what's really going on. Annoyingly, there's a crummy test that generates this bogus warning:

test -e include/linux/autoconf.h -a -e include/config/auto.conf || ( \
echo; \
echo " ERROR: Kernel configuration is invalid."; \
echo " include/linux/autoconf.h or include/config/auto.conf are missing."; \
echo " Run 'make oldconfig && make prepare' on kernel src to fix it."; \

This is a lie, both files are there, and it prints this even in the final working configuration after I fixed the problem. I also saw some warnings about deprecated symbols in the middle and the linker complained about missing things, just like when I try to load the module.

I found some notes about using Virtualbox with a custom kernel in an ubuntu forum which was interesting background but no help. Another nice link that I want to remember for future use shows how to repackage Virtualbox using dpkg to support a customer kernel. But since this was on a RedHat system that wasn't going to help, and ultimately I suspected that the same compilation problem would kick in.

After some more searching, I found the root cause here. Some of the kernel interfaces that existed in the original 2.6.18 kernel I was using last time this worked were changed in the 2.6.25 kernel I was running now, which is why the symbols it was looking for were missing. A Gentoo bug report goes over the ugly details and suggests that Virtualbox versions after 1.6.2 fixes the problem (I had been running 1.5.6). Sure enough, download the latest version, install that one, and the module rebuilds itself just fine now as long as KERN_DIR is set correctly.

This is one of those interesting bits to debate: the reason this problem was introduced was that an API was changed "to fix bugs and add a really important feature". Linux benefits from being free to make such enhancements as necessary. But at the same time, the complete lack of concern for backward compatibility in kernel modules can really make things difficult for regular users of Linux software that must link to the kernel. The obligatory reading here for the developer's side of this argument is Stable API Nonsense.