Monday, 27th December 2010 - 4 Comments
Hey - good to see you! If you are new here, you can subscribe to the RSS feed for updates on this topic.
It is often difficult to replicate your web application hosting environment, not least due to differences in hardware (CPU, RAM, hard disk space and type) but also operating systems. Front-end design/developers in particular will need access to Windows only software such as Photoshop and Illustrator, as well as multiple versions of Internet Explorer in order to effectively test their HTML/CSS compatibility. VirtualBox is a fantastic piece of open source virtualisation software. In this tutorial I will use it to create a local development version of a remote web application server.
I won't dwell on the specifics of configuring Apache, Python, PHP etc as that will be specific to your setup. However, I will provide you with the ability to communicate with the machine via your guest OS. I will also show you how to setup a shared directory between your host operating system and the guest virtual development environment in order to reduce the constant file uploading that is synonymous with iterative web development.
The Benefits of a Virtual Server
Running a localised virtual development environment is practical for multiple reasons. If you use your host machine for tasks other than web development, such as video editing, scientific computation or even video games then you will want to have as much CPU and RAM available as possible. Virtualisation allows you to self-contain all of the processes which usually eat system resources, such as a web database and cacheing servers into an environment that can be fully switched on/off at will. This ensures that you never have arbitrary web processes running in the background utilising resources.
You may be developing web applications across multiple technology platforms. I myself develop applications in ASP.NET MVC 2.0 on Windows Server 2003, which requires Visual Web Developer 2010, SQL Server 2005 and Internet Information Services to all be present. The blog you are reading as well as other sites I develop all run on Django with Apache and Nginx. It clearly isn't practical to install this software on the same host operating system. Virtualisation once again comes to the rescue by providing self-contained virtual images of multiple operating systems that are easily segregated.
Perhaps the most important consideration for web developers is that the virtual development environment can be setup to mirror the remote environment as close as possible. This reduces the need to constantly upload code to a staging server while testing iteratively, preventing "pollution" of your version control branch. Most of the development bugs that are OS-specific can be caught prior to the stage upload saving you a lot of dev time.
I began writing this tutorial prior to the release of VirtualBox 4.0. I am using v3.2 here, but there should be few, if any, differences between the two for the purposes of our setup.
Visit the VirtualBox Downloads page and select the latest version of the software to download. I am using a Windows 7 host machine for this task so you will need to select Windows Hosts -> x86/amd64.
Once it is downloaded, run the setup executable. You can use the defaults for all of the settings. The only aspect that I changed was the default directory for the virtual disk images. However, if you do not have any concerns using the default, then go with that.
Add your first Ubuntu Server virtual machine
Start Virtualbox and you will be presented with a two-paned screen. The left hand side will list all of your virtual machines, while the right hand side will provide details on them. Click "New" to begin creating your first VM.
On the next screen, select "Linux" as your operating system type and "Ubuntu" as your distribution. You can call your VM whatever you want, but for the purposes of this tutorial I will call it "Ubuntu Server 10.04 LTS".
On the next screen you need to decide how much memory to allocate to the machine. Depending upon your uses you may want to increase it from the default 512Mb. In particular, if you are using this as a database test server, you will almost certainly want more RAM. Click Next to continue.
This screen allows you to choose how much to allocate to your VM hard disk. The default is 8Gb and you can adjust this later. Since I presume this is your first time using VirtualBox, you won't be able to use an existing disk, so select "Create New Disk". You'll notice that on my screenshot I already have an existing disk - just ignore that!
Clicking on next gives you the option of choosing a dynamic or fixed size disk. I personally prefer a fixed size disk as that way my VMs don't take over my hard disk as log files grow large, etc. Again, this is a personal choice, so I will leave it up to you. The next screen allows you to adjust the size of the disk. I have left mine at 8Gb, but feel free to choose any size that is available to you. You will also need to choose the location on your host machine's hard disk for the virtual disk. You can keep it in the default directory unless you have other needs.
Upon clicking Next, you will have finished creating your first virtual machine. All of the settings for the VM will be listed on the right hand side of VirtualBox. You will now need to download Ubuntu Server 10.04. I have chosen the 32bit edition of 10.04 LTS. Now, in order to begin installing Ubuntu on your VM you will need to do one of two things. You can either mount the ISO as a virtual IDE device to replicate a CD-ROM drive or create a USB flash drive mount. I've opted for the former, so this is how we will proceed.
Open up the Settings for the guest virtual machine and select the Storage Tab on the left hand side. Select the "Empty" IDE controller and then the folder with the green arrow adjacent to CD/DVD device. Navigate to the Ubuntu Server ISO and select it. This will mount it as a virtual drive which will be checked prior to the virtual hard disk on boot of the VM:
Finally, we can start installing Ubuntu Server. Click "Start" to begin booting the VM. It will attempt to read the boot disk (our virtually mounted ISO drive) and launch the installation process as normal. This tutorial assumes you have installed Ubuntu Server before, but I will run through the steps anyway as there are some minor differences (particularly in hard disk configuration) due to the virtualised nature of the install. The first screen shows the language selection.
Run through the language selection and choose the correct keyboard for your locale. I am in the United Kingdom, so I have selected this as mine. Eventually you will be asked how you want to partition the hard disk. Notice that the virtual machine only has access to the virtual disk you created earlier. This is by design - we do not want the VM tinkering with our host machine files! I always select Guided Partitioning, but if your server has a more complicated setup then you can configure the volumes separately. After perusing the partitition table changes select "Finish partitioning and write changes to disk".
After some more installation, you will eventually be asked to enter your full name and your username. Choose something memorable for your username. You will also be asked for your host name. This also needs to be something memorable, but simple. Eventually you will be asked if the GRUB bootloader should be added to the master boot record. This has no bearing on your host machine bootup - it is only for the initialisation of the virtual machine guest. Select Yes and then proceed with the remainder of the installation. You will also need to unmount the ISO from your Storage settings.
You now have a fully functioning edition of Ubuntu Server 10.04 LTS as a guest operating system on the Windows host. After starting the system, you will be presented with the standard login screen. Please excuse the Star Trek reference - I have an unhealthy habit of naming all of my systems after the Enterprise bridge crew.
Configuring the Virtual Machine
As it stands the Ubuntu server is not hugely useful for our web development needs. We require a properly defined network interface so that we can access it from the host machine. In addition, we require a means of automatically synchronising a host directory with a guest directory so that we're not constantly transferring files between the two machines.
Let's begin by performing an update and an upgrade. This task is the first thing I carry out on every build of Ubuntu Server.
We will also want to connect to the server in the usual manner via SSH, especially if we wish to open multiple terminals. Let's install the OpenSSH libraries:
At this point you will need to configure your server in the manner specific to your web setup. I won't dwell on the details here, but my stack usually consists of Apache, Django and MySQL. I also tend to install Memcached. The configuration of such a stack will be outlined in another post (or two!). Once you have finished installing your stack to your liking, we can begin configuring network access from the host machine.
Configuring Network Access from the Host
SSH is usually defined to accept connections on port 22, while Apache is usually configured to listen on port 80. We are going to forward these ports from the guest Ubuntu Server to the host machine. This will allow us to utilise the host localhost on alternative ports to access the guest operating system. For instance, we can map SSH to port 2222 on the host. This will allow us to use Putty (or equivalent) to access "localhost:2222" as a means of gaining secure access the guest. In addition, we can forward port 80 (or 8000 if you are using a Django development server) to 8888 on the host. In your favourite web browser you will be able to access you guest website at http://localhost:8888.
To enable the port forwarding, you need to open up a Windows prompt by running "cmd" in the Start Menu. Navigate to the directory where VirtualBox is installed (on my host, it is C:\Program Files\Oracle\VirtualBox) and type the following commands. Be sure to replace "django" with your web server and "8000" with the guest port that your guest web server is listening on. Note that the backslashes represent a line continuation, not a character to be typed in!
Reboot the virtual machine and restart the VirtualBox host program as well. Upon reload of the virtual machine you should be able to access the machine via SSH and HTTP over the newly defined ports on the host.
Synchronising the Host and Guest Development Directory
The final stage in the local development server journey is to synchronise your web root (or equivalent) directory on the guest, with that on the host. This will prevent you from having to commit to your repository on the local host and then deploy to the web root on your local development virtual machine (an extremely painful operation after many iterations!). This does require a bit of system administration - so here's how to proceed.
We need to install Guest Additions on the VM guest, so that the guest can access the host machine via a virtual mount point. The first task is to install dkms and the build-essential libraries:
Once the server has booted up again, look to the VirtualBox guest menu for "Devices". Select Devices -> Install Guest Additions. This will add the Guuest Additions ISO image as a readable device to the virtual machine. We still need to manually mount it though. Change to the root media directory and make a new directory called cdrom. Then mount the guest additions disk image to the new cdrom mount point:
Once the ISO is mounted we can view the directory contents and choose the appropriate installer executable for the guest operating system. Since we are running a 32bit edition of Ubuntu Server, we choose VBoxLinuxAdditions-x86.run. Change to the cdrom mount directory and execute the file:
Once the installation is complete you will see a notification that the installer could not find any variant of the X Windows system. This is because the Ubuntu Server does not ship with any windowing systems and as such this error can be safely ignored.
The final task is to add the directory on the host machine that you wish to synchronise to the guest. You will need to shutdown the guest operating system first so that you can access the Settings within VirtualBox. Once the guest VM has been powered off, open up the Settings and look for the "Shared Folders" tab on the left hand side. Click to add a folder on your local hard disk and then select whether you wish to make it read only or not. This is a subtle point because Python code will generate .pyc compiled byte code files when running a website. Thus, for my Django setup, I am unable to select Read Only. Name the share folder appropriately as you will use this name on your guest operating system.
Fire up your guest VM and create a new directory to become your mount point for the host operating system local directory. Once it is created you can use the vboxsf guest additions tool to mount the local and remote directories. Replace yourshare with the name you provided VirtualBox with for your local host folder and the /path/to/mountdir with your remote mounting directory:
And that's it! You now have a fully functional development server which will allow you to test your web stack on your local host operating system. I use this setup daily in order to make changes to the Django stack running this blog as well other sites I work on. I hope that you find continuous integration to be far more straightforward now. Let me know if you have any difficulty setting it up, via the comments. I'm always glad to hear your feedback too.