Twinned Hosting

Twinned hosting uses mirroring between 2 dedicated PC servers to achieve a robust and cost-effective hosting and disaster recovery solution.

The problem with backups

Administrators have to live with the knowledge that an SMS message can arrive at any time of the day or night to let them know that a server has failed and a very stressful few hours or days lie ahead. Just knowing that it could happen any time is stressful.  It hangs over us.

Business managers know their business could be damaged or destroyed by a threat that they can’t quantify or mitigate effectively.  They have to trust their administrators to form and maintain an effective disaster recovery strategy.  They all know someone who thought they had backups, but when they were needed, the backups couldn’t be restored.

Mirroring is a better disaster recovery strategy

There is one disaster recovery strategy that gives managers and administrators the confidence that when a server fails for whatever reason, normal business will be resumed shortly.  An exact replica is placed in a remote data centre, and updated continually.  Then when disaster strikes, the replica becomes the live system.  The administrator just has to reconfigure some domain names.

This is not a new idea.  Big companies have been mirroring their mainframes for many years. There are a range of solutions for PCs, one of which is Twinned Hosting.

How Twinned Hosting works

1. The web server is installed in a virtual machine. The virtual machine’s hard drive is held in a single file on the host. It is this virtual disk file which is synchronised with the remote server.
2. The virtual disk is optimised and flushed before synchronisation. Empty disk space that has been used is wiped by filling it with zeros. This avoids transmitting unnecessary garbage. The database is flushed and locked then the file system is flushed. This ensures that the disk is synchronised in a consistent state.
3. A snapshot is taken of the virtual disk.
   The virtual disk file is stored on the LVM file system, which enables a snapshot to be created. A identical copy of the virtual disk appears, but it doesn’t change. Meanwhile, the original file can continue to be used. The snapshot is instantaneous because the file is not really copied. The file system just starts logging changed to the original and used that deliver the snapshot version.
   As soon as the snapshot is taken, the database lock started in step 2 is released. Steps 2 and 3 happen in an instant, so the web server is not interrupted.
4. The virtual disk is synchronised with the twin
   The twin has a copy of the disk from the last time it was synchronised, so only the changes need to be transmitted.
   When the synchronisation is complete, a checksum of each virtual disk file is taken and compared to verify that the synchronisation was successful.
5. The virtual disk is copied
   Two versions of the virtual disk are kept on the idle server. One is live, and ready to become the new active server if the current active server fails. The other is the synchronised copy, which may be in an inconsistent (unusable) state during synchronisation.
6. The old idle virtual machine is shut down and the new one started.
   Some configuration changes must be made before the new virtual machine becomes live. The IP addresses are changed to addresses that will work on it’s new host network. Any software license keys are updated. (Another good reason to use Open Source software)
7. The twin website can be checked at any time
   The twin website is now live. The twin can be browsed by using the prefix wwt. in place of www. This is because all the websites are configured with dual identities of www.x.com and wwt.x.com. Also, the domain’s name servers are directing requests for www.x.com to the active server and wwt.x.com to the idle server.

When a server fails, the administrator just needs to update the DNS to make the idle server active, and those domains are up and running again.

When is/isn’t twinned hosting appropriate?

• Twinned hosting is ideal for the typical internet connected server hosting multiple low traffic domains with a web server, email server, database, web control panel etc. The guests may contain any operating system.
• Twinned hosting does not work with shared hosting or virtual private servers (VPS). The budget needs to cover two dedicated or co-located servers.
• Twinned hosting does not work with load balancing or seperate web server and database. No domain may exceed the capacity of a single server.
• The servers may be in different data centres, so the bandwidth between them is good, but not good enough for live replication. The mirror lags behind the active guest so the loss of recent updates must be tollerable in the rare event of a disaster.

Why is mirroring better than backups?

Have you ever had to build a new server from data backups and software disks while the phone rings off the hook with irate customers? It’s not recommended. These are some experiences of server failures that I have heard first hand:

• Automated backups have been failing for months but nobody noticed. There are no backups.
• The backup tape drive was also stolen, and we can’t buy a compatible drive.
• Essential files are missing from backups
• There is no record of the many configuration changes or even the FAQs, HoTos and forum posts from which they were gleaned.
• Problems are caused by a mismatch between versions of the software that created the old data, the installed software, the old and new operating system, the application and operating system updates and patches.
• Rebuilding a server from backups and software disks took a long, long time.

When virtual machines are mirrored, recovering from a disaster is quick because there is no need to restore anything. The entire virtual disk is replicated, including Operating System, software, configuration files, registry, permissions, data and all. The virtual machine is already running on the ‘backup’ server.

After the idle server is activated, the administrator can re-establish the failed twin without the pressure of mounting down-time. In fact, re-installing a host is quick because it does very little and its configuration is simple.

Requirements of an implementation

• Both servers are secure. If one server is compromised, the twin is not.
• A range of guest operating systems are support, including Linux and Windows
• The mirroring process is fully automated
• To minimize cost and maximize reliability, proven Open Source software is used.
• Monitoring and alerting software is used to minimize the delay between disaster and recovery.

Implementation considerations

• The twins should be in physically remote data centers operated by different companies. This protects you from a total failure of one data center or the financial collapse of the company that owns the it.
• The DNS records should be configured with a low TTL (e.g. 1 hour) so that DNS changes take effect quickly.
• Spammers will deliver their filth to any server that will accept it, even if it not designated in an MX record. So idle email servers should be configured not to accept incoming mail.
• The virtual disks could be synchronized daily to minimize bandwidth, or continuously to minimize the lag.
• Automated failover may be appropriate in some instances, administrators may prefer to assess whether a server failure can be remedied before switching.
• The best candidate for synchronizing the virtual disks is rsync, which only transfers changes.
• After synchronizing the active guest to the twin, the idle guest needs to be booted and reconfigured e.g. Change IP addresses, software license keys. Clear the outgoing mail queue to prevent messages being sent twice.

Where are the instructions and scripts to set up twinned hosting?

I’m working on it. The central task of mirroring the guests is explained in the CentOS article Backing up a running virtual machine

You are welcome to contribute your own scripts and instructions to fill the gaps.

Where did the idea come from?

Virtualization has been around since the 60s but articles about mirroring virtual machines with rsync started to appear around 2006, made practical by improvements in virtualization software, processors, networks, and disks. Twinned Hosting just takes this idea a step further and turns it into a methodology. The name was given to the methodology so it could be discussed and developed more easily within the open source community.