= Relax and Recover: User guide
:author: Dag Wieers and Jeroen Hoekx
:data-uri:
:encoding: UTF-8
:icons:
:iconsdir: /usr/share/asciidoc/images/icons
:numbered:
:toc:
:toclevels: 2

This document describes everything there is to know regarding Relax and
Recover, an Open Source bare-metal disaster recovery and system migration
solution designed for Linux.


== Introduction
Relax and Recover (Rear) is the leading Open Source disaster recovery
solution, and successor to mkcdrec. It comprises of a modular framework
and ready-to-go workflows for many common situations to produce a bootable
image and restore from backup using this image. As a benefit, it allows to
restore to different hardware and can therefore be used as a migration
tool as well.

Currently Rear supports various boot media (incl. ISO, PXE, OBDR tape,
USB or eSATA storage), a variety of network protocols (incl. sftp, ftp,
http, nfs, cifs) as well as a multitude of backup strategies (incl.
IBM TSM, HP DataProtector, Symantec NetBackup, Bacula, rsync).

Rear was designed to be easy to set up, requires no maintenance and is
there to assist when disaster strikes. Its setup-and-forget nature eradicates
any excuses for not having a disaster recovery solution implemented.

Recovering from disaster is made very straight-forward by a 2-step recovery
process so that it can be executed by operational teams when required.
When used interactively (e.g. when used for migrating systems), menus help
make decisions to restore to a new (hardware) environment.

Extending Rear is made possible by its modular framework. Consistent
logging and optionally extended output help understand the concepts behind
Rear and help debug during development.


== Relax and Recover project
The support and development of the Relax and Recover project takes place
on SourceForge:

Project page::
    http://sourceforge.net/projects/rear

Website::
    http://rear.sourceforge.net/

In case you have questions, ideas or feedback about this document, you
can contact the development team on the Rear mailinglist at:
rear-users@lists.sourceforge.net.

NOTE: Note that you have to be subscribed to be able to send mails to the Rear
mailinglist ! You can subscribed to the list at:
https://lists.sourceforge.net/lists/listinfo/rear-users


== Features and functionality
Relax and Recover has a wide range of features:

// FIXME: Insert the various features from the Release Notes
//        include the information from the presentations
//        and optionally the rescue creation and recovery workflow
// FIXME: Get rid of the below list when everything is in the feature set

 - Improvements to HP SmartArray and CCISS driver integration
 - Improvements to software RAID integration
 - Disk layout change detection for monitoring
 - One-Button-Disaster-Recovery (OBDR) tape support
 - DRBD filesystem support
 - Bacula tape support
 - Multiple DR images per system on single USB storage device
 - USB ext3/ext4 support
 - GRUB bootloader re-implementation
 - Add Relax and Recover entry to local GRUB configuration
 - Nagios and webmin integration
 - Syslinux boot menu
 - Storing rescue/backup logfile on rescue media
 - Restoring to different hardware
 - RHEL4, RHEL5 and RHEL6 support
 - Various usability improvements
 - Serial console support auto-detected
 - Lockless workflows
 - USB udev integration to trigger rear on inserting USB device
 - Beep/UID led/USB suspend integration


== Installing Rear


// FIXME: Make this part more generic and add distribution information

=== Software requirements
Rear can be installed from a native RPM package. Since it is mostly a
collection of Bash scripts, there is no need to make RPM packages for each
Red Hat release. For the supported RHEL distributions it is mostly
version-agnostic.

Rear has few dependencies:

 - recent syslinux (tested with 4.0)

For OBDR tape mode you need the following packages:

 - lsscsi
 - mkisofs
 - sg3_utils

Rear at this time has only been tested on RHEL4, RHEL5 and RHEL6.


=== Limitations
Relax and Recover offers a lot of flexibility in various use-cases, however it
does have some limitations under certain circumstances:

 - Rear depends on the software of the running system. When recovering this
   system to newer hardware, it is possible that the hardware support of the
   original system does not support the newer hardware.
+
This problem has been seen when restoring an older RHEL4 with an older HP
Proliant Support Pack (PSP) to more recent hardware. This PSP did not detect
the newer HP SmartArray controller or its firmware.

 - Rear supports recovering to different hardware, but it cannot always
   automatically adapt to this new environment. In such cases it requires a
   manual intervention to eg.

   * modify the _disklayout.conf_ to indicate the number of controller, disks
     or specific custom desires during restore

   * reduce the partition-sizes/LV-sizes when restoring to smaller storage

   * pull network-media or configure the network interfaces manually

 - Depending on your back-up strategy you may have to perform actions, like:

   * insert the required tape(s)

   * perform commands to restore the backup


=== RPM installation
Simply install (or update) the provided packages using
the command: +rpm -Uhv rear-1.12.0svn497-0.1.noarch.rpm+

You can test your installation by running +rear dump+:

----
[root@system ~]# rear dump
Relax and Recover Version 1.12.0svn497 / 2011-07-11
Dumping out configuration and system information
System definition:
                                    ARCH = Linux-x86_64
                                      OS = GNU/Linux
                               OS_VENDOR = RedHatEnterpriseServer
                              OS_VERSION = 5.6
...
----


== Overview
// FIXME: Describe the concept behind Rear
//  - Creating a rescue environment (bootable media and optionally backup)
//  - Restoring from booted media


== Scenarios
// FIXME: Add the various scenarios, merge with configuration-examples.txt and
//        the below content
//  e.g. using different backup methods,
//       using different output methods

== Using Rear with USB storage devices
Using USB devices with Rear can be appealing for several reasons:

 - If you only need to have a bootable rescue environment, a USB device is
   a *cheap device* for storing only 25 to 60MB to boot from

 - You can leave the USB device inserted in the system and *opt-in booting*
   from it only when disaster hits (although we do recommend storing rescue
   environments off-site)

 - You can *store multiple systems and multiple snapshots* on a single device

 - In case you have plenty of space, it might be a simple solution to store
   complete Disaster Recovery images (rescue + backup) on a single device for
   a set of systems

 - For migrating a bunch of servers having a single device to boot from might
   be very appealing

 - We have implemented a specific workflow: inserting a REAR-000 labeled USB
   stick will invoke +rear udev+ and adds a rescue environment to the USB
   stick (updating the bootloader if needed)

However USB devices may be slow for backup purposes, especially on older
systems or with unreliable/cheap devices.

=== Configuring Rear for USB storage devices
The below configuration (_/etc/rear/local.conf_) gives a list of possible
options when you want to run Rear with USB storage.

[source,bash]
----
BACKUP=BACULA
OUTPUT=USB
USB_DEVICE=/dev/disk/by-label/REAR-000
----

IMPORTANT: On RHEL4 or older there are no _/dev/disk/by-label/_ udev aliases,
           which means we cannot use device by label. However it is possible
           to use +by-path+ references, however this makes it very specific
           to the USB port used.  We opted to use the complete device-name,
           which can be dangerous if you may have other _/dev/sdX_ devices
           (luckily we have CCISS block devices in _/dev/cciss/_).


=== Preparing your USB storage device
To prepare your USB device for use with Rear, do: +rear format /dev/sdX+

This will create a single partition, make it bootable, format it with ext3,
label it +REAR-000+ and disable warnings related filesystem check for the
device.


=== USB storage as rescue media

==== Configuring Rear to have Bacula tools
If the rescue environment needs additional tools and workflow, this can be
spcified by using +BACKUP=BACULA+ in the configuration file
_/etc/rear/local.conf_:

[source,bash]
----
BACKUP=BACULA
OUTPUT=USB
USB_DEVICE=/dev/disk/by-label/REAR-000
----

==== Making the rescue USB storage device
To create a rescue USB device, run +rear -v mkrescue+ as shown below after
you have inserted a *REAR-000* labeled USB device. Make sure the device name
for the USB device is what is configured for +USB_DEVICE+.

----
[root@system ~]# rear -v mkrescue
Relax and Recover Version 1.12.0svn497 / 2011-07-11
Creating disk layout.
Creating root FS layout
Copy files and directories
Copy program files & libraries
Copy kernel modules
Checking udev
Create initramfs
The cleanup phase
Finished in 72 seconds.
----

WARNING: Doing the above may replace the existing MBR of the USB device.
         However any other content on the device is retained.


[[booting-from-usb]]
==== Booting from USB storage device
Before you can recover our DR backup, it is important to configure the BIOS to
boot from the USB device. In some cases it is required to go into to the BIOS
(+F9+ during boot) to change the boot-order of devices. (In BIOS select
+Standard Boot Order (IPL)+)

Once booted from the USB device, select the system you like to recover from
the list. If you don't press a key within 30 seconds, the system will try to
boot from the local disk.

----
      .-------------------------------------------------------------.
      |               Relax and Recover v1.12.0svn497               |
      |-------------------------------------------------------------|
      |  Recovery images                                            |
      |   system.localdomain                                      > |
      |   other.localdomain                                       > |
      |-------------------------------------------------------------|
      |  Other actions                                              |
      |   Help for Relax and Recover                                |
      (>  Boot Local disk (hd1)                                    <)
      |   Boot BIOS disk (0x81)                                     |
      |   Boot Next BIOS device                                     |
      |   Hardware Detection tool                                   |
      |   Memory test                                               |
      |   Reboot system                                             |
      |   Power off system                                          |
      `-------------------------------------------------------------'

             Press [Tab] to edit options or [F1] for Rear help

                        Automatic boot in 30 seconds...
----

WARNING: Booting from a local disk may fail when booting from a USB device.
         This is caused by the fact that the GRUB bootloader on the local
         disk is configured as if it is being the first drive +(hd0)+ but
         it is in fact the second disk +(hd1)+. If you do find menu entries
         not working from GRUB, please remove the +root (hd0,0)+ line from
         the entry.

Then select the image you would like to recover.

----
      .-------------------------------------------------------------.
      |                     system.localdomain                      |
      |-------------------------------------------------------------|
      |  2011-03-26 02:16 backup                                    |
      (> 2011-03-25 18:39 backup                                   <)
      |  2011-03-05 16:12 rescue image                              |
      |-------------------------------------------------------------|
      |  Back                                                     < |
      |                                                             |
      |                                                             |
      |                                                             |
      |                                                             |
      |                                                             |
      |                                                             |
      |                                                             |
      |                                                             |
      |                                                             |
      `-------------------------------------------------------------'

             Press [Tab] to edit options or [F1] for Rear help



Rear backup using kernel 2.6.32-122.el6.x86_64
BACKUP=NETFS OUTPUT=USB OUTPUT_URL=usb:///dev/disk/by-label/REAR-000
----

TIP: When browsing through the images you get more information about the
     image at the bottom of the screen.

==== Restoring from USB rescue media
Then wait for the system to boot until you get the prompt.

On the shell prompt, type +rear recover+.

You may need to answer a few questions depending on your hardware
configuration and whether you are restoring to a (slightly)
different system.

----
RESCUE SYSTEM:/ # rear recover
Relax and Recover Version 1.12.0svn497 / 2011-07-11
NOTICE: Will do driver migration
To recreate HP SmartArray controller 3, type exactly YES: YES
To recreate HP SmartArray controller 0, type exactly YES: YES
Clearing HP SmartArray controller 3
Clearing HP SmartArray controller 0
Recreating HP SmartArray controller 3|A
Configuration restored successfully, reloading CCISS driver...  OK
Recreating HP SmartArray controller 0|A
Configuration restored successfully, reloading CCISS driver...  OK
Comparing disks.
Disk configuration is identical, proceeding with restore.
Type "Yes" if you want DRBD resource rBCK to become primary: Yes
Type "Yes" if you want DRBD resource rOPS to become primary: Yes
Start system layout restoration.
Creating partitions for disk /dev/cciss/c0d0 (msdos)
Creating partitions for disk /dev/cciss/c2d0 (msdos)
Creating software RAID /dev/md2
Creating software RAID /dev/md6
Creating software RAID /dev/md3
Creating software RAID /dev/md4
Creating software RAID /dev/md5
Creating software RAID /dev/md1
Creating software RAID /dev/md0
Creating LVM PV /dev/md6
Creating LVM PV /dev/md5
Creating LVM PV /dev/md2
Creating LVM VG vgrem
Creating LVM VG vgqry
Creating LVM VG vg00
Creating LVM volume vg00/lv00
Creating LVM volume vg00/lvdstpol
Creating LVM volume vg00/lvsys
Creating LVM volume vg00/lvusr
Creating LVM volume vg00/lvtmp
Creating LVM volume vg00/lvvar
Creating LVM volume vg00/lvopt
Creating ext3-filesystem / on /dev/mapper/vg00-lv00
Mounting filesystem /
Creating ext3-filesystem /dstpol on /dev/mapper/vg00-lvdstpol
Mounting filesystem /dstpol
Creating ext3-filesystem /dstpol/sys on /dev/mapper/vg00-lvsys
Mounting filesystem /dstpol/sys
Creating ext3-filesystem /usr on /dev/mapper/vg00-lvusr
Mounting filesystem /usr
Creating ext2-filesystem /tmp on /dev/mapper/vg00-lvtmp
Mounting filesystem /tmp
Creating ext3-filesystem /boot on /dev/md0
Mounting filesystem /boot
Creating ext3-filesystem /var on /dev/mapper/vg00-lvvar
Mounting filesystem /var
Creating ext3-filesystem /opt on /dev/mapper/vg00-lvopt
Mounting filesystem /opt
Creating swap on /dev/md1
Creating DRBD resource rBCK
Writing meta data...
initializing activity log
New drbd meta data block successfully created.
Creating LVM PV /dev/drbd2
Creating LVM VG vgbck
Creating LVM volume vgbck/lvetc
Creating LVM volume vgbck/lvvar
Creating LVM volume vgbck/lvmysql
Creating ext3-filesystem /etc/bacula/cluster on /dev/mapper/vgbck-lvetc
Mounting filesystem /etc/bacula/cluster
Creating ext3-filesystem /var/bacula on /dev/mapper/vgbck-lvvar
Mounting filesystem /var/bacula
Creating ext3-filesystem /var/lib/mysql/bacula on /dev/mapper/vgbck-lvmysql
Mounting filesystem /var/lib/mysql/bacula
Creating DRBD resource rOPS
Writing meta data...
initializing activity log
New drbd meta data block successfully created.
Creating LVM PV /dev/drbd1
Creating LVM VG vgops
Creating LVM volume vgops/lvcachemgr
Creating LVM volume vgops/lvbackup
Creating LVM volume vgops/lvdata
Creating LVM volume vgops/lvdb
Creating LVM volume vgops/lvswl
Creating LVM volume vgops/lvcluster
Creating ext3-filesystem /opt/cache on /dev/mapper/vgops-lvcachemgr
Mounting filesystem /opt/cache
Creating ext3-filesystem /dstpol/backup on /dev/mapper/vgops-lvbackup
Mounting filesystem /dstpol/backup
Creating ext3-filesystem /dstpol/data on /dev/mapper/vgops-lvdata
Mounting filesystem /dstpol/data
Creating ext3-filesystem /dstpol/databases on /dev/mapper/vgops-lvdb
Mounting filesystem /dstpol/databases
Creating ext3-filesystem /dstpol/swl on /dev/mapper/vgops-lvswl
Mounting filesystem /dstpol/swl
Creating ext3-filesystem /dstpol/sys/cluster on /dev/mapper/vgops-lvcluster
Mounting filesystem /dstpol/sys/cluster
Disk layout created.

The system is now ready to restore from Bacula. You can use the 'bls' command
to get information from your Volume, and 'bextract' to restore jobs from your
Volume. It is assumed that you know what is necessary to restore - typically
it will be a full backup.

You can find useful Bacula commands in the shell history. When finished, type
'exit' in the shell to continue recovery.

WARNING: The new root is mounted under '/mnt/local'.

rear>
----


[[restoring-from-bacula-tape]]
==== Restoring from Bacula tape
Now you need to continue with restoring the actual Bacula backup, for this you
have multiple options of which +bextract+ is the most easy and
straightforward, but also the slowest and unsafest.


===== Using a bootstrap file
If you know the JobId of the latest successful full backup, and differential
backups the most efficient way to restore is by creating a bootstrap file with
this information and using it to restore from tape.

A bootstrap file looks like this:

----
Volume = VOL-1234
JobId = 914
Job = Bkp_Daily
----

or

----
Volume = VOL-1234
VolSessionId = 1
VolSessionTime = 108927638
----

Using a bootstrap file with bextract is easy, simply do:
+bextract -b bootstrap.txt Ultrium-1 /mnt/local+

TIP: It helps to know exactly how many files you need to restore, and using
     the +FileIndex+ and +Count+ keywords so +bextract+ does not require to
     read the whole tape. Use the commands in your shell history to access
     an example Bacula bootstrap file.


===== Using bextract
To use +bextract+ to restore *everything* from a single tape, you can do:
+bextract -V VOLUME-NAME Ultrium-1 /mnt/local+

----
rear> bextract -V VOL-1234 Ultrium-1 /mnt/local
bextract: match.c:249-0 add_fname_to_include prefix=0 gzip=0 fname=/
bextract: butil.c:282 Using device: "Ultrium-1" for reading.
30-Mar 16:00 bextract JobId 0: Ready to read from volume "VOL-1234" on device "Ultrium-1" (/dev/st0).
bextract JobId 0: -rw-r-----   1 252      bacula     3623795 2011-03-30 11:02:18  /mnt/local/var/lib/bacula/bacula.sql
bextract JobId 0: drwxr-xr-x   2 root     root          4096 2011-02-02 11:48:28  *none*
bextract JobId 0: drwxr-xr-x   4 root     root          1024 2011-02-23 13:09:53  *none*
bextract JobId 0: drwxr-xr-x  12 root     root          4096 2011-02-02 11:50:00  *none*
bextract JobId 0: -rwx------   1 root     root             0 2011-02-02 11:48:24  /mnt/local/.hpshm_keyfile
bextract JobId 0: -rw-r--r--   1 root     root             0 2011-02-22 12:38:03  /mnt/local/.autofsck
...
30-Mar 16:06 bextract JobId 0: End of Volume at file 7 on device "Ultrium-1" (/dev/st0), Volume "VOL-1234"
30-Mar 16:06 bextract JobId 0: End of all volumes.
30-Mar 16:07 bextract JobId 0: Alert: smartctl version 5.38 [x86_64-redhat-linux-gnu] Copyright (C) 2002-8 Bruce Allen
30-Mar 16:07 bextract JobId 0: Alert: Home page is http://smartmontools.sourceforge.net/
30-Mar 16:07 bextract JobId 0: Alert:
30-Mar 16:07 bextract JobId 0: Alert: TapeAlert: OK
30-Mar 16:07 bextract JobId 0: Alert:
30-Mar 16:07 bextract JobId 0: Alert: Error counter log:
30-Mar 16:07 bextract JobId 0: Alert:            Errors Corrected by           Total   Correction     Gigabytes    Total
30-Mar 16:07 bextract JobId 0: Alert:                ECC          rereads/    errors   algorithm      processed    uncorrected
30-Mar 16:07 bextract JobId 0: Alert:            fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
30-Mar 16:07 bextract JobId 0: Alert: read:       1546        0         0         0       1546          0.000           0
30-Mar 16:07 bextract JobId 0: Alert: write:         0        0         0         0          0          0.000           0
165719 files restored.
----

WARNING: In this case +bextract+ will restore all the Bacula jobs on the
         provided tapes, start from the oldest, down to the latest. As a
         consequence, deleted files may re-appear and the process may take
         a very long time.


==== Finish recovery process
Once finished, continue Rear by typing +exit+.

----
rear> exit
Did you restore the backup to /mnt/local ? Ready to continue ? y
Installing GRUB boot loader

Finished recovering your system. You can explore it under '/mnt/local'.

Finished in 4424 seconds.
----

IMPORTANT: If you neglect to perform this last crucial step, your new system
           will not boot and you have to install a boot-loader yourself
           manually, or re-execute this procedure.


=== USB storage as backup media

==== Configuring Rear for backup to USB storage device
The below configuration (_/etc/rear/local.conf_) gives a list of possible
options when you want to run Rear with USB storage.

[source,bash]
----
BACKUP=NETFS
OUTPUT=USB
USB_DEVICE=/dev/disk/by-label/REAR-000

### Exclude certain items
ONLY_INCLUDE_VG=( vg00 )
EXCLUDE_MOUNTPOINTS=( /data )
----


==== Making the DR backup to USB storage device
Creating a combined rescue device that integrates the backup on USB, it is
sufficient to run +rear -v mkbackup+ as shown below after you have inserted
the USB device. Make sure the device name for the USB device is what is
configured.

----
[root@system ~]# rear -v mkbackup
Relax and Recover Version 1.12.0svn497 / 2011-07-11
Creating disk layout.
Creating root FS layout
Copy files and directories
Copy program files and libraries
Copy kernel modules
Checking udev
Create initramfs
Creating archive 'usb:///dev/sda1/system.localdomain/20110326.0216/backup.tar.gz'
Total bytes written: 3644416000 (3.4GiB, 5.5MiB/s) in 637 seconds.
Writing MBR to /dev/sda
Modifying local GRUB configuration
Copying resulting files to usb location
Finished in 747 seconds.
----

IMPORTANT: It is advised to go into single user mode (+init 1+) before creating
           a backup to ensure all active data is consistent on disk (and no
           important processes are active in memory)


==== Booting from USB storage device
See the section <<booting-from-usb,Booting from USB storage device>> for more
information about how to enable your BIOS to boot from a USB storage device.


==== Restoring a backup from USB storage device
Then wait for the system to boot until you get the prompt.

On the shell prompt, type +rear recover+.

You may need to answer a few questions depending on your hardware
configuration and whether you are restoring to a (slightly)
different system.

----
RESCUE SYSTEM:/ # rear recover
Relax and Recover Version 1.12.0svn497 / 2011-07-11
Backup archive size is 1.2G (compressed)
To recreate HP SmartArray controller 1, type exactly YES: YES
To recreate HP SmartArray controller 7, type exactly YES: YES
Clearing HP SmartArray controller 1
Clearing HP SmartArray controller 7
Recreating HP SmartArray controller 1|A
Configuration restored successfully, reloading CCISS driver...  OK
Recreating HP SmartArray controller 7|A
Configuration restored successfully, reloading CCISS driver...  OK
Comparing disks.
Disk configuration is identical, proceeding with restore.
Start system layout restoration.
Creating partitions for disk /dev/cciss/c0d0 (msdos)
Creating partitions for disk /dev/cciss/c1d0 (msdos)
Creating software RAID /dev/md126
Creating software RAID /dev/md127
Creating LVM PV /dev/md127
Restoring LVM VG vg00
Creating ext3-filesystem / on /dev/mapper/vg00-lv00
Mounting filesystem /
Creating ext3-filesystem /boot on /dev/md126
Mounting filesystem /boot
Creating ext3-filesystem /data on /dev/mapper/vg00-lvdata
Mounting filesystem /data
Creating ext3-filesystem /opt on /dev/mapper/vg00-lvopt
Mounting filesystem /opt
Creating ext2-filesystem /tmp on /dev/mapper/vg00-lvtmp
Mounting filesystem /tmp
Creating ext3-filesystem /usr on /dev/mapper/vg00-lvusr
Mounting filesystem /usr
Creating ext3-filesystem /var on /dev/mapper/vg00-lvvar
Mounting filesystem /var
Creating swap on /dev/mapper/vg00-lvswap
Disk layout created.
Restoring from 'usb:///dev/sda1/system.localdomain/20110326.0216/backup.tar.gz'
Restored 3478 MiB in 134 seconds [avg 26584 KiB/sec]
Installing GRUB boot loader

Finished recovering your system. You can explore it under '/mnt/local'.

Finished in 278 seconds.
----

If all is well, you can now remove the USB device, restore the BIOS boot order
and reboot the system into the recovered OS.


== Using Rear with OBDR tapes
Using One-Button-Disaster-Recovery (OBDR) tapes has a few benefits.

 - Within large organisations tape media is already *part of a workflow*
   for offsite storage and is a *known and trusted technology*

 - Tapes can store large amounts of data reliably and restoring large
   amounts of data is *predictable* in time and effort

 - OBDR offers *booting from tapes*, which is very convenient

 - A single tape can hold both the rescue image as well as a *complete
   snapshot* of the system (up to 1.6TB with LTO4)

However, you need one tape per system as an OBDR tape can only store one
single rescue environment.


=== Configuring Rear for OBDR rescue tapes
The below configuration (_/etc/rear/local.conf_) gives a list of possible
options when you want to run Rear with a tape drive. This example shows how to
use the tape *only* for storing the rescue image, the backup is expected to be
handled by Bacula and so the Bacula tools are included in the rescue
environment to enable a Bacula restore.

[source,bash]
----
OUTPUT=OBDR
TAPE_DEVICE=/dev/nst0
----


=== Preparing your OBDR rescue tape
To protect normal backup tapes (in case tape drives are also used by another
backup solution) Rear expects that the tape to use is labeled *REAR-000*.
To achieve this is to insert a blank tape to use for Rear and run the
+rear format /dev/stX+ command.


=== OBDR tapes as rescue media

==== Configuring Rear to have Bacula tools
If the rescue environment needs additional tools and workflow, this can be
spcified by using +BACKUP=BACULA+ in the configuration file
_/etc/rear/local.conf_:

[source,bash]
----
BACKUP=BACULA
OUTPUT=OBDR
BEXTRACT_DEVICE=Ultrium-1
BEXTRACT_VOLUME=VOL-*
----

Using the +BEXTRACT_DEVICE+ allows you to use the tape device that is
referenced from the Bacula configuration. This helps in those cases where the
discovery of the various tape drives has already been done and configured in
Bacula.

The +BEXTRACT_VOLUME+ variable is optional and is only displayed in the
restore instructions on screen as an aid during recovery.


==== Making the OBDR rescue tape
To create a rescue environment that can boot from an OBDR tape, simply run
+rear -v mkrescue+ with a *REAR-000* -labeled tape inserted.

----
[root@system ~]# rear -v mkrescue
Relax and Recover Version 1.12.0svn497 / 2011-07-11
Rewinding tape
Writing OBDR header to tape in drive '/dev/nst0'
Creating disk layout.
Creating root FS layout
Copy files and directories
Copy program files & libraries
Copy kernel modules
Checking udev
Create initramfs
Making ISO image
Wrote ISO Image /tmp/rear-dag-ops.iso (48M)
Writing ISO image to tape
Modifying local GRUB configuration
Finished in 119 seconds.
----

WARNING: The message above about _/dev/cciss/c1d0_ not being used makes sense
as this is not a real disk but simply an entry for manipulating the controller.
This is specific to CCISS controllers with only a tape device attached.


[[booting-from-obdr]]
==== Booting from OBDR rescue tape
The One Button Disaster Recovery (OBDR) functionality in HP LTO Ultrium drives
enables them to emulate CD-ROM devices in specific circumstances (also known
as being in ''Disaster Recovery'' mode). The drive can then act as a boot
device for PCs that support booting off CD-ROM.

TIP: An OBDR capable drive can be switched into CD-ROM mode by *powering on
     with the eject button held down*. Make sure you keep it pressed when the
     tape drive regains power, and then release the button. If the drive is in
     OBDR mode, the light will blink regularly. This might be easier in some
     cases than the below procedure, hence the name One Button Disaster
     Recovery !


===== Using a HP Smart Array controller
To boot from OBDR, boot your system with the Rear tape inserted. During the
boot sequence, interrupt the HP Smart Array controller with the tape attached
by pressing *F8* (or *Escape-8* on serial console).

----
iLO 2 v1.78 Jun 10 2009 10.5.20.171

Slot 0 HP Smart Array P410i Controller       (512MB, v2.00)   1 Logical Drive
Slot 3 HP Smart Array P401 Controller        (512MB, v2.00)   1 Logical Drive
Slot 4 HP Smart Array P212 Controller          (0MB, v2.00)   0 Logical Drives
     Tape or CD-ROM Drive(s) Detected:
         Port 1I: Box 0: Bay 4
1785-Slot 4 Drive Array Not Configured
     No Drives Detected


  Press <F8> to run the Option ROM Configuration for Arrays Utility
  Press <ESC> to skip configuration and continue
----

Then select *Configure OBDR* in the menu and select the Tape drive by marking
it with *X* (default is on) and press *ENTER* and *F8* to activate this change
so it displays ''Configuration saved''.

Then press *ENTER* and *Escape* to leave the Smart Array controller BIOS.

----
**** System will boot from Tape/CD/OBDR device attached to Smart Array.
----


===== Using an LSI controller
To boot from OBDR when using an LSI controller, boot your system with the Rear
tape inserted. During the boot sequence, interrupt the LSI controller BIOS
that has the tape attached by pressing *F8* (or *Escape-8* on serial console).

----
LSI Logic Corp. MPT BIOS
Copyright 1995-2006 LSI Logic Corp.
MPTBIOS-5.05.21.00
HP Build

<<<Press F8 for configuration options>>>
----

Then select the option +1. Tape-based One Button Disaster Recovery (OBDR).+:

----
Select a configuration option:
1. Tape-based One Button Disaster Recovery (OBDR).
2. Multi Initiator Configuration.                                 <F9 = Setup>
3. Exit.
----

And then select the correct tape drive to boot from:

----
   compatible tape drives found       ->
   NUM   HBA   SCSI ID   Drive information
    0     0       A       - HP       Ultrium 2-SCSI

   Please choose the NUM of the tape drive to place into OBDR mode.
----

If all goes well, the system will reboot with OBDR-mode enabled:

----
    The PC will now reboot to begin Tape Recovery....
----

During the next boot, OBDR-mode will be indicate by:

----
*** Bootable media located, Using non-Emulation mode ***
----


===== Booting the OBDR tape
Once booted from the OBDR tape, select the 'Relax and Recover' menu entry from
the menu. If you don't press a key within 30 seconds, the system will try to
boot from the local disk.

----
      .-------------------------------------------------------------.
      |               Relax and Recover v1.12.0svn497               |
      |-------------------------------------------------------------|
      |  Relax and Recover                                          |
      |-------------------------------------------------------------|
      |  Other actions                                              |
      |   Help for Relax and Recover                                |
      (>  Boot Local disk (hd0)                                    <)
      |   Boot BIOS disk (0x80)                                     |
      |   Boot Next BIOS device                                     |
      |   Hardware Detection tool                                   |
      |   Memory test                                               |
      |   Reboot system                                             |
      |   Power off system                                          |
      `-------------------------------------------------------------'

             Press [Tab] to edit options or [F1] for Rear help

                        Automatic boot in 30 seconds...
----


==== Restoring the OBDR rescue tape
Then wait for the system to boot until you get the prompt.

On the shell prompt, type +rear recover+.

You may need to answer a few questions depending on your hardware
configuration and whether you are restoring to a (slightly)
different system.

----
RESCUE SYSTEM:/ # rear recover
Relax and Recover Version 1.12.0svn497 / 2011-07-11
NOTICE: Will do driver migration
Rewinding tape
To recreate HP SmartArray controller 3, type exactly YES: YES
To recreate HP SmartArray controller 0, type exactly YES: YES
Clearing HP SmartArray controller 3
Clearing HP SmartArray controller 0
Recreating HP SmartArray controller 3|A
Configuration restored successfully, reloading CCISS driver...  OK
Recreating HP SmartArray controller 0|A
Configuration restored successfully, reloading CCISS driver...  OK
Comparing disks.
Disk configuration is identical, proceeding with restore.
Type "Yes" if you want DRBD resource rBCK to become primary: Yes
Type "Yes" if you want DRBD resource rOPS to become primary: Yes
Start system layout restoration.
Creating partitions for disk /dev/cciss/c0d0 (msdos)
Creating partitions for disk /dev/cciss/c2d0 (msdos)
Creating software RAID /dev/md2
Creating software RAID /dev/md6
Creating software RAID /dev/md3
Creating software RAID /dev/md4
Creating software RAID /dev/md5
Creating software RAID /dev/md1
Creating software RAID /dev/md0
Creating LVM PV /dev/md6
Creating LVM PV /dev/md5
Creating LVM PV /dev/md2
Creating LVM VG vgrem
Creating LVM VG vgqry
Creating LVM VG vg00
Creating LVM volume vg00/lv00
Creating LVM volume vg00/lvdstpol
Creating LVM volume vg00/lvsys
Creating LVM volume vg00/lvusr
Creating LVM volume vg00/lvtmp
Creating LVM volume vg00/lvvar
Creating LVM volume vg00/lvopt
Creating ext3-filesystem / on /dev/mapper/vg00-lv00
Mounting filesystem /
Creating ext3-filesystem /dstpol on /dev/mapper/vg00-lvdstpol
Mounting filesystem /dstpol
Creating ext3-filesystem /dstpol/sys on /dev/mapper/vg00-lvsys
Mounting filesystem /dstpol/sys
Creating ext3-filesystem /usr on /dev/mapper/vg00-lvusr
Mounting filesystem /usr
Creating ext2-filesystem /tmp on /dev/mapper/vg00-lvtmp
Mounting filesystem /tmp
Creating ext3-filesystem /boot on /dev/md0
Mounting filesystem /boot
Creating ext3-filesystem /var on /dev/mapper/vg00-lvvar
Mounting filesystem /var
Creating ext3-filesystem /opt on /dev/mapper/vg00-lvopt
Mounting filesystem /opt
Creating swap on /dev/md1
Creating DRBD resource rBCK
Writing meta data...
initializing activity log
New drbd meta data block successfully created.
Creating LVM PV /dev/drbd2
Creating LVM VG vgbck
Creating LVM volume vgbck/lvetc
Creating LVM volume vgbck/lvvar
Creating LVM volume vgbck/lvmysql
Creating ext3-filesystem /etc/bacula/cluster on /dev/mapper/vgbck-lvetc
Mounting filesystem /etc/bacula/cluster
Creating ext3-filesystem /var/bacula on /dev/mapper/vgbck-lvvar
Mounting filesystem /var/bacula
Creating ext3-filesystem /var/lib/mysql/bacula on /dev/mapper/vgbck-lvmysql
Mounting filesystem /var/lib/mysql/bacula
Creating DRBD resource rOPS
Writing meta data...
initializing activity log
New drbd meta data block successfully created.
Creating LVM PV /dev/drbd1
Creating LVM VG vgops
Creating LVM volume vgops/lvcachemgr
Creating LVM volume vgops/lvbackup
Creating LVM volume vgops/lvdata
Creating LVM volume vgops/lvdb
Creating LVM volume vgops/lvswl
Creating LVM volume vgops/lvcluster
Creating ext3-filesystem /opt/cache on /dev/mapper/vgops-lvcachemgr
Mounting filesystem /opt/cache
Creating ext3-filesystem /dstpol/backup on /dev/mapper/vgops-lvbackup
Mounting filesystem /dstpol/backup
Creating ext3-filesystem /dstpol/data on /dev/mapper/vgops-lvdata
Mounting filesystem /dstpol/data
Creating ext3-filesystem /dstpol/databases on /dev/mapper/vgops-lvdb
Mounting filesystem /dstpol/databases
Creating ext3-filesystem /dstpol/swl on /dev/mapper/vgops-lvswl
Mounting filesystem /dstpol/swl
Creating ext3-filesystem /dstpol/sys/cluster on /dev/mapper/vgops-lvcluster
Mounting filesystem /dstpol/sys/cluster
Disk layout created.

The system is now ready to restore from Bacula. You can use the 'bls' command
to get information from your Volume, and 'bextract' to restore jobs from your
Volume. It is assumed that you know what is necessary to restore - typically
it will be a full backup.

You can find useful Bacula commands in the shell history. When finished, type
'exit' in the shell to continue recovery.

WARNING: The new root is mounted under '/mnt/local'.

rear>
----


==== Restoring from Bacula tape
See the section <<restoring-from-bacula-tape,Restoring from Bacula tape>>
for more information about how to restore a Bacula tape.


=== OBDR tapes as backup media
An OBDR backup tape is similar to an OBDR rescue tape, but next to the rescue
environment, it also consists of a complete backup of the system. This is
very convenient in that a single tape can be use for disaster recovery, and
recovery is much more simple and completely automated.

CAUTION: Please make sure that the system fits onto a single tape uncompressed.
         For an LTO4 Ultrium that would mean no more than 1.6TB.


==== Configuring Rear for OBDR backup tapes
The below configuration (_/etc/rear/local.conf_) gives a list of possible
options when you want to run Rear with a tape drive. This example shows how to
use the tape for storing *both* the rescue image and the backup.

[source,bash]
----
BACKUP=NETFS
OUTPUT=OBDR
TAPE_DEVICE=/dev/nst0
----


==== Making the OBDR backup tape
To create a bootable backup tape that can boot from OBDR, simply run
+rear -v mkbackup+ with a *REAR-000* -labeled tape inserted.

----
[root@system ~]# rear -v mkbackup
Relax and Recover Version 1.12.0svn497 / 2011-07-11
Rewinding tape
Writing OBDR header to tape in drive '/dev/nst0'
Creating disk layout
Creating root FS layout
Copy files and directories
Copy program files & libraries
Copy kernel modules
Checking udev
Create initramfs
Making ISO image
Wrote ISO Image /tmp/rear-system.iso (45M)
Writing ISO image to tape
Creating archive '/dev/nst0'
Total bytes written: 7834132480 (7.3GiB, 24MiB/s) in 317 seconds.
Rewinding tape
Modifying local GRUB configuration
Finished in 389 seconds.
----

IMPORTANT: It is advised to go into single user mode (+init 1+) before creating
           a backup to ensure all active data is consistent on disk (and no
           important processes are active in memory)


==== Booting from OBDR backup tape
See the section <<booting-from-obdr,Booting from OBDR rescue tape>> for more
information about how to enable OBDR and boot from OBDR tapes.


==== Restoring from OBDR backup tape

----
RESCUE SYSTEM:~ # rear recover
Relax and Recover Version 1.12.0svn497 / 2011-07-11
NOTICE: Will do driver migration
Rewinding tape
To recreate HP SmartArray controller 3, type exactly YES: YES
To recreate HP SmartArray controller 0, type exactly YES: YES
Clearing HP SmartArray controller 3
Clearing HP SmartArray controller 0
Recreating HP SmartArray controller 3|A
Configuration restored successfully, reloading CCISS driver...  OK
Recreating HP SmartArray controller 0|A
Configuration restored successfully, reloading CCISS driver...  OK
Comparing disks.
Disk configuration is identical, proceeding with restore.
Type "Yes" if you want DRBD resource rBCK to become primary: Yes
Type "Yes" if you want DRBD resource rOPS to become primary: Yes
Start system layout restoration.
Creating partitions for disk /dev/cciss/c0d0 (msdos)
Creating partitions for disk /dev/cciss/c2d0 (msdos)
Creating software RAID /dev/md2
Creating software RAID /dev/md6
Creating software RAID /dev/md3
Creating software RAID /dev/md4
Creating software RAID /dev/md5
Creating software RAID /dev/md1
Creating software RAID /dev/md0
Creating LVM PV /dev/md6
Creating LVM PV /dev/md5
Creating LVM PV /dev/md2
Restoring LVM VG vgrem
Restoring LVM VG vgqry
Restoring LVM VG vg00
Creating ext3-filesystem / on /dev/mapper/vg00-lv00
Mounting filesystem /
Creating ext3-filesystem /dstpol on /dev/mapper/vg00-lvdstpol
Mounting filesystem /dstpol
Creating ext3-filesystem /dstpol/sys on /dev/mapper/vg00-lvsys
Mounting filesystem /dstpol/sys
Creating ext3-filesystem /usr on /dev/mapper/vg00-lvusr
Mounting filesystem /usr
Creating ext2-filesystem /tmp on /dev/mapper/vg00-lvtmp
Mounting filesystem /tmp
Creating ext3-filesystem /boot on /dev/md0
Mounting filesystem /boot
Creating ext3-filesystem /var on /dev/mapper/vg00-lvvar
Mounting filesystem /var
Creating ext3-filesystem /opt on /dev/mapper/vg00-lvopt
Mounting filesystem /opt
Creating swap on /dev/md1
Creating DRBD resource rBCK
Writing meta data...
initializing activity log
New drbd meta data block successfully created.
Creating LVM PV /dev/drbd2
Restoring LVM VG vgbck
Creating ext3-filesystem /etc/bacula/cluster on /dev/mapper/vgbck-lvetc
Mounting filesystem /etc/bacula/cluster
Creating ext3-filesystem /var/bacula on /dev/mapper/vgbck-lvvar
Mounting filesystem /var/bacula
Creating ext3-filesystem /var/lib/mysql/bacula on /dev/mapper/vgbck-lvmysql
Mounting filesystem /var/lib/mysql/bacula
Creating DRBD resource rOPS
Writing meta data...
initializing activity log
New drbd meta data block successfully created.
Creating LVM PV /dev/drbd1
Restoring LVM VG vgops
Creating ext3-filesystem /opt/cache on /dev/mapper/vgops-lvcachemgr
Mounting filesystem /opt/cache
Creating ext3-filesystem /dstpol/backup on /dev/mapper/vgops-lvbackup
Mounting filesystem /dstpol/backup
Creating ext3-filesystem /dstpol/data on /dev/mapper/vgops-lvdata
Mounting filesystem /dstpol/data
Creating ext3-filesystem /dstpol/databases on /dev/mapper/vgops-lvdb
Mounting filesystem /dstpol/databases
Creating ext3-filesystem /dstpol/swl on /dev/mapper/vgops-lvswl
Mounting filesystem /dstpol/swl
Creating ext3-filesystem /dstpol/sys/cluster on /dev/mapper/vgops-lvcluster
Mounting filesystem /dstpol/sys/cluster
Disk layout created.
Restoring from 'tape:///dev/nst0/system/backup.tar'
Restored 7460 MiB in 180 seconds [avg 42444 KiB/sec]
Installing GRUB boot loader

Finished recovering your system. You can explore it under '/mnt/local'.

Finished in 361 seconds.
----

== Tips and tricks using Rear
Recovering a system should not be a struggle against time with poor
tools weighing against you. Rear emphasizes on a relaxing recovery,
and for this it follows three distinct rules:

 1. Do what is generally expected, if possible

 2. In doubt, ask the operator or allow intervention

 3. Provide an environment that is as convenient as possible

This results in the following useful tips and tricks:

[TIP]
====
 1. Rear adds its own GRUB menu entry to your local system's GRUB,
    which is convenient to restore a system without the need for an
    additional boot media. This only works if your system can (still)
    boot from the local disk and the disaster didn't destroy that disk.
    To protect accidental recovery, you need to enter the password:
    *Rec0ver1t*

 2. Rear automatically detects and enables serial console support. This is
    extremely useful if the only way to access the console during disaster
    is a Java-based console riddled with keyboard bugs and slow screen
    refreshes.

 3. Rear conveniently ships with a set of useful commands in its shell
    history. This makes it possible to quickly look for a command to help
    troubleshoot, or modify an important file during recovery.

 4. If you have keyboard problems using a HP iLO 2 Remote Console, type the
    following command +loadkeys -d+ (or simply use the up arrow key to get
    this command from your shell history).

 5. If the original system was configured to log on remotely through the use
    of SSH keys, Rear preserved those keys on the rescue environment and you
    can access the rescue environment from the network as you were used to
    before.

 6. During recovery at any stage you can re-run Rear, modify the layout file
    for recreating the structure and intervene when restoring the backup.
====


== Monitoring your system with Rear
If Rear is not in charge of the backup, but only for creating a rescue
environment, it can be useful to know when a change to the system invalidates
your existing/stored rescue environment, requiring one to update the rescue
environment.

For this, Rear has two different targets, one to create a new baseline (which
is automatically done when creating a new rescue environment successfully. And
one to verify the (old) baseline to the current situation.

With this, one can monitor or automate generating new rescue environments only
when it is really needed.


=== Creating a baseline
Rear automatically creates a new baseline as soon as it successfully has
created a new rescue environment. However if for some reason you want to
recreate the baseline manually, use +rear savelayout+.


=== Detecting changes to the baseline
When you want to know if the latest rescue environment is still valid, you may
want to use the +rear checklayout+ command instead.

----
[root@system ~]# rear checklayout
[root@system ~]# echo $?
0
----

If the layout has changed, the return code will indicate this by a non-zero
return code.

----
[root@system ~]# rear checklayout
[root@system ~]# echo $?
1
----


=== Integration with Nagios and Opsview
If having current DR rescue images is important to your organisation, but they
cannot be automated (eg. a tape or USB device needs inserting), we provide a
Nagios plugin that can send out a notification whenever there is a critical
change to the system that requires updating your rescue environment.

Changes to the system requiring an update are:

 - Changes to hardware RAID
 - Changes to software RAID
 - Changes to partitioning
 - Changes to DRBD configuration
 - Changes to LVM
 - Changes to filesystems

The integration is done using our own _check_rear_ plugin for Nagios.

[source,bash]
----
#!/bin/bash
#
# Purpose: Checks if disaster recovery usb stick is up to date

# Check if rear is installed
if [[ ! -x /usr/sbin/rear ]]; then
    echo "REAR IS NOT INSTALLED"
    exit 2
fi

# rear disk layout status can be identical or changed
# returncode: 0 = ok
if ! /usr/sbin/rear checklayout; then
    echo "Disk layout has changed. Please insert Disaster Recovery USB stick into system !"
    exit 2
fi
----

We also monitor the _/tmp/rear-system.log_ file for +ERROR:+ and +BUG BUG BUG+
strings, so that in case of problems the operator is notified immediately.


== Tested configurations
During the development of Rear we have tested Rear with the following
configurations:

 - With USB storage as rescue and Bacula backup on tape
 - With USB storage as rescue and backup on same USB storage
 - With OBDR tape as rescue and Bacula backup on separate tape
 - With OBDR tape as rescue and backup on same tape

And in the following ways:

 - Booting from second disk in SW RAID1 setup
 - Restoring to different storage configuration
 - Restoring to different hardware
 - Restoring to bad HP SmartArray config


== Design and implementation
// FIXME: This chapter explains the concepts and implementation details
//        This should include the layout information from Jeroen


== Troubleshooting Rear
If you encounter a problem, you may find more information in the log file
which is located at _/tmp/rear-system.log_. During recovery the backup log file
is also available from _/tmp_, for your convenience the history in rescue mode
comes with useful commands for debugging, use the up arrow key in the shell
to find those commands.

There are a few options in Rear to help you debug the situation:

 - use the +-v+ option to show progress output during execution
 - use the +-d+ option to have debug information in your log file
 - use the +-s+ option to see what scripts Rear would be using; this is useful
   to understand how Rear is working internally
 - use the +-S+ option to step through each individual script when
   troubleshooting
 - use the +-D+ option to dump every function call to log file; this is very
   convenient during development or when troubleshooting Rear


=== During backup
During backup Rear creates a description of your system layout in one file
(_disklayout.conf_) and stores this as part of its rescue image. This file
describes the configuration of SmartArray RAID, parititions, software RAID,
DRBD, logical volumes, filesystems, and possibly more.

Here is a list of known issues during backup:

[qanda]
One or more HP SmartArray controllers have errors::
+
--
Rear had detected that one of your HP SmartArray controllers is in ERROR
state and as a result it can not trust the information returned from that
controller. This can be dangerous because we cannot guarantee that the
disk layout is valid when recovering the system.

We discovered that this problem can be caused by a controller that still
has information in its cache that has not been flushed and the only way to
solve it was to reboot the system and pressing F2 during the controller
initialisation when it reports this problem.
--

USB sticks disappear and re-appear with difference device name::
+
--
We have had issues before with a specific batch of JetFlash USB sticks
which, during write operations, reset the USB controller because of a bug
in the Linux kernel. The behaviour is that the device disappears (during
write operations!) and reappears with a different device name. The result
is that the filesystem becomes corrupt and the stick cannot be used.

To verify if the USB stick has any issues like this, we recommend using
the +f3+ tool on Linux or the +h2testw+ tool on Windows. If this tool
succeeds in a write and verify test, the USB stick is reliable.
--


=== During recovery
During restore Rear uses the saved system layout as the basis for recreating a
workable layout on your new system. If your new hardware is very different,
it's advised to copy the layout file _/var/lib/rear/layout/disklayout.conf_
to _/etc/rear_ and modify it according to what is required.

----
cp /var/lib/rear/layout/disklayout.conf /etc/rear/
vi /etc/rear/disklayout.conf
----

Then restart the recovery process: +rear recover+

During the recovery process, Rear translates this layout file into a shell
procedure (_/var/lib/rear/layout/diskrestore.sh_) that contains all the needed
instructions for recreating your desired layout.

If Rear comes across irreconcilable differences, it provides you with a small
menu of options you have. In any case you can Abort the menu, and retry after
cleaning up everything Rear may already have done, incl. +mdadm --stop
--scan+ or +vgchange -a n+.

In any case, you will have to look into the issue, see what goes wrong and
either fix the layout file (_disklayout.conf_) and restart the recovery
process (+rear recover+) or instead fix the shell procedure (_diskrestore.sh_)
and choose +Retry+.

WARNING: Customizations to the shell procedure (_diskrestore.sh_) get
         lost when restarting +rear recover+.


Here is a list of known issues during recovery:

[qanda]
Failed to clear HP SmartArray controller 1::
+
--
This error may be caused by trying to clear an HP SmartArray controller
that does not have a configuration or does not exist. Since we have no
means to know whether this is a fatal condition or not we simply try to
recreate the logical drive(s) and see what happens.

This message is harmless, but may help troubleshoot the subsequent error
message.
--

An error has been detected during restore::
+
--
The (generated) layout restore script _/var/lib/rear/layout/diskrestore.sh_
was not able to perform all necessary steps without error. The system will
provide you with a menu allowing you to fix the _diskrestore.sh_ script
manually and continue from where it left off.

  Cannot create array. Cannot add physical drive 2I:1:5
  Could not configure the HP SmartArray controllers

When the number of physical or logical disks are different, or when other
important system characteristics that matter to recovery are incompatible,
this will be indicated by a multitude of possible error-messages. Rear
makes it possible to recover also in these cases by hand.

You can find more information about your HP SmartArray setup by running one of
the following commands:

    # hpacucli ctrl all show detail
    # hpacucli ctrl all show config
    # hpacucli ctrl all show config detail

TIP: You can find these commands as part of the history of the Rear shell.
--
