
   Synchronizing Disks for HA

   What is this document?
   This  is  only  ONE  way  to keeps nodes in synchronization for a high
   availability cluster.  It suits our needs and may be a good
   starting block for others so I'm sharing it.  Your mileage may vary.

   We used this on a web server to update ~1GB worth of documents in sync
   every 10 minutes.  Depending on how quickly your files change, you may
   be  able  to update more data, more frequently.  I would not recommend
   using  this without a private fast ethernet channel, i.e. an extra NIC
   in each node connected via a crossover cable.
   
   To determine whether this method would be appropriate for you, you may
   wish to do this:
    1. Set the minimum update rate you find acceptable (say, 3 minutes).
    2. Conservatively  approximate the amount of data you would expect to
       change in 3 minutes (200MB)
    3. Assuming  you  can  count  on  6MB/sec  from  your  fast ethernet,
       multiply  this  by  1/3  of  your  update time.  Why?  You need to
       account  for  rsync  (relatively slow executing) to determine what
       needs  updating  and  disk write time.  In our example, this would
       give us 360MB worth of updates per cycle.

   Of  course,  you  can  always  do what I did - set it up and see if it
   updates  fast enough.  The scripts I've provided will notify you if it
   kicks off again before the previous one
   finished.......

   BTW,  for lots of data, or rapidly changing data, you'll want a shared
   disk  or  use [1]drbd.  However, we find rsync is still very useful to
   keep  config  files,  passwd  files,  etc.  consistent  and  use it in
   conjuction with drbd.
   
   One  last  item:   rsync  with the --delete option is dangerous!  Make
   sure  your command options point to the proper destination!  Test this
   on non-critical data.  And then test it again.

   What you'll need:

       1.  OpenSSH
                   You can get it at:
   [2]http://www.openssh.com/portable.html
       2.  A copy of rsync
                   Found at
   [3]http://rsync.samba.org/ftp/rsync/binaries/                     -OR-
   [4]http://rsync.samba.org/ftp/rsync
      3.   Cron and stuff to mirror.

       In actuality, you don't even need SSH.  You could use rsh instead,
   if your security needs permit it.

   Installing SSH

   START HERE for source distribution:
   Untar  your  openssh distribution.  You might want to read the INSTALL
   file or HOWTO, but most can get by with the following:
              ./configure
        make
        make install

   START   HERE   for   rpm   distribution   after   running   "rpm  -ivh
   openssh_<version#>.rpm":
   Once this is done, make sure sshd is started on bootup.  This could be
   done  via  init.d  scripts  or  by placing (if ssh is installed to the
   default  location)  "/usr/local/sbin/sshd"  in  you /etc/rc.d/rc.local
   file.  Type this in now to start it.
   NOTE:        YOU SHOULD READ THE OPENSSH DOCUMENTATION.  THE FOLLOWING
   INSTRUCTIONS WILL INSTRUCT YOU
   ON  ONE  WAY  TO  SET  UP  AUTO-AUTHENTICATION  BETWEEN  YOUR  CLUSTER
   NODES....BUT YOU SHOULD UNDERSTAND WHAT
   YOU ARE DOING!!!

   Make  sure  that /usr/local/bin is in your path and type:  "ssh-keygen
   -d"     This  will  create your ssh "key".   Do not enter a passphrase
   (hit return).
   In  your  ~/.ssh  directory,  there  will  be two files:  "id_dsa" and
   "id_dsa.pub".   "id_dsa"  is  your  private  key, "id_dsa.pub" is your
   public key (please refer to your
   ssh documentation for explanations).  Run the following commands:

   cp id_dsa.pub authorized_keys2

   chmod 400 authorized_keys2

   Now  copy  the  entire .ssh directory (preserving permissions) to your
   home directory on the other node in the cluster.
   You  should be all set.  Try it out by typing on node A:  "ssh nodeB".
   You  should  be  logged  in  to  nodeB without having to type anything
   (except "yes" to accept
   the host key the very first time you connect....)


   Installing Rsync

   Well,  I'm  not  really going to tell you how to do this.  I just used
   the  rpm.   If that's not possible for you, I'm sure the good folks at
   samba will have a nice README.Install for you to follow.  However, the
   binary  link  in  the "What you'll need" section has binaries for just
   about    all    flavors.     Here's    the    link    for   the   rpm:
   [5]http://rsync.samba.org/ftp/rsync/binaries/redhat

   Determining your Rsync command

   For  our  example,  let's  say  you  have  a web server cluster.  As a
   result,  you  need  the  directory  tree "/html" to be current on both
   nodes.   Assuming  node  A is the master, I would use the command from
   node A:
    rsync --rsh=/usr/local/bin/ssh -naurvl --delete /html/ localnetB:/html

   Let's  note  a  few things.  First, since this will be used with cron,
   you  want to be sure that you use the full path for the ssh executable
   (and  the  rsync executable for that matter).  Also, note the "/html/"
   syntax.   This  last "/" is necessary - otherwise you'll have the tree
   "/html/html"  on your slave.  Lastly, I used the "-n" option.  This is
   for  a  dry  run.   You  want  to  do  this to make sure everything is
   copied/deleted  as  you  would  expect  it.  When you put this in your
   crontab  file,  you'll leave off the "-n" option.  Similarly, the "-v"
   verbosity  option  is  only  for this test.  In your crontab entry, it
   will be replaced with the "-q" option for quiet.  Test the command now
   and  make  sure  it  does what you want - the "--delete" option can be
   dangerous!

   Create your sync script and crontab entry

   You now want to create the script which will run rsync every X minutes
   via cron.  Our solution has the same script running on both nodes, but
   it  checks  whether  it  is  running  on  the current primary (the one
   holding  the  services)  or not.  If it's running on the secondary, it
   quits   immediately.    I   use  the  following  perl  script,  called
   "synch_all.pl":

   #!/usr/bin/perl
   use strict;
   use diagnostics;
   use lib "/root/scripts";
   use whohostlib;
   use EnvConfig;
   use Mail::Sendmail;
   #Take care of two-way syncing case during shutdown of one server.
   Skips the first try to sync after failover.
   if ( -e "/var/lock/subsys/mirrorstop" ) {
       unlink "/var/lock/subsys/mirrorstop";
       exit;
   }
   # Only sync if serving IP
   my $dirname = "";
   my $filename = "";
   #Get server that this is running on...
   my $me = "";
   my $other = "";
   ($me,$other) = whohostlib::whohost;
   if (&whohostlib::whostatus() == 1)
   {
       #I'm serving!  You want fries with that?

       #Only sync if other node is running heartbeat:
       my $tst = `$EnvConfig::sshpath local$other
   /etc/rc.d/init.d/heartbeat status`;
       # For 0.4.9.2 and earlier use  --->    if ($tst =~ /running.../) {
       # For 0.4.9 "beta" versions...
       if ($tst =~ /heartbeat OK/) {
           #make sure previous mirror has completed...
           #To avoid hard link probs over partitions, created
   /var/lock/subsys/.DONOTREMOVE file
           if (link("/var/lock/subsys/.DONOTREMOVE",
   "/var/lock/subsys/mirror")) {
               # /home/ -  Home Directories (all users)
               # /root/scripts/ - Root scripts/system scripts - like
   where we put this script...
               # /var/spool/cron/ - System crontabs
                   my @dirlist = ( "/home/", "/root/scripts/",
   "/var/spool/cron/" );
                   foreach $dirname (@dirlist) {
                       system "/usr/bin/rsync --rsh=$EnvConfig::sshpath
   $EnvConfig::rsyncoptions $dirname local$other:$dirname";
               }
               # /etc/password - System Password File
               # /etc/shadow   - Actual Encrypted passwords.
               # /etc/group    - System Group File
               # /etc/ld.so.conf - System Linked Libraries.
               # /etc/shells     - Valid Login Shells
                   my @filelist = ( "/etc/passwd", "/etc/shadow",
   "/etc/group", "/etc/ld.so.conf", "/etc/shells" );
                   foreach $filename (@filelist) {
                       system "/usr/bin/rsync --rsh=$EnvConfig::sshpath
   $EnvConfig::rsyncoptions $filename local$other:$filename";
                   }
               # release rsync process
               unlink "/var/lock/subsys/mirror";
           } else {
               my $ddd = `date`;
               my $subject = "Next RSYNC process starting before previous
   has completed!\n";
               my $message = $subject.$ddd;
               sendmail ('smtp' => $EnvConfig::smtphost,
                   'To' => 'admin@domain.com,admin2@domain.com',
                   'From' => 'cluster@domain.com',
                   'Reply-To' => 'cluster@domain.com',
                   'Subject' => $subject,
                   'Message' => $message)
               or warn "Next RSYNC process starting before previous has
   completed (Couldn't send email)";
               #Try to get to known state by killing all synch procs and
   removing link...
               unlink "/var/lock/subsys/mirror";
               exec "/root/scripts/kill_synchs.sh";
               #"kill_synchs.sh is a one liner:  ps ax | grep synch_all |
   awk {'print $1'} | xargs -n1 kill -9
               exit;
           }
       }
      }

   Mail::Sendmail is a standard Perl module you can get from [6]CPAN, but
   you'll  notice two homemade ones.  There's no magic here, just decided
   to  use them for portability and readability (yes, I know, even though
   it's perl ;-)).  The "EnvConfig" one lets us set the paths and options
   we're using external to the script so that the
   actual  synch_all.pl  script  doesn't  change.  You could even add the
   file and directory lists to this, but ours doesn't change much cluster
   to  cluster.   The  "whohostlib"  has  two  functions used often.  One
   checks  which host the script is running on, and the other (whostatus)
   checks whether the node running the script is holding the services.

   # Configuration file for all scripts, including host definitions, etc.
   package EnvConfig;
   #www_host = "wwwdev.domain.com";
   $www_host = "wwwint.domain.com";
   #www_host = "wwwprod.domain.com";
   $sshpath = "/usr/bin/ssh";
   $rsyncoptions = "-aurlq --delete";
   #$rsyncoptions = "-aurlv";
   $smtphost = 'mail.domain.com';
   $Debug = 0;
      1;

   #!/usr/bin/perl
   use strict;
   use diagnostics;
   package whohostlib;
   sub whohost {
       my $other;
       my $me;
       #Who am I?
       $me = `hostname -s`;
       chomp $me;
       $me =~ s/[a-zA-Z]//g;

       if ($me) {
          $other = 0;
       } else {
          $other = 1;
       }
       return($me,$other);
   }
   #By doing "grep mirror haresources", we peel off the line with the
   info we want...the cluster IP.  Mirror will be added to you
   haresources later...
   sub whostatus {
       my $status = 0;
       my $x = `grep mirror /etc/ha.d/haresources`;
       my @p = split(' ',$x);
       my $ip = $p[1];
       $ip =~ s/IPaddr:://g;
       $x = `/etc/ha.d/resource.d/IPaddr $ip status`;
       # If I am primary, return 1 else return 0
       if ($x =~ /running/) {
          $status = 1;
       } else {
          $status = 0;
       }
       return $status;
   }
      1; #Return True Value

   NOTE:  We use a convention for our clusters that if the clustername is
   "foo"  the two nodes that make up the cluster will be named "foo0" and
   "foo1".   The private data interfaces are given the hostnames "local0"
   and  "local1".   This makes stuff easier with scripting, but you don't
   have to do it.
   NOTE  #2:  You may not want a mail fired off.  You may just want a log
   entry.  You might want both.  If you're unfamiliar with perl, to add a
   log  entry, substitute all the commands with "MAIL" in them above with
   the following:

   open(LOG,">>/var/log/ha-log");
   $dstr = `date +%Y/%m/%d_%T`;
   chomp $dstr;
   print LOG "$dstr RSYNC: Process starting before previous one
   completes!\n";
   close LOG;

   At  this  point  you want to create your crontab entry.  The hard link
   prevents more than one sync process from running at the same time, but
   you  want  to  have a decent idea of how often you need to synchronize
   and how long it will take to synchronize.

   SO,  once  you  determine  how  often  you'll  be  synchronizing, type
   "crontab  -e" to modify your crontab entry.  If you don't like vi, try
   using  "setenv EDITOR /usr/local/bin/emacs" (or export for bash users)
   to  select  emacs  or  a different editor.  If you want to synchronize
   every 10 minutes, your entry would look like this:

*/10 * * * * /script_directory/sync.pl &> /dev/null

   You could also redirect output to some logfile if you desire, but keep
   in mind how often it runs.

   Dealing with Failover

   We're  just  about  there now.  The last thing you need to consider is
   when you shutdown.  You want to do one last synchronization before you
   shut  down.   I  accomplished  this  with  a  heartbeat service called
   "mirror".  On startup, we do nothing other than prevent the very first
   synch_all  (as a precaution) and notify the necessary admins about the
   failover.

   NOTE:    You will want to be sure that any applications writing to the
   synch'ed  areas stop before your last sync.  If these applications are
   controlled via the ipresources configuration file, you can ensure this
   by  listing the "mirror" script right after your IP address.  However,
   if  they  are  not,  you  may  want  to add an application exit to the
   beginning of your mirror script.

   #!/usr/bin/perl
   use lib "/root/scripts";
   use whohostlib;
   # See how we were called.
   if ($#ARGV == 0) {
       $switch = $ARGV[0];
   } else {
       print "Usage: mirror {start|stop|status|restart}\n";
       exit -1;
   }
   #  start)
   if ($switch eq "start") {
       #Make sure to skip first sync.....
       ($me,$other) = whohostlib::whohost;
       system ("/usr/bin/ssh local$me /bin/touch
   /var/lock/subsys/mirrorstop");
       #Notify admins of startup....
       open (XXX,"/root/scripts/.pagelist");
       $hh = `hostname`;
       chomp $hh;
       while (<XXX>) {
       chomp;
       system "/root/scripts/pagescript.pl $_ \"Starting up heartbeat
   services on node $hh\"";
       }
       close XXX;

   } elsif ($switch eq "stop") {
       #stop)
       print "Mirror stop: \n";
       #Am I serving???
       #We'll consider mirror to be "running" if the IP is on this box...
       $x = `grep mirror /etc/ha.d/haresources`;
       @p = split(' ',$x);
       $ip = $p[1];
       $x = `/etc/ha.d/resource.d/IPaddr $ip status`;
       if ($x =~ /running/) {
       #Notify admins of shutdown....
       open (XXX,"/root/scripts/.pagelist");
       $hh = `hostname`;
       chomp $hh;
       while (<XXX>) {
           chomp;
           system "/root/scripts/pagescript.pl $_ \"Shutting down
   heartbeat services on node $hh\"";
       }
       close XXX;
       #You want fries with that?  One last sync to other box, if not in
   the middle of one now...
       if ( -e "/var/lock/subsys/mirror" ) {
           print "Not syncing, already doing so...\n";
       } else {
           print "Synchronizing data on standby node: \n";
           system "/root/scripts/synch_all.pl";
       }
       # To prevent NEW master from syncing back to us too soon, lock
   their first synch_all...
       ($me,$other) = whohostlib::whohost;
       system ("/usr/bin/ssh local$other /bin/touch
   /var/lock/subsys/mirrorstop");
       }

   } elsif ($switch eq "status") {
       #status)
       #We'll consider mirror to be "running" if the IP is on this box...
       $y = `grep mirror /etc/ha.d/haresources`;
       @p = split(' ',$y);
       $ip = $p[1];

       $x = `/etc/ha.d/resource.d/IPaddr $ip status`;
       print $x;
   } elsif ($switch eq "restart") {
       #restart)

       system "/etc/ha.d/resource.d/mirror stop";
       system "/etc/ha.d/resource.d/mirror start";

   } else {
       print "Usage: mirror {start|stop|status|restart}\n";
   }
      exit;

   Finally,  you  need  to install the script in your ipresources on both
   nodes.   You'll want mirror to be the first service listed (after IP).
   For our webserver, it would read:
    nodeA 192.168.85.1 mirror httpd



   Be Careful

   Please  test  your setup on non-critical data first.  There could be a
   bad  typo  above or whatever.  The "--delete" option can be dangerous.
   You've been warned.

   Rev 1.0.0
   Rudy Pawul
   rpawul@iso-ne.com

参照

   1. http://www.complang.tuwien.ac.at/reisner/drbd/
   2. http://www.openssh.com/portable.html
   3. http://rsync.samba.org/ftp/rsync/binaries/
   4. http://rsync.samba.org/ftp/rsync/rsync-2.3.1.tar.gz
   5. http://rsync.samba.org/ftp/rsync/binaries/redhat
   6. file://localhost/tmp/heartbeat-2.1.4-1/doc/www.cpan.org
