Example, bash scrip to check remote server up status via ssh and alert

[dstewart]

I have several large NAS's with mirrored backups. The problem was, as long as the main servers are up the backups may go unnoticed and could go down with out anyone knowing. So I came up with a little bash script to run via cron twice a day to check network status, ssh, and raid volume access. I cron rsync over ssh to backup the main servers so testing the ssh connectivity was important. If any any problems are found the main server will issue an email alert. I spent a few hours searching and many hours experimenting so hopefully posting this here can help someone else along.

#! /bin/bash
#
#Check network to mxnas2 from mxnas1 via ssh
#Check file access to Raid volume 'mxnas2'
#Email alert if exit status is not 0
#
/usr/bin/ssh -n 10.x.x.x test -f /mxnas2/.mxnas_up
exstat=$?
if [ $exstat -ne 0 ]; then
  if [ $exstat -eq 1 ]; then
  /bin/echo "The MXNAS2 server appears to be up but the /mxnas2/.mxnas2_up file can not be found $(date)" > /tmp/mxnas2_stat_report.log
  /bin/mail -s "WARNING-MXNAS2 file system check error" damon.stewart@x.com -c tvcvtrxsan@x.com < /tmp/mxnas2_stat_report.log -- -f "mxnas1@x.com"
    elif [ $exstat -eq 255 ]; then
    /bin/echo "Can not SSH to MXNAS2 $(date)" > /tmp/mxnas2_stat_report.log
    /bin/mail -s "WARNING-MXNAS2 network check error" damon.stewart@x.com -c tvcvtrxsan@x.com < /tmp/mxnas2_stat_report.log -- -f "mxnas1@x.com"
      else 
      /bin/echo "System status check to MXNAS2 with exit status code $exstat $(date)" > /tmp/mxnas2_stat_report.log
      /bin/mail -s "WARNING-MXNAS2 unknown system check error" damon.stewart@x.com -c tvcvtrxsan@x.com < /tmp/mxnas2_stat_report.log -- -f "mxnas1@x.com"
  fi
fi
#Uncomment the following line to report exit status for testing
#echo "exit status to /mxnas2/ = $exstat"
#
#djs/js v1.2 2010-12-09

Just as a note, Im using an rsa key between the main and backup servers to allow the ssh login for both the rsync and status check. I also configured sendmail MTA as a client and added our smtprelay to the submit.mc and submit.cf. Im sure there are many other ways, maybe better, but this works well.

[Affix]

Surely a much easier way would be

exec 3<> /dev/tcp/host/port

For Example

[affix@fedora Desktop]$ exec 3<> /dev/tcp/127.0.0.1/22
[affix@fedora Desktop]$ exec 3<> /dev/tcp/127.0.0.1/2222
bash: connect: Connection refused
bash: /dev/tcp/127.0.0.1/2222: Connection refused

You will see that it doesn't return anything if the host port is open. If not the connection will be refused

[dstewart]

If all you wanted to do is check for a network to port 22, yes. I needed to verify the raid mount and email status on errors.

[banji]

If you also need to to check the raid status and look at mail you could also use expect to automate the logins. But what you are doing works.

[dstewart]

I've cleaned it up and added to to it some since. One of these days I'll play with 'expect'.

rsync will only run if ssh is available to the backup server and the destination raid volume is mounted. If no errors only a log is created. On any error either from the server up test or from rsync, I will get an email with the exit status code. If the error is from rsync I included an exit status legend.

[root@tvc-mxnas1 bin]# cat promo_stat_and_sync.sh 
#!/bin/bash
#
#Check network to mxnas2 from mxnas1 via ssh
#Check file access to Raid volume,'.mxnas_up'
#rsync /mxnas1/ to /mxnas2/ over ssh
#Email if exit status is not 0
#
ssh -n 10.102.63.32 test -f /mxnas2/.mxnas_up
teststat=$?
#
error1="MXNAS1 to MXNAS2 Sync did not occur at $(date) do to the following error;"
error2="The MXNAS2 server appears to be up but the /mxnas2/.mxnas2_up file can not be found"
error3="No ssh path to MXNAS2"
error4="System status check to MXNAS2 with unknown error, exit status code $teststat"
subject1="WARNING-MXNAS2 System check error"
subject2="ERROR-MXNAS1 to MXNAS2 RSYNC FAULT"
rsync_exit_stat="rsync exit status code legend;
0\tSuccess
1\tSyntax or usage error
2\tProtocol incompatibility
3\tErrors selecting input/output files, dirs
4\tRequested  action not supported: an attempt was made to manipulate 64-bit files on a platform that cannot support them; or an option was specified that is supported by the client and not by the server.
5\tError starting client-server protocol
6\tDaemon unable to append to log-file
10\tError in socket I/O
11\tError in file I/O
12\tError in rsync protocol data stream
13\tErrors with program diagnostics
14\tError in IPC code
20\tReceived SIGUSR1 or SIGINT
21\tSome error returned by waitpid()
22\tError allocating core memory buffers
23\tPartial transfer due to error
24\tPartial transfer due to vanished source files
25\tThe --max-delete limit stopped deletions
30\tTimeout in data send/receive"
#
if [ $teststat -eq 0 ]; then
   rsync -a --delete /mxnas1/ 10.102.63.32:/mxnas2/
   syncstat=$?
   echo -e "MXNAS1 to MXNAS2, RSYNC last ran at $(date) with an exit status code of;\n/mxnas1/=$syncstat\n\n$rsync_exit_stat" > /var/log/mxnas2_stat_report.log
        if [ $syncstat -ne 0 ]; then
           mail -s "$subject2" damon.stewart@x.com -c tvcvtrxsan@x.com < /var/log/mxnas2_stat_report.log -- -f "mxnas1@x.com"
        fi
   elif [ $teststat -eq 1 ]; then
        echo -e "$error1\n$error2" > /var/log/mxnas2_stat_report.log
        mail -s "$subject1" damon.stewart@x.com -c tvcvtrxsan@x.com < /var/log/mxnas2_stat_report.log -- -f "mxnas1@x.com"
   elif [ $teststat -eq 255 ]; then
        echo -e "$error1\n$error3" > /var/log/mxnas2_stat_report.log
        mail -s "$subject1" damon.stewart@x.com -c tvcvtrxsan@x.com < /var/log/mxnas2_stat_report.log -- -f "mxnas1@x.com"
   else
        echo -e "$error1\n$error4" > /var/log/mxnas2_stat_report.log
        mail -s "$subject1" damon.stewart@x.com -c tvcvtrxsan@x.com < /var/log/mxnas2_stat_report.log -- -f "mxnas1@x.com"
fi
#
#djs v2.3 2010-12-29

Here is an example email body with a rsync error;

MXNAS1 to MXNAS2, RSYNC last ran at Thu Dec 30 04:31:17 PST 2010 with an exit status code of;
/mxnas1/=24

rsync exit status code legend;
0 Success
1 Syntax or usage error
2 Protocol incompatibility
3 Errors selecting input/output files, dirs
4 Requested action not supported: an attempt was made to manipulate 64-bit files on a platform that cannot support them; or an option was specified that is supported by the client and not by the server.
5 Error starting client-server protocol
6 Daemon unable to append to log-file
10 Error in socket I/O
11 Error in file I/O
12 Error in rsync protocol data stream
13 Errors with program diagnostics
14 Error in IPC code
20 Received SIGUSR1 or SIGINT
21 Some error returned by waitpid()
22 Error allocating core memory buffers
23 Partial transfer due to error
24 Partial transfer due to vanished source files
25 The --max-delete limit stopped deletions
30 Timeout in data send/receive