In this post, we will explain how to setup failover on rundeck. Rundeck officially doesn’t supports failover. This approach we tested on Centos 6 and its working fine. Moreover, use this approach on your own risk 😛

Assumption:

This post assumes that rundeck is running on host1 listens on port 8000. Its standalone server and using mysql database.

Before moving to the post, lets understand basic execution mode on rundeck. There are 2 kind of execution mode available on rundeck

  1. Active Mode: Jobs, scheduled Jobs, and adhoc executions can be run.
  2. Passive Mode:No Jobs or adhoc executions can be run.

We will configure rundeck failover server Passive mode where no jobs will run and whenever master server goes down, rundeck failover server will become a master.

Work Flow:

  1. Checks whether rundeck master server is listening on port 8000 from Failover server
  2. If rundeck master responds, copy the projects, log folders and checks whether failover server is in active mode or passive mode. If rundeck failover server is active mode, change it to passive mode and send notifications. If rundeck failover server is in Passive mode then do nothing.
  3. If rundeck master doesn’t responds, switch the failover rundeck server execution mode from Passive to Active and send notifications.

Fail Over Setup:

Install rundeck on the fail over server.
rpm -Uvh http://repo.rundeck.org/latest.rpm

yum install rundeck

 Configure database details on the rundeck Failover server.

vi /etc/rundeck/rundeck-config.properties

dataSource.url=jdbc:mysql://IPADDRESS/rundeck?autoReconnect=true

dataSource.username=rundeck

dataSource.password=<YOUR PASSWORD>

dataSource.driverClassName=com.mysql.jdbc.Driver

Modify the rundeck port if you wish to listen on different. In our case, we used port 8000

vi /etc/rundeck/profile

RDECK_HTTP_PORT=8000

Grant privileges to mysql user on mysql host

grant all on rundeck.* to ‘rundeck’@’FAILOVER-IPADDRESS’ identified by ‘<YOUR PASSWORD>’;

flush privileges;

Copy ssh key on master to failover server. Execute the below command on failover server

scp /var/lib/rundeck/.ssh

MASTER_IP_ADDRESS:/var/lib/rundeck

Generate API key on FAILOVER SERVER. Move your Pointer to Rundeck -> Settings -> GENERATE

Provide enable_executions, disable_executions  to apitoken.aclpolicy

vi /etc/rundeck/apitoken.aclpolicy

context:

  application: ‘rundeck’

for:

  resource:

    – equals:

        kind: system

      allow: [read,enable_executions,disable_executions] # allow read of system info

Start the rundeck service on failover server.

/etc/init.d/rundeck start

Change active mode to passive on failover server.

Rundeck Server -> Settings -> Change Executions -> Passive -> Save

Copy the failover script and add it cron to run every 15 minutes.

#!/usr/bin/python
#Author:support@pheonixsolutions.com
#Description: Script to setup Failover
#Schedule this script on cron to run every 15 minutes
#Love to do in Python
#Version:1.0
import os,sys,socket
import xml.etree.ElementTree
import smtplib
import email
import email.mime.text
import string
# Required Variables
PROJECT_FOLDER='/var/rundeck/projects'
LOG_FOLDER='/var/log/rundeck'
EXECUTION_LOGS='/var/lib/rundeck/logs'
MASTER_IP_ADDRESS='<MASTER-IP-ADDRESS'
FAILOVER_IP_ADDRESS='<FAILOVER-IP_ADDRESS' RUNDECK_PORT=8000 sender_mail = 'rundeck-failover@domain.tld' # Mention FROM email address recipients =['email1@domain.tld','email2@domain.tld']#Separated by Comma API_TOKEN='YOUR API TOKEN' print "Checking whether Rundeck Master server is responding.." check_socket=socket.socket() try: check_socket.connect((MASTER_IP_ADDRESS,RUNDECK_PORT)); #print "Connected to %s on port %s" % (MASTER_IP_ADDRESS, RUNDECK_PORT) # "Syncing data from Master to Slave" os.system('rsync -ratlz root@'+MASTER_IP_ADDRESS+':'+PROJECT_FOLDER+'/* ' +PROJECT_FOLDER+'/') os.system('rsync -ratlz root@'+MASTER_IP_ADDRESS+':'+LOG_FOLDER+'/* ' +LOG_FOLDER+'/') os.system('rsync -ratlz root@'+MASTER_IP_ADDRESS+':'+EXECUTION_LOGS+'/* ' +EXECUTION_LOGS+'/') os.system("curl -H 'X-Rundeck-Auth-Token:"+API_TOKEN+"' http://"+FAILOVER_IP_ADDRESS+":"+str(RUNDECK_PORT)+"/api/2/system/info>/tmp/output_failover.xml");
  e = xml.etree.ElementTree.parse('/tmp/output_failover.xml').getroot()
  for all_elements in e:
    for executions in all_elements:
      if(executions.tag =='executions'):
        #Checking whether Failover server is active or passive
        if(executions.get('executionMode') == 'active'):
          os.system('curl -H "X-Rundeck-Auth-Token:'+API_TOKEN+'" -X POST http://'+FAILOVER_IP_ADDRESS+':'+str(RUNDECK_PORT)+'/api/14/system/executions/disable');
          #Sending Notifications
          body="Master Server %s is up. Fail Over Server Execution Mode is Active. Disabling Active Mode on %s\n" %(MASTER_IP_ADDRESS,FAILOVER_IP_ADDRESS)
          body_of_the_message= email.mime.text.MIMEText(body,'html');
          message      = email.MIMEMultipart.MIMEMultipart('alternative')
          message['Subject'] ="Rundeck Fail Over Alert"
          message['From']    = sender_mail
          message.attach(body_of_the_message);
          message['To']=",".join(recipients)
          server = smtplib.SMTP('localhost')
          server.sendmail(message['From'],recipients,message.as_string())
          server.quit()
        elif(executions.get('executionMode') == 'passive'):
          print "Fail Over Server Execution Mode is Passive. Do Nothing..";
        else:
          print "Something is wrong. Please check Fail Over Server %s." %(FAILOVER_IP_ADDRESS)
          body="Rundeck Fail Over Alert: Something is Wrong on %s" %(FAILOVER_IP_ADDRESS)
          body_of_the_message= email.mime.text.MIMEText(body,'html');
          message      = email.MIMEMultipart.MIMEMultipart('alternative')
          message['Subject'] ="Urgent:Rundeck Fail Over Alert. Something is wrong on Failover Server"
          message['From']    = sender_mail
          message['To']=",".join(recipients)
          message.attach(body_of_the_message);
          server = smtplib.SMTP('localhost')
          server.sendmail(message['From'],recipients,message.as_string())
          server.quit()
#If the Master server doesn't Responds, switch the connection
except socket.error, e:
  print "Connection Failed. Master Rundeck server failed %s" %(MASTER_IP_ADDRESS);
  print "Starting Failover Server"
  os.system('curl -H "X-Rundeck-Auth-Token:'+API_TOKEN+'" -X POST http://'+FAILOVER_IP_ADDRESS+':'+str(RUNDECK_PORT)+'/api/14/system/executions/enable');
  body="Rundeck Fail Over Alert: Rundeck Master is Down %s. Failover Completed on %s. Please check " %(MASTER_IP_ADDRESS,FAILOVER_IP_ADDRESS)
  body_of_the_message= email.mime.text.MIMEText(body,'html');
  message      = email.MIMEMultipart.MIMEMultipart('alternative')
  message['From']    = sender_mail
  message['Subject'] ="Urgent: Rundeck Master Server %s is Down. Please check " %(MASTER_IP_ADDRESS)
  message.attach(body_of_the_message);
  message['To']=",".join(recipients)
  server = smtplib.SMTP('localhost')
  server.sendmail(message['From'],recipients,message.as_string())
  server.quit()

 
Add the script on cron to run every 15 minutes.
Stop the rundeck server on master and execute the copied script, you can see failover server become Active mode.

Script can be downloaded from github

Post Views: 1,174

Leave a Reply