Introduction:

In this post, we will explain how to set up failover on Rundeck. Rundeck officially doesn’t supports failover. We tested this approach on CentOS 6 and it’s working fine. Moreover, use this approach at your own risk 😛

Prerequisites:

This post assumes that Rundeck is running on host1 and listens on port 8000. Its standalone server and using mysql database.

Before moving to the post, let’s understand basic execution mode on Rundeck. There are 2 kinds of execution mode available on Rundeck

  1. Active Mode: Jobs, scheduled Jobs, and adhoc executions can be run.
  2. Passive Mode: No Jobs or adhoc executions can be run.

We will configure Rundeck failover server in Passive mode, where no jobs will run and whenever the master server goes down, the Rundeck failover server will become a master.

Implementation

  1. Check whether the Rundeck master server is listening on port 8000 from the failover server
  2. If the Rundeck master responds, copy the projects and log folders and check whether the failover server is in active mode or passive mode. If the Rundeck failover server is in active mode, change it to passive mode and send notifications. If the Rundeck failover server is in Passive mode, then do nothing.
  3. If the Rundeck master doesn’t respond, switch the failover Rundeck server execution mode from Passive to Active and send notifications.

Fail Over Setup:

Install Rundeck on the failover server.

$ rpm -Uvh http://repo.rundeck.org/latest.rpm
$ yum install rundeck

 Configure database details on the rundeck Failover server.

$ vi /etc/rundeck/rundeck-config.properties

dataSource.url=jdbc:mysql://IPADDRESS/rundeck?autoReconnect=true

dataSource.username=rundeck

dataSource.password=<YOUR PASSWORD>

dataSource.driverClassName=com.mysql.jdbc.Driver

Modify the rundeck port if you wish to listen on different. In our case, we used port 8000

$ vi /etc/rundeck/profile

RDECK_HTTP_PORT=8000

Grant privileges to mysql user on mysql host

grant all on rundeck.* to ‘rundeck’@’FAILOVER-IPADDRESS’ identified by ‘<YOUR PASSWORD>’;

flush privileges;

Copy ssh key on master to failover server. Execute the below command on failover server

scp /var/lib/rundeck/.ssh

MASTER_IP_ADDRESS:/var/lib/rundeck

Generate API key on FAILOVER SERVER. Move your Pointer to Rundeck -> Settings -> GENERATE

Provide enable_executions, disable_executions  to apitoken.aclpolicy

$ vi /etc/rundeck/apitoken.aclpolicy

context:

  application: ‘rundeck’

for:

  resource:

    – equals:

        kind: system

      allow: [read,enable_executions,disable_executions] # allow read of system info

Start the rundeck service on failover server.

$ /etc/init.d/rundeck start

Change active mode to passive on the failover server.

Rundeck Server -> Settings -> Change Executions -> Passive -> Save

Copy the failover script and add it to cron to run every 15 minutes.

#!/usr/bin/python
#Author:support@pheonixsolutions.com
#Description: Script to setup Failover
#Schedule this script on cron to run every 15 minutes
#Love to do in Python
#Version:1.0
import os,sys,socket
import xml.etree.ElementTree
import smtplib
import email
import email.mime.text
import string
# Required Variables
PROJECT_FOLDER='/var/rundeck/projects'
LOG_FOLDER='/var/log/rundeck'
EXECUTION_LOGS='/var/lib/rundeck/logs'
MASTER_IP_ADDRESS='<MASTER-IP-ADDRESS'
FAILOVER_IP_ADDRESS='<FAILOVER-IP_ADDRESS' RUNDECK_PORT=8000 sender_mail = 'rundeck-failover@domain.tld' # Mention FROM email address recipients =['email1@domain.tld','email2@domain.tld']#Separated by Comma API_TOKEN='YOUR API TOKEN' print "Checking whether Rundeck Master server is responding.." check_socket=socket.socket() try: check_socket.connect((MASTER_IP_ADDRESS,RUNDECK_PORT)); #print "Connected to %s on port %s" % (MASTER_IP_ADDRESS, RUNDECK_PORT) # "Syncing data from Master to Slave" os.system('rsync -ratlz root@'+MASTER_IP_ADDRESS+':'+PROJECT_FOLDER+'/* ' +PROJECT_FOLDER+'/') os.system('rsync -ratlz root@'+MASTER_IP_ADDRESS+':'+LOG_FOLDER+'/* ' +LOG_FOLDER+'/') os.system('rsync -ratlz root@'+MASTER_IP_ADDRESS+':'+EXECUTION_LOGS+'/* ' +EXECUTION_LOGS+'/') os.system("curl -H 'X-Rundeck-Auth-Token:"+API_TOKEN+"' http://"+FAILOVER_IP_ADDRESS+":"+str(RUNDECK_PORT)+"/api/2/system/info>/tmp/output_failover.xml");
  e = xml.etree.ElementTree.parse('/tmp/output_failover.xml').getroot()
  for all_elements in e:
    for executions in all_elements:
      if(executions.tag =='executions'):
        #Checking whether Failover server is active or passive
        if(executions.get('executionMode') == 'active'):
          os.system('curl -H "X-Rundeck-Auth-Token:'+API_TOKEN+'" -X POST http://'+FAILOVER_IP_ADDRESS+':'+str(RUNDECK_PORT)+'/api/14/system/executions/disable');
          #Sending Notifications
          body="Master Server %s is up. Fail Over Server Execution Mode is Active. Disabling Active Mode on %s\n" %(MASTER_IP_ADDRESS,FAILOVER_IP_ADDRESS)
          body_of_the_message= email.mime.text.MIMEText(body,'html');
          message      = email.MIMEMultipart.MIMEMultipart('alternative')
          message['Subject'] ="Rundeck Fail Over Alert"
          message['From']    = sender_mail
          message.attach(body_of_the_message);
          message['To']=",".join(recipients)
          server = smtplib.SMTP('localhost')
          server.sendmail(message['From'],recipients,message.as_string())
          server.quit()
        elif(executions.get('executionMode') == 'passive'):
          print "Fail Over Server Execution Mode is Passive. Do Nothing..";
        else:
          print "Something is wrong. Please check Fail Over Server %s." %(FAILOVER_IP_ADDRESS)
          body="Rundeck Fail Over Alert: Something is Wrong on %s" %(FAILOVER_IP_ADDRESS)
          body_of_the_message= email.mime.text.MIMEText(body,'html');
          message      = email.MIMEMultipart.MIMEMultipart('alternative')
          message['Subject'] ="Urgent:Rundeck Fail Over Alert. Something is wrong on Failover Server"
          message['From']    = sender_mail
          message['To']=",".join(recipients)
          message.attach(body_of_the_message);
          server = smtplib.SMTP('localhost')
          server.sendmail(message['From'],recipients,message.as_string())
          server.quit()
#If the Master server doesn't Responds, switch the connection
except socket.error, e:
  print "Connection Failed. Master Rundeck server failed %s" %(MASTER_IP_ADDRESS);
  print "Starting Failover Server"
  os.system('curl -H "X-Rundeck-Auth-Token:'+API_TOKEN+'" -X POST http://'+FAILOVER_IP_ADDRESS+':'+str(RUNDECK_PORT)+'/api/14/system/executions/enable');
  body="Rundeck Fail Over Alert: Rundeck Master is Down %s. Failover Completed on %s. Please check " %(MASTER_IP_ADDRESS,FAILOVER_IP_ADDRESS)
  body_of_the_message= email.mime.text.MIMEText(body,'html');
  message      = email.MIMEMultipart.MIMEMultipart('alternative')
  message['From']    = sender_mail
  message['Subject'] ="Urgent: Rundeck Master Server %s is Down. Please check " %(MASTER_IP_ADDRESS)
  message.attach(body_of_the_message);
  message['To']=",".join(recipients)
  server = smtplib.SMTP('localhost')
  server.sendmail(message['From'],recipients,message.as_string())
  server.quit()

 
Add the script to cron to run every 15 minutes.
Stop the Rundeck server on the master and execute the copied script; you can see the failover server become Active mode.

The script can be downloaded from github

Conclusion

This solution provides a practical failover mechanism for standalone Rundeck deployments by leveraging automated monitoring, synchronisation, and execution mode switching. While it is not an officially supported high-availability solution, it can significantly improve service continuity in environments where Rundeck availability is critical. Regular testing and monitoring are recommended to ensure the failover process operates as expected.

Post Views: 319

Leave a Reply