In this post, we will explain how to set up failover on Rundeck. Rundeck officially doesn’t supports failover. We tested this approach on CentOS 6 and it’s working fine. Moreover, use this approach at your own risk \ud83d\ude1b<\/p>\n\n\n\n
This post assumes that Rundeck is running on host1 and listens on port 8000. Its standalone server and using mysql database.<\/p>\n\n\n\n
Before moving to the post, let’s understand basic execution mode on Rundeck. There are 2 kinds of execution mode available on Rundeck<\/p>\n\n\n\n
We will configure Rundeck failover server in Passive mode, where no jobs will run and whenever the master server goes down, the Rundeck failover server will become a master.<\/p>\n\n\n\n
Fail Over Setup:<\/strong><\/p>\n\n\n\n Install Rundeck on the failover server.<\/p>\n\n\n\n Configure database details on the rundeck Failover server.<\/p>\n\n\n\n dataSource.url=jdbc:mysql:\/\/IPADDRESS\/rundeck?autoReconnect=true<\/p>\n\n\n\n dataSource.username=rundeck<\/p>\n\n\n\n dataSource.password=<YOUR PASSWORD><\/p>\n\n\n\n dataSource.driverClassName=com.mysql.jdbc.Driver<\/p>\n<\/blockquote>\n\n\n\n Modify the rundeck port if you wish to listen on different. In our case, we used port 8000<\/p>\n\n\n\n RDECK_HTTP_PORT=8000<\/p>\n<\/blockquote>\n\n\n\n Grant privileges to mysql user on mysql host<\/p>\n\n\n\n grant all on rundeck.* to ‘rundeck’@’FAILOVER-IPADDRESS’ identified by ‘<YOUR PASSWORD>\u2019;<\/p>\n\n\n\n flush privileges;<\/p>\n<\/blockquote>\n\n\n\n Copy ssh key on master to failover server. Execute the below command on failover server<\/p>\n\n\n\n scp \/var\/lib\/rundeck\/.ssh<\/p>\n\n\n\n MASTER_IP_ADDRESS:\/var\/lib\/rundeck<\/p>\n<\/blockquote>\n\n\n\n Generate API key on FAILOVER SERVER. Move your Pointer to Rundeck -> Settings -> GENERATE<\/strong><\/p>\n\n\n\n Provide enable_executions, disable_executions <\/strong> to apitoken.aclpolicy<\/strong><\/p>\n\n\n\n context:<\/p>\n\n\n\n application: ‘rundeck’<\/p>\n\n\n\n for:<\/p>\n\n\n\n resource:<\/p>\n\n\n\n – equals:<\/p>\n\n\n\n kind: system<\/p>\n\n\n\n allow: [read,enable_executions,disable_executions]<\/strong> # allow read of system info<\/p>\n<\/blockquote>\n\n\n\n Start the rundeck service on failover server.<\/p>\n\n\n\n Change active mode to passive on the failover server.<\/p>\n\n\n\n Rundeck Server -> Settings -> Change Executions -> Passive -> Save<\/em><\/p>\n\n\n\n Copy the failover script and add it to cron to run every 15 minutes.<\/p>\n\n\n\n \u00a0$ rpm -Uvh http:\/\/repo.rundeck.org\/latest.rpm<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n $ yum install rundeck<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n $ vi \/etc\/rundeck\/rundeck-config.properties<\/code><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n\n
$ vi \/etc\/rundeck\/profile<\/code><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n\n
\n
\n
$ vi \/etc\/rundeck\/apitoken.aclpolicy<\/code><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n\n
$ \/etc\/init.d\/rundeck start<\/code><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n#!\/usr\/bin\/python\n#Author:support@pheonixsolutions.com\n#Description: Script to setup Failover\n#Schedule this script on cron to run every 15 minutes\n#Love to do in Python\n#Version:1.0\nimport os,sys,socket\nimport xml.etree.ElementTree\nimport smtplib\nimport email\nimport email.mime.text\nimport string\n# Required Variables\nPROJECT_FOLDER='\/var\/rundeck\/projects'\nLOG_FOLDER='\/var\/log\/rundeck'\nEXECUTION_LOGS='\/var\/lib\/rundeck\/logs'\nMASTER_IP_ADDRESS='<MASTER-IP-ADDRESS'\nFAILOVER_IP_ADDRESS='<FAILOVER-IP_ADDRESS' RUNDECK_PORT=8000 sender_mail = 'rundeck-failover@domain.tld' # Mention FROM email address recipients =['email1@domain.tld','email2@domain.tld']#Separated by Comma API_TOKEN='YOUR API TOKEN' print \"Checking whether Rundeck Master server is responding..\" check_socket=socket.socket() try: check_socket.connect((MASTER_IP_ADDRESS,RUNDECK_PORT)); #print \"Connected to %s on port %s\" % (MASTER_IP_ADDRESS, RUNDECK_PORT) # \"Syncing data from Master to Slave\" os.system('rsync -ratlz root@'+MASTER_IP_ADDRESS+':'+PROJECT_FOLDER+'\/* ' +PROJECT_FOLDER+'\/') os.system('rsync -ratlz root@'+MASTER_IP_ADDRESS+':'+LOG_FOLDER+'\/* ' +LOG_FOLDER+'\/') os.system('rsync -ratlz root@'+MASTER_IP_ADDRESS+':'+EXECUTION_LOGS+'\/* ' +EXECUTION_LOGS+'\/') os.system(\"curl -H 'X-Rundeck-Auth-Token:\"+API_TOKEN+\"' http:\/\/\"+FAILOVER_IP_ADDRESS+\":\"+str(RUNDECK_PORT)+\"\/api\/2\/system\/info>\/tmp\/output_failover.xml\");\n e = xml.etree.ElementTree.parse('\/tmp\/output_failover.xml').getroot()\n for all_elements in e:\n for executions in all_elements:\n if(executions.tag =='executions'):\n #Checking whether Failover server is active or passive\n if(executions.get('executionMode') == 'active'):\n os.system('curl -H \"X-Rundeck-Auth-Token:'+API_TOKEN+'\" -X POST http:\/\/'+FAILOVER_IP_ADDRESS+':'+str(RUNDECK_PORT)+'\/api\/14\/system\/executions\/disable');\n #Sending Notifications\n body=\"Master Server %s is up. Fail Over Server Execution Mode is Active. Disabling Active Mode on %s\\n\" %(MASTER_IP_ADDRESS,FAILOVER_IP_ADDRESS)\n body_of_the_message= email.mime.text.MIMEText(body,'html');\n message = email.MIMEMultipart.MIMEMultipart('alternative')\n message['Subject'] =\"Rundeck Fail Over Alert\"\n message['From'] = sender_mail\n message.attach(body_of_the_message);\n message['To']=\",\".join(recipients)\n server = smtplib.SMTP('localhost')\n server.sendmail(message['From'],recipients,message.as_string())\n server.quit()\n elif(executions.get('executionMode') == 'passive'):\n print \"Fail Over Server Execution Mode is Passive. Do Nothing..\";\n else:\n print \"Something is wrong. Please check Fail Over Server %s.\" %(FAILOVER_IP_ADDRESS)\n body=\"Rundeck Fail Over Alert: Something is Wrong on %s\" %(FAILOVER_IP_ADDRESS)\n body_of_the_message= email.mime.text.MIMEText(body,'html');\n message = email.MIMEMultipart.MIMEMultipart('alternative')\n message['Subject'] =\"Urgent:Rundeck Fail Over Alert. Something is wrong on Failover Server\"\n message['From'] = sender_mail\n message['To']=\",\".join(recipients)\n message.attach(body_of_the_message);\n server = smtplib.SMTP('localhost')\n server.sendmail(message['From'],recipients,message.as_string())\n server.quit()\n#If the Master server doesn't Responds, switch the connection\nexcept socket.error, e:\n print \"Connection Failed. Master Rundeck server failed %s\" %(MASTER_IP_ADDRESS);\n print \"Starting Failover Server\"\n os.system('curl -H \"X-Rundeck-Auth-Token:'+API_TOKEN+'\" -X POST http:\/\/'+FAILOVER_IP_ADDRESS+':'+str(RUNDECK_PORT)+'\/api\/14\/system\/executions\/enable');\n body=\"Rundeck Fail Over Alert: Rundeck Master is Down %s. Failover Completed on %s. Please check \" %(MASTER_IP_ADDRESS,FAILOVER_IP_ADDRESS)\n body_of_the_message= email.mime.text.MIMEText(body,'html');\n message = email.MIMEMultipart.MIMEMultipart('alternative')\n message['From'] = sender_mail\n message['Subject'] =\"Urgent: Rundeck Master Server %s is Down. Please check \" %(MASTER_IP_ADDRESS)\n message.attach(body_of_the_message);\n message['To']=\",\".join(recipients)\n server = smtplib.SMTP('localhost')\n server.sendmail(message['From'],recipients,message.as_string())\n server.quit()<\/pre>\n\n\n\n
Add the script to cron to run every 15 minutes.
Stop the Rundeck server on the master and execute the copied script; you can see the failover server become Active mode.<\/p>\n\n\n\n