In this post, we will explain how to setup failover on rundeck. Rundeck officially doesn’t supports failover. This approach we tested on Centos 6 and its working fine. Moreover, use this approach on your own risk \ud83d\ude1b<\/p>\n
Assumption:<\/strong><\/p>\n This post assumes that rundeck is running on host1 listens on port 8000. Its standalone server and using mysql database.<\/p>\n Before moving to the post, lets understand basic execution mode on rundeck. There are 2 kind of execution mode available on rundeck<\/p>\n We will configure rundeck failover server Passive mode where no jobs will run and whenever master server goes down, rundeck failover server will become a master.<\/p>\n Work Flow:<\/strong><\/p>\n yum install rundeck<\/span><\/p>\n \u00a0Configure database details on the rundeck Failover server.<\/p>\n dataSource.url=jdbc:mysql:\/\/IPADDRESS\/rundeck?autoReconnect=true<\/span><\/p>\n dataSource.username=rundeck<\/span><\/p>\n dataSource.password=<YOUR PASSWORD><\/span><\/p>\n dataSource.driverClassName=com.mysql.jdbc.Driver<\/span><\/p>\n<\/blockquote>\n Modify the rundeck port if you wish to listen on different. In our case, we used port 8000<\/p>\n RDECK_HTTP_PORT=8000<\/span><\/p>\n<\/blockquote>\n Grant privileges to mysql user on mysql host<\/p>\n grant all on rundeck.* to ‘rundeck’@’FAILOVER-IPADDRESS’ identified by ‘<YOUR PASSWORD>\u2019;<\/span><\/p>\n flush privileges;<\/span><\/p>\n<\/blockquote>\n Copy ssh key on master to failover server. Execute the below command on failover server<\/p>\n scp \/var\/lib\/rundeck\/.ssh<\/span><\/p>\n MASTER_IP_ADDRESS:\/var\/lib\/rundeck<\/span><\/p>\n<\/blockquote>\n Generate API key on FAILOVER SERVER. Move your Pointer to\u00a0Rundeck -> Settings -> GENERATE<\/strong><\/p>\n Provide enable_executions, disable_executions\u00a0<\/strong> to apitoken.aclpolicy<\/strong><\/p>\n context:<\/span><\/p>\n \u00a0 application: ‘rundeck’<\/span><\/p>\n for:<\/span><\/p>\n \u00a0 resource:<\/span><\/p>\n \u00a0 \u00a0 – equals:<\/span><\/p>\n \u00a0 \u00a0 \u00a0 \u00a0 kind: system<\/span><\/p>\n \u00a0 \u00a0 \u00a0 allow: [read,enable_executions,disable_executions]<\/b> # allow read of system info<\/span><\/p>\n<\/blockquote>\n Start the rundeck service on failover server.<\/p>\n Change active mode to passive on failover server.<\/p>\n Rundeck Server -> Settings -> Change Executions -> Passive -> Save<\/em><\/p>\n Copy the failover script and add it cron to run every 15 minutes.<\/p>\n Script can be downloaded from github<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"\n
\n
<\/code>Fail Over Setup:<\/strong><\/p>\n<\/code>Install rundeck on the fail over server.
\nrpm -Uvh http:\/\/repo.rundeck.org\/latest.rpm
\n<\/span><\/p>\nvi \/etc\/rundeck\/rundeck-config.properties<\/code><\/p>\n\n
vi \/etc\/rundeck\/profile<\/code><\/p>\n\n
\n
\n
vi \/etc\/rundeck\/apitoken.aclpolicy<\/code><\/p>\n\n
\/etc\/init.d\/rundeck start<\/code><\/p>\n#!\/usr\/bin\/python\r\n#Author:support@pheonixsolutions.com\r\n#Description: Script to setup Failover\r\n#Schedule this script on cron to run every 15 minutes\r\n#Love to do in Python\r\n#Version:1.0\r\nimport os,sys,socket\r\nimport xml.etree.ElementTree\r\nimport smtplib\r\nimport email\r\nimport email.mime.text\r\nimport string\r\n# Required Variables\r\nPROJECT_FOLDER='\/var\/rundeck\/projects'\r\nLOG_FOLDER='\/var\/log\/rundeck'\r\nEXECUTION_LOGS='\/var\/lib\/rundeck\/logs'\r\nMASTER_IP_ADDRESS='<MASTER-IP-ADDRESS'\r\nFAILOVER_IP_ADDRESS='<FAILOVER-IP_ADDRESS' RUNDECK_PORT=8000 sender_mail = 'rundeck-failover@domain.tld' # Mention FROM email address recipients =['email1@domain.tld','email2@domain.tld']#Separated by Comma API_TOKEN='YOUR API TOKEN' print \"Checking whether Rundeck Master server is responding..\" check_socket=socket.socket() try: check_socket.connect((MASTER_IP_ADDRESS,RUNDECK_PORT)); #print \"Connected to %s on port %s\" % (MASTER_IP_ADDRESS, RUNDECK_PORT) # \"Syncing data from Master to Slave\" os.system('rsync -ratlz root@'+MASTER_IP_ADDRESS+':'+PROJECT_FOLDER+'\/* ' +PROJECT_FOLDER+'\/') os.system('rsync -ratlz root@'+MASTER_IP_ADDRESS+':'+LOG_FOLDER+'\/* ' +LOG_FOLDER+'\/') os.system('rsync -ratlz root@'+MASTER_IP_ADDRESS+':'+EXECUTION_LOGS+'\/* ' +EXECUTION_LOGS+'\/') os.system(\"curl -H 'X-Rundeck-Auth-Token:\"+API_TOKEN+\"' http:\/\/\"+FAILOVER_IP_ADDRESS+\":\"+str(RUNDECK_PORT)+\"\/api\/2\/system\/info>\/tmp\/output_failover.xml\");\r\n e = xml.etree.ElementTree.parse('\/tmp\/output_failover.xml').getroot()\r\n for all_elements in e:\r\n for executions in all_elements:\r\n if(executions.tag =='executions'):\r\n #Checking whether Failover server is active or passive\r\n if(executions.get('executionMode') == 'active'):\r\n os.system('curl -H \"X-Rundeck-Auth-Token:'+API_TOKEN+'\" -X POST http:\/\/'+FAILOVER_IP_ADDRESS+':'+str(RUNDECK_PORT)+'\/api\/14\/system\/executions\/disable');\r\n #Sending Notifications\r\n body=\"Master Server %s is up. Fail Over Server Execution Mode is Active. Disabling Active Mode on %s\\n\" %(MASTER_IP_ADDRESS,FAILOVER_IP_ADDRESS)\r\n body_of_the_message= email.mime.text.MIMEText(body,'html');\r\n message = email.MIMEMultipart.MIMEMultipart('alternative')\r\n message['Subject'] =\"Rundeck Fail Over Alert\"\r\n message['From'] = sender_mail\r\n message.attach(body_of_the_message);\r\n message['To']=\",\".join(recipients)\r\n server = smtplib.SMTP('localhost')\r\n server.sendmail(message['From'],recipients,message.as_string())\r\n server.quit()\r\n elif(executions.get('executionMode') == 'passive'):\r\n print \"Fail Over Server Execution Mode is Passive. Do Nothing..\";\r\n else:\r\n print \"Something is wrong. Please check Fail Over Server %s.\" %(FAILOVER_IP_ADDRESS)\r\n body=\"Rundeck Fail Over Alert: Something is Wrong on %s\" %(FAILOVER_IP_ADDRESS)\r\n body_of_the_message= email.mime.text.MIMEText(body,'html');\r\n message = email.MIMEMultipart.MIMEMultipart('alternative')\r\n message['Subject'] =\"Urgent:Rundeck Fail Over Alert. Something is wrong on Failover Server\"\r\n message['From'] = sender_mail\r\n message['To']=\",\".join(recipients)\r\n message.attach(body_of_the_message);\r\n server = smtplib.SMTP('localhost')\r\n server.sendmail(message['From'],recipients,message.as_string())\r\n server.quit()\r\n#If the Master server doesn't Responds, switch the connection\r\nexcept socket.error, e:\r\n print \"Connection Failed. Master Rundeck server failed %s\" %(MASTER_IP_ADDRESS);\r\n print \"Starting Failover Server\"\r\n os.system('curl -H \"X-Rundeck-Auth-Token:'+API_TOKEN+'\" -X POST http:\/\/'+FAILOVER_IP_ADDRESS+':'+str(RUNDECK_PORT)+'\/api\/14\/system\/executions\/enable');\r\n body=\"Rundeck Fail Over Alert: Rundeck Master is Down %s. Failover Completed on %s. Please check \" %(MASTER_IP_ADDRESS,FAILOVER_IP_ADDRESS)\r\n body_of_the_message= email.mime.text.MIMEText(body,'html');\r\n message = email.MIMEMultipart.MIMEMultipart('alternative')\r\n message['From'] = sender_mail\r\n message['Subject'] =\"Urgent: Rundeck Master Server %s is Down. Please check \" %(MASTER_IP_ADDRESS)\r\n message.attach(body_of_the_message);\r\n message['To']=\",\".join(recipients)\r\n server = smtplib.SMTP('localhost')\r\n server.sendmail(message['From'],recipients,message.as_string())\r\n server.quit()<\/pre>\n
\nAdd the script on cron to run every 15 minutes.
\nStop the rundeck server on master and execute the copied script, you can see failover server become Active mode.<\/p>\n