{"id":1350,"date":"2017-04-17T18:14:02","date_gmt":"2017-04-17T12:44:02","guid":{"rendered":"https:\/\/pheonixsolutions.com\/blog\/?p=1350"},"modified":"2017-04-17T18:14:03","modified_gmt":"2017-04-17T12:44:03","slug":"python-script-take-backup-folder-amazon-s3-windows","status":"publish","type":"post","link":"https:\/\/pheonixsolutions.com\/blog\/python-script-take-backup-folder-amazon-s3-windows\/","title":{"rendered":"Python Script to take backup of folder on amazon s3 – Windows"},"content":{"rendered":"

Python Script to take backup of folder on Amazon s3 – Windows<\/h4>\n

Date Posted:17-04-2017<\/strong><\/p>\n

Amazon S3 is a storage where we can store files in a cloud. We can download the files from anywhere after we stored the files on cloud. There are many way to store the files on cloud. In this post, we will use python script to take backup of folder.<\/p>\n

Assumption:<\/h5>\n
    \n
  1. Windows Machine<\/li>\n
  2. Python<\/li>\n
  3. Boto Library.'”<\/li>\n
  4. IAM Access key and secret key which has privilege to the bucket “bucket-name”<\/li>\n<\/ol>\n

    Incase, if python or boto not installed on windows machine, follow the post to install python\u00a0on windows machine<\/a><\/p>\n

    Implementation.<\/h5>\n

    The below script will take backup of folder\u00a0“c:\\backup”\u00a0<\/em>on s3 bucket “bucket-name”.<\/p>\n

    What needs to be modified on the below script:<\/h5>\n

    Depends on your requirement\/details, please modify the below details<\/p>\n

      \n
    1. AWS_ACCESS_KEY_ID – \u00a0Your Access key. Incase, if we don’t have access key generate one from amazon IAM<\/li>\n
    2. AWS_ACCESS_KEY_SECRET – Your Secret key.<\/li>\n
    3. bucket_name – S3 Bucket name<\/li>\n
    4. sourceDir – Directory which needs to be backed up on s3<\/li>\n<\/ol>\n

      Create a file called s3backup.py<\/p>\n

      nano s3backup.py<\/code><\/p>\n

      import boto\r\nimport boto.s3\r\n\r\nimport os.path\r\nimport sys\r\nimport time\r\nimport datetime\r\n\r\n# Fill these in - you get them when you sign up for S3\r\nAWS_ACCESS_KEY_ID = 'xxxxxxxxxxxx'\r\nAWS_ACCESS_KEY_SECRET = 'xxxxxxxxxxxxxxxxxxxxxxx'\r\n# Fill in info on data to upload\r\n# destination bucket name\r\nbucket_name = 'bucket-name'\r\n# source directory\r\nsourceDir = 'c:\\backup'\r\n# destination directory name (on s3)\r\nutc_timestamp = time.time()\r\nUTC_FORMAT = '%Y%m%d'\r\nutc_time = datetime.datetime.utcfromtimestamp(utc_timestamp)\r\nutc_time = utc_time.strftime(UTC_FORMAT)\r\nprint (utc_time)\r\ndestDir = ''\r\n\r\n#max size in bytes before uploading in parts. between 1 and 5 GB recommended\r\nMAX_SIZE = 20 * 1000 * 1000\r\n#size of parts when uploading in parts\r\nPART_SIZE = 6 * 1000 * 1000\r\n\r\nconn = boto.connect_s3(AWS_ACCESS_KEY_ID, AWS_ACCESS_KEY_SECRET)\r\nbucket = conn.get_bucket(bucket_name)\r\nuploadFileNames = []\r\nfor root, dirs, files in os.walk(sourceDir, topdown=False):\r\n       for name in files:\r\n          fname=os.path.join(root, name)\r\n          print (fname)\r\n          uploadFileNames.append(fname)\r\n          print (uploadFileNames)\r\ndef percent_cb(complete, total):\r\n    sys.stdout.write('.')\r\n    sys.stdout.flush()\r\nfor filenames in uploadFileNames:\r\n        filename=filenames.replace(\"\\\\\", \"\/\")\r\n        print ('filename=' + filename)\r\n        sourcepath = filename\r\n        destpath = utc_time + '\/' + filename\r\n        #print ('Uploading %s to Amazon S3 bucket %s') % \\\r\n               #(sourcepath, bucket_name)\r\n\r\n        filesize = os.path.getsize(sourcepath)\r\n        if filesize > MAX_SIZE:\r\n            print (\"multipart upload\")\r\n            mp = bucket.initiate_multipart_upload(destpath)\r\n            fp = open(sourcepath,'rb')\r\n            fp_num = 0\r\n            while (fp.tell() < filesize):\r\n                fp_num += 1\r\n                print (\"uploading part %i\" %fp_num)\r\n                mp.upload_part_from_file(fp, fp_num, cb=percent_cb, num_cb=10, size=PART_SIZE)\r\n\r\n            mp.complete_upload()\r\n\r\n        else:\r\n            print (\"singlepart upload\")\r\n            k = boto.s3.key.Key(bucket)\r\n            k.key = destpath\r\n            k.set_contents_from_filename(sourcepath,\r\n                    cb=percent_cb, num_cb=10)\r\n<\/pre>\n
       <\/p>\n
      Execute the script on powershell to take backup. The backup will be created on the bucket with today’s date.<\/p>\n
      c:\\python3\\python.exe s3backup.py<\/code><\/p>\n
      We can add the script on schedule tasks to take backup daily.<\/p>\n","protected":false},"excerpt":{"rendered":"
      Python Script to take backup of folder on Amazon s3 – Windows Date Posted:17-04-2017 Amazon S3 is a storage where […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[1019],"tags":[280,292,324],"class_list":["post-1350","post","type-post","status-publish","format-standard","hentry","category-cloud-aws","tag-amazon","tag-python","tag-s3","psol-cat-cloud-aws"],"jetpack_publicize_connections":[],"jetpack_shortlink":"https:\/\/wp.me\/phn2x7-lM","jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/pheonixsolutions.com\/blog\/wp-json\/wp\/v2\/posts\/1350","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/pheonixsolutions.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/pheonixsolutions.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/pheonixsolutions.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/pheonixsolutions.com\/blog\/wp-json\/wp\/v2\/comments?post=1350"}],"version-history":[{"count":0,"href":"https:\/\/pheonixsolutions.com\/blog\/wp-json\/wp\/v2\/posts\/1350\/revisions"}],"wp:attachment":[{"href":"https:\/\/pheonixsolutions.com\/blog\/wp-json\/wp\/v2\/media?parent=1350"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/pheonixsolutions.com\/blog\/wp-json\/wp\/v2\/categories?post=1350"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/pheonixsolutions.com\/blog\/wp-json\/wp\/v2\/tags?post=1350"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}