In this post, we are going to see how to download a file from URL using ansible get_url module in other words ansible curl. As we know it already, Ansible has a lot of built-in modules to accomplish all the tasks we might need for configuration management and automation.
Ansible get_url module is one amongst them which helps us to access the remote URL (HTTP/HTTPS/FTP) and download a file/package from there and save it locally.
Let us consider a situation where we need to download a package or a product online as we are provisioning an infrastructure. Let us say we build a machine with Apache or NGINX or Tomcat.
Based on the flavour of OS you choose you always have an option to download these products using their repository and installation management tool like yum (or) apt-get etc. But that does not suit all our needs. let us say I want to download a package or a file directly from the URL. How can I do that?
In a Simple approach, we can use the CURL or WGET command and get this job done. But why do you have to reinvent something which is already present and provides more features. ( You can always upgrade it )
So let's call this ansible get_url as Ansible CURL.
In this post, we are going to see how to use the ansible curl or ansible get_url module with examples.
If you want to simply access the web page or web site or work with SOAP or RESTFUL web services and crawl through the content of the body. you must consider using the ansible
The ansible get_url module
Some Key points about get_url module to get to know it better.
- ansible get_url module is being used to download files from HTTPS, HTTP and FTP servers (websites/URLs)
- By Default use the default proxy configuration of the node, You can use custom proxy and you can change the proxy address/url by setting environment variables such as
https_proxyor by using the ansible built-in
- you can use get_url to crawl through the page and check if the web page or server is available using
- Supports HTTPS and Secure Site accessing by letting you define the client certificates
- Timeout configuration is available ( something important I believe)
- Can pass through or handle the
Basic Authentication.In other words, you can access the web sites/pages which are secure with Basic Authentication.
checksumvalidation to make sure the downloaded software/package is uncorrupted and legit.
The ansible get_url Examples
- How to download a file from URL [ Download Tomcat version 8.40 from HTTPS URL with checksum validation.(SHA512) ]
- Validate Checksum with SHA256/MD5 keys
- Download a file ( tomcat) with checksum validation and take checksum value from remote URL
- Download a file Secured with Basic Authentication
- Timeout a Request when the URL is not responding (or) taking longer than usual
- Downloading a FILE from FTP server using ansible get_url
Example #1 - Ansible - How to download a file from URL
Here is a quick playbook that downloads Tomcat 8.40 version from
Before I write this playbook. I manually have to be aware of the downloadable link and the SHA512 checksum to validate. I have collected them from here
appserver is the
host group defined in the
ansible_hosts file which has one remote node or server underneath. Named
The Playbook consists of two tasks and a couple of subtasks
- Create a Directory named /opt/tomcat8
- Make Sure the Directory has 0755 permission set
- Set the Owner and the Group of the Directory as
- Download the Tomcat8 tar.gz file into the /opt/tomcat8 directory we created in Step1
- Access the Download link of the tomcat8 using
- Set the Download location as
- Make Sure the downloaded file at
- Make Sure the owner and group of the downloaded tar.gz file are
- Validate the
sha512checksum of the downloaded file, to make sure the file is valid.
- Access the Download link of the tomcat8 using
Here the checksum value was pre declared (or) written in side the playbook. you can optionally specify a URL in which the checksum value of the desired file.
--- - name: Download Tomcat8 from tomcat.apache.org hosts: appserver tasks: - name: Create a Directory /opt/tomcat8 become: yes file: path: /opt/tomcat8 state: directory mode: 0755 owner: tomcat group: tomcat - name: Download Tomcat using get_url become: yes get_url: url: https://www-us.apache.org/dist/tomcat/tomcat-8/v8.5.40/bin/apache-tomcat-8.5.40.tar.gz dest: /opt/tomcat8 mode: 0755 checksum: sha512:5bdea5414713c9ba39e226f062701fa14998b1a798c9750f956a0f59b5edabb8d83af9ec9f81cf9f47fa92c21b560c9b2be1b543d0bd8f1b49579b69101d3a8f group: tomcat owner: tomcat
The Execution Output ( When the CheckSum Match Perfectly)
The Execution Output ( When the Checksum Do not match)
Since we have given checksum validation as part of the get_url module. When the checksum is not matching, The Task would fail and deployment would not be completed
Quick Validation - Ad hoc command to check if the file is present
How to use SHA256 and MD5 Checksums while downloading the file
How to use SHA256 and MD5 Checksums while downloading the file.
To use SHA256 of MD5 checksums. All you have to do is update the checksum parameter of the
If you want to use SHA256 checksum
If you want to use MD5 checksum
You just have to define what is the Key algorithm before the value. That's all. It applies to all SHA256,MD5 or SHA256
Example #2 - Ansible - Download a file and get checksum from the URL
In the last example, we have seen how to download a file and do the checksum validation.
If you have noticed, the checksum was hardcoded or written inside the playbook. as we mentioned earlier you can also tell the ansible get_url module to fetch the checksum value from another URL
There is no major difference between example1 and example2 except the value of the checksum option.
--- - name: Download Tomcat8 from tomcat.apache.org hosts: appserver tasks: - name: Create a Directory /opt/tomcat8 become: yes file: path: /opt/tomcat8 state: directory mode: 0755 owner: tomcat group: tomcat - name: Download Tomcat using get_url become: yes get_url: url: https://www-us.apache.org/dist/tomcat/tomcat-8/v8.5.40/bin/apache-tomcat-8.5.40.tar.gz dest: /opt/tomcat8 mode: 0755 checksum: sha512:https://www.apache.org/dist/tomcat/tomcat-8/v8.5.40/bin/apache-tomcat-8.5.40.tar.gz.sha512 group: tomcat owner: tomcat
Example #3 - Ansible - Download a Protected file with Basic Authentication
Sometimes, we might need to cross the security layer like Basic Authentication Prompt before downloading the file. In those kinds of scenarios, it is important that we pass the credentials and get authenticated before downloading the file.
get_url provides a way to pass
password to Basic Authentication prompts. So that you can cross the security layer and reach and download the file
When you are not passing any credentials to the secured web page/ web site. You will get
401 Unauthorized HTTP Response
A Sample Basic Authentication Enabled File/URL
For testing, we have tweaked a simple Apache WebSite configuration to enable Basic Authentication Security for a specific file named
"secure.txt" under the
/var/www/html/admin document root
These are the apache configuration directives we had to add in httpd.conf file to achieve the Basic Authentication to a specific directory ( /var/www/html/admin )
Here you can see we have three major tasks
- Accessing the URL to make sure it is alive and Secured by validating it returns 401 status_code
- Access the same URL with username and password this time to download the file
- Display the file contents downloaded using a Basic Shell command
--- - name: Get-URL Download the Protected File hosts: appserver tasks: - name: Access the WebSite and make sure the URL is live uri: url: http://mwiweb02/admin/secure.txt status_code: 401 register: validateurl - name: "INFO: HTTP Response for the URL" debug: var=validateurl.msg - name: Access the same URL with Basic Authentication and Download the file get_url: url: http://mwiweb02/admin/secure.txt url_password: Test@123 url_username: sarav dest: /tmp/secure.txt - name: Display the File contents shell: > cat /tmp/secure.txt register: shellout - name: "INFO: Display the File Contents" debug: var=shellout.stdout_lines
The Execution Result
Example#4: Timeout When the URL is not responding ( Default is 10 Seconds)
We will take the same playbook we have used in Example1, and quickly update the timeout option for the get_url module. By default, the ansible get_url module would timeout in 10 seconds if gets no reply from the URL
Here is the modified playbook with the timeout option
--- - name: Download Tomcat8 from tomcat.apache.org hosts: appserver tasks: - name: Create a Directory /opt/tomcat8 become: yes file: path: /opt/tomcat8 state: directory mode: 0755 owner: tomcat group: tomcat - name: Download Tomcat using get_url become: yes get_url: url: https://www-us.apache.org/dist/tomcat/tomcat-8/v8.5.40/bin/apache-tomcat-8.5.40.tar.gz dest: /opt/tomcat8 mode: 0755 group: tomcat owner: tomcat # the request will timeout after 20 seconds timeout: 20
Example#5: Download a file from FTP server using ansible get_url
I have my FTP server running at
ftp://mwiweb02/ and I need to access a Secure file like we shown in the previous example which is available at the URL
Just like the other examples, this is also very simple and all we have to do is replace the HTTP URL with an FTP URL. That's all.
--- - name: Get-URL Download the Protected File hosts: appserver tasks: - name: Access the FTP URL and download the file get_url: url: ftp://mwiweb02/admin/secure.txt dest: /tmp/secure.txt - name: Display the File contents shell: > cat /tmp/secure.txt register: shellout - name: "INFO: Display the File Contents" debug: var=shellout.stdout_lines
The Execution Output
Hope this post helps. Read more Ansible articles here
Any help or feedback. Please let us know in comments.
Signup for Exclusive "Subscriber-only" Content