The free software cURL is one of the oldest and most popular open source projects. The program is written in C and is used to transfer data into computer networks. The name cURL stands for “client URL.” Because it has an open license, pro­gram­mers can use the program for a wide range of purposes. Currently, cURL is being used in countless devices.

What is cURL?

The cURL software consists of two com­pon­ents. The libcurl software library acts as the backbone for data transfer and supports the following protocols:

  • DICT
  • FILE
  • FTP
  • FTPS
  • GOPHER
  • HTTP
  • HTTPS
  • IMAP
  • IMAPS
  • LDAP
  • LDAPS
  • POP3
  • POP3S
  • RTMP
  • RTSP
  • SCP
  • SFTP
  • SMB
  • SMBS
  • SMTP
  • SMTPS
  • TELNET
  • TFTP

The cURL command-line program, on the other hand, acts as a text-based interface and interacts with libcurl via the command line.

This program is an important tool for web de­vel­op­ment. It allows de­velopers to com­mu­nic­ate with servers directly instead of having to go through a browser. Scripts based on cURL commands are used to automate processes, testing, and debugging.

How does cURL work?

The two com­pon­ents in cURL work in different ways.

What is libcurl used for?

The libcurl software library provides functions for trans­fer­ring data in computer networks. There are language bindings for dozens of popular pro­gram­ming languages. These make it easy for libcurl functions to be used in a wide variety of software programs that com­mu­nic­ate with servers.

What is cURL used for?

The cURL command-line program is used for web de­vel­op­ment. The simplest method is to enter cURL commands into the command line. With the right know-how, it can be used to test and debug servers and APIs.

Instead of typing commands into the command line manually, they can be bundled in scripts. This allows complex op­er­a­tions to be stand­ard­ized and automated. This includes both uploading and down­load­ing of data and pro­gram­mat­ic­ally filling out forms and mirroring entire websites.

The following is the general syntax of a cURL command:

# General syntax of a cURL command
curl [options] <url>

We will use the following form in the examples below:

# Curling the following URL
url="www.example.com"
curl [options] "$url"

We have turned the actual URL into a variable. This allows the focus to be placed on the options. The options determine how the cURL request works.

Practical examples of the cURL command in Linux

To use the cURL examples below, you will need the following:

  1. A computer with Linux or a Unix-like operating system — including macOS
  2. Access to the command-line window or terminal
  3. A code editor/plain-text editor for writing commands
Note

You should only use a code editor/plain-text editor to write commands. Under no cir­cum­stances should you use a word pro­cessing program (e.g. Word, Open­Of­fice, or Lib­reOf­fice) to prepare text for the command line.

An initial test with cURL

First, test whether cURL is installed on your system. To do so, open a command-line window and enter the following command there. You may need to press Enter to execute the code.

# Check whether cURL is installed
curl --version

If you now see text starting with “curl” and a version number, this means cURL is installed. If cURL is not installed, please follow the in­struc­tions in “Everything curl” to install the program.

Tip

Use the command “curl --help” and “curl --manual” to learn more about the cURL command.

You should also open a new document in a code editor. You can copy commands to prepare them for use in the command-line window. Once the command is ready, copy it to the command-line window and execute it there.

Note

Commands used in the command-line window are quite powerful. In principle, an in­cor­rectly entered command can cripple the entire system. Therefore, you should not just copy commands you find on the internet and execute them in your home command-line window. Instead, you should have a blank document open in a code editor where you can copy the commands first. This in­ter­me­di­ate step allows you to check each command and modify it when necessary before executing it.

Down­load­ing files with cURL

Re­triev­ing data from a server with cURL

In principle, you can use cURL to retrieve any URL. In this context, the term “curl” is used as a verb to describe re­quest­ing a URL with cURL. Test the following example:

# Curling the following website
site="www.google.com"
curl "$site"

If you execute the code like written, it will output “data salad.” This is because the returned response to the cURL access will be the Google homepage in HTML. The HTML source code will be directly displayed in the command-line window without any format­ting. A browser will be required to display it correctly. It is usually more useful to save the file locally.

Tip

Use the “clear” command in the command-line window to clear the screen. This will get rid of undesired data in the command-line window.

Re­triev­ing data with cURL and saving it locally

Let us download a logo from the English version of Wikipedia. The “-O” option (upper case o, not a zero) tells cURL to use the name of the file at the end of the URL. When you use this option, cURL will save the down­loaded file locally under the same name.

# An image from the English version of Wikipedia
file="https://en.wikipedia.org/static/images/project-logos/enwiki-2x.png"
# Retrieving the image and saving it locally under the same name
curl “$file" -O

But what if the URL does not contain a file name? Test the following code:

# Google homepage
homepage="www.google.com"
# Retrieving the homepage with the -O option
curl "$homepage" -O

As you can see, an error is displayed because the URL of the Google homepage does not contain a file name. In this case, you need to use the “-o” option (lower case o) to assign the file a name yourself:

# Google homepage
homepage="www.google.com"
# Name for the file to be created
name="homepage-google.html"
# Retrieving the homepage and saving it locally under the chosen name
curl "$homepage" -o "$name"

Resuming a download with cURL if it is in­ter­rup­ted

You may have run into this annoying situation before: you are per­form­ing a large download which has already taken hours when it is suddenly in­ter­rup­ted forcing you to start all over again. In this instance, cURL can help with the “-C-” option:

# A large file (human genome) to be downloaded
file="https://ftp.ncbi.nih.gov/genomes/refseq/vertebrate_mammalian/Homo_sapiens/reference/GCF_000001405.39_GRCh38.p13/GCF_000001405.39_GRCh38.p13_genomic.gbff.gz"
# Resuming the download if it is interrupted
curl -C- -O “$file"
Note

Tech­nic­ally, the option is “-C -” with a space after the C, but the similar version “-C-” is easier to read and remember, so we are using that here.

You can also manually cancel a download that was initiated with cURL if you need to. This is useful for larger files when, for instance, you have to leave the house and take your laptop with you.

Tip

You can cancel the current cURL request. To do so, press Ctrl+C. You may need to do this multiple times.

Com­mu­nic­at­ing with the server using cURL

The download func­tion­al­ity of cURL is similar to the popular command-line tool wget. However, cURL is not spe­cific­ally designed for per­form­ing downloads. It is designed for general com­mu­nic­a­tion in networks. cURL can thus do much more.

Using cURL to test whether a server is available

You can use cURL to test whether a server is available. In this regard, the way cURL works is similar to the ping command. However, cURL is more versatile due to the number of protocols and options available. In addition, ping works on an internet layer while cURL runs on the ap­plic­a­tion layer. This means that ping tests whether the machine is on the network. In contrast, the following cURL command checks whether and how a server responds:

# Testing whether a web server is available
server="google.com"
curl -I "$server"

If the status in­form­a­tion is displayed after executing the code, this means that the server is available.

Out­put­ting a header using cURL

In each HTTP request, metadata is exchanged in addition to the actual document. This data is called the HTTP Header and describes both the document and the status of the HTTP request. You have probably already en­countered the 404 Not Found error message at one time or another. The error is a type of metadata – the requested document was not found.

Take a look at the output of the header for the URL “google.com.” To do so, use cURL with the “--head” option:

# Outputting the header
url="google.com"
curl --head "$url"

You should see “301 Moved Per­man­ently,” among other in­form­a­tion. This is the HTTP status code for a permanent redirect. What you are seeing is a redirect from [google.com] (without the “www”) to [www.google.com].

Tip

Instead of using the “--head” option, you can use the equi­val­ent version “-I” (upper case i). It is easy to remember using a mnemonic: you use it to retrieve inform­a­tion for a URL instead of its actual content.

Analysing redirect chains using cURL

HTTP redirects can be linked together in a sequence. We call this a “redirect chain.” For example, take a website with a homepage that displays the address [https://www.example.com/]. If you try to access the website using the address [http://example.com], this may result in the following redirects:

  • [http://example.com] → [https://example.com]
  • [https://example.com] → [https://www.example.com]
  • [https://www.example.com] → [https://www.example.com/]

Chaining redirects results in un­ne­ces­sar­ily long load times and should be avoided. Un­for­tu­nately, this is not an easy problem to deal with. When the website is loaded into the browser, the redirects occur without a user seeing much of it. In this instance, cURL can help with the “--location” option: this tells cURL to follow the redirects to the last one. Here we are using the “--head” option because we are not in­ter­ested in the content of the websites.

# Checking redirects
url="google.com"
curl --location --head  "$url"
Tip

Instead of using the “--location” option, you can use the equi­val­ent version “-L” (upper case l).

Trans­fer­ring data to a server with cURL

In addition to re­triev­ing data, you can also use cURL to transfer data to a server. This may be practical if you want to automate the process of filling out an online form, for example. cURL supports the POST request method as well as GET.

Since trans­fer­ring data is more complex than re­triev­ing it, we can only provide a rough example here. For more detailed in­form­a­tion, read about the HTTP POST method in Everything curl.

# Transferring data to a server
url="example.com"
# Providing the data as key value pairs separated by the “&” symbol
data="firstname=Peter&lastname=Mustermann&age=42"
# This cURL request transfers the data via POST
curl --data "$data" "$url"
# Alternatively, tell cURL to transfer the data via GET
curl --data "$data" "$url" --get

Accessing cookies with cURL

HTTP cookies are small pieces of data that are stored locally on your device when you visit most websites. You can view the cookies from websites you visited in your browser and delete them from there. However, this way is usually too tedious for testing purposes. You can also use the cURL option “--cookie-jar” to access the cookies more directly.

# Accessing cookies
url="www.google.com"
cookies="cookies.txt"
curl --head --cookie-jar "$cookies" "$url"
# Displaying cookies with the cat command
cat "$cookies"
Tip

You can also tell cURL to transfer cookies using the “--cookie” option.

General options

Some of the cURL options available can be combined with the ones we have already gone over.

Dis­play­ing ad­di­tion­al in­form­a­tion

Sometimes, the in­form­a­tion displayed when you execute a cURL command is not quite suf­fi­cient. In such a case, you can use the “--verbose” option. When you execute a cURL command with this option, it will provide more in­form­a­tion.

# Displaying additional information
url="www.google.com"
curl --verbose -I "$url"

Entering a username and password with cURL

Some URLs are protected against un­au­thor­ised access with HTTP au­then­tic­a­tion. But what if you want to access this kind of URL with cURL? If you do not provide a username and password, you will receive the HTTP 401 error message. In this situation, use the following syntax:

# Entering a username and password with cURL
# Password-protected URL 
url="www.example.com/secure/"
# User
user=""
# Password
password=""
curl --user "${user}:${password}" "$url"
Tip

You can also use cURL to retrieve data from an FTP server. To do so, you will once again use the --user option to enter the username and password.

Using a proxy with cURL

A proxy is an in­ter­me­di­ary server. Proxies can be useful for a number of reasons. For example, some security and per­form­ance re­quire­ments can be more easily met by using a proxy.

You need to tell cURL if you are con­nect­ing to the internet using a proxy. To do so, use the “--proxy” option:

# Using a proxy with cURL
url="www.google.com"
proxy="proxy.example.com"
port="8080"
curl --proxy "${proxy}:${port}" "$url"
Note

If you need to enter a username and password for the proxy server, you can do so by using the “--proxy-user” option as well. If you do, enter the username and password in the following format: “user:password”.

Go to Main Menu