How to send HTTP requests with Python

Python requests is a popular library that allows you to send HTTP requests and inspect the responses. Although .get() is the module’s most important method, there are many other options available.

What are Python requests?

Python requests is a standard that allows you to send HTTP requests within the pro­gram­ming language. The basis for this is a clear and effective program interface that makes the in­teg­ra­tion of third-party features par­tic­u­larly easy. The Python requests library provides various tools with which you can send HTTP requests and receive the cor­res­pond­ing responses. Even though the Python requests module is widely used and highly re­com­men­ded, it is not part of the standard Python in­stall­a­tion.

How to install and start the Python requests library

Before you install Python requests, it is re­com­men­ded that you first set up a virtual machine where you can safely test the library. Then use pip for the in­stall­a­tion. This is what the code looks like:

$ python -m pip install requests

To use the library after suc­cess­fully in­stalling it, this is the code you require:

import requests

Func­tion­al­ity, syntax and methods

Similar to other tools, Python’s requests module sends an HTTP request and receives a response upon suc­cess­ful trans­mis­sion. This response is an object con­tain­ing key in­form­a­tion like content, encoding, and status. The syntax for a request like this typically follows this structure:

requests.method(url, **kwargs)

Even though .get() is the function you will use most often, Python requests provides you with other methods. The following functions are available:

• .delete(url, **kwargs): Use .delete() to remove a specific source
• .get(url, parameter=None, **kwargs): Use .get() to request in­form­a­tion from a server
• .head(url, **kwargs): With .head(), you also make a request to a server, but limit yourself to the header
• .patch(url, data=None, **kwargs): You can increase the per­form­ance with .patch(). This only sends changes to the source
• .post(url, data=None, json=None, **kwargs): With .post() you mainly transmit form data to a server
• .put(url, data=None, **kwargs): Use .put() to modify an existing source and generate new data on the server
• .request(method, url, **kwargs): With .request() you send a request with a specific method to the URL

Example for using the .get() method

The .get() method is highly intuitive, making Python’s requests library extremely useful. All that’s required is the method itself and the URL you wish to access, which is enclosed in quotation marks. Here’s an example of how the code looks:

import requests
requests.get("https://example.com")

You will now receive a response from your target server. For demon­stra­tion purposes, you can also display it in a variable. In our example, it would look like this:

import requests
response = requests.get("https://example.com")

Retrieve status codes

The simplest response you will receive to a request is an HTTP status code. This indicates whether your request was suc­cess­ful or not. The status code is three digits long. There are numerous vari­ations that inform you about the progress of your request. The most important cat­egor­ies are as follows:

• 1XX: In­form­a­tion­al responses
• 2XX: Suc­cess­ful responses
• 3XX: Re­dir­ec­tion messages
• 4XX: Client error responses
• 5XX: Server error responses

Some of the most commonly known status codes are ‘200 – OK’ (in­dic­at­ing a suc­cess­ful request) and ‘404 - Not Found’ (when the requested data is not found on the server).

After making a request using Python requests, you can check the status with .status_code. Here’s the code for that:

response.status_code

If the request was suc­cess­ful, you will receive this output:

200

The following code can be used for better visu­al­isa­tion:

if response.status_code == 200:
	print("The request was successful.")
elif response.status_code == 404:
	print("The request was unsuccessful.")

How to view headers with Python requests

The header of an HTTP response contains a lot of useful in­form­a­tion. Among other things, you will find the trans­mit­ted data type, a time limit for buffering and other in­form­a­tion. With the Python requests library, you can also easily display the header. To do this, first execute the function .get() and then .headers:

import requests
response = requests.get("https://example.com")
response.headers

In the output, you’ll get an object that includes the values from the header. You can also retrieve them in­di­vidu­ally by spe­cify­ing the command. For instance, if you’re looking for in­form­a­tion about the data type, use this input:

response.headers["content-type"]

How to access the payload

The data packet trans­ferred between the server and client is referred to as the ‘payload’. It is found in the body of the response. You can retrieve the in­form­a­tion stored there using response. To display it in bytes, this command can be used:

import requests
response = requests.get("https://example.com")
response.content
type(response.content)

If you want to convert the in­form­a­tion collected in this way into a string, use this code:

response.text
type(response.text)

Python requests will take the en­cryp­tion from the header. Al­tern­at­ively, you can instruct the system to use a different method. This code is an example of this:

response.encoding = "utf-8"
response.text

How to check the progress

When you send a request, it is prepared by Python requests to avoid unwanted errors. Among other things, the header is checked. You can monitor the progress of this method using the .request function. For this, use the following code:

import requests
response = requests.post("https://example.com", json={"key": "value"})
print(response.request.headers["content-type"])
print(response.request.url)
print(response.request.body)

This gives you in­form­a­tion about the payload, the header, the URL and many other points.

Au­then­ti­city check via Python requests module

The auth parameter is used in Python requests to send an au­then­tic­a­tion to the server. The server can perform an au­then­tic­a­tion check and therefore check who it is actually in­ter­act­ing with. To use the check, use this code:

import requests
from requests.auth import HTTPBasicAuth
response = requests.get("https://your_site.com/basic-auth/user/passwd", 
	auth=HTTPBasicAuth("user", "passwd"))
print(response.status_code)
print(response.request.headers["Authorization"])

How to use an SSL cer­ti­fic­ate

It is always advisable to use an SSL cer­ti­fic­ate to prevent data loss and un­au­thor­ised access. Python requests offers this option by default. However, if you want to prevent mutual veri­fic­a­tion for a request, you can switch off SSL with this code:

import requests
requests.get("https://example.com", verify=False)

However, you will then receive an explicit warning message.

How to initiate timeout with Python requests

With Python requests you can also limit the maximum time for a response. By default, the client will wait for a response without having a time limit. This can mean that requests back up and per­form­ance suffers as a result. You can prevent this problem with the timeout parameter. In the following code, we instruct Python requests to abort the request after two seconds:

requests.get("https://example.com", timeout=2)