How to send HTTP requests with Python

Python requests is a popular library that allows you to send HTTP requests and inspect the responses. Although .get() is the module’s most important method, there are many other options available.

What are Python requests?

Python requests is a standard that allows you to send HTTP requests within the programming language. The basis for this is a clear and effective program interface that makes the integration of third-party features particularly easy. The Python requests library provides various tools with which you can send HTTP requests and receive the corresponding responses. Even though the Python requests module is widely used and highly recommended, it is not part of the standard Python installation.

How to install and start the Python requests library

Before you install Python requests, it is recommended that you first set up a virtual machine where you can safely test the library. Then use pip for the installation. This is what the code looks like:

$ python -m pip install requests

To use the library after successfully installing it, this is the code you require:

import requests

Functionality, syntax and methods

Similar to other tools, Python’s requests module sends an HTTP request and receives a response upon successful transmission. This response is an object containing key information like content, encoding, and status. The syntax for a request like this typically follows this structure:

requests.method(url, **kwargs)

Even though .get() is the function you will use most often, Python requests provides you with other methods. The following functions are available:

• .delete(url, **kwargs): Use .delete() to remove a specific source
• .get(url, parameter=None, **kwargs): Use .get() to request information from a server
• .head(url, **kwargs): With .head(), you also make a request to a server, but limit yourself to the header
• .patch(url, data=None, **kwargs): You can increase the performance with .patch(). This only sends changes to the source
• .post(url, data=None, json=None, **kwargs): With .post() you mainly transmit form data to a server
• .put(url, data=None, **kwargs): Use .put() to modify an existing source and generate new data on the server
• .request(method, url, **kwargs): With .request() you send a request with a specific method to the URL

Example for using the .get() method

The .get() method is highly intuitive, making Python’s requests library extremely useful. All that’s required is the method itself and the URL you wish to access, which is enclosed in quotation marks. Here’s an example of how the code looks:

import requests
requests.get("https://example.com")

You will now receive a response from your target server. For demonstration purposes, you can also display it in a variable. In our example, it would look like this:

import requests
response = requests.get("https://example.com")

Retrieve status codes

The simplest response you will receive to a request is an HTTP status code. This indicates whether your request was successful or not. The status code is three digits long. There are numerous variations that inform you about the progress of your request. The most important categories are as follows:

• 1XX: Informational responses
• 2XX: Successful responses
• 3XX: Redirection messages
• 4XX: Client error responses
• 5XX: Server error responses

Some of the most commonly known status codes are ‘200 – OK’ (indicating a successful request) and ‘404 - Not Found’ (when the requested data is not found on the server).

After making a request using Python requests, you can check the status with .status_code. Here’s the code for that:

response.status_code

If the request was successful, you will receive this output:

200

The following code can be used for better visualisation:

if response.status_code == 200:
	print("The request was successful.")
elif response.status_code == 404:
	print("The request was unsuccessful.")

How to view headers with Python requests

The header of an HTTP response contains a lot of useful information. Among other things, you will find the transmitted data type, a time limit for buffering and other information. With the Python requests library, you can also easily display the header. To do this, first execute the function .get() and then .headers:

import requests
response = requests.get("https://example.com")
response.headers

In the output, you’ll get an object that includes the values from the header. You can also retrieve them individually by specifying the command. For instance, if you’re looking for information about the data type, use this input:

response.headers["content-type"]

How to access the payload

The data packet transferred between the server and client is referred to as the ‘payload’. It is found in the body of the response. You can retrieve the information stored there using response. To display it in bytes, this command can be used:

import requests
response = requests.get("https://example.com")
response.content
type(response.content)

If you want to convert the information collected in this way into a string, use this code:

response.text
type(response.text)

Python requests will take the encryption from the header. Alternatively, you can instruct the system to use a different method. This code is an example of this:

response.encoding = "utf-8"
response.text

How to check the progress

When you send a request, it is prepared by Python requests to avoid unwanted errors. Among other things, the header is checked. You can monitor the progress of this method using the .request function. For this, use the following code:

import requests
response = requests.post("https://example.com", json={"key": "value"})
print(response.request.headers["content-type"])
print(response.request.url)
print(response.request.body)

This gives you information about the payload, the header, the URL and many other points.

Authenticity check via Python requests module

The auth parameter is used in Python requests to send an authentication to the server. The server can perform an authentication check and therefore check who it is actually interacting with. To use the check, use this code:

import requests
from requests.auth import HTTPBasicAuth
response = requests.get("https://your_site.com/basic-auth/user/passwd", 
	auth=HTTPBasicAuth("user", "passwd"))
print(response.status_code)
print(response.request.headers["Authorization"])

How to use an SSL certificate

It is always advisable to use an SSL certificate to prevent data loss and unauthorised access. Python requests offers this option by default. However, if you want to prevent mutual verification for a request, you can switch off SSL with this code:

import requests
requests.get("https://example.com", verify=False)

However, you will then receive an explicit warning message.

How to initiate timeout with Python requests

With Python requests you can also limit the maximum time for a response. By default, the client will wait for a response without having a time limit. This can mean that requests back up and performance suffers as a result. You can prevent this problem with the timeout parameter. In the following code, we instruct Python requests to abort the request after two seconds:

requests.get("https://example.com", timeout=2)