How to use Python pickle to serialise and deserialise objects

With Python pickle it is possible to serialise and later deserialise objects. Numerous data types can be considered for conversion. However, as malicious code can also be stored in a memory file, you should only convert files from trustworthy sources into the original format.

What is Python pickle?

Python pickle may seem to have an unusual name at first, but if you take a closer look at how the module works and what it is used for, you will quickly understand why it was given this name. The module allows you to save objects (i.e. ‘preserve them’, hence ‘pickle’) in order to use them at a later time for another project. To do this, the objects are converted into a saveable format. This practice is called serialising. Python pickle can also be used to deserialise objects, i.e. to convert them back to their original format. Python pickle is particularly useful if you want to use objects frequently.

The object is converted into a byte stream, whereby all information is transferred unchanged. In addition, Python pickle provides instructions for successful deserialisation, through which the original structure can be reconstructed down to the smallest detail. Using Python pickle saves a lot of time, as once an object is created, it does not have to be recreated for each use. The format for saving is .pkl.

Which data types can be converted?

Python pickle can serialise the following data types:

• Boolean values: ‘true’ and ‘false’, also ‘none’
• Integers and complex numbers
• Strings (normal and Unicode)
• Lists
• Sets
• Python tuples
• Directories consisting exclusively of corresponding objects
• Functions
• Python classes

What are the different methods?

There are four methods for working with Python pickle, which are provided within the module:

• pickle.dump(obj, file, protocol=None, *, fix_imports=True, buffer_callback=None): Is used for serialisation and creates a file with the desired result
• pickle.dumps(obj, protocol=None, *, fix_imports=True, buffer_callback=None): Is also used for serialisation, but returns a byte string
• pickle.load(file, *, fix_imports=True, encoding='ASCII', errors="strict", buffers=None): Is used for deserialisation and reads the saved file for this purpose
• pickle.loads(bytes_object, *, fix_imports=True, encoding="ASCII", errors="strict", buffers=None): Is also used for deserialisation, but operates with a byte string

To differentiate between the methods, you can remember that the ‘s’ in pickle.dumps and pickle.loads stands for ‘String’.

How to use Python pickle with an example

To better illustrate how Python pickle works, we will work with a simple example. We will create a simple list containing four colours. This is our code:

import pickle
colours = ['Blue', 'Red', 'Yellow', 'Orange']

Then we open a text file in the .pkl format and use pickle.dump() to save our list there. This is the code we use for this:

with open('colours_file.pkl', 'wb') as f:
	pickle.dump(colours, f)

The abbreviation wb instructs the system to open the file in binary form. This also outputs the contained data as a bytes object. The ‘colours’ list is then saved in this file with dump(). Finally, the file is closed automatically.

How to convert a saved file to its original format

If you now want to deserialise a binary file again, use the Python method pickle.load(). With the following code, it is possible to convert the object back to its original format and initiate an output. We add the abbreviation rb, which stands for ‘read binary’.

with open('colours_file.pkl', 'rb') as f:
	colours_deserialised = pickle.load(f)
	print(colours_deserialised)

This gives us the following output:

['Blue', 'Red', 'Yellow', 'Orange']

How to serialise dictionary with Python pickle

You can also easily serialise more complex data types such as directories with Python pickle and then convert them back to their original form. To do this, we first create a directory with the name ‘persons’. Here we store different data for different people:

import pickle
persons = {
	'Person 1': {
		'Name': "Maria", 'Age': 56, 'City': "London"
	},
	'Person 2': {
		'Name': "Paul", 'Age': 66, 'City': "London"
	},
	'Person 3': {
		'Name': "Lisa", 'Age': 22, 'City': "Cardiff"
	},
	'Person 4': {
		'Name': "Lara", 'Age': 34, 'City': "Southport"
	}
}

In the following code, we create a new file, convert the data, and then test by converting it back to serialise this directory:

with open("persons_dict.pkl", "wb") as f:
	pickle.dump(persons, f)
with open("persons_dict.pkl", "rb") as f:
	deserialised_dict = pickle.load(f)
	print(deserialised_dict)

The resulting output looks like this:

persons = {
    'Person 1': { 'Name': "Maria", 'Age': 56, 'City': "London"},
    'Person 2': { 'Name': "Paul", 'Age': 66, 'City': "London"},
    'Person 3': { 'Name': "Lisa", 'Age': 22, 'City': "Cardiff"},
    'Person 4': { 'Name': "Lara", 'Age': 34, 'City': "Southport"}
}

You can now access the information as usual. We request the following output as an example:

# Define dictionary
deserialised_dict = {
    'Person 1': {'Name': "Maria", 'Age': 56, 'City': "London"},
    'Person 2': {'Name': "Paul", 'Age': 66, 'City': "London"},
    'Person 3': {'Name': "Lisa", 'Age': 22, 'City': "Cardiff"},
    'Person 4': {'Name': "Lara", 'Age': 34, 'City': "Southport"}
}
# Print output
print(
    "The name of the third person is"
    + deserialised_dict["Person 3"]["Name"]
    + " and they are "
    + str(deserialised_dict["Person 3"]["Age"])
    + " years old."
)

This is what our output looks like:

The name of the third person is Lisa and she is 22 years old.

How to convert a class to a string

In the next example, we use Python pickle to save a class in a string. This class contains completely different data types, but all of them can be taken into account. We create a class called ‘ExampleClass’ and then serialise it. The corresponding code is:

import pickle
class ExampleClass:
	def __init__(self):
		self.a_number = 17
		self.a_list = [5, 10, 15]
		self.a_tuple = (18, 19)
		self.a_string = "hello"
		self.a_dict = {"colour": "blue", "number": 3}
example_object = ExampleClass()
serialised_object = pickle.dumps(example_object)
print(f"This is a serialised object:\n{serialised_object}\n")
example_object.a_dict = None
deserialised_object = pickle.loads(serialised_object)
print(f"This is a_dict from the deserialised object:\n{deserialised_object.a_dict}\n")

After serialising the class and then converting it back to its original format, we get this output:

This is a serialised object:
b'\x80\x03c__main__\nExampleClass\nq\x00)\x81q\x01.'
This is a_dict from the deserialised object:
{'colour': 'blue', 'number': 3}

How to compress serialised objects

In principle, files saved with Python pickle are comparatively compact. Nevertheless, it is also possible and, in some cases, advisable to compress the memory files even more. This works, for example, with the free compression program bzip2, which is part of the programming language’s standard library. In the following example, we create a string, serialise it and then apply the compression program:

import pickle
import bz2
exampleString = """Almost heaven, West Virginia
Blue Ridge Mountains, Shenandoah River
Life is old there, older than the trees
Younger than the mountains, growin' like a breeze
Country roads, take me home
To the place I belong
West Virginia, mountain mama
Take me home, country roads."""
serialised = pickle.dumps(exampleString)
compressed = bz2.compress(serialised)

Security instructions for working with Python pickle

Even though Python pickle is a practical and effective method for converting objects, there is one major drawback that you should be aware of when working with the module, which is that there is the possibility of transferring malicious code via serialised data. Although this is not a problem with your own data, you have to be careful when it comes to third-party files. Therefore, only ever deserialise memory files when you know and trust the source!