How to use the Python Pandas library for data analysis and ma­nip­u­la­tion

Python Pandas is an open-source library spe­cific­ally designed for analysing and ma­nip­u­lat­ing data. It provides pro­gram­mers with data struc­tures and functions that simplify the handling of numerical tables and time series.

What is Python Pandas used for?

The Pandas library is widely used in various areas of data pro­cessing, thanks to its extensive functions that support a range of ap­plic­a­tions:

-Ex­plor­at­ory Data Analysis (EDA): Python Pandas fa­cil­it­ates the ex­plor­a­tion and general un­der­stand­ing of data sets. With functions such as describe(), head() or info(), de­velopers can quickly gain insights into the data sets and recognise stat­ist­ic­al cor­rel­a­tions.

• Data cleansing and pre­pro­cessing: Data from diverse sources often needs to be cleansed and brought into a con­sist­ent format before it can be analysed. Here too, Pandas offers a variety of functions for filtering or trans­form­ing data.
• Data ma­nip­u­la­tion and trans­form­a­tion: The main task of Pandas is the ma­nip­u­la­tion, analysis, and trans­form­a­tion of data sets. Functions such as merge() or groupby() enable complex data op­er­a­tions.
• Data visu­al­isa­tion: Another practical field of ap­plic­a­tion arises in com­bin­a­tion with libraries such as Mat­plot­lib or Seaborn. In this way, Pandas data frames can be converted directly into mean­ing­ful diagrams or plotted.

Ad­vant­ages of Python Pandas

Python Pandas offers numerous ad­vant­ages that make it an in­dis­pens­able tool for data analysts and re­search­ers. The intuitive and easy to un­der­stand API ensures a high level of user-friend­li­ness. Since the central data struc­tures of Python Pandas – DataFrame und Series– are similar to spread­sheets, getting started is not too difficult either.

Another key advantage of Python Pandas is its per­form­ance. Although Python is regarded as a rather slow pro­gram­ming language, Pandas can process even large data sets ef­fi­ciently. This is because the library is written in C and uses optimised al­gorithms.

Pandas supports various data formats, including CSV, Excel, and SQL databases, allowing for easy import and export from diverse sources, which adds im­press­ive flex­ib­il­ity. Its in­teg­ra­tion with existing libraries in the Python ecosystem, such as NumPy or Mat­plot­lib, further enhances its ver­sat­il­ity and enables com­pre­hens­ive data analysis and modelling.

A practical example of the Pandas syntax

To il­lus­trate the basic syntax of Pandas, let’s look at a simple example. Suppose we have a CSV dataset that contains in­form­a­tion about sales. We’ll load this dataset, examine it, and perform some basic data ma­nip­u­la­tion. The data set is struc­tured as follows:

Date,Product,Quantity,Price
2024-01-01,Product A,10,20.00
2024-01-02,Product B,5,30.00
2024-01-03,Product C,7,25.00
2024-01-04,Product A,3,20.00
2024-01-05,Product B,6,30.00
2024-01-06,Product C,2,25.00
2024-01-07,Product A,8,20.00
2024-01-08,Product B,4,30.00
2024-01-09,Product C,10,25.00

Step 1: Importing pandas and loading the data set

Once Python Pandas has been imported, you can create a dataframe from the CSV data using read_csv().

import pandas as pd
# Load the data record from a CSV file named sales_data.csv
df = pd.read_csv('sales_data.csv')

Step 2: Examining the data set

An initial overview of the data can be obtained by dis­play­ing the first lines and a stat­ist­ic­al summary of the data set. The functions head() and describe() are used for this purpose. The latter provides an overview of important stat­ist­ic­al key figures such as the minimum and maximum value, the standard deviation or the mean value.

# Display the first five lines of the data frame
print(df.head())
# Display a statistical summary
print(df.describe())

Step 3: Ma­nip­u­lat­ing the data

Data ma­nip­u­la­tion also works with Python Pandas. In the following code snippet, the sales data is to be ag­greg­ated by product and month:

# Convert the ‘Date’ column into a datetime object so that the dates are recognised as such
df['Date'] = pd.to_datetime(df['Date'])
# Extract the month from the ‘Date’ column and save it in a new column called ‘Month’
df['Month'] = df['Date'].dt.month
# Calculate the revenue (Quantity * Price) and save it in the column called ‘Revenue’
df['Revenue'] = df['Quantity'] * df['Price']
# Aggregate sales data by product and month
sales_summary = df.groupby(['Product', 'Month'])['Revenue'].sum().reset_index()
# Display aggregated data
print(sales_summary)

Step 4: Visu­al­ising the data

Finally, you can visualise the monthly sales figures of a product using the ad­di­tion­al Python library Mat­plot­lib.

import matplotlib.pyplot as plt
# Filter data for a specific product
product_sales = sales_summary[sales_summary['Product'] == 'Product A']
# Create a line diagram 
plt.plot(product_sales['Month'], product_sales['Revenue'], marker='o')
plt.xlabel('Month')
plt.gca().set_xticks(product_sales['Month'])
plt.ylabel('Turnover')
plt.title('Monthly turnover for product A')
plt.grid(True)
plt.show()

The visu­al­ised diagram indicates that in the first month of the year, £940 was generated from product A:

Close