How to parse JSON files with Python JSONPath

JSON is currently one of the most important formats for ex­chan­ging data between ap­plic­a­tions, es­pe­cially online. JSONPath is an ex­pres­sion language that can be used to read specific data from JSON objects. Here, we’ll look at the Python im­ple­ment­a­tion of JSONPath and where to use it using easy-to-un­der­stand examples.

What is Python JSONPath?

JSON is a cross-system file format that can be used to fa­cil­it­ate the exchange of struc­tured data between ap­plic­a­tions. JSON files consist of listed key-value pairs. The values can accept different types of data, both primitive values and objects. Objects, in turn, can contain their own key-value pairs. Since JSON is un­der­stood by almost all modern systems, it can be used for data exchange between any type of ap­plic­a­tion, both locally on the machine and over the internet.

But not every ap­plic­a­tion needs all its data captured in a JSON file. In these cases, JSONPath is a good choice. JSONPath is an ex­pres­sion language that can be used to read specific in­form­a­tion from JSON objects. In most pro­gram­ming languages, JSONPath needs to be imported from an external library. Because these libraries need to be im­ple­men­ted sep­ar­ately for each language, the libraries or im­ple­ment­a­tions may differ slightly.

The Python jsonpath-ng module

jsonpath-ng is probably the most common Python im­ple­ment­a­tion of JSONPath. There are also other JSONPath im­ple­ment­a­tions for Python, such as jsonpath and jsonpath-rw. But these are less popular and com­pre­hens­ive, so we’ll focus ex­clus­ively on jsonpath-ng here.

In­stall­a­tion

You can install jsonpath-ng very easily from your shell. Simply enter the command pip install jsonpath-ng to start.

Syntax

JSONPath can be used to execute complex queries on JSON objects. There are several methods, operators, and atomic ex­pres­sions in the module that can be used to select and query specific data. The two most important JSONPath methods are parse() and find(). With parse() you can define queries, which can then be ref­er­enced and repeated as often as you like. With find() you can run these queries on JSON data to extract concrete values. The following example explains it further.

import json
import jsonpath_ng as jp
raw_data = '''
{
    "name": "John",
    "age": 30,
    "place of residence": "New York"
}
'''
json_object = json.loads(raw_data)
name_query = jp.parse("$.name")
result = name_query.find(json_object)
print(result[0].value) # output: John

In the example above, JSON data in the form of a string was converted into a dic­tion­ary object using json.loads. This is the format that Python works best with. When creating name_query, the request $.name was defined, which should return the value of name. This was then applied to the JSON object with find(). The result of the request was stored in the variable result and read out with result[0].value.

The find method doesn’t just return the requested value, but also other con­tex­tu­al in­form­a­tion, like the path to the searched value. This in­form­a­tion is returned in the form of a list, where the value you are looking for has an index of 0. Now you can use result[0].value to output the value you are looking for.

In the above example, the dollar sign was used when setting the request. This is an atomic ex­pres­sion that is used to refer to the root object of the JSON. All operators and atomic ex­pres­sions are listed in the following table.

In addition to ex­pres­sions and operators, there are filters that you can use to make your search even more specific. In the Python im­ple­ment­a­tion of JSONPath, these can be executed with Python operators. All symbols that can be used with filters are shown with examples in the following table.

Use case for Python JSONPath

import json
import jsonpath_ng as jp
# JSON data as string
import json
import jsonpath_ng as jp
# JSON data as string
data = """
{
    "cities": [
        {
            "name": "Trenton",
            "state": "New Jersey",
            "residents": 90048,
            "iscapital": true,
            "neighborhood Central West": {
                "residents": 1394    
            }
        },
        {
            "name": "Hamburg",
            "state": "Hamburg",
            "residents": 1841000,
            "iscapital": false
        },
        {
            "name": "New York City",
            "state": "New York",
            "residents ": 8804190
            "iscapital": false
        },
        {
            "name": "Los Angeles",
            "state": "California",
            "residents": 3898767
        }
    ]
}
"""
# Convert data from String to dictionary
json_data = json.loads(data)
# Inquiry: Names of all cities
query1 = jp.parse("cities[*].name")
for match in query1.find(json_data):
    print(match.value)     # output: Trenton, Hamburg, New York City, Los Angeles
# jsonpath_ng.ext import to apply filters
import jsonpath_ng.ext as jpx
# Anfrage: Names of all cities with less than 1.5 million residents 
query2 = jpx.parse("$.cities[?@.residents < 1500000].name")
for match in query2.find(json_data):
    print(match.value)     # output: Trenton
# All fields labelled ‘residents’ 
query3 = jp.parse("$.cities..residents")
match = query3.find(json_data)
for i in match:
    print(i.value)     # output: 1394, 1841000, 8804190, 3898767
# The names of all cities that are not called ‘Trenton’
query4 = jpx.parse('$.cities[?(@.name != "Trenton")].name')
for match in query4.find(json_data):
    print(match.value)     # output: Hamburg, New York City, Los Angeles

In this example, JSON data is specified as a string and then converted to a dic­tion­ary object using loads(). There is only a single array in the root object, which in turn contains 4 cities. Each city has 4 fields that contain the following data:

• City name
• State of the city
• Number of in­hab­it­ants
• Whether the city is the capital or not

New Jersey has as an ad­di­tion­al field called ‘Central West’, which also has a number of in­hab­it­ants.

After the data has been converted into a suitable format, 4 different queries are executed. Their functions and outputs are left as comments in the example. You might notice that five values came back on the third request. That’s because the ‘..’ operator re­curs­ively searches for matching fields. This means all objects are searched, as well as all children of these objects. Ac­cord­ingly, the number of in­hab­it­ants of Central West is listed next to the number of in­hab­it­ants of the cities.