Computing quantile values for a pandas DataFrame

Quantile in statistics is a value at which or below lies a given percentage of value of the distribution. The pandas quantile() function of the DataFrame class computes the quantile value for the given fraction.

Dataframe Computations Updated 4 Jun 2019 3 min read

Overview

A class room of students can be grouped based on their height – students with height greater than 160 cm, 170 cm and so on. These measurements 160 cm, 170 cm and 180 cm divide a population into groups wherein the population of the group is of certain percentage of the whole. When this mechanism is applied to a probability distribution the locations that divide the distribution into groups are called quantiles. A quantile divides a group into 100 portions.

Based on the percentage divide they create there are popular quantiles used in statistics. If a distribution is divided in to four groups consisting of below 25 percent, below 50 percent, below 75 percent and below 100 percent – these quantiles are called first quartile, second quantile, third quantile.

The Python [example 1](#Python example that calculates the quartiles for a pandas DataFrame) loads the heights of students in a class into a pandas DataFrame and calculates the quartiles of the distribution. The quartiles are plot as given in the output below using matplotlib. When the quantiles are in the multiples of tens they are called deciles.

Quartiles calculated using pandas

The quantile() function of Pandas DataFrame class computes the value below which a given portion of the data lies. The [example 2](#Deciles in a normal distribution) plots a normal distribution for a value of student scores and marks the deciles in the normal curve as given in the diagram below.

Deciles of a normal distribution

Example 1

The Python example prints the heights of the students of a class at the first quartile (value at 25th percentile), at the second quartile(value at the 50th percentile) and at the third quartile (value at 75th percentile) and plots the heights using normal distribution.

python
# Example Python program that finds

            # the first quartile, second quartile and the

            # third quartile for a given data points in a

            # pandas DataFrame. The quartiles are plot using

            # matplotlib

            import pandas as pds

            import numpy as np

            from scipy import stats

            import matplotlib.pyplot as plt

# Heights of students in a class

            heights = np.linspace(164, 182, 50)

# Load heights into DataFrame

            df         = pds.DataFrame(heights)

# Compute the 25th percentile

            q1 = df.quantile(0.25)

# Compute the 50th percentile

            q2 = df.quantile(0.50)

# Compute the 75th percentile

            q3 = df.quantile(0.75)

print("First quartile:{:.2f}".format(q1[0]))

            print("Second quartile:{:.2f}".format(q2[0]))

            print("Third quartile:{:.2f}".format(q3[0]))

# Find the standard deviation

            stdDev = np.std(heights)

            print("Standard deviation:{:.2f}".format(stdDev))

# Plot using the normal distribution

            heights    = np.sort(heights)

            pd     = stats.norm.pdf(heights, loc = q2, scale = stdDev)

plt.plot(heights, pd, color = 'blue')

plt.axvline(q1[0], color='green', linestyle='--', label=f'Q1={q1[0]:.2f}')

            plt.axvline(q2[0], color='red', linestyle='solid', label=f'Q2={q2[0]:.2f}')

            plt.axvline(q3[0], color='green', linestyle='--', label=f'Q3={q3[0]:.2f}')

plt.title('Student heights - First quartile, Mean, Third quartile')

            plt.xlabel('Student heights')

            plt.ylabel('Probability density')

plt.legend()

            plt.show()

Output

Output
First quartile:168.50

            Second quartile:173.00

            Third quartile:177.50

            Standard deviation:5.30

Example2 - Deciles marked on a normal curve

python
# Example Python program that plots the deciles

            # of a probability distribution

            import pandas as pds

            import numpy as np

            from scipy import stats

            import matplotlib.pyplot as plt

# Student test scores

            scores = np.linspace(0, 100, 50)

# Student scores loaded into a DataFrame

            df        = pds.DataFrame(scores)

fractions = np.arange(0.1, 1.0, 0.1)

            deciles = []

            for i in fractions:

                # Use quantile to get deciles

                dn = df.quantile(i)

    decile = int(round(dn[0]))

                deciles.append(decile)

    decileLoc = int(round(i * 10))

                print("Decile{} location:{}".format(decileLoc , decile))

# Find the standard deviation

            stdDev = np.std(scores)

            print("Standard deviation:{:.2f}".format(stdDev))

# Plot using the normal distribution

            scores    = np.sort(scores)

            pd     = stats.norm.pdf(scores, loc = np.mean(scores), scale = stdDev)

plt.plot(scores, pd, color = 'blue')

            colors = ['blue', 'orange', 'green',

                      'red', 'purple', 'brown',

                      'pink', 'gray', 'olive']

            for decile in deciles:

                loc    = int(decile/10)

                plt.axvline(x = decile, color = colors[loc-1],

                            linestyle = '--',

                            label = f'Decile{loc}')

plt.title('Student scores - deciles')

            plt.xlabel('Student scores')

            plt.ylabel('Probability density')

plt.legend()

            plt.show()

Output

Output
Decile1 location:10

            Decile2 location:20

            Decile3 location:30

            Decile4 location:40

            Decile5 location:50

            Decile6 location:60

            Decile7 location:70

            Decile8 location:80

            Decile9 location:90

            Standard deviation:29.45