Python ProgrammingLevel: Intermediatepython

Data Cleaning With Pandas: A Worked Example

Prepared by the Assignment Achievers data science team.

Free Educational Resource

Problem Statement / Assignment Scope

A dataset of student submissions has duplicate rows, subject names typed in different ways, a missing score and dates in two formats. Clean it with Pandas so it's ready for analysis, then print summary statistics.

Prerequisites: Python basics, a little NumPy, and reading data laid out in rows and columns.

Step-by-Step Worked Solution

1. A cleaning routine you can reuse

Real datasets are rarely ready to analyse, and preparing them often takes longer than the analysis itself. Work through the same checks every time: duplicates, inconsistent text, missing values and data types.

Duplicates: drop_duplicates() removes the second copy of student 204.
Text: stripping spaces and using title case turns " python" and "Python" into one category.
Missing values: the gap in student 202's score is filled with the median.
Dates: pd.to_datetime turns both date styles into proper dates. dayfirst=True reads 02/04/2026 as 2 April, the UK and Australian convention.

2. Why the median?

The median isn't pulled around by one unusually high or low score, so it's the safer choice when the data might be skewed. If the scores are roughly symmetrical, the mean gives a similar result.

3. A note on newer Pandas

Assign the result back (df["score"] = df["score"].fillna(...)) rather than calling fillna(..., inplace=True) on a single column. Recent Pandas versions warn about the in-place form on a column, and it can silently do nothing.

solution.pypython

import numpy as np
import pandas as pd

submissions = pd.DataFrame({
    "student_id": [201, 202, 203, 204, 204, 205],
    "subject": ["Python", " python", "Nursing", "Law", "Law", np.nan],
    "score": [81, np.nan, 94, 70, 70, 58],
    "submitted": ["2026-04-01", "02/04/2026", "2026-04-03", "2026-04-04", "2026-04-04", "2026-04-05"],
})


def tidy(data: pd.DataFrame) -> pd.DataFrame:
    df = data.drop_duplicates().copy()          # 1. remove repeated rows

    df["subject"] = (                           # 2. one spelling per subject
        df["subject"].str.strip().str.title().fillna("Unknown")
    )

    df["score"] = df["score"].fillna(df["score"].median())   # 3. fill gaps

    df["submitted"] = pd.to_datetime(df["submitted"], format="mixed", dayfirst=True)  # 4. real dates
    return df


clean = tidy(submissions)
print(clean)
print(clean["score"].describe())

Key Takeaways & Core Principles

Look at the distribution of a column before choosing how to fill its gaps.
Normalise text with `.str.strip().str.title()` so the same category isn't counted twice.
Say how ambiguous dates should be read, for example with `dayfirst=True`.
Write the cleaned column back instead of relying on `inplace=True`.

Frequently Asked Questions

Should I delete rows with missing data or fill them in?

It depends on how much is missing and why. If a large share of an important column is empty, or the gaps can't be estimated sensibly, dropping rows may be safer. If you need to keep your sample size, fill them and explain the method in your write-up.

Academic References & Standards

[1]McKinney, W. (2022). Python for Data Analysis (3rd ed.). O'Reilly Media. Source Link

Help With Your Own Task

Got a Similar Question for Your Course?

Send it to an Assignment Achievers expert. You'll get an original answer that follows your brief and rubric, the working shown step by step, and a free originality report.