Data

pandas Missing Values: Find, Count, Fill and Drop NaN Without Skewing Your Results

Data · · Published 2026-11-20

A white jigsaw puzzle with one piece missing, leaving a black hole in the middle
One missing piece is easy to spot in a puzzle. In a DataFrame it hides behind a float.

Real data has holes: a sensor that went offline, a form field nobody filled in, a cell that says n/a or just a dash. pandas marks these gaps as missing values, usually displayed as NaN, and most of its functions quietly skip them. That is convenient, but it also means a mean can be computed on half the rows without any warning. This guide shows how to find the gaps, decide what to do with them, and avoid the results that change silently because of them.

All examples were run with pandas 3.0 and Python 3.13. They behave the same on pandas 2.x except where noted.

How pandas represents a missing value

There are three markers you will meet:

The one rule that matters: a missing value is never equal to anything, not even to itself. np.nan == np.nan is False, so df[df["temp"] == np.nan] always returns an empty DataFrame. Always test with isna() or pd.isna(), which recognise all three markers.

Make read_csv recognise your placeholders

Most gaps arrive with a file. Here is a small weather extract with three kinds of holes: empty cells, n/a and a dash.

import io
import pandas as pd

csv = """city,date,temp,rain,station
Lyon,2026-11-01,9.5,0.0,A
Lyon,2026-11-02,,1.2,A
Lyon,2026-11-03,7.8,,
Nantes,2026-11-01,11.0,4.5,B
Nantes,2026-11-02,n/a,3.1,B
Nantes,2026-11-03,10.2,-,B
"""
df = pd.read_csv(io.StringIO(csv))
df.dtypes
# city    str
# date    str
# temp    float64
# rain    str      <- the dash turned the whole column into text

Empty cells and n/a were understood, because they are in the default list of missing markers ("", NA, N/A, n/a, NaN, NULL, null, None, #N/A and a few more). The dash was not, so rain became a text column and any calculation on it fails. Declare your own markers with na_values:

df = pd.read_csv(io.StringIO(csv), na_values=["-"], parse_dates=["date"])
df.dtypes
# city              str
# date    datetime64[us]
# temp           float64
# rain           float64
# station            str

The opposite trap exists too. The default list includes NA, which is also the ISO code for Namibia and a common abbreviation in other fields. If a column legitimately contains such strings, read the file with keep_default_na=False, na_values=[""] so that only empty cells count as missing. When a column stays as text despite this, look for other stray characters with the techniques from ValueError: could not convert string to float; when the file will not open at all, the problem is usually its encoding, covered in fixing UnicodeDecodeError in read_csv.

Find and count the gaps

isna() returns a DataFrame of booleans of the same shape. Summing it counts missing values per column, and averaging it gives the share:

df.isna().sum()
# city       0
# date       0
# temp       2
# rain       2
# station    1

df.isna().mean().round(2)   # temp 0.33, rain 0.33, station 0.17

df[df["temp"].isna()]        # the rows where temp is missing
df.isna().any(axis=1).sum()  # 4 rows have at least one gap

Do this before any analysis. A column that is one-third empty, as temp is here, deserves a decision rather than a default. df.info() shows the same information as non-null counts, and notna() is the inverse of isna().

What NaN does to your calculations

Aggregations skip missing values by default. That is usually what you want, but you should know it is happening:

df["temp"].count()            # 4, not 6
df["temp"].mean()             # 9.625, the mean of 4 values
df["temp"].mean(skipna=False) # nan

pd.Series([None, None], dtype=float).sum()             # 0.0
pd.Series([None, None], dtype=float).sum(min_count=1)  # nan

The last two lines are the subtle ones: the sum of nothing is 0.0, so an entirely empty group in a report shows zero rainfall instead of "no data". Pass min_count=1 when a zero would be misleading.

Grouping has its own rule. Rows whose key is missing are dropped from groupby() unless you ask otherwise, so totals by group no longer add up to the grand total:

df.groupby("station")["temp"].count()                 # A 1, B 2
df.groupby("station", dropna=False)["temp"].count()   # A 1, B 2, NaN 1
df["station"].value_counts(dropna=False)              # B 3, A 2, NaN 1

Our pandas groupby guide covers the other options of groupby(). The same issue appears after joins: an outer or left merge() fills unmatched rows with NaN, which is how you detect them in merge, join and concat.

Drop rows or columns with dropna

dropna() removes every row that contains at least one missing value. On real data that can delete most of the table, so narrow it down:

df.dropna()                 # only 2 of 6 rows survive
df.dropna(subset=["temp"])  # drop rows only when temp is missing
df.dropna(thresh=4)         # keep rows with at least 4 non-missing values
df.dropna(axis=1, how="all")  # drop columns that are entirely empty

Dropping is the honest choice when a row without the value is useless for the question at hand, for example when you study temperature and the temperature is missing. Note in your results how many rows were removed.

Fill gaps with fillna, ffill and interpolate

fillna() replaces missing values. Pass a dictionary to choose a value per column, because a single value rarely suits every column:

df.fillna({"rain": 0, "station": "unknown"})

Filling rain with 0 asserts that it did not rain; that is a claim about the world, not a technical fix. Only do it when the source tells you that an empty cell means zero.

For ordered data such as time series, three methods use the neighbouring values instead of a constant:

s = pd.Series([10.0, None, None, 13.0],
              index=pd.date_range("2026-11-01", periods=4))

s.ffill()          # 10, 10, 10, 13   carry the last known value forward
s.ffill(limit=1)   # 10, 10, NaN, 13  but not across long gaps
s.bfill()          # 10, 13, 13, 13   take the next known value
s.interpolate()    # 10, 11, 12, 13   straight line between known values

The older spelling fillna(method="ffill") was deprecated in pandas 2.1 and no longer works in pandas 3.0; use ffill() and bfill() directly. Sort by date first, and when the table mixes several series, fill within each one with df.groupby("city")["temp"].ffill(), otherwise Lyon's last reading leaks into Nantes.

Filling with a group statistic is a common compromise. transform() returns a value aligned with the original rows, so it can be passed straight to fillna():

city_mean = df.groupby("city")["temp"].transform("mean")
df["temp_filled"] = df["temp"].fillna(city_mean)
# Lyon's gap becomes 8.65, Nantes's gap 10.6

Keep the original column, as here, or add a flag such as df["temp_was_missing"] = df["temp"].isna(). Imputed values reduce the apparent spread of the data, and anyone reading your results should be able to tell measured values from estimated ones.

Integers and missing values

A classic NumPy integer column cannot hold NaN, which is a float. pandas therefore turns [1, None, 3] into a float64 Series, and identifiers such as postcodes or years suddenly print as 2026.0. Converting back fails while gaps remain:

df["temp"].astype(int)
# IntCastingNaNError: Cannot convert non-finite values (NA or inf) to integer.

pd.Series([1, None, 3], dtype="Int64")
# 0       1
# 1    <NA>
# 2       3
# dtype: Int64

The capitalised Int64 is a nullable integer dtype: values stay integers and the gap is stored as pd.NA. You can ask for it when reading a file with dtype={"year": "Int64"}. For codes with leading zeros, such as French postcodes, read the column as text instead.

A short checklist

Sources

Read next

More in Data