pandas Missing Values: Find, Count, Fill and Drop NaN Without Skewing Your Results

Real data has holes: a sensor that went offline, a form field nobody filled in, a cell that says n/a or just a dash. pandas marks these gaps as missing values, usually displayed as NaN, and most of its functions quietly skip them. That is convenient, but it also means a mean can be computed on half the rows without any warning. This guide shows how to find the gaps, decide what to do with them, and avoid the results that change silently because of them.
All examples were run with pandas 3.0 and Python 3.13. They behave the same on pandas 2.x except where noted.
How pandas represents a missing value
There are three markers you will meet:
np.nan(displayedNaN), a special floating-point value. It is the default marker in float columns and, since pandas 3.0, in the defaultstrtext columns too.None, the Python object. When you put it in a numeric Series, pandas converts it toNaN.pd.NA(displayed<NA>), used by the nullable dtypes such asInt64,booleanandstring.
The one rule that matters: a missing value is never equal to anything, not even to itself. np.nan == np.nan is False, so df[df["temp"] == np.nan] always returns an empty DataFrame. Always test with isna() or pd.isna(), which recognise all three markers.
Make read_csv recognise your placeholders
Most gaps arrive with a file. Here is a small weather extract with three kinds of holes: empty cells, n/a and a dash.
import io
import pandas as pd
csv = """city,date,temp,rain,station
Lyon,2026-11-01,9.5,0.0,A
Lyon,2026-11-02,,1.2,A
Lyon,2026-11-03,7.8,,
Nantes,2026-11-01,11.0,4.5,B
Nantes,2026-11-02,n/a,3.1,B
Nantes,2026-11-03,10.2,-,B
"""
df = pd.read_csv(io.StringIO(csv))
df.dtypes
# city str
# date str
# temp float64
# rain str <- the dash turned the whole column into text
Empty cells and n/a were understood, because they are in the default list of missing markers ("", NA, N/A, n/a, NaN, NULL, null, None, #N/A and a few more). The dash was not, so rain became a text column and any calculation on it fails. Declare your own markers with na_values:
df = pd.read_csv(io.StringIO(csv), na_values=["-"], parse_dates=["date"])
df.dtypes
# city str
# date datetime64[us]
# temp float64
# rain float64
# station str
The opposite trap exists too. The default list includes NA, which is also the ISO code for Namibia and a common abbreviation in other fields. If a column legitimately contains such strings, read the file with keep_default_na=False, na_values=[""] so that only empty cells count as missing. When a column stays as text despite this, look for other stray characters with the techniques from ValueError: could not convert string to float; when the file will not open at all, the problem is usually its encoding, covered in fixing UnicodeDecodeError in read_csv.
Find and count the gaps
isna() returns a DataFrame of booleans of the same shape. Summing it counts missing values per column, and averaging it gives the share:
df.isna().sum()
# city 0
# date 0
# temp 2
# rain 2
# station 1
df.isna().mean().round(2) # temp 0.33, rain 0.33, station 0.17
df[df["temp"].isna()] # the rows where temp is missing
df.isna().any(axis=1).sum() # 4 rows have at least one gap
Do this before any analysis. A column that is one-third empty, as temp is here, deserves a decision rather than a default. df.info() shows the same information as non-null counts, and notna() is the inverse of isna().
What NaN does to your calculations
Aggregations skip missing values by default. That is usually what you want, but you should know it is happening:
df["temp"].count() # 4, not 6
df["temp"].mean() # 9.625, the mean of 4 values
df["temp"].mean(skipna=False) # nan
pd.Series([None, None], dtype=float).sum() # 0.0
pd.Series([None, None], dtype=float).sum(min_count=1) # nan
The last two lines are the subtle ones: the sum of nothing is 0.0, so an entirely empty group in a report shows zero rainfall instead of "no data". Pass min_count=1 when a zero would be misleading.
Grouping has its own rule. Rows whose key is missing are dropped from groupby() unless you ask otherwise, so totals by group no longer add up to the grand total:
df.groupby("station")["temp"].count() # A 1, B 2
df.groupby("station", dropna=False)["temp"].count() # A 1, B 2, NaN 1
df["station"].value_counts(dropna=False) # B 3, A 2, NaN 1
Our pandas groupby guide covers the other options of groupby(). The same issue appears after joins: an outer or left merge() fills unmatched rows with NaN, which is how you detect them in merge, join and concat.
Drop rows or columns with dropna
dropna() removes every row that contains at least one missing value. On real data that can delete most of the table, so narrow it down:
df.dropna() # only 2 of 6 rows survive
df.dropna(subset=["temp"]) # drop rows only when temp is missing
df.dropna(thresh=4) # keep rows with at least 4 non-missing values
df.dropna(axis=1, how="all") # drop columns that are entirely empty
Dropping is the honest choice when a row without the value is useless for the question at hand, for example when you study temperature and the temperature is missing. Note in your results how many rows were removed.
Fill gaps with fillna, ffill and interpolate
fillna() replaces missing values. Pass a dictionary to choose a value per column, because a single value rarely suits every column:
df.fillna({"rain": 0, "station": "unknown"})
Filling rain with 0 asserts that it did not rain; that is a claim about the world, not a technical fix. Only do it when the source tells you that an empty cell means zero.
For ordered data such as time series, three methods use the neighbouring values instead of a constant:
s = pd.Series([10.0, None, None, 13.0],
index=pd.date_range("2026-11-01", periods=4))
s.ffill() # 10, 10, 10, 13 carry the last known value forward
s.ffill(limit=1) # 10, 10, NaN, 13 but not across long gaps
s.bfill() # 10, 13, 13, 13 take the next known value
s.interpolate() # 10, 11, 12, 13 straight line between known values
The older spelling fillna(method="ffill") was deprecated in pandas 2.1 and no longer works in pandas 3.0; use ffill() and bfill() directly. Sort by date first, and when the table mixes several series, fill within each one with df.groupby("city")["temp"].ffill(), otherwise Lyon's last reading leaks into Nantes.
Filling with a group statistic is a common compromise. transform() returns a value aligned with the original rows, so it can be passed straight to fillna():
city_mean = df.groupby("city")["temp"].transform("mean")
df["temp_filled"] = df["temp"].fillna(city_mean)
# Lyon's gap becomes 8.65, Nantes's gap 10.6
Keep the original column, as here, or add a flag such as df["temp_was_missing"] = df["temp"].isna(). Imputed values reduce the apparent spread of the data, and anyone reading your results should be able to tell measured values from estimated ones.
Integers and missing values
A classic NumPy integer column cannot hold NaN, which is a float. pandas therefore turns [1, None, 3] into a float64 Series, and identifiers such as postcodes or years suddenly print as 2026.0. Converting back fails while gaps remain:
df["temp"].astype(int)
# IntCastingNaNError: Cannot convert non-finite values (NA or inf) to integer.
pd.Series([1, None, 3], dtype="Int64")
# 0 1
# 1 <NA>
# 2 3
# dtype: Int64
The capitalised Int64 is a nullable integer dtype: values stay integers and the gap is stored as pd.NA. You can ask for it when reading a file with dtype={"year": "Int64"}. For codes with leading zeros, such as French postcodes, read the column as text instead.
A short checklist
- Declare every placeholder your source uses with
na_values, and watch for real values likeNAthat pandas treats as missing. - Count gaps with
df.isna().sum()before computing anything. - Test for missing values with
isna(), never with== np.nan. - Remember that
count(),mean()andsum()skip gaps, and thatgroupby()drops missing keys unlessdropna=False. - Drop with
subsetorthreshrather than a baredropna(). - Fill only when you can justify the value, keep a flag, and fill time series within each group after sorting.
- Use
Int64for integer columns that may contain gaps.
Sources
- pandas documentation, Working with missing data (NaN, NA, filling and interpolation).
- pandas documentation, pandas.read_csv (
na_values,keep_default_naand the default list of missing markers). - pandas documentation, What's new in 3.0.0 (default
strdtype, removal of deprecated arguments).
Datapandas groupby
Split, apply, combine: totals per group, named aggregation, transform, filter and the defaults that changed in pandas 3.
DataUnicodeDecodeError in read_csv
Why pd.read_csv raises UnicodeDecodeError: 'utf-8' codec can't decode byte, and how to fix it with the right encoding, BOM handling and UTF-8 conversion.
Datapandas pivot_table and melt
Turn a long list of records into a readable cross-table, turn it back again, and understand the duplicate-entries error.