Fix UnicodeDecodeError When Reading a CSV File in pandas

Data · Published 2026-10-24

Lines of colourful source code on a dark screen, slightly out of focus — Source code on a dark screen, slightly out of focus.

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x.. in position N means the CSV file was not saved as UTF-8, which is what pd.read_csv assumes by default. Pass the file's real encoding, most often encoding="cp1252" for files exported from Windows or Excel, or encoding="utf-16" for Excel "Unicode Text", and the error goes away.

Bytes, text and why decoding can fail

A file on disk is a sequence of bytes. To get text, those bytes must be decoded with the same encoding that was used to write them. ASCII characters are encoded identically in almost every encoding, which is why the problem only appears once the data contains accented letters, currency symbols or non-Latin scripts:

s = "José 12.50 €"
print(s.encode("utf-8"))
print(s.encode("cp1252"))

b'Jos\xc3\xa9 12.50 \xe2\x82\xac'
b'Jos\xe9 12.50 \x80'

In UTF-8, é takes two bytes and € takes three. In Windows-1252 (cp1252) each is a single byte. UTF-8 has strict rules about which byte sequences are valid; a lone 0xe9 followed by a comma breaks them, so the decoder stops:

import pandas as pd
df = pd.read_csv("sales_cp1252.csv")

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 19: invalid continuation byte

The byte value in the message is a strong hint about the source encoding:

Message fragment	Likely encoding	Try
`byte 0xe9`, `0xe8`, `0xfc`, `0xf6` (é, è, ü, ö)	Windows-1252 / Latin-1	`encoding="cp1252"`
`byte 0x80`, `0x92`, `0x96` (€, curly quote, en dash)	Windows-1252	`encoding="cp1252"`
`byte 0xff in position 0` or `0xfe in position 0`	UTF-16 with BOM	`encoding="utf-16"`
`'ascii' codec can't decode`	Something forced ASCII	Specify `encoding="utf-8"` explicitly

Fix 1: pass the correct encoding

Most CSV files that fail as UTF-8 come from Excel on Windows, legacy databases or older ERP exports, and are encoded in cp1252:

import pandas as pd

df = pd.read_csv("sales_cp1252.csv", encoding="cp1252")
print(df)

   name     city    price
0  José  München  12.50 €
1   Zoë   Kraków   8.00 €

The standard encodings table lists every name Python accepts (cp1252, windows-1252, latin-1, iso-8859-15, shift_jis, gb18030…).

cp1252 or latin-1?

latin-1 (ISO-8859-1) is often recommended because it never raises: every byte from 0 to 255 maps to some character. That is also its weakness. Bytes 0x80–0x9F, where cp1252 puts €, curly quotes and dashes, become invisible control characters:

df_latin = pd.read_csv("sales_cp1252.csv", encoding="latin-1")
print(df_latin["price"].tolist())

['12.50 \x80', '8.00 \x80']

No error, but the euro sign is gone. Prefer cp1252 for Western European data; it raises ('charmap' codec can't decode byte 0x81 ... character maps to <undefined>) on the five bytes it does not define, which is a useful sign that the file is something else entirely.

UTF-16 files from Excel

Excel's "Unicode Text (*.txt)" export writes tab-separated UTF-16 with a byte order mark, hence the failure on the very first byte:

pd.read_csv("excel_unicode.txt", sep="\t")

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

print(pd.read_csv("excel_unicode.txt", sep="\t", encoding="utf-16"))

   name     city
0  José  München

Fix 2: the UTF-8 byte order mark (utf-8-sig)

Excel's "CSV UTF-8" format and many Windows tools prepend a BOM, the bytes EF BB BF, to UTF-8 files. pandas strips it automatically when reading with the default encoding, so it rarely causes a UnicodeDecodeError. It bites elsewhere: the standard csv module and plain open() keep it as an invisible '' glued to the first header, so row["name"] raises KeyError.

import csv

with open("sales_bom.csv", encoding="utf-8") as f:
    print(next(csv.reader(f)))
with open("sales_bom.csv", encoding="utf-8-sig") as f:
    print(next(csv.reader(f)))

['name', 'city', 'price']
['name', 'city', 'price']

utf-8-sig removes a BOM if present and behaves like plain UTF-8 otherwise, so it is a safe default for reading files of unknown origin.

Fix 3: detect the encoding when you don't know it

A file does not record its own encoding, so any detection is a statistical guess. The charset-normalizer package (it is what requests uses) is the usual tool:

pip install charset-normalizer

from charset_normalizer import from_path

result = from_path("reviews.csv").best()
print(result.encoding)
print(str(result).splitlines()[1])

iso8859_10
1,Le café était trčs bon et le service rapide.

This run, with charset-normalizer 3.5 on a short French cp1252 file, shows the limits: the guess decodes without errors, but très came out as trčs. Single-byte encodings are very hard to tell apart on small samples. Always look at the decoded text, especially the accented characters, before trusting a detector. The command-line tool normalizer file.csv prints the same guess as JSON.

In practice a short, ordered list of candidates is more reliable than detection. UTF-8 is strict, so if a file decodes as UTF-8 it almost certainly is UTF-8; try it first, then fall back:

import pandas as pd

def read_csv_any(path, encodings=("utf-8", "cp1252"), **kwargs):
    """Try strict encodings in order; UTF-8 first because it rarely decodes by accident."""
    for enc in encodings:
        try:
            return pd.read_csv(path, encoding=enc, **kwargs), enc
        except UnicodeDecodeError:
            continue
    raise ValueError(f"none of {encodings} could decode {path}")

df, used = read_csv_any("customers_export.csv")
print(used)
print(df.head(3))

cp1252
           customer        city                      note  amount
0       José García      Málaga        livré à l'entrepôt  0.00 €
1  François Lefèvre  Düsseldorf       café gratuit offert  1.50 €
2        Zoë Müller      Québec  Lieferung über Spedition  3.00 €

Do not put latin-1 before other candidates: it never fails, so nothing after it would ever be tried.

Fix 4: replace or ignore undecodable bytes

When a few corrupt bytes are acceptable (log files, scraped data, mixed-encoding dumps), pandas 1.3 and later accept encoding_errors, which takes the same values as open()'s errors argument:

df = pd.read_csv("sales_cp1252.csv", encoding="utf-8", encoding_errors="replace")
print(df)

   name     city    price
0  Jos�  M�nchen  12.50 �
1   Zo�   Krak�w   8.00 �

Every undecodable byte becomes U+FFFD (�). "ignore" drops them silently and "backslashreplace" keeps them visible as \xe9. This is data loss: use it to get a file loaded and inspected, not as a permanent fix for a file whose real encoding you could specify. See the read_csv reference for all options.

Fix 5: convert the file to UTF-8 once

If the same legacy file is read by several scripts, convert it once and use UTF-8 everywhere after that:

from pathlib import Path

src = Path("sales_cp1252.csv")
dst = Path("sales_utf8.csv")
dst.write_text(src.read_text(encoding="cp1252"), encoding="utf-8")
print(dst.read_bytes()[:30])

b'name,city,price\nJos\xc3\xa9,M\xc3\xbcnchen'

On Linux and macOS, iconv does the same from the shell, and file gives a quick (heuristic) guess at the encoding:

iconv -f CP1252 -t UTF-8 sales_cp1252.csv > sales_utf8.csv

For very large files, read and write in chunks (open(..., encoding=...) and iterate over lines) instead of loading the whole file into memory.

Writing CSV files Excel will open correctly

The reverse problem: you write a UTF-8 CSV and Excel shows JosÃ©. That is UTF-8 bytes being decoded as cp1252, which is what Excel does with a CSV that has no BOM. Write with utf-8-sig so Excel recognises the encoding:

df.to_csv("for_excel.csv", index=False, encoding="utf-8-sig")

When in doubt, writing .xlsx with df.to_excel() avoids encoding questions altogether.

Troubleshooting

The error position is huge (e.g. position 81234). The file is mostly ASCII and the first non-ASCII character appears late. The diagnosis is the same; it just took a while to hit an accented character.
It works on your machine but not in Docker or CI. Code that calls open() without encoding= uses the platform's locale encoding, which differs between Windows (often cp1252) and Linux (UTF-8). Always pass encoding explicitly. Python 3.15 is scheduled to make UTF-8 the default (PEP 686); python -X utf8 enables that behaviour today.
The file loads, but text looks like Ã© or â‚¬. That is mojibake: UTF-8 data decoded as cp1252/latin-1. Read it with encoding="utf-8" instead.
The file is actually .xlsx renamed to .csv. The error points at a byte very early in the file (for example byte 0x8f in position 22) and the first two bytes are PK, the ZIP signature (head -c 2 file.csv shows it). Use pd.read_excel().
Compressed files (.gz, .zip) are handled by pandas automatically based on the extension; if the extension is wrong, pass compression= explicitly.

Summary

The error means the bytes are not UTF-8. Look at the byte in the message, then pass the right encoding to read_csv: cp1252 for most Windows/Excel exports, utf-16 for Excel's Unicode Text format, utf-8-sig to strip a BOM when using the csv module. If the encoding is unknown, try UTF-8 then cp1252, use charset-normalizer as a hint while checking the result by eye, fall back to encoding_errors="replace" only to inspect damaged data, and convert recurring legacy files to UTF-8 once.