What's happening:

'JosÃ©' is the visible symptom of UTF-8 bytes being decoded as Latin-1 (ISO-8859-1). The original 'José' (UTF-8: 0x4A 0x6F 0x73 0xC3 0xA9) is being read as 0xC3 0xA9 = two separate characters (Ã + ©) when decoded as Latin-1.

The diagnostic:

Confirm the encoding at each stage.
- What encoding does the source database / file use?
- What encoding does the intermediate system use (ETL, API, file)?
- What encoding does the destination (report) use?
- Mojibake (JosÃ©) happens when there's a mismatch between source encoding and reader encoding.
Check specific bytes. SELECT encode(name::bytea, 'hex') in Postgres shows the actual bytes. 0xC3A9 is 'é' in UTF-8; if you see 0xE9 alone, that's Latin-1 'é'.
Find the mismatch point. Usually one stage is reading/writing differently than the others. Typical culprits:
- Old systems exporting as Latin-1.
- File transfers between OS's (Windows often defaults to CP-1252, Unix to UTF-8).
- APIs that don't specify encoding in headers (assume wrong default).
- Spreadsheet imports (Excel notoriously reads/writes CSV in locale-specific encoding).

The fix:

Set encoding explicitly at every boundary.
- Database connection: SET NAMES 'utf8mb4' (MySQL), SET CLIENT_ENCODING TO 'UTF8' (Postgres).
- File reads: open(filename, encoding='utf-8') in Python, not the default.
- API calls: Content-Type: application/json; charset=utf-8.
Convert existing corrupted data. This requires knowing the mis-decoding step:
```
# If UTF-8 was decoded as Latin-1, re-encode to bytes then decode as UTF-8
fixed = s.encode('latin-1').decode('utf-8')
```
This works if the corruption is simple. Multi-layered mis-decoding (UTF-8 → Latin-1 → UTF-8 again) is harder.
Standardize going forward. UTF-8 everywhere. Every new system/pipeline specifies encoding explicitly.

Special cases:

BOM (Byte Order Mark). UTF-8 BOM is EF BB BF. Some writers add it, some don't; some readers choke on it. Strip if present.
emoji / 4-byte UTF-8. MySQL's default utf8 is actually 3-byte; doesn't handle emoji. Use utf8mb4 for full UTF-8.
Normalization forms. 'é' can be one codepoint (U+00E9) or two codepoints (U+0065 + U+0301). Normalize to NFC or NFD consistently. Rare but real.

Prevention:

Every text column: specify encoding explicitly in DDL.
Every file I/O: specify encoding explicitly in code.
Every API integration: check headers.

Stakeholder framing:

"The names are stored correctly in UTF-8 in the source database. The issue is in the ETL step — the Python pipeline reads the export file without specifying encoding, defaulting to Latin-1 on Windows, UTF-8 on Linux. Fix: add encoding='utf-8' to the file read. Re-run the pipeline for the affected rows to restore correct characters."

What senior candidates add:

Encoding issues often reveal system boundaries. Finding one is a clue that your pipeline has a configuration gap.
UTF-8 is always the right default for new systems. Any modern database, API, or file format supports it.
Test with a diverse character set (include accented Latin, CJK, emoji) to catch encoding bugs early.

How do you debug character encoding issues?

Draft your answer

Stuck? Peek at a hint

More data cleaning questions