Shift_JIS vs CP932: Why ～ Becomes ? and Python Raises UnicodeEncodeError

Two things trigger this problem. One is an error such as UnicodeEncodeError: 'shift_jis' codec can't encode character '\uff5e'. The other is text that silently turns into ? after a round trip through a Windows system. Both come from treating Shift_JIS and CP932 as the same thing.

They are close, not identical

Shift_JIS is the JIS-standard encoding. CP932, also called Windows-31J, is Microsoft's variant: it keeps the same byte layout but maps a few bytes to different characters and adds extra ones. Windows software uses CP932 even when its menus say "Shift_JIS".

The famous case is the wave dash. Bytes 81 60 are 〜 (U+301C) in Python's shift_jis codec and ～ (U+FF5E, fullwidth tilde) in cp932. They look nearly the same, but they are different characters. So encoding ～ with shift_jis fails, and a 〜 read from a standards-based system may show up as ? or a different glyph on Windows.

You can see it yourself:

b = bytes([0x81, 0x60])
print(hex(ord(b.decode("shift_jis"))))   # 0x301c  〜
print(hex(ord(b.decode("cp932"))))       # 0xff5e  ～

"～".encode("cp932")      # works
"～".encode("shift_jis")  # UnicodeEncodeError

What else differs

CP932 also contains characters that strict Shift_JIS lacks, including circled numbers ①, the company symbol ㈱, and the variant kanji 髙 and 﨑 that appear in real people's names. Encoding these with shift_jis raises UnicodeEncodeError.

Which codec should you use?

• Reading or writing files for Windows or Excel: cp932.
• Interoperating with a system documented as strict JIS: shift_jis, but expect failures on the characters above.
• Anything you control end to end: UTF-8.

Fixing the error on write

If you hit the error and the destination expects Shift_JIS-style data, switch to cp932 first. If you really need strict shift_jis, convert the problem characters beforehand:

FIX = str.maketrans({
    "\uff5e": "\u301c",   # ～ -> 〜
    "\uff0d": "\u2212",   # － -> −
    "\u2225": "\u2016",   # ∥ -> ‖
    "\uffe0": "\u00a2",   # ￠ -> ¢
})

text = text.translate(FIX)
data = text.encode("shift_jis")

Characters with no equivalent (such as ①) still fail. Decide per case: replace them (errors="replace"), expand them (① → 1), or stop and use UTF-8.

Avoiding silent damage

The worst case is not an error but a quiet ?. To catch it, round-trip a sample: encode, decode, and compare with the original. If they differ, your chosen encoding loses data.

Quick checklist

• Windows produced the file? Read it as cp932.
• Error on ～, ①, ㈱? That is the CP932-only set.
• Strict JIS target? Map the wave dash and friends before encoding.
• You control both ends? Use UTF-8 and skip the whole problem.