What Is UTF-8 Encoding?

Every file, database record, and HTTP response that contains text must answer one question: which bytes represent which characters? That mapping is called a character encoding. UTF-8 is the answer the web settled on — and for good reason.

Today, 97 % of all websites use UTF-8. If you have ever seen a garbled é instead of é, or a ? box where an emoji should be, you have witnessed two different encodings disagreeing about what those bytes mean. This guide explains how UTF-8 works, why it won, and how to use it correctly in code.

Unicode vs UTF-8: What Is the Difference?

People often use "Unicode" and "UTF-8" interchangeably. They are related but different:

• Unicode is a standard that assigns a unique number (called a code point) to every character in every human writing system. The character A is U+0041; the emoji 🎉 is U+1F389; the Greek letter α is U+03B1. Unicode defines what characters exist.
• UTF-8 is an encoding — a way to turn those code point numbers into actual bytes that can be stored or transmitted. UTF-8 defines how to represent Unicode in binary.

Other encodings exist (UTF-16, UTF-32, Latin-1, Windows-1252), but UTF-8 became the dominant choice for web and network protocols.

How UTF-8 Encodes Characters

UTF-8 is a variable-width encoding: different characters use different numbers of bytes (1–4). The number of bytes depends on the code point value:

The Byte Pattern

UTF-8 encodes each code point using a predictable bit pattern:

• 1-byte (ASCII): 0xxxxxxx — the leading 0 signals a single-byte character.
• 2-byte: 110xxxxx 10xxxxxx
• 3-byte: 1110xxxx 10xxxxxx 10xxxxxx
• 4-byte: 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

The 10 prefix on continuation bytes lets a UTF-8 decoder resync after a partial read — you can always tell whether you are reading a lead byte or a continuation byte.

Why UTF-8 Won Over Other Encodings

1. Backward-compatible with ASCII

The first 128 code points (U+0000–U+007F) — all the letters, digits, and punctuation in plain English — are encoded as single bytes identical to ASCII. Pure ASCII text is valid UTF-8. Legacy C code that processes ASCII bytes does not break.

2. Self-synchronising

Because continuation bytes always start with 10, you can jump into the middle of a UTF-8 byte stream and find the start of the next character by scanning forward. UTF-16 has no equivalent property.

3. No byte-order ambiguity

UTF-16 requires a byte-order mark (BOM) to specify whether bytes come in big-endian or little-endian order. UTF-8 is byte-order neutral. No BOM is needed (though a BOM can appear; it is then the zero-width no-break space U+FEFF, which causes problems in CSV and source files — avoid it).

4. Compact for Western text

ASCII characters take one byte, the same as in ASCII. UTF-16 always takes at least two bytes — English text in UTF-16 is twice as large as UTF-8.

UTF-8 in URLs

URL percent-encoding converts non-ASCII characters to %XX sequences where XX is the hexadecimal value of each UTF-8 byte. The space character (U+0020) becomes %20; the euro sign € (3 UTF-8 bytes: 0xE2 0x82 0xAC) becomes %E2%82%AC.

This is why URL encoding always works on UTF-8 bytes, not on raw Unicode code points.

UTF-8 in HTML

When you save an HTML file as UTF-8 and declare <meta charset="UTF-8">, the browser reads the bytes using UTF-8. Without that declaration, the browser guesses — and gets it wrong roughly 5 % of the time, producing mojibake (garbled text).

You still need HTML entity encoding for <, >, &, ", and ' — those are structural HTML characters that must be escaped regardless of encoding.

Quick Reference

MySQL warning: MySQL's utf8 charset only supports 3-byte sequences (up to U+FFFF). Emoji and rare CJK characters require the utf8mb4 charset. Always use utf8mb4 for new tables.

Code Examples

JavaScript

// Encode a string to UTF-8 bytes
const encoder = new TextEncoder(); // always UTF-8
const bytes = encoder.encode("café");
// Uint8Array [99, 97, 102, 195, 169]  (é = 2 bytes: 0xC3 0xA9)

// Decode UTF-8 bytes back to a string
const decoder = new TextDecoder("utf-8");
const str = decoder.decode(bytes);
// "café"

// Count UTF-8 byte length (not character length)
function byteLength(str) {
  return new TextEncoder().encode(str).length;
}
byteLength("café");   // 5 (4 chars but 5 bytes)
byteLength("hello");  // 5 (5 chars, 5 bytes — ASCII)
byteLength("🎉");     // 4 (1 char, 4 bytes)

Python

# Python 3 strings are already Unicode (code points)
s = "café"

# Encode to UTF-8 bytes
b = s.encode("utf-8")
# b'\x63\x61\x66\xc3\xa9'

# Decode UTF-8 bytes back to a string
s2 = b.decode("utf-8")
# "café"

# Byte length vs character length
len(s)            # 4 characters
len(s.encode())   # 5 bytes

# Read a file as UTF-8 (always specify encoding)
with open("file.txt", encoding="utf-8") as f:
    content = f.read()

# Write a file as UTF-8
with open("output.txt", "w", encoding="utf-8") as f:
    f.write("café")

Go

package main

import (
    "fmt"
    "unicode/utf8"
)

func main() {
    s := "café"

    // len() gives byte count, not rune count
    fmt.Println(len(s))                    // 5 (bytes)
    fmt.Println(utf8.RuneCountInString(s)) // 4 (characters)

    // Iterate over runes (Unicode code points)
    for i, r := range s {
        fmt.Printf("byte offset %d: %c (U+%04X)\n", i, r, r)
    }
    // byte offset 0: c (U+0063)
    // byte offset 1: a (U+0061)
    // byte offset 2: f (U+0066)
    // byte offset 3: é (U+00E9)   ← 2 bytes, so next offset is 5

    // Convert string to bytes and back
    b := []byte(s)
    s2 := string(b)
    fmt.Println(s2) // "café"

    // Validate UTF-8
    fmt.Println(utf8.ValidString(s)) // true
}

PHP

<?php
// PHP strings are byte strings by default — use mbstring for UTF-8

$s = "café";

// Byte length vs character length
strlen($s);           // 5 (bytes)
mb_strlen($s, "UTF-8"); // 4 (characters)

// Substring by character position, not byte position
mb_substr($s, 3, 1, "UTF-8"); // "é"
substr($s, 3, 1);              // broken — cuts inside é

// Convert encoding
$latin1 = mb_convert_encoding($s, "ISO-8859-1", "UTF-8");
$utf8   = mb_convert_encoding($latin1, "UTF-8", "ISO-8859-1");

// Detect encoding
mb_detect_encoding($s, ["UTF-8", "ISO-8859-1", "Windows-1252"]); // "UTF-8"

// Always set default encoding at the top of a script
mb_internal_encoding("UTF-8");
?>

Common Pitfalls

1. Confusing character count with byte count

strlen("café") in PHP returns 5, not 4. In Python 3, len("café") correctly returns 4 because Python 3 strings are Unicode — but len("café".encode()) returns 5. Always know whether your length function counts characters or bytes.

2. MySQL utf8 instead of utf8mb4

MySQL's utf8 is a 3-byte-only subset. Inserting a 4-byte character (any emoji) causes a silent truncation or an error. Migrate to utf8mb4 with utf8mb4_unicode_ci collation.

3. Missing charset declaration

Serving HTML without <meta charset="UTF-8"> causes the browser to guess the encoding. On some servers this defaults to ISO-8859-1, turning é into é.

4. Reading files without specifying encoding

Python 2 open() and Python 3's default depend on the operating system locale. Always pass encoding="utf-8" explicitly.

5. The BOM in CSV and source files

Some Windows editors save UTF-8 files with a BOM (EF BB BF bytes at the start). This breaks CSV parsers, <script> tags, and PHP files. Use "UTF-8 without BOM" in your editor.

6. Double-encoding

URL-encoding an already-encoded string turns %20 into %2520 (the % itself gets encoded). Decode first, then re-encode if needed. The same applies to HTML entities: never run htmlspecialchars() on already-escaped text.

Frequently Asked Questions

Is UTF-8 the same as Unicode?
No. Unicode is a character standard (assigns numbers to characters). UTF-8 is one way to encode those numbers as bytes. UTF-16 and UTF-32 are other encodings of the same Unicode standard.

Why does strlen("é") return 2 in PHP?
Because strlen() counts bytes, not characters. é in UTF-8 is two bytes (0xC3 0xA9). Use mb_strlen($s, "UTF-8") to count characters.

Should I use UTF-8 or UTF-16 for my database?
Use UTF-8 (specifically utf8mb4 in MySQL). UTF-16 is used internally by Java, C#, and JavaScript engines, but for storage and network transfer UTF-8 is more compact and universally supported.

What is the difference between UTF-8 and ISO-8859-1 (Latin-1)?
Latin-1 covers only 256 characters (Western European languages). UTF-8 covers the entire Unicode repertoire (~150,000 characters). For anything beyond basic Western text, Latin-1 is insufficient.

Can a UTF-8 file contain every language?
Yes. UTF-8 can represent every character in the Unicode standard: Latin, Cyrillic, Arabic, Hebrew, Chinese, Japanese, Korean, emoji, mathematical symbols, historic scripts, and more.

What is mojibake?
Mojibake (文字化け) is the garbled text you see when a byte sequence encoded in one charset (e.g. UTF-8) is decoded using a different one (e.g. Latin-1). The é (UTF-8: C3 A9) interpreted as Latin-1 becomes é. Fix it by ensuring consistent UTF-8 throughout: file, database connection, HTTP header, and HTML meta tag.