How to Clean Text Data

Raw text is rarely ready to use. Whether you are processing user input, scraping a website, importing a spreadsheet, or preparing data for an AI model, you will encounter extra whitespace, invisible characters, inconsistent line endings, HTML tags, broken unicode, and other noise. This guide covers the most common text-cleaning operations and how to perform them in JavaScript, Python, Go, and PHP.

Why Text Cleaning Matters

Dirty text causes subtle, hard-to-debug problems:

• A leading space in a name field breaks alphabetical sorting and database lookups
• Inconsistent line endings (\r\n vs \n) produce phantom extra lines or broken diffs
• HTML entities like & or raw tags like <br> appear literally in emails and PDFs
• Non-breaking spaces (\u00A0) look identical to regular spaces but fail .trim() checks
• Invisible control characters (\u0000–\u001F) corrupt CSV files and crash parsers
• Mixed unicode forms (é as one codepoint vs e + combining accent) break string comparisons

Cleaning text early, before any processing, prevents all of these.

The Six Core Cleaning Operations

1. Trim Leading and Trailing Whitespace

This is the most common operation — remove spaces, tabs, and newlines from both ends of the string.

"  hello world  ".trim()        // "hello world"
"\t  line\n".trim()             // "line"

Caution: .trim() in most languages misses non-breaking spaces (\u00A0) and other exotic whitespace. See the unicode section below.

2. Normalize Whitespace (Collapse Internal Spaces)

Multiple consecutive spaces, tabs, or newlines inside text should become a single space.

"too   many    spaces".replace(/\s+/g, " ").trim()
// "too many spaces"

This is essential before word-count, slug generation, or any operation that splits on spaces.

3. Normalize Line Endings

Files edited on Windows use \r\n (CRLF); macOS/Linux use \n (LF); old Mac used \r (CR). Normalize everything to \n:

text.replace(/\r\n/g, "\n").replace(/\r/g, "\n")

Always do this before splitting on \n, counting lines, or comparing text.

4. Strip HTML Tags

When text comes from a rich-text editor or a web scrape, it may contain HTML tags you want to remove.

"<p>Hello <strong>world</strong></p>".replace(/<[^>]*>/g, "")
// "Hello world"

After stripping tags, decode HTML entities too (& → &, < → <,   → ):

const el = document.createElement("div");
el.innerHTML = "& < >  ";
el.textContent; // "& < >  "

Important: Never use regex-based HTML stripping for security (XSS prevention). Use a proper HTML sanitizer like DOMPurify for untrusted input. Regex is fine for extracting visible text from your own data.

5. Remove or Replace Special Characters

Depending on context, you may want to:

• Remove all non-printable control characters (ASCII 0–31, except tab and newline)
• Remove punctuation
• Keep only alphanumeric characters
• Remove emoji

6. Normalize Unicode

Unicode has multiple representations of visually identical characters. The letter é can be encoded as:

• NFC (composed): single codepoint U+00E9 — standard for most text
• NFD (decomposed): two codepoints U+0065 + U+0301 — used by some macOS file systems

String comparison of NFC vs NFD returns false even though they look identical. Always normalize before comparing or indexing:

"é".normalize("NFC") === "é".normalize("NFC") // true

For databases and search indexes, normalize to NFC on input. For sorting, NFD is sometimes preferred.

Code Examples

JavaScript

function cleanText(raw) {
  return raw
    // 1. Normalize line endings to \n
    .replace(/\r\n/g, "\n").replace(/\r/g, "\n")
    // 2. Strip HTML tags
    .replace(/<[^>]*>/g, "")
    // 3. Remove control characters (except \t and \n)
    .replace(/[\x00-\x08\x0B\x0C\x0E-\x1F\x7F]/g, "")
    // 4. Collapse internal whitespace to single space
    .replace(/[^\S\n]+/g, " ")
    // 5. Trim each line
    .split("\n").map(line => line.trim()).join("\n")
    // 6. Remove blank lines (optional)
    .replace(/\n{2,}/g, "\n")
    // 7. Final trim
    .trim()
    // 8. Normalize unicode to NFC
    .normalize("NFC");
}

console.log(cleanText("  <p>Hello &amp;  world\r\n</p>  "));
// "Hello &amp; world"   (entities left for further handling if needed)

To also decode HTML entities in a browser environment:

function decodeHTMLEntities(text) {
  const el = document.createElement("textarea");
  el.innerHTML = text;
  return el.value;
}

Python

import re
import unicodedata
import html

def clean_text(raw: str) -> str:
    # 1. Normalize line endings
    text = raw.replace("\r\n", "\n").replace("\r", "\n")
    # 2. Strip HTML tags
    text = re.sub(r"<[^>]+>", "", text)
    # 3. Decode HTML entities (&amp; → &, &lt; → <, etc.)
    text = html.unescape(text)
    # 4. Remove control characters (except \t and \n)
    text = re.sub(r"[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]", "", text)
    # 5. Collapse horizontal whitespace on each line
    text = "\n".join(
        re.sub(r"[^\S\n]+", " ", line).strip()
        for line in text.split("\n")
    )
    # 6. Collapse multiple blank lines
    text = re.sub(r"\n{2,}", "\n", text)
    # 7. Final strip
    text = text.strip()
    # 8. Normalize unicode to NFC
    text = unicodedata.normalize("NFC", text)
    return text

print(clean_text("  <p>Hello &amp;  world\r\n</p>  "))
# "Hello & world"

Go

package main

import (
    "fmt"
    "html"
    "regexp"
    "strings"
    "unicode"
    "golang.org/x/text/unicode/norm"
)

var (
    reCRLF       = regexp.MustCompile(`\r\n|\r`)
    reHTMLTags   = regexp.MustCompile(`<[^>]+>`)
    reCtrlChars  = regexp.MustCompile(`[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]`)
    reMultiSpace = regexp.MustCompile(`[^\S\n]+`)
    reMultiLine  = regexp.MustCompile(`\n{2,}`)
)

func cleanText(raw string) string {
    // 1. Normalize line endings
    s := reCRLF.ReplaceAllString(raw, "\n")
    // 2. Strip HTML tags
    s = reHTMLTags.ReplaceAllString(s, "")
    // 3. Decode HTML entities
    s = html.UnescapeString(s)
    // 4. Remove control characters
    s = reCtrlChars.ReplaceAllString(s, "")
    // 5. Collapse horizontal whitespace
    lines := strings.Split(s, "\n")
    for i, line := range lines {
        line = reMultiSpace.ReplaceAllString(line, " ")
        lines[i] = strings.TrimFunc(line, unicode.IsSpace)
    }
    s = strings.Join(lines, "\n")
    // 6. Collapse multiple blank lines
    s = reMultiLine.ReplaceAllString(s, "\n")
    // 7. Final trim
    s = strings.TrimSpace(s)
    // 8. NFC normalize (requires golang.org/x/text)
    s = norm.NFC.String(s)
    return s
}

func main() {
    fmt.Println(cleanText("  <p>Hello &amp;  world\r\n</p>  "))
    // Hello & world
}

PHP

<?php
function clean_text(string $raw): string {
    // 1. Normalize line endings
    $text = str_replace(["\r\n", "\r"], "\n", $raw);
    // 2. Strip HTML tags
    $text = strip_tags($text);
    // 3. Decode HTML entities
    $text = html_entity_decode($text, ENT_QUOTES | ENT_HTML5, 'UTF-8');
    // 4. Remove control characters (except \t and \n)
    $text = preg_replace('/[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]/u', '', $text);
    // 5. Collapse horizontal whitespace on each line
    $lines = explode("\n", $text);
    $lines = array_map(function($line) {
        return trim(preg_replace('/[^\S\n]+/', ' ', $line));
    }, $lines);
    $text = implode("\n", $lines);
    // 6. Collapse multiple blank lines
    $text = preg_replace('/\n{2,}/', "\n", $text);
    // 7. Final trim + NFC normalize
    return trim(Normalizer::normalize($text, Normalizer::FORM_C));
}

echo clean_text("  <p>Hello &amp;  world\r\n</p>  ");
// Hello & world
?>

Quick Reference

Frequently Asked Questions

Why doesn't .trim() remove the non-breaking space?
JavaScript's .trim() removes Unicode whitespace, which does include \u00A0 (non-breaking space) in modern engines — but PHP's trim() does not. In PHP, use preg_replace('/^\s+|\s+$/u', '', $text) with the u flag for proper Unicode trim.

Should I strip HTML before or after decoding entities?
Strip HTML tags first, then decode entities. If you decode first, <script> becomes <script> before you strip, which can be a security risk in browser contexts.

What's the difference between NFC and NFD?
NFC (Canonical Decomposition, followed by Canonical Composition) stores composed characters as a single codepoint where possible — é as U+00E9. NFD always decomposes: é becomes e + combining accent U+0301. NFC is the standard for databases, APIs, and most file systems. Use NFC unless you have a specific reason for NFD.

How do I handle emoji in text cleaning?
It depends on your goal. If you want to keep emoji, use NFC normalization and avoid character-range filters that cut off the supplementary Unicode plane (codepoints above \uFFFF). If you want to strip emoji, use /\p{Emoji_Presentation}/gu in JS (requires the u flag) or Python's regex library (not re).

My text has zero-width spaces and other invisible characters — how do I find them?
Zero-width space is U+200B, zero-width non-joiner is U+200C, zero-width joiner is U+200D. Strip them with: text.replace(/[\u200B-\u200D\uFEFF]/g, "") in JS. You can also paste suspicious text into a hex dump tool to see every codepoint.

Is it safe to clean text in the browser (client-side)?
Yes for display purposes. But always re-clean on the server before storing or processing — client-side cleaning can be bypassed. Never trust that the text you receive from a client has been sanitized.