Skip to content

Encoding Detection ​

An HTML document's character encoding comes from Config.Encoding, a BOM, a <meta> declaration, or the data itself. This page covers the decision order, the supported encodings, and what undeclared pages actually produce.

Detection Order ​

Detection has two stages: basic detection proposes a charset, then the candidate is scored — a score of 90 or above is accepted directly; a basic result of utf-8 scoring 70–89 is also accepted; everything else goes to statistical scoring.

PriorityStageSourceNotes
①ForcedConfig.EncodingUsed directly; all later detection is skipped
②BasicBOMUTF-8 (EF BB BF), UTF-16 LE (FF FE), UTF-16 BE (FE FF)
③BasicPure ASCIIUses the <meta> declaration when present, otherwise UTF-8
④BasicValid UTF-8 (non-ASCII present)Treated as UTF-8 and the meta declaration is ignored
⑤BasicMeta declaration<meta charset> or http-equiv="content-type", scanning only the first 1024 bytes
⑥Basicwindows-1252 fallbackWhen nothing above applies
⑦StatisticalScores 13 candidate encodingsRuns when the basic result scores below 90 (a utf-8 basic result scoring 70–89 is accepted directly); samples up to 10KB

The difference between ③ and ④: pure ASCII carries no encoding information of its own, so the declaration decides first; valid UTF-8 containing non-ASCII bytes is trusted as data — some pages declare a legacy charset while actually serving UTF-8.

text
Config.Encoding set? ── yes ──→ use it
        │
        no
        │
BOM? ── yes ──→ use the BOM charset
        │
        no
        │
Pure ASCII? ── yes ──→ meta declaration if present, else UTF-8
        │
        no
        │
Valid UTF-8? ── yes ──→ UTF-8 (meta declaration ignored)
        │
        no
        │
Meta declaration? ── yes ──→ use the declared value
        │
        no
        │
windows-1252 fallback

The charset from basic detection is then scored as well; candidates that miss the acceptance thresholds go to statistical scoring: up to 10KB is sampled, 13 candidate encodings are decoded and scored, and the highest score wins.

The BOM outranks the meta declaration

A BOM is an encoding declaration carried by the data itself, and basic detection checks it first. When a BOM is present the encoding is settled without consulting any meta declaration.

Supported Encodings ​

CategoryEncodingsNotes
UnicodeUTF-8, UTF-16LE, UTF-16BEUTF-8 takes a passthrough fast path and never enters the conversion table; pure ASCII / valid UTF-8 input is treated as UTF-8 directly
Western EuropeanWindows-1252, ISO-8859-1, ISO-8859-15ISO-8859-15 includes the euro sign
Central EuropeanWindows-1250, ISO-8859-2, ISO-8859-3, ISO-8859-4ISO-8859-3 is South European, ISO-8859-4 is Baltic
CyrillicWindows-1251, KOI8-R, ISO-8859-5Russian, etc.
GreekWindows-1253, ISO-8859-7—
TurkishWindows-1254, ISO-8859-9—
HebrewWindows-1255, ISO-8859-8—
ArabicWindows-1256, ISO-8859-6—
VietnameseWindows-1258—
Other ISO-8859ISO-8859-10 (Nordic), ISO-8859-13 (Baltic), ISO-8859-14 (Celtic), ISO-8859-16 (Southeast European)The series omits -11/-12
Simplified ChineseGBK, GB18030The alias gb2312 is normalized to gbk
Traditional ChineseBig5—
JapaneseShift_JIS, EUC-JP, ISO-2022-JP—
KoreanEUC-KR—

The conversion registry supports 32 conversion encodings in total; UTF-8 takes a passthrough fast path and never goes through the conversion table.

Alias normalization ​

Encoding names and aliases are case-insensitive and normalize to canonical names. Common alias examples (the alias table is not exhaustive):

Input aliasNormalized result
gb2312, GB2312gbk
sjis, x-sjis, shift-jisshift_jis
latin1, latin-1iso-8859-1
utf8, utf_8utf-8
8859-1, iso88591iso-8859-1
cp1252, windows1252windows-1252

GBK with a Meta Declaration ​

A GBK-encoded Chinese page that declares its encoding in <meta charset> is identified by basic detection:

go
package main

import (
	"fmt"
	"log"

	"github.com/cybergodev/html"
	"golang.org/x/text/encoding/simplifiedchinese"
)

func main() {
	// GBK-encoded Chinese HTML (the meta tag declares charset=gbk)
	gbkHTML := `<html><head><meta charset="gbk">` +
		`<title>中文网页</title></head>` +
		`<body><article><h1>你好世界</h1>` +
		`<p>这是一段中文内容。</p></article></body></html>`

	// Encode the UTF-8 string to GBK bytes (simulating a real GBK page)
	gbkBytes, err := simplifiedchinese.GBK.NewEncoder().Bytes([]byte(gbkHTML))
	if err != nil {
		log.Fatal(err)
	}

	// Auto-detection: GBK from the meta charset, converted to UTF-8 before extraction
	result, err := html.Extract(gbkBytes)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println("Title:", result.Title)
	// Title: 中文网页

	fmt.Println("Text:", result.Text)
	// Text: # 你好世界
	//       这是一段中文内容。
}

Specifying the Encoding ​

When the meta tag is missing, the declaration is wrong, or the source encoding is known, set Config.Encoding:

go
cfg := html.DefaultConfig()
cfg.Encoding = "gbk"

result, err := html.Extract(gbkBytes, cfg)
Use caseNotes
Known source encodingTake it from the HTTP Content-Type header and specify it
Legacy pages with no meta tagGBK/Shift_JIS pages without <meta charset> fall back to windows-1252 and turn into mojibake
Declaration disagrees with the dataConfig.Encoding wins over the meta declaration

Config.Encoding skips all detection

With Config.Encoding set, the library reads no BOM, no meta declaration, and runs no statistical scoring — it decodes with the given charset. That is what makes it deterministic; conversely, a wrong value produces mojibake or an error immediately.

Undeclared pages decode to mojibake ​

A missing meta tag returns no error

For a GBK or Shift_JIS page without <meta charset>, basic detection falls back to windows-1252. Measured on such pages, the output is mojibake with err == nil — there is no error signal at all. Whether statistical scoring can correct it depends on the content; do not rely on it as a safety net. In production, set Config.Encoding explicitly or take the encoding from the HTTP response header.

Shift_JIS pages need the meta declaration ​

A Shift_JIS-encoded Japanese page is recognized by basic detection once it declares <meta charset="shift_jis">:

go
package main

import (
	"fmt"
	"log"

	"github.com/cybergodev/html"
	"golang.org/x/text/encoding/japanese"
)

func main() {
	// Shift_JIS-encoded Japanese HTML (the meta tag declares charset=shift_jis)
	sjisHTML := `<html><head><meta charset="shift_jis">` +
		`<title>日本語ページ</title></head>` +
		`<body><article><h1>こんにちは</h1>` +
		`<p>東京の天気は晴れです。</p></article></body></html>`

	// Encode to Shift_JIS bytes
	sjisBytes, err := japanese.ShiftJIS.NewEncoder().Bytes([]byte(sjisHTML))
	if err != nil {
		log.Fatal(err)
	}

	// Basic detection reads the meta declaration, converts to UTF-8, then extracts
	result, err := html.Extract(sjisBytes)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println("Title:", result.Title)
	// Title: 日本語ページ

	fmt.Println("Text:", result.Text)
	// Text: # こんにちは
	//       東京の天気は晴れです。
}

Remove the meta declaration from the same bytes and the title becomes mojibake (shaped like “ъ–{ЊкѓyЃ[ѓW) while err stays nil. If a declaration cannot be added to the source, specify the encoding explicitly:

go
cfg := html.DefaultConfig()
cfg.Encoding = "shift_jis"

result, err := html.Extract(sjisBytes, cfg)

Windows-1252 ​

Western European encodings (characters such as é and €) can be specified directly:

go
package main

import (
	"fmt"
	"log"

	"github.com/cybergodev/html"
	"golang.org/x/text/encoding/charmap"
)

func main() {
	// Windows-1252-encoded Western European text
	winHTML := `<html><head><title>Café Menu</title></head>` +
		`<body><article><h1>Café</h1>` +
		`<p>Price: 100 €. Résumé available.</p></article></body></html>`

	winBytes, err := charmap.Windows1252.NewEncoder().Bytes([]byte(winHTML))
	if err != nil {
		log.Fatal(err)
	}

	// Specify the Windows-1252 encoding explicitly
	cfg := html.DefaultConfig()
	cfg.Encoding = "windows-1252"

	result, err := html.Extract(winBytes, cfg)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println("Title:", result.Title)
	// Title: Café Menu

	fmt.Println("Text:", result.Text)
	// Text: # Café
	//       Price: 100 €. Résumé available.
}

Encoding Detection Failures ​

encoding detection failed appears only when the specified encoding disagrees with the data, meaning an encoding was forced incorrectly. There are two cases:

CaseError message
"utf-8" forced on data that is not valid UTF-8encoding detection failed: declared charset "utf-8" but data is not valid UTF-8
An unsupported encoding name on data that is not valid UTF-8encoding detection failed: unsupported encoding "..."
go
// The data is actually GBK, but is forced to decode as UTF-8
cfg := html.DefaultConfig()
cfg.Encoding = "utf-8"

_, err := html.Extract(gbkBytes, cfg)
if err != nil {
    log.Println(err)
    // encoding detection failed: declared charset "utf-8" but data is not valid UTF-8
}

Match on the message prefix; there is no sentinel

This error has no sentinel value, so errors.Is cannot match it — check for the "encoding detection failed" prefix instead. Undeclared data never produces it: the library decodes as windows-1252 and returns mojibake. A "retry on detection failure" fallback therefore cannot catch mojibake pages; determine the encoding from the source.

Audit Logging ​

With auditing enabled, a failed conversion is recorded as AuditEventEncodingIssue (info level):

go
package main

import (
	"fmt"
	"log"

	"github.com/cybergodev/html"
)

func main() {
	cfg := html.DefaultConfig()
	cfg.Audit = html.DefaultAuditConfig()
	cfg.Audit.Enabled = true
	cfg.Encoding = "utf-8" // Force UTF-8 so GBK bytes fail to convert
	// LogEncodingIssues defaults to true (enabled by DefaultAuditConfig)

	p, err := html.New(cfg)
	if err != nil {
		log.Fatal(err)
	}
	defer p.Close()

	// GBK bytes for "你好"; decoding as UTF-8 fails and is logged
	gbkBytes := []byte{0xC4, 0xE3, 0xBA, 0xC3}
	if _, err := p.Extract(gbkBytes); err != nil {
		fmt.Println("extract failed:", err)
		// extract failed: encoding detection failed: declared charset "utf-8" but data is not valid UTF-8
	}

	for _, entry := range p.GetAuditLog() {
		if entry.EventType == html.AuditEventEncodingIssue {
			fmt.Printf("[encoding issue] %s\n", entry.Message)
			// [encoding issue] declared charset "utf-8" but data is not valid UTF-8
		}
	}
}

When the event is recorded

AuditEventEncodingIssue is recorded only when conversion fails, which means a wrong forced encoding. Normal documents never produce it. The event is info level (the lowest): it says the decode may be unreliable, not that security is affected. To filter it out, use NewLevelFilteredSink with a minimum level of warning.

Next Steps ​