Skip to content

Configuration Guide ​

Config is the library's single configuration struct, with 31 fields. Most scenarios need one preset plus a few overrides; the full field reference is in API Reference — Config.

Four Presets ​

PresetUse caseKey difference
DefaultConfig()General extractionAll features on, safe defaults
HighSecurityConfig()Untrusted inputTighter limits, audit enabled, lower depth cap
TextOnlyConfig()Plain text onlyAll media preservation off
MarkdownConfig()Inline Markdown outputImages and links rendered as Markdown
go
package main

import (
	"fmt"
	"log"

	"github.com/cybergodev/html"
)

func main() {
	data := []byte(`<html><body><h1>Title</h1><p>Body content</p></body></html>`)

	// Most scenarios: use the default config directly
	p1, err := html.New()
	if err != nil {
		log.Fatal(err)
	}
	defer p1.Close()
	r1, err := p1.Extract(data)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(r1.Title)
	// Output: Title

	// Plain text only (e.g. search engine indexing)
	p2, err := html.New(html.TextOnlyConfig())
	if err != nil {
		log.Fatal(err)
	}
	defer p2.Close()

	// Images and links appear in Text as Markdown inline syntax (e.g. CMS migration)
	p3, err := html.New(html.MarkdownConfig())
	if err != nil {
		log.Fatal(err)
	}
	defer p3.Close()
	r3, err := p3.Extract(data)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(r3.Text)
	// Output: # Title
	//         Body content
}

A zero-value Config is unusable

Config{} fails Validate() (MaxInputSize and others must be positive), so never start from the zero value. Take a preset, then override fields.

go
cfg := html.HighSecurityConfig()
cfg.PreserveImages = false // additionally disable images on top of high security
processor, _ := html.New(cfg)

TextOnlyConfig() only sets PreserveImages/PreserveLinks/PreserveVideos/PreserveAudios to false; it does not touch the Include* filters used by ExtractAllLinks.

MarkdownConfig() only sets InlineImageFormat and InlineLinkFormat; ExtractToMarkdown always uses Markdown formatting, so it does not need that preset.

Six Field Groups ​

Resource Management ​

These affect memory and throughput. Defaults suit most workloads; adjust for your load.

FieldDefaultDescription
MaxInputSize50 MBMaximum input size; prevents memory exhaustion
MaxCacheEntries2000Cache entry limit; 0 disables the cache
CacheTTL1 hourCache time-to-live
CacheCleanup5 minutesBackground interval for clearing expired entries
WorkerPoolSize4Batch concurrency (1–256)
ProcessingTimeout30 secondsPer-document timeout; 0 means no limit

Caching applies only to Processor instances

Package-level functions (e.g. html.Extract) use a pooled or temporary Processor per call, so the cache never survives across calls. For caching, create a standalone Processor with html.New(). See Processor Reuse & Cache.

Security ​

This group matters in production. See Security Overview for the full picture.

FieldDefaultDescription
EnableSanitizationtrueHTML sanitization (removes dangerous tags and attributes)
MaxDepth500DOM nesting limit; prevents stack overflow
AllowedBaseDir""Sandbox directory for file operations; empty means no restriction
AuditDisabledSecurity audit log configuration

AllowedBaseDir and user-supplied paths

Set AllowedBaseDir whenever a file path comes from user input. It resolves the real path through an OS file handle, which defeats symlink and Windows junction escapes.

Content Extraction ​

Controls what is extracted from the HTML.

FieldDefaultDescription
ExtractArticletrueArticle detection (locates the main content)
ExtractMetadatatrueStructured metadata: parses author, publication time, description, site name, and canonical URL, and lets a metadata title take priority
PreserveImagestrueKeep image information
PreserveLinkstrueKeep link information
PreserveVideostrueExtract videos
PreserveAudiostrueExtract audio

Turning off unneeded media types cuts processing work:

go
cfg := html.DefaultConfig()
cfg.PreserveVideos = false
cfg.PreserveAudios = false
// Extract only text, images, and links

What disabling ExtractMetadata changes

It also changes where the title comes from: Result.Title no longer takes the JSON-LD headline or og:title and falls back to the DOM order <title> → the h1 inside the article → the first document h1 → the first h2. Author, PublishedTime, Description, SiteName, and CanonicalURL stay empty.

Output Formats ​

Controls how images and links appear in the text output. See Output Formats.

FieldDefaultOptions
InlineImageFormat"none""none", "markdown", "html", "placeholder"
InlineLinkFormat"none""none", "markdown", "html"
TableFormat"markdown""markdown", "html"
Encoding"" (auto)"utf-8", "gbk", "shift_jis", "windows-1252", etc.

The three format fields accept exported constants directly, which avoids typos: html.FormatNone ("none"), html.FormatMarkdown ("markdown"), html.FormatHTML ("html"), and html.FormatPlaceholder ("placeholder", accepted only by InlineImageFormat). The constants are equivalent to the string literals and follow the same validation rules (case-insensitive); the constant list is in API Reference: Constants & Errors.

go
cfg := html.DefaultConfig()
cfg.InlineImageFormat = html.FormatMarkdown
cfg.InlineLinkFormat = html.FormatHTML

An empty Encoding goes through auto-detection. Setting it skips the detection step, but only do so when the encoding is known — a wrong value yields mojibake or an encoding detection failed error. See Encoding Detection.

These fields apply to ExtractAllLinks and Reader Mode (the ReadPage family). See Link Extraction.

FieldDefaultDescription
ResolveRelativeURLstrueResolve relative URLs to absolute URLs
BaseURL""Resolution base; auto-detected from the HTML when empty
IncludeImagestrueInclude <img> links
IncludeVideostrueInclude <video> and <iframe>/<embed>/<object> embed links
IncludeAudiostrueInclude <audio> links
IncludeCSStrueInclude <link rel="stylesheet">
IncludeJStrueInclude <script src>
IncludeContentLinkstrueInclude <a> links with a relative href
IncludeExternalLinkstrueInclude <a> links with an absolute or protocol-relative href
IncludeIconstrueInclude favicon/icon

Link extraction and content extraction are independent

The Include* fields affect only ExtractAllLinks. Link retention in Extract is controlled by PreserveLinks.

Extensions ​

FieldDescription
ScorerCustom content scorer; when nil, the built-in scorer (internal) is used

A custom Scorer can tune article detection for a specific site. See Testing & Custom Extensions.

Common Combinations ​

Web Crawler ​

Raise concurrency and shorten the timeout for batch crawling:

go
package main

import (
	"log"
	"time"

	"github.com/cybergodev/html"
)

func main() {
	cfg := html.DefaultConfig()
	cfg.WorkerPoolSize = 8                   // Raise batch concurrency
	cfg.ProcessingTimeout = 10 * time.Second // Shorten the timeout
	cfg.PreserveVideos = false               // Crawlers don't need video
	cfg.PreserveAudios = false

	processor, err := html.New(cfg)
	if err != nil {
		log.Fatal(err)
	}
	defer processor.Close()

	// Batch extraction
	pages := [][]byte{[]byte("<html><body>Page 1</body></html>")}
	batch := processor.ExtractBatch(pages)
	log.Printf("Success %d, Failed %d", batch.Success, batch.Failed)
}

API Backend Service ​

For user-submitted HTML, use the high-security config and restrict the file directory:

go
package main

import (
	"log"

	"github.com/cybergodev/html"
)

func main() {
	cfg := html.HighSecurityConfig()
	cfg.AllowedBaseDir = "/var/www/uploads" // Restrict the file directory

	processor, err := html.New(cfg)
	if err != nil {
		log.Fatal(err)
	}
	defer processor.Close()

	// Process a user-uploaded HTML file
	result, err := processor.ExtractFromFile("/var/www/uploads/user.html")
	if err != nil {
		log.Fatal(err)
	}
	log.Println(result.Title)
}

Content Migration Tool ​

Convert legacy site HTML to Markdown:

go
package main

import (
	"fmt"
	"log"

	"github.com/cybergodev/html"
)

func main() {
	processor, err := html.New() // DefaultConfig
	if err != nil {
		log.Fatal(err)
	}
	defer processor.Close()

	data := []byte(`<html><body><article><h1>Old Article</h1><a href="/post/123">Link</a></article></body></html>`)
	md, err := processor.ExtractToMarkdown(data)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(md)
}

ExtractToMarkdown always uses Markdown formatting, independent of InlineImageFormat/InlineLinkFormat, so MarkdownConfig() is not needed here. BaseURL and ResolveRelativeURLs apply to ExtractAllLinks and Reader Mode, and do not rewrite links in an Extract result.

Validate ​

Configs passed to html.New() are validated automatically. You can also call Validate() first:

go
cfg := html.DefaultConfig()
cfg.MaxInputSize = -1 // Intentionally invalid
if err := cfg.Validate(); err != nil {
    log.Fatal(err) // html: invalid config: MaxInputSize=-1, must be positive
}

Validation covers field ranges and format strings. An invalid config returns *ConfigError, which you can test with errors.Is(err, html.ErrInvalidConfig). Full constraints are in API Reference — Config.

Next Steps ​