Processor Cache & Reuse

This guide explains the difference between package functions and Processor instances, helping you make the right choice for different scenarios and achieve optimal performance.

Two Calling Modes

Package Functions (One-time Calls)

result, err := html.Extract(data)

Internally uses sync.Pool to manage temporary Processors. Each call borrows from the pool and returns it when done.

Use cases: Low-frequency calls (e.g., CLI tools, one-time scripts)

Lifecycle:

Call Extract()
  → Get Processor from sync.Pool (or create new)
  → Execute extraction
  → Return to sync.Pool

Processor Instance (Reuse Mode)

p, _ := html.New()
defer p.Close()

for _, page := range pages {
    result, _ := p.Extract(page)
}

Creates a standalone Processor instance with manual lifecycle management.

Use cases: High-frequency calls (e.g., web services, crawlers)

Lifecycle:

html.New()
  → Create Processor (cache, audit, statistics)
  → Loop calling p.Extract() (reuse cache)
  → defer p.Close()

How to Choose

Scenario	Recommended	Reason
CLI tools, one-time processing	Package functions	Simple, no management needed
Web services, API backends	Processor instance	Cache acceleration, statistics monitoring
Batch crawlers	Processor instance	Cache deduplication, resource control
Test code	Package functions	Stateless, test isolation

Cache Mechanism

Processor instances include content-based caching. Identical HTML input won't be processed twice.

Cache Configuration

cfg := html.DefaultConfig()
cfg.MaxCacheEntries = 2000     // Max cache entries (0=disable)
cfg.CacheTTL = time.Hour       // Cache TTL
cfg.CacheCleanup = 5 * time.Minute // Background cleanup interval

Parameter	Default	Description
MaxCacheEntries	2000	Cache capacity limit; set to 0 to disable
CacheTTL	1 hour	Entry expiration time
CacheCleanup	5 minutes	Interval for background cleanup of expired entries

Cache Key Generation

Cache keys are generated from the UTF-8 converted content:

• Content < 64KB: Hash computed over full content
• Content > 64KB: 5-point sampling algorithm (head + tail + uniform sampling)

Identical HTML content hits the cache on repeated calls, skipping parsing and extraction.

Monitoring Cache Hit Rate

p, _ := html.New()
defer p.Close()

// Process a batch of pages
for _, page := range pages {
    p.Extract(page)
}

// Get statistics
stats := p.GetStatistics()
fmt.Printf("Total processed: %d\n", stats.TotalProcessed)
fmt.Printf("Cache hits: %d\n", stats.CacheHits)
fmt.Printf("Cache misses: %d\n", stats.CacheMisses)

hitRate := float64(stats.CacheHits) / float64(stats.TotalProcessed) * 100
fmt.Printf("Hit rate: %.1f%%\n", hitRate)

Recommended Patterns

Web Service Singleton

In web services, use a singleton Processor:

var processor *html.Processor

func init() {
    cfg := html.DefaultConfig()
    cfg.MaxCacheEntries = 5000
    cfg.CacheTTL = 30 * time.Minute
    cfg.ProcessingTimeout = 10 * time.Second

    var err error
    processor, err = html.New(cfg)
    if err != nil {
        log.Fatal(err)
    }
}

func handleExtract(w http.ResponseWriter, r *http.Request) {
    data, _ := io.ReadAll(r.Body)
    result, err := processor.Extract(data)
    if err != nil {
        http.Error(w, err.Error(), 500)
        return
    }
    json.NewEncoder(w).Encode(result)
}

Crawler Batch Processing

p, _ := html.New(html.DefaultConfig())
defer p.Close()

urls := crawlURLs()
pages := fetchPages(urls) // [][]byte

batch := p.ExtractBatch(pages)
fmt.Printf("Success: %d, Failed: %d\n", batch.Success, batch.Failed)

Periodic Maintenance

Long-running Processors need periodic maintenance:

// Periodic cache cleanup (prevent memory growth)
go func() {
    ticker := time.NewTicker(10 * time.Minute)
    for range ticker.C {
        p.ClearCache()
    }
}()

// Periodic statistics reset (preserve cache)
go func() {
    ticker := time.NewTicker(time.Hour)
    for range ticker.C {
        stats := p.GetStatistics()
        log.Printf("Processed %d times, errors %d",
            stats.TotalProcessed, stats.ErrorCount)
        p.ResetStatistics()
    }
}()

Performance Comparison

Processing the same HTML 1000 times (for reference only):

Mode	First Processing	Cache Hit
Package functions	Baseline	No cache
Processor (no cache)	≈Baseline	≈Baseline
Processor (with cache)	≈Baseline	≈1/10 of baseline

Cache Activation

Cache only works on Processor instances. Package-level functions reuse a Processor via sync.Pool, but the pooled config disables caching (MaxCacheEntries = 0) and clears the cache on return, so caching is unavailable.

Common Pitfalls

Pitfall	Correct Approach
Creating a new Processor with html.New() every call	Reuse the same instance
Forgetting to call p.Close()	Use defer p.Close()
Expecting cache from package functions	Cache only works on Processor instances
Using a closed Processor	Check for ErrProcessorClosed error

Next Steps

• Performance - More performance tuning tips
• API Reference: Processor - Complete method list
• API Reference: Config - Cache configuration details