Skip to content

Log Sampling ​

In high-throughput scenarios (HTTP request logging, event stream processing), logging every entry produces massive data. DD's sampling feature retains logs proportionally, controlling log volume while still reflecting overall trends.

Sampling Principle ​

DD uses a counter-based sampling strategy:

text
┌──────────────────────────────────────────────────────────┐
│  Requests 1-100  →  all logged (Initial phase)           │
│  Request 101     →  skipped                               │
│  Request 102     →  skipped                               │
│  ...                                                      │
│  Request 110     →  logged (1 of every Thereafter=10)     │
│  Request 111     →  skipped                               │
│  ...                                                      │
│  (Tick expires → counter resets, re-enters Initial phase) │
└──────────────────────────────────────────────────────────┘
ParameterDescriptionTypical Value
EnabledEnable samplingtrue
InitialFirst N entries always logged100
ThereafterLog 1 out of every N after Initial10
TickCounter reset interval (0 = never reset)1s / 1m

Quick Start ​

Enable at Configuration ​

go
package main

import (
	"log"
	"time"

	"github.com/cybergodev/dd"
)

func main() {
	cfg := dd.DefaultConfig()
	cfg.Sampling = &dd.SamplingConfig{
		Enabled:    true,
		Initial:    100,         // First 100 always logged
		Thereafter: 10,          // Then log 1 of every 10
		Tick:       time.Second, // Reset counter every second
	}

	logger, err := dd.New(cfg)
	if err != nil {
		log.Fatal(err)
	}
	defer logger.Close()

	// Simulate high-throughput logging
	for i := 0; i < 1000; i++ {
		logger.InfoWith("request processed",
			dd.Int("seq", i),
		)
	}
	// Actual output: first 100 + 90 of remaining 900 = 190 entries
}

dd.DefaultSamplingConfig() returns the recommended defaults: Initial=100, Thereafter=100, Tick=1s — counters reset every second, so traffic below 100 messages/second is never sampled, and only the excess is reduced to 1 in 100:

go
cfg := dd.DefaultConfig()
sampling := dd.DefaultSamplingConfig()
sampling.Initial = 10 // Adjust as needed: first 10 per second, then 1 in 10
cfg.Sampling = &sampling
logger, _ := dd.New(cfg)

Assignment Activates Sampling

DefaultSamplingConfig() returns Enabled=true, so assigning it to Config.Sampling activates sampling immediately. To use it merely as a "template" that you adjust field by field and enable later, you must set Enabled=false manually:

go
sampling := dd.DefaultSamplingConfig()
sampling.Enabled = false // Inert template: assigning it to Config.Sampling will not activate sampling
// ... after adjusting fields, set sampling.Enabled = true when you want it active

Runtime Toggle ​

go
// Enable sampling
logger.SetSampling(&dd.SamplingConfig{
    Enabled:    true,
    Initial:    50,
    Thereafter: 20,
    Tick:       0, // No auto-reset
})

// Disable sampling (resume full logging)
logger.SetSampling(nil)

// Query current sampling config
sc := logger.GetSampling()
if sc != nil {
    fmt.Printf("Sampling: Initial=%d, Thereafter=%d\n", sc.Initial, sc.Thereafter)
}

Global Logger sampling

Package-level functions dd.SetSampling() and dd.GetSampling() operate on the global Logger directly.

Parameter Details ​

Initial: Initial Full-Volume Window ​

Initial guarantees that the first N entries after startup or Tick reset are all logged, ensuring:

  • Startup-phase initialization logs are not lost
  • Short burst traffic has complete records
  • Period-start state is visible after Tick reset

Thereafter: Sampling Rate ​

ThereafterEffectRetention Rate (after Initial)
1Log every entry (= disabled)100%
10Log 1 of every 1010%
100Log 1 of every 1001%
0Stop logging after Initial0%

Thereafter=0

Thereafter=0 means completely stop logging after the Initial phase. Useful in some scenarios (e.g., only startup logs needed), but ensure no important information is missed.

Tick: Periodic Reset ​

go
// Option A: Reset every second (burst detection)
Sampling: &dd.SamplingConfig{
    Enabled: true, Initial: 100, Thereafter: 10,
    Tick: time.Second,
}

// Option B: No reset (global count, long-term reduction)
Sampling: &dd.SamplingConfig{
    Enabled: true, Initial: 1000, Thereafter: 100,
    Tick: 0,
}

After Tick reset, the counter resets to zero and re-enters the Initial full-volume phase. Useful for observing traffic patterns per period.

Typical Scenarios ​

Scenario 1: HTTP Request Logging ​

go
// High-traffic API: first 100 full, then 10% sampled, reset every second
cfg.Sampling = &dd.SamplingConfig{
    Enabled:    true,
    Initial:    100,
    Thereafter: 10,
    Tick:       time.Second,
}

Scenario 2: Background Task Logging ​

go
// Batch processing: first 50 full, then 1 of every 100, no reset
cfg.Sampling = &dd.SamplingConfig{
    Enabled:    true,
    Initial:    50,
    Thereafter: 100,
    Tick:       0,
}

Scenario 3: Debug Mode Toggle ​

go
// Normal: sampled
logger.SetSampling(&dd.SamplingConfig{
    Enabled: true, Initial: 10, Thereafter: 50,
})

// Troubleshooting: disable sampling, full logging
logger.SetSampling(nil)

// Fixed: restore sampling
logger.SetSampling(&dd.SamplingConfig{
    Enabled: true, Initial: 10, Thereafter: 50,
})

Thread Safety ​

Sampling uses atomic operations (atomic.Int64) for the counter and a mutex for Tick reset. No additional synchronization needed for concurrent logging from multiple goroutines.

Fatal logs bypass sampling

Even with sampling enabled, Fatal-level logs are always written — Fatal must be recorded before program exit and should never be skipped by sampling.

Next Steps ​