Skip to content

Sensitive Data Filtering ​

DD has built-in automatic sensitive-data filtering that replaces passwords, API keys, credit card numbers, and other sensitive information with [REDACTED] before logs are written, preventing leakage into logs.

Quick Enable ​

go
cfg := dd.DefaultConfig()
cfg.Security = dd.DefaultSecurityConfig() // Basic filtering (already the DefaultConfig default; made explicit here)
logger, err := dd.New(cfg)
if err != nil {
    log.Fatal(err)
}
defer logger.Close()

// The password field is redacted automatically
logger.InfoWith("user login",
    dd.String("username", "alice"),
    dd.String("password", "s3cr3t123"),    // Output: password=[REDACTED]
)

Built-in Filter Patterns ​

DefaultSecurityConfig() enables basic filtering, covering the following sensitive information:

CategoryMatched Targets
Passwordspassword, passwd, pwd, etc. fields
API Keysapi_key, apikey, access_token, etc.
Credit cardsVisa, MasterCard, and other card-number formats
SSNUS Social Security Number format
Phone numbersGlobal phone-number formats (including international)

SecurityConfigForLevel(dd.SecurityLevelStandard) uses the full pattern set (all of basic plus the following common categories; the former DefaultSecureConfig() is deprecated and returns an equivalent config):

CategoryMatched Targets
Emailsemail address format
IP addressesIPv4/IPv6 addresses
JWT tokensJWT format starting with eyJ
Connection stringspasswords inside database connection strings

Pattern Coverage

The table above only lists common-category examples. DefaultSecurityConfig() (the basic set) actually ships 38 patterns, covering AWS keys, Stripe/GitHub/Slack tokens, Log4Shell, and more; industry presets go higher: HealthcareConfig() 73, FinancialConfig() 74, GovernmentConfig() 75. Note that IBAN and multi-country tax IDs (ABN/NZ-IRD/RUT/CPF/RFC) belong to the full set and industry presets only, not the basic set. See the source internal/patterns.go for the full pattern set.

Custom Filter Patterns ​

Adding Custom Patterns ​

go
filter := dd.NewEmptySensitiveDataFilter()

// Add custom regex patterns (with built-in ReDoS protection)
_ = filter.AddPattern(`(?i)credit_card\s*[:=]\s*\d+`)
_ = filter.AddPattern(`(?i)phone\s*[:=]\s*\d{11}`)

cfg := dd.DefaultConfig()
cfg.Security = &dd.SecurityConfig{
    SensitiveFilter: filter,
}
logger, err := dd.New(cfg)
if err != nil {
    log.Fatal(err)
}
defer logger.Close()

Creating a Custom Filter From Scratch ​

go
filter, err := dd.NewCustomSensitiveDataFilter(
    `(?i)secret_key\s*[:=]\s*\S+`,
    `(?i)private_token\s*[:=]\s*\S+`,
)
if err != nil {
    log.Fatal(err) // Regex safety validation failed
}

ReDoS Protection

NewCustomSensitiveDataFilter has built-in ReDoS validation. If a regex carries catastrophic backtracking risk, it returns an error. Use non-greedy matching and anchored patterns to avoid this.

Security Levels ​

DD provides five security levels, each with a corresponding preset configuration:

go
// Get config by level
cfg := dd.SecurityConfigForLevel(dd.SecurityLevelStandard)
LevelDescriptionUse Case
SecurityLevelDevelopmentMost permissiveLocal development
SecurityLevelBasicBasic filteringTest environments
SecurityLevelStandardStandard filteringGeneral production
SecurityLevelStrictStrict filteringHigh-security requirements
SecurityLevelParanoidMost strictSensitive industries (finance/medical)

Industry-Compliance Presets ​

DD provides three industry-compliance configurations:

HIPAA (Medical) ​

go
cfg := dd.HealthcareConfig()

Extra filtering: ICD-10 codes, medical record numbers (MRN), health insurance claim numbers (HICN), patient identifiers.

PCI-DSS (Financial Payments) ​

go
cfg := dd.FinancialConfig()

Extra filtering: SWIFT codes, IBAN account numbers, CVV/CVC, bank routing numbers.

Government Standard ​

go
cfg := dd.GovernmentConfig()

Extra filtering: passport numbers, driver's license numbers, tax IDs, SSN variants.

Complete Example ​

go
// Medical system: HIPAA-compliant config
cfg := dd.DefaultConfig()
cfg.Format = dd.FormatJSON
cfg.Security = dd.HealthcareConfig()
cfg.Targets = []dd.OutputTarget{
    dd.FileOutput("logs/hipaa-audit.json"),
}
logger, err := dd.New(cfg)
if err != nil {
    log.Fatal(err)
}
defer logger.Close()

// Log messages with sensitive info are auto-redacted
logger.Info("patient record mrn=MRN123456 diagnosis=J18.9 updated")
// MRN and ICD-10 codes in the message match patterns and are redacted

// Structured fields are filtered by key-name sensitivity
logger.InfoWith("user login",
    dd.String("password", "s3cr3t123"),    // -> [REDACTED] (key name is sensitive)
    dd.String("department", "internal"),   // Normal output
)

Filter Statistics ​

Monitor filter runtime state:

go
filter := dd.NewSensitiveDataFilter()
stats := filter.GetFilterStats()
fmt.Printf("active goroutines: %d\n", stats.ActiveGoroutines)
fmt.Printf("filter pattern count: %d\n", stats.PatternCount)
fmt.Printf("total redactions: %d\n", stats.TotalRedactions)
fmt.Printf("timeout count: %d\n", stats.TotalTimeouts)

Disabling Filtering ​

go
// Use the Development security level (no sensitive-data filtering)
cfg := dd.DefaultConfig()
cfg.Security = dd.SecurityConfigForLevel(dd.SecurityLevelDevelopment)
logger, err := dd.New(cfg)
if err != nil {
    log.Fatal(err)
}

// Or manually set an empty SensitiveFilter
cfg = dd.DefaultConfig()
cfg.Security = &dd.SecurityConfig{
    SensitiveFilter: nil,
}
logger, err = dd.New(cfg)
if err != nil {
    log.Fatal(err)
}
defer logger.Close()

Filtering Is Enabled by Default

DefaultConfig() enables basic sensitive-data filtering by default (DefaultSecurityConfig()). Even when the Security field is not set, the default security config is used. To disable filtering, you must explicitly set SensitiveFilter to nil.

Next Steps ​