Model Drift Detection and Bias/Fairness Evaluation in LangSmith

Summary

LangSmith does not provide dedicated out-of-the-box drift detection or bias/fairness scoring systems with automatic baseline comparison. However, it offers flexible building blocks that can be combined to achieve similar observability outcomes for LLMs and AI agents.

Model Drift Detection

What LangSmith Offers

LangSmith provides several capabilities that can be combined to monitor model behavior changes over time:

Key Differences from Traditional ML Drift Detection

Traditional ML drift detection typically involves:

• Statistical comparison of feature distributions (data drift)

• Monitoring prediction probability distributions (concept drift)

• Automatic baseline computation and threshold alerts

LangSmith's approach is tailored for LLM/agent observability:

• Quality is assessed via evaluator scores rather than statistical distribution tests

• Baselines are established through experiments rather than automatic reference windows

• Drift is detected through trend analysis and evaluator score degradation rather than statistical drift tests

Bias and Fairness Detection

What LangSmith Offers

LangSmith does not include built-in bias or fairness metrics. However, custom bias checks can be implemented using:

Implementation Patterns

Custom Code Evaluator Example:

def bias_check_evaluator(run, example):
    output = run.outputs.get("response", "")
    # Custom logic to check for demographic parity, 
    # sensitive term usage, or other bias indicators
    bias_score = custom_bias_detection_logic(output)
    return {"key": "bias_score", "score": bias_score}

LLM-as-Judge for Bias Detection:

• Configure a prompt that instructs an LLM to evaluate outputs for bias

• Can assess nuanced bias that rule-based systems might miss

• Useful for detecting subtle tone, framing, or representation issues

Recommended Approach for AI Observability

For teams seeking comprehensive AI observability across traditional ML, LLMs, and AI agents:

1. Establish Baselines via Experiments

• Create a reference dataset representing expected behavior

• Run experiments to establish baseline scores

• Use comparison views to track deviations in subsequent runs

2. Implement Online Evaluations

• Deploy custom evaluators for quality, bias, and consistency checks

• Configure evaluators to run on sampled production traffic

• Set up automation rules to flag concerning patterns

3. Build Custom Dashboards

• Track key metrics over time (latency, cost, error rates, custom scores)

• Create views segmented by model version, prompt version, or time period

• Use these to visually identify drift trends

4. Set Up Automation Rules

• Configure alerts when scores fall below thresholds

• Route flagged traces to review queues

• Automatically add edge cases to datasets for retraining

What LangSmith Does NOT Provide

To set clear expectations:

• Automatic statistical drift detection - No automatic computation of KL divergence, PSI, or other statistical drift metrics

• Built-in bias/fairness metrics - No prebuilt demographic parity, equalized odds, or disparate impact calculations

• Reference data upload with automatic comparison - Baselines are established through experiments rather than uploaded reference datasets

• Traditional ML model support - LangSmith is designed for LLM/agent observability, not traditional ML models

Related Documentation

• Monitoring dashboards 

• Online evaluations 

• Experiment comparison 

• Automation rules

• Custom code evaluators (UI | SDK) 

• LLM-as-judge evaluators

• Online code evaluators

Summary Table