Context

When using LangSmith for tracing and evaluations, users may experience unexpected spikes in billing due to high trace volumes or extended data retention settings. This can occur particularly when running evaluators across historical traces or having incorrect retention settings configured.

Answer

To prevent unexpected high billing from trace evaluations and manage your LangSmith costs effectively, follow these steps:

1. Monitor Your Trace Volume

   • Be aware of your normal trace volume (e.g., typical monthly traces)

   • Exercise caution when running evaluators on historical data

   • Avoid running evaluators across large historical datasets without first calculating the potential impact

2. Configure Data Retention Settings

   • Review your data retention settings regularly

   • Use default retention settings unless extended retention is specifically needed

   • Configure retention settings at: Data Retention Settings

3. Set Usage Limits

   • Implement usage limits to prevent overspend

   • Configure limits at: Usage Limits

   • Note: Set usage limits at the start of your billing cycle to ensure effectiveness

Advanced Cost Management for LangSmith Evaluations

Understanding Automatic Retention Changes

When running evaluators on traces in LangSmith, the system automatically moves base traces to extended data retention, which significantly increases storage costs. This automatic mechanism is the primary cause of unexpected billing spikes when evaluating historical data.

Key Technical Detail: Running evaluators on any trace – regardless of its original retention setting – will automatically upgrade it to extended retention for the duration of your configured retention period.

Strategic Timing for Usage Limits

While the standard advice is to set usage limits at the start of billing cycles, there are important timing considerations:

• Mid-cycle high usage: If you've already exceeded normal usage mid-cycle, wait until the next billing period to configure limits

• Reason: Setting limits during high‑usage periods may block necessary tracing operations and disrupt ongoing workflows

• Best practice: Monitor usage patterns and set limits proactively rather than reactively

Cost Calculation Before Large Evaluations

Before running evaluators on historical datasets:

1. Calculate retention impact: Multiply the number of traces by your extended retention period cost

2. Factor in evaluation frequency: Consider how often you'll need to re‑run evaluations on the same data

3. Plan data lifecycle: Determine if you need to keep evaluated traces in extended retention or can archive them after analysis

This proactive approach prevents billing surprises and helps you budget accurately for evaluation projects.