Context

When setting up evaluations or automation rules in LangSmith, users may encounter the error "Maximum number of run rules exceeded, please contact support" which prevents them from creating new rules. This occurs when the account has reached its default limit for run rules.

Answer

Each LangSmith account has a default limit of 100 run rules. This limit applies collectively to:

• Automation rules

• Online/custom code evaluators (including LLM-as-a-judge evaluators)

• Binding evaluators to datasets

In some cases, this error message may also appear due to data-parsing issues with run data in your dataset, even if you haven't reached the actual rule limit. This typically occurs when the system encounters malformed JSON data while processing runs.

To resolve this issue, you can:

1. Review and delete any unused evaluation rules or automation rules to free up space within your current limit.

Example for the list of rules call:

curl -X GET "https://api.smith.langchain.com/api/v1/runs/rules" \
-H "x-api-key: YOUR_LANGSMITH_API_KEY

Example for the deletion:

curl -X DELETE "https://api.smith.langchain.com/api/v1/runs/rules/YOUR_RULE_ID" \
-H "x-api-key: YOUR_LANGSMITH_API_KEY"

2. If you're still getting the error after confirming you're under the limit, check your dataset for any runs with malformed or corrupted data that might be causing parsing errors

3. Contact support at support@langchain.dev to request an increase in your run rules limit or help troubleshoot data-parsing issues

Note: Creating an LLM-as-a-judge evaluator or binding an evaluator to a dataset both count as run rules against your limit, regardless of which dataset they're applied to.

For more information about run rules, you can refer to the following documentation:

• Automation Rules Documentation

• Online Evaluations Documentation

• LangSmith API