Context

Users working with multiple LangSmith tracing projects may want to reuse the same evaluator configurations across different projects to maintain consistency in their evaluation process. This is particularly relevant for teams using infrastructure-as-code approaches like Terraform in their CI/CD pipelines.

Answer

Currently, there is no direct way to share evaluators across different LangSmith tracing projects through the user interface. However, there are two alternative approaches you can use:

1. Use the Prompt Hub to reuse prompts from LLM judge evaluators across projects. This allows you to maintain consistency in your evaluation prompts.

2. Implement run rules programmatically using the LangSmith REST API. You can create run rules by making POST requests to the following endpoint:

   https://api.smith.langchain.com/api/v1/runs/rules

   This approach can be integrated into your Terraform-based CI/CD pipeline by incorporating the API calls into your infrastructure code.

   Important: Run rules are counted account-wide regardless of which project they're applied to, with a default limit of 100 rules per organization. When reusing evaluators across multiple projects, you may quickly reach this limit. If you encounter a "maximum number of run rules exceeded" error, you'll need to either clean up unused run rules or contact support to increase your limit. Take a look at our knowledge article for more information.

For detailed API documentation, refer to the LangSmith API Reference.