Context
When upgrading LangSmith to version 0.13, you may encounter issues with the langsmith-ingest-queue pods failing to start due to S3 blob storage authentication errors. This typically manifests as either "Invalid region: region was not a valid DNS name" or "AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity" errors in the pod logs.
Answer
This issue occurs because the IRSA (IAM Roles for Service Accounts) annotation is not properly configured for the ingestQueue service account in LangSmith v0.13. Follow these steps to resolve the problem:
Add IRSA annotation to your values.yaml file:
Include the following configuration in your Helm values file:ingestQueue: serviceAccount: annotations: eks.amazonaws.com/role-arn: "arn:aws:iam::YOUR_ACCOUNT_ID:role/YOUR_LANGSMITH_IRSA_ROLE"Replace
YOUR_ACCOUNT_IDandYOUR_LANGSMITH_IRSA_ROLEwith your actual AWS account ID and IAM role name.Update your IAM role's trust policy:
Ensure that your IAM role's trust policy allows the langsmith-ingest-queue service account to assume the role. The trust policy should include the appropriate conditions for your EKS cluster and service account.Redeploy LangSmith:
Apply the updated configuration by upgrading your Helm deployment with the new values.yaml file.Verify the fix:
Check that the langsmith-ingest-queue pods are now running successfully and no longer showing authentication errors in their logs.
This configuration change resolves both the S3 blob storage authentication issues and related SSO login problems that may occur during the upgrade to v0.13.