LangSmith upgrade fails with SSL certificate verification error when connecting to Azure Cache for Redis

Summary

LangSmith Helm chart upgrade from version 0.11.2 to 0.11.20 fails with SSL certificate verification errors when connecting to Azure Cache for Redis over port 6380. Affected pods include platform-backend, queue, listener, and host-backend services.

Issue Description

After upgrading the LangSmith Helm chart from version 0.11.2 (app version 0.11.9) to version 0.11.20 (app version 0.11.57), multiple pods fail to start with SSL certificate verification errors when attempting to connect to Azure Cache for Redis.

Symptoms:

• Pods in CrashLoopBackOff or failing startup probes

• Multiple pods affected: langsmith-platform-backend, langsmith-platform-host-backend, langsmith-platform-listener, langsmith-platform-queue

• Error message in pod logs:

RuntimeError: Error connecting to Redis. Please make sure the Redis cache is running, 
and that the Redis server is reachable from this service with any provided connection parameters. 
Error 1 connecting to <redis-hostname>.redis.cache.windows.net:6380. 
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1016)

• Deployment succeeds on Kubernetes but application fails to connect to external services

• Issue occurs only with SSL-enabled Redis connections (port 6380)

Environment

Kubernetes Platform: Azure Kubernetes Service (AKS)

Redis Service: Azure Cache for Redis

• SSL enabled (port 6380)

• Using private endpoints with restricted outbound access

Deployment Type: Self-hosted LangSmith

Cause

The root cause is a missing trusted Root Certificate Authority (CA) certificate in the application container's trust store.

Detailed Explanation:

• Certificate Chain Validation Failure: When connecting to Azure Redis, the application receives a certificate chain:

  • Server Certificate (Azure Redis endpoint)

  • Intermediate CA (Microsoft RSA TLS CA 02)

  • Root CA (Microsoft RSA Root CA 2017) ← Missing from trust store

• Connection Rejection: Without the root CA in the trust store, the application cannot validate the certificate chain and rejects the SSL connection with "unable to get local issuer certificate" error.

Workaround

If you need immediate connectivity and can accept reduced security temporarily:

1) Disable SSL verification (NOT RECOMMENDED for production):

- This can be done through connection string parameters depending on the Redis client

• redis://host:6379 (non-SSL port) if available

• rediss://host:6379/o?ssl_cert_reqs=None

3) Rollback to previous version:

helm rollback langsmith -n langsmith

⚠ Warning: These workarounds compromise security and should only be used temporarily while implementing the proper resolution.

Resolution

The proper resolution is to add the Microsoft RSA Root CA 2017 certificate (and other necessary Azure root CAs) to the application's trust store using the LangSmith Helm chart's built-in custom CA configuration.

Step 1: Download Azure Root CA Certificates

# Download Microsoft RSA Root CA 2017
curl -o MicrosoftRSARootCA2017.crt "https://www.microsoft.com/pkiops/certs/Microsoft%20RSA%20Root%20Certificate%20Authority%202017.crt"

# Optional: Also download DigiCert Global Root G2 (recommended)
curl -o DigiCertGlobalRootG2.crt "https://cacerts.digicert.com/DigiCertGlobalRootG2.crt"

Step 2: Convert Certificates from DER to PEM Format

The downloaded certificates are in DER (binary) format and need to be converted to PEM (text) format:

# Convert Microsoft RSA Root CA 2017
openssl x509 -inform DER -in MicrosoftRSARootCA2017.crt -out MicrosoftRSARootCA2017.pem
# Convert DigiCert Global Root G2 (if downloaded)
openssl x509 -inform DER -in DigiCertGlobalRootG2.crt -out DigiCertGlobalRootG2.pem

Verification:

# Verify the conversion
openssl x509 -in MicrosoftRSARootCA2017.pem -noout -subject -enddate

Step 3: Create Combined CA Bundle

Combine the Azure root certificates with the system's existing CA bundle to maintain trust in public certificate authorities:

cat /etc/ssl/certs/ca-certificates.crt MicrosoftRSARootCA2017.pem \
DigiCertGlobalRootG2.pem > custom-ca-bundle.crt

# Alternative: Using Docker to get base CA bundle
docker run --rm alpine:latest cat /etc/ssl/certs/ca-certificates.crt > base-ca-bundle.crt

cat base-ca-bundle.crt MicrosoftRSARootCA2017.pem \
DigiCertGlobalRootG2.pem > custom-ca-bundle.crt

Verification:

# Verify the bundle contains multiple certificates
grep -c "BEGIN CERTIFICATE" custom-ca-bundle.crt

Step 4: Create Kubernetes Secret

Create a Kubernetes secret containing the CA bundle in the same namespace as your LangSmith deployment:

kubectl create secret generic langsmith-custom-ca \
--from-file=ca-bundle.crt=custom-ca-bundle.crt \
-n langsmith

Verification:

kubectl get secret langsmith-custom-ca -n langsmith
kubectl describe secret langsmith-custom-ca -n langsmith

Step 5: Update Helm Values Configuration

Update your Helm values file (e.g., values.yaml) to reference the custom CA bundle:

config:
  customCa:
    secretName: "langsmith-custom-ca"
    secretKey: "ca-bundle.crt"
redis:
  external:
    enabled: true
    # Use rediss:// protocol (note the double 's' for SSL)
    connectionUrl: "rediss://:YOUR_REDIS_PASSWORD@your-redis-host.redis.cache.windows.net:6380"
    # Alternative: Use existing secret for connection URL
    # existingSecretName: "redis-connection-secret"

Important Notes:

- Use rediss:// (with double 's') for SSL connections

- Use port 6380 (Azure's SSL-enabled Redis port)

Step 6: Perform Helm Upgrade

Execute the Helm upgrade with the updated configuration:

helm upgrade langsmith langchain/langsmith --version 0.11.20 \
--namespace langsmith --values values.yaml --wait --timeout 10m

Step 7: Verify the Resolution

Check that pods are running successfully:

# Check pod status
kubectl get pods -n langsmith

# Verify environment variable is set
kubectl exec -it deployment/langsmith-platform-backend -n langsmith -- \
env | grep SSL_CERT_FILE

# Verify certificate file is mounted
kubectl exec -it deployment/langsmith-platform-backend -n langsmith -- \
cat /etc/ssl/certs/custom-ca-certificates.crt | grep -c "BEGIN CERTIFICATE"

# Check pod logs for successful connections
kubectl logs -n langsmith deployment/langsmith-platform-backend --tail=50

# Test Redis connectivity (if redis-cli is available in the pod)
kubectl exec -it deployment/langsmith-platform-backend -n langsmith -- \
redis-cli -h your-redis-host.redis.cache.windows.net -p 6380 \
--tls --cacert /etc/ssl/certs/custom-ca-certificates.crt ping

Troubleshooting

If issues persist after applying the resolution:

1. Verify secret exists and is readable:

kubectl get secret langsmith-custom-ca -n langsmith -o yaml

2. Check that the certificate was mounted correctly:

kubectl describe pod <pod-name> -n langsmith | grep -A 5 "Mounts:"

3. Inspect pod logs for SSL-related errors:

kubectl logs <pod-name> -n langsmith | grep -i "ssl\|certificate\|tls\|redis"

4. Validate certificate expiration dates:

openssl x509 -in MicrosoftRSARootCA2017.pem -noout -enddate
# Microsoft RSA Root CA 2017 expires: Dec 18 23:51:22 2042 GMT

5. Test SSL connection manually:

openssl s_client -connect your-redis-host.redis.cache.windows.net:6380 \
-CAfile custom-ca-bundle.crt \
-servername your-redis-host.redis.cache.windows.net

Known issues

Q: When we combined the 2 certs outlined in the article with the existing certs of the alpine:latest image, the resulting file was larger than the maximum size for a kubernetes secret.

A: You could try to extract the Microsoft RSA Root CA 2017 certificate, e.g.:

openssl x509 -in DigiCertGlobalRootG2.crt -out root-only.crt

then check if it's file size-wise wc -c root-only.crt and create a secret with e.g. --from-file=ca.crt=root-only.crt

Q: After configuring custom certificates without config.customCa - SSO is failing when logging into the LangSmith frontend

A: Custom certificate bundle should be attached to all pods (e.g. platformBackend and backend) if config.customCa is not used, example:

    extraEnv:
      - name: SYSTEM_CERTS_ENABLED
        value: "true"
    volumes:
      - name: custom-cert-bundle
        configMap:
          name: custom-cert-bundle
    volumeMounts:
      - name: custom-cert-bundle
        mountPath: /etc/ssl/certs/ca-certificates.crt
        subPath: custom-bundle.crt

Q: is there a specific public CA certificate for beacon.langchain.com to add to the bundle?

A: This command could show what needs to be included:

echo | openssl s_client -connect beacon.langchain.com:443 -servername beacon.langchain.com -showcerts 2>/dev/null | openssl x509 -noout -issuer -subject

References

LangSmith Self-Hosting Documentation

Azure Cache for Redis TLS Certificate Migration

Microsoft PKI Repository

Azure Cache for Redis SSL Best Practices

OpenSSL Certificate Format Conversion

Kubernetes Secrets Documentation

Configure custom TLS