Context

When deploying LangSmith on Google Kubernetes Engine (GKE) using Gateway API, you may encounter HTTP 405 errors and Gateway programming failures. This typically occurs when the LangSmith operator service template includes GKE Ingress-specific annotations that are incompatible with Gateway API, such as cloud.google.com/backend-config.

Answer

The issue occurs because the cloud.google.com/backend-config annotation is designed for traditional GKE Ingress and is incompatible with Gateway API. When services have this annotation, the GKE Gateway API controller refuses to program the Gateway.

To resolve this issue, follow these steps:

Step 1: Update Your Helm Values

Remove the incompatible annotations from your operator service template in your langsmith-values.yaml file. Revert to the default service template:

operator:
  templates:
    service: |
      apiVersion: v1
      kind: Service
      metadata:
        name: ${name}
        namespace: ${namespace}
      spec:
        type: ClusterIP
        selector:
          app: ${name}
        ports:
        - name: api-server
          protocol: TCP
          port: 8000
          targetPort: 8000

Also remove the entire backendconfig template section if present, as BackendConfig is a GKE Ingress-specific resource not needed with Gateway API.

Step 2: Upgrade Your Helm Chart

Apply the updated values without changing the chart version:

helm upgrade langsmith langsmith/langsmith \
  --namespace langsmith \
  --version 0.x.x \
  --values langsmith-values.yaml

Step 3: Clean Up Existing Resources

Remove existing BackendConfig resources:

kubectl delete backendconfig -n langsmith --all

Step 4: Remove Stale Annotations

The operator only merges annotations from the template into existing services but doesn't remove stale annotations. Manually remove the incompatible annotations from all LGP services:

kubectl annotate service -n langsmith -l app.kubernetes.io/managed-by=lgp-operator cloud.google.com/backend-config-

If you also added custom NEG annotations, remove them as well:

kubectl annotate service -n langsmith -l app.kubernetes.io/managed-by=lgp-operator cloud.google.com/neg-

The trailing - removes the annotation.

Step 5: Monitor Gateway Status

Wait for the Gateway to reconcile and verify it's working:

kubectl get gateway -n langsmith -w

You can also monitor events to ensure the unsupported BackendConfig warnings are resolved:

kubectl get events -n langsmith --field-selector reason=SYNC -w

Important Note

At this moment, LangSmith does not officially support GKE Gateway due to known health check limitations. For production deployments on GKE, consider migrating to Envoy Gateway for better compatibility.

For more information on LangSmith ingress configuration, see the ingress documentation.