Skip to content

For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

gRPC statistics

Verified : Automated tests passed for: Kubernetes resources: The Kubernetes resources on this page were checked against their API schemas. End-to-end run: Every step ran in a fresh test environment and the results were checked.
Page as Markdown
Collect per-service and per-method gRPC metrics from Gateway listeners.
Use the grpcStats field on a ListenerPolicy resource to add Envoy’s grpc_stats HTTP filter to the listeners on a Gateway. The filter records per-service and per-method gRPC metrics, including the gRPC status code that is not visible in ordinary HTTP response-code metrics.
Before you begin

Follow the gRPC routing guide to set up the grpc Gateway, the sample yages.Echo service, and the grpcurl-client pod. The steps in this guide reuse that setup to generate the gRPC traffic that the grpc_stats filter counts.
Show statistics for all methods

Set statsForAllMethods to true to emit per-method statistics for every gRPC method that the listener sees. This option is useful for trusted services with a known, bounded set of method names. If client-supplied method names might create too many metric series, use an allow list instead so that you show statistics for specific methods.
  1. Create a ListenerPolicy that enables the grpcStats filter on the Gateway for all gRPC methods.
    kubectl apply -f- <<EOF
apiVersion: gateway.kgateway.dev/v1alpha1
kind: ListenerPolicy
metadata:
  name: grpc-stats
  namespace: kgateway-system
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: grpc
  default:
    httpSettings:
      grpcStats:
        statsForAllMethods: true
        enableUpstreamStats: true
EOF
    FieldDescription
    spec.targetRefsThe Gateway to apply this listener policy to.
    spec.default.httpSettings.grpcStatsAdds Envoy’s grpc_stats HTTP filter to the listener filter chain. Omit this block to leave the filter disabled. An empty block is rejected.
    statsForAllMethodsSet to true to emit per-method statistics for every gRPC method that the listener sees. Do not set this field together with methodAllowlist.
    enableUpstreamStatsOptional. Emits a histogram for the upstream wire latency of each request.
  2. Port-forward the gateway proxy admin port, 19000.
    kubectl port-forward deployment/grpc -n kgateway-system 19000
  3. Verify that the Envoy configuration includes the grpc_stats filter before the router filter.
    curl -s 127.0.0.1:19000/config_dump | \
jq '.configs[]
  | select(.["@type"] == "type.googleapis.com/envoy.admin.v3.ListenersConfigDump")
  | .dynamic_listeners[].active_state.listener.filter_chains[].filters[].typed_config.http_filters'
    Example output:
    [
  {
    "name": "envoy.filters.http.grpc_stats",
    "typed_config": {
      "@type": "type.googleapis.com/envoy.extensions.filters.http.grpc_stats.v3.FilterConfig",
      "enable_upstream_stats": true,
      "stats_for_all_methods": true
    }
  },
  {
    "name": "envoy.filters.http.router",
    "typed_config": {
      "@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"
    }
  }
]
  4. Send a gRPC request through the Gateway. Because you do not provide a proto file, grpcurl first calls ServerReflectionInfo to resolve the method signature, so you see stats for both calls. The -vv flag prints verbose output.
    kubectl exec -n kgateway-system grpcurl-client -c grpcurl -- \
  grpcurl -plaintext -authority grpc.com -vv grpc:80 yages.Echo/Ping
  5. Query the admin stats endpoint for the per-method grpc. stats. Scope the search to the backend’s cluster so that you don’t also match the kgateway control plane’s own cluster.grpc.<namespace> cluster, which happens to contain the same .grpc. substring.
    curl -s 127.0.0.1:19000/stats | grep '^cluster\.kube_.*\.grpc\.'
    Example output. The kube_<namespace>_<service>_<port> cluster name prefix depends on your backend.
    cluster.kube_kgateway-system_grpc-echo-svc_3000.grpc.grpc.reflection.v1alpha.ServerReflection.ServerReflectionInfo.0: 1
cluster.kube_kgateway-system_grpc-echo-svc_3000.grpc.grpc.reflection.v1alpha.ServerReflection.ServerReflectionInfo.request_message_count: 1
cluster.kube_kgateway-system_grpc-echo-svc_3000.grpc.grpc.reflection.v1alpha.ServerReflection.ServerReflectionInfo.response_message_count: 1
cluster.kube_kgateway-system_grpc-echo-svc_3000.grpc.grpc.reflection.v1alpha.ServerReflection.ServerReflectionInfo.success: 1
cluster.kube_kgateway-system_grpc-echo-svc_3000.grpc.grpc.reflection.v1alpha.ServerReflection.ServerReflectionInfo.total: 1
cluster.kube_kgateway-system_grpc-echo-svc_3000.grpc.yages.Echo.Ping.0: 1
cluster.kube_kgateway-system_grpc-echo-svc_3000.grpc.yages.Echo.Ping.request_message_count: 1
cluster.kube_kgateway-system_grpc-echo-svc_3000.grpc.yages.Echo.Ping.response_message_count: 1
cluster.kube_kgateway-system_grpc-echo-svc_3000.grpc.yages.Echo.Ping.success: 1
cluster.kube_kgateway-system_grpc-echo-svc_3000.grpc.yages.Echo.Ping.total: 1
cluster.kube_kgateway-system_grpc-echo-svc_3000.grpc.yages.Echo.Ping.upstream_rq_time: P0(nan,0) P25(nan,0) P50(nan,0) P75(nan,0) P90(nan,0) P95(nan,0) P99(nan,0) P99.5(nan,0) P99.9(nan,0) P100(nan,0)

    StatDescription
    <status-code> (for example .0)Counter for calls to this method that returned the given gRPC status code. 0 is OK.
    successNumber of calls to this method that completed with a gRPC status that indicates success.
    failureNumber of calls to this method that completed with a gRPC status that indicates failure. Envoy only creates a counter the first time it is incremented, so failure doesn’t appear until at least one call fails.
    totalTotal number of calls to this method, equal to success plus failure.
    request_message_count / response_message_countNumber of request or response messages sent for this method.
    upstream_rq_timeHistogram of the upstream wire latency for calls to this method. Only emitted when enableUpstreamStats is true.
Show statistics for specific methods

Set methodAllowlist to record per-method statistics for only the gRPC methods that you specify. Use this option instead of statsForAllMethods when client-supplied method names might otherwise create too many metric series.
  1. Create a ListenerPolicy that enables the grpcStats filter on the Gateway. Set the methodAllowlist field to the methods that you want to track.
    kubectl apply -f- <<EOF
apiVersion: gateway.kgateway.dev/v1alpha1
kind: ListenerPolicy
metadata:
  name: grpc-stats
  namespace: kgateway-system
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: grpc
  default:
    httpSettings:
      grpcStats:
        methodAllowlist:
        - /yages.Echo/Ping
EOF
    FieldDescription
    spec.targetRefsThe Gateway to apply this listener policy to.
    spec.default.httpSettings.grpcStatsAdds Envoy’s grpc_stats HTTP filter to the listener filter chain. Omit this block to leave the filter disabled. An empty block is rejected.
    methodAllowlistFully qualified gRPC methods to record, in the /package.Service/Method format. The list must contain 1-128 methods. Do not set this field together with statsForAllMethods.
  2. If you completed Show statistics for all methods against this Gateway, restart the gateway proxy. Otherwise, skip this step. Envoy keeps the per-method stats from the earlier policy, and the allow list counts calls in new stats that have the same names. Without a restart, the stats endpoint can keep showing the old values instead of the new counts.
    kubectl rollout restart deployment/grpc -n kgateway-system
kubectl rollout status deployment/grpc -n kgateway-system
  3. Port-forward the gateway proxy admin port, 19000.
    kubectl port-forward deployment/grpc -n kgateway-system 19000
  4. Verify that the Envoy configuration includes the grpc_stats filter before the router filter.
    curl -s 127.0.0.1:19000/config_dump | \
jq '.configs[]
  | select(.["@type"] == "type.googleapis.com/envoy.admin.v3.ListenersConfigDump")
  | .dynamic_listeners[].active_state.listener.filter_chains[].filters[].typed_config.http_filters'
    Example output:
    [
  {
    "name": "envoy.filters.http.grpc_stats",
    "typed_config": {
      "@type": "type.googleapis.com/envoy.extensions.filters.http.grpc_stats.v3.FilterConfig",
      "individual_method_stats_allowlist": {
        "services": [
          {
            "service_name": "yages.Echo",
            "method_names": ["Ping"]
          }
        ]
      }
    }
  },
  {
    "name": "envoy.filters.http.router",
    "typed_config": {
      "@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"
    }
  }
]
  5. Call the allow-listed method and a different method that is not on the allow list. grpcurl’s list subcommand calls the gRPC reflection service, which is a different method (ServerReflectionInfo) than the one you allowed in the methodAllowlist field.
    kubectl exec -n kgateway-system grpcurl-client -c grpcurl -- \
  grpcurl -plaintext -authority grpc.com -vv grpc:80 yages.Echo/Ping

kubectl exec -n kgateway-system grpcurl-client -c grpcurl -- \
  grpcurl -plaintext -authority grpc.com -vv grpc:80 list
  6. Query the admin stats endpoint for the per-method grpc. stats. Verify that you see stats for the yages.Echo.Ping method by name. The ServerReflectionInfo call still counts toward the cluster’s aggregate grpc.* stats, but doesn’t get its own grpc.<service>.<method>.* breakdown because it isn’t listed in the methodAllowlist field.
    curl -s 127.0.0.1:19000/stats | grep '^cluster\.kube_.*\.grpc\.'
    Example output:
    cluster.kube_kgateway-system_grpc-echo-svc_3000.grpc.0: 2
cluster.kube_kgateway-system_grpc-echo-svc_3000.grpc.request_message_count: 2
cluster.kube_kgateway-system_grpc-echo-svc_3000.grpc.response_message_count: 2
cluster.kube_kgateway-system_grpc-echo-svc_3000.grpc.success: 2
cluster.kube_kgateway-system_grpc-echo-svc_3000.grpc.total: 2
cluster.kube_kgateway-system_grpc-echo-svc_3000.grpc.yages.Echo.Ping.0: 1
cluster.kube_kgateway-system_grpc-echo-svc_3000.grpc.yages.Echo.Ping.request_message_count: 1
cluster.kube_kgateway-system_grpc-echo-svc_3000.grpc.yages.Echo.Ping.response_message_count: 1
cluster.kube_kgateway-system_grpc-echo-svc_3000.grpc.yages.Echo.Ping.success: 1
cluster.kube_kgateway-system_grpc-echo-svc_3000.grpc.yages.Echo.Ping.total: 1

    StatDescription
    grpc.<stat> (no service or method in the name)Count of the gRPC calls on this cluster to methods that aren’t in methodAllowlist, such as ServerReflectionInfo here. Calls to allow-listed methods are counted in their own grpc.<service>.<method>.<stat> stats instead.
    grpc.<service>.<method>.<status-code> (for example .0)Counter for calls to this method that returned the given gRPC status code. 0 is OK.
    successNumber of calls that completed with a gRPC status that indicates success.
    failureNumber of calls that completed with a gRPC status that indicates failure. Envoy only creates a counter the first time it is incremented, so failure doesn’t appear until at least one call fails.
    totalTotal number of calls, equal to success plus failure.
    request_message_count / response_message_countNumber of request or response messages sent.
    Tip
    Envoy never clears a stat that it already created, even after you change the ListenerPolicy. If you switch from statsForAllMethods to methodAllowlist without restarting the gateway proxy, the per-method stats from the earlier policy stay. The stats endpoint can then keep showing their old values instead of the new counts. To start from a clean set of stats, restart the gateway proxy as described in step 2.
Clean up

Remove the ListenerPolicy resources that you created.
kubectl delete listenerpolicy grpc-stats -n kgateway-system --ignore-not-found
Destination typesBuffering
Was this page helpful?