Envoy ExternalProcessor Filter Bug 404 Error After Changing Authority

Introduction

Hey guys! Today, we're diving deep into a tricky issue encountered while using Envoy's ExternalProcessor filter to dynamically route requests. The problem? A perplexing 404 error despite the filter correctly modifying the :authority and :path headers. We'll dissect the configuration, analyze the logs, and try to figure out why Envoy isn't routing the request as expected. So, buckle up, and let's get started!

Configuration and Setup

Our setup involves an Envoy proxy configured with the ExternalProcessor filter and the Dynamic Forward Proxy (DFP) filter. The goal is to use the ExternalProcessor to rewrite the request headers, specifically the :authority and :path, based on some external logic. This allows us to dynamically route requests to different backends. Here’s the relevant Envoy configuration:

admin:
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 9901
static_resources:
  listeners:
    - name: main_listener
      address:
        socket_address:
          address: 0.0.0.0
          port_value: 5001
      filter_chains:
      - filters:
        - name: envoy.filters.network.http_connection_manager
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
            stat_prefix: ingress_http
            route_config:
              name: local_route
              virtual_hosts:
              - name: local_service
                domains:
                  - "*"
                routes:
                - match:
                    prefix: "/"
                  route:
                    cluster: dynamic_forward_proxy_cluster
            http_filters:
            - name: envoy.filters.http.ext_proc
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor
                mutation_rules:
                  allow_all_routing: true
                grpc_service:
                  envoy_grpc:
                    cluster_name: ext_afproxy
                  timeout: 15s
                processing_mode:
                  request_header_mode: SEND
                  response_header_mode: "SKIP"
                  request_body_mode: "NONE"
                  response_body_mode: "NONE"
                  request_trailer_mode: "SKIP"
                  response_trailer_mode: "SKIP"
            - name: envoy.filters.http.dynamic_forward_proxy
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.http.dynamic_forward_proxy.v3.FilterConfig
                dns_cache_config:
                  name: dynamic_forward_proxy_cache_config
                  dns_lookup_family: V4_ONLY
                  typed_dns_resolver_config:
                    name: envoy.network.dns_resolver.getaddrinfo
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.network.dns_resolver.getaddrinfo.v3.GetAddrInfoDnsResolverConfig
            - name: envoy.filters.http.router
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
  clusters:
  - name: dynamic_forward_proxy_cluster
    lb_policy: CLUSTER_PROVIDED
    cluster_type:
      name: envoy.clusters.dynamic_forward_proxy
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.clusters.dynamic_forward_proxy.v3.ClusterConfig
        dns_cache_config:
          name: dynamic_forward_proxy_cache_config
          dns_lookup_family: V4_ONLY
          typed_dns_resolver_config:
            name: envoy.network.dns_resolver.getaddrinfo
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.network.dns_resolver.getaddrinfo.v3.GetAddrInfoDnsResolverConfig
  - name: ext_afproxy
    connect_timeout: 5s
    type: LOGICAL_DNS
    dns_lookup_family: V4_ONLY
    lb_policy: ROUND_ROBIN
    typed_extension_protocol_options:
      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
        "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
        explicit_http_config:
          http2_protocol_options: {}
    load_assignment:
      cluster_name: ext_afproxy
      endpoints:
        - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: localhost
                    port_value: 50052

In this configuration, we have a listener on port 5001 that uses an HTTP connection manager. The HTTP filter chain includes the ExternalProcessor filter, the DFP filter, and the router filter. The ExternalProcessor filter is configured to send request headers to a gRPC service (ext_afproxy) and to skip response headers. It's also set to clear the route cache after applying header mutations, which is crucial for dynamic routing changes to take effect. The DFP filter is set up to handle dynamic forward proxying, and the ext_afproxy cluster represents the gRPC service that processes the header mutations.

The Problem: 404 Errors After Header Rewrites

The core issue is that while the ExternalProcessor filter correctly rewrites the headers, Envoy returns a 404 error. For instance, a request to http://localhost:5001/foo/bar is rewritten to have :authority localhost:4001 and :path /bar. The logs indicate that the header mutation is accepted and that the route cache is cleared. However, the following logs reveal the problem:

[2025-07-25 17:06:29.587][27262002][debug][router] [source/common/router/config_impl.cc:2071] route was resolved but final route list did not match incoming request
[2025-07-25 17:06:29.587][27262002][debug][router] [source/common/router/router.cc:471] [Tags: "ConnectionId":"0","StreamId":"12516043724031728105"] no route match for URL ''

This suggests that after the headers are rewritten, Envoy cannot find a matching route for the request. The