Gep 2907 update - TLS mode and allowed routes

[mlavacca]

What type of PR is this?

/kind gep

What this PR does / why we need it:

This PR supersedes #3190.

This PR updates GEP-2907 with two different aspects:

• definition of TLS mode (Terminate and Passthrough)
• what routes can be attached to Listeners that specify TLS configuration.

This PR intends to reach an agreement that will make #2111 and #1474 addressable.

Which issue(s) this PR fixes:

Does this PR introduce a user-facing change?:

NONE

[mlavacca]

/cc @candita @youngnick

[youngnick]

Yeah, this accurately conveys the current state, although I think that we do not have any conformance testing at all about TLSRoute with Terminate behavior.

Edit LGTM, but I'd like to hear from others too.

[mlavacca]

Yeah, this accurately conveys the current state, although I think that we do not have any conformance testing at all about TLSRoute with Terminate behavior.

Edit LGTM, but I'd like to hear from others too.

Yep, I just created #3466 to track conformance tests with TLSRoutes in Terminate Mode.

[mlavacca]

/cc @robscott @shaneutt

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mlavacca, shaneutt

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• geps/OWNERS [mlavacca,shaneutt]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[robscott]

Thanks @mlavacca! Sorry I missed this one!

[robscott]

Why not TLS with passthrough here?

[howardjohn]

If you are doing TLS passthrough and a TCPRoute, its no different than just TCP + TCPRoute. Unless you want to say we allow TLSRoute and TCPRoute to bind to the same listener here, then you would have something like

1. Highest priority: TLS route matching SNI
2. Lowest priority: the (at most 1) TCP route
   ?

[mlavacca]

If you are doing TLS passthrough and a TCPRoute, its no different than just TCP + TCPRoute

Agree with this.

Unless you want to say we allow TLSRoute and TCPRoute to bind to the same listener here

If we allow this, to me it looks like we would overcomplicate an already complex enough problem. Unless we have specific use-cases or requirements for this, I see it as a pointless complication for now, given also the fact that both TCPRoute and TLSRoute are still experimental.

[robscott]

This is a bit confusing to me. Why does the route type matter here? In the case of TLS passthrough, it seems like you'd have two options if you wanted to use SNI to select the backend:

1. Multiple Listeners with different hostnames specified
2. One Listener with multiple TLSRoutes attached, each specifying a different SNI

On the other hand, if you wanted to do TLS termination, you'd have the following options:

1. Route based on HTTP attributes: HTTPRoute
2. Send all traffic attached to a Listener to the same set of backends: HTTPRoute or TCPRoute

Writing that all out makes me continue to think that our <L7 route types may not be entirely necessary (x-ref the related doc).

[howardjohn]

There is a third plausible option, you could do "One Listener with multiple TLSRoutes attached, each specifying a different SNI" but still use terminate. I mean hypothetically, I believe we say this isn't the semantics of the routes.

[robscott]

@howardjohn that's a good point, it's certainly possible with the API surface we currently have, do you know of any common use cases for that?

[mlavacca]

This is good discussion, I have a question: In the case of encrypted L4 traffic, why users should use TCPRoute with TLS Terminate overTLSRoute? This is one of the main aspects I'm trying to capture with this GEP update. The assumption I'm making is: in case you need TLS, use either HTTPRoute or TLSRoute, depending on your needs and use-case.

[keithmattix]

So why do we need both TCPRoute and TLSRoute? Could TCPRoute just have optional TLS config? It's a bit of a sidebar, but the omission. Of TCPRoute here prompts the question I think

[webwurst]

@howardjohn that's a good point, it's certainly possible with the API surface we currently have, do you know of any common use cases for that?

So we don't support the equivilent of this in Istio so I don't have as much concrete data on this. I can only hypothesize that you might want to have 1 GW with a broad wildcard and then scoped TLSRoutes owned by a different persona.

Basically the same story as with HTTP

Yes, we have that use case:

• One GW with wildcard
• TLS termination for HTTP configured in the GW so devs can just add HTTPRoutes to their app deployments
• The app also comes with a MQTT broker. That connection can be terminated the same way in the GW, but at the moment there is no way for the devs to just add a TCPRoute to their several apps. Since it's lacking a discriminator.

For now we need to create separate GW with exact hostnames for each MQTT broker next to the wildcard GW. And for each we need to extend the ReferenceGrant to the wildcard certificate. A less complicated way is much appreciated :)

Either solution is fine for us:

• TLSRoute allowed for terminated TLS connections
• or a TCPRoute with optional hostname field

[mlavacca]

Trying to wrap up this whole discussion here. I think we may need to scope down which route kinds are allowed for each protocol first, because this is something we never (as far as I am concerned) well-defined. What I see as the simplest and most obvious choice here is:

Protocol	Routes	TLS Terminate	TLS Passthough
HTTP	HTTPRoute	yes	no
GRPC	GRPCRoute	yes	no
TLS	TLSRoute	yes	yes
TCP	TCPRoute	no	no
UDP	UDPRoute	no	no

We could definitely discuss if some more complex combinations may exist, such as TCPRoute with protocol TLS and TLS passthrough allowed (described in a comment above), but unless clear demands and use-cases are defined and asked, I'd stick with the most linear and simplest one for now.

This is something we can iterate on and improve over time if new use-cases will pop up.

[mlavacca]

In case this proposal for now looks ok to y'all, I'll use this table in the GEP.

[howardjohn]

I really don't feel like this is the right way to go. It becomes an inconsistent model.

The way I view things, is there are 3 phases of a gateway.

• Downstream accepting -- accepting things from the clinet
• Internal routing and process
• Upstream sending -- what we send to the client

In many cases, these can all be the same protocol. For example, we may accept HTTP, process HTP, and then send HTTP. Easy.

If we look at TLS termination, though, its really more like this:

• Downstream: accept TLS, terminate it, "unwrapping" the inner protocol
• Internal processing: routing based on the inner protocol
• Upstream: either forward the inner protocol, or with BackendTLS, possible wrap in TLS

So for HTTPS with terminate:

• Downstream: accepts HTTPS, unwraps to HTTP
• Routing processes HTTP, as HTTPRoute
• Upstream: send as HTTP

So for TLS with terminate, if we are following this model:

• Downstream: accepts TLS, unwrap to TCP
• Routing: process opaque TCP as TCPRoute
• Upstream: send as opaque TCP data

[howardjohn]

I am not entirely opposed to allowing TLSRoute on terminate - it has value for the exact same reasons ListenerSet does (though perhaps that makes them redundant) - but TCPRoute ought to be allowed on TLS termination.

Another way to look at it: as an application developer, my app is exposed as <HTTP/TCP> so I write a <HTTP/TCP>Route. If the gateway admin decides to add TLS security to the gateway, thats not (necessarily) my problem and I shouldn't have to change my routes.

[keithmattix]

So why do we need both TCPRoute and TLSRoute? Could TCPRoute just have optional TLS config? It's a bit of a sidebar, but the omission. Of TCPRoute here prompts the question I think