netty: drain before closing a gracefully shut down server connection (#13086) Fixes #9566. After the second GOAWAY of a graceful shutdown (`maxConnectionAge`, `maxConnectionIdle`, `Server.shutdown()`), `Http2ConnectionHandler` closes the socket as soon as the last stream's final frame has been written to the kernel. If the client hasn't read the response yet, the response is still in the server's send queue. The client's next frame (a BDP PING or WINDOW_UPDATE) then reaches a socket the server has closed, so the server's kernel answers with RST and drops the unsent bytes, trailers included. The client fails an RPC the server completed, with `UNAVAILABLE: Connection closed after GOAWAY. HTTP/2 error code: NO_ERROR, debug data: max_age`. The response headers had already arrived, so the call is committed and isn't retried. The full analysis and a standalone reproducer are in https://github.com/grpc/grpc-java/issues/9566#issuecomment-5930328142. ### Change Once the second GOAWAY has been sent and no streams are active, `NettyServerHandler` keeps the connection open and reading for up to 1 second before closing. The client normally closes first, as soon as it has read the GOAWAY and its streams have completed, so the drain usually ends early. This is the approach Go's HTTP/2 server took (golang/net@cd69bc3). - It overrides `isGracefulShutdownComplete()`, the hook `NettyClientHandler` already uses, so the close still goes through `Http2ConnectionHandler` and works the same with TLS. - It never extends the grace time: if `maxConnectionAgeGrace` (or the shutdown grace) runs out first, the connection closes then. - The drain is cancelled when the channel becomes inactive. ### Tests - `NettyServerHandlerTest`: 9 tests asserted that the channel closes immediately after the second GOAWAY. They now advance the fake clock by the drain time first. Two new tests: `maxConnectionAge_drainStartsWhenLastStreamCloses` and `maxConnectionAgeGrace_shorterThanDrain_closesAtGrace`. - New `NettyServerGracefulCloseTest`: an end-to-end regression test over a real loopback connection, using only public APIs and the server's default transport. A unary call outlives the max age, and the client's event loop stalls 300 ms while the 256 KiB response is sent. Without the fix it fails on Linux with the error above (3/3 runs); with the fix it passes (3/3, about 6 s). macOS loopback doesn't lose the data, so there it passes either way. Note that `maxConnectionAge` has a 1 s floor, which is why the call runs 1.5 s. - All 655 `grpc-netty` tests pass. The standalone reproducer from the issue, on Linux (`master` against this branch, 20 iterations each, `parkMs` = client event-loop stall): | Scenario | `master` | this PR | |---|---|---| | No hooks | 5/20 fail | 0/20 | | 100 / 500 / 900 ms client stall | 49/60 fail | 0/60 | | 1500 ms client stall (longer than the drain) | 1/20 | 3/20 | A client stalled for longer than the drain can still lose the race, as in Go. ### Relation to Netty The root cause is in Netty's `Http2ConnectionHandler`, so I've also opened netty/netty#17701 (issue netty/netty#17700). It adds an opt-in `gracefulShutdownDrainMillis`. Once a Netty release includes it, the override here can be replaced by calling `gracefulShutdownDrainMillis(1000)`. This PR fixes grpc-java without waiting for that release. ### Not included `OkHttpServerTransport` closes the same way after the second GOAWAY and is probably affected too. I haven't tested it, and I'm happy to follow up if you want both transports to behave the same. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
gRPC-Java supports Java 8 and later. Android minSdkVersion 24 (Nougat) and later are supported with Java 8 language desugaring.
TLS usage on Android typically requires Play Services Dynamic Security Provider. Please see the Security Readme.
Older Java versions are not directly supported, but a branch remains available for fixes and releases. See gRFC P5 JDK Version Support Policy.
| Java version | gRPC Branch |
|---|---|
| 7 | 1.41.x |
For a guided tour, take a look at the quick start guide or the more explanatory gRPC basics.
The examples and the Android example are standalone projects that showcase the usage of gRPC.
Download the JARs. Or for Maven with non-Android, add to your pom.xml:
<dependency> <groupId>io.grpc</groupId> <artifactId>grpc-netty-shaded</artifactId> <version>1.85.0</version> <scope>runtime</scope> </dependency> <dependency> <groupId>io.grpc</groupId> <artifactId>grpc-protobuf</artifactId> <version>1.85.0</version> </dependency> <dependency> <groupId>io.grpc</groupId> <artifactId>grpc-stub</artifactId> <version>1.85.0</version> </dependency>
Or for Gradle with non-Android, add to your dependencies:
runtimeOnly 'io.grpc:grpc-netty-shaded:1.85.0' implementation 'io.grpc:grpc-protobuf:1.85.0' implementation 'io.grpc:grpc-stub:1.85.0'
For Android client, use grpc-okhttp instead of grpc-netty-shaded and grpc-protobuf-lite instead of grpc-protobuf:
implementation 'io.grpc:grpc-okhttp:1.85.0' implementation 'io.grpc:grpc-protobuf-lite:1.85.0' implementation 'io.grpc:grpc-stub:1.85.0'
For Bazel, you can either use Maven (with the GAVs from above), or use @io_grpc_grpc_java//api et al (see below).
Development snapshots are available in Sonatypes's snapshot repository.
For protobuf-based codegen, you can put your proto files in the src/main/proto and src/test/proto directories along with an appropriate plugin.
For protobuf-based codegen integrated with the Maven build system, you can use protobuf-maven-plugin (Eclipse and NetBeans users should also look at os-maven-plugin's IDE documentation):
<build> <extensions> <extension> <groupId>kr.motd.maven</groupId> <artifactId>os-maven-plugin</artifactId> <version>1.7.1</version> </extension> </extensions> <plugins> <plugin> <groupId>org.xolstice.maven.plugins</groupId> <artifactId>protobuf-maven-plugin</artifactId> <version>0.6.1</version> <configuration> <protocArtifact>com.google.protobuf:protoc:3.25.9:exe:${os.detected.classifier}</protocArtifact> <pluginId>grpc-java</pluginId> <pluginArtifact>io.grpc:protoc-gen-grpc-java:1.85.0:exe:${os.detected.classifier}</pluginArtifact> </configuration> <executions> <execution> <goals> <goal>compile</goal> <goal>compile-custom</goal> </goals> </execution> </executions> </plugin> </plugins> </build>
For non-Android protobuf-based codegen integrated with the Gradle build system, you can use protobuf-gradle-plugin:
plugins { id 'com.google.protobuf' version '0.9.5' } protobuf { protoc { artifact = "com.google.protobuf:protoc:3.25.9" } plugins { grpc { artifact = 'io.grpc:protoc-gen-grpc-java:1.85.0' } } generateProtoTasks { all()*.plugins { grpc {} } } }
The prebuilt protoc-gen-grpc-java binary uses glibc on Linux. If you are compiling on Alpine Linux, you may want to use the Alpine grpc-java package which uses musl instead.
For Android protobuf-based codegen integrated with the Gradle build system, also use protobuf-gradle-plugin but specify the ‘lite’ options:
plugins { id 'com.google.protobuf' version '0.9.5' } protobuf { protoc { artifact = "com.google.protobuf:protoc:3.25.9" } plugins { grpc { artifact = 'io.grpc:protoc-gen-grpc-java:1.85.0' } } generateProtoTasks { all().each { task -> task.builtins { java { option 'lite' } } task.plugins { grpc { option 'lite' } } } } }
For Bazel, use the proto_library and the java_proto_library (no load() required) and load("@io_grpc_grpc_java//:java_grpc_library.bzl", "java_grpc_library") (from this project), as in this example BUILD.bazel.
APIs annotated with @Internal are for internal use by the gRPC library and should not be used by gRPC users. APIs annotated with @ExperimentalApi are subject to change in future releases, and library code that other projects may depend on should not use these APIs.
We recommend using the grpc-java-api-checker (an Error Prone plugin) to check for usages of @ExperimentalApi and @Internal in any library code that depends on gRPC. It may also be used to check for @Internal usage or unintended @ExperimentalApi consumption in non-library code.
If you are making changes to gRPC-Java, see the compiling instructions.
At a high level there are three distinct layers to the library: Stub, Channel, and Transport.
The Stub layer is what is exposed to most developers and provides type-safe bindings to whatever datamodel/IDL/interface you are adapting. gRPC comes with a plugin to the protocol-buffers compiler that generates Stub interfaces out of .proto files, but bindings to other datamodel/IDL are easy and encouraged.
The Channel layer is an abstraction over Transport handling that is suitable for interception/decoration and exposes more behavior to the application than the Stub layer. It is intended to be easy for application frameworks to use this layer to address cross-cutting concerns such as logging, monitoring, auth, etc.
The Transport layer does the heavy lifting of putting and taking bytes off the wire. The interfaces to it are abstract just enough to allow plugging in of different implementations. Note the transport layer API is considered internal to gRPC and has weaker API guarantees than the core API under package io.grpc.
gRPC comes with multiple Transport implementations:
