Kernel TLS (KTLS)

KTLS moves TLS record encryption out of NGINX and into the Linux kernel. Once a session's keys are handed to the kernel, static files are served with SSL_sendfile(): file pages go from the page cache straight to encrypted packets, without ever being copied into user space.

Every GetPageSpeed NGINX build — the stock nginx package, nginx-mod and edge — links against openssl35, which we compile with enable-ktls. There is nothing to install or rebuild: KTLS capability is already in the binary you are running, and it stays completely inactive until you switch it on per server block.

Should you enable it?

KTLS is opt-in for a reason. Our measurements (NGINX 1.30.4, OpenSSL 3.5, TLS 1.3, AES-GCM, static file workload):

EL9 stock kernel 5.14, x86_64 — roughly CPU-neutral. The kernel's TLS path and OpenSSL's AES-NI path cost about the same; there is little to gain.
Kernel 6.8+ — a large win: about a third less NGINX worker CPU and substantially higher throughput on the same hardware, thanks to SSL_sendfile() and a faster in-kernel crypto path.

The benefit is concentrated in TLS-heavy static file serving (downloads, package repositories, media). Proxied responses do not use SSL_sendfile() and gain much less. KTLS applies to TCP only — HTTP/3 (QUIC) is unaffected.

We run this setup in production: extras.getpagespeed.com, the repository these packages come from, has served its HTTPS (TCP) downloads over KTLS since September 2026 (NGINX on kernel-lt 6.18, about 6 million KTLS sessions in the first month). Kernel transmit errors stayed at zero and served files matched their on-disk checksums.

Do not enable on Rocky/RHEL 9.7 kernel 5.14.0-611.24.1.el9_7

This specific EL 9.7 kernel corrupts data sent through KTLS sendfile(): clients receive files whose checksums do not match. We found this during release validation (our OpenSSL test suite carries a skip for exactly this kernel). Since stock EL9 gains ~nothing from KTLS anyway, the safe rule is simple: only enable KTLS when running a modern kernel, e.g. kernel-lt / kernel-ml 6.x. Verify integrity after enabling (below) — a checksum comparison over HTTPS takes a minute.

Enabling KTLS

Requirements: the tls kernel module (present in EL distro kernels and our kernel-lt/kernel-ml builds; it auto-loads on first use) and TLS 1.2/1.3 with an AES-GCM or ChaCha20-Poly1305 cipher.

In the server block where you want it:

server {
    listen 443 ssl;
    ...
    sendfile on;
    ssl_conf_command Options KTLS;
}

sendfile on; is usually already inherited from the http block; it is what unlocks the zero-copy SSL_sendfile() path.

ssl_conf_command inheritance

ssl_conf_command directives are inherited from the http level only when the server block defines none of its own. If you already use ssl_conf_command at the http level, repeat those commands in any server block where you add Options KTLS.

Verifying

The kernel counts KTLS sessions and errors:

$ cat /proc/net/tls_stat
TlsCurrTxSw    3
TlsTxSw        412
TlsTxSwError   0
TlsDecryptError 0
...

TlsTxSw climbing while you make HTTPS requests confirms KTLS transmit is active. With OpenSSL 3.5 the receive side is offloaded too, so TlsRxSw climbs alongside it.

TlsTxSwError must stay at 0. TlsDecryptError counts records from clients that failed authentication on receive: on an Internet-facing server, scanners and broken clients sending garbage make it creep up slowly (on our own package repository: under 300 across 6 million sessions in a month, nearly all from a single scanner IP). A steady trickle is normal. A jump that follows legitimate traffic is not.

New [alert] lines in the error log

With KTLS receive active, a client that sends a TLS alert or a non-data record while NGINX reads the request (typically a scanner) shows up as recv() failed (5: Input/output error) at [alert] level. Before KTLS, the same disconnects either logged nothing or logged at info. They are harmless: the connection was ending anyway. On our repository server none of them came from a package-manager download.

Then confirm integrity end-to-end — this is the check that catches a bad kernel:

$ curl -so /tmp/f https://your.server/some-large-file
$ sha256sum /tmp/f   # compare against the file on the server

If checksums mismatch, disable KTLS and update the kernel before re-trying.

Gotcha: `tcp_notsent_lowat` throttles HTTP/1.1 downloads

Symptom: with KTLS on, HTTP/1.1 downloads (every yum client on EL7, for example) crawl at about 100 kB/s, even over loopback, while HTTP/2 transfers of the same file run at full speed.

Cause: net.ipv4.tcp_notsent_lowat=16384, a widely copied sysctl from HTTP/2 prioritization tuning guides. A KTLS record is 16384 bytes of payload plus 22 bytes of overhead, so one queued record keeps the socket at or above the low-water mark. After every record SSL_sendfile() gets EAGAIN and waits 40-200 ms for the socket to become writable. It is not a kernel or OpenSSL bug, and tcp_nopush makes no difference.

Loopback A/B on a 6 MB file over HTTP/1.1:

`tcp_notsent_lowat`	Throughput
16384	104 kB/s
131072	34.7 MB/s
kernel default (4294967295)	37.9 MB/s
KTLS off	35 MB/s

Check:

$ sysctl net.ipv4.tcp_notsent_lowat

Fix: raise it to at least 131072, or drop the override so the kernel default applies, and persist it:

$ grep -r tcp_notsent_lowat /etc/sysctl.conf /etc/sysctl.d/ /usr/lib/sysctl.d/
$ echo 'net.ipv4.tcp_notsent_lowat = 131072' | sudo tee /etc/sysctl.d/90-ktls-notsent-lowat.conf
$ sudo sysctl --system

If another file sets the old value, edit or remove that line: the file that sorts last wins. The change applies immediately, no NGINX restart needed.

Verify: an HTTP/1.1 download should now be in the same range as HTTP/2:

$ curl --http1.1 -so /dev/null -w '%{speed_download}\n' https://your.server/some-large-file

Kernel TLS (KTLS)

Should you enable it?

Enabling KTLS

Verifying

Gotcha: tcp_notsent_lowat throttles HTTP/1.1 downloads

Gotcha: `tcp_notsent_lowat` throttles HTTP/1.1 downloads