Rust-Quiche: quiche is an implementation of the QUIC transport protocol and HTTP/3 as specified by the IETF.

quiche

crates.io docs.rs license build

quiche is an implementation of the QUIC transport protocol and HTTP/3 as specified by the IETF. It provides a low level API for processing QUIC packets and handling connection state. The application is responsible for providing I/O (e.g. sockets handling) as well as an event loop with support for timers.

A live QUIC server based on quiche is available at https://quic.tech:4433/, and an HTTP/3 one at https://quic.tech:8443/, that can be used for experimentation.

For more information on how quiche came about and some insights into its design you can read a post on Cloudflare's (where this library is used in production) blog that goes into some more detail.

Getting Started

Command-line apps

Before diving into the quiche API, here are a few examples on how to use the quiche tools provided as part of the quiche-apps crate.

The client can be run as follows:

 $ cargo run --manifest-path=tools/apps/Cargo.toml --bin quiche-client -- https://quic.tech:8443/

while the server can be run as follows:

 $ cargo run --manifest-path=tools/apps/Cargo.toml --bin quiche-server -- \
      --cert tools/apps/src/bin/cert.crt \
      --key tools/apps/src/bin/cert.key

(note that the certificate provided is self-signed and should not be used in production)

Use the --help command-line flag to get a more detailed description of each tool's options.

Connection setup

The first step in establishing a QUIC connection using quiche is creating a configuration object:

let config = quiche::Config::new(quiche::PROTOCOL_VERSION)?;

This is shared among multiple connections and can be used to configure a QUIC endpoint.

On the client-side the connect() utility function can be used to create a new connection, while accept() is for servers:

// Client connection.
let conn = quiche::connect(Some(&server_name), &scid, &mut config)?;

// Server connection.
let conn = quiche::accept(&scid, None, &mut config)?;

Handling incoming packets

Using the connection's recv() method the application can process incoming packets that belong to that connection from the network:

loop {
    let read = socket.recv(&mut buf).unwrap();

    let read = match conn.recv(&mut buf[..read]) {
        Ok(v) => v,

        Err(e) => {
            // An error occurred, handle it.
            break;
        },
    };
}

Generating outgoing packets

Outgoing packet are generated using the connection's send() method instead:

loop {
    let write = match conn.send(&mut out) {
        Ok(v) => v,

        Err(quiche::Error::Done) => {
            // Done writing.
            break;
        },

        Err(e) => {
            // An error occurred, handle it.
            break;
        },
    };

    socket.send(&out[..write]).unwrap();
}

When packets are sent, the application is responsible for maintaining a timer to react to time-based connection events. The timer expiration can be obtained using the connection's timeout() method.

let timeout = conn.timeout();

The application is responsible for providing a timer implementation, which can be specific to the operating system or networking framework used. When a timer expires, the connection's on_timeout() method should be called, after which additional packets might need to be sent on the network:

// Timeout expired, handle it.
conn.on_timeout();

// Send more packets as needed after timeout.
loop {
    let write = match conn.send(&mut out) {
        Ok(v) => v,

        Err(quiche::Error::Done) => {
            // Done writing.
            break;
        },

        Err(e) => {
            // An error occurred, handle it.
            break;
        },
    };

    socket.send(&out[..write]).unwrap();
}

Sending and receiving stream data

After some back and forth, the connection will complete its handshake and will be ready for sending or receiving application data.

Data can be sent on a stream by using the stream_send() method:

if conn.is_established() {
    // Handshake completed, send some data on stream 0.
    conn.stream_send(0, b"hello", true)?;
}

The application can check whether there are any readable streams by using the connection's readable() method, which returns an iterator over all the streams that have outstanding data to read.

The stream_recv() method can then be used to retrieve the application data from the readable stream:

if conn.is_established() {
    // Iterate over readable streams.
    for stream_id in conn.readable() {
        // Stream is readable, read until there's no more data.
        while let Ok((read, fin)) = conn.stream_recv(stream_id, &mut buf) {
            println!("Got {} bytes on stream {}", read, stream_id);
        }
    }
}

HTTP/3

The quiche HTTP/3 module provides a high level API for sending and receiving HTTP requests and responses on top of the QUIC transport protocol.

Have a look at the examples/ directory for more complete examples on how to use the quiche API, including examples on how to use quiche in C/C++ applications (see below for more information).

Calling quiche from C/C++

quiche exposes a thin C API on top of the Rust API that can be used to more easily integrate quiche into C/C++ applications (as well as in other languages that allow calling C APIs via some form of FFI). The C API follows the same design of the Rust one, modulo the constraints imposed by the C language itself.

When running cargo build, a static library called libquiche.a will be built automatically alongside the Rust one. This is fully stand-alone and can be linked directly into C/C++ applications.

Building

quiche requires Rust 1.39 or later to build. The latest stable Rust release can be installed using rustup.

Once the Rust build environment is setup, the quiche source code can be fetched using git:

 $ git clone --recursive https://github.com/cloudflare/quiche

and then built using cargo:

 $ cargo build --examples

cargo can also be used to run the testsuite:

 $ cargo test

Note that BoringSSL, which is used to implement QUIC's cryptographic handshake based on TLS, needs to be built and linked to quiche. This is done automatically when building quiche using cargo, but requires the cmake, go and perl commands to be available during the build process. On Windows you also need NASM. The official BoringSSL documentation has more details.

In alternative you can use your own custom build of BoringSSL by configuring the BoringSSL directory with the QUICHE_BSSL_PATH environment variable:

 $ QUICHE_BSSL_PATH="/path/to/boringssl" cargo build --examples

Building for Android

To build quiche for Android, you need the following:

  • Install the Android NDK (13b or higher), using Android Studio or directly.
  • Set ANDROID_NDK_HOME environment variable to NDK path, e.g.
 $ export ANDROID_NDK_HOME=/usr/local/share/android-ndk
  • Install the Rust toolchain for Android architectures needed:
 $ rustup target add aarch64-linux-android arm-linux-androideabi armv7-linux-androideabi i686-linux-android x86_64-linux-android

Note that the minimum API level is 21 for all target architectures.

Depending on the NDK version used, you can take one of the following procedures:

NDK version >= 19

For NDK version 19 or higher (21 recommended), you can build in a simpler way using cargo-ndk. You need to install cargo-ndk first.

 $ cargo install cargo-ndk

You can build the quiche library using the following procedure. Note that --target and --android-platform are mandatory.

 $ cargo ndk --target aarch64-linux-android --android-platform 21 -- build

See build_android_ndk19.sh for more information.

Note that building with NDK version 18 appears to be broken.

NDK version < 18

If you need to use NDK version < 18 (gcc), you can build quiche in the following way.

To prepare the cross-compiling toolchain, run the following command:

 $ tools/setup_android.sh

It will create a standalone toolchain for arm64/arm/x86 architectures under the $TOOLCHAIN_DIR/arch directory. If you didn't set TOOLCHAIN_DIR environment variable, the current directory will be used.

After it run successfully, run the following script to build libquiche:

 $ tools/build_android.sh --features ndk-old-gcc

It will build binaries for aarch64, armv7 and i686. You can pass parameters to this script for cargo build. For example if you want to build a release binary with verbose logs, do the following:

 $ tools/build_android.sh --features ndk-old-gcc --release -vv

Building for iOS

To build quiche for iOS, you need the following:

  • Install Xcode command-line tools. You can install them with Xcode or with the following command:
 $ xcode-select --install
  • Install the Rust toolchain for iOS architectures:
 $ rustup target add aarch64-apple-ios x86_64-apple-ios
  • Install cargo-lipo:
 $ cargo install cargo-lipo

To build libquiche, run the following command:

 $ cargo lipo

or

 $ cargo lipo --release

iOS build is tested in Xcode 10.1 and Xcode 11.2.

Building Docker images

In order to build the Docker images, simply run the following command:

 $ make docker-build

You can find the quiche Docker images on the following Docker Hub repositories:

The latest tag will be updated whenever quiche master branch updates.

cloudflare/quiche

Provides a server and client installed in /usr/local/bin.

cloudflare/quiche-qns

Provides the script to test quiche within the quic-interop-runner.

Copyright

Copyright (C) 2018-2019, Cloudflare, Inc.

See COPYING for the license.

Comments

  • cannot use quiche and openssl together
    cannot use quiche and openssl together

    Jun 11, 2020

    It seems that when I use both quiche and openssl in the same Rust package, the final build fails to link. Is this known issue? Could this be related to the fact that quiche uses BoringSSL?

    When I use the same openssl related code without quiche, the build was successful.

    The error I got is something like this:

     = note: /usr/bin/ld: /home/myproject/target/debug/deps/libopenssl-b9de86a917260a58.rlib(openssl-b9de86a917260a58.openssl.e9esesv9-cgu.4.rcgu.o): in function `openssl::symm::Crypter::finalize':
              /home/myproject/.cargo/registry/src/github.com-1ecc6299db9ec823/openssl-0.10.29/src/symm.rs:587: undefined reference to `EVP_CipherFinal'
    

    The crates I used:

    quiche = "0.4.0"
    openssl = "0.10"
    
    Reply
  • try to retransmit old frames on PTO
    try to retransmit old frames on PTO

    Jun 15, 2020

    Currently when sending a PTO probe we either include new data frames or a PING frame in the packet to make sure that the probe is ack-eliciting. However if no new data is availabe, we should try to include old data that has not been marked as lost yet if there is any, instead of just sending a PING.

    This changes that by iterating over old sent packets (that have not been acked or lost yet) and copying CRYPTO and STREAM frames into the outgoing packet when creating PTO probes.

    This requires cloning the frames, which is somewhat unfortunate, but it should only happen when PTO expires, which has exponential backoff so it shouldn't be possible to use this as a DoS vector.

    Reply
  • Quiche drops HANDSHAKE_DONE as invalid packet and doesn’t send an ACK
    Quiche drops HANDSHAKE_DONE as invalid packet and doesn’t send an ACK

    Jun 15, 2020

    A HANDSHAKE_DONE frame sent by packetdrill as soon as the handshake completes. As it is an ACK-eliciting packet, the library should acknowledge. During our adoption of quiche to run packetdrill tests, we noticed that quiche drops the HANDSHAKE_DONE as invalid packet and doesn’t send an ACK.

    Reply
  • Quiche drops the packet with the invalid stream id instead of sending CONNECTION_CLOSE
    Quiche drops the packet with the invalid stream id instead of sending CONNECTION_CLOSE

    Jun 15, 2020

    When we inject a stream frame with a stream ID larger than the max streams limit advertised by the QUIC library. The library must close the connection, with STREAM_LIMIT_ERROR.

    Example (below stream is not permitted by the peer) STREAM[id=1157, off=0, len=100]

    Currently, quiche drops the packet with the invalid stream id and keeps sending ping PTO probes.

    Reply
  • Quiche drops the packet with OOB stream id instead of sending CONNECTION_CLOSE
    Quiche drops the packet with OOB stream id instead of sending CONNECTION_CLOSE

    Jun 15, 2020

    When we inject a MAX_STREAMS frame with a value greater than 2^60 . The QUIC library must close the connection immediately with a connection error STREAM_LIMIT_ERROR.

    Example, MAX_STREAMS_BIDI[id=1152921504606846977]

    Similar test for injecting OOB STREAMS_BLOCKED has same result, i.e. the packet is dropped without connection close.

    Reply
  • C version examples [server] and [client] connection failed
    C version examples [server] and [client] connection failed

    Jun 16, 2020

    I am trying c version examples in quiche/examples and I have some trouble using them. Use master branch, build these examples by make in quiche/examples Just changed one line in client.c:

    -    quiche_config *config = quiche_config_new(0xbabababa);
    +    quiche_config *config = quiche_config_new(QUICHE_PROTOCOL_VERSION);
    

    Here are logs: server

    ~/.../quiche/examples >>> ./server 127.0.0.1 1234                                           
    stateless retry
    sent 93 bytes
    recv would block
    new connection
    quiche: bed00effe987dca943b3a186386830a7 rx pkt Initial version=ff00001d dcid=c4c07e8f8588b0f65febb00f9c62d991 scid=57fb5bcfc3946bf7cd5800bb49a75417 token=7175696368650200ed7f7f0000010000000000000000c4c07e8f8588b0f65febb00f9c62d991 len=1120 pn=1
    quiche: bed00effe987dca943b3a186386830a7 rx frm CRYPTO off=0 len=512
    quiche::tls: checking peer ALPN Ok("hq-29") against Ok("hq-29")
    quiche::tls: bed00effe987dca943b3a186386830a7 write message lvl=Initial len=90
    quiche::tls: bed00effe987dca943b3a186386830a7 set write secret lvl=Handshake
    quiche::tls: bed00effe987dca943b3a186386830a7 write message lvl=Handshake len=1178
    quiche::tls: bed00effe987dca943b3a186386830a7 set write secret lvl=OneRTT
    quiche::tls: bed00effe987dca943b3a186386830a7 set read secret lvl=Handshake
    quiche: bed00effe987dca943b3a186386830a7 rx frm PADDING len=587
    recv 1200 bytes
    recv would block
    quiche: bed00effe987dca943b3a186386830a7 tx pkt Initial version=ff00001d dcid=57fb5bcfc3946bf7cd5800bb49a75417 scid=bed00effe987dca943b3a186386830a7 len=116 pn=0
    quiche: bed00effe987dca943b3a186386830a7 tx frm ACK delay=1118 blocks=[1..1]
    quiche: bed00effe987dca943b3a186386830a7 tx frm CRYPTO off=0 len=90
    quiche::recovery: bed00effe987dca943b3a186386830a7 timer=999.850593ms latest_rtt=0ns srtt=None min_rtt=0ns rttvar=0ns loss_time=[None, None, None] loss_probes=[0, 0, 0] cwnd=14520 ssthresh=18446744073709551615 bytes_in_flight=159 app_limited=true congestion_recovery_start_time=None delivered=0 delivered_time=162.421µs recent_delivered_packet_sent_time=163.121µs app_limited_at_pkt=0  hystart=window_end=None last_round_min_rtt=None current_round_min_rtt=None rtt_sample_count=0 lss_start_time=None  
    sent 159 bytes
    quiche: bed00effe987dca943b3a186386830a7 tx pkt Handshake version=ff00001d dcid=57fb5bcfc3946bf7cd5800bb49a75417 scid=bed00effe987dca943b3a186386830a7 len=1154 pn=0
    quiche: bed00effe987dca943b3a186386830a7 tx frm CRYPTO off=0 len=1134
    quiche::recovery: bed00effe987dca943b3a186386830a7 timer=999.711623ms latest_rtt=0ns srtt=None min_rtt=0ns rttvar=0ns loss_time=[None, None, None] loss_probes=[0, 0, 0] cwnd=14520 ssthresh=18446744073709551615 bytes_in_flight=1355 app_limited=true congestion_recovery_start_time=None delivered=0 delivered_time=296.695µs recent_delivered_packet_sent_time=297.22µs app_limited_at_pkt=0  hystart=window_end=None last_round_min_rtt=None current_round_min_rtt=None rtt_sample_count=0 lss_start_time=None  
    sent 1196 bytes
    quiche: bed00effe987dca943b3a186386830a7 tx pkt Handshake version=ff00001d dcid=57fb5bcfc3946bf7cd5800bb49a75417 scid=bed00effe987dca943b3a186386830a7 len=64 pn=1
    quiche: bed00effe987dca943b3a186386830a7 tx frm CRYPTO off=1134 len=44
    quiche::recovery: bed00effe987dca943b3a186386830a7 timer=999.600422ms latest_rtt=0ns srtt=None min_rtt=0ns rttvar=0ns loss_time=[None, None, None] loss_probes=[0, 0, 0] cwnd=14520 ssthresh=18446744073709551615 bytes_in_flight=1461 app_limited=true congestion_recovery_start_time=None delivered=0 delivered_time=410.724µs recent_delivered_packet_sent_time=411.244µs app_limited_at_pkt=0  hystart=window_end=None last_round_min_rtt=None current_round_min_rtt=None rtt_sample_count=0 lss_start_time=None  
    sent 106 bytes
    done writing
    quiche: bed00effe987dca943b3a186386830a7 rx pkt Handshake version=ff00001d dcid=bed00effe987dca943b3a186386830a7 scid=57fb5bcfc3946bf7cd5800bb49a75417 len=21 pn=0
    quiche: bed00effe987dca943b3a186386830a7 rx frm CONNECTION_CLOSE err=8 frame=0 reason=[]
    quiche: bed00effe987dca943b3a186386830a7 dropped epoch 0 state
    recv 61 bytes
    recv would block
    done writing
    quiche: bed00effe987dca943b3a186386830a7 draining timeout expired
    timeout
    done writing
    connection closed, recv=2 sent=3 lost=0 rtt=500000000ns cwnd=14520
    

    client

    ~/.../quiche/examples >>> ./client 127.0.0.1 1234             
    quiche::tls: 57fb5bcfc3946bf7cd5800bb49a75417 write message lvl=Initial len=512
    quiche: 57fb5bcfc3946bf7cd5800bb49a75417 tx pkt Initial version=ff00001d dcid=c4c07e8f8588b0f65febb00f9c62d991 scid=57fb5bcfc3946bf7cd5800bb49a75417 len=1157 pn=0
    quiche: 57fb5bcfc3946bf7cd5800bb49a75417 tx frm CRYPTO off=0 len=512
    quiche: 57fb5bcfc3946bf7cd5800bb49a75417 tx frm PADDING len=625
    quiche::recovery: 57fb5bcfc3946bf7cd5800bb49a75417 timer=999.544142ms latest_rtt=0ns srtt=None min_rtt=0ns rttvar=0ns loss_time=[None, None, None] loss_probes=[0, 0, 0] cwnd=14520 ssthresh=18446744073709551615 bytes_in_flight=1200 app_limited=true congestion_recovery_start_time=None delivered=0 delivered_time=472.271µs recent_delivered_packet_sent_time=472.809µs app_limited_at_pkt=0  hystart=window_end=None last_round_min_rtt=None current_round_min_rtt=None rtt_sample_count=0 lss_start_time=None  
    sent 1200 bytes
    done writing
    quiche: 57fb5bcfc3946bf7cd5800bb49a75417 rx pkt Retry version=ff00001d dcid=57fb5bcfc3946bf7cd5800bb49a75417 scid=c4c07e8f8588b0f65febb00f9c62d991 token=7175696368650200ed7f7f0000010000000000000000c4c07e8f8588b0f65febb00f9c62d991
    recv 93 bytes
    recv would block
    done reading
    quiche::tls: 57fb5bcfc3946bf7cd5800bb49a75417 write message lvl=Initial len=512
    quiche: 57fb5bcfc3946bf7cd5800bb49a75417 tx pkt Initial version=ff00001d dcid=c4c07e8f8588b0f65febb00f9c62d991 scid=57fb5bcfc3946bf7cd5800bb49a75417 token=7175696368650200ed7f7f0000010000000000000000c4c07e8f8588b0f65febb00f9c62d991 len=1119 pn=1
    quiche: 57fb5bcfc3946bf7cd5800bb49a75417 tx frm CRYPTO off=0 len=512
    quiche: 57fb5bcfc3946bf7cd5800bb49a75417 tx frm PADDING len=587
    quiche::recovery: 57fb5bcfc3946bf7cd5800bb49a75417 timer=999.582502ms latest_rtt=0ns srtt=None min_rtt=0ns rttvar=0ns loss_time=[None, None, None] loss_probes=[0, 0, 0] cwnd=14520 ssthresh=18446744073709551615 bytes_in_flight=1200 app_limited=true congestion_recovery_start_time=None delivered=0 delivered_time=1.48449ms recent_delivered_packet_sent_time=1.48498ms app_limited_at_pkt=0  hystart=window_end=None last_round_min_rtt=None current_round_min_rtt=None rtt_sample_count=0 lss_start_time=None  
    sent 1200 bytes
    done writing
    quiche: 57fb5bcfc3946bf7cd5800bb49a75417 rx pkt Initial version=ff00001d dcid=57fb5bcfc3946bf7cd5800bb49a75417 scid=bed00effe987dca943b3a186386830a7 token= len=117 pn=0
    quiche: 57fb5bcfc3946bf7cd5800bb49a75417 rx frm ACK delay=1118 blocks=[1..1]
    quiche::recovery: 57fb5bcfc3946bf7cd5800bb49a75417 packet newly acked 1
    quiche::recovery: 57fb5bcfc3946bf7cd5800bb49a75417 timer=none latest_rtt=10.321509ms srtt=Some(10.321509ms) min_rtt=10.321509ms rttvar=5.160754ms loss_time=[None, None, None] loss_probes=[0, 0, 0] cwnd=14520 ssthresh=18446744073709551615 bytes_in_flight=0 app_limited=true congestion_recovery_start_time=None delivered=1200 delivered_time=157.869µs recent_delivered_packet_sent_time=11.539471ms app_limited_at_pkt=0  hystart=window_end=None last_round_min_rtt=None current_round_min_rtt=None rtt_sample_count=0 lss_start_time=None  
    quiche: 57fb5bcfc3946bf7cd5800bb49a75417 rx frm CRYPTO off=0 len=90
    quiche::tls: 57fb5bcfc3946bf7cd5800bb49a75417 set write secret lvl=Handshake
    quiche::tls: 57fb5bcfc3946bf7cd5800bb49a75417 set read secret lvl=Handshake
    recv 159 bytes
    quiche: 57fb5bcfc3946bf7cd5800bb49a75417 rx pkt Handshake version=ff00001d dcid=57fb5bcfc3946bf7cd5800bb49a75417 scid=bed00effe987dca943b3a186386830a7 len=1155 pn=0
    quiche: 57fb5bcfc3946bf7cd5800bb49a75417 rx frm CRYPTO off=0 len=1134
    failed to process packet
    recv 106 bytes
    recv would block
    done reading
    quiche: 57fb5bcfc3946bf7cd5800bb49a75417 tx pkt Handshake version=ff00001d dcid=bed00effe987dca943b3a186386830a7 scid=57fb5bcfc3946bf7cd5800bb49a75417 len=20 pn=0
    quiche: 57fb5bcfc3946bf7cd5800bb49a75417 tx frm CONNECTION_CLOSE err=8 frame=0 reason=[]
    quiche::recovery: 57fb5bcfc3946bf7cd5800bb49a75417 timer=44.094416ms latest_rtt=10.321509ms srtt=Some(10.321509ms) min_rtt=10.321509ms rttvar=5.160754ms loss_time=[None, None, None] loss_probes=[0, 0, 0] cwnd=14520 ssthresh=18446744073709551615 bytes_in_flight=61 app_limited=true congestion_recovery_start_time=None delivered=1200 delivered_time=1.560008ms recent_delivered_packet_sent_time=12.941538ms app_limited_at_pkt=0  hystart=window_end=None last_round_min_rtt=None current_round_min_rtt=None rtt_sample_count=0 lss_start_time=None  
    quiche: 57fb5bcfc3946bf7cd5800bb49a75417 dropped epoch 0 state
    sent 61 bytes
    done writing
    timeout
    done writing
    quiche: 57fb5bcfc3946bf7cd5800bb49a75417 draining timeout expired
    timeout
    done writing
    connection closed, recv=1 sent=3 lost=0 rtt=10321509ns
    

    Thanks for your help!

    Reply
  • Why is the Connection boxed by default?
    Why is the Connection boxed by default?

    May 5, 2019

    Hey,

    I was just skimming over the API and I was wondering why connect and accept return a boxed Connection? Connection is neither a trait object nor dynamically sized, so why the need for the box?

    Reply
  • quiche_conn_recv() == -10 on windows
    quiche_conn_recv() == -10 on windows

    Apr 19, 2020

    I run client.c and server.c on windows but I got this error: failed to process packet: -10 returned from the quiche_conn_recv() function

    I set port:5555 and host="localhost"

    I can't understand why this error occurred? Can I get more debug info or logging somehow to better understand the details of this failure?

    Capture20 Capture21

    Reply
  • tools: initial fuzzers
    tools: initial fuzzers

    Sep 22, 2019

    This adds a couple of fuzzers based on honggfuzz that can be used to fuzz incoming packet processing from both the server and client side.

    Reply
  • Unexpected http0.9 protocol when using cURL
    Unexpected http0.9 protocol when using cURL

    Nov 6, 2019

    :wave: Hey There (Thanks for the great library!),

    I seem to be running into something unexpected while testing quiche (through the C FFI), and using a locally built cURL. When using cURL I must specify the: --http0.9 option, but only when using my version (not the rust example).

    In my C++ Code I run:

     // _config has been initialized with: quiche_config_new(QUICHE_PROTOCOL_VERSION);
    quiche_config_set_application_protos(
          _config.get(),
          const_cast<uint8_t *>(
              reinterpret_cast<const uint8_t *>(QUICHE_H3_APPLICATION_PROTOCOL)),
          sizeof(QUICHE_H3_APPLICATION_PROTOCOL) - 1);
    

    The definition of: QUICHE_H3_APPLICATION_PROTOCOL comes from quiche.h, and seems to be properly set to: #define QUICHE_H3_APPLICATION_PROTOCOL "\x05h3-23", which is what I expect.

    When I try to use cURL though I get:

    *   Trying ::1:8080...
    * Sent QUIC client Initial, ALPN: h3-23
    * quiche: recv() unexpectedly returned -1 (errno: 111, socket 3)
    * connect to ::1 port 8080 failed: Connection refused
    *   Trying 127.0.0.1:8080...
    * Sent QUIC client Initial, ALPN: h3-23
    * h3 [:method: GET]
    * h3 [:path: /]
    * h3 [:scheme: https]
    * h3 [:authority: localhost:8080]
    * h3 [user-agent: curl/7.67.0-DEV]
    * h3 [accept: */*]
    * Using HTTP/3 Stream ID: 0 (easy handle 0x55e202f95280)
    > GET / HTTP/3
    > Host: localhost:8080
    > user-agent: curl/7.67.0-DEV
    > accept: */*
    > 
    * Received HTTP/0.9 when not allowed
    
    * Connection #0 to host localhost left intact
    curl: (1) quiche: recv() unexpectedly returned -1 (errno: 111, socket 3)
    

    This error goes away if I provide the: --http0.9 flag to cURL, and the request completes successfully.

    When running the example http-server written in rust: ./target/debug/examples/http3-server --listen 0.0.0.0:8080 with cURL I am able to successfully query without providing the http0.9 flag:

    ./src/curl -k --tlsv1.3 --http3 https://localhost:8080/ && echo ""
    Not Found!
    

    (It should be noted while running the C example with: ./examples/http3-server 0.0.0.0 8080, I get a straight up failure).

    I'm curious if this is the only place I call: quiche_config_set_application_protos, and it doesn't (to the best of my knowledge) include http0.9, how is curl seeing this 0.9 data? Is there a config option I need to set when using the C API?

    Reply
  • Nginx configures multiple sites to share 443/udp ports
    Nginx configures multiple sites to share 443/udp ports

    Oct 25, 2019

    [email protected] conf 18:30:35 # pwd
    /usr/local/nginx/conf
    [email protected] conf # grep -n quic nginx.conf
    177:        listen      443 quic    reuseport;
    248:        listen      443 quic    reuseport;
    [email protected] conf # nginx -t               
    nginx: [emerg] duplicate listen options for 0.0.0.0:443 in /usr/local/nginx/conf/nginx.conf:248
    nginx: configuration file /usr/local/nginx/conf/nginx.conf test failed
    

    Is this a bug?

    Reply
  • Nginx http3 closes connection with big html page
    Nginx http3 closes connection with big html page

    Nov 26, 2019

    I got a problem while testing an nginx server patched with Quiche implementation of HTTP/3 with curl: when I try to send multiple consecutive request for a small html page (~1kb), nginx responds correctly

        [email protected]:~# ./curl/src/curl https://192.168.19.128?[1-5] -Ik --http3
    
    [1/5]: https://192.168.19.128?1 --> <stdout>
    --_curl_--https://192.168.19.128?1
    HTTP/3 200
    server: nginx/1.16.1
    date: Mon, 25 Nov 2019 13:44:21 GMT
    content-type: text/html
    content-length: 924
    last-modified: Mon, 25 Nov 2019 12:07:59 GMT
    etag: "5ddbc41f-39c"
    alt-svc: h3-23=":443"; ma=86400
    accept-ranges: bytes
    
    
    [2/5]: https://192.168.19.128?2 --> <stdout>
    --_curl_--https://192.168.19.128?2
    HTTP/3 200
    server: nginx/1.16.1
    date: Mon, 25 Nov 2019 13:44:21 GMT
    content-type: text/html
    content-length: 924
    last-modified: Mon, 25 Nov 2019 12:07:59 GMT
    etag: "5ddbc41f-39c"
    alt-svc: h3-23=":443"; ma=86400
    accept-ranges: bytes
    
    
    [3/5]: https://192.168.19.128?3 --> <stdout>
    --_curl_--https://192.168.19.128?3
    HTTP/3 200
    server: nginx/1.16.1
    date: Mon, 25 Nov 2019 13:44:21 GMT
    content-type: text/html
    content-length: 924
    last-modified: Mon, 25 Nov 2019 12:07:59 GMT
    etag: "5ddbc41f-39c"
    alt-svc: h3-23=":443"; ma=86400
    accept-ranges: bytes
    
    
    [4/5]: https://192.168.19.128?4 --> <stdout>
    --_curl_--https://192.168.19.128?4
    HTTP/3 200
    server: nginx/1.16.1
    date: Mon, 25 Nov 2019 13:44:21 GMT
    content-type: text/html
    content-length: 924
    last-modified: Mon, 25 Nov 2019 12:07:59 GMT
    etag: "5ddbc41f-39c"
    alt-svc: h3-23=":443"; ma=86400
    accept-ranges: bytes
    
    
    [5/5]: https://192.168.19.128?5 --> <stdout>
    --_curl_--https://192.168.19.128?5
    HTTP/3 200
    server: nginx/1.16.1
    date: Mon, 25 Nov 2019 13:44:21 GMT
    content-type: text/html
    content-length: 924
    last-modified: Mon, 25 Nov 2019 12:07:59 GMT
    etag: "5ddbc41f-39c"
    alt-svc: h3-23=":443"; ma=86400
    accept-ranges: bytes
    

    If I try to make a single request to a medium/big html file, nginx respond correctly again, but when I try to make multiple consecutive request to a medium/big html page (>=30kb), nginx stop responding after an arbitrary number of requests (2-5 requests normally). Here's an example made of 10 requests to the https://cloudflare-quic.com html page (which I downloaded on my server):

       [email protected]:~# ./curl/src/curl -Ik https://192.168.19.128/cloudflare.html?[1-10] --http3 -v
    
    [1/10]: https://192.168.19.128/cloudflare.html?1 --> <stdout>
    --_curl_--https://192.168.19.128/cloudflare.html?1
    *   Trying 192.168.19.128:443...
    * Sent QUIC client Initial, ALPN: h3-23
    * h3 [:method: HEAD]
    * h3 [:path: /cloudflare.html?1]
    * h3 [:scheme: https]
    * h3 [:authority: 192.168.19.128]
    * h3 [user-agent: curl/7.67.0-DEV]
    * h3 [accept: */*]
    * Using HTTP/3 Stream ID: 0 (easy handle 0x5614ee569460)
    > HEAD /cloudflare.html?1 HTTP/3
    > Host: 192.168.19.128
    > user-agent: curl/7.67.0-DEV
    > accept: */*
    >
    < HTTP/3 200
    HTTP/3 200
    < server: nginx/1.16.1
    server: nginx/1.16.1
    < date: Mon, 25 Nov 2019 13:53:43 GMT
    date: Mon, 25 Nov 2019 13:53:43 GMT
    < content-type: text/html
    content-type: text/html
    < content-length: 106072
    content-length: 106072
    < vary: Accept-Encoding
    vary: Accept-Encoding
    < etag: "5ddbdc21-19e58"
    etag: "5ddbdc21-19e58"
    < alt-svc: h3-23=":443"; ma=86400
    alt-svc: h3-23=":443"; ma=86400
    < accept-ranges: bytes
    accept-ranges: bytes
    
    <
    * Excess found: excess = 27523 url = /cloudflare.html (zero-length body)
    * Connection #0 to host 192.168.19.128 left intact
    
    [2/10]: https://192.168.19.128/cloudflare.html?2 --> <stdout>
    --_curl_--https://192.168.19.128/cloudflare.html?2
    * Found bundle for host 192.168.19.128: 0x5614ee56db00 [can multiplex]
    * Re-using existing connection! (#0) with host 192.168.19.128
    * Connected to 192.168.19.128 (192.168.19.128) port 443 (#0)
    * h3 [:method: HEAD]
    * h3 [:path: /cloudflare.html?2]
    * h3 [:scheme: https]
    * h3 [:authority: 192.168.19.128]
    * h3 [user-agent: curl/7.67.0-DEV]
    * h3 [accept: */*]
    * Using HTTP/3 Stream ID: 4 (easy handle 0x5614ee56b2b0)
    > HEAD /cloudflare.html?2 HTTP/3
    > Host: 192.168.19.128
    > user-agent: curl/7.67.0-DEV
    > accept: */*
    >
    * Got h3 for stream 0, expects 4
    * Got h3 for stream 0, expects 4
    * Got h3 for stream 0, expects 4
    * Got h3 for stream 0, expects 4
    [...]
    

    access.log of this requests:

    192.168.19.129 - - [26/Nov/2019:12:41:56 +0100] "HEAD /cloudflare.html?1 HTTP/3" 200 106072 "-" "curl/7.67.0-DEV" "-" 3d6f3588d11d45d4cffa3c9ee8e3dcbe
    192.168.19.129 - - [26/Nov/2019:12:41:56 +0100] "HEAD /cloudflare.html?2 HTTP/3" 200 106072 "-" "curl/7.67.0-DEV" "-" d4b886015b99af3c53618895959f90f2
    

    error.log is empty.

    It stucks on this screen repeating "Got h3 for stream 0, expects 4". Also I noticed that, when testing on smaller pages, that the smallest the file the bigger is the number of requests fullfilled before stop responding and start printing the error "Got h3 for stream x, expecting y", whith the relation that y=x+4. Also the access.log and the error.log are clean, meaning that it could maybe be some king of parameter missing in the server configuration, but I'm not sure about it. Does anyone have an idea of what the problem could be?

    My config

    nginx version:

    nginx version: nginx/1.16.1
    built by gcc 9.2.1 20191008 (Ubuntu 9.2.1-9ubuntu2)
    built with OpenSSL 1.1.0 (compatible; BoringSSL) (running with BoringSSL)
    TLS SNI support enabled
    configure arguments: 
    --prefix=/root/nginx-1.16.1 
    --with-http_ssl_module 
    --with-http_v2_module 
    --with-http_v3_module 
    --with-openssl=../quiche/deps/boringssl 
    --with-quiche=../quiche
    

    nginx.conf:

    user root;
    # you must set worker processes based on your CPU cores, nginx does not benefit from setting more than that
    worker_processes auto; #some last versions calculate it automatically
    
    # number of file descriptors used for nginx
    # the limit for the maximum FDs on the server is usually set by the OS.
    # if you don't set FD's then OS settings will be used which is by default 2000
    worker_rlimit_nofile 100000;
    
    # only log critical errors
    error_log logs/error.log crit;
    error_log  logs/error.log debug;
    error_log  logs/error.log  notice;
    error_log  logs/error.log  info;
    
    # provides the configuration file context in which the directives that affect connection processing are specified.
    events {
        # determines how much clients will be served per worker
        # max clients = worker_connections * worker_processes
        # max clients is also limited by the number of socket connections available on the system (~64k)
        worker_connections 4000;
    
        # optimized to serve many clients with each thread, essential for linux -- for testing environment
        use epoll;
    
        # accept as many connections as possible, may flood worker connections if set too low -- for testing environment
        multi_accept on;
    }
    
    http {
        # cache informations about FDs, frequently accessed files
        # can boost performance, but you need to test those values
        open_file_cache max=200000 inactive=20s;
        open_file_cache_valid 30s;
        open_file_cache_min_uses 2;
        open_file_cache_errors on;
    
        # to boost I/O on HDD we can disable access logs
        access_log on;
    
        # copies data between one FD and other from within the kernel
        # faster than read() + write()
        sendfile on;
    
        # send headers in one piece, it is better than sending them one by one
        tcp_nopush on;
    
        # don't buffer data sent, good for small data bursts in real time
        tcp_nodelay on;
    
        # reduce the data that needs to be sent over network -- for testing environment
        gzip on;
        # gzip_static on;
        gzip_min_length 10240;
        gzip_comp_level 1;
        gzip_vary on;
        gzip_disable msie6;
        gzip_proxied expired no-cache no-store private auth;
        gzip_types
            # text/html is always compressed by HttpGzipModule
            text/css
            text/javascript
            text/xml
            text/plain
            text/x-component
            application/javascript
            application/x-javascript
            application/json
            application/xml
            application/rss+xml
            application/atom+xml
            font/truetype
            font/opentype
            application/vnd.ms-fontobject
            image/svg+xml;
    
        # allow the server to close connection on non responding client, this will free up memory
        reset_timedout_connection on;
    
        # request timed out -- default 60
        client_body_timeout 10;
    
        # if client stop responding, free up memory -- default 60
        send_timeout 2;
    
        # server will close connection after this time -- default 75
        keepalive_timeout 30;
    
        # number of requests client can make over keep-alive -- for testing environment
        keepalive_requests 100000;
        
        log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                          '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for"';
      ########################################################
    	########################################################  
            server {
    
    
            access_log  logs/access.log  main;
         	  sendfile on;
        	  tcp_nopush on;
        	  tcp_nodelay on;
        	  keepalive_timeout 65;
        	  types_hash_max_size 2048;
        	  # server_tokens off;
            gzip  on;
          
    		    # Enable QUIC and HTTP/3.
            listen 443 quic reuseport;
    
            # Enable HTTP/2 (optional).
            listen 443 ssl http2;
    
            ssl_certificate      certificate.pem;
            ssl_certificate_key  key.pem;
    
            # Enable all TLS versions (TLSv1.3 is required for QUIC).
            ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
            
            # Add Alt-Svc header to negotiate HTTP/3.
            add_header alt-svc 'h3-23=":443"; ma=86400';
     
            listen       80;
            server_name  localhost;
    
            location / {
                root   html;
                index  index.html index.htm;
            }
     
            error_page   500 502 503 504  /50x.html;
            location = /50x.html {
                root   html;
            } 
            ###Limits the maximum number of concurrent HTTP/3 streams in a connection.
            http3_max_concurrent_streams 256;
            
            ###Limits the maximum number of requests that can be served on a single HTTP/3 connection, 
            ###after which the next client request will lead to connection closing and the need of establishing a new connection.
            http3_max_requests 20000;
            
            ###Limits the maximum size of the entire request header list after QPACK decompression.
            http3_max_header_size 100000k;
            
            ###Sets the per-connection incoming flow control limit.
            http3_initial_max_data 2000000m;
            
            ###Sets the per-stream incoming flow control limit.
            http3_initial_max_stream_data 1000000m;
            
            ###Sets the timeout of inactivity after which the connection is closed.
            http3_idle_timeout 1500000m;
        }
     ########################################################
    	########################################################
    }
    

    Curl version

    curl 7.67.0-DEV (x86_64-pc-linux-gnu) libcurl/7.67.0-DEV BoringSSL zlib/1.2.11 nghttp2/1.39.2 quiche/0.1.0
    Release-Date: [unreleased]
    Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
    Features: AsynchDNS HTTP2 HTTP3 HTTPS-proxy IPv6 Largefile libz NTLM NTLM_WB SSL UnixSockets
    
    Reply