Rust-Flate2 rs: A streaming compression/decompression library DEFLATE-based streams in Rust.

flate2

Crates.io Documentation

A streaming compression/decompression library DEFLATE-based streams in Rust.

This crate by default implemented as a wrapper around the miniz_oxide crate, a port of miniz.c to Rust. This crate can also optionally use other backends like the zlib library or miniz.c itself.

Supported formats:

  • deflate
  • zlib
  • gzip
# Cargo.toml
[dependencies]
flate2 = "1.0"

Compression

use std::io::prelude::*;
use flate2::Compression;
use flate2::write::ZlibEncoder;

fn main() {
    let mut e = ZlibEncoder::new(Vec::new(), Compression::default());
    e.write_all(b"foo");
    e.write_all(b"bar");
    let compressed_bytes = e.finish();
}

Decompression

use std::io::prelude::*;
use flate2::read::GzDecoder;

fn main() {
    let mut d = GzDecoder::new("...".as_bytes());
    let mut s = String::new();
    d.read_to_string(&mut s).unwrap();
    println!("{}", s);
}

Backends

Using zlib instead of the (default) Rust backend:

[dependencies]
flate2 = { version = "1.0", features = ["zlib"], default-features = false }

The cloudflare optimized version of zlib is also available. While it's significantly faster it requires a x86-64 CPU with SSE 4.2 or ARM64 with NEON & CRC. It does not support 32-bit CPUs at all and is incompatible with mingw. For more information check the crate documentation.

[dependencies]
flate2 = { version = "1.0", features = ["cloudflare_zlib"], default-features = false }

Using miniz.c:

[dependencies]
flate2 = { version = "1.0", features = ["miniz-sys"], default-features = false }

License

This project is licensed under either of

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this project by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Comments

  • Do not compile unsafe FFI code when using Rust backend
    Do not compile unsafe FFI code when using Rust backend

    Oct 31, 2019

    flate2 includes plenty of unsafe code that's only used for interaction C backends, and can be omitted from the Rust backend builds. We should use conditional compilation to leave it out of the build when it's not in use, so that any auditing or refactoring efforts can focus on the unsafe code that's actually executed.

    For example, the entirety of src/ffi/c.rs can be left out of builds that use Rust backend only; this brings down the unsafe expression count from 107 to 43 according to cargo-geiger.

    Reply
  • Creating a slice of uninit memory is unsound (equivalent to mem::uninitialized)
    Creating a slice of uninit memory is unsound (equivalent to mem::uninitialized)

    Oct 31, 2019

    flate2 creates slices of uninitialized memory in several places. The only two places where it happens when using the Rust backend are here and here.

    This is equivalent to the use of the now-deprecated mem::uninitialized. Instead, either a slice of MaybeUninit<T> should be constructed and passed to the backends, or the backends should receive a structure that does not expose uninitialized memory by design such as Vec or a Vec-like fixed-capacity view of memory.

    If the backend does not overwrite the entire slice, this can become an exploitable security vulnerability.

    Reply
  • Expose zlib fine-tuning options in high-level API
    Expose zlib fine-tuning options in high-level API

    Apr 14, 2020

    Just moving from a comment (https://github.com/alexcrichton/flate2-rs/pull/230#issuecomment-592104767) to a proper tracking issue:

    @RReverser > @alexcrichton Somewhat related. I've noticed that higher-level APIs currently don't expose these advanced zlib configurations, which seems a shame because they're much easier to use than Compress / Decompress directly, and sometimes you really need that fine-tuning. Would you be open to adding either new methods to ZlibEncoder / ZlibDecoder to propagate corresponding params, or adding implementations of From<Compress> / From<Decompress> so that one could create a high-level wrapper for a preconfigured instance?

    @alexcrichton > In terms of adding higher-level APIs, yes, that seems fine to me too. I would prefer it if were a separate PR though and for this to not balloon too much.

    Reply
  • Inconsistent GzBuilder() example file name
    Inconsistent GzBuilder() example file name

    Apr 20, 2020

    The example for GzBuilder() should create "hello_world.txt.gz" because compressed file name is "hello_word.txt"

    If you run the example as is, then:

    $ gunzip -l hello_world.gz
         compressed        uncompressed  ratio uncompressed_name
                     72                  11 -18.2% hello_world
    

    and file name is not "hello_word.txt".

    Not a big deal though.

    Reply
  • Mediocre gzip compression using flate2 with miniz_oxide
    Mediocre gzip compression using flate2 with miniz_oxide

    May 14, 2020

    Hi there, I'm using flate2 with the default backend (apparently miniz_oxide) via the async_compression crate.

    I have a "file" like this (in a test):

    const TEXT: &'static str = concat![
        "Chunk one\n",
        "data data\n",
        "\n",
        "Chunk two\n",
        "data data\n",
        "\n",
        "Chunk three\n",
        "data data\n",
    ];
    

    flate2 appears to compress that to the following (hex dump from binary):

    00000000: 1f8b 0800 0000 0000 00ff 6dca b109 0030  ..........m....0
    00000010: 0805 d1fe 4fe1 2e4e 2244 1002 0ac1 90f5  ....O..N"D......
    00000020: 4352 59d8 5cf1 38b6 ed93 c215 4352 e805  CRY.\.8.....CR..
    00000030: e08f 79a2 415b 5adf 0bde f2d7 8140 0000  ..y.A[[email protected]
    00000040: 00                                       .
    

    However, this test is being partially ported from Node.js, where I did something similar a while ago. Node.js uses zlib. The hex resulting from that binary is this:

    00000000: 1f8b 0800 0000 0000 0013 73ce 28cd cb56  ..........s.(..V
    00000010: c8cf 4be5 4a49 2c49 5400 115c 5cce 60c1  ..K.JI,IT..\\.`.
    00000020: 92f2 7c2c 8219 45a9 c86a 01de f2d7 8140  ..|,[email protected]
    00000030: 0000 00                                  ...
    

    The zlib one here is quite a bit smaller.

    Both of these do appear to be valid, and both decompress fine with other tools.

    Addendum: this is with the "default" quality setting (zlib level 6 or 7 (whichever it is), default flate2). Changing flate2 to "best" does not appear to make a difference.

    Reply
  • feat: allow full range of window bits for zlib
    feat: allow full range of window bits for zlib

    May 20, 2020

    According to https://github.com/madler/zlib/blob/cacf7f1d4e3d44d871b605da3b647f07d718623f/zlib.h#L839-L870 the Window Bits parameter for inflateInit2 should support values in the range of -15 to 47. This PR allows for those ranges in the C implementation.

    Reply
  • Partial file decompress support
    Partial file decompress support

    May 19, 2020

    I'm working on an application that pulls parts of compressed files out of S3. I don't want to pull the entire file out of s3 due to file sizes. I should be able to pass part of the file to GzDecode and decompress just the chunk.

    let client = S3Client::new(Region::default());
    let req = GetObjectRequest {
        bucket: self.bucket.clone(),
        key: self.key.clone(),
        version_id: Some(self.version_id.clone()),
        part_number: Some(part),
        ..Default::default()
    };
    
    let response = client.get_object(req).await;
    if response.is_err() {
        let err = response.err().unwrap();
        error!("{}", err);
        panic!("Unable to fetch object from S3");
    }
    let response = response.unwrap();
    // More Code Here...
    
    let body = response.body.unwrap();
    let mut buff = BytesMut::with_capacity(512);
    match body.into_async_read().read_buf(&mut buff).await {
    // More Code Here...
    let frozen_bytes = buff.to_vec();
    let mut deflater = GzDecoder::new(&frozen_bytes[..]);
    let mut s = String::new();
    let read_response_length = deflater.read_to_string(&mut s);
    

    However, I get a corrupt deflate stream error. Is it not possible to pass only part of a gzip compressed file to the GzDecoder?

    Reply
  • Decoder eats whole input stream
    Decoder eats whole input stream

    Feb 21, 2015

    As explained to me in http://stackoverflow.com/a/28641354/506962

    However, it [ByRefReader] does not work when reading. It looks like reader::ZlibDecoder might consume all the way to the end of the underlying Reader. This could potentially be a bug or an oversight in the flate2 library.

    Reply
  • Rename internal types to match the public types
    Rename internal types to match the public types

    Apr 4, 2017

    The write/read/bufread modules all repeat the same type names:

    pub use gz::EncoderReader as flate2::read::GzEncoder;
    pub use deflate::EncoderReader as flate2::read::DeflateEncoder;
    

    Unfortunately the rustdoc for flate2::read::GzEncoder shows the private names:

    impl<R: Read> EncoderReader<R>
    fn new(r: R, level: Compression) -> EncoderReader<R>
    

    This is a rustdoc bug but even aside from that, this pattern is confusing to people browsing the code because what they see clicking through a [src] link does not match how they will be using the library. Let's try to make the real type names match the public names they are exported as.

    Relevant API guideline: https://github.com/brson/rust-api-guidelines/issues/5 Rustdoc bug: https://github.com/rust-lang/rust/issues/41072

    easy help wanted 
    Reply
  • Add a pure-Rust backend
    Add a pure-Rust backend

    Mar 27, 2017

    flate2 gets its actual compression from either libz or miniz, both C libraries. Ultimately, a pure-Rust stack is better for Rust, mostly because of build simplicity.

    Write a miniz replacement in Rust, publish it to crates.io and add another compile-time feature to select it.

    Then make the performance better than miniz and make it the default.

    help wanted 
    Reply
  • undefined reference to `__assert_func'
    undefined reference to `__assert_func'

    Oct 2, 2015

    For some reason I cannot compile this project: https://github.com/viperscape/font-atlas-example/tree/master/atlas-gen, specifically that sub-project atlas-gen. Below is the output during linking. I am using mingw gcc from cygwin (x86_64-w64-mingw32-gcc) with rust-nightly 32bit. It's very possible that this is an issue on my end, as I was able to compile this yesterday; I'm not sure what changed :-( I'm hesitant to install mingw directly because it's housed on sourceforge, which I try to avoid now a days. Any clues to what is going on here? Thanks!

    note: C:\Users\chris\font-atlas-example\atlas-gen\target\debug\deps\libminiz_sys-d19b88f9ef21a81d.rlib(miniz.o): In function `tinfl_decompress':
    /cygdrive/c/Users/Chris/.cargo/registry/src/github.com-121aea75f9ef2ce2/miniz-sys-0.1.6/miniz.c:1707: undefined reference to `__assert_func'
    C:\Users\chris\font-atlas-example\atlas-gen\target\debug\deps\libminiz_sys-d19b88f9ef21a81d.rlib(miniz.o): In function `tdefl_start_dynamic_block':
    /cygdrive/c/Users/Chris/.cargo/registry/src/github.com-121aea75f9ef2ce2/miniz-sys-0.1.6/miniz.c:2024: undefined reference to `__assert_func'
    /cygdrive/c/Users/Chris/.cargo/registry/src/github.com-121aea75f9ef2ce2/miniz-sys-0.1.6/miniz.c:2026: undefined reference to `__assert_func'
    /cygdrive/c/Users/Chris/.cargo/registry/src/github.com-121aea75f9ef2ce2/miniz-sys-0.1.6/miniz.c:2027: undefined reference to `__assert_func'
    /cygdrive/c/Users/Chris/.cargo/registry/src/github.com-121aea75f9ef2ce2/miniz-sys-0.1.6/miniz.c:2030: undefined reference to `__assert_func'
    C:\Users\chris\font-atlas-example\atlas-gen\target\debug\deps\libminiz_sys-d19b88f9ef21a81d.rlib(miniz.o):/cygdrive/c/Users/Chris/.cargo/registry/src/github.com-121aea75f9ef2ce2/miniz-sys-0.1.6/miniz.c:2031: more undefined references to `__assert_func' follow
    
    $ gcc --version
    gcc (GCC) 4.9.2
    Copyright (C) 2014 Free Software Foundation, Inc.
    This is free software; see the source for copying conditions.  There is NO
    warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
    
    $ rustc --version
    rustc 1.5.0-nightly (168a23ebe 2015-10-01)
    
    $ cargo --version
    cargo 0.6.0-nightly (7f21e73 2015-10-01)
    
    Reply
  • failed to run custom build command for `miniz-sys v0.1.7`
    failed to run custom build command for `miniz-sys v0.1.7`

    Dec 28, 2015

    Some PistonDeveloper projects are implicit referencing miniz-sys. On my Desktop i run into problems compiling them (named: conrod examples and imgproc examples).

    OS: Windows 8.1 (64Bit) rustc: 1.5.0 (Stable)

    failed to run custom build command for `miniz-sys v0.1.7`
    Process didn't exit successfully: `C:\Users\username\Documents\GitHub\conrod\target\
    release\build\miniz-sys-f1c39e8e406fa25f\build-script-build` (exit code: 101)
    --- stdout
    TARGET = Some("x86_64-pc-windows-gnu")
    OPT_LEVEL = Some("3")
    PROFILE = Some("release")
    TARGET = Some("x86_64-pc-windows-gnu")
    debug=false opt-level=3
    HOST = Some("x86_64-pc-windows-gnu")
    TARGET = Some("x86_64-pc-windows-gnu")
    TARGET = Some("x86_64-pc-windows-gnu")
    HOST = Some("x86_64-pc-windows-gnu")
    CC_x86_64-pc-windows-gnu = None
    CC_x86_64_pc_windows_gnu = None
    HOST_CC = None
    CC = None
    TARGET = Some("x86_64-pc-windows-gnu")
    HOST = Some("x86_64-pc-windows-gnu")
    CFLAGS_x86_64-pc-windows-gnu = None
    CFLAGS_x86_64_pc_windows_gnu = None
    HOST_CFLAGS = None
    CFLAGS = None
    running: "gcc.exe" "-O3" "-ffunction-sections" "-fdata-sections" "-m64" "-o" "C:
    \\Users\\username\\Documents\\GitHub\\conrod\\target\\release\\build\\miniz-sys-f1c3
    9e8e406fa25f\\out\\miniz.o" "-c" "miniz.c"
    ExitStatus(ExitStatus(1))
    
    command did not execute successfully, got: exit code: 1
    
    --- stderr
    gcc.exe: error: CreateProcess: No such file or directory
    thread '<main>' panicked at 'explicit panic', C:\Users\username\.cargo\registry\src\
    github.com-0a35038f75765ae4\gcc-0.3.21\src\lib.rs:772
    

    My %PATH% (relevant parts) looks like this:

    C:\Program Files\Rust stable 1.5\bin;C:\Program Files\Rust stable 1.5\bin\rustlib\x86_64-pc-windows-gnu\lib\;C:\Program Files\Rust stable 1.5\bin\rustlib\x86_64-pc-windows-gnu\bin\;C:\Program Files\Rust stable1.5\bin\;
    

    manual executing build-scrip-build.exe:

    C:\Users\username\Documents\GitHub\conrod\target\debug\build\miniz-sys-f1c39e8e406fa
    25f>build-script-build.exe
    thread '<main>' panicked at 'called `Option::unwrap()` on a `None` value', ../sr
    c/libcore\option.rs:366
    

    gcc -v output:

    Using built-in specs.
    COLLECT_GCC=gcc
    Target: x86_64-w64-mingw32
    Configured with: ../../../src/gcc-4.9.1/configure --host=x86_64-w64-mingw32 --bu
    ild=x86_64-w64-mingw32 --target=x86_64-w64-mingw32 --prefix=/mingw64 --with-sysr
    oot=/c/mingw491/x86_64-491-win32-seh-rt_v3-rev1/mingw64 --with-gxx-include-dir=/
    mingw64/x86_64-w64-mingw32/include/c++ --enable-shared --enable-static --disable
    -multilib --enable-languages=ada,c,c++,fortran,objc,obj-c++,lto --enable-libstdc
    xx-time=yes --enable-threads=win32 --enable-libgomp --enable-libatomic --enable-
    lto --enable-graphite --enable-checking=release --enable-fully-dynamic-string --
    enable-version-specific-runtime-libs --disable-isl-version-check --disable-cloog
    -version-check --disable-libstdcxx-pch --disable-libstdcxx-debug --enable-bootst
    rap --disable-rpath --disable-win32-registry --disable-nls --disable-werror --di
    sable-symvers --with-gnu-as --with-gnu-ld --with-arch=nocona --with-tune=core2 -
    -with-libiconv --with-system-zlib --with-gmp=/c/mingw491/prerequisites/x86_64-w6
    4-mingw32-static --with-mpfr=/c/mingw491/prerequisites/x86_64-w64-mingw32-static
     --with-mpc=/c/mingw491/prerequisites/x86_64-w64-mingw32-static --with-isl=/c/mi
    ngw491/prerequisites/x86_64-w64-mingw32-static --with-cloog=/c/mingw491/prerequi
    sites/x86_64-w64-mingw32-static --enable-cloog-backend=isl --with-pkgversion='x8
    6_64-win32-seh-rev1, Built by MinGW-W64 project' --with-bugurl=http://sourceforg
    e.net/projects/mingw-w64 CFLAGS='-O2 -pipe -I/c/mingw491/x86_64-491-win32-seh-rt
    _v3-rev1/mingw64/opt/include -I/c/mingw491/prerequisites/x86_64-zlib-static/incl
    ude -I/c/mingw491/prerequisites/x86_64-w64-mingw32-static/include' CXXFLAGS='-O2
     -pipe -I/c/mingw491/x86_64-491-win32-seh-rt_v3-rev1/mingw64/opt/include -I/c/mi
    ngw491/prerequisites/x86_64-zlib-static/include -I/c/mingw491/prerequisites/x86_
    64-w64-mingw32-static/include' CPPFLAGS= LDFLAGS='-pipe -L/c/mingw491/x86_64-491
    -win32-seh-rt_v3-rev1/mingw64/opt/lib -L/c/mingw491/prerequisites/x86_64-zlib-st
    atic/lib -L/c/mingw491/prerequisites/x86_64-w64-mingw32-static/lib '
    Thread model: win32
    
    Reply