Rust-Bzip2 rs: A streaming compression/decompression library for rust with bindings to libbz2.

bzip2

Documentation

A streaming compression/decompression library for rust with bindings to libbz2.

# Cargo.toml
[dependencies]
bzip2 = "0.4"

License

This project is licensed under either of

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in Serde by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Comments

  • Shouldnt bzip2-sys have the bzip2 license?
    Shouldnt bzip2-sys have the bzip2 license?

    Feb 15, 2018

    Right now, Cargo.toml of the bzip2-sys crate claims MIT/Apache-2.0 but it contains the entire source code of the bzip2 library.

    Reply
  • bzip2::Compression type does not allow arbitrary values from 1-9
    bzip2::Compression type does not allow arbitrary values from 1-9

    Mar 9, 2018

    The bzip2::Compression type only allows the options Fastest (1), Default (6), or Best (9). There seems to be no way to specify a compression level of, say, 4.

    Perhaps it would be better to make bzip2::Compression work like flate2::Compression (which does allow arbitrary numbers), since the two crates already have very similar APIs otherwise?

    Reply
  • Re-implement in pure Rust
    Re-implement in pure Rust

    Jan 25, 2019

    Are you interested in re-implementing bzip2 in pure Rust or do you intend to keep it as a binding? (i.e. will you accept such work if anyone will be interested in it or not?) bzip2-sys implementation could be kept as a feature-gated disabled by default option.

    Potential rewrite will help with the licensing issue and will help crates which target WASM.

    Reply
  • Reading Multiple Blocks
    Reading Multiple Blocks

    Mar 14, 2019

    I'm encountering a slight issue when using the library: I'm only getting the first block of bz2 files. In my case, the block size is 900k, and when reading off BzDecoder, I get a total of 900k then EOF (size zero reads thereafter).

    This happens on both the crates.io release and master here on github. Am I confused here? Would appreciate any suggestions. Thanks!

    Reply
  • Readme has a bzip2 crate version that isn't published yet
    Readme has a bzip2 crate version that isn't published yet

    Feb 16, 2020

    I tried making an example program with bzip2 = "0.4" but it's not published:

    $ cargo run
        Updating crates.io index
    error: failed to select a version for the requirement `bzip2 = "^0.4"`
      candidate versions found which didn't match: 0.3.3, 0.3.2, 0.3.1, ...
      location searched: crates.io index
    required by package `bzip2test v0.1.0 (/home/x/dev/rust/bzip2test)`
    
    Reply
  • bzip2-sys: features=[
    bzip2-sys: features=["static"]?

    May 12, 2020

    Would it be possible to have static feature like libz-sys does?:

    https://github.com/rust-lang/libz-sys/blob/master/Cargo.toml#L35

    /cc https://github.com/alexcrichton/xz2-rs/issues/62

    Reply
  • Use partial-io to shake out Interrupted-related bugs
    Use partial-io to shake out Interrupted-related bugs

    May 30, 2017

    partial-io is a helper library that lets users wrap readers and writers and generate partial reads, or fail with temporary errors like Interrupted. It also integrates with quickcheck to provide random sequences of test instances.

    partial-io found a bug on the write side: we weren't handling Interrupted, even though AsyncWrite requires that Interrupted errors be handled at this layer.

    Closes #19.

    Reply
  • Update partial-io requirement to ^0.3.0
    Update partial-io requirement to ^0.3.0

    Jul 5, 2018

    Updates the requirements on partial-io to permit the latest version.

    Commits

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

    Dependabot will not automatically merge this PR because this dependency is pre-1.0.0.


    Note: This repo was added to Dependabot recently, so you'll receive a maximum of 5 PRs for your first few update runs. Once an update run creates fewer than 5 PRs we'll remove that limit.

    You can always request more updates by clicking Bump now in your Dependabot dashboard.

    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot ignore this [patch|minor|major] version will close this PR and stop Dependabot creating any more for this minor/major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot badge me will comment on this PR with code to add a "Dependabot enabled" badge to your readme

    Additionally, you can set the following in your Dependabot dashboard:

    • Update frequency (including time of day and day of week)
    • Automerge options (never/patch/minor, and dev/runtime dependencies)
    • Pull request limits (per update run and/or open at any time)
    • Out-of-range updates (receive only lockfile updates, if desired)
    • Security updates (receive only security updates, if desired)

    Finally, you can contact us by mentioning @dependabot.

    dependencies 
    Reply
  • Error trying to install crate bzip2-sys
    Error trying to install crate bzip2-sys

    Oct 4, 2018

    I'm trying to install the zip crate that depends on bzip2 but I'm getting the following error.

    Using Rust nightly 1.31.0 ("2018 edition")

    Compiling bzip2-sys v0.1.6
    error: failed to run custom build command for `bzip2-sys v0.1.6`
    process didn't exit successfully: `C:\Dev\[project]\target\debug\build\bzip2-sys-33f8cb5c7dfcf9ea\build-script-build` (exit code: 101)
    --- stdout
    TARGET = Some("x86_64-pc-windows-msvc")
    OPT_LEVEL = Some("0")
    HOST = Some("x86_64-pc-windows-msvc")
    CC_x86_64-pc-windows-msvc = None
    CC_x86_64_pc_windows_msvc = None
    HOST_CC = None
    CC = Some("C:\\mingw-w64\\x86_64-7.3.0-posix-seh-rt_v5-rev0\\mingw64\\bin\\gcc.exe")
    CFLAGS_x86_64-pc-windows-msvc = None
    CFLAGS_x86_64_pc_windows_msvc = None
    HOST_CFLAGS = None
    CFLAGS = None
    DEBUG = Some("true")
    running: "C:\\mingw-w64\\x86_64-7.3.0-posix-seh-rt_v5-rev0\\mingw64\\bin\\gcc.exe" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-I" "bzip2-1.0.6" "-D_WIN32" "-DBZ_EXPORT" "-DBZ_NO_STDIO" "/FoC:\\Dev\\[project]\\target\\debug\\build\\bzip2-sys-351245291296f0d0\\out\\bzip2-1.0.6\\blocksort.o" "/c" "bzip2-1.0.6/blocksort.c"
    cargo:warning=gcc.exe: error: /FoC:\Dev\[project]\target\debug\build\bzip2-sys-351245291296f0d0\out\bzip2-1.0.6\blocksort.o: Invalid argument
    cargo:warning=gcc.exe: error: /c: No such file or directory
    exit code: 1
    
    --- stderr
    thread 'main' panicked at '
    
    Internal error occurred: Command "C:\\mingw-w64\\x86_64-7.3.0-posix-seh-rt_v5-rev0\\mingw64\\bin\\gcc.exe" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-I" "bzip2-1.0.6" "-D_WIN32" "-DBZ_EXPORT" "-DBZ_NO_STDIO" "/FoC:\\Dev\\[project]\\target\\debug\\build\\bzip2-sys-351245291296f0d0\\out\\bzip2-1.0.6\\blocksort.o" "/c" "bzip2-1.0.6/blocksort.c" with args "gcc.exe" did not execute successfully (status code exit code: 1).
    
    ', C:\Users\[user]\.cargo\registry\src\github.com-1ecc6299db9ec823\cc-1.0.25\src\lib.rs:2260:5
    note: Run with `RUST_BACKTRACE=1` for a backtrace.
    
    Reply
  • BzCompressor fails on empty buffer?
    BzCompressor fails on empty buffer?

    Mar 2, 2015

    The Read implementation of BzCompressor can return an error when the underlying buffer return zero bytes: https://github.com/alexcrichton/bzip2-rs/blob/master/src/reader.rs#L67.

    I believe this is now incorrect behaviour, as EOF is now signalled through a zero-byte read instead of an error. That's what I get from the docs, anyway: https://doc.rust-lang.org/std/io/trait.Read.html#tymethod.read

    Reply
  • Build failure at gcc level: blocksort.c not found
    Build failure at gcc level: blocksort.c not found

    Jul 6, 2018

    bzip2-sys seems to refuse to build due to missing blocksort.c (it is indeed not present in the related build directory).

       Compiling bzip2-sys v0.1.6
    error: failed to run custom build command for `bzip2-sys v0.1.6`
    process didn't exit successfully: `c:\Local\poligon\extract_blobs\target\release\build\bzip2-sys-9937a6f77c12f052\build-script-build` (exit code: 101)
    --- stdout
    TARGET = Some("x86_64-pc-windows-gnu")
    OPT_LEVEL = Some("3")
    TARGET = Some("x86_64-pc-windows-gnu")
    HOST = Some("x86_64-pc-windows-gnu")
    TARGET = Some("x86_64-pc-windows-gnu")
    TARGET = Some("x86_64-pc-windows-gnu")
    HOST = Some("x86_64-pc-windows-gnu")
    CC_x86_64-pc-windows-gnu = Some("C:\\Local\\poligon\\.rustup\\toolchains\\nightly-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\bin\\gcc.exe")
    TARGET = Some("x86_64-pc-windows-gnu")
    HOST = Some("x86_64-pc-windows-gnu")
    CFLAGS_x86_64-pc-windows-gnu = None
    CFLAGS_x86_64_pc_windows_gnu = None
    HOST_CFLAGS = None
    CFLAGS = None
    DEBUG = Some("false")
    running: "C:\\Local\\poligon\\.rustup\\toolchains\\nightly-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\bin\\gcc.exe" "-O3" "-ffunction-sections" "-fdata-sections" "-m64" "-I" "bzip2-1.0.6" "-D_WIN32" "-DBZ_EXPORT" "-DBZ_NO_STDIO" "-o" "c:\\Local\\poligon\\extract_blobs\\target\\release\\build\\bzip2-sys-f19fa19517d295e2\\out\\bzip2-1.0.6\\blocksort.o" "-c" "bzip2-1.0.6/blocksort.c"
    cargo:warning=gcc.exe: error: CreateProcess: No such file or directory
    exit code: 1
    

    I'm using rustc 1.28.0-nightly-gnu-x64 with Win 7.

    Reply
  • Stuck busy looping
    Stuck busy looping

    Aug 11, 2017

    First please take whit a grain of salt, and assume I might be doing something wrong, or the data is corrupted.

    I am doing long running rdedup stressttest I caught it hanged, using 100% of one of the CPUs. I was able to attach gdb and turns out it looping on a drop/finish in this code of mine:

      8     fn decompress(&self, buf: SGData) -> io::Result<SGData> {                                                       
      7         let mut decompressor =                                                                                      
      6             bzip2::write::BzDecoder::new(Vec::with_capacity(buf.len()));                                            
      5                                                                                                                     
      4         for sg_part in buf.as_parts() {                                                                             
      3             decompressor.write_all(&sg_part)?;                                                                      
      2         }                                                                                                           
      1         Ok(SGData::from_single(decompressor.finish()?))                                                             
    114     }
    

    Note that 114 might be off, as I might have added/removed a line or two. I'm guessing it is finish() call.

    #0  bzip2::write::{{impl}}::write<collections::vec::Vec<u8>> (self=0x7fff68365488, data=...)                            
        at /home/dpc/.cargo/registry/src/github.com-1ecc6299db9ec823/bzip2-0.3.2/src/write.rs:254                           
    #1  0x00005567a4127d20 in core::ptr::drop_in_place<bzip2::write::BzDecoder<collections::vec::Vec<u8>>> ()               
        at /home/dpc/.cargo/registry/src/github.com-1ecc6299db9ec823/bzip2-0.3.2/src/write.rs:219                           
    #2  0x00005567a4127952 in rdedup_lib::compression::{{impl}}::decompress (self=<optimized out>, buf=...)                 
        at lib/src/compression.rs:114                           
    #3  0x00005567a4153913 in rdedup_lib::{{impl}}::read_chunk_into (self=0x7fff6837fd38, digest=...,                       
        chunk_type=<optimized out>, data_type=rdedup_lib::DataType::Data, writer=...) at lib/src/lib.rs:352                 
    #4  0x00005567a4151807 in rdedup_lib::ReadContext::read_recursively (self=0x7fff68367aa0, accessor=..., digest=...)     
        at lib/src/lib.rs:262    
    

    If I keep doing next in my gdb, I get:

    core::ptr::drop_in_place<bzip2::write::BzDecoder<collections::vec::Vec<u8>>> ()                                         
        at /home/dpc/lab/rust/rdedup/lib/<try macros>:3         
    3       /home/dpc/lab/rust/rdedup/lib/<try macros>: No such file or directory.                                          
    (gdb)                         
    218             while !self.done {                          
    (gdb)                         
    219                 try!(self.write(&[]));                  
    (gdb)                         
    3       /home/dpc/lab/rust/rdedup/lib/<try macros>: No such file or directory.                                          
    (gdb)                         
    218             while !self.done {                          
    (gdb)                         
    219                 try!(self.write(&[]));                  
    (gdb) n                       
    3       /home/dpc/lab/rust/rdedup/lib/<try macros>: No such file or directory.                                          
    (gdb)                         
    218             while !self.done {                          
    (gdb)                         
    219                 try!(self.write(&[]));                  
    (gdb)                         
    3       /home/dpc/lab/rust/rdedup/lib/<try macros>: No such file or directory.                                          
    (gdb)                         
    218             while !self.done {                          
    (gdb)                         
    219                 try!(self.write(&[]));                  
    (gdb)                         
    3       /home/dpc/lab/rust/rdedup/lib/<try macros>: No such file or directory.    
    

    If I do step, I get:

    3       /home/dpc/lab/rust/rdedup/lib/<try macros>: No such file or directory.                                          
    (gdb) s                       
    218             while !self.done {                          
    (gdb)                         
    219                 try!(self.write(&[]));                  
    (gdb)                         
    bzip2::write::{{impl}}::write<collections::vec::Vec<u8>> (self=0x7fff68365488, data=...)                                
        at /home/dpc/.cargo/registry/src/github.com-1ecc6299db9ec823/bzip2-0.3.2/src/write.rs:254                           
    254             if self.done {
    (gdb)                         
    258                 try!(self.dump());                      
    (gdb)                         
    258                 try!(self.dump());                      
    (gdb)                         
    bzip2::write::BzDecoder<collections::vec::Vec<u8>>::dump<collections::vec::Vec<u8>> (self=0x7fff68365488)               
        at /home/dpc/.cargo/registry/src/github.com-1ecc6299db9ec823/bzip2-0.3.2/src/write.rs:200                           
    200             while self.buf.len() > 0 {                  
    (gdb)                         
    204             Ok(())        
    (gdb)                         
    205         }                 
    (gdb)                         
    bzip2::write::{{impl}}::write<collections::vec::Vec<u8>> (self=0x7fff68365488, data=...)                                
        at /home/dpc/lab/rust/rdedup/lib/<try macros>:3                                                                     
    3       /home/dpc/lab/rust/rdedup/lib/<try macros>: No such file or directory.                                          
    (gdb)                                                                                                                   
    267             (self.raw.total_in_lo32 as u64) |                                                                       
    (gdb)                                                                                                                   
    268             ((self.raw.total_in_hi32 as u64) << 32)                                                                 
    (gdb)                                                                                                                   
    261                 let res = self.data.decompress_vec(data, &mut self.buf);                                            
    (gdb)                                                                                                                   
    bzip2::mem::Decompress::decompress_vec (self=0x7fff68365488, input=..., output=0x7fff683654a8)                          
        at /home/dpc/.cargo/registry/src/github.com-1ecc6299db9ec823/bzip2-0.3.2/src/mem.rs:243                             
    243                 let before = self.total_out();                                                                      
    (gdb)                                                                                                                   
    bzip2::mem::Decompress::total_out (self=<optimized out>)                                                                
        at /home/dpc/.cargo/registry/src/github.com-1ecc6299db9ec823/bzip2-0.3.2/src/mem.rs:243                             
    243                 let before = self.total_out();                                                                      
    (gdb)                         
    bzip2::mem::Stream<bzip2::mem::DirDecompress>::total_out<bzip2::mem::DirDecompress> (self=<optimized out>)              
        at /home/dpc/.cargo/registry/src/github.com-1ecc6299db9ec823/bzip2-0.3.2/src/mem.rs:272                             
    272             (self.raw.total_out_lo32 as u64) |          
    (gdb)                         
    273             ((self.raw.total_out_hi32 as u64) << 32)    
    (gdb)                         
    bzip2::mem::Decompress::decompress_vec (self=0x7fff68365488, input=..., output=0x7fff683654a8)                          
        at /home/dpc/.cargo/registry/src/github.com-1ecc6299db9ec823/bzip2-0.3.2/src/mem.rs:245                             
    245                     let ptr = output.as_mut_ptr().offset(len as isize);                                             
    (gdb)                         
    core::ptr::{{impl}}::offset<u8> (count=<optimized out>, self=<optimized out>) at /checkout/src/libcore/ptr.rs:617       
    617     /checkout/src/libcore/ptr.rs: No such file or directory.                                                        
    (gdb)         
    (...)
    
    Reply