Clone this repo:

Branches

  1. e767c88 Fix VPCLMULQDQ CRC32 build with partial AVX-512 baselines by Nathan Moin Vaziri · 8 days ago main
  2. 341f2d7 Add fallback defines to skip generic C code when native intrinsics exist by Nathan Moinvaziri · 8 days ago
  3. 8db6717 Use __attribute__((constructor)) to initialize the functable by Vladislav Shchapov · 8 days ago
  4. 9babb29 [CI] Switch CMake workflow to use MSYS2 for MinGW32. by Mika Lindqvist · 8 days ago
  5. 8f19573 Add ACCUM_ROUND macro to crc32_chorba_sse2 by Nathan Moin Vaziri · 8 days ago
CIStableDevelop
GitHub ActionsStable CMake
Stable Configure
Develop CMake
Develop Configure
CodeFactorCodeFactorCodeFactor
OSS-FuzzFuzzing StatusFuzzing Status
CoverallsCoverage StatusCoverage Status

zlib-ng

zlib data compression library for the next generation systems

Maintained by Hans Kristian Rosbach aka Dead2 (zlib-ng àt circlestorm dót org)

Features

  • Zlib compatible API with support for dual-linking
  • Modernized native API based on zlib API for ease of porting
  • Major speed improvements, x86-64 can be about 4x faster than stock zlib for example.
  • Modern C11 syntax and a clean code layout
  • Deflate medium and quick algorithms based on Intel’s zlib fork
  • Support for CPU intrinsics when available
    • Adler32 implementation using SSSE3, SSE4.2, AVX2, AVX512, AVX512-VNNI, Neon, VMX & VSX, LSX, LASX, RVV
    • CRC32-B implementation using SSE2, SSE4.1, (V)PCLMULQDQ, ARMv8, ARMv8.2 PMULL+EOR3, Power8, IBM Z, LoongArch, ZBC
    • Slide hash implementations using SSE2, AVX2, ARMv6, Neon, Power8, VMX & VSX, LSX, LASX
    • Compare256 implementations using SSE2, AVX2, AVX512, Neon, Power9, LSX, LASX, RVV
    • Inflate chunk copying using SSE2, SSSE3, AVX2, AVX512, Neon, Power8, VSX, LSX, LASX, RVV
    • Support for hardware-accelerated deflate using IBM Z DFLTCC
  • Safe unaligned memory read/writes and large bit buffer improvements
  • Includes improvements from Cloudflare and Intel forks
  • Configure and CMake build system support
  • Comprehensive set of CMake unit tests
  • Code sanitizers, fuzzing, and coverage
  • GitHub Actions continuous integration on Windows, macOS, and Linux
    • Native CI for Aarch64, S390x, x86, x86-64
    • Emulated CI for ARM, LoongArch, PPC, PPC64, RISCV, SPARC64, S390x using qemu

History

The motivation for this fork was seeing several 3rd party contributions with new optimizations not getting implemented into the official zlib repository.

Mark Adler has been maintaining zlib for a very long time, and he has done a great job and hopefully he will continue for a long time yet. The idea of zlib-ng is not to replace zlib, but to co-exist as a drop-in replacement with a lower threshold for code change.

zlib has a long history and is incredibly portable, even supporting many systems that predate the Internet.
That is great, but it can complicate further development and maintainability. The zlib code contains many workarounds for really old compilers or to accommodate systems with limitations such as operating in a 16-bit environment.

Many of these workarounds are only maintenance burdens, some of them are pretty huge code-wise. With many workarounds cluttered throughout the code, it makes it harder for new programmers with an idea/interest for zlib to contribute.

I decided to make a fork, merge all the Intel optimizations, some of the Cloudflare optimizations, plus a couple other smaller patches. Then started cleaning out workarounds, various dead code, all contrib and example code.
The result is a better performing and easier to maintain zlib-ng.

A lot of improvements have gone into zlib-ng since its start, and numerous people and companies have contributed both small and big improvements, or valuable testing.

Build

Please read LICENSE.md, it is very simple and very liberal.

There are two ways to build zlib-ng:

Cmake

To build zlib-ng using the cross-platform makefile generator cmake.

cmake .
cmake --build . --config Release
ctest --verbose -C Release

Alternatively, you can use the cmake configuration GUI tool ccmake:

ccmake .

Configure

To build zlib-ng using the bash configure script:

./configure
make
make test

Build Options

CMakeconfigureDescriptionDefault
ZLIB_COMPAT--zlib-compatCompile with zlib compatible APIOFF
ZLIB_ALIASESProvide zlib compatible CMake targetsON
WITH_GZFILEOP--without-gzfileopsCompile with support for gzFile related functionsON
WITH_OPTIM--without-optimizationsBuild with optimisationsON
WITH_NEW_STRATEGIES--without-new-strategiesUse new strategiesON
WITH_CRC32_CHORBABuild with Chorba optimized CRC32ON
WITH_REDUCED_MEM--with-reduced-memReduce zlib-ng memory usage, affects performance and compression ratioOFF
WITH_GTESTBuild tests using GTest frameworkON
WITH_BENCHMARKSBuild benchmarks using Google Benchmark frameworkOFF
INSTALL_UTILSCopy minigzip and minideflate during installOFF
BUILD_TESTINGBuild test binariesON

Install

WARNING: We do not recommend manually installing unless you really know what you are doing, because this can potentially override the system default zlib library, and any incompatibility or wrong configuration of zlib-ng can make the whole system unusable, requiring recovery or reinstall. If you still want a manual install, we recommend using the /opt/ path prefix.

For Linux distros, an alternative way to use zlib-ng (if compiled in zlib-compat mode) instead of zlib, is through the use of the LD_PRELOAD environment variable. If the program is dynamically linked with zlib, then the program will temporarily attempt to use zlib-ng instead, without risking system-wide instability.

LD_PRELOAD=/opt/zlib-ng/libz.so.1.2.13.zlib-ng /usr/bin/program

Cmake

To install zlib-ng system-wide using cmake:

cmake --build . --target install

Configure

To install zlib-ng system-wide using the configure script:

make install

CPack

After building with cmake, an installation package can be created using cpack. By default a tgz package is created, but you can append -G <format> to each command to generate alternative packages types (TGZ, ZIP, RPM, DEB). To easily create a rpm or deb package, you would use -G RPM or -G DEB respectively.

cd build
cpack --config CPackConfig.cmake
cpack --config CPackSourceConfig.cmake

Vcpkg

Alternatively, you can build and install zlib-ng using the vcpkg dependency manager:

git clone https://github.com/Microsoft/vcpkg.git
cd vcpkg
./bootstrap-vcpkg.sh # "./bootstrap-vcpkg.bat" for powershell
./vcpkg integrate install
./vcpkg install zlib-ng

The zlib-ng port in vcpkg is kept up to date by Microsoft team members and community contributors. If the version is out of date, please create an issue or pull request on the vcpkg repository.

Contributing

Zlib-ng is aiming to be open to contributions, and we would be delighted to receive pull requests on github. Help with testing and reviewing pull requests etc is also very much appreciated.

Please check the Wiki for more info: Contributing

Acknowledgments

Thanks go out to all the people and companies who have taken the time to contribute code reviews, testing and/or patches. Zlib-ng would not have been nearly as good without you.

The deflate format used by zlib was defined by Phil Katz.
The deflate and zlib specifications were written by L. Peter Deutsch.

zlib was originally created by Jean-loup Gailly (compression) and Mark Adler (decompression).

Advanced Build Options

CMakeconfigureDescriptionDefault
WITH_NATIVE_INSTRUCTIONSCompiles with full instruction set supported on this host (gcc/clang -march=native)OFF
WITH_RUNTIME_CPU_DETECTIONCompiles with runtime CPU detectionON
WITH_SSE2x86: Build with SSE2 intrinsicsON
WITH_SSSE3x86: Build with SSSE3 intrinsicsON
WITH_SSE41x86: Build with SSE41 intrinsicsON
WITH_SSE42x86: Build with SSE42 intrinsicsON
WITH_PCLMULQDQx86: Build with PCLMULQDQ intrinsicsON
WITH_AVX2x86: Build with AVX2 intrinsicsON
WITH_AVX512x86: Build with AVX512 intrinsicsON
WITH_AVX512VNNIx86: Build with AVX512VNNI intrinsicsON
WITH_VPCLMULQDQx86: Build with VPCLMULQDQ intrinsicsON
WITH_ARMV6--without-armv6arm: Build with ARMv6 intrinsicsON
WITH_ARMV8--without-armv8arm: Build with ARMv8 intrinsicsON
WITH_NEON--without-neonarm: Build with NEON intrinsicsON
WITH_ALTIVEC--without-altivecppc: Build with AltiVec (VMX) intrinsicsON
WITH_POWER8--without-power8ppc: Build with POWER8 intrinsicsON
WITH_POWER9--without-power9ppc: Build with POWER9 intrinsicsON
WITH_RVV--without-rvvriscv: Build with RVV intrinsicsON
WITH_RISCV_ZBC--without-zbcriscv: Build with RiscV ZBC intrinsicsON
WITH_S390_VX--without-s390-vxs390x: Build with S390 VX intrinsics on IBM ZON
WITH_DFLTCC_DEFLATE--with-dfltcc-deflates390x: Build with DFLTCC intrinsics for compression on IBM ZOFF
WITH_DFLTCC_INFLATE--with-dfltcc-inflates390x: Build with DFLTCC intrinsics for decompression on IBM ZOFF
WITH_LSXloongarch: Build with LSX intrinsicsON
WITH_CRC32_LA--without-crc32-laloongarch: Build with vectorized CRC32ON
WITH_INFLATE_STRICTBuild with strict inflate distance checkingOFF
WITH_INFLATE_ALLOW_INVALID_DISTBuild with zero fill for inflate invalid distancesOFF
WITH_BENCHMARK_APPSBuild benchmark apps (currently libpng)OFF
WITH_ALL_FALLBACKSBuild with all c-fallbacks (useful for Gbench comparisons)OFF
WITH_MAINTAINER_WARNINGSBuild with project maintainer warningsOFF
WITH_SANITIZERBuild with sanitizer (memory, address, undefined)OFF
WITH_FUZZERSBuild test/fuzzOFF
WITH_CODE_COVERAGEEnable code coverage reportingOFF

Related Projects