Google Git
Sign in
fuchsia / third_party / github.com / zlib-ng / zlib-ng / 267c3ea68c4cf1492cec706fbfa1cacea6a3f570
commit267c3ea68c4cf1492cec706fbfa1cacea6a3f570[log] [tgz]
authorUnknown Author <devnull@fuchsia.dev>Thu Oct 13 14:59:18 2022 +0100
committerVinny <felizardo@google.com>Wed Nov 09 18:54:26 2022 -0800
tree8af724096ff6b6a4a79ca152184573165d7e9cc0
parent30df9f9364ca736c864f4201a683d1f609b38cbf [diff]
Project import generated by Copybara.

GitOrigin-RevId: b3dcf11b4204a16fde71fa8224d7b7054e225b93
Change-Id: I8061e59c4a423d98b5e5db572cebe992d8f66106
289 files changed
tree: 8af724096ff6b6a4a79ca152184573165d7e9cc0
  1. .github/
  2. arch/
  3. cmake/
  4. doc/
  5. test/
  6. tools/
  7. win32/
  8. .codecov.yaml
  9. .gitattributes
  10. .gitignore
  11. .shellcheckrc
  12. adler32.c
  13. adler32_fold.c
  14. adler32_fold.h
  15. adler32_p.h
  16. chunkset.c
  17. chunkset_tpl.h
  18. CMakeLists.txt
  19. compare256.c
  20. compress.c
  21. configure
  22. cpu_features.c
  23. cpu_features.h
  24. crc32_braid.c
  25. crc32_braid_comb.c
  26. crc32_braid_comb_p.h
  27. crc32_braid_p.h
  28. crc32_braid_tbl.h
  29. crc32_fold.c
  30. crc32_fold.h
  31. deflate.c
  32. deflate.h
  33. deflate_fast.c
  34. deflate_huff.c
  35. deflate_medium.c
  36. deflate_p.h
  37. deflate_quick.c
  38. deflate_rle.c
  39. deflate_slow.c
  40. deflate_stored.c
  41. fallback_builtins.h
  42. FAQ.zlib
  43. functable.c
  44. functable.h
  45. gzguts.h
  46. gzlib.c
  47. gzread.c.in
  48. gzwrite.c
  49. INDEX.md
  50. infback.c
  51. inffast.c
  52. inffast.h
  53. inffixed_tbl.h
  54. inflate.c
  55. inflate.h
  56. inflate_p.h
  57. inftrees.c
  58. inftrees.h
  59. insert_string.c
  60. insert_string_roll.c
  61. insert_string_tpl.h
  62. LICENSE.md
  63. Makefile.in
  64. match_tpl.h
  65. PORTING.md
  66. README.md
  67. slide_hash.c
  68. trees.c
  69. trees.h
  70. trees_emit.h
  71. trees_tbl.h
  72. uncompr.c
  73. zbuild.h
  74. zconf-ng.h.in
  75. zconf.h.in
  76. zendian.h
  77. zlib-ng.h.in
  78. zlib-ng.map
  79. zlib.h.in
  80. zlib.map
  81. zlib.pc.cmakein
  82. zlib.pc.in
  83. zlib_name_mangling-ng.h.in
  84. zlib_name_mangling.h.empty
  85. zlib_name_mangling.h.in
  86. zutil.c
  87. zutil.h
  88. zutil_p.h
README.md

zlib-ng

zlib data compression library for the next generation systems

Maintained by Hans Kristian Rosbach aka Dead2 (zlib-ng àt circlestorm dót org)

CIStatus
GitHub ActionsMaster Branch Status Master Branch Status Master Branch Status
BuildkiteBuild status
CodeFactorCodeFactor
OSS-FuzzFuzzing Status
Codecovcodecov.io

Features

  • Zlib compatible API with support for dual-linking
  • Modernized native API based on zlib API for ease of porting
  • Modern C11 syntax and a clean code layout
  • Deflate medium and quick algorithms based on Intel’s zlib fork
  • Support for CPU intrinsics when available
    • Adler32 implementation using SSSE3, AVX2, AVX512, AVX512-VNNI, Neon, VMX & VSX
    • CRC32-B implementation using PCLMULQDQ, VPCLMULQDQ, ACLE, & IBM Z
    • Hash table implementation using CRC32-C intrinsics on x86 and ARM
    • Slide hash implementations using SSE2, AVX2, Neon, VMX & VSX
    • Compare256 implementations using SSE2, AVX2, Neon, & POWER9
    • Inflate chunk copying using SSE2, AVX, Neon & VSX
    • Support for hardware-accelerated deflate using IBM Z DFLTCC
  • Unaligned memory read/writes and large bit buffer improvements
  • Includes improvements from Cloudflare and Intel forks
  • Configure, CMake, and NMake build system support
  • Comprehensive set of CMake unit tests
  • Code sanitizers, fuzzing, and coverage
  • GitHub Actions continuous integration on Windows, macOS, and Linux
    • Emulated CI for ARM, AARCH64, PPC, PPC64, SPARC64, S390x using qemu

History

The motivation for this fork came after seeing several 3rd party contributions containing new optimizations not getting implemented into the official zlib repository.

Mark Adler has been maintaining zlib for a very long time, and he has done a great job and hopefully he will continue for a long time yet. The idea of zlib-ng is not to replace zlib, but to co-exist as a drop-in replacement with a lower threshold for code change.

zlib has a long history and is incredibly portable, even supporting lots of systems that predate the Internet. This is great, but it does complicate further development and maintainability. The zlib code has numerous workarounds for old compilers that do not understand ANSI-C or to accommodate systems with limitations such as operating in a 16-bit environment.

Many of these workarounds are only maintenance burdens, some of them are pretty huge code-wise. For example, the [v]s[n]printf workaround code has a whopping 8 different implementations just to cater to various old compilers. With this many workarounds cluttered throughout the code, new programmers with an idea/interest for zlib will need to take some time to figure out why all of these seemingly strange things are used, and how to work within those confines.

So I decided to make a fork, merge all the Intel optimizations, merge the Cloudflare optimizations that did not conflict, plus a couple of other smaller patches. Then I started cleaning out workarounds, various dead code, all contrib and example code as there is little point in having those in this fork for various reasons.

A lot of improvements have gone into zlib-ng since its start, and numerous people and companies have contributed both small and big improvements, or valuable testing.

Please read LICENSE.md, it is very simple and very liberal.

Build

There are two ways to build zlib-ng:

Cmake

To build zlib-ng using the cross-platform makefile generator cmake.

cmake .
cmake --build . --config Release
ctest --verbose -C Release

Alternatively, you can use the cmake configuration GUI tool ccmake:

ccmake .

Configure

To build zlib-ng using the bash configure script:

./configure
make
make test

Build Options

CMakeconfigureDescriptionDefault
ZLIB_COMPAT--zlib-compatCompile with zlib compatible APIOFF
ZLIB_ENABLE_TESTSBuild test binariesON
WITH_GZFILEOP--without-gzfileopsCompile with support for gzFile related functionsON
WITH_OPTIM--without-optimizationsBuild with optimisationsON
WITH_NEW_STRATEGIES--without-new-strategiesUse new strategiesON
WITH_NATIVE_INSTRUCTIONS--nativeCompiles with full instruction set supported on this host (gcc/clang -march=native)OFF
WITH_SANITIZERBuild with sanitizer (memory, address, undefined)OFF
WITH_FUZZERSBuild test/fuzzOFF
WITH_BENCHMARKSBuild test/benchmarksOFF
WITH_MAINTAINER_WARNINGSBuild with project maintainer warningsOFF
WITH_CODE_COVERAGEEnable code coverage reportingOFF

Install

WARNING: We do not recommend manually installing unless you really know what you are doing, because this can potentially override the system default zlib library, and any incompatibility or wrong configuration of zlib-ng can make the whole system unusable, requiring recovery or reinstall. If you still want a manual install, we recommend using the /opt/ path prefix.

For Linux distros, an alternative way to use zlib-ng (if compiled in zlib-compat mode) instead of zlib, is through the use of the LD_PRELOAD environment variable. If the program is dynamically linked with zlib, then zlib-ng will temporarily be used instead by the program, without risking system-wide instability.

LD_PRELOAD=/opt/zlib-ng/libz.so.1.2.12.zlib-ng /usr/bin/program

Cmake

To install zlib-ng system-wide using cmake:

cmake --build . --target install

Configure

To install zlib-ng system-wide using the configure script:

make install

Vcpkg

Alternatively, you can build and install zlib-ng using the vcpkg dependency manager:

git clone https://github.com/Microsoft/vcpkg.git
cd vcpkg
./bootstrap-vcpkg.sh # "./bootstrap-vcpkg.bat" for powershell
./vcpkg integrate install
./vcpkg install zlib-ng

The zlib-ng port in vcpkg is kept up to date by Microsoft team members and community contributors. If the version is out of date, please create an issue or pull request on the vcpkg repository.

Contributing

Zlib-ng is a aiming to be open to contributions, and we would be delighted to receive pull requests on github. Just remember that any code you submit must be your own and it must be zlib licensed. Help with testing and reviewing of pull requests etc is also very much appreciated.

If you are interested in contributing, please consider joining our IRC channel #zlib-ng on the Freenode IRC network.

Acknowledgments

Thanks to Servebolt.com for sponsoring my maintainership of zlib-ng.

Thanks go out to all the people and companies who have taken the time to contribute code reviews, testing and/or patches. Zlib-ng would not have been nearly as good without you.

The deflate format used by zlib was defined by Phil Katz. The deflate and zlib specifications were written by L. Peter Deutsch.

zlib was originally created by Jean-loup Gailly (compression) and Mark Adler (decompression).

Advanced Build Options

CMakeconfigureDescriptionDefault
FORCE_SSE2--force-sse2Skip runtime check for SSE2 instructions (Always on for x86_64)OFF (x86)
FORCE_TZCNT--force-tzcntSkip runtime check for TZCNT instructionsOFF
WITH_AVX2Build with AVX2 intrinsicsON
WITH_AVX512Build with AVX512 intrinsicsON
WITH_AVX512VNNIBuild with AVX512VNNI intrinsicsON
WITH_SSE2Build with SSE2 intrinsicsON
WITH_SSE41Build with SSE41 intrinsicsON
WITH_SSE42Build with SSE42 intrinsicsON
WITH_PCLMULQDQBuild with PCLMULQDQ intrinsicsON
WITH_VPCLMULQDQ--without-vpclmulqdqBuild with VPCLMULQDQ intrinsicsON
WITH_ACLE--without-acleBuild with ACLE intrinsicsON
WITH_NEON--without-neonBuild with NEON intrinsicsON
WITH_ALTIVEC--without-altivecBuild with AltiVec (VMX) intrinsicsON
WITH_POWER8--without-power8Build with POWER8 optimisationsON
WITH_CRC32_VX--without-crc32-vxBuild with vectorized CRC32 on IBM ZON
WITH_DFLTCC_DEFLATE--with-dfltcc-deflateBuild with DFLTCC intrinsics for compression on IBM ZOFF
WITH_DFLTCC_INFLATE--with-dfltcc-inflateBuild with DFLTCC intrinsics for decompression on IBM ZOFF
WITH_UNALIGNED--without-unalignedAllow optimizations that use unaligned reads if safe on current archON
WITH_INFLATE_STRICTBuild with strict inflate distance checkingOFF
WITH_INFLATE_ALLOW_INVALID_DISTBuild with zero fill for inflate invalid distancesOFF
INSTALL_UTILSCopy minigzip and minideflate during installOFF
ZLIBNG_ENABLE_TESTSTest zlib-ng specific APION

Related Projects