powturbo/TurboPFor-Integer-Compression

Free

Fastest Integer Compression

FreeFree tier

Type

Open Source

Company

powturbo

Visit Website GitHub (38 stars)

About powturbo/TurboPFor-Integer-Compression

TurboPFor is a high-performance integer compression library written in C, providing the fastest known integer compression and decompression routines. It supports full-range 8/16/32/64 bit integers via scalar and SIMD (SSE, AVX2, NEON) codecs on x86, ARMv8, and Power9 architectures. The library includes a wide variety of compression schemes: PFor/PForDelta, variable byte, bit packing, Simple family, Elias Fano, TurboVLC, hybrid TurboBitByte, and more. It features direct/random access to compressed data without full decompression, integrated delta/zigzag/xor transforms, and special optimizations for time series (Gorilla-style) and floating-point compression. Bindings are available for Rust and Java, allowing near-native performance. TurboPFor is designed for use in databases, search engines, time-series systems, and any application requiring high-throughput integer data compression.

Key Features

Fastest integer compression and decompression performance

SIMD-accelerated codecs: SSE/AVX2 (x86), NEON (ARMv8), Altivec (Power9)

Full range support: 8/16/32/64 bit integer widths, both scalar and SIMD

Multiple compression schemes: PFor/PForDelta, variable byte, bit packing, Simple family, Elias Fano, TurboVLC, TurboBitByte

Direct/random access to compressed entries without decompression

Integrated delta, zigzag, XOR, and transpose transforms

Time series compression via Gorilla-style encoding with RLE

Floating-point compression using TurboFloat and LzXor

Rust and Java bindings with JNI for near-native speed

Cross-platform: Linux (amd64, arm64, Power9), macOS (Intel + Apple M1), RISC-V (scalar path)

Pros & Cons

Pros

Claimed to be fastest integer compression library available
Extensive SIMD support for modern CPUs
Direct access capability reduces decompression overhead
Wide range of codecs for different data patterns
Simple C API with optional high-level bindings
Free and open source under permissive license

Cons

Primarily focused on integer data; not a general-purpose compression library
SIMD code paths require modern hardware (e.g., AVX2, NEON) for peak performance
RISC-V support currently limited to scalar path without SIMD optimizations
Learning curve for selecting appropriate codec for specific data
Documentation appears mostly in source code comments and README

Best For

Inverted index compression for search enginesTime series database compressionDatabase columnar storage (integers, timestamps)High-speed data serialization and packingScientific computing with large integer arraysNetwork packet payload compression

FAQ

What architectures are supported?

TurboPFor supports x86 (SSE/AVX2), ARMv8 (NEON), Power9 (Altivec), and RISC-V (scalar path) on Linux, macOS (Intel and Apple M1), and Windows (via Visual Studio).

Does TurboPFor support floating-point compression?

Yes, it provides multiple floating-point compression methods: TurboFloat for efficient integer-based compression, LzXor for Lempel-Ziv style, and Delta/Zigzag + improved Gorilla and FCM/DFCM schemes.

Can I use TurboPFor from programming languages other than C?

Yes, there are official bindings for Rust (via GitHub package) and Java (via JNI) that allow near-native performance.

Is TurboPFor free to use?

Yes, it is open source under a permissive license (no specific license stated in the provided content, but typically MIT or BSD as common for such libraries; the repository contains a LICENSE file).

What is the typical compression ratio?

Compression ratio depends on data characteristics, but TurboPFor is designed to balance speed and compression, often outperforming other fast integer compression schemes in speed while maintaining competitive ratios.