DotFastLZ.Compression 2024.1.1

There is a newer version of this package available.
See the version list below for details.

dotnet add package DotFastLZ.Compression --version 2024.1.1

NuGet\Install-Package DotFastLZ.Compression -Version 2024.1.1

This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.

<PackageReference Include="DotFastLZ.Compression" Version="2024.1.1" />

For projects that support PackageReference, copy this XML node into the project file to reference the package.

paket add DotFastLZ.Compression --version 2024.1.1

The NuGet Team does not provide support for this client. Please contact its maintainers for support.

#r "nuget: DotFastLZ.Compression, 2024.1.1"

#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.

// Install DotFastLZ.Compression as a Cake Addin
#addin nuget:?package=DotFastLZ.Compression&version=2024.1.1

// Install DotFastLZ.Compression as a Cake Tool
#tool nuget:?package=DotFastLZ.Compression&version=2024.1.1

The NuGet Team does not provide support for this client. Please contact its maintainers for support.

<h1 align="center">DotFastLZ</h1> <p align="center"><i> DotFastLZ is C# fastlz, a port of <a href="https://github.com/ariya/FastLZ">ariya/FastLZ</a><br/> DotFastLZ can be used in Unity3D, C# server, network packet, game data </i></p>

Usage: DotFastLZ.Compression

for (int level = 1; level <= 2; ++level)
{
    // compress
    var input = GetInputSource();
    var estimateSize = FastLZ.EstimateCompressedSize(input.Length);
    var comBuf = new byte[estimateSize];
    var comBufSize = FastLZ.CompressLevel(level, input, input.Length, comBuf);

    // decompress
    byte[] decBuf = new byte[input.Length];
    var decBufSize = FastLZ.Decompress(comBuf, comBufSize, decBuf, decBuf.Length);

    // compare
    var compareSize = FastLZ.MemCompare(input, 0, decBuf, 0, decBufSize);

    // check
    Assert.That(decBufSize, Is.EqualTo(input.Length), "decompress size error");
    Assert.That(compareSize, Is.EqualTo(input.Length), "decompress compare error");
}

Usage: DotFastLZ.Compression.Packaging

const string targetFileName = "soruce.txt";
string packagingFileName = targetFileName + ".fastlz";

// pack/unpack
SixPack.PackFile(2, targetFileName, packagingFileName, Console.Write);
SixPack.UnpackFile(packagingFileName, Console.Write);

Usage: DotFastLZ.Packaging.Tools

$ dotnet tool install --global 6pack
$ dotnet tool 6pack --help

6pack: high-speed file compression tool
Copyright (C) Ariya Hidayat, Choi Ikpil(ikpil@naver.com)
 - https://github.com/ikpil/DotFastLZ

Usage: 6pack [options] input-file output-file

Options:
  -1    compress faster
  -2    compress better
  -v    show program version
  -d    decompression (default for .fastlz extension)
  -mem  check in-memory compression speed

Overview

FastLZ (MIT license) is an ANSI C/C90 implementation of Lempel-Ziv 77 algorithm (LZ77) of lossless data compression. It is suitable to compress series of text/paragraphs, sequences of raw pixel data, or any other blocks of data with lots of repetition. It is not intended to be used on images, videos, and other formats of data typically already in an optimal compressed form.

The focus for FastLZ is a very fast compression and decompression, doing that at the cost of the compression ratio. As an illustration, the comparison with zlib when compressing enwik8 (also in more details):

	Ratio	Compression	Decompression
FastLZ	54.2%	159 MB/s	305 MB/s
zlib -1	42.3%	50 MB/s	184 MB/s
zlib -9	36.5%	11 MB/s	185 MB/s

FastLZ is used by many software products, from a number of games (such as Death Stranding) to various open-source projects (Godot Engine, Facebook HHVM, Apache Traffic Server, Calligra Office, OSv, Netty, etc). It even serves as the basis for other compression projects like BLOSC.

For other implementations of byte-aligned LZ77, take a look at LZ4, Snappy, Density, LZO, LZF, LZJB, LZRW, etc.

Block Format

Let us assume that FastLZ compresses an array of bytes, called the uncompressed block, into another array of bytes, called the compressed block. To understand what will be stored in the compressed block, it is illustrative to demonstrate how FastLZ will decompress the block to retrieve the original uncompressed block.

The first 3-bit of the block, i.e. the 3 most-significant bits of the first byte, is the block tag. Currently the block tag determines the compression level used to produce the compressed block.

Block tag	Compression level
0	Level 1
1	Level 2

The content of the block will vary depending on the compression level.

Block Format for Level 1

FastLZ Level 1 implements LZ77 compression algorithm with 8 KB sliding window and up to 264 bytes of match length.

The compressed block consists of one or more instructions. Each instruction starts with a 1-byte opcode, 2-byte opcode, or 3-byte opcode.

Instruction type	Opcode[0]	Opcode[1]	Opcode[2]
Literal run	`000`, L₅-L₀	-	-
Short match	M₂-M₀, R₁₂-R₈	R₇-R₀	-
Long match	`111`, R₁₂-R₈	M₇-M₀	R₇-R₀

Note that the very first instruction in a compressed block is always a literal run.

Literal run instruction

For the literal run instruction, there is one or more bytes following the code. This is called the literal run.

The 5 least-significant bits of opcode[0], L, determines the number of literals following the opcode. The value of 0 indicates a 1-byte literal run, 1 indicates a 2-byte literal run, and so on. The minimum literal run is 1 and the maximum literal run is 32.

The decompressor copies (L + 1) bytes of literal run, starting from the first one right after opcode.

Example: If the compressed block is a 4-byte array of [0x02, 0x41, 0x42, 0x43], then the opcode is 0x02 and that means a literal run of 3 bytes. The decompressor will then copy the subsequent 3 bytes, [0x41, 0x42, 0x43], to the output buffer. The output buffer now represents the (original) uncompressed block, [0x41, 0x42, 0x43].

Short match instruction

The 3 most-significant bits of opcode[0], M, determines the match length. The value of 1 indicates a 3-byte match, 2 indicates a 4-byte match and so on. The minimum match length is 3 and the maximum match length is 8.

The 5 least-significant bits of opcode[0] combined with the 8 bits of the opcode[1], R, determines the reference offset. Since the offset is encoded in 13 bits, the minimum is 0 and the maximum is 8191.

The following C code retrieves the match length and reference offset:

M = opcode[0] >> 5;
R = 256 * (opcode[0] << 5) + opcode[1];

The decompressor copies (M+2) bytes, starting from the location offsetted by R in the output buffer. Note that R is a back reference, i.e. the value of 0 corresponds the last byte in the output buffer, 1 is the second to last byte, and so forth.

Example 1: If the compressed block is a 7-byte array of [0x03, 0x41, 0x42, 0x43, 0x44, 0x20, 0x02], then there are two instructions in the there. The first instruction is the literal run of 4 bytes (due to L = 3). Thus, the decompressor copies 4 bytes to the output buffer, resulting in [0x41, 0x42, 0x43, 0x44]. The second instruction is the short match of 3 bytes (from M = 1, i.e 0x20 >> 5) and the offset of 2. Therefore, the compressor goes back 2 bytes from the last position, copies 3 bytes ([0x42, 0x43, 0x44]), and appends them to the output buffer. The output buffer now represents the complete uncompressed data, [0x41, 0x42, 0x43, 0x44, 0x42, 0x43, 0x44].

Example 2: If the compressed block is a 4-byte array of [0x00, 0x61, 0x40, 0x00], then there are two instructions in there. The first instruction is the literal run of just 1 byte (L = 0). Thus, the decompressor copies the byte (0x61) to the output buffer. The output buffer now becomes [0x61]. The second instruction is the short match of 4 bytes (from M = 2, i.e. 0x40 >> 5) and the offset of 0. Therefore, the decompressor copies 4 bytes starting using the back reference of 0 (i.e. the position of 0x61). The output buffer now represents the complete uncompressed data, [0x61, 0x61, 0x61, 0x61, 0x61].

Long match instruction

The value of opcode[1], M, determines the match length. The value of 0 indicates a 9-byte match, 1 indicates a 10-byte match and so on. The minimum match length is 9 and the maximum match length is 264.

The 5 least-significant bits of opcode[0] combined with the 8 bits of opcode[2], R, determines the reference offset. Since the offset is encoded in 13 bits, the minimum is 0 and the maximum is 8191.

The following C code retrieves the match length and reference offset:

M = opcode[1];
R = 256 * (opcode[0] << 5) + opcode[2];

The decompressor copies (M+9) bytes, starting from the location offsetted by R in the output buffer. Note that R is a back reference, i.e. the value of 0 corresponds to the last byte in the output buffer, 1 is for the second to last byte, and so forth.

Example: If the compressed block is a 4-byte array of [0x01, 0x44, 0x45, 0xE0, 0x01, 0x01], then there are two instructions in there. The first instruction is the literal run with the length of 2 (due to L = 1). Thus, the decompressor copies the 2-byte literal run ([0x44, 0x45]) to the output buffer. The second instruction is the long match with the match length of 10 (from M = 1) and the offset of 1. Therefore, the decompressor copies 10 bytes starting using the back reference of 1 (i.e. the position of 0x44). The output buffer now represents the complete uncompressed data, [0x44, 0x45, 0x44, 0x45, 0x44, 0x45, 0x44, 0x45, 0x44, 0x45, 0x44, 0x45].

Decompressor Reference Implementation

The following 40-line C function implements a fully-functional decompressor for the above block format. Note that it is intended to be educational, e.g. no bound check is implemented, and therefore it is absolutely unsafe for production.

void fastlz_level1_decompress(const uint8_t* input, int length, uint8_t* output) {
  int src = 0;
  int dest = 0;
  while (src < length) {
    int type = input[src] >> 5;
    if (type == 0) {
      /* literal run */
      int run = 1 + input[src];
      src = src + 1;
      while (run > 0) {
        output[dest] = input[src];
        src = src + 1;
        dest = dest + 1;
        run = run - 1;
      }
    } else if (type < 7) {
      /* short match */
      int ofs = 256 * (input[src] & 31) + input[src + 1];
      int len = 2 + (input[src] >> 5);
      src = src + 2;
      int ref = dest - ofs - 1;
      while (len > 0) {
        output[dest] = output[ref];
        ref = ref + 1;
        dest = dest + 1;
        len = len - 1;
      }
    } else {
      /* long match */
      int ofs = 256 * (input[src] & 31) + input[src + 2];
      int len = 9 + input[src + 1];
      src = src + 3;
      int ref = dest - ofs - 1;
      while (len > 0) {
        output[dest] = output[ref];
        ref = ref + 1;
        dest = dest + 1;
        len = len - 1;
      }
    }
  }
}

Product	Compatible and additional computed target framework versions.
.NET	net5.0 was computed. net5.0-windows was computed. net6.0 is compatible. net6.0-android was computed. net6.0-ios was computed. net6.0-maccatalyst was computed. net6.0-macos was computed. net6.0-tvos was computed. net6.0-windows was computed. net7.0 is compatible. net7.0-android was computed. net7.0-ios was computed. net7.0-maccatalyst was computed. net7.0-macos was computed. net7.0-tvos was computed. net7.0-windows was computed. net8.0 is compatible. net8.0-android was computed. net8.0-browser was computed. net8.0-ios was computed. net8.0-maccatalyst was computed. net8.0-macos was computed. net8.0-tvos was computed. net8.0-windows was computed.
.NET Core	netcoreapp2.0 was computed. netcoreapp2.1 was computed. netcoreapp2.2 was computed. netcoreapp3.0 was computed. netcoreapp3.1 was computed.
.NET Standard	netstandard2.0 is compatible. netstandard2.1 is compatible.
.NET Framework	net461 was computed. net462 was computed. net463 was computed. net47 was computed. net471 was computed. net472 was computed. net48 was computed. net481 was computed.
MonoAndroid	monoandroid was computed.
MonoMac	monomac was computed.
MonoTouch	monotouch was computed.
Tizen	tizen40 was computed. tizen60 was computed.
Xamarin.iOS	xamarinios was computed.
Xamarin.Mac	xamarinmac was computed.
Xamarin.TVOS	xamarintvos was computed.
Xamarin.WatchOS	xamarinwatchos was computed.

Compatible target framework(s)

Included target framework(s) (in package)

Learn more about Target Frameworks and .NET Standard.

.NETStandard 2.0
- No dependencies.
.NETStandard 2.1
- No dependencies.
net6.0
- No dependencies.
net7.0
- No dependencies.
net8.0
- No dependencies.

NuGet packages (1)

Showing the top 1 NuGet packages that depend on DotFastLZ.Compression:

Package	Downloads
DotFastLZ.Packaging DotFastLZ - a port of FastLZ, Small and portable byte-aligned LZ77 compression for C#, Unity3D	1.1K

GitHub repositories

This package is not used by any popular GitHub repositories.

Version	Downloads	Last updated
2024.1.2	474	3/2/2024
2024.1.1	157	3/2/2024
2023.10.5	570	10/29/2023
2023.10.4	189	10/20/2023
2023.10.3	183	10/20/2023
2023.10.2	204	10/3/2023
2023.10.1	176	10/3/2023
2023.8.6.1	168	8/6/2023
2023.8.6	163	8/6/2023