LittleLittleCloud.TorchSharp.BitsAndBytes
0.0.4
dotnet add package LittleLittleCloud.TorchSharp.BitsAndBytes --version 0.0.4
NuGet\Install-Package LittleLittleCloud.TorchSharp.BitsAndBytes -Version 0.0.4
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="LittleLittleCloud.TorchSharp.BitsAndBytes" Version="0.0.4" />
For projects that support PackageReference, copy this XML node into the project file to reference the package.
<PackageVersion Include="LittleLittleCloud.TorchSharp.BitsAndBytes" Version="0.0.4" />
<PackageReference Include="LittleLittleCloud.TorchSharp.BitsAndBytes" />
For projects that support Central Package Management (CPM), copy this XML node into the solution Directory.Packages.props file to version the package.
paket add LittleLittleCloud.TorchSharp.BitsAndBytes --version 0.0.4
The NuGet Team does not provide support for this client. Please contact its maintainers for support.
#r "nuget: LittleLittleCloud.TorchSharp.BitsAndBytes, 0.0.4"
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
#addin nuget:?package=LittleLittleCloud.TorchSharp.BitsAndBytes&version=0.0.4
#tool nuget:?package=LittleLittleCloud.TorchSharp.BitsAndBytes&version=0.0.4
The NuGet Team does not provide support for this client. Please contact its maintainers for support.
TorchSharp.BitsAndBytes
The TorchSharp.BitsAndBytes is a C# binding library for bitsandbytes library from Huggingface. It provides 4Bit and 8Bit quantization for TorchSharp models.
Usage
4Bit Quantization && Dequantization
[!NOTE] 4Bit quantization is only available for CUDA devices.
var input = torch.rand([dim * 4, dim], dtype: ScalarType.Float32).cuda(); // FP32 tensor, must be on cuda device
string quantizedDType = "fp4"; // Available options: "fp4", "nf4"
int blockSize = 64; // can be [64, 128, 256, 512, 1024]
// Quantize to 4Bit
(var quantizedTensor, var absMax, blockSize, var n) = BitsAndByteUtils.Quantize4Bit(input, quantizedDType, blockSize);
// Dequantize to FP32
var dequantizedTensor = BitsAndByteUtils.Dequantize4Bit(quantiedTensor, absMax, input.dtype, quantizedDType, n, input.shape, blockSize);
| Product | Versions Compatible and additional computed target framework versions. |
|---|---|
| .NET | net8.0 is compatible. net8.0-android was computed. net8.0-browser was computed. net8.0-ios was computed. net8.0-maccatalyst was computed. net8.0-macos was computed. net8.0-tvos was computed. net8.0-windows was computed. net9.0 was computed. net9.0-android was computed. net9.0-browser was computed. net9.0-ios was computed. net9.0-maccatalyst was computed. net9.0-macos was computed. net9.0-tvos was computed. net9.0-windows was computed. |
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.
-
net8.0
- No dependencies.
NuGet packages
This package is not used by any NuGet packages.
GitHub repositories
This package is not used by any popular GitHub repositories.