Skip To Content
Toggle navigation
Packages
Upload
Statistics
Documentation
Downloads
Blog
Sign in
Advanced search filters
Frameworks
Include compatible frameworks
Framework Filter Mode
ALL
ANY
.NET
net8.0
net7.0
net6.0
net5.0
.NET Core
netcoreapp3.1
netcoreapp3.0
netcoreapp2.2
netcoreapp2.1
netcoreapp2.0
netcoreapp1.1
netcoreapp1.0
.NET Standard
netstandard2.1
netstandard2.0
netstandard1.6
netstandard1.5
netstandard1.4
netstandard1.3
netstandard1.2
netstandard1.1
netstandard1.0
.NET Framework
net481
net48
net472
net471
net47
net462
net461
net46
net452
net451
net45
net40
net35
net30
net20
Package type
All types
Dependency
.NET tool
Template
Options
Include prerelease
47 packages returned for Tags:"Tokenizer"
Sort by
Relevance
Downloads
Recently updated
Universal.
OpenAI
by:
ong.andrew
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
326 total downloads
last updated
3/9/2023
Latest version:
1.0.0
Universal
OpenAI
Token
Tokenizer
GPT-2
Class library containing utilities that are useful in conjunction with OpenAI's models and APIs.
Ccr.
Parsing
by:
vreniose95
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
2,206 total downloads
last updated
5/13/2020
Latest version:
3.1.0
Ccr
Ccr
Framework
Ccr.Parsing
Ccr
Parsing
Parsing
Tokenizing
Tokenizer
Lexing
More tags
Ccr.Parsing package for writing parsers, lexers, tokenizers, etc.
BytecodeApi.
Lexer
by:
bytecode77
.NET 7.0
251 total downloads
last updated
9/27/2023
Latest version:
3.0.1
lexer
tokenizer
parse
tree
syntax
grammar
token
General purpose tokenizer for parsing of any syntax.
Forge.
OpenAI.
GPT
by:
JZO
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
129 total downloads
last updated
3/3/2024
Latest version:
1.0.3
OpenAI
GPT
GPT3
Tokenizer
Tokens
AI
OpenAI GPT utils, e.g. GPT3 Tokenizer
PortedFastBertTokenizer
by:
siew24
.NET 5.0
.NET Core 3.0
.NET Standard 2.1
175 total downloads
last updated
11/24/2023
Latest version:
0.4.6-beta
bert
tokenizer
wordpiece
llm
semantic-kernel
performance
ai
artificial-intelligence
ml
machine-learning
Fast and memory-efficient WordPiece tokenizer as it is used by BERT and others. Tokenizes text for further processing using NLP/language models. This fork only supports .NET Standard 2.1
VisualFA.
Compiler
by:
honey.the.codewitch
.NET 6.0
172 total downloads
last updated
4/24/2024
Latest version:
1.1.0
finite
automata
compiler
regex
regular
expression
lexer
tokenizer
Adds dynamic compilation support for the Visual FA runtime
vforteli.
WordPieceTokenizer
by:
vforteli
.NET 8.0
45 total downloads
last updated
4/29/2024
Latest version:
0.1.0
llm
bert
dotnet
.net
c#
ml.net
wordpiece
tokenizer
Wordpiece tokenizer for using eg onnx models with ML.Net
Previous
Next