Skip To Content
Toggle navigation
Packages
Upload
Statistics
Documentation
Downloads
Blog
Sign in
Advanced search filters
Frameworks
Include compatible frameworks
Framework Filter Mode
ALL
ANY
.NET
net8.0
net7.0
net6.0
net5.0
.NET Core
netcoreapp3.1
netcoreapp3.0
netcoreapp2.2
netcoreapp2.1
netcoreapp2.0
netcoreapp1.1
netcoreapp1.0
.NET Standard
netstandard2.1
netstandard2.0
netstandard1.6
netstandard1.5
netstandard1.4
netstandard1.3
netstandard1.2
netstandard1.1
netstandard1.0
.NET Framework
net481
net48
net472
net471
net47
net462
net461
net46
net452
net451
net45
net40
net35
net30
net20
Package type
All types
Dependency
.NET tool
Template
Options
Include prerelease
46 packages returned for Tags:"Tokenizer"
Sort by
Relevance
Downloads
Recently updated
TokenCs
by:
zeplar_exe
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
2,208 total downloads
last updated
8/10/2022
Latest version:
1.0.5
token
tokenizer
lexer
parse
parser
A simple library to abstract the gritty process of lexical parsing.
NReco.
NLQuery
by:
nreco
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.5
9,800 total downloads
last updated
8/29/2022
Latest version:
1.1.1
NLP
NER
NLQ
search
search-interface
natural
language
query
named
entity
More tags
NLQuery: natural language query parser recognizes entities in context of structured sources (like tabular dataset). Can be used for building natural language interface to SQL database or OLAP cube, implementing...
More information
LLMSharp.
OpenAi.
Tokenizer
by:
veerash-ayyagari
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
4,019 total downloads
last updated
10/4/2023
Latest version:
2.0.3
openai
gpt
3.5
gpt
4
tokenizer
dotnet
tiktoken
Open AI Chat Completion Models (GPT 3.5/GPT 4) BPE Tokenizer unofficial implementation
Brainfuck-
Runner
by:
nikolayresh
.NET 5.0
9,701 total downloads
last updated
11/9/2021
Latest version:
1.0.14
brainfuck
brain
brainfuck
interpreter
brainfuck
tokenizer
brainfuck
validator
brainfuck
engine
More tags
Powerful Brainfuck language interpreter with ability to tokenize & validate the Brainfuck code
XLemmatizer
by:
rchristen
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
838 total downloads
last updated
5/29/2020
Latest version:
0.1.0
nlp
lemmatizer
tokenizer
Pre-release version. API might change later. A lemma is the canonical form of the word. For example, the words "run", "runs", "ran" and "running" can be lemmatized to "run" XLemmatizer tokenizes and lemmatizes...
More information
GParse
by:
gggkiller
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
2,206 total downloads
last updated
6/15/2021
Latest version:
5.0.0-alpha.10
parser
lexer
parsing
lexing
tokenizer
tokenizing
Parsing and lexing utilities to create your own parser and lexer
GParse.
Extensions.
StateMachines
by:
gggkiller
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
1,030 total downloads
last updated
6/7/2021
Latest version:
5.0.0-alpha09
parser
lexer
parsing
lexing
tokenizer
tokenizing
Extension methods to integrate GParse with Tsu.StateMachines
Virastyar
by:
senobari
.NET Framework
5,406 total downloads
last updated
12/11/2011
Latest version:
2.0.0
spell-checker
transliterator
unicode
farsi
persian
NLP
language
tokenizer
datetime
lemmatizer
More tags
A Farsi (Persian) language checking and NLP library. This package includes both library and it's required data files.
VisualFA.
SourceGenerator
by:
honey.the.codewitch
424 total downloads
last updated
4/1/2024
Latest version:
1.2.1
lexer
tokenizer
dfa
regex
Generates fast DFA lexers and matchers in C# during the build process
Tweekenizer
by:
cristipufu
.NET 5.0
438 total downloads
last updated
11/30/2020
Latest version:
1.0.0
tokenizer
hashtag
nlp
emoji
Tokenizer for social media posts and comments
GBertTokenizer
by:
GeorgeTer
.NET 6.0
283 total downloads
last updated
2/4/2024
Latest version:
1.1.1
BERT
Tokenizer
charp
dotnet
Package Description
Cynic-
Magnit.
Tokenization
by:
catapart
.NET 6.0
1,039 total downloads
last updated
10/20/2022
Latest version:
1.0.1
string
parse
parsing
token
tokenizer
tokenization
Tokenize strings into custom tokens using ordered regex operations.
Noise
by:
m-wantia
.NET 6.0
248 total downloads
last updated
6/19/2022
Latest version:
0.0.0.1-alpha
lexer
lexical
tokenizer
Package Description
PrismSharp
by:
tkubec
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
186 total downloads
last updated
2/18/2022
Latest version:
1.0.0-beta
syntax
highlighting
highlighter
tokenizer
PrismSharp is a syntax highlighting library based on an excellent javascript library PrismJS, fully written in C#. It currently supports over 270 programming languages and has 44 built-in visual themes, also...
More information
APCSharp
by:
Dotch
.NET 5.0
.NET Core 3.1
3,569 total downloads
last updated
4/11/2021
Latest version:
1.0.8
lexer
parser
parsing
language
tokenizer
lexical
semantic
analysis
compiler
combinator
More tags
Another Parser Combinator for C#. A library for building optimized and flexible parsers.
Virastyar.
Lib
by:
senobari
.NET Framework
3,753 total downloads
last updated
12/11/2011
Latest version:
2.0.0
spell-checker
transliterator
unicode
farsi
persian
NLP
language
tokenizer
datetime
lemmatizer
More tags
A Farsi (Persian) language checking and NLP library.
Virastyar.
Data
by:
senobari
3,816 total downloads
last updated
12/11/2011
Latest version:
2.0.0
spell-checker
transliterator
unicode
farsi
persian
NLP
language
tokenizer
datetime
lemmatizer
More tags
Required data files for Virastyar library.
Leeax.
Parsing.
CSS
by:
leeax
.NET 5.0
198 total downloads
last updated
1/12/2021
Latest version:
1.0.0-beta
css
css3
tokenizer
parser
A lightweight CSS tokenizer/parser with no dependencies.
Universal.
OpenAI
by:
ong.andrew
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
323 total downloads
last updated
3/9/2023
Latest version:
1.0.0
Universal
OpenAI
Token
Tokenizer
GPT-2
Class library containing utilities that are useful in conjunction with OpenAI's models and APIs.
TKN
by:
Gulg
.NET Framework 4.7.2
192 total downloads
last updated
7/22/2022
Latest version:
1.0.0
Tokenizer
Lexer
Parser
Regex
An advanced tokenizer by GulgDevs
Previous
Next