Skip To Content
Toggle navigation
Packages
Upload
Statistics
Documentation
Downloads
Blog
Sign in
Frameworks
.NET
net7.0
net6.0
net5.0
.NET Core
netcoreapp3.1
netcoreapp3.0
netcoreapp2.2
netcoreapp2.1
netcoreapp2.0
netcoreapp1.1
netcoreapp1.0
.NET Standard
netstandard2.1
netstandard2.0
netstandard1.6
netstandard1.5
netstandard1.4
netstandard1.3
netstandard1.2
netstandard1.1
netstandard1.0
.NET Framework
net481
net48
net472
net471
net47
net462
net461
net46
net452
net451
net45
net40
net35
net30
net20
Package type
All types
Dependency
.NET tool
Template
Options
Include prerelease
42 packages returned for Tags:"tokenizer"
Sort by
Relevance
Downloads
Recently updated
Tokenizer
by:
flipbit
294,841 total downloads
last updated
1/4/2021
Latest version:
2.2.2
tokenizer
tokens
pattern
matching
Tokenizer extracts structured information from blocks of text and reflects them onto .NET objects
Jokenizer.
Net
by:
umutozel
83,153 total downloads
last updated
2/10/2020
Latest version:
1.1.1
csharp
expression
parser
tokenizer
C# Expression parser and evaluator, inspired from jokenizer project.
AI.
Dev.
OpenAI.
GPT
by:
devis
55,497 total downloads
last updated
10/20/2022
Latest version:
1.0.2
OpenAI
GPT
GPT3
Tokenizer
Tokens
AI
OpenAI GPT utils, e.g. GPT3 Tokenizer
OpenNLP.
NET
by:
sergey_tihon
73,540 total downloads
last updated
8/23/2022
Latest version:
1.9.4.1
NLP
OpenNLP
Tokenizer
Categorizer
Chunker
Parser
POS
NER
IKVM
The Apache OpenNLP library is a machine learning based toolkit for the processing of natural language text. It supports the most common NLP tasks, such as tokenization, sentence segmentation, part-of-speech...
More information
SoftCircuits.
Parsing.
Helper
by:
SoftCircuits
20,985 total downloads
last updated
7/16/2023
Latest version:
5.0.0
text
parse
parser
parsing
text-parser
text-parsing
parsing-helper
.net
csharp
tokenizer
A .NET class library that makes it easier to parse text. The library tracks the current position within the text, ensures your code never attempts to access a character at an invalid index, and includes many...
More information
BERTTokenizers
by:
NMZivkovic
7,917 total downloads
last updated
9/9/2022
Latest version:
1.2.0
BERT
Tokenizer
charp
dotnet
This package contains tokenizers for following models: · BERT Base · BERT Large · BERT German · BERT Multilingual · BERT Base Uncased · BERT Large Uncased
Conekta.
Xamarin
by:
bscheiman
10,156 total downloads
last updated
3/1/2015
Latest version:
1.0.6
boris
scheiman
bscheiman
conekta
xamarin
tokenizer
Tokenizador (Xamarin) para Conekta. Necesitas tener alguna libreria de servidor para usar el token.
Loretta.
CodeAnalysis.
Common
by:
gggkiller
6,510 total downloads
last updated
6/25/2023
Latest version:
0.2.12-nightly.24
parser
lexer
parsing
lexing
tokenizer
tokenizing
A shared package used by Loretta. Do not install this package manually, it will be added as a prerequisite by other packages that require it.
NltkNet
by:
skeleton
11,740 total downloads
last updated
1/1/2021
Latest version:
1.0.14
nltk
nlp
parser
tokenizer
categorizer
POS
NER
chunker
IronPython
NLTK python library wrapper for .NET
VBF.
Compilers.
Scanners
by:
Ninputer
5,845 total downloads
last updated
10/24/2015
Latest version:
1.0.5
VBF
Compiler
scanner
lexer
lex
tokenizer
VBF.Compilers.Scanners is a scanner builder. It contains a regular expression to DFA engine, can generate high performance scanners for unicode source text.
Loretta.
CodeAnalysis.
Lua
by:
gggkiller
5,742 total downloads
last updated
6/25/2023
Latest version:
0.2.12-nightly.24
parser
lexer
parsing
lexing
tokenizer
tokenizing
A GLua/Lua lexer, parser, code analysis, transformation and generation library.
Loretta.
CodeAnalysis.
Lua.
Experimental
by:
gggkiller
4,407 total downloads
last updated
6/25/2023
Latest version:
0.2.12-nightly.24
parser
lexer
parsing
lexing
tokenizer
tokenizing
Experimental code that might become part of Loretta.CodeAnalysis.Lua.
Tortoise
by:
m-wantia
2,054 total downloads
last updated
6/17/2022
Latest version:
0.1.1
lexer
lexical
tokenizer
Package Description
Trl.
PegParser
by:
wcoetser
4,686 total downloads
last updated
11/26/2020
Latest version:
1.3.0
Tokenizer
PEG
Grammer
Regular_Expressions
Regex
Parser
Parser_Generator
Trl.PegParser contains a tokenizer and a parser. The tokenizer uses regular expressions to define tokens, and exposes both matched and unmatched character ranges. The PEG Parser uses parsing expression grammers...
More information
MetaParser
by:
dsisco11
3,049 total downloads
last updated
2/23/2023
Latest version:
1.0.7
parsing
tokenization
tokenizer
text
parser
Package Description
Stringes
by:
Berkin
15,191 total downloads
last updated
11/8/2015
Latest version:
1.4.8
stringes
tokenizer
token
lexer
string
substring
parser
ast
syntax
grammar
More tags
The Stringe is a wrapper for the .NET String object that tracks line, column, offset, and other metadata for substrings.
owl
by:
SplittyDev
5,939 total downloads
last updated
10/21/2014
Latest version:
1.0.5407.37498
html
preprocessor
compiler
lexer
parser
scanner
tokenizer
owl
A html preprocessor which allows you to write html code using a beautiful syntax
TokenCs
by:
zeplar_exe
2,117 total downloads
last updated
8/10/2022
Latest version:
1.0.5
token
tokenizer
lexer
parse
parser
A simple library to abstract the gritty process of lexical parsing.
CEM.
Utilities.
Tokenizer
by:
dominic.proulx
1,349 total downloads
last updated
7/29/2022
Latest version:
1.0.1
CEM
Utilities
Tokenizer
Classe qui gère le découpage d'une chaine en paires de clés/valeurs et permet de la reconstruire avec des valeurs de remplacement ou des valeurs additionnelles
XLemmatizer
by:
rchristen
778 total downloads
last updated
5/29/2020
Latest version:
0.1.0
nlp
lemmatizer
tokenizer
Pre-release version. API might change later. A lemma is the canonical form of the word. For example, the words "run", "runs", "ran" and "running" can be lemmatized to "run" XLemmatizer tokenizes and lemmatizes...
More information
Previous
Next