Sep 0.1.0-preview.7

Sep - Possibly the World's Fastest .NET CSV Parser

Modern, minimal, fast, zero allocation, reading and writing of separated values (csv, tsv etc.). Cross-platform, trimmable and AOT/NativeAOT compatible. Featuring an opinionated API design and pragmatic implementation targetted at machine learning use cases.

• ⭐ Modern - utilizes modern features such as Span<T>, Generic Math, ref struct and other .NET 7+/C# 11 features.
• 🔎 Minimal - a succinct yet expressive API with few moving parts, configurations and no hidden changes to input or output. What you read/write is what you get. This means there is no "automatic" escaping/unescaping of quotes, for example.
• 🚀 Fast - blazing fast with both architecture specific and cross-platform SIMD vectorized parsing and using csFastFloat for fast parsing of floating points. Reads or writes one row at a time efficiently with detailed benchmarks to prove it.
• 🗑️ Zero allocation - intelligent and efficient memory management allowing for zero allocations after warmup incl. supporting use cases of reading or writing arrays of values (e.g. features) easily without repeated allocations.
• 🌐 Cross-platform - works on any platform, any architecture supported by .NET. 100% managed and written in beautiful modern C#.
• ✂️ Trimmable and AOT/NativeAOT compatible - no problematic reflection or dynamic code generation. Hence, fully trimmable and Ahead-of-Time compatible. With a simple console tester program executable possible in just a few MBs. 💾
• 🗣️ Opinionated and pragmatic - conforms to the essentials of RFC-4180, but takes an opinionated and pragmatic approach towards this especially with regards to quoting and line ends. See section RFC-4180.

var text = """
           A;B;C;D;E;F
           Sep;🚀;1;1.2;0.1;0.5
           CSV;✅;2;2.2;0.2;1.5
           """;

using var reader = Sep.Reader().FromText(text);  // Infers separator 'Sep' from header
using var writer = reader.Sep.Writer().ToText(); // Writer defined from reader 'Sep'
                                                 // Use .FromFile(...)/ToFile(...) for files
var idx = reader.Header.IndexOf("B");
var nms = new[] { "E", "F" };

foreach (var readRow in reader)           // Read one row at a time
{
    var a = readRow["A"].Span;            // Column as ReadOnlySpan<char>
    var b = readRow[idx].ToString();      // Column to string (allocates new string per call)
    var c = readRow["C"].Parse<int>();    // Parse any T : ISpanParsable<T>
    var d = readRow["D"].Parse<float>();  // Parse float/double fast via csFastFloat
    var s = readRow[nms].Parse<double>(); // Parse multiple columns as Span<T>
                                          // - Sep handles array allocation and reuse
    foreach (ref var v in s) { v *= 10; }

    using var writeRow = writer.NewRow(); // Start new row. Row written on Dispose.
    writeRow["A"].Set(a);                 // Set by ReadOnlySpan<char>
    writeRow["B"].Set(b);                 // Set by string
    writeRow["C"].Set($"{c * 2}");        // Set via InterpolatedStringHandler, no allocs
    writeRow["D"].Format(d / 2);          // Format any T : ISpanFormattable
    writeRow[nms].Format(s);              // Format multiple columns directly
}

var expected = """
               A;B;C;D;E;F
               Sep;🚀;2;0.6;1;5
               CSV;✅;4;1.1;2;15
               """;
Assert.AreEqual(expected, writer.ToString());

// Above example code is for demonstration purposes only.
// Short names and repeated constants are only for demonstration.

Comparison Benchmarks

Reader Comparison Benchmarks

AMD 5950X Platform Information

BenchmarkDotNet=v0.13.2, OS=Windows 10 (10.0.19044.2846/21H2/November2021Update)
AMD Ryzen 9 5950X, 1 CPU, 32 logical and 16 physical cores

Machine Learning Reader Comparison Benchmarks

Benchmark demonstrating what Sep is built for. Namely floating points or features as in machine learning. Here a simple CSV-file is randomly generated with 20 ground truth values, 20 predicted result values and some typical extra columns leading that, but which aren't parsed or read in the benchmark. For Scope=Floats the benchmark will parse those features as two spans of floats; one for ground truth values and one for predicted result values. Finally, as an example calculates the mean squared error (MSE) of those i.e. loss.

AMD 5950X - Machine Learning Benchmark Results

NCsvPerf PackageAssets Reader Comparison Benchmarks

AMD 5950X - PackageAssets Benchmark Results

AMD 5950X - PackageAssets with Quotes Benchmark Results

RFC-4180

While the RFC-4180 requires \r\n (CR,LF) as line ending, the well-known line endings (\r\n, \n and \r) are supported similar to .NET. Environment.NewLine is used when writing. Quoting is supported by simply matching pairs of quotes, no matter what. With no automatic escaping. Hence, you are responsible and in control of this at this time.

Note that some libraries will claim conformance but the RFC is, perhaps naturally, quite strict e.g. only comma is supported as separator/delimiter. Sep defaults to using ; as separator if writing, while auto-detecting supported separators when reading. This is decidedly non-conforming.

The RFC defines the following condensed ABNF grammar:

file = [header CRLF] record *(CRLF record) [CRLF]
header = name *(COMMA name)
record = field *(COMMA field)
name = field
field = (escaped / non-escaped)
escaped = DQUOTE *(TEXTDATA / COMMA / CR / LF / 2DQUOTE) DQUOTE
non-escaped = *TEXTDATA
COMMA = %x2C
CR = %x0D ;as per section 6.1 of RFC 2234 [2]
DQUOTE =  %x22 ;as per section 6.1 of RFC 2234 [2]
LF = %x0A ;as per section 6.1 of RFC 2234 [2]
CRLF = CR LF ;as per section 6.1 of RFC 2234 [2]
TEXTDATA =  %x20-21 / %x23-2B / %x2D-7E

Note how TEXTDATA is restricted too, yet many will allow any character incl. emojis or similar (which Sep supports), but is not in conformance with the RFC.

Quotes inside an escaped field e.g. "fie""ld" are only allowed to be double quotes. Sep currently allows any pairs of quotes and quoting doesn't need to be at start of or end of field (col or column in Sep terminology).

All in all Sep takes a pretty pragmatic approach here as the primary use case is not exchanging data on the internet, but for use in machine learning pipelines or similar.