DbfSharp.Core 0.2.4

DbfSharp.Core

A high-performance, memory-efficient DBF (dBase) file reader for .NET with support for all major DBF versions and memo files.

Installation

dotnet add package DbfSharp.Core

Features

• Support for all major DBF versions (dBase III, IV, Visual FoxPro, etc.)
• High performance with streaming and loaded access patterns
• Memory efficient streaming by default, optional loading for random access
• Full memo file support (FPT, DBT files) for large text and binary data
• Plugin architecture for custom field parsers
• Modern C# with Span<T>, Memory<T>, and latest .NET features
• Support for all standard field types and character encodings

Quick Start

Basic Usage

using DbfSharp.Core;

// Open and read a DBF file
using var reader = DbfReader.Open("data.dbf");

foreach (var record in reader.Records)
{
    var name = record.GetValue<string>("NAME");
    var birthDate = record.GetValue<DateTime?>("BIRTHDATE");
    var salary = record.GetValue<decimal?>("SALARY");

    Console.WriteLine($"{name}, born {birthDate}, salary: {salary:C}");
}

Async Operations

// Async file operations
using var reader = await DbfReader.OpenAsync("data.dbf");

await foreach (var record in reader.ReadRecordsAsync())
{
    var name = record.GetValue<string>("NAME");
    Console.WriteLine($"Processing: {name}");
}

Performance Optimized

// Use performance-optimized settings for large files
var options = DbfReaderOptions.CreatePerformanceOptimized();
using var reader = DbfReader.Open("large_file.dbf", options);

foreach (var record in reader.Records)
{
    var id = record[0]; // Access by index for speed
    var name = record["NAME"]; // Access by name
}

Memory Optimized

// Use memory-optimized settings for huge files
var options = DbfReaderOptions.CreateMemoryOptimized();
using var reader = DbfReader.Open("huge_file.dbf", options);

// Process one record at a time with minimal memory usage
foreach (var record in reader.Records)
{
    ProcessRecord(record);
}

Loaded Mode for Analysis

// Load all records into memory for random access
using var reader = DbfReader.Open("analysis.dbf");
reader.Load();

// Access records by index
var firstRecord = reader[0];
var lastRecord = reader[reader.Count - 1];

// Use LINQ for queries
var highSalaries = reader.Records
    .Where(r => r.GetValue<decimal?>("SALARY") > 50000)
    .ToList();

Configuration

DbfReaderOptions

var options = new DbfReaderOptions
{
    Encoding = Encoding.UTF8,           // Override auto-detected encoding
    IgnoreCase = true,                  // Case-insensitive field names
    LowerCaseFieldNames = false,        // Convert field names to lowercase
    LoadOnOpen = false,                 // Load all records immediately
    IgnoreMissingMemoFile = true,       // Don't fail if memo file missing
    TrimStrings = true,                 // Trim whitespace from strings
    ValidateFields = false,             // Skip field validation for speed
    MaxRecords = 10000,                 // Limit number of records read
    BufferSize = 64 * 1024,            // I/O buffer size
    UseMemoryMapping = true             // Use memory-mapped files (64-bit)
};

using var reader = DbfReader.Open("data.dbf", options);

Preset Configurations

// Maximum performance
var perfOptions = DbfReaderOptions.CreatePerformanceOptimized();

// Minimum memory usage
var memOptions = DbfReaderOptions.CreateMemoryOptimized();

// Maximum compatibility
var compatOptions = DbfReaderOptions.CreateCompatibilityOptimized();

Supported Field Types

Advanced Features

Custom Field Parsers

public class CustomFieldParser : FieldParserBase
{
    public override bool CanParse(FieldType fieldType, DbfVersion dbfVersion)
    {
        return fieldType == FieldType.Character;
    }

    public override object? Parse(DbfField field, ReadOnlySpan<byte> data,
        IMemoFile? memoFile, Encoding encoding, DbfReaderOptions options)
    {
        // Custom parsing logic
        var text = encoding.GetString(data);
        return text.ToUpperInvariant(); // Example: convert to uppercase
    }
}

var options = new DbfReaderOptions
{
    CustomFieldParser = new CustomFieldParser()
};

Error Handling

try
{
    using var reader = DbfReader.Open("data.dbf");
    foreach (var record in reader.Records)
    {
        // Check for invalid values
        foreach (var kvp in record)
        {
            if (kvp.Value is InvalidValue invalid)
            {
                Console.WriteLine($"Invalid value in field {kvp.Key}: {invalid.ErrorMessage}");
            }
        }
    }
}
catch (DbfNotFoundException ex)
{
    Console.WriteLine($"File not found: {ex.FilePath}");
}
catch (MissingMemoFileException ex)
{
    Console.WriteLine($"Memo file missing: {ex.MemoFilePath}");
}
catch (FieldParseException ex)
{
    Console.WriteLine($"Parse error in field {ex.FieldName}: {ex.Message}");
}

Working with Deleted Records

var options = new DbfReaderOptions
{
    SkipDeletedRecords = false // Include deleted records
};

using var reader = DbfReader.Open("data.dbf", options);

// Access only deleted records
await foreach (var deletedRecord in reader.ReadDeletedRecordsAsync())
{
    Console.WriteLine($"Deleted Record ID: {deletedRecord.GetValue<int>("ID")}");
}

Metadata and Statistics

using var reader = DbfReader.Open("data.dbf");
var stats = reader.GetStatistics();

Console.WriteLine($"Table: {stats.TableName}");
Console.WriteLine($"Version: {stats.DbfVersion}");
Console.WriteLine($"Records: {stats.TotalRecords:N0}");
Console.WriteLine($"Fields: {stats.FieldCount}");
Console.WriteLine($"Encoding: {stats.Encoding}");
Console.WriteLine($"Last Updated: {stats.LastUpdateDate}");

// Inspect field definitions
foreach (DbfField field in reader.Fields)
{
    Console.WriteLine($"- {field.Name} ({field.Type}), Length: {field.ActualLength}");
}

Performance

DbfSharp.Core is optimized for both high throughput and low memory usage. Performance benchmarks measured on AMD Ryzen 5 7600X with .NET 9.0:

Key Performance Metrics (1,000,000 records)

Performance Scaling Across Dataset Sizes

Memory Usage Comparison

Code Examples by Use Case

Maximum Performance - 7.3M rows/sec

Use when you need the fastest possible processing:

using var dbf = DbfReader.Create("data.dbf");

foreach (var record in dbf.EnumerateSpanRecords())
{
    var nameBytes = record.GetFieldBytes(1); // Zero allocations
    // Process raw bytes directly
}

Memory Constrained - 3.5M rows/sec, 23MB for 1M rows

Use for large files when memory usage is critical:

var options = new DbfReaderOptions { UseMemoryMapping = true };
using var dbf = DbfReader.Create("large_file.dbf", options);

foreach (var record in dbf.EnumerateSpanRecords())
{
    var name = record.GetString(1); // Fast string conversion
    var id = record.GetInt32(0);
}

Balanced Approach - 1.3M rows/sec

Default streaming mode balances performance and memory:

using var dbf = DbfReader.Create("data.dbf");

foreach (var record in dbf.Records)
{
    var name = record.GetString("NAME");
    var id = record.GetInt32("ID");
}

Random Access - 500K rows/sec (after loading)

Best for analysis requiring frequent lookups:

using var dbf = DbfReader.Create("data.dbf");
dbf.Load(); // One-time load cost

// Now random access is very fast
var record = dbf[1000];
var filteredRecords = dbf.Records
    .Where(r => r.GetDecimal("SALARY") > 50000)
    .ToList();

When to Use Each Approach

Raw Field Bytes (GetFieldBytes()) - Maximum throughput

• Use when: Processing millions of records, custom parsing logic, performance-critical applications
• Avoid when: You need convenient object access or complex field parsing

Zero-Allocation Span API (EnumerateSpanRecords()) - Best overall performance

• Use when: Need both speed and convenience, processing large datasets
• Avoid when: Memory is extremely constrained

Memory-Mapped Files - Large files with memory constraints

• Use when: Files >500MB, running on memory-limited systems, need decent performance
• Avoid when: File size <10MB (overhead not worth it)

Default Streaming - General purpose

• Use when: Moderate file sizes, need convenient object access, typical business applications
• Avoid when: Processing millions of records frequently

Load Into Memory - Analysis and random access

• Use when: Need LINQ queries, random access by index, multiple passes over data
• Avoid when: File size >100MB or memory is limited

Optimization Tips

1. Choose the right access pattern:

   • Use EnumerateSpanRecords() for maximum performance
   • Use Records for convenient object access
   • Use Load() only when you need random access

2. Memory management:

   • Enable memory mapping for files >100MB: options.UseMemoryMapping = true
   • Use memory-optimized options: DbfReaderOptions.CreateMemoryOptimized()

3. Field access optimization:

   • Access by index (record[0]) is faster than by name (record["NAME"])
   • Use GetFieldBytes() for raw data processing
   • Cache field indices for repeated access