MemoryChunks.SCU 1.0.1

SCU.MemoryChunks

Minimal, Effective, Safe and Simple extension with single functionality - Split strings or arrays by size into chunks, without allocation redundant intermediate arrays like in LINQ version (Chunk method).

This extension was originally designed to split strings, because when you use linq Chunk(N) you need to convert char arrays to strings.

Issues with LINQ Approach: When using text.Chunk(N).Select(c => new string(c)).ToArray(), you incur:

1. N redundant temporary char[] allocations (from Chunk())
2. N new string allocations
3. Internal buffer resizing in Chunks() method

This Solution:

• Zero intermediate allocations
• Direct slicing of source string
• Controlled allocation points via .ToString()

This extension split strings without this redundant allocations, only N strings in result. This extension can be used for arrays too (not only for strings). It will be better than Linq too, because in this situation you will have N allocations of arrays, but Linq version has logic for array resize, and there is a possibility to have more than N allocations.

• No redundant memory allocations
• Almost zero cognitive complexity
• Without 'ref struct enumerators', without self-written enumerators, can be used on almost all .net f/c versions (copy-paste in project)
• Logic in 3 lines of code
• Async-compatible and Memory-safe (uses Memory<T> instead of Span<T>)
• 28-30x faster than LINQ in synchronous operations

Less code, less problems.

You can use Nuget package MemoryChunks.SCU, or copy-past code of this extension in you project, or not use it. It's up to you 😃

Code:

using System.Runtime.CompilerServices;
namespace SCU.MemoryChunks
{
    public static class MemoryExtensions
    {
        /// <summary>
        /// Splits to Memory<char> chunks with zero-memory allocation. 
        /// Safe for using in async methods. Strings are immutable, so, you can use this enumaration in code with long-running enumaration.
        /// </summary>
        /// <param name="chunkSize">max size of chunk</param>
        [MethodImpl(MethodImplOptions.AggressiveInlining)]
        public static IEnumerable<ReadOnlyMemory<char>> MemoryChunks(this string input, int chunkSize) => MemoryChunks<char>(input.AsMemory(), chunkSize);

        /// <summary>
        /// Splits to Memory<<typeparamref name="T"/>> chunks with zero-memory allocation. 
        /// Safe for using in async methods. 
        /// Make sure <paramref name="input"/> won't be modified during enumeration, otherwise if your enumaration is long-running then use ToArray/ToList to materialize/execute this enumeration before changes in original array.
        /// </summary>
        /// <param name="chunkSize">max size of chunk</param>
        [MethodImpl(MethodImplOptions.AggressiveInlining)]
        public static IEnumerable<ReadOnlyMemory<T>> MemoryChunks<T>(this T[] input, int chunkSize) => MemoryChunks<T>(input.AsMemory(), chunkSize);

        /// <summary>
        /// Splits to Memory<<typeparamref name="T"/>> chunks with zero-memory allocation. 
        /// Safe for using in async methods. 
        /// Make sure <paramref name="input"/> won't be modified during enumeration, otherwise if your enumaration is long-running then use ToArray/ToList to materialize/execute this enumeration before changes in original array.
        /// </summary>
        /// <param name="chunkSize">max size of chunk</param>
        public static IEnumerable<ReadOnlyMemory<T>> MemoryChunks<T>(this ReadOnlyMemory<T> source, int chunkSize)
        {
            if (chunkSize <= 0) throw new ArgumentOutOfRangeException(nameof(chunkSize));
            for (int i = 0; i < source.Length; i += chunkSize)
            {
                yield return source.Slice(i, Math.Min(chunkSize, source.Length - i));
            }
        }
    }
}

Nuget (only .net8.0 for now)

https://www.nuget.org/packages/MemoryChunks.SCU Note: Nuget not passing name SCU.MemoryChunks, so in Nuget this extension has name MemoryChunks.SCU, but in C# code name of lib is SCU.MemoryChunks

dotnet add package MemoryChunks.SCU

or

NuGet\Install-Package MemoryChunks.SCU

using SCU.MemoryChunks;

MemoryChunks vs LINQ Chunk Performance Benchmark

Test Environment:

• .NET 8.0.8
• Intel Core i5-11320H (4 cores @ 3.20GHz)
• Windows 10 20H2
• BenchmarkDotNet v0.14.0

Results

Note about Async tests:
The async benchmarks simulate await points during iteration. While they show MemoryChunks' consistent advantage (1.7x less memory), these scenarios are primarily included to demonstrate thread safety with Memory<T>. In real-world usage, the synchronous versions will always be more efficient.

1. Performance:

   • MemoryChunks is 28-30x faster than LINQ for synchronous operations
   • Async overhead adds ~80μs regardless of method

2. Memory Efficiency:

   • MemoryChunks allocates 315x less memory than LINQ version
   • String allocation multiplies memory usage by:
     • 315x for MemoryChunks
     • 636x for LINQ

3. Async Impact:

   • MemoryChunks still uses 1.7x less memory than LINQ in async mode
   • Maintains consistent performance advantage in async scenarios

Recommended Use Cases

1. For maximum performance:

   foreach (var chunk in text.MemoryChunks(100)) // No allocations
   {
       // Process chunk directly
   }
   

   or when allocation is needed for string, it will be better than Linq version because Linq produces redundant intermediate arrays (and Linq has logic for Array resize in Chunks method)

   foreach (var chunk in text.MemoryChunks(100)) // No allocations
   {
       // use chunk.ToString()
   }
   

   or when allocation is needed for arrays, it will be better than Linq version because Linq produces redundant intermediate arrays (and Linq has logic for Array resize in Chunks method)

   foreach (var chunk in text.MemoryChunks(100)) // No allocations
   {
       // use chunk.ToArray()
   }
   

• Can be used in Async methods, because use Memory<>, not Span<> types
• Safe for strings (immutable by nature)
• For arrays: ensure source isn't modified during enumeration
• Safe for long-lived enumerations

License

Free MIT license (https://github.com/sapozhnikovv/SCU.MemoryChunks/blob/main/LICENSE)