ASCDataAccessLibrary 2.3.0

ASCDataAccessLibrary v2.3 Documentation

What's New in Version 2.3

Major Changes

• Simplified Queue Architecture: Each QueueData now represents ONE independent queue (one StateList)
• Independent Queue Management: Cleaner separation between multiple queues
• Enhanced Queue Operations: New methods for managing individual queues
• Improved Statistics: Better tracking per queue and category-wide statistics
• Backward Compatibility: Existing code continues to work with clearer semantics

Key Improvements

• Fixed queue implementation for proper one-to-one queue/StateList mapping
• Added per-queue operations (GetQueueAsync, PeekQueueAsync)
• Category-wide statistics and status tracking
• Simplified mental model: 1 QueueData = 1 Queue = 1 StateList
• Better support for parallel queue processing

Table of Contents

1. Installation
2. Basic Concepts
3. StateList - Position-Aware Collections
4. Queue Management (UPDATED)
5. Dynamic Entities
6. Working with Data Access
7. Session Management
8. Error Logging
9. Blob Storage Operations
10. Advanced Features
11. Performance Optimizations
12. Migration Guide
13. Best Practices

Installation

Install the package from NuGet:

Install-Package ASCDataAccessLibrary -Version 2.3.0

Or using the .NET CLI:

dotnet add package ASCDataAccessLibrary --version 2.3.0

Basic Concepts

Core Components

• DataAccess<T>: Generic class for CRUD operations with Azure Table Storage
• TableEntityBase: Base class for all table entities with automatic field chunking
• StateList<T>: Position-aware collection that maintains state through serialization
• QueueData<T>: Represents a SINGLE queue with one StateList (Updated in v2.3)
• DynamicEntity: Pre-built implementation for schema-less operations
• Session: Advanced session management for stateful applications
• ErrorLogData: Comprehensive error logging with caller information
• AzureBlobs: Blob storage operations with tag-based indexing

Architecture Overview (v2.3)

┌─────────────────────────────────────┐
│         Your Application            │
├─────────────────────────────────────┤
│     Multiple Independent Queues     │ ← NEW: Each QueueData = 1 Queue
│            ↓                        │
│     StateList<T> (1 per Queue)      │ ← Position-aware collection
│            ↓                        │
│     QueueData<T> Storage            │ ← One row = one queue
├─────────────────────────────────────┤
│     Strongly-Typed Entities         │
│            ↓                        │
│     TableEntityBase                 │
│            ↓                        │
│     ITableExtra                     │
├─────────────────────────────────────┤
│      Dynamic Entities               │
│            ↓                        │
│      DynamicEntity                  │
│            ↓                        │
│     IDynamicProperties              │
├─────────────────────────────────────┤
│      DataAccess<T>                  │
├─────────────────────────────────────┤
│      Azure Table Storage            │
└─────────────────────────────────────┘

StateList - Position-Aware Collections

Overview

StateList<T> is a powerful collection that maintains its current position through serialization/deserialization. Perfect for:

• Processing large datasets with interruption recovery
• Maintaining state across application restarts
• Queue processing with position tracking
• Paginated data processing with resume capability

Creating and Using StateList

// Create a StateList from existing data
var items = new List<string> { "Item1", "Item2", "Item3", "Item4", "Item5" };
var stateList = new StateList<string>(items);

// Set description for context
stateList.Description = "Customer Processing Queue";

// Navigate through the list
stateList.First();  // Move to first item
Console.WriteLine($"Current: {stateList.Current}"); // "Item1"

stateList.MoveNext(); // Move forward
Console.WriteLine($"Current: {stateList.Current}"); // "Item2"
Console.WriteLine($"Position: {stateList.CurrentIndex}"); // 1

// Check navigation capabilities
if (stateList.HasNext)
    stateList.MoveNext();

if (stateList.HasPrevious)
    stateList.MovePrevious();

// Get current item with index
var peek = stateList.Peek; // Returns (Data: "Item2", Index: 1)

Position Preservation Through Serialization

// Process part of a list
var stateList = new StateList<Order>(orders);
stateList.Description = "Order Processing";

// Process some items
while (stateList.CurrentIndex < 50 && stateList.MoveNext())
{
    ProcessOrder(stateList.Current);
}

// Save state (position is preserved)
string json = JsonConvert.SerializeObject(stateList);
await SaveToStorage(json);

// Later: Restore and continue from last position
var restored = JsonConvert.DeserializeObject<StateList<Order>>(json);
Console.WriteLine($"Resuming from position {restored.CurrentIndex}");

// Continue processing from where we left off
while (restored.MoveNext())
{
    ProcessOrder(restored.Current);
}

Queue Management (UPDATED)

Overview - v2.3 Architecture

In version 2.3, the queue architecture has been simplified and clarified:

• One QueueData = One Queue = One StateList
• Each QueueData row in the table represents an independent queue
• Multiple queues can share the same category (Name/PartitionKey)
• Each queue has its own unique ID (QueueID/RowKey)

Basic Queue Operations

// Create a single queue from StateList
var items = new StateList<Invoice>(invoices);
items.Description = "Q1 2024 Invoice Processing";
items.CurrentIndex = 25; // Already processed 25 items

// Create a queue (represents ONE queue)
var queue = QueueData<Invoice>.CreateFromStateList(
    items, 
    "InvoiceProcessing",  // Category name (PartitionKey)
    "Q1_2024_Queue"       // Unique queue ID (RowKey)
);

queue.ProcessingStatus = "In Progress";
queue.PercentComplete = 25.0;
queue.TotalItemCount = items.Count;
queue.LastProcessedIndex = items.CurrentIndex;

// Save this single queue
await queue.SaveQueueAsync(accountName, accountKey);

// Retrieve a specific queue by ID
var retrievedQueue = await QueueData<Invoice>.GetQueueAsync(
    "Q1_2024_Queue",      // Specific queue ID
    accountName, 
    accountKey,
    deleteAfterRetrieve: false
);

if (retrievedQueue != null)
{
    var stateList = retrievedQueue.GetData();
    Console.WriteLine($"Queue {retrievedQueue.QueueID}: Position {stateList.CurrentIndex} of {stateList.Count}");
}

Working with Multiple Independent Queues

// Create multiple independent queues in the same category
var categories = new[] { "Q1_2024", "Q2_2024", "Q3_2024", "Q4_2024" };

foreach (var quarter in categories)
{
    var quarterInvoices = GetInvoicesForQuarter(quarter);
    var stateList = new StateList<Invoice>(quarterInvoices)
    {
        Description = $"{quarter} Invoice Processing"
    };
    
    var queue = QueueData<Invoice>.CreateFromStateList(
        stateList,
        "InvoiceProcessing",  // Same category for all
        $"{quarter}_Queue"    // Unique ID for each queue
    );
    
    await queue.SaveQueueAsync(accountName, accountKey);
}

// Get all queues in a category
var allQueues = await QueueData<Invoice>.GetQueuesAsync(
    "InvoiceProcessing",  // Category name
    accountName, 
    accountKey,
    deleteAfterRetrieve: false
);

// Each queue is independent with its own StateList
foreach (var kvp in allQueues)
{
    var queueId = kvp.Key;
    var stateList = kvp.Value;
    Console.WriteLine($"Queue {queueId}: {stateList.CurrentIndex}/{stateList.Count} items processed");
}

Processing Individual Queues

// Process a specific queue
public async Task ProcessSpecificQueue(string queueId)
{
    // Get the specific queue
    var queue = await QueueData<Order>.GetQueueAsync(
        queueId, 
        accountName, 
        accountKey, 
        deleteAfterRetrieve: false
    );
    
    if (queue == null)
    {
        Console.WriteLine($"Queue {queueId} not found");
        return;
    }
    
    var stateList = queue.GetData();
    Console.WriteLine($"Processing queue {queueId} from position {stateList.CurrentIndex}");
    
    // Process items
    while (stateList.MoveNext())
    {
        await ProcessOrder(stateList.Current);
        
        // Save checkpoint every 10 items
        if (stateList.CurrentIndex % 10 == 0)
        {
            await QueueData<Order>.SaveProgressAsync(
                stateList, 
                accountName, 
                accountKey
            );
            Console.WriteLine($"Checkpoint saved at position {stateList.CurrentIndex}");
        }
    }
    
    // Mark queue as complete
    queue.ProcessingStatus = "Completed";
    queue.PercentComplete = 100;
    queue.LastProcessedIndex = stateList.Count - 1;
    await queue.SaveQueueAsync(accountName, accountKey);
}

Parallel Processing of Multiple Queues

// Process multiple independent queues in parallel
public async Task ProcessAllQueuesInParallel(string category)
{
    // Get all queues in the category
    var allQueues = await QueueData<WorkItem>.GetQueuesAsync(
        category, 
        accountName, 
        accountKey,
        deleteAfterRetrieve: false
    );
    
    Console.WriteLine($"Found {allQueues.Count} queues to process");
    
    // Process each queue in parallel
    var tasks = allQueues.Select(kvp => Task.Run(async () =>
    {
        var queueId = kvp.Key;
        var stateList = kvp.Value;
        
        Console.WriteLine($"Starting processing of queue {queueId}");
        
        while (stateList.MoveNext())
        {
            await ProcessWorkItem(stateList.Current);
        }
        
        Console.WriteLine($"Completed queue {queueId}");
        
        // Delete completed queue
        await QueueData<WorkItem>.DeleteQueuesAsync(
            new List<string> { queueId }, 
            accountName, 
            accountKey
        );
    }));
    
    await Task.WhenAll(tasks);
    Console.WriteLine("All queues processed");
}

Queue Monitoring and Statistics

// Get status of all queues in a category
var statuses = await QueueData<Report>.GetQueueStatusesAsync(
    "ReportGeneration", 
    accountName, 
    accountKey
);

foreach (var status in statuses)
{
    Console.WriteLine($"Queue: {status.QueueId}");
    Console.WriteLine($"  Status: {status.ProcessingStatus}");
    Console.WriteLine($"  Progress: {status.PercentComplete:F1}%");
    Console.WriteLine($"  Items: {status.ProcessedItems}/{status.TotalItems}");
}

// Get category-wide statistics
var stats = await QueueData<Report>.GetCategoryStatisticsAsync(
    "ReportGeneration", 
    accountName, 
    accountKey
);

Console.WriteLine($"Category Statistics:");
Console.WriteLine($"  Total Queues: {stats.TotalQueues}");
Console.WriteLine($"  Active Queues: {stats.ActiveQueues}");
Console.WriteLine($"  Completed Queues: {stats.CompletedQueues}");
Console.WriteLine($"  Total Items: {stats.TotalItems}");
Console.WriteLine($"  Processed Items: {stats.TotalProcessedItems}");
Console.WriteLine($"  Average Progress: {stats.AverageProgress:F1}%");

Paged Queue Retrieval

// Get queues in pages (each queue is independent)
var pagedResult = await QueueData<Task>.GetPagedQueuesAsync(
    "TaskProcessing",
    accountName,
    accountKey,
    pageSize: 10,  // Get 10 queues at a time
    continuationToken: null
);

Console.WriteLine($"Retrieved {pagedResult.Queues.Count} queues");

foreach (var kvp in pagedResult.Queues)
{
    var queueId = kvp.Key;
    var stateList = kvp.Value;
    var metadata = pagedResult.QueueMetadata[queueId];
    
    Console.WriteLine($"Queue {queueId}:");
    Console.WriteLine($"  Status: {metadata.ProcessingStatus}");
    Console.WriteLine($"  Progress: {metadata.PercentComplete:F1}%");
    Console.WriteLine($"  Position: {stateList.CurrentIndex}/{stateList.Count}");
}

// Get next page if available
if (pagedResult.HasMore)
{
    var nextPage = await QueueData<Task>.GetPagedQueuesAsync(
        "TaskProcessing",
        accountName,
        accountKey,
        pageSize: 10,
        continuationToken: pagedResult.ContinuationToken
    );
}

Advanced Queue Patterns

// Pattern 1: Queue with retry logic
public async Task ProcessQueueWithRetry(string queueId, int maxRetries = 3)
{
    for (int attempt = 1; attempt <= maxRetries; attempt++)
    {
        try
        {
            var queue = await QueueData<Job>.GetQueueAsync(
                queueId, accountName, accountKey, false);
            
            if (queue == null) return;
            
            var stateList = queue.GetData();
            
            while (stateList.MoveNext())
            {
                try
                {
                    await ProcessJob(stateList.Current);
                }
                catch (Exception ex)
                {
                    // Log error but continue processing
                    Console.WriteLine($"Error processing item {stateList.CurrentIndex}: {ex.Message}");
                }
                
                // Save progress frequently
                if (stateList.CurrentIndex % 5 == 0)
                {
                    await QueueData<Job>.SaveProgressAsync(
                        stateList, accountName, accountKey);
                }
            }
            
            // Success - delete the queue
            await QueueData<Job>.DeleteQueuesAsync(
                new List<string> { queueId }, accountName, accountKey);
            break;
        }
        catch (Exception ex)
        {
            Console.WriteLine($"Attempt {attempt} failed: {ex.Message}");
            if (attempt == maxRetries) throw;
            await Task.Delay(TimeSpan.FromSeconds(Math.Pow(2, attempt))); // Exponential backoff
        }
    }
}

// Pattern 2: Priority queue processing
public async Task ProcessPriorityQueues(string category)
{
    var statuses = await QueueData<PriorityTask>.GetQueueStatusesAsync(
        category, accountName, accountKey);
    
    // Sort by priority (stored in ProcessingStatus for this example)
    var prioritizedQueues = statuses
        .OrderBy(s => s.ProcessingStatus) // "1-High", "2-Medium", "3-Low"
        .ToList();
    
    foreach (var queueStatus in prioritizedQueues)
    {
        await ProcessSpecificQueue(queueStatus.QueueId);
    }
}

// Pattern 3: Time-based queue segmentation
public async Task CreateTimeBasedQueues(List<LogEntry> allLogs)
{
    // Group logs by hour
    var hourlyGroups = allLogs.GroupBy(log => 
        new DateTime(log.Timestamp.Year, log.Timestamp.Month, 
                    log.Timestamp.Day, log.Timestamp.Hour, 0, 0));
    
    foreach (var group in hourlyGroups)
    {
        var stateList = new StateList<LogEntry>(group.ToList())
        {
            Description = $"Logs for {group.Key:yyyy-MM-dd HH:00}"
        };
        
        var queue = QueueData<LogEntry>.CreateFromStateList(
            stateList,
            "LogProcessing",
            $"Logs_{group.Key:yyyyMMddHH}"
        );
        
        await queue.SaveQueueAsync(accountName, accountKey);
    }
}

Dynamic Entities

Overview

Dynamic entities allow you to work with Azure Table Storage without defining compile-time schemas. Perfect for:

• Multi-tenant applications with varying schemas
• Integration with external systems
• Rapid prototyping
• Scenarios where table structure changes frequently

Creating Dynamic Entities

// Create a dynamic entity with runtime-defined table name
var entity = new DynamicEntity("SystemName_EntityType", "partitionKey", "rowKey");

// Add properties using indexer syntax
entity["EmployeeName"] = "John Doe";
entity["HireDate"] = DateTime.UtcNow;
entity["Salary"] = 95000.50;
entity["IsActive"] = true;

// Alternative: Use method syntax
entity.SetProperty("Department", "Engineering");
entity.SetProperty("YearsExperience", 8);

Working with Data Access

Creating Strongly-Typed Entities

public class Customer : TableEntityBase, ITableExtra
{
    public string CompanyId
    {
        get => this.PartitionKey;
        set => this.PartitionKey = value;
    }
    
    public string CustomerId
    {
        get => this.RowKey;
        set => this.RowKey = value;
    }
    
    public string Name { get; set; }
    public string Email { get; set; }
    public string LargeDataField { get; set; } // Auto-chunked if >32KB
    
    public string TableReference => "Customers";
    public string GetIDValue() => this.CustomerId;
}

CRUD Operations

var dataAccess = new DataAccess<Customer>(accountName, accountKey);

// Create/Update
var customer = new Customer
{
    CompanyId = "company123",
    CustomerId = "cust456",
    Name = "Acme Corporation",
    Email = "contact@acme.com"
};
await dataAccess.ManageDataAsync(customer);

// Retrieve with lambda
var active = await dataAccess.GetCollectionAsync(c => 
    c.Email == "contact@acme.com" && c.Status == "Active");

// Pagination
var page = await dataAccess.GetPagedCollectionAsync(pageSize: 50);

// Batch operations with progress
var progress = new Progress<BatchUpdateProgress>(p =>
    Console.WriteLine($"Progress: {p.PercentComplete:F1}%"));

await dataAccess.BatchUpdateListAsync(
    customers, 
    TableOperationType.InsertOrReplace, 
    progress);

Session Management

Creating and Using Sessions

// Asynchronous creation (recommended)
var session = await Session.CreateAsync(accountName, accountKey, sessionId);

// Store values
session["UserName"] = "John Doe";
session["LastLogin"] = DateTime.Now.ToString();
session["CartItems"] = JsonConvert.SerializeObject(items);

// Retrieve values
string userName = session["UserName"]?.Value;

// Auto-commit with using statement
await using var session = await Session.CreateAsync(accountName, accountKey, sessionId);
session["Key"] = "Value";
// Automatically committed on disposal

Error Logging

Enhanced Error Logging

try
{
    // Your code
}
catch (Exception ex)
{
    // With automatic caller information
    var errorLog = ErrorLogData.CreateWithCallerInfo(
        "Failed to process customer data",
        ErrorCodeTypes.Error,
        customerId);
    
    await errorLog.LogErrorAsync(accountName, accountKey);
}

// Clear old error logs
await ErrorLogData.ClearOldDataAsync(accountName, accountKey, daysOld: 30);

Blob Storage Operations

Upload and Search with Tags

var azureBlobs = new AzureBlobs(accountName, accountKey, "my-container");

// Upload with tags
var uri = await azureBlobs.UploadFileAsync(
    "path/to/file.pdf",
    indexTags: new Dictionary<string, string>
    {
        { "category", "invoices" },
        { "year", "2024" }
    });

// Search by tags using lambda
var invoices = await azureBlobs.GetCollectionAsync(b => 
    b.Tags["category"] == "invoices" && b.Tags["year"] == "2024");

// Batch operations
var downloaded = await azureBlobs.DownloadFilesAsync(
    b => b.Tags["type"] == "report" && b.UploadDate > DateTime.Today.AddDays(-7),
    @"C:\Downloads");

Advanced Features

Complex Queue Workflow Management

public class WorkflowManager
{
    private readonly string _accountName;
    private readonly string _accountKey;
    
    // Create a multi-stage workflow with separate queues
    public async Task CreateWorkflowQueues(List<Document> documents)
    {
        // Stage 1: Validation Queue
        var validationList = new StateList<Document>(documents)
        {
            Description = "Document Validation"
        };
        var validationQueue = QueueData<Document>.CreateFromStateList(
            validationList, "DocumentWorkflow", "Stage1_Validation");
        await validationQueue.SaveQueueAsync(_accountName, _accountKey);
        
        // Stage 2: Processing Queue (empty initially)
        var processingList = new StateList<Document>()
        {
            Description = "Document Processing"
        };
        var processingQueue = QueueData<Document>.CreateFromStateList(
            processingList, "DocumentWorkflow", "Stage2_Processing");
        await processingQueue.SaveQueueAsync(_accountName, _accountKey);
        
        // Stage 3: Archive Queue (empty initially)
        var archiveList = new StateList<Document>()
        {
            Description = "Document Archival"
        };
        var archiveQueue = QueueData<Document>.CreateFromStateList(
            archiveList, "DocumentWorkflow", "Stage3_Archive");
        await archiveQueue.SaveQueueAsync(_accountName, _accountKey);
    }
    
    // Process workflow stage and move items to next stage
    public async Task ProcessWorkflowStage(string currentStage, string nextStage)
    {
        // Get current stage queue
        var currentQueue = await QueueData<Document>.GetQueueAsync(
            currentStage, _accountName, _accountKey, false);
        
        if (currentQueue == null) return;
        
        var stateList = currentQueue.GetData();
        var processedItems = new List<Document>();
        
        // Process items
        while (stateList.MoveNext())
        {
            var doc = stateList.Current;
            
            try
            {
                // Process based on stage
                switch (currentStage)
                {
                    case "Stage1_Validation":
                        ValidateDocument(doc);
                        break;
                    case "Stage2_Processing":
                        ProcessDocument(doc);
                        break;
                    case "Stage3_Archive":
                        ArchiveDocument(doc);
                        break;
                }
                
                processedItems.Add(doc);
            }
            catch (Exception ex)
            {
                Console.WriteLine($"Error processing {doc.Id}: {ex.Message}");
                // Item stays in current queue for retry
            }
            
            // Save progress
            if (stateList.CurrentIndex % 10 == 0)
            {
                await QueueData<Document>.SaveProgressAsync(
                    stateList, _accountName, _accountKey);
            }
        }
        
        // Move processed items to next stage
        if (processedItems.Any() && !string.IsNullOrEmpty(nextStage))
        {
            var nextQueue = await QueueData<Document>.GetQueueAsync(
                nextStage, _accountName, _accountKey, false);
            
            if (nextQueue != null)
            {
                var nextList = nextQueue.GetData();
                nextList.AddRange(processedItems);
                
                await QueueData<Document>.SaveProgressAsync(
                    nextList, _accountName, _accountKey);
            }
        }
        
        // Clear processed items from current queue
        currentQueue.PutData(new StateList<Document>());
        await currentQueue.SaveQueueAsync(_accountName, _accountKey);
    }
}

Queue Health Monitoring Dashboard

public class QueueMonitor
{
    public async Task<DashboardData> GetQueueDashboard(string category)
    {
        var stats = await QueueData<object>.GetCategoryStatisticsAsync(
            category, accountName, accountKey);
        
        var statuses = await QueueData<object>.GetQueueStatusesAsync(
            category, accountName, accountKey);
        
        var dashboard = new DashboardData
        {
            Category = category,
            TotalQueues = stats.TotalQueues,
            ActiveQueues = stats.ActiveQueues,
            CompletedQueues = stats.CompletedQueues,
            TotalItems = stats.TotalItems,
            ProcessedItems = stats.TotalProcessedItems,
            AverageProgress = stats.AverageProgress,
            
            QueueDetails = statuses.Select(s => new QueueDetail
            {
                QueueId = s.QueueId,
                Status = s.ProcessingStatus,
                Progress = s.PercentComplete,
                ItemsProcessed = s.ProcessedItems,
                TotalItems = s.TotalItems,
                LastModified = s.LastModified,
                EstimatedCompletion = EstimateCompletion(s)
            }).ToList(),
            
            HealthStatus = DetermineHealthStatus(stats, statuses)
        };
        
        return dashboard;
    }
    
    private string DetermineHealthStatus(
        QueueCategoryStatistics stats, 
        List<QueueMetadata> statuses)
    {
        if (stats.ActiveQueues == 0 && stats.TotalQueues > 0)
            return "Idle";
        
        if (stats.AverageProgress < 25)
            return "Starting";
        
        if (stats.AverageProgress > 75)
            return "Completing";
        
        if (statuses.Any(s => s.ProcessingStatus == "Error"))
            return "Attention Required";
        
        return "Healthy";
    }
}

Performance Optimizations

Performance Metrics (v2.3)

Best Practices for Performance

1. One queue per logical work unit - Don't mix unrelated items
2. Use categories for grouping - Leverage PartitionKey for related queues
3. Process queues in parallel - Each queue is independent
4. Save checkpoints frequently - But not after every item
5. Clean up completed queues - Don't let them accumulate

Migration Guide

From v2.2 to v2.3

The main change is conceptual - each QueueData now clearly represents ONE queue:

// v2.2 (still works but semantics are clearer)
var queue = new QueueData<Item>();
queue.PutData(stateList); // This is ONE queue

// v2.3 (recommended - explicit queue creation)
var queue = QueueData<Item>.CreateFromStateList(
    stateList, 
    "CategoryName",  // Groups related queues
    "UniqueQueueId"  // Identifies this specific queue
);

// v2.3 - Working with multiple queues
// Each queue is independent
var queues = await QueueData<Item>.GetQueuesAsync(
    "CategoryName", accountName, accountKey);

foreach (var kvp in queues)
{
    var queueId = kvp.Key;      // Unique queue identifier
    var stateList = kvp.Value;  // This queue's StateList
    // Process each queue independently
}

Key Changes to Understand

1. Queue Identity: Each QueueData has a unique QueueID (RowKey)
2. Queue Category: Multiple queues can share a Name (PartitionKey)
3. Independence: Each queue has its own StateList and progress
4. Clearer Methods: New methods like GetQueueAsync for single queues

Upgrading Queue Processing Code

// OLD (v2.2) - Unclear if dealing with one or many queues
var data = await QueueData<Order>.GetQueuesAsync(
    "Orders", accountName, accountKey);
// 'data' could be confusing - is it one queue or many?

// NEW (v2.3) - Clear that we're getting multiple queues
var queues = await QueueData<Order>.GetQueuesAsync(
    "Orders", accountName, accountKey);
// 'queues' is clearly a dictionary of QueueID -> StateList

// NEW (v2.3) - Get a specific queue
var specificQueue = await QueueData<Order>.GetQueueAsync(
    "OrderQueue_2024Q1", accountName, accountKey);
// Clear that we're getting ONE queue

Best Practices

Queue Design Best Practices (v2.3)

1. One Queue = One Logical Unit of Work

   • Don't mix different types of work in one queue
   • Each queue should represent a cohesive set of items

2. Use Meaningful Queue IDs

   • Include date/time stamps for time-based processing
   • Include source or type information
   • Examples: "Orders_2024Q1", "CustomerImport_20240315", "Report_Daily_20240315"

3. Leverage Categories for Organization

   • Use the Name (PartitionKey) to group related queues
   • Makes it easy to process all queues of a type
   • Examples: "OrderProcessing", "ReportGeneration", "DataImport"

4. Monitor Queue Health

   • Regularly check category statistics
   • Set up alerts for stalled queues
   • Clean up completed queues

5. Process Queues Appropriately

   • Use parallel processing for independent queues
   • Use sequential processing for dependent workflows
   • Save checkpoints based on processing time, not item count

StateList Best Practices

1. Always set Description for debugging and monitoring
2. Save checkpoints regularly for large processing tasks
3. Use CurrentIndex for progress reporting
4. Leverage Find() for navigation - automatically sets current
5. Use implicit conversions for cleaner code

General Guidelines

1. Choose the right granularity:

   • Too many small queues = management overhead
   • Too few large queues = less parallelization opportunity
   • Find the right balance for your use case

2. Implement proper error handling:

   • Save state before risky operations
   • Log errors with queue and position information
   • Consider dead letter queues for failed items

3. Clean up regularly:

   • Delete completed queues
   • Archive old queue data if needed
   • Monitor storage usage

Troubleshooting

Common Issues (v2.3)

Issue: Confusion about queue identity

• Solution: Remember: QueueID = unique queue, Name = category
• Check: Use GetQueueAsync for single queue, GetQueuesAsync for category

Issue: Queues not processing in parallel

• Solution: Each queue is independent - process them concurrently
• Check: Ensure you're not serializing queue processing unnecessarily

Issue: Lost queue progress

• Solution: Save checkpoints more frequently
• Check: Verify SaveProgressAsync is being called

Issue: Too many queues accumulating

• Solution: Delete completed queues
• Check: Implement queue lifecycle management

Support and Resources

• NuGet Package: ASCDataAccessLibrary
• Source Code: GitHub Repository
• Issues: GitHub Issues
• Documentation: This document
• Version: 2.3.0
• License: MIT

Changelog

Version 2.3.0 (Current)

• Clarified queue architecture: 1 QueueData = 1 Queue = 1 StateList
• Added GetQueueAsync for single queue retrieval
• Added GetQueueStatusesAsync for monitoring
• Added GetCategoryStatisticsAsync for analytics
• Improved queue management methods
• Enhanced documentation with clearer examples
• Better support for parallel queue processing

Version 2.2.0

• Added StateList<T> for position-aware collections
• Enhanced QueueData<T> with StateList integration
• Added position-based pagination methods
• Implemented progress tracking and monitoring
• Added checkpoint and recovery support

Version 2.1.0

• Added DynamicEntity for runtime-defined schemas
• Introduced IDynamicProperties interface
• Implemented TableEntityTypeCache for performance
• Enhanced TableEntityBase serialization

Version 2.0.0

• Lambda expression support
• Hybrid server/client filtering
• Batch operations with progress tracking
• Pagination support
• Async-first pattern
• Blob storage with tag indexing

Last Updated: 2024 Version: 2.3.0