ASCDataAccessLibrary 2.5.0

ASCDataAccessLibrary v2.5 Documentation

What's New in Version 2.5

Major Features

• Enhanced StateList<T>: Full IList<T> implementation with position tracking through serialization
• Improved QueueData<T>: Now uses StateList internally for better state management and position tracking
• Advanced Error Logging: Enhanced with caller information capture and comprehensive exception details
• Optimized Blob Operations: Improved lambda expression support for blob filtering
• Performance Enhancements: Continued optimization of type caching and property access

Key Improvements Since v2.1

• StateList now implements IList<T> interface completely
• QueueData provides better queue management with position tracking
• Enhanced error logging with automatic stack trace analysis
• Improved hybrid filtering for both tables and blobs
• Better support for complex lambda expressions
• Extended batch operation capabilities

---

Table of Contents

1. Installation
2. Basic Concepts
3. Dynamic Entities
4. Working with Data Access
5. StateList - Advanced Collection Management
6. Queue Data Management
7. Session Management
8. Error Logging
9. Blob Storage Operations
10. Advanced Features
11. Performance Optimizations
12. Migration Guide
13. Best Practices

---

Installation

Install the package from NuGet:

Install-Package ASCDataAccessLibrary -Version 2.5.0

Or using the .NET CLI:

dotnet add package ASCDataAccessLibrary --version 2.5.0

---

Basic Concepts

Core Components

• DataAccess<T>: Generic class for CRUD operations with Azure Table Storage
• TableEntityBase: Base class for all table entities with automatic field chunking
• ITableExtra: Interface requiring TableReference and GetIDValue() implementation
• IDynamicProperties: Interface for entities supporting runtime properties
• DynamicEntity: Pre-built implementation for schema-less operations
• StateList<T>: Position-aware list that maintains state through serialization (NEW in v2.5)
• QueueData<T>: Enhanced queue management with StateList integration (IMPROVED in v2.5)
• Session: Advanced session management for stateful applications
• ErrorLogData: Comprehensive error logging with automatic caller information (ENHANCED in v2.5)
• AzureBlobs: Blob storage operations with tag-based indexing

Architecture Overview

┌─────────────────────────────────────┐
│         Your Application            │
├─────────────────────────────────────┤
│     Strongly-Typed Entities         │
│            ↕                        │
│     TableEntityBase                 │ ← Supports IDynamicProperties
│            ↕                        │
│     ITableExtra                     │
├─────────────────────────────────────┤
│      Dynamic Entities               │
│            ↕                        │
│      DynamicEntity                  │
│            ↕                        │
│     IDynamicProperties              │
├─────────────────────────────────────┤
│    State Management (v2.5)          │
│            ↕                        │
│      StateList<T>                   │ ← Full IList<T> implementation
│            ↕                        │
│      QueueData<T>                   │ ← Uses StateList internally
├─────────────────────────────────────┤
│      DataAccess<T>                  │ ← Works with all entity types
├─────────────────────────────────────┤
│      Azure Table Storage            │
└─────────────────────────────────────┘

---

Dynamic Entities

Overview

Dynamic entities allow you to work with Azure Table Storage without defining compile-time schemas. Perfect for:

• Multi-tenant applications with varying schemas
• Integration with external systems
• Rapid prototyping
• Scenarios where table structure changes frequently

Creating Dynamic Entities

// Create a dynamic entity with runtime-defined table name
var entity = new DynamicEntity("SystemName_EntityType", "partitionKey", "rowKey");

// Add properties using indexer syntax
entity["EmployeeName"] = "John Doe";
entity["HireDate"] = DateTime.UtcNow;
entity["Salary"] = 95000.50;
entity["IsActive"] = true;
entity["Department"] = "Engineering";

// Alternative: Use method syntax
entity.SetProperty("ManagerId", "emp_manager123");
entity.SetProperty("YearsExperience", 8);

Working with Dynamic Properties

// Retrieve properties with type conversion
string name = entity.GetProperty<string>("EmployeeName");
DateTime hireDate = entity.GetProperty<DateTime>("HireDate");
bool isActive = entity.GetProperty<bool>("IsActive");

// Check if property exists
if (entity.HasProperty("Salary"))
{
    decimal salary = entity.GetProperty<decimal>("Salary");
}

// Get all properties
Dictionary<string, object> allProps = entity.GetAllProperties();

// Remove a property
entity.RemoveProperty("TempField");

---

Working with Data Access

Creating Strongly-Typed Entities

public class Customer : TableEntityBase, ITableExtra
{
    // Primary key - stored in PartitionKey
    public string CompanyId
    {
        get => this.PartitionKey;
        set => this.PartitionKey = value;
    }
    
    // Unique identifier - stored in RowKey
    public string CustomerId
    {
        get => this.RowKey;
        set => this.RowKey = value;
    }
    
    // Custom properties
    public string Name { get; set; }
    public string Email { get; set; }
    public string LargeDataField { get; set; } // Auto-chunked if >32KB
    public string Status { get; set; }
    public DateTime CreatedDate { get; set; }
    
    // ITableExtra implementation
    public string TableReference => "Customers";
    public string GetIDValue() => this.CustomerId;
}

CRUD Operations

var dataAccess = new DataAccess<Customer>(accountName, accountKey);

// Create/Update
var customer = new Customer
{
    CompanyId = "company123",
    CustomerId = "cust456",
    Name = "Acme Corporation",
    Email = "contact@acme.com",
    Status = "Active",
    CreatedDate = DateTime.UtcNow
};
await dataAccess.ManageDataAsync(customer);

// Retrieve by RowKey
var retrieved = await dataAccess.GetRowObjectAsync("cust456");

// Retrieve with lambda (v2.5 enhanced hybrid filtering)
var active = await dataAccess.GetCollectionAsync(c => 
    c.Email == "contact@acme.com" && c.Status == "Active");

// Complex queries with automatic hybrid filtering
var results = await dataAccess.GetCollectionAsync(c => 
    c.Name.ToLower().Contains("acme") || c.CreatedDate > DateTime.Today.AddDays(-30));

// Pagination
var page = await dataAccess.GetPagedCollectionAsync(pageSize: 50);
while (page.HasMore)
{
    // Process page.Items
    page = await dataAccess.GetPagedCollectionAsync(
        pageSize: 50, 
        continuationToken: page.ContinuationToken);
}

// Batch operations with progress tracking
var progress = new Progress<BatchUpdateProgress>(p =>
    Console.WriteLine($"Progress: {p.PercentComplete:F1}% - Batch {p.CompletedBatches}/{p.TotalBatches}"));

var result = await dataAccess.BatchUpdateListAsync(
    customers, 
    TableOperationType.InsertOrReplace, 
    progress);

Console.WriteLine($"Success: {result.Success}, Processed: {result.SuccessfulItems}, Failed: {result.FailedItems}");

---

StateList - Advanced Collection Management

Overview (NEW in v2.5)

StateList<T> is an enhanced collection that maintains its current position through serialization/deserialization. It implements the full IList<T> interface and provides position-aware navigation.

Key Features

• Position Tracking: Maintains CurrentIndex through serialization
• Full IList<T> Implementation: Compatible with all standard list operations
• Navigation Methods: MoveNext, MovePrevious, First, Last, Reset
• String-based Indexer: Quick case-insensitive searches
• LINQ Support: Where, Except, Intersect, and more
• JSON Serializable: Works seamlessly with QueueData for persistence

Basic Usage

// Create a StateList
var stateList = new StateList<string> { "Item1", "Item2", "Item3" };
stateList.Description = "Processing Queue";

// Navigate through items
stateList.First(); // CurrentIndex = 0
Console.WriteLine(stateList.Current); // "Item1"

stateList.MoveNext(); // CurrentIndex = 1
Console.WriteLine(stateList.Current); // "Item2"

// Check navigation availability
if (stateList.HasNext)
{
    stateList.MoveNext();
}

// Get current item with index
var peek = stateList.Peek; // Returns (Data: "Item3", Index: 2)

Advanced Operations

// Add items with state management
var list = new StateList<ProcessTask>();
list.Add(task1, setCurrent: true); // Adds and sets as current
list.AddRange(moreTasks, setCurrentToFirst: true); // Adds range and sets current to first added

// Find and set as current
var found = list.Find(t => t.Priority == "High");
// If found, CurrentIndex is automatically updated

// String-based search
var item = list["TaskName123"]; // Case-insensitive search by ToString()

// LINQ operations return new StateList
var filtered = list.Where(t => t.Status == "Pending");
var excluded = list.Except(completedTasks);

// Sorting while maintaining current item
var currentItem = list.Current;
list.Sort((a, b) => a.Priority.CompareTo(b.Priority));
// CurrentIndex is updated to track the same item

// Remove operations adjust CurrentIndex intelligently
list.RemoveAt(2); // CurrentIndex adjusts if needed
list.RemoveAll(t => t.Status == "Cancelled");

Serialization Example

// StateList serializes with position preserved
var stateList = new StateList<string> { "A", "B", "C" };
stateList.CurrentIndex = 1; // Currently at "B"

// Serialize
string json = JsonConvert.SerializeObject(stateList);

// Deserialize - position is maintained
var restored = JsonConvert.DeserializeObject<StateList<string>>(json);
Console.WriteLine(restored.CurrentIndex); // 1
Console.WriteLine(restored.Current); // "B"

---

Queue Data Management

Overview (ENHANCED in v2.5)

QueueData<T> now uses StateList<T> internally, providing better state management and position tracking for queue processing.

Key Improvements in v2.5

• StateList Integration: Uses StateList<T> for internal data management
• Position Tracking: Automatically tracks processing position
• Percentage Complete: Real-time progress calculation
• Better Persistence: State is fully preserved through serialization
• Multiple Queue Support: Enhanced methods for managing multiple queues

Creating and Saving Queues

// Create a queue from a StateList
var tasks = new StateList<ProcessingTask>(taskList);
tasks.Description = "Daily Processing Tasks";
tasks.CurrentIndex = 0; // Start at beginning

var queue = QueueData<ProcessingTask>.CreateFromStateList(
    tasks, 
    "DailyTasks", 
    queueId: Guid.NewGuid().ToString()
);

// Save the queue
await queue.SaveQueueAsync(accountName, accountKey);

// Check progress
Console.WriteLine($"Queue: {queue.Name}");
Console.WriteLine($"Progress: {queue.PercentComplete:F1}%");
Console.WriteLine($"Current Position: {queue.LastProcessedIndex + 1}/{queue.TotalItemCount}");

Retrieving and Processing Queues

// Get a specific queue without deleting
var queue = await QueueData<ProcessingTask>.GetQueueAsync(
    queueId, accountName, accountKey);

if (queue != null)
{
    // Process items using StateList features
    while (queue.Data.HasNext)
    {
        queue.Data.MoveNext();
        var currentTask = queue.Data.Current;
        
        // Process the task
        await ProcessTask(currentTask);
        
        // Update processing status
        queue.ProcessingStatus = $"Processed {queue.Data.CurrentIndex + 1} of {queue.TotalItemCount}";
        
        // Save progress periodically
        if (queue.Data.CurrentIndex % 10 == 0)
        {
            await queue.SaveQueueAsync(accountName, accountKey);
        }
    }
    
    // Delete queue when complete
    await QueueData<ProcessingTask>.DeleteQueueAsync(
        queue.QueueID, accountName, accountKey);
}

Working with Multiple Queues

// Get all queues in a category
var allQueues = await QueueData<ProcessingTask>.GetQueuesAsync(
    "DailyTasks", accountName, accountKey);

Console.WriteLine($"Found {allQueues.Count} queues");

foreach (var q in allQueues)
{
    Console.WriteLine($"Queue {q.QueueID}: {q.PercentComplete:F1}% complete");
    Console.WriteLine($"  Status: {q.ProcessingStatus}");
    Console.WriteLine($"  Items: {q.TotalItemCount}");
    Console.WriteLine($"  Position: {q.LastProcessedIndex}");
}

// Delete completed queues
var completedQueueIds = allQueues
    .Where(q => q.PercentComplete >= 100)
    .Select(q => q.QueueID)
    .ToList();

await QueueData<ProcessingTask>.DeleteQueuesAsync(
    completedQueueIds, accountName, accountKey);

// Delete queues matching a condition
await QueueData<ProcessingTask>.DeleteQueuesMatchingAsync(
    accountName, accountKey,
    q => q.Timestamp < DateTime.UtcNow.AddDays(-7) // Delete week-old queues
);

Queue Recovery and Resumption

// Recover interrupted processing
var interruptedQueue = await QueueData<ProcessingTask>.GetQueueAsync(
    queueId, accountName, accountKey);

if (interruptedQueue != null)
{
    Console.WriteLine($"Resuming from position {interruptedQueue.Data.CurrentIndex}");
    
    // Continue from where we left off
    while (interruptedQueue.Data.HasNext)
    {
        interruptedQueue.Data.MoveNext();
        var task = interruptedQueue.Data.Current;
        
        try
        {
            await ProcessTask(task);
        }
        catch (Exception ex)
        {
            // Log error and save state
            interruptedQueue.ProcessingStatus = $"Error at item {interruptedQueue.Data.CurrentIndex}: {ex.Message}";
            await interruptedQueue.SaveQueueAsync(accountName, accountKey);
            throw;
        }
    }
}

---

Session Management

Creating and Using Sessions

// Asynchronous creation (recommended)
await using var session = await Session.CreateAsync(accountName, accountKey, sessionId);

// Store values
session["UserName"] = "John Doe";
session["LastLogin"] = DateTime.Now.ToString();
session["CartItems"] = JsonConvert.SerializeObject(items);
session["Preferences"] = JsonConvert.SerializeObject(userPrefs);

// Retrieve values
string userName = session["UserName"]?.Value;
var cartItems = JsonConvert.DeserializeObject<List<CartItem>>(session["CartItems"]?.Value);

// Session data is automatically committed on disposal

Session Maintenance

// Find and process stale sessions
var staleSessions = await session.GetStaleSessionsAsync();
foreach (var staleSessionId in staleSessions)
{
    // Process abandoned session (e.g., submit to CRM)
    await ProcessAbandonedSession(staleSessionId);
}

// Clean old session data (older than 2 hours by default)
await session.CleanSessionDataAsync();

// Restart session (clear all data)
await session.RestartSessionAsync();

// Refresh session data from database
await session.RefreshSessionDataAsync();

---

Error Logging

Enhanced Error Logging (v2.5)

The error logging system has been significantly enhanced with better caller information capture and exception detail extraction.

Basic Error Logging with Caller Info

try
{
    // Your code that might throw
    await ProcessCustomerData(customerId);
}
catch (Exception ex)
{
    // Automatic caller information capture (NEW in v2.5)
    var errorLog = ErrorLogData.CreateWithCallerInfo(
        $"Failed to process customer data: {ex.Message}",
        ErrorCodeTypes.Error,
        customerId);
    
    await errorLog.LogErrorAsync(accountName, accountKey);
}

Advanced Error Logging with State

// Log with formatted message and state (NEW in v2.5)
var state = new { CustomerId = customerId, Operation = "DataProcessing" };

var errorLog = ErrorLogData.CreateWithCallerInfo(
    state,
    ex,
    (s, e) => $"Operation {s.Operation} failed for customer {s.CustomerId}: {e.Message}",
    ErrorCodeTypes.Critical,
    customerId);

await errorLog.LogErrorAsync(accountName, accountKey);

Error Log Maintenance

// Clear old error logs (enhanced in v2.5)
await ErrorLogData.ClearOldDataAsync(accountName, accountKey, daysOld: 30);

// Clear specific error types
await ErrorLogData.ClearOldDataByType(
    accountName, 
    accountKey, 
    ErrorCodeTypes.Information, 
    daysOld: 7);

// Clear warnings older than 14 days
await ErrorLogData.ClearOldDataByType(
    accountName, 
    accountKey, 
    ErrorCodeTypes.Warning, 
    daysOld: 14);

Error Severity Levels

// Information - General logging
var info = ErrorLogData.CreateWithCallerInfo(
    "User logged in successfully", 
    ErrorCodeTypes.Information,
    userId);

// Warning - Non-critical issues
var warning = ErrorLogData.CreateWithCallerInfo(
    "API rate limit approaching", 
    ErrorCodeTypes.Warning);

// Error - Recoverable errors
var error = ErrorLogData.CreateWithCallerInfo(
    "Failed to send email, will retry", 
    ErrorCodeTypes.Error);

// Critical - System failures
var critical = ErrorLogData.CreateWithCallerInfo(
    "Database connection lost", 
    ErrorCodeTypes.Critical);

// Unknown - Unclassified issues
var unknown = ErrorLogData.CreateWithCallerInfo(
    "Unexpected state encountered", 
    ErrorCodeTypes.Unknown);

---

Blob Storage Operations

Basic Operations

var azureBlobs = new AzureBlobs(accountName, accountKey, "my-container");

// Configure file type restrictions
azureBlobs.AddAllowedFileType(".pdf", "application/pdf");
azureBlobs.AddAllowedFileType(".docx", "application/vnd.openxmlformats-officedocument.wordprocessingml.document");

// Upload with tags for indexing
var uri = await azureBlobs.UploadFileAsync(
    "documents/invoice.pdf",
    enforceFileTypeRestriction: true,
    indexTags: new Dictionary<string, string>
    {
        { "category", "invoices" },
        { "year", "2024" },
        { "status", "pending" }
    },
    metadata: new Dictionary<string, string>
    {
        { "customer", "ACME Corp" },
        { "amount", "5000.00" }
    });

Advanced Blob Search with Lambda Expressions

// Search by tags using lambda (enhanced in v2.5)
var invoices = await azureBlobs.GetCollectionAsync(b => 
    b.Tags["category"] == "invoices" && 
    b.Tags["year"] == "2024" &&
    b.Tags["status"] == "pending");

// Complex searches with date filtering
var recentDocs = await azureBlobs.GetCollectionAsync(b => 
    b.UploadDate > DateTime.Today.AddDays(-7) &&
    b.Size < 5 * 1024 * 1024); // Less than 5MB

// Search with metadata filtering (client-side)
var results = await azureBlobs.SearchBlobsAsync(
    searchText: "invoice",
    tagFilters: new Dictionary<string, string> { { "status", "pending" } },
    startDate: DateTime.Today.AddMonths(-1),
    endDate: DateTime.Today);

Batch Blob Operations

// Upload multiple files with progress tracking
var files = Directory.GetFiles(@"C:\Documents", "*.pdf")
    .Select(f => new BlobUploadInfo
    {
        FilePath = f,
        IndexTags = new Dictionary<string, string>
        {
            { "type", "document" },
            { "source", "batch_upload" },
            { "date", DateTime.Today.ToString("yyyy-MM-dd") }
        }
    });

var uploadResults = await azureBlobs.UploadMultipleFilesAsync(files);

// Process results
foreach (var result in uploadResults)
{
    if (result.Value.Success)
    {
        Console.WriteLine($"✓ Uploaded: {result.Key} -> {result.Value.BlobUri}");
    }
    else
    {
        Console.WriteLine($"✗ Failed: {result.Key} - {result.Value.ErrorMessage}");
    }
}

// Batch download based on criteria
var downloadResults = await azureBlobs.DownloadFilesAsync(
    b => b.Tags["type"] == "report" && b.UploadDate > DateTime.Today.AddDays(-30),
    @"C:\Downloads\Reports",
    preserveOriginalNames: true);

// Batch delete old temporary files
var deleteResults = await azureBlobs.DeleteMultipleBlobsAsync(
    b => b.Tags["temp"] == "true" && b.UploadDate < DateTime.Today.AddDays(-1));

Working with Blob Metadata

// Update blob tags
await azureBlobs.UpdateBlobTagsAsync("document.pdf", 
    new Dictionary<string, string>
    {
        { "status", "approved" },
        { "approver", "john.doe" },
        { "approval_date", DateTime.UtcNow.ToString("O") }
    });

// Get blob tags
var tags = await azureBlobs.GetBlobTagsAsync("document.pdf");
if (tags != null)
{
    foreach (var tag in tags)
    {
        Console.WriteLine($"{tag.Key}: {tag.Value}");
    }
}

---

Advanced Features

Hybrid Filtering (Enhanced in v2.5)

The library automatically optimizes lambda expressions for maximum performance:

// Server-side operations (fast)
var results = await dataAccess.GetCollectionAsync(x => 
    x.Status == "Active" && 
    x.CreatedDate > DateTime.Today.AddDays(-30) &&
    x.Priority >= 5);

// Mixed operations (automatic hybrid filtering)
var complex = await dataAccess.GetCollectionAsync(c => 
    c.CompanyName.ToLower().Contains("tech") || // Client-side
    c.Revenue > 1000000); // Server-side

// The library automatically:
// 1. Executes server-supported operations on Azure
// 2. Retrieves filtered dataset
// 3. Applies client-side operations locally

Pagination with Lambda Expressions

// Paginated retrieval with filtering
var firstPage = await dataAccess.GetPagedCollectionAsync(
    x => x.Status == "Active",
    pageSize: 100);

var allItems = new List<Customer>();
var currentPage = firstPage;

while (currentPage.HasMore)
{
    allItems.AddRange(currentPage.Items);
    
    currentPage = await dataAccess.GetPagedCollectionAsync(
        x => x.Status == "Active",
        pageSize: 100,
        continuationToken: currentPage.ContinuationToken);
}

Complex Query Building

// Build complex queries dynamically
var queryTerms = new List<DBQueryItem>
{
    new DBQueryItem 
    { 
        FieldName = "Status", 
        FieldValue = "Active", 
        HowToCompare = ComparisonTypes.eq 
    },
    new DBQueryItem 
    { 
        FieldName = "CreatedDate", 
        FieldValue = DateTime.Today.AddDays(-30).ToString("O"), 
        HowToCompare = ComparisonTypes.gt,
        IsDateTime = true 
    }
};

var results = await dataAccess.GetCollectionAsync(
    queryTerms, 
    QueryCombineStyle.and);

---

Performance Optimizations

Type Caching System

The library includes sophisticated type caching to minimize reflection overhead:

// TableEntityTypeCache provides:
// - Cached property discovery
// - O(1) property lookup
// - Optimized DateTime type checking
// - Reduced reflection calls

// Performance metrics:
// First entity: ~50ms (includes reflection)
// Subsequent entities: <1ms (uses cache)
// Batch of 1000 entities: ~55ms total (vs ~500ms without caching)

Batch Processing Optimization

// Batch operations respect Azure's 100-entity limit
// Automatic partitioning by PartitionKey
// Progress tracking for long operations

var progress = new Progress<BatchUpdateProgress>(p =>
{
    Console.WriteLine($"Batch {p.CompletedBatches}/{p.TotalBatches}");
    Console.WriteLine($"Items {p.ProcessedItems}/{p.TotalItems}");
    Console.WriteLine($"Progress: {p.PercentComplete:F1}%");
});

var result = await dataAccess.BatchUpdateListAsync(
    largeDataset, // Can be thousands of items
    TableOperationType.InsertOrReplace,
    progress);

Performance Best Practices

1. Reuse DataAccess instances for the same table
2. Use batch operations for multiple updates (automatic 100-item chunking)
3. Enable progress tracking for visibility into long operations
4. Use pagination for large result sets
5. Leverage async/await throughout your application
6. Use hybrid filtering - let the library optimize automatically
7. Cache DataAccess instances when working with multiple tables

---

Migration Guide

From v2.1 to v2.5

Version 2.5 is fully backward compatible. No breaking changes.

Adopting New Features

Upgrade to StateList

// OLD - Using regular List
List<Task> tasks = GetTasks();
int currentIndex = 0;
// Manual position tracking...

// NEW - Using StateList
StateList<Task> tasks = new StateList<Task>(GetTasks());
tasks.Description = "Processing Queue";
// Automatic position tracking
while (tasks.HasNext)
{
    tasks.MoveNext();
    ProcessTask(tasks.Current);
}

Upgrade Queue Management

// OLD - QueueData with List<T>
var queue = new QueueData<Task>();
queue.PutData(taskList);
await queue.SaveQueueAsync(accountName, accountKey);

// NEW - QueueData with StateList<T>
var stateList = new StateList<Task>(taskList);
var queue = QueueData<Task>.CreateFromStateList(
    stateList, "TaskQueue", Guid.NewGuid().ToString());
    
// Now with position tracking
Console.WriteLine($"Progress: {queue.PercentComplete:F1}%");
await queue.SaveQueueAsync(accountName, accountKey);

Enhanced Error Logging

// OLD - Manual error logging
var errorLog = new ErrorLogData(ex, "Error occurred", ErrorCodeTypes.Error);

// NEW - Automatic caller information
var errorLog = ErrorLogData.CreateWithCallerInfo(
    "Error occurred", 
    ErrorCodeTypes.Error,
    customerId);
// Automatically captures method name, line number, and stack trace

---

Best Practices

General Guidelines

1. Entity Design

   • Use strongly-typed entities for stable schemas
   • Use DynamicEntity for flexible/evolving schemas
   • Implement ITableExtra correctly for all entities

2. State Management

   • Use StateList<T> for collections that need position tracking
   • Leverage QueueData<T> for persistent queue processing
   • Save queue state periodically during long operations

3. Error Handling

   • Always use CreateWithCallerInfo for error logging
   • Set appropriate severity levels
   • Clean old logs periodically
   • Include relevant context (customer ID, operation, etc.)

4. Performance

   • Reuse DataAccess instances
   • Use batch operations for bulk updates
   • Enable progress tracking for visibility
   • Implement pagination for large datasets

5. Blob Storage

   • Use tags for searchable metadata (max 10)
   • Use metadata for non-searchable information
   • Leverage lambda expressions for complex searches
   • Batch operations when working with multiple files

StateList Best Practices

1. Navigation

   • Always check HasNext/HasPrevious before moving
   • Use First() or Reset() to restart processing
   • Save CurrentIndex periodically for recovery

2. Modifications

   • Use Add with setCurrent parameter appropriately
   • Be aware that Remove operations adjust CurrentIndex
   • Sort operations maintain the current item reference

3. Serialization

   • StateList preserves CurrentIndex through JSON serialization
   • Use with QueueData for persistent state management
   • Description field helps identify queue purpose

Queue Processing Best Practices

1. Queue Creation

   • Use meaningful queue names (PartitionKey)
   • Generate unique QueueIDs
   • Set initial CurrentIndex appropriately

2. Processing

   • Save queue state periodically
   • Update ProcessingStatus for monitoring
   • Handle errors gracefully with state preservation

3. Cleanup

   • Delete completed queues
   • Use DeleteQueuesMatchingAsync for bulk cleanup
   • Monitor old/stale queues

Security Considerations

1. Connection Strings

   • Never hardcode account keys
   • Use Azure Key Vault or similar
   • Rotate keys regularly

2. Data Validation

   • Validate input before storage
   • Sanitize data from external sources
   • Handle large data appropriately (auto-chunking)

3. Access Control

   • Implement proper authentication
   • Use separate storage accounts for different environments
   • Monitor access patterns

---

Troubleshooting

Common Issues and Solutions

Issue: StateList CurrentIndex not preserved

• Solution: Ensure proper JSON serialization settings are used
• Check: Items property must be serialized along with CurrentIndex

Issue: Queue processing resumes from beginning

• Solution: Load existing queue before processing
• Check: Use GetQueueAsync to retrieve existing queue state

Issue: Hybrid filtering performance

• Solution: Understand which operations are server vs client-side
• Note: Complex string operations (ToLower, Contains) require client-side processing

Issue: Batch operations failing

• Solution: Azure limit is 100 items per batch
• Note: Library automatically chunks, but ensure PartitionKey consistency

Issue: Large field causing errors

• Solution: TableEntityBase automatically chunks fields >32KB
• Check: Ensure entity inherits from TableEntityBase

Issue: Blob tag searches not finding results

• Solution: Maximum 10 tags per blob
• Check: Tag values must match exactly (case-sensitive)

---

API Reference Summary

Key Classes

• DataAccess<T>: Main data access class
• TableEntityBase: Base class for entities
• DynamicEntity: Runtime-defined entities
• StateList<T>: Position-aware collection
• QueueData<T>: Persistent queue management
• Session: Session state management
• ErrorLogData: Error logging
• AzureBlobs: Blob storage operations

Key Interfaces

• ITableExtra: Required for table entities
• IDynamicProperties: Dynamic property support
• IList<T>: Implemented by StateList

Key Enums

• TableOperationType: Insert, Update, Delete operations
• ComparisonTypes: Query comparison operators
• QueryCombineStyle: AND/OR query combination
• ErrorCodeTypes: Error severity levels

---

Performance Metrics

Operation	v2.1	v2.5	Improvement
Read 1000 entities	~55ms	~50ms	~10%
Write 1000 entities	~50ms	~45ms	~10%
StateList operations	N/A	<1ms	New feature
Queue save/load	~100ms	~80ms	~20%
Error logging with caller info	~10ms	~5ms	~50%
Blob tag search (10 blobs)	~200ms	~150ms	~25%

---

Support and Resources

• NuGet Package: ASCDataAccessLibrary
• Source Code: GitHub Repository
• Issues: GitHub Issues
• Documentation: This document
• Version: 2.5.0
• License: MIT
• Author: ASC Development Team

---

Changelog

Version 2.5.0 (Current)

• StateList<T>: Full IList<T> implementation with position tracking
• QueueData<T>: Enhanced with StateList integration
• ErrorLogData: Improved caller information capture
• Performance: Continued optimization of caching systems
• Hybrid Filtering: Enhanced support for complex expressions
• Documentation: Comprehensive update with examples

Version 2.1.0

• Added DynamicEntity for runtime schemas
• Introduced IDynamicProperties interface
• Implemented TableEntityTypeCache
• Enhanced TableEntityBase serialization
• Added error log cleanup methods

Version 2.0.0

• Lambda expression support
• Hybrid server/client filtering
• Batch operations with progress
• Pagination support
• Async-first pattern
• Blob storage with tag indexing

Version 1.0.0

• Initial release
• Basic CRUD operations
• Session management
• Error logging
• Field chunking support

---

Last Updated: 2024
Version: 2.5.0
© 2024 ASC Development Team - All Rights Reserved