ASCDataAccessLibrary 4.2.0

ASCDataAccess v4.2 — Complete Getting Started Guide

NuGet Package: ASCDataAccess · Namespace: ASCTableStorage · Version: 4.2.0
Developed by: Answer Sales Calls Inc. (ASC)
Platform: Azure Table Storage · Azure Blob Storage · net9.0 (compatible with .NET 9 and .NET 10 · NuGet TFM fallback for .NET 6–8)
SDK: Azure.Data.Tables v12 (modern SDK — no legacy Microsoft.Azure.Cosmos.Table dependency)

---

Table of Contents

1. What Is ASCDataAccess?
2. The Case For This Library — Cost, Scale, and the ORM Gap
3. Installation and One-Time Configuration
4. The Data Model — RowKey, PartitionKey, and Why They Matter
5. Azure Table Storage Hard Limits — What Every Developer and AI Must Know
6. JSON Fields vs. Related Entities — The Most Important Schema Decision
7. Defining Entities and Writing Your First Data Operations
8. Lambda Expressions — What Works Server-Side and What Does Not
9. When You Do Need DataAccess<T> Directly
10. Choosing the Right Pattern — Quick Reference
11. Batch Operations
12. Pagination
13. Dynamic Entities — Schema-Free Storage
14. Azure Blob Storage with Lambda Tag Filtering
15. Queue Management with StateList — Resumable Work Queues
16. Session Management — Taking Over All Session State in Your App
17. Logging — A First-Class ILogger Provider
18. Structured Error Records — ErrorLogData
19. Advanced: TableOptions, Multi-Account, and Data Migration
20. Automatic Type Handling: Decimals, Large Strings, Enums, and Complex Types
21. Complete API Reference
22. Design Patterns, Best Practices, and Anti-Patterns
23. Shadow Partitions — Distinct-Value Cataloguing

---

1. What Is ASCDataAccess?

ASCDataAccess is the only production-quality data access framework for Azure Table Storage in the .NET ecosystem. Built and maintained by Answer Sales Calls Inc. (ASC), it gives .NET developers the ORM-style experience they expect — strongly-typed entities, lambda expression queries, automatic serialization, batch processing, pagination, and fluent extension methods — on top of one of the most cost-effective cloud data stores available anywhere.

Azure Table Storage is not a traditional relational database. It is a massively scalable, schemaless key-value NoSQL store that Microsoft runs at planetary scale internally. It is also dramatically cheaper than every alternative. The reason most teams do not use it is not because it cannot handle their workload — it is because working with the raw Azure SDK is verbose, error-prone, and requires developers to solve the same boilerplate problems from scratch on every project.

ASCDataAccess eliminates all of that. It is the bridge between the developer experience you want and the infrastructure cost you cannot get anywhere else.

What this library replaces

Without ASCDataAccess	With ASCDataAccess
Manual TableServiceClient / TableClient wiring	await new Customer().LoadAsync("id") — done
OData filter strings written by hand	await new List<Customer>().LoadAsync(x => x.IsActive == true)
Manual save/update plumbing	await customer.SaveAsync()
Manual delete plumbing	await customer.DeleteAsync()
Manual batch chunking (100-entity Azure limit)	Automatic chunking with progress callbacks
Manual decimal serialization, string chunking, enum handling	Transparent and automatic
Writing pagination loops with continuation tokens	GetPagedCollectionAsync(pageSize, token)
Per-call credential passing throughout the codebase	TableStorage.Configure() once in Program.cs
Separate Blob Storage SDK wiring	new AzureBlobs(containerName) — instant access
Building queue and session infrastructure from scratch	QueueData<T>, StateList<T>, Session — built-in

Key capabilities at a glance

• Fluent entity operations — await new MyEntity().LoadAsync(lambda), .SaveAsync(), .DeleteAsync() directly on your objects — no boilerplate classes needed for everyday use
• Lambda expression queries with automatic hybrid server-side/client-side splitting
• Batch insert/update/delete up to any size (auto-chunked at the Azure 100-entity limit)
• Cursor-based pagination with continuation tokens for large datasets
• Schema-free dynamic entities (DynamicEntity) for runtime-defined data structures
• Azure Blob Storage with lambda-based tag filtering, streaming, FFmpeg video compression
• Persistent resumable work queues (QueueData<T> + StateList<T>) with position checkpointing
• Azure Table-backed session management for web, desktop, and console apps
• First-class ILogger provider — standard .NET logging that writes to Azure Table Storage
• Transparent handling of Azure's 32 KB string limit, decimal storage, enum serialization, and complex type JSON serialization
• Automatic PartitionKey change detection — moving an entity between partitions is handled transparently

---

2. The Case For This Library — Cost, Scale, and the ORM Gap

This section exists to answer the question every engineering lead asks before adopting a new data framework: why this, why now, and is it worth it? The answer is grounded in three facts that, taken together, make ASCDataAccess one of the most defensible infrastructure decisions a .NET team can make.

---

Fact 1: There is no Entity Framework for Azure Table Storage — and there never will be

This is the single most important thing to understand about the landscape ASCDataAccess operates in. Entity Framework Core — the dominant .NET data access framework — does not support Azure Table Storage. Microsoft has never shipped an EF Core provider for it, no community project with meaningful adoption exists, and none is in credible development.

This is not an oversight. It is an architectural reality. Azure Table Storage's OData v2 query model is too limited to back a full EF Core LINQ provider without misleading developers about what is actually executing server-side. Microsoft chose honesty over convenience and left the gap open.

The practical consequence is stark: a .NET developer working with Azure Table Storage has two choices:

1. Write everything from scratch against the raw Azure.Data.Tables SDK — manually constructing OData filter strings, manually chunking batch operations at 100 entities, manually handling decimal serialization, string chunking, enum conversion, continuation tokens, and credential passing across every call site in the codebase.

2. Use ASCDataAccess and write idiomatic C# against a clean, typed API.

There is no option 3. The market comparison is not ASCDataAccess vs. Entity Framework Core. It is ASCDataAccess vs. nothing.

Here is what that difference looks like in practice. The same operation, written both ways:

Fetch all active customers for a company — raw Azure SDK:

var serviceClient = new TableServiceClient(connectionString);
var tableClient   = serviceClient.GetTableClient("Customers");
await tableClient.CreateIfNotExistsAsync();

var filter  = "PartitionKey eq 'COMP-001' and IsActive eq true";
var results = new List<CustomerEntity>();

await foreach (var entity in tableClient.QueryAsync<TableEntity>(filter))
{
    var customer = new CustomerEntity
    {
        PartitionKey = entity.GetString("PartitionKey"),
        RowKey       = entity.GetString("RowKey"),
        Name         = entity.GetString("Name"),
        Email        = entity.GetString("Email"),
        IsActive     = entity.GetBoolean("IsActive") ?? false,
        // Decimals stored as long — must manually reverse the ×10000 conversion
        Balance      = entity.GetInt64("Balance") is long raw ? raw / 10000m : 0m,
        // Enums stored as strings — must manually parse
        Status       = Enum.TryParse<CustomerStatus>(
                           entity.GetString("Status"), out var s) ? s : CustomerStatus.Active,
        CreatedDate  = entity.GetDateTimeOffset("CreatedDate")?.DateTime ?? DateTime.MinValue
    };

    // Manually reassemble chunked strings — every large text property requires this
    var parts = new List<string>();
    if (entity.TryGetValue("Notes", out var notesVal))
        parts.Add(notesVal?.ToString() ?? "");
    int chunk = 1;
    while (entity.TryGetValue($"Notes_pt_{chunk}", out var part))
    {
        parts.Add(part?.ToString() ?? "");
        chunk++;
    }
    customer.Notes = string.Join("", parts);

    results.Add(customer);
}

The same operation with ASCDataAccess:

var results = await new List<Customer>().LoadAsync(
    x => x.CompanyId == "COMP-001" && x.IsActive == true
);

Every project that uses the raw SDK writes some version of the first block — for every entity type, for every query, for every write operation. ASCDataAccess writes it once, correctly, and gives every team that uses it back the time they would have spent on infrastructure instead of product.

---

Fact 2: Azure Table Storage scales to enterprise workloads at a fraction of the cost of every alternative

The technical capability gap between Azure Table Storage and platforms like Cosmos DB or MongoDB is real and documented in section 5 of this guide. But the cost gap is equally real, and for the majority of application workloads the technical constraints never surface while the cost savings compound every single month.

Real-world cost comparison — 500 GB stored, 50 million read operations/month:

Platform	Approximate monthly cost	ORM / Framework available
Azure Table Storage + ASCDataAccess	~$25–40	ASCDataAccess (this library)
Azure Cosmos DB (NoSQL API)	~$300–600	EF Core Cosmos provider
MongoDB Atlas (M30 cluster)	~$200–400	MongoDB.Driver with LINQ
Azure SQL Database (GP, 4 vCores)	~$370–500	Entity Framework Core / Dapper
Marten on Azure PostgreSQL (GP Gen5_4)	~$250–400	Marten (excellent)
Amazon DynamoDB (on-demand)	~$150–300	AWS SDK (no dedicated ORM)

Azure Table Storage is not marginally cheaper. It is 6 to 15 times cheaper than every named alternative for equivalent data volumes and request rates. For a startup, that gap is the difference between a sustainable infrastructure budget and one that threatens the business. For an enterprise, it is a six-figure annual saving on data tier costs that can be redirected to product development.

The platforms with richer query capabilities cost more precisely because of those capabilities. Teams that need full-text search, complex aggregation pipelines, or arbitrary secondary indexes on large datasets should evaluate Cosmos DB or MongoDB — and section 5 of this guide is honest about exactly where those needs arise. But the data shows that most application workloads — user management, order history, session state, audit logs, telemetry, configuration, queue management, file metadata — are dominated by key-based lookups and partition scans. These are the access patterns Azure Table Storage executes at scale with sub-15ms latency and at a cost no competitor comes close to matching.

The throughput ceiling is not a constraint for most applications. Azure Table Storage processes approximately 20,000 transactions per second per storage account. An application supporting 10,000 simultaneous active users — each generating an average of one read/write per second — produces roughly 10,000 TPS. That is well within the single-account ceiling, and storage account sharding is available for workloads that exceed it. The platforms with richer query languages are not faster at key lookups — they are more expensive because they do more things. If your workload does not need those things, you are paying for capability you are not using.

Schema migrations do not exist. Azure Table Storage is schemaless. Adding a new property to an entity requires changing one C# class. There are no ALTER TABLE scripts, no migration files, no downtime windows, no rollback procedures. For teams that ship frequently, this is not a minor convenience — it is a meaningful reduction in deployment complexity and risk.

---

Fact 3: The workloads that define most applications map perfectly to Azure Table Storage's strengths

The capabilities Azure Table Storage lacks — server-side joins, arbitrary secondary indexes, aggregation pipelines, full-text search — are real limitations. But examine the actual data access patterns that constitute the majority of code in most business applications:

Access pattern	Azure Table Storage capability
"Give me this user by their ID"	✅ O(1) RowKey lookup — optimal
"Give me all orders for this customer"	✅ PartitionKey scan — fast and efficient
"Save this record"	✅ Single-entity upsert — fast
"Save these 500 records"	✅ Auto-batched — handled by the library
"Give me the next page of results"	✅ Continuation token pagination — built-in
"Process this work queue and resume after a crash"	✅ QueueData<T> + StateList<T> — built-in
"Store this session between requests"	✅ Session class — built-in
"Log this error"	✅ ErrorLogData — built-in
"Store this uploaded file"	✅ AzureBlobs — built-in
"Find all active customers by company"	✅ Lambda query — server-side OData
"Find customers whose name contains 'Smith'"	⚠️ Client-side filter — use a normalized key field
"Count all orders this month"	⚠️ Client-side aggregation — pre-compute in summary entity
"Full-text search across all documents"	❌ Not supported — use Azure Cognitive Search

The honest picture is that the first eleven rows describe the overwhelming majority of operations in a typical business application. The last two rows describe capabilities that belong in supplemental infrastructure regardless of which primary data store you use — even Cosmos DB teams typically add Azure Cognitive Search for full-text requirements.

ASCDataAccess exists to make the first eleven rows — the ones that define your application's core data layer — as clean, type-safe, and low-friction as Entity Framework Core makes SQL. On a platform that costs a fraction of any alternative. With zero schema migration overhead. On infrastructure that requires no servers to manage, no connection pools to tune, and no query plans to analyze.

---

The bottom line

If your application is greenfield and your team is evaluating data platforms, the question to ask is: which of my access patterns actually require capabilities that Azure Table Storage cannot provide? If the honest answer is "none" or "only in a small supplemental component," Azure Table Storage with ASCDataAccess is the correct choice for your primary data tier and delivers savings that compound every month for the life of the product.

If your application is already on Azure Table Storage and you are using the raw SDK, ASCDataAccess eliminates the infrastructure boilerplate immediately. There is no migration — your entities, your tables, and your data are unchanged. You swap the raw SDK calls for the library's API and get the same results with a fraction of the code.

In either case, ASCDataAccess is not competing with Entity Framework Core on Azure Table Storage. Entity Framework Core on Azure Table Storage does not exist. ASCDataAccess is the only framework available for this platform, and it was built by a team — Answer Sales Calls Inc. — that runs it in production across multiple enterprise applications.

---

3. Installation and One-Time Configuration

Install via NuGet

dotnet add package ASCDataAccess

Or via Package Manager Console:

Install-Package ASCDataAccess

Runtime compatibility: Ships as net9.0. Builds and runs on .NET 9 and .NET 10 SDKs. .NET 6, 7, and 8 projects can install the package and receive the net9.0 target automatically via NuGet TFM fallback.

appsettings.json

{
  "Azure": {
    "TableStorageName": "your-storage-account-name",
    "TableStorageKey":  "your-storage-account-key"
  }
}

Program.cs — configure once, use everywhere

using ASCTableStorage;

// Called once at startup — all DataAccess<T>, AzureBlobs, Session,
// and QueueData<T> instances resolve credentials automatically from this point.
TableStorage.Configure(
    builder.Configuration["Azure:TableStorageName"]!,
    builder.Configuration["Azure:TableStorageKey"]!
);

After this single call, every DAL class can be instantiated without passing credentials:

var da      = new DataAccess<Customer>();
var blobs   = new AzureBlobs("documents");
var session = new Session(sessionId);
var queue   = await QueueData<Order>.GetQueueAsync(queueId);

Backward compatibility: All explicit-credential constructors (new DataAccess<T>(accountName, accountKey)) still work unchanged. TableStorage.Configure() is additive, not a breaking change.

TableStorage static class

if (TableStorage.IsConfigured)
{
    var da = new DataAccess<MyEntity>();
}

string name = TableStorage.AccountName;
string key  = TableStorage.AccountKey;

---

4. The Data Model — RowKey, PartitionKey, and Why They Matter

This is the most important conceptual section in this guide. AI assistants and developers new to Azure Table Storage must understand this before writing any entity classes.

Azure Table Storage is a key-value store with a two-part composite key. Every row is uniquely identified by the combination of PartitionKey and RowKey. Understanding the semantic role of each is critical to designing a schema that performs well.

RowKey — the Primary Key

RowKey is the unique identifier for a row within its partition. Think of it as the primary key in a traditional relational database table. It must be unique within its PartitionKey group.

• Use a GUID, a natural business identifier (order number, email address, product SKU), or any unique string
• Must not contain: /, \, #, ?, or control characters
• Maximum length: 1 KB
• Azure uses RowKey for O(log n) point lookups — fetching by RowKey is the fastest possible query

PartitionKey — the Foreign Key / Grouping Key

PartitionKey is the grouping key that defines which logical partition a row belongs to. Think of it as a foreign key relationship — all rows with the same PartitionKey belong to the same logical group.

• Azure physically co-locates rows with the same PartitionKey on the same storage node
• Querying by PartitionKey alone is a fast, server-efficient operation
• Querying across multiple PartitionKey values requires a full table scan (client-side)
• Use PartitionKey to represent the parent entity: a customer ID, a company ID, a date bucket, a user ID
• Maximum length: 1 KB

Mandatory C# Property Pattern — RowKey and PartitionKey

When defining entity classes, the way you expose RowKey and PartitionKey as named properties determines the correctness and safety of every write to Azure Table Storage. Follow these patterns without exception.

RowKey — the Primary Key. ALWAYS use Guid.NewGuid().ToString() as the null fallback:

/// <summary>Unique user identifier (RowKey — primary key).</summary>
public string UserID
{
    get => RowKey ?? Guid.NewGuid().ToString();
    set => RowKey = value ?? Guid.NewGuid().ToString();
}

⚠️ NEVER use string.Empty, "", or any non-GUID fallback for RowKey. RowKey is the primary key — it must be unique per partition. A blank RowKey is not unique. If two rows share the same PartitionKey and both have an empty RowKey, the second write silently overwrites the first, corrupting data without any error. A Guid.NewGuid() fallback guarantees uniqueness even if the property is never explicitly set.

PartitionKey — the Foreign Key / Grouping Key. string.Empty is an acceptable fallback:

/// <summary>Company this user belongs to (PartitionKey — foreign key / grouping key).</summary>
public string CompanyId
{
    get => PartitionKey ?? string.Empty;
    set => PartitionKey = value ?? string.Empty;
}

PartitionKey groups rows — a blank PartitionKey is legal (it means "the default partition") and does not risk data corruption. In production always set a meaningful grouping value, but the fallback is safe.

The relationship in one sentence: PartitionKey is the parent (foreign key); RowKey is the child's unique identity (primary key). Every RowKey must be unique within its PartitionKey group, and that uniqueness must be guaranteed at the property definition level via a GUID fallback.

---

Practical examples of PartitionKey / RowKey design

// Orders for a customer
PartitionKey = "CUST-12345"       // Customer ID (foreign key relationship)
RowKey       = "ORD-98765"        // Order ID (primary key)

// Log entries by date
PartitionKey = "2025-04"          // Year-Month bucket (groups all April 2025 logs)
RowKey       = Guid.NewGuid()     // Unique log entry ID

// User profile
PartitionKey = "users"            // All users in one partition (small dataset only)
RowKey       = "user@example.com" // Email as natural unique key

// Songs by artist
PartitionKey = "artist-456"       // Artist ID
RowKey       = "song-789"         // Song ID

Why this matters for DataAccess<T>

The library uses PartitionKey and RowKey to route all queries. The most efficient query shapes are:

1. GetRowObjectAsync(rowKey) — O(1), the fastest possible lookup
2. GetCollectionAsync(partitionKey) — Fast, returns all rows for a partition
3. GetCollectionAsync(x => x.CustomerId == "CUST-123") — Server-side OData if mapped to PartitionKey; otherwise hybrid

Design your PartitionKey to match your most common query pattern. The most common access pattern should be a PartitionKey scan, not a full table scan.

---

5. Azure Table Storage Hard Limits — What Every Developer and AI Must Know

AI assistants generating entity classes and schema designs must read this section carefully. Violating these limits causes runtime failures, silent data truncation, or severe performance degradation. Azure Table Storage is a capable but constrained platform — designing within its limits is non-negotiable.

Entity (row) size limit — 1 MB maximum

This is the single most important limit in Azure Table Storage. Every entity — the combined total of all its properties, keys, and system fields — must not exceed 1 megabyte. This is not a soft limit. Azure will reject the write with an error if an entity exceeds 1 MB.

This limit has significant implications for schema design:

• An entity with 10 properties averaging 100 KB each fails at write time
• An entity that stores a large JSON blob, several chunked strings, and many scalar fields can approach this limit without the developer realizing it
• The library's automatic string chunking handles the 32 KB per-property limit transparently, but it does not protect you from the 1 MB entity limit — chunking just means a large string becomes many properties, all of which still count toward the total entity size

Design guidance: Keep entities lean. If a single row is approaching hundreds of kilobytes, that is a signal that some data should be moved to Azure Blob Storage or split into related child entities.

Complete hard limits table

Limit	Value	Notes
Entity (row) size	1 MB	Hard limit — writes fail above this
Property count per entity	255	Includes 4 system properties (PartitionKey, RowKey, Timestamp, ETag) — effective max is 251 custom properties
String property size	32 KB practical / 64 KB theoretical	Library auto-chunks at 32 KB; each chunk counts as a separate property toward the 255 limit
PartitionKey length	1 KB
RowKey length	1 KB
Table name	3–63 characters	Alphanumeric only, must start with a letter
Batch transaction size	100 entities, all same PartitionKey	Total batch payload must also be ≤ 4 MB
Query result page	1,000 entities	Azure never returns more than 1,000 per page regardless of page size requested
Account throughput	~20,000 transactions/second	Per storage account
Forbidden key characters	/ \ # ? and control chars	In both PartitionKey and RowKey

How the string chunking limit compounds

When a string property exceeds 32 KB, the library splits it into chunks:

MyText         → first 32 KB     (1 property slot)
MyText_pt_1    → next 32 KB      (1 property slot)
MyText_pt_2    → remainder       (1 property slot)

A single 96 KB string consumes 3 of your 255 property slots and 96 KB of your 1 MB entity budget. An entity with five large text fields could consume 15 property slots before any scalar fields are counted. Plan accordingly — and move large content to Azure Blob Storage where it belongs.

No secondary indexes — only PartitionKey and RowKey are indexed

In Azure Table Storage, only PartitionKey and RowKey receive native index treatment. Every other property filter is executed as either an OData filter during a table scan or a client-side filter applied in memory after Azure returns results.

This means:

• You cannot efficiently query by email address unless email is the RowKey or PartitionKey
• You cannot efficiently query by date unless date is encoded into the key
• You cannot efficiently query by any computed or secondary field without fetching and filtering in memory

The design implication: If you need to query by multiple access patterns, maintain multiple entity types with different key designs that store the same logical data indexed differently. This is the standard Azure Table Storage pattern for secondary access paths and is documented with full examples in section 6.

No server-side joins

Azure Table Storage has no concept of a JOIN. Every cross-entity relationship must be resolved in application code with multiple queries.

// ❌ Not possible in Azure Table Storage
// SELECT c.*, o.* FROM Customers c JOIN Orders o ON c.CustomerId = o.CustomerId

// ✅ Two separate queries, assembled in code
var customer = await customerDa.GetRowObjectAsync(customerId);
var orders   = await orderDa.GetCollectionAsync(customerId);  // PartitionKey = customerId

No server-side aggregates

Azure Table Storage has no COUNT, SUM, AVG, MIN, MAX, or GROUP BY. All aggregation must be performed client-side after fetching records. For heavy reporting, supplement with Azure Synapse Analytics or pre-compute aggregates into summary entities at write time.

No cross-partition transactions

Batch operations are atomic only within a single PartitionKey. You cannot perform an atomic batch that spans two different PartitionKey values.

No server-side string search (LIKE / CONTAINS)

OData v2 does not support LIKE, CONTAINS, or STARTSWITH as server-side indexed operations. When you write a lambda using string methods, the library fetches the partition (or table) and evaluates client-side. For large tables, store a normalized search key at write time and filter by equality.

// ⚠️ Client-side full scan
await da.GetCollectionAsync(x => x.ProductName.Contains("widget"));

// ✅ Normalize at write time, query by equality
await da.GetCollectionAsync(x => x.SearchKey == "widget");

Comparison to other NoSQL platforms

Capability	Azure Table Storage	Amazon DynamoDB	Azure Cosmos DB (NoSQL)	MongoDB
Max row / document size	1 MB	400 KB	2 MB	16 MB
Secondary indexes	❌ None	✅ GSI / LSI	✅ Composite indexes	✅ Any field
Server-side aggregates	❌ None	❌ Limited	✅ Full pipeline	✅ Full pipeline
Cross-partition transactions	❌ No	✅ Transactions API	✅ Multi-document ACID	✅ Multi-document ACID
Query language	OData v2 (minimal)	PartiQL / Conditions	SQL-like	MQL / Aggregation
String search server-side	❌ No	✅ contains()	✅ CONTAINS()	✅ $regex
ORM / Framework	ASCDataAccess	None	EF Core provider	MongoDB.Driver
Approximate cost (500 GB, 50M reads/mo)	~$25–40	~$150–300	~$300–600	~$200–400

The cost column is the decisive column for any team that has honestly evaluated their query requirements and confirmed that Azure Table Storage's access patterns meet their needs. The technical constraints are real. The 6–15x cost advantage over every alternative is equally real.

---

6. JSON Fields vs. Related Entities — The Most Important Schema Decision

This section is critical for AI-assisted code generation. The single most common schema design mistake when using ASCDataAccess — whether by developers or by AI tools — is choosing incorrectly between JSON-serialized fields and proper related entities. Getting this decision wrong leads to queries that cannot filter, data that cannot be searched, and entity sizes that approach or exceed the 1 MB limit.

The core rule

Use a JSON field when:

• The child data is small, bounded, and always travels with the parent
• The child data is never filtered, sorted, or searched independently
• The child data does not need to be accessed without its parent
• The data is configuration or display metadata — not a queryable entity in its own right

Use a related entity (separate table row) when:

• The child data needs to be queried, filtered, or sorted on its own
• You need to count, aggregate, or report on the child data
• The child data can grow to an unbounded number of items
• Any lambda expression will ever need to filter on the child data
• The child data is logically independent and has meaning outside the parent context

Why JSON fields break lambda queries

When a property is stored as a JSON string, the library has no way to filter on its internal structure. The lambda expression visitor only understands top-level entity properties. To filter by anything inside a JSON field, you must fetch all candidate rows and deserialize each one — a full table scan with expensive per-row deserialization.

// ❌ Broken — Tags is a JSON string; lambda cannot filter inside it
public class Product : TableEntityBase, ITableExtra
{
    public string? TagsJson { get; set; }  // Stored as "[\"electronics\",\"wireless\"]"
}

// Fetches ALL products, filters client-side — semantically unreliable
await da.GetCollectionAsync(x => x.TagsJson.Contains("electronics"));  // Wrong and slow

The correct approach: related entity for queryable child data

// ✅ Product tags as a proper related entity — fully queryable
public class ProductTag : TableEntityBase, ITableExtra
{
    public string ProductId   // PartitionKey — groups all tags for a product
    {
        get => PartitionKey ?? string.Empty;
        set => PartitionKey = value;
    }

    public string TagName     // RowKey — unique tag within the product
    {
        get => RowKey ?? Guid.NewGuid().ToString();
        set => RowKey = value ?? Guid.NewGuid().ToString();
    }

    public string TableReference => "ProductTags";
    public string GetIDValue()   => TagName;
}

var tagDa = new DataAccess<ProductTag>();

// All tags for a specific product — fast PartitionKey scan
var tags = await tagDa.GetCollectionAsync(productId);

When JSON fields ARE the right choice

public class UserProfile : TableEntityBase, ITableExtra
{
    public string UserId   { get => RowKey ?? Guid.NewGuid().ToString();  set => RowKey = value ?? Guid.NewGuid().ToString(); }
    public string Region   { get => PartitionKey ?? string.Empty;          set => PartitionKey = value ?? string.Empty; }

    // ✅ Small, bounded, never queried independently — JSON is correct
    public DisplayPreferences? Preferences  { get; set; }
    public ContactInfo?        ContactInfo  { get; set; }

    // ❌ Wrong — subscription tier affects queries; use a related entity
    // public List<string> SubscriptionFeatures { get; set; }

    public string TableReference => "UserProfiles";
    public string GetIDValue()   => UserId;
}

public class DisplayPreferences
{
    public string Theme        { get; set; } = "light";
    public string Language     { get; set; } = "en";
    public bool   ShowTutorials { get; set; } = true;
    public int    ItemsPerPage  { get; set; } = 25;
}

Decision checklist

Question	If YES →	If NO →
Will a lambda ever filter on this data?	Related entity	JSON field is safe
Does this data grow to an unbounded list?	Related entity	JSON field may be safe
Is this data accessed without its parent?	Related entity	JSON field is safe
Does this data need to be counted or aggregated?	Related entity	JSON field may be safe
Is this larger than ~50 KB when serialized?	Blob Storage or related entity	JSON field is safe
Is this configuration or display metadata?	JSON field is appropriate	—

The secondary index pattern for multi-access-path queries

// Primary entity — found by CustomerId (RowKey)
public class Customer : TableEntityBase, ITableExtra
{
    public string CompanyId   { get => PartitionKey ?? string.Empty;          set => PartitionKey = value ?? string.Empty; }
    public string CustomerId  { get => RowKey ?? Guid.NewGuid().ToString();  set => RowKey = value ?? Guid.NewGuid().ToString(); }
    public string? Email      { get; set; }
    public string TableReference => "Customers";
    public string GetIDValue()   => CustomerId;
}

// Email index — found by Email (RowKey), points back to primary
public class CustomerEmailIndex : TableEntityBase, ITableExtra
{
    public string CompanyId   { get => PartitionKey ?? string.Empty; set => PartitionKey = value ?? string.Empty; }
    public string Email       { get => RowKey ?? Guid.NewGuid().ToString();  set => RowKey = value ?? Guid.NewGuid().ToString(); }
    public string? CustomerId { get; set; }
    public string TableReference => "CustomerEmailIndex";
    public string GetIDValue()   => Email;
}

// Write both together (same PartitionKey enables atomic batch)
var customer   = new Customer { CompanyId = "COMP-001", CustomerId = "CUST-001", Email = "jane@example.com" };
var emailIndex = new CustomerEmailIndex { CompanyId = "COMP-001", Email = "jane@example.com", CustomerId = "CUST-001" };

await customer.SaveAsync();
await emailIndex.SaveAsync();

// Lookup by email — RowKey lookup on index, then primary lookup
var index    = await new CustomerEmailIndex().LoadAsync("jane@example.com");
var customer = await new Customer().LoadAsync(index.CustomerId!);

---

7. Defining Entities and Writing Your First Data Operations

The most important section for AI code generation. Read this section before writing any data access code. The idiomatic ASCDataAccess pattern is built around entity objects that carry their own load, save, and delete operations as extension methods. You almost never need to instantiate DataAccess<T> directly in day-to-day application code. Define your entity, call .LoadAsync(), .SaveAsync(), or .DeleteAsync() on it — that is the entire API for the vast majority of use cases.

Step 1 — Define your entity

Every entity inherits from TableEntityBase and implements ITableExtra. That is all that is required.

using ASCTableStorage.Models;
using ASCTableStorage.Common;

public class Customer : TableEntityBase, ITableExtra
{
    // PartitionKey — the "foreign key" grouping (see section 4)
    // Map a meaningful business property onto it for clarity
    public string CompanyId
    {
        get => PartitionKey ?? string.Empty;
        set => PartitionKey = value;
    }

    // RowKey — the "primary key", unique within the CompanyId partition
    public string CustomerId
    {
        get => RowKey ?? Guid.NewGuid().ToString();
        set => RowKey = value ?? Guid.NewGuid().ToString();
    }

    // All other properties are plain C# — the library handles serialization automatically
    public string?        Name        { get; set; }
    public string?        Email       { get; set; }
    public bool           IsActive    { get; set; }
    public decimal        Balance     { get; set; }   // Stored as long ×10000 automatically
    public DateTime       CreatedDate { get; set; }
    public CustomerStatus Status      { get; set; }   // Stored as JSON string automatically

    // ✅ JSON is appropriate here — small, non-queryable companion data
    public ContactInfo?   ContactInfo { get; set; }

    // ITableExtra — the Azure table name for this entity
    public string TableReference => "Customers";

    // ITableExtra — the unique identifier value
    public string GetIDValue() => CustomerId;
}

public class ContactInfo
{
    public string? Phone   { get; set; }
    public string? Address { get; set; }
    public string? City    { get; set; }
}

public enum CustomerStatus { Active, Inactive, Suspended }

Step 2 — Use it. No DataAccess<T> required.

Once your entity is defined, every data operation is an extension method on the entity itself or on a List<T> of it. This is the primary pattern for all everyday data access in ASCDataAccess. You call operations directly on your objects — exactly the way you think about your data.

using ASCTableStorage.Common;  // brings in the extension methods

Load a single entity by its RowKey (primary key):

var customer = await new Customer().LoadAsync("CUST-042");

Load a single entity by lambda expression:

var customer = await new Customer().LoadAsync(x => x.Email == "jane@example.com");

Load a collection by lambda expression:

var customers = await new List<Customer>().LoadAsync(x => x.CompanyId == "COMP-001" && x.IsActive == true);

Modify and save:

customer.Balance  = 2500.00m;
customer.IsActive = true;
await customer.SaveAsync();

Create and save a new entity:

var newCustomer = new Customer
{
    CompanyId  = "COMP-001",
    CustomerId = Guid.NewGuid().ToString(),
    Name       = "Jane Smith",
    Email      = "jane@example.com",
    IsActive   = true,
    Balance    = 0m,
    Status     = CustomerStatus.Active
};

await newCustomer.SaveAsync();

Delete:

await customer.DeleteAsync();

Save or delete a collection:

await customers.SaveAsync();
await customers.DeleteAsync();

That is it. For the overwhelming majority of application code — service classes, controllers, background workers — this is all you ever write. No DataAccess<T>, no constructor arguments, no credential passing. Define your entity, call extension methods on it.

The complete lifecycle in one place

// Load → modify → save
var order = await new Order().LoadAsync(x => x.OrderId == orderId);
order.Status = OrderStatus.Shipped;
order.ShippedDate = DateTime.UtcNow;
await order.SaveAsync();

// Load a collection → filter further in memory → delete
var staleOrders = await new List<Order>().LoadAsync(
    x => x.CompanyId == companyId && x.Status == OrderStatus.Cancelled
);
await staleOrders.DeleteAsync();

// Load collection → update all → save all
var pendingOrders = await new List<Order>().LoadAsync(x => x.Status == OrderStatus.Pending);
pendingOrders.ForEach(o => o.Priority = Priority.High);
await pendingOrders.SaveAsync();

ITableExtra interface

public interface ITableExtra
{
    string TableReference { get; }  // The Azure table name for this entity type
    string GetIDValue();            // The primary identifier value (maps to RowKey)
}

Automatic type handling — nothing to configure

C# Type	Storage reality	What the library does
decimal	Not natively supported	Multiplied by 10,000, stored as long; reversed on read
string > 32 KB	32 KB per-property limit	Auto-chunked into sequential fields; reassembled on read
enum	Not natively supported	JSON-serialized as string name
Complex objects (List<T>, custom classes)	Not natively supported	JSON-serialized; deserialized on read
bool	Multiple formats in storage	Parses true/false, 1/0, yes/no
DateTime / DateTimeOffset	Multiple formats in storage	Intelligent multi-format parsing

Entity size discipline

• Keep scalar properties scalar: string, int, bool, decimal, DateTime, Guid
• Use JSON only for small, bounded, non-queryable companion data (see section 6)
• Move large content — HTML, source code, binary files — to Blob Storage; store the URL
• 255 properties maximum per entity; chunked strings consume multiple slots
• Call entity.GetDiagnostics() in development to inspect sizes before production

PartitionKey change tracking

If you change the PartitionKey of an existing entity and save it, the library automatically deletes the old row and inserts under the new partition — transparent to the caller.

var customer = await new Customer().LoadAsync("CUST-001");
customer.CompanyId = "NEW-COMPANY";  // PartitionKey change
await customer.SaveAsync();          // Delete-from-old + insert-to-new handled automatically

---

8. Lambda Expressions — What Works Server-Side and What Does Not

AI code generation rule for lambda queries: Always write the lambda on the entity or collection directly using extension methods. Use new Customer().LoadAsync(lambda) or new List<Customer>().LoadAsync(lambda) — not new DataAccess<Customer>().GetCollectionAsync(lambda). The result is identical; the extension method form is shorter, cleaner, and requires no boilerplate.

ASCDataAccess translates C# lambda expressions into OData filter strings for server-side execution wherever possible. When a lambda contains operations OData v2 cannot evaluate, the library splits automatically: the server-evaluable part runs as an OData query, the remainder is evaluated client-side on the results. The two call sites that accept lambda predicates are the LoadAsync extension methods and DataAccess<T>.GetCollectionAsync — both go through exactly the same expression analysis pipeline, so everything in this section applies to both.

// Extension method — preferred for everyday use
var results = await new List<Order>().LoadAsync(x => x.CustomerId == customerId);

// DataAccess<T> directly — same engine, same result; reach for this only when you
// also need batching, pagination, or explicit account credentials
var da      = new DataAccess<Order>();
var results = await da.GetCollectionAsync(x => x.CustomerId == customerId);

---

Supported server-side operations — use freely

These expressions are fully translated to OData and execute on Azure Table Storage before any data is returned to your application. They perform well regardless of table size.

// Equality and comparison — any value type, string, or enum
var active  = await new List<Customer>().LoadAsync(x => x.Status == CustomerStatus.Active);
var wealthy = await new List<Customer>().LoadAsync(x => x.Balance > 1000m);
var recent  = await new List<Customer>().LoadAsync(x => x.CreatedDate >= new DateTime(2024, 1, 1));

// Logical AND / OR — compose freely
var vip     = await new List<Customer>().LoadAsync(x => x.IsActive == true && x.Balance > 500m);
var flagged = await new List<Customer>().LoadAsync(
    x => x.Status == CustomerStatus.Active || x.Status == CustomerStatus.Suspended);

// Null / empty check — translated to (Email eq '')
var noEmail = await new List<Customer>().LoadAsync(x => string.IsNullOrEmpty(x.Email));

// Boolean fields — both forms work; the bare form is expanded internally to (IsActive eq true)
var active2 = await new List<Customer>().LoadAsync(x => x.IsActive == true);
var active3 = await new List<Customer>().LoadAsync(x => x.IsActive);

// String prefix — translated to an OData range: (Name ge 'Sm' and Name lt 'Sm~')
// Efficient on large tables when combined with a PartitionKey filter
var smiths  = await new List<Customer>().LoadAsync(x => x.Name.StartsWith("Sm"));

// Single entity by lambda — returns the first match or default
var customer = await new Customer().LoadAsync(x => x.Email == "jane@example.com");

---

QueryExtensions — rich server-side and client-side operators

QueryExtensions is a set of extension methods in the ASCTableStorage.Common namespace that the expression visitor intercepts and translates to efficient OData clauses or in-memory predicates. Add using ASCTableStorage.Common; to any file that uses them.

Important: QueryExtensions methods are marker methods. They are only valid inside lambda expressions passed to LoadAsync or GetCollectionAsync. Calling them directly at runtime will throw InvalidOperationException. The expression visitor intercepts them before they ever execute.

Membership — In

Tests whether a field's value is one of a supplied set. Translates to an OR-joined equality chain on the server: (field eq 'a' or field eq 'b' or field eq 'c').

using ASCTableStorage.Common;

// Params-array form — inline values
var errors = await new List<AppLog>().LoadAsync(
    x => x.Severity.In("Critical", "Error", "Warning"));

// IEnumerable form — values from a variable (closure is evaluated at query-build time)
var targetStatuses = new[] { OrderStatus.Pending, OrderStatus.Processing };
var open = await new List<Order>().LoadAsync(
    x => x.Status.In(targetStatuses));

// Works with strings, enums, ints, or any value type
var byId = await new List<Product>().LoadAsync(
    x => x.CategoryId.In(1, 2, 5, 9));

// Via DataAccess<T> directly — identical result
var da     = new DataAccess<AppLog>();
var errors = await da.GetCollectionAsync(x => x.Severity.In("Critical", "Error"));

Returns: All rows where the field matches any of the supplied values. An empty options list returns no rows.

---

Range — Between

Tests whether a field falls within an inclusive range [low, high]. Translates to (field ge low and field le high). Works with any IComparable<T>: int, long, double, decimal, DateTime, DateTimeOffset, string.

// Numeric range
var midTier = await new List<Order>().LoadAsync(
    x => x.TotalAmount.Between(100m, 500m));

// Integer range
var highError = await new List<AppLog>().LoadAsync(
    x => x.ErrorCount.Between(10, 100));

// DateTime range — bounds are explicit; use InDateRange for relative windows
var q1 = await new List<Sale>().LoadAsync(
    x => x.SaleDate.Between(new DateTime(2025, 1, 1), new DateTime(2025, 3, 31)));

// String lexicographic range
var namesAtoF = await new List<Customer>().LoadAsync(
    x => x.LastName.Between("A", "G"));

Returns: All rows where low <= field <= high (inclusive on both ends).

---

String presence — IsNotNullOrEmpty

Tests that a string field contains a non-empty value. Translates to (field ne ''). This is the complement of string.IsNullOrEmpty(), which translates to (field eq '').

// Find customers who have supplied a phone number
var withPhone = await new List<Customer>().LoadAsync(
    x => x.PhoneNumber.IsNotNullOrEmpty());

// Combine with other filters
var actionable = await new List<Lead>().LoadAsync(
    x => x.IsActive == true && x.Email.IsNotNullOrEmpty());

// The pair — both are server-side
var hasEmail  = await new List<Customer>().LoadAsync(x => x.Email.IsNotNullOrEmpty());
var noEmail   = await new List<Customer>().LoadAsync(x => string.IsNullOrEmpty(x.Email));

---

Relative date windows — IsToday, IsThisWeek, IsThisMonth, IsThisYear

These methods test whether a DateTime? or DateTimeOffset? field falls within a calendar window anchored to the current UTC time. The window is calculated once at query-build time and emitted as a static (field ge start and field lt end) OData clause. There is no runtime recalculation — the bounds are baked into the OData string sent to Azure.

• IsToday — midnight-to-midnight UTC today
• IsThisWeek — Monday 00:00 UTC through Sunday 23:59:59 UTC of the current calendar week
• IsThisMonth — first day of the current calendar month through the last
• IsThisYear — January 1 through December 31 of the current year

// All of these translate to a concrete ge/lt OData pair — no client-side work

// Today's log entries
var todayLogs = await new List<AppLog>().LoadAsync(
    x => x.Timestamp.IsToday());

// This week's orders
var weekOrders = await new List<Order>().LoadAsync(
    x => x.CreatedDate.IsThisWeek());

// This month's sales
var monthSales = await new List<Sale>().LoadAsync(
    x => x.SaleDate.IsThisMonth());

// This year's customers
var yearCustomers = await new List<Customer>().LoadAsync(
    x => x.JoinDate.IsThisYear());

// Combine with other filters
var todayErrors = await new List<AppLog>().LoadAsync(
    x => x.Severity == "Critical" && x.Timestamp.IsToday());

// Works with DateTime? and DateTimeOffset? — both overloads behave identically
var logs = await new List<AppLog>().LoadAsync(x => x.Timestamp.IsToday());

// Via DataAccess<T>
var da   = new DataAccess<AppLog>();
var logs = await da.GetCollectionAsync(x => x.Timestamp.IsThisMonth());

Returns: All rows where the date field falls within the named calendar window, evaluated in UTC at the moment the query is built.

---

Explicit date range — InDateRange

Tests whether a date field falls within [start, end) (inclusive start, exclusive end). Translates to (field ge start and field lt end). Use this when the window is dynamic or does not correspond to a calendar boundary.

// Rolling 30-day window
var from    = DateTimeOffset.UtcNow.AddDays(-30);
var to      = DateTimeOffset.UtcNow;
var recent  = await new List<Order>().LoadAsync(
    x => x.CreatedDate.InDateRange(from, to));

// Fiscal quarter — explicit bounds
var q2Start = new DateTime(2025, 4, 1);
var q2End   = new DateTime(2025, 7, 1);
var q2Sales = await new List<Sale>().LoadAsync(
    x => x.SaleDate.InDateRange(q2Start, q2End));

// Accepts both DateTimeOffset and DateTime — pick whichever matches your property type

Returns: All rows where start <= field < end. The end bound is exclusive (standard range convention).

---

Bitwise flag testing — HasAnyFlag (client-side)

Tests whether any of the specified bits are set in a numeric field. OData v2 has no bitwise operators, so this predicate always executes in-memory after the server fetch. The library automatically splits the lambda: any server-evaluable parts run as OData, HasAnyFlag applies to the fetched rows.

// Fetch rows where the Permissions field has any of the read or write bits set
const int ReadBit  = 0x01;
const int WriteBit = 0x02;

var readWrite = await new List<UserPermission>().LoadAsync(
    x => x.Permissions.HasAnyFlag(ReadBit | WriteBit));

// Combine with a server-side filter — PartitionKey filter runs server-side,
// HasAnyFlag runs in-memory on the returned subset
var adminRW = await new List<UserPermission>().LoadAsync(
    x => x.TenantId == tenantId && x.Permissions.HasAnyFlag(ReadBit | WriteBit));

// long overload
var flagged = await new List<Feature>().LoadAsync(
    x => x.FeatureFlags.HasAnyFlag(0x0000_0003L));

Performance note: When using HasAnyFlag, pair it with a server-side filter that meaningfully reduces the row count first (PartitionKey, status, date window, etc.). The server-side part runs as OData; HasAnyFlag evaluates only on those returned rows. Avoid using it alone on large tables with no other predicate.

Returns: All rows (from the server-filtered set) where (field & flags) != 0.

---

Operations that force a full table fetch — avoid on large tables

// ⚠️ Contains and EndsWith are not translatable to OData — fetches all rows, filters in memory
var found = await new List<Customer>().LoadAsync(x => x.Name.Contains("Smith"));
var found = await new List<Customer>().LoadAsync(x => x.Email.EndsWith("@gmail.com"));

// ⚠️ Case transformation — not expressible in OData
var found = await new List<Customer>().LoadAsync(x => x.Name.ToLower() == "smith");

// ⚠️ Null equality on string properties forces a full scan
var noEmail = await new List<Customer>().LoadAsync(x => x.Email == null);

// ❌ Properties inside JSON-serialized complex type fields — never use in lambdas
var boston = await new List<Customer>().LoadAsync(x => x.ContactInfo.City == "Boston");

For Contains and EndsWith on large tables, store a pre-normalized or indexed field (e.g. a domain-extracted email field, a lowercase name field) and query that instead.

---

Null-guarded optional filters

When a filter variable is nullable and may be null at runtime — meaning "no filter on this field" — write the guard as variable == null || field.In(variable). The expression analyzer folds the true || ... short-circuit at query-build time and omits the entire branch from both the OData clause and the in-memory predicate, preventing null-reference exceptions.

// categoryFilter is null → the entire OR branch is dropped; only the other conditions apply
// categoryFilter has values → In(...) runs server-side as an OR-equality chain
List<NotificationCategory>? categoryFilter = ParseCategories(request.Categories);

var notifications = await new List<Notification>().LoadAsync(n =>
    n.UserId == currentUserId &&
    n.IsActive == true &&
    (categoryFilter == null || n.Category.In(categoryFilter)));

---

Query performance summary

| Pattern | Executes | Performance | Guidance | |---|---|---|---| | .LoadAsync("rowKey") | Server — O(1) point read | Excellent | Always prefer when both keys are known | | .LoadAsync(x => x.CompanyId == id) (PartitionKey) | Server | Excellent | Primary query pattern | | ==, !=, >, <, >=, <= | Server | Good | Use freely | | string.IsNullOrEmpty() / IsNotNullOrEmpty() | Server | Good | Safe on any size table | | .StartsWith("prefix") | Server — range approximation | Good | Safe; less precise than exact equality | | .In(values) | Server — OR equality chain | Good | Efficient for small-to-moderate value sets | | .Between(lo, hi) | Server — ge/le range | Good | Use freely | | IsToday() / IsThisWeek() / IsThisMonth() / IsThisYear() | Server — static range | Good | Use freely; window is baked in at query time | | InDateRange(start, end) | Server — ge/lt range | Good | Use freely | | HasAnyFlag(bits) | Client — in-memory | Acceptable if server filter narrows first | Always pair with a server-side predicate | | .Contains() / .EndsWith() | Client — full scan then filter | Poor on large tables | Pre-index or store a normalized field | | Lambda on JSON field internals | Never translatable | Always wrong | Use a related entity instead | | No predicate (load all) | Server — full table scan | Worst | Only for small or admin tables |

9. When You Do Need DataAccess<T> Directly

DataAccess<T> is the engine that powers all extension methods. For everyday single-entity and collection operations you never need to touch it — the extension method pattern handles everything. You reach for DataAccess<T> directly when you need capabilities the extension methods do not expose:

• Batch operations with progress tracking across thousands of records
• Cursor-based pagination for large result sets
• Raw OData filter strings for edge-case queries
• Dynamic query builder (DBQueryItem) for runtime-constructed queries
• Multi-account or migration scenarios targeting a different storage account
• Table name or PartitionKey property overrides via TableOptions

// Only instantiate DataAccess<T> when you need one of the above capabilities
var da = new DataAccess<Customer>();                              // Default credentials
var da = new DataAccess<Customer>("accountName", "accountKey");  // Explicit credentials
var da = new DataAccess<Customer>(new TableOptions              // Table/key override
{
    TableName                = "CustomersArchive",
    PartitionKeyPropertyName = "RegionId"
});

Do not instantiate DataAccess<T> for simple load/save/delete operations. Use the extension methods on your entity objects. They resolve DataAccess<T> internally — you get the same result with a fraction of the code.

When DataAccess<T> is the right choice

// ✅ Batch — extension methods don't expose progress callbacks or batch results
var da     = new DataAccess<Customer>();
var result = await da.BatchUpdateListAsync(largeCustomerList, TableOperationType.InsertOrReplace, progress);
Console.WriteLine($"{result.SuccessfulItems} succeeded, {result.FailedItems} failed");

// ✅ Pagination — extension methods don't expose continuation tokens
var page = await da.GetPagedCollectionAsync(
    x => x.Status == CustomerStatus.Active, pageSize: 50, continuationToken: token);

// ✅ Raw OData — for queries that cannot be expressed as a lambda
var results = await da.GetCollectionByFilterAsync(
    "Status eq 'Active' and CreatedDate gt datetime'2024-01-01'");

// ✅ Dynamic query — runtime-constructed filter
var queryTerms = new List<DBQueryItem>
{
    new DBQueryItem { FieldName = "Status",   FieldValue = "Active", HowToCompare = ComparisonTypes.eq },
    new DBQueryItem { FieldName = "IsActive", FieldValue = "true",   HowToCompare = ComparisonTypes.eq },
};
var results = await da.GetCollectionAsync(queryTerms, QueryCombineStyle.And);

// ✅ Data migration between accounts
var source = new DataAccess<Customer>("source-account", "source-key");
var dest   = new DataAccess<Customer>("dest-account",   "dest-key");
var data   = await source.GetAllTableDataAsync();
await dest.BatchUpdateListAsync(data, TableOperationType.InsertOrReplace);

TableOperationType — used by both extension methods and DataAccess<T>

Value	Behavior
InsertOrReplace	Upsert — full replace if exists (default for .SaveAsync())
InsertOrMerge	Upsert — merge properties if exists (preserves unset fields)
Delete	Delete the entity (default for .DeleteAsync())

---

10. Choosing the Right Pattern — Quick Reference

AI code generation: always start here. Pick the simplest pattern that satisfies the requirement. Only escalate to DataAccess<T> when the extension methods genuinely cannot do what you need.

// ─────────────────────────────────────────────────────────────────────────────
// PATTERN 1: Single entity — load by primary key
// Use when: you know the RowKey (ID) of the record
// ─────────────────────────────────────────────────────────────────────────────
var customer = await new Customer().LoadAsync("CUST-042");


// ─────────────────────────────────────────────────────────────────────────────
// PATTERN 2: Single entity — load by lambda
// Use when: you need to find one record by a non-key field
// ─────────────────────────────────────────────────────────────────────────────
var customer = await new Customer().LoadAsync(x => x.Email == "jane@example.com");


// ─────────────────────────────────────────────────────────────────────────────
// PATTERN 3: Collection — load by lambda
// Use when: you need all matching records for a query
// ─────────────────────────────────────────────────────────────────────────────
var customers = await new List<Customer>().LoadAsync(
    x => x.CompanyId == "COMP-001" && x.IsActive == true);


// ─────────────────────────────────────────────────────────────────────────────
// PATTERN 4: Create or update
// Use when: saving any entity regardless of whether it already exists
// ─────────────────────────────────────────────────────────────────────────────
var order = new Order { CompanyId = "COMP-001", OrderId = "ORD-001", Total = 599.99m };
await order.SaveAsync();                                          // InsertOrReplace (default)
await order.SaveAsync(TableOperationType.InsertOrMerge);         // Merge into existing


// ─────────────────────────────────────────────────────────────────────────────
// PATTERN 5: Load → modify → save (the most common real-world pattern)
// Use when: updating an existing record
// ─────────────────────────────────────────────────────────────────────────────
var order = await new Order().LoadAsync("ORD-001");
order.Status    = OrderStatus.Shipped;
order.ShippedAt = DateTime.UtcNow;
await order.SaveAsync();


// ─────────────────────────────────────────────────────────────────────────────
// PATTERN 6: Delete
// Use when: removing a record
// ─────────────────────────────────────────────────────────────────────────────
await order.DeleteAsync();


// ─────────────────────────────────────────────────────────────────────────────
// PATTERN 7: Load collection → modify all → save all
// Use when: bulk update of a set of related records
// ─────────────────────────────────────────────────────────────────────────────
var orders = await new List<Order>().LoadAsync(
    x => x.CompanyId == "COMP-001" && x.Status == OrderStatus.Pending);

orders.ForEach(o => o.Priority = Priority.High);
await orders.SaveAsync();


// ─────────────────────────────────────────────────────────────────────────────
// PATTERN 8: Load collection → delete all matching
// Use when: removing a set of records
// ─────────────────────────────────────────────────────────────────────────────
var cancelled = await new List<Order>().LoadAsync(x => x.Status == OrderStatus.Cancelled);
await cancelled.DeleteAsync();


// ─────────────────────────────────────────────────────────────────────────────
// PATTERN 9: Batch — only when saving/deleting thousands of records at once
// Escalate to DataAccess<T> for progress tracking and batch result inspection
// ─────────────────────────────────────────────────────────────────────────────
var da     = new DataAccess<Order>();
var result = await da.BatchUpdateListAsync(largeOrderList, TableOperationType.InsertOrReplace, progress);


// ─────────────────────────────────────────────────────────────────────────────
// PATTERN 10: Pagination — only when driving a paged UI or API endpoint
// Escalate to DataAccess<T> for continuation token control
// ─────────────────────────────────────────────────────────────────────────────
var da   = new DataAccess<Order>();
var page = await da.GetPagedCollectionAsync(x => x.CompanyId == "COMP-001", 50, token);

---

11. Batch Operations

ASCDataAccess handles batch operations with automatic chunking at Azure Table Storage's strict 100-entity-per-transaction limit.

Batch constraint: All entities in a single batch transaction must share the same PartitionKey. The library handles this automatically by grouping entities by partition before submitting.

Basic batch

var result = await da.BatchUpdateListAsync(customers);
var result = await da.BatchUpdateListAsync(customers, TableOperationType.InsertOrMerge);
bool ok    = da.BatchUpdateList(customers);

With progress tracking

var progress = new Progress<BatchUpdateProgress>(p =>
    Console.WriteLine($"{p.ProcessedItems}/{p.TotalItems} ({p.PercentComplete:F1}%) " +
                      $"Success: {p.SuccessfulItems}, Failed: {p.FailedItems}")
);

var result = await da.BatchUpdateListAsync(customers, TableOperationType.InsertOrReplace, progress);

Console.WriteLine($"{result.SuccessfulItems} succeeded, {result.FailedItems} failed");

BatchUpdateResult

Property	Type	Description
Success	bool	True if all items succeeded
SuccessfulItems	int	Count of successful operations
FailedItems	int	Count of failed operations
Errors	List<string>	Error details for failed items

Batch delete

await da.BatchUpdateListAsync(staleRecords, TableOperationType.Delete);
await staleRecords.DeleteAsync();

---

12. Pagination

Azure page size ceiling: Azure Table Storage never returns more than 1,000 entities per page regardless of the pageSize value requested.

Basic pagination loop

string? continuationToken = null;

do
{
    var page = await da.GetPagedCollectionAsync(100, continuationToken);

    foreach (var customer in page.Items)
    {
        // Process each customer
    }

    continuationToken = page.ContinuationToken;

} while (!string.IsNullOrEmpty(continuationToken));

By PartitionKey

var page = await da.GetPagedCollectionAsync("COMP-001", pageSize: 50, continuationToken: null);

By lambda expression

var page = await da.GetPagedCollectionAsync(
    predicate:         x => x.Status == CustomerStatus.Active,
    pageSize:          25,
    continuationToken: previousToken
);

// page.Items             — current page items
// page.ContinuationToken — null when done
// page.HasMore           — true if additional pages exist
// page.TotalReturned     — count of items in this page

Initial data load

var initialPage = await da.GetInitialDataLoadAsync(initialLoadSize: 50);
var initialPage = await da.GetInitialDataLoadAsync(x => x.IsActive == true, initialLoadSize: 50);

---

13. Dynamic Entities — Schema-Free Storage

DynamicEntity is one of the most powerful and underutilized features in ASCDataAccess. Where strongly-typed entities require you to know the schema at compile time, DynamicEntity lets you define, store, and retrieve data whose shape is determined at runtime. It uses the same Azure Table Storage infrastructure as typed entities — the same PartitionKey/RowKey model, the same automatic serialization, the same 1 MB entity budget — but with a dictionary-based property model instead of a class definition.

The 1 MB entity limit applies equally to DynamicEntity. Do not use it as an escape hatch for storing large arbitrary payloads.

When DynamicEntity is the right choice

Scenario	Why DynamicEntity fits
Telemetry and event tracking	Each event type has different properties — you cannot pre-define them all
Multi-tenant configuration	Each tenant defines their own custom fields
ETL / import pipelines	You are processing external data whose schema arrives at runtime (CSV, JSON API, webhook payload)
Form builder / custom field systems	Users define their own data structures in your app
Feature flags and A/B test metadata	Properties added and removed without deploying new entity classes
Prototype / rapid iteration	You want to persist data before you have committed to a final schema
Bridging legacy data	Existing data in Table Storage whose columns do not map to a typed class
AI / ML feature stores	Feature sets evolve continuously without a migration cycle

---

Use case 1: Telemetry and event tracking

Different event types carry completely different properties. Rather than maintaining a giant event class with 50 nullable fields, store each event as a DynamicEntity with exactly the properties it carries:

// Page view event — has URL, referrer, duration
var pageView = new DynamicEntity("TelemetryEvents",
    partitionKey: $"2025-04",           // Date bucket — efficient time-range queries
    rowKey:       Guid.NewGuid().ToString());

pageView["EventType"]   = "PageView";
pageView["UserId"]      = userId;
pageView["Url"]         = "/products/headphones";
pageView["Referrer"]    = "google.com";
pageView["DurationMs"]  = 1240;
pageView["SessionId"]   = sessionId;

var da = new DataAccess<DynamicEntity>();
await da.ManageDataAsync(pageView);

// Purchase event — completely different properties, same table
var purchase = new DynamicEntity("TelemetryEvents",
    partitionKey: "2025-04",
    rowKey:       Guid.NewGuid().ToString());

purchase["EventType"]   = "Purchase";
purchase["UserId"]      = userId;
purchase["OrderId"]     = "ORD-9871";
purchase["Amount"]      = 149.99;
purchase["Currency"]    = "USD";
purchase["ProductSku"]  = "SKU-HDPH-001";
purchase["IsFirstBuy"]  = true;

await da.ManageDataAsync(purchase);

Both events live in the same table, partitioned by month. You query them the same way:

// All events for April 2025
var aprilEvents = await da.GetCollectionAsync("2025-04");

// Filter by event type after fetch
var purchases = aprilEvents
    .Where(e => e["EventType"]?.ToString() == "Purchase")
    .ToList();

---

Use case 2: Multi-tenant custom configuration

Each tenant in your SaaS app defines their own configuration keys. You cannot define a typed entity because you do not know what keys each tenant will use:

// Tenant A configures their own notification rules
var tenantConfig = new DynamicEntity("TenantConfig",
    partitionKey: "TENANT-A",
    rowKey:       "NotificationSettings");

tenantConfig["EmailEnabled"]       = true;
tenantConfig["EmailAddress"]       = "alerts@tenant-a.com";
tenantConfig["SlackEnabled"]       = true;
tenantConfig["SlackWebhook"]       = "https://hooks.slack.com/...";
tenantConfig["AlertThreshold"]     = 95.5;
tenantConfig["TimeZone"]           = "America/New_York";

var da = new DataAccess<DynamicEntity>();
await da.ManageDataAsync(tenantConfig);

// Later, read back and access individual settings
var config = await da.GetRowObjectAsync("TENANT-A", ComparisonTypes.eq, "NotificationSettings");

bool emailEnabled  = bool.Parse(config["EmailEnabled"]?.ToString() ?? "false");
string? webhook    = config["SlackWebhook"]?.ToString();
double threshold   = double.Parse(config["AlertThreshold"]?.ToString() ?? "0");

---

Use case 3: Import pipeline — persisting external data at runtime

You receive a JSON payload from a webhook or API whose schema you do not control. Rather than deserializing into a fixed class that breaks whenever the external schema changes, map it directly to a DynamicEntity:

// Incoming Stripe webhook payload — schema varies by event type
string webhookJson = await request.Content.ReadAsStringAsync();

var entity = DynamicEntity.CreateFromJson(
    tableName:   "StripeEvents",
    json:        webhookJson,
    patternConfig: new DynamicEntity.KeyPatternConfig
    {
        PartitionKeyPatterns = new List<DynamicEntity.PatternRule>
        {
            new() { Pattern = @"type",    Priority = 10, KeyType = DynamicEntity.KeyGenerationType.DirectValue }
        },
        RowKeyPatterns = new List<DynamicEntity.PatternRule>
        {
            new() { Pattern = @"id",      Priority = 10, KeyType = DynamicEntity.KeyGenerationType.DirectValue }
        }
    }
);

var da = new DataAccess<DynamicEntity>();
await da.ManageDataAsync(entity);

// The entire webhook payload is now persisted with its native structure
// No class needed, no schema migration when Stripe adds fields

---

Use case 4: Bridging any C# object without a typed entity class

You have an existing C# object — perhaps from a third-party library or a DTO from an external API — and you want to persist it to Table Storage without creating a TableEntityBase subclass for it:

// Any C# object — no TableEntityBase required
public class ExternalProductDto
{
    public string ProductId    { get; set; } = "";
    public string CategoryId   { get; set; } = "";
    public string Name         { get; set; } = "";
    public decimal Price       { get; set; }
    public bool IsAvailable    { get; set; }
    public List<string> Tags   { get; set; } = new();
}

var product = new ExternalProductDto
{
    ProductId  = "prod-001",
    CategoryId = "electronics",
    Name       = "Wireless Headphones",
    Price      = 149.99m,
    IsAvailable = true,
    Tags       = new List<string> { "audio", "wireless", "bluetooth" }
};

// Convert any object to DynamicEntity — reflection maps all public properties
var entity = DynamicEntity.CreateFromObject(
    product,
    tableName:               "Products",
    partitionKeyPropertyName: "CategoryId"  // Which property becomes PartitionKey
);

var da = new DataAccess<DynamicEntity>();
await da.ManageDataAsync(entity);

---

Use case 5: Feature flag and A/B test store

Feature flags evolve constantly — flags are added, removed, and modified without a deployment cycle. DynamicEntity stores them without requiring entity class changes:

var da = new DataAccess<DynamicEntity>();

// Write a feature flag — any shape, any properties
var flag = new DynamicEntity("FeatureFlags",
    partitionKey: "v2-rollout",
    rowKey:       "new-checkout-flow");

flag["Enabled"]          = true;
flag["RolloutPercent"]   = 25;
flag["TargetRegions"]    = "US,CA,GB";
flag["ExcludedUserIds"]  = "user-123,user-456";
flag["ExpiresAt"]        = DateTime.UtcNow.AddDays(30).ToString("O");
flag["Notes"]            = "Phase 1 rollout — watch conversion rate";

await da.ManageDataAsync(flag);

// Read back and use
var allFlags = await da.GetCollectionAsync("v2-rollout");

var checkoutFlag = allFlags.FirstOrDefault(f => f.RowKey == "new-checkout-flow");
bool isEnabled   = bool.Parse(checkoutFlag?["Enabled"]?.ToString() ?? "false");
int rollout      = int.Parse(checkoutFlag?["RolloutPercent"]?.ToString() ?? "0");

---

Explicit key construction

When you know exactly what your PartitionKey and RowKey should be, use the explicit constructor:

var entity = new DynamicEntity(
    tableName:    "AuditLog",
    partitionKey: $"{userId}",          // All audit entries for a user in one partition
    rowKey:       $"{DateTime.UtcNow:O}_{Guid.NewGuid()}"  // Timestamp + GUID for uniqueness
);

entity["Action"]     = "PasswordChanged";
entity["IpAddress"]  = ipAddress;
entity["UserAgent"]  = userAgent;
entity["Success"]    = true;

var da = new DataAccess<DynamicEntity>();
await da.ManageDataAsync(entity);

---

Pattern-based automatic key detection

When you have a dictionary of properties and want the library to figure out which fields should be PartitionKey and RowKey based on naming patterns:

var config = new DynamicEntity.KeyPatternConfig
{
    PartitionKeyPatterns = new List<DynamicEntity.PatternRule>
    {
        // Any field whose name matches ".*company.*id.*" becomes the PartitionKey
        new() { Pattern = @".*company.*id.*", Priority = 10,
                KeyType = DynamicEntity.KeyGenerationType.DirectValue },

        // Fields matching ".*date.*" become a date-bucket PartitionKey (YYYY-MM)
        new() { Pattern = @".*created.*date.*", Priority = 5,
                KeyType = DynamicEntity.KeyGenerationType.DateBased }
    },
    RowKeyPatterns = new List<DynamicEntity.PatternRule>
    {
        new() { Pattern = @".*order.*id.*", Priority = 10,
                KeyType = DynamicEntity.KeyGenerationType.DirectValue },

        // If no ID field found, generate a reverse timestamp (newest-first ordering)
        new() { Pattern = @".*",            Priority = 1,
                KeyType = DynamicEntity.KeyGenerationType.ReverseTimestamp }
    },
    FuzzyMatchThreshold = 0.8  // 1.0 = exact match only; 0.0 = everything matches
};

var entity = new DynamicEntity("Orders", new Dictionary<string, object>
{
    ["CompanyId"]    = "COMP-001",   // → PartitionKey (matches company.*id)
    ["OrderId"]      = "ORD-999",   // → RowKey (matches order.*id)
    ["Amount"]       = 499.00,
    ["Status"]       = "Pending",
    ["CreatedDate"]  = DateTime.UtcNow
}, config);

KeyGenerationType values

Value	Description	Best for
DirectValue	Use field value as-is	IDs, codes, natural keys
DateBased	Convert to YYYY-MM bucket	Time-series data, log archives
Sequential	Generate sequential key	Ordered processing queues
ReverseTimestamp	Newest-first timestamp	Activity feeds, notifications
Composite	Combine multiple fields	Compound natural keys
Generated	Generate a new GUID	When no natural key exists

Reading back and accessing properties

var da = new DataAccess<DynamicEntity>();

// Fetch a single entity
var entity = await da.GetRowObjectAsync("ORD-999");

// Access properties by key — returns object?
string? status = entity["Status"]?.ToString();
double  amount = double.Parse(entity["Amount"]?.ToString() ?? "0");

// Diagnostics — see what the entity contains
Console.WriteLine(entity.ToString());      // Human-readable property dump
Console.WriteLine(entity.GetDiagnostics()); // Property counts and sizes

Lambda expressions with DynamicEntity

Lambda expressions work with DataAccess<DynamicEntity> the same way they do with typed entities — with one important constraint to understand.

What is queryable server-side: DynamicEntity exposes PartitionKey, RowKey, and Timestamp as real C# properties. Lambda predicates on those three properties are translated to OData and execute server-side efficiently.

What is not queryable server-side: The dictionary-based properties — entity["Status"], entity["Amount"], etc. — are not real C# properties. They cannot be translated into an OData filter. Predicates that reference them are evaluated client-side after Azure returns the results.

This means the right query strategy for DynamicEntity is: use PartitionKey to scope the query server-side, then filter by dynamic properties in memory.

var da = new DataAccess<DynamicEntity>();

// ✅ Server-side — PartitionKey and RowKey are real properties, fully translated to OData
var aprilEvents  = await da.GetCollectionAsync("2025-04");          // All April 2025 events
var specificFlag = await da.GetRowObjectAsync("new-checkout-flow");  // Single entity by RowKey

// ✅ Server-side lambda on PartitionKey
var tenantData = await da.GetCollectionAsync(
    x => x.PartitionKey == "TENANT-A"
);

// ✅ Server-side lambda on Timestamp — useful for time-range queries
var recentEntries = await da.GetCollectionAsync(
    x => x.Timestamp >= DateTimeOffset.UtcNow.AddHours(-1)
);

// ✅ Server-side — PartitionKey scope + client-side filter on dynamic property
// Azure executes the PartitionKey filter; your code filters the results in memory
var aprilPurchases = (await da.GetCollectionAsync("2025-04"))
    .Where(e => e["EventType"]?.ToString() == "Purchase")
    .ToList();

// ✅ Pattern: PartitionKey scopes efficiently, in-memory filter refines
var activeFlags = (await da.GetCollectionAsync("v2-rollout"))
    .Where(f => bool.TryParse(f["Enabled"]?.ToString(), out var enabled) && enabled)
    .ToList();

// ⚠️ This executes client-side — dynamic properties cannot reach OData
// Fine for small partitions, expensive for large ones
var da2 = new DataAccess<DynamicEntity>();
var allPurchases = await da2.GetCollectionAsync(
    x => x["EventType"]!.ToString() == "Purchase"  // ⚠️ Not an OData-translatable expression
);

The practical query pattern for DynamicEntity

Because dynamic properties cannot drive a server-side filter, the most efficient query pattern is always:

1. Scope with PartitionKey — let Azure handle the coarse filter
2. Refine with LINQ — filter by dynamic properties in memory on the scoped result

This is why good PartitionKey design matters even more for DynamicEntity than for typed entities. A well-chosen partition (user ID, date bucket, tenant ID, event category) keeps the in-memory filter set small. A poor partition (everything in one partition, or a globally unique value per row) forces either a full table scan or defeats the purpose of partitioning entirely.

// ✅ Good pattern — narrow partition scope, small in-memory filter set
var da = new DataAccess<DynamicEntity>();

var userEvents = (await da.GetCollectionAsync($"user-{userId}"))   // Server: one user's events
    .Where(e => e["EventType"]?.ToString() == "Purchase"           // Memory: purchase events only
             && DateTime.TryParse(e["CreatedAt"]?.ToString(),       // Memory: last 7 days
                    out var dt) && dt >= DateTime.UtcNow.AddDays(-7))
    .OrderByDescending(e => e["CreatedAt"]?.ToString())
    .Take(10)
    .ToList();

---

14. Azure Blob Storage

AzureBlobs provides a full-featured Azure Blob Storage client with tag-based indexing, lambda expression filtering, multi-file operations, FFmpeg video support, and stream uploads.

Use Blob Storage for large content. Any property that would exceed ~50 KB serialized — HTML, source code, binary files, large text, serialized datasets — belongs in Blob Storage. Store the blob URL as a string property on the entity. This is how you keep entity sizes lean and use the correct storage primitive for each data type.

Instantiation

var blobs = new AzureBlobs("documents");                                       // Default credentials, private container
var blobs = new AzureBlobs("videos", defaultMaxFileSizeBytes: 500*1024*1024);  // Custom size limit, private container
var blobs = new AzureBlobs("accountName", "accountKey", "documents");          // Explicit credentials, private container

// Public-blob container (anonymous read access for blobs; container listing remains private)
var blobs = new AzureBlobs("chatbot-files", PublicAccessType.Blob);
var blobs = new AzureBlobs("acct", "key", "chatbot-files", PublicAccessType.Blob);

The constructor always calls CreateIfNotExistsAsync(containerAccess) so the container is auto-provisioned on first use — exactly the same way the DAL auto-creates Azure tables on first write. The default PublicAccessType.None preserves backward compatibility; existing call sites that do not specify an access level continue to produce fully private containers.

Dynamic public container provisioning (anonymous-read URLs)

Most blob containers should remain private — files are accessed via the typed AzureBlobs.DownloadAsync / GetBlobWithContentAsync methods that authenticate with the storage account key. But some scenarios genuinely need anonymously-readable URLs:

• Chatbot attachments (screenshots and files uploaded during a conversation that an external triage LLM later needs to fetch by URL)
• Public marketing assets, OG / share-link preview images
• AI-readable artifacts where the consuming model has no Azure credentials to present

For those cases pass PublicAccessType.Blob to the constructor and the container is created (or reused) with anonymous read access on blobs. The container itself remains non-enumerable — outside parties cannot list its contents, they can only fetch blobs whose names they already know.

using ASCTableStorage.Blobs;
using Azure.Storage.Blobs.Models;

// One-line setup: container is created with PublicAccessType.Blob on first use.
var blobs = new AzureBlobs("chatbot-files", PublicAccessType.Blob);

// The returned Uri is anonymously fetchable by any client.
Uri url = await blobs.UploadStreamAsync(
    content:     screenshotStream,
    fileName:    $"{Guid.NewGuid()}-bug.png",
    contentType: "image/png");
// → https://{account}.blob.core.windows.net/chatbot-files/{guid}-bug.png

EnsureContainerAccessAsync — fix or escalate an existing container

The constructor's container provisioning runs fire-and-forget (it does not block; it does not throw to the caller). If a container was created on a prior run with a different access level — for example originally private, now needing to be public — or if you want exceptions surfaced instead of swallowed, call the awaitable helper:

var blobs = new AzureBlobs("chatbot-files");

try
{
    // Ensures the container exists AND has Blob-level public access.
    // First call creates it; subsequent calls flip the access policy if it differs.
    await blobs.EnsureContainerAccessAsync(PublicAccessType.Blob);
}
catch (Azure.RequestFailedException ex) when (ex.ErrorCode == "PublicAccessNotPermitted")
{
    // The storage account has "Allow Blob anonymous access" disabled at the account level.
    // CreateIfNotExists succeeded (container exists), but SetAccessPolicy was rejected.
    // The blobs are reachable only via SAS or shared-key authentication.
    _logger.LogWarning("Storage account forbids public blob access; URLs will require SAS.");
}

Behavior summary:

• CreateIfNotExistsAsync(containerAccess) — runs first; container is guaranteed to exist when this returns.
• SetAccessPolicyAsync(containerAccess) — runs second; flips access if it differs from what the container already has.
• If the account-level "Allow Blob anonymous access" toggle is OFF in the Azure Portal, only the SetAccessPolicy step throws; the container still exists privately so the caller can fall back gracefully.

When NOT to use PublicAccessType.Blob

Scenario	Recommendation
User PII (medical records, payment details, identity documents)	PublicAccessType.None + SAS-token-gated downloads
Internal-only documents	None + authenticated access
Anything subject to data-residency / GDPR right-to-erasure	None — public URLs cannot be reliably retracted from caches
Public marketing assets, share-link previews, OG images	Blob — exactly what it is for
Bug-report screenshots / triage evidence	Blob — downstream agents need URL access without tokens
AI-readable artifacts (model inputs, evaluation outputs)	Blob — the LLM cannot present an Azure auth token

PublicAccessType reference

Azure.Storage.Blobs.Models.PublicAccessType ships with the Azure SDK; AzureBlobs accepts the value directly so the caller can choose any level Azure supports.

Value	Behavior
None	Container and blobs are private. Access requires shared key, SAS, or AAD authentication. (Default.)
Blob	Blobs are anonymously readable by URL; the container itself cannot be enumerated anonymously.
BlobContainer	Both blobs and container listings are anonymously readable. Rarely needed — prefer Blob.

Upload

// From file path
Uri uri = await blobs.UploadFileAsync(
    filePath:  "/tmp/invoice.pdf",
    indexTags: new Dictionary<string, string> { ["Year"] = "2025", ["Status"] = "Pending" },
    metadata:  new Dictionary<string, string> { ["UploadedBy"] = "system" }
);

// From stream
Uri uri = await blobs.UploadStreamAsync(stream, "photo.jpg",
    indexTags: new Dictionary<string, string> { ["UserId"] = "user-123" });

// From string (HTML, CSS, JS, source code, text)
Uri uri = await blobs.UploadStringDataAsync("report.html", "<html>...</html>", "text/html",
    tags: new Dictionary<string, string> { ["Year"] = "2025" });

// Multiple files
var results = await blobs.UploadMultipleFilesAsync(fileList);

Query with lambda expressions on tags

List<BlobData> list = await blobs.GetCollectionAsync(
    x => x.Tags["Year"] == "2025" && x.Tags["Department"] == "Finance"
);

// Streaming — start processing before download completes
await foreach (var blob in blobs.GetCollectionAsyncEnumerable(
    x => x.Tags["Status"] == "Approved", loadContent: false))
{
    Console.WriteLine($"{blob.Name} — {blob.Size} bytes");
}

Blob tag limits: Maximum 10 index tags per blob. Plan your 10 tags around your most important query dimensions.

Search, download, delete, and tag management

// Search
var results = await blobs.SearchBlobsByTagsAsync("Year = '2025' AND Status = 'Active'");
var results = await blobs.SearchBlobsAsync("invoice", tagFilters, startDate, endDate);

// Download
bool ok             = await blobs.DownloadFileAsync("invoice.pdf", "/tmp/invoice.pdf");
BlobData? blob      = await blobs.GetBlobWithContentAsync("invoice.pdf");
List<BlobData> many = await blobs.GetBlobsWithContentAsync(new[] { "a.pdf", "b.pdf" });

// Delete
bool ok      = await blobs.DeleteBlobAsync("invoice.pdf");
var results  = await blobs.DeleteMultipleBlobsAsync(new[] { "a.pdf", "b.pdf" });
await blobs.DeleteMultipleBlobsAsync(x => x.Tags["Status"] == "Expired");

// Tags
await blobs.UpdateBlobTagsAsync("invoice.pdf", new Dictionary<string, string> { ["Status"] = "Paid" });
var tags = await blobs.GetBlobTagsAsync("invoice.pdf");

// List
var all = await blobs.ListBlobsAsync(prefix: "invoices/2025/", loadContent: false);
await foreach (var b in blobs.ListBlobsAsyncEnumerable("invoices/")) { }

File type restrictions

blobs.AddAllowedFileType(".pdf", "application/pdf");
blobs.AddAllowedFileType(".jpg", "image/jpeg");
bool ok = blobs.IsFileTypeAllowed("document.pdf");
blobs.RemoveAllowedFileType(".pdf");

FFmpeg video support

// Auto-compress if upload exceeds size limit
Uri uri = await blobs.UploadStreamAsync(videoStream, "demo.mp4",
    maxFileSizeBytes: 50 * 1024 * 1024);

// Direct FFmpeg operations
byte[] compressed = await FFmpegManager.CompressVideoAsync(videoBytes, targetBitrateKbps: 2000);
byte[] thumbnail  = await FFmpegManager.ExtractThumbnailAsync(videoBytes, timeSeconds: 3.0);
byte[] stitched   = await FFmpegManager.StitchVideosAsync(videoChunks, "output.mp4");

Blob streaming with IAsyncEnumerable — process results as they arrive

AzureBlobs exposes three IAsyncEnumerable<BlobData> methods that stream results from Azure as they are found rather than waiting for the entire result set to be assembled in memory first. This is a fundamentally different execution model from the standard GetCollectionAsync / ListBlobsAsync methods, and it enables scenarios those methods cannot support efficiently.

The difference in plain terms: GetCollectionAsync fetches all matching blobs, builds the full list, then hands it to you. GetCollectionAsyncEnumerable hands you each blob the moment Azure returns it — your processing code runs in parallel with Azure's continued search. For a container with 10,000 blobs, the first result arrives in milliseconds and you begin working immediately. With the non-streaming version you wait until all 10,000 are fetched.

All three streaming methods accept a CancellationToken, which means the entire operation — including the ongoing Azure query — can be cancelled mid-stream without waiting for completion.

// Stream by lambda tag filter — yields each matching blob as Azure finds it
await foreach (var blob in blobs.GetCollectionAsyncEnumerable(
    x => x.Tags["Year"] == "2025" && x.Tags["Department"] == "Finance",
    loadContent: false,          // Don't download bytes — metadata only for fast scanning
    cancellationToken: cts.Token))
{
    Console.WriteLine($"{blob.Name} — {blob.Size:N0} bytes");
}

// Stream all blobs in a container or prefix — useful for inventory and migration
await foreach (var blob in blobs.ListBlobsAsyncEnumerable(
    prefix:      "invoices/2025/",
    loadContent: false,
    cancellationToken: cts.Token))
{
    await ProcessInvoiceMetadataAsync(blob);
}

// Stream by raw tag query string
await foreach (var blob in blobs.SearchBlobsByTagsAsyncEnumerable(
    tagQuery:          "Status = 'Pending' AND Priority = 'High'",
    loadContent:       true,   // Download content for each blob — use only when needed
    cancellationToken: cts.Token))
{
    await ProcessDocumentAsync(blob.Data!);
}

When to use streaming vs standard collection methods

Scenario	Use
Large container scan where you want to start processing immediately	GetCollectionAsyncEnumerable
Building a real-time dashboard that populates as results arrive	GetCollectionAsyncEnumerable
Feeding a Blazor or SignalR component that renders items progressively	GetCollectionAsyncEnumerable
Processing pipeline where each item triggers async work	GetCollectionAsyncEnumerable
Bulk export or migration where stopping mid-way must cancel the Azure query	ListBlobsAsyncEnumerable + CancellationToken
User-initiated search with a cancel button	Any AsyncEnumerable variant + CancellationToken
Small result set where you need all items before proceeding	GetCollectionAsync
Displaying a complete sorted/grouped list in a UI	GetCollectionAsync

Streaming with early exit — processing until a condition is met

Because IAsyncEnumerable is a lazy pull sequence, you can stop consuming it at any point with break and the Azure query is cancelled cleanly. No wasted network traffic for results you will never use:

// Find the first 5 blobs over 10MB — stop as soon as we have them
var largeFiles = new List<BlobData>();

await foreach (var blob in blobs.ListBlobsAsyncEnumerable("uploads/", loadContent: false))
{
    if (blob.Size > 10 * 1024 * 1024)
        largeFiles.Add(blob);

    if (largeFiles.Count >= 5)
        break;  // Cancels the Azure enumeration — no further network calls
}

Streaming into a Blazor component

The async enumerable pattern maps naturally to progressive UI rendering in Blazor. The component begins rendering results immediately without waiting for the full dataset:

// In a Blazor component
private List<BlobData> _documents = new();

protected override async Task OnInitializedAsync()
{
    var blobs = new AzureBlobs("documents");

    await foreach (var doc in blobs.GetCollectionAsyncEnumerable(
        x => x.Tags["ClientId"] == ClientId,
        loadContent: false))
    {
        _documents.Add(doc);
        StateHasChanged();  // Re-render after each item — UI populates progressively
        await Task.Yield(); // Yield control back to the renderer between items
    }
}

Memory profile comparison

For a container with 50,000 blobs:

Method	Peak memory	Time to first result	Cancellable mid-operation
ListBlobsAsync()	~50,000 objects allocated before processing	After all 50,000 fetched	No
ListBlobsAsyncEnumerable()	1 object at a time	Milliseconds	Yes

For large containers or any scenario where the full result set would be unwieldy in memory, the streaming variants are the correct choice. They are not a convenience alias for the list methods — they represent a genuinely different and more efficient execution path.

---

15. Queue Management with StateList — Resumable Work Queues

QueueData<T> and StateList<T> provide a persistent, resumable work queue backed by Azure Table Storage — replacing Azure Service Bus or Azure Queue Storage for scenarios where you need position tracking and crash recovery without message broker complexity.

Queue entity size applies. The serialized StateList<T> is stored as a JSON property. For very large queues, split work across multiple QueueData<T> instances grouped by the same Name (PartitionKey).

StateList<T>

var list = new StateList<Order>(orders);

list.MoveNext();                        // Forward — returns true if successful
list.MovePrevious();                    // Backward
list.JumpTo(42);                        // Jump to index
list.ResetNavigation();                 // Reset to before first item

Order? current = list.Current;
int    index   = list.CurrentIndex;     // -1 if not started

var checkpoint = list.GetCheckpoint();
list.RestoreCheckpoint(checkpoint);     // Roll back to saved position

List<Order> remaining = list.GetRemainingItems();

list.Add(newOrder);
list.AddRange(moreOrders);
list.RemoveAll(o => o.Status == "Cancelled");
list.Sort((a, b) => a.CreatedDate.CompareTo(b.CreatedDate));

StateList<Order> sl = orders.ToStateList();

QueueData<T> — create and inspect

var queue = QueueData<Order>.CreateFromList(pendingOrders, name: "PendingOrders");
await queue.SaveQueueAsync();

var queue = QueueData<Order>.CreateFromStateList(stateList, name: "OrderBatch-2025-04");
await queue.SaveQueueAsync();

var queue = await QueueData<Order>.GetQueueAsync(queueId);

Console.WriteLine($"Status:   {queue.ProcessingStatus}");    // "In Progress (42 of 500)"
Console.WriteLine($"Progress: {queue.PercentComplete:F1}%");
Console.WriteLine($"Items:    {queue.TotalItemCount}");
Console.WriteLine($"Position: {queue.LastProcessedIndex}");

Processing with crash recovery

var queue = await QueueData<Order>.GetQueueAsync(queueId);
if (queue == null) return;

while (queue.Data.MoveNext())
{
    var order = queue.Data.Current!;
    try
    {
        await ProcessOrderAsync(order);

        if ((queue.Data.CurrentIndex + 1) % 10 == 0)
            await queue.SaveQueueAsync();  // Persist position — survives crashes
    }
    catch
    {
        await queue.SaveQueueAsync();
        throw;
    }
}

await QueueData<Order>.DeleteQueueAsync(queueId);

Retrieval and batch management

var q   = await QueueData<Order>.GetQueueAsync(queueId);
var p   = await QueueData<Order>.PeekQueueAsync(queueId);       // Non-destructive
var gd  = await QueueData<Order>.GetAndDeleteQueueAsync(queueId); // One-shot
var all = await QueueData<Order>.GetQueuesAsync("PendingOrders");

int deleted = await QueueData<Order>.DeleteQueuesAsync(new List<string> { id1, id2 });
int deleted = await QueueData<Order>.DeleteQueuesMatchingAsync(q => q.PercentComplete == 100);
List<StateList<Order>> data = await QueueData<Order>.DeleteAndReturnAllAsync("PendingOrders");

---

16. Session Management — Taking Over All Session State in Your App

ASCDataAccess does not just provide a Session class — it provides a complete session management takeover for any .NET application type. Once configured, you interact with session state using the same familiar API regardless of whether you are building a web app, a desktop app, a console tool, or a background service. The session layer automatically detects the platform, manages session IDs in the right storage mechanism for that platform, batches writes to Azure Table Storage, and cleans up stale data on a configurable schedule.

The key principle: you do not manage session objects directly in your application code. You register the session infrastructure once in Program.cs and then access session state anywhere in your app through SessionManager.Current — one line that always returns the right session for the current user, regardless of platform.

How SessionManager works

SessionManager.Current is the unified entry point. It automatically determines the correct session ID for the current execution context using this priority chain:

1. Authenticated user identity (web apps) — the authenticated user's Identity.Name or NameIdentifier claim
2. Session cookie (web apps, unauthenticated) — reads or creates a HttpOnly cookie scoped to the browser
3. Explicitly configured session ID (desktop / console)
4. Custom ID provider — your own Func<string> for special scenarios
5. Platform-specific detection — Username_MachineName for desktop; MachineName_ProcessId for services
6. Generated fallback — a stable GUID persisted to LocalApplicationData for survival across restarts

Sessions are stored in a ConcurrentDictionary keyed by session ID — fully thread-safe and designed to scale to thousands of simultaneous users without contention.

SessionIdStrategy — choosing the right strategy for your app

Strategy	Best for	Session ID source
Auto	Let the library detect the platform	Platform-appropriate automatic detection
HttpContext	ASP.NET Core web apps	HTTP cookie (per browser) or authenticated user identity
UserMachine	Desktop apps	Username_MachineName — stable across app restarts
MachineProcess	Windows services, background workers	MachineName_ProcessId
Custom	Multi-tenant, special routing	Your own Func<string> provider
Guid	Isolated one-off sessions	New GUID per session

---

Program.cs setup — ASP.NET Core web application

using ASCTableStorage;
using ASCTableStorage.Sessions;

// ── 1. Global DAL credentials ──────────────────────────────────────────────
TableStorage.Configure(
    builder.Configuration["Azure:TableStorageName"]!,
    builder.Configuration["Azure:TableStorageKey"]!
);

// ── 2. Logging — first-class ILogger provider writing to Azure Table Storage
builder.Logging.ClearProviders();
builder.Logging.AddAzureTableLogging(options =>
{
    options.AccountName   = builder.Configuration["Azure:TableStorageName"]!;
    options.AccountKey    = builder.Configuration["Azure:TableStorageKey"]!;
    options.MinimumLevel  = LogLevel.Warning;
    options.RetentionDays = 60;
});

// ── 3. Session management — takes over ALL session state for the application
builder.Services.AddAzureTableSessions(options =>
{
    options.AccountName       = builder.Configuration["Azure:TableStorageName"]!;
    options.AccountKey        = builder.Configuration["Azure:TableStorageKey"]!;
    options.IdStrategy        = SessionIdStrategy.HttpContext; // Cookie per browser / user identity
    options.CleanupInterval   = TimeSpan.FromMinutes(30);
    options.BatchWriteDelay   = TimeSpan.FromMilliseconds(500);
    options.AutoCommitInterval = TimeSpan.FromMilliseconds(500);
    options.StaleDataCleanupAge = TimeSpan.FromMinutes(45);
    options.EnableAutoCleanup = true;
});

// ── 4. Build the app ────────────────────────────────────────────────────────
var app = builder.Build();

// ── 5. Post-build initialization ────────────────────────────────────────────
// RemoteLogger must be wired after DI is built so ILoggerFactory is available
RemoteLogger.Initialize(app.Services.GetRequiredService<ILoggerFactory>());

// ── IMPORTANT: Do NOT call app.UseSession()
// AzureTableSessions registers its own middleware and manages session state
// internally. Calling app.UseSession() would conflict with it.

---

Program.cs setup — desktop application (WinForms, WPF, Avalonia)

Desktop apps do not have a DI container or IHostBuilder by default, but SessionManager works without one. The session ID is derived from Username_MachineName, persisted to LocalApplicationData, and survives application restarts.

using ASCTableStorage;
using ASCTableStorage.Sessions;
using ASCTableStorage.Logging;
using Microsoft.Extensions.Logging;

// ── 1. Global DAL credentials ──────────────────────────────────────────────
TableStorage.Configure("your-account-name", "your-account-key");

// ── 2. Logging ─────────────────────────────────────────────────────────────
var loggerFactory = LoggerFactory.Create(logging =>
{
    logging.ClearProviders();
    logging.AddAzureTableLogging(options =>
    {
        options.AccountName   = "your-account-name";
        options.AccountKey    = "your-account-key";
        options.MinimumLevel  = LogLevel.Information;
        options.RetentionDays = 30;
    });
});

// Wire RemoteLogger so the rest of the DAL can log through it
RemoteLogger.Initialize(loggerFactory);

// ── 3. Session management ──────────────────────────────────────────────────
await SessionManager.InitializeAsync(
    "your-account-name",
    "your-account-key",
    options =>
    {
        options.IdStrategy   = SessionIdStrategy.UserMachine; // Username_MachineName
        options.AutoCommitInterval = TimeSpan.FromSeconds(2);
        options.EnableAutoCleanup  = true;
        options.CleanupInterval    = TimeSpan.FromHours(1);
    }
);

// ── 4. Application entry point ─────────────────────────────────────────────
// Now SessionManager.Current is available anywhere in the app
Application.Run(new MainForm());

---

Program.cs setup — console / background service

using ASCTableStorage;
using ASCTableStorage.Sessions;

var host = Host.CreateDefaultBuilder(args)
    .ConfigureServices((context, services) =>
    {
        // ── 1. Session management via IHostBuilder extension ───────────────
        services.AddAzureTableSessions(options =>
        {
            options.AccountName  = context.Configuration["Azure:TableStorageName"]!;
            options.AccountKey   = context.Configuration["Azure:TableStorageKey"]!;
            options.IdStrategy   = SessionIdStrategy.MachineProcess;
            options.EnableAutoCleanup = true;
        });
    })
    .ConfigureLogging((context, logging) =>
    {
        logging.ClearProviders();
        logging.AddAzureTableLogging(options =>
        {
            options.AccountName  = context.Configuration["Azure:TableStorageName"]!;
            options.AccountKey   = context.Configuration["Azure:TableStorageKey"]!;
            options.MinimumLevel = LogLevel.Information;
        });
    })
    .Build();

TableStorage.Configure(
    host.Services.GetRequiredService<IConfiguration>()["Azure:TableStorageName"]!,
    host.Services.GetRequiredService<IConfiguration>()["Azure:TableStorageKey"]!
);

RemoteLogger.Initialize(host.Services.GetRequiredService<ILoggerFactory>());

await host.RunAsync();

---

Using session state anywhere in your application

Once initialized, access session state through SessionManager.Current from anywhere — a controller, a service, a background task, or a form. The correct session is always returned for the current user context automatically.

// ── Store any typed value ──────────────────────────────────────────────────
var session = SessionManager.Current;

session.SetString("Theme",  "dark");
session.SetInt32("PageSize", 25);
session.SetObject("UserPreferences", new UserPrefs { Language = "en", ShowHelp = true });

// ── Retrieve typed values ──────────────────────────────────────────────────
string? theme    = session.GetString("Theme");
int?    pageSize = session.GetInt32("PageSize");
UserPrefs? prefs = session.GetObject<UserPrefs>("UserPreferences");

// ── DateTime helpers ───────────────────────────────────────────────────────
session.SetString("LastLogin", DateTime.UtcNow.ToString("O"));
DateTime lastLogin    = session.GetDateTime("LastLogin");
DateTime lastLoginUtc = session.GetUtcDateTime("LastLogin");

// ── Raw AppSessionData access (when you need full control) ─────────────────
session["CartTotal"] = new AppSessionData { Value = 149.99m };
var cartData = session["CartTotal"];
decimal total = (decimal)cartData!.Value!;

// ── Check, remove, and clear ───────────────────────────────────────────────
bool exists  = session.ContainsKey("Theme");
bool removed = await session.RemoveAsync("Theme");
await session.ClearAsync();   // Wipes all data for this session
await session.FlushAsync();   // Force immediate write to Azure (bypass batch delay)

Commit behavior — automatic and manual

By default, session writes are batched and flushed to Azure Table Storage every 500ms (AutoCommitInterval). This prevents chatty per-keystroke writes under rapid updates. You can also commit explicitly:

// Manual commit — use when you need the data persisted immediately
await session.CommitDataAsync();

// Check if uncommitted changes exist
bool dirty = !session.DataHasBeenCommitted;

// Refresh from Azure — picks up changes made by another process or request
await session.RefreshSessionDataAsync();

// Full session restart — clears all stored data for this session ID
await session.RestartSessionAsync();

Multi-session management

SessionManager maintains isolated sessions for every concurrent user automatically. You can also explicitly access any session by ID — useful in admin tools, impersonation, or multi-user desktop apps:

// Access a specific session by ID
Session adminView = SessionManager.GetSession("user-456-session");

// Find and clean up stale sessions
List<string> staleIds = await SessionManager.Current.GetStaleSessionsAsync();
foreach (var id in staleIds)
{
    var stale = SessionManager.GetSession(id);
    await stale.RestartSessionAsync();
}

// Statistics
var stats = SessionManager.GetStatistics();
Console.WriteLine($"Total writes: {stats?.TotalWrites}");

SessionOptions reference

Option	Default	Description
AccountName	Required	Azure Storage account name
AccountKey	Required	Azure Storage account key
IdStrategy	Auto	How session IDs are determined
CustomIdProvider	null	Func<string> for Custom strategy
TableName	AppSessionData	Azure table name for session storage
AutoCommitInterval	500ms	How often pending writes are flushed
StaleDataCleanupAge	45min	Age at which session rows are considered stale
CleanupInterval	30min	How often the cleanup background job runs
CleanupStartDelay	30sec	Delay before first cleanup run after startup
EnableAutoCleanup	true	Run automatic stale data cleanup
TrackActivity	true	Track session activity for stale detection
ApplicationName	null	Optional grouping label

---

17. Logging — A First-Class ILogger Provider

The most important thing to understand about logging in ASCDataAccess: you do not use ErrorLogData directly in your day-to-day application code. AzureTableLogger is a first-class ILogger provider that plugs into the standard .NET logging pipeline. Once registered, you log exactly the same way you do with any other .NET logger — via constructor injection, ILoggerFactory, or RemoteLogger.CreateLogger<T>(). All of that writes to Azure Table Storage automatically, with background batching, automatic retention cleanup, and per-level retention policies.

ErrorLogData is a secondary, specialized API for cases where you want to write a rich structured error record — with customer ID, session ID, caller info, and a full exception trace — into a dedicated error table separate from the application log stream. It is not the everyday logging path.

Register the logger — ASP.NET Core

This registration replaces the default console/debug providers with Azure Table Storage. Add it to your Program.cs builder block before app.Build():

builder.Logging.ClearProviders();
builder.Logging.AddAzureTableLogging(options =>
{
    options.AccountName       = builder.Configuration["Azure:TableStorageName"]!;
    options.AccountKey        = builder.Configuration["Azure:TableStorageKey"]!;
    options.MinimumLevel      = LogLevel.Warning;       // Trace, Debug, Information, Warning, Error, Critical
    options.RetentionDays     = 60;                     // Default retention for all levels
    options.EnableAutoCleanup = true;                   // Runs cleanup on a background timer
    options.CleanupInterval   = TimeSpan.FromDays(1);  // How often cleanup runs
    options.RetentionByLevel  = new Dictionary<LogLevel, int>
    {
        [LogLevel.Information] = 7,    // Information logs kept 7 days
        [LogLevel.Warning]     = 30,   // Warning logs kept 30 days
        [LogLevel.Error]       = 90,   // Error logs kept 90 days
        [LogLevel.Critical]    = 365   // Critical logs kept 1 year
    };
});

Wire RemoteLogger after build

RemoteLogger is the DAL's internal logger bridge — it allows DAL classes (session manager, batch operations, etc.) to emit logs through your registered ILoggerFactory. Wire it in the post-build initialization block:

var app = builder.Build();

// Must be called after Build() so ILoggerFactory is available from DI
RemoteLogger.Initialize(app.Services.GetRequiredService<ILoggerFactory>());

Register the logger — desktop / console (non-DI)

var loggerFactory = LoggerFactory.Create(logging =>
{
    logging.ClearProviders();
    logging.AddAzureTableLogging(options =>
    {
        options.AccountName   = "your-account-name";
        options.AccountKey    = "your-account-key";
        options.MinimumLevel  = LogLevel.Information;
        options.RetentionDays = 30;
    });
});

// Wire RemoteLogger so DAL internals can log through this factory
RemoteLogger.Initialize(loggerFactory);

Using the logger — standard .NET logging API throughout your app

Once registered, inject ILogger<T> anywhere in your application — controllers, services, background workers, anywhere. This is identical to using any other .NET logger:

public class OrderService
{
    private readonly ILogger<OrderService> _logger;

    public OrderService(ILogger<OrderService> logger)
    {
        _logger = logger;
    }

    public async Task ProcessOrderAsync(Order order)
    {
        _logger.LogInformation("Processing order {OrderId} for customer {CustomerId}",
            order.OrderId, order.CustomerId);

        try
        {
            // ... processing logic ...

            _logger.LogInformation("Order {OrderId} completed. Total: {Total:C}",
                order.OrderId, order.Total);
        }
        catch (PaymentException ex)
        {
            _logger.LogError(ex, "Payment failed for order {OrderId}. Amount: {Amount:C}",
                order.OrderId, order.Amount);
            throw;
        }
        catch (Exception ex)
        {
            _logger.LogCritical(ex, "Unexpected failure processing order {OrderId}", order.OrderId);
            throw;
        }
    }
}

All of that writes to Azure Table Storage — no other configuration needed. The logger is the provider. You write standard .NET structured log messages; ASCDataAccess handles the persistence.

Getting a logger outside of DI

RemoteLogger provides static access to loggers for code that is not wired through DI — legacy code, static helpers, extension methods:

// By type name (category = "MyNamespace.MyComponent")
ILogger<MyComponent> typed = RemoteLogger.CreateLogger<MyComponent>();

// By explicit category string
ILogger logger = RemoteLogger.CreateLogger("Security.Suspicion");
ILogger logger = RemoteLogger.CreateLogger("BackgroundSync");

// Then use normally
logger.LogWarning("Rate limit exceeded for IP {IpAddress}", ip);
logger.LogError(ex, "Sync failed for tenant {TenantId}", tenantId);

Log level guide

Level	When to use
LogTrace	Verbose diagnostic data — disabled in production
LogDebug	Developer diagnostics — disabled in production
LogInformation	Normal application events: user logged in, order placed
LogWarning	Recoverable issues: retry occurred, rate limit approaching
LogError	Operation failed but application continues: payment declined, API timeout
LogCritical	Application-level failure requiring immediate attention: data corruption, auth service down

Log partitioning

Logs are automatically partitioned by YYYY-MM timestamp bucket. This means:

• Querying logs by date range is fast (PartitionKey scan, not full table scan)
• Retention cleanup deletes by partition — no row-by-row scanning
• High log volumes distribute across monthly partitions automatically

---

18. Structured Error Records — ErrorLogData

ErrorLogData is the specialized API for capturing a rich, structured error record in a dedicated error table, separate from the application log stream. Use it when you need a record that carries business context — customer ID, session ID, company ID, caller info — that your standard ILogger call does not include.

The everyday logging case is _logger.LogError(ex, "message"). Use ErrorLogData when you need a queryable, business-context-enriched error record that a support team or automated system can retrieve by customer or session.

// Structured error with business context — writes to dedicated error table
var error = ErrorLogData.CreateWithCallerInfo(
    "Payment processing failed",
    ErrorCodeTypes.Critical,
    customerID: "CUST-001",
    sessionID:  "sess-abc"
);
await error.LogErrorAsync();  // Uses TableStorage credentials; also runs retention cleanup

// With structured state object and exception
var error = ErrorLogData.CreateWithCallerInfo(
    state:     new { OrderId = "ORD-789", Amount = 599.99, Gateway = "Stripe" },
    exception: ex,
    formatter: (s, ex) => $"Order {s.OrderId} for ${s.Amount} failed via {s.Gateway}: {ex?.Message}",
    severity:  ErrorCodeTypes.Error
);
await error.LogErrorAsync();

// Retention cleanup (also runs automatically after each LogErrorAsync call)
await ErrorLogData.ClearOldDataAsync("accountName", "accountKey", daysOld: 60);
await ErrorLogData.ClearOldDataByType("accountName", "accountKey",
    ErrorCodeTypes.Information, daysOld: 7);

Generic age-based retention — ClearOldDataAsync<T>

The same age-based purge that ErrorLogData has shipped with since v1 is now reusable for any TableEntityBase / ITableExtra entity. The body — GetCollectionAsync(e => e.Timestamp < cutoff) followed by BatchUpdateListAsync(old, TableOperationType.Delete) — runs against whatever entity type you supply. The constraint guarantees compile-time safety (where T : TableEntityBase, ITableExtra, new() — the same constraint DataAccess<T> already enforces) and the method is co-located with the existing non-generic overload so any consumer already calling ErrorLogData.ClearOldDataAsync(...) on a retention schedule can extend the same retention to a new entity by adding one line.

using ASCTableStorage;
using ASCTableStorage.Models;

// Original ErrorLog retention call — unchanged behavior, identical to all prior versions.
// Internally now delegates to the generic overload with T = ErrorLogData.
await ErrorLogData.ClearOldDataAsync(
    TableStorage.AccountName,
    TableStorage.AccountKey,
    daysOld: 60);

// Generic overload — purge any entity by Timestamp age. Same query, same batch delete.
await ErrorLogData.ClearOldDataAsync<MyEntity>(
    TableStorage.AccountName,
    TableStorage.AccountKey,
    daysOld: 90);

// Real-world example: the companion ASCCommonCode library defines a BugReport
// entity for in-chat bug reporting. It uses the same retention contract as the
// error log — wired into the calling app's existing retention schedule without
// duplicating any cleanup code.
await ErrorLogData.ClearOldDataAsync<ASCCommonCode.BugReport>(
    TableStorage.AccountName,
    TableStorage.AccountKey,
    daysOld: 60);

Why the generic method lives on ErrorLogData rather than on DataAccess<T>: ErrorLogData is already the canonical retention surface that consuming apps schedule — most apps already call ErrorLogData.ClearOldDataAsync from a daily timer, hosted service, or Azure Function. Putting the generic overload right next to it means an app extending retention to a new entity adds one line — ErrorLogData.ClearOldDataAsync<NewEntity>(...) — alongside the existing call. There is nothing new to register, no constructor to wire, no service to inject.

Filtering by additional fields: if you need to combine age-based cleanup with another predicate (only delete Completed records older than N days, or only delete records from a specific tenant), write your own BatchUpdateListAsync call against DataAccess<T> directly. The generic method handles only the common case (everything older than N days, full stop). The existing ErrorLogData.ClearOldDataByType method is the canonical example of the more selective pattern — copy its body when you need a similar customized purge.

// Pattern for "selective" age-based retention — copy/adapt ErrorLogData.ClearOldDataByType
public static async Task PurgeCompletedOldAsync<T>(
    string accountName, string accountKey, int daysOld)
    where T : TableEntityBase, ITableExtra, new()
{
    var da  = new DataAccess<T>(accountName, accountKey);
    var old = await da.GetCollectionByFilterAsync(
        $"Status eq 'Completed' and Timestamp lt datetime'{DateTime.UtcNow.AddDays(-daysOld):O}'");
    await da.BatchUpdateListAsync(old, TableOperationType.Delete);
}

ErrorCodeTypes

Value	Description
Information	General informational events
Warning	Non-critical issues requiring attention
Error	Errors that affect functionality
Critical	Severe errors requiring immediate intervention
Unknown	Fallback / unclassified

When to use ILogger vs ErrorLogData

Scenario	Use
Normal application event (user action, process step)	_logger.LogInformation(...)
Recoverable issue (retry, fallback used)	_logger.LogWarning(...)
Operation failure (exception caught and handled)	_logger.LogError(ex, ...)
Application-level failure (service down)	_logger.LogCritical(ex, ...)
Business error needing customer/session context for support	ErrorLogData.CreateWithCallerInfo(...)
Error that a support system queries by customer ID	ErrorLogData.CreateWithCallerInfo(...)

---

19. Advanced: TableOptions, Multi-Account, and Data Migration

// Override table name and PartitionKey property
var da = new DataAccess<Order>(new TableOptions
{
    TableName                = "OrdersArchive",
    PartitionKeyPropertyName = "RegionId"
    // Null values fall back to TableStorage.AccountName / AccountKey
});

// Explicit different account (migrations, multi-tenant)
var da = new DataAccess<Order>(new TableOptions
{
    TableName        = "Orders",
    TableStorageName = "migration-account",
    TableStorageKey  = "migration-key"
});

// Data migration between accounts
var source = new DataAccess<Customer>("source-account", "source-key");
var dest   = new DataAccess<Customer>("dest-account",   "dest-key");
var data   = await source.GetAllTableDataAsync();
await dest.BatchUpdateListAsync(data, TableOperationType.InsertOrReplace);

---

20. Automatic Type Handling

Decimal precision

Stored as long × 10,000. Preserves 4 decimal places exactly. Transparent on read and write.

123.4567m  → stored as 1234567L
0.0001m    → stored as 1L

Large string chunking (> 32 KB)

Split into sequential properties; reassembled on read. Each chunk consumes one of 255 property slots and contributes to the 1 MB entity budget.

MyText        → first 32 KB   (slot 1)
MyText_pt_1   → next 32 KB    (slot 2)
MyText_pt_2   → remainder     (slot 3)

Complex type serialization

Custom classes, List<T>, Dictionary<K,V> — JSON-serialized to a single string column. Limitation: lambda expressions cannot filter inside the serialized JSON. See section 6.

Enum serialization

Stored as string name — human-readable, survives renumbering.

Boolean and DateTime

Booleans parsed from true/false, 1/0, yes/no. DateTimes parsed intelligently from multiple historical formats.

---

21. Complete API Reference

TableStorage static class

Member	Description
TableStorage.Configure(name, key)	Set global credentials — call once in Program.cs
TableStorage.AccountName	Get configured account name
TableStorage.AccountKey	Get configured account key
TableStorage.IsConfigured	Check if Configure has been called

DataAccess<T> — constructors

Signature	Description
DataAccess()	Uses TableStorage.AccountName/Key
DataAccess(string name, string key)	Explicit credentials
DataAccess(TableOptions options)	Full configuration override

DataAccess<T> — CRUD

Method	Description
ManageDataAsync(T, TableOperationType)	Async upsert/delete single typed entity
ManageDataAsync(object, TableOperationType)	Async upsert/delete any object via DynamicEntity
ManageData(T, TableOperationType)	Sync variant
ManageData(object, TableOperationType)	Sync variant

DataAccess<T> — queries

Method	Description
GetRowObjectAsync(string rowKey)	O(1) fetch by RowKey — always prefer
GetRowObjectAsync(Expression<Func<T,bool>>)	Fetch by lambda (hybrid)
GetRowObjectAsync(field, ComparisonTypes, value)	Fetch by field comparison
GetCollectionAsync(string partitionKey)	All rows in a partition — fast scan
GetCollectionAsync(Expression<Func<T,bool>>)	Lambda query (hybrid)
GetCollectionAsync(List<DBQueryItem>, QueryCombineStyle)	Dynamic multi-field query
GetCollectionByFilterAsync(string odataFilter)	Raw OData filter
GetAllTableDataAsync()	Full table scan — use sparingly
All async methods have sync equivalents	Remove Async suffix

DataAccess<T> — batch

Method	Description
BatchUpdateListAsync(List<T>, TableOperationType, IProgress?)	Async batch typed
BatchUpdateListAsync(List<DynamicEntity>, TableOperationType, IProgress?)	Async batch dynamic
BatchUpdateList(List<T>, TableOperationType)	Sync batch typed
BatchUpdateList(List<DynamicEntity>, TableOperationType)	Sync batch dynamic

DataAccess<T> — pagination

Method	Description
GetPagedCollectionAsync(int pageSize, string? token, string? filter)	Page by OData filter
GetPagedCollectionAsync(string partitionKey, int pageSize, string? token)	Page by partition
GetPagedCollectionAsync(Expression<Func<T,bool>>, int pageSize, string? token)	Page by lambda
GetInitialDataLoadAsync(int size, string? filter)	Quick initial load
GetInitialDataLoadAsync(Expression<Func<T,bool>>, int size)	Quick initial load by lambda

TableOperationType

Value	Behavior
InsertOrReplace	Upsert — full replace if exists (default)
InsertOrMerge	Upsert — merge properties if exists
Delete	Delete the entity

ComparisonTypes

Value	Meaning
eq	Equal
ne	Not Equal
gt	Greater Than
ge	Greater Than or Equal
lt	Less Than
le	Less Than or Equal

Extension methods (ASCTableStorage.Common.Extensions)

Method	Scope	Direction
entity.Load(rowKey)	Single	Read sync
entity.Load(lambda)	Single	Read sync
entity.Save(direction?)	Single	Write sync
entity.Delete(direction?)	Single	Delete sync
entity.LoadAsync(rowKey)	Single	Read async
entity.LoadAsync(lambda)	Single	Read async
entity.SaveAsync(direction?)	Single	Write async
entity.DeleteAsync(direction?)	Single	Delete async
collection.Load(lambda)	List	Read sync
collection.Save(direction?)	List	Write sync
collection.LoadAsync(lambda)	List	Read async
collection.SaveAsync(direction?)	List	Write async
collection.DeleteAsync(direction?)	List	Delete async
stateList.Save(direction?)	StateList	Write sync
stateList.SaveAsync(direction?)	StateList	Write async
stateList.DeleteAsync(direction?)	StateList	Delete async
enumerable.ToStateList()	Any IEnumerable	Convert
errorLog.LogErrorAsync()	ErrorLogData	Write async

---

22. Design Patterns, Best Practices, and Anti-Patterns

✅ Best Practices

Map PartitionKey and RowKey to meaningful business properties

// ✅ Semantic clarity — the Azure key model mapped to your domain
public string CompanyId  { get => PartitionKey ?? string.Empty; set => PartitionKey = value; }
public string CustomerId { get => RowKey ?? Guid.NewGuid().ToString();  set => RowKey = value ?? Guid.NewGuid().ToString(); }

Use GetRowObjectAsync(rowKey) whenever you have the key

// ✅ O(1) — fastest operation in the entire library
var customer = await da.GetRowObjectAsync(customerId);

Batch all bulk writes — never loop ManageDataAsync

// ✅ One batch call — auto-chunked at 100 per Azure transaction
await da.BatchUpdateListAsync(largeList);

Move large content to Blob Storage; store the URL

// ✅ Entity stays lean; content is in the right storage tier
public string? HtmlContentUrl { get; set; }  // Points to AzureBlobs

Pre-compute aggregates into summary entities at write time

// ✅ Summary entity updated on each write — instant read performance
public class MonthlySummary : TableEntityBase, ITableExtra
{
    public string CompanyId  { get => PartitionKey ?? string.Empty; set => PartitionKey = value ?? string.Empty; }
    public string YearMonth  { get => RowKey ?? Guid.NewGuid().ToString();  set => RowKey = value ?? Guid.NewGuid().ToString(); }
    public int    OrderCount { get; set; }
    public decimal Revenue   { get; set; }
    public string TableReference => "MonthlySummaries";
    public string GetIDValue()   => YearMonth;
}

Use explicit boolean comparisons in lambda queries

// ✅ Always explicit
await da.GetCollectionAsync(x => x.IsActive == true);

❌ Anti-Patterns

Instantiating DataAccess<T> for simple load/save/delete

// ❌ Unnecessary boilerplate — DataAccess<T> is not needed for simple operations
var da       = new DataAccess<Customer>();
var customer = await da.GetRowObjectAsync("CUST-042");
await da.ManageDataAsync(customer, TableOperationType.InsertOrReplace);

// ✅ Use extension methods — same result, no ceremony
var customer = await new Customer().LoadAsync("CUST-042");
await customer.SaveAsync();

// ❌ Lambda cannot filter inside JSON — full table scan + broken semantics
public string? OrdersJson { get; set; }
await da.GetCollectionAsync(x => x.OrdersJson.Contains("ORD-1"));

// ✅ Use a related entity
public class Order : TableEntityBase, ITableExtra { ... }

String methods in lambdas on large tables

// ❌ Full table scan
await da.GetCollectionAsync(x => x.Name.Contains("Smith"));

// ✅ Normalize at write time; query by equality
await da.GetCollectionAsync(x => x.NameKey == "smith");

Unbounded property growth toward the 255 limit

// ❌ Dynamic tag fields on the entity — hits 255 limit as tags grow
public string? Tag1 { get; set; }
public string? Tag2 { get; set; }
// ... Tag50 { get; set; }

// ✅ Related entity — scales to any number of tags
public class ProductTag : TableEntityBase, ITableExtra { ... }

"all" or a single value as PartitionKey for all records

// ❌ Everything in one partition — single storage node bottleneck
PartitionKey = "all";

// ✅ Meaningful distribution
PartitionKey = companyId;

Reusing a DataAccess<T> instance across different tables

// ❌ One instance targets one table
var da = new DataAccess<Customer>();  // Only queries the Customers table

// ✅ Separate instances per type
var customerDa = new DataAccess<Customer>();
var orderDa    = new DataAccess<Order>();

Loading entire tables for filtered reads

// ❌ Downloads everything, filters in memory
var all    = await da.GetAllTableDataAsync();
var active = all.Where(x => x.IsActive).ToList();

// ✅ OData filter reduces data returned from Azure
var active = await da.GetCollectionAsync(x => x.IsActive == true);

Storing binary or large text as entity properties

// ❌ Drives entity toward the 1 MB limit
public byte[]?  ThumbnailBytes { get; set; }
public string?  HtmlContent   { get; set; }

// ✅ Blob Storage for content; URL on the entity
public string? ThumbnailUrl { get; set; }
public string? ContentUrl   { get; set; }

---

23. Shadow Partitions — Distinct-Value Cataloguing

The problem this solves: Azure Table Storage has no SELECT DISTINCT, no GROUP BY, and no aggregate functions. There is no built-in way to answer "what values exist for this field?" without scanning the entire table. Shadow Partitions solve this — automatically and at zero cost to the primary write path.

What Shadow Partitions are

Every time an entity is written to the database, the library silently catalogues the unique values it observes for each tracked field into a dedicated Azure table called AppShadowPartitions. Each row in that catalog represents one unique (table, field, value) triple that has been observed in the live data. The write is:

• Fire-and-forget — decoupled entirely from the primary write; your caller never waits for it
• Idempotent — a process-lifetime in-memory cache ensures each unique triple is only written to storage once per application run, and a deterministic RowKey (hash of field + value) makes existence checks O(1) point reads rather than partition scans
• Recursion-safe — the shadow catalog tracks your domain entities only; it never catalogues its own rows

The catalog stores, per observed value:

Property	Content
FieldName	The C# property name
FieldValue	The unique value observed
DataType	The C# type name (String, Int32, Boolean, etc.)
FieldPurpose	The XML <summary> documentation comment from the entity property

What gets catalogued

The library tracks fields whose types produce useful categorical values: string, int, long, double, and all enum types. Fields with values longer than 200 characters are skipped (free-text, base64, HTML — not useful as filter options). Guid fields are excluded because their values are almost always unique per entity and serve no purpose as a filter catalog.

The result is a catalog that contains exactly what you need for distinct-style queries: status codes, role names, boolean flags, category identifiers, and the like — with no noise from identifiers or unstructured content.

Reading from the catalog

The catalog is a standard typed entity. Query it directly:

// All catalogued schema for a specific table
var da      = new DataAccess<ShadowPartition>();
var catalog = await da.GetCollectionByFilterAsync($"PartitionKey eq 'AppOrders'");

// All distinct values for a specific field
var statusValues = catalog
    .Where(s => s.FieldName == "Status")
    .Select(s => s.FieldValue)
    .Distinct()
    .OrderBy(v => v)
    .ToList();
// → ["Cancelled", "Completed", "Pending", "Processing", "Shipped"]

// All distinct fields observed in a table (schema discovery)
var knownFields = catalog
    .GroupBy(s => s.FieldName)
    .Select(g => new { Field = g.Key, Type = g.First().DataType, Purpose = g.First().FieldPurpose })
    .OrderBy(f => f.Field)
    .ToList();

The ShadowPartition entity:

public class ShadowPartition : TableEntityBase, ITableExtra
{
    public string  TableName    { get => PartitionKey ?? string.Empty; set => PartitionKey = value ?? string.Empty; }
    public string  ShadowID     { get => RowKey ?? Guid.NewGuid().ToString();  set => RowKey = value ?? Guid.NewGuid().ToString(); }
    public string? FieldName    { get; set; }
    public string? FieldValue   { get; set; }
    public string? DataType     { get; set; }
    public string? FieldPurpose { get; set; }

    public string TableReference => "AppShadowPartitions";
    public string GetIDValue()   => ShadowID ?? string.Empty;
}

---

Use cases

Use case 1: Dynamic filter dropdowns

You are building an admin UI that lets users filter a data table by any column. Without the shadow catalog, populating the "Status" dropdown requires a full table scan, a pre-seeded enum list, or hard-coded values in the UI. With the catalog, it is a single point read:

// Populate a filter dropdown for the Status field in the Orders table
var da      = new DataAccess<ShadowPartition>();
var catalog = await da.GetCollectionByFilterAsync("PartitionKey eq 'AppOrders'");

var statusOptions = catalog
    .Where(s => s.FieldName == "Status")
    .Select(s => s.FieldValue!)
    .OrderBy(v => v)
    .ToList();

// Result: ["Cancelled", "Completed", "Pending", "Processing", "Shipped"]
// No table scan. No hard-coded list. Always accurate as new statuses appear in production.

This pattern works for any categorical field — order types, user roles, region codes, product categories, approval states — anything that has a bounded set of values. The dropdown always reflects what actually exists in the data, not what someone thought would exist when writing the UI.

---

Use case 2: Schema discovery without reading data

In multi-tenant or schema-flexible systems, different tenants may write different field sets. The catalog gives you an always-current view of what fields and values exist for any table without querying the entities themselves:

// Discover what fields the TelemetryEvents table actually contains in production
var da      = new DataAccess<ShadowPartition>();
var catalog = await da.GetCollectionByFilterAsync("PartitionKey eq 'TelemetryEvents'");

foreach (var field in catalog.GroupBy(s => s.FieldName).OrderBy(g => g.Key))
{
    var sample = field.First();
    var valueCount = field.Count();
    Console.WriteLine($"{sample.FieldName} ({sample.DataType}): {valueCount} distinct value(s) — {sample.FieldPurpose}");
}

// Output:
// EventType  (String): 4 distinct value(s) — The type of telemetry event
// IsError    (Boolean): 2 distinct value(s) — Whether the event represents an error condition
// Region     (String): 12 distinct value(s) — Azure region where the event originated
// Severity   (String): 3 distinct value(s) — Event severity level

This is especially useful for data teams onboarding onto an existing system, for generating data dictionaries, and for surfacing what a table actually looks like in production as opposed to what the original schema declared.

---

Use case 3: Data quality and validation monitoring

When a field should have a controlled set of values (an effective enum stored as a string, for example), the catalog makes it trivial to detect values that fall outside the expected set — indicating a bug in a data pipeline, a missing validation rule, or an unhandled code path:

var expectedStatuses = new HashSet<string> { "Pending", "Processing", "Shipped", "Completed", "Cancelled" };

var da      = new DataAccess<ShadowPartition>();
var catalog = await da.GetCollectionByFilterAsync("PartitionKey eq 'AppOrders'");

var unexpectedValues = catalog
    .Where(s => s.FieldName == "Status" && !expectedStatuses.Contains(s.FieldValue ?? ""))
    .Select(s => s.FieldValue)
    .ToList();

if (unexpectedValues.Any())
{
    // Alert — unexpected Status values found in production: ["InProgress", "CANCELLED", "unknown"]
    // → Case mismatch, renamed value, or missing migration detected
}

Running this check as a scheduled diagnostic catches data inconsistencies before they affect downstream logic. It requires no table scan and no sampling — the shadow catalog is already the deduplicated universe of what has actually been observed.

---

Use case 4: AI-assisted and natural-language queries

Systems that let users describe what they want in plain English — rather than building query UI — need the model to know not just what fields exist, but what values are actually in use. The shadow catalog delivers exactly that context:

var da      = new DataAccess<ShadowPartition>();
var catalog = await da.GetCollectionByFilterAsync($"PartitionKey eq '{tableName}'");

// Build schema context for the AI model
var schemaContext = new StringBuilder();
foreach (var g in catalog.GroupBy(s => s.FieldName))
{
    var field  = g.First();
    var values = g.Select(s => s.FieldValue).Where(v => !string.IsNullOrEmpty(v))
                  .Distinct().OrderBy(v => v).ToList();

    schemaContext.Append($"  {field.FieldName} ({field.DataType})");
    if (!string.IsNullOrWhiteSpace(field.FieldPurpose))
        schemaContext.Append($" — {field.FieldPurpose}");
    if (values.Count <= 12)
        schemaContext.Append($" — known values: {string.Join(", ", values.Select(v => $"'{v}'"))}");
    schemaContext.AppendLine();
}

// The model now knows: Status is 'Pending'|'Shipped'|'Completed', not just "a string called Status"
// Result: "Show me all shipped orders from last week" → Status eq 'Shipped' and ShippedDate ge '...'
// Without the catalog: the model guesses "Shipped", "shipped", "SHIPPED", "status_shipped" — any of which fail

The known-values context is the difference between an AI query layer that works reliably and one that generates valid-looking OData that Azure rejects because the value casing is wrong.

---

Use case 5: Audit and compliance reporting

Compliance requirements often call for demonstrating what values a field has taken over time — every user role that has been granted, every approval state a record has passed through, every geographic region where data has been stored. The shadow catalog answers these questions without any bespoke audit infrastructure:

// All distinct user roles that have ever been assigned in the system
var da      = new DataAccess<ShadowPartition>();
var catalog = await da.GetCollectionByFilterAsync("PartitionKey eq 'SoSyncableUsers'");

var allRoles = catalog
    .Where(s => s.FieldName == "UserRole")
    .Select(s => s.FieldValue!)
    .OrderBy(v => v)
    .ToList();
// → ["Admin", "Agency", "Artist", "LicenseHolder", "ReadOnly"]
// Demonstrates to an auditor every role level that has been in active use

Note that the catalog represents values that have been written — not necessarily values that still exist in live entities. If a role was retired and all entities updated, the shadow row persists as a historical record unless explicitly deleted. Whether that is a feature or a limitation depends on your compliance posture.

---

Use case 6: Multi-tenant feature flag and configuration discovery

In a SaaS application where each tenant configures their own settings using dynamic entities or structured config tables, the catalog provides a cross-tenant view of which configuration keys are actually in use — useful for deprecating old keys, validating required settings, or generating tenant health dashboards:

var da      = new DataAccess<ShadowPartition>();
var catalog = await da.GetCollectionByFilterAsync("PartitionKey eq 'TenantConfig'");

// Which PartitionKey values (tenant IDs) have a config entry?
var tenantsWithConfig = catalog
    .Where(s => s.FieldName == "PartitionKey")
    .Select(s => s.FieldValue!)
    .OrderBy(v => v)
    .ToList();

// Which tenants have explicitly enabled the new checkout flow?
var tenantIds = catalog
    .Where(s => s.FieldName == "NewCheckoutEnabled" && s.FieldValue == "True")
    .Select(s => s.TableName)
    .ToList();

---

Performance characteristics

The shadow write path has been designed to add zero latency to your primary writes:

Scenario	Cost
Repeated writes of same entity (hot path)	Zero network calls — process-lifetime in-memory cache hit
First write of a new unique value (cold path)	One O(1) point read + one insert — runs on a background thread
Application restart — already-seen values	One O(1) point read per unique value on first occurrence after restart, then cache hits forever
Delete operations	Cache eviction + one O(1) point read + one delete if the shadow row exists

The existence check uses a deterministic RowKey derived from a hash of the field name and value, so lookups are always GetEntityIfExistsAsync(partitionKey, rowKey) — the fastest operation Azure Table Storage supports. There are no partition scans in the shadow write path.

---

Cleanup after schema changes

If a field is renamed, removed, or its valid values change, stale shadow rows can be deleted directly:

// Remove all shadow entries for a retired field
var da      = new DataAccess<ShadowPartition>();
var catalog = await da.GetCollectionByFilterAsync("PartitionKey eq 'AppOrders'");

var staleRows = catalog.Where(s => s.FieldName == "LegacyStatusCode").ToList();
await da.BatchUpdateListAsync(staleRows, TableOperationType.Delete);

Deleting a domain entity via the normal write path automatically removes its shadow entries — the catalog stays in sync with your live data by design.

---

ASCDataAccess v4.2 — Developed and maintained by Answer Sales Calls Inc. (ASC)
The only production-quality data access framework for Azure Table Storage in the .NET ecosystem.

---

Appendix: Migrating from ASCDataAccess v4.0 to v4.1

v4.1 is fully backward-compatible with v4.0. No breaking changes. All changes are additive:

1. TableStorage.Configure(name, key) — new static class for one-time credential registration
2. No-arg constructors on DataAccess<T>, AzureBlobs, Session, and QueueData<T>
3. TableOptions falls back to TableStorage credentials when name/key are null
4. QueueData<T> static methods have credential-free overloads
5. Session.CreateAsync(sessionId) and Session(sessionId) — new no-credential overloads

Optional migration steps (v4.0 code continues to work unchanged)

1. Add TableStorage.Configure(name, key) to Program.cs
2. Replace new DataAccess<T>(accountName, accountKey) → new DataAccess<T>()
3. Replace new AzureBlobs(name, key, container) → new AzureBlobs(container)
4. Replace new Session(name, key) → new Session()
5. Replace new Session(name, key, id) → new Session(id)

---

Appendix: Migrating from ASCDataAccess v4.1 to v4.2

v4.2 is fully backward-compatible with v4.1. No breaking changes. All changes are additive:

1. Shadow Partition cataloguing — automatic, fire-and-forget. Begins the moment you install v4.2 and make any entity write. No configuration required. A new AppShadowPartitions table is created in your Azure Storage account on first write. See section 23.
2. ShadowPartition entity — new public model in ASCTableStorage.Models. Use it to query the catalog directly via DataAccess<ShadowPartition>. See section 23.
3. GetRowObjectAsync(partitionKey, rowKey) overload — new O(1) point-read overload added to DataAccess<T>. Can be used in your own code wherever you know both keys.
4. AzureBlobs(containerName, PublicAccessType) constructors + EnsureContainerAccessAsync(PublicAccessType) method. Two new constructor overloads on AzureBlobs accept a PublicAccessType parameter so the container is dynamically created with anonymous-read-blob access on first use — required by features that need URLs readable by external clients (downstream AI agents, public marketing assets, share-link previews). Existing constructors default to PublicAccessType.None (private) so call sites that do not opt in are unaffected. See section 14.
5. ErrorLogData.ClearOldDataAsync<T>(...) — generic age-based retention. The original ErrorLogData.ClearOldDataAsync(name, key, daysOld) method has been refactored to a one-line wrapper around a new generic overload. Any entity that inherits from TableEntityBase and implements ITableExtra can now be purged on the same age-based contract — without duplicating the query/delete code in every consumer. The non-generic call sites in existing apps continue to compile and behave identically. See section 18.

Enhanced QueryExtensions (cumulative through v4.2)

The lambda expression engine in section 8 has been progressively expanded over the v4.x line. As of v4.2 the full set of server-side and hybrid-translatable operators available inside any LoadAsync / GetCollectionAsync lambda is:

• Equality / comparison / logical: ==, !=, >, >=, <, <=, &&, || — fully server-side
• String prefix: .StartsWith("...") — translated to an OData ge / lt range
• String null/empty: string.IsNullOrEmpty(x.Field) and x.Field.IsNotNullOrEmpty() — server-side
• Membership: x.Field.In(...) (params or IEnumerable<T>) — translates to an OR-equality chain
• Range: x.Field.Between(low, high) — inclusive, works with any IComparable<T>
• Date windows (calendar): IsToday, IsThisWeek, IsThisMonth, IsThisYear — bounds baked at query-build time
• Date windows (explicit): InDateRange(start, end) — [start, end) half-open range
• Bitwise flags: HasAnyFlag(bits) — client-side evaluation after server narrowing

All operators except HasAnyFlag are translated to OData and execute on Azure before any data is returned. HasAnyFlag is the lone explicit client-side helper — pair it with a server-side filter that narrows row count first. See section 8 for the full operator reference with example code.

Optional steps after upgrading

• The AppShadowPartitions table will be populated progressively as entities are written. No back-fill needed — the catalog builds itself organically from normal write traffic.
• If you want to pre-populate the catalog for an existing table, iterate your entities once and call SaveAsync() or ManageDataAsync(). The shadow path will catalogue them automatically.
• Review the 200-character value cap and the string/int/long/double/enum tracked-type list in section 23 and confirm they align with your schema. Both are configurable via the ShadowMaxValueLength constant and _shadowTrackedTypes set in DataAccess<T> if you need to adjust them.
• For any new blob container that needs anonymous-read URLs (chatbot attachments, public marketing assets), pass PublicAccessType.Blob to the AzureBlobs constructor — first call creates the container with the right access level automatically.
• To extend retention cleanup to a new entity, schedule ErrorLogData.ClearOldDataAsync<MyEntity>(...) alongside the existing ErrorLogData.ClearOldDataAsync(...) call in your retention cron / hosted service. Same call site, same interval, one extra line per entity.

The only production-quality data access framework for Azure Table Storage in the .NET ecosystem.