BetterCommerce.Webhooks 1.0.0

BetterCommerce.Webhooks

A robust, scalable webhook and outbox pattern implementation for microservices with multi-tenant support, OAuth 2.0 authentication, per-service decentralized storage, templating, retry logic, and comprehensive observability.

🚀 Key Features

• Transactional Outbox Pattern: Ensures reliable event publishing with ACID guarantees
• Multi-Tenant Support: Full OrgId and DomainId isolation for all webhook operations
• OAuth 2.0 Authentication: Complete OAuth client credentials flow with token caching
• Decentralized Architecture: Each microservice owns its webhook schema and data
• Flexible Payload Templating: Liquid templating engine for custom webhook payloads
• Robust Retry Logic: Configurable exponential backoff with dead letter queue
• Multiple Authentication Methods: OAuth 2.0, HMAC, Bearer, Basic, API Key support
• Comprehensive Observability: OpenTelemetry metrics and distributed tracing
• Schema Migration System: Automatic database schema management with versioning
• High Performance: PostgreSQL with SKIP LOCKED for concurrent processing
• Background Processing: Built-in hosted service for automatic webhook dispatch

📦 Solution Structure

BetterCommerce.Webhooks/
├── src/
│   └── BetterCommerce.Webhooks.Core/         # Core library (NuGet package)
│       ├── Domain/                           # Domain entities and value objects
│       ├── Infrastructure/                   # Database context and migrations
│       ├── Services/                         # Business logic services
│       ├── Workers/                          # Background webhook dispatcher
│       └── Extensions/                       # Service collection extensions
├── samples/
│   ├── BetterCommerce.Webhooks.Api/          # REST API for webhook management
│   └── BetterCommerce.Webhooks.ConsoleDemo/  # Console demo application
└── tests/
    └── BetterCommerce.Webhooks.Tests/        # Unit and integration tests

🔧 Installation & Setup

1. Add NuGet Package

dotnet add package BetterCommerce.Webhooks.Core

2. Configure in Program.cs

using BetterCommerce.Webhooks.Core.Extensions;

var builder = WebApplication.CreateBuilder(args);

// Configure webhook services
var webhookConfig = new WebhookConfiguration
{
    ConnectionString = builder.Configuration["ConnectionStrings:Database"],
    SchemaName = "commerce_webhooks",  // Per-service schema
    ServiceName = "Commerce.Api",
    EnableBackgroundWorker = true,
    PollingIntervalSeconds = 5,
    BatchSize = 10,
    MaxConcurrency = 5
};

builder.Services.AddBetterCommerceWebhooks(webhookConfig);

// Add user context for multi-tenancy
builder.Services.AddHttpContextAccessor();
builder.Services.AddScoped<IUserContext, UserContext>();

var app = builder.Build();

// Initialize database schema
await app.Services.InitializeWebhooksAsync();

3. Configure Multi-Service Support

In appsettings.json:

{
  "ConnectionStrings": {
    "Database": "Host=localhost;Database=webhooks;Username=postgres;Password=postgres"
  },
  "ConfigService": {
    "Microservices": "commerce:commerce_webhooks;pim:pim_webhooks;inventory:inventory_webhooks"
  },
  "Webhooks": {
    "SchemaName": "commerce_webhooks",
    "ServiceName": "Commerce.Api",
    "EnableBackgroundWorker": true,
    "PollingIntervalSeconds": 5,
    "BatchSize": 10,
    "MaxConcurrency": 5
  }
}

💻 Usage Examples

Publishing Events

public class OrderService
{
    private readonly IOutboxService _outboxService;
    
    public async Task<Order> CreateOrderAsync(CreateOrderRequest request)
    {
        // Create order in database
        var order = new Order { ... };
        await _dbContext.SaveChangesAsync();
        
        // Publish webhook event (guaranteed delivery via outbox pattern)
        await _outboxService.PublishEventAsync(
            entityName: "Order",
            eventName: "OrderCreated",
            entityPk: order.Id,
            payload: order,
            orgId: order.OrgId,
            domainId: order.DomainId
        );
        
        return order;
    }
}

Creating Webhook Endpoints

POST /api/webhooks/commerce/endpoints
Authorization: Bearer {token}

{
  "name": "Shopify Order Sync",
  "targetUrl": "https://shop.mystore.com/webhooks/orders",
  "httpMethod": "POST",
  "authType": "OAuth2",
  "oauthTokenUrl": "https://shop.mystore.com/oauth/token",
  "oauthClientId": "client123",
  "oauthClientSecret": "secret456",
  "retryPolicy": {
    "maxAttempts": 5,
    "backoffMultiplier": 2,
    "maxDelaySeconds": 3600
  }
}

Creating Webhook Subscriptions

POST /api/webhooks/commerce/subscriptions
Authorization: Bearer {token}

{
  "endpointId": "550e8400-e29b-41d4-a716-446655440000",
  "entityName": "Order",
  "eventName": "OrderCreated",
  "payloadTemplate": "{\"OrderNo\":\"{{CustomNo}}\",\"Status\":\"{{Status}}\",\"Total\":\"{{GrandTotal}}\",\"Customer\":\"{{CreatedBy}}\"}",
  "isActive": true
}

🔄 Webhook Processing Flow

1. Event Publishing: Business logic publishes events via IOutboxService
2. Outbox Storage: Events stored in outbox_messages table with status "Pending"
3. Background Processing: WebhookDispatcherHostedService polls for pending messages
4. Template Rendering: Liquid templates transform payload using actual data
5. Authentication: OAuth tokens acquired/refreshed, signatures generated
6. HTTP Delivery: Webhook sent to target endpoint with retry logic
7. Status Updates: Message marked as Delivered, Failed, or Dead

📊 Database Schema

Each microservice gets its own schema with these tables:

• {schema}.webhook_endpoints - Target URLs and authentication
• {schema}.webhook_subscriptions - Event to endpoint mappings
• {schema}.outbox_messages - Pending/sent webhook messages
• {schema}.webhook_attempts - Delivery attempt history
• {schema}.oauth_tokens - Cached OAuth tokens

🔐 Authentication Methods

OAuth 2.0

{
  "authType": "OAuth2",
  "oauthTokenUrl": "https://api.example.com/oauth/token",
  "oauthClientId": "client_id",
  "oauthClientSecret": "client_secret",
  "oauthScope": "webhooks:write"
}

HMAC Signature

{
  "authType": "HMAC",
  "secret": "shared_secret_key"
}

Bearer Token

{
  "authType": "Bearer",
  "secret": "token123"
}

🎨 Liquid Template Examples

Simple Property Mapping

{
  "orderId": "{{Id}}",
  "orderNo": "{{CustomNo}}",
  "status": "{{Status}}",
  "total": {{GrandTotal}},
  "customer": "{{CreatedBy}}"
}

Conditional Logic

{
  "order": "{{CustomNo}}",
  "status": "{% if Status == 9 %}Dispatched{% else %}Processing{% endif %}",
  "priority": {% if GrandTotal > 1000 %}true{% else %}false{% endif %}
}

Date Formatting

{
  "orderDate": "{{OrderDate | date: 'yyyy-MM-dd'}}",
  "timestamp": {{timestamp}},
  "processedAt": "{{now | date: 'o'}}"
}

🚦 Health Checks & Monitoring

Health Check Endpoint

GET /health

Metrics Endpoint (Prometheus)

GET /metrics

OpenTelemetry Metrics

• webhooks_published_total - Total webhooks published
• webhooks_delivered_total - Total webhooks delivered
• webhooks_failed_total - Total webhook failures
• webhook_delivery_duration - Delivery duration histogram

🔧 Troubleshooting

Common Issues and Solutions

1. PostgreSQL Enum Type Errors

Error: column "status" is of type webhook_status but expression is of type text

Solution: The system uses explicit PostgreSQL enum casting (::schema.webhook_status) in all queries. Ensure:

• Database migrations have created the enum types
• All enum values in C# match PostgreSQL (case-insensitive)
• The system handles: Pending, Delivered, Failed, Dead (Note: InProgress removed from DB operations)

2. Connection Management Issues

Error: Connection property has not been initialized
OR
Error: A command is already in progress

Solutions:

• Each database operation creates its own connection when not in a transaction
• For transactional operations, connection is properly scoped
• Ensure connection string has sufficient pool size:

Host=localhost;Database=webhooks;Username=postgres;Password=postgres;Pooling=true;Min Pool Size=5;Max Pool Size=100

3. Template Rendering & Data Binding

Error: Template variable '{{CustomNo}}' not found or data not binding

Solution: The PayloadRenderer now:

• Adds all root-level JSON properties directly to Liquid context
• Supports both {{propertyName}} and {{data.propertyName}} syntax
• Provides special variables: {{now}}, {{timestamp}}, {{RecordId}} (alias for Id)
• Example working template:

{
  "OrderNo": "{{CustomNo}}",
  "OrderId": "{{Id}}",
  "RecordId": "{{RecordId}}",
  "Status": "{{Status}}",
  "Total": {{GrandTotal}}
}

4. SaveChangesAsync Not Completing

Issue: Code execution stops after SaveChangesAsync without errors

Solution: Check for:

• Enum casting errors in SQL queries (fixed with explicit casting)
• Transaction scope issues (each operation now properly manages connections)
• Enable detailed logging to see actual SQL errors

5. Background Worker Processing Failures

Error: invalid input value for enum webhook_status: 'inprogress'

Solution:

• The InProgress status has been removed from background processing
• Messages remain in Pending status during processing
• Only transitions to Delivered, Failed, or Dead after processing

6. Enum Parsing Case Sensitivity

Error: Requested value 'pending' was not found

Solution: All enum parsing now uses case-insensitive matching:

Enum.Parse<WebhookStatus>(value, ignoreCase: true)

7. OAuth Token Expiration

The system automatically refreshes OAuth tokens. Check logs for token refresh failures.

📝 API Reference

Endpoints API

• GET /api/webhooks/{service}/endpoints - List endpoints
• POST /api/webhooks/{service}/endpoints - Create endpoint
• PUT /api/webhooks/{service}/endpoints/{id} - Update endpoint
• DELETE /api/webhooks/{service}/endpoints/{id} - Delete endpoint

Subscriptions API

• GET /api/webhooks/{service}/subscriptions - List subscriptions
• POST /api/webhooks/{service}/subscriptions - Create subscription
• PUT /api/webhooks/{service}/subscriptions/{id} - Update subscription
• DELETE /api/webhooks/{service}/subscriptions/{id} - Delete subscription

Messages API

• GET /api/webhooks/{service}/messages - List messages
• GET /api/webhooks/{service}/messages/{id}/attempts - Get delivery attempts
• POST /api/webhooks/{service}/messages/{id}/retry - Retry failed message

🏗️ Architecture Decisions

1. Per-Service Schemas: Each microservice owns its webhook data in a separate schema
2. Outbox Pattern: Guarantees at-least-once delivery with transactional consistency
3. SKIP LOCKED: PostgreSQL feature for efficient concurrent processing
4. Liquid Templates: Flexible payload transformation without code changes
5. Token Caching: Reduces OAuth token requests with in-memory cache
6. Exponential Backoff: Prevents overwhelming failed endpoints

📋 Implementation Notes

Database Connection Management

• Isolated Connections: Each database operation creates its own connection when not in a transaction
• Transaction Scoping: Proper connection lifecycle management within transactions
• Concurrent Operations: Supports high concurrency without connection conflicts
• Connection Pooling: Leverages Npgsql connection pooling for performance

Enum Handling

• PostgreSQL Enums: Custom enum types per schema (e.g., commerce_webhooks.webhook_status)
• Explicit Casting: All enum parameters use ::schema.enum_type casting in SQL
• Case-Insensitive Parsing: Handles PostgreSQL lowercase returns (pending) to C# PascalCase (Pending)
• Status Values: Pending, Delivered, Failed, Dead (no InProgress in DB)

Template Rendering Engine

• Direct Property Access: All root JSON properties available as {{PropertyName}}
• Nested Access: Supports {{data.PropertyName}} for nested objects
• Built-in Variables:
  • {{now}} - Current UTC DateTime
  • {{timestamp}} - Unix timestamp
  • {{RecordId}} - Alias for {{Id}}
• Fluid Parser: Uses Fluid library for Liquid template processing

Multi-Tenant Isolation

• Context-Based Filtering: All queries filtered by OrgId/DomainId from UserContext
• Background Worker: Processes messages across all tenants (configurable)
• Schema Isolation: Each service has isolated schema preventing cross-service access

Error Recovery

• Transient Failures: Automatic retry with exponential backoff
• Dead Letter Queue: Messages marked as Dead after max attempts
• Detailed Logging: Comprehensive error logging for troubleshooting
• Attempt History: Full audit trail of delivery attempts

📈 Performance Considerations

• Batch Processing: Process multiple webhooks concurrently (configurable batch size)
• Connection Pooling: Efficient database connection management
• Async/Await: Non-blocking I/O throughout the pipeline
• SKIP LOCKED: Prevents lock contention in high-throughput scenarios
• Index Strategy: Proper indexes on status, next_attempt_at columns

🔒 Security

• Multi-Tenant Isolation: All queries filtered by OrgId/DomainId
• Secret Encryption: Sensitive data encrypted at rest
• HMAC Verification: Request signatures with timestamps
• OAuth Scopes: Fine-grained permission control
• TLS/HTTPS: Enforced for all webhook deliveries

📚 Additional Documentation

• BACKGROUND_WORKER_GUIDE.md - Detailed worker configuration
• TESTING_AND_RUNNING_GUIDE.md - Testing strategies
• SERVICE_CONFIG.md - Multi-service configuration
• USAGE_GUIDE.md - Detailed usage examples

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add/update tests
5. Submit a pull request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🆘 Support

For issues and questions:

• Create an issue in the repository
• Contact the development team
• Check the troubleshooting guide above

Version: 1.0.1
Last Updated: December 2024
Maintained By: BetterCommerce Development Team

Recent Updates (v1.0.1)

• Fixed connection management for concurrent operations
• Improved PostgreSQL enum handling with explicit casting
• Enhanced Liquid template data binding with direct property access
• Resolved SaveChangesAsync transaction issues
• Removed InProgress status from background worker
• Added case-insensitive enum parsing
• Comprehensive error handling improvements