Codelightly.IntegrationEngine 1.0.0

Codelightly Integration Engine

Package Information

<PackageId>Codelightly.IntegrationEngine</PackageId>
<Version>1.0.0</Version>
<Authors>Sidney Jones</Authors>
<Company>BPM Software Solutions</Company>
<Description>Core services and interfaces for entity resolution, mapping, and profile execution in BPM's Codelightly Integration Engine.</Description>
<PackageTags>integration;data-mapping;entity-resolution;BPM;automation</PackageTags>
<PackageLicenseExpression>MIT</PackageLicenseExpression>
<RepositoryUrl>https://github.com/bpmcloudsolutions/BPM.IntegrationEngine</RepositoryUrl>
<PackageProjectUrl>https://bpmcloudsolutions.com</PackageProjectUrl>

Overview

This system enables dynamic, declarative mapping from input data (e.g., flat files, DTOs) into rich entity models using a JSON configuration file. It decouples the logic of resolving references, creating missing data (e.g., companies, pay periods), and assigning values from the application logic.

Key Components

FieldMappingEngine

• Reads a JSON mapping profile.
• Uses reflection to extract fields from the input object.
• Resolves each mapped field using either:
  • A literal
  • Runtime context
  • Source field value
  • A foreign entity via EntityResolverService
• Applies optional transformations like trim, uppercase, and type coercion (int, datetime, etc.).
• Instantiates and populates a target object of any type TTarget.

EntityResolverService

• Resolves foreign key relationships dynamically via repository methods.
• Supports GetByXAsync, GetByXAndYAsync, or explicitly defined repository method names.
• Can auto-create missing records if createIfNotFound = true in the JSON.
• Allows specifying how new records should be created (createWith block).
• Dynamically determines repository and model types via namespace strings.

EntityPersistenceService

• Enables JSON-driven mapping and persistence of entities from DTOs to database.
• Determines if entities should be created or updated based on configuration.
• Handles property exclusions during updates.
• Applies pre-save transformations to entities.
• Uses the appropriate repository methods for persistence.

IntegrationProfileRunner

• Orchestrates the entire integration process for batches of data.
• Coordinates mapping and persistence operations.
• Provides detailed results including success/failure information.
• Supports conditional processing and error handling.

JSON Mapping Structure

Each field in the mapping file must include:

{
  "targetField": "TargetPropertyName",
  "value": "$SourceFieldName"
}

Or for foreign key resolution:

{
  "targetField": "CompanyId",
  "value": {
    "entity": "Company",
    "keyFields": [
      { "field": "CompanyCode", "source": "$CompanyCode" }
    ],
    "repositoryMethod": "GetByCompanyCodeAsync",
    "outputField": "CompanyId",
    "createIfNotFound": true,
    "createWith": {
      "CompanyCode": "$CompanyCode",
      "Name": "$WorkLocation",
      "EmployerId": "[EmployerId]",
      "IsActive": "'true'"
    }
  }
}

Token Types

Transformation Support

Each field can define a format object:

"format": {
  "trim": true,
  "uppercase": true,
  "type": "datetime"
}

Installation

dotnet add package Codelightly.IntegrationEngine

Example Use

Using the Field Mapping Engine Directly

var inputRow = FieldMappingEngine.BuildInputRow(employeeDto);
var runtimeContext = new DefaultRuntimeContext
{
    ["EmployerId"] = employer.EmployerId,
    ["NowUtc"] = DateTime.UtcNow
};

var mappingJson = JsonDocument.Parse(File.ReadAllText("PayPeriodEmployee.mapping.json")).RootElement;
var payPeriodEmployee = await _fieldMappingEngine.MapRowToEntityAsync<PayPeriodEmployee>(inputRow, mappingJson, runtimeContext);

Using the Integration Profile Runner

// Create a runtime context with any variables needed during mapping
var runtimeContext = new DefaultRuntimeContext
{
    ["EmployerId"] = employer.EmployerId,
    ["NowUtc"] = DateTime.UtcNow
};

// Execute an integration profile with automatic persistence
var results = await _integrationProfileRunner.ExecuteProfileAsync<EmployeeDto, Employee>(
    "EmployeeIntegration",
    employeeDtos,
    runtimeContext);

// Process the results
foreach (var result in results)
{
    if (result.Success)
    {
        Console.WriteLine($"Successfully processed employee: {result.Entity.FirstName} {result.Entity.LastName}");
        Console.WriteLine($"Entity was {(result.IsNewlyCreated ? "created" : "updated")}");
    }
    else
    {
        Console.WriteLine($"Error processing employee: {result.ErrorMessage}");
    }
}

Extensibility

• Add validation blocks per field (required, minLength, regex, etc.)
• Add custom transformation strategies (e.g., date format parsing, default values)
• Register repository/model namespaces dynamically via config or dependency injection
• Implement custom persistence strategies for specific entity types

Additional Features

File Integration

The FileIntegrationProcessor provides capabilities for processing data from various file formats:

• CSV, Excel, and JSON file support
• Automatic header detection and mapping
• Batch processing with configurable chunk sizes
• Detailed error reporting and validation

API Integration

The ApiIntegrationProcessor enables integration with external APIs:

• Send and receive data from RESTful endpoints
• Configure authentication and headers via JSON
• Transform data before sending and after receiving
• Handle pagination and rate limiting

Authors

• Engine Design: Sidney Jones
• Integration Pattern: BPM Software Solutions

For questions or extending this tool, contact your BPM architect or visit the documentation hub at https://bpmcloudsolutions.com

Appendix:

This section contains extensive examples to facilitate unit testing.

Mapping File example: PayPeriodEmployee.mapping.json

[
  {
    "targetField": "CompanyId",
    "value": {
      "entity": "Company",
      "keyFields": [
        { "field": "CompanyCode", "source": "$CompanyCode" }
      ],
      "repositoryMethod": "GetByCompanyCodeAsync",
      "outputField": "CompanyId",
      "createIfNotFound": true,
      "createWith": {
        "Name": "$WorkLocation",
        "CompanyCode": "$CompanyCode",
        "EmployerId": "[EmployerId]",
        "IsActive": "'true'"
      }
    }
  },
  {
    "targetField": "PayPeriodId",
    "value": {
      "entity": "PayPeriod",
      "keyFields": [
        { "field": "EmployerId", "source": "[EmployerId]" },
        { "field": "PayDate", "source": "$PayDate" }
      ],
      "repositoryMethod": "GetByEmployerIdAndPayDateAsync",
      "outputField": "PayPeriodId",
      "createIfNotFound": true,
      "createWith": {
        "EmployerId": "[EmployerId]",
        "PayDate": "$PayDate",
        "StartDate": "$PayrollPeriodStartDate",
        "EndDate": "$PayrollPeriodEndDate",
        "Status": "'Pending'",
        "DateGenerated": "[NowUtc]",
        "GeneratedBy": "'API'"
      }
    }
  },
  { "targetField": "CompanyCode", "value": "$CompanyCode" },
  { "targetField": "PayrollEmployeeId", "value": "$PayrollEmployeeId" },
  { "targetField": "FirstName", "value": "$FirstName" },
  { "targetField": "LastName", "value": "$LastName" },
  {
    "targetField": "BirthDate",
    "value": "$BirthDate",
    "format": { "type": "datetime" }
  },
  { "targetField": "WorkEmail", "value": "$WorkEmail" },
  { "targetField": "WorkLocation", "value": "$WorkLocation" },
  { "targetField": "EmploymentType", "value": "$EmploymentType" },
  { "targetField": "Address1Address2", "value": "$Address1Address2" },
  { "targetField": "City", "value": "$City" },
  { "targetField": "State", "value": "$State" },
  { "targetField": "Country", "value": "$Country" },
  { "targetField": "PayrollFrequency", "value": "$PayrollFrequency" },
  {
    "targetField": "HireDate",
    "value": "$HireDate",
    "format": { "type": "datetime" }
  },
  {
    "targetField": "TerminationDate",
    "value": "$TerminationDate",
    "format": { "type": "datetime" }
  },
  { "targetField": "Status", "value": "$Status" },
  {
    "targetField": "PayDate",
    "value": "$PayDate",
    "format": { "type": "datetime" }
  },
  {
    "targetField": "PayrollPeriodStartDate",
    "value": "$PayrollPeriodStartDate",
    "format": { "type": "datetime" }
  },
  {
    "targetField": "PayrollPeriodEndDate",
    "value": "$PayrollPeriodEndDate",
    "format": { "type": "datetime" }
  },
  {
    "targetField": "BasePayAmount",
    "value": "$BasePayAmount",
    "format": { "type": "decimal" }
  },
  { "targetField": "CreatedBy", "value": "'API'" },
  { "targetField": "CreatedDate", "value": "[NowUtc]" },
  { "targetField": "LastModifiedBy", "value": "'API'" },
  { "targetField": "LastModifiedDate", "value": "[NowUtc]" }
]