WillPower.FileParser 1.2025.6.5

WillPower.IO.FileParser

A library intended to facilitate quickly parsing and loading data files for export (ETL/ELT) or consumption. Most commonly used to parse binary EBCDIC files that include packed variable types or other "strange" data. As this assembnly utilizes a common IOC pattern, it can easily be overridden at any level for customization by implementing the relevent interface and base class in a custom class instead of using a default instance type.

Abstract Classes

• FileParserBase : A common base class for a file parsing engine. Must be inherited.

• FileRecordBase : A common base class for record-level data, their fields, and any possible Exceptions. Must be inherited.

• FileFieldBase : A common base class containing the necessary properties and methods for processing a single field. Must be inherited.

Interfaces

• IFileParser : A common Interface for a file parsing engine.

• IFileLayout : The layout of the file, including any header, footer, or conditional rows. For binary files the RecordLength must be greater than 0. For text files the default TextLineTerminator is typically '\n' (nextline), but can be set at any time.

• IFileParserEncoder : An Interface for containing the necessary Encodings to parse a file (Source and Destination).

• IFileRecord : A common interface for record-level data, their fields, and any possible errors.

• IFileConditional : An interface to encapsulate expected functionality (rules) to determine an action.

• IFileField : A common Interface containing the necessary properties and methods for processing a single value.

Instance Classes

• AbortableTask : A simple container for handling Task cancellation (maintains its own CancellationTokenSource). Can be inherited.

• FieldException : An object for containing exception data occurring during field read. Can be inherited.

• FileConditional : A concrete class to encapsulate expected functionality (a rule) to determine an action. Cannot be inherited.

• FileField : A concrete class containing the necessary properties and methods for processing a single field. Cannot be inherited.

• FileTable : A table field that functions more or less like a miniature file reader within a record/field. Used for 3D files. Can be inherited.

• FileRecord : A concrete class for record-level data, their fields, and any possible Exceptions. Cannot be inherited.

• FileLayout : The layout of the file, including any header, footer, or conditional rows. See IFileLayout. Cannot be inherited.

• FileParserEncoder : A concrete class for containing the necessary Encoding properties to parse a file. Cannot be inherited.

• FileParser : A concrete class for file parsing. Cannot be inherited.

• TaskManager : A class for managing active tasks without any complex magic. Can be dialed up or down and inherited.

• SerializableField : A class that can be converted into a FileField while retaining serializability.

• SerializableDecision : A container for serializing in and out of a Conditional.

• SerializableCondition : A serializable FileConditional.

• SerializableLayout : The master class for serializing file layouts.

Sample Code

        public List<Entities.Account> Read3DFile()
        {
            if (!File.Exists("MyFile.dat"))
            {
                Assert.Fail($"Test file ({Layouts.TEST3DFILE}) doesn't exist. Run the write test first.");
                return;
            }
            IFileParser parser = new FileParser(Get3DLayout(false));
            parser.LoadFile("MyFile.dat");
            List<Entities.Account> accounts = [];
            foreach (IFileRecord record in parser.Records)
            {
                string acctNum = record.Get<string>("Account Number").Trim();
                if (accounts.Any(x => x.Account_Number == acctNum))
                {
                    Entities.Account ac = accounts.First(x => x.Account_Number == acctNum);
                    ac.AddData(record);
                }
                else
                {
                    accounts.Add(new Entities.Account(record, true));
                }
            }
            return accounts;
        }

        static FileLayout Get3DLayout(bool write) => new(FileParserEncoder.GetDefaultEncoder(write))
        {
            FillByte = 0x00, // write only
            RecordLength = 80,
            Name = "Binary EBCDIC 3D File",
            HeaderRecord = Header(write),
            FooterRecord = Footer(write),
            MasterFields =
                [
                    new FileField(write)
                    {
                        DataFormat = FileFieldDataFormat.String,
                        FillCharacter = ' ', // write only
                        Type = typeof(string),
                        StartPosition = 0,
                        Length = 10,
                        Name = "Account Number"
                    },
                    new FileField(write)
                    {
                        DataFormat = FileFieldDataFormat.String,
                        FillCharacter = ' ', // write only
                        Type = typeof(string),
                        StartPosition = 10,
                        Length = 1,
                        Name = "RecordType"
                    },
                ],
            Conditions =
                [
                    new FileConditional(x => x.Get<string>("RecordType") == "A")
                    {
                        Name = "Master Record",
                        Fields =
                        [
                            new FileField(write)
                            {
                                DataFormat = FileFieldDataFormat.IBMPacked,
                                FillByte = 0x00, // write only
                                Precision = 2,
                                Type = typeof(decimal),
                                StartPosition = 11,
                                Length = 8,
                                Name = "Balance"
                            },
                            new FileField(write)
                            {
                                DataFormat = FileFieldDataFormat.String,
                                FillCharacter = ' ', // write only
                                Type = typeof(string),
                                StartPosition = 19,
                                Length = 3,
                                Name = "Account Type"
                            },
                            new FileField(write)
                            {
                                DataFormat = FileFieldDataFormat.IBMPacked,
                                FillByte = 0x00, // write only
                                Precision = 2,
                                Type = typeof(decimal?),
                                StartPosition = 22,
                                Length = 6,
                                Name = "Payment Amount"
                            },
                            new FileField(write)
                            {
                                DataFormat = FileFieldDataFormat.String,
                                FillCharacter = ' ', // write only
                                Type = typeof(DateTime),
                                StartPosition = 30,
                                Length = 6,
                                YearFirst = true,
                                Name = "File Date"
                            },
                            new FileField(write)
                            {
                                DataFormat = FileFieldDataFormat.String,
                                FillCharacter = ' ', // write only
                                Type = typeof(string),
                                StartPosition = 36,
                                Length = 40,
                                Name = "Annotation"
                            },
                            new FileField(write)
                            {
                                DataFormat = FileFieldDataFormat.IBMPacked,
                                FillCharacter = ' ', // write only
                                Type = typeof(DateTime?),
                                StartPosition = 76,
                                Length = 4,
                                Name = "Next Payment Due"
                            },
                        ]
                    },
                    new FileConditional(x => x.Get<string>("RecordType") == "B")
                    {
                        Name = "Interest Accrual",
                        Fields =
                        [
                            new FileField(write)
                            {
                                DataFormat = FileFieldDataFormat.IBMPacked,
                                FillByte = 0x00, // write only
                                Type = typeof(decimal),
                                Precision = 3,
                                StartPosition = 11,
                                Length = 6,
                                Name = "Rate"
                            },
                            new FileField(write)
                            {
                                DataFormat = FileFieldDataFormat.IBMPacked,
                                FillByte = 0x00, // write only
                                Type = typeof(decimal),
                                Precision = 2,
                                StartPosition = 26,
                                Length = 6,
                                Name = "Amount per Annum"
                            },
                            new FileField(write)
                            {
                                DataFormat = FileFieldDataFormat.IBMPacked,
                                FillByte = 0x00, // write only
                                Type = typeof(decimal),
                                Precision = 2,
                                StartPosition = 32,
                                Length = 6,
                                Name = "Amount per Month"
                            },
                            new FileField(write)
                            {
                                DataFormat = FileFieldDataFormat.IBMPacked,
                                FillByte = 0x00, // write only
                                Type = typeof(DateTime?),
                                StartPosition = 40,
                                Length = 2,
                                Name = "Next Accrual"
                            },
                            new FileField(write)
                            {
                                DataFormat = FileFieldDataFormat.String,
                                FillCharacter = ' ', // write only
                                Type = typeof(string),
                                StartPosition = 42,
                                Length = 30,
                                Name = "Remarks"
                            },
                        ]
                    },
                    new FileConditional(x => x.Get<string>("RecordType") == "C")
                    {
                        Name = "Fees",
                        Fields =
                        [
                            new FileTable(FileParserEncoder.GetDefaultEncoder(write))
                            {
                                StartPosition = 11,
                                Length = 68,
                                Name = "Fees",
                                Layout = new FileLayout(FileParserEncoder.GetDefaultEncoder(write))
                                {
                                    FillByte = 0x00, // write only
                                    RecordLength = 20,
                                    Name = "Fee Table",
                                    MasterFields =
                                    [
                                        new FileField(write)
                                        {
                                            DataFormat = FileFieldDataFormat.String,
                                            FillCharacter = ' ', // write only
                                            Type = typeof(string),
                                            StartPosition = 0,
                                            Length = 8,
                                            Name = "Fee Type"
                                        },
                                        new FileField(write)
                                        {
                                            DataFormat = FileFieldDataFormat.String,
                                            FillCharacter = ' ', // write only
                                            Type = typeof(string),
                                            StartPosition = 8,
                                            Length = 3,
                                            Name = "Workstation"
                                        },
                                        new FileField(write)
                                        {
                                            DataFormat = FileFieldDataFormat.IBMPacked,
                                            FillByte = 0x00, // write only
                                            Type = typeof(decimal),
                                            Precision = 2,
                                            StartPosition = 11,
                                            Length = 4,
                                            Name = "Amount"
                                        },
                                        new FileField(write)
                                        {
                                            DataFormat = FileFieldDataFormat.IBMPacked,
                                            Type = typeof(DateTime?),
                                            StartPosition = 15,
                                            Length = 2,
                                            Name = "Assessment Date"
                                        },
                                    ]
                                }
                            },
                        ]
                    },
                    new FileConditional(x => x.Get<string>("RecordType") == "D")
                    {
                        Name = "Collateral",
                        Fields =
                        [
                            new FileField(write)
                            {
                                DataFormat = FileFieldDataFormat.IBMPacked,
                                FillByte = 0x00, // write only
                                Type = typeof(double),
                                Precision = 0,
                                StartPosition = 11,
                                Length = 9,
                                Name = "LO Code"
                            },
                            new FileTable(FileParserEncoder.GetDefaultEncoder(write))
                            {
                                StartPosition = 20,
                                Length = 20,
                                Name = "Collateral",
                                Layout = new FileLayout(FileParserEncoder.GetDefaultEncoder(write))
                                {
                                    FillByte = 0x00, // write only
                                    RecordLength = 20,
                                    Name = "Collateral Table",
                                    MasterFields =
                                    [
                                        new FileField(write)
                                        {
                                            DataFormat = FileFieldDataFormat.String,
                                            FillCharacter = ' ', // write only
                                            Type = typeof(string),
                                            StartPosition = 0,
                                            Length = 5,
                                            Name = "Collateral Type"
                                        },
                                        new FileField(write)
                                        {
                                            DataFormat = FileFieldDataFormat.String,
                                            FillCharacter = ' ', // write only
                                            Type = typeof(string),
                                            StartPosition = 5,
                                            Length = 10,
                                            Name = "Remarks"
                                        },
                                        new FileField(write)
                                        {
                                            DataFormat = FileFieldDataFormat.IBMPacked,
                                            FillByte = 0x00, // write only
                                            Type = typeof(decimal),
                                            Precision = 2,
                                            StartPosition = 15,
                                            Length = 5,
                                            Name = "Value"
                                        },
                                    ]
                                }
                            },
                        ]
                    },
                ]
        };

        static FileRecord Header(bool write) => new(FileParserEncoder.GetDefaultEncoder(write))
        {
            Fields =
                [
                    new FileField(write)
                    {
                        DataFormat = FileFieldDataFormat.String,
                        FillCharacter = ' ', // write only
                        Type = typeof(string),
                        StartPosition = 0,
                        Length = 20,
                        Name = "Title"
                    },
                    new FileField(write)
                    {
                        DataFormat = FileFieldDataFormat.IBMPacked,
                        Type = typeof(DateTime),
                        StartPosition = 20,
                        Length = 4,
                        Name = "Create Date"
                    },
                    new FileField(write)
                    {
                        DataFormat = FileFieldDataFormat.IBMPacked,
                        FillCharacter = ' ', // write only
                        Type = typeof(int),
                        StartPosition = 24,
                        Length = 6,
                        RightAlign = true,
                        Name = "Record Count"
                    },
                    new FileField(write)
                    {
                        DataFormat = FileFieldDataFormat.String,
                        Type = typeof(bool),
                        StartPosition = 40,
                        Length = 1,
                        BoolAsString = BooleanStringRepresentation.Yes_No,
                        Name = "Processed"
                    }
                ]
        };

        static FileRecord Footer(bool write) => new FileRecord(FileParserEncoder.GetDefaultEncoder(write))
        {
            Fields =
                [
                    new FileField(write)
                    {
                        DataFormat = FileFieldDataFormat.String,
                        FillCharacter = ' ', // write only
                        Type = typeof(string),
                        StartPosition = 0,
                        Length = 20,
                        RightAlign = true, // write only
                        Name = "Label01",
                        Value = "Totals:"
                    },
                    new FileField(write)
                    {
                        DataFormat = FileFieldDataFormat.IBMPacked,
                        Type = typeof(decimal),
                        Precision = 2,
                        StartPosition = 20,
                        Length = 4,
                        Name = "Total Balances"
                    },
                    new FileField(write)
                    {
                        DataFormat = FileFieldDataFormat.IBMPacked,
                        FillCharacter = ' ', // write only
                        Type = typeof(int),
                        StartPosition = 44,
                        Length = 6,
                        Name = "Total Offsets"
                    },

                ]
        };