Encamina.Enmarcha.SemanticKernel.Connectors.Document 8.1.6-preview-01

This is a prerelease version of Encamina.Enmarcha.SemanticKernel.Connectors.Document.
There is a newer version of this package available.
See the version list below for details.
dotnet add package Encamina.Enmarcha.SemanticKernel.Connectors.Document --version 8.1.6-preview-01                
NuGet\Install-Package Encamina.Enmarcha.SemanticKernel.Connectors.Document -Version 8.1.6-preview-01                
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="Encamina.Enmarcha.SemanticKernel.Connectors.Document" Version="8.1.6-preview-01" />                
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add Encamina.Enmarcha.SemanticKernel.Connectors.Document --version 8.1.6-preview-01                
#r "nuget: Encamina.Enmarcha.SemanticKernel.Connectors.Document, 8.1.6-preview-01"                
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
// Install Encamina.Enmarcha.SemanticKernel.Connectors.Document as a Cake Addin
#addin nuget:?package=Encamina.Enmarcha.SemanticKernel.Connectors.Document&version=8.1.6-preview-01&prerelease

// Install Encamina.Enmarcha.SemanticKernel.Connectors.Document as a Cake Tool
#tool nuget:?package=Encamina.Enmarcha.SemanticKernel.Connectors.Document&version=8.1.6-preview-01&prerelease                

Semantic Kernel - Document Connectors

Nuget package

Document Connectors specializes in reading information from files in various formats and subsequently chunking it. The most typical use case is, within the context of generating document embeddings, reading information from a variety of file formats (pdf, docx, pptx, etc.) and chunks its content into smaller parts.

Setup

Nuget package

First, install NuGet. Then, install Encamina.Enmarcha.SemanticKernel.Connectors.Document from the package manager console:

PM> Install-Package Encamina.Enmarcha.SemanticKernel.Connectors.Document

.NET CLI:

First, install .NET CLI. Then, install Encamina.Enmarcha.SemanticKernel.Connectors.Document from the .NET CLI:

dotnet add package Encamina.Enmarcha.SemanticKernel.Connectors.Document

How to use

Starting from a Program.cs or a similar entry point file in your project, add the following code:

// Entry point
var builder = WebApplication.CreateBuilder(new WebApplicationOptions
{
   // ...
});

// ...

services.AddDefaultDocumentContentExtractor();

This extension method will add the default implementation of the IDocumentContentExtractor interface as a singleton. The default implementation is DefaultDocumentContentExtractor. With this, we can resolve the IDocumentContentExtractor interface and obtain the chunks of a file:

Construction injection

public class MyClass
{
    private readonly IDocumentContentExtractor documentContentExtractor;

    public MyClass(IDocumentContentExtractor documentContentExtractor)
    {
        this.documentContentExtractor = documentContentExtractor;
    }

    public IEnumerable<string> GetPdfChunks()
    {
        using var file = File.OpenRead("example.pdf");

        var pdfChunks = documentContentExtractor.GetDocumentContent(file, ".pdf");

        return pdfChunks;
    }
}

Service Provider

var serviceProvider = services.BuildServiceProvider();
var documentContentExtractor = serviceProvider.GetRequiredService<IDocumentContentExtractor>();

using var file = File.OpenRead("example.pdf");
var fileChunks = documentContentExtractor.GetDocumentContent(file, ".pdf");

For the above code to be fully functional, it is necessary to configure some additional services, specifically the ITextSplitter interface and a function to calculate the length of each chunk.

The previous code, based on the file extension, searches for a suitable IDocumentConnector for the file type, processes the file to extract its text and finally, it uses an ITextSplitter to split the text into chunks.

Details about the IDocumentConnector

The default implementation DefaultDocumentContentExtractor, uses the following IDocumentConnectors:

  • WordDocumentConnector: For .docx files, it extracts the text from the file by adding each paragraph on a new line.

  • CleanPdfDocumentConnector: For .pdf files, it extracts the raw text from the file (with all words separated by spaces) and removes common words, typically headers or footers that appear in at least 25% of the document.

  • ParagraphPptxDocumentConnector: For .pptx files, it extracts the text from the file, with one line per paragraph found in each slide.

  • TxtDocumentConnector: For .txt files, it extracts the raw text from the file using UTF-8 as the character encoding.

  • TxtDocumentConnector: For .md files, it extracts the raw text from the file using UTF-8 as the character encoding.

  • VttDocumentConnector: For .vtt files, it extracts the text from the subtitles while removing the timestamp marks. Use UTF-8 as the character encoding.

For other formats, it throws a NotSupportedException.

Others available IDocumentConnector

  • SlidePptxDocumentConnector: For .pptx files, it extracts the text from the file with just one line for each slide found.

  • PdfDocumentConnector: For .pdf files, it extracts the raw text from the file for each page (all words separated by spaces) and add a line break between the text of each page.

  • PdfWithTocDocumentConnector: For .pdf files, it retrieve the Table of Contents and generates, for each Table of Contents item, a text with the section title, a colon mark (:), and the content text of the section (e.g. Title1: Content of the Title1 section). Add a line break between each section. The output format of the text is configurable with the TocItemFormat property. Additionally, remove common words, typically headers or footers that appear in at least 25% of the document.

  • StrictFormatCleanPdfDocumentConnector: For .pdf files, it extracts the text from the file and attempts to preserve the document's formatting, including paragraphs, titles, and other structural elements. Additionally, it removes common words, typically headers or footers that appear in at least 25% of the document, and it excludes non-horizontal text. During the text extraction process, an effort is made to retain the document's format; however, it is important to note that this process relies on OCR recognition, which is not perfect, and the results may vary depending on the quality of the PDF.

Use your own IDocumentConnector

To use your own IDocumentConnectors, you can use the base class DocumentContentExtractorBase and override the GetDocumentConnector method. This way, you can return your own IDocumentConnectors to handle a specific file format based on the file extension.

public class MyCustomDocumentContentExtractor : DocumentContentExtractorBase
{
    public MyCustomDocumentContentExtractor(ITextSplitter textSplitter, Func<string, int> lengthFunction) : base(textSplitter, lengthFunction)
    {
    }

    protected override IDocumentConnector GetDocumentConnector(string fileExtension)
    {
        return fileExtension.ToUpperInvariant() switch
        {
            @".rtf" => new MyCustomRtfDocumentConnector(),
            @".pdf" => new PdfWithTocDocumentConnector(),
            @".txt" => new TxtDocumentConnector(Encoding.UTF8),
            _ => throw new NotSupportedException(fileExtension),
        };
    }
}

Don't forget to register it.

// Entry point
var builder = WebApplication.CreateBuilder(new WebApplicationOptions
{
   // ...
});

// ...

// Now we use our own implementation
// services.AddDefaultDocumentContentExtractor();

services.AddSingleton<IDocumentContentExtractor, MyCustomDocumentContentExtractor>();

With this, you will be able to use the extractor you need for each type of file.

Product Compatible and additional computed target framework versions.
.NET net8.0 is compatible.  net8.0-android was computed.  net8.0-browser was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last updated
8.2.0 313 10/22/2024
8.2.0-preview-01-m01 92 9/17/2024
8.1.9-preview-03 205 11/19/2024
8.1.9-preview-02 65 10/22/2024
8.1.9-preview-01 202 10/4/2024
8.1.8 162 9/23/2024
8.1.8-preview-07 284 9/12/2024
8.1.8-preview-06 142 9/11/2024
8.1.8-preview-05 141 9/10/2024
8.1.8-preview-04 213 8/16/2024
8.1.8-preview-03 133 8/13/2024
8.1.8-preview-02 88 8/13/2024
8.1.8-preview-01 100 8/12/2024
8.1.7 104 8/7/2024
8.1.7-preview-09 128 7/3/2024
8.1.7-preview-08 107 7/2/2024
8.1.7-preview-07 86 6/10/2024
8.1.7-preview-06 89 6/10/2024
8.1.7-preview-05 110 6/6/2024
8.1.7-preview-04 90 6/6/2024
8.1.7-preview-03 98 5/24/2024
8.1.7-preview-02 105 5/10/2024
8.1.7-preview-01 98 5/8/2024
8.1.6 133 5/7/2024
8.1.6-preview-08 56 5/2/2024
8.1.6-preview-07 99 4/29/2024
8.1.6-preview-06 277 4/26/2024
8.1.6-preview-05 97 4/24/2024
8.1.6-preview-04 106 4/22/2024
8.1.6-preview-03 96 4/22/2024
8.1.6-preview-02 127 4/17/2024
8.1.6-preview-01 103 4/15/2024
8.1.5 109 4/15/2024
8.1.5-preview-15 88 4/10/2024
8.1.5-preview-14 141 3/20/2024
8.1.5-preview-13 85 3/18/2024
8.1.5-preview-12 118 3/13/2024
8.1.5-preview-11 89 3/13/2024
8.1.5-preview-10 123 3/13/2024
8.1.5-preview-09 97 3/12/2024
8.1.5-preview-08 80 3/12/2024
8.1.5-preview-07 98 3/8/2024
8.1.5-preview-06 203 3/8/2024
8.1.5-preview-05 92 3/7/2024
8.1.5-preview-04 92 3/7/2024
8.1.5-preview-03 84 3/7/2024
8.1.5-preview-02 146 2/28/2024
8.1.5-preview-01 144 2/19/2024
8.1.4 187 2/15/2024
8.1.3 141 2/13/2024
8.1.3-preview-07 77 2/13/2024
8.1.3-preview-06 106 2/12/2024
8.1.3-preview-05 93 2/9/2024
8.1.3-preview-04 100 2/8/2024
8.1.3-preview-03 123 2/7/2024
8.1.3-preview-02 80 2/2/2024
8.1.3-preview-01 88 2/2/2024
8.1.2 162 2/1/2024
8.1.2-preview-9 96 1/22/2024
8.1.2-preview-8 91 1/19/2024
8.1.2-preview-7 87 1/19/2024
8.1.2-preview-6 90 1/19/2024
8.1.2-preview-5 94 1/19/2024
8.1.2-preview-4 107 1/19/2024
8.1.2-preview-3 90 1/18/2024
8.1.2-preview-2 80 1/18/2024
8.1.2-preview-16 73 1/31/2024
8.1.2-preview-15 85 1/31/2024
8.1.2-preview-14 190 1/25/2024
8.1.2-preview-13 91 1/25/2024
8.1.2-preview-12 97 1/23/2024
8.1.2-preview-11 84 1/23/2024
8.1.2-preview-10 76 1/22/2024
8.1.2-preview-1 98 1/18/2024
8.1.1 129 1/18/2024
8.1.0 113 1/18/2024
8.0.3 175 12/29/2023
8.0.1 157 12/14/2023
8.0.0 156 12/7/2023
6.0.4.3 160 12/29/2023
6.0.4.2 165 12/20/2023
6.0.4.1 104 12/19/2023
6.0.4 170 12/4/2023
6.0.3.20 122 11/27/2023
6.0.3.19 133 11/22/2023