Yoti.Fcm.Api.x86 0.0.11-alpha

.NET Framework 4.5
This is a prerelease version of Yoti.Fcm.Api.x86.
The owner has unlisted this package. This could mean that the package is deprecated, has security vulnerabilities or shouldn't be used anymore.
dotnet add package Yoti.Fcm.Api.x86 --version 0.0.11-alpha
NuGet\Install-Package Yoti.Fcm.Api.x86 -Version 0.0.11-alpha
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="Yoti.Fcm.Api.x86" Version="0.0.11-alpha" />
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add Yoti.Fcm.Api.x86 --version 0.0.11-alpha
#r "nuget: Yoti.Fcm.Api.x86, 0.0.11-alpha"
#r directive can be used in F# Interactive, C# scripting and .NET Interactive. Copy this into the interactive tool or source code of the script to reference the package.
// Install Yoti.Fcm.Api.x86 as a Cake Addin
#addin nuget:?package=Yoti.Fcm.Api.x86&version=0.0.11-alpha&prerelease

// Install Yoti.Fcm.Api.x86 as a Cake Tool
#tool nuget:?package=Yoti.Fcm.Api.x86&version=0.0.11-alpha&prerelease

Yoti Face Capture Module Net45

<p align="center"> <img src="yoti-logo.png" alt="Yoti Logo" height="250" /> </p>

<div align="center">

Build version

</div> <br>

.NET Framework client-side Face Detection, Validation and Face Capture

<br> <p align="center"> <em>To be used in congunction with <a href="https://developers.yoti.com/age-estimation/integration-guide">Yoti age estimation APIs</a> this Class Library aims to validate frames from a Video Capture (or individual images) to confirm suitability for use with the Yoti Age Estimation API. </em> </p>

<br> <br>

Overview

  • Numerous attributes of each frame are analyzed in order to both validate and rank (assuming found valid) each frame.
  • Batches are analyzed to "smooth" the output and the best frame from any successful batch is passed to consuming applications via the raising of a (thread-safe) event.
  • Any detection or validation failures are exposed to the consuming application so feedback can be presented to users. Consumers can either use simple summarized feedback, or to take account of detailed per-frame analysis.

Aims

This component aims to reduce the:

  • Number of API calls
  • Network bandwidth / latency
  • End-user effort/time
  • Cost

required to successfully age estimate end-user against <a href="https://developers.yoti.com/age-estimation/integration-guide">Yoti AI services age estimation APIs</a>.

Demos

Two demo projects have been created:

  • <b>DevDemo:</b> A development & QA tool (provides UI to cover most of the functionality provided): <a href="https://github.com/lampkicking/yoti-dotnet-fcm-devdemo/tree/development" target="_blank">https://github.com/lampkicking/yoti-dotnet-fcm-devdemo</a>
  • <b>UserDemo:</b> Lightweight implementation to show only core functionality (and integrated API calls): <a href="https://github.com/lampkicking/yoti-dotnet-fcm-userdemo/tree/development" target="_blank">https://github.com/lampkicking/yoti-dotnet-fcm-userdemo</a>

<br>

Requirements

Platforms

Using AnyCPU is <b>not recommended</b>.

Numerous nuget packages that the Yoti.Fcm rely on make use of native dlls and were intended to be used only for platform specific compilation. So please set consuming projects to x64 or x86 appropriately and choose the corresponding nuget(s) e.g.

Yoti.Fcm.Core.x86 or Yoti.Fcm.Core.x64

Cross Platform .NET

This codebase could be re-packaged to provide .NET Core and/or Mono version(s) to target Linux, macOS, iOS and Android

A POC was successfully built for and run on <b>Ubuntu 20.04</b>.

Optional Nugets

Two additional <b>(optional)</b> nugets are available, these provide additonal functionality:

  • <b>Yoti.Fcm.Wpf:</b> provides BindableSessionCreator to use implementations (of inputs and outputs) which facilitate binding to UI in WPF applications
  • <b>Yoti.Fcm.Api:</b> provides a lightweight IApiCaller implementation (which functionally matches <i>relevant</i> parts of the <a href="https://github.com/getyoti/yoti-dotnet-sdk" target="_blank">Yoti dotnet SDK</a> version 3.11.0)

They reference Yoti.Fcm.Core and also provide <b>platform specific</b> versions.

<br> <br>

Features

Validation

  • Frame resolution
  • Bounding box resolution
  • Number of faces (should be only one)
  • Position of Face (bounding box) within frame (center deviation)
  • Brightness below maximum (ROI)
  • Brightness above minimum (ROI)
  • Sharpness (ROI) three measures are combined to provide "anti-blur" validation

Detection

Uses modern, lightweight, optimized, pre-trained machine learning models to detect faces, their locations and associated probabilities within frames.

Highly Configurable

We have attempted to cater for a host of different hardware, software and deployment scenarios by providing a full range of customisation options to allow Integrators to focus on what matters most in their specific case, and to allow for a wide range of hardware capabilities. Consumers can choose to focus on performance or accuracy - or opt for a balance somewhere in the middle.

For more details see <a href="/Configuration.md">Configuration.md</a>

<br> <br>

Application Flow

Common

The FCM flow takes any one image (either a frame from video capture or individual images) and passes it through several steps of assessment:

Validation

  • If a face detected
  • Is there is only one face
  • Is the Probability of the one face higher than the configured value
  • Is the Bounding Box Resolution higher than the configured value
  • Is the Frame Resolution higher than the configured value
  • Is the Bounding Box centre deviation (compared to the centre of the Frame) higher than the configured value (and config is not set to ignore this check)

Validation-Ranking

If all of the above are success then we calculate further metrics; brightness and three edge detection/sharpness metrics (Sobel, Laplacian, Canny), the system then validates the following:

  • Is the Bounding Box ROI (Region of Interest) brighter than config minimum (and config is not set to ignore the brightness check)
  • Is the Bounding Box ROI (Region of Interest) darker than config maximum (and config is not set to ignore the brightness check)
  • Is the sharpnessScore (the config weighted combination of three normalised blur/edge detection scores) greater than the configured threshold (and config is not set to ignore this check)

The Validation-Ranking section is relatively expensive CPU-wise so only runs if the "Validation" checks all pass.

Batches (Sessions)

When using a capture session to process frames from a video stream:

  • The system batches frames into groups (configurable - default 10)
  • Assuming the configurable number of “excellent” frames per batch is met or exceeded (defaults to 100% / 10)
  • The system chooses the top “scoring” frame (based on the sharpnessScore)
  • System fires FaceCaptured Event and passes the image to event subscribers (consuming code)
  • If configured the system will automatically pass the image to the IApiCaller implementation
  • ApiImplementation prepares the image and uses configurable values to call the API and expose the result

Additonal "With Render"

If an Integrator wants more information than the primary result of the batch processing the (FaceCaptured event) they can request a more advanced session (based on the generic IOpenCvSessionWithRender interface). This provides the folowing:

  • FrameCaptured event which can be used to inspect or render to UI each frame as soon as it has been captured and processed.
  • GetFrameDetails method returns List<FrameData> can be used to inspect all the detailed results of each processed frame (and provides access to each image)

<br> <br> <br>

Sessions

The standard way to use the Yoti.Fcm.Core nuget is to create and use one of the capture session objects.

<b>Sessions are intended to be short lived (a number of seconds).</b>

<br>

IDisposable

Sessions implement IDisposable and as <b>such should always be disposed of</b> correctly (either calling session.Dispose(), with the using syntax or for desktop by wrapping in a Component).

<br>

Status

All the Session types expose a Status property which can be used to interrogate the overal status at any instant.

TStatus Status { get; } 

where TStatus : Output.IStatus, new()

<br>

The following are the simplest ways to construct capture sessions:

IOpenCvSessionDefault session = SessionCreator.GetOpenCvSession();

...for additonal features use "With Render" version:

IOpenCvSessionDefaultWithRender sessionWithRender = SessionCreator.GetOpenCvSessionWithRender();

<br>

To create an OpenCv based face detection and capture session with UI bindable (INotifyPropertyChanged) inputs (IConfig), and outputs (IStatus) reference the optional Yoti.Fcm.Wpf nuget:

using Yot.Fcm.Wpf;

var bindableSession = Yoti.Fcm.Wpf.BindableSessionCreator.GetOpenCvSession();
DataContext = bindableSessionWithRender.Config;
bindableSession.FaceCaptured += session_FaceCaptured;

...for additonal features use "With Render" version:

using Yot.Fcm.Wpf;

var bindableSessionWithRender = Yoti.Fcm.Wpf.BindableSessionCreator.GetOpenCvSessionWithRender();
DataContext = bindableSessionWithRender.Status;
var allFrames = bindableSessionWithRender.GetFrameDetails();
bindableSession.FaceCaptured += session_FaceCaptured;
bindableSession.FrameCaptured += session_FrameCaptured;

<br>

Generics (full customization)

If for any reason the default versions of the inputs or outputs for sessions are not appropriate an Integrator can choose to provide their own custom implementations and pass these to the relevant SessionCreate method overloads:

GetOpenCvSessionGeneric<TConfig, TStatus, TImage>(Input.IApiCaller apiCaller = null)
		 where TConfig : Input.IConfig, new()
		 where TStatus : Output.IStatus, new()
		 where TImage : class

...for additonal features use "With Render" version:

GetOpenCvSessionWithRenderGeneric<TConfig, TStatus, TImage, TFrame>(Input.IApiCaller apiCaller = null)
                where TConfig : Input.IConfig, new()
                where TStatus : Output.IStatus, new()
                where TImage : class
                where TFrame : class

<br> <br>

Integration

Use package manage GUI in Visual Studio (or VS Code) or run the following command from NuGet Package Manager Console:

Install-Package Yoti.Fcm.Core.x86 

or

Install-Package Yoti.Fcm.Core.x64 

<b>Note:</b> currently SDK-Style projects are required to handle the transitive dependencies and content files properly (see demos for full samples)

Video Capture Session

Required:

  • Reference nuget package Yoti.Fcm.Core.x86 or Yoti.Fcm.Core.x64
  • Create a session object using the appropriate SessionCreator method

Optional:

  • Subscribe to events
  • Customise configuration with App.config
  • Customise configuration with IConfig instance
  • Pass an IApiCaller implementation for automatic API calls
  • Handle results (exposed by session.Status) <br> <br>
using Yoti.Fcm.Core;
using OpenCvSharp.Extensions;

namespace WpfApp1
{
    public partial class MainWindow : Window
    {
        public MainWindow()
        {
            InitializeComponent();
            session = SessionCreator.GetOpenCvSessionWithRender();
            session.FaceCaptured += session_FaceCaptured;
            session.FrameCaptured += session_FrameCaptured;
            Closing += mainWindow_Closing;
        }

        void session_FaceCaptured(object sender, FaceCapturedEventArgs<BitmapSource> e)
        {
            //access captured image data, call api... 
            var image = e.Image; 
        }

        void session_FrameCaptured(object sender, FrameCapturedEventArgs<OpenCvSharp.Mat> e)
        {
            //render the Frame to your UI and or examine session.Status (optional)
            Dispatcher.Invoke(() => FrameImage.Source = e.Frame.ToWriteableBitmap());
        }

        async void buttonStart_Click(object sender, RoutedEventArgs e) => await session.StartCaptureAsync();
        async void buttonStop_Click(object sender, RoutedEventArgs e) => await session.StopCaptureAsync();

        void buttonPause_Click(object sender, RoutedEventArgs e) => session.PauseCapture();
        void buttonResume_Click(object sender, RoutedEventArgs e) => session.ResumeCapture();

        void buttonGetData_Click(object sender, RoutedEventArgs e)
        {
            //get all captured frames and all their details
            var allFrames = session.GetFrameDetails();
        }

        async void mainWindow_Closing(object sender, CancelEventArgs e)
        {
            await session.StopCaptureAsync();
            session.Dispose();
        }

        IOpenCvSessionDefaultWithRender session;
    }
}

<br> <br>

Single Image Validation

Required:

  • Reference nuget package Yoti.Fcm.Core.x86 or Yoti.Fcm.Core.x64
  • Pass your image to a SingleImageAssessor.Assess() overload
  • Handle the simple results

Optional:

  • Customise configuration with App.config
  • Customise configuration with IConfigSingleImage instance
using Yoti.Fcm.Core;

class Program 
{
    static void Main() 
    {
        var resultFromPath = SingleImageAssessor.Assess("some_potential_face.jpg");
        if(resultFromPath.Decision == Output.Results.Good)
            // Call Api...
        else
            MessageBox.Show(resultFromPath.Result.ToString("G");

        
        //Choose appropriate data type for your image input:
        
        byte[] imageFileBytes = YourImageService.GetImageBytes();
        var resultFromBytes = SingleImageAssessor.Assess(imageFileBytes);

        MemoryStream imageStream = YourImageService.GetImageStream();
        var resultFromStream = SingleImageAssessor.Assess(imageStream);

        Mat imageOpenCvMat =  YourImageService.GetImageMatrix();
        var resultFromOpenCvMat = SingleImageAssessor.Assess(imageOpenCvMat);


        //Handle more detailed information...

        Console.Writeline(resultFromStream.TimesStamp);
        Console.Writeline(resultFromStream.Rank);
        Console.Writeline(resultFromStream.Validation.BadReason);
    }
}

<br> <br>

Configuration

If you would like to tweak your configuration for your particular deployment scenario, or simply would like more information about the operation of the module see <a href="Configuration.md">Configuration.md</a>.

<br> <br>

Benchmarking

Initial testing has shown around 56ms for each ML detection run and around 10 FPS total throughput (including detection, frame assessment, augmentation and rendering - with default configuration).

Video capture sessions seem to consume around 50-100 MB RAM usage and around 25% CPU usage (obviously highly dependant on H/W and S/W).

Practical Performance

Overall (with responsive/attentive users monitoring feedback) we tend to see successful (auto) capture in the region of 800ms - 2.5s

<br> <br>

Terms

  • See <a href="Terms.md" alt="Terms.md">Terms.md</a> for more info
  • © Yoti Ltd. 2022 <a href="https://www.yoti.com/" alt="Yoti Ltd" target="_blank">yoti.com</a>

More Info

For further documentation, see https://developers.yoti.com/

Product Versions
.NET Framework net45 net451 net452 net46 net461 net462 net463 net47 net471 net472 net48
Compatible target framework(s)
Additional computed target framework(s)
Learn more about Target Frameworks and .NET Standard.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last updated