Yoti.Fcm.Wpf.x86
0.0.11-alpha
dotnet add package Yoti.Fcm.Wpf.x86 --version 0.0.11-alpha
NuGet\Install-Package Yoti.Fcm.Wpf.x86 -Version 0.0.11-alpha
<PackageReference Include="Yoti.Fcm.Wpf.x86" Version="0.0.11-alpha" />
paket add Yoti.Fcm.Wpf.x86 --version 0.0.11-alpha
#r "nuget: Yoti.Fcm.Wpf.x86, 0.0.11-alpha"
// Install Yoti.Fcm.Wpf.x86 as a Cake Addin
#addin nuget:?package=Yoti.Fcm.Wpf.x86&version=0.0.11-alpha&prerelease
// Install Yoti.Fcm.Wpf.x86 as a Cake Tool
#tool nuget:?package=Yoti.Fcm.Wpf.x86&version=0.0.11-alpha&prerelease
Yoti Face Capture Module Net45
<p align="center"> <img src="yoti-logo.png" alt="Yoti Logo" height="250" /> </p>
<div align="center">
</div> <br>
.NET Framework client-side Face Detection, Validation and Face Capture
<br> <p align="center"> <em>To be used in congunction with <a href="https://developers.yoti.com/age-estimation/integration-guide">Yoti age estimation APIs</a> this Class Library aims to validate frames from a Video Capture (or individual images) to confirm suitability for use with the Yoti Age Estimation API. </em> </p>
<br> <br>
Overview
- Numerous attributes of each frame are analyzed in order to both validate and rank (assuming found valid) each frame.
- Batches are analyzed to "smooth" the output and the best frame from any successful batch is passed to consuming applications via the raising of a (thread-safe) event.
- Any detection or validation failures are exposed to the consuming application so feedback can be presented to users. Consumers can either use simple summarized feedback, or to take account of detailed per-frame analysis.
Aims
This component aims to reduce the:
- Number of API calls
- Network bandwidth / latency
- End-user effort/time
- Cost
required to successfully age estimate end-user against <a href="https://developers.yoti.com/age-estimation/integration-guide">Yoti AI services age estimation APIs</a>.
Demos
Two demo projects have been created:
- <b>DevDemo:</b> A development & QA tool (provides UI to cover most of the functionality provided): <a href="https://github.com/lampkicking/yoti-dotnet-fcm-devdemo/tree/development" target="_blank">https://github.com/lampkicking/yoti-dotnet-fcm-devdemo</a>
- <b>UserDemo:</b> Lightweight implementation to show only core functionality (and integrated API calls): <a href="https://github.com/lampkicking/yoti-dotnet-fcm-userdemo/tree/development" target="_blank">https://github.com/lampkicking/yoti-dotnet-fcm-userdemo</a>
<br>
Requirements
- .NET Framework 4.5.X
- Windows (x64 or x86)
Platforms
Using AnyCPU is <b>not recommended</b>.
Numerous nuget packages that the Yoti.Fcm rely on make use of native dlls and were intended to be used only for platform specific compilation. So please set consuming projects to x64 or x86 appropriately and choose the corresponding nuget(s) e.g.
Yoti.Fcm.Core.x86
or Yoti.Fcm.Core.x64
Cross Platform .NET
This codebase could be re-packaged to provide .NET Core and/or Mono version(s) to target Linux, macOS, iOS and Android
A POC was successfully built for and run on <b>Ubuntu 20.04</b>.
Optional Nugets
Two additional <b>(optional)</b> nugets are available, these provide additonal functionality:
- <b>Yoti.Fcm.Wpf:</b> provides
BindableSessionCreator
to use implementations (of inputs and outputs) which facilitate binding to UI in WPF applications - <b>Yoti.Fcm.Api:</b> provides a lightweight
IApiCaller
implementation (which functionally matches <i>relevant</i> parts of the <a href="https://github.com/getyoti/yoti-dotnet-sdk" target="_blank">Yoti dotnet SDK</a> version 3.11.0)
They reference Yoti.Fcm.Core and also provide <b>platform specific</b> versions.
<br> <br>
Features
Validation
- Frame resolution
- Bounding box resolution
- Number of faces (should be only one)
- Position of Face (bounding box) within frame (center deviation)
- Brightness below maximum (ROI)
- Brightness above minimum (ROI)
- Sharpness (ROI) three measures are combined to provide "anti-blur" validation
Detection
Uses modern, lightweight, optimized, pre-trained machine learning models to detect faces, their locations and associated probabilities within frames.
Highly Configurable
We have attempted to cater for a host of different hardware, software and deployment scenarios by providing a full range of customisation options to allow Integrators to focus on what matters most in their specific case, and to allow for a wide range of hardware capabilities. Consumers can choose to focus on performance or accuracy - or opt for a balance somewhere in the middle.
For more details see <a href="/Configuration.md">Configuration.md</a>
<br> <br>
Application Flow
Common
The FCM flow takes any one image (either a frame from video capture or individual images) and passes it through several steps of assessment:
Validation
- If a face detected
- Is there is only one face
- Is the Probability of the one face higher than the configured value
- Is the Bounding Box Resolution higher than the configured value
- Is the Frame Resolution higher than the configured value
- Is the Bounding Box centre deviation (compared to the centre of the Frame) higher than the configured value (and config is not set to ignore this check)
Validation-Ranking
If all of the above are success then we calculate further metrics; brightness and three edge detection/sharpness metrics (Sobel, Laplacian, Canny), the system then validates the following:
- Is the Bounding Box ROI (Region of Interest) brighter than config minimum (and config is not set to ignore the brightness check)
- Is the Bounding Box ROI (Region of Interest) darker than config maximum (and config is not set to ignore the brightness check)
- Is the sharpnessScore (the config weighted combination of three normalised blur/edge detection scores) greater than the configured threshold (and config is not set to ignore this check)
The Validation-Ranking section is relatively expensive CPU-wise so only runs if the "Validation" checks all pass.
Batches (Sessions)
When using a capture session to process frames from a video stream:
- The system batches frames into groups (configurable - default 10)
- Assuming the configurable number of “excellent” frames per batch is met or exceeded (defaults to 100% / 10)
- The system chooses the top “scoring” frame (based on the sharpnessScore)
- System fires FaceCaptured Event and passes the image to event subscribers (consuming code)
- If configured the system will automatically pass the image to the IApiCaller implementation
- ApiImplementation prepares the image and uses configurable values to call the API and expose the result
Additonal "With Render"
If an Integrator wants more information than the primary result of the batch processing the (FaceCaptured
event) they can request a more advanced session (based on the generic IOpenCvSessionWithRender
interface). This provides the folowing:
FrameCaptured
event which can be used to inspect or render to UI each frame as soon as it has been captured and processed.GetFrameDetails
method returnsList<FrameData>
can be used to inspect all the detailed results of each processed frame (and provides access to each image)
<br> <br> <br>
Sessions
The standard way to use the Yoti.Fcm.Core nuget is to create and use one of the capture session objects.
<b>Sessions are intended to be short lived (a number of seconds).</b>
<br>
IDisposable
Sessions implement IDisposable
and as <b>such should always be disposed of</b> correctly (either calling session.Dispose()
, with the using
syntax or for desktop by wrapping in a Component
).
<br>
Status
All the Session types expose a Status
property which can be used to interrogate the overal status at any instant.
TStatus Status { get; }
where TStatus : Output.IStatus, new()
<br>
Default Implementations (recommended)
The following are the simplest ways to construct capture sessions:
IOpenCvSessionDefault session = SessionCreator.GetOpenCvSession();
...for additonal features use "With Render" version:
IOpenCvSessionDefaultWithRender sessionWithRender = SessionCreator.GetOpenCvSessionWithRender();
<br>
INotifyPropertyChanged / WPF Bindings (recommended)
To create an OpenCv based face detection and capture session with UI bindable (INotifyPropertyChanged
) inputs (IConfig
), and outputs (IStatus
) reference the optional Yoti.Fcm.Wpf nuget:
using Yot.Fcm.Wpf;
var bindableSession = Yoti.Fcm.Wpf.BindableSessionCreator.GetOpenCvSession();
DataContext = bindableSessionWithRender.Config;
bindableSession.FaceCaptured += session_FaceCaptured;
...for additonal features use "With Render" version:
using Yot.Fcm.Wpf;
var bindableSessionWithRender = Yoti.Fcm.Wpf.BindableSessionCreator.GetOpenCvSessionWithRender();
DataContext = bindableSessionWithRender.Status;
var allFrames = bindableSessionWithRender.GetFrameDetails();
bindableSession.FaceCaptured += session_FaceCaptured;
bindableSession.FrameCaptured += session_FrameCaptured;
<br>
Generics (full customization)
If for any reason the default versions of the inputs or outputs for sessions are not appropriate an Integrator can choose to provide their own custom implementations and pass these to the relevant SessionCreate method overloads:
GetOpenCvSessionGeneric<TConfig, TStatus, TImage>(Input.IApiCaller apiCaller = null)
where TConfig : Input.IConfig, new()
where TStatus : Output.IStatus, new()
where TImage : class
...for additonal features use "With Render" version:
GetOpenCvSessionWithRenderGeneric<TConfig, TStatus, TImage, TFrame>(Input.IApiCaller apiCaller = null)
where TConfig : Input.IConfig, new()
where TStatus : Output.IStatus, new()
where TImage : class
where TFrame : class
<br> <br>
Integration
Use package manage GUI in Visual Studio (or VS Code) or run the following command from NuGet Package Manager Console:
Install-Package Yoti.Fcm.Core.x86
or
Install-Package Yoti.Fcm.Core.x64
<b>Note:</b> currently SDK-Style projects are required to handle the transitive dependencies and content files properly (see demos for full samples)
Video Capture Session
Required:
- Reference nuget package
Yoti.Fcm.Core.x86
orYoti.Fcm.Core.x64
- Create a session object using the appropriate
SessionCreator
method
Optional:
- Subscribe to events
- Customise configuration with
App.config
- Customise configuration with
IConfig
instance - Pass an
IApiCaller
implementation for automatic API calls - Handle results (exposed by
session.Status
) <br> <br>
using Yoti.Fcm.Core;
using OpenCvSharp.Extensions;
namespace WpfApp1
{
public partial class MainWindow : Window
{
public MainWindow()
{
InitializeComponent();
session = SessionCreator.GetOpenCvSessionWithRender();
session.FaceCaptured += session_FaceCaptured;
session.FrameCaptured += session_FrameCaptured;
Closing += mainWindow_Closing;
}
void session_FaceCaptured(object sender, FaceCapturedEventArgs<BitmapSource> e)
{
//access captured image data, call api...
var image = e.Image;
}
void session_FrameCaptured(object sender, FrameCapturedEventArgs<OpenCvSharp.Mat> e)
{
//render the Frame to your UI and or examine session.Status (optional)
Dispatcher.Invoke(() => FrameImage.Source = e.Frame.ToWriteableBitmap());
}
async void buttonStart_Click(object sender, RoutedEventArgs e) => await session.StartCaptureAsync();
async void buttonStop_Click(object sender, RoutedEventArgs e) => await session.StopCaptureAsync();
void buttonPause_Click(object sender, RoutedEventArgs e) => session.PauseCapture();
void buttonResume_Click(object sender, RoutedEventArgs e) => session.ResumeCapture();
void buttonGetData_Click(object sender, RoutedEventArgs e)
{
//get all captured frames and all their details
var allFrames = session.GetFrameDetails();
}
async void mainWindow_Closing(object sender, CancelEventArgs e)
{
await session.StopCaptureAsync();
session.Dispose();
}
IOpenCvSessionDefaultWithRender session;
}
}
<br> <br>
Single Image Validation
Required:
- Reference nuget package
Yoti.Fcm.Core.x86
orYoti.Fcm.Core.x64
- Pass your image to a
SingleImageAssessor.Assess()
overload - Handle the simple results
Optional:
- Customise configuration with
App.config
- Customise configuration with
IConfigSingleImage
instance
using Yoti.Fcm.Core;
class Program
{
static void Main()
{
var resultFromPath = SingleImageAssessor.Assess("some_potential_face.jpg");
if(resultFromPath.Decision == Output.Results.Good)
// Call Api...
else
MessageBox.Show(resultFromPath.Result.ToString("G");
//Choose appropriate data type for your image input:
byte[] imageFileBytes = YourImageService.GetImageBytes();
var resultFromBytes = SingleImageAssessor.Assess(imageFileBytes);
MemoryStream imageStream = YourImageService.GetImageStream();
var resultFromStream = SingleImageAssessor.Assess(imageStream);
Mat imageOpenCvMat = YourImageService.GetImageMatrix();
var resultFromOpenCvMat = SingleImageAssessor.Assess(imageOpenCvMat);
//Handle more detailed information...
Console.Writeline(resultFromStream.TimesStamp);
Console.Writeline(resultFromStream.Rank);
Console.Writeline(resultFromStream.Validation.BadReason);
}
}
<br> <br>
Configuration
If you would like to tweak your configuration for your particular deployment scenario, or simply would like more information about the operation of the module see <a href="Configuration.md">Configuration.md</a>.
<br> <br>
Benchmarking
Initial testing has shown around 56ms for each ML detection run and around 10 FPS total throughput (including detection, frame assessment, augmentation and rendering - with default configuration).
Video capture sessions seem to consume around 50-100 MB RAM usage and around 25% CPU usage (obviously highly dependant on H/W and S/W).
Practical Performance
Overall (with responsive/attentive users monitoring feedback) we tend to see successful (auto) capture in the region of 800ms - 2.5s
<br> <br>
Terms
- See <a href="Terms.md" alt="Terms.md">Terms.md</a> for more info
- © Yoti Ltd. 2022 <a href="https://www.yoti.com/" alt="Yoti Ltd" target="_blank">yoti.com</a>
More Info
For further documentation, see https://developers.yoti.com/
Product | Versions Compatible and additional computed target framework versions. |
---|---|
.NET Framework | net45 is compatible. net451 was computed. net452 was computed. net46 was computed. net461 was computed. net462 was computed. net463 was computed. net47 was computed. net471 was computed. net472 was computed. net48 was computed. net481 was computed. |
-
.NETFramework 4.5
- PresentationFramework (>= 4.6.0)
- Yoti.Fcm.Core.x86 (>= 0.0.11-alpha)
NuGet packages
This package is not used by any NuGet packages.
GitHub repositories
This package is not used by any popular GitHub repositories.
Version | Downloads | Last updated |
---|