Clapsode.DataLayerTS.WeatherDataLoader 1.0.13

DWD Weather Ingestion for Data Layer TS

A high-performance .NET library for ingesting weather forecast data from the German Weather Service (DWD) into a Data Layer TS time-series database.

This tool is essential for applications that rely on weather data for forecasting, such as predicting photovoltaic power production or wind turbine output.

The Challenge with DWD Data

The German Weather Service (DWD) provides a wealth of free, commercially-usable weather forecast data. However, using it directly for time-series analysis presents several challenges:

• Data Volatility: Forecasts are overwritten every 24 hours, making historical analysis impossible without a storage solution.
• Data Format: The data is provided in GRIB2 files, where each file contains data for hundreds of thousands of grid points but only for a single point in time.
• Analysis Needs: Effective model training and forecasting require the opposite format: a long series of many time steps for just one or a few grid points.

This library bridges that gap, transforming the DWD's data into a format ready for analysis and storing it in the high-performance Data Layer TS database.

Key Features

• Continuous Ingestion: Runs as a background service to continuously fetch the latest forecasts as they are published.
• Broad Model Support: Ingests data from a wide range of DWD models:
  • ICON (Global)
  • ICON-EU (Europe)
  • ICON-D2 (DACH region)
  • ICON-D2-RUC (Rapid Update, DACH)
  • ICON-ART (Aerosols and Reactive Trace gases, Global & EU)
• Flexible Grid Support: Handles both regular-lat-lon and icosahedral grids.
• Dynamic Configuration: Can automatically detect new variables and coordinates added to your Data Layer TS instance and begin ingesting them without a restart.
• Performance Tuning: Allows configuration of parallel ingestion tasks to maximize throughput by matching the IOPS of your target Data Layer TS instance.

Requirements

• .NET 9 or later.
• An active Data Layer TS instance. (Currently available exclusively in the Microsoft Azure Marketplace).

Installation

Install the package into your .NET project using the NuGet Package Manager or the dotnet CLI.

dotnet add package Clapsode.DataLayerTS.WeatherDataLoader

Quick Start

The following example demonstrates how to configure and run the IngestionScheduler.

Dependencies for Example

To use the console logger as shown in the example, you will also need the Microsoft.Extensions.Logging.Console package.

dotnet add package Microsoft.Extensions.Logging.Console

Example Code

using Clapsode.DataLayerTS.WeatherDataLoader;
using Clapsode.DataLayerTS.WeatherDataLoader.Enumerations;
using Clapsode.DataLayerTS.WeatherDataLoader.Models;
using Microsoft.Extensions.Logging;
using System.Threading.Tasks;

// 1. Initialize Logging (Optional but Recommended)
using ILoggerFactory loggerFactory = LoggerFactory.Create(builder =>
{
    builder
        .AddFilter("Microsoft", LogLevel.Warning)
        .AddFilter("System", LogLevel.Warning)
        .AddFilter("Clapsode", LogLevel.Information)
        .AddSimpleConsole(c => c.TimestampFormat = "[HH:mm:ss] ");
});

// 2. Create the Scheduler with Base Configuration
// Replace with your Data Layer TS instance URL and auth token.
var schedulerConfig = new SchedulerConfiguration("https://YOUR_[INSTANCE.com/](https://INSTANCE.com/)", "Basic YOUR_TOKEN");
await using var ingestionScheduler = new IngestionScheduler(schedulerConfig, loggerFactory);

// 3. Define Coordinates to Extract per Model
var coordsToLoad = new Dictionary<NumericalWeatherPredictionModel, Dictionary<GridType, HashSet<Coordinate>>>
{
    [NumericalWeatherPredictionModel.ICON_D2] = new()
    {
        [GridType.REGULAR_LAT_LON] = new HashSet<Coordinate> { new(50.93, 6.95) } // Cologne, DE
    },
    [NumericalWeatherPredictionModel.ICON_EU] = new()
    {
        [GridType.REGULAR_LAT_LON] = new HashSet<Coordinate> { new(48.85, 2.35) } // Paris, FR
    }
};

// 4. Define Weather Variables to Ingest
var contentToLoad = new HashSet<ContentConfiguration>
{
    // Temperature at 2m from ICON-D2
    new(NumericalWeatherPredictionModel.ICON_D2, GridType.REGULAR_LAT_LON, LevelType.SINGLE_LEVEL, level: null, "t_2m", TimeSpan.FromMinutes(60)),
    // Surface Downward Short-Wave Diffuse Radiation (ASWDIFD_S) from ICON-D2
    new(NumericalWeatherPredictionModel.ICON_D2, GridType.REGULAR_LAT_LON, LevelType.SINGLE_LEVEL, level: null, "aswdifd_s", TimeSpan.FromMinutes(15)),
    // Temperature at 2m from ICON-EU
    new(NumericalWeatherPredictionModel.ICON_EU, GridType.REGULAR_LAT_LON, LevelType.SINGLE_LEVEL, level: null, "t_2m", TimeSpan.FromMinutes(60))
};

// 5. Create and Run the Ingestion Request
var request = new SchedulingRequest
{
    RunContinuously = true,
    ConfigurationsToLoad = contentToLoad,
    CoordinatesToLoad = coordsToLoad,
    CoordinateMapping = CoordinateMapping.MAP_TO_NEAREST
};

// This will run indefinitely until the application is shut down or the token is cancelled.
await ingestionScheduler.RunAsync(request, CancellationToken.None);

Advanced Configuration

Dynamic Loading from Data Layer TS

Instead of defining coordinates and variables in code with a SchedulingRequest, you can configure the scheduler to pull its configuration directly from your Data Layer TS instance.

Set UseRepositoryForConfiguration to true in the SchedulingRequest. The scheduler will then:

1. Query the configured ILoadingConfigurationRepository (which defaults to your Data Layer TS instance).
2. Continuously ingest all time series it finds.
3. A background job will periodically check for new time series in the database and automatically start ingesting them.

State Management

The scheduler's state (which files have been processed) is managed via the IProcessingHistoryRepository. By default, this uses local files. A cold start (with no history) is not expensive, as Data Layer TS is optimized for writes based on the number of series, not the number of data points.

Performance Tuning

The main bottleneck will likely be the write IOPS of your Data Layer TS instance. You can control the number of parallel ingestion processes in the SchedulerConfiguration. The optimal setting is the lowest number of parallel tasks that keeps your Data Layer TS instance's IOPS near 100%, ensuring maximum ingestion throughput while preserving read capacity.

License

Usage of this library is granted under the condition that it is used in conjunction with a Data Layer TS instance.