Azure.Storage.DataMovement.Blobs 12.0.0-beta.3

Azure Storage Data Movement Blobs client library for .NET

Server Version: 2020-04-08, 2020-02-10, 2019-12-12, 2019-07-07, and 2020-02-02

Azure Storage is a Microsoft-managed service providing cloud storage that is highly available, secure, durable, scalable, and redundant. Azure Storage includes Azure Blobs (objects), Azure Data Lake Storage Gen2, Azure Files, and Azure Queues.

The Azure Storage Data Movement library is optimized for uploading, downloading and copying customer data.

The Azure.Storage.DataMovement.Blobs library provides infrastructure shared by the other Azure Storage client libraries.

Source code | Package (NuGet) | API reference documentation | REST API documentation | Product documentation

Getting started

Install the package

Install the Azure Storage client library for .NET you'd like to use with NuGet and the Azure.Storage.DataMovement.Blobs client library will be included:

dotnet add package Azure.Storage.DataMovement --prerelease
dotnet add package Azure.Storage.DataMovement.Blobs --prerelease

Prerequisites

You need an Azure subscription and a Storage Account to use this package.

To create a new Storage Account, you can use the Azure Portal, Azure PowerShell, or the Azure CLI. Here's an example using the Azure CLI:

az storage account create --name MyStorageAccount --resource-group MyResourceGroup --location westus --sku Standard_LRS

Authenticate the client

The Azure.Storage.DataMovement.Blobs library uses clients from the Azure.Storage.Blobs package to communicate with the Azure Blob Storage service. For more information see the Azure.Storage.Blobs authentication documentation.

Key concepts

The Azure Storage Common client library contains shared infrastructure like [authentication credentials][auth_credentials] and RequestFailedException.

Thread safety

We guarantee that all client instance methods are thread-safe and independent of each other (guideline). This ensures that the recommendation of reusing client instances is always safe, even across threads.

Additional concepts

Client options | Accessing the response | Long-running operations | Handling failures | Diagnostics | Mocking | Client lifetime

Examples

Examples using BlobContainerClient extension methods to upload and download directories.

Instantiate the BlobContainerClient

BlobServiceClient service = new BlobServiceClient(serviceUri, credential);

BlobContainerClient container = service.GetBlobContainerClient(containerName);

Upload a local directory to the root of the container

DataTransfer transfer = await container.StartUploadDirectoryAsync(localPath);

await transfer.AwaitCompletion();

Upload a local directory to a virtual directory in the container by specifying a directory prefix

DataTransfer transfer = await container.StartUploadDirectoryAsync(localPath, blobDirectoryPrefix);

await transfer.AwaitCompletion();

Upload a local directory to a virtual directory in the container specifying more advanced options

BlobContainerClientTransferOptions options = new BlobContainerClientTransferOptions
{
    BlobContainerOptions = new BlobStorageResourceContainerOptions
    {
        DirectoryPrefix = blobDirectoryPrefix
    },
    TransferOptions = new TransferOptions()
    {
        CreateMode = StorageResourceCreateMode.Overwrite,
    }
};

DataTransfer transfer = await container.StartUploadDirectoryAsync(localPath, options);

await transfer.AwaitCompletion();

Download the entire container to a local directory

DataTransfer transfer = await container.StartDownloadToDirectoryAsync(localDirectoryPath);

await transfer.AwaitCompletion();

Download a directory in the container by specifying a directory prefix

DataTransfer tranfer = await container.StartDownloadToDirectoryAsync(localDirectoryPath2, blobDirectoryPrefix);

await tranfer.AwaitCompletion();

Download from the container specifying more advanced options

BlobContainerClientTransferOptions options = new BlobContainerClientTransferOptions
{
    BlobContainerOptions = new BlobStorageResourceContainerOptions
    {
        DirectoryPrefix = blobDirectoryPrefix
    },
    TransferOptions = new TransferOptions()
    {
        CreateMode = StorageResourceCreateMode.Overwrite,
    }
};

DataTransfer tranfer = await container.StartDownloadToDirectoryAsync(localDirectoryPath2, options);

await tranfer.AwaitCompletion();

Examples using BlobContainerClient extension methods to upload and download directories.

Create Instance of TransferManager with Options

// Create BlobTransferManager with event handler in Options bag
TransferManagerOptions transferManagerOptions = new TransferManagerOptions();
TransferOptions options = new TransferOptions()
{
    MaximumTransferChunkSize = 4 * Constants.MB,
    CreateMode = StorageResourceCreateMode.Overwrite,
};
TransferManager transferManager = new TransferManager(transferManagerOptions);

Start Upload from Local File to Block Blob

DataTransfer dataTransfer = await transferManager.StartTransferAsync(
    sourceResource: new LocalFileStorageResource(sourceLocalPath),
    destinationResource: new BlockBlobStorageResource(destinationBlob));
await dataTransfer.AwaitCompletion();

Apply Options to Block Blob Download

await transferManager.StartTransferAsync(
    sourceResource: new BlockBlobStorageResource(sourceBlob, new BlockBlobStorageResourceOptions()
    {
        DestinationConditions = new BlobRequestConditions(){ LeaseId = "xyz" }
    }),
    destinationResource: new LocalFileStorageResource(downloadPath2));

Start Directory Upload

// Create simple transfer directory upload job which uploads the directory and the contents of that directory
DataTransfer dataTransfer = await transferManager.StartTransferAsync(
    sourceResource: new LocalDirectoryStorageResourceContainer(sourcePath),
    destinationResource: new BlobStorageResourceContainer(
        container,
        new BlobStorageResourceContainerOptions() { DirectoryPrefix = "sample-directory2" }),
    transferOptions: options);

Start Directory Download

DataTransfer downloadDirectoryJobId2 = await transferManager.StartTransferAsync(
    sourceDirectory2,
    destinationDirectory2);

Simple Logger Sample for Transfer Manager Options

// Create BlobTransferManager with event handler in Options bag
TransferManagerOptions options = new TransferManagerOptions();
TransferOptions transferOptions = new TransferOptions();
transferOptions.SingleTransferCompleted += (SingleTransferCompletedEventArgs args) =>
{
    using (StreamWriter logStream = File.AppendText(logFile))
    {
        logStream.WriteLine($"File Completed Transfer: {args.SourceResource.Path}");
    }
    return Task.CompletedTask;
};

Simple Failed Event Delegation for Container Transfer Options

transferOptions.TransferFailed += (TransferFailedEventArgs args) =>
{
    using (StreamWriter logStream = File.AppendText(logFile))
    {
        // Specifying specific resources that failed, since its a directory transfer
        // maybe only one file failed out of many
        logStream.WriteLine($"Exception occured with TransferId: {args.TransferId}," +
            $"Source Resource: {args.SourceResource.Path}, +" +
            $"Destination Resource: {args.DestinationResource.Path}," +
            $"Exception Message: {args.Exception.Message}");
    }
    return Task.CompletedTask;
};

Troubleshooting

All Azure Storage services will throw a RequestFailedException with helpful ErrorCodes.

Next steps

Get started with our Blob DataMovement samples.

Contributing

See the Storage CONTRIBUTING.md for details on building, testing, and contributing to these libraries.

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit cla.microsoft.com.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.