Upload files larger than 4 MB in chunks using Azure blob storage

How to upload files larger than 4MB in size into chunks to Windows Azure Storage using PutBlockList method.

 Windows Azure   
 

Posted: October 25, 2017. | By: mustafa

In the previous article we have seen how to upload file using Azure blob storage. However it had a major limitation of being able to handle files of sizes which is less than the max size handled by IIS (by default 4MB).

Today I am going to explain how to files larger than 4 MB in Windows Azure Storage. Azure blob storage allows us to store any unstructured data file such as text, images, video, etc.

The idea is to split a block blob into smaller chunk of blocks, uploading them one-by-one or in-parallel and lastly combine them all into single block by calling PutBlockList().

 

Define internal class model as FileBlock.

internal class FileBlock
 {
 public string Id
 {
 get;
 set;
 }
 public byte[] Content
 {
 get;
 set;
 }
 }
public Uri UploadBlob(byte[] fileContent, string blobName)
 {
 CloudBlobContainer container = cloudBlobClient.GetContainerReference(ContainerName);
 //container.CreateIfNotExist();
 CloudBlockBlob blob = container.GetBlockBlobReference(blobName);
 HashSet<string> blocklist = new HashSet<string>();
 foreach (FileBlock block in GetFileBlocks(fileContent))
 {
 blob.PutBlock(
 block.Id,
 new MemoryStream(block.Content, true),
 null
 );
 blocklist.Add(block.Id);
 }

 blob.PutBlockList(blocklist);
 return blob.Uri;
 }
 private IEnumerable<FileBlock> GetFileBlocks(byte[] fileContent)
 {
 HashSet<FileBlock> hashSet = new HashSet<FileBlock>();
 //check if fileContent length is 0.
 if (fileContent.Length == 0)
 return new HashSet<FileBlock>();
 int blockId = 0;
 int index = 0;
 int currentBlockSize = MaxBlockSize;
 while (currentBlockSize == MaxBlockSize)
 {
 
 //check if currentBlock length is greater than total fileContent length.
 if ((index + currentBlockSize) > fileContent.Length)
 currentBlockSize = fileContent.Length - index;
 
 byte[] chunk = new byte[currentBlockSize];

 Array.Copy(fileContent, index, chunk, 0, currentBlockSize);

 hashSet.Add(
 new FileBlock()
 {
 Content = chunk,
 Id = Convert.ToBase64String(System.BitConverter.GetBytes(blockId)) //add block id in form of base64 string.
 });

 index += currentBlockSize;
 blockId++;
 }
 return hashSet;
 }

As we have to get the file blocks lesser than 4 MB in size. We are going to set upper limit 2 MB to split the file to be able to uploaded easily through IIS.

private const int MaxBlockSize = 4 * 1024 *1024; // Approx. 4MB chunk size

 

While using foreach loop we are going to perform upload by calling blob.PutBlock() and adding block id into blocklist as base64 string to determine the blocks are in one group. Calling PutListBlock() will commit all the blocks that we’ve uploaded previously.

 

Thanks for reading, enjoy uploading files to Windows azure cloud storage.


related posts

Back to top