Amazon S3 Bucket Management with C#: Part 10 – Uploading all files in a directory recursively to an S3 Bucket

Before getting started

Skill Level: Intermediate

Assumptions:

  1. You already gone through Parts 1-9 of Managing Amazon AWS with C#.

Additional information: I sometimes cover small sub-topics in a post. Along with AWS, you will also be exposed to:

  • Rhyous.SimpleArgs
  • Single Responsibility Principle (S in S.O.L.I.D.)
  • async, await, parallelism
  • 10/100 rule

Doing things by convention.

Step 1 – Add a method to get the list of files in a local directory

This isn’t the focus of our post, however, in order to upload all files in a directory recursively, we have to be able to list them. We are going to create a method that is 10 lines of code. The method has one single repsponsibility, to return all files in a directory recursively. It is not the responsibility of BucketManager.cs to do this. Hence we need a new class that has this responsibility.

Another reason to move this method to its own file is that this method is itself 10 lines of code. While you can have methods longer than ten lines, more than ten lines is usually the first sign that the Single Responsibility principal is broken. Most beginning developers have a hard time seeing the many ways a method may be breaking the single responsibility principle. So a much easier rule, is the 10/100 rule. In the 10/100 rule, a method can only have 10 lines. This rule is pretty soft. Do brackets count? It doesn’t matter. What matters is that the 10 line mark, with or without brackets, is where you start looking at refactoring the method by splitting it into two or more smaller and simpler methods. This is a a Keep It Super Simple (K.I.S.S.) rule.

  1. Add the following utility class: FileUtils.cs.
    using System.Collections.Generic;
    using System.IO;
    using System.Linq;
    using System.Threading.Tasks;
    
    namespace Rhyous.AmazonS3BucketManager
    {
        public static class FileUtils
        {
            public static async Task<List<string>> GetFiles(string directory, bool recursive)
            {
                var files = Directory.GetFiles(directory).ToList();
                if (!recursive)
                    return files;
                var dirs = Directory.GetDirectories(directory);
                var tasks = dirs.Select(d => GetFiles(d, recursive)).ToList();
                while (tasks.Any())
                {
                    var task = await Task.WhenAny(tasks);
                    files.AddRange(task.Result);
                    tasks.Remove(task);
                }
                return files;
            }
        }
    }
    

Notice: The above class will get all files and directories, recursively. It will do it in parallel. Parallelism likely isn’t needed most of the time. Any non-parallel code that could list files and directories recursively would work. But if you were going to sync directories with tens of thousands of files each, parallelism might be a huge benefit.

Step 2 – Add an UploadFiles method to BucketManager.cs

  1. Edit file called BucketManager.cs.
  2. Enter this new method:
            public static async Task UploadFiles(TransferUtility transferUtility, string bucketName, string directory)
            {
                var files = await FileUtils.GetFiles(directory, true);
                var directoryName = Path.GetFileName(directory); // This is not a typo. GetFileName is correct.            
                var tasks = files.Select(f => UploadFile(transferUtility, bucketName, f, f.Substring(f.IndexOf(directoryName)).Replace('\\', '/')));
                await Task.WhenAll(tasks);
            }
    

Notice 1: We follow the “Don’t Repeat Yourself (DRY) principle by having UploadFiles() forward each file to the singular UploadFile().
Notice 2: We don’t use the await keyword when we redirect each file UploadFile. Instead we capture the returned Task objects and then we will await the completion of each of them.

Step 3 – Update the Action Argument

We should be very good at this by now. We need to make this method a valid action for the Action Argument.

  1. Edit the ArgsHandler.cs file to define an Action argument.
                        ...
                        AllowedValues = new ObservableCollection<string>
                        {
                            "CreateBucket",
                            "CreateBucketDirectory",
                            "CreateTextFile",
                            "DeleteBucket",
                            "DeleteBucketDirectory",
                            "ListFiles",
                            "UploadFile",
                            "UploadFiles"
                        },
                        ...
    

Note: There are enough of these now that I alphabetized them.

Step 4 – Delete the Parameter dictionary

In Part 4, we created a method to pass different parameters to different methods.We took note in Part 8 and Part 9 that we now have more exceptions than we have commonalities. It is time to refactor this.

Another reason to refactor this is because the OnArgumentsHandled method is seriously breaking the 10/100 rule.

Let’s start by deleting what we have.

  1. Delete the Dictionary line from Program.cs.
            static Dictionary<string, object[]> CustomParameters = new Dictionary<string, object[]>();
    
  2. Delete the section where we populated the dictionary.
                // Use the Custom or Common pattern
                CustomParameters.Add("CreateBucketDirectory", new object[] { s3client, bucketName, Args.Value("Directory") });
                CustomParameters.Add("CreateTextFile", new object[] { s3client, bucketName, Args.Value("Filename"), Args.Value("Text") });
                CustomParameters.Add("DeleteBucketDirectory", new object[] { s3client, bucketName, Args.Value("Directory") });
                CustomParameters.Add("DeleteFile", new object[] { transferUtility, bucketName, Args.Value("Filename") });
                CustomParameters.Add("UploadFile", new object[] { transferUtility, bucketName, Args.Value("File"), Args.Value("RemoteDirectory") });
    

Step 5 – Implement parameters by convention

To refactor the parameter passing, To refactor this, we are going use a convention.

A convention is some arbitrary rule that when followed makes the code work. You have to be very careful when using conventions because they are usually not obvious. Because they are not obvious, the first rule of using a convention is this: Conventions must be documented.

The convention is this: Make the Argument names match the method parameters. Argument names are not case sensitive, so we don’t have to worry about case. Just name.

There are two exceptions to this convention. AmazonsS3Client and TransferUtility. We will handle those exceptions statically in code.

Now, let’s implement our convention.

  • For each Argument, make sure the associated parameter is the same name.
    • Change bucketName to bucket in all methods.
    • Change file to filename in the DeleteFile method.
    • Change UploadLocation to RemoteDirectory in the UploadFile method.
    • Change directory to LocalDirectory in the UploadFiles method.
  • Create the following MethodInfoExtension.cs.
    using Amazon;
    using Amazon.S3;
    using Amazon.S3.Transfer;
    using Rhyous.SimpleArgs;
    using System;
    using System.Collections.Generic;
    using System.Configuration;
    using System.Reflection;
    
    namespace Rhyous.AmazonS3BucketManager
    {
        public static class MethodInfoExtensions
        {
            public static List<object> DynamicallyGenerateParameters(this MethodInfo mi)
            {
                var parameterInfoArray = mi.GetParameters();
                var parameters = new List<object>();
                var region = RegionEndpoint.GetBySystemName(ConfigurationManager.AppSettings["AWSRegion"]);
                foreach (var paramInfo in parameterInfoArray)
                {
                    if (paramInfo.ParameterType == typeof(AmazonS3Client) || paramInfo.ParameterType == typeof(TransferUtility))
                        parameters.Add(Activator.CreateInstance(paramInfo.ParameterType, region));
                    if (paramInfo.ParameterType == typeof(string))
                        parameters.Add(Args.Value(paramInfo.Name));
                }
    
                return parameters;
            }
        }
    }
    

    Notice this class will dynamically query the parameters. AmazonS3Client and TransferUtility are exceptions. The rest of the parameters are created using a convention and pulled from Argument values.

  • Update Program.cs to use this new extension method.
            internal static void OnArgumentsHandled()
            {
                var action = Args.Value("Action");
                var flags = BindingFlags.NonPublic | BindingFlags.Public | BindingFlags.Static | BindingFlags.FlattenHierarchy;
                MethodInfo mi = typeof(BucketManager).GetMethod(action, flags);
                List<object> parameters = mi.DynamicallyGenerateParameters();
                var task = mi.Invoke(null, parameters.ToArray()) as Task;
                task.Wait();
            }   
    

 

Notice: Look how simple Program.OnArgumentsHandled method has become. By using this convention, and by moving the parameter creation to an extension method, we are down to six lines. The total size for the Program.cs class is 25 lines, including spaces.

You can now move a directory to an Amazon S3 bucket using C#.

<h3>Design Pattern: Facade</h3>

Yes, we have just implement the popular Facade design pattern.

Our project, and most specifically BucketManger.cs, represent an entire system: Amazon S3. When code is written to represent an entire system or substem, that code is called a Facade.

Go to: Rhyous.AmazonS3BucketManager on GitHub to see the full example project from this 10 part tutorial.

Return to: Managing Amazon AWS with C#

Amazon S3 Bucket Management with C#: Part 9 – Uploading a file with its path to a Bucket

Before getting started

Skill Level: Beginner

Assumptions:

  1. You already gone through Parts 1-8 of Managing Amazon AWS with C#.

Additional information: I sometimes cover small sub-topics in a post. Along with AWS, you will also be exposed to:

  • Rhyous.SimpleArgs

Step 1 – Alter the existing UploadFile method in BucketManager.cs

We need the UploadFile method to take in a parameter that specifies the remote directory, which is the directory path on the S3 bucket. However, if no directory is specified, the key should simply be the file name.

  1. Edit file called BucketManager.cs.
  2. Enter this new method:
    Note: We are in luck, the TransferUtility object has an overload that takes in the key.

            public static async Task UploadFile(TransferUtility transferUtility, string bucketName, string file, string uploadLocation = null)
            {
                var key = Path.GetFileName(file);
                if (!string.IsNullOrWhiteSpace(uploadLocation))
                {
                    uploadLocation = uploadLocation.EndsWith("/") ? uploadLocation : uploadLocation + "/";
                    key = uploadLocation + key;
                }
                await Task.Run(() => transferUtility.Upload(file, bucketName, key));
            }
    

Note: This method is already added to the Action Argument, so we don’t need to update it.

Step 2 – Add a RemoteDirectory Argument

If we are going to upload a file to a specific location, we should know what that specific location is. So add an Argument for it.

  1. Add the following argument to ArgsHandler.cs.
                    ...
                    new Argument
                    {
                        Name = "RemoteDirectory",
                        ShortName = "rd",
                        Description = "The remote directory on the S3 Bucket.",
                        Example = "{name}=My/Remote/Directory",
                        Action = (value) =>
                        {
                            Console.WriteLine(value);
                        }
                    }
                    ...
    

Step 4 – Update the parameter array passed to UploadFile

We’ve already create a custom parameter array for the UploadFile action. We simply need to add a method for it

            // Use the Custom or Common pattern
            CustomParameters.Add("CreateBucketDirectory", new object[] { s3client, bucketName, Args.Value("Directory") });
            CustomParameters.Add("CreateTextFile", new object[] { s3client, bucketName, Args.Value("Filename"), Args.Value("Text") });
            CustomParameters.Add("DeleteBucketDirectory", new object[] { s3client, bucketName, Args.Value("Directory") });
            CustomParameters.Add("DeleteFile", new object[] { transferUtility, bucketName, Args.Value("Filename") });
            CustomParameters.Add("UploadFile", new object[] { transferUtility, bucketName, Args.Value("File"), Args.Value("RemoteDirectory") });

Note: There are enough of these now that I alphabetized them.

You can now specify the remote directory when uploading a file.

Go to: Part 10 – Uploading all files in a directory recursively to an S3 Bucket

Return to: Managing Amazon AWS with C#

Amazon S3 Bucket Management with C#: Part 8 – Deleting a file in a Bucket

Before getting started

Skill Level: Beginner

Assumptions:

  1. You already gone through Parts 1-7 of Managing Amazon AWS with C#.

Additional information: I sometimes cover small sub-topics in a post. Along with AWS, you will also be exposed to:

  • Rhyous.SimpleArgs
  • Don’t Repeat Yourself (DRY) Principal

Step 1 – Add a DeleteFile method to BucketManager.cs

  1. Edit file called BucketManager.cs.
  2. Enter this new method:
            public static async Task CreateTextFile(AmazonS3Client client, string bucketName, string filename, string text)
            {
                var dirRequest = new PutObjectRequest
                {
                    BucketName = bucketName,
                    Key = filename,
                    InputStream = text.ToStream()
                };
                await client.PutObjectAsync(dirRequest);
                Console.WriteLine($"Created text file in S3 bucket: {bucketName}/{filename}");
            }
    

Notice: The code is almost identical to that of deleting a directory, with only one exception. We aren’t ending with a /. We really should not have duplicate code. So lets fix this in the next step.

Step 2 – Solve the Repetitive Code

It is best practice to avoid having duplicate code. This is often called the “Don’t Repeat Yourself” principal. So let’s update the DeleteBucketDirectory code to forward to the DeleteFile code.

  1. Update the DeleteDirectory method so that both methods share code.
           public static async Task DeleteBucketDirectory(AmazonS3Client client, string bucketName, string directory)
            {
                if (!directory.EndsWith("/"))
                    directory = directory += "/";
                await DeleteFile(client, bucketName, directory);
            }
    

Now the delete directory code is no longer repetitive. A directory is the same as a file, just with a slash. So the Delete directory correctly makes sure that the directory name ends with a slash, then forwards the call to delete file.

Step 3 – Update the Action Argument

We should be very good at this by now. We need to make this method a valid action for the Action Argument.

  1. Edit the ArgsHandler.cs file to define an Action argument.
                        ...
                        AllowedValues = new ObservableCollection<string>
                        {
                            "CreateBucket",
                            "DeleteBucket",
                            "ListFiles",
                            "UploadFile",
                            "CreateBucketDirectory",
                            "DeleteBucketDirectory",
                            "CreateTextFile",
                            "DeleteFile"
                        },
                        ...
    

Note: There are no additional Arguments to add. To delete a file, we need the bucket name and a file name, which we already have Arguments for.

Step 4 – Fix the parameter mismatch problem

In Part 4, we created a method to pass different parameters to different methods. Let’s use that to pass in the correct parameters.

However, take note that we now have more exceptions than we had commonalities. This suggests that it is about time to refactor this code. For now, we will leave it.

            // Use the Custom or Common pattern
            CustomParameters.Add("UploadFile", new object[] { transferUtility, bucketName, Args.Value("File") });
            CustomParameters.Add("CreateBucketDirectory", new object[] { s3client, bucketName, Args.Value("Directory") });
            CustomParameters.Add("DeleteBucketDirectory", new object[] { s3client, bucketName, Args.Value("Directory") });
            CustomParameters.Add("CreateTextFile", new object[] { s3client, bucketName, Args.Value("Filename"), Args.Value("Text") });

You can now add a text file to an Amazon S3 bucket using C#.

Homework: There is some repetitiveness between CreateFolder and DeleteFolder. What is it? (Hint: Directories end with a slash.)

Go to: Part 9 – Uploading a file with its path to a Bucket

Return to: Managing Amazon AWS with C#

Amazon S3 Bucket Management with C#: Part 7 – Creating a text file in a Bucket

Before getting started

Skill Level: Beginner

Assumptions:

  1. You already gone through Parts 1-6 of Managing Amazon AWS with C#.

Additional information: I sometimes cover small sub-topics in a post. Along with AWS, you will also be exposed to:

  • Rhyous.SimpleArgs
  • Rhyous.StringLibrary

Step 1 – Add NuGet Package

  1. Right-click on your project and choose Management NuGet Packages.
  2. Search for Rhyous.StringLibrary.
    This is a simple library for string extensions methods and more. String code that is not in .Net by default, yet the methods have proven over time to be commonly used.
  3. Install the Rhyous.StringLibrary NuGet package.

Step 2 – Add a CreateTextFile method to BucketManager.cs

  1. Edit file called BucketManager.cs.
  2. Enter this new method:
            public static async Task CreateTextFile(AmazonS3Client client, string bucketName, string filename, string text)
            {
                var dirRequest = new PutObjectRequest
                {
                    BucketName = bucketName,
                    Key = filename,
                    InputStream = text.ToStream()
                };
                await client.PutObjectAsync(dirRequest);
                Console.WriteLine($"Created text file in S3 bucket: {bucketName}/{filename}");
            }
    

Notice: The code is almost identical to that of creating a directory, with two exceptions. We aren’t ending with a /. And instead of assigning zero bytes to InputStream, we assigned text.ToStream(). Rhyous.StringLibrary provides us the ToStream() extension method.

Step 2 – Update the Action Argument

We now need to make this method a valid action for the Action Argument.

  1. Edit the ArgsHandler.cs file to define an Action argument.
                        ...
                        AllowedValues = new ObservableCollection&amp;amp;lt;string&amp;amp;gt;
                        {
                            "CreateBucket",
                            "DeleteBucket",
                            "ListFiles",
                            "UploadFile",
                            "CreateBucketDirectory",
                            "DeleteBucketDirectory",
                            "CreateTextFile",
                        },
                        ...
    

Step 3 – Add FileName and Text Arguments

If we are going to create a text file, we need to know the file name and the text to insert.

  1. Edit the ArgsHandler.cs file to define an Action argument.
                    ...
                    new Argument
                    {
                        Name = "FileName",
                        ShortName = "N",
                        Description = "The name of text a file to create.",
                        Example = "{name}=MyTextfile.txt",
                        Action = (value) =>
                        {
                            Console.WriteLine(value);
                        }
                    },
                    new Argument
                    {
                        Name = "Text",
                        ShortName = "T",
                        Description = "The text to put in a text file.",
                        Example = "{name}=\"This is some text!\"",
                        Action = (value) =>
                        {
                            Console.WriteLine(value);
                        }
                    }
                    ...
    

Step 4 – Fix the parameter mismatch problem

In Part 4, we created a method to pass different parameters to different methods. Let’s use that to pass in the correct parameters.

However, take note that we now have more exceptions than we had commonalities. This suggests that it is about time to refactor this code. For now, we will leave it.

            // Use the Custom or Common pattern
            CustomParameters.Add("UploadFile", new object[] { transferUtility, bucketName, Args.Value("File") });
            CustomParameters.Add("CreateBucketDirectory", new object[] { s3client, bucketName, Args.Value("Directory") });
            CustomParameters.Add("DeleteBucketDirectory", new object[] { s3client, bucketName, Args.Value("Directory") });
            CustomParameters.Add("CreateTextFile", new object[] { s3client, bucketName, Args.Value("Filename"), Args.Value("Text") });

You can now add a text file to an Amazon S3 bucket using C#.

Go to: Deleting a file in a Bucket

Return to: Managing Amazon AWS with C#

Amazon S3 Bucket Management with C#: Part 6 – Deleting a Directory in a Bucket

Before getting started

Skill Level: Beginner

Assumptions:

  1. You already gone through Parts 1-5 of Managing Amazon AWS with C#.

Additional information: I sometimes cover small sub-topics in a post. Along with AWS, you will also be exposed to:

  • Rhyous.SimpleArgs

Step 1 – Add a DeleteBucketDirectory method to BucketManager.cs

  1. Edit file called BucketManager.cs.
  2. Enter this new method:
            public static async Task DeleteBucketDirectory(AmazonS3Client client, string bucketName, string directory)
            {
                var dirRequest = new DeleteObjectRequest
                {
                    BucketName = bucketName,
                    Key = directory + "/"
                };
                await client.DeleteObjectAsync(dirRequest);
                Console.WriteLine($"Created S3 bucket folder: {bucketName}/{directory}/");
            }
    

Note: Amazon S3 uses objects with a key ending in a / as a directory, so we have to call DeleteObjectAsync.

Step 2 – Update the Action Argument

We now need to make this method a valid action for the Action Argument.

  1. Edit the ArgsHandler.cs file to define an Action argument.
                        ...
                        AllowedValues = new ObservableCollection&amp;lt;string&amp;gt;
                        {
                            "CreateBucket",
                            "DeleteBucket",
                            "ListFiles",
                            "UploadFile",
                            "CreateBucketDirectory",
                            "DeleteBucketDirectory"
                        },
                        ...
    

Step 3 – Fix the parameter mismatch problem

In Part 4, we created a method to pass different parameters to different methods. Let’s use that to pass in the correct parameters.

            // Use the Custom or Common pattern
            CustomParameters.Add("UploadFile", new object[] { transferUtility, bucketName, Args.Value("File") });
            CustomParameters.Add("CreateBucketDirectory", new object[] { s3client, bucketName, Args.Value("Directory") });
            CustomParameters.Add("DeleteBucketDirectory", new object[] { s3client, bucketName, Args.Value("Directory") });

You can now Delete a directory on S3, using C#.

Go to:

Return to: Managing Amazon AWS with C#