Archive for the ‘csharp’ Category.

Reuseable Building Block Development

If you haven’t paid attention to the development world, you might have missed the current movement called “Reuseable Building Block development.” You know, as a kid, we could get blocks are build anything with them. We only had to stack them. Well, having a n-tier stack is very common, now, so stacking isn’t the issue. It is having blocks that are easy to stack. Some are calling it the open source movement, and while most reusable building blocks are open source, not all of them are. Many of the building blocks don’t have to be open source, but can simply be well-documented and work well.

With NuGet and Npm, building blocks are being created and published daily. The problem now is helping other developers recognize this movement. Changing our mindset from, “we can’t use if it wasn’t invented here,” to something more like, “this is our unique stack of building blocks for a unique problem and this stack was invented here.”

I have created a bunch of building blocks for C#. Check out my github account at https://github.com/rhyous. You will see a few reusable building blocks:

  • Rhyous.Collections – You know all those pesky extension methods your write for collections that are missing from the collections or from linq. I have a lot of them in here.
  • Rhyous.EasyCsv – A simple tool for working with csv files.
  • Rhyous.EasyXml – A simpel tool for working with Xml. (You might ask why I don’t have one for JSON, and that is because Newtonsoft.Json and fast.jsona already exist , so another one isn’t needed.)
  • Rhyous.EntityAnywhere – Wow, have a full rest api and only have to create the model class. Are you kidding, this is probably the coolest project for Web Service APIs since the REST pattern was introduced.Rhyous.SimplePluginLoader – Easily load plugins in your app.
  • Rhyous.SimpleArgs – Writing a tool with command line arguments? This tool allows you to configure your arguments in a model class and be done. It will output usage and force required parameters and allow for events when a parameter is set, etc.
  • Rhyous.StringLibrary – You know all those pesky extension methods you write for string manipulations missing from .NET Framework. They are in this library, along with a pluralization tool. Every heard of the The oft forgotten Middle Trim, well, it is in this library, too.
  • WPFSharp.Globalizer – The best localization library for WPF that exists, allowing you to change language and style (including left to right flow for certain languages) at runtime.

I actually have many more building blocks. Take a look.

DateTime Within Extension Method

I wrote and extension method to DateTime today. I want to call something simple to see if one date is within a two days of another date. There isn’t a within method. I set out to create one and this what I came up with.

Let me know what you think.

using System;
using System.Collections.Generic;

namespace Rhyous.WebFramework.Handlers.Extensions
{
    public enum DateTimeInterval
    {
        Miliseconds,
        Seconds,
        Minutes,
        Hours,
        Days,
        Weeks,
        Months,
        Years
    }

    internal class DateTimeIntervalActionDictionary : Dictionary<DateTimeInterval, Func<double, TimeSpan>>
    {
        #region Singleton

        private static readonly Lazy<DateTimeIntervalActionDictionary> Lazy = new Lazy<DateTimeIntervalActionDictionary>(() => new DateTimeIntervalActionDictionary());

        public static DateTimeIntervalActionDictionary Instance { get { return Lazy.Value; } }

        internal DateTimeIntervalActionDictionary()
        {
            Add(DateTimeInterval.Miliseconds, TimeSpan.FromMilliseconds);
            Add(DateTimeInterval.Seconds, TimeSpan.FromSeconds);
            Add(DateTimeInterval.Minutes, TimeSpan.FromMinutes);
            Add(DateTimeInterval.Hours, TimeSpan.FromHours);
            Add(DateTimeInterval.Days, TimeSpan.FromDays);
            Add(DateTimeInterval.Weeks, (double d) => { return TimeSpan.FromDays(d * 7); });
            Add(DateTimeInterval.Months, (double d) => { return TimeSpan.FromDays(d * 30); });
            Add(DateTimeInterval.Years, (double d) => { return TimeSpan.FromDays(d * 365); });
        }

        #endregion
    }

    public static class DateExtensions
    {
        public static bool IsWithin(this DateTime dateTime, double interval, DateTimeInterval intervalType, DateTime comparisonDateTime)
        {
            TimeSpan allowedDiff = DateTimeIntervalActionDictionary.Instance[intervalType].Invoke(interval);
            TimeSpan diff = dateTime - comparisonDateTime;
            return allowedDiff <= diff;
        }
    }
}

The problems with the Tiobe Index in regards to .Net

The Tiobe index is really missing one piece of information about .Net for its users. Java is #1. So users should use Java, right? Well, maybe not. Let’s talk about the problems with it before we move on.

I am going to make an argument that:

  1. Java is actually a more clear #1 than suggested.
  2. .Net is #2 behind Java, but not as far behind as the Tiobe index makes it appear.

Problem 1 – DotNet Framework is not listed as one a language

.Net has more languages writing against it than just one. That makes it appear less popular because the language is more fragmented. In fact, two of them are in the top 5 or 6. However, the fact that a dll compiled in either language can be consumed by either language is really not described here. I am not saying this should be on the same list of programming languages, but Tiobe should make it clear that the combined .Net languages show .Net as being used more heavily. Similary for Java, there are other languages that compile to the JVM. Perhaps there should be a page on compile target: What percent of languages compile to .Net’s Common Intermediary Language or compile to the Java Virtual Machine or to machine code or don’t compile at all?

As for intermediary languages, there are only two that stand out: Java and .Net. And Java is #1 but it only has 1 in the top 10. .Net has two in the top 10 and the combined languages are easily a rival to the combined JVM languages.

Look at the Tiobe index and add up the .Net Framework languages:

.Net Framework Languages

Language2019 Tiobe Rating
Visual Basic .Net5.795%
C#3.515%
F#0.206%
Total9.516%

Notice that combined, the number of the three main .Net languages is %9.516. That puts .Net in the #3 position behind Java, C, and C++.

Problem 2 – Some .Net languages are missing and may be lumped in other languages

What about Visual C++? Yes, you can write .Net code in C++. However, that language is completely missing from Tiobe. Or is it? Is all Visual C++ searches lumped in with C++? If so, shouldn’t Visual C++ be separate out from C++. What is the Tiobe raiting Visual C++ would get? That would be hard to guess. But it is a language has been around for almost two decades. Let’s assume that a certain percentage of C++ developers are actually doing Visual C++. Let’s say it is more than F# but quite a lot less than C#. Let’s just guess because unlike Tiobe, I don’t have have this data. Let’ say it was .750. Again, this is a wild guess. Perhaps Tiobe could comment on this, perhaps they couldn’t find data on it themselves.

.Net Framework Languages

Language2019 Tiobe Rating
Visual Basic .Net5.795%
C#3.515%
F#0.206%
F#0.206%
Total10.266%

As you see, .Net combined is clearly #3 just by combining the .Net languages. Well past Python, which in fact can be used to both code for .Net (IronPython) and for the Java JVM (Jython). What percent of python is used for that?

Here is a wikipedia list of .Net-based languages: https://en.wikipedia.org/wiki/List_of_CLI_languages.

Similarly, for Java, languages like Groovy up it’s score. Here is a wikipedia list of Jvm-based languages: https://en.wikipedia.org/wiki/List_of_JVM_languages.

Problem 3 – Visual Studio is Awesome

For all the problems and complaints of bloat, Visual Studio is the move feature rich IDE by such a long ways that I doubt any other IDE will ever catch up to it, except may Visual Studio Code, which, however, is just as much part of the Tiobe index problem as Visual Studio is.

The better the tool, the less web searching is needed. The breadth of the features in Visual Studio is staggering. The snippets, the Intellisense, the ability to browse and view and even decompile existing code means that .Net developers are not browsing the web as often as other languages. My first search always happens in Intellisense in Visual Studio, not on Google. The same features and tooling in other IDEs for other languages just isn’t there. Maybe Exclipse, but only with hundreds for plugins that most developers don’t know about.

After Visual Studio 2012 released, the need to search the web has decreased with every single release of Visual Studio. I am claiming that C#, which is the primary .Net Framework language microsoft code for in Visual Studio, is used far more than Visual Basic .Net. Tiobe has Visual Basic .Net at 5.795% and C# at 3.515%, but reality doesn’t match Tiobe’s statististics. C# is used far more than Visual Basic .Net.

I am making the hypothesis that as the primarily coded language in Visual Studio, C# would appear to go down in the Tiobe index since the release of Visual Studio 2012. Let’s test my hypothesis by looking at the Tiobe year-by-year chart for C#. Do we see the Tiobe index going down starting with the release of VS 2012?

After looking at the Tiobe index, I am upgrading my claim from a hypothesis to a theory.

Other .Net languages may not experience the same as C# as the tooling in .Net is primarily focussed around C#.

So the reality is that the Tiobe index is showing the data it can find from search engines, but the data for C# is just not there because a lot of the number of ways C# deflects the need to search.

I hypothesise that C# reached a peak Tiobe index of 8.763% and it’s usage has not actually gone down. Instead, it has gone up. However, the data doesn’t exist to prove it. Assuming the hypothesis is correct, and C# usage has gone up, then the rate it should be is closer to 9 or 10. That means the C# is probably #3 on it’s own.

If we adjust to take this problem into account, simply by using the 2012 index and not assuming the the usage rate has gone up, we see the following:

.Net Framework Languages

Language2019 Tiobe Rating
Visual Basic .Net5.795%
C#8.7%
F#0.206%
F#0.206%
Total17.606%

Now, I am not saying .Net is above Java with my hypothesized 17.505% adjusted rating. Java has other languages as well that compile to the JVM that would similarly raise it and it is still #1.

Problem 4 – Direct linking to or searching on Microsoft.com

Microsoft has done a great job with a lot of their documentation. Some of this could be attributed to Visual Studio as well. After clicking a link in Visual Studio, we are taking directly to a site like https://msdn.microsft.com where I do a lot of my language searches.

Also, Microsoft has built a community where customers can ask questions and get data.

Tiobe has a nice document that clearly states which search enginers did not qualify and what the reason they didn’t qualify was.

  • Microsoft.com: NO_COUNTERS

See: https://www.tiobe.com/tiobe-index/programming-languages-definition/

I would argue that a significant amount of searches for .Net languages are done primarily on Microsoft.com. I can only provide personal data. I often go directly to the source documentation on Microsoft.com and search on Microsoft’s site. And once I am there almost all further searches for .Net data occur there.

Microsoft has more C# developers in their company that many programming languages have world wide. Are they doing web searches through the list of qualified search engines?

Problem 5 – Better documentation

I hypothesize that the better the documentation, the less searching on the web is required. I also hypothesize that Microsoft is one of the best at providing documentation for it’s languages.

Because the documentation for .Net framework is so excellent, the question is usually answered in a single search instead of multiple searches that languages that are less well documented may require.

Problem 6 – Education

Colleges are teaching certain languages. Python and C++ are top languages taught in college. I would estimate that because of these, the languages primarily taught in college have far higher good search rates. Unfortunately, .Net languages, because of their former proprietary nature (which is no longer the case with the open source of .Net Core), were shunned by colleges.

It would be interesting to filter out searches by college students. Unfortunately, how would Tiobe know that a search came from a college student or not.

Problem 7 – Limited Verbage

Tiobe is only looking at certain words. The words that are being queried are:

  • C#: C#, C-Sharp, C Sharp, CSharp, CSharp.NET, C#.NET

Further, Tiobe says:

The ratings are calculated by counting hits of the most popular search engines. The search query that is used is

+"&lt;language&gt; programming"

This problem piggy backs on Problems 3, 4, and 5. Visual Studio is so awesome, that we know exactly what we are looking for. As a C# developer, I don’t type C# into my searches hardly at all. I type something like: WebApi, WCF, WPF, System.Net.Http or Entity Framework or LINQ, Xamarin, and many other seaches. Microsoft documentation is so clear and specific (Problem 5) that we can do highly specific searches without including the word C#.

Yes, other languages have libraries, too, but do other languages have Microsoft’s marketing department that brands libraries with trademarks and logos and makes that brand the goto phrase to search? I don’t think there is a single other programming language other than C# that does this. Microsoft is lowing the web searches for C# by their awesome marketing.

This is further evidence to explain why the actual usage of C# has gone way up while the Tiobe index has gone way down. Asp.Net, Ado.Net, Razor, WCF, WebApi, WPF, WF,  etc. What other language has logos and brands around specific parts of a language?

Problem 8 – Is C# always seen as C# in search engines

I don’t always add C# to my google searches. However, when I do, it is somehow changed to just C. The sharp symbol, #, is often removed. This recently stopped happening on Google, but it used to happen with every search in every browser. It was frustrating.

Has this been addressed in search engine stats?

Conclusion

The belief that C# is in the 3% range is an unfortunate error of circumstances. And .Net should be looked at is the second most important tool for a programmer, second only to Java, and above all other programming languages.

 

 

Amazon S3 Bucket Management with C#: Part 10 – Uploading all files in a directory recursively to an S3 Bucket

Before getting started

Skill Level: Intermediate

Assumptions:

  1. You already gone through Parts 1-9 of Managing Amazon AWS with C#.

Additional information: I sometimes cover small sub-topics in a post. Along with AWS, you will also be exposed to:

  • Rhyous.SimpleArgs
  • Single Responsibility Principle (S in S.O.L.I.D.)
  • async, await, parallelism
  • 10/100 rule

Doing things by convention.

Step 1 – Add a method to get the list of files in a local directory

This isn’t the focus of our post, however, in order to upload all files in a directory recursively, we have to be able to list them. We are going to create a method that is 10 lines of code. The method has one single repsponsibility, to return all files in a directory recursively. It is not the responsibility of BucketManager.cs to do this. Hence we need a new class that has this responsibility.

Another reason to move this method to its own file is that this method is itself 10 lines of code. While you can have methods longer than ten lines, more than ten lines is usually the first sign that the Single Responsibility principal is broken. Most beginning developers have a hard time seeing the many ways a method may be breaking the single responsibility principle. So a much easier rule, is the 10/100 rule. In the 10/100 rule, a method can only have 10 lines. This rule is pretty soft. Do brackets count? It doesn’t matter. What matters is that the 10 line mark, with or without brackets, is where you start looking at refactoring the method by splitting it into two or more smaller and simpler methods. This is a a Keep It Super Simple (K.I.S.S.) rule.

  1. Add the following utility class: FileUtils.cs.
    using System.Collections.Generic;
    using System.IO;
    using System.Linq;
    using System.Threading.Tasks;
    
    namespace Rhyous.AmazonS3BucketManager
    {
        public static class FileUtils
        {
            public static async Task&amp;lt;List&amp;lt;string&amp;gt;&amp;gt; GetFiles(string directory, bool recursive)
            {
                var files = Directory.GetFiles(directory).ToList();
                if (!recursive)
                    return files;
                var dirs = Directory.GetDirectories(directory);
                var tasks = dirs.Select(d =&amp;gt; GetFiles(d, recursive)).ToList();
                while (tasks.Any())
                {
                    var task = await Task.WhenAny(tasks);
                    files.AddRange(task.Result);
                    tasks.Remove(task);
                }
                return files;
            }
        }
    }
    

Notice: The above class will get all files and directories, recursively. It will do it in parallel. Parallelism likely isn’t needed most of the time. Any non-parallel code that could list files and directories recursively would work. But if you were going to sync directories with tens of thousands of files each, parallelism might be a huge benefit.

Step 2 – Add an UploadFiles method to BucketManager.cs

  1. Edit file called BucketManager.cs.
  2. Enter this new method:
            public static async Task UploadFiles(TransferUtility transferUtility, string bucketName, string directory)
            {
                var files = await FileUtils.GetFiles(directory, true);
                var directoryName = Path.GetFileName(directory); // This is not a typo. GetFileName is correct.            
                var tasks = files.Select(f =&amp;gt; UploadFile(transferUtility, bucketName, f, f.Substring(f.IndexOf(directoryName)).Replace('\\', '/')));
                await Task.WhenAll(tasks);
            }
    

Notice 1: We follow the “Don’t Repeat Yourself (DRY) principle by having UploadFiles() forward each file to the singular UploadFile().
Notice 2: We don’t use the await keyword when we redirect each file UploadFile. Instead we capture the returned Task objects and then we will await the completion of each of them.

Step 3 – Update the Action Argument

We should be very good at this by now. We need to make this method a valid action for the Action Argument.

  1. Edit the ArgsHandler.cs file to define an Action argument.
                        ...
                        AllowedValues = new ObservableCollection&amp;lt;string&amp;gt;
                        {
                            "CreateBucket",
                            "CreateBucketDirectory",
                            "CreateTextFile",
                            "DeleteBucket",
                            "DeleteBucketDirectory",
                            "ListFiles",
                            "UploadFile",
                            "UploadFiles"
                        },
                        ...
    

Note: There are enough of these now that I alphabetized them.

Step 4 – Delete the Parameter dictionary

In Part 4, we created a method to pass different parameters to different methods.We took note in Part 8 and Part 9 that we now have more exceptions than we have commonalities. It is time to refactor this.

Another reason to refactor this is because the OnArgumentsHandled method is seriously breaking the 10/100 rule.

Let’s start by deleting what we have.

  1. Delete the Dictionary line from Program.cs.
            static Dictionary&amp;lt;string, object[]&amp;gt; CustomParameters = new Dictionary&amp;lt;string, object[]&amp;gt;();
    
  2. Delete the section where we populated the dictionary.
                // Use the Custom or Common pattern
                CustomParameters.Add("CreateBucketDirectory", new object[] { s3client, bucketName, Args.Value("Directory") });
                CustomParameters.Add("CreateTextFile", new object[] { s3client, bucketName, Args.Value("Filename"), Args.Value("Text") });
                CustomParameters.Add("DeleteBucketDirectory", new object[] { s3client, bucketName, Args.Value("Directory") });
                CustomParameters.Add("DeleteFile", new object[] { transferUtility, bucketName, Args.Value("Filename") });
                CustomParameters.Add("UploadFile", new object[] { transferUtility, bucketName, Args.Value("File"), Args.Value("RemoteDirectory") });
    

Step 5 – Implement parameters by convention

To refactor the parameter passing, To refactor this, we are going use a convention.

A convention is some arbitrary rule that when followed makes the code work. You have to be very careful when using conventions because they are usually not obvious. Because they are not obvious, the first rule of using a convention is this: Conventions must be documented.

The convention is this: Make the Argument names match the method parameters. Argument names are not case sensitive, so we don’t have to worry about case. Just name.

There are two exceptions to this convention. AmazonsS3Client and TransferUtility. We will handle those exceptions statically in code.

Now, let’s implement our convention.

  • For each Argument, make sure the associated parameter is the same name.
    • Change bucketName to bucket in all methods.
    • Change file to filename in the DeleteFile method.
    • Change UploadLocation to RemoteDirectory in the UploadFile method.
    • Change directory to LocalDirectory in the UploadFiles method.
  • Create the following MethodInfoExtension.cs.
    using Amazon;
    using Amazon.S3;
    using Amazon.S3.Transfer;
    using Rhyous.SimpleArgs;
    using System;
    using System.Collections.Generic;
    using System.Configuration;
    using System.Reflection;
    
    namespace Rhyous.AmazonS3BucketManager
    {
        public static class MethodInfoExtensions
        {
            public static List&amp;lt;object&amp;gt; DynamicallyGenerateParameters(this MethodInfo mi)
            {
                var parameterInfoArray = mi.GetParameters();
                var parameters = new List&amp;lt;object&amp;gt;();
                var region = RegionEndpoint.GetBySystemName(ConfigurationManager.AppSettings["AWSRegion"]);
                foreach (var paramInfo in parameterInfoArray)
                {
                    if (paramInfo.ParameterType == typeof(AmazonS3Client) || paramInfo.ParameterType == typeof(TransferUtility))
                        parameters.Add(Activator.CreateInstance(paramInfo.ParameterType, region));
                    if (paramInfo.ParameterType == typeof(string))
                        parameters.Add(Args.Value(paramInfo.Name));
                }
    
                return parameters;
            }
        }
    }
    

    Notice this class will dynamically query the parameters. AmazonS3Client and TransferUtility are exceptions. The rest of the parameters are created using a convention and pulled from Argument values.

  • Update Program.cs to use this new extension method.
            internal static void OnArgumentsHandled()
            {
                var action = Args.Value("Action");
                var flags = BindingFlags.NonPublic | BindingFlags.Public | BindingFlags.Static | BindingFlags.FlattenHierarchy;
                MethodInfo mi = typeof(BucketManager).GetMethod(action, flags);
                List&amp;lt;object&amp;gt; parameters = mi.DynamicallyGenerateParameters();
                var task = mi.Invoke(null, parameters.ToArray()) as Task;
                task.Wait();
            }   
    

 

Notice: Look how simple Program.OnArgumentsHandled method has become. By using this convention, and by moving the parameter creation to an extension method, we are down to six lines. The total size for the Program.cs class is 25 lines, including spaces.

You can now move a directory to an Amazon S3 bucket using C#.

<h3>Design Pattern: Facade</h3>

Yes, we have just implement the popular Facade design pattern.

Our project, and most specifically BucketManger.cs, represent an entire system: Amazon S3. When code is written to represent an entire system or substem, that code is called a Facade.

Go to: Rhyous.AmazonS3BucketManager on GitHub to see the full example project from this 10 part tutorial.

Return to: Managing Amazon AWS with C#

Amazon S3 Bucket Management with C#: Part 9 – Uploading a file with its path to a Bucket

Before getting started

Skill Level: Beginner

Assumptions:

  1. You already gone through Parts 1-8 of Managing Amazon AWS with C#.

Additional information: I sometimes cover small sub-topics in a post. Along with AWS, you will also be exposed to:

  • Rhyous.SimpleArgs

Step 1 – Alter the existing UploadFile method in BucketManager.cs

We need the UploadFile method to take in a parameter that specifies the remote directory, which is the directory path on the S3 bucket. However, if no directory is specified, the key should simply be the file name.

  1. Edit file called BucketManager.cs.
  2. Enter this new method:
    Note: We are in luck, the TransferUtility object has an overload that takes in the key.

            public static async Task UploadFile(TransferUtility transferUtility, string bucketName, string file, string uploadLocation = null)
            {
                var key = Path.GetFileName(file);
                if (!string.IsNullOrWhiteSpace(uploadLocation))
                {
                    uploadLocation = uploadLocation.EndsWith("/") ? uploadLocation : uploadLocation + "/";
                    key = uploadLocation + key;
                }
                await Task.Run(() => transferUtility.Upload(file, bucketName, key));
            }
    

Note: This method is already added to the Action Argument, so we don’t need to update it.

Step 2 – Add a RemoteDirectory Argument

If we are going to upload a file to a specific location, we should know what that specific location is. So add an Argument for it.

  1. Add the following argument to ArgsHandler.cs.
                    ...
                    new Argument
                    {
                        Name = "RemoteDirectory",
                        ShortName = "rd",
                        Description = "The remote directory on the S3 Bucket.",
                        Example = "{name}=My/Remote/Directory",
                        Action = (value) =>
                        {
                            Console.WriteLine(value);
                        }
                    }
                    ...
    

Step 4 – Update the parameter array passed to UploadFile

We’ve already create a custom parameter array for the UploadFile action. We simply need to add a method for it

            // Use the Custom or Common pattern
            CustomParameters.Add("CreateBucketDirectory", new object[] { s3client, bucketName, Args.Value("Directory") });
            CustomParameters.Add("CreateTextFile", new object[] { s3client, bucketName, Args.Value("Filename"), Args.Value("Text") });
            CustomParameters.Add("DeleteBucketDirectory", new object[] { s3client, bucketName, Args.Value("Directory") });
            CustomParameters.Add("DeleteFile", new object[] { transferUtility, bucketName, Args.Value("Filename") });
            CustomParameters.Add("UploadFile", new object[] { transferUtility, bucketName, Args.Value("File"), Args.Value("RemoteDirectory") });

Note: There are enough of these now that I alphabetized them.

You can now specify the remote directory when uploading a file.

Go to: Part 10 – Uploading all files in a directory recursively to an S3 Bucket

Return to: Managing Amazon AWS with C#

Amazon S3 Bucket Management with C#: Part 8 – Deleting a file in a Bucket

Before getting started

Skill Level: Beginner

Assumptions:

  1. You already gone through Parts 1-7 of Managing Amazon AWS with C#.

Additional information: I sometimes cover small sub-topics in a post. Along with AWS, you will also be exposed to:

  • Rhyous.SimpleArgs
  • Don’t Repeat Yourself (DRY) Principal

Step 1 – Add a DeleteFile method to BucketManager.cs

  1. Edit file called BucketManager.cs.
  2. Enter this new method:
            public static async Task CreateTextFile(AmazonS3Client client, string bucketName, string filename, string text)
            {
                var dirRequest = new PutObjectRequest
                {
                    BucketName = bucketName,
                    Key = filename,
                    InputStream = text.ToStream()
                };
                await client.PutObjectAsync(dirRequest);
                Console.WriteLine($"Created text file in S3 bucket: {bucketName}/{filename}");
            }
    

Notice: The code is almost identical to that of deleting a directory, with only one exception. We aren’t ending with a /. We really should not have duplicate code. So lets fix this in the next step.

Step 2 – Solve the Repetitive Code

It is best practice to avoid having duplicate code. This is often called the “Don’t Repeat Yourself” principal. So let’s update the DeleteBucketDirectory code to forward to the DeleteFile code.

  1. Update the DeleteDirectory method so that both methods share code.
           public static async Task DeleteBucketDirectory(AmazonS3Client client, string bucketName, string directory)
            {
                if (!directory.EndsWith("/"))
                    directory = directory += "/";
                await DeleteFile(client, bucketName, directory);
            }
    

Now the delete directory code is no longer repetitive. A directory is the same as a file, just with a slash. So the Delete directory correctly makes sure that the directory name ends with a slash, then forwards the call to delete file.

Step 3 – Update the Action Argument

We should be very good at this by now. We need to make this method a valid action for the Action Argument.

  1. Edit the ArgsHandler.cs file to define an Action argument.
                        ...
                        AllowedValues = new ObservableCollection<string>
                        {
                            "CreateBucket",
                            "DeleteBucket",
                            "ListFiles",
                            "UploadFile",
                            "CreateBucketDirectory",
                            "DeleteBucketDirectory",
                            "CreateTextFile",
                            "DeleteFile"
                        },
                        ...
    

Note: There are no additional Arguments to add. To delete a file, we need the bucket name and a file name, which we already have Arguments for.

Step 4 – Fix the parameter mismatch problem

In Part 4, we created a method to pass different parameters to different methods. Let’s use that to pass in the correct parameters.

However, take note that we now have more exceptions than we had commonalities. This suggests that it is about time to refactor this code. For now, we will leave it.

            // Use the Custom or Common pattern
            CustomParameters.Add("UploadFile", new object[] { transferUtility, bucketName, Args.Value("File") });
            CustomParameters.Add("CreateBucketDirectory", new object[] { s3client, bucketName, Args.Value("Directory") });
            CustomParameters.Add("DeleteBucketDirectory", new object[] { s3client, bucketName, Args.Value("Directory") });
            CustomParameters.Add("CreateTextFile", new object[] { s3client, bucketName, Args.Value("Filename"), Args.Value("Text") });

You can now add a text file to an Amazon S3 bucket using C#.

Homework: There is some repetitiveness between CreateFolder and DeleteFolder. What is it? (Hint: Directories end with a slash.)

Go to: Part 9 – Uploading a file with its path to a Bucket

Return to: Managing Amazon AWS with C#

Amazon S3 Bucket Management with C#: Part 7 – Creating a text file in a Bucket

Before getting started

Skill Level: Beginner

Assumptions:

  1. You already gone through Parts 1-6 of Managing Amazon AWS with C#.

Additional information: I sometimes cover small sub-topics in a post. Along with AWS, you will also be exposed to:

  • Rhyous.SimpleArgs
  • Rhyous.StringLibrary

Step 1 – Add NuGet Package

  1. Right-click on your project and choose Management NuGet Packages.
  2. Search for Rhyous.StringLibrary.
    This is a simple library for string extensions methods and more. String code that is not in .Net by default, yet the methods have proven over time to be commonly used.
  3. Install the Rhyous.StringLibrary NuGet package.

Step 2 – Add a CreateTextFile method to BucketManager.cs

  1. Edit file called BucketManager.cs.
  2. Enter this new method:
            public static async Task CreateTextFile(AmazonS3Client client, string bucketName, string filename, string text)
            {
                var dirRequest = new PutObjectRequest
                {
                    BucketName = bucketName,
                    Key = filename,
                    InputStream = text.ToStream()
                };
                await client.PutObjectAsync(dirRequest);
                Console.WriteLine($"Created text file in S3 bucket: {bucketName}/{filename}");
            }
    

Notice: The code is almost identical to that of creating a directory, with two exceptions. We aren’t ending with a /. And instead of assigning zero bytes to InputStream, we assigned text.ToStream(). Rhyous.StringLibrary provides us the ToStream() extension method.

Step 2 – Update the Action Argument

We now need to make this method a valid action for the Action Argument.

  1. Edit the ArgsHandler.cs file to define an Action argument.
                        ...
                        AllowedValues = new ObservableCollection&amp;amp;lt;string&amp;amp;gt;
                        {
                            "CreateBucket",
                            "DeleteBucket",
                            "ListFiles",
                            "UploadFile",
                            "CreateBucketDirectory",
                            "DeleteBucketDirectory",
                            "CreateTextFile",
                        },
                        ...
    

Step 3 – Add FileName and Text Arguments

If we are going to create a text file, we need to know the file name and the text to insert.

  1. Edit the ArgsHandler.cs file to define an Action argument.
                    ...
                    new Argument
                    {
                        Name = "FileName",
                        ShortName = "N",
                        Description = "The name of text a file to create.",
                        Example = "{name}=MyTextfile.txt",
                        Action = (value) =>
                        {
                            Console.WriteLine(value);
                        }
                    },
                    new Argument
                    {
                        Name = "Text",
                        ShortName = "T",
                        Description = "The text to put in a text file.",
                        Example = "{name}=\"This is some text!\"",
                        Action = (value) =>
                        {
                            Console.WriteLine(value);
                        }
                    }
                    ...
    

Step 4 – Fix the parameter mismatch problem

In Part 4, we created a method to pass different parameters to different methods. Let’s use that to pass in the correct parameters.

However, take note that we now have more exceptions than we had commonalities. This suggests that it is about time to refactor this code. For now, we will leave it.

            // Use the Custom or Common pattern
            CustomParameters.Add("UploadFile", new object[] { transferUtility, bucketName, Args.Value("File") });
            CustomParameters.Add("CreateBucketDirectory", new object[] { s3client, bucketName, Args.Value("Directory") });
            CustomParameters.Add("DeleteBucketDirectory", new object[] { s3client, bucketName, Args.Value("Directory") });
            CustomParameters.Add("CreateTextFile", new object[] { s3client, bucketName, Args.Value("Filename"), Args.Value("Text") });

You can now add a text file to an Amazon S3 bucket using C#.

Go to: Deleting a file in a Bucket

Return to: Managing Amazon AWS with C#

Amazon S3 Bucket Management with C#: Part 6 – Deleting a Directory in a Bucket

Before getting started

Skill Level: Beginner

Assumptions:

  1. You already gone through Parts 1-5 of Managing Amazon AWS with C#.

Additional information: I sometimes cover small sub-topics in a post. Along with AWS, you will also be exposed to:

  • Rhyous.SimpleArgs

Step 1 – Add a DeleteBucketDirectory method to BucketManager.cs

  1. Edit file called BucketManager.cs.
  2. Enter this new method:
            public static async Task DeleteBucketDirectory(AmazonS3Client client, string bucketName, string directory)
            {
                var dirRequest = new DeleteObjectRequest
                {
                    BucketName = bucketName,
                    Key = directory + "/"
                };
                await client.DeleteObjectAsync(dirRequest);
                Console.WriteLine($"Created S3 bucket folder: {bucketName}/{directory}/");
            }
    

Note: Amazon S3 uses objects with a key ending in a / as a directory, so we have to call DeleteObjectAsync.

Step 2 – Update the Action Argument

We now need to make this method a valid action for the Action Argument.

  1. Edit the ArgsHandler.cs file to define an Action argument.
                        ...
                        AllowedValues = new ObservableCollection&amp;lt;string&amp;gt;
                        {
                            "CreateBucket",
                            "DeleteBucket",
                            "ListFiles",
                            "UploadFile",
                            "CreateBucketDirectory",
                            "DeleteBucketDirectory"
                        },
                        ...
    

Step 3 – Fix the parameter mismatch problem

In Part 4, we created a method to pass different parameters to different methods. Let’s use that to pass in the correct parameters.

            // Use the Custom or Common pattern
            CustomParameters.Add("UploadFile", new object[] { transferUtility, bucketName, Args.Value("File") });
            CustomParameters.Add("CreateBucketDirectory", new object[] { s3client, bucketName, Args.Value("Directory") });
            CustomParameters.Add("DeleteBucketDirectory", new object[] { s3client, bucketName, Args.Value("Directory") });

You can now Delete a directory on S3, using C#.

Go to:

Return to: Managing Amazon AWS with C#

Amazon S3 Bucket Management with C#: Part 8 – Deleting a file in a Bucket

Before getting started

Skill Level: Beginner

Assumptions:

  1. You already gone through Parts 1-7 of Managing Amazon AWS with C#.

Additional information: I sometimes cover small sub-topics in a post. Along with AWS, you will also be exposed to:

  • Rhyous.SimpleArgs
  • Don’t Repeat Yourself (DRY) Principal

Step 1 – Add a DeleteFile method to BucketManager.cs

  1. Edit file called BucketManager.cs.
  2. Enter this new method:
            public static async Task CreateTextFile(AmazonS3Client client, string bucketName, string filename, string text)
            {
                var dirRequest = new PutObjectRequest
                {
                    BucketName = bucketName,
                    Key = filename,
                    InputStream = text.ToStream()
                };
                await client.PutObjectAsync(dirRequest);
                Console.WriteLine($"Created text file in S3 bucket: {bucketName}/{filename}");
            }
    

Notice: The code is almost identical to that of deleting a directory, with only one exception. We aren’t ending with a /. We really should not have duplicate code. So lets fix this in the next step.

Step 2 – Solve the Repetitive Code

It is best practice to avoid having duplicate code. This is often called the “Don’t Repeat Yourself” principal. So let’s update the DeleteBucketDirectory code to forward to the DeleteFile code.

  1. Update the DeleteDirectory method so that both methods share code.
           public static async Task DeleteBucketDirectory(AmazonS3Client client, string bucketName, string directory)
            {
                if (!directory.EndsWith("/"))
                    directory = directory += "/";
                await DeleteFile(client, bucketName, directory);
            }
    

Now the delete directory code is no longer repetitive. A directory is the same as a file, just with a slash. So the Delete directory correctly makes sure that the directory name ends with a slash, then forwards the call to delete file.

Step 3 – Update the Action Argument

We should be very good at this by now. We need to make this method a valid action for the Action Argument.

  1. Edit the ArgsHandler.cs file to define an Action argument.
                        ...
                        AllowedValues = new ObservableCollection<string>
                        {
                            "CreateBucket",
                            "DeleteBucket",
                            "ListFiles",
                            "UploadFile",
                            "CreateBucketDirectory",
                            "DeleteBucketDirectory",
                            "CreateTextFile",
                            "DeleteFile"
                        },
                        ...
    

Step 3 – Add FileName and Text Arguments

If we are going to create a text file, we need to know the file name and the text to insert.

  1. Edit the ArgsHandler.cs file to define an Action argument.
                    ...
                    new Argument
                    {
                        Name = "FileName",
                        ShortName = "N",
                        Description = "The name of text a file to create.",
                        Example = "{name}=MyTextfile.txt",
                        Action = (value) =>
                        {
                            Console.WriteLine(value);
                        }
                    },
                    new Argument
                    {
                        Name = "Text",
                        ShortName = "T",
                        Description = "The text to put in a text file.",
                        Example = "{name}=\"This is some text!\"",
                        Action = (value) =>
                        {
                            Console.WriteLine(value);
                        }
                    }
                    ...
    

Step 4 – Fix the parameter mismatch problem

In Part 4, we created a method to pass different parameters to different methods. Let’s use that to pass in the correct parameters.

However, take note that we now have more exceptions than we had commonalities. This suggests that it is about time to refactor this code. For now, we will leave it.

            // Use the Custom or Common pattern
            CustomParameters.Add("UploadFile", new object[] { transferUtility, bucketName, Args.Value("File") });
            CustomParameters.Add("CreateBucketDirectory", new object[] { s3client, bucketName, Args.Value("Directory") });
            CustomParameters.Add("DeleteBucketDirectory", new object[] { s3client, bucketName, Args.Value("Directory") });
            CustomParameters.Add("CreateTextFile", new object[] { s3client, bucketName, Args.Value("Filename"), Args.Value("Text") });

You can now add a text file to an Amazon S3 bucket using C#.

Homework: There is some repetitiveness between CreateFolder and DeleteFolder. What is it? (Hint: Directories end with a slash.)

Go to:

Return to: Managing Amazon AWS with C#

Amazon S3 Bucket Management with C#: Part 5 – Creating a Directory in a Bucket

Before getting started

Skill Level: Beginner

Assumptions:

  1. You already gone through Parts 1-4 of Managing Amazon AWS with C#.

Additional information: I sometimes cover small sub-topics in a post. Along with AWS, you will also be exposed to:

  • Rhyous.SimpleArgs

Step 1 – Add a CreateBucketDirectory method to BucketManager.cs

  1. Edit file called BucketManager.cs.
  2. Enter this new method:
            public static async Task CreateBucketDirectory(AmazonS3Client client, string bucketName, string directory)
            {
                var dirRequest = new PutObjectRequest
                {
                    BucketName = bucketName,
                    Key = directory + "/",
                    InputStream = new MemoryStream(new byte[0])
                };
                await client.PutObjectAsync(dirRequest);
                Console.WriteLine($"Created S3 bucket folder: {bucketName}/{directory}/");
            }
    

Note: Amazon S3 uses objects with a key ending in a / as a directory, so all we do is put a new empty object with a slash.

Step 2 – Update the Action Argument

We now need to make this method a valid action for the Action Argument.

  1. Edit the ArgsHandler.cs file to define an Action argument.
                        ...
                        AllowedValues = new ObservableCollection&amp;lt;string&amp;gt;
                        {
                            "CreateBucket",
                            "DeleteBucket",
                            "ListFiles",
                            "UploadFile",
                            "CreateBucketDirectory"
                        },
                        ...
    

Step 3 – Fix the parameter mismatch problem

In Part 4, we created a method to pass different parameters to different methods. Let’s use that to pass in the correct parameters.

            // Use the Custom or Common pattern
            CustomParameters.Add("UploadFile", new object[] { transferUtility, bucketName, Args.Value("File") });
            CustomParameters.Add("CreateBucketDirectory", new object[] { s3client, bucketName, Args.Value("Directory") });

You can now add a directory on S3, using C#.

Go to:

Return to: Managing Amazon AWS with C#

Amazon S3 Bucket Management with C#: Part 4 – Uploading a file to a Bucket

Before getting started

Skill Level: Beginner

Assumptions:

  1. You already gone through Parts 1, 2, and 3 of Managing Amazon AWS with C#.

Additional information: I sometimes cover small sub-topics in a post. Along with AWS, you will also be exposed to:

  • async, await, Task
  • Rhyous.SimpleArgs
  • Dependency Inject (D in S.O.L.I.D.)
  • Reflection
  • Custom or Common pattern

Step 1 – Add an UploadFile method to BucketManager.cs

  1. Edit file called BucketManager.cs.
  2. Enter this new method:
            public static async Task UploadFile(TransferUtility transferUtility, string bucketName, string file)
            {
                await Task.Run(() => transferUtility.Upload(file, bucketName));
            }
    

    Notice 1: This method has different parameters. We are going to have to fix Program.cs later. Up until now, all of our methods had the same parameters.

  3. Notice 2: The content of this method is an action we would like to run asynchronously, but it is not asynchronous. Task.Run is a static method that runs any method you call inside it asynchronously.

Step 2 – Update the Action Argument

We now need to make this method a valid action for the Action Argument.

    1. Edit the ArgsHandler.cs file to define an Action argument.
                          ...
                          AllowedValues = new ObservableCollection<string>
                          {
                              "CreateBucket",
                              "DeleteBucket",
                              "ListFiles",
                              "UploadFile",
                          },
                          ...
      

Step 3 – Add a File Argument

If we are going to upload a file, we should know which file it is.

  1.                 new Argument
                    {
                        Name = "File",
                        ShortName = "f",
                        Description = "The file.",
                        Example = "{name}=c:\\some\file.txt",
                        CustomValidation = (value) =>
                        {
                            return File.Exists(value);
                        },
                        Action = (value) =>
                        {
                            Console.WriteLine(value);
                        }
                    }
    

    Notice: One of the cool features of SimpleArgs is the ability to declare custom validation. We don’t have to check elsewhere in our code if the file exists. We can use that to validate whether the File parameter is valid or not.

Step 4 – Fix the parameter mismatch problem

To be nice and S.O.L.I.D., our program now needs to determine the method’s dependencies and inject them into the method. It needs to do this dynamically at runtime.

Well, with only one exception, it is most easy to use the Custom or Common pattern. The Custom or Common pattern simply means checking for a customization and if no customization exists, use the common implementation.

To implement this, we could just use a simple IF condition. But just to demonstrate how a Dictionary can be used for the Custom or Common pattern, I will use it.

    class Program
    {
        static void Main(string[] args)
        {
            new ArgsManager<ArgsHandler>().Start(args);
        }

        static Dictionary<string, object[]> CustomParameters = new Dictionary<string, object[]>();

        internal static void OnArgumentsHandled()
        {
            var action = Args.Value("Action");
            var bucketName = Args.Value("Bucket");
            var file = Args.Value("File");

            var flags = BindingFlags.NonPublic | BindingFlags.Public | BindingFlags.Static | BindingFlags.FlattenHierarchy;
            MethodInfo mi = typeof(BucketManager).GetMethod(action, flags);

            var region = RegionEndpoint.GetBySystemName(ConfigurationManager.AppSettings["AWSRegion"]);
            var s3client = new AmazonS3Client(region);
            var transferUtility = new TransferUtility(region);

            // Use the Custom or Common pattern
            CustomParameters.Add("UploadFile", new object[] { transferUtility, bucketName, file });
            object[] parameters;
            if (!CustomParameters.TryGetValue(action, out parameters))
                parameters = new object[] { s3client, bucketName };

            var task = mi.Invoke(null, parameters) as Task;
            task.Wait();
        }
    }

Notice in line 8 we create a dictionary. In line 24 we populate the dictionary with a customization. In Lines 26 and 27, we try to get a custom parameter array and if we don’t find it, we use the default one.

Homework: What if every method had different parameters? What would you do?

Go to: Part 5 – Creating a Directory in a Bucket

Return to: Managing Amazon AWS with C#

Amazon S3 Bucket Management with C#: Part 3 – Listing files in a Bucket

Before getting started

Skill Level: Beginner

Assumptions:

  1. You already gone through Part 1 – Creating a Bucket and Part 2 – Deleting a Bucket

Additional information: I sometimes cover small sub-topics in a post. Along with AWS, you will also be exposed to:

  • async, await, Task
  • Rhyous.SimpleArgs

Step 1 – Add a ListFiles method to BucketManager.cs

  1. Edit file called BucketManager.cs.
  2. Enter this new method:
            public static async Task ListFiles(AmazonS3Client client, string bucketName)
            {
                var listResponse = await client.ListObjectsV2Async(new ListObjectsV2Request { BucketName = bucketName });
                if (listResponse.S3Objects.Count > 0)
                {
                    Console.WriteLine($"Listing items in S3 bucket: {bucketName}");
                    listResponse.S3Objects.ForEach(o => Console.WriteLine(o.Key));
                }
            }
    

    Note: I noticed there was a ListObjectsAsync and a ListObjectsV2Async. I assumed the one with V2 is newer and should be used for new code. The documentation for ListObjectsV2Async confirmed this.

Step 2 – Update the Action Argument

We now need to make this method a valid action for the Action Argument.

  1. Edit the ArgsHandler.cs file to define an Action argument.
                        ...
                        AllowedValues = new ObservableCollection<string>
                        {
                            "CreateBucket",
                            "DeleteBucket",
                            "ListFiles"
                        },
                        ...
    

Notice: We didn’t have a step 3. We wrote some S.O.L.I.D. code in Part 1 and Part 2, which made it really easy for us to implement this method.

Homework: I also read in the documentation that only 1000 files will be listed when a call to ListObjectsV2Async is made. What if you have more than 1000 files, how would you list them all?

Go to: Part 4 – Uploading a file to a Bucket

Return to: Managing Amazon AWS with C#

Amazon S3 Bucket Management with C#: Part 2 – Deleting a Bucket

Before getting started

Skill Level: Beginner

Assumptions:

  1. You already gone through Part 1 – Creating a Bucket where you have already:
    1. Created a new Console Application Project
    2. Added NuGet Package
    3. Created a BucketManager.cs
    4. Coded Program.cs.
    5. Added command line arguments

Additional information: I sometimes cover small sub-topics in a post. Along with AWS, you will also be exposed to:

  • async, await, Task
  • Reflection
  • Rhyous.SimpleArgs
  • Single Responsibility Principal (S of S.O.L.I.D.) or Don’t Repeat Yourself (DRY)
  • Dependency Injection – Method Injection (D of S.O.L.I.D.)

Step 1 – Add a DeleteBucket method to BucketManager.cs

  1. Edit file called BucketManager.cs.
  2. Enter this new method:
            public static async Task DeleteBucket(string bucketName)
            {
                var region = RegionEndpoint.GetBySystemName(ConfigurationManager.AppSettings["AWSRegion"]);
                var client = new AmazonS3Client(region);
                await AmazonS3Util.DeleteS3BucketWithObjectsAsync(client, bucketName);
                Console.WriteLine($"Deleted S3 bucket: {bucketName}");
            }
    

    Notice that there is more involved with deleting a bucket than creating a bucket. A bucket may not be empty. It could have files in it already. Because of this we call a helper method, DeleteS3BucketWithObjectsAsync, that deletes a bucket even if it has objects, i.e. files, in it.

Step 2 – Configure an Argument for the Action to call

Since BucketManager can now can both Create and Delete a bucket, we need an argument to specify what action we would like to call.

  1. Edit the ArgsHandler.cs file to define an Action argument.
    SimpleArgs allows for arguments to be declarative and provides most the features you would want in command line arguments without having to write those features for every new application.

                    new Argument
                    {
                        Name = "Action",
                        ShortName = "a",
                        Description = "The action to run.",
                        Example = "{name}=default",
                        DefaultValue = "Default",
                        AllowedValues = new ObservableCollection&lt;string&gt;
                        {
                            "CreateBucket",
                            "DeleteBucket"
                        },
                        IsRequired = true,
                        Action = (value) =&gt;
                        {
                            Console.WriteLine(value);
                        }
                    }
    

    Notice: We don’t have to validate that the correct method was passed in as a variable because SimpleArgs will do this for us by simply declaring them as AllowedValues. When AllowedValues is declared, only those values are allowed. Any other value will result in the application stopping and outputting the list of valid arguments.

Step 3 – Edit the Program.cs

We are now going to use reflection to call the appropriate function based on our action parameter.

    1. Edit the OnArgumentsHandled method of Program.cs.
              internal static void OnArgumentsHandled()
              {
                  var action = Args.Value("Action");
                  var bucketName = Args.Value("Bucket");
                  var flags = BindingFlags.NonPublic | BindingFlags.Public | BindingFlags.Static | BindingFlags.FlattenHierarchy;
                  MethodInfo mi = typeof(BucketManager).GetMethod(action, flags);
      
                  var task = mi.Invoke(null, new[] { bucketName }) as Task;
                  task.Wait();
              }
      

Step 4 – Make the code S.O.L.I.D.

We have broken the Single Responsibility Principal (S in S.O.L.I.D.) or Don’t Repeat Yourself (DRY) rule. Let’s notice it and fix it.

    1. Notice in BucketManager.cs that both methods are breaking two rules:
      • Don’t Repeat Yourself or DRY: We have two methods repeating the same two lines of code.
                    var region = RegionEndpoint.GetBySystemName(ConfigurationManager.AppSettings["AWSRegion"]);
                    var client = new AmazonS3Client(region);											
        
      • Single Responsibility Principal: Each method has one repsponsibility. CreateBucket should only create a bucket on the Amazon S3 server. DeletBucket should only delete a bucket. Currently, both methods are having to instantiate a client and figure out the region. That isn’t the responsibility of these methods.
    2. Let’s solve this with Method Injection. Method Injection is a form of Dependency Injection (D in S.O.L.I.D.). We will pass the client into the method.
      Note: I am using method injection because all the methods are static, so Constructor Injection is not an option. Property Injection is an option. A Lazy-injectable Property would also be a very good option here.
    3. Alter the methods to take in an AmazonS3Client object.
      using Amazon.S3;
      using Amazon.S3.Util;
      using System;
      using System.Threading.Tasks;
      
      namespace Rhyous.AmazonS3BucketManager
      {
          public class BucketManager
          {
              public static async Task CreateBucket(AmazonS3Client client, string bucketName)
              {
                  await client.PutBucketAsync(bucketName);
                  Console.WriteLine($"Created S3 bucket: {bucketName}");
              }
      
              public static async Task DeleteBucket(AmazonS3Client client, string bucketName)
              {
                  await AmazonS3Util.DeleteS3BucketWithObjectsAsync(client, bucketName);
                  Console.WriteLine($"Deleted S3 bucket: {bucketName}");
              }
          }
      }
      
    4. Update the Program.cs to instantiate the client and pass it into the methods.
              internal static void OnArgumentsHandled()
              {
                  var action = Args.Value("Action");
                  var bucketName = Args.Value("Bucket");
      
                  var flags = BindingFlags.NonPublic | BindingFlags.Public | BindingFlags.Static | BindingFlags.FlattenHierarchy;
                  MethodInfo mi = typeof(BucketManager).GetMethod(action, flags);
      
                  var region = RegionEndpoint.GetBySystemName(ConfigurationManager.AppSettings["AWSRegion"]);
                  var client = new AmazonS3Client(region);
      
                  var task = mi.Invoke(null, new object[] { client, bucketName }) as Task;
                  task.Wait();
              }
      

      Notice we now only create a client one time.

      Note: The OnArgumentsHandled in our Program.cs is now doing four things. As you can see above, it has each thing it does in a pair of lines, with each pair of lines separated by a double space. This is where we want to be careful to not overdo it when following design patterns. We only have eight lines of code. Program.cs is supposed to be a program. It is fine for now. Let’s not change it. Notice it. Consider changing it. This time we decided to leave it, but we can keep it in mind. If more lines are added, then that method probably should be changed.

Go to: Part 3 – Listing files in a Bucket

Return to: Managing Amazon AWS with C#

Amazon S3 Bucket Management with C#: Part 1 – Creating a Bucket

Before getting started

Skill Level: Beginner

Assumptions:

  1. You already have Visual Studio installed.
  2. You are familiar with creating projects in Visual Studio.
  3. We assume you have already gone to AWS and registered with them. If you haven’t done that already, stop and go there now. Amazon has a free tier and you can create an account here: https://aws.amazon.com/free

Additional Information: I sometimes cover small sub-topics in a post. Along with AWS, you will also be exposed to:

  • .NET Core 2.0 – If you use .NET Framework, the steps will be slightly different, but as this is a beginner level tutorial, it should be simple.
  • async, await, Task
  • Rhyous.SimpleArgs

Step 1 – Create the project

  1. Open Visual Studio.
  2. Go to File | New Project.
  3. Choose Console Application.
    Give it any name you want.
    I am going to call my project Rhyous.AmazonS3BucketManager.

Step 2 – Add NuGet Packages

  1. Right-click on your project and choose Management NuGet Packages.
  2. Search for AWSSDK.S3.
  3. Install the NuGet package and all the dependencies.
  4. Search for System.Configuration.ConfigurationManager.
  5. Install it.

Step 3 – Create a BucketManager.cs file

  1. Create a new file called BucketManager.cs.
  2. Enter this code:
    using Amazon;
    using Amazon.S3;
    using System;
    using System.Configuration;
    using System.Threading.Tasks;
    
    namespace Rhyous.AmazonS3BucketManager
    {
        public class BucketManager
        {
            public static async Task CreateBucket(string bucketName)
            {
                var region = RegionEndpoint.GetBySystemName(ConfigurationManager.AppSettings["AWSRegion"]);
                var client = new AmazonS3Client(region);
                await client.PutBucketAsync(bucketName);
                Console.WriteLine($"Created S3 bucket: {bucketName}");
            }
        }
    }
    

Step 4 – Edit the Program.cs

  1. Add the following to Program.cs.
            static void Main()
            {
                var task = BucketManager.CreateBucket("my.new.bucket");
                task.Wait();
            }
    

Step 5 – Create/Edit the App.config

  1. If there isn’t an app.config in your project, create one.
  2. Right-click on your project and choose Add | New Item.
  3. Search for Application Configuration File.
    Make sure it is name app.config.
  4. Add an appSetting for your AWS profile name.
    Add an additional appSetting for your chosen AWS region.

    <?xml version="1.0" encoding="utf-8" ?>
    <configuration>
      <appSettings>
        <add key="AWSProfileName" value="yourprofilename"/>
        <add key="AWSRegion" value="us-west-2" />
      </appSettings>
    </configuration>
    

Step 6 – Configure an Argument for the bucket name.

We are going to be adding to this program in subsequent posts. For this reason, we are going to use Rhyous.SimpleArgs library for our command line arguments as it provides ready-made command line argument features.

  1. Install another NuGet Package.
  2. Right-click on your project and choose Management NuGet Packages.
  3. Search for Rhyous.SimpleArgs
  4. Install it.
  5. Create an ArgsHandler.cs file to define the arguments:
    Note: If you used a .NET core project you have to create this file. If you created a .NET Framework file, this file should have been created for you and you have but to edit it.

    using Rhyous.SimpleArgs;
    using System;
    using System.Collections.Generic;
    using System.Text.RegularExpressions;
    
    namespace Rhyous.AmazonS3BucketManager
    {
        public class ArgsHandler : ArgsHandlerBase
        {
            public override void InitializeArguments(IArgsManager argsManager)
            {
                Arguments.AddRange(new List<Argument>
                {
                    new Argument
                    {
                        Name = "Bucket",
                        ShortName = "b",
                        Description = "The bucket name to create. No uppercase or underscores allowed.",
                        Example = "{name}=my.first.bucket",
                        DefaultValue = "my.first.bucket",
                        IsRequired = true,
                        CustomValidation = (value) => 
                        {
                            return Regex.IsMatch(value, "^[a-z0-9.]+$");
                        },
                        Action = (value) =>
                        {
                            Console.WriteLine(value);
                        }
                    }
                });
            }
    
            public override void HandleArgs(IReadArgs inArgsHandler)
            {
                base.HandleArgs(inArgsHandler);
                Program.OnArgumentsHandled();
            }
        }
    }
    
  6. Update Program.cs as follows:
    using Rhyous.SimpleArgs;
    using System;
    
    namespace Rhyous.AmazonS3BucketManager
    {
        class Program
        {
            static void Main(string[] args)
            {
                new ArgsManager<ArgsHandler>().Start(args);
            }
    
            internal static void OnArgumentsHandled()
            {
                var bucketName = Args.Value("Bucket");
                var task = BucketManager.CreateBucket(bucketName);
                task.Wait();
            }
        }
    }
    

Now for fun, you can delete the app.config and change them to parameters.

Next:

  • Deleting a Bucket
  • Return to: Managing Amazon AWS with C#

    Understanding async and await, Task.WaitAll, Task.Run, and parallelism

    Ok. You are probably new to async and await or maybe you aren’t new but you’ve never deep dived into it. You may not understand some simple truths:

    1. aync/await does NOT give you parallelism for free.
    2. Tasks are not necessary parallel. They can be if you code them to be.
    3. The recommendation “You should always use await” is not really true when you want parallelism, but is still sort of true.
    4. Task.WhenAll is both parallel and async.
    5. Task.WaitAll only parallel.

    Here is a sample project that will help you learn.

    There is more to learn in the comments.
    There is more to learn by running this.

    Note: I used Visual Studio 2017 and compiled with .Net 7.1, which required that I go to the project properties | Build | Advanced | Language Version and set the language to C# 7.1 or C# latest minor version.

    using System;
    using System.Diagnostics;
    using System.Threading.Tasks;
    
    namespace MultipleAwaitsExample
    {
        class Program
        {
            static async Task Main(string[] args)
            {
                Console.WriteLine("Running with await");
                await RunTasksAwait();
                Console.WriteLine("Running with Task.WaitAll()");
                await RunTasksWaitAll();
                Console.WriteLine("Running with Task.WhenAll()");
                await RunTasksWhenAll();
                Console.WriteLine("Running with Task.Run()");
                await RunTasksWithTaskRun();
                Console.WriteLine("Running with Parallel");
                RunTasksWithParallel();
            }
    
            /// <summary>
            /// Pros: It works
            /// Cons: The tasks are NOT run in parallel.
            ///       Code after the await is not run while the await is awaited
            ///       **If you want parallelism, this isn't even an option.**
            ///       Slowest. Because of no parallelism.
            /// </summary>
            public static async Task RunTasksAwait()
            {
                var group = "await";
                Stopwatch watcher = new Stopwatch();
                watcher.Start();
                await MyTaskAsync(1, 500, group);
                await MyTaskAsync(2, 300, group);
                await MyTaskAsync(3, 100, group);
                Console.WriteLine("Code immediately after tasks.");
                watcher.Stop();
                Console.WriteLine($"{group} runtime: {watcher.ElapsedMilliseconds}");
            }
    
            /// <summary>
            /// WaitAll behaves quite differently from WhenAll
            /// Pros: It works
            ///       The tasks run in parallel
            /// Cons: It isn't clear whether the code is parallel here, but it is.
            ///       It isn't clear whether the code  is async here, but it is NOT.
            ///       There is a Visual Studio usage warning. You can remove async to get rid of it because it isn't an Async method.
            ///       The return value is wrapped the Result property of the task
            ///       Breaks Aync end-to-end
            ///       Note: I can't foresee usecase where WaitAll would be preferred over WhenAll.
            /// </summary>
            public static async Task RunTasksWaitAll()
            {
                var group = "WaitAll";
                Stopwatch watcher = new Stopwatch();
                watcher.Start();
                var task1 = MyTaskAsync(1, 500, group);
                var task2 = MyTaskAsync(2, 300, group);
                var task3 = MyTaskAsync(3, 100, group);
                Console.WriteLine("Code immediately after tasks.");
                Task.WaitAll(task1, task2, task3);
                watcher.Stop();
                Console.WriteLine($"{group} runtime: {watcher.ElapsedMilliseconds}");
            }
    
            /// <summary>
            /// WhenAll gives you the best of all worlds. The code is both parallel and async.
            /// Pros: It works
            ///       The tasks run in parallel
            ///       Code after the tasks run while the task is running
            ///       Doesn't break end-to-end async
            /// Cons: It isn't clear you are doing parallelism here, but you are.
            ///       There is a Visual Studio usage warning
            ///       The return value is wrapped the Result property of the task
            /// </summary>
            public static async Task RunTasksWhenAll()
            {
                var group = "WaitAll";
                Stopwatch watcher = new Stopwatch();
                watcher.Start();
                var task1 = MyTaskAsync(1, 500, group);  // You can't use await if you want parallelism
                var task2 = MyTaskAsync(2, 300, group);
                var task3 = MyTaskAsync(3, 100, group);
                Console.WriteLine("Code immediately after tasks.");
                await Task.WhenAll(task1, task2, task3); // But now you are calling await, so you are sort of still awaiting
                watcher.Stop();
                Console.WriteLine($"{group} runtime: {watcher.ElapsedMilliseconds}");
            }
    
            /// <summary>
            /// Pros: It works
            ///       The tasks run in parrallel
            ///       Code can run immediately after the tasks but before the tasks complete
            ///       Allows for running non-async code asynchonously
            /// Cons: It isn't clear whether the code is doing parallelism here. It isn't.
            ///       The lambda syntax affects readability
            ///       Breaks Aync end-to-end
            /// </summary>
            public static async Task RunTasksWithTaskRun()
            {
                var group = "Task.Run()";
                Stopwatch watcher = new Stopwatch();
                watcher.Start();
                await Task.Run(() => MyTask(1, 500, group));
                await Task.Run(() => MyTask(2, 300, group));
                await Task.Run(() => MyTask(3, 100, group));
                Console.WriteLine("Code immediately after tasks.");
                watcher.Stop();
                Console.WriteLine($"{group} runtime: {watcher.ElapsedMilliseconds}");
            }
    
            /// <summary>
            /// Pros: It works
            ///       It is clear in the code you want to run these tasks in parallel.
            ///       Code can run immediately after the tasks but before the tasks complete
            ///       Fastest
            /// Cons: There is no async or await.
            ///       Breaks Aync end-to-end. You can workaround this by wrapping Parallel.Invoke in a Task.Run method. See commented code.
            /// </summary>
            public /* async */ static void RunTasksWithParallel()
            {
                var group = "Parallel";
                Stopwatch watcher = new Stopwatch();
                watcher.Start();
                //await Task.Run(() => 
                Parallel.Invoke(
                    () => MyTask(1, 500, group),
                    () => MyTask(2, 300, group),
                    () => MyTask(3, 100, group),
                    () => Console.WriteLine("Code immediately after tasks.")
                );
                //);
                
                watcher.Stop();
                Console.WriteLine($"{group} runtime: {watcher.ElapsedMilliseconds}");
            }
    
            public static async Task MyTaskAsync(int i, int milliseconds, string group)
            {
                await Task.Delay(milliseconds);
                Console.WriteLine($"{group}: {i}");
            }
    
            public static void MyTask(int i, int milliseconds, string group)
            {
                var task = Task.Delay(milliseconds);
                task.Wait();
                Console.WriteLine($"{group}: {i}");
            }
        }
    }
    

    And here is the same example but this time with some return values.

    using System;
    using System.Diagnostics;
    using System.Threading.Tasks;
    
    namespace MultipleAwaitsExample
    {
        class Program1
        {
            static async Task Main(string[] args)
            {
                Console.WriteLine("Running with await");
                await RunTasksAwait();
                Console.WriteLine("Running with Task.WaitAll()");
                await RunTasksWaitAll();
                Console.WriteLine("Running with Task.WhenAll()");
                await RunTasksWhenAll();
                Console.WriteLine("Running with Task.Run()");
                await RunTasksWithTaskRun();
                Console.WriteLine("Running with Parallel");
                RunTasksWithParallel();
            }
    
            /// <summary>
            /// Pros: It works
            /// Cons: The tasks are NOT run in parallel.
            ///       Code after the await is not run while the await is awaited
            ///       **If you want parallelism, this isn't even an option.**
            ///       Slowest. Because of no parallelism.
            /// </summary>
            public static async Task RunTasksAwait()
            {
                var group = "await";
                Stopwatch watcher = new Stopwatch();
                watcher.Start();
                // You just asign the return variables as normal.
                int result1 = await MyTaskAsync(1, 500, group);
                int result2 = await MyTaskAsync(2, 300, group);
                int result3 = await MyTaskAsync(3, 100, group);
                Console.WriteLine("Code immediately after tasks.");
                watcher.Stop();
                // You now have access to the return objects directly.
                Console.WriteLine($"{group} runtime: {watcher.ElapsedMilliseconds}");
            }
    
            /// <summary>
            /// WaitAll behaves quite differently from WhenAll
            /// Pros: It works
            ///       The tasks run in parallel
            /// Cons: It isn't clear whether the code is parallel here, but it is.
            ///       It isn't clear whether the code  is async here, but it is NOT.
            ///       There is a Visual Studio usage warning. You can remove async to get rid of it because it isn't an Async method.
            ///       The return value is wrapped the Result property of the task
            ///       Breaks Aync end-to-end
            ///       Note: I can't foresee usecase where WaitAll would be preferred over WhenAll.
            /// </summary>
            public static async Task RunTasksWaitAll()
            {
                var group = "WaitAll";
                Stopwatch watcher = new Stopwatch();
                watcher.Start();
                var task1 = MyTaskAsync(1, 500, group);
                var task2 = MyTaskAsync(2, 300, group);
                var task3 = MyTaskAsync(3, 100, group);
                Console.WriteLine("Code immediately after tasks.");
                Task.WaitAll(task1, task2, task3);
                watcher.Stop();
                // You now have access to the return object using the Result property.
                int result1 = task1.Result;
                int result2 = task2.Result;
                int result3 = task3.Result;
                Console.WriteLine($"{group} runtime: {watcher.ElapsedMilliseconds}");
            }
    
            /// <summary>
            /// WhenAll gives you the best of all worlds. The code is both parallel and async.
            /// Pros: It works
            ///       The tasks run in parallel
            ///       Code after the tasks run while the task is running
            ///       Doesn't break end-to-end async
            /// Cons: It isn't clear you are doing parallelism here, but you are.
            ///       There is a Visual Studio usage warning
            ///       The return value is wrapped the Result property of the task
            /// </summary>
            public static async Task RunTasksWhenAll()
            {
                var group = "WaitAll";
                Stopwatch watcher = new Stopwatch();
                watcher.Start();
                var task1 = MyTaskAsync(1, 500, group);  // You can't use await if you want parallelism
                var task2 = MyTaskAsync(2, 300, group);
                var task3 = MyTaskAsync(3, 100, group);
                Console.WriteLine("Code immediately after tasks.");
                await Task.WhenAll(task1, task2, task3); // But now you are calling await, so you are sort of still awaiting
                watcher.Stop();
                Console.WriteLine($"{group} runtime: {watcher.ElapsedMilliseconds}");
            }
    
            /// <summary>
            /// Pros: It works
            ///       The tasks run in parrallel
            ///       Code can run immediately after the tasks but before the tasks complete
            ///       Allows for running non-async code asynchonously
            /// Cons: It isn't clear whether the code is doing parallelism here. It isn't.
            ///       The lambda syntax affects readability
            ///       Breaks Aync end-to-end
            /// </summary>
            public static async Task RunTasksWithTaskRun()
            {
                var group = "Task.Run()";
                Stopwatch watcher = new Stopwatch();
                watcher.Start();
                int result1 = await Task.Run(() => MyTask(1, 500, group));
                int result2 = await Task.Run(() => MyTask(2, 300, group));
                int result3 = await Task.Run(() => MyTask(3, 100, group));
                Console.WriteLine("Code immediately after tasks.");
                watcher.Stop();
                // You now have access to the return objects directly.
                Console.WriteLine($"{group} runtime: {watcher.ElapsedMilliseconds}");
            }
    
            /// <summary>
            /// Pros: It works
            ///       It is clear in the code you want to run these tasks in parallel.
            ///       Code can run immediately after the tasks but before the tasks complete
            ///       Fastest
            /// Cons: There is no async or await.
            ///       Breaks Aync end-to-end. You can workaround this by wrapping Parallel.Invoke in a Task.Run method. See commented code.
            /// </summary>
            public /* async */ static void RunTasksWithParallel()
            {
                var group = "Parallel";
                Stopwatch watcher = new Stopwatch();
                watcher.Start();
                // You have to declare your return objects before hand.
                //await Task.Run(() => 
                int result1, result2, result3;
                Parallel.Invoke(
                    () => result1 = MyTask(1, 500, group),
                    () => result2 = MyTask(2, 300, group),
                    () => result3 = MyTask(3, 100, group),
                    () => Console.WriteLine("Code immediately after tasks.")
                );
                //);
                // You now have access to the return objects directly.
                watcher.Stop();
                Console.WriteLine($"{group} runtime: {watcher.ElapsedMilliseconds}");
            }
    
            public static async Task<int> MyTaskAsync(int i, int milliseconds, string group)
            {
                await Task.Delay(milliseconds);
                Console.WriteLine($"{group}: {i}");
                return i;
            }
    
            public static int MyTask(int i, int milliseconds, string group)
            {
                var task = Task.Delay(milliseconds);
                task.Wait();
                Console.WriteLine($"{group}: {i}");
                return i;
            }
        }
    }