Background

I started a new development project a while back, one that I noodle on ‘when I find time’, and wanted to document some of the thought processes behind the development process and project itself. Anytime I find myself in some sort of a mentorship role, I encourage developers to utilize a lot of little side projects. This isn’t the ‘trite’, 120%-time or the ‘going above and beyond’ side projects that I’m referring to, but rather small, easily set-up, easily tossed away projects that can aid in quickly iterating on ideas. I’ve found these to provide a lot of rapid feedback and over time, have found that they’ve influenced the way I approach software development and the architecture of the systems I write. I’ve utilized many prototyping tools to aid in this process: CodePen, DotNetFiddle, StackBlitz, Docker and my local machine.

A project that I took on in the recent past was a good example of this for me. It required quite a number of these prototypes and has led to a full-fledged iteratively developed application. First I would like to describe the project at its inception, describe the process I’ve used along the way to create it and then provide some documentation of where it stands now with an eye towards future growth.

Application description

There is an organization I’m involved with that provides a number of services. One of them is educational classes. The classes are drop-in(no registration required), happen on a regular basis, usually are around 1hr long and are audio recorded. Some are stand alone classes, while others are short class series and some are long running series. The initial goal of the application was to be able to quickly setup an application that would take in the recorded classes, store them and present them to users via a web interface so attendees could catch up on lectures they missed, sample new topics or explore new vistas. As it happened, there was a long-running series that was going to start in the near future that seemed a great use-case for this app. The goal was that the initial implementation would be done quickly to enable the best adoption.

Initial decisions

Seeing as I wanted the initial implementation to be very delivered on a very short timeline, it seemed logical to start with the development stack I know best. The elements of the development stack that I use regularly are: Angular for the front-end, C# for the back-end, Azure as the cloud provider and GitHub as the source repository as well as the host for the static web application assets. While Azure DevOps would have also been a logical choice for the source code, I was hoping to get others involved in the development of the application and the public model that GiHub has, seemed to be the easiest way to enable that. GitHub also has a very easy way of hosting static applications via HTTPS.

I’ve written before about the segmentation of projects I use for a C# micro-service and while I’m aware of other micro-service architectures, this is still the one that I’ve seen to be most effective and easy to understand. I wanted to follow web best practices and also wanted to challenge myself in the realm of cost to run the application. The organization the application was being built for is a not for profit organization and I would be footing the bill as a part of my contribution to the organization.

Knowing that I was going to utilize Azure as the cloud provider for the application, I looked into its offerings regarding file storage and creating REST apis. Azure Blob Storage seemed logical for the first piece, but the second decision had a few more options. I wanted something as light weight as possible and landed on Azure Functions for the job.

Micro-project #1

Getting started with Azure Blob Storage SDK

With those decisions made, I began my first micro-project. I set out trying to get a bare-bones app that could upload files to Azure Blob Storage. For this, I decided to use WinForms. While there are many jokes around the battleship looking applications that WinForms produces, the drag and drop style of creating controls and the small amount of code required to interact with those controls made it a great option. This allowed me to quickly iterate on the data model to use for the metadata associated with each lecture uploaded and allowed me a sandbox for understanding the Azure Blob Storage apis. As I never planned on distributing the WinForms app out to the lecturers, the initial release required the lecturer to email the class to me every day, which I would then download and use the app to upload. To achieve a faster time to delivery, I also utilized embedding some of the metadata(Lecturer name, Date of class, etc) for each class in the name of the file itself.

Micro-project #2

Getting started with audio file manipulation

As mentioned previously, I wanted to not only challenge myself regarding the ongoing costs of the application, but I also wanted to try to be a responsible contributor to the world wide web. To enable the most users benefit from the application, I wanted the class’ on-disk sizes to be the smallest they could. No matter a user’s type of connection, I wanted the data download required to run the app to be as small as possible. This would make the app faster, but also reduce the cost to the end user if they are paying for cellular data. The classes that were being uploaded, on average, were an hour in length. The recording device used and the settings selected on it, could cause the file to be anywhere from 40MB to 1GB each. While this was surprising to me at first, after doing some research and understanding better how audio files are created, I came to realize that there are different standards around encoding audio files and as time has gone on, newer standards have a number of efficiencies built in that weren’t there previously. I also learned that the ‘sampling rate’ of the audio file plays a big role and for regular speech, the sampling rate can be dialed way down without any noticeable difference in the perceptible audio quality. Now I experimented with my second micro-project in which I created a proof-of-concept of how to do this locally via an Azure Functions project running through Visual Studio. By the end of this iteration, the classes that were downloaded by the users were now consistently in the 10MB range…quite an improvement.

With those items in place, I manually created an Azure Function in the cloud on a Windows machine, created an azure-pipelines.yml file by walking through the Azure DevOps UI, finding the steps I wanted and then copying the yaml they produced for each step to my file. The tools worked so well that I was able to complete all of this in a week and a half, most of which was spent trying to get the audio file manipulation working correctly.

The process of utilizing micro-projects to prove certain aspects of the application and then working on stitching them together to create the complete application is similar in concept to unit tests. I was able to prove aspects of my design with the micro-projects, but that didn’t prove that they all worked together to create a functioning application. There was definitely some time spent on getting it worked when deployed to Azure as I had to learn how to connect the function to blob storage and what directories were available to me in the running function to find the deployed executable used for audio manipulation as well as which directories I could write the output, reduced-file-size file to. At this point though, I was confident that it could work due to the micro-projects.

Micro-project #3

Learning about media controls in the browser

At this point, I still needed two things: a datastore to hold the metadata for the class information(not all of it was a part of the filename) and a frontend application. GitHub pages provides a very convenient, free and fast option for hosting static web sites. The code is all public and the site has to adhere to their terms and agreements, but again, the goal was to get this out as quickly and cheaply as possible. There is one small DNS entry that needs to be entered in your DNS provider and a CNAME file needs to be added to the repo and GitHub then does all of the work for the application to be completely HTTPS. GitHub handles the certs and everything in that regard for you. Once support was there for the first series of classes, I was anticipating a much slower-paced iteration of development from there on out. I created a very small Angular application with two pages: one to show the list of recordings and one that contained the audio player for a given class. As for the datastore, I decided to utilize a json file, hosted as a part of my GitHub repository that I would have to manually update after every class upload.

Initial delivery

For starters, that was it. As far as the attendees were concerned, there was a fully functioning website that they could go to to catch up on or review any lectures that they wanted to. From the administrative perspective, the lecturer would need to email me the class, I would utilize the WinForm app to upload the lecture to Azure blob storage, a queue based Azure function listened for changes to the blob storage container which then kicked off a process to read the file, manipulate its sample-rate/etc and re-upload. I was then required to go into the GitHub repo and update the metadata json file. With this complete, the new lecture would show up on the website. The initial lecture in this series started on Jan 5, 2020 and the initial application was live Jan 17, 2020.

I’ve worked in the industry for over 10 years now and every place I’ve worked at has been ‘agile’. Every place has had VERY different interpretations as to what agile means though. I don’t fully know how to define ‘agile’, but to me, this project seems to have embraced an agile approach. Deliver MVP, then iterate in small iterations, delivering value all the way. Try to make decisions now that won’t cripple a specific development approach later on, but don’t over-engineer it to do so either.

At this point, while I had a fully functioning application for the end user, the manual effort on my end was small, but daily. That was to be the next focus of my efforts. Deliver value to the site maintainer :) The only long term decisions I’d made at this point were that I wasn’t maintaining the original filename of the file that was sent to me by the lecturer, but I would retain the initial file. While the filename decision resulted in a small loss of data, I deemed it to be insignificant and a worthwhile sacrifice to enable the quick release of the application. I did retain the original file and the reduced file but was only exposing the reduced file via the web application.

As I intended this series to document some of my development/thought processes, I also wanted to emphasize the power of the ‘micro-project’. For this application prototype, I already mentioned three that I had utilized. Before I document the next development evolution of the application, I’d like to utilize the next few posts to discuss the micro-projects I’ve mentioned so far. I hope to dive into what proof-of-concept each helped me iterate on and just as importantly, what potentially complicating issues each helped me to temporarily ignore. I’ve found this to be a very powerful way to break large difficult problems into much smaller and more manageable ones.