JonKnopp.com

Other Recommended Blogs for .NET Developers

2020-12-28T01:00:00-07:00

Here is a list of other blogs that I recommend following for any developers in the .NET space, in no particular order.

Randomly Select Item from Array Based on Frequency

2020-12-12T10:40:00-07:00

While working on a clients project to develop a JavaScript game based around the catching of falling objects, we needed a way to determine the frequency at which objects would fall, in relation to one another. Meaning some “rarer” objects needed to fall less frequently than other “common” objects, while still mainting a random nature to the generation.

Doing some research into algorithms, I found the following example on generating a random number based on an arbitrary probability distribution. While this sounds quite complicated, it’s really a pretty simple concept. You have 2 sets of data, 1 is the set you are looking to generate randomly from, the other dataset is arbitrary numbers representing the frequency of each item in the first dataset occuring. The number is arbirary because it only determines frequency in relation to the other numbers in the frequency dataset. Bigger numbers occur more than smaller numbers, and this is exactly what we needed to weight our “rarer” objects to occur less often.

The article had some good insights into how to accomplish the task at hand, while maintaining O(n) complexity. The following code is my JavaScirpt implementation of the algorithm along with an example of how it could be used.

// Utility function to find index of {random} in prefix[start..end]
function findRandomInPrefixArray(prefix, random, start, end) {
    let mid;
    while (start < end) {
        mid = Math.floor((start + end) / 2);
        if (random > prefix[mid]) {
            start = mid + 1;
        } else {
            end = mid;
        }
    }
    return (prefix[start] >= random) ? start : -1;
}

// Returns a random item from array[]
// according to distribution array
// defined by frequency[].
export const arrayItemArbitraryProbabilityDistribution = (array, frequency) => {
    const size = array.length;
    // Create and fill prefix array
    let prefix = [];
    prefix[0] = frequency[0];
    for (let i = 1; i < size; ++i) {
        prefix[i] = prefix[i - 1] + frequency[i];
    }

    // prefix[n-1] is sum of all frequencies. 
    // Generate a random number with  
    // value from 1 to this sum  
    const random = ((Math.floor((Math.random() * 323567))) % prefix[size - 1]) + 1;
    // Find find index of {random} in prefix array
    const index = findRandomInPrefixArray(prefix, random, 0, size - 1);
    return array[index];
}

And how it might be used:

// Define our array of object we want to randomly select from
const fallingItems = 
[{
    id: 1,
    value: 150,
    image: 'rare.png',
    velocity: 5000,
    size: 60
},
{
    id: 2,
    value: 25,
    image: 'average.png',
    velocity: 2000
    size: 110
},
{
    id: 3,
    value: 2,
    image: 'common.png',
    velocity: 900,
    size: 50
}];
// Define the freqeuncy we want each of the fallingItems to be occur at, 
// relative to one another
const frequencyOfFallingItems = [2, 4, 8];

// Randomly get falling item based on frequency array
const itemToQueue =
    arrayItemArbitraryProbabilityDistribution(fallingItems, frequencyOfFallingItems);

We define an array of 3 items: rare, average, and common. Then we assign the frequencies such that ‘average’ items are twice as likely as ‘rare’ items, and ‘common’ items are twice as likely as ‘average’ items, making them 4 times more likely than ‘rare’ items.

Questions, comments or thoughts? Feel free to contact me, or discuss on the gist.

Hope you learned something with me; till next time.

My Cross Platform Development Environment

2020-12-09T10:40:00-07:00

Setting up a cross-platform development environment which works on Windows, Mac, and Linux.

https://chimerical.ca/posts/creating-my-awesome-windows-10-dev-setup

A Windows Docker Container Series (Part 2)

2017-10-05T03:31:00-06:00

After Part 1 where we created a build template in Packer for a Docker compatible image of Windows Server, capable of running containers, I was inspired by the West Wind Album Viewer sample project, and decided it was time to start developing a containerized app for the server to run. With Visual Studio 2017 and .NET Framework 4.7, there is native support for running containerized applications with Docker.

From the Docker for Windows website: “Docker for Windows requires 64bit Windows 10 Pro with Hyper-V available”, but I opted for Enterprise which worked fine also. The trick for me was enabling Hyper-V nested virtualization since my Windows 10E image is running in a VM, and also needs the “Virtualization”, and “Containerization” Windows Features enabled, which I will cover more on later. Visual Studio 2017 also contains a native template for .NET Core 2.0 Web API Projects with Docker support via Docker Compose. The goal is to combine this with their Angular Template and a Docker SQL Server container to create a 3 container app debuggable from Visual Studio and SSMS.

An Introduction to Visual Studio with Docker

In order to run Docker and Visual Studio together you must have the following prerequisites: Install Microsoft Visual Studio 2017 with the .NET Core workload and setup Shared Drives in Docker for Windows. With a new project, there are 2 ways to add docker support to you .NET Core project: 1) Using Visual Studio, create a new ASP.NET Core Web Application. When the application is loaded, either select Add Docker Support from the Project Menu or 2) right-click the project from the Solution Explorer and select Add > Docker Support. In this solution, we will use both ways.

Enabling support for Docker inside a Hyper-V VM

One of the tricks I face in getting docker working with Visual Studio was compatibility for Docker for Windows inside a Hyper-V VM. Since I wanted to create as much of this project in an automated format as I could, I opted to use fresh VMs for the different steps of the project. This allows me the reparability and reliance we discussed in Part 1. With that comes a catch, Docker for Windows requires Hyper-V Virtualization in order to run containers, which means Nested Virtualization for me since my Windows 10E instance is already running inside a Hyper-V VM. Luckily, I was able to find this post which had the following solution to enabling Nested Virtualization in Hyper-V:

Set-VMProcessor -VMName ContainerVM01 -ExposeVirtualizationExtensions $true

How I want to setup the project

As mentioned above, the goal I set out to achieve with this solution was 2 different projects, 1 angular front-end container, and 1 back-end API container that would be networked together to allow for communication. Finally I wanted to run SQL Server as a database solution inside a third container which would be networked with the API container. Since Visual Studio natively uses Docker Compose for multi-container apps, I also wanted to craft my solution in such a way that all the Visual Studio Debugging tools and SQL Server Management functions still worked liked a traditional .NET application.

Walkthrough

Setting up the Projects

To start of I created the Web API project using the Visual Studio Project Menu. From the menu, I used option #1 from above to enable support for Docker directly when the project was created. What this does is creates a Dockerfile and a docker-compose.yml file which can be used with the Docker engine to run your application inside a container. For the Angular project which is presented in the Project Menu the Docker Support menu is not enabled so I started off creating this project without it. I then used option #2 from above to enable the Docker Support later. More on that, and the SQL Server container to come. Breakdown of tiers:

Tier	Visual Studio Docker Support	Container Support
Angular	Yes, but not natively	Yes
Web API	Yes, natively	Yes
SQL Server	Not Available	Yes

As noted by the chart, the SQL Server container will not have a Visual Studio project but will be contained within the Visual Studio solutions docker-compose.yml file so that it is run with the solution when debugging.

Should I run Linux Containers or Windows Containers?

Windows supports the ability to develop and run both Linux-based and Windows-based containers, however, I don’t believe there is a way to develop for both simultaneously in the same solution. When starting off I chose Windows-based containers however, I quickly decided this wasn’t a good choice and started the solution over as Linux-based. The reason I chose this was because my front-end container is being created in Angular and could benefit from having NodeJS running. Also, being a backend developer by trade, I like the idea that the frontend project would be completely changed, and modified (even ported to another language) without affecting any of the API logic. Trying to install NodeJS fully from Powershell in a Dockerfile was proving extremely cumbersome so I opted for Linux in favor of a simple Bash install:

RUN apt-get update  
RUN apt-get -f install  
RUN apt-get install -y wget gnupg
RUN wget -qO- https://deb.nodesource.com/setup_8.x | bash -  
RUN apt-get install -y build-essential nodejs

SQL Server is available as a Linux container and .NET Core 2.0 is also compatible on Linux so for this solution it seemed like the right fit.

The API Project

In the name of keeping things simple and straightforward at this point in the tutorial, I have the left the API project very close to what was auto-generated, with a few exceptions:

In the sampledatacontroller.cs class I have the attributes:
[EnableCors(“CorsPolicy”)] - needed to allow calls from another container
[Route(“api/[controller]”)] - setup our API prefix

In the startup.cs file I added:

services.AddCors(options =>
{
  options.AddPolicy("CorsPolicy",
  	builder => builder
      			.AllowAnyOrigin()
                  .AllowAnyMethod()
                  .AllowAnyHeader()
                  .AllowCredentials());
});

The Angular Project

This project was the trickiest to get running in a Docker container. As mentioned above, I had to add the bash script lines to the generate Dockerfile. I also had to update the ClientApp\app\components\fetchdata\fetchdata.component.ts file to account for the API call instead of a local call. In addition to updating the call I also added an app.config.ts file to ClientApp\app\ which held the variable ‘apiBaseUrl’ in a class, so that the “http://localhost:PORT/ value could be configurable and global. The reason this project is separated is because it is still served by .NET Core, meaning it uses Startup.cs, Program.cs and a HomeContoller to bootstrap the routing of the Angular SPA, similar to how Express might be used in NodeJS.

As previously mentioned, in theory this project could be moved entirely out of .NET Core, and run solely on a NodeJS web server which could then communicate to the existing API back-end. In this scenario, I could see the Front-end Angular project being excluded from the Visual Studio solution, and possibly developed in a lighter-weight editor with better NodeJS support, like Sublime. The final trick was making sure that the port was predictable for the API container so that the Angular calls could reach, which I will discuss in the next section.

The Docker Compose files

The setup of the docker-compose.yml was pretty straight-forward based on the generated file with a few modifications:

services:
  angular.web:
    image: angular.web
    build:
      context: ./Angular
      dockerfile: Dockerfile
    depends_on:
      - angular.api

  angular.api:
    image: angular.api
    build:
      context: ./Angular.API
      dockerfile: Dockerfile
    depends_on:
      - db

  db:
    image: "microsoft/mssql-server-linux:2017-latest"
    environment:
        SA_PASSWORD: "SuperSecretPassword"
        ACCEPT_EULA: "Y"

I combined all 3 containers into 1 file and added the depends_on sections, which helps ensure the containers are launched in the proper order.

The other file of importance that is included is the docker-compose.override.yml file. This allows you to set some development overrides. Following the template that was auto-generated I ended up with:

services:
  angular.web:
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
    ports:
      - "80"

  angular.api:
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
    ports:
      - 4828:80 # Explicitly map this port because Angular.Web project expects it 

  db:
    ports:
      - 1433:1433 # Explicitly map this port because SSMS expects it 

Here is where I was able to explicitly define both the API port, and the DB port so that it would be predictable and constant, not auto-assigned by Docker.

SQL Server Container

Getting the initial SQL Server container up and running was pretty straight-forward thanks to the surprisingly comprehensive documentation and examples from both Microsoft and Docker. However, in my initial attempt to get the container running I ran into a bug which I could not get around and finally assumed must have been in the Docker for Windows platform or SQL Container. It centered around the Memory requirement (4GB RAM) for the SQL Server container which I clearly met. I was able to figure out the problem thanks to rabih who posted the code to print the stdout messages in the powershell console:

docker run -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=abc@123" -p 1433:1433 -a stdin -a stdout -a stderr microsoft/mssql-server-linux:2017-latest

After letting a few versions pass, I updated all my clients and all was well and running.

Next Steps

The next steps I would like to take with the projects in this solution are (in no particular order):

add full database support to the API so data can be pulled out of tables
continue to improve upon the front-end project by adding other NodeJS libraries and tools
look into swapping the Angular frontend for React frontend based on some feedback from a front-end team-member

Hope you learned something with me; till next time.

A Windows Docker Container Series (Part 1)

2017-06-26T05:05:00-06:00

In order to begin my journey into exploring Docker, I felt the need to incorporate as many of the 12 Factor principles as I could into my development pipeline. I also wanted to have a clean place to get started on a Docker workflow, that was isolated from my normal .NET Development environment so as not to conflict, or pickup and unneeded dependencies. I had heard a lot about Hashicorp, and the tools they build for “provisioning, securing, connecting, and running any infrastructure for any application”. So, I decided I wanted to start there, with a fresh Packer Image that could be built, and reused from “scratch” as many times as I would need.

An Introduction to Packer

Packer is an open source tool that was made to create machine images (“single static unit that contains a pre-configured operating system and installed software which is used to quickly create new running machines”) for multiple platforms from a single .json configuration file. It allows for faster deployments because provisioning time is eliminated, by allowing machines to pre-provisioned and configured, ahead of time, by Packer. This can also improve stability of deployment because Packer installs and configures all software for a machine when the image is built. If there are any issues with install scripts or configurations, they’ll be caught early, rather than several minutes after a machine is launched. Due to nature of how Packer creates machine images it allows for identical images to be created for multiple platforms (GCP, AWS, OpenStack, VMware, VirtualBox, HyperV, etc.), as transitioning between them can be nearly seamless.

Principle #10

12 Factor principle #10 is on the “Dev/Prod parity”, and by keeping any images used identical between environments, we can take a large step towards reducing the gap that commonly exists. The. json which describes the machine images can also be used in a CI/CD scenario to have a blank machine created with each new build as a way to prevent configuration drift.

Why does it matter?

I believe this can be a crucial step in reducing the “works on my machine but not on his/hers” issue that is common in development. I also feel that in the context of Agency business, it would be unreasonable to try and have someone maintain VMs for each client’s environment (especially at scale), in order to cater to their specific needs or requirements. Therefore, we end up with shared environments. With Packer, it would be much easier to maintain a few Packer files in the client’s repo that could be an exact environment, which matches prod, for any developer joining the project, no background knowledge required.

Walkthrough

Windows Requirements

To start off, I began learning about what would be required to run Docker Containers on Windows. Windows Containers are offered with two different base images, Windows Server Core and Nano Server. There are also 2 different mythologies supported for running containers “Windows Server Container” and “Hyper-V Container”. Windows Server Containers share the kernel between containers and the host, therefore mismatched versions are not supported. Hyper-V Containers utilize their own instance of the Windows kernel so you can miss-match the container host and container image versions. That lead me into determining which Windows version would support which of the different base-images, and container methodologies. Microsoft offer the following table under their “Windows Container Requirements” documentation:

Host Operating System	Windows Server Container	Hyper-V Container
Windows Server 2016 (Standard or Datacenter)	Server Core / Nano Server	Server Core / Nano Server
Nano Server	Nano Server	Server Core / Nano Server
Windows 10 Pro / Enterprise	Not Available	Server Core / Nano Server

Based on this chart I felt it would be best use a workflow centered around Windows Server 2016, as it had the broadest support, covering all variations, of running Docker Containers on Windows. After determining I wanted to use Windows Server 2016 as my staring point I began to browse GitHub to see if I could find any existing Windows Server 2016 Packer templates.

Packer Template

Packer.io had the below template for getting started with Hyper-V (which is what comes by default in Windows 10 Enterprise, so that’s what I went with):

{
  "builders": [
    {
      "vm_name":"windows2012r2",
      "type": "hyperv-iso",
      "disk_size": 61440,
      "floppy_files": [],
      "secondary_iso_images": [
        "./windows/windows-2012R2-serverdatacenter-amd64/answer.iso"
      ],
      "http_directory": "./windows/common/http/",
      "boot_wait": "0s",
      "boot_command": [
        "a<wait>a<wait>a"
      ],
      "iso_url": "http://download.microsoft.com/download/6/2/A/62A76ABB-9990-4EFC-A4FE-C7D698DAEB96/9600.16384.WINBLUE_RTM.130821-1623_X64FRE_SERVER_EVAL_EN-US-IRM_SSS_X64FREE_EN-US_DV5.ISO",
      "iso_checksum_type": "md5",
      "iso_checksum": "458ff91f8abc21b75cb544744bf92e6a",
      "communicator":"winrm",
      "winrm_username": "vagrant",
      "winrm_password": "vagrant",
      "winrm_timeout" : "4h",
      "shutdown_command": "f:\\run-sysprep.cmd",  
      "ram_size": 4096,
      "cpu": 4,
      "generation": 2,
      "switch_name":"LAN",
      "enable_secure_boot":true
    }
  ],
  "provisioners": [
    {
      "type": "powershell",
      "elevated_user":"vagrant",
      "elevated_password":"vagrant",
      "scripts": [
        "./windows/common/install-7zip.ps1",
        "./windows/common/install-chef.ps1",
        "./windows/common/compile-dotnet-assemblies.ps1",
        "./windows/common/cleanup.ps1",
        "./windows/common/ultradefrag.ps1",
        "./windows/common/sdelete.ps1"
      ]
    }
  ],
  "post-processors": [
    {
      "type": "vagrant",
      "keep_input_artifact": false,
      "output": "_windows-2012r2_chef.box"
    }
  ]
}

Also, want to give credit to the most helpful public Github repo I came across from Stefan Scherer: Packer Windows.

Since this file is the main configuration file, I will do my best to explain what it’s doing.

The first section defines a builder object. Builders are how you write definitions for creating different machines, and for generating an image for plaforms such as Azure, Docker, EC2, VMware, VirtualBox, and of course, Hyper-V.

Within the builder, there are many parameters defined, without going into them all indivdually, they provide Packer with the information needed to create a machine image via the command-line on the specified platform. In this case we are creating a Windows Server 2016 image so we need to provide Packer all the variables need to perform a “silent install” of the OS.

The next section defines a provisioner object. According to Packer documentation, “Provisioners use builtin, and third-party software, to install, and configure, the machine image after booting to prepare the system for use”. Commom uses are:

installing packages (zip utility, windows features)
patching the kernel (installing Windows updates
creating users (creating the default ‘vagrant’ user)
downloading application code (Docker)

The final section defines an optional post-processor object. Post-processers are used to do things with your image after they have been prepeared. In my case, like the example from Packer above also, I chose to use the Vagrant post-processor, which prepares the Hyper-V VM for use with another Hashicorp tool Vagrant in conjunction with Hyper-V.

Running Packer

The brilliance of Packer is in it’s simplicity. Once the configuration is complete, all one has to do to create the machine image is the following command from PowerShell using elevated privileges:

packer build -var 'hyperv_switchname=Ethernet' .\windows_2016_docker.json

I specified as a variable my exisitng Hyper-V switch, so that when the machine boots during the install, it is able to connect to the internet for Windows Updates (optional), and various other tasks in the provisioning step.

Next Steps

The next steps I would like to take with packer are (in no particular order):

continuing to add provisiong scripts which add my presonal preferences to the Windows Server machine.
create new builders for other plaforms so I can ensure my machine image is able to run anywhere
use my Packer Image Builds in a full CI/CD workflow with Jenkins and Kubernetes

Hope you learned something with me; till next time.