Kevin LaBranche

Azure Podcast Guest– Leading Teams Through DevOps

Kevin LaBranche — Thu, 06 Jul 2023 00:14:03 GMT

I’ve been a part of an Architect Forum that Clear Measure (@ClearMeasure) and Jeffrey Palermo (@Jeffreypalermo) have been hosting monthly. With many conversations over these forums Jeffrey asked me to join him on his podcast. I am honored to join the multitude of amazing alums on the podcast and definitely feel inadequate. However, I won’t let that stop me and don’t let that stop you. Show up, give your best, it will help someone and help you grow as well. We all have something to share. While we conversed about DevOps and build scripts the underlying thread is how I approach my architecture position with mentorship as a very strong and necessary component to success.

Listen to this Azure DevOps Podcast episode, Leading Teams through DevOps, with Kevin LaBranche. https://t.co/Ai9fOiFa0X

Learn what a build script is @jeffreypalermo @klabranche #azuredevopspodcast #devops #buildscripts #API pic.twitter.com/X77GK9zqUL
— Clear Measure (@ClearMeasure) July 4, 2023

Conventions For Using Variables Groups In Azure DevOps

Kevin LaBranche — Thu, 15 Jun 2023 00:02:01 GMT

Variable groups allow for sharing of configuration values that can be shared by all pipelines and releases in the same project.

I have authored and maintained a few hundred Azure DevOps pipelines and releases over the last 3-5 years. One of the conventions that I developed is around the use of variable groups.

The conventions I have set in our team provide for quick recognition of where a value is sourced from. The magic of variable groups matching on their name to replace a value in a configuration for a stage is too magical. It’s super convenient until there’s a mismatch. Not all tokenize replacement tasks are created equal either. Some can be configured to error with mismatches, some can’t. I have found this can easily be a source of wasted effort when troubleshooting an issue and/or worse the green release but a configuration value didn’t get replaced.

Conventions

1. Add a prefix of your choosing to the name of each variable in the group.

2. Explicitly map variables in your pipeline to them.

3. Use Release scope for variable groups mapped to stages.

4. How to override a variable group value.

5. Don’t lock a variable (make it a secret, pad lock icon) if it really doesn’t need to be hidden including logs.

6. If you have to lock a value consider instead using Azure Key Vault and link it to a variable group.

7. Create a developer wiki for the variable groups.

Add a prefix

Adding a prefix like vg-variableName quickly allows a team member to know where this value is sourced from. If it’s just a value directly sourced in the pipeline variables we don’t add a prefix.

Bonus – If you have infrastructure based configurations such as what to name an IIS application, I recommend you prefix that as well.

The prefixes or lack thereof is information that the team can use to more quickly assess pipeline variables.

Explicitly map

By explicitly mapping our variable’s used in a variable group in our pipeline variables we remove the magic in replacing variables by name match for groups. At a glance we know if variables are coming from a group and which one’s in the group we are using. While this means some initial extra setup, it pays long term dividends in maintenance and troubleshooting.

Our team chose symbols for brevity sake for our prefixes.

Underscore _VariableName means it’s an IIS infrastructure configuration. We deploy to many shared IIS environments with the Manage IIS and Deploy IIS tasks.

Caret ^VariableName means it comes from a variable group.

No prefix, it’s a local variable.

Use Release scope for variable groups mapped to stages

I covered the next two conventions in an earlier post. If your variable groups are mapped to stages you can use one definition in your pipeline variables scoped to Release. The stages will properly map.

This:

Not this:

How to override a variable group value

Given a pipeline variable mapped to a variable group is using the Release scope one can then overwrite a stage by adding a staged scoped variable and value.

Doing this makes it obvious we are overriding and makes it easy to remove and go back to the variable groups value. It also avoids having to look through logs to see what was used if one checked the “settable at release time” option.

Don’t lock

As a starting point, don’t lock if it isn’t a secret. If it is a secret, consider this carefully. My reason isn’t to thwart proper security concerns. I’ve seen too many times where the actual secret was not stored elsewhere and/or where it was stored elsewhere was no longer accurate. Once it’s locked, you will never see it again in DevOps. Troubleshooting this especially if one does not have a way to verify the configuration is rough.

Instead of locking use Azure Key Vault

If you determine that the secret must be protected from casual viewing then please use a mechanism where the source feeds the variable group in a secure fashion. Azure Key Vault is a great option for this.

Create a Developer Wiki for the Variable Groups

Document the variable groups, share it, make sure the developers know about them. I’ve ran into countless cases of pipeline variables that were setup with local values when a variable group existed.

Learn More

Add & Use variable groups
Connecting Azure Key Vault to a variable group

Node rookie discovery with NPM and NODE_ENV Production

Kevin LaBranche — Thu, 08 Jun 2023 00:38:28 GMT

To the node pro this post will not surprise.

TLDR; If NODE_ENV is set to production npm install and npm ci commands won’t install developer dependencies without passing --include=dev (-–also=dev for npm older than v7).

Why would you install developer dependencies in a Production environment? Agree, 100%. However, we do, gulp in particular. Not that I’m happy about it either.

This functionality isn’t hidden but it is interesting how the instructions returned when we tried to use a developer dependency (gulp) ended up throwing the team for a loop and a few wasted cycles.

Story Time

It’s the age ole story of unwritten knowledge within the company, lack of expertise and lack of keeping things up to date.

For us, it all started about seven years back during an ITS merger. The merger brought several disparate development teams into one and those teams used different development stacks (Java, Node, .Net). The team members from those teams continued to maintain the apps until there were no more staff from those teams left. Some apps were retired, some were rewritten to the chosen platform (.Net), some remained entombed in times gone by. We did a fairly good job with documentation and even had recorded trainings from the staff leaving.

Several years went by, apps happily running, a few glitches here and there with failures that we rectified through normal operational tasks. I’m a node newb and so are my fellow team members who had the task of updating one of these node applications. The process to update the node app is not what I would consider modern. There is no CI/CD chain.

The process to update the application on the server(s) was:

1. git pull to get the latest code.
2. npm ci
3. gulp to do all the needed minifying, bundling, copying, etc.

The typical local development machine steps or on a build server. We happily updated the code needed and dutifully followed the process laid out on our development and test servers. We moved to the production systems and when we ran the gulp step we got:

Local gulp not found in <application folder>

Try running: npm install gulp

Scratching our heads, we compared dev/test and prod environments. All seemed properly aligned. We poured over our docs again not finding anything. So we ran `npm install gulp` and we got: updated 1 package in 26.605s

To our surprise when we manually ran our `gulp` command we still got: Local gulp not found in application folder Try running: npm install gulp

So after more research and not finding anything we tried a force install. No change. We could see that gulp wasn’t in the node_modules folder.

The Aha Moment

We found in the npm-ci docs in the omit parameter section the following two pieces of information.

“The default state is to omit developer dependencies when NODE_ENV is set to production.” Makes sense except it’s telling us it updated gulp?

“Note that these dependencies are still resolved and added to the package-lock.json or npm-shrinkwrap.json file. They are just not physically installed on disk.”

Totally not what we would expect to occur. Long story short, it resolved and updated the package-lock.json file but didn’t actually install.

To install developer dependencies when in production mode pass `—include=dev` ( `-–also=dev` for npm older than v7) to have it REALLY install your developer dependencies to disk.

Learn More

https://docs.npmjs.com/cli/v9/commands/npm-ci#omit

https://worknme.wordpress.com/2021/12/19/how-to-install-dev-dependencies-in-node-env-production/

Azure DevOps - Library Variables, Stages and Release Scope–Did you know?

Kevin LaBranche — Thu, 01 Jun 2023 23:53:29 GMT

Our team has over 100 different repositories (applications) with pipelines and release definitions in Azure DevOps. We have many on-premise web servers that share hosting the applications. With this setup we have taken advantage of using Library / Variable Groups to share settings that are reused. In our team we recently “rediscovered” a little gem in how Azure DevOps works with library variables / variable groups, stages and Release scope. I wanted to share as there was confusion in the team that was adding unnecessary pipeline variable setup.

Pipeline Variables

Given a three stage pipeline like dev->test->prod and you need to define a variable:

For a variable that is the same for all stages, use the Release scope. AppSettings.AppName for example in the above snippet doesn’t change for the stage.

For a variable that is different for each stage, define it three times, setting the value differently per the stage.

Straight forward, no fuss. What about variables groups?

Variable Groups

Every release definition of ours has variable groups that describe the iis deployment information such as the root folder path for where we “install” the application folder and other IIS information that we have normalized for all the applications on that host (deployment group and/or environment). We also have settings for our single sign on system that all the applications use and it’s settings for the various stages (dev/test/prod).

If the information for each stage does not change then we could pick the Release Scope when we setup the Variable Group. Straight forward, no fuss.

If you are using variable groups scoped to a stage then do you have to declare the variable per stage, three times (dev/test/prod)?

Below, we have hooked up variable groups to each stage (dev/test/prod) for our .Net Core apps logging settings.

So do you create three pipeline variables, one per stage in the pipeline variables? Since the values are different and/or scoped to the stage, this seems like the logical choice and you can but it’s just adding clutter.

You can define them once and use Release for the scope and the variable groups use the scoped value per the stage during the release to each stage*.

* I know someone reading this is saying BUT BUT if the variable name (Logging.LogLevel.Default) in the variable group matches with what is in your settings file you don’t have to reference it at all in pipeline variables. Correct. However, there are times you might have to due to a mismatch or as we have by convention chosen to. You might notice we start our variable reference with a caret ^. This is part of the convention we have adopted. More on that in a later post.

Overriding the Release scope

Now, what if you wanted to override a value for a stage that has a Release based scope?

Add the new pipeline variable, set the stage and set the value to override with. The variable group value for the stage will be overridden with your new value. This is a great option when you are deviating from your convention or need to have a different setting for a while. It’s easy to cleanup. Simply delete the pipeline variable for the stage when done. We use this technique for a 3rd party vendor solution connection string that changes during their upgrade cycles. During the upgrade timeline we override then go back to normal when the upgrade cycle completes.

You can also mark the Release scope variable as “settable at release time” instead of creating a stage variable override like shown above. It is “faster” and if one was doing it for quick troubleshooting, no problem. For deviating from the normal/convention or prolonged usage I prefer an overridden stage variable. The extra information provided by this tells anyone looking at the variables this isn’t “normal”. It’s also less obtuse than having to view the logs of a release to see what the setting was set to at release time when using the “settable at release time” option.

Never, mix the two approaches for the same variable. The overridden stage variable will win which might feel counterintuitive. I’ll explain more on our conventions around variable groups in a later post.

Learn More

Custom Variables and Scope
Variable Groups (Library)

Managing a Windows File Share as part of your Azure DevOps Release

Kevin LaBranche — Thu, 25 May 2023 19:49:10 GMT

It’s important to have everything possible defined in your release or in an Infrastructure As Code (IAC) process. The holy grail is immutable infrastructure but that isn’t always possible and maybe not worth trying to achieve. We still have many on premise IIS virtual machines that host several applications and the IAC setup handles the basic setup and configuration of IIS and the Windows environment itself only. Nothing for the applications.

In this situation, we have tried to setup our releases to be Application Infrastructure As Code (AIAC). If we have to move to another IIS environment we can change our deployment group and/or environment and deploy. We get the added benefit of the release serving as further documentation to the requirements for the application that often go undocumented and makes for lift and shifts an exercise in frustration and rediscovery. I call that application archeology. Really not a position anyone wants to be in.

In an earlier post I described how one could add a separate virtual directory as part of the release. A few of our applications also have file shares including a case where a virtual directory is also an upload folder for staff to include some very large files for the application to use. Let’s get that into the release.

A PowerShell task is well suited to accomplish this. I’ve included links in the Learn More section that goes over the nuances of file share rights and setup with PowerShell.

Create a PowerShell task that is after the task that creates the folder you want to share. That’s about the only prerequisite. It can go anywhere in the task step order otherwise.

We chose to use the inline type with the below code:

$shareExists = Get-SmbShare -Name upload -ErrorAction Ignore
if($shareExists -eq $null)

{
   New-SmbShare -Name upload -Description "Upload For ABC App" -Path c:\somefolderpath
   Grant-SmbShareAccess -Name upload -AccountName domain\adgroup1 -AccessRight Change -Force
   Grant-SmbShareAccess -Name upload -AccountName domain\adgroup2 -AccessRight Change -Force
   Revoke-SmbShareAccess -Name upload -AccountName Everyone -Force

}

else

{
   Set-SmbShare -Name upload -Description "Upload For ABC App" -Force
   Grant-SmbShareAccess -Name upload -AccountName domain\adgroup1 -AccessRight Change -Force
   Grant-SmbShareAccess -Name upload -AccountName domain\adgroup2 -AccessRight Change -Force
   Revoke-SmbShareAccess -Name upload -AccountName Everyone -Force

}

$ACL = Get-ACL -Path "c:\somefolderpath" $AccessRuleOwner = New-Object System.Security.AccessControl.FileSystemAccessRule("domain\adgroup1","FullControl","ContainerInherit,ObjectInherit","None","Allow") $AccessRuleStaff = New-Object System.Security.AccessControl.FileSystemAccessRule("domain\adgroup2","Modify","ContainerInherit,ObjectInherit","InheritOnly","Allow") $ACL.SetAccessRule($AccessRuleOwner) $ACL.SetAccessRule($AccessRuleStaff) $ACL | Set-Acl -Path "c:\somefolderpath"

First, we try to get the share and if it doesn’t exist we set it up (New-SmbShare) and then setup the rights to the share, not the underlying folder/files. If it does exist, we update the share. Then we setup the folder/files in the underlying path for the share’s access rights. We have an “owner” we setup with full rights, say our team member’s group and then a second group that represents the users and carefully set them up with change rights to everything below the root path but not the root itself.

Share access rights and folder/file access rights are different and the most restrictive rights will win.

Share access for the adgroup2 will have contribute since they can’t modify the root path but adgroup1 has read/write since they can as we granted them full rights to the root and below.

Share rights

Folder/File rights:

The PowerShell could be improved upon. Instead of forcing an update we could interrogate the settings and only update if it’s changed. Additionally, what if we wanted to use a new folder path or share name? We’d want to add some concept of old/new and clean up after ourselves. We didn’t do either in our case since these are long established paths and shares. We don’t plan on changing them anytime soon and if we did the existing release is now clearly documented making it easy to amend for such a change.

Learn More

Managing Windows file shares with PowerShell
How To Manage NTFS Permissions With PowerShell
Differences Between Share and NTFS Permissions
NTFS ACL's: What is the difference between object and container inheritance?
ACL Propagation Rules
PropagationFlags Enum
InheritanceFlags Enum
Setting Inheritance and Propagation flags with set-acl and powershell

Adding a virtual directory with your IIS web application manage task in Azure DevOps

Kevin LaBranche — Fri, 19 May 2023 05:17:00 GMT

Using the IIS web app manage task in Azure DevOps offers an easy and idempotent way of defining your web application on an IIS server. You can think of it as Application Infrastructure As Code. This task defines how the application should be setup/configured on an existing IIS server. It allows for a deployment to a new IIS server to seamlessly work.

What if you need to add a virtual directory as part of this setup? An upload folder for example located outside the web applications virtual directory.

Let’s take a look at three options, using the “Additional appcmd.exe commands” in the IIS web app manage task itself, using a second IIS web app manage task or a PowerShell task.

Of these options the “Additional appcmd.exe commands” immediately seems the way to go since it’s part of the setup task for the web app.

TL;DR – Use a second IIS web app manage task for most cases.

Using the “Additional appcmd.exe commands” in the IIS web app manage task itself

I really want to like this option. However, this option is too limiting to work for a scenario to add or change the virtual. The IIS web app manage task itself is idempotent*. It will add if an object doesn’t exist, change it if it does. * (maybe not fully idempotent…)

A naïve appcmd to use:

add vdir /app.name:”mywebsitename/mywebapp” /path:”/content” /physicalpath:"c:\somefolderpath”

This will work the first time but subsequent deploys will fail with duplicate collection element “/content”.

You could change it to be a set statement but that will fail if the virtual directory doesn’t exist.

The text area only allows for appcmd commands but these commands can be piped together and the appcmd has the /in parameter to pass items to the next command in the pipe. Since I’m trying to setup the release to be idempotent as much as possible this is looking a bit grim.

What about Pipes and /in for appcmd?

So, let’s see if the virtual exists and set it if it does.

list vdir /physicalpath:"c:\somefolderpath" /xml | set vdir /in /physicalpath:"c:\somefolderpath" /path:"/content"

Great, that works. However, what if the virtual doesn’t exist? This could be my own failing but it doesn’t appear possible to pipe this out in a way to add. It also doesn’t appear possible to if/else to do a set or an add with pipes and the appcmd.

What about piping to other functions? How about list piped to find with an or to add for the not found case.

list vdir /physicalPath:"c:\somefolderpath" 
| findstr "somefolderpath" || add vdir /app.name:"mywebsitename/mywebapp" /path:"/content" /physicalpath:"c:\somefolderpath"

You’ll get errors about invalid tokens. You can only use appcmd commands in this text area.

While one can pipe appcmd’s together and use the /in parameter for input to the next pipe, the inability (or at least my inability) to change the appcmd from an add if the virtual doesn’t exist to a set if it does makes this a no go.

What about Continue On Error and Run this task Even if a previous task failed combination?

Well, one could include add and set as two different lines in the text area and set Continue On Error for the task and a downstream task to run Even if a previous task failed. However, this has the downside of hiding real errors with this task. Plus, I can’t stand seeing the partially succeeded icon. It always make me think something is broken and must be addressed.

I even tried to NUL out the output to swallow the error.

add vdir /app.name:”mywebsitename/mywebapp” /path:”/content” /physicalpath:"c:\somefolderpath” > NUL

No go, errors. So, at the moment, I don’t see this as a viable option for adding or setting a virtual directory.

Using a second IIS web app manage task

You can have more than one IIS web app manage task. Let’s create a second one after the initial task to setup the web app.

On this task, select for the Configuration type, IIS Virtual Directory.

The parameters might be a bit confusing if you know you way around the appcmd but for the parameters as used in above examples:

Parent website name: mywebsitename

Virtual Path: /mywebapp/content

Physical Path: c:\somefolderpath

Since the task is smart enough to add or change we have achieved the desired result quickly and easily.

YAML example

If you happen to be using YAML, below is the equivalent commands for the IIS Web App Manage Task:

- task: IISWebAppManagementOnMachineGroup@0
  displayName: 'Manage IISVirtualDirectory - Add or Set content virtual directory'
  inputs:
    IISDeploymentType: IISVirtualDirectory
    ParentWebsiteNameForVD: 'mywebsitename'
    VirtualPathForVD: '/mywebapp/content'
    PhysicalPathForVD: 'c:\somefolderpath'

Using a PowerShell task

I like PowerShell. I’m not the best at it but I use it often for automation tasks around DevOps. So how would we accomplish with a PowerShell task.

The code below was heavily inspired by the source for the IIS web app manage task.

$vdirNameToFind="mywebsitename/mywebapp/content"
c:\windows\system32\inetsrv\appcmd.exe list vdir /vdir.name:$vdirNameToFind
$vdir=c:\windows\system32\inetsrv\appcmd.exe list vdir /vdir.name:$vdirNameToFind

if($vdir -ne $null -and $vdir -like "*`"$vdirNameToFind`"*") # like needed as vdir name can match parent vdirs { Write-Host "virtual found, updating" c:\windows\system32\inetsrv\appcmd.exe set vdir /vdir.name:$vdirNameToFind /path:"/content" /physicalPath:"c:\somefolderpath" } else { Write-Host "virtual not found, adding" c:\windows\system32\inetsrv\appcmd.exe add vdir /app.name:"mywebsitename/mywebapp" /path:"/content" /physicalPath:"c:\somefolderpath" }

The source from the IIS web app manage task has a far better flushed out solution so I recommend using the task but maybe one needs to do other work in the PowerShell task and having it all it scripted is desired.

Learn More

IIS Web App Manage Task
Getting started with AppCmd
Piping AppCmd

Passing variables between PowerShell file tasks in classic Azure DevOps pipelines

Kevin LaBranche — Fri, 12 May 2023 23:03:32 GMT

I ran into a bump when trying to pass variables around between PowerShell files in two tasks in the same job. I found the PowerShell task acts differently when using inline vs file path type.

Inline vs File Path

Set variables in scripts - Azure Pipelines | Microsoft Learn shows how to pass variables around between inline tasks in the same job.

set it:

Write-Host "##vso[task.setvariable variable=myVar;]foo"

Subsequent inline tasks in the same job can use it: Write-Host "myVar is $(myVar)"

File Path Type

Referencing variables in File Path Type PowerShell Tasks changes slightly. If you try to access the variable $(myVar) like the inline example you get an error:

myVar : The term 'myVar' is not recognized as the name of a cmdlet, function, script file, or operable program.

There are two ways to accomplish this with the File Path Type.

Environment Variable

Use the environment ($env):

$env:myVar

Output Variable

Use an output variable and pass it as a parameter into the next script task. This also works for tasks not in the same job.

The syntax to set the variable in the Set it step’s script file changes to:

Write-Host "##vso[task.setvariable variable=myVar;isoutput=true]foo"

In the task, set the Reference name in the task to a name of your choosing. I used setOutput.

The consuming task, uses an argument in the consuming PowerShell file and uses the Reference Name from the previous task. If you called the argument in the PowerShell file $myVarArg then you pass it in –myVarArg “$(setOutput.myVar)”

The PowerShell file needs to accept that parameter:

Param(

[string]$myVarArg

)

Write-Host "myVar: $myVarArg"

See Classic release and artifacts variables - Azure Pipelines | Microsoft Learn for more information.

The advantage of the second option is that it also works between jobs. If you are using the variables in the same job, consider using the environment variable.