• 00:28 A quick overview of PowerShell & AWS Lambda
• 01:07 Anatomy of a PowerShell Lambda function
• 01:35 Setting up a development environment
• 03:00 AWSLambdaPSCore modules
• 03:42 Packaging a PowerShell module
• 04:19 Handling the handler
• 05:14 Publishing your PowerShell Lambda function
• 05:42 Understanding the AWS Lambda filesystem

Resources:

• https://www.powershellgallery.com/packages/AWSLambdaPSCore
• https://dotnet.microsoft.com/en-us/download/dotnet/8.0
• https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell?view=powershell-7.4
• https://code.visualstudio.com/download
• https://github.com/PowerShell/PowerShell

• 00:00 Intro
• 1:20 The AWS Services that power the Meatgrinder
• 1:51 AWS S3
• 2:18 AWS EventBridge
• 2:50 AWS Step Functions
• 4:10 PowerShell on Lambda & Step Functions interaction
• 7:05 Python Lamba function
• 7:33 Logging success in AWS CloudWatch

Resources:

• https://vbrownbag.com
• https://aws.amazon.com/serverless/
• https://damiankarlson.com/2024/04/21/whats-in-the-bag-behind-the-scenes-at-vbrownbag-com/

In the previous iteration of the Meatgrinder process, new posts were added by email. However, I didn’t like the idea of using an email in the process because I wanted the new Meatgrinder to be as self-contained as possible, and using PowerShell’s Invoke-WebRequest with the REST API meets that requirement.

Some notes on the process: A WordPress application password is recommended as it simplifies authentication. Author, tags, and categories are their respective unique IDs in each WordPress database. Featured images can’t be uploaded along with posts; they have to be uploaded and then their unique image ID associated with the post’s unique ID. The thumbnail is read from S3 into the /tmp directory of the Lambda function at run time using Read-S3Object. The decrypt cmdlet in this sample is the helper function I wrote to decrypt AWS Lambda environment variables.

# Creating the credential object
$app_pwd = decrypt -ciphertext $env:app_pwd

$securepass = ConvertTo-SecureString $app_pwd -AsPlainText -Force

$credential = New-Object System.Management.Automation.PSCredential($env:app_user, $securepass)

# Begin building the post
$title = "New post title!"

$content = "Here's a brand new vBrownBag.com post!"

# Fun one-liner that creates an excerpt if the content is longer than 124 chars
$excerpt = $content.Length -gt 124 ? $content.Substring(0, 124) + " ..." : $content

# Retrieving the featured image from S3 as $thumbnail, and reading in the bytes
$thumbnail = Read-S3Object -BucketName $env:bucket -Key "$thumbnail.jpg" -File (Join-Path $env:TEMP "$thumbnail.jpg")

$thumbnailBytes = [System.IO.File]::ReadAllBytes($thumbnail)

# Uploading the featured image and retrieving the $imageId
$response = Invoke-WebRequest -Uri "$env:url/wp-json/wp/v2/media" -Method Post -Authentication Basic -Credential $credential -Headers @{
  "Content-Type"        = "image/jpeg"
  "Content-Disposition" = "attachment; filename=`"$thumbnail.jpg`""
} -Body $thumbnailBytes

$imageId = ($response.Content | ConvertFrom-Json).id

# Updating the featured image title & alt text for $imageId
$imageJson = @{
  "date"     = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
  "title"    = $title
  "alt_text" = "Featured image for " + $title
} | ConvertTo-Json -Depth 10

Invoke-WebRequest -Uri "$env:url/wp-json/wp/v2/media/$imageId" -Method Post -Authentication Basic -Credential $credential -Headers @{"Content-Type" = "application/json" } -Body $imageJson

$postJson = @{
  "date"           = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
  "author"         = $authorId
  "title"          = $title
  "excerpt"        = $excerpt
  "content"        = $content
  "comment_status" = "closed"
  "ping_status"    = "closed"
  "featured_media" = $imageId
  "status"         = $status # "publish" or "pending"
  "categories"     = @("1", "2")
} | ConvertTo-Json -Depth 10

$response = Invoke-WebRequest -Uri "$env:url/wp-json/wp/v2/posts" -Method Post -Authentication Basic -Credential $credential -Headers @{"Content-Type" = "application/json" } -Body $postJson

Next, let’s cover some XML code that I’m using to update our RSS file. The process is rather simple: download the latest version of the XML file to /tmp using Invoke-WebRequest, save a time-stamped backup copy to S3, read in the contents of the XML file, create a new entry, remove any entries that are than 2 years old, make sure there’s no more than 300 entries in the file (just in case!), write the updated XML to disk, then upload it to the website. The createRssElement helper function was from the original Meatgrinder script, and its job is to simplify XML element creation. Lastly, the Send-SFTPFile cmdlet is from the Transferetto module.

function createRssElement {
  param([string]$elementName, [string]$value, $parent, [string]$namespaceURI = $null)
  if ($namespaceURI) {
    $thisNode = $xml.CreateElement("itunes", $elementName, "http://www.itunes.com/dtds/podcast-1.0.dtd")
  }
  else {
    $thisNode = $xml.CreateElement($elementName)
  }
  $thisNode.InnerText = $value
  $null = $parent.AppendChild($thisNode)
    return $thisNode
}

$xmlUrl = $env:rssxml_url + $env:rssxml_filename
$xmlbackupFileName = (Get-Date -Format "yyyyMMddHHmmss") + " $($env:rssxml_filename)"
$xmlbackupPath = Join-Path $env:TEMP $env:rssxml_filename
# Download the file to /tmp
Invoke-WebRequest -Uri $xmlUrl -OutFile $xmlbackupPath | Out-Null
# Backup to S3
Write-S3Object -BucketName $bucket -File $xmlbackupPath -Key $xmlbackupFileName

# Read in the contents of the file
$xml = [xml] (Get-Content $xmlbackupPath)

# Remove anything with a bad date or more than 2 years old
$xml.rss.channel.Item | Where-Object {
  $pubDate = [DateTime]::MinValue
  $parseSuccessful = [DateTime]::TryParse($_.pubDate, [ref]$pubDate)
  if (-not $parseSuccessful -or $pubDate -lt (Get-Date).AddYears(-2)) {
    $xml.rss.channel.RemoveChild($_) | Out-Null
  }
}

# Create the new entry
$rssChannel = $xml.rss.channel
$thisItem = createRssElement -elementName 'item' -value '' -parent $rssChannel
$null = createRssElement -elementName 'title' -value $title -parent $thisItem
$null = createRssElement -elementName 'link' -value 'http://vbrownbag.com' -parent $thisItem
$null = createRssElement -elementName 'description' -value $description -parent $thisItem
$null = createRssElement -elementName 'guid' -value $url -parent $thisItem
$enclosure = createRssElement -elementName 'enclosure' -parent $thisItem
$null = $enclosure.SetAttribute('url', $url)
$null = $enclosure.SetAttribute('length', $mp4file.Size)
$null = $enclosure.SetAttribute('type', 'video/mp4')
$null = createRssElement -elementName 'category' -value "Podcasts" -parent $thisItem
$pubDate = Get-Date -Format 'r'
$null = createRssElement -elementName 'pubDate' $pubDate -parent $thisItem

# Sort by date and remove any duplicates
$orderedItems = $rssChannel.Item | Group-Object -Property pubdate | ForEach-Object { 
  if ($_.Group.Count -gt 1) {
    for ($i = 1; $i -lt $_.Group.Count; $i++) {
        Write-Output "Duplicate item: $($_.Group[$i].title)"
    }
  }
  $duplicatesCount += ($_.Group.Count - 1)
  $_.Group[0]
} |  Sort-Object { [DateTime]::Parse($_.pubDate) } -Descending

# Only use the first 300 because Apple doesn't like more than that
if ($orderedItems.Count -gt 300) {
  $orderedItems = $orderedItems | Select-Object -First 300
}

# Remove all items from the channel
$itemsToRemove = $xml.rss.channel.item.Clone()
foreach ($item in $itemsToRemove) {
  $xml.rss.channel.RemoveChild($item) | Out-Null
}
        
# Add the ordered items back to the channel
foreach ($item in $orderedItems) {
  $xml.rss.channel.AppendChild($item) | Out-Null
}

# Timestamp this newest version of the file and write to disk
$xml.rss.channel.lastBuildDate = (Get-Date).ToUniversalTime().ToString("r")
$xmlWriterSettings = new-object System.Xml.XmlWriterSettings
$xmlWriterSettings.CloseOutput = $true
$xmlWriterSettings.IndentChars = "    "
$xmlWriterSettings.Indent = $true
$xmlWriterSettings.NewLineOnAttributes = $true
$xmlWriterSettings.NewLineHandling = [System.Xml.NewLineHandling]::Replace
$xmlWriter = [System.Xml.XmlWriter]::Create($xmlbackupPath, $xmlWriterSettings)
$null = $xml.WriteTo($xmlWriter)
$xmlWriter.Close()

# SFTP the file to where it lives
$sftp_password = decrypt -ciphertext $env:sftp_password
$sftp_credential = New-Object System.Management.Automation.PSCredential($env:sftp_username, (ConvertTo-SecureString $sftp_password -AsPlainText -Force))
$sftpClient = Connect-SFTP -Server $env:sftp_host -Credential $sftp_credential -Port $env:sftp_port
Send-SFTPFile -SftpClient $SftpClient -LocalPath $filepath -RemotePath (Join-Path $env:sftp_destination $env:rssxml_filename) -AllowOverride

Well, I think I’ve reached the logical end of this series. We’ve covered a lot of ground, and I’ve learned a whole lot. Thanks for reading along with me!

Google OAuth with PSAuthClient

Some quick introductory info: there are two ways of authenticating with the YouTube data API. The first is through an API key created through https://console.cloud.google.com. It’s an alpha-numeric string that can be appended to REST API calls against endpoints such as https://www.googleapis.com/youtube/v3/videos. However, the API key doesn’t allow privileged access to an account, and for that you need to use OAuth.

There’s a ton of helpful documentation on Google OAuth, but my goal is not to repeat all of that here, nor talk about creating an app in Google’s Cloud console. Rather, I’d like to talk about how I used PSAuthClient to meet my OAuth needs using PowerShell. The two most important pieces of information needed to get started are the client_id and client_secret. The first uniquely identifies the Google app, and the second is a client secret string.

After installing PSAuthClient using Install-Module, I used Invoke-OAuth2AuthorizationEndpoint to make the initial OAuth request in order to begin building my $authorization object. The customParameters hashtable is important as I need to be able to refresh the OAuth token without requiring human interaction every time, since this is running in a Lambda function. Google OAuth also needs the client_secret in the $authorization object, so I add it to the object. Finally, I use the Invoke-OAuth2TokenEndpoint cmdlet to get the $token object by splatting the $authorization object (passing a collection of parameter values to a command as a unit).

$authorization = Invoke-OAuth2AuthorizationEndpoint -uri $uri -client_id $client_id -redirect_uri $redirect_uri -response_type "code" -scope $scope -customParameters @{ access_type = "offline"; include_granted_scopes = "true" }
$authorization.Add("client_secret", $client_secret)
$token = Invoke-OAuth2TokenEndpoint -uri $uri @authorization

$token is a PSCustom object that contains access_token and refresh_token strings. The access_token expires in about an hour, which means that I need to get a new one whenever I’m making an OAuth request. This is accomplished through Invoke-OAuth2TokenEndpoint.

$token = Invoke-OAuth2TokenEndpoint -uri $uri -refresh_token $refresh_token -client_id $client_id -client_secret $client_secret

Now I can begin building a new $youtubeService object for uploading new videos, assigning them to playlists, searching, etc.

$googleCredential = [Google.Apis.Auth.OAuth2.GoogleCredential]::FromAccessToken($token.access_token).CreateScoped($token.scope)

# Create a new YouTube service
$youtubeService = New-Object Google.Apis.YouTube.v3.YouTubeService(
    New-Object Google.Apis.Services.BaseClientService+Initializer -Property @{
        HttpClientInitializer = $googleCredential
        ApplicationName       = "YouTube Uploader"
    }
)

I’d like to note for the purposes of clarity, that I had to use Invoke-OAuth2AuthorizationEndpoint from my desktop, as it uses WebView2 to handle web interaction/approving the credentials. PSAuthClient is a PowerShell Core & Desk module which throws errors when I require the entire module in my Lambda script. However, Invoke-OAuth2TokenEndpoint is a PSCore cmdlet, so I’m able to include the .ps1 file directly in my Lambda zip file.

Encrypted environment variables in Lambda

As you can see above, I’m storing a lot of sensitive information in variables. I decided during the early stages of working on Meatgrinder to use Lambda environment variables instead of storing those values directly in the script, or a secrets file, etc. The variables & their values are defined in the AWS Lambda configuration panel. When the Lambda function runs, those variables are available to the script with the $env prefix, so the value of oauth_client_id is available as $env:oauth_client_id, and so on. For the sensitive variables, I chose to create an AWS KMS symmetric key to encrypt & decrypt my environment variables, and require the AWS.Tools.KeyManagementService module in my script, which includes the Invoke-KMSDecrypt cmdlet. Now you might be thinking, “oh, I can just use that cmdlet to directly decrypt my variable!” and you’d be wrong just like I was. There’s actually a few extra steps required.

First, the encryption context needs to be defined, and in this case, it’s the environment variable AWS_LAMBDA_FUNCTION_NAME. (The encryption context is defined in the execution role policy that allows the function the kms:decrypt action using the resource arn of the KMS key.) Then the ciphertext value stored in the encrypted environment variable needs to be converted from a base64-encoded string to a byte array. The context and the encrypted byte array are passed to the Invoke-KMSDecrypt cmdlet with the -Select Plaintext parameter which asks for the plaintext value instead of the whole service response (which is the default), and a decrypted byte array is returned. That byte array is converted to a string which is the decrypted plaintext value. That’s a lot, right? If you were setting up the encryption configuration for environment variables chose the “enable helpers” checkbox, AWS would present you with popup containing a .NET 8 snippet that does all of this, but it doesn’t show it in PowerShell. So, here’s the PowerShell equivalent:

<#
.SYNOPSIS
   Helper function to decrypt a ciphertext using the AWS Key Management Service (KMS).
.PARAMETER ciphertext
    The ciphertext to decrypt.
.OUTPUTS
    Returns the plain text
.EXAMPLE
   decrypt -ciphertext "ciphertext"
#>
function decrypt {
    param (
        [Parameter(Mandatory = $true)]
        [string]$ciphertext
    )
   
    $encryptionContext = @{"LambdaFunctionName" = $env:AWS_LAMBDA_FUNCTION_NAME }
    $cipherbytes = [System.Convert]::FromBase64String($ciphertext)
    $response = Invoke-KMSDecrypt -CiphertextBlob $cipherbytes -EncryptionContext $encryptionContext -Select Plaintext
    return [System.Text.Encoding]::UTF8.GetString($response.ToArray())
}

Well, if you read this far, congratulations! Or, hi Mom, whichever the case may be. In the next post, I’d like to cover WordPress REST API and RSS XML updates. Cheers!

One of this function’s primary design goals was for it to be entirely event-driven. I didn’t want to be concerned with giving future friends of the vBrownBag access to invoking the Lambda function directly, or to be concerned with any other details than simply uploading a video file along with a metadata file. However, I couldn’t quite figure out a graceful way to make sure both the .mp4 & .csv files were present before moving forward with the automation process. If you’ve dealt with AWS, you’re probably well aware that there are many different ways to solve a problem. That’s the benefits of AWS’ focus on cloud primitives (vs a cloud like Azure that tends to focus more on solutions). Here’s a few examples: 1.) I could’ve simply triggered the Lambda function from an S3 PUT and then let Lambda exit if both files weren’t present, but that would’ve introduced unnecessary Lambda executions that would’ve been unsuccessful half the time. 2.) I could’ve allowed a human to invoke the function directly, after uploading both files. This wasn’t a bad idea, but still required more interaction than I wanted. 3.) I could’ve had Lambda run regularly to check for new files in an S3 bucket, but that’s a clumsier approach. Honestly, the list could go on and on. I finally decided to build an AWS EventBridge rule that would start a Step Functions state machine execution after a .csv file is uploaded to an S3 bucket. As these notifications happen in millisecond time frames, the first thing Step Functions does is wait before asking the Lambda function to check if both files are present. Ultimately, I decided that being event-driven doesn’t mean that event responses have to happen immediately, as there’s no point in having millisecond responses to find files that aren’t done uploading.

As you can see in the graph, there’s an initial 5 minute wait to allow for large .mp4 files to finish uploading. After that, the Lambda function is invoked to check for matching .mp4 & .csv files in a private S3 bucket. If they exist, JSON is returned to the state machine indicating that the process can move forward. If not, then JSON is returned that it’s still waiting. This loop eventually breaks out and exits after running a few times, just to make sure that this doesn’t run indefinitely during an incomplete upload. Each JSON output is interpreted and acted upon by the Lambda function, which also constructs JSON responses to the state machine upon completion.

Here’s example JSON that EventBridge sends to Step Functions (using InputPath filtering on $.detail.object). Step Functions invokes the Lambda function with the JSON as input. Inside the Lambda function, the input is received as $LambdaInput which is a PSObject that contains input to the function’s configured handler method.

{
  "key": "test2-long.csv",
  "size": 644,
  "etag": "26f04fed485a8b0487b17974b313b52d",
  "version-id": "w9ywc77rdDz49dYmUPRRTEVzcQqAifIM",
  "sequencer": "00663D74124FC29E94"
}

The Meatgrinder Lambda function processes this input, checks to see if a matching .mp4 file exists, and then responds with JSON to the Step Function state machine. All of the JSON returned by the function is built as a PSCustomObject and then converted to JSON using ConvertTo-JSON.

{
  "action": "upload",
  "video": {
    "name": "test2-long.mp4",
    "privacyStatus": "public",
    "csv": "test2-long.csv"
  }
}

Meatgrinder refreshes the OAuth token used to authenticate to our YouTube channel, builds a Google.Apis.YouTube.v3.Data.Video object containing information about the video, uploads to YouTube, and then copies the .mp4 from the S3 working bucket over to our public bucket named as videoId.mp4, which reflects the unique ID that YouTube assigned the upload. At this point in the process, the YouTube data API is the source of truth for publish times, content duration, thumbnail images, etc. so an OAuth authenticated Google.Apis.YouTube.v3.Data.VideoListResponse object is converted to JSON and saved in the working bucket.

An additional wait period is introduced to wait on the YouTube data API to signal that the video is done processing. As there’s no exponential backoff in Step Functions, I chose a wait period of 15 minutes, as an hour of HD video may take 20-30 minutes to process. Once the API says the video has been processed, Meatgrinder is invoked again to finish the vBrownBag.com blog post & Apple Podcasts RSS update.

{
  "action": "finalize",
  "video": {
    "id": "ZshoNZd2-_g"
  },
  "status": "processed"
}

I spent some time trying to figure out how to make sure that the function was idempotent, as I didn’t want to accidentally duplicate vBrownBag.com blog posts or RSS entries. I solved this by appending a wpPost_Id key/value pair in the JSON which won’t trigger a new blog post if the value isn’t null (meaning, a blog post with that id exists). Additionally, by using the YouTube videoId as a unique identifier, I’m able to find any XML entries that match that videoId and not add a new one if it already exists.

Lambda function internal structure

In addition to the handler() function itself, the rest of the Meatgrinder Lambda function consists of PowerShell functions that are named like PowerShell cmdlets with approved verbs such as New-YouTubeVideo, Get-OAuthRefresh, Update-RSS, etc. I also wrote extensive comment-based help for the PowerShell functions. Parameters are defined (string, object, etc), consistently named, and enforced when passed between other PowerShell functions. I’ve also tried to consistently use try/catch blocks for error handling, and logging Meatgrinder process steps to AWS CloudWatch using PowerShell’s Write-Host. These were intentional choices designed to document the code and to make it easier to understand for future maintainers (or myself, in 6 months!)

Stay tuned for Part 5, where I’ll cover Google OAuth & PSAuthClient, decrypting AWS Lambda environment variables, and more. Thanks for reading!

I decided to write this Lambda function on my Windows 11 machine because I’d started this effort by writing code to “catch up” the website & the podcast RSS file, and naively thought that I could just move the code from Windows to Lambda. While it’s mostly true that the code is cross-platform there are some differences, such as case-sensitivity for paths/filenames/variables on Linux vs Windows’ case-insensitivity. Having the benefit of hindsight, I should’ve developed in an environment that resembles where the code was running as I probably would’ve had fewer portability issues.

That being said, here’s the environment I ended up with:

• Primary development on Windows 11 running PowerShell 7 and Visual Studio Code, connected to a GitHub repository and using GitHub Copilot
• Amazon Linux running in WSL2.
  • This was a multi-step process, as I spun up a Ubuntu distro on WSL, then installed Docker.
  • After Docker was setup, I installed Amazon Linux in Docker, then exported the filesytem and imported into WSL as the default distro. Instructions here. Note, I could’ve installed Docker on Windows to perform the import/export, but I didn’t.
  • The default Amazon Linux image runs a minimal set of packages, so I used yum to groupinstall “Development Tools”, and installed wget to help pull down the rest of what I needed.
• Both operating systems run the dotnet 8 SDK, aws.lambda.tools for dotnet, PowerShell 7, the modular AWS PowerShell modules, and the AWS Lambda Tools for PowerShell.
• Other PowerShell modules I’m using include PSAuthClient for handling OAuth2 tokens and Posh-SSH to move a file from Lambda to vbrownbag.com.

I prefer the modular AWS PowerShell modules as I can limit the module groups that are loaded into my Lambda function vs the monolithic AWS PowerShell module that includes everything but add unneeded start times to the function.

The dotnet 8 SDK is a requirement for packaging Lambda functions on any platform. The aws.lambda.tools for dotnet are very helpful in creating new dotnet projects that will run on Lambda. The AWS Lambda Tools for PowerShell are effectively wrapper cmdlets that simplify packaging of a function that lives in a single script, or is part of a bigger project.

Thoughts on GitHub Copilot

First off – Copilot is darn near magical at times. Being able to have a conversation about code, solving specific problems, or even getting answers to things like “why am I getting this error?” is simply amazing.

That said, it’s certainly not perfect. In fact, there have been a few times that I swear Copilot was drinking on the job. Sometimes it would answer in circles, give me bad ideas on how to solve things, forget the context of our conversation, or start using the “I’m just an AI” response when I asked it broader questions about my code, which is nearing about 1000 lines at this point. One time in particular stands out to me where it suggested a block of code that repeated a line twice. When I asked “why did you suggest doing this twice?” it denied that it did. When I said said that it absolutely did suggest it to me, it remembered that it did and then apologized about it.

I think it would be helpful if there was clearer communication that Copilot can’t “see” everything you’re working on because it isn’t always clear. Its responses will say that it’s referencing lines of code, but why it chooses those lines isn’t obvious if it’s based on cursor position, the question itself, or what’s in the VS Code view port. Copilot also doesn’t ask clarifying questions; often it will just give an answer that’s not appropriate in context of the code being written or where it’s running.

All of that said, I’m really excited to see where Copilot is headed. Having a tool like this one lowers the barrier to entry for everyone as Copilot can explain unfamiliar code, help translate from one type of code to another, and even translate into another spoken language. If you’re curious about trying out Copilot, as of this writing there’s a 30 day free trial for individual use.

Coming up next

I’ve spent a lot of time detailing what I want to do and the tools I’m using to do it. Next up in the series, let’s start looking at code with specific examples. Talk to you then!

Quick note

Before we get started, I’d like to talk about what’s going on in a general sense. The majority of the new meatgrinder functionality will be done in PowerShell. Why PowerShell? Well, it’s what Al wrote for the previous iteration of the meatgrinder and I want to continue using it because I love it. Most of the PowerShell Internet calls will either be Invoke-RestMethod or Invoke-WebRequest. The AWS calls will use cmdlets from AWS.Tools.Common and AWS.Tools.S3 for reading/writing/deleting S3 objects. Output logging writes to AWS Cloudwatch, due to the way that Lambda handles output. Packaging the PowerShell function for Lambda will be covered later.

Meatgrinder process

• After a vBrownBag recording is live-streamed, the YouTube recording is manually downloaded locally for editing & fancy on-screen graphics, YouTube shorts creation, etc.
• A file containing metadata such as video title, description, tags, etc. and an .mp4 file with same name (but different extension) are uploaded to a dedicated AWS S3 bucket via AWS CLI or the AWS S3 PowerShell module.
• The Lambda function is invoked via the AWS CLI along with a JSON payload that specifies the video.mp4 & video metadata file, and the function proceeds to do a number of things:
  • Verifies both the video file & metadata file exist in the S3 bucket
  • Parses the metadata to create variables with the title, description, tags, etc.
  • Adds on a promotional content text block to the description, refreshes the OAuth token and then POSTs the video to the YouTube API videos:insert endpoint, which responds in JSON. The unique YouTube video id is parsed from this response. Note: I’ll cover the OAuth token component in a future post about PSAuthClient.
  • Lambda asks for more information on that video id from YouTube’s videos:list endpoint which responds in JSON with video id, title, excerpt, long description, thumbnail details, publishedAt (and more) and saves a copy of the media file (named as $videoId.mp4) & $videoId.jpg thumbnail to our public S3 bucket.
    • I really just want the YouTube excerpt and publishedAt values versus reusing the original metadata, as I want the blog post & RSS feed entry to have the same excerpt & timestamp as it makes the blog post look cleaner in case of a longer description, or if I want to backdate the blog post in case this step is done later. An example of this would be the “catch-up” process I went through to get the website & RSS feed caught up on the last 6 months of the YouTube channel when the former iteration of the meatgrinder script was inoperable.
  • Lambda deletes the working bucket upload, as processing is done and any video & thumbnails in our public bucket should avoid name collisions anyway by way of YouTube’s naming conventions.
  • Lambda creates the vBrownBag.com blogpost from the YouTube details using the original description (without the promotional text block), and POSTs to vBrownBag.com using the WordPress REST API. More on that in a future post.
  • Lambda then POSTs the YouTube thumbnail image to be used as the featured media (the post thumbnail), then links the featured media to the post. The WordPress REST API doesn’t allow side-loading the media, so we have to create the post, upload the media, then associate the post & media.
  • Finally, Lambda grabs the Apple podcast RSS XML file, saves a backup to S3, adds the latest post to the top of the XML body, and sends it back to where it lives. I’ll show more on that process later, as it’s also pretty nifty. If you’re wondering “hey, what about us Android folks?!” the answer is that Google has decided YouTube will be its podcasting source too, so it’s effectively built-in to the vBrownBag YouTube channel.

Here’s a rather simplified flow that I made with Lucidchart. It’s not 100% exact, but it’s enough to get the idea across. In my mind, I’d like to actually map out the POSTs and responses, but that feels a bit too… extra.

That’s all for now. The next post in this series will be a brief overview of the development environment, change management, and the tools necessary to make all of this work.

A few weeks ago, I asked my good friends Alastair Cooke & Chris Williams how I could be useful to them & the vBrownBag.com website. Al suggested that I take a look at the automation script, affectionately known as “the meatgrinder” that had been running behind the scenes for years but hadn’t been running for about the past 6 months. It was written mostly in PowerShell and ran on a Windows VM and its job was to process a video file (such as one from a live vBrownBag TechTalk recorded at a conference, or a recording from the weekly video podcast). The script would upload the file to AWS S3, post to the YouTube channel, update the Apple Podcasts RSS feed, and post by email to the WordPress install that powers vBrownBag. Finally, Al wanted it to run as an AWS Lambda function.

Excitement intensifies

I was very excited, and still am, because I just love PowerShell. I find it fascinating and always have. I think it’s because I used to love bash scripting and PowerShell’s pipeline reminds me of that. Also, there’s super cool things like splatting that just make life easier. Looking back on the first few weeks of working through this process, I realize that I’ve learned way more than merely reacquainting myself with PowerShell in a very short period of time. And, due to the fact that I’m running PowerShell on Lambda (which is an Amazon Linux image underneath) I’ve run into some unique issues that there aren’t a lot of Google results about. So I decided that I should start writing these down in public for a few reasons. The first one is documentation for myself & any friend of the vBrownBag who may need to work on the meatgrinder in the future. Second, it’s for folks like me who are stuck trying to figure out things like the WordPress REST API, working through Google’s OAuth framework in order to post to YouTube, optimizing Lambda layers, and more.

Bad data & a good ending

However, before I get to all of that, I’d like to start with the first problem I had to solve which was (what appeared to be) corrupted data in the vBrownBag.com database. I’m not entirely sure when it happened, but there were a lot of characters that didn’t belong in the posts, pingbacks, and comments such as Â, €, or . There were even some non-printable characters that wouldn’t render properly in a decent text editor like Notepad++. I’m not sure if it was a bad automated restore, or a past friend of the vBrownBag that accidentally messed it up, but it was a problem that impacted the majority of the site content. I tried a simple find & replace but working with database dump files is a risky business and I could’ve easily made the problem worse. I did a lot of googling too, like any decent technologist would do. 😉 Finally, I found the name of the problem: mojibake. It’s garbled or gibberish text that is the result of text being decoded using an unintended character encoding. That took me down a wild rabbit hole full of arcane magic and dark spells, but eventually I stumbled onto this page. Turns out that a fellow nerd and former MySQL guy for Yahoo decided to write down all sorts of random database things on one very long page. The fix that I was looking for was in the “fixes for various cases” section. Specifically, this command:

UPDATE tbl SET col = CONVERT(BINARY(CONVERT(col USING latin1)) USING utf8mb4);

The command converts the col column to latin1 encoding, then converts it to a binary string, then converts the string to utf8mb4 encoding. The fix was as easy as that. I was amazed. Thank you, Mr Rick “MySQL Superman” James for helping to fix the website.

For my next post in the series, I’ll break down the meatgrinder steps into smaller pieces and more of the things I’ve learned along the way.

Edit: Read part 2 in the series. You know you want to.

A recurring theme of many conversations was VMware’s focus on AWS. In fact, one off-hand comment that resonated with me was “whose conference is this?” While that was said in jest, it is interesting to see how the relationship between VMware and AWS has shifted over the years from competing on all fronts to the realization that many customers’ IT strategies feature offerings from both companies and that joint solutions are needed.One of the biggest announcements was Amazon RDS on VMware, which should prove to be a rather interesting solution marrying the best of both VMware and AWS.

It will be interesting to see what the future holds for both companies because, while there are great synergies, there is still strong competition.

VMworld has published most (all?) of the content from the conference, and there’s a tremendous amount of sessions related to AWS in the catalog. One of the highest rated sessions covered migrating to VMware Cloud on AWS (video & PDF) and I found it to be quite interesting. There’s a ton of great sessions that have been published – too many to list! Be sure to check them out.

Community

As always, the VMworld community is one of the best things about the show. Two things that stood out to me – and I fully admit that I’m a little bit biased – were the VMunderground party and the live #vBrownBag TechTalks. Both have been going on for many years, and while this year was the first that I wasn’t involved in some way with either (whether it was organizing, presenting, or supporting) it gave me the opportunity to experience them with fresh eyes. The VMunderground party continues to be the best community networking event, and this year’s was no exception. The #vBrownBag TechTalks have continued to mature and expand, and its success is due to a combination of passionate leadership and an even more passionate technical community. There’s over 60 recorded sessions on #vBrownBag’s YouTube playlist, and they’re full of solid content, with minimal fluff. I highly recommend checking them out.

Earlier today I was trying to add application blueprints from the VMware Solution Exchange Marketplace to my Application Services appliance. The “Try” page for the blueprints features an option to login to your Application Services appliance and deploy the blueprint directly. That, however, doesn’t work. As it turns out, Marketplace access isn’t available on 6.1 (release notes).

In order to import the blueprints, download each file and upload it to the Application Services appliance. Then SSH into the appliance and start darwin-cli.jar. Login to the Application Services darwin instance and run the import-package command with the correct flags in order to import the XML. Be sure to SSH into the appliance itself, as import-package looks for importFilePath on the local machine.

Pay special attention to the ConflictResolutionAction flag as overwriting or skipping a piece of the import could have undesired consequences.