I’ve needed to make a YAML pipeline on Azure DevOps, using ADO Git hosting. It’s not the edgiest stack out there, but as a consultant it’s rare that you get to choose the tools you use. Sometimes you just need to get something automated today. It’s easy to set up pipeline triggers from changes in the Git repository:
trigger:
branches:
include:
- '*'
Later on, one wants to ensure that a job runs in Pull Requests, so that people can get feedback on those PR’s before they get merged: it’s a powerful way to go from “LGTM” to”let’s fix that vulnerability first”.
pr:
branches:
include:
- main
- feature-*
It’s possible to spend a lot of time wondering why this doesn’t work. Here’s the thing: presently, it doesn’t work for Azure’s Git hosting. You’ll need to go to the Repository in Azure DevOps and set up a Build validation policy. This is a pointy-click change, but you can use the Azure CLI if you want a hands-off workaround.
I can empathise with the Azure DevOps team here: they’ve had to evolve this product over 17 years to keep up with changing fashions in development. This feature hasn’t been migrated to YAML yet. The world has changed a lot, Enterprise customers are looking for stability, and there’s a workaround to keep the pipelines flowing.
]]>We want to stop bad things from happening early, not pick them up after publishing or deployment.
Insecure containers expose the system to application vulnerabilities. Let’s go find some!
You want to know if you’re building containers with weak security. How do you find out? By Linting the Dockerfile before you build it. I chose Hadolint as the tool to do this on a sample application, and added it to the Makefile:
lint:
docker run --rm -i hadolint/hadolint < Dockerfile
docker: app.war lint
docker build --platform linux/amd64 --build-arg JAR_FILE=build/libs/\demo-$(VERSION).war -t demo:$(VERSION) .
Hadolint found issues with my 6 line Dockerfile, some of which could have impacted security: for example you shouldn’t leave package cache files sitting around if you happened to install a package in your Docker build.
Any good container repository should let you scan for vulnerabilities: Azure and AWS Container repositories do, the Docker service also does. But why wait? I added Trivy (hat tip to Phil) in order to expose any issues. Oh my Lord there were issues:
demo:1.0.0 (alpine 3.14.0)
Total: 36 (UNKNOWN: 0, LOW: 0, MEDIUM: 6, HIGH: 26, CRITICAL: 4)
┌──────────────┬────────────────┬──────────┬───────────────────┬───────────────┬─────────────────────────────────────────────────────────────┐
│ Library │ Vulnerability │ Severity │ Installed Version │ Fixed Version │ Title │
├──────────────┼────────────────┼──────────┼───────────────────┼───────────────┼─────────────────────────────────────────────────────────────┤
│ apk-tools │ CVE-2021-36159 │ CRITICAL │ 2.12.5-r1 │ 2.12.6-r0 │ libfetch before 2021-07-26, as used in apk-tools, xbps, and │
│ │ │ │ │ │ other products, mishandles... │
│ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2021-36159 │
├──────────────┼────────────────┼──────────┼───────────────────┼───────────────┼─────────────────────────────────────────────────────────────┤
│ busybox │ CVE-2021-42378 │ HIGH │ 1.33.1-r2 │ 1.33.1-r6 │ busybox: use-after-free in awk applet leads to denial of │
│ │ │ │ │ │ service and possibly... │
│ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2021-42378 │
│ ├────────────────┤ │ │ ├─────────────────────────────────────────────────────────────┤
│ │ CVE-2021-42379 │ │ │ │ busybox: use-after-free in awk applet leads to denial of │
There were so many issues found in the Docker image and the Java app inside it, that I had to pause writing and go fix it. Even then I couldn’t find a clean Docker image, and there are still transitive dependencies from Spring Boot that cause noise. That’s a challenge managing vulnerabilities in a live project, so I configured it to exclude unfixed findings, and only fail at a threshold. Right now it fails the build if there are Critical vulnerabilities discovered.
docker: app.war lint
docker build --platform linux/amd64 --build-arg JAR_FILE=build/libs/\demo-$(VERSION).war -t demo:$(VERSION) .
trivy image demo:${VERSION} --exit-code 1 --ignore-unfixed --severity CRITICAL #,HIGH
It takes about 2 seconds to build and check the Docker image (excluding building the toy Spring Boot app). See below for details. Doing this for all your container images gives a high ROI for your time: you’ll reduce your exposure to application and container vulnerabilities.
time make docker
docker run --rm -i hadolint/hadolint < Dockerfile
docker build --platform linux/amd64 --build-arg JAR_FILE=build/libs/\demo-1.0.0.war -t demo:1.0.0 .
[+] Building 1.0s (7/7) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 37B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/openjdk:18 0.9s
=> [internal] load build context 0.0s
=> => transferring context: 104B 0.0s
=> [1/2] FROM docker.io/library/openjdk:18@sha256:ac8bc800c76446207675f93939a8977e032043a182c1a6be23c6cdd300db0cc5 0.0s
=> CACHED [2/2] COPY build/libs/demo-1.0.0.war /app.war 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:8ff141dc971724c6602aa51bb709b6ae804c720cdfa44e19fc866a4ffe214a8a 0.0s
=> => naming to docker.io/library/demo:1.0.0 0.0s
trivy image demo:1.0.0 --exit-code 1 --ignore-unfixed --severity CRITICAL #,HIGH
2022-10-24T12:45:49.662+1300 INFO Vulnerability scanning is enabled
2022-10-24T12:45:49.662+1300 INFO Secret scanning is enabled
2022-10-24T12:45:49.662+1300 INFO If your scanning is slow, please try '--security-checks vuln' to disable secret scanning
2022-10-24T12:45:49.662+1300 INFO Please see also https://aquasecurity.github.io/trivy/v0.32/docs/secret/scanning/#recommendation for faster secret detection
2022-10-24T12:45:49.692+1300 INFO Detected OS: oracle
2022-10-24T12:45:49.692+1300 INFO Detecting Oracle Linux vulnerabilities...
2022-10-24T12:45:49.699+1300 INFO Number of language-specific files: 1
2022-10-24T12:45:49.699+1300 INFO Detecting jar vulnerabilities...
demo:1.0.0 (oracle 8.6)
Total: 0 (CRITICAL: 0)
real 0m2.011s
user 0m0.439s
sys 0m0.095s
So I downloaded a year’s worth of withdrawals in CSV format, and did this:
egrep -i "powerco|gasco" account.csv | awk -F "," '{s+=$4} END {printf "%.0f\n", s}'
I could write the whole thing in awk, but these things evolve by tacking commands onto the pipelines, and there’s no need to get clever.
We spent more than I’d expected to, so my next job is to find out what is chewing up so much energy. Of course, normal people would use a spreadsheet.
]]>I started from first principles with a basic Spring Boot application, tried some different distributions of Kubernetes, and made an application deploy. Then made the thing visible from my local computer. That took a long time. Here’s what I learned:
Apple Silicon has teething issues: I love my M1 MacBook Pro. I found the following gumption traps in my first session:
platform
parameter everywhere, to run Intel/AMD containers instead of ARM.The ecosystem is a loud bazaar: On the desktop there’s several different distributions of Kubernetes. Each cloud provider supports at least one implementation. The ecosystem that interacts with these is huge. Similar itches are being scratched many different ways. My view is that you needs to work out what Dev and Ops cultures matter in your team:
It’s hard to choose tools correctly the first time: as communities and systems evolve, yesterday’s right choice might be today’s technical debt. If you can keep an open mind about where to apply different approaches, then you may be able to incubate newer tools and get some payoff from them. If you stick to the tools that you’re comfortable with, you may find yourself with an obsolete stack.
Where I got to with my playpen application: I’m building local images using the Docker CLI against Rancher, orchestrated with Make, and calling down to Gradle to build Java stuff. After experimenting with plain YAML and kubetcl, exploring Tanka, I finished with the Terraform provider for Kubernetes. I would want to deploy AKS or EKS infrastructure with Terraform, and to kick the tyres I will continue with Terraform to get apps lifted and shifted to Kubernetes. That won’t scale forever, and at some point I’ll find the right tools to manage the inside of the cluster. But for today, I’m going to add one new tool to the toolbelt.
Update: I got around to Securikube Part 1.
]]>the fish rots from the head down
: if your CEO allows awful behaviours to happen at work without intervention, then they should take the blame when those behaviours become normalised in the organisation that they run. This post is not about that.When you’re getting a pen test for your web application, testers are duty bound to give you findings about your missing HTTP headers. It’s important, but it also muddies the waters with findings that anyone could generate. If you visit Security Headers and ensure that your application has a clean scan before the pen testers show up, you’ll get a cleaner report.
I scanned this site last week and got a D. After reading the recommendations and adding some Zeit config, it’s an A+. Probably not a huge acheivement for this static site, but a good tool to have in the shed.
]]>I’ve since pulled nearly everything back to AWS so I can manage it all with Terraform in a GitHub repo. That raises a question: how secure are my AWS resources? Inspecting stuff with AWS is possible, with tools like AWS Security Hub. It would be nice to know that I were making secure Terraform code before I pushed it and applied. I chose Checkov. There are other tools out there, like Snyk IaC.
I installed the Checkov package from PiPi using Pipenv, and then invoked pipenv shell
to call the checkov
script from a Makefile. Here’s the kind of output that Checkov gives you:
(terraform) bash-3.2$ checkov --quiet -d .; exit
_ _
___| |__ ___ ___| | _______ __
/ __| '_ \ / _ \/ __| |/ / _ \ \ / /
| (__| | | | __/ (__| < (_) \ V /
\___|_| |_|\___|\___|_|\_\___/ \_/
By bridgecrew.io | version: 2.0.338
terraform scan results:
Passed checks: 91, Failed checks: 3, Skipped checks: 43
Check: CKV_AWS_19: "Ensure all data stored in the S3 bucket is securely encrypted at rest"
FAILED for resource: aws_s3_bucket.media-build-doctor-com
File: /build-doctor.com.tf:84-90
Guide: https://docs.bridgecrew.io/docs/s3_14-data-encrypted-at-rest
84 | resource "aws_s3_bucket" "media-build-doctor-com" {
88 | bucket = "foo.build-doctor.com"
89 | acl = "private"
90 | }
Here’s what I learned:
Checkov will tell you in no uncertain terms that your Terraform code is insecure; you’ll end up questioning your life choices. It was especially bad for my poorly composed Terraform code - I had lot of similar resources in my code, each of which picked up many warnings.
Doing the right thing and extracting modules for my static websites was a good thing - it drove consistency and reduced the howls of outage from Checkov.
The age old struggle between convenience and security played out again - I spent a couple of hours happily smashing security vulnerabilities that Checkov reported, before finding that I’d completely broken the sites, with no access from S3 buckets to Cloudfront. The log buckets were so tightly locked down that I couldn’t even see what happened. After quite some time kicking tyres, I rolled the changes back.
I made some some really simple tests using curl, that I can easily run from make. This gives me confidence that the websites still work. I try to get early feedback by applying the changes in Terraform and then testing afterward.
The many findings about S3 security make a great deal of sense in many contexts - the way we use S3 has changed during the lifetime of the S3 service. Nobody wants to win a Bucket Negligence Award. In my case, where I host some very basic static websites, I’m happy to suppress those:
resource "aws_s3_bucket" "bucket" {
#checkov:skip=CKV2_AWS_6:We have a public access block
#checkov:skip=CKV_AWS_18:this kind of content doesn't need encryption
#checkov:skip=CKV_AWS_19:this kind of content doesn't need encryption
#checkov:skip=CKV_AWS_20=Public ACLs are fine for this
#checkov:skip=CKV_AWS_52:MFA delete hard to apply via TF
#checkov:skip=CKV_AWS_144:Cross Region Replication not important
#checkov:skip=CKV_AWS_145:We're not going to encrypt public website stuff
Suppression helps me to focus on the things that matter in my context.
]]>I’ve collected a few easy options for deploying SAST tools in this post.
First on the list is SonarQube. As it supports many tools under the hood, it can give you an overwhelming list of things to fix. With a bit of setup, it’s useful for tracking the health of a codebase over time. The neat thing is that it will also detect and track security issues. If you’re being asked by clients about your SAST or DAST tooling, it’s a great way to check the box. SonarQube also works with the SonarLint IDE plugin, which is very helpful. You’ll need to buy the cloud service, or run your own SonarQube server.
Next up, is GitHub Advanced Security. Being a GitHub feature, it’s tightly integrated with GitHub and can be configured to run on push, or as an Action. You can also add third party scanners, and use webhooks to add Jira tickets to the backlog. Advanced Security needs an Enterprise plan, so if you’re looking to throw money at the problem, this could be the solution.
Finally, there’s two other tools that I think deserve a mention: Snyk, because they have good IDE support (why not squash them at the source?), and it integrates with BitBucket and GitLab. Also, of note is Muse; Sonatype acquired this recently and will be releasing hosted and free versions. It’ll be branded as Sonatype Lift and will integrate with the Maven ecosystem, to help address supply chain compromises.
]]>JEKYLL_ENV
environment variable in Vercel, which stopped Google Analytics from loading, but I still had leftover Google Analytics code lying around, and Discuss includes rotting away.Very satisfying to fix tech debt. There’s always tech debt.
]]>Staying alive beats everything else. Like a living thing, startups will endure all sorts of horrors to stay alive. Crappy work conditions, penny-pinching, super economy travel: all necessary unless you have the enough runway to upgrade. It’s the job of the management team to keep the business afloat, so don’t complain about having to buy your own lunch. The present amount of funding available means that startup employees get amazing benefits: it’s not normal.
Silos will hurt your startup. It’s easier than ever for organisations to spread around the world. It’s also easy to create geographic as well as organisational silos. You need to work out how to prevent that in your organisation: having Sales based in one region and Engineering in another can easily lead to silos. We’re tribal creatures, and will naturally create ingroups and outgroups based on department and geography. Plan accordingly.
Risk and compliance will matter. Sure, go ahead and pivot all you like while you’re trying to find your startup’s place in the world. Kick as many cans down the road as you can: you might not even stay on that road. Once you’re on that road: start investing in the tools to make compliance and security easier. You don’t have to start with much; perhaps some advice on how to get started. I’ve seen several organisations make compliance difficult, or be unaware of the risk that they’re running by uh, not managing risk.
While you’re reading about risk: after ten long years in startups I’m moving back to consulting. I’ll be joining Safe Advisory at the beginning of May to do security and DevSecOps advisory. I’m looking forward to the change: helping people to demystify security and fearlessly secure their organisations feels like a worthwhile pursuit. It also gets me out of the house from time to time.
]]>This activity gives me a lot of time to think about twisted metaphors like this one: weeds are your security issues and technical debt.
The grass grows differently around our garden: there’s different species of grass, like the Kikuyu that spreads everywhere, and other species that don’t grow so fast. Some areas seem to get more water flowing downhill, and the grass grows like crazy there.
Then there’s everything else that grows: the wisteria, the honeysuckle I just found this afternoon, the olive trees on the berm: we’re constantly finding things as we rip out the weeds.
How does that compare to your organisation? Where do the weeds crop up? What’s under your Kikuyu?
]]>