What’s in a name?
As a definition of what *Client Platform Engineering* is, we’d be the team that ‘manages the life cycle’ of workstation platforms. In the day-to-day it would be less helping end users in real time, have less interactions with locked-down ‘appliance’-style mobile devices, but it’s definitely about making sure computers get patches, secure configurations, and groups can be identified that an appropriate environment is maintained on applicable devices.

Is CPE For You?

Starting at the beginning, why am I intending to encourage the talent pool of CPE’s to increase, and saying it’s both a great time to be one and ok if you don’t want to do this work (forever)? Years ago the MacAdmin community would lose talent to Linux sysadminery/devops-y folk. Then being fluent in Continuous Integration/Continuous Development supporting the building of iOS apps became prolific. Around the same time securing Corp meant pulling the people who know how to patch and inspect configs on the fleet (we even have the highest count of CPUs) into Security-dedicated roles. Now there’s enough of a gap in tooling from the MDM perspective that vendors are recognizing you don’t get better without people from the trenches with problem space expertise. Opportunities abound! (And we can get compensated accordingly – it’s valuable work.)

Speaking for myself, I’ve enjoyed making this my career. But one of my favorite quotes from a manager I’ve heard in passing was ‘this isn’t going to be your last job, so I’m hoping we can help you improve in the way you want to evolve’. My personality means I’ve followed Mac blogs since it was hard to blog, I’m a lifer for this work. But as a devops mindset and its topics crept in to how MacAdmins can get their job done, we (should be able to have) increased who can do this job. Tech workers are kept afloat in seemingly all economic conditions and are finally getting salaries to show proper respect for the breadth of skill we display across various parts of the IT ‘stack’. Apple’s continuous churn, breaking and giving us QA tasks for the disruption they cause ~being courageous~ ‘innovating’ means we also have to continuously learn and stay on top of change, more than almost any other tech discipline (besides I guess JavaScript frameworks).

Be You, Too

I, for one, don’t need people who religiously follow the blogs or Apple’s mercurial shifts, but I do want folks who know how important gitops and code review and visualization of metrics are and yeah, also happen to be able to deliver operating system/workstation platform lifecycle management. If it’s a given this will not be your last job, I have no problem eventually losing teammmates to other departments and fields that use osquery or devtools scripting or containerized GitLab CI/CD or web-facing cloud service delivery, as long as we get the mutual benefit of our time together in enough of the core CPE disciplines.

At (several, failed) Dropbox interviews I detected I wasn’t respected because I hadn’t gone to university, same due to my lack of ‘real’ Computer Science bonafides when I interviewed at Google (although they were still super nice about it). Some employers train interviewers on bias and explicitly call out ensuring qualified candidates get the consideration they deserve. At most interviews I can still fake a ‘culture fit’ for pampered people havens because I’m male-identifying and cis and have worked in places with unconscionable practices, with really bad personalities that provided cover for my own toxicity. I want us all to realize we should grow people that aren’t like the archetypes we can be perceived as having benefitted from in our careers. When we do that, our service delivery will get feedback from the widest customer base possible because our members will be that much more approachable. Any drop off of interest from the already slim subset of people who care about our IT product hits our respect as peers in an engineering organization, and the dynamicity of our team can only be assisted by showing better representation of the customer population as a whole.

The MacAdmins Foundation recognizing the need for a mentorship program (see #maf-mentorship) is a great space to watch, and I hope we can all find resources at our current employers to get experience both being mentored and mentoring so that we can get REALLY GOOD at investment in people as a community. (And I guess therapy/diversity training to unlearn some of the toxic aspects of how we had previously thought it was acceptable to operate would be a Good Thing.)

It’s Dangerous To Go Alone, Bring This

Now we come to what I hope is AS REMEMBERED as the preceding topics in this post, which is the things someone with my career ‘bent’ would like to see people choose as aspects they can work on if they feel their skill set doesn’t yet match what a CPE team member should be expected to have. And admittedly it’s an unreasonably broad range of skills, but at this point I have well-trodden, go-to topics I’d like to see a passing familiarity with, and links to preferred resources to consult can hopefully get folks started.

Not just any scripting language, python is what I need people to be relatively fluent in, and Google’s online python class was what really challenged me but also provided a great set of resources to come back to about ‘what was that super basic thing again’. Being able to tool around in the interpreter is way more cool than fish shell prompt bling IMO. All of us who live or die by shell scripting know you hit a line length or complexity where it’s the wrong tool for the job, and I certainly learned by copy-pasting and modifying others code, but you need to reason with and solve problems from scratch ‘on your own’ with a language that intends to give you one good-enough common-case way to navigate a complex control flow. As a community, we benefit from a truly immensely valuable turn of events from over 15 years ago: Greg Neagle wanted to learn python, worked with Google engineers to add features to Munki, and writes well-commented, as straightforward as possible code with compact functions and not so much sprawl in either number of library files nor line length. Listing all the functions in Reposado and tracing the logic control flow (for a PSU MacAdmins presentation) was one of the best things I did for my understanding of solving non-trivial problems (in reality just tangentially related to the MacAdmin space) in python.

You may find other training resources that stick with you (I also liked the Coursera python class and like the Google resource above, regularly frequent the pymotw reference series), and hopefully you come across approachable problems to take apart. And THEN you should definitely put your flag in the ground and share what you’ve come up with, open source has been a huge part of what CPE needs at any kind of scale. I think running a high-ish visibility project has helped Greg find peers in the community that solve problems in different ways, code review is a hard discipline to be immersed in and we all start somewhere – reporting issues and writing documentation is super helpful and valuable! Similar to how I benefitted from having to present on the topic of Reposado and my blogging about various things, being able to reason about a stance you’re taking in text is a hugely valuable skill. Greg wrote literal magazine articles on MacAdmin topics, and as the length of subscriptions in MacAdmins Slack’s blog-feed list and infrequency of articles to AFP548 may indicate… there’s always readers in this space looking for cogent content and ways to reason about the challenges this work presents.

Show ‘Em What You Got

Contribution to open source online has to get slid through the perilous slot of version control, so yet another side benefit of sharing code has been it proves you can slay the git dragon. (VS Code will never get me, but the war was long ago lost for mercurial, good riddance svn etc.) API wrangling and glue code will never stop being a well-worn topic, so if you’re in search of where to show prowess, python modules that handle web and ‘process’ (read = wrapping command line tools to automate things) interactions are what you’ll be seeing for most of your career. (Built-in standard library should be preferred, but if a problem-space-specific pip module is trustable-enough, investigate that and come up with your own qualification criteria!) If you can do some or most of that ‘glue’ in a language like ObjectiveC or Go or Swift that’s all to the better, but these days pseudocode python when designing solutions feels like a common-enough language for design and discussion.

From a similar web and platform perspective, I want to interview people who know how IP subnetting and DNS records work enough and the basic moving parts to troubleshoot connectivity – I have no clue why this is overlooked so often. Getting at log files is another live-or-die skill for some, for me just being able to process large volumes of text across various formats (json, xml, etc.) to do basic analysis was a fun task that helped in other problem spaces over the years.

If I can be opinionated and brief (ever), don’t learn Jenkins, learn GitLabCI/CD or GitHub Actions and understand convergence time periods and document warranty’d behavior and engage with the pitfalls in both the dependency stack and network links that connect your automation from start to finish. (Also internalize why you should strive to never curl -k, ffs.) Likewise, document your entire DEP deploy process, all the things from an out-of-the-box Mac to the warranty’d Standard Operating Environment. Automation is a contract, put your name at the top of the wiki/notion page and own that you are responsible for what you said your engineering delivers. (It’s ok if the world has shifted before the ink is dry: the time stamp on the page is as much versioning as you need to be held to – just at least do it once for yourself/future you.)

Get Through The Door

More opinions! One (pdf) page for a resume SHOULD BE enough – anyone will help you whittle things down and format it well enough if you ask them, and a LinkedIn or Google Sites page with your entire history can fill in the rest of the details if you want to be verbose. Phrase achievements during your time with your previous employers in quantifiable numbers things like budget and/or time and/or complexity, impact-wise. Include stuff you have working knowledge of at least maintaining, don’t list alphabet soup. Do check the recruiter boxes in a job post! I want you to get past the recruiter pulse check, you’ll save them time if you include tangible experience with the tools we’re asking for familiarity with. Go ahead and tell me you have PHP or ColdFusion or Novell eDirectory or VectorWorks ~or trapeze~ or whatever other things may be considered inapplicable/passé experience as long as you delivered real value with it! Recruiters may not care, but I’ll be intrigued that you met the business challenges where they needed to be solved; demonstrate-able, broad excellence in anything is a great indication you can be good at Just CPE Thangs.

In recent years ‘service delivery’ (meaning hosting Crypt or an MDM or monitoring or inventory or CloudFunction/Lambda/glue-y stuffs or whatever other platforms end up not being solely SaaS nowadays) has meant ‘the cloud’, and for some that has also meant Terraform (or whatever platform-specific way of delivering your needed infra to your cloud is). My recent dalliances with compliance also brings us back to the fundamental need and unique problem space configuration management tools like Puppet, Chef, SaltStack and Ansible address. I cannot overstate the importance of approaching high-stakes infra or config delivery with that mindstate: you must get as much of your setup into repeatable code as possible, and at least understand the off-ramp one must take when the workflow around basic scripts ain’t gonna cut it anymore. You’ll definitely be leveling up past where I’m at when you can reason about where ‘data’ and/or config-specific boundaries ‘naturally’ lie and can be dealt with separately in Infra As Code and server-full (or, even cooler, serverless+metrics) config mgmt deployments.

Ok, pardon that may have been a bit of a firehose. I write like I talk, and I’m a recovering New Yorker. Hopefully the whole audience sees things in this they can get value from, and I’m glad to discuss things further in the comments or Slack or elsewhere online! (No I do not has a WoolyMammothdon.) Best of luck!

‘Observability’ and ‘governance’ are my favorite buzzwords of late, because the Reporting Structure Above incentivizes us Client Platform Engineers to display the slog of busywork we ship when preparing for being audited as it immediately turns into proof of a compliant state – in being rigorous, we mix in things that prove we’re doing our actual job, and the org validates that every effort is proven to be worthwhile or dashboards wouldn’t need to be looked at. I won’t spend too many characters on ‘feelings’ about this whole CIS thing, I think I got enough of that out of my system, so in this post we’ll elaborate on the client and server and the data moving parts of actually shipping those checks.

As we use Zentral to get the job done, it’s what I’ll be referencing a bunch in this post. Bundling Grafana/Prometheus/OpenSearch behind its osquery service gave us flexible options at an enterprise scale, but that’s admittedly a lot of stack if you’re looking at setting up the ~dozen or so core moving parts (our Terraform apply is ~250 AWS resources between all the alarms and backups and secure infra configs and tweaks and customizations we’ve built up).

Unfortunately if you were hoping for a clean shot where you wouldn’t need to be a subject matter expert on various parts of the systems you need to audit AND the osquery servers moving parts… ur in teh rong biz/are perhaps in need of reminding what that ‘engineer’ part refers to when the whole world has done gone SRE/DevOps-y for a while now. Fortunately, we’re all lucky to be in the biz with flexible, rock-solid tools like Zentral and the open source communities helping smooth over the rough spots that undoubtedly occur with the various agents that can send data through it. Even our relatively lean team got a long way towards telling a good story in graphs (lies, damn lies, statistics, etc.)… at least until the delayed and new/future tech debt comes a callin’ .

Zooming back in on the examples referenced in my presentation, jmespath and osquery are the two formats you can write ‘compliance check’s in. The ‘pass/fail’ data for these can then be surfaced as ‘filters’ a.k.a. widgets on the Zentral inventory view. You can even include a collection of them at a RESTful URL to easily point folks at. But… Zentral has a GUI only a developer could love – just as vital to the dependability of the backend (by leveraging cloud-primitives like subscriptions and message queues), and frontend (via ‘stores’ like open/elasticsearch/splunk/datadog/snowflake…) that accompanying Prometheus and Grafana stack I referred to lets us tap into and slice and dice the data for even easier consumption – which folks usually need logging infra and vendor-specific specialization to get at.

Jmespath is a way of querying json data (in our implementation from events earmarked to also get streamed into prometheus buckets) which has usage syntax not unlike jq. Zentral implements common-case, low-hanging fruit/golden-path ways of querying e.g. the presence of mobile configuration profiles delivered to a device (as inventoried by the ‘monolith’ wrapper custom to Zentral in case you’re hosting munki somewhere else – CERTAIN overpriced MDM’s cannot efficiently return full inventory data about what profiles are installed on a device… no comment.) But you should keep in mind jmespath is not a full scripting language, you may soon realize how shallow you dip into jq when parsing stuff. Many of the ‘sources’ you can plug in as modules to Zentral (Jamf/Munki/Puppet being the most notable) have a section in each inventory record with the last recorded value for ‘extra facts’ – that custom/arbitrary inventory data can have simpler jmespath queries run and stop smoke from coming out of your ears. (I’m leaving out some specifics and details about which facts you may benefit from, but… you know where to find us in the Zentral community if you need blanks filled in .)

Helpfully, you can use the DevTool built into Zentral to take real collected data for a device and confirm that the jmespath check gives you the result you expect, and conversely devices that are in the opposite state are parsed correctly. Besides ‘scoping’ a check to a source or an operating system platform, tags also allow you to take other groupings you can sync into Zentral and apply arbitrary constraints as needed. For example, cloud-hosted Ubuntu systems wouldn’t need full disk encryption and screensaver locks, nor do macOS-based conference room and signage ‘appliance’ Mac Mini’s, so identifying those via tags or sites ensures the checks only run on applicable devices. Here’s an example of confirming the presence of a mobile configuration profile with WSOne or Jamf:

contains(keys(@),profiles) && contains(profiles[*].uuid,E81C883A-6938-4F43-8E28-10428836FB2B)

Breakinitdown:

• First, confirm which source sends the inventory information for the top-level key you’re going to follow down the tree for (e.g. on Mac, munki gathers profile information)
• Then confirm it’s present (as there may be a slow convergence period e.g. right after DEP bootstrap) by using ‘contains’ and the key name wrapped in tildes: contains(keys(@), profiles)
• As mentioned, you can customize this for other top-level keys, under extra_facts. Note those would be source-specific, so e.g. you’d choose Jamf if you need EAs
• Next, continue evaluating with a logical and, &&
• Then enumerate the uuids across all nested profiles with a glob/asterisk in square brackets after the top-level key, and then dot-index into the value we want to check: profiles[*].uuid
• Nest that inside a contains() and follow the profiles[*].uuid above with a comma and then the value we’re checking for wrapped in tildes

To reiterate a point I made in the presentation, I find these types of ‘is profile present’ checks… naïve, because you’re trusting that the responsible framework(s) did in fact apply the change/restriction and it’s not pending a reboot or transition to a specific ‘happy state’. So to contrast the previous example, here’s how you’d check that the condition puppet tests for via a fact to run idempotently is in the applicable complaint state, we use funny logic that ends up making enough sense that expressing a boolean in text doesn’t drive us nuts:

puppet_node.extra_facts.contains(keys(@),indeed_nfc_disabled) && puppet_node.extra_facts.indeed_nfc_disabled == 'OK'

(We’d always see the puppet_node in the root of the json inventory representation, so we’re going two nodes down the tree to make sure we have the extra_fact populated in the first place, otherwise Zentral would give us a third state, ‘UNKNOWN’ which obvs isn’t ideal.)

To explain how you’d write SQLite to check compliance with osquery, it’s best to start with another naïve example – this checks the hash of one file and is wrapped with keywords Zentral looks for when parsing its boolean result:

SELECT sha256,
CASE
WHEN sha256 = 'omgwtfbbqrandomchars00112233445566' THEN 'OK'
ELSE 'FAILED'
END ztl_status
FROM hash
WHERE path = '/path/to/file';

To explain the syntax in use here even if you’re not SQLite/osquery-savvy,
* starting with SELECT and then sha256 as the ‘key’/column ing the result whose value we’re trying to evaluate results based on
* then CASE so we enter the sqlite ‘logic flow’ where pass/fail criteria can be assigned and different paths can be followed as a result
* then WHEN sha256 = ‘sha256werelookingforomgwtfbbq’ THEN ‘OK’ ← which designates the ‘pass’ case if the osquery hash table says the result of checking that file is as we expect
* ELSE ‘FAILED’ (one possible keyword Zentral is looking for in the ‘failure’ state)
* END ztl_status ← this actually renames the adhoc key/column in our results to ztl_status, and is how Zentral knows where it’s trying to read a value from. The END closes the case statement
* FROM hash ← tells it what table we’re getting the sha256 of the file via
* WHERE path = ‘/path/to/conf’; ← these are our sql-isms to stretch how you think a database works, since it’s not something statically stored that we’re retrieving, it’s ‘at-time-of-query’ that it does the lookup on the device in question to determine what the sha256 of the path stated is

And being more rigorous with an advanced example of a query for the NTP config which SHOULD be parsed and applied as expected on most recent macOS versions… (which… we never got the red team to confirm does anything worthwhile? But we felt there wouldn’t be collateral damage/much downside to making this the default, and the GUI is still accessible to confirm SOMETHING we consider accurate is being applied):

with compliance_check as (
select sha256,
case when sha256 = 'omgwtfbbqrandomchars00112233445566' then 'OK' else 'FAILED' end ztl_status
from hash
where path = '/path/to/conf'
)
select compliance_check.sha256,
coalesce(compliance_check.ztl_status, 'FAILED') ztl_status
from (select 1 as x) x
left join compliance_check on x.x = 1;

For osquery via sqlite, ‘join’s allow us to pull results to correlate or otherwise mash up from multiple tables. Dot-notation is used with those table names to map the ‘field’/column label to either return in the subsequent results or match on when sewing tables together. You can additionally adhoc create new tables to work with or wrap the output from a select statement looking up values in a table into a newly named one.

Zentral provided code to cover the case when the thing we’re auditing may actually not be present and therefore not return a result, whereas we’d prefer to fail the check in that context instead of letting it be unknown, to improve on the outcome of the v1 query above. To explain what’s going on in the v2 example code provided:

• First we’re overriding the name of the table ‘nesting’ the results of the query in parentheses using the ‘as’ keyword to make it ‘compliance_check’
• We’re then requesting two columns/rows –
• sha256,
• and the results of the case statement, labeled ‘ztl_status’.
• From that new ‘metatable’, we say we’d like to end up with its sha256 column and a default of failed for ztl_status (via coalesce)
• In the process/same line we’re also re-applying the ztl_status label (since the way we’re asking for it would be the column header otherwise)
• To ‘trick’ the query into returning/generating/allowing an empty/NULL result (when as in this case the ‘where path’ isn’t found/the query would normally return no result) we make a SECOND, new table to join against with one key/value, x = 1, and re-lable that single column ‘x’.
• Left joining excludes that new tables content but now we can return an empty sha256 and failed for that case on the right-side’s table. Think of it like a venn diagram where our join only cares about the values distinctly only in the ‘compliance_check’… metatable

And now that we’ve broken that down, to bring it all home, here’s the true not-so-secret sauce of the setup: compliance checks status change events get written out to Prometheus metrics, which Grafana can dynamically pull in via variables to a self-maintaining dashboard! The one we use reflects the raw numbers alongside percentages as a time series trend a.k.a. a burndown chart (if we had a threshold, which you can also trivially represent in Grafana as an arbitrary horizontal line, to work towards getting the fleet under). Looks impressive on slides! Get yours today!

Zentral actually has more exciting visualizations, but I’ll round out this article underlying the governance benefits of using this tool for proof when being audited; on Windows and Linux via puppet we check in the code, on Mac we checked in the Munki pkg with the metadata and payload that made the change, then with Terraform to manage the compliance check or gitops to manage the osquery pack we verified that the job got done. And THAT is how you apply rigor to your processes so you can own the ‘engineer’ in your title as Client Platform Engineers, if I do say so myself. I hope you agree and get to work with tooling that enables rigor like this!

To even be able to look at the style used (instead of extracting the code from whatever prod system it’s running on…) we need to get it in a more share-able format at least. Although we’ve already touched on git, you don’t actually need to know proper care and feeding of a repo to let others see your code at whatever state it’s in; sites like Pastebin have been around forever, GitLab has a snippets concept, and GitHub gists can even accept multiple files in a single page. GitHub then tracks versions for you mimicking a simplified repo. Some of these sites accept anonymous posts, and some links can be shared ‘privately’ (for anyone with the link, not even limited to logged-in users with an account) without being discoverable/indexed by search engines – handy when wanting to share (ahem hopefully sanitized) log files or debug output, as a more general-purpose usage example.

It’s all downhill from here

Now before I go further, don’t get me wrong – my pet peeves are not important, and neither is my opinion. (Until it is, intuition is a real thing dedicated practitioners might develop over time. If at all, ever.) Don’t take it personally if you blissfully do the things I’m warning against, or ignore the things I’m advocating for in your code, even though I obviously consider these important enough to write (at length!) about. Maybe I’m just ‘owning the brand’ of being a snob/elitist, but as with anything in life, you either complete tasks where you only care about the goal/’it appears to do the thing’, or you show care about the process and let it influence how you operate (on code). All of this is for as much as you both find relevant, personally, and like, remember/take the time/effort to do. Real scientists publish ship.

But, except for XML, use spaces, not tabs. Or be wrong, up to you. :sarcmark:

Trust no one

Taking it from the top, she bangs! #!'s! Pay attention to how you’re finding an interpreter in whatever execution environment it is that the code is supposed to/ is intended to be running in, especially if you’re pointing to a path that’s actually a symlink. (Like /bin/sh, which actually points to… /bin/bash…) With python 3, using an env shebang is definitely a risky click if not exactly YOLOMODE. (Unless you’re in a venv, natch.) It’s more reliable to be as explicit/exact as you can, at least feel free to explain what e.g. version of python and what 3rd party modules you’re expecting (if any) in comments or associated docs. Keep in mind when contributing to other projects that they have probably considered this fundamental piece carefully; sometimes you may actually not want to specify an interpreter, for example in ‘library’-like modules loaded by other code. Reasonable people can disagree, but it’s house/maintainer rules. Or you can go fork yourself. (That might sound aggressive, I mean it that you’re more than welcome to show the world how it’s done in your own repo – I’m happy as long as others can see how you’re solving a problem at all, if you’re kind enough to share.)

Continuing with not trusting PATHs, anytime you’re essentially shelling out or leveraging a binary, when I’m the reviewer, I’m going to expect you to use the full path. (Again, including symlinks, if you weren’t already aware, var, tmp, and etc are symlinks to inside root /private in macOS). This is definitely a style point, but instead of assuming the contents of a defined path is what you expect, (or worse, assume that e.g. launchd or the agent running this code knows what paths/lookup order has the version you need) doing this is (again) potentially better/more defensive by being explicit/exact. With Apple SIP-protecting some defined paths you may find it less important, and that’s ok, but they’re also requiring us to ship our own interpreters more and more. (Targeting a SIP-protected path in your shebang is probably good practice for as long as those paths can be relied on :sweat_smile:)

I usually check for the complete path with which <binaryname>, but also consider the output of type <binaryname> – if it’s not a shell builtin, it may not be portable to *nix, on the off chance that’s important for you to know/handle. And don’t take it for granted that Apple won’t move stuff around or delete it on a whim/in a point release. (Yes it was a major OS release, but we all recall how telnet got pulled.)

If you’re talking to an API or expecting specific schema for a data structure your code parses, consider putting your assumption of that criteria in comments/docstrings, a sanitized example is even more straightforward. That leaves a paper trail/gives context for why you operated on it in a specific way (and what was valid input to your program at some point in time), with the bonus feature that people can take that mockup and replicate your experience/confirm your assumptions (even without access to the same system). Same for links to API docs or other references that were used to understand the system being interacted with. It’s ideal if someone other than just the creator can follow the references and see the ‘shadow’ inputs operated on by the design, this really shortens ramping-up time to reason about whether the decisions made were the best way to approach the problem.

Easiest computer to throw out a window: yours (ideally into the ocean)

You don’t run random code from the internet without at least attempting to confirm it does what you need, I hope/presume? The same goes with boilerplate/scaffolding/auto-generated stuff that bootstrapping tools may ‘crap up’ your repo with. Some of it can certainly be helpful when you’re getting started, but it’s all code you’ll end up owning, and you want to strike a good balance of ‘I think I have to include at least this amount of bloat to ship’ with ‘now that I understand enough of the moving parts, let’s strip this back to focus on what we actually need/use’. Convention is a good thing to follow when collaborating with a larger community that imposes boilerplate on us or wants us to grapple with what they consider best practices, but otherwise let’s leave it as close to brass tacks as possible so we don’t have trouble remembering where we actually started making the code do things. Unused functions that are mostly stubbed out or exercise nothing should be purged until activated. Ideally, everything we commit to git should have a reason to exist that we can explain.
To mix all the metaphors, code both rots and is flammable, don’t have too much of it in one place. Teach a person to code, they’re on fire for a lifetime, amirite?

As part of these style nitpicks, this may not be as valuable/important to you/your team, but consider removing extra/extraneous comments, verbose manual logging, or debug echo/print() statements (unless gated by arguments/a flag and/or logging level) – ideally debug would stick around in something like integration tests that help you validate assumptions/confirm serviceability, but elsewhere in your code it could leak secrets or otherwise add unnecessary overhead/bloat to sift through when put into service/deployed to ‘prod’.

Can’t believe I have to say this

I think besides being overtly… ‘particular’/borderline-cargo-culty in points above, I’ve been a benevolent rubocop dictator. (All the metaphors!) Time for bad cop.
I don’t know who needs to hear this (jk, you unconscionable no-good-niks, you know who you are)…

Log.

It’s usually not hard and will save your butt later, it’s certainly cheaper and more practical than a time machine or ‘god mode’. Put whatever output you generate somewhere discoverable, e.g. in /private/var/log for system-wide or ~/Library/Logs, and check out the python and bash follow-up post for examples. If you then have those logs shipped/aggregated, know what value it will actually provide to put stuff at different ‘levels’ – ‘info’, ‘warning’, ‘error’, ‘debug’, etc., are those labels too numerous and too open to interpretation for your team? Maybe just assume only qualified individuals even look at/read this stuff anyway, so shove anything helpful in there and let your log ingest system discard whatever you don’t need, maybe? Consider writing json blobs if appropriate (or some other parseable format, just throw us a bone with UTC timestamps and delimiters or something, especially if you want to quickly turn the corner from shipped data to metrics). Got state you want to track locally? Consider shoving stuff into a database format, primarily if you are generating data that would benefit from ledger-style operations. Just leave a record somewhere, optimized for retrieval!
For more ‘vanilla’ logs, rotate and purge using system-native facilities (logrotate for Linux, apple system logging conf’s on Mac, etc. Don’t be like vanilla Docker.) Or don’t purge! If you don’t think it’ll be more than a couple hundred MBs in your average computer lifespan, maybe who cares, maybe disk is cheap and plentiful enough and this historical data local is helpful to have? Hey, if you break it you get to keep the pieces. (Or Apple will just helpfully dump it for you on OS point upgrades, surprise!)

Continuing, ffs, go out of your way to trust (and confirm your code validates) TLS certs when transiting the series of tubes. It’s worth it not just because it encrypts bits on the move, it also assists in validating the host you’re talking to. Another side effect is, when it’s wrong and due to our inspection it gets fixed, we’re being part of the solution. Calling protecting people’s privacy a side effect is almost forgetting the oath we all took as sysadmins.

Are you pulling random dookie over the network to then pass to processes running as root? Checksum it. It’s literally the least you could do. Don’t let you and me get joined at the MacMule. Otherwise you may be able to join that startup working on Privesc-as-a-Service.

And finally, to wrap up Things You Should Already Know: don’t put api keys, passwords, or secrets-in-general in plaintext or trivially-reversible formats. Options are available, but for crissakes obfuscation shouldn’t be one of them – truffle hogs look for that stuff, you’re not being clever, herr hax0r. Private repo’s are not an excuse, what happens in obscurity ends up dehydrated on a casino rooftop in Vegas. Environment variables? Decoupling a credentials file? Those are (arguably) Less Turrible, but Still Not Great, but again I’m not in security and can’t tell what your threat model is (although hopefully it’s not Mossad). Is SAML hard? Yes. Should we all find ways we should integrate with it when appropriate? It’s often the only game in town, take yer lumps. Using libraries written/evaluated by crypto pro’s? Yes pls. I’d add that PKI (e.g. client certs signed by a trusted CA) is another potential avenue to handle authentication/identification which you can therefore leverage as an ingredient when at least authorizing access, but don’t listen to me! I’m a rando on the interwebs. (They says, ~1800 words later.)

Tune in for our next installment where we get into the actual programming language slingin’ specifics!
Got other general coding ‘best practices’ thoughts? Use your tweet horn to holler at me!

Since we’re talking code, we should come to grips with the ungainly topic of version control itself before going much further, which in 2020 still means git. On this site (~6 years ago! with not the most generous title) we went over what could be considered a common-enough basic workflow and tips for git beginners, this time around I’m going to point out a more team-focused workflow and newer developments or caveats about that post:

Originally I encouraged forking into your user’s namespace/account instead of potentially piling up branches, or having to be explicitly added to the destination repo so you can create them there. This helped folks who often forgot to click the box to delete their short-lived branches on merge, which the web tools can even clean up in the (remote) forked space for you. I recommended this guidance as good hygiene, but there’s one caveat: automated testing/CI/CD runners may only be available on the main repo/project, so if it’s configured that way then sending a pull/merge request could trigger tests to be run on the remote/’userspace’ repo, and it could complain/not work. If the runner can’t be configured for individual users it’s no big deal to branch ‘locally’ to the repo, especially if common ‘guard rails’ like approvals/protected branches are in place. If you haven’t run into that as an issue, the overarching guidance remains. (Or if it’s too obscure to understand what I’m warning against, like, y’know, git is weird, please follow up with us in the comments or on the Twitters.)

One of the first times I really started to hate git’s user experience was squashing commits or trying to not include updates/’catch up’ commits in a fork/branch in my user account/space. You lose no ‘internet cool’ points if you delete your fork entirely in the web GUI and start over wherever the main branch you’d previously forking off of is at now, but there’s also this post about catching up your fork in a way that people have had success with.

If your team tracks work in a bug tracking/ticketing system, it may provide good context and extra metadata (or just be convention) to prefix your branch names and/or commit messages with the related identifier. (Some Jira-specific integrations I’ve seen actually rely on it and can provide context/feed metrics.) No, the code is not the documentation, unlike my obscure elusive magical unicorn comments, even the most verbose/perfectly-written comments are probably not enough context… This is just to say when an out-of-band communication method is preferred for a project, please consider using it. Wiki’s similarly exist for a reason.

Quick Tangent

Definitely don’t be like me and call ‘everything my beginners mind gets tripped up on’ a ‘frequently asked question’, although these getting started questions could use recording somewhere in my opinion. Imagine someone new to your team, new to the language, the framework, new to coding/version control, and help them in a discoverable way like you would have appreciated!

While we’re talking docs, something that previous teams I worked with in sysadminery aspired to, (at least take the time to think about actually doing eventually) is straight up test plans – formal documentation on how to exercise the code, how to pull together the moving parts or get access to applicable test data/accounts/systems. If it’s a trickier deployment, what’s the (set of) success metric(s), rollback steps to follow, etc. This may actually be coupled with the idea that your design document (something else we’ll cover in a subsequent post) led to these decisions. Side benefit, it can become a preview of potential maintenance tasks to support a measurable service-level objective. If we want to get buzz-wordy. (I promise, no story points or other stuff I haven’t actually been required to show ‘agility’ with! Got a reason why scrummy work is helpful/horrible? Holler at us on the Twitters or in the comments!)

Back to the-actual-name-is-Linus’s-Monster, there’s all sorts of opinions on styles and conventions you could use for the commits themselves, and the first reference I recall coming across was this one. The gist is, code diffs already explicitly say what changed, so you’re encouraged to think about it from a workflow perspective: the specifics are less important than why it changed over time: imagine scanning the commit history to notice when a ‘shift’ occurred, when intent or functionality shifted. Issue tracker/ticket notes and the full body/description of a commit are where you can provide additional context. Git was designed to use email as its primary communication method, all of this back-and-forth should be considered written communication and approached accordingly. Or maybe you don’t want to think about it too much, all of this is to say record more of what you were trying to achieve (to use a Neagle-ism), less of how you mechanically did it. (Typo commits can get rolled up and squashed anyway.)

Another less commonly discussed topic is how often to commit, what or how often to squash, and I guess we can relate it to why we’re putting this under ‘revision control’ in the first place. Just like why you backup your files – it’s for the restore, right? Only your team knows how much value seeing the code’s evolution over time actually has, keep in mind tags and ‘topic’ branches that you use while working on a specific release/feature/issue may provide a good-enough record of signposts in time that will still be around in retrievable history, even if every iteration isn’t in the primary branch those events are formulated in. This is all to say I personally tend to not see value in making granular commits e.g. per file changed, nor does every commit from the topic branch require preserving when merging; I often squash in the GUI and not worry what iterations were involved. It’s almost as if the mainline history should consist only of merge ‘events’ and the state of commits in branches don’t matter until after it gets eyes on/potentially altered in a review. (‘Draft’/WIP/Work In Progress designations are similarly used to conserve on commits that need to get merged with a topic branch into the ‘main’ branch.)

Maybe you don’t need to gpg/pgp-sign commits (via @groob). Maybe you work in ‘security’ and want to show you’re tall enough to ride/can attest to managing some extra sort of complication for benefits you perceive as important. I’m basically doing it for the ‘gram internet cool points fo sho, via a system/app that may already be abandonware (like GPGTools tends to look like at times with slower new-OS-release adoption).

In web GUIs/systems like Gitlab and GitHub, committing tweaks to the branch you sent a pull/merge request from will automagically update the changelist/merge/pull request accordingly. Just in case that’s news to some. Also specific to those webapps, reviewers can make code change suggestions to be accepted/applied by the author right there in the web interface while commenting on lines/reviewing, which is often ideal – it’s certainly more collaborative, since the reviewer can spell out exactly how they expect requested changes to be implemented.

In our next post we’ll get into semi language-agnostic style generalities, stay tuned!
Got other hot gossgit tips/code review thoughts? Use your tweet horn to holler at me!

It’s not helpful to be opinionated AND inconsistent, so I wanted to… codify some guidelines, and an approach that the folks in my team (and hopefully the community at large) can use as a starting point to collaborate on/argue about in the service of getting us all on the same page regarding style/design ‘quality’/etc. Even open source projects/loosely affiliated groupings of individuals shipping code may benefit from writing this down, but I didn’t have a good example of how to approach the task for us sysadmin-y types in particular. (If you have references you’ve found/like for any of the topics in this series, please share on Twitter or in the comments!) It turns out this is a big enough thing for several posts, so I’m breaking it up and will cover the scripting languages MacAdmins commonly use later on. ‘Code reviews’ as a concept may be new to some, so that’s where I’ll start in this post.

Why We’re Here

The point of code review is not to ‘rubber stamp’ as a mindless formality, nor is it to just confirm it kindof looks like a familiar file extension the author wants to spread blame for prior to shipping into prod. In more rigorous environments, it is also not a given that the reviewer will fully grasp the problem the code is attempting to solve, or be able to exercise all parts/functions in the code, or safely run it against ‘production’ data/live systems to 100% validate it,[1]

the point of code review is to have someone else spot-check the code, to… y’know, read it, every line, and to make sure it passes some degree of muster for maintainability/whole-team support/adoption that could achieve something almost like endorsement when it ships. The reviewer should display empathy and hopefully have TIL (today I learned) moments, they should reason about the design used and offer alternatives, constructively argue when individual nitpicks clash and help the team collaborate on and grow the style guide with things that are important enough to at least bury hatchets about,[2]

and the changes proposed in the review can be followed through/integrated into the code before/after shipping if/when we have the time/see the value. Because rhetoric is all well and good, but as they say, real artists ship.

1. BTW, ‘correctness’ by the original author is not taken for granted, there’s ways to help prove it to the reviewer we’ll get into later, but it’s the author’s name in git blame who is considered primarily responsible.
2. Sometimes a style guide contains headstones for your litany of hangups that got sacrificed for team unity/sanity/productivity.

The statement right before these sort-of footnotes is important to expand, because developers/traditional ‘engineers’ have different incentives and expectations set on them than sysadmins who ship code. Many startups can get years down the road before the YOLO/eff-it-ship-it attitude starts losing its luster and editing code on the live production instance actually causes enough of an outage that business processes are interrupted. Or maybe you get interested in verification processes when you’re getting audited/acquired, or the lack of rigor just becomes unseemly. For organizations that don’t value computers, or have low-enough standards, or a high-enough tolerance for pain, maybe that time never comes. For MacAdmins, we know that on the technology treadmill there’s constantly demands to ship product and anticipate issues. Most (if not all) of our tasks are not purely intellectual exercises, and our managers are incentivized to help us ship better than ‘working prototypes’ more often than not. At least, after a company’s culture evolves to a point that it reaches a certain scale (or plateau) and decides to emphasize stability over velocity.

Ideally, ultimately, we hope things ship with increasing reliability. And in the process of getting muscle memory going through a process, we hope reviewers understand both what was written and why, agree that the implementation should ship, potentially help introduce efficiencies, and, where the REAL value comes from, (which is the reason management encourages us to do this in the first place,) reduce/prevent bugs/unwanted side effects.

Opposition to taking the time to check the code into version control in the first place, let alone do a review, usually starts with ‘well if there’s an emergency…’ Define what an emergency is to your team, otherwise they get the Princess Bride quote, or the ‘lack of planning on your part doesn’t constitute an emergency on mine’ tough love. Getting everyone to stop being independent operators is not easy, so management needs to be on board and lower the (usually false) sense of urgency. We’ll talk about asynchronous tips below, but anecdotally, having a distributed team increasing turnaround expectations actually helps with this. (When folks aren’t chained to Slack or have Zoom sessions open all day also helps, because often code design benefits from your brain operating in system two and the stimulation of a busy chat with instant gratification tends to prevent you from being idle at regular intervals.)

It’s dangerous in the code base, take this

I’ve already droned on waving my arms for a bit, but needless to say Google shipped their code review and style documentation years back as well-edited references. An important point they discuss for reviews is calling out the good things you see as a reviewer, since this can often look like a critique that the humans submitting the code can feel bruised by, especially in public/on the official record. Breaking the computers, less people are worried about (eventually we all have to Escape from LA). They get understandably annoyed if you break the malleable humans.

I also benefitted from this series about being on a distributed, asynchronous/non-time-zone-overlapping team landing code together. It does a great job illustrating that this is all about context transfer/communication by taking action and thinking about the other side of the coin and our shared goals. AFP548 lets me publish text in english on the internet, but only you as the reader can judge if I made my point/got across what I set out to say. (Communication is judged on the receiving end, exhibit A.) Positive code review experiences are the result of good back-and-forth communication.

Out in the real world for us sysadmins where reviews may be less codified, if you’re lucky enough that the project you’re sending code to already has contributing guidelines publicly stated (e.g. in a CONTRIBUTING.md), then that’s your defined jumping-off point. We often can’t choose who reviews our code nor express tone clearly in text from either side of the interaction, so each has to provide context and be as welcoming as possible. As the author, be equal parts ego-less and greedy – showing you really care that others help you is a great look. If a review process is still in flux, hopefully you find these concepts helpful to mull over, and we can all have fun painting bike sheds later when we get into code style details.

Up next, it’s unavoidable to talk about the version control tooling itself. TTFN!

What’s in a name?

Names are just labels. Your inventory and/or management system could contain an identifier like the serial number, hardware UUID, MAC address, and/or device certificate to more accurately track a computer (and help with linking it to a user), so IMO it’s not necessary to stop people from renaming their computer in System Preferences in general. We all know that by default it chooses the computer model to name itself, and on a home computer will update that name with whoever was the user created in Setup Assistant (E.g. Testy McTesterston’s Compy386).

It could be that your management tools are… silly, or security agents only pick up that easily-changed name and therefore you need to extract whatever other unique identifier may be stored locally on the client and ship it to the aforementioned mythical inventory tool for correlation. What those vendors of silly tools might need a primer on is the actual three possible labels we could be using for ‘name’ information on a Mac, as query-able by scutil, a mnemonic for system configuration utility. Let’s do it!

1. ComputerName – that’s the label we’ve been referring to above, accessible from the System Preferences → Sharing preference pane. Some docs historically have referred to it as the ‘user-friendly’ name, one claim to fame is that Apple Remote Desktop leverages this e.g. in school lab environments.
2. LocalHostName – this designates the computer on a local broadcast domain or subnet/vlan. This is the name that will be visible through file sharing such as AirDrop, and is critical to ZeroIP/Bonjour (also formerly Apple-branded as Rendezvous) addressing for computers so services can benefit from name resolution even when the device only has a link-local address or otherwise doesn’t have DHCP.

Fun Fact!
Ever see Computer Name (2) or similar number increments in parentheses? That could be because a resolution lookup returned the local name and Apple’s framework told the computer to keep incrementing until it saw an unused increment. In more rocky releases of macOS, it was thought to even occur just if both WiFi and Ethernet were connected to the same network…

When setting ‘ComputerName’ in System Preferences → Sharing, the change is inherited by this value, but you’ll see it append ‘.local’ in the text below that field.

3. HostName – by default, this is not set on out-of-the-box computers, and in testing we see the hostname command inherits the LocalHostName as a substitute. That default ‘unset’ state would result in the bash or zsh prompt matching the LocalHostName, unless HostName is set, in which case both prompt and the hostname command would use that. This is also why opening a terminal/command prompt may show a random name pulled via some quirk with name resolution on that particular network.

<code class="bash">~ allister$ scutil --get HostName
HostName: not set
air:~ allister$ scutil --get ComputerName
air
air:~ allister$ hostname
air.local
air:~ allister$ sudo scutil --set LocalHostName michaeljordon
air:~ allister$ hostname
michaeljordon.local

Last login: Tue Nov  5 15:16:11 on ttys000
michaeljordon:~ allister$
</code>

In many business settings, all three names are set to the same value at deploy time, and management tools sometimes enforce this. Some prefer to base device certificate names off the UUID or serial or a combination of derived values when applicable.

Thank you for coming to my Ted Talk, and belated welcome to 2020!

What makes me come out of an unintentional blogging sabbatical to talk about 10.12? Well, a pretty widespread issue we saw is time went out of sync all of a sudden recently, across a bunch of networks and even with a known-good image. NTP as a protocol is supposed to be lossy and expensive, so it should be resilient to failure to the point that you may not notice it skewing for quite some time, and querying it even shows a ‘resolution’ or tolerance that is measured across the various command line tools in 5+ digits/decimal places. Some implementations can’t even correct themselves if you’re off by greater than 15 minutes (or 1000 seconds). Apple has had its ups and downs, from the only other time a silent patch was pushed (previous to 10.13.2’s passwordless-root bug) to treating it like DNS resolution and rewriting its mechanisms every couple of releases. (Remember lookupd? AFP548 alumni have blogged about DNS confusion for quite some time.) I was particularly flummoxed when this issue cropped up again because it had previously occurred for all of the signage iPads in one of our offices. You could directly reproduce the issue by trying ntpq -p time.euro.apple.com and get time-ios.g.aaplimg.com: timed out, nothing received. Nothing was being explicitly blocked or… delayed at the firewall, but in that instance Apple’s NTP servers were not able to send a response back to the device due to an issue with how the wireless LAN controller was configured.

In this case, however, we were seeing ntpq: read: Connection refused, as if the process wasn’t running. But a trip to launchctl as sudo would disagree, telling you that the job was loaded/running, so… what gives? There are a few stackoverflows that will tell you about the mindblowingly deep new-math Apple’s been using to just win so hard, dunking on us with the swiftness by never meeting a good-enough that it couldn’t rewrite, but none of it seemed applicable anymore – ain’t no pacemaker in /usr/libexec on my systems, and knowing that timed is a completely new animal as of 10.13 anyway means I wasn’t too hot on the idea of learning older stuff now.

Luckily Mr. Oakley wrote this post, which pointed me to the /usr/libexec/ntpd-wrapper… shell script (:jackie:) which Sierra still relies on. Inspect that loaded launchd job at /System/Library/LaunchDaemons/org.ntp.ntpd.plist and you’ll notice it has a KeepAlive/PathState key with a value that it only runs as long as… /private/etc/ntp.conf is true, meaning the file exists. The contents would be something like server time.euro.apple.com., but for whatever reason over the past few weeks that file went missing entirely across hundreds of computers under our care. Putting it back in place immediately synced the clock. To the fixing machine! We use a lot of ugly Ruby in an elegant way via facter to do site detection, and even already had a ‘location’ fact for use with Simian that told us overall region (although Apple only shows 3 NTP server choices, and we consider India separate). Just detect if the file is missing, and if so, echo the appropriate server into that path, and Bob’s Your Uncle, Muriel’s Your Aunt. But then we hit another stumbling block – SIP!

You can’t echo text to /private/etc. You can mv files to /private/etc. Because, y’know, reasons. Hope this helps whoever out there is still clinging to 10.12 become less crazy. (But really, we just have an appliance-type use case that we haven’t migrated to 10.13 yet, everyone accepts the fact we’re on borrowed time and should upgrade ASAP. Cheers!)

(Paragraph-long parenthetical: We are going to stop calling it FV2 at some point and just let it own the ‘FileVault’ name, since we all know that’s what we’re talking about, right? FV1 mattered, but it sure didn’t catch on like its successor. FV1 was a neat parlor trick, like Time Machine doing hard links to :all_the_things:. Anyway.)

Although Apple always likes to implement things that would only make sense to its design/engineering/marketing departments, they still need to comply with the (U)EFI spec by having a partition available just for updating boot ROM, aka firmware. I’m a simple lad, so here’s the long ‘disambiguation’ section: EFI applied to the boot ROM chip has nothing to do with firmware on lots of other components jangling around in your Mac, like the TouchBar (running embeddedOS) or wifi controller, and removing the partition is pretty harmless pre-TouchBar because it only had been a staging area to apply an update.

TouchBar Macs came along and now we had A Problem: part of the activation process of what the TouchBar is, which is to say an iOS device very similar to the Apple Watch, was now embedded inside that ‘invisible’ EFI partition on the ‘host’ Mac. OS updates also updated the eOS image living in that same EFI partition, which needs to stay in-sync: if you leave the EFI partition but upgrade the version of the Mac OS via imaging, you haven’t delivered the new eOS image to the EFI partition, and you’ll see a nice black screen with Apple logo while it reaches across the internet to grab it. Which you had better hope is available and no proxies are in the way and you have an active internet connection. (eOS activation servers may have been up the other day, but we couldn’t get that image downloaded, even after multiple attempts at internet recovery, because whatever server hosts eOS versions was down. All Friday. Thanks Obama!)

Now APFS is bearing down on us, looking pretty ominous on the horizon. I’ll love the day-to-day speed boost for common operations, and the promise of Time Machine (and Siracusa’s dreams) being fulfilled, but otherwise it’s a very ‘hold on to your butts’ moment. The tricky part of where EFI overlaps with a filesystem is the APFS driver – as announced at the WWDC session, it’s a filesystem-based driver so that they can modify the format in the future without hard-coding it into the boot ROM directly. This means your firmware needs to be able to 1. recognize the APFS container format and 2. look for the driver inline (I guess is one way to put it) with that APFS container/volume(s).

So how do we get the firmware updated so that a 10.13 APFS image can restore successfully? Well a technique made reference to previously by others including @jazzace and adapted by @bruienne and @bochoven is to grab the FirmwareUpdate.pkg out of the full OS installer app bundle, then make sure the tools and scripts are in place so that it is universal across all supported hardware. Here’s what you’d run at the commandline, assuming you have the High Sierra Beta app in it’s default location and munkipkg installed and in your path:

/usr/bin/hdiutil mount /Applications/Install\ macOS\ High\ Sierra\ Beta.app/Contents/SharedSupport/InstallESD.dmg
/usr/sbin/pkgutil --expand "/Volumes/InstallESD/Packages/FirmwareUpdate.pkg" /tmp/FirmwareUpdate
munkipkg --create /tmp/FirmwareUpdateStandalone
/bin/cp /tmp/FirmwareUpdate/Scripts/postinstall_actions/update /tmp/FirmwareUpdateStandalone/scripts/postinstall
/bin/cp -R /tmp/FirmwareUpdate/Scripts/Tools /tmp/FirmwareUpdateStandalone/scripts/
munkipkg /tmp/FirmwareUpdateStandalone

And then it’s up to you to put it in your deployment mechanism of choice. Cheers!

Here’s a way to use one of the more popular collection of puppet-specific hooks to add checks before you commit locally across the modules you maintain and deploy.

First, for consistencies sake, you probably want to make sure you have a consistent directory in your home folder to point at. If you don’t have one already, you can create a binary directory to store stuff like csshX, munkipkg, and other fun items that you want in your path.

mkdir bin
chmod 700 bin
chmod +a "group:everyone deny delete" bin

(if you want to add it to your path temporarily, you’d run PATH=$PATH:~/bin or add it on each login by creating or appending ~/.bash_profile with the following export PATH=${PATH}:~/bin)

Download the current master version of the code here https://github.com/drwahl/puppet-git-hooks/archive/master.zip and unpack it into that bin directory.

Now in case you haven’t had the pure unadultered joy, it’s time to have fun with ruby via bundler and all of the things you’re actually going to perform the checks with:

sudo gem install -n /usr/local/bin bundler r10k rspec puppet-lint avoids the default gem install behavior of trying to write to SIP-protected directories cd ~/bin/puppet-git-hooks-master/
bundle install --path vendor/bundle

Now you just go to the git repo directory you’re hoping to lint or perform pre-commit checks on and run the following: ln -s ~/bin/puppet-git-hooks-master/pre-commit .git/hooks/pre-commit

Which brought to mind a vaguely-related tweet by a sometimes-visitor to the #security channel in MacAdmins Slack

https://twitter.com/osxreverser/status/860910010276216832

If I may pervert its purpose a bit and editorialize to do a send-up of that HN post…

I’m going to take this opportunity to plug my favourite open source project – the AutoPkg framework that gets stuff done with software, safely.

It can work as a homebrew replacement (and is custom-built to work on the Mac), comes with a humble collection of recipes that the community has expanded exponentially, and while the code is spare and written in python, it uses ‘trust info’ to fingerprint every moving part. Better than homebrew, many recipes check code signatures of signed downloaded artifacts, but also doesn’t require manual interaction to verify what the sha256 on an unsigned binary is. Unlike something like homebrew-cask, it doesn’t have homebrew in its name.

It can also work as a great way of bootstrapping an admin machine or just patching it out-of-band while testing a new release because it has install functionality! All the advantages of a package manager, without actually using *nix. Due to its functional nature, it comes with a wealth of advantages over homebrew and other hipster package managers. Once you get past the relatively trivial learning curve due to its huge adoption among Mac Admins, creating your own recipes or modifying existing ones is a breeze. It can create metadata artifacts that allow you to automatically ingest the software into whatever management system you work with, and one of the extensions even adds VirusTotal integration! Check out the AutoPkg wiki for more information.

It’s so flexible that people have built support for really out-there workflows: fetching Windows software, patching Macs with SCCM, uploading artifacts to random destinations via rsync or scp… and then another couple of doozies that help prove the extensibility of the framework, which we haven’t discussed here before.

Really, we’re going off the rails – you can put up a wall at the kernel level and delegate security to a product like BlockBlock, or take a hands-off, watch-and-know approach by just alerting wherever a new launchd job or executable shows up via osquery. But we’re already well down the autopkg path, so…

Since late 2014, AutoPkg has had a feature called CodeSignatureVerification that will look at a signed pkg or app and check it against a ‘known-good’ value. The certificate chain that ships with macOS means your computer trusts that artifact was signed by someone with access to the developers public/private key pair. Santa from the MacOps team at Google can monitor or lock down what apps or binaries can run, based on a certificate. But say you don’t have their fancy internal voting webapp with which to log and crowd-source the approval of unsigned/new binaries for your organization. Or you want to be able to tell the moment the cert on an app you provide via your org’s software mgmt system differs from the one you expect. Santa logs almost all script or binary executions, and you should really be aggregating those logs to build your whitelist, you should really ship down those whitelists from something like Zentral or Moroz in the absence of that server Google has yet to release. But if you haven’t yet, and use Munki, you can integrate this processor in your recipes to whitelist the new binary being installed to a system that has Santa during the preflight. This comes in handy when certain vendors can’t quite figure out how to track down all their moving parts and sign them. And if you’ve got a vendor with a less than helpful build process changing the cert in use, or there IS an actual hijack of the vendor’s release site where a new certificate is in use, you can get a head start by seeing the mismatch in your AutoPkg results report with this processor.

So to finish (wait, AFP548 still publishes posts?), you can either listen to reason

If Handbrake is one of your #autopkg recipes, make sure you check for the bad SHAs; Santa or @osquery can also help detect it. https://t.co/t8WegzIAbq

— Victor (groob) (@wikiwalk) May 6, 2017

or rants like mine. /me resumes sitting on hands