Clixel Blog

Implementing PHP Background Tasks With Beanstalk

Mon, 03 Feb 2025 00:00:00 +0000

I have a sprawling custom PHP codebase for a client that I started nearly a quarter century ago (!!), which has rapidly been increasing in complexity with database-intensive caching of various aggregate sums for customers and orders, a complex stock report / on order pipeline for products + distributors in various warehouse locations, and more long-running email + API operations that I’d like to keep out of the normal web request cycle.

Over the years, I slowly expanded on a rudimentary delayed task queue, with several periodically run cron jobs which query a cronjobs table for matching tasks based on type, item_id and date_added fields. This has worked well for all sorts of needs, especially when accumulating a list of long-running tasks to do in batches late at night or every few hours, but I’ve found myself having more urgent background tasks that need to run more quickly than “sometime in the next 1-60 seconds.” The user needs to see results from their actions within a few seconds, or they’ll think it isn’t working. I considered querying cronjobs and showing users a notice like “Hey! There’s a pending update for this item. Refresh in a minute or two to see changes.” — but that just feels dumb. Also, managing the lifecycle and schedules of application <-> mysql <-> stand-alone script <-> cronjob is a pain and annoyingly distributed (e.g., the crontab config isn’t stored in git).¹

After much searching around, I first tried Gearman since it had been around so long, assuming it was feature complete and solid. Overall it was pretty simple to set up, but it was a bit of a mess trying to get it running locally on my M1 MacBook, where I use homebrew to switch between a few versions of PHP². I kept having a “Please install libgearman” error barfed up when running pecl install gearman. I tried all sorts of fixes until I realized I could easily install it from source (duh! c’mon Nate), allowing me to specify necessary homebrew paths with ./configure --with-php-config=/opt/homebrew/opt/php@8.3/bin/php-config --with-gearman=$(brew --prefix gearman).

Once I had it running, I was able to set up a few worker classes and fired up Gearman Manager to handle daemonizing. Everything was working great until I realized I have some workers I would like to queue up in the future, and a few searches later I realized that would introduce even more dependencies for me to manage, possibly even reverting to using my cronjob setup with some tweaks for async tasks. I really wanted to keep this as simple and self-contained as possible.³

While searching around for delayed queue support in Gearman, I saw mention of Beanstalk as an alternative which supported future-dated tasks. It also seemed like Beanstalk was a bit more straightforward. Ok! Tear it all down! Let’s try this again!

Switching to Beanstalk was pretty painless, but ironically there was an aspect of Gearman that it didn’t support: unique keys for each task, to avoid queueing up duplicates. I saw a suggested solution that reminded me of what I was just trying to avoid: setting up your own middleware to track keys and avoid duplication. Argh! I decided to worry about it later.⁴

I started with two workers, one for order-related hooha, and one for stock report giggles. The workers basically look like this:

<?php
require(__DIR__.'/bootstrap.php');

use Pheanstalk\Pheanstalk;
use Pheanstalk\Values\TubeName;

$pheanstalk = Pheanstalk::create($_ENV['BEANSTALKD_SERVER']);
$tube = new TubeName('orders');
$pheanstalk->watch($tube);

// This hangs until a Job is produced
$job = $pheanstalk->reserve();

try {

  $data = json_decode($job->getData(), true);

  // Merge in "force-run" flag to opts to trigger running task immediately
  $opts = array_merge(($data['opts'] ?? []), [
    'force-run' => 1
  ]);

  // Run method with 'force-run' flag + original params
  APP()->orders->doFooBar($opts);

  // Delete job from queue
  $pheanstalk->delete($job);

} catch(\Exception $e) {
  
  $error_msg = $e->getMessage();

  // If mysql timed out, no need to log error, just release for next worker
  if (!preg_match('/MySQL server has gone away/i', $error_msg)) {
    echo sprintf("%s: Error running Job %s (#%s): %s\n", date('Y-m-d H:i:s'), $data['task'], $job->getId(), $error_msg);
  }

  // Let another worker retry this job
  $pheanstalk->release($job);

}

I manage these workers with Supervisor, configured in /etc/supervisor/conf.d/workers.conf:

[program:orders]
command=/usr/bin/php /var/www/app/workers/%(program_name)s.php
process_name=%(program_name)s_%(process_num)02d
stdout_logfile=/var/logs/supervisor/%(program_name)s.log
stderr_logfile=/var/logs/supervisor/%(program_name)s-error.log
user=www-data
autostart=true
autorestart=true
numprocs=3
startsecs=0

[program:stock-report]
command=/usr/bin/php /var/www/app/workers/%(program_name)s.php
process_name=%(program_name)s_%(process_num)02d
stdout_logfile=/var/logs/supervisor/%(program_name)s.log
stderr_logfile=/var/logs/supervisor/%(program_name)s-error.log
user=www-data
autostart=true
autorestart=true
numprocs=3
startsecs=0

I was having issues keeping workers alive, as many of my “long-running” tasks turned out to be much quicker than supervisor liked, and it would stop respawning after a few requests (by default, startretries=3). These tasks hang while waiting for a job, but as soon as a job enters the queue, they process it and quit in a fraction of a second. Supervisor was upset until I added startsecs=0 to the config, which specifies “the number of seconds a program must stay running after a startup to be considered successfully started.”

Originally I thought these would be daemon processes, and I might switch to that method, using a while statement with sleep after each job to keep it from hogging CPU. But for now, it seems to be working fine to just fire up a new process to handle each job.

I set up several of my existing methods that I want to support async with a force-run flag. If it’s not present, they will add the task to a Beanstalk queue⁵ with a JSON string of the params and return. The worker then appends force-run to the params, and runs the same method in the background process.

Everything has been working well for a first dabble in background/async tasks. I had a hard time finding comprehensive explanations of the entire setup + configs, so hopefully this helps any other poor sucker hacking away at a custom PHP stack.

All said, my homegrown crontab system is still in use for several scripts. I could improve the system by having a central cron pipeline, and only one check-pending-cronjobs script that runs every minute. It just hasn’t hit that threshold of pain yet to inspire refactoring. ↩︎
I have tried several times to get on the Docker train, but it always seems to introduce more problems than solutions for my local development setup. I just have so many legacy projects + a handful of more modern Craft/Statamic sites, and for the most part they all work fine sharing the same instances of Apache/MariaDB/PHP 8.x. ↩︎
I tend to dive into all sorts of obscure tech solutions, learn enough to get it working for my needs, then not touch for a year or more. When I come back to it I have to reverse engineer my own dang work. I do leave copious comments everywhere for future Nate, but that doesn’t help much when I invariably have to run updates and suddenly there are incompatibilities and changes that make my old notes useless. ↩︎
Looking back now, I realize this isn’t worth worrying about, as most of my tasks run immediately, and should run again if a duplicate is queued up soon afterward. ↩︎
Only if Beanstalk is running, otherwise it just runs the method immediately. This allows the staging site to not require its own set of supervisor’d PHP processes constantly running. ↩︎

Getting US Timezones From Zipcodes in PHP Without an API

Thu, 28 Mar 2024 00:00:00 +0000

A client wanted to batch update a customer database with timezones so their sales team could have a better idea when best to harass folks by phone. Looking for solutions to determine timezones from an address really sent me down a rabbit hole. Stack Overflow has answers that go all over the place over the last decade plus, most of them coming down to using Google Maps API, or Nominatim/OpenStreetMap via geopy in python. The first step is to determine latitude and longitude, and then either make another API call to get timezone, or some manner of querying via boundary maps. Another solution I found was to have a giant SQL table of zipcode to timezone lookups, but it was out of date, and required quite a bit of storage for such a simple task.

I really didn’t need extreme accuracy, so I figured using zipcodes would be ideal. Searching around I found TZip, a long-abandoned-but-oft-forked Ruby gem which did just this, and without any external API requirements. I seem to keep finding abandoned Ruby libraries that are almost perfect solutions to obscure requirements I have, like the ONIX gem that I forked and hacked away at myself, which I used in a simple Sinatra app to convert book data in JSON to ugly ONIX XML.

When I peeked under the hood, I realized this Ruby library would be easy to adapt to PHP, so I could keep from having another decoupled service running that needed updating separate from my behemoth custom PHP codebase. I refactored output to match the terminology my client requested (“Eastern” instead of “America/Eastern”), and iirc made a few changes to zip lookups for changes in the last 15 years.

Usage:

$timezone = new Timezone();
echo $timezone->findTimezone(27705);
// 'Eastern'

I haven’t heard any complaints after using this for over a year, so I think it’s as accurate as needed! Hopefully it’s of use to someone else.

Revisiting this post (which I originally started writing in early 2023, egads), I had to track down the Ruby library I mentioned, and in doing so found Ziptz, a newer Ruby gem that would do the trick if you need this in Ruby.

/**
 * Timezone lookups based on ZIP code
 */
class Timezone {

  // Zonings and mappings for determining timezone as accurately as possible
  public $zonings = [];
  public $mappings = [];

  /**
   * Init
   */
  public function __construct() {
    $this->zonings['Alaska'] = explode(' ', '995 996 997 998 999');
    $this->zonings['Central'] =  explode(' ', '35 36 370 3720 3723 381 383 39');
    $this->zonings['Central'] = array_merge($this->zonings['Central'], explode(' ', '4641'));
    $this->zonings['Central'] = array_merge($this->zonings['Central'], explode(' ', '50 51 52 53 54 55 56 57 581 582 585'));
    $this->zonings['Central'] = array_merge($this->zonings['Central'], explode(' ', '60 61 62 63 64 65 660 661 6660 672 680 681 685'));
    $this->zonings['Central'] = array_merge($this->zonings['Central'], explode(' ', '7'));
    $this->zonings['Central'] = array_merge($this->zonings['Central'], explode(' ', '371 372 373 375 377 380 382 384 385 386 387 388 389 401 420 421 422 423 424 426 427 498 498 499 580 583 584 587 588 662 664 665 666 667 668 669 670 671 673 674 675 676 677 678 679 683 684 686 687 688 689 690 691 692'));
    # Indiana zip prefixes span Eastern/Central timezones so the full 5 digits is needed
    $this->zonings['Central'] = array_merge($this->zonings['Central'], explode(' ', '47514 47515 47520 47525 47551 47574 47576 47586 47588 46301 46302 46303 46304 46307 46308 46310 46311 46312 46319 46320 46321 46322 46323 46324 46325 46327 46340 46342 46345 46346 46347 46348 46349 46350 46352 46355 46356 46360 46361 46365 46368 46371 46372 46373 46375 46376 46377 46379 46380 46381 46382 46383 46384 46385 46390 46391 46392 46393 46394 46366 46401 46402 46403 46404 46405 46407 46409 46531 46534 46968 46341 46374 46532 46552 47523 47537 47577 47601 47639 47640 47660 47943 47948'));

    $this->zonings['Eastern'] = explode(' ', '0 1 2');
    $this->zonings['Eastern'] = array_merge($this->zonings['Eastern'], explode(' ', '30 31 32 33 34 376 379 398 399'));
    $this->zonings['Eastern'] = array_merge($this->zonings['Eastern'], explode(' ', '402 405 410 43 44 45 460 462 462 480 481 482 483 485 488 489 490 495 496 '));
    $this->zonings['Eastern'] = array_merge($this->zonings['Eastern'], explode(' ', '569'));
    $this->zonings['Eastern'] = array_merge($this->zonings['Eastern'], explode(' ', '021 034 037 038 040 041 042 046 047 048 049 053 056 058 063 064 066 067 068 069 070 074 077 085 086 088 373 374 377 378 400 401 403 404 407 408 409 411 412 413 414 415 417 418 484 486 487 491 492 493 494 497 498 499'));
    $this->zonings['Eastern'] = array_merge($this->zonings['Eastern'], explode(' ', '400 401 403 404 406 407 408 409 411 412 413 414 415 416 417 418 420 421 422 423 424 425 426 427'));
    # Indiana zip prefixes span Eastern/Central timezones so the full 5 digits is needed
    $this->zonings['Eastern'] = array_merge($this->zonings['Eastern'], explode(' ', '46112 46113 46121 46124 46130 46133 46148 46158 46160 46164 46180 46181 46186 46406 46408 46506 46511 46550 46562 46574 46705 46723 46725 46732 46747 46761 46767 46770 46776 46783 46788 46792 46910 46919 46929 46940 46950 46960 46970 46982 46991 47001 47006 47018 47040 47041 47060 47106 47118 47122 47125 47138 47140 47147 47170 47177 47224 47230 47232 47235 47244 47250 47264 47272 47281 47325 47336 47352 47354 47359 47368 47373 47384 47432 47433 47436 47438 47448 47456 47468 47513 47532 47553 47833 47834 47837 47842 47858 47868 47874 47917 47918 47926 47932 47952 47960 47970 46102 46103 46104 46105 46106 46107 46110 46111 46115 46117 46118 46120 46122 46123 46125 46126 46127 46128 46129 46131 46135 46140 46142 46143 46144 46146 46147 46149 46150 46151 46154 46155 46156 46157 46161 46162 46163 46165 46166 46167 46168 46170 46171 46172 46173 46175 46176 46182 46183 46184 46501 46502 46504 46507 46508 46510 46513 46514 46515 46516 46517 46524 46526 46527 46528 46530 46536 46537 46538 46539 46540 46542 46543 46544 46545 46546 46553 46554 46555 46556 46561 46563 46565 46567 46570 46571 46572 46573 46580 46581 46582 46590 46595 46601 46613 46614 46615 46616 46617 46619 46624 46626 46628 46634 46635 46637 46660 46680 46699 46701 46702 46703 46704 46706 46710 46711 46713 46714 46721 46730 46731 46733 46737 46738 46740 46741 46742 46743 46745 46746 46748 46750 46755 46759 46760 46763 46764 46765 46766 46769 46771 46772 46773 46774 46777 46778 46779 46780 46781 46782 46784 46785 46786 46787 46789 46791 46793 46794 46795 46796 46797 46798 46799 46801 46802 46803 46804 46805 46806 46807 46808 46809 46814 46815 46816 46818 46819 46825 46835 46845 46850 46851 46852 46853 46854 46855 46856 46857 46858 46859 46860 46861 46862 46863 46864 46865 46866 46867 46868 46869 46885 46895 46896 46897 46898 46899 46901 46902 46903 46904 46911 46912 46913 46914 46915 46916 46917 46920 46921 46922 46923 46926 46928 46930 46931 46932 46933 46935 46936 46937 46938 46939 46941 46942 46943 46945 46946 46947 46951 46952 46953 46957 46958 46959 46961 46962 46965 46967 46971 46974 46975 46977 46978 46979 46980 46984 46986 46987 46988 46989 46990 46992 46994 46995 46998 47003 47010 47012 47016 47017 47021 47022 47023 47024 47025 47030 47031 47032 47033 47034 47035 47036 47037 47039 47042 47102 47104 47107 47108 47110 47111 47112 47114 47115 47117 47119 47120 47124 47126 47129 47130 47131 47132 47133 47134 47135 47136 47141 47142 47143 47144 47146 47150 47151 47160 47161 47162 47163 47164 47165 47166 47167 47172 47190 47199 47201 47202 47203 47220 47223 47225 47226 47227 47228 47229 47231 47234 47236 47240 47243 47245 47246 47247 47249 47260 47261 47263 47265 47270 47273 47274 47280 47282 47283 47302 47303 47304 47305 47306 47307 47308 47320 47322 47324 47326 47327 47330 47331 47334 47335 47337 47338 47339 47340 47341 47342 47344 47345 47346 47348 47351 47353 47355 47356 47357 47358 47360 47361 47362 47366 47367 47369 47370 47371 47374 47375 47380 47381 47382 47383 47385 47386 47387 47388 47390 47392 47393 47394 47396 47401 47402 47403 47404 47405 47406 47407 47408 47420 47421 47424 47426 47427 47429 47431 47434 47435 47437 47439 47441 47443 47445 47446 47449 47451 47452 47453 47454 47455 47458 47459 47460 47462 47463 47464 47465 47467 47469 47470 47471 47531 47536 47550 47552 47556 47579 47610 47611 47612 47613 47615 47616 47617 47618 47619 47620 47629 47630 47631 47633 47634 47635 47637 47638 47647 47648 47649 47654 47665 47666 47670 47683 47701 47702 47703 47704 47705 47706 47708 47710 47711 47712 47713 47714 47715 47716 47719 47720 47721 47722 47724 47725 47728 47730 47731 47732 47733 47734 47735 47736 47737 47740 47747 47750 47801 47802 47803 47804 47805 47807 47808 47809 47830 47831 47832 47836 47838 47840 47841 47845 47846 47847 47848 47849 47850 47851 47852 47853 47854 47855 47857 47859 47860 47861 47862 47863 47865 47866 47869 47870 47871 47872 47875 47876 47878 47879 47880 47881 47882 47884 47885 47901 47902 47903 47904 47905 47906 47907 47909 47916 47920 47921 47922 47923 47924 47925 47928 47929 47930 47933 47940 47941 47942 47944 47949 47950 47951 47954 47955 47958 47959 47962 47963 47964 47965 47966 47967 47968 47969 47971 47974 47975 47977 47978 47980 47981 47982 47983 47984 47986 47987 47988 47989 47990 47991 47992 47993 47994 47995 47996 47997 47116 47123 47137 47145 47174 47175 47457 47501 47512 47516 47519 47521 47522 47524 47527 47528 47529 47535 47541 47542 47545 47546 47547 47549 47557 47558 47561 47562 47568 47573 47575 47578 47580 47581 47591 47596 47597 47564 47567 47584 47585 47590 47598 46985 46996 47946 47957 47011 47019 47020 47038 47043'));

    $this->zonings['Hawaii'] = explode(' ', '967 968');

    $this->zonings['Mountain'] = explode(' ', '577');
    $this->zonings['Mountain'] = array_merge($this->zonings['Mountain'], explode(' ', '59'));
    $this->zonings['Mountain'] = array_merge($this->zonings['Mountain'], explode(' ', '798 799'));
    $this->zonings['Mountain'] = array_merge($this->zonings['Mountain'], explode(' ', '80 81 82 83 831 84 8501 87 88'));
    $this->zonings['Mountain'] = array_merge($this->zonings['Mountain'], explode(' ', '979'));
    $this->zonings['Mountain'] = array_merge($this->zonings['Mountain'], explode(' ', '586 677 678 690 691 692 693'));

    $this->zonings['Pacific'] = explode(' ', '835 838 889 89');
    $this->zonings['Pacific'] = array_merge($this->zonings['Pacific'], explode(' ', '90 91 92 93 94 95 960 961 962 963 964 965 969 972 973 975 977 98 990 991 992 993 994'));
    $this->zonings['Pacific'] = array_merge($this->zonings['Pacific'], explode(' ', '970 971 974 976 978'));
    $this->zonings['Pacific'] = array_merge($this->zonings['Pacific'], explode(' ', '85 86')); // Arizona

    $this->zonings['Unused'] = explode(' ', '000 002 003 004 099');
    $this->zonings['Unused'] = array_merge($this->zonings['Unused'], explode(' ', '213 269'));
    $this->zonings['Unused'] = array_merge($this->zonings['Unused'], explode(' ', '343 348 353'));
    $this->zonings['Unused'] = array_merge($this->zonings['Unused'], explode(' ', '419 429'));
    $this->zonings['Unused'] = array_merge($this->zonings['Unused'], explode(' ', '517 518 519 529 533 536 552 568 578 579 589'));
    $this->zonings['Unused'] = array_merge($this->zonings['Unused'], explode(' ', '621 632 642 643 659 663 682 694 695 696 697 698 699'));
    $this->zonings['Unused'] = array_merge($this->zonings['Unused'], explode(' ', '702 709 715 732 742 771'));
    $this->zonings['Unused'] = array_merge($this->zonings['Unused'], explode(' ', '817 818 819 839 848 849 854 858 861 862 866 867 868 869 876 886 887 888 892 896 899'));
    $this->zonings['Unused'] = array_merge($this->zonings['Unused'], explode(' ', '909 929 987'));

    // Create lookup array for determining timezone
    foreach ($this->zonings as $zone => $prefixes) {
      foreach ($prefixes as $prefix) {
        $this->mappings[$prefix] = $zone;
      }
    }
  }

  /**
   * Find timezone for a ZIP code
   */
  public function findTimezone($zipcode) {
    $zipcode = substr(trim($zipcode), 0, 5);
    // Loop through levels of specificity to try finding a timezone
    foreach([5,4,3,2,1] as $i) {
      $chk = substr($zipcode, 0, $i);
      if (array_key_exists($chk, $this->mappings)) {
        if ($this->mappings[$chk] == 'Unused')
          return '';
        else
          return $this->mappings[$chk];
      }
    }
    return '';
  }

}

Where I Live, 2023 Edition

Sun, 08 Jan 2023 00:00:00 +0000

I used to do these every few years, listing out all the software I use on a daily basis. But looking back at the last one I posted ten years ago (gulp!), I realize not that much has changed. Sadly? Gladly? I’m not sure tbh. Good software is timeless software, apparently. Though as a developer, I know that keeping any sort of software afloat over time is a lot of work, no matter the platform, so I appreciate that some of these are still around, let alone still best of breed.

Sublime Text (now at v4) is where I’ve now spent a good portion of my life for 12+ years. I know everybody’s salivating over VS Code these days, but I recently gave it a shot, as well as another earnest go at Nova, and I just keep coming back to Sublime Text. It’s so damn fast, and it has come a long way with context-aware autocompletes, especially for highly-custom codebases, which I spend most of my time in. I have a few sites that now stretch back 20 years, and their quirkiness is welcome in ST-land, where VS Code and Nova don’t recognize my weird brain patterns 90% of the time.
I’ve switched to mostly using Safari these days, because of how fast & lean it is, but it’s probably 50/50 with Brave usage. I prefer how Brave handles View Source, and Safari drives me crazy how it so often treats a manually-entered *.localhost URL as a search. If the path isn’t in your history, you have to prepend http:// like an animal, or you end up on a useless search page for clixel.localhost/whatever. For the life of me, I can’t figure out how to tell Safari to always resolve *.localhost to 127.0.0.1 for dev domains.
I used The Hit List for ages, but it started crashing consistently at some point, and seems to be abandoned by the folks that bought it (Karelia) about 5 years ago. I decided to finally give Things a try, and haven’t looked back. It’s a very streamlined, opinionated to-do app that I love using on my Mac and iPhone (and Watch).
Fork has taken over as my git client from GitX. They’re very similar, but Fork has excelled at making everything a bit more elegant and streamlined, and is a paid app, so hopefully it keeps on keepin’ on. It’s an amazing git GUI client, well worth the $50.
I’m elated to say that Pixelmator Pro has been able to totally replace Photoshop for me. It’s insane what you get $50, and they often have sales where you can get it for an even more insane $20. (After leaving Firebelly in 2022, I lost my access to a CC subscription, and I refuse to give Adobe a monthly fee for their garbage apps.) I also keep up my registration and fire up Acorn often for quick image needs. Both Acorn and Pixelmator Pro are super powerful options. My dream of Adobe having strong indie competition on the Mac has finally been fulfilled.
I still use ForkLift 24/7, now at version 3. It’s an amazing dual-pane local/remote file manager, which I’ve used daily since I started these silly posts 13 years ago. I still pay for Transmit (and Nova, which has Transmit built-in), but honestly never use it unless ForkLift barfs at some obscure FTP requirement (I’m looking at you, nearlyfreespeech).
Sequel Ace has taken over for Sequel Pro, as it’s regularly updated. It’s pretty much the same layout and functionality, and still a great app for local mysql management, though I do miss the old syrup-drenched Sequel Pro icon.
iTerm2 is still open 24/7 on my machine. I’ve tried various other terminals but always come back to iTerm.
For notes, I switched from Notational Velocity to using The Archive for several years, but in the last year have been using Drafts full time as a daily to-do and archival note repository. It’s infinitely configurable and oft-updated, plus I appreciate that it has a healthy pay-to-play setup so it will stick around for the long run.
I’ve honestly been pretty disgusted with how Apple software has been falling apart (really…it’s been so bad), and Music is one of the worst. Doppler is a lean & focused local music player for Mac that I’ve been enjoying as a replacement. The dev is super responsive and has added several features from feedback I’ve sent.
Another app still in regular use is Tweetbot, but since Elon has taken over Twitter, I’ve been spending more and more time at Mastodon, and look forward to Tapbots’ Ivory client. In the meantime I’ve been using the very well crafted Mastonaut for Mac, and the default Mastodon client for iOS.
I still pay for CodeKit but only use it on a few personal projects. Usually I have a gulp or webpack setup. Hilarious version update notes are still a bonus.
Apple Mail is on the list because I’ve been doing my best to return to owning my data. I deleted my Facebook, then switched from using 100% Gmail to hosted email via nate@clixel.com on Opalstack (which came about when WebFaction shit the bed), and recently even deleted my Instagram, which I’m still ambivalent about. Everybody’s still on Instagram! But I kept finding myself hating my time spent there, and the direction they’ve been taking the app, chasing TikTok as if all services must compete for the exact same experience. Instagram was such a great site when it was solely focused on photography (eventually with a dash of short-form video) from your friends. I miss ye olde Instagram as much as I miss Flickr.
For invoicing, I used On The Job for much of the last 10 years, but when Catalina came out it stopped working. After struggling with several betas and frequent issues + slow updates, I ended up writing my own web-based invoicing software (!!), and have been using that since, even utilizing it as a menubar app via Unite which creates a Mac app from a webpage (previously I used Fluid until that also stopped being updated).

Moving from Fabric 1.x to Fabric 2.x

Thu, 07 Jan 2021 17:48:39 -0500

I’ve long used Fabric for simple deploy scripts to essentially ssh in, git pull, composer install or bundle, restart apps, clear caches, etc. It’s also handy for local dev shortcuts like fab assets as a stand-in for the more wordy npx gulp --production or fab dev to simultaneously fire up sphinx, a tornado websocket app, gulp, etc.

With a new M1 MacBook Air, I ran into issues getting fabric@1.4 to play nicely with homebrew, and decided to once again try my hand at converting my simple deploy scripts to the new v2 syntax. Information is scarce on sample Fabric 2 scripts, but I did find an aptly titled post “Why Is Fabric 2 so Hard?” which helped me get started.

There’s an upgrade guide, but my god it’s complicated, until it’s not complicated enough with the sample v1->v2 migration at the end of the page.

Here’s my simple v1 fabfile.py:

from fabric.api import *

env.hosts = ['natebeaty.opalstacked.com']
env.user = 'deploy'
env.remotepath = '/home/natebeaty/apps/nb-craft-staging'
env.git_branch = 'master'
env.forward_agent = True
env.php_binary = 'php74'

def production():
  env.remotepath = '/home/natebeaty/apps/nb-craft-staging'
  env.hosts = ['natebeaty.com']

def assets():
  local('npx gulp --production')

def deploy(composer='y'):
  update()
  if composer == 'y':
    composer_install()
  clear_cache()

def update():
  with cd(env.remotepath):
    run('git pull origin {0}'.format(env.git_branch))

def composer_install():
  with cd(env.remotepath):
    run('%s ~/bin/composer.phar install' % env.php_binary)

def clear_cache():
  with cd(env.remotepath):
    run('./craft clear-caches/compiled-templates')
    run('./craft clear-caches/data')

And this is where I’m at so far with a functional v2 version:

from fabric import task
from invoke import run as local

remote_path = "/home/natebeaty/apps/nb-craft-staging"
remote_hosts = ["deploy@natebeaty.opalstacked.com"]
php_command = "php74"

# set to production
@task
def production(c):
    global remote_hosts, remote_path
    remote_hosts = ["deploy@natebeaty.com"]
    remote_path = "/home/natebeaty/apps/nb-craft-production"

# deploy
@task(hosts=remote_hosts)
def deploy(c):
    update(c)
    composer_update(c)
    clear_cache(c)

def update(c):
    c.run("cd {} && git pull".format(remote_path))

def composer_update(c):
    c.run("cd {} && {} ~/bin/composer.phar install".format(remote_path, php_command))

def clear_cache(c):
    c.run("cd {} && ./craft clear-caches/compiled-templates".format(remote_path))
    c.run("cd {} && ./craft clear-caches/data".format(remote_path))

# local commands
@task
def assets(c):
    local("npx gulp --production")

Still a few things to port over, but it’s working well enough. The part that feels ugly is the use of global in production(), but I couldn’t find any examples of how folks are overriding connection configs in a Fabric v2 file.

Using c.local() I was getting errors finding npx because my $PATH was different, missing the M1 homebrew /opt/homebrew/bin. This StackOverflow comment pointed me to using from invoke import run as local then local() instead of c.local(). This fixed my $PATH issues and npx gulp worked fine.

Gitlog Automagical Timesheet Filler Outer Redux

Tue, 21 May 2019 00:00:00 +0000

I’ve been using variations on this script for several years to spit out my git commit log in a timesheet-friendly format, initially just as a single command.

I’ve now refined it to a li’l bash script that has a few extra tricks up its sleeve (updated yet again in 2025):

#!/bin/bash
# Pulls git log entries for a day formatted for pasting in a timesheet

# Usage:
# gitlog = all commits today
# gitlog -2 = all commits -2 days ago
# gitlog 2024-10-03 or 10/3/2024 or 10-3

args=$1
# Replace / with - in args for alternate date formats e.g. 10/3/2024
args=${args//\//\-}

# Default day to today if no args are sent
day=`date +%F`

# Convert various argument formats to YYYY-MM-DD
if [[ $args =~ ^[0-9]{1,4}\-[0-9]{1,2}\-[0-9]{1,2}$ ]]; then
	# already in format needed
	day=$args
elif [[ $args =~ ^[0-9]{1,2}\-[0-9]{1,2}$ ]]; then
	# 10-10 -> 2024-10-10
	day=`date -jf %m-%d +%Y-%m-%d $args`
elif [[ $args =~ ^[0-9]{1,2}\-[0-9]{1,2}\-[0-9]{1,4}$ ]]; then
	# 10-10-2024 -> 2024-10-10
	day=`date -jf %m-%d-%Y +%Y-%m-%d $args`
elif [[ $args =~ ^\-[0-9]{1,2}$ ]]; then
	# -2 (days ago)
	day=`date -v"$args"d +%F`
fi

# Store output to var so we can spit it out to console then shove it in the clipboard
logs=$(git log --all --author="`git config user.name`" --no-merges --format="%s %ai" | grep $day | perl -pe 's/ (\d{4}\-\d{2}\-\d{2} \d{2}:\d{2}:\d{2} \-\d{4})//g' | perl -pe 's/^(compiled assets|pedant|cleanup|typo)\n$//g' | tail -r | perl -pe 's/\n/; /' | sed -e "s/; $//g")

if [ "$logs" = "" ]; then
	echo "No git logs found for $day"
else
	echo -e "Log for $day (also in clipboard):\n----\n$logs"
	echo -e $logs | pbcopy
fi

You set your name in the script, drop it in your ~/bin dir and chmod +x it, then run gitlog from a repo dir with various options, the default being the current day. Use gitlog -1 to show commits from yesterday, or gitlog 2025-01-01 to show a specific date. Your git commit messages are concatenated in a format suitable for timesheets, ready to be pasted into whatever time-tracking program you prefer. e.g.:

$ gitlog
Log for 2025-02-20 (also in clipboard):
----
slight copy edits; update highlightjs and change theme to agate; host highlightjs locally; simple dark theme

Chrome 63 breaks local *.dev domains

Tue, 12 Dec 2017 00:00:00 +0000

I did a complete wipe and reinstall of macOS High Sierra on my aging 2012 mbp, thinking it would solve all my random woes (false). As I was setting up my dev environment, I was baffled when both Safari and Chrome were redirecting all *.dev domain requests to https. Console showed no 301 redirects. Wth? I thought it was something I messed up in Apache config and spent a few hours beating my head against the keyboard to no avail.

Turns out .dev is a TLD owned by Google, and they recently preloaded HSTS rules for the TLD which means you cannot use http with *.dev any more. Fun!

I decided to switch all my dev domains to *.localhost. (I also toyed around with adding self-signed SSL certs for each *.dev domain, with some luck on my home computer and less luck on my work computer. In the end I decided to just go with switching to *.localhost.)

Here’s a quick rundown of things I had to do. Hopefully it helps someone.

I use dnsmasq to avoid having to edit /etc/hosts for every dev domain I work on. This gist has the latest way to do this.

First, add *.localhost to dnsmasq setup:

echo 'address=/.localhost/127.0.0.1' >> $(brew --prefix)/etc/dnsmasq.conf
sudo bash -c 'echo "nameserver 127.0.0.1" > /etc/resolver/localhost'
sudo brew services stop dnsmasq && sudo brew services start dnsmasq

ping foo.localhost for sanity to see if it works. May have to reboot if it doesn’t. I did, but it may have been from other finagling.

I use Homebrew Apache and mod_vhost_alias (good tutorial here) so open up the vhost config with subl /usr/local/etc/httpd/extra/httpd-vhosts.conf and add this block:

<virtualhost>
  ServerAlias localhost *.localhost
  VirtualDocumentRoot /Users/nate/Sites/%1/web/
  UseCanonicalName Off
</virtualhost>

sudo apachectl restart

We use Bedrock to make Wordpress development 17% less painful (note document root above with …/web/). The only thing you need to do is update your local .env file with WP_HOME=http://cfs.localhost. If you use the super handy wp-cli you can run wp search-replace 'cfs.dev' 'cfs.localhost' to update db references.

We have long been using this excellent guide from Grav as a jumping off point to set up our dev environments, which already has an update noting this change from Chrome 63.

WebFaction Https Redirects

Tue, 04 Oct 2016 00:00:00 +0000

Yet another thing to file under “beating my head against a wall for hours before figuring out something simple"—argh!

I wanted to redirect all requests to https for a WebFaction-hosted site, and wasn’t having any luck with the normal:

RewriteEngine On
RewriteCond %{HTTPS} !on
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]

I was getting "too many redirects” errors. After digging around I found mention of https redirects in (now-defunct) WebFaction docs with this code:

RewriteEngine On
RewriteCond %{HTTP:X-Forwarded-SSL} !on
RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]

Note the %{HTTP:X-Forwarded-SSL} instead of %{HTTPS}.

Well howdy-doody it works now, hallelujah, back to more important coding matters like my giant SVG head navigation.

libsass, node-sass, gulp-sass, my-ass

Tue, 17 Nov 2015 00:00:00 +0000

This error:

Error: `libsass` bindings not found. Try reinstalling `node-sass`

was showing up in older Bedrock projects, causing two devs to pull some hair out—well one of them still has hair—and it turned out to be a simple update of gulp-sass in the packages.json file:

-    "gulp-sass": "^1.3.3",
+    "gulp-sass": "^2.0.0",

npm update and you’re good to go.

Sequel Pro bombing after MySQL update

Tue, 17 Nov 2015 00:00:00 +0000

After a brew update and brew upgrade, Sequel Pro suddenly stopped connecting to localhost, bombing without much explanation. Several google searches later, I found this ticket which mentioned running mysql_upgrade.

It’s possible I missed a homebrew notice about this, as it updated a ton of packages, but this ended up solving the issue:

mysql_upgrade -u root -p

Followed by restarting mysql:

launchctl unload ~/Library/LaunchAgents/homebrew.mxcl.mysql.plist launchctl load ~/Library/LaunchAgents/homebrew.mxcl.mysql.plist

Duotone Images With ImageMagick In Wordpress

Tue, 12 May 2015 00:00:00 +0000

My dayjob studio Firebelly loves duotone images in their site designs. When we’re using Rails, it’s pretty easy to use Paperclip or Dragonfly to send along commands to ImageMagick on the fly, e.g.

# Paperclip
has_attached_file :image_file, 
  styles: { wallpaper: "1600x1000>" },
  convert_options: {
    wallpaper: "-quality 60 -strip -fill Black -colorize 27% -background Black \\( +clone -fill White -colorize 100 -background \"Gray(70%)\" -vignette 0x65+25%+5% \\) -compose multiply -composite"
  }

# Dragonfly

We’re using this technique on a Bedrock/Sage Wordpress site, and I wanted to avoid applying this to every uploaded image via a WP hook. I ended up with this beast of a function (fugly PHP alert) that creates a duotone image using two hex values:

/**
 * Get header bg for post, duotone treated with the random IHC_BACKGROUND + Dark Blue 
 */
function get_header_bg($post) {
  $header_bg = false;
  if (has_post_thumbnail($post->ID)) {
    $thumb_id = get_post_thumbnail_id($post->ID);
    $background_image = get_attached_file($thumb_id, 'full', true);
    $upload_dir = wp_upload_dir();
    $base_dir = $upload_dir['basedir'] . '/backgrounds/';

    // Build treated filename with thumb_id in case there are filename conflicts
    $treated_filename = preg_replace("/.+\/(.+)\.(\w{2,5})$/", $thumb_id."-$1-".IHC_BACKGROUND.".$2", $background_image);
    $treated_image = $base_dir . $treated_filename;

    // If treated file doesn't exist, create it
    if (!file_exists($treated_image)) {

      // Create background directory if necessary
      if(!file_exists($base_dir)) {
        mkdir($base_dir);
      }

      exec('/usr/local/bin/convert '.$background_image.' -colorspace gray -level +10% +level-colors "#44607f","#'.IHC_BACKGROUND.'" '.$treated_image);
    }    
    $header_bg = ' style="background-image:url(' . $upload_dir['baseurl'] . '/backgrounds/' . $treated_filename . ');"';
  }
  return $header_bg;
}

It’s always so difficult finding info on this sort of thing in ImageMagick forums, thought I’d give it a home in case someone else needs it. The basic command is:

convert source.jpg -colorspace gray -level +10% +level-colors "#44607f","#c1d6d8" treated.jpg

Which will give you this sort of treatment:

Update 4/2017: If newer versions of ImageMagick are giving you trouble, you can use this variation to get a similar duotone effect:

convert in.jpg +profile "*" -resize 1400x -quality 65 -modulate 100,0 -size 256x1! gradient:#44607f-#C2D6D9 -clut out.jpg

String Interpolation Using Rails ActiveRecord Objects

Mon, 23 Feb 2015 00:00:00 +0000

I often find myself wanting to use several attributes of a ActiveRecord object in a string, but have until now been unable to track down exactly how to do that.

Say I have this simple model with a custom method:

class Applicant < ActiveRecord::Base
    def full_name
      [first_name,last_name].reject{ |e| e.empty? }.join ' '
    end
end

If I want to build an email header using that custom method and a few standard attributes, trying this:

subject = "New Application from %{full_name} at %{city}, %{state}" % @applicant

will cough up the error:

ArgumentError: one hash required

Ok, so we need a hash out of our AR object. How about:

subject = "New Application from %{full_name} at %{city}, %{state}" % @applicant.as_json

gives us this:

KeyError: key{full_name} not found

Right! We have to add our custom method manually:

subject = "New Application from %{full_name} at %{city}, %{state}" % @applicant.as_json(methods: :full_name)

Crap! Still failing:

KeyError: key{full_name} not found

Turns out the hash needs to have keys that are symbols. No fear, symbolize_keys to the rescue:

subject = "New Application from %{full_name} at %{city}, %{state}" % @applicant.as_json(methods: :full_name).symbolize_keys

And voila:

"New Application from Nate Beaty at Chicago, IL"

For what seems like such a simple task, this has tripped me up for quite some time, and it’s been difficult to Google exactly what was needed to make this happen.

Getting Image Dimensions For Sublime

Sat, 21 Feb 2015 00:00:00 +0000

Sublime Text doesn’t have any way to get dimensions for an image in the sidebar yet, though a sidebar API may be coming soon. ¹

I’ve found myself writing this simple script a few times on various machines, so I thought I’d jot it down:

#!/bin/bash

FILENAME=`basename $1`
DIMS=`/usr/local/bin/identify -format "width: %wpx;\nheight: %hpx;" $1`
printf "background: url($FILENAME) no-repeat;\n$DIMS\n" | pbcopy

This uses a Homebrew installed Imagemagick’s identify command to toss CSS-ready width/height into your clipboard. I then set up a custom Tool entry inside the-dusty-albeit-still-my-fav-dual-pane-file-manager ForkLift:

And if you want, you can set a keyboard shortcut also. (Note that you have to close prefs after adding your tool, and then cmd-, again to have it show up in the Keyboard Shortcuts pane—one of several minor longstanding bugs I try to ignore.)

Now you can select an image in ForkLift, hit the key command you specified, and then cmd-shift-v will paste it into Sublime Text respecting indentation.

Of course you may need to fix the relative path of the image filename. Still somewhat useful until ST3 manages to support this type of functionality from the sidebar.

This is something I miss from TextMate, where you can just drag an image into your code to do this, even in the ancient 1.x releases: (Granted, it’s getting the relative image path wrong here anyway! You just can’t win.) ↩︎

Daily Git Log For Your Timesheet

Wed, 18 Feb 2015 00:00:00 +0000

Update: I made this into a bash script that has sensible defaults (gitlog for today's log) and a few tricks (e.g., gitlog -1 to yank yesterday's log into the clipboard).

This is probably the most satisfying command-line lifehack of the year for me:

git log --author=Nate --since=1.day.ago --all-match --format='%s' | tail -r | paste -s -d : - | sed -e 's/:/; /g' | pbcopy

This takes all your commit messages for the last day, concatenates them with a semicolon, and shoves them into your clipboard to paste into Harvest (or On The Job if I’m working freelance).

Breakdown:

git log options seem pretty obvious except --format='%s' which outputs just the commit message on each line
tail -r reverses order of lines
paste -s -d : - concatenates & separates each line with a colon
sed -e 's/:/; /g' replaces the colon with a semicolon and a space (couldn’t figure out a way to do this in one command)
pbcopy shoves the result into your clipboard on OS X

I’m guessing there’s a slightly prettier way to do this, but there’s a reason I keep trying to get everyone to start calling me Hacksaw because I’ve always really wanted a nickname and then I could get a tattoo.

You can also do a a custom span of time e.g. --since=3.days.ago --until=1.day.ago if you want more than just the last day of commits.

If you are in the thick of push battles with your coworker on a project and want to avoid the “Merge branch” lines from git, you can just add sed '/Merge branch/d' to the mix:

git log --author=Nate --since=1.day.ago --all-match --format='%s' | sed '/Merge branch/d' | tail -r | paste -s -d : - | sed -e 's/:/; /g' | pbcopy

Yosemite Pip Install Fix

Sat, 14 Feb 2015 00:00:00 +0000

I was getting compile errors when pip installing anything requiring compilation after upgrading to Yosemite, barfing out all sorts of cryptic nonsense ending with:

error: command '/usr/local/bin/gcc-4.2' failed with exit status 1

Googling that error didn’t help, and I tried all sorts of fixes until I realized the real problem was this line in my ~/.bash_profile:

export CC=/usr/local/bin/gcc-4.2

When I removed that, opened a new terminal, reactivated my virtualenv and pip install -r requirements.txt everything worked perfectly. Hopefully this helps someone else who’s been having issues after upgrading to Yosemite and have some lingering bash remnants from earlier dev setups.

It's a Prepending Party

Thu, 10 Jul 2014 00:00:00 +0000

I just realized a very powerful and simple feature of CodeKit that has made my life immensely easier, as well as giving a speed boost to my sites.

You can prepend all the libraries you’re using on a project at the top of your main javascript file like so:

// @codekit-prepend "mootools-core-1.5.0.js"
// @codekit-prepend "mootools-more-1.5.0.js"
// @codekit-prepend "shadowbox.js"
// @codekit-prepend "chosen.js"
// @codekit-prepend "slider.js"

And then set the output to be Minified + source map (for easier debugging):

And then in your footer you can get all your behavior with one HTTP request:

<script src="https://blog.clixel.com/js/min/application-ck.js" type="text/javascript"></script>

Stupid simple, but for some reason it just hit me that this was possible, and now I use it all the time. I love finding new ways to improve my workflow with CodeKit.

(Underwear drawer notes: yes, I still use MooTools on several of my older sites, as well as scrappy ol’ PHP. I have been wooed to jQuery and sexier Ruby + Python methods of coding over the years, but it’s good to keep a wide variety of skills in your tool belt.)

Flat File Future

Sun, 12 Jan 2014 00:00:00 +0000

I’m not sure exactly where I entered the rabbit hole, but I saw a mention of Statamic somewhere, and one link led to another until I realized there is a myriad of very slick, flat-file-based frameworks to choose from, in just about any language you prefer.

Perhaps the idea of a barebones, well-orchestrated platform was such a relief because I’d been spending my entire weekend hand-coding PHP to send multi-part MIME order confirmation emails with crusty table markup, in the thick of aging procedural code. From “GOOD GOD WHAT AM I DOING WITH MY LIFE” to “Oh cool, Statamic looks nice.”

Statamic just needs PHP 5.3. Pretty slim requirements¹. It uses YAML for all config files, and creating plugins looks damn simple, enough to make PHP appear not-too-shabby as a language.

One of the earliest to start this trend is Jekyll, which has been around for some time, looking just gorgeous in its Ruby minimalism suit of adult soothing dark techy hues. Installing it is ridiculously simple:

gem install jekyll
jekyll create hotsauce
cd hotsauce && jekyll serve

Oo-lala! Like Python? I do. How about Cactus?

sudo easy_install cactus
cactus create holysmokes
cd holysmokes && cactus serve

Looks familiar! Jekyll comes a little more pre-styled, but they’re both very similar concepts. And along with Statamic, they all use a variant of the Django templating language (Cactus uses the real deal), which is fantastic.

But every time I get excited about using one of these handsomely simple frameworks, like Ghost, I remember that most sites I’ve worked on just aren’t minimal enough to be served entirely by these solutions. As soon as you need products or some custom data type the client can manage, or a page that mixes & matches data types (just about every site we do at Firebelly to my chagrin), I’d be fighting against their opinionated limitations.

All said, the next time I decide to redo my portfolio site, or have a simple site to build, I’m definitely going to give one of these a try. Being able to just create a new folder to extend heirarchy, work with markup entirely in inheriting templates, and version control the entire stack sounds damn refreshing.

For now, back to my PHP spaghetti.

Though one thing about Statamic: it’s not free. I’ve noticed a trend emerging where folks are actually -gasp- charging for their work, and as a fellow developer I think this is a fine direction to go in. ↩︎

Hiding Settings Tab in Refinery 2

Tue, 07 Jan 2014 00:00:00 +0000

Quick note of something that took me a while to find: if you install the Refinery Blog engine but don’t need the Settings tab showing in the admin, you can easily hide it by adding config/initializers/refinery/settings.rb with:

Refinery::Settings.configure do |config|
  config.enable_interface = false
end

(This is for refinerycms / refinerycms-blog v2.1.0)

Where I Live, 2013 Edition

Sat, 05 Oct 2013 00:00:00 +0000

Sublime Text (now at v3 beta) is where I’ve spent a good portion of my life over the last 2-½ years. It’s a great cave as far as code caves go.
Sadly Photoshop CC still sits in my toolbelt, Acorn is getting so damn good (same with Pixelmator) but it’s still not quite enough to supplant this required, ever-worsening POS. I pine for the day indie apps kick the crap out of the CC suite.
Tweetbot because I spend entirely too much time blathering in 140-char chunks.
GitX (L) because it’s a perfect git GUI. And free. I’ve tried so many and keep coming back to GitX (L).
On The Job for time tracking & invoicing. It’s not perfect but it’s the closest I could find to my simple needs. For many years I used Billable and only switched because I didn’t like the successor, Profit Train.
Chromium because I trust Google less every day. It’s always a pain finding a download for this. I can’t even find one now.
ForkLift 2 is still in my belt, because there’s nothing quite like it. I still use Transmit for DockSend. ForkLift has sadly been left to pasture by BinaryNights while they focus on Locko, a 1Password clone.
The Hit List is still the best damn to-do app. I don’t opt to pay for the syncing & iOS app. Reminders is my on-the-go queue for tasks that I dump into Hit List when back at the desk.
Sequel Pro for when you want to navigate MySQL locally. It’s crazy this is free. I just realized you can donate and dropped $10 in the beer fund.
CodeKit was a random purchase to compile Less files, but has become an essential tool for Scss compiling, Javascript combining/minimizing, image optimizing, etc. Hilarious version update notes are a bonus.
iTerm2 became stable enough to replace Terminal.
Notational Velocity is the sharpest text-editing knife. Plays nice with Dropbox folder of plain text files.

My Instapaper Reader

Wed, 05 Oct 2011 00:00:00 +0000

A more appropriate branding for my Kindle.

Where I Live, 2011 Edition

Wed, 05 Oct 2011 00:00:00 +0000

TextMate has been supplanted by a new fling, Sublime Text 2, shown here with my own icon:

to replace the gray turd that comes supplied. Acorn 3 has finally come along far enough to replace Photoshop for most mockups and simple graphic tasks. Chrome’s slightly-enhanced version of WebKit’s Inspector is now good enough to replace the only reason to use Geezer Firefox: Firebug. TaskPaper usurped the forever-beta-expiration-updating Hit List, though there’s been murmurs of activity in the long-dormant lands of Potion Factory. I’ve been leaving Transmit open for the single feature ForkLift lacks: DockSend.

Dark gray version: