Ben Hepworth

Using command line tool jq to parse structured json

Ben Hepworth — Thu, 10 Oct 2019 20:31:10 +0000

Prerequisite: have brew installed. Visit this website to install brew – https://brew.sh/

Install jq:

brew install jq

Let’s say you have a file in tmp.json that contains the following (yes, I’m scripting against AWS if you can’t tell):

{
      "Credentials": {
          "SecretAccessKey": "abcdefghj1234567", 
          "SessionToken": "lkasjdhlfkjhalskdjfhlkajsdhlfkj", 
          "Expiration": "2019-10-12T06:50:05Z", 
          "AccessKeyId": "AHSDJF1234DSHFG"
      }
  }

To pull out the value of SecretAccessKey in the Credentials object, run the following command:

$ cat tmp.json| jq -r .Credentials.SecretAccessKey
$ abcdefghj1234567

Wow, that is so easy and powerful! Previously I was using a combination of grep and sed to parse it out of there. Not very efficient, super cryptic, and doesn’t scale well for nested objects. See the comparison:

cat tmp.json | grep SecretAccessKey|sed 's/",.*$//'|sed 's/^.*"//'
cat tmp.json| jq -r .Credentials.SecretAccessKey

You judge for yourself which one is easier. The first one looks like a 2 year old got a hold of a keyboard.

Arrays are indexed starting at zero. For example, let’s say you have an array in tmp2.json called Items that has multiple items:

{
  "Items": [
    { "name": "Ben",
      "lastName": "Hepworth"
    },
    { "name": "Dingus",
      "lastName": "Hepworth"
    }
  ]
}

To get the first item in the array, run the following command:

$ cat tmp2.json | jq -r .Items[0]
 {
   "name": "Ben",
   "lastName": "Hepworth"
 }

As you can see, it just returned Ben Hepworth and not Dingus Hepworth. You can imagine what else you can do with this once you know how to properly reference items in the json. You could easily write a for loop to loop through all items in the array and do something with them. grep, sed and awk, I still love you. Like my aging grandparents.

Fixing or installing Subversion SVN after upgrading to Mac OS Sierra

Ben Hepworth — Mon, 26 Sep 2016 17:01:48 +0000

I upgraded to Mac OS Sierra the other day and my command line subversion broke. It sure seems like every time there is an update to the Mac OS, I spend a few days working through little kinks with the various programs that I use until everything stabilizes. After updating to Mac OS Sierra recently, I tried to run a standard svn up command and got the following error message:

$ svn up xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun

The solution is pretty simple and also applies if you want to install command line SVN. It appears that the svn command line tools were uninstalled when I upgraded from Mac OS El Capitan to Mac OS Sierra. To install or reinstall the svn command line tools, run the following command:
xcode-select --install

After running this command, you’ll get a popup window prompting you to confirm installation of the XCode command line tools. You don’t have to install the entire 4GB+ installation of XCode in order to get command line tools like svn.

One more thing – when I was done installing, I tried to run a svn up and got the following error message:

tapir:$ svn up svn: E155036: Please see the 'svn upgrade' command svn: E155036: The working copy at '/Users/bhepworth/svn/trunk' is too old (format 29) to work with client version '1.9.4 (r1740329)' (expects format 31). You need to upgrade the working copy first.

I went into the root of the trunk that I had checked out and ran the command:

svn upgrade

After that, I was able to update from svn:

tapir:$ svn upgrade Upgraded '.' tapir:$ svn up Updating '.': U share/scripts/deploy/deployBuild.sh Updated to revision 1372.

I hope this helps you out! Every time Apple upgrades their OS, there are several kinks to work though. Luckily this one was a fairly easy fix.

Here is a quick youtube video I made that shows this fix in action:

~Ben

Turning Off Apple Live Photos

Ben Hepworth — Tue, 16 Aug 2016 02:15:57 +0000

Why, oh why, would Apple introduce Live Photos with iOS 9? In my opinion it is one of the worst decisions ever made by Apple. Phil, come on. You are definitely smarter than that. Why would you want a small video clip with every single photo you take that includes audio? Before you read any further, please turn off Live Photos on your iPhone. Launching the camera app and click the yellow circle that is between the HDR and Timer button.

I made a quick video to show how to do this as well:

I discovered this by accident when I had posted a picture to Facebook. It had this yellow circle on it and said something like “Live Photo”.

I was like, “um, what’s that?”

I did a little Googling and found out that for some reason Apple thought it would be a good idea to turn still photos into “unforgettable living memories” that include what happened right before and right after the photo, INCLUDING AUDIO. I held down my finger on the picture, and sure enough, it was a mini video with audio.

Now there are tons of people who think they are sharing a photo with someone on Facebook, text, or whatever social media platform you prefer, when they are actually sending a small video – let me repeat – THAT INCLUDES AUDIO.

WTF Apple!

Talk about a privacy issue. Not to mention the fact that Apple turned it on by default for everyone when they updated to iOS 9. This should be an “opt-in” feature, not an “opt-out” feature. Yes, if you read about all of the features ahead of time, you could have know about this and been on the lookout for it – but who does that? I think I vaguely remember them announcing it, but it was one of those “um, ok – what’s the use case for that” kinda moments during the Apple event of Sep 2015.

Do you agree? The only valid use case I’ve found is one that my son showed me where you can take a live photo and put that as your lock screen. Then your lockscreen comes to life everytime you unlock it. Big fat whoop-dee-doo! I’ve waited 9 iOS releases for this feature – woohoo! Seriously though, what if you were having some sort of private conversation while snapping a quick photo and then sent it to a friend, not realizing that your audio was captured too? This one is almost as bad as the “Siri raise to speak” that was introduced in iOS 8…but that’s another topic for another blog post.

Peace out – and turn off your Live Photos for crying out loud!

The Best Way to Delete Old WordPress Revisions

Ben Hepworth — Tue, 28 Jun 2016 00:45:03 +0000

By default, WordPress stores every modification to a post in a separate revision. The idea is that you can roll back to a previous revision. The problem with that, though, is that all of these old revisions can clog up your database and slow it down significantly. For my wife’s blog, her WordPress Admin was running very slow. She had tens of thousands of old revisions. I did a bunch of googling and there are several different ways posted on the internet on how to do this. There is also a WordPress plugin to do this, but I just wanted to use the build-in way of deleting revisions without having to run some database hack or load a plugin.

The problem with removing old posts is that the metadata is stored in multiple tables across the WordPress database. In order to ensure that all necessary rows are removed from all tables, you can call the wp_delete_post WordPress API. It is a native method in WordPress that handles the cleanup of all the necessary data everywhere in the database.

In pseudocode, this is what you do:

Load WordPress Config so you have access to the wp_delete_post method
Query the database and get the list of old revisions
Loop through the list of revisions and call wp_delete_post for each one

Here is the code for the script:
require_once('WP_ROOT/wp-load.php');


        $counter=0;
        $post_revisions = $wpdb->get_results(

                "SELECT `ID` AS revision_id

                ,`POST_TITLE` AS post_title

                FROM ($wpdb->posts)

                WHERE `post_type` = 'revision'

                ORDER BY `ID` DESC"

        );

foreach($post_revisions as $post) { echo "{$post->revision_id} : {$post->post_title}\n"; wp_delete_post($post->revision_id,true); $counter++; } echo "Deleted $counter revisions\n"; ?>

The only thing you should have to modify in this script is <> and replace it with the full path to the root directory of your wordpress install. If you’re not sure where this is, SSH into your server and run the command:
find . -name "wp-load.php"

This will tell you where the wp-load.php file is located and will give you access to the internal WordPress APIs.

Save the script to a file called deleteOldRevisions.php. Then, to execute it, run the following command from within the same directory where the script is located:

php -f deleteOldRevisions.php

Now, let’s break down what the script is doing:
require_once('WP_ROOT/wp-load.php');
This line loads the wp libraries so that you have access to the wp_delete_post method. We then set a counter variable so we can count how many revisions we end up deleting:
$counter=0;
We then query the database for all posts that are of type revision. We return the ID to pass into the wp_delete_post method and the post_title for a nice output to the screen when running.
$post_revisions = $wpdb->get_results( "SELECT `ID` AS revision_id ,`POST_TITLE` AS post_title FROM ($wpdb->posts) WHERE `post_type` = 'revision' ORDER BY `ID` DESC" );
This returns an array that is now stored in the $post_revisions variable. We then loop through the $post_revisions variable and call wp_delete_post against each one. The echo we print before calling the method so that we can see which post revision it is deleting. The counter at the end increments the counter for how many revisions we are deleting.
foreach($post_revisions as $post) { echo "{$post->revision_id} : {$post->post_title}\n"; wp_delete_post($post->revision_id,true); $counter++; }
Then, last but not least, we print how many revisions were deleted by printing the $counter variable:
echo "Deleted $counter revisions\n";

This method of removing old posts is the safest because you are getting down at the core functions that are built and maintained by the WordPress team. Once I ran this on my wife’s server, her WordPress Admin sped up significantly. Before it would sometimes take 30 seconds to a minute just to save a post. Now it is almost instantaneous. Good luck!

Tech Tip: Mute Messages on Mac – Do Not Disturb Mode

Ben Hepworth — Thu, 16 Jun 2016 05:04:19 +0000

Have you ever received a text message to your Mac while you’re in a meeting? Or maybe you’re showing someone something at work and then a text message pops up. Or maybe you’re on a screen share using Join.Me or WebEx and you don’t want any notifications. The notification center is useful, but at the same time can be very annoying if you don’t want any notifications popping up – mail, messages, etc. Here is how you turn it off and go into Do Not Disturb Mode.

Step 1: click on the 3 lines in the upper right corner of your Mac:

The notification center will slide in from the right.

Step 2: Scroll up while inside the notification bar – this is the trick. By default, the Do Not Disturb button is hidden at the top. Once you know where this is, it is quite easy to turn on/off.

By default when you turn notifications off, you’ll see a message indicating that they are off for the day and will automatically turn back on tomorrow. Click on any other window, the Desktop to get the notification bar to disappear.

Here is a quick YouTube video I made showing how easy this is.

Install python to separate directory on linux in 5 easy steps

Ben Hepworth — Wed, 18 May 2016 22:37:40 +0000

In some cases you may not want to touch the system-wide python installation on a linux box. I recently ran into this where we had a need to install python in it’s own directory. To install it in a separate directory, it is quite simple. Several years ago I wrote about how to install perl in it’s own directory in this post. Surprisingly enough, the first page of Google did not point me in the direction that I wanted. The use cases for this are:

You are developing custom software and need a newer version of python than your server has. You don’t want to mess with the system-wide python that other programs and scripts may depend on.
You want to install some custom python modules and don’t want to touch the system-wide python. I know you can create your own virtaulenv using pip, but what if you need newer functionality or a module is not compatible with the version of python that you have?

The install for this is quite simple. The key is to use the –prefix= option in your configure command. DONE.

So, for example, my home directory is /home/bhepworth. Let’s say I want to install the latest python, which at the time of writing this is 3.5.1. My python install choice is /home/bhepworth/python3. You can technically install it to any directory you prefer, I’m just using /home/bhepworth as the example. Here are the commands, starting from in my home directory of /home/bhepworth:

Step 1: Download Python
wget https://www.python.org/ftp/python/3.5.1/Python-3.5.1.tgz

Step 2: Uncompress the binaries
tar -zxvf Python-3.5.1.tgz

Step 3: Go into the directory with the Python source code
cd Python-3.5.1

Step 4: Configure (change /home/bhepworth/python3 to whatever directory you want to install python to)
./configure --prefix=/home/bhepworth/python3

Step 5: Compile and install
make
make test (Optional)
make install

Verification
Run python with the -V flag and you will see the version:
$ /home/bhepworth/python3/bin/python3 -V Python 3.5.1

In my case, the system-wide python is in /usr/bin/python and is on still on the older version, which I can verify by running the same command and using the system-wide python:
$ which python /usr/bin/python $ python -V Python 2.4.3

IMPORTANT: Referencing your new and shiny python in your scripts.

Make sure at the top of any python scripts you write, that you reference your new python and not /usr/bin/python – the first line should look like this:

#!/home/bhepworth/python3/bin/python3

Good luck, and happy python-ing!

Tivo Hacks: Episode 2 – Unboxing Tivo Roamio Multi-Room Setup

Ben Hepworth — Sat, 30 Apr 2016 02:47:09 +0000

This is part 2 in my series of setting up my own multi-room DVR setup using a real for real TiVo. You can view episode 1 here. This episode contains a video of the unboxing of my TiVo Roamio. This kit contains all of the equipment necessary to setup a multi-room TiVo setup in my house – a four room setup with streaming outside of the house to be exact. Have fun watching the unboxing video! Future episodes will contain tutorials for how to set this up yourself.

Install Warcraft 3 Natively on Modern Mac

Ben Hepworth — Tue, 05 Apr 2016 04:24:05 +0000

Ever since OS X Mavericks you have not been able to install Warcraft 3 on a mac…until now. Blizzard released patch 1.27a in March of 2016. The installer now supports modern day Mac computers and runs the game natively on your Mac. The video I made shows the download and install being done on OS X El Capitan in April of 2016. Previously he only way to install Warcraft III on a Mac was to use an open source windows emulater called wineskin, which was a bit of a pain to setup. The game ran ok on it, but not like running natively. Warcraft 3 is now running natively and extremely smooth on my Mac – woohoo! Now I can introduce my son to old school RTS gaming. Can you believe Warcraft 3 is now 14 years old?

Annoying iPhone Group Text? Here’s how to silence it

Ben Hepworth — Sat, 06 Feb 2016 19:46:30 +0000

Have you ever been in a meeting and your phone is on silent when a group text comes in out of nowhere? Then people start replying left and right, creating an embarrassing buzzing during the meeting? Even worse, you have an iPad next to you as well and it is also buzzing with every new message that comes in since it is hooked up to your iCloud account with Messages. Here is a quick tip on how you can silence that group text so your phone and/or iPad will stop buzzing. You will still receive all of the messages and can catch up on the thread later, you just won’t disturb your clients or the people that you are in a meeting with a buzzing iPhone and/or iPad.

Step 1: Open Messages (iPhone and/or iPad)
Step 2: Open the group text
Step 3: Click Details in the upper right
Step 4: Scroll to the bottom
Step 5: Select “Do Not Disturb”

Here are a few screenshots to show how to do this:

~~~~~~~~~~~~~~~~

~~~~~~~~~~~~~~~~

Resolving issues during LEMP server installation

Ben Hepworth — Tue, 15 Dec 2015 00:01:10 +0000

I recently moved my wife’s blog from the terrible hosting of HostGator to my own VPS at XMission and setup a LEMP server (Linux, nginx, Mariadb, PHP) on CentOS 7. During the process I ran into a handful of issues that I had to work through.

First off, I used this website as a rough tutorial for setting up a LEMP server. The tutorial was written for someone on a debian server, though, and I am on CentOS 7. I also used this server for some of the CentOS 7 differences. I still ran into several issues during the process. Here are the issues I encountered and how I resolved them.

Table of Contents for this post (click on links to go to the appropriate section):
Issue #1: No feedback from systemctl
Issue #2: Permission denied error when starting nginx
Issue #3: nginx blank PHP pages
Issue #4: WordPress error – 502 Bad Gateway
Issue #5: Some Blog Posts were displaying blanks

Issue #1: No feedback from systemctl

The first thing I noticed about CentOS 7 is that the commands for starting/stopping are different than what I’m used to on redhat systems. /etc/init.d or service refers you over to systcmctl. For example, to start nginx:
systemctl start nginx
I’m used to seeing feedback with [ OK ] or something like that when stopping or starting services. On CentOS 7 it gives you nothing. After some googling, apparently this is by desing, and it only gives you an error when there are issues with the systemctl command. For example:
# systemctl restart nginx Job for nginx.service failed. See 'systemctl status nginx.service' and 'journalctl -xn' for details.
You can then run the command systemctl status nginx.service to get the details about what went wrong during the start. This one really wasn’t technically an issue with the install, but was something that I learned while using CentOS 7 for the first time.

Issue #2: Permission denied error when starting nginx

After setting up the virtualhost in nginx, I was getting the following error when starting nginx:
nginx: [emerg] open() ".../logs/access.log" failed (13: Permission denied) nginx: configuration file /etc/nginx/nginx.conf test failed nginx.service: control process exited, code=exited status=1 Failed to start The nginx HTTP and reverse proxy server. Unit nginx.service entered failed state.
Running “nginx -t” did not return any errors. I verified that the entire directory tree up to logs was 755 and that the user I was running nginx as has permission to it all. After a lot of googling, the issue ended up being with CentOS 7 SELinux. It was set to Enforcing. Changing it to Permissive resolved the issue. This website helped resolve the issue for me. First, I ran the command getenforce, which returned Enforcing:
# getenforce Enforcing
Then I set it to permissive:
# setenforce 0
After that, it returned Permissive:
# getenforce Permissive
Then a restart of nginx and the error went away:
# systemctl start nginx

Issue #3: nginx blank PHP pages

Once I got the virtualhost issue with SELinux resolved, I could get regular HTMl pages to render just fine, but any PHP page would not work. nginx doesn’t actually do the php processing. It hands off any PHP pages to PHP-FPM for the processing. PHP-FPM runs as a separate process on the server. The tutorial for the debian setup uses the unix sock setup, whereas CentOS 7 uses regular process listening on a port. At the end of the day, the following two configs needed to match to get PHP to render:
In /etc/php-fpm.d/www.conf :
listen = 127.0.0.1:9000
The fastcgi_pass line in the location section of the virtualhost configuration for nginx:
location ~ \.php$ { try_files $uri =404; fastcgi_split_path_info ^(.+\.php)(/.+)$; fastcgi_pass 127.0.0.1:9000; fastcgi_index index.php; include fastcgi.conf; }

The debian instructions had the line like this:
fastcgi_pass unix:/var/run/php5-fpm.sock;

With the sock location, it didn’t match the php-fpm config. Once I updated it to point to 127.0.0.1:9000 then nginx was able to properly pass the PHP requests to the php-fpm process. Totally makes sense now, but I was sure hitting my head against a brick wall until I figured this out.

Issue #4: WordPress error – 502 Bad Gateway

I migrated my wordpress installation over by to the new server and was receiving a 502 Bad Gateway error when attempting to hit the site. After much Googling, apparently nginx does not like PHP shortcuts, whereas Apache is more forgiving. For example, to properly do a php block of code, you are supposed to start it like this:
...your php code ?>
If you are lazy and using Apache, you can use a shortcut for your PHP like this:
...your php code ?>
I was seeing the following errors in the nginx virtualhost error log:
[error] *15 FastCGI sent in stderr: "PHP message: PHP Parse error: syntax error, unexpected 'else' (T_ELSE) in wp-content/plugins/wp-gravatar/gravatars.php on line 752" while reading response header from upstream
I opened up the referenced file (gravatars.php) and found two spots in it that had the php shortcut. I just added php after the open &lt? and saved the file. In googling there were other people referencing this as well. I will need to keep this in mind after future wordpress updates. If there are any other PHP shortcuts, they’ll need to be edited like this in order for nginx to work with it. I found this solution at here.

Issue #5: Some Blog Posts were displaying blanks

PHP message: PHP Warning: Unknown: Failed to write session data (files). Please verify that the current setting of session.save_path is correct (/var/lib/php/session)

; Set session path to a directory owned by process user php_value[session.save_handler] = files php_value[session.save_path] = /var/lib/php/session

This directory was owned by root:apache.
# ls -l /var/lib/php total 4 drwxrwx---. 2 root apache 4096 Jun 23 17:20 session
I changed ownership on the session directory to the user and group that nginx is running as.
# chown user:group /var/lib/php/session

After doing that, the session errors went away in my virtual host error.log, but there were still proxy errors:

[crit] 1295#0: *1369 open() "/var/lib/nginx/tmp/fastcgi/1/17/0000000171" failed (13: Permission denied) while reading upstream, client: 62.210.101.170, server: houseofhepworths.com, request: "GET /2012/01/10/mounting-your-tv-to-the-wall-and-hiding-all-the-cords/ HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.houseofhepworths.com"
In the virtualhost error.log, I was seeing a lot of these errors. When I punched in the URL from any of the GET calls, it was blank. Upon further analysis, it looked like it was trying to access /var/lib/nginx/tmp. When I first setup nginx, it was set to run as the ngnix id and group. I updated it to the user and group that I am running nginx as now since it didn’t have permission to that directory:
# cd /var/lib # chown -R user:group nginx
As soon as I made the nginx user and group own /var/lib/nginx recursively, the errors went away in the virtualhost error log and the blog posts loaded.

After working through all of these issues, I have a functioning LEMP server. I still need to do some other things like setup caching, but everything looks god for now. Hopefully if you have seen any of these errors, this post will help you so you don’t have to spend hours googling and digging through posts to find the answers.