There are two things I want to be able to do with Ubuntu Server VMs: establish an SSH connection as root and navigate the file system using applications such as Forklift and FileZilla. Purists and more security-minded individuals may balk at this, but these VMs are local, non-production servers purely for testing.

NOTE: If you have both VMware Fusion and Multipass installed on the same machine, take a look at Possible Conflict between VMware Fusion and Multipass at the foot of this article.

Throughout this article the following terms are used:

Glossary of terms

To demonstrate, I’ll be using macOS as the host OS and a Multipass guest instance named foo with an IP address of 172.16.170.4 running Ubuntu 20.04 LTS. A standard user with sudo privileges named ubuntu is created by default on every new guest instance:

multipass list

Name                State             IPv4             Image
foo                 Running           172.16.170.4     Ubuntu 20.04 LTS

There are two built-in commands to connect with a guest instance. The first is the exec command which simply executes a command on the running guest instance. If it isn’t running, the exec command fails:

multipass exec foo -- whoami

ubuntu

exec failed: instance "foo" is not running

The second is shell or sh which opens an interactive login shell on the running guest instance. If the guest instance isn’t running, an attempt is made to first start it:

multipass shell foo

Neither of these methods can be used to configure connections in applications like Forklift or FileZilla. These and other file transfer applications typically use SFTP to transfer files to and from the client and server. Using SFTP, an encrypted SSH connection is first established by the client to the server. Files are then transferred using FTP over the SSH connection.

When establishing an SSH connection, we need to be mindful of the authentication method used by the server. Typically this is public key authentication and Multipass guest instances are no exception. Public key authentication requires both a public and private key. Multipass generates this key pair using the RSA algorithm with a 2048-bit key length and uses the same key pair for both the ubuntu and root users.

The private key is stored in an identity file on the host. For macOS, this identity file is /var/root/Library/Application Support/multipassd/ssh-keys/id_rsa¹ and is created when Multipass is installed. The corresponding public key is included in the vendor-data that Multipass gives to cloud-init when the guest instance is initialised and is written to /home/ubuntu/.ssh/authorized_keys and /root/.ssh/authorized_keys on the guest instance.

¹ On a Linux host with Multipass installed via Snap the identity file is /var/snap/multipass/common/data/multipassd/ssh-keys/id_rsa.

Because of its location on the host, superuser privileges are required to read the identity file. Using sudo on the command line is not an issue although by doing so the guest instance’s public host keys (ed25519, rsa and ecdsa) are added to /var/root/.ssh/known_hosts on the host not ~/.ssh/known_hosts. This is important to remember if attempting to remove public host keys for a given instance from the known_hosts file:

sudo ssh-keygen -R 172.16.170.4 -f /var/root/.ssh/known_hosts

However, the requirement for using sudo is a problem when configuring SFTP connections in Forklift or FileZilla as it’s not possible to elevate an admin user’s privileges to those of the superuser. One way to overcome this is to copy the identity file to a new location accessible to the admin user:

mkdir -p ~/.ssh/multipass && sudo cp /var/root/Library/Application\ Support/multipassd/ssh-keys/id_rsa $_

This first creates a sub-directory named multipass in the admin user’s .ssh directory, then copies the identity file to it. The $_ at the end of the cp command is a special parameter which expands to the last argument – ~/.ssh/multipass – passed to the previous command – mkdir.

The duplicate file has the same root:wheel ownership as the original. This needs to be changed to $USER:staff:

sudo chown $USER:staff ~/.ssh/multipass/id_rsa

In addition, the duplicate identity file’s permissions are read-only (400), the same as the original. When using FileZilla, you may need to change the format of the private key. See FileZilla can’t Read the Private Key at the end of this article. To do so, the permissions must first be changed to read-write (600):

chmod 600 ~/.ssh/multipass/id_rsa

Now the private key is in the correct location, let’s try to establish an SSH connection to the guest instance named foo using the standard user ubuntu. The location of the identity file is specified using the -i option. The instance must first be running or the connection will timeout:

ssh -i ~/.ssh/multipass/id_rsa ubuntu@172.16.170.4

The credentials used to successfully establish an SSH connection to the guest instance can now be used to configure an SFTP connection in Forklift or FileZilla by specifying the location of the identity file, but what if we want to connect as the guest instance’s root user.

It’s possible to open a non-login shell with root access on a running guest instance using the built-in exec command:

multipass exec foo -- sudo su

This is no different from executing sudo su having first logged-in using the standard user account ubuntu on the guest instance and as such doesn’t allow for the configuration of SFTP connections in file transfer applications.

Let’s see if it’s possible to establish an SSH connection to the guest instance using its root user. We use the same command syntax as before, but substitute root for ubuntu:

ssh -i ~/.ssh/multipass/id_rsa root@172.16.170.4

Please login as the user "ubuntu" rather than the user "root".

Connection to 172.16.170.4 closed.

It’s pretty clear from the output that this isn’t possible. However, let’s take a look at the /root/.ssh/authorized_keys file on the guest instance:

sudo cat /root/.ssh/authorized_keys

no-port-forwarding,no-agent-forwarding,no-X11-forwarding,command="echo 'Please login as the user \"ubuntu\" rather than the user \"root\".';echo;sleep 10;exit 142" ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz3PNy+AIaX4jTgvTw0VtrAxnH53kpOjiVnbjM27fMkXFUIZN1Fa16XynrVzg6pGaUrFYUJhK5OCXzZJ6DwcWQG1QYxI1TGytraJa3osTU72EZh71vsD+7EP6+M2MzxK7B7oQDh6GTt17f4dG5GPOHEdueG0qHhRM5A9WhiWvDCYUcFHm/eFfYAGQTEz103faMl7frl5Vq5VdJQBjVEVIKfnEtfP3vRGcuUfi+IRfzYS4L1dxX5blnj5Im39YMh31aBXxl/Fo4Atb6PKOAnoaR4wHBTDik+2q4vHhmfPDXPAcX3gUyWjQ9nyBLqpjptVF6rnPIMv228/WwwnVRB/ED ubuntu@localhost

As is evident from the output, the public key is prefixed with:

no-port-forwarding,no-agent-forwarding,no-X11-forwarding,command="echo 'Please login as the user \"ubuntu\" rather than the user \"root\".';echo;sleep 10;exit 142"

This string is the default value of the disable_root_opts configuration key used by cloud-init and the highlighted code is responsible for displaying an error message, waiting 10 seconds and then exiting, effectively denying root login.

To be able to establish an SSH connection as root, this code has to be removed either manually by opening the file with nano or similar editor or alternatively using sed:

;

sudo sed -i.$(date +"%Y%m%d-%H%M%S") 's/,command.*exit 142"//' /root/.ssh/authorized_keys

Alternatively, if you pass user-data to cloud-init to bootstrap Multipass guest instances you could include either the disable_root or the disable_root_opts configuration keys in your cloud-init configuration file to control the root user’s access.

The default value for disable_root is true which causes the value of disable_root_opts to be pre-pended to the public key in the authorized_keys file. We’ve already seen what this default value is, so the first method is to provide a new value for the disable_root_opts configuration key in your cloud-init configuration file, omitting the offending code:

#cloud-config
...
disable_root_opts: no-port-forwarding,no-agent-forwarding,no-X11-forwarding
...

In the second method, we change the disable_root configuration key to false which effectively ignores the the disable_root_opts configuration key and results in nothing being pre-pended to the public key:

#cloud-config
...
disable_root: false
...

Multipass guest instances can now be created using:

multipass launch --name foo --cloud-init config.yaml

It should now be possible to establish an SSH connection to the guest instance using its root user:

ssh -i ~/.ssh/multipass/id_rsa root@172.16.170.4

These same credentials can be used to configure SFTP connections to the guest instance using its root user in Forklift and FileZilla.

NOTES

Possible Conflict between VMware Fusion and Multipass

If both Multipass and VMware Fusion are installed on your machine it’s worth noting that by default VMware Fusion VMs are configured to share the IP address of the Mac on the external network under Settings > Network Adapter > Share with my Mac. Before starting a VMware Fusion VM ensure that no Multipass guest instances are running by executing multipass stop --all. If any Multipass guest instances are running, VMware Fusion displays the error Could not connect 'Ethernet0' to virtual network '/dev/vmnet8'. The VMware Fusion VM will start, but will be unable to connect to the Internet.

A workaround to having both Multipass guest instances and VMware Fusion VMs running at the same time is to reconfigure the VMware Fusion VM’s network adapter to use Wi-Fi with Settings > Network Adapter > Wi-Fi before starting the VMware Fusion VM.

However, starting a VMware Fusion VM while a Multipass guest instance is running will most likely cause issues with Multipass as well. If this occurs, first stop and then start the Multipass daemon – multipassd – before restarting the guest instance:

sudo launchctl unload /Library/LaunchDaemons/com.canonical.multipassd.plist && sudo launchctl load /Library/LaunchDaemons/com.canonical.multipassd.plist

FileZilla can’t Read the Private Key

FileZilla appears unable to read the private key in the identity file. It displays a Could not load key file error. The header of the private key is -----BEGIN PRIVATE KEY----- and the Base64-encoded text begins MII...IBADAN which identifies it as an unencrypted private key in Base64-encoded PKCS#8 format.

I couldn’t find any documentation to confirm if or explain why FileZilla doesn’t like this format, but FileZilla will accept the private key if it is first converted to the new OpenSSH format. This conversion appears to be as simple as setting a new empty passphrase – the original passphrase is also empty – using ssh-keygen:

ssh-keygen -p -N '' -f ~/.ssh/multipass/id_rsa

The private key remains unencrypted, but its header is now -----BEGIN OPENSSH PRIVATE KEY-----.

Using The Python Code Examples

All the Python code examples in this post were tested using Python 3.8.2 (macOS) and Python 3.6.9 (Ubuntu) and can be run using the Python interpreter.

The examples assume the file macos.json is located in your home directory: /User/username (macOS) and /home/username (Ubuntu).

If pasting code examples containing a for loop or with statement, press enter twice to execute the code and return to the >>> Python interpreter prompt.

The Structure of JSON-encoded Data

A more detailed description on the structure of JSON-encoded data can be found here, but for the purposes of this post:

• JSON-encoded data consists of arrays, objects, values and name/value pairs.
• An array is surrounded by square brackets [...] and contains a comma-separated list of values.
• An object is surrounded by braces {...} and contains a comma-separated list of name/value pairs.
• A name/value pair consists of a name in double quotes "...", followed by a colon :, followed by a value.
• A value can be a string, a number, a boolean, null, another array or another object.

Consider the following file containing JSON-encoded data on two versions of macOS:

{
    "updated": "2020-07-09",
    "versions": [
        {
            "family": "macOS",
            "version": "10.14",
            "codename": "Mojave",
            "announced": "2018-06-04",
            "released": "2018-09-24"
        },
        {
            "family": "macOS",
            "version": "10.15",
            "codename": "Catalina",
            "announced": "2019-06-03",
            "released": "2019-10-07"
        }
    ]
}

• The top-level object defines two name/value pairs.
• The first is updated and has a string value of 2020-07-09 representing the date the information in the file was last updated.
• The second is versions whose value is an array.
• Each value in the versions array is an object representing a single macOS version.
• Each object representing a single macOS version contains information on that version in the form of five name/value pairs: family, version, codename, announced and released. All values are strings.

Accessing JSON-encoded Data in Python

To allow Python to access the JSON-encoded data we first need to open the file and then read its contents into memory. The latter is known as decoding or deserializing and in Python is performed using the load() method from Python’s json library. As part of this deserializing process a JSON array is converted to a Python list [] and a JSON object is converted to a Python dictionary {}.

import os
import json
f = open(os.environ["HOME"] + "/macos.json", "r", encoding="utf-8")
data = json.load(f)
f.close()

The deserialized JSON-encoded data is now stored in the variable data. The output has been re-formatted for clarity:

print(data)

{   
	'updated': '2020-07-09',
    'versions': [
        {
            'family': 'macOS',
            'version': '10.14',
            'codename': 'Mojave',
            'announced': '2018-06-04',
            'released': '2018-09-24'
        },
        {   
            'family': 'macOS',
            'version': '10.15',
            'codename': 'Catalina',
            'announced': '2019-06-03',
            'released': '2019-10-07'
        }
    ]
}

Apart from double-quotes " being replaced with single-quotes ', this output looks identical to the JSON-encoded data in our file. Remember however, what constitutes an array in the JSON-encoded data is now a Python list [] and what are objects in that same data are now Python dictionaries {}.

To access a specific piece of data we can use bracket notation. For example, to get the date the information was last updated:

print(data["updated"])

2020-07-09

To get specific information on a particular macOS version is a little more involved:

• Information on both macOS versions is contained in the versions list.
• Each value of the versions list is a dictionary containing information on a single macOS version.
• We can access the information in a specific dictionary by using its position or index in the versions list. As Python lists are zero-indexed, the first dictionary has an index of 0, the second an index of 1.

NOTE: JSON Editor Online is a useful tool that displays JSON-encoded data showing the number of values in a list (JSON array) and each value’s index within that list. Please note you need to switch to tree view. Also, the code editor Visual Studio Code has an Outline view that displays JSON-encoded data in a similar way.

So, to get the date macOS Mojave was announced we need to target the first dictionary of the versions list:

print(data["versions"][0]["announced"])

2018-06-04

To get the dates both macOS Mojave and macOS Catalina were released we need to target the first dictionary of the versions list and then its second dictionary:

print(data["versions"][0]["released"]); \
print(data["versions"][1]["released"])

2018-09-24
2019-10-07

Let’s make things a little more interesting and add some more information for each macOS version to our JSON-encoded data:

{
    "updated": "2020-07-09",
    "versions": [
        {
            "family": "macOS",
            "version": "10.14",
            "codename": "Mojave",
            "announced":"2018-06-04",
            "released": "2018-09-24",
            "requirements": [
                "iMac (Late 2012 or newer)",
                "iMac Pro (2017)",
                "Mac Mini (Late 2012 or newer)",
                "Mac Pro (Late 2013; Mid 2010 and Mid 2012 models with recommended Metal-capable graphics cards)",
                "MacBook (Early 2015 or newer)",
                "MacBook Air (Mid 2012 or newer)",
                "MacBook Pro (Mid 2012 or newer)",
                "2 GB of memory",
                "12.5 - 18.5 GB of available avaialable disk space",
                "OS X 10.8 or later"
            ],
            "releases": [
                {
                    "version": "10.14",
                    "build": "18A391",
                    "darwin": "18.0.0",
                    "released": "2018-09-24"
                },
                {
                    "version": "10.14.1",
                    "build": "18B75",
                    "darwin": "18.2.0",
                    "released": "2018-10-30"
                },
                {
                    "version": "10.14.2",
                    "build": "18C54",
                    "darwin": "18.2.0",
                    "released": "2018-12-05"
                },
                {
                    "version": "10.14.3",
                    "build": "18D42",
                    "darwin": "18.2.0",
                    "released": "2019-01-22"
                },
                {
                    "version": "10.14.4",
                    "build": "18E226",
                    "darwin": "18.5.0",
                    "released": "2019-03-25"
                },
                {
                    "version": "10.14.5",
                    "build": "18F132",
                    "darwin": "18.6.0",
                    "released": "2019-05-13"
                },
                {
                    "version": "10.14.6",
                    "build": "18G84",
                    "darwin": "18.7.0",
                    "released": "2019-07-22"
                }
            ]
        },
        {
            "family": "macOS",
            "version": "10.15",
            "codename": "Catalina",
            "announced":"2019-06-03",
            "released": "2019-10-07",
            "requirements": [
                "iMac (Late 2012 or newer)",
                "iMac Pro (2017)",
                "Mac Mini (Late 2012 or newer)",
                "Mac Pro (Late 2013)",
                "MacBook (Early 2015 or newer)",
                "MacBook Air (Mid 2012 or newer)",
                "MacBook Pro (Mid 2012 or newer)",
                "4 GB of memory",
                "12.5 GB of available avaialable disk space",
                "OS X 10.11.5 or later"
            ],
            "releases": [
                {
                    "version": "10.15",
                    "build": "19A583",
                    "darwin": "19.0.0",
                    "released": "2019-10-07"
                },
                {
                    "version": "10.15.1",
                    "build": "19B88",
                    "darwin": "19.0.0",
                    "released": "2019-10-29"
                },
                {
                    "version": "10.15.2",
                    "build": "19C57",
                    "darwin": "19.2.0",
                    "released": "2019-12-10"
                },
                {
                    "version": "10.15.3",
                    "build": "19D76",
                    "darwin": "19.3.0",
                    "released": "2020-01-28"
                },
                {
                    "version": "10.15.4",
                    "build": "19E266",
                    "darwin": "19.4.0",
                    "released": "2020-03-24"
                },
                {
                    "version": "10.15.5",
                    "build": "19F96",
                    "darwin": "19.5.0",
                    "released": "2020-05-26"
                }
            ]
        }
    ]
}

In Python, the updated JSON-encoded data in the file needs to be deserialized again:

import os
import json
with open(os.environ["HOME"] + "/macos.json", "r", encoding="utf-8") as f:
    data = json.load(f)

Here we use a with statement with the open function. This ensures better exception handling and doesn’t require the close() function to be called explicitly as the with statement handles the proper acquisition and release of resources.

The updated deserialized data. The output has been re-formatted for clarity:

print(data)

{
    'updated': '2020-07-09',
    'versions': [
        {
            'family': 'macOS',
            'version': '10.14',
            'codename': 'Mojave',
            'announced': '2018-06-04',
            'released': '2018-09-24',
            'requirements': [
                'iMac (Late 2012 or newer)',
                'iMac Pro (2017)',
                'Mac Mini (Late 2012 or newer)',
                'Mac Pro (Late 2013; Mid 2010 and '
                'Mid 2012 models with recommended '
                'Metal-capable graphics cards)',
                'MacBook (Early 2015 or newer)',
                'MacBook Air (Mid 2012 or newer)',
                'MacBook Pro (Mid 2012 or newer)',
                '2 GB of memory',
                '12.5 - 18.5 GB of available '
                'avaialable disk space',
                'OS X 10.8 or later'
            ],
            'releases': [
                {
                    'version': '10.14',
                    'build': '18A391',
                    'darwin': '18.0.0',
                    'released': '2018-09-24'
                },
                {
                    'version': '10.14.1',
                    'build': '18B75',
                    'darwin': '18.2.0',
                    'released': '2018-10-30'
                },
                {
                    'version': '10.14.2',
                    'build': '18C54',
                    'darwin': '18.2.0',
                    'released': '2018-12-05'
                },
                {
                    'version': '10.14.3',
                    'build': '18D42',
                    'darwin': '18.2.0',
                    'released': '2019-01-22'
                },
                {
                    'version': '10.14.4',
                    'build': '18E226',
                    'darwin': '18.5.0',
                    'released': '2019-03-25'
                },
                {
                    'version': '10.14.5',
                    'build': '18F132',
                    'darwin': '18.6.0',
                    'released': '2019-05-13'
                },
                {
                    'version': '10.14.6',
                    'build': '18G84',
                    'darwin': '18.7.0',
                    'released': '2019-07-22'
                }
            ]
        },
        {
            'family': 'macOS',
            'version': '10.15',
            'codename': 'Catalina',
            'announced': '2019-06-03',
            'released': '2019-10-07',
            'requirements': [
                'iMac (Late 2012 or newer)',
                'iMac Pro (2017)',
                'Mac Mini (Late 2012 or newer)',
                'Mac Pro (Late 2013)',
                'MacBook (Early 2015 or newer)',
                'MacBook Air (Mid 2012 or newer)',
                'MacBook Pro (Mid 2012 or newer)',
                '4 GB of memory',
                '12.5 GB of available avaialable '
                'disk space',
                'OS X 10.11.5 or later'
            ],
            'releases': [
                {
                    'version': '10.15',
                    'build': '19A583',
                    'darwin': '19.0.0',
                    'released': '2019-10-07'
                },
                {
                    'version': '10.15.1',
                    'build': '19B88',
                    'darwin': '19.0.0',
                    'released': '2019-10-29'
                },
                {
                    'version': '10.15.2',
                    'build': '19C57',
                    'darwin': '19.2.0',
                    'released': '2019-12-10'
                },
                {
                    'version': '10.15.3',
                    'build': '19D76',
                    'darwin': '19.3.0',
                    'released': '2020-01-28'
                },
                {
                    'version': '10.15.4',
                    'build': '19E266',
                    'darwin': '19.4.0',
                    'released': '2020-03-24'
                },
                {
                    'version': '10.15.5',
                    'build': '19F96',
                    'darwin': '19.5.0',
                    'released': '2020-05-26'
                }
            ]
        }
    ]
}

• Each dictionary in the versions list now contains two additional name/value pairs: requirements and releases.
• requirements is a list whose values are all strings with each string representing a minimum system requirement to run that particular macOS version.
• releases is also a list, but each of its values is a dictionary representing information on each release of a single macOS version provided by four name/value pairs: version, build, darwin and released.

To get the first minimum system requirement to run macOS Mojave:

print(data["versions"][0]["requirements"][0])

iMac (Late 2012 or newer)

To get the first and second minimum system requirement to run macOS Catalina:

print(data["versions"][1]["requirements"][0]); \
print(data["versions"][1]["requirements"][1])

iMac (Late 2012 or newer)
iMac Pro (2017)

To get the build number of the first release of macOS Catalina:

print(data["versions"][1]["releases"][0]["build"])

19A583

To get the build number of the eighth release of macOS Mojave. Whoops! There have only been seven releases of macOS Mojave:

print(data["versions"][0]["releases"][7]["build"])

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: list index out of range

To get the build number and Darwin version of the first release of macOS Catalina:

print(data["versions"][1]["releases"][0]["build"] +" "+ data["versions"][1]["releases"][0]["darwin"])

19A583 19.0.0

To get the version, build number and Darwin version of the first, second and third release of macOS Mojave:

print(data["versions"][0]["releases"][0]["version"] +" "+ data["versions"][0]["releases"][0]["build"] +" "+ data["versions"][0]["releases"][0]["darwin"]); \
print(data["versions"][0]["releases"][1]["version"] +" "+ data["versions"][0]["releases"][1]["build"] +" "+ data["versions"][0]["releases"][1]["darwin"]); \
print(data["versions"][0]["releases"][2]["version"] +" "+ data["versions"][0]["releases"][2]["build"] +" "+ data["versions"][0]["releases"][2]["darwin"])

10.14 18A391 18.0.0
10.14.1 18B75 18.2.0
10.14.2 18C54 18.2.0

Looping Through JSON-encoded Data

As is evident from the examples above, accessing multiple values in lists is cumbersome and error prone. Each value – whether it be a string, a number, boolean, null or a dictionary – has to be explicitly targeted using its position or index in the list. Often, the total number of values in a list is unknown beforehand or the number of values in similarly named lists differ resulting in index out of range errors. For example, the releases list for macOS Mojave contains seven values (dictionaries), but for macOS Catalina contains only six.

To overcome this we can use a for statement to loop or iterate through every value in the list repeating the same steps during each loop or iteration.

For example, to get the codename of every macOS version without using a for loop:

print(data["versions"][0]["codename"]); \
print(data["versions"][1]["codename"])

Mojave
Catalina

Alternatively, to get the codename of every macOS version using a for loop:

for _version in data["versions"]: 
    print(_version["codename"])

Mojave
Catalina

The loop repeats for every value (dictionary) in the versions list. On each loop, the current list value is written to the variable _version. Subsequently, we use _version["codename"] to get the codename of that macOS version.

To get the codename of every macOS version together with their minimum system requirements without using a for loop:

print(data["versions"][0]["codename"]); \
print("  " + data["versions"][0]["requirements"][0]); \
print("  " + data["versions"][0]["requirements"][1]); \
print("  " + data["versions"][0]["requirements"][2]); \
print("  " + data["versions"][0]["requirements"][3]); \
print("  " + data["versions"][0]["requirements"][4]); \
print("  " + data["versions"][0]["requirements"][5]); \
print("  " + data["versions"][0]["requirements"][6]); \
print("  " + data["versions"][0]["requirements"][7]); \
print("  " + data["versions"][0]["requirements"][8]); \
print("  " + data["versions"][0]["requirements"][9]); \
print(data["versions"][1]["codename"]); \
print("  " + data["versions"][1]["requirements"][0]); \
print("  " + data["versions"][1]["requirements"][1]); \
print("  " + data["versions"][1]["requirements"][2]); \
print("  " + data["versions"][1]["requirements"][3]); \
print("  " + data["versions"][1]["requirements"][4]); \
print("  " + data["versions"][1]["requirements"][5]); \
print("  " + data["versions"][1]["requirements"][6]); \
print("  " + data["versions"][1]["requirements"][7]); \
print("  " + data["versions"][1]["requirements"][8]); \
print("  " + data["versions"][1]["requirements"][9])

Mojave
  iMac (Late 2012 or newer)
  iMac Pro (2017)
  Mac Mini (Late 2012 or newer)
  Mac Pro (Late 2013; Mid 2010 and Mid 2012 models with recommended Metal-capable graphics cards)
  MacBook (Early 2015 or newer)
  MacBook Air (Mid 2012 or newer)
  MacBook Pro (Mid 2012 or newer)
  2 GB of memory
  12.5 - 18.5 GB of available avaialable disk space
  OS X 10.8 or later
Catalina
  iMac (Late 2012 or newer)
  iMac Pro (2017)
  Mac Mini (Late 2012 or newer)
  Mac Pro (Late 2013)
  MacBook (Early 2015 or newer)
  MacBook Air (Mid 2012 or newer)
  MacBook Pro (Mid 2012 or newer)
  4 GB of memory
  12.5 GB of available avaialable disk space
  OS X 10.11.5 or later

Alternatively, to get the same information using a for loop:

for _version in data["versions"]: 
	print(_version["codename"])
	for _requirement in _version["requirements"]: 
		print("  " + _requirement)

Mojave
  iMac (Late 2012 or newer)
  iMac Pro (2017)
  Mac Mini (Late 2012 or newer)
  Mac Pro (Late 2013; Mid 2010 and Mid 2012 models with recommended Metal-capable graphics cards)
  MacBook (Early 2015 or newer)
  MacBook Air (Mid 2012 or newer)
  MacBook Pro (Mid 2012 or newer)
  2 GB of memory
  12.5 - 18.5 GB of available avaialable disk space
  OS X 10.8 or later
Catalina
  iMac (Late 2012 or newer)
  iMac Pro (2017)
  Mac Mini (Late 2012 or newer)
  Mac Pro (Late 2013)
  MacBook (Early 2015 or newer)
  MacBook Air (Mid 2012 or newer)
  MacBook Pro (Mid 2012 or newer)
  4 GB of memory
  12.5 GB of available avaialable disk space
  OS X 10.11.5 or later

In this instance the for loops are nested. The outer for loop is the same as in the previous example. The inner
for loop repeats for every value (string) in the requirements list. On each loop, the current list value is written to the variable _requirement. Subsequently, to get each requirement we simply use _requirement.

Because the requirements list is contained by the versions list we have to nest the for loops. Not doing so will give incomplete results where only the requirements list of the last value in the versions list is targeted:

for _version in data["versions"]: 
	print(_version["codename"])
for _requirement in _version["requirements"]: 
	print("  " + _requirement)

Mojave
Catalina
  iMac (Late 2012 or newer)
  iMac Pro (2017)
  Mac Mini (Late 2012 or newer)
  Mac Pro (Late 2013)
  MacBook (Early 2015 or newer)
  MacBook Air (Mid 2012 or newer)
  MacBook Pro (Mid 2012 or newer)
  4 GB of memory
  12.5 GB of available avaialable disk space
  OS X 10.11.5 or later

Finally, to additionally include the version of every macOS release without using a for loop:

print(data["versions"][0]["codename"]); \
print("  " + data["versions"][0]["requirements"][0]); \
print("  " + data["versions"][0]["requirements"][1]); \
print("  " + data["versions"][0]["requirements"][2]); \
print("  " + data["versions"][0]["requirements"][3]); \
print("  " + data["versions"][0]["requirements"][4]); \
print("  " + data["versions"][0]["requirements"][5]); \
print("  " + data["versions"][0]["requirements"][6]); \
print("  " + data["versions"][0]["requirements"][7]); \
print("  " + data["versions"][0]["requirements"][8]); \
print("  " + data["versions"][0]["requirements"][9]); \
print("    " + data["versions"][0]["releases"][0]["version"]); \
print("    " + data["versions"][0]["releases"][1]["version"]); \
print("    " + data["versions"][0]["releases"][2]["version"]); \
print("    " + data["versions"][0]["releases"][3]["version"]); \
print("    " + data["versions"][0]["releases"][4]["version"]); \
print("    " + data["versions"][0]["releases"][5]["version"]); \
print("    " + data["versions"][0]["releases"][6]["version"]); \
print(data["versions"][1]["codename"]); \
print("  " + data["versions"][1]["requirements"][0]); \
print("  " + data["versions"][1]["requirements"][1]); \
print("  " + data["versions"][1]["requirements"][2]); \
print("  " + data["versions"][1]["requirements"][3]); \
print("  " + data["versions"][1]["requirements"][4]); \
print("  " + data["versions"][1]["requirements"][5]); \
print("  " + data["versions"][1]["requirements"][6]); \
print("  " + data["versions"][1]["requirements"][7]); \
print("  " + data["versions"][1]["requirements"][8]); \
print("  " + data["versions"][1]["requirements"][9]); \
print("    " + data["versions"][1]["releases"][0]["version"]); \
print("    " + data["versions"][1]["releases"][1]["version"]); \
print("    " + data["versions"][1]["releases"][2]["version"]); \
print("    " + data["versions"][1]["releases"][3]["version"]); \
print("    " + data["versions"][1]["releases"][4]["version"]); \
print("    " + data["versions"][1]["releases"][5]["version"])

Mojave
  iMac (Late 2012 or newer)
  iMac Pro (2017)
  Mac Mini (Late 2012 or newer)
  Mac Pro (Late 2013; Mid 2010 and Mid 2012 models with recommended Metal-capable graphics cards)
  MacBook (Early 2015 or newer)
  MacBook Air (Mid 2012 or newer)
  MacBook Pro (Mid 2012 or newer)
  2 GB of memory
  12.5 - 18.5 GB of available avaialable disk space
  OS X 10.8 or later
    10.14
    10.14.1
    10.14.2
    10.14.3
    10.14.4
    10.14.5
    10.14.6
Catalina
  iMac (Late 2012 or newer)
  iMac Pro (2017)
  Mac Mini (Late 2012 or newer)
  Mac Pro (Late 2013)
  MacBook (Early 2015 or newer)
  MacBook Air (Mid 2012 or newer)
  MacBook Pro (Mid 2012 or newer)
  4 GB of memory
  12.5 GB of available avaialable disk space
  OS X 10.11.5 or later
    10.15
    10.15.1
    10.15.2
    10.15.3
    10.15.4
    10.15.5

Alternatively, using a for loop:

for _version in data["versions"]: 
	print(_version["codename"])
	for _requirement in _version["requirements"]: 
		print("  " + _requirement)
	for _release in _version["releases"]: 
		print("    " + _release["version"])

Mojave
  iMac (Late 2012 or newer)
  iMac Pro (2017)
  Mac Mini (Late 2012 or newer)
  Mac Pro (Late 2013; Mid 2010 and Mid 2012 models with recommended Metal-capable graphics cards)
  MacBook (Early 2015 or newer)
  MacBook Air (Mid 2012 or newer)
  MacBook Pro (Mid 2012 or newer)
  2 GB of memory
  12.5 - 18.5 GB of available avaialable disk space
  OS X 10.8 or later
    10.14
    10.14.1
    10.14.2
    10.14.3
    10.14.4
    10.14.5
    10.14.6
Catalina
  iMac (Late 2012 or newer)
  iMac Pro (2017)
  Mac Mini (Late 2012 or newer)
  Mac Pro (Late 2013)
  MacBook (Early 2015 or newer)
  MacBook Air (Mid 2012 or newer)
  MacBook Pro (Mid 2012 or newer)
  4 GB of memory
  12.5 GB of available avaialable disk space
  OS X 10.11.5 or later
    10.15
    10.15.1
    10.15.2
    10.15.3
    10.15.4
    10.15.5

The last for loop repeats for every value (dictionary) in the releases list. On each loop, the current list value is written to the variable _release. Subsequently, we use _release["version"] to get the version of each release.

Similar to the requirements list, the releases list is also contained within the versions list, so the for loop targeting the releases list is also nested within the for loop targeting the versions list.

Notes

• The information on macOS (Mac OS X) releases contained in the file macos.json is for demonstration purposes only. The version of the file used in this post is a subset of the data contained in the original. The latest, complete version can be found here. While every attempt has been made to ensure this data is correct, its accuracy is not guaranteed.

• parse-json.py is a Python script used to parse the JSON-encoded data in the file macos.json and is based on the examples in this post. A Node.js application that runs this Python script and displays the results can be found at macos.tech-otaku.com

• The GitHub repository for the Node.js application can be found at macos-versions.

For information on installing WP-CLI, some basic configuration tips and examples of using aliases please refer to Installing and Configuring WP-CLI on macOS.

This site uses the Typology theme by Meks. The theme option that controls which posts appear in the slider is front_page_cover_posts_manual and requires a comma-separated string of post IDs.

I can use WP-CLI to get the current value of this option:

wp @production option pluck typology_settings front_page_cover_posts_manual

1789,6090,1860,5524,1282

The constituent parts of the command string are:

To update which posts appear in the slider I replace the WP-CLI command option pluck with option patch update and provide a new comma-separated list of post IDs:

wp @production option patch update typology_settings front_page_cover_posts_manual '4825,5232,2163,6090,245'

Success: Updated 'typology_settings' option.

Rather than manually creating a comma-separated list of post IDs, one can be automatically generated using post list:

wp @production post list --posts_per_page=5 --post_status=publish --post_type=post --orderby=rand --format=ids

4737 940 6554 250 6473

The post list command uses any argument supported by WP_Query to return post data. Here I’m selecting 5 published posts in random order. The --format=ids argument ensures that only the ID of each post is returned.

The output from this command isn’t in the correct format. The post IDs are separated by spaces not commas, but by piping the output of the command to tr the spaces can be changed to commas:

wp @production post list --posts_per_page=5 --post_status=publish --post_type=post --orderby=rand --format=ids | tr ' ' ,

4737,940,6554,250,6473

By passing the command string that generates random post IDs as an argument to the command string that updates the theme option I can change what appears in the slider with a set of randomly selected posts:

wp @production option patch update typology_settings front_page_cover_posts_manual $(wp @production post list --posts_per_page=5 --post_status=publish --post_type=post --orderby=rand --format=ids | tr ' ' ,)

Success: Updated 'typology_settings' option.

For convenience the command can be placed in a script:

#!/bin/bash

# USAGE [/bin/bash] /path/to/slider-posts.sh <number-of-posts> <alias>
# $1 = Number of posts to include in the slider
# $2 = The alias to the WordPress installation

if [ $# -ne 2 ]; then
	printf "ERROR: 2 arguments are required: <number-of-posts> <alias>"
	exit 1
elif [[ ! $(/usr/local/bin/wp cli alias get @"${2}" 2>/dev/null) ]]; then 
	printf "ERROR: The alias \"@${2}\" doesn't exist."
	exit 1
fi

/usr/local/bin/wp @"${2}" option patch update typology_settings front_page_cover_posts_manual $(/usr/local/bin/wp @"${2}" post list --posts_per_page="${1}" --post_status=publish --post_type=post --orderby=rand --format=ids | tr ' ' ,)

NOTE: The script explicitly uses the full path to the WP-CLI binary: /usr/local/bin/wp. This is to allow the script to be executed by launchd. When the script is executed from the command-line, it searches the directories defined in the user’s $PATH environment variable for the WP-CLI binary. The $PATH environment variable will invariably contain the /usr/local/bin directory. However, launchd has a default search path limited to /usr/bin:/bin:/usr/sbin:/sbin. Consequently, if launchd attempted to run this script without the script using the full path to the WP-CLI binary it would fail.

The script requires two arguments: the number of posts to include in the slider and the alias to the WordPress install being targeted. In addition, the script uses WP-CLI to check the alias and exit with an error if the alias doesn’t exist.

Make the script executable:

chmod +x /path/to/slider-posts.sh

Run the script passing the number of posts and alias as arguments:

/path/to/slider-posts.sh 5 production

Updated 'typology_settings' option.

Now I have a working script I can use launchd to execute the script at a regular interval by first creating a job definition file. This file should have a .plist extension and be located in the ~/Library/LaunchAgents directory:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.tech-otaku.slider.posts</string>
    <key>ProgramArguments</key>
    <array>
        <string>/path/to/slider-posts.sh</string>
        <string>5</string>
        <string>production</string>
    </array>
    <key>StandardErrorPath</key>
    <string>/tmp/com.tech-otaku.slider.posts.stderr</string>
    <key>StandardOutPath</key>
    <string>/tmp/com.tech-otaku.slider.posts.stdout</string>
    <key>StartCalendarInterval</key>
    <array>
        <dict>
            <key>Hour</key>
            <integer>23</integer>
            <key>Minute</key>
            <integer>0</integer>
        </dict>
    </array>
</dict>
</plist>

When active, this job definition will execute /path/to/slider-posts.sh 5 production at 23:00 everyday.

To activate a job definition, it needs to be loaded. All enabled job definitions are loaded automatically when your Mac powers-on or you login, but to load it explicitly:

launchctl load "${HOME}/Library/LaunchAgents/com.tech-otaku.slider.posts"

Authorisation requests are routinely handled by macOS when an application requiring access is first launched. Occasionally, an application may need to be authorised by manually granting it access.

Before looking at how an application can be manually authorised, we need to configure Finder windows to display hard disks in the sidebar. Select Finder > Preferences > Sidebar and ensure that under Locations the Hard disk option is checked:

To manually authorise an application it needs to be given full disk access. Select System Preferences > Security & Privacy > Privacy > Full Disk Access and click the + button:

To give, say, the GitHub Desktop application full disk access, navigate to the /Applications directory, select GitHub Desktop.app and click Open:

With macOS Catalina, full disk access may need to be given to a program not located in the /Applications directory. An example is the bash interpreter that currently ships with macOS and whose binary is in the /bin directory.

To grant /bin/bash full disk access, first click Macintosh HD in the sidebar of the Open dialog:

The /bin directory – amongst others – isn’t shown until the ⇧+⌘+. keys are pressed simultaneously:

Navigate to the /bin directory, select bash then click Open to grant the bash binary full disk access:

This Open dialog method can also be used to grant full disk access to an application by selecting a symbolic link that points to that application. An example is /usr/local/bin/bash which is a symbolic link to the bash binary /usr/local/Cellar/bash/5.0.16/bin/bash installed through Homebrew:

An alternative to the Open dialog method is to first navigate to the application in the Finder, then drag and drop the application icon onto the Full Disk Access pane of System Preferences:

This drag and drop method doesn’t work with symbolic links in macOS Catalina. While it is possible to drag and drop the symbolic link icon onto the Full Disk Access pane of System Preferences, the application the symbolic link points to doesn’t appear to be granted full disk access. Consequently, only the Open dialog method described above should be used with symbolic links.

Method 1 – Using the MySQL Command Line Client

The Google Authenticator plugin settings are stored on a per-user basis in the *_usermeta table. Before taking a look at these settings from the command prompt of your server’s remote shell you’ll need the following information from your site’s wp-config.php file:

mysql-user-name:     DB_USER
mysql-user-password: DB_PASSWORD
database-name:       DB_NAME
table-prefix_:       $table_prefix

To get the appropriate wp-user-id take a look at How to Find a WordPress User ID but instead of username, hover over the number of posts and look for author=

Armed with this information, use the following command to interrogate the database (you’ll be prompted for the mysql-user-name password):

mysql -u mysql-user-name -p -e "USE database-name; SELECT user_id, meta_key, meta_value FROM table-prefix_usermeta WHERE user_id=wp-user-id AND meta_key LIKE 'googleauthenticator_%';"

+---------+----------------------------------+-----------------------------------------------------------------------+
| user_id | meta_key                         | meta_value                                                            |
+---------+----------------------------------+-----------------------------------------------------------------------+
|       2 | googleauthenticator_description  | My Site                                                               |
|       2 | googleauthenticator_enabled      | enabled                                                               |
|       2 | googleauthenticator_lasttimeslot | 52734943                                                              |
|       2 | googleauthenticator_passwords    | {"appname":"Default","password":"HCACyh@eH-XnL*ALCJJ3-yrKVBVw2m62jm"} |
|       2 | googleauthenticator_pwdenabled   | disabled                                                              |
|       2 | googleauthenticator_relaxedmode  | disabled                                                              |
|       2 | googleauthenticator_secret       | 2TXZUABQROZJAH2E                                                      |
+---------+----------------------------------+-----------------------------------------------------------------------+

The option we’re interested in is googleauthenticator_enabled which can either have a value of enabled (active) or disabled (inactive). To change the value to disabled use the following command:

mysql -u mysql-user-name -p -e "USE database-name; UPDATE table-prefix_usermeta SET meta_value='disabled' WHERE user_id=wp-user-id AND meta_key='googleauthenticator_enabled';"

To confirm the change is successful and the value is now disabled:

mysql -u mysql-user-name -p -e "USE database-name; SELECT user_id, meta_key, meta_value FROM table-prefix_usermeta WHERE user_id=wp-user-id AND meta_key='googleauthenticator_enabled';"

+---------+----------------------------------+-----------------------------------------------------------------------+
| user_id | meta_key                         | meta_value                                                            |
+---------+----------------------------------+-----------------------------------------------------------------------+
|       2 | googleauthenticator_enabled      | disabled                                                              |
+---------+----------------------------------+-----------------------------------------------------------------------+

We could combine the command to update the option with the command to confirm the update was successful, but the resulting command is a little lengthy so we could place the necessary SQL statements into a file named say disable-ga.sql:

USE database-name;
UPDATE table-prefix_usermeta SET meta_value='disabled' WHERE user_id=wp-user-id AND meta_key='googleauthenticator_enabled';
SELECT user_id, meta_key, meta_value FROM table-prefix_usermeta WHERE user_id=wp-user-id AND meta_key='googleauthenticator_enabled';

The command to disable 2FA and confirm the change is now:

mysql --table -u mysql-user-name -p < /path/to/disable-ga.sql

+---------+----------------------------------+-----------------------------------------------------------------------+
| user_id | meta_key                         | meta_value                                                            |
+---------+----------------------------------+-----------------------------------------------------------------------+
|       2 | googleauthenticator_enabled      | disabled                                                              |
+---------+----------------------------------+-----------------------------------------------------------------------+

Note the addition of the mysql command line option --table which is necessary when running mysql in batch mode to ensure results are displayed in a tabular format.

Similarly, we could create another file named say enable-ga.sql in order to enable 2FA:

USE database-name;
UPDATE table-prefix_usermeta SET meta_value='enabled' WHERE user_id=wp-user-id AND meta_key='googleauthenticator_enabled';
SELECT user_id, meta_key, meta_value FROM table-prefix_usermeta WHERE user_id=wp-user-id AND meta_key='googleauthenticator_enabled';

Now, to enable 2FA:

mysql --table -u mysql-user-name -p < /path/to/enable-ga.sql

+---------+----------------------------------+-----------------------------------------------------------------------+
| user_id | meta_key                         | meta_value                                                            |
+---------+----------------------------------+-----------------------------------------------------------------------+
|       2 | googleauthenticator_enabled      | enabled                                                               |
+---------+----------------------------------+-----------------------------------------------------------------------+

We now have a way of disabling/enabling 2FA from the command line, but this is being actioned server-side remotely. On a Unix-like OS such as Linux or macOS the same can be achieved from a local shell:

ssh -p 22 user@12.34.56.789 "mysql --table -u mysql-user-name -p < /path/to/disable-ga.sql"

Enter password: ERROR 1045 (28000): Access denied for user 'mysql-user-name'@'localhost' (using password: YES)

We're simply placing our command in double-quotes and prefixing it with the SSH credentials required to login to the server. However, this will cause an error as we're unable to enter the password for mysql-user-name. To avoid this error amend or create the file .my.cnf in your home directory on the server:

[client-database-name]
user=mysql-user-name
password='mysql-user-password'

This provides mysql with the password for the MySQL user mysql-user-name. Note the group name in square brackets: client-database-name. It must begin with the word client, but can be followed with any string. I chose - followed by the database-name.

Now, to disable 2FA from a local shell:

ssh -p 22 user@12.34.56.789 "mysql --defaults-group-suffix=-database-name --table -u mysql-user-name < /path/to/disable-ga.sql"

+---------+----------------------------------+-----------------------------------------------------------------------+
| user_id | meta_key                         | meta_value                                                            |
+---------+----------------------------------+-----------------------------------------------------------------------+
|       2 | googleauthenticator_enabled      | disabled                                                              |
+---------+----------------------------------+-----------------------------------------------------------------------+

Note the addition of the mysql command line option --defaults-group-suffix= and its value -database-name which instructs mysql to use the settings associated with the group named client-database-name in the ~/.my.cnf file.

Optionally, create shell aliases to each of these commands by adding the following code to your user profile. For bash, this file is either ~/.bashrc or ~/.bash_profile. For zsh this is ~/.zshrc.

To create an alias named disga:

alias disga='ssh -p 22 user@12.34.56.789 "mysql --defaults-group-suffix=-database-name --table -u mysql-user-name < /path/to/disable-ga.sql"'

To create an alias named enga:

alias enga='ssh -p 22 user@12.34.56.789 "mysql --defaults-group-suffix=-database-name --table -u mysql-user-name < /path/to/enable-ga.sql"'

Now, to disable or enable 2FA simply type disga or enga respectively at the local command prompt:

enga

+---------+----------------------------------+-----------------------------------------------------------------------+
| user_id | meta_key                         | meta_value                                                            |
+---------+----------------------------------+-----------------------------------------------------------------------+
|       2 | googleauthenticator_enabled      | enabled                                                               |
+---------+----------------------------------+-----------------------------------------------------------------------+

Method 2 - Using WP-CLI

The second method uses WP-CLI, the command line interface for WordPress and requires WP-CLI to be installed both locally and remotely with some configuration required for the local install. However, once installed it can be used for not only toggling 2FA. See Installing and Configuring WP-CLI on macOS for details.

The following commands use the WP-CLI user meta get and user meta update commands.

To check the status of 2FA provided by the Google Authenticator plugin using WP-CLI locally:

wp @production user meta get "wp-user-id" googleauthenticator_enabled

enabled

To disable 2FA using WP-CLI locally:

wp @production user meta update "wp-user-id" googleauthenticator_enabled "disabled"

Success: Updated custom field 'googleauthenticator_enabled'.

To enable 2FA using WP-CLI locally:

wp @production user meta update "wp-user-id" googleauthenticator_enabled "enabled"

Success: Updated custom field 'googleauthenticator_enabled'.

These commands can also be aliased. To create an alias named disgawp:

alias disgawp='wp @production user meta update "wp-user-id" googleauthenticator_enabled "disabled"'

To create an alias named engawp:

alias engawp='wp @production user meta update "wp-user-id" googleauthenticator_enabled "enabled"'

Paste the following at a command prompt to create a script that demonstrates this:

cat << 'EOF' > "${HOME}/which-interpreter.sh"
#!/bin/bash
ps -p "$$" | tail -1 | awk '{print $4}'
EOF

This script simply displays the interpreter that parses it, but before running the script let’s look at what the line ps -p "$$" | tail -1 | awk '{print $4}' does.

The ps command with the -p flag outputs information about the process matching the process ID (PID). The $$ represents the PID of the script. The output of ps -p "$$" is:

 PID TTY           TIME CMD
3740 ttys000    0:00.01 /bin/bash /Users/steve/which-interpreter.sh

This output is piped to tail to remove the header:

3740 ttys000    0:00.01 /bin/bash /Users/steve/which-interpreter.sh

The CMD portion of the output shows the command that launched the script: /bin/bash and the parameter passed to it /Users/steve/which-interpreter.sh. Only the command is of interest so the output is piped to awk to print the fourth field:

/bin/bash

Make the script executable:

chmod +x "${HOME}/which-interpreter.sh"

Now let’s run the script:

"${HOME}/which-interpreter.sh"

/bin/bash

It doesn’t matter if the shell being used to run the script is a different version of bash or even another shell like zsh. The shebang will still be honoured and the script parsed by the /bin/bash interpreter unless another interpreter is explicitly used to execute the script at the command prompt.

Here are some examples:

NOTE: I have two bash interpeters on my system. The one shipped with macOS: /bin/bash and the one installed via Homebrew: /usr/local/bin/bash. The latter is a symbolic link to /usr/local/Cellar/bash/5.0.16/bin/bash.

/usr/local/bin/bash "${HOME}/which-interpreter.sh"

/usr/local/bin/bash

/bin/zsh "${HOME}/which-interpreter.sh"

/bin/zsh

/usr/bin/env bash "${HOME}/which-interpreter.sh"

bash

This last example only shows that the script is being parsed by a bash interpeter, but not which one. It could be /bin/bash or /usr/local/bin/bash or another bash interpreter.

To see which interpreter is being used, overwrite the existing script to include the which command:

cat << 'EOF' > "${HOME}/which-interpreter.sh"
#!/bin/bash
which $(ps -p "$$" | tail -1 | awk '{print $4}')
EOF

Now run the amended script:

/usr/bin/env bash "${HOME}/which-interpreter.sh"

/usr/local/bin/bash

Using /usr/bin/env bash from the command prompt or as part of a shebang instructs the script to use a bash interpreter to parse its contents, but doesn’t explicitly state which one. That is determined by the user’s search path which is stored in the PATH environmental variable:

echo $PATH

/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

The script is parsed by the first instance of bash found in the search path. In my case this is /usr/local/bin/bash, not /bin/bash.

Overwrite the existing script changing the shebang to #!/usr/bin/env bash:

cat << 'EOF' > "${HOME}/which-interpreter.sh"
#!/usr/bin/env bash
which $(ps -p "$$" | tail -1 | awk '{print $4}')
EOF

Run the amended script:

"${HOME}/which-interpreter.sh"

/usr/local/bin/bash

Care should be taken when using launchd to execute scripts with this type of shebang as launchd has a default search path limited to /usr/bin:/bin:/usr/sbin:/sbin. Consequently, if this script were run by launchd it would be parsed by the only available bash interpeter in that search path: /bin/bash.

Getting to this stage wasn’t easy, but not the nightmare it is often portrayed as. I relied heavily on these excellent guides to get the mail server up-and-running: How to Easily Set Up a Full-Featured Mail Server on Ubuntu 18.04 with iRedMail and How to Host Multiple Mail Domains in iRedMail with Nginx and then the official iRedMail documentation to fine tune the components.

During testing I re-provisioned the mail server many, many, many times and developed my own way of doing things, some of which may be useful to others.

Please note that the following were tested using iRedMail version 1.3.1.

1. Create New Mail Domains from the Command Line

During installation iRedMail configures a single mail domain. Additional mail domains can be added using iRedAdmin – the iRedMail web admin panel. During testing a more efficient, consistent and convenient way for me to create new mail domains was from the MySQL command-line client.

To invoke the MySQL client as root:

sudo mysql -u root

To create the new mail domain:

SET @DOMAIN:='domain.com';

INSERT INTO `vmail`.`domain` (`domain`, `transport`, `settings`, `created`, `active`) VALUES (@DOMAIN, 'dovecot', 'default_user_quota:512;', NOW(), '1');

To display the newly created mail domain record:

SELECT `domain`, `transport`, `settings`, `created`, `active` FROM `vmail`.`domain` WHERE `domain`=@DOMAIN;

2. Create New Mail Accounts from the Command Line

iRedMail ships with a script – create_mail_user_SQL.sh – to create mail accounts from the command line. The script requires two parameters – email address and password:

ADDRESS='steve@example.com'; \
PASSWORD='pAss71woRd'; \
cd ${HOME}/iRedMail-1.3.1/tools; \
bash create_mail_user_SQL.sh $ADDRESS $PASSWORD > user.sql; \
sudo mysql -u root -p -D vmail -e "SOURCE user.sql"; \
rm -f user.sql

The script does not allow the user’s quota to be set on an individual basis without amending the hard-coded value in the script file each time nor does it capitalise the first letter of the user’s name – steve instead of Steve – which is derived from the user’s email address.

Using sed the original script can be amended so that it accepts the user’s quota as a third parameter and automatically capitalises the user’s first name. The differences between the original and amended script are detailed here:

sed -i.$(date +"%Y%m%d-%H%M%S") -e "s/^DEFAULT_QUOTA='1024'/DEFAULT_QUOTA=\$3/" -e "s/X'2'/X'3'/" -e "s/ltd plain_password/ltd plain_password mail_quota/" -e "s/'\${username}'/\"\$(echo \${username} | awk '{print toupper(substr(\$0,0,1))tolower(substr(\$0,2))}')\"/" $HOME/iRedMail-1.3.1/tools/create_mail_user_SQL.sh

The script can now be run with with a value for the user’s quota. Note that a value of 0 means there is no restriction on the size of the mailbox:

ADDRESS='steve@example.com'; \
PASSWORD='pAss71woRd'; \
QUOTA='0'; \
cd ${HOME}/iRedMail-1.3.1/tools; \
bash create_mail_user_SQL.sh $ADDRESS $PASSWORD $QUOTA > user.sql; \
sudo mysql -u root -p -D vmail -e "SOURCE user.sql"; \
rm -f user.sql

3. SOGo and Memcached

If you opted to have SOGo installed by the iRedMail installer you may see a lot of errors afterward containing SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY in /var/log/sogo/sogo.log and when the script /var/vmail/backup/backup_sogo.sh is executed (usually as a daily cronjob). This is due to Memcached either not being installed or the Memcached server not running.

On Ubuntu, to check if Memcached is installed:

dpkg -L memcached

dpkg-query: package 'memcached' is not installed
Use dpkg --info (= dpkg-deb --info) to examine archive files,
and dpkg --contents (= dpkg-deb --contents) to list their contents.

/.
/etc
/etc/default
/etc/default/memcached
/etc/init.d
/etc/init.d/memcached
/lib
/lib/systemd
/lib/systemd/system
/lib/systemd/system/memcached.service
/usr
/usr/bin
/usr/bin/memcached
/usr/include
/usr/include/memcached
/usr/include/memcached/protocol_binary.h
/usr/lib
/usr/lib/tmpfiles.d
/usr/lib/tmpfiles.d/memcached.conf
/usr/share
/usr/share/doc
/usr/share/doc/memcached
/usr/share/doc/memcached/NEWS.Debian.gz
/usr/share/doc/memcached/NEWS.gz
/usr/share/doc/memcached/README.Debian
/usr/share/doc/memcached/README.md
/usr/share/doc/memcached/changelog.Debian.gz
/usr/share/doc/memcached/copyright
/usr/share/doc/memcached/protocol.txt.gz
/usr/share/man
/usr/share/man/man1
/usr/share/man/man1/memcached.1.gz
/usr/share/memcached
/usr/share/memcached/memcached.conf.default
/usr/share/memcached/scripts
/usr/share/memcached/scripts/README.damemtop
/usr/share/memcached/scripts/damemtop
/usr/share/memcached/scripts/damemtop.yaml
/usr/share/memcached/scripts/memcached-tool
/usr/share/memcached/scripts/start-memcached
/usr/share/memcached/scripts/systemd-memcached-wrapper

To install Memcached:

sudo apt-get install -y memcached

After installing Memcached you’d be forgiven for thinking the Memcached server has started and is running without issue. To check the status of Memcached:

sudo systemctl status memcached

● memcached.service - memcached daemon
   Loaded: loaded (/lib/systemd/system/memcached.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2019-12-29 07:32:50 GMT; 2s ago
     Docs: man:memcached(1)
 Main PID: 19952 (memcached)
    Tasks: 10 (limit: 4583)
   CGroup: /system.slice/memcached.service
           └─19952 /usr/bin/memcached -u memcache -l 127.0.0.1

● memcached.service - memcached daemon
   Loaded: loaded (/lib/systemd/system/memcached.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sun 2019-12-29 07:20:23 GMT; 7min ago
     Docs: man:memcached(1)
  Process: 19857 ExecStart=/usr/share/memcached/scripts/systemd-memcached-wrapper /etc/memcached.conf (code=exited, status=64)
 Main PID: 19857 (code=exited, status=64)

Dec 29 07:20:23 mail.example.com systemd[1]: memcached.service: Service hold-off time over, scheduling restart.
Dec 29 07:20:23 mail.example.com systemd[1]: memcached.service: Scheduled restart job, restart counter is at 5.
Dec 29 07:20:23 mail.example.com systemd[1]: Stopped memcached daemon.
Dec 29 07:20:23 mail.example.com systemd[1]: memcached.service: Start request repeated too quickly.
Dec 29 07:20:23 mail.example.com systemd[1]: memcached.service: Failed with result 'exit-code'.
Dec 29 07:20:23 mail.example.com systemd[1]: Failed to start memcached daemon.

When failing to start, the status messages don’t really give any clues as to why, but looking at /var/log/syslog specifically for the Failed to start memcached daemon message gives a clue: can't run as root without the -u switch

sudo grep -B 8 'Failed to start memcached daemon' /var/log/syslog

Dec 29 07:20:22 mail systemd-memcached-wrapper[19857]: can't run as root without the -u switch
Dec 29 07:20:22 mail systemd[1]: memcached.service: Main process exited, code=exited, status=64/n/a
Dec 29 07:20:22 mail systemd[1]: memcached.service: Failed with result 'exit-code'.
Dec 29 07:20:23 mail systemd[1]: memcached.service: Service hold-off time over, scheduling restart.
Dec 29 07:20:23 mail systemd[1]: memcached.service: Scheduled restart job, restart counter is at 5.
Dec 29 07:20:23 mail systemd[1]: Stopped memcached daemon.
Dec 29 07:20:23 mail systemd[1]: memcached.service: Start request repeated too quickly.
Dec 29 07:20:23 mail systemd[1]: memcached.service: Failed with result 'exit-code'.
Dec 29 07:20:23 mail systemd[1]: Failed to start memcached daemon.

To start the Memcached server – not as root, but as user memcache – add the following line to the file /etc/memcached.conf:

-u memcache
-l 127.0.0.1

Start the Memcached server:

sudo systemctl start memcached

4. Fail2ban Log File

The Fail2ban log file at /var/log/fail2ban.log contains notifications of IP addresses first banned and subsequently unbanned from accessing your mail server.

The log file is initially created as owned by user syslog with read/write permissions and group adm with read-only permissions:

ls -lah /var/log/fail2ban.log

-rw-r----- 1 syslog adm 13K Sep 15 14:15 /var/log/fail2ban.log

The Faile2ban log file is rotated on a weekly basis when /var/log/fail2ban.log becomes /var/log/fail2ban.log.1 and a new /var/log/fail2ban.log is created. However, the newly created file is now owned by root and consequently notifications will no longer be written to the new Fail2ban log file and it will remain empty:

ls -lah /var/log/fail2ban.log

-rw-r----- 1 root adm 0K Sep 20 06:25 /var/log/fail2ban.log

The reason for this change in ownership is due to the Logrotate configuration file for Fail2ban:

cat /etc/logrotate.d/fail2ban

/var/log/fail2ban.log {

    ...

    # If fail2ban runs as non-root it still needs to have write access
    # to logfiles.
    # create 640 fail2ban adm
    create 640 root adm
}

The line responsible for the Fail2ban log’s change in ownership is create 640 root adm. Change this to create 640 syslog adm and in future and any /var/log/fail2ban.log will be writable:

sudo sed -i.$(date +"%Y%m%d-%H%M%S") 's/create 640 root adm/create 640 syslog adm/' /etc/logrotate.d/fail2ban && cat /etc/logrotate.d/fail2ban

/var/log/fail2ban.log {

    ...

    # If fail2ban runs as non-root it still needs to have write access
    # to logfiles.
    # create 640 fail2ban adm
    create 640 syslog adm
}

If you have an existing /var/log/fail2ban.log owned by root, change its ownership:

sudo chown syslog /var/log/fail2ban.log

Changing the port Webmin listens on or using Webmin’s IP Access Control to allow access to a limited number of IP addresses may reduce the risk, but not eliminate it. As I’m the only one who requires access to Webmin, it makes more sense to block remote access entirely and allow only limited local access using local port forwarding through an SSH tunnel.

Remote Access to Webmin

Figure 1 shows applications on a local computer – client – sending requests to a remote host – server – and the appropriate applications on the server handling those requests and sending data back to the client applications. The server’s firewall blocks all incoming requests except to those ports that have been explicitly opened.

On the server, ufw will show which ports are open:

sudo ufw status 

To                         Action      From
--                         ------      ----
10000/tcp                  ALLOW       Anywhere
80/tcp                     ALLOW       Anywhere
22/tcp                     LIMIT       Anywhere
10000/tcp (v6)             ALLOW       Anywhere (v6)
80/tcp (v6)                ALLOW       Anywhere (v6)
22/tcp (v6)                LIMIT       Anywhere (v6)

The Webmin server – Miniserv – is listening for requests on port 10000. As port 10000 is open in the server’s firewall, any client requests on port 10000 will reach and be handled by the Webmin server rendering the Webmin login page accessible to anyone with Internet access.

Blocking Access to Webmin

The first step in securing access to Webmin is to close the port in the server firewall that the Webmin server listens on for client requests. We do this with ufw by deleting the rule that explicitly opens port 10000:

sudo ufw delete allow 10000/tcp

Rule deleted
Rule deleted (v6)

Figure 2 shows the client browser still sending requests on port 10000 to the Webmin server and the Webmin server still listening for requests on that same port. However, as port 10000 is no longer open in the server firewall, the Webmin server will never receive the client requests.

Webmin is – for now – inaccessible to everyone.

Accessing Webmin Through an SSH Tunnel

I’m going to provide access to Webmin by routing browser requests for it through an encrypted SSH connection between the client and the server and have these requests forwarded to Webmin. Figure 3 demonstrates this process:

1. The client establishes an encrypted SSH connection to the server on port 22…

2. …simultaneously opening local port 10005 on the client.

3. The browser no longer sends requests directly to Webmin on the server, but to local port 10005 instead.

4. All requests received on local port 10005 are then forwarded or tunnelled through the encrypted SSH connection to the server.

5. The server forwards the requests onward to the Webmin server on port 10000.

The advantage of this approach is the ability to limit Webmin access to only those users who can connect to the server via SSH. In addition, SSH provides an encrypted connection. Any plain text data being tunneled through it will benefit from that secure connection. This means that Webmin can be accessed over HTTP as the plain text data being sent to and from the Webmin server doesn’t need the encryption of TLS/SSL that HTTPS provides.

In general, on a macOS or Ubuntu Desktop client the command to establish local port forwarding through an SSH tunnel is:

ssh -p 22 user@host -L <local-port>:<remote-host>:<remote-port> -N &

More specifically, for the example in figure 3 the command is:

ssh -p 22 penny@203.0.113.5 -L 10005:127.0.0.1:10000 -N &

[1] 5907

Please note the following:

• -p 22 does not need to be included in the command if the SSH server is listening on the default port 22. However, if it’s listening on say port 5522, -p 5522 would have to be part of the command.

• Thinking that 127.0.0.1 refers to a localhost on a client, it took me a while to understand why it is being used for the <remote-host> until I realised it’s viewed from the perspective of the server not the client.

• & puts SSH into the background which is useful if you want to continue using the same Terminal window. I prefer this to the -f option which is part of ssh as I find it easier to terminate the SSH tunnel having used the former option. On a Ubuntu Desktop client you may need to press enter to drop back to the command prompt after using &.

Having established the SSH tunnel and forwarded the local client port 10005 to the remote server port 10000, Webmin can be accessed from a browser using:

http://localhost:10005

You may receive an error that Webmin should be accessed using https:// instead of http://. This is more than likely because SSL is enabled in the Webmin server configuration. You can continue by using https://localhost:10005, but your browser will warn you that the connection is not private.

We know that any plain text data bound for the Webmin server will be sent through the encrypted SSH tunnel, so my preferred method is to disable SSL on the Webmin server and access Webmin through http://localhost:10005.

To disable SSL on the Webmin server we need to change ssl=1 to ssl=0 in the file /etc/webmin/miniserv.conf on the server. To do this, on the server type:

sudo sed -i.$(date +"%Y%m%d-%H%M%S") 's/^ssl=1$/ssl=0/' /etc/webmin/miniserv.conf

Then restart the Webmin server:

sudo systemctl restart webmin

Terminating the SSH Tunnel

When you’ve finished your Webmin session, it’s good practice to terminate the SSH tunnel. If you used the & option when creating the SSH tunnel, from the same Terminal window type:

jobs

[1]+  Running                 ssh -p 22 penny@203.0.113.5 -L 10005:127.0.0.1:10000 -N &

This will display any process running in the background each of which will start with a number in square brackets. Bring the relevant process to the foreground by typing:

fg 1

ssh -p 22 penny@203.0.113.5 -L 10005:127.0.0.1:10000 -N

To end the process type Ctrl+C and the SSH tunnel is no more.

]]> https://www.tech-otaku.com/networking/securing-access-to-webmin-through-an-ssh-tunnel/feed/ 5 5567 https://www.tech-otaku.com/mac/installing-configuring-wp-cli-macos/ Thu, 27 Jun 2019 16:28:32 +0000 https://www.tech-otaku.com/?p=5524 WP-CLI is the command-line interface for WordPress. It allows you to manage your WordPress site from the command line without using a web browser. I currently use WP-CLI to temporarily disable the WordPress Google Authenticator plugin, create and configure a version of this site to run in a local AMP environment on my Mac and periodically change the posts that appear in the slider on this site’s home page.

Installing WP-CLI

There are several ways of installing WP-CLI. I chose the Homebrew method as Homebrew is already installed on my Mac. In Terminal or similar terminal emulator type:

brew install wp-cli

The Homebrew method creates the following directory structure:

/usr/local/bin/wp -> /usr/local/Cellar/wp-cli/2.2.0/bin/wp
/usr/local/Cellar/wp-cli
/usr/local/Cellar/wp-cli/2.2.0
/usr/local/Cellar/wp-cli/2.2.0/.brew
/usr/local/Cellar/wp-cli/2.2.0/.brew/wp-cli.rb
/usr/local/Cellar/wp-cli/2.2.0/bin
/usr/local/Cellar/wp-cli/2.2.0/bin/wp
/usr/local/Cellar/wp-cli/2.2.0/INSTALL_RECEIPT.json
/usr/local/opt/wp-cli -> /usr/local/Cellar/wp-cli/2.2.0
/usr/local/var/homebrew/linked/wp-cli -> /usr/local/Cellar/wp-cli/2.2.0

To show information on WP-CLI, in Terminal type:

wp cli info

OS:	Darwin 18.6.0 Darwin Kernel Version 18.6.0: Thu Apr 25 23:16:27 PDT 2019; root:xnu-4903.261.4~2/RELEASE_X86_64 x86_64
Shell:	/bin/bash
PHP binary:	/usr/bin/php
PHP version:	7.1.23
php.ini used:
WP-CLI root dir:	phar://wp-cli.phar/vendor/wp-cli/wp-cli
WP-CLI vendor dir:	phar://wp-cli.phar/vendor
WP_CLI phar path:	/Users/steve
WP-CLI packages dir:
WP-CLI global config:
WP-CLI project config:
WP-CLI version:	2.2.0

A WP-CLI Command Example

The basic format for using WP-CLI is wp <command> <sub-command> <options>. For example, to get a site’s url, in Terminal type:

wp option get siteurl

Error: This does not seem to be a WordPress installation.
Pass --path=`path/to/wordpress` or run `wp core download`.

However – as can be seen from the output – this throws an error as wp is looking for a non-existent WordPress installation in the current directory. To remedy this we need to supply the path to the WordPress installation to wp. This will most likely be the same as the site’s DocumentRoot as defined in the Apache configuration files.

To get the url for a site running in a local AMP environment, in Terminal type:

wp option get siteurl --path=/local/path/to/wordpress/install

http://local.tech-otaku

When getting a url for a remote site we need to pass SSH login details in the form of name, address and port to wp along with the remote path to the WordPress installation:

wp option get siteurl --ssh=ben@12.34.56.789:2222 --path=/remote/path/to/wordpress/install

https://www.tech-otaku.com

NOTE: When accessing WordPress data on a remote host, WP-CLI must also be installed on that remote host. For a Linux host use the recommended install method.

In the examples above, --ssh and --path are two of WP-CLI’s global parameters. When used on the command line they should be placed at the end of the command string or immediately after wp.

For a complete list of WP-CLI commands see WP-CLI Commands.

Creating an Alias to a WordPress Installation

Passing global parameters on the command line can prove cumbersome. Fortunately, we can create a one-word alias to each site we want to access and use that alias on the command line instead of passing global parameters. Each alias along with its relevant information is stored in the WP-CLI global configuration file: ~/.wp-cli/config.yml.

On a fresh WP-CLI installation this global configuration file doesn’t exist. You can create and edit it using a text editor or – as I prefer – create it from the command line and then use WP-CLI to create each alias. In Terminal type:

mkdir ~/.wp-cli && touch ~/.wp-cli/config.yml

An alternative to creating this global configuration file in your home directory is to create the structure in a directory of a file sharing service like iCloud Drive or Dropbox, then create a symbolic link to the directory containing the configuration file in your home directory. This way you’re able to share a single global configuration with more than one Mac.

In this instance I name the directory .wp-cli_shared and the symbolic link .wp-cli.

To create the global configuration file in iCloud Drive, in Terminal type:

mkdir ~/Library/Mobile\ Documents/com~apple~CloudDocs/.wp-cli_shared && touch ~/Library/Mobile\ Documents/com~apple~CloudDocs/.wp-cli_shared/config.yml

To create the the symbolic link in your home drectory, in Terminal type:

ln -s ~/Library/Mobile\ Documents/com~apple~CloudDocs/.wp-cli_shared .wp-cli

Having created the global configuration file, we can now create each alias. To create an alias named @local for a local WordPress installation, in Terminal type:

wp cli alias add @local --set-path=/local/path/to/wordpress/install --set-user=emily

To create an alias named @production for a remote WordPress installation, in Terminal type:

wp cli alias add @production --ssh=ben@12.34.56.789:2222 --path=/remote/path/to/wordpress/install --set-user=chikako

To view the contents of the global configuration file, in Terminal type:

cat ~/.wp-cli/config.yml

---
@local:
  path: /local/path/to/wordpress/install
  user: emily
@production:
  ssh: ben@12.34.56.789:2222
  path: /remote/path/to/wordpress/install
  user: chikako

To get either the local or remote site’s url I can now use the appropriate alias:

wp @local option get siteurl

http://local.tech-otaku

wp @production option get siteurl

https://www.tech-otaku.com

NOTE: When using an alias on the command line it can only be placed immediately after wp.

A comparison of the two folders shows that of the four entries in the temp folder, the file named file.txt and the folder named folder don’t exist on iCloud.

This example only shows a fraction of the contents of each folder and so having written the Bash script below to loop through the contents of the temp folder checking for the existence of each entry on iCloud, I assumed the results would confirm the visual comparison.

#!/usr/bin/env bash

shopt -s nullglob
shopt -u dotglob

src="/Users/steve/temp"
trg="/Users/steve/Documents"

for entry in "$src"/{..?,.[!.],}*; do
	name=$(basename -- "$entry")
	if [ ! -e "$trg/$name" ]; then	# file doesn't exist on target
		echo "'$name' DOESN'T EXIST in $trg"
	else
		echo "'$name' already exists in $trg"
	fi
done

shopt -u nullglob
shopt -s dotglob

'file.txt' DOESN'T EXIST in /Users/steve/Documents
'folder' DOESN'T EXIST in /Users/steve/Documents
'Wordpress Plugins.docx' already exists in /Users/steve/Documents
'world_cup_2018.xlsx' DOESN'T EXIST in /Users/steve/Documents

Not so. Clicking the OUTPUT tab above shows the results are as expected except for the file world_cup_2018.xlsx which the script has determined as not being on iCloud when the Finder clearly shows it is.

The clue to the script’s unexpected result is the icon next to the world_cup_2018.xlsx file on iCloud which denotes the file has yet to be downloaded to the local drive.

Listing the Document folder’s contents on iCloud from the command line shows that such files are subject to a different naming convention not evident in the Finder where the file name is prefixed with . – traditionally denoting a hidden file – and ends with .icloud.

ls -a /Users/steve/Documents

.world_cup_2018.xlsx.icloud
Wordpress Plugins.docx
workout.xlsx

Ammending the script to include an additional test for this other naming convention gives the expected results:

#!/usr/bin/env bash

shopt -s nullglob
shopt -u dotglob

src="/Users/steve/temp"
trg="/Users/steve/Documents"

for entry in "$src"/{..?,.[!.],}*; do
	name=$(basename -- "$entry")
	if [ ! -e "$trg/$name" ] && [ ! -e "$trg/.$name.icloud" ]; then	# file doesn't exist on target
		echo "'$name' DOESN'T EXIST in $trg"
	else
		echo "'$name' already exists in $trg"
	fi
done

shopt -u nullglob
shopt -s dotglob

'file.txt' DOESN'T EXIST in /Users/steve/Documents
'folder' DOESN'T EXIST in /Users/steve/Documents
'Wordpress Plugins.docx' already exists in /Users/steve/Documents
'world_cup_2018.xlsx' already exists in /Users/steve/Documents