Signing a message or text file

We’ll start by signing a simple text file, using the --clearsign option. This includes the signature in the message, which we can then distribute to people to read. Here’s the contents of message.txt:

This is a public message from Tom Ryder.

We’ll sign that with our new private key like so:

$ gpg --clearsign message.txt

We’re prompted for our passphrase for the private key:

You need a passphrase to unlock the secret key for
user: "Tom Ryder (Test Key Only) <tom@sanctum.geek.nz>"
4096-bit RSA key, ID 040FE79B, created 2013-03-23

Having provided that, the file message.txt.asc is created, with PGP sections and a plaintext ASCII signature:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

This is a public message from Tom Ryder.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iQIcBAEBCAAGBQJRTQcTAAoJEGIHQ4oED+ebtjoP/19PlndkGhR46BA6YZmDVdC1
Snk9aXe4Eo42kRpW13bjo8xg+pb+U26ylHkH52OBB7fJ3/BR5eZMug/RXJLGIf+U
aiylbGVz4dGjkeTCSxtg1TvcyEtzhm1ETfIWLarboj9PPBJfO1QIh4uPDkD7kb6+
+OnpdxPURbiJ03osu2Mj2fFq5wYT0La+I9BdKUAJmS8zt+CaTirKw+xF6l0sEAVv
lqslWjEwF9JCfumKAj8aMBeZMndoKRqW18ZHoYJdP0x3g1SUKAjZ/NRRUGug6Eg+
JTJ82ETCRKGKmYFLkHJ6iCaucrmhLTd9IYyEQZE/weUuClKqtshollhNHF1dD3SF
fWMPq0+29DInjlXwIkXyzVDln1wULNzbd5zv5Wg5b6lSCZlwH0xCrNjiYO7f413c
Ty4q6SqtRFUommnpMA5XcmX7ebbUpMfqfqqzoLqeTpA15Yuhh3DDR6NoMN82oLyF
FFt7UZh/JlYMc8GOnEqyZfT7d57FbKSLn3vpZbH9QXNFWG6/oZabFFyRm8r7k8F2
FzYTdyp59O0dW4T5OJ6a/xo/OnZutUN1RqW6ZJS19Xb4/5eEohFAFL9cDVLuC6zo
HmlA2m5zwv8aYbJad6Rk6+vpQAAdxHgNq/VYdcOfLOtcJAE6Jnm3alVyeXb1PbMB
WBxaM998Z6R8VmeMB7gQ
=WMzO
-----END PGP SIGNATURE-----

Note that the message itself is plainly readable; this message isn’t encrypted, it’s just verified as having been written by a particular person, and not altered since it was written.

Now anyone who has our public key on their keyring (as we ourselves do) can verify that it was actually us who wrote this message:

$ gpg --verify message.txt.asc
gpg: Signature made Sat 23 Mar 2013 14:32:17 NZDT using RSA key ID 040FE79B
gpg: Good signature from "Tom Ryder (Test Key Only) <tom@sanctum.geek.nz>"

If anybody tampers with the message, even something like removing a period from the end of a sentence, the verification will fail, suggesting the message was tampered with:

$ gpg --verify message.txt.asc
gpg: Signature made Sat 23 Mar 2013 14:32:17 NZDT using RSA key ID 040FE79B
gpg: BAD signature from "Tom Ryder (Test Key Only) <tom@sanctum.geek.nz>"

Signing and verifying binary files

For all other files, we likely need to make the signature file separate with a detached signature:

$ gpg --armor --detach-sign archive.tar.gz

This produces a file archive.tar.gz.asc in the same directory, containing the signature. We use --armor to make the signature in ASCII, which makes for a longer file but easier distribution online.

In this case, both the file and the signature are required for verification; put the signature file first when you check this:

$ gpg --verify archive.tar.gz.asc archive.tar.gz

You can use this method to verify software downloads from trusted sources, such as the Apache HTTPD development team. First, we would download and import all their public keys at the URL they nominate:

$ wget http://www.apache.org/dist/httpd/KEYS
$ gpg --import KEYS

We could then download an Apache HTTPD release, along with its key, from an arbitrary mirror:

$ wget http://www.example.com/apache/httpd/httpd-2.4.4.tar.gz
$ wget http://www.apache.org/dist/httpd/httpd-2.4.4.tar.gz.asc

We can then use the key and signature to verify that it’s an uncompromised copy of the original file signed by the developers:

$ gpg --verify httpd-2.4.4.tar.gz.asc httpd-2.4.4.tar.gz
gpg: Signature made Tue 19 Feb 2013 09:28:39 NZDT using RSA key ID 791485A8
gpg: Good signature from "Jim Jagielski (Release Signing Key) <jim@apache.org>"
gpg:                 aka "Jim Jagielski <jim@jaguNET.com>"
gpg:                 aka "Jim Jagielski <jim@jimjag.com>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: A93D 62EC C3C8 EA12 DB22  0EC9 34EA 76E6 7914 85A8

Note that the gpg output cautions that this is still not perfect assurance that the release actually came from Jim Jagielski, because we’ve never met him and can’t absolutely, definitely say that this is his public key. Looking him up on public keyservers, we can however see a lot of other Apache developers have signed his key, which looks promising, but do we know who they are?

Despite the lack of absolute certainty, when downloading from mirrors this is a lot better (and harder to exploit) than simply downloading without validating or checksumming at all, given that the signature and the KEYS file were downloaded from Apache’s own site.

You will need to decide for yourself how sure you need to be whether a person’s public key really corresponds to them. This might extend to the point of arranging to meet them with government-issued identification!

Encrypting a file

We can encrypt a file so that only nominated people can decrypt and read it. In this case, we encrypt it not with our own private key, but with the recipient’s public key. This means that they will be able to decrypt it using their own private key.

Here’s the contents of secret-message.txt:

This is a secret message from Tom Ryder.

Now we need at least one recipient. Let’s say this message was intended for my friend John Public. He’s given me his public key in a file called john-public.asc on a USB drive in person; he even brought along his birth certificate and driver’s license (which is weird, because I’ve known him since I was four).

To start with, I’ll import his key into my keychain:

$ gpg --import john-public.asc
gpg: key 695195A5: public key "John Public (Main key) <johnpublic@example.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)

Now I can encrypt the message for only John to read. I like to use the 8-digit hex code for the key for --recipient, to make sure I’ve got the right person. You can see it in the output above, or in the output of gpg --list-keys.

$ gpg --armor --recipient 695195A5 --encrypt secret-message.txt

The encrypted message is written to secret-message.txt.asc:

-----BEGIN PGP MESSAGE-----
Version: GnuPG v1.4.10 (GNU/Linux)

hQEMAxiBb8eWSupuAQgAgOUQvqbTh60N6RQhDtP/bY9l+gjm4Grx5XcuhgQqK6pn
YtyPTKcpHdPK679lhbv0vE0RYe7pL+nBOngU1hCQYuGbRDZDxIXTIZW/rBvXbtHA
jgeSxrquad2totfh2nc7upePVCqXncPrLraJyDJBLLMrBHVvmOZymDabJbemOFuq
A/NbcmT3+osptvaEPFdlbgAW+J3vGxXMuQQYkT8GSnuutfEhZRb7SEL1ktaXwaMc
AA6NAan5ak7nCyDDHhDSDFMS9SQQHd8TDvQPF6OzRXlq26EOFD8HvlbDcgc51lbS
+N5nWaHM/CiuPh9dIOEV0H4Y8WDBdgkxp6kXKQfqb9JzAdwQ047r82SJAA7MSqCS
HRVtCRf5SNM12HqTRzF9XXum4uG+HXT6Bpy+K/lYpLgmHcHoUVKh8c2OcGaCHWQh
UC9B+aaThKdkxUfD/9tVIRmugjutgj7KdtDTGm+qLeCoJqp6HK5z5SX8Ha+P6/P5
hxinyw==
=kqUG
-----END PGP MESSAGE-----

Note that even I can’t read it, because I didn’t list myself as a recipient, and I don’t have access to John’s private key:

tom@tombox:~$ gpg --decrypt secret-message.txt.asc
gpg: encrypted with 2048-bit RSA key, ID 964AEA6E, created 2013-03-10
    "John Public (Main key) <johnpublic@example.com>"
gpg: decryption failed: secret key not available

However, on John’s computer, using his private key, he can decrypt and read it:

john@johnbox:~$ gpg --decrypt secret-message.txt.asc
gpg: encrypted with 2048-bit RSA key, ID 964AEA6E, created 2013-03-10
    "John Public (Main key) <johnpublic@example.com>"
This is a private, secret message from Tom Ryder.

If I wanted to make sure I could read the message too, I’d add my own public key to identify myself as a recipient when I encrypt it. Then either of us will be able to read it with our private keys (independently of the other):

$ gpg --recipient 695195A5 --recipient 040FE79B \
    --armor --encrypt secret-message.txt

Just to be thorough, we can sign the message as well to prove it came from us:

$ gpg --recipient 695195A5 --recipient 040FE79B \
    --armor --sign --encrypt secret-message.txt

Then when John runs the --decrypt, gpg will automatically verify the signature for us too, provided he has my public key in his keyring:

$ gpg --decrypt secret-message.txt.asc
gpg: encrypted with 2048-bit RSA key, ID 964AEA6E, created 2013-03-10
    "John Public (Main key) <johnpublic@example.com>"
gpg: encrypted with 4096-bit RSA key, ID AA159E5B, created 2013-03-23
    "Tom Ryder (Test Key Only) <tom@sanctum.geek.nz>"
This is a private, secret message from Tom Ryder.
gpg: Signature made Sat 23 Mar 2013 17:23:20 NZDT using RSA key ID 040FE79B
gpg: Good signature from "Tom Ryder (Test Key Only) <tom@sanctum.geek.nz>"

These are all the basic functions of GnuPG that will be useful to most people. We haven’t considered here submitting keys to public servers, or participating in the web of trust; you should only look into this once you’re happy with how your key setup is working, and are ready to publish your key for public use.

You can install GnuPG with its gpg(1) frontend on Debian like so:

# apt-get install gnupg

You can do a lot of very cool things with GPG, but it boils down to four central ideas:

• Generation of keypairs, randomly-generated and mathematically linked pairs of files, one of which is kept permanently secret (the private key) and one of which is published (the public key). This is the basis of asymmetric key cryptography.
• Managing keys, both your own public and private key, along with other people’s public keys, so that you can verify others’ messages and files, or encrypt them so that only those people can read them. This might include publishing your public key to online keyservers, and getting people to sign it to confirm that the key is really yours.
• Signing files and messages with your private key to enable others to verify that a file or message was authored or sighted by you, and not edited in transmission over untrusted channels like the internet. The message itself remains readable to everybody.
• Encrypting files and messages with other people’s public keys, so that only those people can decrypt and read them with their private keys. You can also sign such messages with your own private key so that people can verify that it was sent by you.

We’ll run through the fundamentals of each of these. We won’t concern ourselves too much with the mathematics or algorithms behind these operations; the Wikipedia article for asymmetric key cryptography explains this very well for those curious for further details.

Generating a keypair

Let’s start by generating a 4096-bit RSA keypair, which should be more than sufficient for almost everyone at the time of writing. We’ll observe a few of the best practices recommended for the Debian developers.

Doing this on a private, up-to-date desktop machine is best, as it’s easier to generate entropy this way. It’s still possible on an SSH-only headless server, but you may have to resort to less cryptographically sound methods to generate proper randomness.

Create or edit the file ~/.gnupg/gpg.conf on your system, and add the following lines:

personal-digest-preferences SHA256
cert-digest-algo SHA256
default-preference-list SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAST5 ZLIB BZIP2 ZIP Uncompressed

These lines tell GnuPG to use the cryptographically stronger SHA256 hashing algorithm for signatures in preference to the long-broken SHA1 algorithm.

With that done, we can get down to generating some keys:

$ gpg --gen-key

You will be prompted to choose the type of keypair you want. The default ought to be RSA and RSA, which means we’ll generate one master key for signing, and one subkey for encryption:

Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
Your selection? 1

For the key length, choose the maximum 4096 bit RSA:

What keysize do you want? (2048) 4096
Requested keysize is 4096 bits

The expiry date is up to you; if you’re just tinkering with GnuPG for the moment, you should feel free to set a short expiry. However, if you’re in it for the long haul and you’re confident you can keep your key secure indefinitely, feel free to set it to never expire, as I’ll do here:

Please specify how long the key should be valid.
       0 = key does not expire
    <n>  = key expires in n days
    <n>w = key expires in n weeks
    <n>m = key expires in n months
    <n>y = key expires in n years
Key is valid for? (0) 0
Key does not expire at all
Is this correct? (y/N) y

Next, we’re prompted for some basic information to name the key. In almost all circumstances you should use your real name, as without a real-world means to actually verify your identity, public keys are much less useful long-term. For the comment, you can include the key’s purpose, or your public aliases, or any other information relevant to the key:

Real name: Tom Ryder
Email address: tom@sanctum.geek.nz
Comment: Test Key Only
You selected this USER-ID:
    "Tom Ryder (Test Key Only) <tom@sanctum.geek.nz>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O

Key passphrase

Next, we’re prompted for a passphrase to encrypt the key, so that if it ever falls into the wrong hands, nobody will be able to use it without knowing the passphrase.

You need a Passphrase to protect your secret key.

Choose a sequence of random words, or possibly a unique sentence you can easily memorise in any language, the longer the better. Don’t choose anything that might be feasibly guessable, like proverbs or movie quotes. You will also need to remember how you typed the passphrase exactly; I recommend using all-lowercase and no punctuation. Wikipedia has a few guidelines here.

You’ll need to type the passphrase twice to confirm it, and it won’t echo on your terminal, much as if you were typing a password.

Entropy generation

Finally, the system will prompt us to generate some entropy:

We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.

Not enough random bytes available.  Please do some other work to give
the OS a chance to collect more entropy! (Need 283 more bytes)

This step is necessary for the computer to generate sufficient random information to ensure that the private key being generated could not feasibly be reproduced. Moving the mouse around and using the keyboard on a desktop system is ideal, but generating any kind of hardware activity (including spinning disks up) should do the trick. Running expensive find(1) operations over a filesystem (with contents that couldn’t be reasonably predicted or guessed) helps too.

This step benefits from patience. You might find discussion online about forcing the use of the non-blocking PRNG random device /dev/urandom instead, using a tool like rngd(1). This definitely speeds up the process, but if you’re going to be using your key for anything serious, I recommend actually interacting with the computer and using hardware noise to seed the randomness adequately, if you can.

When adequate entropy is read and the key generation is done, you’ll be presented with some details for your master key and its signing subkey, and the private and public keys for each are automatically added to your keyring for use:

gpg: /home/tom/.gnupg/trustdb.gpg: trustdb created
gpg: key 040FE79B marked as ultimately trusted
public and secret key created and signed.
gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
pub   4096R/040FE79B 2013-03-23
    Key fingerprint = 7A28 5ADA 7680 6813 48DF  401B 6207 438A 040F E79B
uid                  Tom Ryder (Test Key Only) <tom@sanctum.geek.nz>
sub   4096R/AA159E5B 2013-03-23

Managing keys

With this done, we have our own keys added to the private and public keychain:

$ gpg --list-secret-keys
/home/tom/.gnupg/secring.gpg
----------------------------
sec   4096R/040FE79B 2013-03-23
uid                  Tom Ryder (Test Key Only) <tom@sanctum.geek.nz>
ssb   4096R/AA159E5B 2013-03-23

$ gpg --list-public-keys
/home/tom/.gnupg/pubring.gpg
----------------------------
pub   4096R/040FE79B 2013-03-23
uid                  Tom Ryder (Test Key Only) <tom@sanctum.geek.nz>
sub   4096R/AA159E5B 2013-03-23

The directory ~/.gnupg contains the managed keys. It’s very, very important to keep this directory private and to back it up securely, preferably to removable media that you keep in some physically secure place. Don’t lose it!

In most contexts in GnuPG, you can refer to a key by the name of its owner, or by its eight-digit hex ID. I prefer the latter method. Here, the short ID of my main key is 040FE79B. While you shouldn’t use this for any actual verification, it’s sufficiently unique that you can use it to identify a specific key on your keyring with which you want to work.

For example, if we want to provide someone with a copy of our public key, a friendly way to do so is to export it in ASCII format with --armor, providing the appropriate key’s short ID:

$ gpg --armor --export 040FE79B > tom-ryder.public.asc

While you can export private keys the same way with --export-secret-key, you should never, ever provide anyone with your private key, so this shouldn’t be necessary.

Revocation certificate

After generating your keys, you should generate a revocation certificate:

$ gpg --output revoke.asc --gen-revoke 040FE79B

sec  4096R/040FE79B 2013-03-23 Tom Ryder (Test Key Only) <tom@sanctum.geek.nz>

Create a revocation certificate for this key? (y/N) y
Please select the reason for the revocation:
0 = No reason specified
1 = Key has been compromised
2 = Key is superseded
3 = Key is no longer used
Q = Cancel
(Probably you want to select 1 here)
Your decision? 1
Enter an optional description; end it with an empty line:
>
Reason for revocation: Key has been compromised
(No description given)
Is this okay? (y/N) y

You need a passphrase to unlock the secret key for
user: "Tom Ryder (Test Key Only) <tom@sanctum.geek.nz>"
4096-bit RSA key, ID 040FE79B, created 2013-03-23

ASCII armored output forced.
Revocation certificate created.

Please move it to a medium which you can hide away; if Mallory gets
access to this certificate he can use it to make your key unusable.
It is smart to print this certificate and store it away, just in case
your media become unreadable.  But have some caution:  The print system of
your machine might store the data and make it available to others!

You should store the resulting revoke.asc file somewhere safe. You can use this certificate to revoke your key later on if the private key is ever compromised, so that people know the key should no longer be used or trusted. You may even like to print it out and keep a hard copy, as the output of gpg suggests.

With the above setup done, we can proceed with some basic usage of GnuPG, as discussed in the next article.

Subkeys

In the output of both commands, you’ll note we actually have two private and two public keys. The sub line refers to the encryption subkey automatically generated for you. The master key is used for cryptographic signing, and the subkey for encryption; this is how GnuPG does things by default with RSA keypairs.

For extra security, it might be appropriate to physically remove the master private key from your computer, and instead use a second generated subkey for signing files as well. This is desirable because it allows you to keep the master key secure on some removable media (preferably with a backup), and not loaded on your main computer in case you get compromised.

This means you can sign and encrypt files as normal with your signing subkey and encryption subkey. If those keys ever get compromised, you can simply revoke them and generate new ones with your uncompromised master key; everyone who has signed your public master key or otherwise indicated they trust it will not have to do that all over again.

For details on how to do this, I suggest reading the Debian Wiki article on subkey management. However, it’s not necessary for performing basic GPG operations.

I’ll be working on Debian GNU/Linux, but most of these tools should adapt well to other open source UNIX-likes, including BSD. Please feel free to comment on the articles with details relevant to your own implementations, or with extra security considerations for interested readers.

As a disclaimer, I’m not myself an expert on cryptographic algorithms or key security. If you are, and you find an error or security problem with any of my explanations or suggestions, please let me know and I will correct it and credit you.

I’ll be covering the following topics:

• GnuPG key generation and management
• GnuPG signing, verifying, encryption, and decryption
• SSH key generation and authentication
• gpg-agent(1), ssh-agent(1), and the use of keychain(1)
• The pass(1) UNIX password manager

If you already know about a specific topic, feel free to skip around through the other articles.

<a>  97,  Hex 61,  Octal 141

This information can be useful, but it’s worth it to extend it to include some other relevant information, including the Unicode point and name of the character, its HTML entity name (if applicable), and any digraph entry method. This can be done by installing the characterize plugin by Tim Pope.

With this plugin installed, pressing ga over a yields a bit more information:

<a> 97, \141, U+0061 LATIN SMALL LETTER A

This really shines however when inspecting characters that are available as HTML entities, or as Vim digraphs, particularly commonly used characters like an EM DASH:

<—> 8212, U+2014 EM DASH, ^K-M, &mdash;

Or a COPYRIGHT SYMBOL:

<©> 169, \251, U+00A9 COPYRIGHT SIGN, ^KCo, ^KcO, :copyright:, &copy;

Or as one of the eyes in a look of disapproval:

<ಠ> 3232, U+0CA0 KANNADA LETTER TTHA

Note that ga shows you all the Unicode information for the character, along with any methods to type it as a digraph, and an appropriate HTML entity if applicable.

If you work with multibyte characters a lot, whether for internationalization reasons or for typographical correctness in web pages, this may be very useful to you.

In the man page for tmux(1), the feature is described as follows, under the details for the resize-pane command:

With -Z, the active pane is toggled between zoomed (occupying the
whole of the window) and unzoomed (its normal position in the
layout).

This command is bound to <prefix> z by default; for most users, this will be Ctrl-a z. The effect can be observed by pressing this key sequence in any window with at least two panes, to toggle the zoomed state for the active pane:

Note the Z suffix that appears after the window title in the status bar while the pane is zoomed.

For most users, the new feature should mean that any custom maximize/minimize style bindings they may be using are no longer needed. This works particularly smoothly given that the new release also includes support for reflowing text when panes and windows are resized, something GNU Screen has supported for some time.

Be sure to take a look at some of the other changes in the newest release of tmux. If you’re using a DPKG or RPM based packaging system, you might like to build it from source and install it with checkinstall(8).

Newsbeuter refers to itself as “the Mutt of RSS readers”, alluding to its keystroke-driven ncurses(3) interface, plaintext configuration with many options, and extensive feature set. If you like the idea of using a client-side RSS reader in a terminal, then this may be ideal for you.

Having a client-side reader is particularly valuable if you follow feeds which aren’t available on the public internet, or if you would prefer to keep your subscriptions relatively private. While Google Reader’s search is very good, it’s also handy to have a local cache of feed items to search, which is a feature of Newsbeuter.

Installing Newsbeuter

Newsbeuter can be downloaded and built from source, or there are packages available in most Linux distributions. On Debian-derived systems, it’s available in the newsbeuter package:

# apt-get install newsbeuter

Newsbeuter will throw an error if you try to start it with no feeds defined. We’ll be fixing that shortly.

Exporting Google Reader feeds

If you’re using Google Reader, you should start by exporting your feeds in OPML format format using Google Takeout. You can do this by going to Reader Settings -> Import/Export -> Download your data through Takeout:

This leads you to the Google Takeout page, and offers you a download of all of your Google Reader data, which you can retrieve by clicking Create Archive. The downloaded zip file will contain (within a couple of directories) a file called subscriptions.xml. This is the OPML file containing the URLs and categorizations of all the feeds to which you were subscribed. Save that somewhere accessible on the Linux or BSD machine on which you intend to run Newsbeuter.

Importing feeds into Newsbeuter

Once you have your subscriptions.xml file ready for import, you can import the data straight into Newsbeuter using the -i option:

$ newsbeuter -i subscriptions.xml
Import of subscriptions finished.

With this done, you should be able to start Newsbeuter with no options, and its main interface will start with the URLs to all your feeds:

You’ll note that none of these have any items yet; this is because the defaults for Newsbeuter are to fetch the articles only on demand, not automatically. You can start this process by pressing R for Reload All, at which point the titles of your feeds will appear, along with a count of their unread items:

Some useful keystrokes

From here, the basics are pretty intuitive; you can move around with the cursor keys, and select feeds and items within them with Enter. You can press q to move up a screen, and to quit the program; Q will quit unconditionally from any screen.

You can move to next and previous feed items with Ctrl-j and Ctrl-k. A nice quick way to read everything is to cycle through unread items across all feeds with n. You can save the complete text of an article with s, and search for articles matching a string (not a regular expression) with /.

You can press o to open the feed’s URL in a browser; this works fine if you’re using an X server, but you can also configure this to be a command-line browser like lynx if you’d prefer with the browser option in the configuration file. If you’re using PuTTY and you’re going to be copy-pasting URLs from your terminal window, it helps to make sure you’ve configured it to easily select URLs on double-click.

A complete list of all the keystrokes is available by pressing ?.

Managing feeds

Adding, removing, and tagging feeds is all done with the urls file. This might be saved in either ~/.config/newsbeuter/urls, or ~/.newsbeuter/urls. Either way, you can edit it directly within the program using E, which will start your $EDITOR to manage the URLs. Add and remove feed URLs, save the file, quit, and you’re done; Newsbeuter will reload its defined feeds automatically once the editor is closed.

Tags

If you imported your feeds from Google Reader and you were using folders to keep your feeds organised, you may note that in your urls file in Newsbeuter the names of the folders are included in quotes at the end of each line:

http://www.debian-administration.org/atom.xml "Tech"
http://www.jerkcity.com/jc.rss "Comics"
http://www.kiwiblog.co.nz/feed "Politics"

These are tags, Newsbeuter’s way of organising feeds non-hierarchically. If you have such tags defined, you can limit your view of feeds to a particular tag by pressing t to show only those matching feeds. You can press Ctrl-t to back out of that view and show all feeds again.

Creating a new tag is done by editing the urls file as above. Add the tag in quotes after the appropriate feed URLs. Note that you can have more than one tag for each URL:

http://www.debian-administration.org/atom.xml "Tech" "Debian"

Configuration

The Newsbeuter configuration file might be in either ~/.config/newsbeuter/config or ~/.newsbeuter/config. The following options might be useful:

• auto-reload yes — Check all feeds for new items on startup, and periodically thereafter.
• reload-time 30 — Re-check all feeds automatically every 30 minutes.
• notify-beep yes — Send a console beep every time new items are found. You will probably only want this if you are dealing sensibly with bells, for example with a visual bell system in GNU Screen or tmux, otherwise you may find an audible bell annoying.
• confirm-exit yes — Prompt before quitting. Tapping q to get to the top screen is a little error-prone, and it’s easy to quit accidentally.

The colorscheme for the application can also be customized here, and the keybindings too. See the Newsbeuter documentation for a complete list of configuration options.

User agents

You may find that some feeds don’t return any information when you use Newsbeuter, probably because the user agent string it sends is not recognised as an RSS reader. The feed for the Abstruse Goose comic is an example, as is Toothpaste for Dinner.

The easiest way to work around this is to make Newsbeuter identify itself as a better-known RSS reader. I’ve found that pretending to be Liferea works:

user-agent "Liferea/1.4.14 (Linux; en_US.UTF8; http://liferea.sf.net/)"

With this done and Newsbeuter restarted, the feeds seem more willing to yield their items for reading.

Daniel Aleksandersen points out in the comments that this is probably because Newsbeuter used a suspicious user agent string until his patch for 2.6. If you are using Newsbeuter 2.6 or newer, then you may not need to do the above.

Special feeds

If you can’t directly retrieve your feed from a URL, but need to generate it programatically from a script or use a tool like curl to retrieve it, you can use special exec: URLs in the urls file to manage this. For example, to retrieve an RSS feed of my work’s network changelist, I do something like this:

"exec:ssh work curl http://changelog.worknet/rss.xml"

This retrieves the feed using curl(1) over ssh(1), and presents it as a normal feed in Newsbeuter. Note the quotes are required for any command that includes spaces.

Though I will miss Google Reader, I’ve found Newsbeuter a great replacement, and it fits very nicely as a permanent window in my tmux(1) session. Hopefully you’ll find it suits you too, and works well with your terminal-based workflow.

$ TERM=xterm-256color

This misinformation sometimes extends to suggesting that users put the forced TERM change into their shell startup scripts. The reason this is such a bad idea is that it forces your shell to assume what your terminal is, and thereby disregards the initial terminal identity string sent by the emulator. This leads to a lot of confusion when one day you need to connect with a very different terminal emulator.

Accounting for differences

All terminal emulators are not created equal. Certainly, not all of them are xterm(1), although many other terminal emulators do a decent but not comprehensive job of copying it. The value of the TERM environment variable is used by the system running the shell to determine what the terminal connecting to it can and cannot do, what control codes to send to the program to use those features, and how the shell should understand the input of certain key codes, such as the Home and End keys. These things in particular are common causes of frustration for new users who turn out to be using a forced TERM string.

Instead, focus on these two guidelines for setting TERM:

1. Avoid setting TERM from within the shell, especially in your startup scripts like .bashrc or .bash_profile. If that ever seems like the answer, then you are probably asking the wrong question! The terminal identification string should always be sent by the terminal emulator you are using; if you do need to change it, then change it in the settings for the emulator.

2. Always use an appropriate TERM string that accurately describes what your choice of terminal emulator can and cannot display. Don’t make an rxvt(1) terminal identify itself as xterm; don’t make a linux console identify itself as vt100; and don’t make an xterm(1) compiled without 256 color support refer to itself as xterm-256color.

In particular, note that sometimes for compatibility reasons, the default terminal identification used by an emulator is given as something generic like xterm, when in fact a more accurate or comprehensive terminal identity file is more than likely available for your particular choice of terminal emulator with a little searching.

An example that surprises a lot of people is the availability of the putty terminal identity file, when the application defaults to presenting itself as an imperfect xterm(1) emulator.

Configuring your emulator’s string

Before you change your terminal string in its settings, check whether the default it uses is already the correct one, with one of these:

$ echo $TERM
$ tset -q

Most builds of rxvt(1), for example, should already use the correct TERM string by default, such as rxvt-unicode-256color for builds with 256 colors and Unicode support.

Where to configure which TERM string your terminal uses will vary depending on the application. For xterm(1), your .Xresources file should contain a definition like the below:

XTerm*termName: xterm-256color

For rxvt(1), the syntax is similar:

URxvt*termName: rxvt-unicode-256color

Other GTK and Qt emulators sometimes include the setting somewhere in their preferences. Look for mentions of xterm, a common fallback default.

For Windows PuTTY, it’s configurable under the ”’Connections > Data”’ section:

More detail about configuring PuTTY for connecting to modern systems can be found in my article on configuring PuTTY.

Testing your TERM string

On Linux systems, an easy way to test the terminal capabilities (particularly effects like colors and reverse video) is using the msgcat(1) utility:

$ msgcat --color=test

This will output a large number of tests of various features to the terminal, so that you can check their appearance is what you expect.

Finding appropriate terminfo(5) definitions

On Linux systems, the capabilities and behavior of various terminal types is described using terminfo(5) files, usually installed as part of the ncurses package. These files are often installed in /lib/terminfo or /usr/share/terminfo, in subdirectories by first letter.

In order to use a particular TERM string, an appropriate file must exist in one of these directories. On Debian-derived systems, a large collection of terminal types can be installed to the system with the ncurses-term package.

For example, the following variants of the rxvt terminal emulator are all available:

$ cd /usr/share/terminfo/r
$ ls rxvt*
rxvt-16color  rxvt-256color  rxvt-88color  rxvt-color  rxvt-cygwin
rxvt-cygwin-native  rxvt+pcfkeys  rxvt-unicode-256color  rxvt-xpm 

Private and custom terminfo(5) files

If you connect to a system that doesn’t have a terminfo(5) definition to match the TERM definition for your particular terminal, you might get a message similar to this on login:

setterm: rxvt-unicode-256color: unknown terminal type
tput: unknown terminal "rxvt-unicode-256color"
$

If you’re not able to install the appropriate terminal definition system-wide, one technique is to use a private .terminfo directory in your home directory containing the definitions you need:

$ cd ~/.terminfo
$ find
.
./x
./x/xterm-256color
./x/xterm
./r
./r/rxvt-256color
./r/rxvt-unicode-256color
./r/rxvt
./s
./s/screen
./s/screen-256color
./p
./p/putty-256color
./p/putty

You can copy this to your home directory on the servers you manage with a tool like scp:

$ scp -r .terminfo server:

TERM and multiplexers

Terminal multiplexers like screen(1) and tmux(1) are special cases, and they cause perhaps the most confusion to people when inaccurate TERM strings are used. The tmux FAQ even opens by saying that most of the display problems reported by people are due to incorrect TERM settings, and a good portion of the codebase in both multiplexers is dedicated to negotiating the differences between terminal capacities.

This is because they are “terminals within terminals”, and provide their own functionality only within the bounds of what the outer terminal can do. In addition to this, they have their own type for terminals within them; both of them use screen and its variants, such as screen-256color.

It’s therefore very important to check that both the outer and inner definitions for TERM are correct. In .screenrc it usually suffices to use a line like the following:

term screen

Or in .tmux.conf:

set-option -g default-terminal screen

If the outer terminals you use consistently have 256 color capabilities, you may choose to use the screen-256color variant instead.

If you follow all of these guidelines, your terminal experience will be much smoother, as your terminal and your system will understand each other that much better. You may find that this fixes a lot of struggles with interactive tools like vim(1), for one thing, because if the application is able to divine things like the available color space directly from terminal information files, it saves you from having to include nasty hacks on the t_Co variable in your .vimrc.

While it’s very popular and useful, PuTTY’s defaults are quite old, and are chosen for compatibility reasons rather than to take advantage of all the features of a more complete terminal emulator. For new users, this is likely an advantage as it can avoid confusion, but more advanced users who need to use a Windows client to connect to a modern Linux system may find the defaults frustrating, particularly when connecting to a more capable and custom-configured server.

Here are a few of the problems with the default configuration:

• It identifies itself as an xterm(1), when terminfo(5) definitions are available named putty and putty-256color, which more precisely define what the terminal can and cannot do, and their various custom escape sequences.
• It only allows 16 colors, where most modern terminals are capable of using 256; this is partly tied into the terminal type definition.
• It doesn’t use UTF-8 by default, which should be used whenever possible for reasons of interoperability and compatibility, and is well-supported by modern locale definitions on Linux.
• It uses Courier New, a workable but rather harsh monospace font, which should be swapped out for something more modern if available.
• It uses audible terminal bells, which tend to be annoying.
• Its default palette based on xterm(1) is rather garish and harsh; softer colors are more pleasant to read.

All of these things are fixable.

Terminal type

Usually the most important thing in getting a terminal working smoothly is to make sure it identifies itself correctly to the machine to which it’s connecting, using an appropriate $TERM string. By default, PuTTY identifies itself as an xterm(1) terminal emulator, which most systems will support.

However, there’s a terminfo(5) definition for putty and putty-256color available as part of ncurses, and if you have it available on your system then you should use it, as it slightly more precisely describes the features available to PuTTY as a terminal emulator.

You can check that you have the appropriate terminfo(5) definition installed by looking in /usr/share/terminfo/p:

$ ls -1 /usr/share/terminfo/p/putty*
/usr/share/terminfo/p/putty  
/usr/share/terminfo/p/putty-256color  
/usr/share/terminfo/p/putty-sco  
/usr/share/terminfo/p/putty-vt100

On Debian and Ubuntu systems, these files can be installed with:

# apt-get install ncurses-term

If you can’t install the files via your system’s package manager, you can also keep a private repository of terminfo(5) files in your home directory, in a directory called .terminfo:

$ ls -1 $HOME/.terminfo/p
putty
putty-256color

Once you have this definition installed, you can instruct PuTTY to identify with that $TERM string in the Connection > Data section:

Here, I’ve used putty-256color; if you don’t need or want a 256 color terminal you could just use putty.

Once connected, make sure that your $TERM string matches what you specified, and hasn’t been mangled by any of your shell or terminal configurations:

$ echo $TERM
putty-256color

Color space

Certain command line applications like Vim and Tmux can take advantage of a full 256 colors in the terminal. If you’d like to use this, set PuTTY’s $TERM string to putty-256color as outlined above, and select Allow terminal to use xterm 256-colour mode in Window > Colours:

You can test this is working by using a 256 color application, or by trying out the terminal colours directly in your shell using tput:

$ for color in $(seq 0 255); do
> tput setaf $color && echo -n "test"
> done

If you see the word test in many different colors, then things are probably working. Type reset to fix your terminal after this:

$ reset

Using UTF-8

If you’re connecting to a modern GNU/Linux system, it’s likely that you’re using a UTF-8 locale. You can check which one by typing locale. In my case, I’m using the en_NZ locale with UTF-8 character encoding:

$ locale
LANG=en_NZ.UTF-8
LANGUAGE=en_NZ:en
LC_CTYPE="en_NZ.UTF-8"
LC_NUMERIC="en_NZ.UTF-8"
LC_TIME="en_NZ.UTF-8"
LC_COLLATE="en_NZ.UTF-8"
LC_MONETARY="en_NZ.UTF-8"
LC_MESSAGES="en_NZ.UTF-8"
LC_PAPER="en_NZ.UTF-8"
LC_NAME="en_NZ.UTF-8"
LC_ADDRESS="en_NZ.UTF-8"
LC_TELEPHONE="en_NZ.UTF-8"
LC_MEASUREMENT="en_NZ.UTF-8"
LC_IDENTIFICATION="en_NZ.UTF-8"
LC_ALL=

If the output of locale does show you’re using a UTF-8 character encoding, then you should configure PuTTY to interpret terminal output using that character set; it can’t detect it automatically (which isn’t PuTTY’s fault; it’s a known hard problem). You do this in the Window > Translation section:

While you’re in this section, it’s best to choose the Use Unicode line drawing code points option as well. Line-drawing characters are most likely to work properly with this setting for UTF-8 locales and modern fonts:

If Unicode and its various encodings is new to you, I highly recommend Joel Spolsky’s classic article about what programmers should know about both.

Fonts

Courier New is a workable monospace font, but modern Windows systems include Consolas, a much nicer terminal font. You can change this in the Window > Appearance section:

There’s no reason you can’t use another favourite Bitmap or TrueType font instead once it’s installed on your system; DejaVu Sans Mono, Inconsolata, and Terminus are popular alternatives. I personally favor Ubuntu Mono.

Bells

Terminal bells by default in PuTTY emit the system alert sound. Most people find this annoying; some sort of visual bell tends to be much better if you want to use the bell at all. Configure this in Terminal > Bell:

Given the purpose of the alert is to draw attention to the window, I find that using a flashing taskbar icon works well; I use this to draw my attention to my prompt being displayed after a long task completes, or if someone mentions my name or directly messages me in irssi(1).

Another option is using the Visual bell (flash window) option, but I personally find this even worse than the audible bell.

Default palette

The default colours for PuTTY are rather like those used in xterm(1), and hence rather harsh, particularly if you’re used to the slightly more subdued colorscheme of terminal emulators like gnome-terminal(1), or have customized your palette to something like Solarized.

If you have decimal RGB values for the colours you’d prefer to use, you can enter those in the Window > Colours section, making sure that Use system colours and Attempt to use logical palettes are unchecked:

There are a few other default annoyances in PuTTY, but the above are the ones that seem to annoy advanced users most frequently. Dag Wieers has a similar post with a few more defaults to fix.

Since these restricted networks almost always allow port 443 out, since it’s the destination port for outgoing HTTPS requests, an easy workaround is to have your OpenSSH server listen on port 443 if it isn’t already using the port.

This is sometimes given as a rationale for changing the sshd port completely, but you don’t need to do that; you can simply add another Port directive to sshd_config(5):

Port 22
Port 443

After restarting the OpenSSH server with this new line in place, you can verify that it’s listening with ss(8) or netstat(8)

# ss -lnp src :22
State      Recv-Q Send-Q    Local Address:Port      Peer Address:Port
LISTEN     0      128                  :::22                  :::*
users:(("sshd",3039,6))
LISTEN     0      128                   *:22                   *:*
users:(("sshd",3039,5))
# ss -lnp src :443
State      Recv-Q Send-Q    Local Address:Port      Peer Address:Port
LISTEN     0      128                  :::443                 :::*
users:(("sshd",3039,4))
LISTEN     0      128                   *:443                  *:*
users:(("sshd",3039,3))

You’ll then be able to connect to the server on port 443, the same way you would on port 22. If you intend this setup to be permanent, it would be a good idea to save the configuration in your ssh_config(5) file, or whichever SSH client you happen to use.

• Letters with diacritical marks like ä, é, and ô — Vim refers to these as digraphs — a particular problem when using a US keyboard layout
• Unicode characters like typographic dashes — or copyright symbols ©, or other symbols from the multi-byte portion of the UTF-8 character set (including foreign languages)
• Literal control characters like <Tab>

Vim has a method for inserting each of these within the editor, rather than having to copy-paste them from another document. We won’t discuss Vim’s alternative multibyte input methods here, and will assume that you’re using a keyboard with a US or UK layout, and predominantly type in English – apologies to international readers, but I do not have another type of keyboard to test this out!

Some of the following assumes that you’re using Vim in a UTF-8 capable terminal, and with the encoding option in your .vimrc set to utf-8, which is highly recommended for the vast majority of editing requirements:

set encoding=utf-8

It also assumes that your font is capable of displaying all of the characters concerned; monospace fonts with workable symbol coverage include Consolas, Inconsolata, and Ubuntu Mono.

Digraphs

Vim has a special shorthand for entering characters with diacritical marks. If you need some familiar variant of a Latin alphabet character with a diacritical mark or embellishment, it’s likely you’ll be able to input it with the digraph system. It also has support for some other sometimes-needed characters like thorn Þ and eszett ß, and Cyrillic characters.

Digraph input is started in insert or command mode (but not normal mode) by pressing Ctrl-k, then two printable characters in succession; the first is often the “base” form of the letter, and the second denotes the appropriate embellishment.

Some simple examples that might occasionally be needed for English speakers to correctly type one of the language’s many “loan words”:

• Ctrl-k c , -> ç
• Ctrl-k e ' -> é
• Ctrl-k o ^ -> ô
• Ctrl-k a ! -> à
• Ctrl-k u : -> ü
• Ctrl-k = e -> €

This is just a small sample; Vim has support for a great many digraphs. Take a look at the relevant section of the documentation for a complete treatment of the feature. You can also type :digraphs within Vim to get a complete list of digraphs — several screenfuls of them!

Note that you can enter all of these characters using the Unicode mode discussed later in this article as well; two-character mnemonic digraphs simply happen to be easier to remember than four-digit codes.

Unicode characters

For characters not covered in the digraph set, you can also enter unicode characters by referring to their code page number. In insert or command mode (but not normal mode) this is done by typing Ctrl-v and then u, followed by the hexadecimal number. Some potentially useful examples:

• Ctrl-v u 2018 -> ‘, a LEFT SINGLE QUOTATION MARK
• Ctrl-v u 2019 -> ’, a RIGHT SINGLE QUOTATION MARK
• Ctrl-v u 2014 -> —, an EM DASH
• Ctrl-v u 00a9 -> ©, a COPYRIGHT SIGN

These are handy in some cases when writing HTML documents, as an alternative to using HTML entities like — or ©. An exhaustive summary of these characters and their codes is available on the Unicode website.

Other non-printable characters

The unicode character input method is actually a specialised case of inputting literal characters with a Ctrl-v prefix. We can input other non-printable and control characters using this prefix:

• Ctrl-v <Enter> -> ^M
• Ctrl-v <Tab> -> ^I

This is sometimes handy when conforming to someone else’s tab style, and can also be handy when searching for characters literally in searches.