Ricard Torres dev

Scaling Mastodon: moving media assets to Object Storage

Ricard Torres — Mon, 19 Dec 2022 07:10:59 +0000

When I installed my private personal Mastodon instance y quickly noticed Mastodon has storage issues, so I tried to improve it by setting up a bunch of cron jobs to purge cached files.

All the commands used are out of the box, no hacking involved but it's something that's not really highlighted in the installation guides and it's quite important.

I've been battling with a steady reduction of disk space for the last month. Not terrible but not great.

After reading a lot from other users that the actual costs of moving the cached files with a CDN in front were so low I deiced to make the jump. External object storage provider was the next step.

Why?

Disk space, obviously 😄 The reality is that VPS providers don't give you that much disk space. My VPS provider Linode, offers great shared CPU plans such as:

5$ for 1 CPU, 1GB of RAM and 25 GB of disk space

It's quite cheap, if you ask me 💜

But that's not enough for Mastodon's media usage. Nowadays the space should be cheap, and it actually is, but you have to use object storage. Regular SSD disk space was not meant for this use case.

Linode also offers object storage, nice. In fact (at the time of the writing) they offer 250GB for only 5$ extra. Five dollars is not much but it's a flat fee 😢 If you go with AWS or BackBlaze the price starts from 0$ 🤩 Unless you already have two hundred GB of media, you're probably better off with a dynamically priced provider to start with just like I did 😉

What's my setup?

BackBlaze: the object storage provider with lower prices than Amazon's famous S3 but it has a compatible S3 API (be mindful when creating an account, select the correct region US vs EU). It has partnership with Cloudflare for very cheap close to free CDN. You need however to move your domain's DNS to Cloudflare if you want to use their CDN.
CloudFront from Amazon Web Services: I didn't move my domain's DNS to Cloudflare so I used AWS. It has an absurdly high free tier which I can't possibly make use of. My CDN forecasted usage is less than 2% of the free tier 🤭
AWS CLI: I've used to do the initial sync between my disk and the S3 bucket in BackBlaze. It takes a while, sit tight and grab a coffee.

Configuring your .env.production file

From your Mastodon folder ~/live/.env.production

Here's what I had to add, customize the values based on your S3 and URL's:

S3_ENABLED=true
S3_PROTOCOL=https
S3_ENDPOINT=https://XXXXX.backblazeb2.com
S3_HOSTNAME=XXXX.backblazeb2.com
S3_BUCKET=XXXX
AWS_ACCESS_KEY_ID=XXXX
AWS_SECRET_ACCESS_KEY=XXXX
S3_ALIAS_HOST=media.ricard.social
Code language: Bash (bash)

What's the Mastodon retention policy?

I've set up the purging of attachments and previews to 4 days running every night.

Before doing the S3 migration I had it to one day but considering how cheap it is, seven feels like a nice number.

How cheap is the whole thing?

I'm honestly surprised how low the cost of the whole move is. For the past 2 weeks:

BackBlaze S3 bucket:
- 💾 Disk usage (38.7 GB): 0.04€
- 📈 Transactions: 0.08€
- 📉 Downloaded GB: 0€
- 💰 Cost: 0.13€
Amazon Web Services CloudFront CDN:
- 📈 Transactions: 1% from the free tier
- 📉 Downloaded GB: 1.2% from the free tier
- 💰 Monthly cost: free

Conclusion

There is no doubt a move to object storage should be your default if you attempt to host a Mastodon instance. It doesn't matter if for a private single-user, 10 users, or 10.000 users. The cost of hosting the media files in a S3-compatible bucket will be far cheaper than locally on the same server.

Have you moved your files to an object storage? Have you encountered any issues?

Any other cloud storage provider you recommend?

The post Scaling Mastodon: moving media assets to Object Storage appeared first on Ricard Torres dev.

How to add web mentions to your WordPress site

Ricard Torres — Tue, 29 Nov 2022 07:39:29 +0000

A Webmention enables enables cross-site conversations. Letting site A know that site B is mentioning them (be it via link, a like, a re-post... there are many times of mentions).

One of the cool use cases is to link your Fediverse account (Mastodon, in my case) with your blog. So every time you share a link from your blog and people react to it, these reactions (comments, or likes) will show up as native comments in WordPress. Pretty cool, huh ? 😁

Requirement: Enable pingbacks

There are 2 things you should enable in Settings > Discussion

Attempt to notify any blogs linked to from the post
Allow link notifications from other blogs (pingbacks and trackbacks) on new posts

Otherwise even if webmentions hit your site, you wouldn't see them.

Requirement: Install the Webmention plugin

Webmention

Requirement: Bridgy

This free service is used to ping your site when, in my case my Mastodon account, makes a post linking to one of my articles. The likes and mentions from that initial post on Mastodon will be pinged back to my WordPress site.

https://brid.gy/

Requirement: Install Semantic-Linkbacks

It modifies the way your comments are printed, adding the mentions, likes in a nice way. It includes CSS and JS for these rich features.

Semantic-Linkbacks

Conclusion

It might need a little tweaking to show the comments to your liking, if the out of the box doesn't please you. A part from that, quite straightforward.

The post How to add web mentions to your WordPress site appeared first on Ricard Torres dev.

Improving Mastodon’s disk usage

Ricard Torres — Tue, 08 Nov 2022 13:44:24 +0000

Mastodon's built-in CLI gives you the availability to clean attachments and previews from remote accounts, purging the disk cache. This is fantastic and you couldn't possible survive with out it.

My current crontab looks something like this:

0 3 * * * RAILS_ENV=production /home/mastodon/live/bin/tootctl media remove --days 1
0 4 * * * RAILS_ENV=production /home/mastodon/live/bin/tootctl preview_cards remove --days 1
0 5 * * * RAILS_ENV=production /home/mastodon/live/bin/tootctl statuses remove --days 1
Code language: Bash (bash)

I basically run these 3 jobs at 3AM, 4AM and 5AM every day to keep the single-user instance thin. It works, the size is kept down to something like this.

Attachments:	494 MB (0 Bytes local)
Custom emoji:	43.4 MB (0 Bytes local)
Preview cards:	13.3 MB
Code language: plaintext (plaintext)

Q: Why do you run the commands cleaning 1 day old media?
A: My server doesn't have much space and I want to keep it as lean as possible.

How about avatars and headers?

Here's where things get dissapointing, a GitHub issue from 2018 outlines the problem of caching avatars and headers for the the users from outside your instance. Yes, it makes sense that these are downloaded to your instance but it doesn't make sense that are kept forever (?).

There should be a way to purge them, to check if these are accounts you're interacting with or not. In a way you have:

tootctl accounts refresh --all

Which does:

Refetch remote user data and files for one or multiple accounts.

But this is not good enough ❌

You have yet another command at your disposal:

tootctl accounts cull

Remove remote accounts that no longer exist. Queries every single remote account in the database to determine if it still exists on the origin server, and if it doesn't, then remove it from the database. Accounts that have had confirmed activity within the last week are excluded from the checks, in case the server is just down.

Again, this is not good enough ❌

I refuse to believe that after running these commands I'm still stuck with:

Avatars:	920 MB (61.3 KB local)
Headers:	1.95 GB (101 KB local)
Code language: plaintext (plaintext)

I couldn't care less for user headers 🤷‍♂️

💡Idea: replace all headers with 1x1 pixel

Update: I've ended up simply removing all the files inside the accounts/headers folder. Nothing seems to break and it's just more fast and clean ✅
Edited: November 14th 2022

What if we replace all header files (jpg, png, jpeg, webp...) with the smallest image file possible? A 1x1 pixel file.

Yes, you would still have thousands of files but taking way less space.

Well, that's exactly what I did. A small bash script (I'm no expert!) that loops through all the image files inside the cache/headers folder and replaces them with a pixel.

#!/bin/bash

for file in $(find /home/mastodon/live/public/system/cache/accounts/headers -type f \( -iname \*.jpg -o -iname \*.jpeg -o -iname \*.png -o -iname \*.webp \) -type f); do
  rm "$file"

  if [ "${file: -5}" == ".jpeg" ]
  then
    echo "This is a JPEG! $file"
    SOURCE="pixel.jpeg"
  fi

  if [ "${file: -4}" == ".jpg" ]
  then
    echo "This is a JPG! $file"
    SOURCE="pixel.jpg"
  fi

  if [ "${file: -4}" == ".png" ]
  then
    echo "This is a PNG! $file"
    SOURCE="pixel.png"
  fi

  if [ "${file: -5}" == ".webp" ]
  then
    echo "This is a webP! $file"
    SOURCE="pixel.webp"
  fi

  cp "$SOURCE" "$file"
done
Code language: Bash (bash)

GitHub repo with the code ready for you to checkout

Did it work?

Yes! The headers folder has considerably reduced (for a single-user instance):

2GB ➡️ 250MB

Nothing seems to be broken. The user profiles load, the images load, the only thing is that they're the replaced 1 pixel images instead of the originally cached header.

⚠️ The only thing is that Mastodon's built in usage check still thinks I have 2GB in headers. My guess is the size of the images must be stored in the database (?)

tootctl media usage

💡 Re-compress instead of replace

We can also re-compress the files, if you don't want to delete the headers or avatars. Personally I run the following on the avatars cache:

find -name '*.jpg' -print0 | xargs -0 jpegoptim --verbose --preserve --threshold=1 --max=45
find -name '*.jpeg' -print0 | xargs -0 jpegoptim --verbose --preserve --threshold=1 --max=45
find -name '*.png' -print0 | xargs -0 pngquant --verbose --ext=.png --force --speed 10 --quality 45-50 --skip-if-larger
Code language: JavaScript (javascript)

Conclusion

Keeping the size of a single-user Mastodon is not trivial.

Had I known this before getting started I would have probably installed a Pleroma or Akkoma instance instead. Which are way more lightweight. Granted the UI is not as good (if you want multi-column layout) but maybe you don't need all Mastodon's features. I am currently too invested to switch but I would highly encourage you to check out Pleroma and Akkoma (a fork) before installing Mastodon.

Do you have any other tips on how to keep a Mastodon instance lean?

⚠️ Don't miss the follow up post: Scaling Mastodon: moving media assets to Object Storage

The post Improving Mastodon’s disk usage appeared first on Ricard Torres dev.

Hosting my own Mastodon instance

Ricard Torres — Sun, 23 Oct 2022 10:34:18 +0000

⚠️ Read this before deep diving

I've been dealing with Mastodon's high disk usage for the last week. Have a read before installing your own Mastodon instance. I would recommend checking out Pleroma and Akkoma before taking a decision. They have way less system requirements than Mastodon.
EDIT November 8th

Self-hosting

That's right, I know have ricard.social as my Mastodon home. Here's how this went down...

(I will assume that you're at least a little bit familiar with Mastodon and the Fediverse. If not, have a quick read before diving into my post 🤓)

🛑 mastodon.technology is shutting down

This great instance of 24.000 users is shutting down. The owner can't maintain it anymore, but that is fine! The beauty of the Fediverse is that you can move somewhere else, take your followers and continue sharing, reading, communicating...

At this point I thought I'd try to host my own Mastodon instance. I did try in the past and failed 🫠 I had hopes now it would be different. Lucky me, it was 🍀

😁 I'm proud to say that all the setup was done from my hotel room, in Busan 🇰🇷 (the big city at the boom of South Korea) ✈️ 👨‍💻

🤷‍♂️ Why host your own instance?

Because we can. Unlike with Twitter.
Complete control over what you see, who can talk to you, etc. You could block entire instances, mute people and so on with a regular account in someone else's instance. Having your own just gives you more super powers.
Hopefully by self-hosting I won't have to move again.
Self-hosting gives you extra authority. As anybody could create an account with your username in any instance, by having my own instance with my domain, it's easier to know who is behind this account.

👨‍💻 Was it hard to have it up and running?

A little bit, yes. I'm not a black belt in system engineering but I know my way around NGINX and basic Ubuntu administration.

The official documentation is good. A great step by step that in most cases should be enough. It wasn't for me.
The official mastodon forum is helpful as many people post there issues and you end up there after googling your error.
Still, I ended up in a state in which I saw no error and didn't know what to do. I slept on it and finally understood I had low memory issues.
I had to double my server capacity to 2GB RAM. Truth is, it already was at its limit. I'm self hosting a dozen sites and services. So adding a Mastodon instance on top was too much for it. Even tring to add swap space wasn't possible as I was at near full disk.

As soon as I upgraded it all worked out.

I'm very happy with the process and the end result. I hope to be more engaged on Mastodon now. I mean, now I really have to use it 😁

Do you usually self-host services?

Have you tried self-hosting a Mastodon instance?

The post Hosting my own Mastodon instance appeared first on Ricard Torres dev.

Migrating from Gatsby to custom Node.js script

Ricard Torres — Tue, 30 Aug 2022 08:00:00 +0000

Two years ago I created a small site to track what movies and shows I watch. I did it with Gatbsy because well, it was a new thing for me and I wanted to experiment with Jamstack.

The idea is to create a new Markdown file for each movie or tv episode watched. A TV show season has 10 episodes, then I have 10 files. Using GraphQL I would query the files and show those on the site. Neat!

Using GitHub Actions, of course, I would simply build the Gatbsy site and send it to GitHub Pages. No issues. Life was good.

The site never needed much update. It was working fine. Until I noticed it wasn't that fast, it was taking a while to build, around 2 minutes on GH Actions. The naive version of me thought: "oh; i'll try to upgrade the NPM dependencies see if the newer versions speed things up" 🤭

🤦‍♂️ Dependency hell

You might have experienced this. The pain of trying to upgrade an old build system, the deprecated packages, the incompatibilities of the old versions with the newer and more secure versions of Node and NPM.

It was painful. Nothing worked anymore. Gatsby would no longer compile of course. Upgrading to breaking change versions was bound to break stuff. I did try to fix but I eventually gave up.

I'm just fed up with this nonsense. Having dozens of packages in your package.json is a recipe for headache if you're not constantly upgrading it, checking the changelogs, etc. For work projects it's fine, but for a side project with 0 maintenance... no good.

🤔 What's the alternative?

Custom Node script with just 1 dependency. It feels good, I just feel clean! 🥰

That's right. The only thing I really needed for this site is a way to convert Markdown files into HTML. Well, there's just a package to do that. A single one, your package.json will never look better.

Have a look at markdown-it/markdown-it if you're interested.

const MarkdownIt = require('markdown-it');

const md = new MarkdownIt();

const result = md.render('# markdown-it rulezz!');
Code language: JavaScript (javascript)

🤓 Yeah, yeah... but how?

Using what Node already gives you functions to read folders, to read files and to write files. What else do you need?

Sure, you won't have all the featues Gatbsy gives you out of the box, that's obvious. But I don't really needed them for this site anyway. This is a very boiled down version but has all the essentials:

Read a folder to get all the files in it.
Read each of the files inside a folder.
Process the Markdown into HTML.
Save a new HTML file with the processed content.

const movies = await fs.promises.readdir(`my/folder`);

movies.forEach((movie) => {
  const movieContent = fs.readFileSync(`source/folder/${movie}/index.md`, { encoding: 'utf8', flag: 'r' });

  const md = new MarkdownIt();
  
  const html = md.render(movieContent);
  
  fs.writeFileSync(`new-file-html`, html);
});
Code language: JavaScript (javascript)

The final main.js is surely a little bit more complex but that's because I wanted to add the extra features. Have a look here.

Final thoughts

Running through close to 6.000 (thousand!) Markdown files and creating the new HTML takes on GitHub actions takes about 20 seconds 😮 The whole build is 45 seconds 🏎

Am I going to be in dependency hell in the future?

Very doubtful. There just 2 things I need to take care of, the main dependency and the Node.js version which has very long LTS versions. I should be fine for a couple of years.

I'm very happy with the result. Not only to learn more about file handling in Node.js but to speed things up. Taking less build time and resources is just a win-win for me, the data-center running the build, and the environment 💜

Repository

https://github.com/quicoto/reviews

The post Migrating from Gatsby to custom Node.js script appeared first on Ricard Torres dev.

How to dynamically generate images with Node.js

Ricard Torres — Wed, 11 May 2022 13:14:01 +0000

🚫 Problem

Having to manually 😴 create unique images for Social Media (Open Graph) for your new blog posts.

✅ Solution

Let's use Node.js to engineer a script to take care of this for us 😎

❤️ Generated image

This is the final result, a dynamically generated image:

🤔 How does this work?

A part from playing with the file system (fs) module from Node.js, the actual image creation will be done using canvas:

https://www.npmjs.com/package/canvas

I assume you're comfortable adding NPM packages to your package.json file.

Require all functions needed from canvas

We'll start our index.js file by requiring this, as well as the file system module:

const fs = require('fs');
const {
  createCanvas,
  loadImage,
  registerFont,
} = require('canvas');
Code language: JavaScript (javascript)

Creating a canvas

To define the area we will be working with, define it like so:

const imageCanvas = createCanvas(1200, 630);
const context = imageCanvas.getContext('2d');
Code language: JavaScript (javascript)

Loading a custom font (optional)

In case the system you're creating the images in does not have the fonts you want, simply include the .ttf files and register the custom font like so:

registerFont('comicsans.ttf', { family: 'Comic Sans' })
Code language: JavaScript (javascript)

Adding a background color to the canvas

You'll notice the width and height match the canvas size we defined earlier.

context.fillStyle = '#343a40';
context.fillRect(0, 0, 1200, 630);
Code language: JavaScript (javascript)

Adding text to the canvas

Use these 3 properties to define properties of the text, and then add the text in the position you want relative to the canvas.

context.fillStyle = '#fff';
context.font = 'bold 30pt Menlo';
context.fillText('ricard.dev', 580, 520);
Code language: JavaScript (javascript)

Writing to disk

The final step is to simply write a file with the canvas as PNG format:

fs.writeFileSync(
  `./images/myImage.png`,
  imageCanvas.toBuffer('image/png'),
);
Code language: JavaScript (javascript)

At this point you would be done 👏. You now know how to create images with text with canvas from within Node. You don't have to continue if you don't use WordPress. Just jump to the end to grab the final code.

What improvements can be done?

Use a database.txt file to list all the images you want to create (this can be found in the final code).
Create this on the fly. You could even serve the image on the fly rather than storing it to disk (not covered in this blog post).

WordPress (optional)

If you're using WordPress here are small snippets you'll be needing:

Generate the database

List of post and their ID's:

$path = preg_replace('/wp-content.*$/','',__DIR__);

include($path.'wp-load.php');

$args = array(
  'numberposts' => -1
);

$posts = get_posts( $args );

echo "";

if ( $posts ) {
  foreach ( $posts as $post ) {
    print_r($post->ID . "|" . $post->post_title . "\n");
  }
}

echo "
";
Code language: PHP (php)

It will output something like this:

9099|How to list recently edited pages in Hugo
9072|Moving from Docusaurus to Hugo
Code language: plaintext (plaintext)

Loop the database

From the database you just created, it would only make sense to loop it while creating the images.

const databaseContent = fs.readFileSync('./database.txt', { encoding: 'utf8', flag: 'r' });
const lines = databaseContent.split(/\r?\n/)

lines.forEach((line) => {
  const content = line.split('|');
  const id = content[0];
  const text = content[1];

  // Do stuff with the canvas
  // Add text, colors, etc
  // ...
});
Code language: JavaScript (javascript)

Overwrite Yoast SEO plugin output

By default the plugin will get the actual featured image from the post, you want to customize that output. Should you have placed the script inside your theme, you will also want this in your functions.php file

function wp_open_graph_overwrite_YOAST( $url ) {
  if (!is_singular('post')) {
    return $url;
  }

  // Check if the file exists on disk
  $image = '/image-generator/images/' . get_the_ID() . ".png";
  $imageURL = get_stylesheet_directory_uri() . $image;
  $imagePath = get_stylesheet_directory() . $image;

  if (file_exists($imagePath)) {
    return $imageURL;
  }

  return $url;
}

add_filter( 'wpseo_opengraph_image', 'wp_open_graph_overwrite_YOAST' );
Code language: PHP (php)

👨‍💻 Grab the complete code

Check out the GitHub repository:

https://github.com/quicoto/image-generator

The post How to dynamically generate images with Node.js appeared first on Ricard Torres dev.

Defer CSS loading with WordPress

Ricard Torres — Thu, 03 Mar 2022 09:51:05 +0000

So you're learning about Critical Rendering Path, you have already optimized your WordPress theme's stylesheet but still there are WordPress core CSS files that you want to lazy load. I have a present for you 🎁

One of several techniques is to load the stylesheet using media="print" which tells the browser this file is meant for print view and not when regular browsing the page.

The trick is to use JavaScript to tell the browser that after finishing loading change that stylesheet from print to all, thus loading it for the regular view.

This will effectively defer the CSS loading until the end. In my case I want to defer the WordPress Block Library, I don't use it a lot and I saw a reduction of render time by applying this change:

function custom_use_print_block_library( $html, $handle ) {
  $handles = array( 'wp-block-library' );
  if ( in_array( $handle, $handles ) ) {
    $html = str_replace( 'media=\'all\'', 'media=\'print\' onload="this.onload=null;this.media=\'all\'"', $html );
  }
  return $html;
}
add_filter( 'style_loader_tag', 'custom_use_print_block_library', 10, 2 );
Code language: PHP (php)

To be placed in your functions.php theme's file.

The post Defer CSS loading with WordPress appeared first on Ricard Torres dev.

How to list recently edited pages in Hugo

Ricard Torres — Sun, 20 Feb 2022 15:59:09 +0000

The following snippet will list your Hugo site pages ordered by last modification date. You simply have to reverse it and limit to your liking.

Shortcode

themes/YOUR_THEME/layouts/shortcodes/last-edited-pages.html

<h2>Recently Editedh2>

{{ $byLastMod :=  .Site.Pages.ByLastmod  }}
{{ $recent := ($byLastMod | last 5).Reverse }}

<ul>
{{ range  $recent }}
  <li>
    <a href="{{ .Permalink }}">{{ .Title }}a>
  li>
{{ end }}
ul>
Code language: HTML, XML (xml)

How to use?

{{< last-edited-pages >}}
Code language: HTML, XML (xml)

The post How to list recently edited pages in Hugo appeared first on Ricard Torres dev.

Moving from Docusaurus to Hugo

Ricard Torres — Tue, 15 Feb 2022 08:00:00 +0000

For static generated sites I'm loving more everyday Hugo. It's speed and zero JavaScript dependency (unlike Gatsby) is just what the doctor ordered.

I have a private wiki for my, well, not public notes. The first version was done with WordPress (I even open sourced the theme). It worked okay, nice writing experience, one click publish/update. My problem with WordPress (and I do like WordPress) is security (of the PHP files, the database, etc) as well as maintenance of them. Plus in the end I just want to write plain text files.
I then moved to Docusaurus because I wanted to get rid of the database and increase speed (no cache plugins, no PHP cache, CPU load on each HTTP request to query the database and process the PHP files... Don't get me started!). Setup simplicity overall at the cost of writing experience.

Now, I'm about to change it again, to hopefully a even better setup. You tell me. My reasons? The same I had before. I want more simplicity and speed.

🤔 Why move to Hugo?

It's pointless to compare the size of the two systems. Hugo, running in Go and installed globally on your machine vs hundreds if not thousands of NPM packages with Docusaurus. It's a far cry from a simple HTML and CSS file we used back in the good ol' days.

My pain points:

Dependency maintenance: I hate having to maintain NPM dependencies, because of security reasons or when I want to update to latest versions/features. You end up with a big package.json (and even bigger node_modules folder) that sometimes takes puzzle work to match dependencies with NodeJS and NPM versions, without breaking a feature.
With Hugo, if it works, you don't ever need to update the Hugo binaries. Just build the site, it works and it's fast, like really fast.
Build speed: might not be a fair comparison as the final generated product is different, but it's important to illustrate my use case.
- Local macOS:
  - Docusaurus:
    - Develop: 8 seconds
    - Production: 23 seconds
  - Hugo:
    - Develop: 1 second
    - Production: 1 second
- On GitHub Actions, build for production:
  - Docusaurus: average of 1.5 minutes.
  - Hugo: 45 seconds.

📝 Content

🗂 Categories

Docusaurus handles post categories with a _category_.json which has a content similar to:

{
  "label": "🎸 Guitar",
  "collapsed": true
}
Code language: JSON / JSON with Comments (json)

This need to go. With Hugo you create a _index.md file under the folder. I like this better because with this file you can already include content for the category, before creating the child pages under it. You end up with something like:

/content/guitar/_index.md
/content/guitar/songs.md

📄 Pages

Regarding your content, your Markdown files, the "only" change is to add Front Matter to each .md file

I didn't have many files so I did it manually but on a high content site you might need a bash script or similar to automate the process. This would be the bare minimum:

---
title: Hello World
---

🏎 The need for speed?

With Hugo, as mentioned, you have full control over what JS and CSS you include in your theme.

Why not use PostCSS to remove unused CSS from your theme? 😁

👨‍💻 Conclusion

The move was quite painless. Since you'r content is practically the same, Markdown files, you can easily switch to another CMS if needed.

The most work, as you can imagine, is creating a theme in the new system. Less work if you get an open sourced theme, I forked one and stripped down everything I didn't need to make it lean.

I'm sure there are plenty of valid use cases for Docusaurus, it's actively developed and it looks promising. For my particular simple use case, I didn't need the added complexity (even if they claim: "Build optimized websites quickly, focus on your content").

What's your take? Which one do you use? What's your use case?

The post Moving from Docusaurus to Hugo appeared first on Ricard Torres dev.

Generating book lists from my Goodreads profile

Ricard Torres — Wed, 09 Feb 2022 12:03:41 +0000

If you read books you have perhaps heard about Goodreads.com, the site has book reviews, season awards, and more. The most interesting part is that you can track you read, currently reading and want to read books.

Yes, you could use a notepad or a spreadsheet 🤷‍♂️ but using the site is nice. You can review the books you read, rate them and so on.

🔒 Owning your data

They key thing is not to give your data away with risk of losing it. I want to create a backup / export of my data, should Goodreads eventually go away. Simple, don't lose your data 😉

🤓 RSS XML

The good folks at Goodreads are nice enough to offer RSS XML files for your profile ❤️ For each of the mentioned shelves (Currently Reading, Read, and Want to Read) 💯

We can use these 3 XML to store the output in .txt files. The text files can be used (regardless of the RSS XML availability) to generate a nice HTML output (should you want to showcase the lists in your site).

https://www.goodreads.com/review/list_rss/YOUR_PROFILE_ID?key=&shelf=currently-reading
https://www.goodreads.com/review/list_rss/YOUR_PROFILE_ID?key=&shelf=to-read
https://www.goodreads.com/review/list_rss/YOUR_PROFILE_ID?key=&shelf=read
Code language: HTML, XML (xml)

📚 Fetching the RSS

While we wait for Node 17.5 to have native fetch() support we need to find a workaround. This nice NPM package node-fetch will provide what we need ✅

Additionally we need to parse de XML, arraybuffer-xml-parser is the other package we make use of ✅

With these two we can do something like:

fetch(url)
  .then(response => response.text())
  .then(data => {
    const xmlData = encoder.encode(data);
    const object = parse(xmlData);
    const items = object.rss.channel.item;

    items.forEach(item => {
      console.log(item.title)
    })
  });
Code language: JavaScript (javascript)

🆕 Creating files

Ok, so we've fetched the RSS files and after parsing them we need to create a file. With Node is easy:

import fs from 'fs';

fs.writeFile(fileName, data, (err) => {
  if (!err) {
    console.log('File created: ' + fileName);
  }
});
Code language: JavaScript (javascript)

The content of the .txt files look like this:

book_id|year|Book Name|URL
book_id|year|Book Name|URL
book_id|year|Book Name|URL

👓 Reading files

Reading your files with Node is trivial:

import fs from 'fs';

function readFile(shelve) {
  return fs.readFileSync(`./database/${shelve}.txt`, { encoding:'utf8', flag:'r'});
}
Code language: JavaScript (javascript)

Reading the lines in the file

Once you read the file with fs you can get the lines with a simple split:

function getLines(content) {
  return content.split(/\r?\n/);
}
Code language: JavaScript (javascript)

🔥 GitHub Actions

You know me, I'm a fan of Github Actions 🤩 We'll use this simple action to run our Node.js script and deploy the output to a different branch 🚀

The idea is to have our scripts create the files in a the public folder (or whatever name you want it to be!). Then after running the script copy that folder to another branch, for persistence, history, etc.

Create the file in your repo: .github/workflows/build.yml

name: Build and Deploy
on:
  push:
    branches:
      - main
jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout 🛎️
        uses: actions/checkout@v2
        with:
            persist-credentials: false
      - name: Install and Build
        run: |
          npm ci
          npm run build
      - name: Deploy 🚀
        uses: JamesIves/github-pages-deploy-action@releases/v4
        with:
          BRANCH: output # The branch the action should deploy to.
          FOLDER: public # The folder the action should deploy.
Code language: YAML (yaml)

🤔 One possible improvement could be to have the action do a commit to the main branch with the changes. Otherwise I have to manually copy the updated files from the output branch to the main branch. Small price to pay, though.

👨‍💻 Conclusion

Now you know how to easily fetch, write files to disk and read them with NodeJS.

The rest is up to you know to build your app. Make use of the free and powerful GitHub Actions and own your data!

😎 Live

See it live here ricard.blog/books

🍴 GitHub

Play with it, modify it, fork it!

github.com/quicoto/goodreads-profile-generator

The post Generating book lists from my Goodreads profile appeared first on Ricard Torres dev.

How to create a news digest using RSS and GitHub Actions

Ricard Torres — Thu, 03 Feb 2022 14:54:48 +0000

One of my daily morning routines is to scan several news sites and read headlines to see if anything piques my interest. Or at least to have a vague recollection of topics happening nowadays.

I go to Hacker News, The Verge, The Guardian… and others. A mixed bag, really of tech and world news.

Because of the mind of an engineer… I wanted to optimize the time spent doing this. How? Glad you asked 🌝 Using the magic of RSS 🥰 I love RSS, Google tried to end RSS with the killing of Google Reader but it is still going strong. I use it to follow the blogs I like and now, the news I want to consume.

💡 Idea

Have a simply HTML page of containing news headlines with links to the articles from different sites (RSS feeds). For instance two of the feeds I use:

Scraping the sites would be another approach but it would complicate things so much more 👎 If they ever change classes or markup, your selectors might stop working. With RSS you know what you're getting. As long as they keep updating the content on their feeds, we're good! 😌

🙅‍♂️ I don't want a backend

I don't want any backend, database or API service. Just give me the stuff on a plain HTML page. Fast.

To do that we need to use a CI to build this HTML, so we don't have a server to pull on demand. Rather, why don't we schedule the build of this HTML before I wake up?

🤔 NodeJS and GitHub Actions

We will use NodeJS because I know JavaScript but you could use another language. As long as you can fetch the RSS feeds and parse them, you're good.

With NodeJS I used this cool NPM package:

npm install --save rss-parser
Code language: Bash (bash)

Then something similar to this:

let Parser = require('rss-parser');
let parser = new Parser();

(async () => {

  let feed = await parser.parseURL('url-of-the-feed.xml');
  console.log(feed.title);

  feed.items.forEach(item => {
    console.log(item.title + ':' + item.link)
  });

})();
Code language: JavaScript (javascript)

In my case I will loop through my RSS feed URLs which I have stored in a JSON file so I can easily add/remove items without changing the actual implementation.

I'm building an array of promises and using Promise.all I'll parse all the contents and build up an HTML page. A list of links, really.

GitHub Actions

I'm in love with the simplicity of GitHub Actions, so easy to implement, so little friction, and yet so powerful.

I'm going to create an action to run my NodeJS script, take the generated HTML and deploy it to GitHub Pages. Which will simply host HTML and CSS.

In my package.json I have a single command build:

{
  "name": "news-digest",
  "version": "1.0.0",
  "description": "A single place to check all my daily news sites in 1 go",
  "dependencies": {
    "rss-parser": "^3.12.0"
  },
  "scripts": {
    "build": "node index.js"
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}
Code language: JSON / JSON with Comments (json)

Then on the Github Action workflow:

name: Build and Deploy
on:
  push:
    branches:
      - main
  schedule:
      - cron: "0 6 * * *"
jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout 🛎️
        uses: actions/checkout@v2
        with:
            persist-credentials: false
      - name: Install and Build
        run: |
          npm ci
          npm run build
      - name: Deploy 🚀
        uses: JamesIves/github-pages-deploy-action@releases/v4
        with:
          BRANCH: gh-pages # The branch the action should deploy to.
          FOLDER: dist # The folder the action should deploy.
Code language: YAML (yaml)

Don't forget to setup the Github Pages on the repository settings. You have to set it to the gh-pages branch.

😎 See it in action

Here's the live site which I check every morning ☕️ 🥱 Have also a look on GitHub

Would you do this differently? If so, how?

The post How to create a news digest using RSS and GitHub Actions appeared first on Ricard Torres dev.

How to defer scripts in WordPress

Ricard Torres — Mon, 16 Aug 2021 07:00:00 +0000

You don't need me explaining you how important it is to defer scripts for performance. If your application doesn't need render-blocking scripts you should simply defer them.

Most of the WordPress blogs/sites do not need render blocking scripts, for toggling menu visibility, swapping emojis for SVG's, embed media... All these can be deferred no problem.

The script below will handle all non-Dashboard (WordPress admin) scripts as defer.

You might want to have an allow-list array using the $handle param. In my case I want everything, to speed up the page render time as much as possible.

/*
 * Add defer attribute to enqueued scripts
 *
 * @param string $tag
 * @param string $handle
 * @param string $src
 * @return void
 */

function defer_scripts( $tag, $handle, $src ) {
  if (is_admin()) return $tag;

  return '';
}

add_filter( 'script_loader_tag', 'defer_scripts', 10, 3 );
Code language: PHP (php)

The post How to defer scripts in WordPress appeared first on Ricard Torres dev.

How to create custom commands for Mac’s Spotlight

Ricard Torres — Fri, 06 Aug 2021 13:02:07 +0000

Out of the box doesn't seem to be possible 🙁 BUT! I've found a workaround that works almost as nicely 🎉
There are paid apps out there, but we as developers should do it ourselves 👨🏻‍💻 To learn from it and to feel like a hacker.

💡 Create a bash script

I want to able to go quickly to a site from anywhere on my Mac. Sure that's what bookmarks are for, but you need to focus on Firefox, find the bookmark and click on it. That's just too much 😁

Imagine just typing a command in spotlight and be done with it ✅ The bash script below does exactly that.

Opens Firefox Developer Edition
Goes to the URL
Terminal closes itself (I have configured Terminal to not ask for confirmation when closing with open processes).

#! /bin/bash

open -a Firefox\ Developer\ Edition -g https://ricard.dev/wp-admin/
osascript -e 'tell application "Terminal" to quit'
Code language: JavaScript (javascript)

As you know you can do practically anything with a bash script, so just let loose! It doesn't have to be my browser use case 😎

🤔 Easy find/write syntax

I've created my own syntax for naming the files cXZY.command

In the example below: cdev.command will open my dev blog Dashboard in Firefox.

This syntax might not work for your brain just come up with something unique so Spotlight does not give you the wrong result.

Known issues

If I'm watching a video with VLC and run the command it will open Firefox but it will not focus on that app. I've tried to use activate but doesn't work. Ideas?

osascript -e 'tell application "Firefox\ Developer\ Edition" to activate'
Code language: JavaScript (javascript)

🐙 Github

I've created a repo for them: https://github.com/quicoto/shortcut-commands

The post How to create custom commands for Mac’s Spotlight appeared first on Ricard Torres dev.

How to do multiple StaticQuery in Gatsby

Ricard Torres — Thu, 05 Aug 2021 08:00:00 +0000

If you need to do more than one GraphQL StaticQuery in Gatbsy you can assign each query to a name.

Then simply access data.yourName and you're ready to go.

Here's a full example of how I use it on my reviews site:

import React from "react"
import { StaticQuery, graphql } from "gatsby"

const TotalTime = () => (
  <StaticQuery
    query={graphql`
      {
        movies: allMarkdownRemark(filter: { frontmatter: { type: { eq: "movie" } } }) {
          edges {
            node {
              id
            }
          }
        }
        series: allMarkdownRemark(filter: { frontmatter: { type: { eq: "series" } } }) {
          edges {
            node {
              id
            }
          }
        }
      }
    `}
    render={data => {
      let movies = JSON.parse(JSON.stringify(data.movies, null, 4)).edges;
      let series = JSON.parse(JSON.stringify(data.series, null, 4)).edges;

      return <div>
          Do something...
        div>
    }}
  >StaticQuery>
)

export default TotalTime
Code language: JavaScript (javascript)

The post How to do multiple StaticQuery in Gatsby appeared first on Ricard Torres dev.

How to remove client-side JavaScript from Gatsby

Ricard Torres — Sat, 31 Jul 2021 15:49:17 +0000

I build my reviews site using Gatsby. It's cool, the graphql is powerful but there's something that was buggering me.

Why does my static site need client-side JavaScript?

There's no reason for it. Gatsby will argue about faster route changing this way, sure. For a high-traffic site where users visit multiple pages on the same visit, it makes sense. Not my case.

Astonishingly Gatsby does not provide an out-of-the-box setting or flag to amend this issue. I would've loved for a boolean in the config static if I need JS. To it builds a a truly static site, which can be done with Jekyll, Hugo or Eleventy.

No JavaScript plugin

I found this plugin but it doesn't seem to work with sites under a subfolder such as example.com/mysite

Then I encountered this Gist with the small amendment needed so it works in a subdirectory installation. It's based on the plugin but you don't need the plugin installed to work.

Create a gatsby-ssr.js file in the root directory of your site. it will remove all JavaScript from the final build (not when running the develop task).

⚠️ It did not fully work in my Gatsby 2.25.3, it needed a small optional chaining around the polyfill condition.

Here's my updated version:

let pageScripts
/*
 * The "scripts" variable is not documented by Gatsby, https://www.gatsbyjs.org/docs/ssr-apis/#onRenderBody, and that is probably for a good
 * reason. The variable contains the scripts the Gatsby internals, https://github.com/gatsbyjs/gatsby/blob/d9cf5a21403c474846ebdf7a0508902b9b8a2ea9/packages/gatsby/cache-dir/static-entry.js#L270-L283,
 * puts into the head and post body. We will be relying on this undocumented variable until it does not work anymore as the alternative is
 * to read the webpack.stats.json file and parse it ourselves.
 */
export function onRenderBody({ scripts }) {
  if (process.env.NODE_ENV !== "production") {
    // During a gatsby development build (gatsby develop) we do nothing.
    return
  }
  // TODO maybe we should not even wait and see if Gatsby removes this internal "script" variable and code around the issue if the variable is not there.
  if (!scripts) {
    throw new Error(
      "gatsby-plugin-no-javascript: Gatsby removed an internal detail that this plugin relied upon, please submit this issue to https://www.github.com/itmayziii/gatsby-plugin-no-javascript."
    )
  }
  pageScripts = scripts
}
// Here we rely on the fact that onPreRenderHTML is called after onRenderBody so we have access to the scripts Gatsby inserted into the HTML.
export function onPreRenderHTML(
  {
    getHeadComponents,
    pathname,
    replaceHeadComponents,
    getPostBodyComponents,
    replacePostBodyComponents,
  },
  pluginOptions
) {
  if (
    process.env.NODE_ENV !== "production" ||
    checkPathExclusion(pathname, pluginOptions)
  ) {
    // During a gatsby development build (gatsby develop) we do nothing.
    return
  }
  replaceHeadComponents(
    getHeadComponentsNoJS(getHeadComponents(), pluginOptions)
  )
  replacePostBodyComponents(
    getPostBodyComponentsNoJS(getPostBodyComponents(), pluginOptions)
  )
}

function getHeadComponentsNoJS(headComponents, pluginOptions) {
  return headComponents.filter(headComponent => {
    // Not a react component and therefore not a