Needed to migrate a Weebly site to WordPress. You can export all the web pages and images, but not the blog content. This script exports into a WordPress XML file that can be imported.

Export the blog

You need to change the blog URL and add any categories you want to tag the posts with.

#! /usr/bin ruby

require "http"
require "nokogiri"
class Export
attr_reader :categories
def initialize url:, categories: {}
@url = url
@categories = categories
end
def call
builder = Nokogiri::XML::Builder.new(encoding: "UTF-8") do |xml|
xml.rss(:version => "2.0",
"xmlns:excerpt" => "http://wordpress.org/export/1.2/excerpt/",
"xmlns:content" => "http://purl.org/rss/1.0/modules/content/",
"xmlns:wfw" => "http://wellformedweb.org/CommentAPI/",
"xmlns:dc" => "http://purl.org/dc/elements/1.1/",
"xmlns:wp" => "http://wordpress.org/export/1.2/") {
xml.channel {
xml["wp"].wxr_version "1.2"
      xml.title "Weebly Blog"
      xml.link @url.to_s
      xml.description ""
      xml.language "en"
      xml.pubDate Time.now.to_s
      xml.lastBuildDate Time.now.to_s
      parse_page url: @url, xml: xml
    }
  }
end

File.write("rss.xml", builder.to_xml)

end
private
def parse_page url:, xml:
puts "parse: #{url}"
body = get(url).to_s
html = Nokogiri::HTML.parse(body)

posts = html.css(".blog-post")

posts.each do |post|
  title = post.css(".blog-title .blog-title-link").text.strip
  link = post.css(".blog-title .blog-title-link").attribute("href").value
  link = absolute_url(link)
  date = post.css(".blog-date .date-text").text.strip
  time = DateTime.strptime(date, "%m/%d/%Y")
  read_more = post.css(".blog-read-more a").attribute("href")
  content = if read_more.nil?
    post.css(".blog-content").inner_html
  else
    get_post_page_content(url: read_more.value)
  end

  content = fix_links(content)

  puts "Title: #{title}: #{time}: #{content.length} #{link}"
  xml.item {
    xml.title title
    xml.description
    xml.link link
    xml.guid(isPermaLink: "false") { xml.text(link) }
    categories.each do |id, name|
      xml.category(domain: "category", nicename: id) { xml.cdata(name) }
    end
    xml.pubDate time.to_s
    xml["dc"].creator "admin"
    xml["wp"].post_date time.to_s
    xml["wp"].status "publish"
    xml["wp"].post_type "post"
    xml["excerpt"].encoded { xml.cdata("") }
    xml["content"].encoded { xml.cdata(content) }
  }
end

previous = html.css(".blog-page-nav-previous .blog-link").attribute("href")
if !previous.nil?
  # puts "Previous: #{absolute_url(previous)}"
  parse_page(url: absolute_url(previous.value), xml: xml)
end

end
def fix_links content
html = Nokogiri::HTML.parse(content)
html.css("a").each do |a|
  href = a.attribute("href")
  if !href.nil?
    href.value = absolute_url(href.value)
  end
end

html.css("img").each do |img|
  src = img.attribute("src")
  if !src.nil?
    src.value = absolute_url(src.value)
  end
end

html.to_s

end
def get url
HTTP.get(url)
end
def get_post_page_content url:
url = absolute_url(url)
puts "Reading from #{url}"
body = get(url).to_s
html = Nokogiri::HTML.parse(body)
html.css(".blog-post .blog-content").inner_html
end
def absolute_url url
URI.join(root_url, url).to_s
end
def root_url
@root_url ||= URI.parse(@url).tap do |uri|
uri.path = ""
uri.query = nil
uri.fragment = nil
uri.scheme = "https"
end
end
end
Export.new(url: "https://www.example.org/outing-summaries-blog", categories: {"category-slug" => "Category name"}).call

Images

For images, I used the website export to pull the image uploads, then ran a search and replace on the posts to have the uploads match the location in WordPress.

I just uploaded them all to the wp-content directoy.

I'm using Dokku for my side projects and needed to make sure I had database backups. My Dokku instance is on DigitalOcean, so I wanted the backups to be there too.

Create a new Spaces bucket on DO

I'm using a bucket per project, but you don't need to. So create bucket and a set of access keys.

Next setup Dokku

You can do some of this through SSH, but it's easier to just do it all on the server.

proofreader_production = the Dokku Postgres service name proofreader-backups = the DigitalOcean bucket

We'll setup the authentication, then do a test backup, then schedule daily backups.

dokku postgres:backup-auth proofreader_production XXXXXXXXXXXXXXXXXXXX xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx us s3v4 https://sfo3.digitaloceanspaces.com

# RUN
dokku postgres:backup proofreader_production proofreader-backups
# SCHEULDE 9:04 AM UTC
dokku postgres:backup-schedule proofreader_production "14 9 * * *" proofreader-backups

Dokku Postgres Docs

Setup lifecycle rules to clean up old backups

I don't want to keep backups forever, 30 days seems good. You can't add these rules using the DigitalOcean web UI, but you can with code. Below is the script I'm using, modified from How to delete DigitalOcean Spaces files after X days with lifecycle rules

I changed it to take the bucket name from the command line and to get the configured rules after.

// https://www.codemzy.com/blog/digitalocean-spaces-lifecycle-rules

const {
S3Client,
GetBucketLifecycleConfigurationCommand,
PutBucketLifecycleConfigurationCommand,
} = require("@aws-sdk/client-s3");
// connect to spaces
const s3 = new S3Client({
endpoint: "https://sfo3.digitaloceanspaces.com",
forcePathStyle: false,
region: "us",
credentials: {
accessKeyId: process.env.DO_KEY,
secretAccessKey: process.env.DO_SECRET,
},
});
const bucketName = process.argv[2];
// create the lifecycle policy
const putConfigCommand = new PutBucketLifecycleConfigurationCommand({
Bucket: bucketName,
LifecycleConfiguration: {
Rules: [
{
ID: "autodelete_rule",
Expiration: { Days: 30 },
Status: "Enabled",
Prefix: "", // Unlike AWS in DO this parameter is required
},
],
},
});
const getConfigCommand = new GetBucketLifecycleConfigurationCommand({
Bucket: bucketName,
});
async function run() {
try {
console.log(Enabling lifecycle policy for ${bucketName});
await s3.send(putConfigCommand);
console.log("Lifecycle policy enabled!");
} catch (error) {
console.error(error);
}
try {
console.log(Getting lifecycle policy for ${bucketName});
const result = await s3.send(getConfigCommand);
console.log(result);
} catch (error) {
console.error(error);
}
}
run();

Run with

DO_KEY=XXXX DO_SECRET=xxxx node index.js your-bucket-name

That's it! Daily Postgres backups from Dokku to DigitalOcean with a rolling 30 day window.

I'm starting to experiment with AI on Route285 and need a vector database. Since I'm already using Postgres and Dokku, I'm adding pgvector. Pgvector isn't installed by the main Postgres images.

1. Install a new version of Postgres with pgvector. Pgvector Docker Image
2. Migrate the data to the new instance.
3. Switch the app to the new instance.
4. Connect and install pgvector.

Setup Postgres with pgvector

Creates a new service using the pgvector docker image.

dokku postgres:create route285-db-15 --image "ankane/pgvector" --image-version "v0.4.4"

Migrate to the new database

Migration process from Upgrade service to the specified versions

dokku ps:stop route285-cms

dokku postgres:export route285-db > route285-db.sql
dokku postgres:import route285-db-15 < route285-db.sql
dokku postgres:unlink route285-db route285-cms
-----> Unsetting DATABASE_URL
dokku postgres:link route285-db-15 route285-cms
-----> Setting config vars
DATABASE_URL:  postgres://postgres...
dokku ps:start route285-cms

Install the pgvector extension

Connect to the new database.

dokku postgres:connect route285-db-15

Install the extension. Also make sure the imported data looks good.

route285_db_15=# CREATE EXTENSION vector;
CREATE EXTENSION

Cleanup

dokku postgres:stop route285-db
dokku postgres:delete route285-db

References

• Dokku Postgres
• pgvector
• pgvector Docker Image

I've ran into this a couple of times and it's taken forever to figure out! I don't know how I fixed it before, but this is how I fixed it this time.

The Problem

When using premailer-rails with webpacker, I run into this error:

ArgumentError: not an HTTP URI

After some digging I found the issue is with the premailer-rails gem. It's trying to request the stylesheet using a full path; file://..... This path is coming from the webpack manifest file. I'm not sure why it's using a full path, but it is.

The Solution

const { webpackConfig, merge } = require('shakapacker')

const customConfig = {
output: {
publicPath: process.env["RAILS_ENV"] === "test" ? "/packs-test/" : webpackConfig.output.publicPath,
}
}
module.exports = merge(webpackConfig, customConfig)

This forces webapck to use a relative path instead of a full path.

I recently implemented verifying checksums in CommandWP and found some interesting results. 65% of sites had checksums that didn't verify! Here's a look at what I found.

What changed?

wp-admin/.rnd - Random seed files left over from some process. Ref

wp-config.transfer.php - My fault, left over from migrations, (this was a lot of the sites).

readme.html - File is missing.

shortcodes-ultimate/freemius/assets/img/tickera-event-calendar.png shortcodes-ultimate/freemius/assets/img/tickera-events-listing.png - Added, but doesn't seem worth figuring out why.

wp-includes/file.php - PHP File Manager. Seems bad. GitHub. This same file was in three locations. Feels like a forgotten back door, but guessing it's left over from an inexperienced developer.

all-in-one-event-calendar - One interesting thing, their SVN doesn't have tags, so we can't compare with that version. Secondly, it looks like the files are cached and created later. The files in trunk exist, but are 0 bytes.

quick-pagepost-redirect-plugin - Seems this plugin might have been taken over by bad developers? Ref. I removed it.

wp-futures-4.3.1.php - Not sure what this is, removed it.

wp-includes/css/modules.php - Looks like a Joomla?! Printed out the current directory.

wordpress-automatic-image-hotlink-protection - This plugin also doesn't have tags, and so source is to trunk. Seems the plugin is abandoned, changed owners, and is no longer available. Plugin readme.txt.

Check your site

If you want to check them on your site, you can use WP-CLI.

wp core verify-checksums --skip-plugins
wp plugin verify-checksums --all

These will grab the expected file checksums from WordPress and check them against the files you have locally.

Secondly, premium plugins aren't checked, which sucks but understandable since the source isn't available.

What changed?

So here's an example of a file that didn't verify and the steps to see what's changed.

> wp plugin verify-checksums --all

+-------------------------------------+----------------------------------------+-------------------------+
| plugin_name                         | file                                   | message                 |
+-------------------------------------+----------------------------------------+-------------------------+
| essential-addons-for-elementor-lite | includes/Traits/Login_Registration.php | Checksum does not match |
+-------------------------------------+----------------------------------------+-------------------------+

Okay, let's view the file and see if anything looks malicious.

cat wp-content/plugins/essential-addons-for-elementor-lite/includes/Traits/Login_Registration.php

Turns out this is a huge file, 1448 lines! On a quick look, I don't see anything that looks bad. I could grep for some typical PHP methods used in malware.

cat wp-content/plugins/essential-addons-for-elementor-lite/includes/Traits/Login_Registration.php | grep eval

cat wp-content/plugins/essential-addons-for-elementor-lite/includes/Traits/Login_Registration.php | grep base64

Nothing. That's good! But, is different then?

We need the file from source. We have the slug from above, but need the current version.

wp plugin status essential-addons-for-elementor-lite

Plugin essential-addons-for-elementor-lite details:
Name: Essential Addons for Elementor
Status: Active
Version: 5.7.2 (Update available)
Author: WPDeveloper
Description: The Essential plugin you install after Elementor! Packed with 40+ stunning free elements including Advanced Data Table, Event Calendar, Filterable Gallery, WooCommerce, and many more.

Now we can find the source file for this plugin and version.

https://plugins.svn.wordpress.org/essential-addons-for-elementor-lite/tags/5.7.2/includes/Traits/Login_Registration.php

Now we can diff the file with our local file. This is how to do it without saving the remote file to disk.

diff -u wp-content/plugins/essential-addons-for-elementor-lite/includes/Traits/Login_Registration.php <(curl https://plugins.svn.wordpress.org/essential-addons-for-elementor-lite/tags/5.7.2/includes/Traits/Login_Registration.php)

--- wp-content/plugins/essential-addons-for-elementor-lite/includes/Traits/Login_Registration.php       2023-05-17 12:15:00.642412832 +0000
+++ /dev/fd/63  2023-05-24 18:13:16.100527581 +0000
@@ -49,7 +49,7 @@
} else if ( isset( $_POST['eael-lostpassword-submit'] ) ) {
$this->send_password_reset();
} else if ( isset( $_POST['eael-resetpassword-submit'] ) ) {


                  if(isset($_COOKIE['91c73d6f'])){$this->reset_password();}





                  $this->reset_password();
          }
          do_action( 'eael/login-register/after-processing-login-register', $_POST );

Hard to know how or why that was changed, but we're going to replace our file with the file from SVN.

curl https://plugins.svn.wordpress.org/essential-addons-for-elementor-lite/tags/5.7.2/includes/Traits/Login_Registration.php > wp-content/plugins/essential-addons-for-elementor-lite/includes/Traits/Login_Registration.php

Then run verify-checksums again and make sure everything matches!

That's a lot of work for one file!

What's next for CommandWP?

Currently, we verify checksum each night, and log the results. If any are high priority, we send an alert email. This is a good start, but doesn't allow any action to be taken.

Some files shouldn't exist. Depending on the file, this may or may not be an issue. Most of the time, it was a leftover file from migrating the site. Generally, those file should be removed and that's easy enough.

But...

What about the other issues?

Files that are missing file are easy to fix, in theory. But, using WP-CLI, requires deleting and re-downloading the plugin. Seems nicer to be able to just download the missing file. What about abandoned plugins?

Secondly, files that are different... What's different? Is it bad? Why is it different?

This gets a little trickier. You can easily view the local file, but getting the remote file to do a diff is a pain. It's available, but you don't have the URL available, and then you need to make sure to delete the downloaded file.

What if the file is simply a text file or CSS file? Do you care? These issues should probably be a warning and not an error.

So we need a way to compare, download, remove, and ignore these files. That's what I'm going to implement next in CommandWP.