The web has changed so much I don’t really know how discovery works any more, if I ever did, but I thought I’d have a poke around Bing webmaster tools and Google Search Console to see if I could at least try to get my pages indexed there, not that anyone uses search any more —and given that web search itself doesnlt really seem to work very well any more anyway…

The pages on New Troubadours are typically paramterised — things like https://newtroubadours.org/performers.html?performer=hannah-moore and https://newtroubadours.org/venues.html?venue=exeter-phoenix — and javascript does the job of rendering the content into the page from a JSON data feed. The Google live page previewer shows that the Google crawler will have a go at running the js and then indexing the rendered page, but Bing just seems to to show the skeleton body, though js rendered metadata into the head seems to work.

Hmmm… that means, for Bing (and hence Duck Duck Go etc), most of the content won’t make it into the index, and most of the parametrised pages, even though the parameterised URLs are declared canonical, won’t be be discoverable…

(The metadata in the parameterised pages is a bit shonky and I should probably do something about that…)

Anyway… what to do… Seems like Cloudflare to the rescue again in the form of a pre-rendering Cloudflare worker. I didn’t really have a model of how things work to make use of the few standlaone docs that Cloudflare provides, and very few folk blog how to recipes anymore, so I turned to chatgpt (I was out of free tokens on claude). Then went round in circles with its shonky advice, til I got the next round of free Claude tokens and got a solution….

…which seems to be two workers:

• one to render a page using a headless browser and generate the parsed HTML;
• one to make that pre-rendering available to web search engines.

The following is pretty much all Claude, built up from the original demo pre-render code on the Cloudflare blog.

In the Cloudflare dashboard, Build menu, there is a Workers and Pages option. From that, Create an Application (i.e. a worker), and go for the Hello World. Give it a name (I used new-troubadours-prerender). From its dashboard page, Settings, select Bindings and add a new one, selecting Browser Run from the menu and naming it BROWSER.

From the worker dashboard, select edit (top right) and use code based on the Cloudflare example, tweaked to point to my domain:

// Only render pages you control. This prevents the Worker from becoming
// an open browser-rendering proxy for arbitrary websites.
const ALLOWED_HOSTNAMES = new Set(["newtroubadours.org", "www.newtroubadours.org"]);

const getTargetUrl = (request) => {
	const requestUrl = new URL(request.url);
	const target = requestUrl.searchParams.get("url");

	if (!target) {
		throw new Error("Missing url query parameter");
	}

	const targetUrl = new URL(target);

	// Only render HTTP(S) pages. Other protocols are not valid web pages.
	if (!["http:", "https:"].includes(targetUrl.protocol)) {
		throw new Error("Only HTTP and HTTPS URLs are allowed");
	}

	if (!ALLOWED_HOSTNAMES.has(targetUrl.hostname)) {
		throw new Error("This hostname is not allowed");
	}

	return targetUrl;
};

const renderHtml = async (env, targetUrl) => {
	// The /content Quick Actions endpoint loads the page in Browser Run and returns
	// a JSON envelope containing the rendered HTML in the result field.
	const response = await env.BROWSER.quickAction("content", {
		url: targetUrl.toString(),
		gotoOptions: {
			waitUntil: "networkidle2",
			timeout: 30000,
		},
		// If your page has a specific readiness signal, use waitForSelector
		// instead of relying only on network activity.
		// waitForSelector: { selector: "[data-prerender-ready='true']", timeout: 30000 },
	});

	if (!response.ok) {
		const detail = (await response.text()).slice(0, 500);
		throw new Error(`Browser Run failed with ${response.status}: ${detail}`);
	}

	const data = await response.json();

	if (!data.success || typeof data.result !== "string") {
		throw new Error("Browser Run returned an unsuccessful response");
	}

	return data.result;
};

export default {
	async fetch(request, env) {
		try {
			// Read and validate the URL before sending it to Browser Run.
			const targetUrl = getTargetUrl(request);
			const html = await renderHtml(env, targetUrl);

			// Return the rendered HTML to the crawler or integration.
			return new Response(html, {
				headers: {
					"content-type": "text/html; charset=utf-8",
				},
			});
		} catch (error) {
			return Response.json(
				{ error: error instanceof Error ? error.message : "Unknown error" },
				{ status: 400 },
			);
		}
	},
};

That does the browser rendering.

Deploy that, go back to the workers page, and create another new one (another new application). Select hellow world template again, and give it a name (I used newtroubadours-gate).

Again in the Bindings section (from the flow chart on the Overview or the Settings tab, this time to select a Service binding and select new-troubadours-prerender. This second worker is going to be fed from the first.

Also on the Settings page, create a variable (an evironment variable) called DEBUG_SECRET and give it a value, eg letmein.

This worker’s code will parse an http request for a page, and if it comes from a web crawler, will try to render the page via the renderer worker.

const CRAWLER_UA_RE = /bingbot|googlebot|duckduckbot|slurp|baiduspider|yandexbot|facebookexternalhit|twitterbot|linkedinbot|discordbot|slackbot/i;
const ASSET_EXTENSION_RE = /\.(js|mjs|css|png|jpe?g|gif|svg|webp|ico|avif|woff2?|ttf|eot|json|xml|txt|map|mp4|webm|pdf)$/i;
const CACHE_TTL_SECONDS = 60 * 60 * 24;
const DEBUG_HEADER = "x-force-render";
const PURGE_HEADER = "x-purge-cache";

function isCrawler(request) {
  return CRAWLER_UA_RE.test(request.headers.get("user-agent") || "");
}

function isAuthorized(request, env, headerName) {
  return env.DEBUG_SECRET && request.headers.get(headerName) === env.DEBUG_SECRET;
}

function looksLikeHtmlRequest(request, url) {
  if (ASSET_EXTENSION_RE.test(url.pathname)) return false;
  const dest = request.headers.get("sec-fetch-dest");
  if (dest && dest !== "document") return false;
  return true;
}

export default {
  async fetch(request, env, ctx) {
    const url = new URL(request.url);
    const cache = caches.default;
    const cacheKey = new Request(url.toString(), request);

    // Purge: delete this URL's cached render and stop — doesn't render anything.
    if (isAuthorized(request, env, PURGE_HEADER)) {
      const deleted = await cache.delete(cacheKey);
      return new Response(
        deleted ? `Purged cache for ${url.toString()}` : `No cache entry found for ${url.toString()}`,
        { status: 200 }
      );
    }

    const forced = isAuthorized(request, env, DEBUG_HEADER);

    if (!forced && (!isCrawler(request) || !looksLikeHtmlRequest(request, url))) {
      return fetch(request);
    }

    if (!forced) {
      const cached = await cache.match(cacheKey);
      if (cached) return cached;
    }

    const renderUrl = new URL("https://renderer/");
    renderUrl.searchParams.set("url", url.toString());
    const rendered = await env.RENDERER.fetch(renderUrl);

    if (!rendered.ok) {
      return fetch(request);
    }

    const response = new Response(await rendered.text(), {
      headers: {
        "content-type": "text/html; charset=utf-8",
        "cache-control": forced ? "no-store" : `public, max-age=${CACHE_TTL_SECONDS}`,
        "x-prerendered": "true",
      },
    });

    if (!forced) {
      ctx.waitUntil(cache.put(cacheKey, response.clone()));
    }
    return response;
  },
};

It will also attempt to cache the result. If the renderer can’t return the rendered page, the original unrendered page is returned.

This second worker also needs a route which passes traffic to it. In my case, (with the newtroubadours.org DNS already being served using Cloudlare in a proxied way), from the Domains tab, I added the route newtroubadours.org/*.

And that’s it. When a request is made for a page on newtroubadours.org, Cloudflare intercepts it (it’s already doing some URL forwarding/rewrites, which I should blog/document) and the newtroubadours-gate worker checks who’s making the request. If it’s a search engine indexer, etc, it passes the request to the new-troubadours-prerender which tries to render the page and return the result. (All other requests simply return the original unrendered page.) The worker passes on the pre-rendered page if it can, else falls back to the original unrendered page.

Via the gate worker, there are also a couple of diagnostics I can run by passing in my secret. If the secret is correct, I’m treated as if I were one of the allowed search engine agents.

In the first case, inspect the rendered page:

# View a rendered page (also caches it on Cloudflare)
curl -H "x-force-render: MYSECRET" "https://newtroubadours.org/venues.html?venue=exeter-phoenix"

We can also purge the cached page:

# Purge one page's cached render
curl -H "x-purge-cache: MYSECRET" "https://newtroubadours.org/venues.html?venue=exeter-phoenix"

P.S. if the renderer has out-quota-ed itself, we can tell the search engine to bacjk off for a bit…

    if (!rendered.ok) {
      // Render failed (e.g. hit the daily Browser Rendering quota).
      if (crawler) {
        // Tell the crawler this response is incomplete/temporary — don't index it.
        return new Response(
          "Prerendering temporarily unavailable — please retry later.",
          {
            status: 503,
            headers: {
              "content-type": "text/plain; charset=utf-8",
              "retry-after": String(RENDER_RETRY_AFTER_SECONDS),
            },
          }
        );
      }
      // Not a crawler (e.g. a debug ping) — fall back to the plain origin page.
      return fetch(request);
    }

(I really should have been documenting the build over the last few months — pretty much all the HTML/JS/CSS coding has been done using claude, with me exploring features and making the odd code nudge — but will maybe get round to retroposting the process at some point.)

One of the features of the site is that it supports view switching deep links into a page. So for example, the generic storyclub page with URL https://newtroubadours.org/storyclub.html:

also lets you link in to view information about a particular club using a url of the form https://newtroubadours.org/storyclub.html?club=leicesterstorytree :

This use of the path and URL parameters meant that when migrating the subdomain URL to a new domain, any legacy links out there would break: I’m using a Github static pages to host the site (so no server side redirects) and was using GoDaddy, where the domain is registered, for the DNS hosting; and whilst GoDaddy would be able to silently redirect from newtroubadours.montystoryteller.org to newtroubadours.org, it wouldn’t be able to pass along the path and URL argument gubbins… So even though there are very few links out there to the old site, which, along with the new domain, currently has pretty much zero search or genAI presence — (can you find it?!;-) — I’m still a persistent URL kind of guy and donlt want to break any links that are out there…

Add to that the page names, which originally had the form new_troubadours_storyclub.html, for example, and which I wanted to simplify to the form storyclub.html, whilst not breaking any original links or passed through/redirected page names. Again, a need for a server side redirect.

Or not… because a little poke around suggested that if I moved to Cloudflare free DNS hosting, I would be able to do the forwarding, would be able to pass through arguments, and would be able to do filename rewrites. (I’ll share some details of how in another post.)

And as an added bonus, I’d also be able to write some MX (email) handling records into the DNS and get free email forwarding for emails on the newtroubadours.org domain. (Again, I’ll post more on that at a later date. Maybe…)

But for now, here’s the site:

New Troubadours — Storytelling, Music & Punk Poetry —A comprehensive guide to traditional and performance storytelling in the UK: story clubs, storytelling shows, folk nights, punk poets, acoustic musicians, grass roots venues and touring performers across the UK. Aimed at adults; appropriate all ages — https://newtroubadours.org/

PS for any old bloggers out there, you know the game… The web is still about links, so for anyone posting “ooh, interesting to see Tony (or is it ‘Monty’ now?) posting about a new site he has set up to promote tradtional, live, storytelling events, called New Troubadours …” etc. etc. along with a link, thanks in advance!:-)

The break was pretty much a clean one, except for a visiting position to see out getting JupyterLite and browser executed R code deployed to support a student in a secure environment (one of His Majesty’s pleasure facilities).

I’m not sure if any other modules are aware that such a thing is possible — depending on which direction the planned update to the data analysis and management module went, there is no reason at all why that couldn’t have offered all the student practical activities as runnable via and within the browser, and as such, been offerable from a technical point of view, and from what I’ve seen of many of the science and other computing modules that include coding activities, they could also in principle execute code solely within the browser (which includes on smart phones) — but I hope that we at least manage to set a precedent showing that it is technically possible to support student coding activities inside such environments. (The OU is a bit like the old NASA model, where it’s difficiult to fly things in space that haven’t already been flown. The very few successes I had in the OU were setting precedents…)

Anyway — this blog is largely now defunt. I was minded to post very briefly just now on a blog I haven’t looked at for getting on for a decade, the Digital Worlds — Distored Reality blog — The Web as Alternate Reality — and may start to use that as a dumping ground to further document the nightmare descent of that parallel nonsense world, but I spend most of my time now in the world of story.

That said, if you must, be critical. Paul Bradshaw is as good a read as anyone I follow in my feeds on being measured in how you make use of such tools. For example, 4 ways you can ‘role play’ with AI and “Journey prompts” and “destination prompts”: how to avoid becoming deskilled when using AI.

But even so, pretty much everything about it still all leaves me cold (not even considering the consequences for financial markets in the short term, then social consequences of the bail outs for those who had in the crash aftermath), and I see no real fun in exploring this tech at all, just the potential for misdirection, manipulation, misapprehension and lots of other negatives beginning with m.

I’m so glad I got of out working for an org that, as with most others, feels obliged to engage with it. The tech I embraced over the years, I knew where it fell short / didn’t cope well, I had a sense of whwre it was likely to go over the next couple of years (and into the five year + horizon that OU courses were developoed and had to deliver into), and I understood why that meant other folk felt they couldn’t use it (because it didn’t”always” work, or didn’t necessarily work “100% properly”, even if you knew where / how it wouldn’t work and what its failure modes were). Now, pretty much everyone is embracing something that doesn’t work “100% properly”, etc., and for which it can be really hard to get a proper sense or understanding of what and how it’s broken. Which it is. Innately. And they’re rushing into a fog bank of uncertainty about how it will develop and where the risks / harms are, not forgetting that the techbro cos. are using the tobacco bros. playbook when it comes to, yeah, well, harm, whatever. Tobacco / sugar etc. crossed paths with biological evolution. Solcial media has been messing with psychology (which leadds to personal and social harm at the individual and family level). GenAI is messing with culture, which knackers things at a generational level. Turn it off.

This includes scripted book cover generation using GOFC — good old fashioned code — that runs the same way each time, and a quarto publishing route that churns quarto markdown into LaTeX and then PDF or epub format.

Two or three years ago I used Lulu to run off a few copies of an earlier version these booklets using a Jupyer Book initiated production route, but the Amazon pricing is better when buying author copies that the Lulu route, so that was another reason to switch to Amazon.

Ideally, I’d do a larger print run with a printing house, but that requires an up-front order in the hundreds to get the price breaks, which not only means an up-front investment in the hundreds of squids for each booklet, but also storage and sales outlet finding problem, as well as shipping and logistics costs and overheads.

Along the way, I did use a bit of AI assist to customise some LaTeX includes to modify the quarto default formatting, to help with some of the regexes in creating a script to convert my legacy Jupyter Book MyST style markdown to qmd, and to fettle image production, such as resizing, autocropping and transparency handling using ImageMagick CLI commands.

I’ve also been exploring generating map related images…

Anyway, the books are in a series — Island Tales — and are available as both ebook and paperback from Amazon:

• A Legend of Godshill reproduces Elder’s version of the tale as well as my take on it, Percy Stone’s lay, excerpts from 19th century tourist guides, and a biographical note on Abraham Elder; available here.
• A Legend of Puckaster Cove includes Elder’s original version of the tale, something approximating my first retelling of it a couple or more years ago, and some historical notes that once again pull on 19th century travel guides to the Island; available here.
  

Now that I’m “retired”, I’m also looking for merch opportunities to try to claw back some of the costs associated with telling tales (a 2-3 hour folk night or open mic, at a pint of shandy every 40-50 minutes, = 15 squids of cost…). So I’ve started looking at producing booklets I can print-on-demand and then sell for a fiver.

I started off with a couple of runs using Lulu, (which I’d used to test publish some Island Tales storynotes a couple of years ago), using the Lulu book cover desginer to generate simple covers, but now I’m looking at Amazon, which is as cheap, if not cheaper, for one-off print on demand, plus it has the global reach in terms of marketing not just print-on-demand books but also ebook variants.

Up till now I’ve been using Jupyter Book (sphinx) but publishing, but Jupyter Book never really met its early promise, is a faff when trying to generate non-HTML outputs (PDF, docx, e-book), and, as I understand it, is in the process of moving to a myst/node rather sphinx/python build process. So instead, I’m moving to quarto, which is well-supported and rich featured, although I haven’t yet worked out a clean way of generating an index. (That said, the epub doesnlt seem to work for me with Amazon KDP; instaed, I have to load a docx into the KDP app, then generate the KPF file. That may simply be down to how i config the epub in the quarto build, or they may be some other issue.)

As my main output will be PDF, I’ve started hacking together LaTeX templates, admittedly with quite a lot of support from Claude.ai and ChatGPT, partly because they can generated the code quick than I can, partly because I don’t (yet) know how to write LaTex all that well.

For the inner book content, I’ve started putting together core templates for the different booklets I want to strat producing, handling things like page layout, headers, header undlerines, page numbering etc. (I’ve still not settled on if / how I want to use footnotes; I currently inline footnotes…)

I also have templates for blurb pages (copyright / about) and title pages etc.

For the covers, I’m using LaTeX templates that allow me to fix a design and then customise it with text, fine-tuned positing, and colour themes. (The idea is that books will be in series, and I want each series to have a similar look and feel.

As part of the cover, I can also (optinally) overlay debug lines.

To improve reproducibility further, I’m using customised .devcontainer in VS Code for the build process so that I can build a book or cover independently of whatever machine I’m on. (This extends to making sure the fonts I want to use are available. My original tests were done on a Mac and I’d unwittingly used fonts that ship with Mac but are commercial fonts. Adding fonts into the build process and then using those means I keep that part of the build reporducible too.)

For the Amazon publishing, the cover needs to mount an ISBN. Again, this is configurable.

The designs are rather utlitiarian (I’ve only just started out on this process too…), but then, so are the designs in the cover design tools on Lulu and Amazon.

The most important point, though, is that while I used genAI to help write the LaTeX, everyhting in the publishing process is scripted and tuned using static configuration files. The process is idempotent. I am in control. There is no genAI in the actual publication route that is likely to give a different result every time you run it.

Anywhere, it’s here at the relaunched montystoryteller.org site…

Next up, I’m pondering porting the tistales.org.uk site away from WordPress so I can drop the Reclaim Hosting plan and save pennies now I’m “retired”, particulalry given they’re stopping email hosting (which has been really unreliable for ages what with being spam blocked etc. Small email providers stand no chance…); wp2hugo looks like it might be interesting for this, but that’s for another day — I’m really trying to limnit my screentime, and have already spent too long at the keyboard today…