PeacockMedia

Progress with the Mac GUI application for Minipro (GUMP) : Use a TL866 programmer to program ROM chips

2024-07-17T02:08:00.000-07:00

The passage of time is frightening. I see that it has been three years since I first posted about GUMP. I had expected a greater response.

In the mean time I've continued to use my GUI wrapper for minipro when burning ROM chips with my TL866 clone because that has been far friendlier than using the command line. It takes away the need to remember the switches or type the path to the binary.

I've used one or two other of my apps in conjunction with GUMP - Peep for 'hexeditor'-style viewing of the data read from a chip or checking the binary to be written. And a makeshift script for taking one file (or a blank file padded with FF or 00) and incorporating a smaller file at a given start address.

This has all been very clunky and disjointed. Plus there are some other features that I wanted. For example, a lookup for the supported device names and a list of 'favourites' to save having to keep a note of which device name I last used for the chips that I burn regularly.

A new version

So I've been doing the work to roll all of this together and add the new features. The result is a new version of GUMP. I'm still testing and developing, but it's now in a state where I can call it 'beta' and unleash it on anyone who wants to try it. [update 22 Jul 24] Now released as a free beta

Above the tabbed area is the 'device' field. You can type the name of your chip if you know it, or press the search button, type a partial string, see results and choose the one you want.

Once selected you can add it to your 'favourites' which are available in a drop-down list at any time.

Below that is the tabbed area. The 'Read' tab obviously allows you to read the chip and see the data in a 'hexeditor' style viewer. You can export this to a file if you want. The 'combine' checkbox allows you to use the data read from the chip as the starting point to build the output data.

Below that is a console area which relays the minipro output. This has minimal value but at least tells you whether the operation succeeded or failed, and sometimes why.

The 'Write/Verify' tab displays the data that you'll be writing to the chip. You can open a binary file, specify a start address, optionally combine this with the data read in from the chip and padd it to a certain size (although a switch allows you to suppress the error from minipro if it's shorter than the capacity of your chip). You can edit this byte by byte.

There are a few useful switches presented as checkboxes.

The Prefs window allows you to specify the location of minipro. This must be installed separately but if you use homebrew that's as simple as brew install minipro You can specify the full path here if minipro isn't in your search path.

[update 22 Jul 24] Now released as a free beta

Structured Data in Scrutiny Scans

2022-10-21T04:02:00.001-07:00

Back in April we launched version 12 of Scrutiny (skipping 11).

Since then the topic of structured data has been raised by users.

Scrutiny has long collected Twitter and open graph meta data, and has displayed those fields in its SEO > Meta Data table. But not structured data provided via json-ld.

It's easy to collect this but not so easy to display it en masse as it has a 'tree' structure.

Here are some first steps. I'm open to discussion on where to go from here. The following is all possible with the current version of Scrutiny, 12.5.0

The json-ld is collected from each page and parsed. Perhaps the most important pieces of information are the @types. Like headings there may be multiple, and at different levels, so like headings they can be displayed in the table, in a single column using a separator and in the order that they appear.

The column can be switched on using the Columns selector above the SEO > Meta Data table

That allows you to see at a glance what pages have the structured data and what type(s) are used.

For more detail on a particular page, a double click in this table opens the Page Inspector:

This is the Meta data > json-ld tab of the Page Inspector. Note that we're not seeing the original json string from the page. It has been parsed and formatted. It's the way that a machine would read it, and if Scrutiny can't read the json, then there will be an error message here. (You can see that I've used @graph here in order to display both the software application and the breadcrumb list within the same chunk of json-ld. If there are multiple chunks of json-ld on the page then they'll be amalgamated here.)

Also in the same inspector is a Twitter Card viewer. This will display information from the Twitter meta data if present, and fall back on Open Graph (og:) fields, as Twitter does. This doesn't look exactly the way that Twitter would display the information - Twitter also draws on info from the Twitter account referenced.

Milestone for MIDISID

2022-07-28T16:44:00.001-07:00

MIDISID has two SID chips (modern SID replacements, SwinSID Nanos) driven by a Pi Pico. It has MIDI in and audio out.

I'd hoped that this version of the boards (above) would be the final 'release' version. There are some very minor tweaks to make but I'll be building 5 of these - one for myself and four for people interested in an 'earlybird' deal. I'll try a black version too.

The firmware has taken many many hours, and it's clear that there will always be new features on the horizon. If it sells then I look forward to the ongoing development.

It's working very well, see the video below, in which it is in 'General MIDI' mode, I'm playing in the MIDI files for the DOOM levels.

Picoterm v1.1 released, choice of colours and Turbo Pascal 3

2022-07-09T15:59:00.002-07:00

A user reported that when using Turbo Pascal 3 on his RC2014, scrolling within the source code editor wasn't working.

My way of investigating issues with escape codes is to load up a modified version of PicoTerm which prints every character as a number. I also had the RC2014 connected to a terminal on my Mac, so I could still use TP3.

Investigating these numbers showed that the scroll up / scroll down (and also the scrolls that happen when you cursor to the top or bottom of the window) all rely on two escape codes that PicoTerm didn't support:

<esc>[1L

and

<esc>[1M

These are not listed on the Wikipedia ANSI escape code page, which has been my main reference. PicoTerm didn't support them before today. (And neither did PiGFX as far as I can see.) But they do work within minicom and Teraterm, so it seems that they're reasonably 'standard'.

It appears that these codes mean 'insert a line at the cursor position, scrolling everything down' and 'delete a line at the cursor position, scrolling everything up'. I have assumed that the number is the number of lines to scroll, although I can't be sure about this until I find some documentation.

Picoterm is built with scrolling in mind - each line of text is a structure, and the display buffer is an array of pointers to these structures, so scrolling is no more than switching 30 pointers around, and clearing a line.

So adding support for these escape codes was straightforward. Each is a variation on the scroll that is already used when the cursor reaches the bottom of the screen.

Here is a quick demo, first showing another new feature of the textmode (mono) version of PicoTerm - the ability to switch colour on startup, according to button A, B or C being held on power-up. White, Green or Amber.

Then a quick demo of the scrolling up and down within Turbo Pascal 3's editor.

More information about PicoTerm generally, and downloads, are here.

Clock, temperature and humidity display using Badger2040

2022-06-21T16:02:00.007-07:00

The Badger2040 is a RP2040 with e-ink display. Or possibly an e-ink display with a RP2040 on the back:

It has other goodies like 2Mb of flash ram, some buttons and lots of connectors such as a battery connector, stemma/qwiic socket and breakout pads for various serial connections. And USB-C for programming and serial in/out.

I've added an Adafruit temperature / humitity sensor using the stemma connector. This particular sensor is AHT20. The only negative thing about that for this project is that it has a fixed i2c address, and I would have liked to hook up two, one for indoor and one for outdoor temperature. I'll have to buy another which is a different type and has a different address.

For now my program is reading the data from the one sensor and displaying the temp and humidity. This took me some time because the libraries for this sensor (that I could find) were either circuit python or arduino. If you want to use either of those things, you only need to write a couple of lines of code. For C (or in this case C++) I had to write my own code starting with i2c_write_blocking() and i2c_read_blocking() and referring to the AHT20 data sheet and the Arduino library. An annoying bug (on my part) meant that I spent more hours than I should have on that.

With that done it was a breeze to print the stuff to the screen (using the Badger C++ example code). I wrote the clock from scratch using the time.h functions within the Pi Pico C SDK to access and process the hardware timer - get_absolute_time(), to_ms_since_boot() etc. I've got the up and down buttons adjusting the hours or minutes up or down, with buttons A and B switching between adjusting hours and minutes.

It looks great. The e-ink display is very easy to read in any light and uses no power, except when updating. I've got the screen updating every 30s or if you push a button to adjust the time. Another curious thing about this type of display is that it keeps displaying its contents even with power disconnected. Meaning if you wear it as an actual badge, you don't need to power it once it's displaying what you want!

The battery pack that comes with the Badger is a bit chunky (two AAAs inside). I don't know how long that lasts, but I might use a small lithium battery instead, that can be stowed more easily on the back.

UPDATE

I've now got it running off my first found-free-in-the-street battery. This is a relatively small one compared to the 500+ mAh ones Big Clive has been finding but it is nice and flat and it has been powering another pico project of mine and showed no signs of flagging. I'll be fascinated to see how long it runs the Badger!

Another improvement I've made today is to make the clock periodically write some details to the flash memory, so that if power is removed, it'll start up displaying the same time as it did when disconnected. That saves a lot of button-pressing after a battery change.

I've been struggling with writing to the flash memory on the Pico and on this device, but today I solved my issue. Basically don't have USB stdio enabled while trying to write to the flash.

Milestone for Scrutiny - v12

2022-04-25T08:06:00.004-07:00

Last year we spent many months rewriting our website crawling engine, the reasons are here.

Since then, Integrity went through a beta period and eventually Integrity, Integrity Plus and Pro version 12 were released.

There have been many updates, those apps are becoming mature, the problems found are now increasingly obscure.

As well as fixing issues with the new Integrity, Scrutiny has been catching up.

It's now based on Integrity. That approach comes with some compromises (see notes below) but the benefits for all of us are great.

It also means a totally different interface. Or rather reverting, because up until version 4 Scrutiny had the open, tabbed Integrity-like interface. The move away from that wasn't popular and Scrutiny's UI has been its weak point.

Notes: (also in the release notes within the .dmg)

There will be no autosaved data for a site until you've run a scan. Your previous scans will only be visible in the previous version. V12 requires new scans before autosaved data is available. This is due to the format of some of the data changing a little,

V12 isn't document-based. This is part of consolidating Scrutiny & Integrity. Being document-based requires a different architecture within the app, introduces the need for all sorts of complexity and workarounds. The compromise is that it's no longer possible to run multiple scans at the same time. Until there's a 'queueing' system built into Scrutiny 12, you'll need to make sure that any scheduled scans don't overlap time-wise.

V12 doesn't have monitoring or orphan check. I don't think either of these features will be missed, and if there is a call for the monitoring functionality, it should be in a separate app.

The layout is different. I should say 'reverted' because until version 4, Scrutiny shared the Integrity-style interface, and the move away from that wasn't popular. Scrutiny's UI for managing websites hasn't been very user-friendly and had come in for some criticism. It also made it impossible for Voiceover to work (requires visible tabs).

Automatic ftp of sitemaps is no longer available. Scrutiny saves the xml sitemap, you must transfer it yourself. ftp and ftp+SSL (ftps) has been fine but sftp (which in my experience is now more common) has been problematic

French localisation is unfortunately no longer available. I'm grateful to French-speaking paying customers and people who have offered to help with the translation. The problem isn't paying for translation, but the fact that any new feature or change to the interface (which are happening all the time) requires an interaction with the translator and I prefer a much more agile process. The apps haven't been noticeably more popular in French-speaking countries than without the localisation.

2022-04-21T09:30:00.002-07:00

Having got so far with the MIDI SID project, I had this real mess of wires and boards. Then I started to add to the carnage with knobs and a display.

The problem was that if something stopped working I couldn't be sure whether it was because I'd got something wrong, or had just jogged a connection.

Happily, prototype boards are so cheap that you can order something without worrying about the cost. This isn't even really a prototype yet, not all of the components are on the board. I just wanted the Pico, two SIDs (in this case swinsids) and the MIDI in circuit soldered firmly in place so that I could experiment with knobs, buttons and displays, and develop the software without fear of breaking it by jogging the desk!

To get you up-to-date if you haven't seen my ramblings about this, I'd love a box with MIDI input, SID chips inside, audio out, and simple controls. I'd like to be able to play a .mid file into it, or plug in a keyboard, with options for polyphonic or monophonic output. (I've done some of this using a C64 and MIDI cartridge, but it'll be good to have a standalone box.) I'd like to try a larger version with enough SIDs to be able to map each midi channel to a voice. Maybe in time I could add the components needed to support real SIDs, but being able to use ARMSIDs would be good as they're a great replacement. (I'm not sure that it's ethical to encourage anyone to use precious real SIDs in something like this.

Quick demo of the thing working (just one channel in, one voice playing). I've got Logic Pro playing a simple tune, with a USB-MIDI converter allowing me to plug that into this board.

If you're interested in the project, I'm going to use the hashtag #MIDISID on Twitter.

2022-03-06T02:28:00.001-08:00

Minstrel the 4th is a modern recreation of the Jupiter Ace, a lesser-known home computer from 1982. It's notable for its use of Forth rather than BASIC which was common at the time.

The Ace had a ZX Spectrum-like rubber keyboard. The Minstrel's is an improvement but its small tactile switches don't really allow for fast typing.

The Minstrel and similar computer scan the keyboard by reading a group of hardware ports which correspond with half-rows of the keyboard. They expose the bus as an expansion port, so it's possible to intercept those port reads. Here's my working prototype, it uses quite a bit of logic. The Pico's main job is to act as a USB host and allow a modern keyboard to be plugged into its own micro-USB port. and maps the keypresses it receives to the appropriate rows and columns of the Minstrel's keyboard matrix.

It's not only the size and arrangement of the keys that limit typing speed, but the fact that the keyboard matrix is scanned every 1/50th second and then not buffered. So even with modern key switches, this makes typing on computers like the Ace, ZX81 and Spectrum is a bit of a plod. It's easy for the Pico to buffer the keyboard input, and present it to the Minstrel one key at a time, at a speed that it can handle.

This also opens up the possibility of connecting a serial terminal to the device to allow listings to be pasted to the Minstrel. It also means that anything you type using a USB keyboard can be echoed to the terminal giving you a paper tape of In the top picture you can see that the finished product has a USB-FTDI adaptor attached for this purpose.

My interface is now in production in small batches, please contact me for details.

Integrity v12 launched

2022-02-28T08:03:00.001-08:00

You may already know that we spent a chunk of last year bringing Integrity's engine up-to-date. You can read the reasons for this in an earlier post:

https://blog.peacockmedia.software/2021/08/the-future-for-integrity-and-scrutiny.html

In the new year, this was released in the form of Integrity Pro 12 beta which was free and unrestricted. Jan and Feb have seen gradual fixes and improvements to that and this week v12 has quietly become the general release (with v10 also available and supported for a while as a safety net).

In case you're not familiar with the range, Integrity is a link checker, it's free and no-frills.

Integrity Plus allows you to manage settings for multiple sites, adds xml sitemap functionality, csv exporting and search box / filter button.

Integrity Pro adds many of Scrutiny's features; a SEO table, HTML validation table and spell-checking.

If you already hold a licence for Plus or Pro then that key will work with version 12, it's free for you. If you'd like to go from Plus to Pro then there's a lose-nothing upgrade, and also from Plus or Pro to Scrutiny.

(Note that if you're moving from v10 Integrity Plus to v12, or from Plus to Pro, this will be useful.)

If you're a blogger or journalist, I've updated the Press Materials to include the new icons.

Scrutiny 12

Scrutiny 12 is very much in progress. As with Integrity, there will be a revision of the pricing, and again, there'll be no fee for existing licence holders (v7 onwards). (Hint: now is a good time to buy or upgrade).

One big question: Scrutiny moved to a very different layout at version 5, which wasn't a popular decision, people liked the more open-plan tabbed layout of Integrity and the classic Scrutiny. Should we return Scrutiny to that look? Your opinion on this would be gratefully received.

2022-02-22T02:25:00.001-08:00

This is a quick help document showing how to transfer website configurations between Integrity Plus, Integrity Pro or Scrutiny.

For legacy reasons*, you may also need to do this if you're moving from the web download of Integrity Plus (non-App Store version) to the new v12 Integrity Plus.

First of all, run the new version to see whether this applies to you. You may see the websites that you set up in your previous version. If you don't, then read on.

This is simply an export / import.

Using version 10 of Integrity Plus, File > Export > Settings > All websites

Integrity will save a .lcsbatch file. Choose a location and confirm.

Using the new version 12, File > Open... and choose the file that you just saved.

*Previously, the web downloads of Integrity Plus and Pro were not sandboxed, while the app store versions were sandboxed. Also, the web download and App Store versions of Integrity Plus had different bundle id's (meaning MacOS will see them as different apps).

Version 12 corrects these inconsistencies, but means that the new Integrity Plus can't 'see' the location where the previous version stored its data, making an automatic import impossible.

In simple terms, the best settings for Integrity and Scrutiny

2022-02-16T04:06:00.000-08:00

The default settings for Integrity, Integrity Plus, Integrity Pro and Scrutiny have been tweaked over 15 years. Generally speaking, they will be the best settings and the most likely* to perform a successful, full and useful scan.

The very short version of this post is: go with the defaults, and only adjust them if you understand and want to use an additional feature, or if you have a problem that may be cured by making a change. Please contact support if you're unsure about anything.

The rest of this post gives a very basic 'layman' explanation of the site-specific options and settings.

In version 12 (in beta as I write this) these settings have been rearranged and grouped into a more logical order. They're listed below as they are grouped in version 12.

Options

These are optional features. In general, only enable them (or change the default) if you understand what they mean and are prepared to troubleshoot if the option causes unexpected results.

This page only: Simple - sometimes you may want to scan a single page. If you want to scan an entire site, leave this switched off.
Check linked js and css files: This will drill more deeply into the site. If you're looking for a straightforward link check, leave this off.
Check for broken images: Finding broken images is probably as important as finding broken links, leave this on
Check lazyload images/load images: It's possible that your site uses lazyloading of images. If you know that it does then you may want to enable this. NB there is no standard for lazyloading images. Integrity will try to find the image urls in a couple of likely places, but this option can lead to false positives or duplication. Be prepared for troubleshooting.
Check anchors: An anchor link takes you, not just to a page, but to a specific point on a page. With this option on, Integrity will check that the anchor point exists on the target page. If you know that your site uses this type of link, and you want to test them, enable this option.
Flag missing link url: Sometimes during development, you'll create links with empty targets, or use # as a placeholder. This is a way to find those 'unfinished' links.

Advanced

Here we have the controls that may sometimes need to be altered to 'tune' Integrity to your site. In general, the default values should work, only change them if you have a reason.

User-agent string: The default values should work almost all of the time. If the user-agent string is how Integrity identifies itself. If this is set to that of a real browser (which is now the default value) then that should be fine. (Occasionally a site will give different pages for a mobile browser / desktop browser. Or to Googlebot.)
Accept language: can be used to check specific language pages of a multilingual site.
Timeouts and delays: Use the defaults. If you have problems with timeouts or certain errors then it may be necessary to adjust these settings.

Site characteristics

Here are a few settings which may need to be adjusted for your particular site. Again, the defaults should be fine, but refer to this guide or ask for help if you have problems.

Ignore querystrings: This is the option that is most likely to need changing to suit your particular site. The default is off and that'll probably be fine. However sometimes a session id or other things can be included in the querystring (the part after the ? in a url.) and sometimes these can cause loops or duplications. In that case the setting should be on. On the flip side, sometimes important information can be included in the querystring, such as a page id, and so for those sites you definitely need the setting to be off.
Page urls have no file extension (more recently renamed 'Starting url has page name without file extension'): The explanation of what this box actually does is lengthy and it's more than likely that you don't need it switched on. In the case where it's required, Integrity should recognise this and ask you an explicit question, and set this box accordingly.
Ignore trailing slash: It's very unlikely that this needs to be switched off (default is on). It has become less important in version 12 because its inner workings are slightly different.

If you're using version 10 or earlier, then you'll have the option to Check links on error pages. I strongly advise leaving this switched off, as it's pretty likely to cause problems or confusion. v12 doesn't have the option.

If you have a custom error page (which is likely to be one page) and want to test the links on it, then test it separately by setting up a single-page configuration to a non-existent url (such as mysite.com/xyzabc)

Rules

If you have a specific problem, then you can sometimes cure that with a targeted 'ignore' or 'don't check' rule.

The other very useful use for rules is to either ignore an entire section of a site, or to limit the crawl to a specific part of a site.

*It may not seem that way if yours is one of the sites that needs a change from default settings, but that's probability for you. In practice, only the querystring setting is an unpredictable 'it depends' setting. Go with the default, contact support if you need help.

Locate : an overlooked feature in Integrity and Scrutiny

2022-01-31T06:51:00.000-08:00

The Locate feature is an overlooked feature in Integrity and Scrutiny. It answers the common question, "Integrity is reporting a broken link on a page but I don't know where to find that page" or "that page shouldn't exist any more".

It tells you, as a user, how to find the link in question and the page it's on. These hyperlinks are clickable.

There may be more than one route to click through from the home page (or your starting url) to the link in question, but this tool should show the shortest.

It's important to distinguish here between link urls, and a single instance of a link.

In this example, I've selected a link url which has tested as good. There may be links with that target url on multiple pages (or multiple links on the same page). For example a link to the home page probably exists on every page of a site, maybe in more than one place on a page.

A context menu* triggered by a right-click or control-click on that url row will only show options that are relevant to that url, or the page at that url. In order to access the Locate feature, you need to right-click one of the link instances, which are revealed when you expand the row.

The By page and By status views both show link instances when the page / status is expanded, so Locate can be accessed in those views after expanding a page/status. All Links is a flat table showing link instances, so Locate will appear when you right-click any row. In all of these cases it's important to only select a single row, Locate can't work on multiple selected items.

Similarly, if you open the link inspector, it concerns a link url and the status of that url when tested, and it has a table listing the instances or occurrences of that url in links. Before using the Locate button (or context menu in that table) it's important to select one of the instances in the table.

Recent versions of Integrity and Scrutiny may have Locate in these context menus but it may not appear to do anything. This is fixed in Integrity and Scrutiny v10.4.12+

UPLOAD.COM for Z80 CP/M - usage

2022-01-16T04:48:00.000-08:00

I feel very honoured to have written a utility which will be distributed as part of the system included with RC2014 kits with CP/M or ROMWBW.

Since upgrading my first RC2014 to CP/M, transferring files has been a bit tricky. My storage medium is a compact flash card, which I can't read with my modern computer system. I can "dd" it to make a backup, which is easy to transfer to a new card. But that doesn't allow me to put a file onto the card, or copy one off using my Mac.

There are methods of reading the CF card on a new system, but I've not made anything work for me.

My favourite method of transferring my files onto the card is a combination of the DOWNLOAD.COM utility which comes with the CP/M system on the A: drive, and my own utility for creating a .PKG file. That's a hex format that works with the download utility. I've written about that here. Spencer now provides a web utility which creates the same pkg file.

The problem

That still leaves me unable to acquire a file that I've created on the RC2014. I tend to develop in Forth and assembly on my modern computer, transferring the files to the RC to run them, so I rarely create files there on the card. But there have been times when I've done so, and it would also be good to selectively back up files from the card rather than making an image of the whole thing.

Spencer asked me to have a go at writing UPLOAD.COM, the partner to DOWNLOAD.COM. I've written about the process of creating the utility separately if you're interested in Z80 programming and the CP/M's BDOS.

Using UPLOAD.COM

Sit UPLOAD.COM in A: alongside DOWNLOAD.COM

Then browse to any drive and use it like this:

A:UPLOAD MYFILE.COM

It is happy for you to give it wildcards in the standard way:

A:UPLOAD MAT?????.COM or

A:UPLOAD *.* for every file in the directory, as I've done below.

Note that my terminal (minicom) doesn't do wrapping, and it doesn't allow me to select and copy the characters that are off the right-hand side. Those characters seem to be lost. The pkg format doesn't allow for breaks in the stream of hex characters.

The answer to this is to use the capture feature of your terminal program. In the case of minicom it's -C filename. With that option, everything that comes to the console is also written to the file.

Here is my capture file, after UPLOADing all files in my B: drive. The process happens very quickly, and I was able to make a very convenient backup of every file in every drive on my CF card in a matter of minutes.

It's very easy to copy and paste any of those files back into CP/M. The first line of the PKG is the instruction to use DOWNLOAD to put the file into the current drive.

UPLOAD.COM for Z80 CP/M - writing the utility

2022-01-16T04:45:00.005-08:00

Below is a pkg file. It's easy to see what's going on. Your source file is in hex and padded to the 128-byte boundary. There's a count and checksum at the end. The first line is the instruction to use 'download.com' to put the file into the current directory as the name specified.

I wrote this utility for my Mac some time ago, it allows me to convert any file, paste the result into the Terminal app I'm using to communicate with my RC2014 and have the file copied onto one of the drives.

What has been missing is a way to do the opposite. To grab a file from the RC2014 as the original binary file. I've written about that problem here. The very neat solution would be to have a Z80 program which does this same conversion on the RC2014 and displays a file that you can copy and paste - and either convert back into a binary or store for re-downloading at some later point.

Conveniently, there's an example program in the CP/M Operating System Manual called A Sample File Dump Utility (5.4). This very nearly does what we want. It takes a filename as an argument and dumps that file as hex to the console (formatted neatly with numbering and spaces, which were easy to take out).

It was also easy to add the necessary count and checksum, output A:UPLOAD at the start along with the filename.

I decided to look up the actual user number, rather than just printing "U0". This is the start of my deep dive into BDOS - a series of operating system calls, documented and available for you to use in your programs. They tend to involve setting up parameters in Z80 registers, setting the C register to the number of the call that you want to make, and then CALLing the base of the BDOS.

I've dabbled in this before, it provides convenient ways to receive keys and output characters to and from the console.

Getting the user number is simple.

But then we came to parsing the passed argument for wildcards and finding matching files. The example program doesn't do this. It simply OPENs the file that's specified by the user, and if that happens to contain wildcards, the first matching file is opened.

UPLOAD.COM would be a whole lot more useful if it does pattern matching and spews out all of the matching files. It now does and I've demonstrated this in my guide.

The CP/M documentation isn't bad. It allowed me to discover that you can pass a filename to a 'first' and 'next' functions (17 and 18) which give you the first and subsequent matches for your filename which contains ? or *.

I learned that the system has a 128-character buffer at a known location, which is used for passing results for certain functions. For filenames It's divided into 4 32-byte buffers and you get a number 0-3 in A which you shift-multiply by 32 and add to the base address of the buffer for the result you need.

After calling the 'first match' function, you have to call the 'next match' immediately, and then again, as long as you receive a valid result. (You can use this same process without checking whether your filename contains wildcard characters. If it doesn't, then 'first match' just returns your file, and 'next' will return 255 meaning 'no more results').

It would seem sensible to use the 'first' and 'next' system at the start and make an array of matching filenames. Instead I chose to do something less efficient, but that I thought was simpler to write, which is a 'get_nth_match' function, which passes the original filename to 'first' and then calls 'next' the appropriate number of times.

UPLOAD.COM is available here and included alongside download.com with the CF cards that you can buy for your RC2014.

Creating a custom template for custom cards in Dendrite

2022-01-06T07:28:00.001-08:00

As is sometimes the case, I'm writing this post primarily for my own future reference.

Dendrite is a development environment that allows you to work with tokens for commonly-used blocks of text when hand-coding a site. Further to that, it gives you simple dialogs for special blocks with structured text and images, known as 'cards'.

It allows me to paste an image and fill in a few fields when I add stories to a page like the one above, this is a newsletter I manage.

There's a block of code near the top of this page, which generates a feature image. This is the code:

It's an image (with 2x and 3x versions) with a caption, all within a div which is floating right.

Each month for some years I've created the resized images and edited this code manually, because that's less effort than remembering how to set up a new template for a custom card in Dendrite.

So today I've gone to the trouble, and have also gone to the trouble of documenting it here for you and for myself.

This text only appears once on the page, but the real benefit is being able to use the 'insert card' dialog to paste the image and write the caption. Turning multiple fiddly copy-and-paste operations into a few slick and mindless strokes.

So here's the procedure. You're welcome, future-me.

Adding a new template to a Dendrite website

First copy one of the existing default templates, found in <website folder>/Templates. (use 'News' because that one uses all of the tokens, which will serve as a temporary reference). Change its name to something memorable (In this case, FeatureImage.txt)

A template file consists of the code for that card, with dendrite-style tokens inserted where information from the card dialog is to appear. The example above should be self-explanatory.

With that in place, Insert Card in Dendrite offers this dialog, which you can use to fill in the information, and Add Image allows you to paste the image (past it once, Dendrite can create the 2x and 3x versions. All you have to do is give it a base name, which also serves as the id for this particular object.)

I use a keystroke to grab a roughly-square image to the clipboard. A paste into the image well in the dialog above crops it to the selected proportions and resizes it with 2x and 3x versions. It's a remarkably efficient way to add 'cards' containing images to a hand-coded web page. (Note that in this example, my new template doesn't use the body, the link text or the link target. So those can be ignored when filling in this card.)

Important things to remember

Remember to select your new template before OKing this card dialog (the drop-down button at the very top of the Insert Card dialog.) I think it defaults to 'News' rather than the last one you chose.
If you use a different image format (as I do here, 200w and locked proportions rather than 150x150 for the regular news stories) remember to make sure those settings are right for this special card and future regular ones.

Dimming an RGB lightstrip using a Tiny2040 (Pi Pico)

2021-12-18T03:13:00.006-08:00

This is just a simple experimental project to try out Pimoroni's Tiny2040 for the first time.

In order to dim LEDs we have to use PWM; change the duty cycle of a square wave. The 2040 chip handles this and so this program is really simple, it just sets up PWM on one pin and handles two inputs, one for brighter, one for dimmer.

The harder part was controlling 12v LED strip using a 3.3v output. I've used a small transistor to amplify to 5V and then a mosfet to step that up to 12v. A 7805 gives us a 5v supply to power the microcontroller and to use in the amplification circuit.

It would be possible to generate three square waves with independent duty cycles, and thus have smooth colour changing with the RGB LEDs.

PicoTerm, terminal application for Raspberry Pi Pico, supported escape codes and user-definable characters

2021-12-17T04:56:00.004-08:00

PicoTerm is an application for the Pi Pico which provides a serial terminal with VGA output and USB keyboard input.

It was developed for the RC2014 and therefore is configured for that computer; eg 115,200 baud and the RC2014 logo at startup. However, if you'd like a custom version, please get in touch.

The Pico has resources that allow for a 640*480 output with a character mode, or 320x240 with full 32,000 colour.

I've recently added support for uploading user-defined character data. I've made this page in order to document the escape codes currently supported, along with information about the new UD character functionality.

Here is the list of supported escape codes, followed by the information about defining characters.

\ESC[?25l | Cursor invisible
\ESC[?25h | Cursor visible
\ESC[H | Move to 0-0
\ESC[s | Save the cursor position
\ESC[u | Move cursor to previously saved position
\ESC[-Row-;-Col-H | Move to -Row-,-Col-
\ESC[0K | Clear from cursor to the end of the line
\ESC[1K | Clear from the beginning of the current line to the cursor
\ESC[2K | Clear the whole line
\ESC[2J | Clear the screen and move the cursor to 0-0
\ESC[-n-A | Move the cursor up -n- lines
\ESC[-n-B | Move the cursor down -n- lines
\ESC[-n-C | Move the cursor forward -n- characters
\ESC[-n-D | Move the cursor backward -n- characters
\ESC[0m | normal text (should also set foreground & background colours to normal)
\ESC[7m | reverse text
\ESC[0J | clear screen from cursor
\ESC[1J | clear screen to cursor
\ESC[3J | same as \ESC[2J
\ESC[nS | scroll whole page up by n rows (default 1 if n missing)

40 col colour only: (sequence is ignored, no effect in 80 col b/w)

\ESC[38;5-n-m | Set foreground colour to -n- (0-255)
\ESC[48;5;-n-m | Set background colour to -n- (0-255)
\ESC[38;2;r;g;bm | Set foreground colour to r/g/b (values 0-255, scaled by picoterm to 0-31 to give 32,000 colours)
\ESC[48;2;r;g;bm | Set background colour to r/g/b (values 0-255, scaled by picoterm to 0-31 to give 32,000 colours)

80-col textmode version only: (from version 1.1)

\ESC[1L | Insert line at cursor position, scrolling down from the cursor's row
\ESC[1M | Delete line at cursor position, scrolling up to the cursor's row

User-definable characters

C/PM (which is widely used on the RC2014) uses 7-bit ascii characters, which means that there are 128 characters (128-255 inclusive) unused and available for defining by the user. In the case of the colourmode version of Picoterm, the character data is simply stored as pixel data (foreground colour/background colour, 8 bytes per character) and drawn to the screen when needed, which means that the 'reverse graphics' doesn't rely on those reverse characters being defined, and therefore works on the default fixed characters and any that you define.

Note that this functionality is present in picoterm colour v0.2.0 onwards, which is available here.

\ESC[?-n-U | followed by 8 bytes, which will be inserted into the user-defined character space for character n (n can be 128-255 inclusive)

When displaying custom characters, the current foreground and background colours apply, these being reversed if reversed graphics is on.

Here is the simple basic program used in the screenshot above to display pacman: (yes, it's possible to use a loop to define the four characters, but beware of MS Basic's tendency to add a space when using STR$. It's expanded here for readability.)

10 PRINT CHR$(27);"[?128U";
20 FOR C=1TO8
30 READ A
40 PRINT CHR$(A);
50 NEXT C

60 PRINT CHR$(27);"[?129U";
70 FOR C=1TO8
80 READ A
90 PRINT CHR$(A);
100 NEXT C

110 PRINT CHR$(27);"[?130U";
120 FOR C=1TO8
130 READ A
140 PRINT CHR$(A);
150 NEXT C

160 PRINT CHR$(27);"[?131U";
170 FOR C=1TO8
180 READ A
190 PRINT CHR$(A);
200 NEXT C

205 PRINT CHR$(27);"[38;5;11m";
210 PRINT CHR$(128);CHR$(129)
220 PRINT CHR$(130);CHR$(131)

230 END

1000 DATA 15,63,127,127,255,252,224,252
1010 DATA 0,192, 224,224,128,0,0,0
1020 DATA 255,127,127,63,15,0,0,0
1030 DATA 128,224,224,192,0,0,0,0

Inconsistent results with Integrity

2021-12-09T09:53:00.002-08:00

Running the existing release alongside a potential new release is a great sanity check to make sure that any new changes haven't had any undesirable effects on the results.

Today this happened (the new Integrity Pro and existing release are showing different results):

It was a head scratcher and I had to sort the sitemap tables and carefully compare them to find the differences.

The reason is that I'd put some A/B testing in place, there are a bunch of pages with differences and A or B in the querystring. It's in an area of the site I'd normally ignore for reasons, but didn't today.

Therefore, the list of pages may feature both the A version and the B version (because the url of the page is the unique key for Integrity). So those pages may be duplicated in the results, or maybe not, and that's a matter of chance.

Ignoring querystrings fixed the problem, and now gives consistent results every time. (Querystrings aren't used for any other reason on this site, so it's fine to ignore them).

Electronic Dice with the Pi Pico

2021-12-09T08:21:00.000-08:00

This was my first project with the Pi Pico. It's a 'Hello World' for hardware/software.

The real reason for two buttons is that I already had the box drilled and fitted with the two buttons for a different project. It made sense to switch the power using an arcade button, because it's automatically switched off when you let go of it. The second button grounds the 'run' of the Pico, which means that it performs the shuffle and display once more. In reality, just the one power button would do, because letting go of that and pressing it again does the same thing as the 'run again' button.

I wrote the software in Micropython because this was my introduction to the Pico. More recent projects use C which suits me much better.

* That should probably be electronic die, because one die is shuffled and displayed. The pico has enough pins to shuffle and display 3 dice, and probably many more with a bit of logic to decode the binary numbers 0-6 (3 GPIO pins) to 7 LEDs.

'Not allowed to load local resource' error with Safari

2021-11-29T04:29:00.001-08:00

If, after archiving a site in 'browsable' / 'processed' form using WebArch or Website Watchman, parts of the page aren't visible, it may be because of Safari's security restrictions.

If a page uses a js file and that file has been properly saved locally, the browser may still not allow access to that file. Open up your browser's console and check:

Here you can see my browser isn't allowing access to mmenudom.js, even though that file exists in the same folder as the index.html file.

The answer is simple, from the Developer menu, select 'Disable Local File Restrictions'.

I should add that this restriction exists for a reason, and that you shouldn't leave it disabled permanently.

Terminal software for the Raspberry Pi Pico

2021-11-07T15:05:00.009-08:00

I am very proud to have been involved in this hardware / software project with Spencer, creator of the RC2014.

The Pi Pico is a powerful microcontroller, it runs at 133 Mhz by default but can be overclocked to 400Mhz. It has two cores and eight state machines. The state machines have their own assembly language and can independently manage i/o or dma tasks.

It is capable of buffering and generating a video signal, VGA or DVI. Using the TinyUSB library it's capable of receiving USB input.

With these features, it's the perfect candidate for this task. With a few supporting components and the software/firmware installed, it's acting as a serial terminal with USB keyboard input and VGA output.

The RC2014 is a retro-style computer running a Z80 at up to 7.372Mhz. It does require you to use some kind of terminal for input and output. Usually this is a terminal program running on a modern computer and that's where this new module comes in. It allows the computer to be used standalone, as seen here with CP/M running on the RC2014. It communicates with the computer using UART serial via two of the Pico's GPIO pins and the Rx and Tx lines on the RC2014 bus. It receives keyboard input via the micro USB socket on the Pico (with an OTG adaptor. NB TinyUSB on the pico seems to work with some keyboards and not others.)

The project required custom software to be written for the Pico in C.

The video output is taken care of by libraries, although the example code certainly did take a while to understand. That part of Textmode PicoTerm largely draws on the text mode example code.

The USB handling uses TinyUSB. The Pico must act as a host (with the keyboard connected via an OTG adaptor).

That leaves the terminal functionality. You need to maintain a buffer to store the characters (because the video functionality makes callbacks, requesting the data for each scanline as it's generated). Scrolling is an important feature of a terminal, so I created a structure to store the character codes for each row of characters, and an array of pointers to those rows. So scrolling is a matter of shuffling 30 pointers rather than 2400 characters.

One of the Pico's cores can handle the scanline generation for the video while the other listens for characters sent from the computer and from the keyboard, buffers and handles those. [update - with the textmode Picoterm, polling was fine and works without dropped characters. With the newer colourmode PicoTerm, because of the increased load of putting characters on the screen, it was necessary to implement interrupts on Rx to receive and buffer each incoming character].

As well as just feeding characters to the screen and scrolling when necessary, it's also important to parse and handle certain escape sequences. These codes can position the cursor on the screen or change the attributes (eg reverse characters). The full list of sequences that we've implemented so far are here on the module's documentation page.

RC2014 Pi Pico VGA Terminal

Submitting page changes to Bing Webmaster Tools / IndexNow

2021-10-26T08:01:00.000-07:00

IndexNow was recently brought to my attention. It seems that Bing are trying to gain some ground on Google by promoting IndexNow for fast indexing as well as a 'one-stop-shop' - submit once to Bing and 'participating search engines' (which I assume won't ever include Google). Read more

With a well-established and open 'XML sitemap' schema, naturally Microsoft have decided to invent their own xml format and submission protocol. To be fair it's very simple, it is no more than a list of urls and doesn't include the last modified and priority fields, which may not have had any effect anyway.

To submit your xml to Bing can't be done by web upload or ftp, but by http POST request. I guess the idea is that content management systems can submit page(s) as soon as they're edited or created, which is a terrific idea.

It's very easy for Scrutiny / Integrity to generate this XML and make the POST request. It's a simpler variation on what these apps do already with sitemap XML files.

This functionality will appear on Scrutiny 10.4.6 (Scrutiny isn't always the first to get a new feature, it happens that the user who has been helping me with this with is a Scrutiny user). Here's the relevant part of the updated manual:

Exporting and submitting XML for Bing Webmaster Tools / IndexNow

IndexNow is designed to allow faster indexing of your page updates by multiple search engines.

From version 10.4.6, Scrutiny can generate the xml in the correct format, split into batches of 500 if necessary, and submit them to Bing for IndexNow.

In Preferences > Sitemap, the options for saving locally and ftp'ing have been altered slightly to read 'transfer' rather than ftp, and these options apply to both traditional XML Sitemaps and the Bing Webmaster Tools XML format.

Bing Webmaster Tools require a http POST request, which contains an API key. To use this feature you'll need to enter your key in this box in Preferences. Here's where to find your key:

As before, you'll need to scan your site and go to Sitemap results. Examine the list to make sure that it contains the pages that you expect. This time the last modified and priority are irrelevant (The Bing format only allows for a list of urls).

Find "Bing Webmaster XML" in the Export button above the Sitemap table, or use File > Export > Bing Webmaster XML.

"Leveraging tools like Scrutiny and Bing's IndexNow can significantly improve a website's visibility by ensuring that search engines quickly recognize and index important updates. Staying ahead in SEO means not just optimizing content but also optimizing how it's discovered." - Maria Harutyunyan, CEO of professional SEO company Loopex Digital

First look at new app LinkDoc (for testing links within locally-stored pdf or docx documents)

2021-09-21T13:54:00.006-07:00

One of the most frequently-asked questions on the Integrity support desk is how to test the links within a local document (.pdf or .doc). It should be possible with Scrutiny, it can parse a pdf or doc but only when Scrutiny encounters it as part of a website crawl.

Rather than shoehorn the functionality into the existing apps, this sounds more like a job for a 'single button' app built for this one purpose. Here it is.

It happens that I'm well into a ground-up rewrite of the Integrity / Scrutiny crawling engine. It's at the point where it runs. There's plenty to do, but for parsing a single page (document in this case) and checking the links, it should be fine. Of course as the 'V12 engine' develops, then any apps that use it, such as the new LinkDoc will receive those updates.

If you'd like to try it, it's available for download now. It's free but in return, please contact us with any problems or suggestions.

Frustrating loop with NSURLSession when implementing URLSession: task: willPerformHTTPRedirection: newRequest: completionHandler:

2021-09-14T06:58:00.002-07:00

This isn't really a question, I think I've solved the problem. But I think I've seen this in the past and solved it then. So I'm posting this here in case anyone else - most likely future me - has this same problem and wants to save some time.

Context: I'm working on the next version of my crawling engine - it moves to NSURLSession rather than NSURLConnection**. Therefore (with the connection stuff at least) it's a ground-up rewrite.

Here's the problem I hit this morning. A Youtube link is redirecting from a shortened link to a longer one. But some unexpected things are happening.

My URLSession: task: willPerformHTTPRedirection: newRequest: completionHandler: is being called and at first I simply put in the minimum code needed to simply continue with the redirection:

if(completionHandler){
completionHandler(request)
}

According to example code found in various places, that should work. The documentation suggests that this is fine: "completionHandler: A block that your handler should call with either the value of the request parameter, a modified URL request object, or NULL to refuse the redirect and return the body of the redirect response."

As you can see from the screenshot above, our url is being redirected over and over, each time with some garbage appended to the querystring.

Integrity and Scrutiny handle this fine. (Though at this point they are using the old NSURLConnection.) However, they have a lot more code in the redirect delegate method than my new one. Notably, they make a mutable copy of the new request and make some changes. Why do they need to do that?

I've a funny feeling that I've seen this problem before. Indeed, adding code to make a clone of the new request and modify it is what cures this problem.

It's not enough to simply clone and return the new request. This is the line that makes the difference:

[clonedRequest setValue: [[request URL] host] forHTTPHeaderField: @"Host"];

(There's more to it than that, but there are reasons why my code isn't open source! Also, I use Objective C, apologies if you're a Swift person*).

In short, when we create the original request, we set the Host field. In the past I have found this to be necessary in certain cases. In fact, the documentation says that the Host field must be present with http 1.1 and a server may return a 4xx status if it's not present and correct.

If we capture the header fields of the proposed new request in our delegate method, the original Host field appears unchanged. Therefore, it no longer matches the actual host of the updated request url. Here, the url is being redirected to https://www.youtube.com/watch?v=xyz, but the Host field remains "youtu.be"

As I mentioned, I've written the fix into Integrity and Scrutiny in the dim and distant past, presumably because I have spent time on this problem before.

I'm guessing that this isn't a problem if you don't explicitly set the Host field yourself in your original request, but if you don't then you may find other problems pop up occasionally.

If future me is reading this after starting another ground-up project and running into the same problem: you're welcome.

** The important delegate methods of NSURLSessionTask are very similar to the old connection ones. Because I'm seeing similar behaviour with both, I do believe that beneath the skin, NSURLSession and NSURLConnection are just the same.

* I'm aware that many folks are using Swift today, and it's getting harder to find examples and problem fixes written in Objective C. I'm expecting that (as with java-cocoa) Apple will eventually remove support. But I won't switch and I'm grateful for Objective C support as long as it lasts.

429 status codes when crawling sites

2021-09-09T01:31:00.005-07:00

I've had a few conversations with friends about Maplin recently. I have very good memories of the Maplin catalogue way back when they sold electronic components. The catalogue grew bigger each year and featured great spaceship artwork on the cover. They opened high street shops, started selling toys and then closed their shops.

The challenge with this site is that it would finish early after receiving a bunch of 429 status codes.

This code means "too many requests". So the server would respond normally for a while before deciding not to co-operate any more. When this happens, it's usually solved by throttling the crawler; limiting its number of threads, or imposing a limit on the number of requests per minute.

With Maplin I went back to a single thread and just 50 requests per minute (less than one per second) and even at this pedestrian speed, the behaviour was the same. So I guess that it's set to allow a certain number of requests from a given IP address within a certain time. It didn't block my IP and so after a break would respond again.

I managed to get through the site using a technique which is a bit of a hack but works. It's the "Pause and Continue" technique. When you start to receive errors, pausing and waiting for a while allows us to continue and make a fresh start with the server. A useful feature of Integrity and Scrutiny's engine is that on Continue, it doesn't just continue from where it left off. It will start at the top of its list, ignore the good statuses but re-check any bad links. This leads to the fun spectacle of the number of bad links counting backwards!

On finish, there seems to be around 50 genuinely broken links. Easily fixed once found.