These days crawling the web¹ is becoming more and more difficult with websites trying to ensure that “only humans” access them. While there are some good arguments in favour of these efforts, I also have some good arguments:

• Trying to crawl an auction website to feed the images / descriptions into an LLM to enhance it, because sellers don’t always provide the details I’m interested in
• Trying to crawl the premium website of a recently deceased comic author because I’m not sure how much longer it will be around…

Anyway, here are some tips to help out in creating a crawler:

• Firs and most important: run this code under a separate user! These days with all the supply chain attacks, using some kind of separation is highly recommended. Whether it’s a separate user (without admin rights!) or even a dedicated virtual machine, run it isolated.
• The idea is to run a full browser and programmatically navigate to the pages we want to crawl. This (a) helps avoid bot detection systems (b) if a CAPTCHA or similar obstacle is displayed, you can intervene and unblock the process and (c) you can help out with steps like logging in to the website.
• What I used (repo with example scripts):
  • The latest Chrome controlled through the CDP (Chrome DevTools Protocol). This seems the best way currently to “blend in”. Browser automation libraries like Playwright or Selenium come with their own pre-packaged browsers, but those are usually older versions and they get quickly flagged as “bots” by the automated systems.
    
    To start Chrome with CDP, just run it on the command line with --remote-debugging-port=9222 --user-data-dir=Chrome_Profile The “user-data-dir” part is technically not needed, since we’re already running under a separate user (you did create a separate user, right!?), but it provides an extra level of isolation. It does mean that if you want to use the cookies for downloading videos for example, you would have to use an extension like “Get cookies.txt LOCALLY” (Chrome extensions are a notorious source for malware, but you are running under a separate user and only providing minimal information to Chrome, right!?)
  • I used Playwright in current project. It supports a couple of different programming languages, has support for connecting to Chrome through CDP and LLMs do a decent job of generating code that mostly does what you need it to do (I found knowledge of web technologies like Javascript and CSS – selectors for example – useful in guiding it in the right direction). I stuck to the synchronous API for this project, since my aim was a one-off archival, rather than a long running performant system. Also, “high performance” systems can easily overwhelm the site we are trying to archive!
  • I saved the website in three ways (see the code):
    • As a dump of all the HTML content (obviously this doesn’t cover additional assets like style sheets, javascript or images). Also, it wouldn’t cover embedded iframes (which this website didn’t use)
    • As a “PDF print” which preserves the visual feel better (and also keeps text as searchable text)
    • All the requests proxied through mitmproxy
  • Speaking of mitmproxy, here are a couple of things I’ve learned:
    • MacOS ships with Python3, but to get the latest version you need to update / install it from Python.org (makes sense in hindsight, but I was used to the GNU/Linux way of “just update all the packages which will get you the latest version of the software”)
    • Run mitmdump: mitmdump --listen-host 127.0.0.1 -w dump.mitm -s strip_hsts.py (the source for strip_hsts.py is in the example repo). This will save all traffic in the “dump.mitm” file.
    • To have Chrome use mitmproxy, use Proxy SwitchyOmega (V3) extension (again, this extension seems alright for limited use in a sandbox environment). Other ways of setting up the proxy suggests setting it system wide – which could capture other, unintended traffic and would cut internet access when I stop the proxy.
    • To set up TLS (aka. SSL/HTTPS) capture – which we almost certainly need in todays world – start with the mitmproxy documentation, but again, I would strongly recommend to only install the certificates for this instance instead of system-wide, as the documentation suggests (again, I want to only capture traffic from this instance, not system wide!)
    • You can convert the dump (which now contains all the data used to display the website – all the images, style sheets, javascript, etc²) from the mitmproxy specific format to the more generic HAR format using the command mitmdump -r dump.mitm --set hardump=dump-9.har Note that this is somewhat inefficient in that it first needs to load the entire dump file to memory, so try splitting up the dumps into chunks not larger than 3/4 of your RAM in advance³. Theoretically you can also use the -q command line parameter to avoid dumping all the URLs on the screen again, but I found that with that option the conversion process sometimes hangs ¯\(ツ)/¯. And finally, these are text based formats, so they benefit from compression with bzip2 / xz / 7zip.

1. Image from pixabay ︎
2. This still might not be enough to later completely recreate the website, because the Javascript might do some client-side processing that involves the current time, random values (used for cache-busting for example) or other sources of randomness. That’s why I also archive the pages as PDFs. And, in my particular case, I am confident that I only need a subset of data from the captured pages which is not affected by these potential problems ︎
3. I tried looking for solutions that would convert each request one by one (“streaming”) instead of loading the entire file in memory, but couldn’t find any ︎

I thought to myself: wouldn’t it be nice if I could glance at the device being charged and quickly understand if it’s fully charged or not? For example the MagSafe connectors have an LED which turns green once the charging is done.

I came up with a couple (yes, literally two) of possible solutions for this:

• There are chargers which display the voltage / current / power being delivered (the Pinepower for example, which I like because it takes a grounded input – hopefully it actually does something useful with it internally!)
• Cables which show the power being delivered through them. It just so happens that the wast majority of the devices we want to charge use USB-C (thank you EU!) and for the remainder I’m fine with sticking an adapter on top the cable or using a separate cable which doesn’t have a charging indicator.

But even after deciding to go with the second solution, I wasn’t done, because manufacturers can surprise you in many ways. For example, here is one cable claiming to be passing through ~120W of power, when – in fact – it’s passing through about half of it:

This is annoying, because it means that it can’t be used to answer the question: “can this charger deliver more power to my laptop or not?”.

Even worse, here is the same cable still claiming to be delivering about 120W, while – in fact – the laptop is fully charged and drawing less than 1W of power:

Which makes it totally useless for the intended purpose (to ascertain if the device is still charging or not). Also, tip: the displays of these cables are frequently very flickery. Which isn’t an issue normally, since I don’t intend to look at it for more than a second. But they have such a low refresh rate that my phone camera captures “in between” images. However, I found that shooting a short video and then exporting a frame of it works for capturing a reasonable image.

To end on a more positive note: I found two cables which, based on my testing, do provide more reliable measurement of the current flowing through them. Here are the links to them on AliExpress (although, given how frequently things change there, the links might be already obsolete…):

• 240W Type C Fast Charging Cable
• PZOZ retractable usb c to type c

I even tested the cables by connecting my phone to my laptop and they do support some data transfer (didn’t test the speeds though). Here are the leads that are connected for the first one:

And here are the ones for the PZOZ retractable one:

For reference, here is snapshot from an “official” high speed/high power cable that came with a Dell (?) monitor:

Interestingly, although it has almost all pins connected, it’s missing two of them – which are connected in the first cable.

But anyway, I’ve went on for long enough. May this be useful to you in picking the right cable for your needs!

On the one hand it has some pretty nice security features, decent hardware (although the effort for disassembling / fixing is higher than other laptops – and the insistence of soldering memory / storage to the mainboard is also annoying).

On the other hand, when using LuLu (a very nice free and open source application firewall for MacOS), one can see that there are dozens of Apple processes which want to talk to the network and it’s hard to understand the purpose of each :(. It’s safe to assume that Apple has very detailed telemetry about each of their devices which is out there.

Well, at least there are the APIs to implement a firewall like LuLu.

PS. Since this is an older machine, I had to install AppTamer and TB Switcher to avoid overheating (yes, I also repasted the CPU/GPU), and, surprise surprise, one of the processes using the most CPU is “siriactionsd” even though I disabled every option related to Siri I could find. So, MacOS also has it’s own type of bloat, like Windows…

It’s nice that more and more devices have USB-C interfaces. Yet, they are not without their particularities:

• Not all chargers / cables are the same. They support different volts and amps, so, to support the fastest charging possible for your device (phone / laptop / tablet), make sure to select the right one.
  • In particular I was surprised that my phone wasn’t “fast charging” with several different laptop chargers I tried. Turns out (with the help of an USB voltage analyzer) that they were delivering 2 Amps at 9V and my phone was expecting 3 Amps for fast charging. Those (Dell / HP) laptop chargers were plenty powerful to deliver the total watts at higher voltages, but I guess 9V just wasn’t their priority.
  • Quick tip if you don’t want to poke around with USB analyzers for finding a good charger: look for something that has “fast charger” in the name and then test it with your equipment (hopefully you can still send it back if it doesn’t work for you). I had good experiences with the Fairphone chargers and with the UGreen chargers / cables.
• USB-C is seemingly symmetric (ie. you can plug it in both ways and “it works”). That said, some cheap products / cables only implement half of the connections, so if something is not working, try rotating the cable 180° (↺) and see if that works.
  • That said, dust/lint buildup can also be a problem, here is a page on the iFixit forums discussing how to clean the ports.
• You can have “USB-C” products which don’t work / don’t charge unless a USB-A to C cable is used (so they don’t work with a USB-C laptop charger for example for a USB-C to C cable). What’s that about?
  • USB-A always delivers a small amount of current at 5V when something gets plugged in (and then the device / source can negotiate higher voltages / amps if supported). USB-C in turn doesn’t output anything though, unless the consumer has some specific resistors. The “USB-C” (and I put USB-C in quotes here, since they can’t really claim to be an USB-C equipment, since they don’t conform to the spec) gadgets which only work with the A-to-C cables are missing these resistors, most probably because the manufacturer wanted to save a couple of cents (I’ve only seen this on cheap devices). Not much that can be done here, other than using the A-to-C cables ¹

Happy charging!

Photo taken from Andreas G with permission.

1. Yes, theoretically we could add the resistors if we have the time / equipment. Also, theoretically it would be easy to make some adapters / cables that have the resistors in them – I’m surprised that I haven’t seen anything like that. Although, I guess, they have a high risk of being used incorrectly, so maybe it’s better this way. ︎

Noting this here in the hopes that it will be useful for somebody (or even me :)) in the future, since it took me longer than I expected to figure this out.

Scenario: you have a Google account with entries in Google Calendar that you would like to see on your iOS device (iPad / iPhone). Some (well, many) sites suggests that you need to install Google Calendar on the device. This is in fact not necessary!

All you need to do is to link your Google account with the native iOS Calendar App by going to Settings > Apps > Calendar and under “Accounts” add your Google Account. È voilà! That’s all folks! Hat tip to this article for reminding me about the procedure.

PS. You might need to do an additional step for the shared calendars, as described by Scott Hanselmann here: go to https://www.google.com/calendar/syncselect and select the calendars you want to be sync-ed to the iOS devices. I’m not sure if this is needed, since I’ve done it several years ago and it was already enabled when I recently set up a new iOS device.

PS. PS. There is now a new option for ad blocking for iOS devices that is well worth considering: uBlock Origin Lite for iOS.

Image taken from John Davey’s Flickr album with permission.

To start with: good ad blocking is the best one can do for their security / privacy online these days!

For a long time adblocking on iOS devices (iPad / iPhone) was limited. One could use DNS based blocking (Pi-Hole or NextDNS are good options), but many advertisement circumvent these blocks. There was AdBlockPlus which worked well, however:

• I feel uneasy about the company behind it given past behaviour
• Even today, when installing on iOS, ABP forces the user to activate their subscription for a 7 days free trial and then one has to dig through the settings to deactivate it – less one gets charged after the 7 days
  • For reference, the setting is under Settings > Apple Account > Subscriptions
• Ad blocking extensions have full access to the content of the websites you’re visiting (by necessity – since they have to remove part of it). I’m not comfortable with giving this company access to everything I visit / type in to a browser.

There is the current gold standard in adblocking – UBlockOrigin – however it’s not available for Firefox under iOS due to some dubious practices of Apple (Apple makes it so that you can’t really install alternative browsers on iOS – all the “alternatives” are actually reskins of Safari – so Firefox for iOS can’t support the Firefox extensions, less it gets banned from the AppStore – hopefully the EU will look into this anti-competitive practice and make it end soon).

So, I was excited when I’ve learned that uBlock Origin Lite is available for iOS: https://apps.apple.com/us/app/ublock-origin-lite/id6745342698

Some thoughts:

• first and foremost: it works well!
• second: it’s free – although I would like to provide some support for the author, they don’t seem to be interested in it currently. So here I am at least spreading the news about it.
• third: I trust this extension more since it has a long reputation of being trustworthy and it’s F/LOSS.
  • Sidenote: Apple doesn’t provide a way to verify that the source code published is the same as was used to build the binary in the AppStore like f-droid does for example – but maybe one day.
• what I also liked: theoretically it also has a minimal mode which doesn’t require access to the current website (see my concerns above about this kind of software). Unfortunately that doesn’t seem to work on the websites I frequent, but maybe it could be used in some situations where higher situations are needed.

And the setup:

• Install it from the AppStore
• Open Settings. Navigate to Safari -> Extensions
• Turn it on (optionally for private windows also)
• To avoid constant prompting, set the websites to “allow all” (the last option on the page)

Enjoy a safer, more private and faster browsing!

Update: also, don’t forget to disable the Advertising ID on your devices. Here is an EFF tutorial about how to.

Noting this here because all the information I could find on the Interned told me “use MakeMKV for it”. And while MakeMKV seems like a fine product, it’s a bit harder to install, since it doesn’t seem to be packaged for Debian at least. I still gave Handbrake a try (since it is packaged for Debian) and lucky me, it turns out it does work with BlueRay disks.

Now, the following instructions are a bit fuzzy, since I used a bit of a round-about way to achieve this, but here is what I did:

• Installed VLC (just from the repositories, nothing fancy)
• Made sure that it could play BDs
  • It was giving the error “Missing AACS configuration file”. Luckily this askUbuntu answer pointed me in the right direction (and worked for Debian 13)
• Once VLC could play back the BDs, turns out Handbrake could also read them!

Since this is short and sweet, here is an extra tip, courtesy of /r/linuxquestions: how to update the chapter names after encoding? Turns out ffmpeg can do it (què surprise :)):

$ ffmpeg -i movie.mp4 -f ffmetadata movie.metadata.txt
# edit movie.metadata.txt
$ ffmpeg -i movie.mp4 -i movie.metadata.txt -c copy -map_chapters 1 movie_out.mp4

(image taken from bluecinetech.co.uk with permission under a CC license)

It’s a very useful little fanless (!) machine. The DisplayPort outputs even support HDMI-CEC when connected to a TV, out of the box. Don’t forget to update the TPM chips if you get one of these nice machines.

It was fun to see

I’ve picked up a Lenovo Yoga 370 because (amongst other features) it has a removable / upgradable RAM slot (yes, two slots would have been even better :)). You can find various sources (including ChatGPT) claiming it to be 16GB, however, if we look up the processor for this particular model on the Intel website, we’ll see that it supports up to 32GB. And indeed, here is the laptop, happily passing memtest86+ with a 32GB module (specifically this one):

In the same vein, Asus claims a maximum of 2x16GB for their Q170T, however the Intel website says 64GB should work with the i5 processor that’s in the machine, and indeed, here is a machine with 64GB happily passing memtest86+:

The lesson is: processor specs are more relevant than (in)official documentation when determining the maximum supported RAM size.