Lev Lafayette blogs

Ten Years of Daily Duolingo

lev_lafayette — Sun, 30 Aug 2026 14:00:00 +0000

Recently, I passed a ten-year streak of doing Duolingo every day. It's not like I do the bare minimum either; for the past three years, at least, my end-of-year Duo report records me in the 0.1% of learners on their application. My profile lists a rather impossible list of languages, many of which I looked in my early years and, I must admit, were handy whilst travelling through Europe. These days I'm concentrating on standard Chinese and French, whilst last year I completed the Spanish course as I was travelling to South America. In other words, over the years, my language learning has become more functionally-oriented rather than experimental.

Despite this regular use, I have mixed opinions about Duolingo. I will argue that Duolingo is the best language-learning application currently available. Nothing else comes to mind that has an extensive range of courses, that has a similar depth of content, that has regular updates, expansions of content and alignment to CEFR language competence. It continues to improve in areas such as spoken content, grammar, and conversational use of a language.

Likewise, I will also argue that Duolingo is the worst when it comes to business practises. It has built itself on countless hours of willing volunteers who gave advice and highlighted bugs over many years when the free version of the application was useful. Now that Duolingo has reached a position of apparently unassailable market dominance for language learning it has turned the screws to exclude community input (e.g., closing the forums, closing the language incubators) and, with aggressive and often gross advertising, has made the free version of the application almost unusable.

As a result of these practises, Duoling is the most profitable venture of its type. In terms of market evolution, it successfully outmanoeuvred potential alternatives in the competitive stage of the market development by engaging in the highest levels of community input but without providing community empowerment. Now it has reached the stage of market dominance, it has what is erroneously called "competitive advantage" in business studies, but is really "monopolistic advantage" when viewed through the lens of economic analysis.

I am sure the leaders at Duolingo are very well aware of this; whilst they could be true to their origins and actually contribute substantially to such a development, I suspect their business logic will run contrary to it. Duolingo argued that: "Our mission is to develop the best education in the world and make it universally available". They have claimed that: "The freemium business model is good for our mission and our business. We grow by offering an incredible free product and monetize by making the paid version worth it. This fuels a growth flywheel...". The reality is, however, that they don't really have a freemium model anymore.

However, at is core, Duolingo is actually a fairly simple product. In terms of computer design, flashcards (whether words, sentence gaps, etc) are simply an associative array of text and audio, text for grammar with a simple user interface (the simpler the better; Duo's distracting and unnecessary animations are awful) and spaced repetition algorithms. At the moment, numerous community-built Anki cards provide the highest level of development in this regard. Ultimately, however, Duolingo is a very tempting target for a community project, which I think is inevitable, which leads to an interesting conclusion that, in the near future, Duolingo's functionality will be open-sourced.

Attachment	Size
tenyearduo.png	10.31 KB

Maximum-Likelihood (ML) Phylogenetics in High Performance Computing

lev_lafayette — Tue, 04 Aug 2026 12:15:36 +0000

Given a DNA or protein sequence, Maximum-Likelihood (ML) Phylogenetics searches for an evolutionary tree that would most probably have produced the observed sequences. This probability is based on a statistical model of how sequences evolve.

For example, consider the DNA sequences for the following primates with standard DNA nucleotide abbreviations: adenine (A), cytosine (C), guanine (G), and thymine (T). For what it's worth, a single 7-letter sequence like "ACGTTAGC" is normally too short to build a meaningful tree.

Species Sequence Difference
Human ACGTTAGC
Chimp ACGTTAGT Differs from Human at position 7
Gorilla ACGCTAGT Differs from Human at position 4, 7
Orangutan ATGCTAGT Differs from Human at positions 2, 4, 7

Apart from the quantity of differences, there are qualitative changes as well, with the probability correlated to the actual physical alteration required. Changes between adenine and guanine (A to G), for example, are more common than adenine and thymine; the former is a transition mutation, swapping one purine base for the other purine, while the latter is a transversion mutation, swapping a purine for a pyrimidine base.

With our toy example, there are a number of possible evolutionary trees. It is possible, for example, that the four species are entirely separate and have different common ancestors. Or you might notice that the difference between Gorilla and Orangutan is a slight mutation, between cytosine and thymine, which is relatively common. With the four species, there are only three possible unrooted trees. But once the number increases, the potential variation increases; with ten species, there are 2 million potential evolutionary trees, and with a hundred species, the number is 2*10^182.

There is currently more than 6,500 living mammal species. When one starts looking at other taxonomic levels (phylum, kingdoms, domains) one quickly ends up with far more potential genonic trees than atoms in the known universe. As a result, phylogenetic programs such as RAxML, IQ-TREE, PhyML, and FastTree-2 do not examine every possible tree. Rather, they search the tree space, repeatedly making small rearrangements based on whether the observed DNA alignment is more likely than in previous iterations. Eventually, this results in a tree with a very high likelihood of accuracy. There are many algorithms for calculating likelihood, from the deliberately simplest and most unrealistic (Jukes-Cantor model from 1969) to contemporary and complex models, such as the Newton-Raphson method.

As can be imagined, accurate and fast computation of these complex phylogenetic searches is almost implausible even on powerful desktop computing systems; one will invariably look to a high-performance computing cluster with GPU acceleration. So, in addition to variation in the algorithms that are used, Maximum-Likelihood (ML) Phylogenetics searches use a different architecture. IQ-TREE 2 is designed to use GPU accelerators to accelerate maximum-likelihood scoring functions and tree evaluations for genomic datasets, whereas RAxML/RAxML-NG uses fine-grained shared-memory parallelisation and coarse-grained MPI for scalability.

Further, just to make matters even more complex, maximum likelihood is not the only approach for phylogenetics. Whereas maximum likelihood, which finds the most probable tree from the observed data, calculates a single tree, Bayesian inference yields a range of probable trees and their relative probabilities, incorporating prior knowledge (e.g., the fossil record). Bayesian inference is computationally more intensive and therefore typically slower, and ML and Bayesian approaches are often implemented in the same application. Applications such as BEAGLE perform inference with GPU acceleration for core likelihood and gradient calculations, whereas MrBayes uses GPU-accelerated likelihood calculations with BEAGLE and also supports MPI and OpenMP for inference.

Ultimately, one has to make choices about speed and accuracy; a valiant example of such an inquiry is an ongoing project by Paul Gardner at the University of Otago, who has differentiated between bioinformatic tools along these dimensions. He has written several papers on RNA sequencing alignment programs and metagenomic analysis tools, noting significant divergence. Notably, he has discovered that it is not the discipline the software comes from, nor the academic "status" of the institution, but rather developer commitment to fixing issues: "active, long-term software maintenance is a key factor for producing accurate tools". This author hypothesises that the same approach can be applied to phylogenetics.

Further Reading

DNA Replication and Causes of Mutation
http://www.nature.com/scitable/topicpage/dna-replication-and-causes-of-m...

List of phylogenetics software
https://en.wikipedia.org/wiki/List_of_phylogenetics_software

Substitution Models
https://en.wikipedia.org/wiki/Substitution_model
https://evomics.org/resources/substitution-models/nucleotide-substitutio...

Hybrid Parallelization of the MrBayes & RAxML Phylogenetics Codes
https://cme.h-its.org/exelixis/resource/doc/Phylo100225.pdf

Image from:
Comparative genomic profiling of transport inhibitor Response1/Auxin signaling F-box (TIR1/AFB) genes in eight Pyrus genomes revealed the intraspecies diversity and stress responsiveness patterns
https://www.researchgate.net/publication/380360506_Comparative_genomic_p...

GNOME File Picker Issues

lev_lafayette — Fri, 12 Jun 2026 05:18:25 +0000

Recently, I had noticed that when I add an attachment (e.g. to email, via a web browser), GNOME's GTK file picker was only providing a list from recently opened items. Previously, it provided a PATH selection, which was a lot more useful for me. When the file picker opens, normally, there is a left-hand sidebar containing items like "Home", "Documents", "Downloads", etc. A normal workaround if this isn't available is to select Ctrl+L to open a location-based entry box, but this wasn't available either. The following question arises: How do I revert to the previous PATH-based menu? What changed, and why?

The first step to check the version of the operating system, desktop environment, and version of GNOME:

$ cat /etc/os-release PRETTY_NAME="Ubuntu 22.04.5 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.5 LTS (Jammy Jellyfish)" $ echo $XDG_CURRENT_DESKTOP ubuntu:GNOME $ gnome-shell --version GNOME Shell 42.9

There are reasons why I'm using a 2022 LTS version of Ubuntu, but that's for another post; what's important is that this version of Ubuntu and the version of GNOME provided a clue toward what happened.

The next step was to search in the gsettings for the file picker's "recent" choice, and that confirmed what was being experienced:

$ gsettings list-recursively | grep recent ... org.gtk.gtk4.Settings.FileChooser startup-mode 'recent'

GNOME 42 uses a mix of GTK3, and GTK4 plus xdg-desktop-portal, and many applications now use the GTK4 file chooser, which starts in the "Recent" view. Many developers (including GNOME developers) are increasingly adopting the "Recent" view and document-centric workflows over traditional filesystem navigation. This a terrible idea, albeit very common in a brain-dead Sharepoint-style mentality where the implicit knowledge gained by showing the filesystem and PATH is removed from the user. It will make users increasingly ignorant of computer systems, and it will make it harder for those who want to use the system to their advantage. The only people it benefits are the wilfully ignorant. Remember: Stupid is wrong and must be defeated.

The first suspect was that a GTK4 package or Flatpak update changed the setting. Another test resulted in some interesting answers, which added to this suspicion:

$ gsettings get org.gtk.gtk4.Settings.FileChooser startup-mode 'cwd' $ gsettings range org.gtk.gtk4.Settings.FileChooser startup-mode enum 'recent' 'cwd' $ gsettings describe org.gtk.gtk4.Settings.FileChooser startup-mode Either "recent" or "cwd"; controls whether the file chooser starts up showing the list of recently-used files, or the contents of the current working directory.

To fix this bug, change gsettings to 'current working directory' and verify:

$ gsettings set org.gtk.gtk4.Settings.FileChooser startup-mode 'cwd' $ gsettings get org.gtk.gtk4.Settings.FileChooser startup-mode 'cwd'

Whilst that's fixed the problem, the question still remains of what caused it in the first place?

National Supercomputing Centre and Sunway TaihuLight

lev_lafayette — Mon, 06 Apr 2026 12:34:25 +0000

Wuxi is home to China's National Supercomputing Centre and the Sunway TaihuLight supercomputer. This system came to global attention in June 2016, when it topped the global Top500 list of the world's most powerful supercomputers, far surpassing the second system (also Chinese, the Tianhe-2A) let alone the third (Titan, from the United States). Not only that, Sunway TaihuLight held the top position for an unprecedented and never-repeated two years in succession. until the US system, Summit, hosted at Oak Ridge National Laboratory, took the top position in the June 2018 metrics. Nevertheless, almost ten years later, Sunway TaihuLight has remained not only a Top500 supercomputer, but remains at in the top 25 systems (placed #24 in November 2025) without any change to the original configuration in all that time, a blunt indication of how advanced it was at the time.

The Sunway TaihuLight consists of 10,649,600 cores and a Rmax of 93.01 petaflops and an Rpeak of 125.44 petaflops. With various import restrictions in place (those who preach free trade don't like practising it with a competitor), the processors are a home-grown variety, Sunway SW26010 260C, a 64-bit RISC running at 1.45GHz. The clock rate might seem low, but it matches the requirements of a manycore processor and, as the manufacturer code suggests, the Sunway SW26010 consists of a truly impressive 260 cores per processor, arranged as four clusters of 64 Compute-Processing Elements (CPEs) in an eight-by-eight array. These CPEs support SIMD instructions, making the chip (at a very high level) seem like something between a traditional CPU and a GPU architecture. The CPE clusters also have a more conventional general-purpose core, the Management Processing Element (MPE), that provides supervisory functions. As each node has 260 cores, there are "supernodes" of 256 nodes, and each cabinet holds 4 supernodes. There are 40 cabinets in total, providing over 10 million cores. Sunway has its own interconnect, with a five-level integrated hierarchy: (i) computing node, (ii) computing board, (iii) super-nodes, (iv) cabinet, and (v) complete system with a network link bandwidth of 16GB/s and total I/O bandwidth of 288 GB/s.

The operating system is also custom-built, Sunway Raise OS, based on Linux. Common compilers (C, C++, Fortran), an automatic vectorisation tool, basic math libraries, and a customised version of OpenACC are available. The software build system is also specialised, targetting the Sunway processor. Whilst minimal modifications have been sought, those applications designed for GPUs have been "signiﬁcantly more challenging". Nevertheless, dozens of applications have been written which can, in theory, scale to use the entire system, with early tests of atmospheric simulations scaling effectively to eight million cores. Other early simulations of note include atomistic simulations of silicon nanowires and ultra-high-resolution global ocean surface wave numerical simulations. Parallel software compilation at the node level generally uses MPI. For the four CGs within the same processor, software can use either MPI or OpenMP, but within each CG, Sunway OpenACC is used. Sunway OpenACC uses the OpenACC 2.0 syntax but targets the CPE clusters and includes parallel task management, heterogeneous code extraction, and data transfer descriptions. Syntax extensions from the original OpenACC 2.0 standard include ﬁner control over multi-dimensional array buﬀering, and packing distributed variables for data transfer.

The Sunway TaihuLight will be remembered alongside other systems that are "giants" in history for their performance, architecture, and lasting impact, and contributions to science, such as ENIAC, UNIVAC, CDC-6600, Cray-1, Beowulf, and RoadRunner. A system as powerful, innovative, and novel as this comes along perhaps once a decade, and after 10 years of operation, Sunway TaihuLight has earned its place in computing history. All the engineers and administrators who have built, operated, and maintained this system deserve respect for their contributions. What is especially remarkable is that, due to the political climate, this system had an additional requirement for novel design. However, I have been informed by a trustworthy source to "watch this space"; there are plans for a system ten times as powerful in the very near future.

Guizhou National Data Centre and Radio Telescope

lev_lafayette — Mon, 06 Apr 2026 11:14:33 +0000

A few weeks ago, I had the opportunity to visit Guizhou, China's National Big Data Comprehensive Pilot Zone. At first blush, the choice of Guizhou seems to be an unusual one. With a population of 38 million, it ranks 17th in the list of administrative divisions and 18th in population density, and it is even further behind in terms of GDP (22nd). From the principle that it is most efficient to conduct compute near where data is located, one would expect that such a data centre would be located in provinces with a greater population and economic activity, such as Guangdong, Jiangsu, or Shandong, or maybe even in accord to scientific output, which would also include Beijing and Shanghai. Instead, Guizhou, with its rugged karst formations and dense forests and lower level of economic development (fourth lowest in GDP per capita in 2020), has been part of a "Big Data Guizhou" strategic plan launched in 2014 which includes a threefold approach; "Big Data", "Big Poverty Reduction", and "Big Ecology" by then governer, Chen Min'er.

With low energy costs and a consistently cool climate, Guizhou has established the Guizhou Cloud, sponsored by the Guizhou Big Data Development Administration and supervised by the Board of Supervisors of Guizhou State-Owned Enterprises. This in turn has attracted major national corporations to its numerous data centers, notably Apple's iCloud China and Huawei's Cloud Service Base, along with Tencent, Alibaba Cloud, the AI firm SenseTime, as well as hosting the annual China International Big Data Industry Expo since 2015. These corporate decisions are notable enough in their own right, but what really makes Guizhou such an attractive place for a national data hub is the presence of the Five-hundred-meter Aperture Spherical Telescope (FAST) which, as is evident from the name, has a 500m diameter dish (I call it a "wok") making it the world's largest single-dish telescope.

Radio telescopes are essentially antennas and receivers for radio waves, with frequencies ranging from around 20 kHz to around 300 GHz, just as an optical telescope collects data from the visible portion of the spectrum. Primary local sources for radio waves include the Sun, Jupiter (due to its magnetosphere), and Jupiter's moon, Ganymede. The Galactic Centre of the Milky Way is an especially powerful source, as are supernova remnants, such as Cassiopeia, and neutron stars, such as pulsars and Rotating Radio Transients (RRATs). Primordial black holes and extraterrestrial intelligence communications are two other speculative, currently unobserved sources. The main point is that to derive information about these radio sources, one needs to collect radio wave data, and the more data you collect, the greater the chance you will find something interesting. Thus, to collect more data, one wants a larger receiver.

FAST is a very big receiver and it collects a lot of data, roughly 100TB per day. Construction began in 2011, testing began in 2016, and it was fully operational in 2020. Even before becoming operational it discovered two pulsars and by 2021 it has discovered an incredible 500. Hardware innovations are continuing; late last year, China Environment for Network Innovation (CENI), announced that they had conducted a (somewhat contrived) data transfer test of 72TB between FAST to Huazhong University of Science and Technology in Central China's Hubei province. The data transfer test, which would normally take 699 days was completed in a mere 1.6 hours. Further, China has announced that it will build an additional 24 radio dishes of 40m diameter around FAST, creating an array that will mimic a massive 10km diameter dish, and boost telescope resolution by 30 times.

Mention must also be made of the rest of the Tianyan Scenic Area, which hosts the FAST system. Apart from the stunning natural beauty of the region, and the FAST Observation Platform (complete with bungy jump during holiday weeks), the site also hosts an impressive Astronomy Experience Hall, the Astronomical Space-Time Tower (at 99.999 metres), the Nan Rendong Memorial Hall (the astronomer who was the main drive for setting up FAST), the Dome Flight Cinema, and a Aerospace Science and Technology Museum. Like many massive engineering projects in China, they have also turned the site into an informative destination for national and international visitors.

Critical Issues for the Global Climate

lev_lafayette — Tue, 03 Mar 2026 06:40:13 +0000

This presentation covers the science of Earth's "energy budget" of heat inputs and losses, and describes the overall climate system. It continues with a description of the Industrial Age, the introduction of direct temperature measurement, and the resulting temperature rise from burning fossil fuels. This is followed by a description of how Greenhouse Gases operate on a molecular level, the increase since the pre-industrial period, and the carbon cycle. After this, human activity and projections are considered, followed by changes to species' habitats and the possibility of an Anthropocene Extinction Event, then energy trajectories and future global policy directions. Concluding remarks identify climate change as a critical issue and one subject to "race conditions", and note that the policy route, whilst necessary, is currently falling short of requirements.

This was a presentation to Future Day 2026, March 2-4. A transcript is provided along with the accompanying slide deck.

Transcript:
http://levlafayette.com/files/2026FutureDayGlobalClimateTranscript.pdf

Slides:
http://levlafayette.com/files/2026FutureDay_GlobalClimate.pdf

Youtube
https://www.youtube.com/watch?v=i-bAZ4NVhCM

Attachment	Size
2026FutureDayGlobalClimateTranscript.pdf	87.13 KB
2026FutureDay_GlobalClimate.pdf	1.37 MB

The Environmental Management of Artificial Intelligence

lev_lafayette — Sun, 07 Dec 2025 02:21:32 +0000

With at least three disciplines of interest - energy and climatology, public economics, and high-performance computing - there is the issue of whether current trends in artificial intelligence are environmentally sustainable. The following is a basic sketch of electricity usage, needs, and costings.

The promise of artificial intelligence is as old as computing itself, and, in some ways, it is difficult to distinguish from computing in general. As the old joke goes, it is no contest for real stupidity, and an issue that became all too evident to Charles Babbage when he developed the idea of a programmable computer:

On two occasions I have been asked, - "Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?" ... I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.
-- Passages from the Life of a Philosopher (1864), ch. 5 "Difference Engine No. 1"

Of course, we now have computational devices that can attempt to solve problems with incorrect inputs and perhaps provide a correct answer. That, if anything, is what distinguishes classical computation from contemporary artificial intelligence. Time does not permit a thorough exploration of the rise and fall of several attempts to implement AI; however, the most recent version of the last decade, which involves the application of transformer deep learning and the use of Graphics Processing Units, continues to attract investment and interest.

Transformer architectures for artificial neural networks are a fascinating topic in their own right; attention (pun intended) is directed to the use of GPUs as the main issue. Whilst the physical architecture of GPUs makes them particularly suitable for graphics processing, it was also realised that they could be used for a variety of vector processing, providing massive data parallelism, i.e., "general purpose (computing on) graphics processing units", GPGPUs. However, physics gets in the way of the pure mathematical potential of GPUs; they generate a significant amount of heat and require substantial electricity, and that's where the environmental question arises.

The current global electricity consumption by data centres is approximately 1.5%, according to the International Energy Agency. However, that is expected to reach 3.0% by 2030, primarily due to the growth of AI, which would include not just the GPUs themselves, but also the proportional contributions by CPU hosts, cooling, transport, installation, and so forth. A doubling of energy consumption over a period of a few years (from c400TWh in 2024 to c900TWh in 2030) is very significant and, if the estimates prove to be even roughly correct, then further energy utilisation needs to be considered as an ongoing trajectory for at least another two decades until it becomes ubiquitous.

There are essentially two ways of managing energy used in production with an environmental perspective, given a particular policy. One approach is high-energy and high-production, concentrating on renewables or non-GHG energy sources. The other is a reduced-energy, high-efficiency approach that concentrates on better outcomes, "doing more with less". More important than either of these, in my opinion, is the incorporation of externalised costs into the internal price of an energy source. One graphic example of this is the deaths per Terawatt-hour by energy source. Solar, for example, is more than three orders of magnitude safer than coal.

To satisfy existing and expected demand, the AI industry is, in part, turning to nuclear for its energy needs. Data centres tend to be located within population centres, partially due to latency reasons, whereas renewables like wind and solar require a significant amount of land area. Additionally, where existing nuclear power plants and infrastructure are already in place, it is relatively inexpensive, even compared to battery technologies. Nuclear provides sustained power generation not just throughout the day, but across months and seasons. With approximately 5% of generation lost in transmission, nearby power sources are more efficient.

The main weakness of nuclear power is the time and cost associated with the construction of new plants, and in this regard, the big data centre and technology groups are taking a gamble. They assume that there will be sufficient demand for AI and that they can generate enough income over the next decade to cover the costs. Whilst they are very likely to be correct in this assessment, and certainly the choice of nuclear is preferable to the fossil fuel sources that are currently driving most data centres (e.g., methane gas in the United States, coal in China).

As for demand-side considerations, these can include the energy efficiency of the data centres themselves, the way models are designed, and the way AI is utilised. Cooling is an especially interesting case; as mentioned, GPUs run quite hot, and to avoid catastrophic failure, they require effective cooling. This is usually done with evaporative cooling, which means significant water loss, or by chillers, which doesn't mean much loss, but a huge amount of water for cooling. A third option is dielectric liquids, such as mineral oil, which results in a data centre that is quiet and at room temperature, while servers operate at an optimal temperature. The main disadvantage is the messy and time-consuming procedures for upgrading system units.

The model design also presents some opportunities for improvement. The typical approach is to train neural networks with large quantities of data; however, the more indiscriminate the data collection is, the greater the possibility of conventional error. As some critics suggest, an LLM is essentially a language interface that sits in front of a search engine. A smaller but more accurate collection of data can be more accurate, as well as being less resource-intensive to train in the first place. A number of smaller models can operate with connective software for matters outside of the initial module's scope instead of a monolithic approach.

Finally, there is the matter of what AIs are being used for. Certainly, there are some powerful and important success stories such as the key designers behind AlphaFold winning the 2024 Nobel Prize in Chemsitry for protein structure prediction and, as many contemporary workers (especially in computer science) are all too aware, the ability of AIs to produce code is quite good, assuming th developer knows how to structure the questions with care and engages in thorough testing. Additionally, the increasing application of these technologies in robotics and autonomous vehicles is disconcerting, as illustrated by the predictive and plausible video "Slaughterbots".

On the consumer level, an AI can perform tasks which a human is less efficient at. So rather than simply asking "how many tonnes of a GHG does AI cause", a net emissions question should be asked, appending "... compared to human activity", that is, productivity substitution. However, with effectiveness comes the lure of convenience, as it attempts to extend the use of AI to everything, even when human energy usage would be less than that of an AI-mediated task. Ultimately, it is the combination of human failings, a combination of laziness (always choosing convenience), wilful ignorance (not knowing and caring about energy efficiency), distractibility (extending AI for trivialities rather than tasks of importance), and powerlust (commercial or political), that present a continuing challenge to the prospect of implementing an environmentally-sustainable development of artificial intelligence.

Against Default Modules

lev_lafayette — Tue, 21 Oct 2025 05:57:21 +0000

Environment Modules allow users to dynamically modify the $PATHs in their shell environment. In particular, it provides the ability to switch between different software applications and versions. It is a very handy tool for developers who want to test against multiple versions or compilation options of an application and for users on a multi-user system (e.g., high performance computing) where both consistency and the opportunity to introduce newer features needs to exist for different research groups. The two major environment modules systems are the older, Tcl-based system, and the newer Lua-based system (Lmod). Both allow default versions of a software application to be set as in the $PATH when invoked. This can cause problems and should be disabled.

With a standard module system, one can see the modules for an application. For example, using a LMod system:

$ module av GCC/ ------------------------------------ /apps/easybuild-2022/easybuild/modules/all/Core ---------------------------- GCC/11.3.0 (L) GCC/12.3.0 GCC/13.3.0 (D)

This illustrates that from the GNU Compiler suite (GCC), the application GCC/11.3.0 is loaded (by default in this case as part of the login) and that GCC/11.3.0 is the default as part of a wider toolchain. That is, if the command

module load GCC

is invoked, the existing environment $PATH will change so that it points to the binary, libraries, etc for GCC/12.3.0 instead of GCC/11.3.0. In the older, Tcl-based modules system, it was possible to have both versions loaded simultaneously; the most recent version would be the initial search path, but if a particular library, for example, couldn't be found, it would search whatever was in the $PATH. That could be an interesting issue for the replication of results.

Having a default obviously saves a few keystrokes, but problems can result. The major issue is that when new software is introduced by the local friendly HPC engineers, then the default changes. For example, the latest version of GCC, at the time of writing, is 15.1 (April 2025). If that was installed on a system, one would witness:

$ module av GCC/ ------------------------------------ /apps/easybuild-2022/easybuild/modules/all/Core ---------------------------- GCC/11.3.0 (L) GCC/12.3.0 GCC/13.3.0 GCC/15.1.0 (D)

Instead of loading GCC/12.3.0 from the

module load GCC

command, LMod will add GCC/15.1.0 to the $PATH. Is this a problem? Most certainly because when software (version, compiler) changes, the results can change. For example, with the release of GCC/13 it included the statement: -Ofast, -ffast-math and -funsafe-math-optimizations will no longer add startup code to alter the floating-point environment when producing a shared object with -shared. In a nutshell, this means that one can get different numerical results by compiling the same code between GCC/12 and GCC/13 due to different ways the compiler would handle rounding, precision, and exceptions when using floating-point calculations. This could be very interesting if one cares about that supposed hallmark of science, "reproducibility".

To avoid this, at a user-level, one should always use the fully-qualified version of a software application. For example, instead of loading a module with

module load GCC

, whether in an interactive session or job script, one should use the specific version, e.g.,

module load GCC/13.3.0

. Of course, sometimes users don't follow the friendly and sensible advice from their system engineers, and that's where a blunter tool needs to be invoked.

Site-wide LMod configuration (at let's face it, LMod is the overwhelmingly dominant version of environment modules these days) occurs in the lmod_config.lua file,

$LMOD_DIR/../init/lmod_config.lua

. Adding the following lines is sufficient:

-- Do not automatically pick a version when multiple options are available always_load_default = false -- Do not automatically swap modules from the same family auto_swap = false -- Disallow default-version resolution and force the user to specify which version site_defaults = { load_default = false, }

Another option is to export optins in

/etc/profile.d/lmod.sh

or equivalent. This avoids making changes to the lmod_config file.

export LMOD_DISABLE_NAME_AUTOSWAP=yes export LMOD_DISABLE_VERSION_AUTODEFAULT=yes

If a system is using EasyBuild for software installs (and many of us are), EasyBuild will create

.version

files which include the default; this can be removed by changing the

easybuild.cfg

file to have

module_install_defaults = False

One issue that arises from the above, from a support perspective, is that existing user expectations and job submission scripts may come into conflict with a new policy. For example, if a job submission script included

module load GCC

, the job would fail as LMod would be asking for user input on which specific version of GCC they wish to load.

This is a valid concern in the short term, but in the longer term, the benefits of providing consistent environment variables, reproducibility, and educating users far outweigh the costs. Further, the longer one persists with the policy of allowing application defaults the greater the damage.

Command-Line CD Extraction and Formatting

lev_lafayette — Sun, 12 Oct 2025 00:47:48 +0000

LAME (recursive acronym Lame ain't an MP3 encoder) has been an exceptional application for creating MP3 audio files from other formats (e.g., WAV, FLAC, MPEG-1, MPEG-2) or even for re-encoding existing MP3 files. Creating MP3 files is popular due to their extensive use in portable media players, as they combine an impressive reduction in file size with minimal loss from uncompressed audio. LAME was first released 27 years ago, has been in regular development since, and is now bundled with the very popular audio editor, Audacity, and other excellent audio conversion applications like FFmpeg. The following are several examples of how to use LAME on the command line, convert files, and automate this process, along with the use of cdparanoia and ffmpeg. This might be handy if you want to (for example) back up a collection of CDs.

To extract files from existing media, use cdparanoia, a delightfully old and stable tool where the last release was September 2008 and the front-end for libparanoia, which is also used in the cdrtools suite, which can be used for the creation of audio and data CDs. To use cdparanoia to extract files from disk, first use the command to get the information about what drives you have, i.e., $ cdparanoia -vsQ, a verbose search and query. Then, to extract the files, simply use $ cdparanoia -B, batch mode, saving each track to a separate file. The default format is WAV; if one wants RAW, use $ cdparanoia -rB.

Using LAME to convert files, the simplest action is to convert a file with no options. e.g., $ lame input.wav output.mp3. This can be extended with the numerous options offered by LAME. Perhaps the most common is the -V% (variable bit rate) (e.g., $ lame -V0 input.wav output.mp3. Values are from 0 to 9, where 0 is the highest quality (but largest file size), which is good for very high-fidelity recording, whilst 9.999 is the lowest one, which is fine for low fidelity needs (e.g., a monophonic voice recording); the default is 4. Run a test on a WAV file to see if you can detect the difference in audio quality and check the file size differences. Note that despite being a good encoder, LAME is not recommended for archives; use a lossless audio codec (e.g., FLAC).

If one has multiple files to convert, a short loop is an effective strategy (e.g., $ for item in *.wav; do lame "$item" "${item%.wav}.mp3"; done). This is certainly preferable to opening files with a GUI application and running the conversion one file at a time. If one has a number of large files, then use of GNU Parallel is an excellent choice (e.g., $ parallel lame -V0 {} {.}.mp3 ::: *.wav). On a sample CD, the loop conversion took 57.201 seconds; with parallel, 13.264 seconds; it is fairly clear how this makes a difference if you have a large number of CDs. Note that with a smaller number and size of files, the use of GNU Parallel will actually take a longer time than a loop due to the overhead of splitting up the tasks. Certainly, if you are converting the WAV files from a CD, use GNU Parallel. Note that GNU Parallel will use all the cores available on a system; the number of concurrent jobs can be limited with the -j option (e.g., parallel -j2 lame -V0 {} {.}.mp3 ::: *.wav).

Finally, there is a great little tool called mp3wrap, which can wrap multiple MP3 files into a single MP3, but keeps filenames and metadata information, which can be later split back to original files (using the mp3splt command). The output file will be named OUTPUTFILE_MP3WRAP.mp3; keep the MP3WRAP string so that the split program detects that the file is wrapped with this utility. To create a single file, simply use $mp3wrap album.mp3 track*.mp3.

Research Software Engineering New Zealand Conference 2025

lev_lafayette — Tue, 30 Sep 2025 06:49:44 +0000

The 2025 New Zealand Research Software Engineering Conference was held on September 23-24th. It has run for almost a decade, starting in 2016 as the CRI Coding Conference before becoming the Science Coding Conference, a title it kept until 2020, when it became the Research Software Engineering Conference. Throughout this time, it has focused on science and engineering, high performance computing, and cloud computing and is aimed at various coders, sysadmins, software engineers, data analysts, and IT managers from public research institutions and has the endorsement of the RSE Association of Australia and New Zealand.

As the programme indicates, this year's selection of presentations and BoFs has a strong emphasis on machine learning and developments in artificial intelligence. This was evident right from the very start with Nick Jones' keynote address on the first day, and with numerous presentations throughout the conference. A concern must be raised when this pivot to AI/ML involves a recursive comparative testing being conducted with other AI/ML systems. Ultimately, the validity of computational modelling must come not only from the quality of the inputs but the real-world predictive (and hindcasting) value of the outputs.

Thankfully, there was little in the way of "AI/ML marketing hype" at this conference, which really was firmly dedicated to actual computational practice, development of skills and knowledge, and research outputs. Further, there was only a moderate amount of theory, and that's primarily for foundations, as it should be. Being in Aotearoa New Zealand, it is perhaps unsurprising that there were several presentations on Earth sciences, but also of note was the emphasis on climatology, oceanography, and new developments in forensics.

Interestingly, Australian research software development and education were present in a number of presentations, including speakers from WEHI and CSIRO. From Research Computing Services at the University of Melbourne received a surprising highlight with my own presentation, "Programming Principles in a High Performance Computing Environment", the first presentation of the conference and Daniel Tosello's "VSCode on the node" being the third. It bodes well for current and future Trans-Tasman collaboration.

Finally, the Research Software Engineering New Zealand Conference included a small number of explicit community-building presentations. In conferences such as these, presentations are often of a special interes,t but with a much wider and latent feature being the awareness of the directions that other institutions and individuals are taking and the strength of professional connections, a critical requirement not only for an individual's development, but also for raising the collective knowledge of the institutions that they belong to.