Lev Lafayette blogs

National Supercomputing Centre and Sunway TaihuLight

lev_lafayette — Mon, 06 Apr 2026 12:34:25 +0000

Wuxi is home to China's National Supercomputing Centre and the Sunway TaihuLight supercomputer. This system came to global attention in June 2016, when it topped the global Top500 list of the world's most powerful supercomputers, far surpassing the second system (also Chinese, the Tianhe-2A) let alone the third (Titan, from the United States). Not only that, Sunway TaihuLight held the top position for an unprecedented and never-repeated two years in succession. until the US system, Summit, hosted at Oak Ridge National Laboratory, took the top position in the June 2018 metrics. Nevertheless, almost ten years later, Sunway TaihuLight has remained not only a Top500 supercomputer, but remains at in the top 25 systems (placed #24 in November 2025) without any change to the original configuration in all that time, a blunt indication of how advanced it was at the time.

The Sunway TaihuLight consists of 10,649,600 cores and a Rmax of 93.01 petaflops and an Rpeak of 125.44 petaflops. With various import restrictions in place (those who preach free trade don't like practising it with a competitor), the processors are a home-grown variety, Sunway SW26010 260C, a 64-bit RISC running at 1.45GHz. The clock rate might seem low, but it matches the requirements of a manycore processor and, as the manufacturer code suggests, the Sunway SW26010 consists of a truly impressive 260 cores per processor, arranged as four clusters of 64 Compute-Processing Elements (CPEs) in an eight-by-eight array. These CPEs support SIMD instructions, making the chip (at a very high level) seem like something between a traditional CPU and a GPU architecture. The CPE clusters also have a more conventional general-purpose core, the Management Processing Element (MPE), that provides supervisory functions. As each node has 260 cores, there are "supernodes" of 256 nodes, and each cabinet holds 4 supernodes. There are 40 cabinets in total, providing over 10 million cores. Sunway has its own interconnect, with a five-level integrated hierarchy: (i) computing node, (ii) computing board, (iii) super-nodes, (iv) cabinet, and (v) complete system with a network link bandwidth of 16GB/s and total I/O bandwidth of 288 GB/s.

The operating system is also custom-built, Sunway Raise OS, based on Linux. Common compilers (C, C++, Fortran), an automatic vectorisation tool, basic math libraries, and a customised version of OpenACC are available. The software build system is also specialised, targetting the Sunway processor. Whilst minimal modifications have been sought, those applications designed for GPUs have been "signiﬁcantly more challenging". Nevertheless, dozens of applications have been written which can, in theory, scale to use the entire system, with early tests of atmospheric simulations scaling effectively to eight million cores. Other early simulations of note include atomistic simulations of silicon nanowires and ultra-high-resolution global ocean surface wave numerical simulations. Parallel software compilation at the node level generally uses MPI. For the four CGs within the same processor, software can use either MPI or OpenMP, but within each CG, Sunway OpenACC is used. Sunway OpenACC uses the OpenACC 2.0 syntax but targets the CPE clusters and includes parallel task management, heterogeneous code extraction, and data transfer descriptions. Syntax extensions from the original OpenACC 2.0 standard include ﬁner control over multi-dimensional array buﬀering, and packing distributed variables for data transfer.

The Sunway TaihuLight will be remembered alongside other systems that are "giants" in history for their performance, architecture, and lasting impact, and contributions to science, such as ENIAC, UNIVAC, CDC-6600, Cray-1, Beowulf, and RoadRunner. A system as powerful, innovative, and novel as this comes along perhaps once a decade, and after 10 years of operation, Sunway TaihuLight has earned its place in computing history. All the engineers and administrators who have built, operated, and maintained this system deserve respect for their contributions. What is especially remarkable is that, due to the political climate, this system had an additional requirement for novel design. However, I have been informed by a trustworthy source to "watch this space"; there are plans for a system ten times as powerful in the very near future.

Guizhou National Data Centre and Radio Telescope

lev_lafayette — Mon, 06 Apr 2026 11:14:33 +0000

A few weeks ago, I had the opportunity to visit Guizhou, China's National Big Data Comprehensive Pilot Zone. At first blush, the choice of Guizhou seems to be an unusual one. With a population of 38 million, it ranks 17th in the list of administrative divisions and 18th in population density, and it is even further behind in terms of GDP (22nd). From the principle that it is most efficient to conduct compute near where data is located, one would expect that such a data centre would be located in provinces with a greater population and economic activity, such as Guangdong, Jiangsu, or Shandong, or maybe even in accord to scientific output, which would also include Beijing and Shanghai. Instead, Guizhou, with its rugged karst formations and dense forests and lower level of economic development (fourth lowest in GDP per capita in 2020), has been part of a "Big Data Guizhou" strategic plan launched in 2014 which includes a threefold approach; "Big Data", "Big Poverty Reduction", and "Big Ecology" by then governer, Chen Min'er.

With low energy costs and a consistently cool climate, Guizhou has established the Guizhou Cloud, sponsored by the Guizhou Big Data Development Administration and supervised by the Board of Supervisors of Guizhou State-Owned Enterprises. This in turn has attracted major national corporations to its numerous data centers, notably Apple's iCloud China and Huawei's Cloud Service Base, along with Tencent, Alibaba Cloud, the AI firm SenseTime, as well as hosting the annual China International Big Data Industry Expo since 2015. These corporate decisions are notable enough in their own right, but what really makes Guizhou such an attractive place for a national data hub is the presence of the Five-hundred-meter Aperture Spherical Telescope (FAST) which, as is evident from the name, has a 500m diameter dish (I call it a "wok") making it the world's largest single-dish telescope.

Radio telescopes are essentially antennas and receivers for radio waves, with frequencies ranging from around 20 kHz to around 300 GHz, just as an optical telescope collects data from the visible portion of the spectrum. Primary local sources for radio waves include the Sun, Jupiter (due to its magnetosphere), and Jupiter's moon, Ganymede. The Galactic Centre of the Milky Way is an especially powerful source, as are supernova remnants, such as Cassiopeia, and neutron stars, such as pulsars and Rotating Radio Transients (RRATs). Primordial black holes and extraterrestrial intelligence communications are two other speculative, currently unobserved sources. The main point is that to derive information about these radio sources, one needs to collect radio wave data, and the more data you collect, the greater the chance you will find something interesting. Thus, to collect more data, one wants a larger receiver.

FAST is a very big receiver and it collects a lot of data, roughly 100TB per day. Construction began in 2011, testing began in 2016, and it was fully operational in 2020. Even before becoming operational it discovered two pulsars and by 2021 it has discovered an incredible 500. Hardware innovations are continuing; late last year, China Environment for Network Innovation (CENI), announced that they had conducted a (somewhat contrived) data transfer test of 72TB between FAST to Huazhong University of Science and Technology in Central China's Hubei province. The data transfer test, which would normally take 699 days was completed in a mere 1.6 hours. Further, China has announced that it will build an additional 24 radio dishes of 40m diameter around FAST, creating an array that will mimic a massive 10km diameter dish, and boost telescope resolution by 30 times.

Mention must also be made of the rest of the Tianyan Scenic Area, which hosts the FAST system. Apart from the stunning natural beauty of the region, and the FAST Observation Platform (complete with bungy jump during holiday weeks), the site also hosts an impressive Astronomy Experience Hall, the Astronomical Space-Time Tower (at 99.999 metres), the Nan Rendong Memorial Hall (the astronomer who was the main drive for setting up FAST), the Dome Flight Cinema, and a Aerospace Science and Technology Museum. Like many massive engineering projects in China, they have also turned the site into an informative destination for national and international visitors.

Critical Issues for the Global Climate

lev_lafayette — Tue, 03 Mar 2026 06:40:13 +0000

This presentation covers the science of Earth's "energy budget" of heat inputs and losses, and describes the overall climate system. It continues with a description of the Industrial Age, the introduction of direct temperature measurement, and the resulting temperature rise from burning fossil fuels. This is followed by a description of how Greenhouse Gases operate on a molecular level, the increase since the pre-industrial period, and the carbon cycle. After this, human activity and projections are considered, followed by changes to species' habitats and the possibility of an Anthropocene Extinction Event, then energy trajectories and future global policy directions. Concluding remarks identify climate change as a critical issue and one subject to "race conditions", and note that the policy route, whilst necessary, is currently falling short of requirements.

This was a presentation to Future Day 2026, March 2-4. A transcript is provided along with the accompanying slide deck.

Transcript:
http://levlafayette.com/files/2026FutureDayGlobalClimateTranscript.pdf

Slides:
http://levlafayette.com/files/2026FutureDay_GlobalClimate.pdf

Attachment	Size
2026FutureDayGlobalClimateTranscript.pdf	87.13 KB
2026FutureDay_GlobalClimate.pdf	1.37 MB

The Environmental Management of Artificial Intelligence

lev_lafayette — Sun, 07 Dec 2025 02:21:32 +0000

With at least three disciplines of interest - energy and climatology, public economics, and high-performance computing - there is the issue of whether current trends in artificial intelligence are environmentally sustainable. The following is a basic sketch of electricity usage, needs, and costings.

The promise of artificial intelligence is as old as computing itself, and, in some ways, it is difficult to distinguish from computing in general. As the old joke goes, it is no contest for real stupidity, and an issue that became all too evident to Charles Babbage when he developed the idea of a programmable computer:

On two occasions I have been asked, - "Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?" ... I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.
-- Passages from the Life of a Philosopher (1864), ch. 5 "Difference Engine No. 1"

Of course, we now have computational devices that can attempt to solve problems with incorrect inputs and perhaps provide a correct answer. That, if anything, is what distinguishes classical computation from contemporary artificial intelligence. Time does not permit a thorough exploration of the rise and fall of several attempts to implement AI; however, the most recent version of the last decade, which involves the application of transformer deep learning and the use of Graphics Processing Units, continues to attract investment and interest.

Transformer architectures for artificial neural networks are a fascinating topic in their own right; attention (pun intended) is directed to the use of GPUs as the main issue. Whilst the physical architecture of GPUs makes them particularly suitable for graphics processing, it was also realised that they could be used for a variety of vector processing, providing massive data parallelism, i.e., "general purpose (computing on) graphics processing units", GPGPUs. However, physics gets in the way of the pure mathematical potential of GPUs; they generate a significant amount of heat and require substantial electricity, and that's where the environmental question arises.

The current global electricity consumption by data centres is approximately 1.5%, according to the International Energy Agency. However, that is expected to reach 3.0% by 2030, primarily due to the growth of AI, which would include not just the GPUs themselves, but also the proportional contributions by CPU hosts, cooling, transport, installation, and so forth. A doubling of energy consumption over a period of a few years (from c400TWh in 2024 to c900TWh in 2030) is very significant and, if the estimates prove to be even roughly correct, then further energy utilisation needs to be considered as an ongoing trajectory for at least another two decades until it becomes ubiquitous.

There are essentially two ways of managing energy used in production with an environmental perspective, given a particular policy. One approach is high-energy and high-production, concentrating on renewables or non-GHG energy sources. The other is a reduced-energy, high-efficiency approach that concentrates on better outcomes, "doing more with less". More important than either of these, in my opinion, is the incorporation of externalised costs into the internal price of an energy source. One graphic example of this is the deaths per Terawatt-hour by energy source. Solar, for example, is more than three orders of magnitude safer than coal.

To satisfy existing and expected demand, the AI industry is, in part, turning to nuclear for its energy needs. Data centres tend to be located within population centres, partially due to latency reasons, whereas renewables like wind and solar require a significant amount of land area. Additionally, where existing nuclear power plants and infrastructure are already in place, it is relatively inexpensive, even compared to battery technologies. Nuclear provides sustained power generation not just throughout the day, but across months and seasons. With approximately 5% of generation lost in transmission, nearby power sources are more efficient.

The main weakness of nuclear power is the time and cost associated with the construction of new plants, and in this regard, the big data centre and technology groups are taking a gamble. They assume that there will be sufficient demand for AI and that they can generate enough income over the next decade to cover the costs. Whilst they are very likely to be correct in this assessment, and certainly the choice of nuclear is preferable to the fossil fuel sources that are currently driving most data centres (e.g., methane gas in the United States, coal in China).

As for demand-side considerations, these can include the energy efficiency of the data centres themselves, the way models are designed, and the way AI is utilised. Cooling is an especially interesting case; as mentioned, GPUs run quite hot, and to avoid catastrophic failure, they require effective cooling. This is usually done with evaporative cooling, which means significant water loss, or by chillers, which doesn't mean much loss, but a huge amount of water for cooling. A third option is dielectric liquids, such as mineral oil, which results in a data centre that is quiet and at room temperature, while servers operate at an optimal temperature. The main disadvantage is the messy and time-consuming procedures for upgrading system units.

The model design also presents some opportunities for improvement. The typical approach is to train neural networks with large quantities of data; however, the more indiscriminate the data collection is, the greater the possibility of conventional error. As some critics suggest, an LLM is essentially a language interface that sits in front of a search engine. A smaller but more accurate collection of data can be more accurate, as well as being less resource-intensive to train in the first place. A number of smaller models can operate with connective software for matters outside of the initial module's scope instead of a monolithic approach.

Finally, there is the matter of what AIs are being used for. Certainly, there are some powerful and important success stories such as the key designers behind AlphaFold winning the 2024 Nobel Prize in Chemsitry for protein structure prediction and, as many contemporary workers (especially in computer science) are all too aware, the ability of AIs to produce code is quite good, assuming th developer knows how to structure the questions with care and engages in thorough testing. Additionally, the increasing application of these technologies in robotics and autonomous vehicles is disconcerting, as illustrated by the predictive and plausible video "Slaughterbots".

On the consumer level, an AI can perform tasks which a human is less efficient at. So rather than simply asking "how many tonnes of a GHG does AI cause", a net emissions question should be asked, appending "... compared to human activity", that is, productivity substitution. However, with effectiveness comes the lure of convenience, as it attempts to extend the use of AI to everything, even when human energy usage would be less than that of an AI-mediated task. Ultimately, it is the combination of human failings, a combination of laziness (always choosing convenience), wilful ignorance (not knowing and caring about energy efficiency), distractibility (extending AI for trivialities rather than tasks of importance), and powerlust (commercial or political), that present a continuing challenge to the prospect of implementing an environmentally-sustainable development of artificial intelligence.

Against Default Modules

lev_lafayette — Tue, 21 Oct 2025 05:57:21 +0000

Environment Modules allow users to dynamically modify the $PATHs in their shell environment. In particular, it provides the ability to switch between different software applications and versions. It is a very handy tool for developers who want to test against multiple versions or compilation options of an application and for users on a multi-user system (e.g., high performance computing) where both consistency and the opportunity to introduce newer features needs to exist for different research groups. The two major environment modules systems are the older, Tcl-based system, and the newer Lua-based system (Lmod). Both allow default versions of a software application to be set as in the $PATH when invoked. This can cause problems and should be disabled.

With a standard module system, one can see the modules for an application. For example, using a LMod system:

$ module av GCC/ ------------------------------------ /apps/easybuild-2022/easybuild/modules/all/Core ---------------------------- GCC/11.3.0 (L) GCC/12.3.0 GCC/13.3.0 (D)

This illustrates that from the GNU Compiler suite (GCC), the application GCC/11.3.0 is loaded (by default in this case as part of the login) and that GCC/11.3.0 is the default as part of a wider toolchain. That is, if the command

module load GCC

is invoked, the existing environment $PATH will change so that it points to the binary, libraries, etc for GCC/12.3.0 instead of GCC/11.3.0. In the older, Tcl-based modules system, it was possible to have both versions loaded simultaneously; the most recent version would be the initial search path, but if a particular library, for example, couldn't be found, it would search whatever was in the $PATH. That could be an interesting issue for the replication of results.

Having a default obviously saves a few keystrokes, but problems can result. The major issue is that when new software is introduced by the local friendly HPC engineers, then the default changes. For example, the latest version of GCC, at the time of writing, is 15.1 (April 2025). If that was installed on a system, one would witness:

$ module av GCC/ ------------------------------------ /apps/easybuild-2022/easybuild/modules/all/Core ---------------------------- GCC/11.3.0 (L) GCC/12.3.0 GCC/13.3.0 GCC/15.1.0 (D)

Instead of loading GCC/12.3.0 from the

module load GCC

command, LMod will add GCC/15.1.0 to the $PATH. Is this a problem? Most certainly because when software (version, compiler) changes, the results can change. For example, with the release of GCC/13 it included the statement: -Ofast, -ffast-math and -funsafe-math-optimizations will no longer add startup code to alter the floating-point environment when producing a shared object with -shared. In a nutshell, this means that one can get different numerical results by compiling the same code between GCC/12 and GCC/13 due to different ways the compiler would handle rounding, precision, and exceptions when using floating-point calculations. This could be very interesting if one cares about that supposed hallmark of science, "reproducibility".

To avoid this, at a user-level, one should always use the fully-qualified version of a software application. For example, instead of loading a module with

module load GCC

, whether in an interactive session or job script, one should use the specific version, e.g.,

module load GCC/13.3.0

. Of course, sometimes users don't follow the friendly and sensible advice from their system engineers, and that's where a blunter tool needs to be invoked.

Site-wide LMod configuration (at let's face it, LMod is the overwhelmingly dominant version of environment modules these days) occurs in the lmod_config.lua file,

$LMOD_DIR/../init/lmod_config.lua

. Adding the following lines is sufficient:

-- Do not automatically pick a version when multiple options are available always_load_default = false -- Do not automatically swap modules from the same family auto_swap = false -- Disallow default-version resolution and force the user to specify which version site_defaults = { load_default = false, }

Another option is to export optins in

/etc/profile.d/lmod.sh

or equivalent. This avoids making changes to the lmod_config file.

export LMOD_DISABLE_NAME_AUTOSWAP=yes export LMOD_DISABLE_VERSION_AUTODEFAULT=yes

If a system is using EasyBuild for software installs (and many of us are), EasyBuild will create

.version

files which include the default; this can be removed by changing the

easybuild.cfg

file to have

module_install_defaults = False

One issue that arises from the above, from a support perspective, is that existing user expectations and job submission scripts may come into conflict with a new policy. For example, if a job submission script included

module load GCC

, the job would fail as LMod would be asking for user input on which specific version of GCC they wish to load.

This is a valid concern in the short term, but in the longer term, the benefits of providing consistent environment variables, reproducibility, and educating users far outweigh the costs. Further, the longer one persists with the policy of allowing application defaults the greater the damage.

Command-Line CD Extraction and Formatting

lev_lafayette — Sun, 12 Oct 2025 00:47:48 +0000

LAME (recursive acronym Lame ain't an MP3 encoder) has been an exceptional application for creating MP3 audio files from other formats (e.g., WAV, FLAC, MPEG-1, MPEG-2) or even for re-encoding existing MP3 files. Creating MP3 files is popular due to their extensive use in portable media players, as they combine an impressive reduction in file size with minimal loss from uncompressed audio. LAME was first released 27 years ago, has been in regular development since, and is now bundled with the very popular audio editor, Audacity, and other excellent audio conversion applications like FFmpeg. The following are several examples of how to use LAME on the command line, convert files, and automate this process, along with the use of cdparanoia and ffmpeg. This might be handy if you want to (for example) back up a collection of CDs.

To extract files from existing media, use cdparanoia, a delightfully old and stable tool where the last release was September 2008 and the front-end for libparanoia, which is also used in the cdrtools suite, which can be used for the creation of audio and data CDs. To use cdparanoia to extract files from disk, first use the command to get the information about what drives you have, i.e., $ cdparanoia -vsQ, a verbose search and query. Then, to extract the files, simply use $ cdparanoia -B, batch mode, saving each track to a separate file. The default format is WAV; if one wants RAW, use $ cdparanoia -rB.

Using LAME to convert files, the simplest action is to convert a file with no options. e.g., $ lame input.wav output.mp3. This can be extended with the numerous options offered by LAME. Perhaps the most common is the -V% (variable bit rate) (e.g., $ lame -V0 input.wav output.mp3. Values are from 0 to 9, where 0 is the highest quality (but largest file size), which is good for very high-fidelity recording, whilst 9.999 is the lowest one, which is fine for low fidelity needs (e.g., a monophonic voice recording); the default is 4. Run a test on a WAV file to see if you can detect the difference in audio quality and check the file size differences. Note that despite being a good encoder, LAME is not recommended for archives; use a lossless audio codec (e.g., FLAC).

If one has multiple files to convert, a short loop is an effective strategy (e.g., $ for item in *.wav; do lame "$item" "${item%.wav}.mp3"; done). This is certainly preferable to opening files with a GUI application and running the conversion one file at a time. If one has a number of large files, then use of GNU Parallel is an excellent choice (e.g., $ parallel lame -V0 {} {.}.mp3 ::: *.wav). On a sample CD, the loop conversion took 57.201 seconds; with parallel, 13.264 seconds; it is fairly clear how this makes a difference if you have a large number of CDs. Note that with a smaller number and size of files, the use of GNU Parallel will actually take a longer time than a loop due to the overhead of splitting up the tasks. Certainly, if you are converting the WAV files from a CD, use GNU Parallel. Note that GNU Parallel will use all the cores available on a system; the number of concurrent jobs can be limited with the -j option (e.g., parallel -j2 lame -V0 {} {.}.mp3 ::: *.wav).

Finally, there is a great little tool called mp3wrap, which can wrap multiple MP3 files into a single MP3, but keeps filenames and metadata information, which can be later split back to original files (using the mp3splt command). The output file will be named OUTPUTFILE_MP3WRAP.mp3; keep the MP3WRAP string so that the split program detects that the file is wrapped with this utility. To create a single file, simply use $mp3wrap album.mp3 track*.mp3.

Research Software Engineering New Zealand Conference 2025

lev_lafayette — Tue, 30 Sep 2025 06:49:44 +0000

The 2025 New Zealand Research Software Engineering Conference was held on September 23-24th. It has run for almost a decade, starting in 2016 as the CRI Coding Conference before becoming the Science Coding Conference, a title it kept until 2020, when it became the Research Software Engineering Conference. Throughout this time, it has focused on science and engineering, high performance computing, and cloud computing and is aimed at various coders, sysadmins, software engineers, data analysts, and IT managers from public research institutions and has the endorsement of the RSE Association of Australia and New Zealand.

As the programme indicates, this year's selection of presentations and BoFs has a strong emphasis on machine learning and developments in artificial intelligence. This was evident right from the very start with Nick Jones' keynote address on the first day, and with numerous presentations throughout the conference. A concern must be raised when this pivot to AI/ML involves a recursive comparative testing being conducted with other AI/ML systems. Ultimately, the validity of computational modelling must come not only from the quality of the inputs but the real-world predictive (and hindcasting) value of the outputs.

Thankfully, there was little in the way of "AI/ML marketing hype" at this conference, which really was firmly dedicated to actual computational practice, development of skills and knowledge, and research outputs. Further, there was only a moderate amount of theory, and that's primarily for foundations, as it should be. Being in Aotearoa New Zealand, it is perhaps unsurprising that there were several presentations on Earth sciences, but also of note was the emphasis on climatology, oceanography, and new developments in forensics.

Interestingly, Australian research software development and education were present in a number of presentations, including speakers from WEHI and CSIRO. From Research Computing Services at the University of Melbourne received a surprising highlight with my own presentation, "Programming Principles in a High Performance Computing Environment", the first presentation of the conference and Daniel Tosello's "VSCode on the node" being the third. It bodes well for current and future Trans-Tasman collaboration.

Finally, the Research Software Engineering New Zealand Conference included a small number of explicit community-building presentations. In conferences such as these, presentations are often of a special interes,t but with a much wider and latent feature being the awareness of the directions that other institutions and individuals are taking and the strength of professional connections, a critical requirement not only for an individual's development, but also for raising the collective knowledge of the institutions that they belong to.

Parallel wget with xargs or parallel

lev_lafayette — Wed, 14 May 2025 02:12:12 +0000

The following is an illustration of how to use xargs to conduct parallel operations on single-threaded applications, specifically wget.

GNU wget is a great tool for downloading content from websites. The wget command is a non-interactive network downloader; by "non-interactive" what is meant is that it can be run in the background. Some very hand options include -c (continue, for partially downloaded files), -m (mirror, for an entire website), -r --no-parent (recursive, no parent, to download part of a website and its subdirectories). The cURL application has a wider range of protocols and includes upload options, but is non-recursive.

Recently, I had the need to download a small number of PDF files. The wildcard-based approach would be:

$ wget -r -nd --no-parent -A 'rpgreview_*.pdf' http://rpgreview.net/files/

The -r and --no-parent options have already been explained. The -nd option allows one to save all files to the current directory, without hierarchy of directories. The -A option ('accept', or -R 'reject) allows one to specify comma-separated lists of file name suffixes or patterns to accept or reject. Note that if any of the wildcard characters, *, ?, or ranges [] to be in an acclist or rejlist.

Running the above has the following time:

real 2m19.353s
user 0m0.836s
sys 0m2.998s

An alternatiive, looping through each file one at a time, would have been something like:

for issue in {1..53} do wget "https://rpgreview.net/files/rpgreview_$issue.pdf" done

(Just for the record, wget can get a bit gnarly when dealing with http requests because for some webservers there is no requirement for path delimiters to match directory delimiters. For the purposes of this discussion it is assumed that we're dealing with a rational being where the two are equivalent.)

Using a combination of the printf command and the xargs command a list of the URLs can be constructed which is then passed to xargs which can split the list to run in parallel.

By itself, xargs simply reads items from standard input, delimited by blanks or newlines, and executes commands from that list of items as arguments. This is somewhat different to the pipe command which, by itself, sends the output of one command as the input stream to another. In contrast, xargs takes data from standard input and executes a command which, by default, the data is appended to the end of the command as an argument. However, the data can be inserted anywhere however, using a placeholder for the input; the typical placeholder is {}.

The value -P 8 is entirely arbitrary here and should be modified according to available resources. Adding -nc prevents xargs attempting to download the same file more than once (wget will not overwrite an existing file, but rather append a new file with .1, etc. The -n option ensures that only one argument is run per process.

printf "https://rpgreview.net/files/rpgreview_%d.pdf\n" {1..53} | xargs -n 1 -P 8 wget -q -nc

The time of the above comes to:

real 1m23.534s
user 0m1.567s
sys 0m2.732s

Yet another choice is to use GNU parallel and seq.

seq 53 | parallel -j8 wget "https://rpgreview.net/files/rpgreview_{}.pdf"

real 1m57.647s user 0m1.830s sys 0m4.214s

A final option, most common in high-performance computing systems with job schedulers, is to make use of a job array. This is effective assuming resource availability. This is a very effective option if each task in the array takes more than a couple of minutes each (given that there is an overhead involved in constructing the job, submitting it to the queue, etc). In Slurm, a script the directives and code would look like the following:

#SBATCH --job-name="file-array" #SBATCH --ntasks=1 #SBATCH --time=0-00:15:00 #SBATCH --array=1-53

wget "https://rpgreview.net/files/rpgreview_${SLURM_ARRAY_TASK_ID}.pdf"

Quantum Computing and Quantum Computers

lev_lafayette — Mon, 14 Apr 2025 13:13:36 +0000

It is appropriate, on World Quantum Day, to talk about quantum computing and quantum computers, as the two are often confused. Quantum computing is any method to generate quantum effects whereby qubit states can exist in superposition (o,1, both) rather than binary states (0,1). Binary states are represented in classical computing in low-level software as logical 0s and 1s but in hardware as high and low-voltage states.

The typical system to do quantum computing, or at least simulate it, is usually High Performance Computing (HPC). That works, it's a proven technology that has a rate of return of $44 per $1 invested - and higher when COVID research is considered. The development of HPC clusters with message passing is one of the most successful technological developments in computing in the last thirty years.

In contrast, a quantum computer directly uses a quantum mechanical system and requires appropriate specialised hardware. For example, GENCI in France uses a photonic computer, LRZ in Germany uses superconducting qubits, PSNC in Poland uses trapped ions, etc. David P. DiVincenzo offers the most significant physical challenges that face quantum computers, regardless of what technology is used; these include scaling qubits, initialisation of values, developing a universal gate for the construction of a quantum operation, developing gates that are faster than decoherence from a quantum state due to environmental interactions, and reading qubits (especially considering that can alter the quantum state).

As a result, classical computers outperform quantum computers in all real-world applications. Not only that, there is a serious issue of whether quantum computers will ever be able to outperform classical computers. Mikhail Dyakonov points out that the rudimentary qubits used in quantum computing systems is insufficient for useful calculations.

"Experts estimate that the number of qubits needed for a useful quantum computer, one that could compete with your laptop in solving certain kinds of interesting problems, is between 1,000 and 100,000. So the number of continuous parameters describing the state of such a useful quantum computer at any given moment must be at least 21,000, which is to say about 10^300. That's a very big number indeed. How big? It is much, much greater than the number of subatomic particles in the observable universe."

The promise of quantum computers is, of course, very significant in theory. In theory, it can perform some calculations incredibly fast, and the larger the task, the more impressive the result, to the extent that common secure encryption systems could be broken, as well as the more prosaic use in quantum simulations. In reality, the physical implementation has been more than challenging, to put it mildly. Classical computers, in principle, can solve the same problems as a quantum computer can, in principle solve as well. For a classical computer the problem is the sheer quantity of time that is required. For quantum computers, the problem is the implementation in reality. For the time being, and for the foreseeable future, it seems that quantum computing will continue to be done on classical computers.

Attachment	Size
IBM System 1 Quantum Computer	189.81 KB

Scheduler Parallelisation

lev_lafayette — Mon, 17 Mar 2025 11:51:34 +0000

The standard computing model uses unithreaded instructions and data with automation through looping and conditional branching. Automation is encouraged as it results in the computer doing the work that it is designed for. However, this can be inefficient when using a multicore system. An alternative in HPC systems is to make use of job arrays, which use a job to allocate resources to sub-jobs which can be individually controlled, whether directed toward instruction sets or datasets. Further, job arrays can be combined with job dependencies, allowing for conditional chains of job submission and runs. Finally, job arrays can be simulated through the use of heredocs with looped submission. This may even allow more familiar control with shell scripting.

This slidedeck is derived from a presentation to the Univerity of Melbourne's "Spartan Champions" group on March 7, 2025.

https://levlafayette.com/files/2025Champions_Arrays.pdf