Marcin Juszkiewicz - ubuntu

“Ten” years at Linaro

2023-04-17T12:36:00+02:00

Some time ago was a day when I reached “ten” years at Linaro. Why “”? Because it was 3 + 7 rather than 10 years straight. First three years as Canonical contractor now seven years at Red Hat employee assigned as Member Engineer.

My first three years at Linaro

NewCo or NewCore? Or Ubuntu on ARM?

In 2010 I signed contract with Canonical as “Foundation OS Engineer”. Once there I signed another paper which moved me to NewCo project (also called NewCore but NewCo name is on paper I signed).

On 30th April 2010 I got “Welcome to Linaro” e-mail.

Then UDS-M happened where we were hiding under “Ubuntu on Arm” name (despite the fact that Ubuntu had such team).

Ah, it is Linaro now :D

On 3rd June 2010 Linaro was officially announced. No more hiding, we went public with name.

My team

I became a part of Developer Platform team and worked mostly on toolchain packages for Ubuntu and later Debian. There were funny moments at sprints/summits/connects when I joked that I have two managers to listen to (one from Toolchain Working Group and one from Developer Platform).

There were Debian/Ubuntu developers there and people from other environments.

At some moment it was renamed to “Base and Baselines”. Or “Bed and Breakfast” as most of time “BB” was used instead of full name.

AArch64 bring up

In 2012 I dusted off my OpenEmbedded knowledge and started working on getting AArch64 architecture bring up. Lot of not-yet-public patches was in use. The fun of seeing “Hello world!” message in emulator printed by OS image I built from scratch was something I hope to never forget.

Each time I am choosing mug for my coffee I see Pac-Man one I bought during Linaro/Arm AArch64 sprint we had in October 2012.

My AArch64 mug

The End

There was lot of noise in 2011 about deal between Canonical and Linaro. Several engineers at Linaro were from Canonical and there was some messy situation related with money.

It ended with retiring of people every quarter. Some moved back to Canonical, some changed job and got hired directly by Linaro. There were also people who moved to Linaro member companies and stayed with their Linaro position. Some people left both companies and went to other jobs.

I was supposed to leave Linaro in 2012 but it was postponed by half a year. So I left after about 37 months.

Second round

Time passed, I was working at Red Hat on getting AArch64 first class citizen in RHEL and Fedora Linux distributions. And one day my manager asked do I want to work at Linaro again.

I did some research, discussed with friends at Linaro and on 8th April 2016 I was back.

My team II

This time I became part of LEG: Linaro Enterprise Group. Servers, data centres etc. There were several teams to choose and I ended in SDI (Software Defined Infrastructure) one. We were behind LDC (Linaro Developer Cloud) project.

At some moment LEG became LDCG (Linaro Datacenter and Cloud Group). Some years later LDCG lost “and Cloud” part as AWS and other cloud providers started offering AArch64 systems so we did not had to deal with it any more.

OpenStack all over

First version of LDC was Debian based with OpenStack ‘liberty’. Then were weird months when we had to reinvent deployment several times. It was mess most of time.

So we abandoned own solutions and went with OpenStack Kolla. I quickly became one of core developers there. LDC moved to be container based.

In 2022 we ended working on OpenStack. LDC is now used only for internal projects.

Building stuff

With my “give me software and I will build it” mantra I ended also as kind of CI jobs developer for LEG teams. Apache Bigtop, Apache Arrow, TensorFlow, EDK2 and several other projects. Some used containers, some were running shell scripts, some Ansible.

SBSA Reference Platform

I was involved in some work around getting QEMU to emulate SBSA Reference Platform (“sbsa-ref” machine). Created some CI jobs to run test suites, build firmware images etc. After running test suites I created a bunch of issues in Jira so we can track how things go.

During recent months I became more involved. I am testing patches, running them through both SBSA and BSA Arm Compliance Suites and reporting results.

I have own set of scripts to handle logs to make it easier to track how things are now.

Summary

At last Linaro Connect events there was always a moment when they announced people who worked at Linaro for 5 (or later) 10 years. I have to admit — I felt envy several times.

And when I was 5 years straight at Linaro we had COVID-19 pandemic so there was no event.

EBBR on EspressoBin

2021-02-15T21:53:00+01:00

SBBR or GTFO

Me.

Yeah, right. But world is not so nice and there are many cheap SBC on market which are not SBBR compliant and probably never will. And with small amount of work they can do EBBR (Embedded Base Boot Requirements).

NOTE: I have similar post about EBBR on RockPro64 board.

WTH is EBBR?

It is specification for devices which are not servers and do not pretend to be such. U-Boot is all they have and with properly configured one they have some subset of EFI Boot/Runtime Services to load distribution bootloader (grub-efi usually) like it is done on servers.

ACPI is not required but may be present. DeviceTree is perfectly fine. You may provide both or one of them.

Firmware can be stored wherever you wish. Even MBR partitioning is available if really needed.

Few words about board itself

EspressoBin has 4MB of SPI flash on board. Less than on RockPro64 but still enough for storing firmware (U-Boot takes less than 1MB).

This SBC is nothing new — first version was released in 2016. There were several revisions with different memory type, amount of ram chips, type of them (ddr3 or ddr4), CPU speed and some more changes.

I got EspressoBin revision 5 with 1GB ram of ddr3 in 2 chips. And 1GHz processor.

It may sound silly that I repeat that information but it matters when you start building firmware for that board.

So let us build fresh firmware

This is Marvell so abandon all hope for sanity.

Thanks to Arm Trusted Firmware authors there is good documentation on how to build firmware for EspressoBin which guides step by step and explains all arguments you need. For me it was several git clone calls and then two make calls:

make -C u-boot CROSS_COMPILE=aarch64-linux-gnu- \
mvebu_espressobin-88f3720_defconfig u-boot.bin -j12

make -C trusted-firmware-a CROSS_COMPILE=aarch64-linux-gnu- \
CROSS_CM3=arm-none-linux-gnueabihf- PLAT=a3700 \
CLOCKSPRESET=CPU_1000_DDR_800 DDR_TOPOLOGY=2 \
MV_DDR_PATH=$PWD/marvell/mv-ddr-marvell/ \
WTP=$PWD/marvell/A3700-utils-marvell/ \
CRYPTOPP_PATH=$PWD/marvell/cryptopp/ \
BL33=$PWD/u-boot/u-boot.bin \
mrvl_flash -j12

And I had to install cross toolchain for 32-bit arm because the one I had was for building kernels/bootloaders only.

Is your U-Boot friendly or not?

First you need to check which version of U-Boot and hardware you have. Then check does it recognize SPI flash or not:

Marvell>> sf probe
SF: unrecognized JEDEC id bytes: 9d, 70, 16
Failed to initialize SPI flash at 0:0 (error -2)

I had bad luck as my board used SPI chip not recognized by any official U-Boot build…

Armbian to the rescue

I asked in few places did anyone had some experiences with this board. One of them was #debian-arm IRC channel where I got hint from Xogium that Armbian may have U-Boot builds.

And they have whole page about EspressoBin. With information how to choose firmware files etc.

So I downloaded archive with proper files for UART recovery. The important thing to remember is that once you move jumpers and load all firmware files over serial they are not written in SPI flash so reset of board means you start over.

Quick check is SPI flash detected:

Marvell>> sf probe
SF: Detected is25wp032 with page size 256 Bytes, erase size 4 KiB, total 4 MiB

Yeah! Now can start USB and flash own firmware build:

Marvell>> bubt flash-image.bin spi usb
Burning U-Boot image "flash-image.bin" from "usb" to "spi"
Bus usb@58000: Register 2000104 NbrPorts 2
Starting the controller
USB XHCI 1.00
Bus usb@5e000: USB EHCI 1.00
scanning bus usb@58000 for devices... 1 USB Device(s) found
scanning bus usb@5e000 for devices... 2 USB Device(s) found
Image checksum...OK!
SF: Detected is25wp032 with page size 256 Bytes, erase size 4 KiB, total 4 MiB
Erasing 991232 bytes (242 blocks) at offset 0 ...Done!
Writing 990944 bytes from 0x6000000 to offset 0 ...Done!

Quick reset and board boots to fresh, mainline U-Boot:

TIM-1.0
WTMI-devel-18.12.1-1a13f2f
WTMI: system early-init
SVC REV: 5, CPU VDD voltage: 1.108V
NOTICE:  Booting Trusted Firmware
NOTICE:  BL1: v2.4(release):v2.4-345-g04c122310 (Marvell-devel-18.12.2)
NOTICE:  BL1: Built : 17:11:19, Feb 15 2021
NOTICE:  BL1: Booting BL2
NOTICE:  BL2: v2.4(release):v2.4-345-g04c122310 (Marvell-devel-18.12.2)
NOTICE:  BL2: Built : 17:11:20, Feb 15 2021
NOTICE:  BL1: Booting BL31
NOTICE:  BL31: v2.4(release):v2.4-345-g04c122310 (Marvell-devel-18.12.2)
NOTICE:  BL31: Built : 18:07:02, Feb 15 2021


U-Boot 2021.01 (Feb 15 2021 - 19:25:41 +0100)

DRAM:  1 GiB
Comphy-0: USB3_HOST0    5 Gbps    
Comphy-1: PEX0          2.5 Gbps  
Comphy-2: SATA0         5 Gbps    
SATA link 0 timeout.
AHCI 0001.0300 32 slots 1 ports 6 Gbps 0x1 impl SATA mode
flags: ncq led only pmp fbss pio slum part sxs 
PCIE-0: Link down
MMC:   sdhci@d0000: 0, sdhci@d8000: 1
Loading Environment from SPIFlash... SF: Detected is25wp032 with page size 256 Bytes, erase size 4 KiB, total 4 MiB
OK
Model: Globalscale Marvell ESPRESSOBin Board
Card did not respond to voltage select! : -110
Net:   eth0: neta@30000
Hit any key to stop autoboot:  0

Final steps

OK, so SBC has fresh, mainline firmware. Nice. But still some stuff needs to be done.

First note MAC addresses of Ethernet ports. Use printenv command to check stored environment and note few variables:

eth1addr=f0:ad:4b:aa:97:01
eth2addr=f0:ad:4b:aa:97:02
eth3addr=f0:ad:4b:aa:97:03
ethaddr=f0:ad:4e:72:10:ef

Of course you may also skip that step and rely on random ones or choose own ones (I had router with C0:FF:EE:C0:FF:EE in past).

Then reset environment to default values stored in U-Boot binary and set those MAC addresses by hand:

=> env default -a -f
=> setenv eth1addr f0:ad:4b:aa:97:01
=> setenv eth2addr f0:ad:4b:aa:97:02
=> setenv eth3addr f0:ad:4b:aa:97:03
=> setenv ethaddr f0:ad:4e:72:10:ef
=> saveenv
Saving Environment to SPIFlash... Erasing SPI flash...Writing to SPI flash...done
OK
=>

What EBBR brings?

Now your board is ready to boot Debian, Fedora and several other distribution install media with two commands:

=> set boot_targets usb0
=> boot

It will find EFI bootloader and start it. Just like on any other boring SBBR/EBBR system.

Distributions with old style ‘boot.scr’ script (like OpenWRT for example) will also work so no functionality loss.

Standards are boring

2021-01-20T17:33:00+01:00

We have made Arm servers boring.

Jon Masters

Standards are boring. Satisfied users may not want to migrate to other boards the market tries to sell them.

So Arm market is flooded with piles of small board computers (SBC). Often they are compliant to standards only when it comes to connectors.

But our hardware is not standard

It is not a matter of ‘let produce UEFI ready hardware’ but rather ‘let write EDK2 firmware for boards we already have’.

Look at Raspberry/Pi then. It is shitty hardware but got popular. And group of people wrote UEFI firmware for it. Probably without vendor support even.

Start with EBBR

Each new board should be EBBR compliant at start. Which is easy — do ‘whatever hardware’ and put properly configured U-Boot on it. Upstreaming support for your small device should not be hard as you often base on some already existing hardware.

Add 16MB of SPI flash to store firmware. Your users will be able to boot ISO without wondering where on boot media they need to write bootloaders.

Then work on EDK2 for board. Do SMBIOS (easy) and keep your existing Device Tree. You are still EBBR. Remember about upstreaming your work — some people will complain, some will improve your code.

Add ACPI, go SBBR

Next step is moving from Device Tree to ACPI. May take some time to understand why there are so many tables and what ASL is. But as several other systems show it can be done.

And this brings you to SBBR compliance. Or SystemReady ES if you like marketing.

SBSA for future design

Doing new SoC tends to be “let us take previous one and improve a bit”. So this time change it a bit and make your next SoC compliant with SBSA level 3. All needed components are probably already included in your Arm license.

Grab EDK2 support you did for previous board. Look at QEMU SBSA Reference Platform support, look at other SBSA compliant hardware. Copy, reuse their drivers, their code.

Was it worth?

At the end you will have SBSA compliant hardware running SBBR compliant firmware.

Congratulations, your board is SystemReady SR compliant. Your marketing team may write that you are on same list as Ampere with their Altra server.

Users buy your hardware and can install whatever BSD, Linux distribution they want. Some will experiment with Microsoft Windows. Others may work on porting Haiku or other exotic operating system.

But none of them will have to think “how to get this shit running”. And they will tell friends that your device is as boring as it should be when it comes to running OS on it == more sales.

So long, and thanks for all the fun

2020-12-10T12:33:00+01:00

During last days I tried to get my Applied Micro Mustang running again. And it looks like it is no more. Like that Norwegian Blue parrot.

Tried some things

By default Mustang outputs information on serial console. It does not here. Checked serial cables, serial to usb dongles. Nothing.

Tried to load firmware from SD card instead of on-board flash. Nope.

Time to put it to rest.

How it looked

When I got it in June 2014 it came in 1U server case. With several loud fans. Including one on cpu radiator. So I took the board out and put into PC Tower case. Also replaced 50mm processor fan with 80mm one:

Top view of Mustang

Side view

All that development…

I did several things on it:

tested USB support
tested PCI Express support
run X11 on hardware
debugged why X11 does not start out of the box
wrote new instruction how to unbrick it
run 64 and 32-bit virtual machines
used it as a desktop (day 1, day 2, last day)
tested OpenStack

Some of them were done for first time on AArch64.

Board gave me lot of fun. I built countless software packages on it. For CentOS, Debian, Fedora, RHEL. Tested installers of each of them.

Was running OpenStack on it since ‘liberty’ (especially after moving from 16GB to 32GB ram).

What next?

I am going to frame it. With few other devices which helped me during my career.

Replacement?

It would be nice to replace Mustang with some newer AArch64 hardware. From what is available on mass market SolidRun HoneyComb looks closest. But I will wait for something with Armv8.4 cores to be able to play with nested virtualization.

UPDATE: I got it working again.

8 years of my work on AArch64

2020-09-23T17:33:00+02:00

Back in 2012 AArch64 was something new, unknown yet. There was no toolchain support (so no gcc, binutils or glibc). And I got assigned to get some stuff running around it.

OpenEmbedded

As there was no hardware cross compilation was the only way. Which meant OpenEmbedded as we wanted to have wide selection of software available.

I learnt how to use modern OE (with OE Core and layers) by building images for ARMv7 and checking them on some boards I had floating around my desk.

Non-public toolchain work

Some time later first non-public patches for binutils and gcc arrived in my inbox. Then eglibc ones. So I started building and on 12th September 2012 I was able to build helloworld:

12:38 hrw@puchatek:aarch64-oe-linux$ ./aarch64-oe-linux-gcc ~/devel/sources/hello.c -o hello
12:38 hrw@puchatek:aarch64-oe-linux$ file hello
hello: ELF 64-bit LSB executable, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.39, not stripped
12:39 hrw@puchatek:aarch64-oe-linux$ objdump -f hello

hello:     file format elf64-littleaarch64
architecture: aarch64, flags 0x00000112: 
EXEC_P, HAS_SYMS, D_PAGED 
start address 0x00000000004003e0

Then images followed. Several people at Linaro (and outside) used those images to test misc things.

At that moment we ran ARMv8 Fast models (quite slow system emulator from Arm). There was a joke that Arm developers formed a queue for single core 10 GHz x86-64 cpus to get AArch64 running faster.

Toolchain became public

Then 1st October 2012 came. I entered Linaro office in Cambridge for AArch64 meeting and was greeted with “glibc patches went to public ML” information. So I rebased my OpenEmbedded repository, updated patches, removed any traces of non-public ones and published whole work.

Building on AArch64

My work above added support for AArch64 as a target architecture. But can it be used as a host? One day I decided to check and ran OpenEmbedded on AArch64.

After one small patch it worked fine.

X11 anyone?

As I had access to Arm Fast model I was able to play with graphics. So one day in January 2013 I did a build and and started Xorg. Through next years I had fun when people wrote that they got X11 running on their AArch64 devices ;D

Two years later I had Applied Micro Mustang at home (still have it). Once it had working PCI Express support I added graphics card and started X11 on hardware.

Then went debugging why Xorg requires configuration file and one day with help from Dave Airlie, Mark Salter and Matthew Garrett I got two solutions for the problem. Do not remember did any of them went upstream but some time later problem was solved.

Few years later I met Dave Airlie at Linux Plumbers. We introduced to each other and he said “ah, you are the ‘arm64 + radeon guy’” ;D

AArch64 Desktop week

One day in September 2015 I had an idea. PCIe worked, USB too. So I did AArch64 desktop week. Connected monitors, keyboard, mouse, speakers and used Mustang instead of my x86-64 desktop.

It was fun.

Distributions

First we had nothing. Then I added AArch64 target into OpenEmbedded.

Same month Arm released Foundation model so anyone was able to play with AArch64 system. No screen, just storage, serial and network but it was enough for some to even start building whole distributions like Debian, Fedora, OpenSUSE, Ubuntu.

At that moment several patches were shared by all distributions as it was faster way than waiting for upstreams. I saw multiple versions of some of them during my journey of fixing packages in some distributions.

Debian and Ubuntu

In February 2013 Debian/Ubuntu team presented their AArch64 port. It was their first architecture bootstrapped without using external toolchains. Work was done in Ubuntu due to different approach to development than Debian has. All work was merged back so some time later Debian also had AArch64 port.

Fedora

Fedora team started early — October 2012, right after toolchain became public. Used Fedora 17 packages and switched to Fedora 19 during work.

When I joined Red Hat in September 2013 one of my duties was fixing packages in Fedora to get them built on AArch64.

OpenSUSE

In January 2014 first versions of QEMU support arrived and people moved from using Foundation model. March/April OpenSUSE team did massive amount of builds to get their distribution built that way.

RHEL

Fedora bootstrap also meant RHEL 7 bootstrap. When I joined Red Hat there were images ready to use in models. My work was testing them and fixing packages. There were multiple times when AArch64 fix helped to build also on ppc64le and s390x architectures.

Hardware I played with

First Linux capable hardware was announced in June 2013. I got access to it at Red Hat. Building and debugging was much faster than using fast models ;D

Applied Micro Mustang

Soon Applied Micro Mustangs were everywhere. Distributions used them to build packages etc. Even without support for half of hardware (no PCI Express, no USB).

I got one in June 2014. Running UEFI firmware out of the box. At first months I had a feeling that firmware is developed at Red Hat as we had fresh versions often right after first patches for missing hardware functionality were written. In reality it was maintained by Applied Micro and we had access to sources so there were some internal changes in testing (that’s why I had firmware versions like ‘0.12-rh’).

All those graphics cards I collected to test how PCI Express works. Or testing USB before it was even merged into Linux mainline kernel. Using virtualization for development of armhf build fixes (8 cores, 12 gigabytes of ram and plenty of storage beat all armv7 hardware I had).

I stopped using Mustang around 2018. It is still under my desk.

For those who use: make sure you have 3.06.25 firmware.

96boards

In February 2015 Linaro announced 96boards initiative. The plan was to make small, unified SBC with different Arm chips. Both 32- and 64-bit ones.

First ones were ‘Consumer Edition’. Small, limited to basic connectivity. Now there are tens of them. 32-bit, 64-bit, fpga etc. Choose your poison ;D

Second ones were ‘Enterprise Edition’. Few attempts existed, most of them did not survived prototype phase. There was joke that full length PCI Express slot and two USB ports requirements are there because I wanted to have AArch64 desktop ;D

Too bad that nothing worth using came from EE spec.

Servers

As Linaro assignee I have access to several servers from Linaro members. Some are mass-market ones, some never made to market. We had over hundred X-Gene1 based systems (mostly as m400 cartridges in HPe Moonshot chassis’) and shutdown them in 2018 as they were getting more and more obsolete.

Main system I use for development is one of those ‘never went to mass-market’ ones. 46 cpu cores, 96 GB of ram make it nice machine for building container images, Debian packages or running virtual machines in OpenStack.

Desktop

For some time I was waiting for some desktop class hardware to have development box more up-to-date than Mustang. Months turned into years. I no longer wait as it looks like there will be no such thing.

Solidrun has made some attempts in this area. First with Macchiatobin and later with Honeycomb. I did not used any of them.

Cloud

When I (re)joined Linaro in 2016 I became part of team working on getting OpenStack working on AArch64 hardware. We used Liberty, Mitaka, Newton releases and then changed way we work and started contributing more. And more. Kolla, Nova, Dib and other projects. Added aarch64 nodes to OpenDev CI.

The effect of it was Linaro Developer Cloud used by hundreds of projects to speed-up their aarch64 porting, tens of projects hosting their CI systems etc.

Two years later Amazon started offering aarch64 nodes in AWS.

Summary

I spent half of my life with Arm on AArch64. Had great moments like building helloworld as one of first people outside of Arm Ltd. Got involved in far more projects then ever thought. Met new friends, visited several places in the world I would probably never go otherwise.

I also got grumpy and complained far too many times that AArch64 market is ‘cheap but limited sbc or fast but expensive servers and nearly nothing in between’. Wrote some posts about missing systems targeting software developers and lost hope that such will happen.

NOTE: It is 8 years of my work on AArch64. I work with Arm since 2004.

EBBR on RockPro64

2020-06-17T17:53:00+02:00

SBBR or GTFO

Me.

But Arm world no longer ends on “SBBR compliant or complete mess”. For over a year there is new specification called EBBR (Embedded Base Boot Requirements).

WTH is EBBR?

In short it is kind of SBBR for devices which can not comply. So you still need to have some subset of UEFI Boot/Runtime Services but it can be provided by whatever bootloader you use. So U-Boot is fine as long it’s EFI implementation is enabled.

ACPI is not required but may be present. DeviceTree is perfectly fine. You may provide both or one of them.

Firmware can be stored wherever you wish. Even MBR partitioning is available if really needed.

Make it nice way

RockPro64 has 16MB of SPI flash on board. This is far more than needed for storing firmware (I remember time when it was enough for palmtop Linux).

During last month I sent a bunch of patches to U-Boot to make this board as comfortable to use as possible. Including storing of all firmware parts into on board SPI flash.

Needed files

To have U-Boot in SPI flash there you need to fetch two files:

idbloader.img (SPL + TPL)
u-boot.itb (U-Boot itself)

Store them as files on USB pen drive.

Flashing RockPro64 board

NOTE: I assume that you already have a way to boot your board to U-Boot shell (most common is to use microSD card with U-Boot on it).

Reboot board to U-Boot shell. Plug USB pen drive into any of RockPro64 USB ports.

Next do this set of commands to update U-Boot:

Hit any key to stop autoboot:  0 
=> usb start

=> ls usb 0:1
   163807   idbloader.img
   867908   u-boot.itb

2 file(s), 0 dir(s)

=> sf probe
SF: Detected gd25q128 with page size 256 Bytes, erase size 4 KiB, total 16 MiB

=> load usb 0:1 ${fdt_addr_r} idbloader.img
163807 bytes read in 16 ms (9.8 MiB/s)

=> sf update ${fdt_addr_r} 0 ${filesize}
device 0 offset 0x0, size 0x27fdf
163807 bytes written, 0 bytes skipped in 2.93s, speed 80066 B/s

=> load usb 0:1 ${fdt_addr_r} u-boot.itb
867908 bytes read in 53 ms (15.6 MiB/s)

=> sf update ${fdt_addr_r} 60000 ${filesize}
device 0 offset 0x60000, size 0xd3e44
863812 bytes written, 4096 bytes skipped in 11.476s, speed 77429 B/s

And reboot board.

After this your RockPro64 will have firmware stored in on board SPI flash. No need for wondering which offsets to use to store them on SD card etc.

Booting installation media

The nicest part of it is that no longer you need to mess with installation media. Fetch Debian/Fedora installer ISO, write it to USB pen drive, plug into port and reboot board.

Should work with any generic AArch64 installation media. Of course kernel on media needs to support RockPro64 board. I played with Debian ‘testing’ and Fedora 32 and rawhide and they booted fine.

Testing U-Boot on microSD

By default RockPro64 loads U-Boot from SPI flash. If you need/want to boot from microSD then follow instruction from official wiki:

If you mess-up your SPI and are unable to boot, jumpering pins 23 (CLK) and 25 pin (GND) on the PI-2-bus header will disable the SPI as a boot device.

You have to remove the jumper 2 seconds after having started your RP64 (before the white LED turns ON) otherwise the SPI will be missing and you won’t be able to flash it.

My setup has ‘disable SPI’ button next to ‘disconnect Tx serial line’ switch — both on small breadboard next to the board.

RockPro64 setup on my desk

OpenDev CI speed-up for AArch64

2020-06-15T17:53:00+02:00

I work with OpenDev CI for a while. My first Kolla patches were over three years ago. We (Linaro) added AArch64 nodes few times — some nodes were taken down, some replaced, some added.

Speed or lack of it

Whenever you want to install some Python package using pip it is downloaded from Pypi (directly or mirror). If there is a binary package then you get it, if not then “noarch” package is fetched.

In worst case source tarball is downloaded and whole build process starts. You need to have all required compilers installed, development headers for Python and all required libraries and rest of needed tools. And then wait. And wait as some packages require a lot of time.

And then repeat it again and again as you are not allowed to upload packages into Pypi for projects you do not own.

Argh you, protobuf

There was a new release of protobuf package. OpenStack bot picked it up, sent patch for review and it got merged.

And all AArch64 CI jobs failed…

Turned out that protobuf 3.12.0 was released with x86 wheels only. No source tarball. At all.

This turned out to be new maintainer mistake — after 2-3 weeks it was fixed in 3.12.2 release.

Another CI job then

So I started looking at ‘requirements’ project and created a new CI job for it. To check are new package versions are available for AArch64. Took some time and several side updates as well (yak shaving all the way again).

Stuff got merged and works now.

Wheels cache

While working on above CI job I had a discussion with OpenDev infra team how to make it work properly. Turned out that there were old jobs doing exactly what I wanted: building wheels and caching them for next CI tasks.

It took several talks and patches from Ian Wienand, Clark Boylan, Jeremy ‘fungi’ Stanley and others. Several CI jobs got renamed, some were moved from one project to another. Servers got configuration changes etc.

Now we have wheels built for both x86-64 and AArch64 architectures. Covering CentOS 7/8, Debian ‘buster’ and Ubuntu ‘xenial/bionic/focal’ releases. For OpenStack ‘master’ and few stable branches.

Effect

Requirements project has quick ‘check-uc’ job running on AArch64 to make sure that all packages are available for both architectures. All OpenStack projects profit from it.

In Kolla ‘openstack-base’ image went from 23:49 to just 5:21 minutes. Whole Debian/source build is now 57 minutes instead of 2 hours 20 minutes.

Nice result, isn’t it?

CirrOS 0.5.0 released

2020-03-04T11:53:00+01:00

Someone may say that I am main reason why CirrOS project does releases.

In 2016 I got task at Linaro to get it running on AArch64. More details are in my blog post ‘my work on changing CirrOS images’. Result was 0.4.0 release.

Last year I got another task at Linaro. So we released 0.5.0 version today.

But that’s not how it happened.

Multiple contributors

Since 0.4.0 release there were changes done by several developers.

Robin H. Johnson took care of kernel modules. Added new ones, updated names. Also added several new features.

Murilo Opsfelder Araujo fixed build on Ubuntu 16.04.3 as gcc changed preprocessor output.

Jens Harbott took care of lack of space for data read from config-drive.

Paul Martin upgraded CirrOS build system to BuildRoot 2019.02.1 and bumped kernel/grub versions.

Maciej Józefczyk took care of metadata requests.

Marcin Sobczyk fixed starting of Dropbear and dropped creation of DSS ssh key which was no longer supported.

My Linaro work

At Linaro I got Jira card with “Upgrade CirrOS’ kernel to Ubuntu 18.04’s kernel” title.

This was needed as 4.4 kernel was far too old and gave us several booting issues. Internally we had builds with 4.15 kernel but it should be done properly and upstream.

So I fetched code, did some test builds and started looking how to improve situation. Spoke with Scott Moser (owner of CirrOS project) and he told me about his plans to migrate from Launchpad to GitHub. So we did that in December 2019 and then fun started.

Continuous Integration

GitHub has several ways of adding CI to projects. First we tried GitHub Actions but turned out that it is paid service. Looked around and then I decided to go with Travis CI.

Scott generated all required keys and integration started. Soon we had every pull request going through CI. Then I added simple script (bin/test-boot) so each image was booted after build. Scott improved script and fixed Power boot issue.

Next step was caching downloads and ccache files. This was huge improvement!

In meantime Travis bumped free service to 5 simultaneous builders which got our builds even faster.

CirrOS supports building only under Ubuntu LTS. But I use Fedora so we merged two changes to make sure that proper ‘grub(2)-mkimage’ command is used.

Kernel changes

4.4 kernel had to go. First idea was to move to 4.18 from Ubuntu 18.04 release. But if we upgrade then why not going for HWE one? I checked 5.0 and 5.3 versions. As both worked fine we decided to go with newer one.

Modules changes

During start of CirrOS image several kernel modules are loaded. But there were several “no kernel module found” like messages for built-in ones.

We took care of it by querying /sys/module/ directory so now module loading is quiet process. At the end a list of loaded ones is printed.

VirtIO changes

Lot of things happened since 4.4 kernels. So we added several VirtIO modules.

One of results is working graphical console on AArch64. Thanks to ‘virtio-gpu’ providing framebuffer and ‘hid-generic’ handling usb input devices.

As lack of entropy is common issue in VM instances we added ‘virtio-rng’ module. No more ‘uninitialized urandom read’ messages from kernel.

Final words

Yesterday Scott created 0.5.0 tag and CI built all release images. Then I wrote release notes (based on ones from pre-releases). Kolla project got patch to move to use new version.

When next release? Looking at history someone may say 2023 as previous one was in 2016 year. But know knows. Maybe we will get someone with “please add s390x support” question ;D