tag:blogger.com,1999:blog-278767652024-03-18T18:36:00.999+01:00Day to day stuffExperiences from a hard core JVM programmer. Likes to keep things simple.Erik van Oostenhttp://www.blogger.com/profile/15976519439979651010noreply@blogger.comBlogger140125tag:blogger.com,1999:blog-27876765.post-43934488501722416232024-01-24T09:13:00.001+01:002024-01-24T09:17:35.244+01:00Scheduling tasks and sharing state with streams
Recently we built a system that needs to perform 2 tasks. Taks 1 runs every 15 minutes,
task 2 runs every 2 minutes. Task 1 kicks off some background jobs (an upload to
BigQuery), task 2 checks upon the results of these background jobs and does some cleanup
when they are done (delete the uploaded files). The two tasks need to share information back and forth about what
jobs are running in the Erik van Oostenhttp://www.blogger.com/profile/15976519439979651010noreply@blogger.com0tag:blogger.com,1999:blog-27876765.post-10118344911227862242023-11-26T09:58:00.001+01:002023-11-26T09:58:56.827+01:00Discovering scala-cli while fixing my digital photo archive
Over the years I built up a nice digital photo library with my family. It is a messy process. Here are some of the things that can go wrong:
Digital cameras that add incompatible exif metadata.
Some files have exif tag CreateDate, others DateTimeOriginal.
Images shared via Whatsapp or Signal do not have an exif date tag at all.
Wrong rotation.
Fuzzy, yet memorable jpeg images wich take 15MB Erik van Oostenhttp://www.blogger.com/profile/15976519439979651010noreply@blogger.com0tag:blogger.com,1999:blog-27876765.post-57311837130899840542023-10-08T09:15:00.003+02:002023-10-08T09:20:20.170+02:00Dependabot, Gradle and Scala
Due to a series of unfortunate circumstances we have to deal with a couple of projects that use Gradle as build tool at work.
For these projects we wanted automatic PR generation for updated dependencies.
Since we use Github Enterprise, using Dependabot seems logical.
However, this turned out to be not very straightforward.
This article documents one way that works for us.
As we Erik van Oostenhttp://www.blogger.com/profile/15976519439979651010noreply@blogger.com0tag:blogger.com,1999:blog-27876765.post-30559928158543673432023-04-20T14:22:00.012+02:002023-04-23T20:21:39.814+02:00Zio-kafka hacking day
Not long ago I contacted Steven (committer of the zio-kafka library) to get some better understanding of how the library works. April 12, not more than 2 months later I am a committer, and I was sitting in a room together with Steven, Jules Ivanic (another committer) and wildcard Pierangelo Cecchetto (contributor), hacking on zio-kafka.
The meeting was an idea of Jules who was ‘in the Erik van Oostenhttp://www.blogger.com/profile/15976519439979651010noreply@blogger.com0tag:blogger.com,1999:blog-27876765.post-83809933083801068982023-01-29T10:20:00.004+01:002023-01-29T10:25:13.961+01:00Kafka is good for transport, not for system boundaries
In the last years I have learned that you should not run Kafka as a system boundary. A system boundary in this
article is the place where messages are passed from one
autonomy domain
to another.
Now why is that? Let’s look at two classes of problems: connecting to Kafka and the long feedback loop. To
prove my points, I am going to bore you with Erik van Oostenhttp://www.blogger.com/profile/15976519439979651010noreply@blogger.com0tag:blogger.com,1999:blog-27876765.post-45032717870956593792022-12-04T20:28:00.000+01:002022-12-04T20:28:14.609+01:00ZIO service layer pattern
While reading about ZIO-config in 2.0.4, the following pattern to create services caught my eye. I am copying it here for easy lookup. Enjoy.
val myLayer: ZLayer[PaymentRepo, Nothing, MyService] =
ZLayer.scoped {
for {
repo <- ZIO.service[PaymentRepo]
config <- ZIO.config(MyServiceImpl.config)
ref <- Ref.make(MyState.Initial)
impl <- ZIO.succeed(new Erik van Oostenhttp://www.blogger.com/profile/15976519439979651010noreply@blogger.com0tag:blogger.com,1999:blog-27876765.post-17508256956019929702022-11-05T15:58:00.008+01:002024-01-24T09:26:21.176+01:00Speed up ZIOs with memoization
TLDR: You can do ZIO memoization in just a few lines, however, use zio-cache for more complex use cases.
Recently I was working on fetching Avro schema's from a schema registry.
Avro schema's are immutable and therefore perfectly cacheable.
Also, the number of possible schema's is limited so cache evictions are not needed.
We can simply cache every schema for ever in a plain hash-map.
So, we Erik van Oostenhttp://www.blogger.com/profile/15976519439979651010noreply@blogger.com2tag:blogger.com,1999:blog-27876765.post-56821546167984199492022-06-08T09:35:00.005+02:002022-06-08T09:37:23.842+02:00Zigzag bytesI was playing around with a goofy idea for which I needed zigzag encoding for bytes. Zigzag encoding is often used in combination with variable length encoding in things like Avro, Thrift and Protobuf.
In zigzag encoded integers, the least significant bit is used for sign. To convert from regular encoding (2-complement) to zigzag (and back) you can use the following Scala code:
def i32ToZigZag(Erik van Oostenhttp://www.blogger.com/profile/15976519439979651010noreply@blogger.com0tag:blogger.com,1999:blog-27876765.post-81850729996066938802022-03-12T17:09:00.008+01:002023-12-06T19:57:27.689+01:00Upgrading Libreoffice with HomebrewUpdate 2023-06-06 brew now asks for your password so it can install everything directly. Much better!!
The text below is no longer applicable and only kepts as reference.
Reminder to self: this is the procedure to upgrade Libreoffice with Homebrew:
brew update
brew upgrade
open -a /Applications/LibreOffice.app
Quit the application
brew reinstall libreoffice-language-pack, enter Erik van Oostenhttp://www.blogger.com/profile/15976519439979651010noreply@blogger.com0tag:blogger.com,1999:blog-27876765.post-1796488393606049202022-01-26T14:17:00.000+01:002022-01-26T14:17:16.939+01:00Having fun with Ordering in ScalaChallenge: sort a list of objects by name, but some names have priority. If these names appear, they should be ordered by the position they have in the priority list.
For example:
val priorityList = Seq("Willow", "James", "Ezra")
val input = Seq("Olivier", "Charlotte", "Willow", "Declan", "Aurora", "Ezra")
val ordered = ???
assert(ordered == Seq("Willow", "Ezra", "Aurora", "Charlotte", "Declan"Erik van Oostenhttp://www.blogger.com/profile/15976519439979651010noreply@blogger.com0tag:blogger.com,1999:blog-27876765.post-32328591035242361362022-01-10T14:21:00.005+01:002022-01-10T15:12:20.783+01:00From Adoptopenjdk to Temurin on a Mac using HomebrewAdoptopenjdk joined the Eclipse foundation and renamed their JDK to Temurin. Here are instructions on how to migrate on Macs with Homebrew.
The following instructions removes Adoptopenjdk JDKs you may still have:
brew remove adoptopenjdk/openjdk/adoptopenjdk8
brew remove adoptopenjdk/openjdk/adoptopenjdk11
brew untap AdoptOpenJDK/openjdk
brew remove adoptopenjdk8
brew remove adoptopenjdk11
brewErik van Oostenhttp://www.blogger.com/profile/15976519439979651010noreply@blogger.com0tag:blogger.com,1999:blog-27876765.post-12412615250160144472021-12-19T17:44:00.004+01:002021-12-19T17:52:24.278+01:00Customizing the Jitsi Meet UI in a Docker deployment
I manage a Jitsi instance for a small for-benefit organization. I wanted
to make some changes to the UI to make it visually belong to the organization.
Unfortunately, Jitsi doesn't make it easy to do this. Upon every upgrade
your changes are gone. This post describes a workaround for Jitsi deployments that use Docker.
Although the details can be hairy, the idea is quite simple. We Erik van Oostenhttp://www.blogger.com/profile/15976519439979651010noreply@blogger.com0tag:blogger.com,1999:blog-27876765.post-74479489872576107942021-12-08T14:36:00.003+01:002021-12-08T17:50:00.013+01:00Akka graceful shutdown - continuedSome time ago I wrote on how to gracefully shutdown Akka HTTP servers, crucial to prevent aborted requests during re-deployments or in elastic (cloud) environments where instances come and go. Look at the previous post for more details on how graceful shutdown works and some common caveats in setting it up.
This post refreshes how this works for newer Akka versions, and it gives some tips on howErik van Oostenhttp://www.blogger.com/profile/15976519439979651010noreply@blogger.com0tag:blogger.com,1999:blog-27876765.post-52504636762913378222020-07-07T12:03:00.003+02:002021-03-21T15:06:15.137+01:00Avoiding Scala's Option.foldScala's Option.fold is a bit nasty to use sometimes because type inference is not always great. Here is a stupid toy example:
val someValue: Option[String] = ???
val processed = someValue.fold[Either[Error, UserName]](
Left(Error("No user name given"))
)(
value => Right(UserName(value))
)
The [Either[Error, UserName]] on fold is necessary otherwise the scala compiler can not derive the Erik van Oostenhttp://www.blogger.com/profile/15976519439979651010noreply@blogger.com0tag:blogger.com,1999:blog-27876765.post-19922938668291938012020-04-15T11:29:00.001+02:002020-04-15T11:32:51.195+02:00Traefik v2 enable HSTS, Docker and nextcloudThis took me days to figure out how to configure Traefik v2. Here it is for posterity.
This is a docker-compose.yaml fragment to append to a service section:
labels:
- "traefik.enable=true"
- "traefik.http.routers.service.rule=Host(`www.example.com`)"
- "traefik.http.routers.service.entrypoints=websecure"
- "traefik.http.routers.service.tls.certresolver=myresolver"
Erik van Oostenhttp://www.blogger.com/profile/15976519439979651010noreply@blogger.com4tag:blogger.com,1999:blog-27876765.post-91682011202510159502020-04-10T17:03:00.001+02:002021-12-08T14:40:04.522+01:00Akka-http graceful shutdownWhy?
By default, when you restart a service, the old instance is simply killed. This means that all current requests are aborted; the caller will be left with a read timeout. We can do better!
What?
A graceful shutdown looks as follows:
The scheduler (Kubernetes, Nomad, etc.) sends a signal (usually SIGINT) to the service.
The service gets the signal and closes all server-ports; it can no Erik van Oostenhttp://www.blogger.com/profile/15976519439979651010noreply@blogger.com0tag:blogger.com,1999:blog-27876765.post-84202442427967265842020-03-03T10:56:00.000+01:002020-03-05T15:06:03.486+01:00Push GaugesA colleague was complaining to me that Micrometer gauges didn't work the way he expected. This led to some interesting work.
What is a gauge?
In science a gauge is a device for making measurements. In computer systems a gauge is very similar: a 'metric' which tracks something in your system over time. For example, you could track the number of items in a job queue. Libraries like Micrometer andErik van Oostenhttp://www.blogger.com/profile/15976519439979651010noreply@blogger.com0tag:blogger.com,1999:blog-27876765.post-15002828921243121292017-07-12T11:05:00.004+02:002017-07-12T11:05:59.571+02:00Continuation parsers and encoders
I finally got around to writing about my hack project of last year. It was an exploration of what can be done with continuation parsers and encoders in order to implement a very fast single-copy asynchronous Thrift implementation.
Continuation parsers and encoders try to decode (read)/encode (write) their data directly from/to a network buffer. When the buffer has been fully read/written, it Erik van Oostenhttp://www.blogger.com/profile/15976519439979651010noreply@blogger.com0tag:blogger.com,1999:blog-27876765.post-31867658774683348532017-04-18T11:50:00.003+02:002017-04-21T10:16:56.518+02:00Bash history backupI like my bash history, and I proudly have this in my .bash_profile:
# Increase bash history size, append instead of overwrite history on exit
export HISTSIZE=10000000
export HISTCONTROL=erasedups
shopt -s histappend
However, after reading about Historian I realised I have no backup. Instead of installing Historian, I decided to take a simpler approach. Here it is:
#!/bin/bash
set -euo Erik van Oostenhttp://www.blogger.com/profile/15976519439979651010noreply@blogger.com0tag:blogger.com,1999:blog-27876765.post-33247579709704803002016-09-13T10:51:00.000+02:002017-02-05T07:54:10.915+01:00Exploring Rkt on UbuntuI have been using docker in my home server since 0.4 in 2013. For me the most attractive property of docker is that it provides a way to decrease the amount of stuff one has to install on a server. I have only one server, but it has many different tasks of which some need to be rock solid (my family email) and other are experimental. Containers provide a nice way to clean up experiments. Erik van Oostenhttp://www.blogger.com/profile/15976519439979651010noreply@blogger.com0tag:blogger.com,1999:blog-27876765.post-44562631052606259502016-03-03T21:19:00.000+01:002016-03-03T21:19:32.696+01:00Don’t call your state ‘state’In the OO world you are frowned upon if you call something object. Its time to extend this principle: don’t call your state state. This post is about why this is a bad idea and what you can do about it.
Recently we sat down to discuss the new data model for our messaging system at eBay’s Classifieds Group. One of the things we inherited from the past is the entity called conversation with a Erik van Oostenhttp://www.blogger.com/profile/15976519439979651010noreply@blogger.com2tag:blogger.com,1999:blog-27876765.post-13444393872349925332016-02-14T14:08:00.001+01:002016-02-14T14:09:12.806+01:00Generate a certificate signing request (CSR) as a 1 linerAs I run my own (secure) web and mail server I frequently have to get certificates. You could run with a selfsigned certificate, but that is not ideal; desktop browsers are very noisy about them, and mobile browsers are outright hostile. Installing certificates on a mobile phone are no fun (though, hat of for CAdroid).
Now that Letsencrypt is live, I wanted to try it out. However, Letsencrypt Erik van Oostenhttp://www.blogger.com/profile/15976519439979651010noreply@blogger.com0tag:blogger.com,1999:blog-27876765.post-47823168446603993522015-05-20T21:35:00.000+02:002015-05-20T21:35:00.988+02:005 problems in an hour@svpino posted a nice chalenge: solve 5 problems within an hour or denounce your title of software engineer.
It took me 40 minutes, of which 10 minutes was fighting around a limitation of the scala REPL.
Here are my solutions. They can all be pasted as is in the Scala REPL.
/////////////////////////////////////////
// Problem 1 (Yuc! This is not a Scala problem!)
def p1_for(xs: Seq[Int]): IntErik van Oostenhttp://www.blogger.com/profile/15976519439979651010noreply@blogger.com0tag:blogger.com,1999:blog-27876765.post-62891582537401050542013-11-01T10:29:00.001+01:002013-11-01T10:30:29.021+01:00Installing gnutar on MaverickUnfortunately Apple decided to remove /usr/bin/gnutar from Maverick (Mac OSX 10.9). This is a pain because most of the tarring I do on my mac is to transfer the file to a GNU based linux (e.g. Debian/Ubuntu). Apple's bsd-tar is not compatible with gnu-tar.
This is my solution:
brew install gnu-tar
cd /usr/bin
sudo ln -s /usr/local/opt/gnu-tar/libexec/gnubin/tar gnutar
Erik van Oostenhttp://www.blogger.com/profile/15976519439979651010noreply@blogger.com3tag:blogger.com,1999:blog-27876765.post-88628331730541968932013-09-18T21:18:00.001+02:002013-09-18T21:18:36.152+02:00Configuring Postfix/Dovecot for Microsoft Windows Live MailPersonal mail gets no love from Microsoft. The last 10 year I have not seen their product change a lot. Notable I see name changes (always a bad sign) and some visual changes. The actual implementation is still the same: not respecting standards.
I run a Postfix/Dovecot installation for my family mail. I have had many many different email clients connect to it without large problems. With Erik van Oostenhttp://www.blogger.com/profile/15976519439979651010noreply@blogger.com0