Holden's Blog

New year new job, same projects

2026-02-12T07:51:00.000-08:00

I’m stoked to announce that I’ve joined Snowflake to continue working on OSS Apache Spark :) I’ve got a post on the Snowflake blog talking about the work we’re doing https://careers.snowflake.com/us/en/blogarticle/building-apache-spark-in-the-open-at-snowflake —

Making Hibernate work on Ubuntu 22.04 (jammy) on the Framework Laptop w/full disk encryption

2022-05-30T17:44:00.002-07:00

Making Hibernate work on Ubuntu 22.04 (jammy) on the Framework Laptop w/full disk encryption

The 0th step is disabling secure boot in the bios (reboot, press F2 to get into the BIOS, disable secure boot).

The first step is to make a swapfile large enough for your device to suspend to.

sudo swapoff /swapfile

sudo dd if=/dev/zero of=/swapfile bs=$(cat /proc/meminfo | awk '/MemTotal/ {print $2}') count=1024 conv=notrunc

sudo mkswap /swapfile

sudo swapon /swapfile

Then make sure it's enabled on reboot, e.g: echo '/swapfile swap swap defaults 0 0' | sudo tee -a /etc/fstab

If there is another swap partition or file in /etc/fstab you may need to take it out.

Find the device your swapfile is located on (e.g. findmnt -no source -T /swapfile), note here some guides recommend using the UUID, in my experience, this resulted in some errors.

To find the resume offset of your file run filefrag -v /swapfile and then look at the first value under physical_offset.

Set `GRUB_CMDLINE_LINUX_DEFAULT="splash resume=${RESUME_DEVICE} resume_offset=${RESUME_OFFSET} nvme.noacpi=1"` in `/etc/default/grub` and then update grub with sudo update-grub

If you want to enable hibernation on lid close set `HandleLidSwitch=hibernate` in `/etc/systemd/logind.conf`.

Taking a Break (surgery)

2020-10-31T10:18:00.001-07:00

Once again I am having surgery. As with all surgery, there is a chance for complications, so while I hope to be back up and at most of my usual things by the end of November, there is a non-zero chance that plan won't pan out. Unlike last time (fixing bones from a Vespa/motorcycle crash), this is one I've had in the back of my mind for a bit (facial feminization surgery) and I'm really excited about it.

I am going to try and do some blogging during my recovery period -- if I feel up to doing so (predicting how recovery is going to be is HARD). I do know that when I feel like doing streams again, that'll be without my face visible before I feel comfortable with myself on camera again -- thanks for understanding. I will continue to do code reviews and fix any bugs I've introduced whenever possible, but I will be slower to respond.

If you are an American citizen, please vote! And regardless of where you are when you can, vote for love over hate. Local elections can have huge impacts too, and even in liberal cities, I've seen people running on platforms that would severely hurt myself, my friends, and our community. I love San Francisco, and I don't want to have to leave (especially after finally getting my green-card this year).

Socially Distant Air Hugs & Kisses,

Holden

Scaling Python for ML (new blog)

2020-10-26T16:46:00.002-07:00

I've started a new blog over at http://scalingpythonml.com/ exploring the different tools to scale Python ML code. I'm just getting started but you can also follow along with my YouTube channel.

The Importance of Reviewing PR comments @ Scala Days EU 2019

2019-06-12T02:42:00.001-07:00

Come join me on Wednesday 12 June @ 17:45 for The Importance of Reviewing PR comments.I'll update this post with the slides soon.Come see to the talk or comment bellow to join in the discussion :).Talk feedback is appreciated at http://bit.ly/holdenTalkFeedback

Validating Spark ML Jobs-Stopping Failures Before Production on Apache Spark @ @SparkAISummit SF 2019

2019-04-24T11:10:00.001-07:00

Thanks for joining me on 2019-04-24 at @SparkAISummit SF 2019 San Francisco, CA, USA for Validating Spark ML Jobs-Stopping Failures Before Production on Apache Spark.

The slides are at http://bit.ly/2KYpqig.

Validating spark ml jobs stopping failures before production on Apache Spark @ Spark Summit 2019 from Holden Karau

Comment bellow to join in the discussion :).Talk feedback is appreciated at http://bit.ly/holdenTalkFeedback

Powering Tensorflow with Big Data @ CERN Computing Seminar

2019-04-03T23:26:00.001-07:00

Thanks for joining me on 2019-04-03 for Powering Tensorflow with Big Data.

The slides are at http://bit.ly/2MWFpi3.

Powering tensor flow with big data using apache beam, flink, and spark cern 2019 (3) from Holden Karau

Comment bellow to join in the discussion :).Talk feedback is appreciated at http://bit.ly/holdenTalkFeedback

PySpark on Kuberntes @ Python Barcelona March Meetup

2019-03-21T10:25:00.001-07:00

Thanks for joining me on 2019-03-21 at Python Barcelona March Meetup 2019 Barcelona, Spain for PySpark on Kuberntes.

The slides are at http://bit.ly/2Fv7Uwj.

PySpark on Kubernetes @ Python Barcelona March Meetup from Holden Karau

Comment bellow to join in the discussion :).Talk feedback is appreciated at http://bit.ly/holdenTalkFeedback

Contributing to Spark 3 @ Spark BCN Meetup

2019-03-19T10:17:00.001-07:00

Thanks for joining me on 2019-03-19 at Spark BCN Meetup 2019 Barcelona, Spain for Contributing to Spark 3.

The slides are at http://bit.ly/2HwlXFf.

Contributing to Apache Spark 3 from Holden Karau

Comment bellow to join in the discussion :).Talk feedback is appreciated at http://bit.ly/holdenTalkFeedback

PyData Hong Kong - Making the Big Data ecosystem work together with Python: Apache Arrow, Spark, Flink, Beam, and Dask @ PyData Hong Kong

2019-02-18T18:02:00.001-08:00

Thanks for joining me on 2019-02-19 at PyData Hong Kong 2019 Hong Kong for PyData Hong Kong - Making the Big Data ecosystem work together with Python: Apache Arrow, Spark, Flink, Beam, and Dask.I'll update this post with the slides soon.Comment bellow to join in the discussion :).Talk feedback is appreciated at http://bit.ly/holdenTalkFeedback

Validating Big Data Jobs An exploration with @ApacheSpark & @ApacheAirflow (+ friends) @ FOSDEM

2019-02-05T06:50:00.001-08:00

Thanks for joining me on 2019-02-03 at FOSDEM 2019 Brussels, Belgium for Validating Big Data Jobs An exploration with @ApacheSpark & @ApacheAirflow (+ friends).

The slides are at http://bit.ly/2wMwRiF.

Validating big data pipelines - FOSDEM 2019 from Holden Karau

Comment bellow to join in the discussion :).Talk feedback is appreciated at http://bit.ly/holdenTalkFeedback

Introducing @Kubeflow (w. Special Guests Tensorflow and @ApacheSpark) @ FOSDEM

2019-02-02T01:30:00.001-08:00

Thanks for joining us (@holdenkarau,@rawkintrevo) on 2019-02-03 at FOSDEM 2019 Brussels, Belgium for Introducing @Kubeflow (w. Special Guests Tensorflow and @ApacheSpark).I'll update this post with the slides soon.Comment bellow to join in the discussion :).Talk feedback is appreciated at http://bit.ly/holdenTalkFeedback

Apache Spark on Kubernetes -- Avoiding the pain of YARN @ Pre-FOSDEM Belgium Kubernetes Meetup

2019-02-01T02:34:00.001-08:00

Thanks for joining me on 2019-02-02 for Apache Spark on Kubernetes -- Avoiding the pain of YARN.The talk covered:

Apache Spark is one of the most popular big data tools, and starting last year has had integrated support for running on Kubernetes. This talk will introduce some of the use cases of Apache Spark quickly (machine learning, ETL, etc.) and then look at the current cluster managers Spark runs on and their limitations. Most of the focus will be around running non-Java code, and the challenges associated with dependencies along with general challenges like scale-up & down. Its not all sunshine and roses though, I will talk about some of the limitations of our current approach and the work being done to improve this.

.I'll update this post with the slides soon.Comment bellow to join in the discussion :).Talk feedback is appreciated at http://bit.ly/holdenTalkFeedback

Predicting areas for PR Comments based on Code Vectors & Mailing List Data @ FOSDEM

2019-01-02T00:07:00.001-08:00

Thanks for joining us (@holdenkarau,@krisnova) on 2019-02-03 at FOSDEM 2019 Brussels, Belgium for Predicting areas for PR Comments based on Code Vectors & Mailing List Data.I'll update this post with the slides soon.Comment bellow to join in the discussion :).Talk feedback is appreciated at http://bit.ly/holdenTalkFeedback

Understanding Spark Tuning with Auto Tuning (or how to stop your pager going off @ Data Day Texas

2018-12-31T15:44:00.001-08:00

Thanks for joining me on 2019-01-26 at Data Day Texas 2019 Austin, TX, USA for Understanding Spark Tuning with Auto Tuning (or how to stop your pager going off.I'll update this post with the slides soon.Comment bellow to join in the discussion :).Talk feedback is appreciated at http://bit.ly/holdenTalkFeedback

Validating Big Data Jobs - Stopping Failures before Production (w/ Spark, BEAM, & friends!) -- now with Scooters @ Scala eXchange 2018

2018-12-14T06:18:00.001-08:00

Thanks for joining me on 2018-12-14 at Scala eXchange 2018 London, UK for Validating Big Data Jobs - Stopping Failures before Production (w/ Spark, BEAM, & friends!) -- now with Scooters.I'll update this post with the slides soon.Comment bellow to join in the discussion :).Talk feedback is appreciated at http://bit.ly/holdenTalkFeedback

Holden @ Kiwi Code Mania: talk title TBD @ Code Mania

2018-12-14T03:17:00.001-08:00

Thanks for joining me on 2019-05-15 at Code Mania 2019 Auckland, New Zealand for Holden @ Kiwi Code Mania: talk title TBD.I'll update this post with the slides soon.Comment bellow to join in the discussion :).Talk feedback is appreciated at http://bit.ly/holdenTalkFeedback

Validating Big Data Jobs - Stopping Failures before Production (w/ Spark, BEAM, & friends!) @ Big Data Spain

2018-11-14T02:56:00.001-08:00

Thanks for joining me on 2018-11-14 at Big Data Spain 2018 Madrid, Spain for Validating Big Data Jobs - Stopping Failures before Production (w/ Spark, BEAM, & friends!).

The slides are at http://bit.ly/2S1y3HJ.The video of the talk is up at http://bit.ly/2Gxk23h.

Validating Big Data Pipelines - Big Data Spain 2018 from Holden Karau

Comment bellow to join in the discussion :).Talk feedback is appreciated at http://bit.ly/holdenTalkFeedback

Big Data w/Python on Kubernetes (PySpark on K8s) @ Big Data Spain

2018-11-14T02:53:00.001-08:00

Thanks for joining me on 2018-11-15 for Big Data w/Python on Kubernetes (PySpark on K8s).

The slides are at http://bit.ly/2RWsxpA.The video of the talk is up at http://bit.ly/2R9x4bE.

Big data with Python on kubernetes (pyspark on k8s) - Big Data Spain 2018 from Holden Karau

And some related links: recorded demos, setup livestreamComment bellow to join in the discussion :).Talk feedback is appreciated at http://bit.ly/holdenTalkFeedback

A Basic Introduction to PySpark Dataframes by exploring ASF Gender Diversity Data - workshop @ @pyconca

2018-11-09T17:49:00.001-08:00

Thanks for joining us (@holdenkarau,@math_foo) on 2018-11-10 at @pyconca 2018 Toronto, ON, Canada for A Basic Introduction to PySpark Dataframes by exploring ASF Gender Diversity Data - workshop.

You can find the code for this talk at https://github.com/holdenk/diversity-analytics/.The slides are at http://bit.ly/2RNhG1g.There is a related video you might want to check out.

Comment bellow to join in the discussion :).Talk feedback is appreciated at http://bit.ly/holdenTalkFeedback

PyConCanada After Lunch Keynote Sunday @ @pyconca

2018-11-08T21:02:00.001-08:00

Thanks for joining me on 2018-11-08 for PyConCanada After Lunch Keynote Sunday.I'll update this post with the slides soon.Comment bellow to join in the discussion :).Talk feedback is appreciated at http://bit.ly/holdenTalkFeedback

Dealing With Contributor Overload @ Festival de Software Libre

2018-11-02T13:58:00.001-07:00

Thanks for joining us (@holdenkarau,@griscz) on 2018-11-02 at Festival de Software Libre 2018 Puerto Vallarta, Jalisco, Mexico for Dealing With Contributor Overload.I'll update this post with the slides soon.Comment bellow to join in the discussion :).Talk feedback is appreciated at http://bit.ly/holdenTalkFeedback

Diversity in OSS with Holden & Gris @ Festival de Software Libre

2018-11-02T10:27:00.001-07:00

Thanks for joining us (@holdenkarau,@griscz) on 2018-11-02 at Festival de Software Libre 2018 Puerto Vallarta, Jalisco, Mexico for Diversity in OSS with Holden & Gris.

The slides are at http://bit.ly/2JBYgcw.

Keynote Open Source Diversity - Festival del Software Libre from Holden Karau

Comment bellow to join in the discussion :).Talk feedback is appreciated at http://bit.ly/holdenTalkFeedback

End to End ML with Kubeflow: Scaling with Big & Tiny Data (+ deep learning of course) @ signal

2018-10-13T16:55:00.001-07:00

Thanks for joining me on 2018-10-17 at signal 2018 San Francisco, CA, USA for End to End ML with Kubeflow: Scaling with Big & Tiny Data (+ deep learning of course).

The slides are at http://bit.ly/2pWFUKj.The video of the talk is up at http://bit.ly/2GBMp0B.And if you want there is a related codelab you can try out.

Intro - End to end ML with Kubeflow @ SignalConf 2018 from Holden Karau

Comment bellow to join in the discussion :).Talk feedback is appreciated at http://bit.ly/holdenTalkFeedback

Building Recoverable (and optionally Async) Spark Pipelines @ Scylla Summit 2018

2018-10-13T16:34:00.001-07:00

Thanks for joining me on 2018-11-07 for Building Recoverable (and optionally Async) Spark Pipelines.

The slides are at http://bit.ly/2D90dfy.

Building Recoverable (and optionally async) Pipelines with Apache Spark (+ small revisions) from Holden Karau

Comment bellow to join in the discussion :).Talk feedback is appreciated at http://bit.ly/holdenTalkFeedback