bjg.io

SerenityOS Office Hours (with friends!)

2022-07-24T00:00:00+00:00

On June 24th, 2022 Andreas asked Dániel Bertalan and I to join him during his weekly office hours stream. It was great to talk with the community and answer questions!

SerenityOS Office Hours On YouTube:

SerenityOS Office Hours (with friends!) was originally published by Brian Gianforcaro at bjg.io on July 24, 2022.

SerenityOS Developer Interview: Me!

2021-12-14T00:00:00+00:00

In November of 2021 Andreas Kling asked if he could interview me for his youtube channel. He has a very entertaining series where he interviews developers who contribute to the SerenityOS project.

It was wonderful talking with Andreas, I really enjoyed the experience. SerenityOS has been a significant part of my life over the past few years, so it was nice to let the community know a little bit more about me.

The Interview On YouTube:

SerenityOS Developer Interview: Me! was originally published by Brian Gianforcaro at bjg.io on December 14, 2021.

Mixed Extents 12: SQLPAL

2021-11-19T00:00:00+00:00

Back in September of 2021 I was invited to talk about SQLPAL on the SQL Server podcast Mixed Extents. The Mixed Extents folks were a lot of fun to chat with. It’s always awesome getting to talk to a group of passionate and informed power users of SQL Server On Linux!

The Episode On YouTube:

Mixed Extents 12: SQLPAL was originally published by Brian Gianforcaro at bjg.io on November 19, 2021.

Tricks for debugging python internals on Windows

2020-08-09T00:00:00+00:00

Background

At work I was debugging and issue where I had to dig into the internals of the Python runtime running on Windows. Along the way I learned a few tips and tricks that might come in handy for someone else. So I figured I’d do a quick little write up to share the knowledge. To start you’ll want to debug python with the windows system debugger: WinDbg. Debugging runtime like Python is convenient in Visual Studio, but in the end it’s not as powerful and features available in WinDbg will make things a lot easier.

Symbol Support

The first thing you’ll want is symbols for your python release. There are multiple ways to get Python symbols:

You can download them manually using this helpful guide.
You can use Sean Cline’s symbol server, and the debugger will download symbols for you.

The symbol server support seems to work well, and is easy, so I would recommend that. To add the symbol server to your WinDbg symbol path, run:

1:044> .sympath+ http://pythonsymbols.sdcline.com/symbols

Source Support

Now that we have symbols, next we need to get the source to line up with the code you are debugging and stepping through. First we need to use the debugger to find out what version of python we are debugging:

1:044> !filever python38.dll
filever python38.dll
GetFileVersionInfo
Version: 3.8.150.1013 
0000000f`02ca0000 0000000f`030c9000   python38
    Loaded symbol image file: python38.dll
    Image path: python38.dll
    Image name: python38.dll
    Browse all global symbols  functions  data
    Timestamp:        Mon Oct 14 12:38:17 2019 (5DA4CEA9)
    CheckSum:         0040A466
    ImageSize:        00429000
    Product version:  3.8.150.1013
    File flags:       0 (Mask 3F)
    File OS:          4 Unknown Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0000.04b0
    Information from resource tables:
        CompanyName:      Python Software Foundation
        ProductName:      Python
        InternalName:     Python DLL
        OriginalFilename: python38.dll
        ProductVersion:   3.8.0
        FileVersion:      3.8.0

The version I’m debugging appears to be based on Python 3.8.0, using that information we can clone a copy of the cpython repository, and checkout the correct branch:

c:\> git clone https://github.com/python/cpython.git
c:\> pushd cpython
c:\cpython> git tag -l | findstr 3.8.0
v3.8.0
...

c:\cpython> git checkout v3.8.0

Finally we can fix up the source path in our debugger to point to our python source code:

1:044> .srcpath+ c:\cpython

Dynamic Breakpoints

Another trick that might need, is the ability to inject dynamic breakpoints into the python code in order to narrow down a bug reproduction, or catch the system in a certain state. Unfortunately there is no way to write inline assembly in python. However we can take advantage of the Python ctypes library. It exposes the windll API with support for kernel32.dll exported functions. Using windll we can call into the windows implementation of DebugBreak() via the foreign function wrapper. This will trigger a breakpoint exception in the process, and the debugger will break into handle it.

Here’s an example of using it to instrument some code:

# Call DebugBreak export from kernel32
from ctypes import windll
windll.kernel32.DebugBreak()

thing_you_are_trying_to_debug()

The PyExt WinDbg Extension

Discovering the PyExt WinDbg extension allowed me to start being productive and narrowing down my issue. It’s very useful for gleaning bits of information about the python runtime while debugging and issue or analyzing a crash dump.

To install, download the latest version from the PyExt Release Page. Extract the zip file somewhere on disk. Then you can load the extension into the debugger with the the following command.

.load C:\PyExt-x64-Release\x64\Release\pyext.dll

The extension exposes a few useful commands:

1:044> !pyext.help
Commands for C:\PyExt-x64-Release\x64\Release\pyext.dll:
  !help     - Displays information on available extension commands
  !pyobj    - Prints information about a Python object
  !pystack  - Prints the Python stack for the current thread, or starting at a
              provided PyFrameObject
  !pysymfix - Adds python symbols to the symbol path.

As a short cut to the steps listed above the extension provides a command to quickly setup the python symbol server: !pysymfix

1:044> !pysymfix
Current symbol path: srv*
Adding symbol server to path...

************* Path validation summary **************
Response                         Time (ms)     Location
Deferred                                       http://pythonsymbols.sdcline.com/symbols
New symbol path: http://pythonsymbols.sdcline.com/symbols
Loading symbols...

The extension also allows you to traverse the runtime stack frame object and display a stack trace of the executing python code, using the !pystack command. The utilize !pystack we’ll need to find a python stack frame pointer somewhere on on the native call stack. Lets look around and try to find one:

044> .f+
0000000f`0044bfe0 0000000f`02ce9770 python38!_PyEval_EvalFrameDefault+0xaba

044> dv
prv param @r12  struct _frame * f = 0x0000000f`0886e800

We simply pass the address of the _frame object to the !pystack command and let it work:

1:045> !pystack 0x0000000f0886e800 
Stack trace starting at (PyFrameObject*)(00000008`85b46130):
    File "C:\python-3.8.0.amd64\lib\site-packages\pip\__main__.py", line 26, in <module>
                    [Locals] [Globals] 
    File "C:\python-3.8.0.amd64\lib\runpy.py", line 85, in _run_code
                    [Globals] 
    File "C:\python-3.8.0.amd64\lib\runpy.py", line 192, in _run_module_as_main
                    [Globals]

Next the !pyobj extension allows you dump python runtime variables. Objects in the CPython implementation are of type _object, this is what !pyobj operates on. Lets find an example in our test case, and try dumping it’s value:

1:049> g
Breakpoint 11 hit
python38!_io_FileIO___init___impl:
0000000f`02d3652c 48895c2410      mov     qword ptr [rsp+10h],rbx ss:0000000f`0044aff8=0000000000000002

1:049> dv nameobj
prv param  @rdx              @rdx              struct _object * nameobj = 0x0000000f`04cc2f10

1:049> !pyobj 0xf04cc2f10
PyASCIIObject at address: 0000000f`04cc2f10
    RefCount: 81
    Type: str
    Repr: 'C:\\python-3.8.0.amd64\\lib\\tempfile.py'

Conclusion

These resources, tips, and tricks helped me dig into the problem I was working on and figure out what was going on. I hope they might help out someone else out there debugging python on windows. Thanks for reading!

Tricks for debugging python internals on Windows was originally published by Brian Gianforcaro at bjg.io on August 09, 2020.

PR Dash: A command line pull request dashboard

2020-07-03T00:00:00+00:00

Many Code Review Systems

I’m a strong believer in the value of code review. I spend a large portion of my days reviewing my peers’ code at my day job. At Microsoft the universal tool for code review is CodeFlow, a UI for reviewing code. It’s great at showing you all of the details of a singular code review. Azure DevOps, GitHub, and GitLab all have a way of viewing assigned reviews for given project. Many developers today participate in projects across many different services; some are involved in several internal projects during their work week and then participate in or lead multiple open source projects in their free time.

This whole system ends up being a bit of a time suck. All of these various platforms rely on you to check them constantly to monitor progress on code reviews that you are the author of, as well as reviews where you are a reviewer. This made me wonder if it would be possible to create a grand, unified view of the status of all your required work.

Jane Street’s “Iron”

Years ago, I ran across the series of talks and blogs that the folks over at Jane Street put together about their internal code review system, Iron.

There are a ton of interesting things about the system they describe. Including Iron ability to manage all review activities from a singular UI. For instance, there is a global overview of all the reviews you are an author of as well as, those you are a reviewer of. The overview gives you an idea of how many lines are changed per review which allows you to quickly unblock your peer by choosing to review their small pull requests before working on your own projects.

Introducing pr-dash

Late last year, I finally had enough of manually tracking pull requests across the multiple repositories I contribute to every day. The faint memory of the Iron user interface inspired me to create a tool for managing my pull requests. I ended writing pr-dash, a console-based pull request dashboard.

It’s currently written in C# using dotnet core, and lives on github. Currently it is able abile to monitor Azure DevOps repositories, but it was built with future extensibility in mind.

The main features the tool offers are:

Ability to filter only those PRs that are considered “actionable”, which filters PRs which you are waiting on feedback from the author, or have already signed off on, or are in draft state;
Ability to switch between actionable, signed off, draft, or PRs you are waiting on;
Ability to switch to just the PRs that you created and are waiting for feedback on.

When you want to view a PR you can open it in your default web browser by pressing enter while it’s selected. This allows you to take action on the pull request. The user interface refreshes automatically in the background to make sure you are always up to date!

It works well enough that I can use it for my work, and I have a few of my peers at work using it as well. I’m eager to get more feedback and even contributions from the broader community.

Feel free to get in touch with feedback on the projects issue tracker or reach out to me on Twitter!

PR Dash: A command line pull request dashboard was originally published by Brian Gianforcaro at bjg.io on July 03, 2020.

All Systems Go - 2019

2019-10-07T00:00:00+00:00

Last month my colleagues Argenis Fernandez, Eugene Birukov, and I attended the All Systems Go! - 2019 conference held from September 20-22 in Berlin, Germany. All Systems Go! is a conference dedicated to Linux user space technologies. The audience appears to be a mix of systems developers working on user space and kernel adjacent projects from Facebook, RedHat, and various startups, to the consumers of these systems in industry, and the community at large. The conference was dual track, with each talk clocking in at around 35 minutes on Friday and Saturday with a few lighting talks sprinkled throughout. Sunday was a dedicated “Hackathon” for attendees to collaborate on whatever projects they want.

Microsoft was one of the conference sponsors, my team members and I attended to give a talk about the unique technologies we used to ship SQL Server on Linux, as well as evangelize Azure SQL Database Edge, running on ARM64.

Links:

Conference website: link
Conference schedule: link
Recordings of all the conference talks: link

Our Talk:

Our talk seemed to be well received, although attendance did suffer with our time slot being post lunch break and many people were still in line for the food trucks. The audience had good questions during the QA, as well as after the talk.

The slides are available on speaker deck:

The recording is available on YouTube:

All Systems Go - 2019 was originally published by Brian Gianforcaro at bjg.io on October 07, 2019.

Mystery: HORIPAD switch controller not working with Steam on Linux?

2019-01-13T00:00:00+00:00

I just picked up a HORIPAD for the Switch. Hoping I could use it with Steam on my Linux box in addition to the switch, I plugged it into my system running the 5.0-rc1 kernel. It was a bit to my surprise, but it showed up as a recognized device:

$ dmesg
... snip ...
[    2.665567] input: HORI CO.,LTD. HORIPAD S as /devices/pci0000:00/0000:00:14.0/usb1/1-1/1-1:1.0/0003:0F0D:00C1.0001/input/input8
[    2.665671] hid-generic 0003:0F0D:00C1.0001: input,hidraw0: USB HID v1.11 Gamepad [HORI CO.,LTD. HORIPAD S] on usb-0000:00:14.0-1/input0

We can find the same information in lsusb:

$ lsusb -v 

Bus 001 Device 002: ID 0f0d:00c1 Hori Co., Ltd 
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0        64
  idVendor           0x0f0d Hori Co., Ltd
  idProduct          0x00c1 
  bcdDevice            5.72
  iManufacturer           1 
  iProduct                2 
  iSerial                 0 
  bNumConfigurations      1

  ... snip ...

So we know it’s recognized, lets see if jstest can read input coming from the device:

$ jstest --normal /dev/input/js0

Driver version is 2.1.0.
Joystick (HORI CO.,LTD. HORIPAD S) has 6 axes (X, Y, Z, Rz, Hat0X, Hat0Y)
and 14 buttons (BtnA, BtnB, BtnC, BtnX, BtnY, BtnZ, BtnTL, BtnTR, BtnTL2, BtnTR2, BtnSelect, BtnStart, BtnMode, BtnThumbL).
Testing ... (interrupt to exit)
Axes:  0:     0  1:     0  2:     0  3:     0  4:     0  5:     0 Buttons:  0:off  1:off  2:off  3:off  4:off  5:off  6:off  7:off  8:off  9:off 10:off 11:off 12:off 13:off 
Axes:  0:     0  1:     0  2:     0  3:     0  4:     0  5:     0 Buttons:  0:off  1:on   2:off  3:off  4:off  5:off  6:off  7:off  8:off  9:off 10:off 11:off 12:off 13:off 
Axes:  0:     0  1:     0  2:     0  3:     0  4:     0  5:     0 Buttons:  0:off  1:on   2:off  3:off  4:off  5:off  6:off  7:off  8:off  9:off 10:off 11:off 12:off 13:off 
Axes:  0:     0  1:     0  2:     0  3:     0  4:     0  5:     0 Buttons:  0:off  1:off  2:on   3:off  4:off  5:off  6:off  7:off  8:off  9:off 10:off 11:off 12:off 13:off
Axes:  0:     0  1:     0  2:     0  3:     0  4:     0  5:     0 Buttons:  0:off  1:off  2:on   3:off  4:off  5:off  6:off  7:off  8:off  9:off 10:off 11:off 12:off 13:off
Axes:  0:  9458  1: -2703  2:     0  3:     0  4:     0  5:     0 Buttons:  0:off  1:off  2:off  3:off  4:off  5:off  6:off  7:off  8:off  9:off 10:off 11:off 12:off 13:off 
Axes:  0:-32767  1:     0  2:     0  3:     0  4:     0  5:     0 Buttons:  0:off  1:off  2:off  3:off  4:off  5:off  6:off  7:off  8:off  9:off 10:off 11:off 12:off 13:off
Axes:  0: -1014  1: 32767  2:-18580  3: 32767  4:     0  5:     0 Buttons:  0:off  1:off  2:off  3:off  4:off  5:off  6:off  7:off  8:off  9:off 10:off 11:off 12:off 13:off
Axes:  0: -1690  1: 32767  2:-18580  3: 32767  4:     0  5:     0 Buttons:  0:off  1:off  2:off  3:off  4:off  5:off  6:off  7:off  8:off  9:off 10:off 11:off 12:off 13:off
Axes:  0: -1690  1: 32767  2:-18580  3: 32767  4:     0  5:     0 Buttons:  0:off  1:off  2:off  3:off  4:off  5:off  6:off  7:off  8:off  9:off 10:off 11:off 12:off 13:off
Axes:  0: -2027  1: 32767  2:-18580  3: 32767  4:     0  5:     0 Buttons:  0:off  1:off  2:off  3:off  4:off  5:off  6:off  7:off  8:off  9:off 10:off 11:off 12:off 13:off
Axes:  0:     0  1:     0  2:     0  3:     0  4:     0  5:     0 Buttons:  0:off  1:off  2:off  3:off  4:off  5:off  6:off  7:off  8:off  9:off 10:off 11:off 12:off 13:off

All the buttons, sticks, and dpad seem to work and jstest is able to read signal from all inputs! I opened up steam, thinking I was going to play Celeste. However steam reported that it didn’t recognize any controllers… Going into Settings -> Controllers -> General Controller Settings, I had the “Generic Gamepad Configuration Support” option checked, as this wasn’t one of the known Xbox/PS/Switch controllers. Steam wasn’t able to see the device for some reason.

After digging around online for a bit, I discovered the steam-devices package, which was recommended by folks debugging other controllers. I installed that package, and nothing changed. :(

Wondering what exactly this package does, I started to dig into what the package provided.

$ dpkg -L steam-devices

/.
/lib
/lib/udev
/lib/udev/rules.d
/lib/udev/rules.d/60-HTC-Vive-perms.rules
/lib/udev/rules.d/99-steam-controller-perms.rules
/usr
/usr/share
/usr/share/doc
/usr/share/doc/steam-devices
/usr/share/doc/steam-devices/changelog.Debian.gz
/usr/share/doc/steam-devices/copyright

That 99-steam-controller-perm.rules file stood out as interesting.

$ head /lib/udev/rules.d/99-steam-controller-perms.rules

# Valve USB devices
SUBSYSTEM=="usb", ATTRS{idVendor}=="28de", MODE="0666"
# Steam Controller udev write access
KERNEL=="uinput", SUBSYSTEM=="misc", TAG+="uaccess"

# Valve HID devices over USB hidraw
KERNEL=="hidraw*", ATTRS{idVendor}=="28de", MODE="0666"

# Valve HID devices over bluetooth hidraw
KERNEL=="hidraw*", KERNELS=="*28DE:*", MODE="0666"

It looks like a bunch of udev rules for configuring the permission for hidraw and uinput devices. We saw in the dmesg output above that our controller is a hidraw device, maybe we need to setup permissions for the device to be read by non elevated users?

To test this out I appended the following to the 99-steam-controller-perm.rules file. I found the vendor id and product id for the HORIPAD using lsusb -v (as seen above).

# HORIPAD S over USB hidraw
KERNEL=="hidraw*", ATTRS{idVendor}=="0f0d", ATTRS{idProduct}=="00c1", MODE="0666"

Not wanting to reboot I quickly dug up how to force reload udev rules:

$ sudo udevadm control --reload-rules 
$ sudo udevadm trigger

A few seconds later, steam popped up a window informing me that it had recognized a new controller!

I started up Celeste and everything worked like a charm. Off to play the game!

Mystery: HORIPAD switch controller not working with Steam on Linux? was originally published by Brian Gianforcaro at bjg.io on January 13, 2019.

Link: C++ Interlude 2015

2016-01-02T00:00:00+00:00

Agustin Bergé wrote up a great summary of the changes 2015 brought to the C++ language, and what might be coming in C++17. The post is available here, it’s well worth the read.

Just A Quick Aside:

Looking at the list of papers from the C++ Standards Committee in 2015, one of the recent entries P0155R0 - “Handling Disappointment in C++”, peeked my interest a bit. During the past few years I’ve always been very interested in error handling techniques besides the standard return status code, or exception based solutions. In Handling Disappointment in C++ Lawrence Crowl compares two proposals N4109 - Addition of the Expected Monad and N4233 - A Class for Status and Optional Value.

I need to take some time and play with the Expected<T> style error handling and see how well it would fair in production code. Maybe in a future post?

Link: C++ Interlude 2015 was originally published by Brian Gianforcaro at bjg.io on January 02, 2016.

Mystery: Why is MSYS2 executing so slowly?

2015-03-16T00:00:00+00:00

I’ve always enjoyed using *nix tools, even when working on a Windows machine. In the past I had always used Cygwin, but recently I have been playing around with the MSYS2 project, which is a down stream fork of the Cygwin project. It has a few interesting features, one of the most impressive being a real package manager.

Recently I got a new development machine, so I decided to try a pure MSYS2 setup to see how well it would work. After installing MSYS2 I noticed two particularly strange behaviors:

The first issue was that it took an excessive amount of time to start a MSYS2 shell, on the order of 30 seconds. However once the shell was started everything was very snappy.
I often put the MSYS2/Cygwin tools in my normal windows console path. They are normally quite useful during my normal development. When executing any of the MSYS2 tools which perform file access (like ls, grep, etc) there was a significant delay during execution. If you let the program continue to execute it would eventually finish, but it was obviously an unproductive work flow…

Initially I didn’t connect the two issues together, as I was seeing them in two entirely separate usage patterns. After digging around in the documentation for a while and coming up with nothing, I finally thought hey lets just use strace! Strace immediately revealed the issue, there were thousands of references to a single function when ever a program which dealt with file access was executing.

$ strace ls
  ... snip ...
  497 5276872 [main] ls 9004 pwdgrp::fetch_account_from_windows: line: <xxxxxxxxxxxxxx1:>
  492 5277364 [main] ls 9004 pwdgrp::fetch_account_from_windows: line: <xxxxxxxxxxxxxx2:>
  520 5277884 [main] ls 9004 pwdgrp::fetch_account_from_windows: line: <xxxxxxxxxxxxxx3:>
  521 5278405 [main] ls 9004 pwdgrp::fetch_account_from_windows: line: <xxxxxxxxxxxxxx4:>
  495 5278900 [main] ls 9004 pwdgrp::fetch_account_from_windows: line: <xxxxxxxxxxxxxx5:>
  496 5279396 [main] ls 9004 pwdgrp::fetch_account_from_windows: line: <xxxxxxxxxxxxxx6:>
  493 5279889 [main] ls 9004 pwdgrp::fetch_account_from_windows: line: <xxxxxxxxxxxxxx7:>
  493 5280382 [main] ls 9004 pwdgrp::fetch_account_from_windows: line: <xxxxxxxxxxxxxx8:>
  497 5280879 [main] ls 9004 pwdgrp::fetch_account_from_windows: line: <xxxxxxxxxxxxxx9:>
  ... snip ...

Catching A Lead

Armed with the actual function call which was wreaking havoc, I hit up Google. I eventually ran across this news group post, where Michael Klemm seemed to be having similar problems as I was. Reading through the mail, Andrey Repin points to a FAQ entry which describes in detail the exact issue I was hitting:

Starting a new terminal window is slow. What’s going on?

There are many possible causes for this.

If your terminal windows suddenly began starting slowly after a Cygwin upgrade, it may indicate issues in the authentication setup.

For almost all its lifetime, Cygwin has used Unix-like /etc/passwd and /etc/group files to mirror the contents of the Windows SAM and AD databases. Although these files can still be used, since Cygwin 1.7.34, new installations now use the SAM/AD databases directly.

To switch to the new method, move these two files out of the way and restart the Cygwin terminal. That runs Cygwin in its new default mode. If you are on a system that isn’t using AD domain logins, this makes Cygwin use the native Windows SAM database directly, which may be faster than the old method involving /etc/passwd and such. At worst, it will only be a bit slower. (The speed difference you see depends on which benchmark you run.) For the AD case, it can be slower than the old method, since it is trading a local file read for a network request. Version 1.7.35 will reduce the number of AD server requests the DLL makes relative to 1.7.34, with the consequence that you will now have to alter /etc/nsswitch.conf in order to change your Cygwin home directory, instead of being able to change it from the AD configuration.

This explains the issue I was experiencing, my machine was joined to a very large AD deployment. The Cygwin/MSYS2 runtime was attempting to enumerate the thousands of users and groups as part of the process of loading the shell process initially, resulting in slow terminal start up.

The Cygwin DLL queries information about every group you’re in to populate the local cache on startup.

The fact that this is an in memory cache, populated on the load of runtime dll explains the second symptom. When not running under the Cygwin/MSYS2 shell, the runtime dll isn’t going to be already loaded and used by the utilities so every invocation of tools like ls or grep results in a separate load of the runtime and as a result an enumeration of the AD groups.

Solution

If you continue reading the FAQ, the steps necessary to cache your users and group memberships is listed. However given my previously discussed situation of being part of a very large AD deployment I only wanted to include my current user info as well as my current users groups to avoid a huge cache (see a related mail thread). To achieve this I did the following:

$ mkpasswd -l -c > /etc/passwd
$ mkgroup -l -c > /etc/group

Once we have the local cache populated, we can configure Cygwin/MSYS2 to read from the files, instead of querying AD. This is achieved by editing /etc/nsswitch.conf and modifying passwd and group sections to read from “files” instead of “files db”, on lines 2 and 3 below:

1
2
3
4
5
6
7
8
# Begin /etc/nsswitch.conf
passwd: files
group: files
db_enum: cache builtin
db_home: cygwin desc
db_shell: cygwin desc
db_gecos: cygwin desc
# End /etc/nsswitch.conf

Once you restart you Cygwin/MSYS2 console, the change should take effect immediately.

Mystery: Why is MSYS2 executing so slowly? was originally published by Brian Gianforcaro at bjg.io on March 16, 2015.

Presentation: Papers We Love - Managing Update Conflicts in Bayou

2015-03-15T00:00:00+00:00

Description From the video:

Peter Bailis presents the “Managing Update Conflicts in Bayou, A Weakly Connected Replicated Storage System” paper by Doug Terry, Marvin Theimer, Karin Petersen, Alan J. Demers, Mike J. Spreitzer, and Carl H. Hauser in SOSP 1995.

Peter does wonderful job summarizing the paper, and discussing the design implications of the project in relation to modern day systems. You can find the original paper here.

Presentation: Papers We Love - Managing Update Conflicts in Bayou was originally published by Brian Gianforcaro at bjg.io on March 15, 2015.

Learning Elixir

2014-11-30T00:00:00+00:00

Intro

Elixir has been on my radar for about a year now, however despite my best effort I haven’t had a chance to take a thorough look yet. I decided I’ve been waiting long enough, I just need to sit down and try to hack out some code. If you haven’t looked into the language yet it appears very cool on paper:

Elixir is a functional, concurrent, general-purpose programming language built atop the Erlang Virtual Machine (BEAM). Elixir builds on top of Erlang to provide distributed, fault-tolerant, soft real-time, non-stop applications but also extends it to support metaprogramming with macros and polymorphism via protocols.

Installation was super easy, just head over to the install page on http://elixir-lang.org. The windows web installer will download and install everything you need, a totally seamless experience.

I have a little bit of familiarity with Erlang, but besides that this is the first time I’ve actually looked into the Elixir syntax. So the rest of this post will be entirely, unfiltered stream of consciousness.

Trying Out Some Code

Lets see if we can use Elixir to solve the first problem on the Project Euler site.

Multiples of 3 and 5: If we list all the natural numbers below 10 that are multiples of 3 or 5, we get 3, 5, 6 and 9. The sum of these multiples is 23. Find the sum of all the multiples of 3 or 5 below 1000.

First lets try to fire up the REPL, I think I remember front page mentioning it was called iex.

$ iex
Interactive Elixir (1.0.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)>

Great, so lets try to start working on the problem. Seems like we can naively solve this by traversing a list, lets see if we can figure out the syntax for that. Maybe it’s like Haskell? [1..10.]

iex(1)> [1...1000]
** (SyntaxError) iex:2: syntax error before: '...'

Hrmm… that didn’t work. Maybe lets try dropping the brackets? I think i vaguely remember Erlang only using two dots in the notation?

iex(1)> 1 .. 1000
1..1000

That looks like it could be a representation of a list! I wonder how we could take the head it?

Lets try just a function?

iex(2)> head(1 .. 1000)
** (RuntimeError) undefined function: head/1

Maybe it’s prolog/erlang style, using the symbolic syntax?

iex(3)> x = 1 .. 1000
1..1000
iex(4)> x
1..1000
iex(5)> [y|z] = x
** (MatchError) no match of right hand side value: 1..1000

Looks like that’s not right… something seems off I thought that would work… It seems like the language is probably erlang with some syntax. I vaguely remember the head function in erlang being hd/1, let’s try that on our list.

iex(5)> hd
** (RuntimeError) undefined function: hd/0

iex(5)> hd(x)
** (ArgumentError) argument error
    :erlang.hd(1..1000)
iex(5)> [x]
[1..1000]

iex(6)> head([x])
** (RuntimeError) undefined function: head/1

iex(6)> hd([x])
1..1000

iex(7)> hd([1 .. 2])
1..2

Hrm… it seems like the 1..1000 syntax doesn’t create a list, maybe it’s a symbol? Time to go read the docs …

So after reading the docs for lists it seems like, the bracket syntax is correct. hd/1 is head as we thought and tl/1b is tail. However they don’t seem to cover the range syntax on that page. Need to dig in to the docs a little bit more I guess, maybe there is no range syntax?

Ahha! It looks like on this page that we were actually kind of correct. 1..1000 seems to represent a lazy range, not a list. So we need to use functions to operate on top of the lazy range.

Looks like there is a Stream package for lazy sequences, which has map/filter:

iex(8)> 1..1000 |> Stream.filter
** (UndefinedFunctionError) undefined function: Stream.filter/1
    (elixir) Stream.filter(1..1000)

Need to figure out how to write a lambda, lets try to filter out odd numbers.. one of the map examples in the docs page uses this syntax:

    fn x -> x*2 end

Seems to make sense, lets try it out.

iex(8)> 1..1000 |> Stream.filter(fn x -> rem(x,2 == 0) end)
** (ArithmeticError) bad argument in arithmetic expression
             :erlang.rem(1, false)
    (elixir) lib/enum.ex:666: anonymous fn/3 in Enum.filter/2
    (elixir) lib/range.ex:77: Enumerable.Range.reduce/6
    (elixir) lib/enum.ex:666: Enum.filter/2

I would have thought that would have worked… wait the parenthesis don’t look right. I forgot to properly close the rem(..)!

iex(8)> 1..1000 |> Stream.filter(fn x -> rem(x,2) == 0 end)
[2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42,
 44, 46, 48, 50, 52, 54, 56, 58, 60, 62, 64, 66, 68, 70, 72, 74, 76, 78, 80, 82,
 84, 86, 88, 90, 92, 94, 96, 98, 100, ...]

cool.

I think I saw a Enum.sum function as well, putting all the pieces of the question together we get a nice one liner.

iex(12)> multiples = fn x -> rem(x,3) == 0 || rem(x,5) == 0 end
#Function<6.90072148/1 in :erl_eval.expr/5>
iex(13)> 1..999 |> Stream.filter(multiples) |> Enum.sum
<redacted-from-article>

That wasn’t so bad, I like the pipelining syntax. It seems similar to the syntax used in ML based language, like F#/OCaml.

Resources

If you are on board with Elixir and want to continue on your own Elixir journey there are a couple of really great resources to continue exploring the language.

How I Start: Elixir
Elixir-Lang Getting Started Tutorial
The 30 Days of Elixir Repository
Learning X in Y Minutes: Where X = Elixir
The Elixir Cheat Sheet

Learning Elixir was originally published by Brian Gianforcaro at bjg.io on November 30, 2014.

Article: The Network is Reliable

2014-11-04T00:00:00+00:00

One of my favorite excerpts from the article:

DRBD Split-brain

When a two-node cluster partitions, there are no cases in which a node can reliably declare itself to be the primary. When this happens to a DRBD file system, as one user reported (http://serverfault.com/questions/485545/dual-primary-ocfs2-drbd-encountered-split-brain-is-recovery-always-going-to-be), both nodes can remain online and accept writes, leading to divergent file system-level changes.

In this article, Peter Bailis and Kyle Kingsbury provide an awesome sampling of real life network failures in a variety of actively deployed distributed systems.

It’s definitely worth the read, especially interesting given the data from Google, Amazon, and Microsoft. You can view the living version of the document on github: https://github.com/aphyr/partitions-post.

Article: The Network is Reliable was originally published by Brian Gianforcaro at bjg.io on November 04, 2014.

Embracing The Mirage [Part One]

2014-10-20T00:00:00-00:00

As of late I have become increasingly enamored with this great research project Mirage OS. The team is building a library operating system, they use the term unikernel. In essence this means that you write the code for your system, and at compilation time the system builds you an image which you can then deploy to a hypervisor running on bare metal. This image is then run virtualized, entirely on its own. No operating system is needed to host your application, resulting the ability to better utilize cloud resources (less overhead) and much better isolation guarantees due to VM sandboxing which is inherit in virtualized environments.

Besides the promises of isolation and efficiency, I’m also interested in the project because of the team’s choice of language for implementing their library OS. They have chosen to use the OCaml language, for the uninitiated OCaml is an awesome language. It has inspired tons of other languages that you might use everyday. OCaml has an great type system, which allows for quite in depth static analysis to be performed during compilation. It has full support for functional, imperative, and object oriented programming styles.

To me the project appears to be a recipe for success, and I want to help Mirage OS become even more successful. The project currently has the ability to compile down to both Xen images, as well as unix user space process. In the coming months, I will be working on adding user space windows process support, as well as a future goal of supporting Hyper-V images.

I’ve expressed my interest to the Mirage OS developers mailing list, and they seem to agree this would be valuable work. This is super exciting to me, and although there are many challenges ahead I’m sure I will learn a ton.

I’ll be documenting the journey here.

Embracing The Mirage [Part One] was originally published by Brian Gianforcaro at bjg.io on September 22, 2014.

How To Kill An Azure Linux VM in One Second

2014-09-20T00:00:00-00:00

It’s super easy, just try to install the Xen HyperVisor on the Ubuntu 14.0 Image.

> sudo apt-get install xen-system-amd64
> reboot

The machine will never come up, as it will be hung on boot. This appears to stem from the fact that Hyper-V does not support nesting Hypervisors.

How To Kill An Azure Linux VM in One Second was originally published by Brian Gianforcaro at bjg.io on September 20, 2014.