Sebastian Spier

InnerSource is like Sourdough

2022-11-17T00:00:00+00:00

At a talk at the InnerSource Summit 2022, I shared some thoughts about why “InnerSource is like Sourdough”.

While this might not be scientifically correct, the metaphor has helped me as a mental picture to explain the concept of InnerSource to myself and to others.

Summary

When baking bread with sourdough you are dealing with the sometimes surprizing behavior of wild yeasts, creating a bubbly dough that eventually turns into tasty bread.

Learning how to handle sourdough takes time, practices, and the willingness to experiment and fail. However once you master it, you are rewarded with more healthy bread, richer in flavor and with a longer shelf life!

InnerSource has many similarities to sourdough in this regards.

This talk highlights those similarities in a (hopefully) thought-provoking and fun way. And to make you hungry there will be a lot of bread pictures ;)

Video

Knowledge Sharing as a Catalyst for Professional Growth

2022-08-30T00:00:00+00:00

We had another Devopsicon — our internal engineering (un)conference. We record most of these company-internal sessions to ensure we can share the knowledge with those who were unable to attend, and to build a knowledge base over time.

We have decided to go one step further and share some of these sessions publicly. This post describes how this approach has benefitted both us as a company and the professional growth of our engineers.

Creating a Knowledge Base

The last unconference produced a lot of video material! About 130 sessions were recorded and shared throughout the organization. Many sessions happen in parallel, so recording the sessions helps the audience to overcome their FOMO, at least a little bit. :)

Since we ran recent Devopsicon events fully remotely, we got a lot better at recording, cataloging, and sharing those recordings after the event. This has created a great knowledge base of 500+ videos about all sorts of topics:

A live hacking JavaScript session
Understanding brain chemicals as a way to strengthen teams
Using terraform with GitHub
Backstage as a tool for improving Developer Productivity

We figured that some of these sessions would also be interesting for people outside of Meltwater Engineering. These sessions will give you an idea of who we are, how we work, and what we are working on.

What’s in it for Meltwater?

Let’s come clean: We are always looking to hire new, great colleagues for our Engineering teams. To allow people outside of Meltwater to understand our work culture and showcase that we are an attractive employer in Tech, we need to provide sufficient opportunities for them to get to know Meltwater Engineering.

But how do we create relevant material?

We found that writing a blog post has a high barrier to entry for many of our colleagues. However, at our internal unconference many of those people deliver short internal presentations. Apparently that is easier than writing! So we decided to turn some of that existing video content into material that we can share publicly.

In addition, we want our engineers to grow! These presentations are a way for us to support their personal and professional growth.

Reasons for Meltwater Engineers to Present in Public

We selected some of the best presentations from our unconference and asked the presenters to re-record them. Ufff — extra work for them! Naturally they asked, “What’s in it for me?”

There is a lot to learn from presenting something publicly:

Showcase your work — In a B2B product like Meltwater’s, it can be difficult to explain or show what you do. Creating a video about something that you are working on provides a glimpse into it.
Personal development — Learning how to articulate complex topics, while making it clear yet simultaneously interesting, is a difficult task. Therefore, presenting a topic in public greatly contributes to the personal development of the presenter. As a bonus, we help our engineers by providing constructive feedback on slides and demos. We also handle most of the editing, resulting in professional-looking video with relatively little effort.
Professional branding — Through these videos, our engineers present themselves as knowledgeable professionals or even experts in a certain field. This aids them in building their professional brand, which among other things might help them land speaking opportunities at conferences.

Where Do I Find the Videos?

We are starting with a session about “KEDA in Terraform - The good, the bad, the savings”, by the fantastic Patrick Nanys from our Budapest office. Thanks for going first Patrick!

Centralizing Developer Docs in Backstage

2022-07-19T00:00:00+00:00

A growing Engineering org may reach a point where answering questions that used to be simple have become hard. At least that is true for us! Questions such as: “Who owns this?” “Do we have a component that I can reuse?” “Where are the docs for that API?”

This post shares the challenges we want to solve, and the experiences we made while centralizing our internal developer docs in Backstage, an open platform for building developer portals.

Challenges of Autonomy and Growth

In 2017 the Meltwater Engineering organization started a transformation process towards autonomous teams with full end-to-end responsibility. We shared a lot more about this transformation in various podcasts.

During the same period we also grew significantly in size, now with 500+ people in our org, spread over 10+ sites.

The transformation was a great success! We significantly improved delivery speed, quality, and developer happiness.

However any significant change has side-effects. In our case the increased autonomy and growth of the org led to a fragmented ecosystem of developer tooling. It became a lot harder to understand simple things like: “Who owns this?” “Do we have a component that I can reuse?” “Where are the docs for that API?” “What is the recommended way to get started with doing X?” etc

Often the only way to figure it out was to ask another colleague with more tenure in the org, an approach that Gary Niemen from Spotify so aptly called rumour-driven development.

What Do We Want Instead?

While we want to maintain the benefits of team autonomy, we can improve in areas where the standardization of tooling doesn’t reduce the ability of our teams to move fast and innovate on their core mission.

We can boil down the goal for our Product & Engineering org to:

Faster onboarding of new hires and acquisitions.

This is the experience (and metric) that we will push to improve, while periodically reevaluating the needs of our org and business.

First Experiments with Backstage

When we found Backstage (backstage.io) we were excited. Spotify’s description of the problem space resonated so well with us that we wanted to try it out.

As we did not immediately have the capacity in our Platform teams to give Backstage a spin, a small group of us started to experiment with Backstage at an internal hackathon.

During that hackathon we explored questions like:

Are the TechDocs a solution to centralize our developer documentation?
Does the Software Catalog clarify the question of component ownership and does it help to explain the interdependencies between components?
How can we make it simple for developers to use and contribute to our Backstage installation?
Can we use the tech-radar plugin to visualize the tech used within Meltwater? What other Backstage plugins could help our engineers?

By the end of the hackathon we had a working Backstage instance, with some components imported into the Software Catalog already.

We were especially impressed with TechDocs and its potential to help us build out our centralized developer documentation. Therefore, we decided to continue our experiment after the hackathon and focus on TechDocs first.

Milestone 1: Starting with TechDocs Only!

For about 2 years we maintained a readme.com portal, where we hosted cross-team developer documentation. We learned a lot: How to process suggestions for documentation improvements coming from other teams, or how to indicate effectively which team owns which component, especially for pieces that are shared between multiple teams.

However, we were never able to get sufficient teams to contribute to the portal, nor were we happy with the documentation living outside of our source code repos. When we saw the docs-like-code approach that Backstage offers through TechDocs, we were easily sold (also see Docs as Code).

We ran multiple experiments to validate whether we could migrate all docs from readme.com to TechDocs. One of the teams even wrote a little converter script to ease the migration.

From a pure documentation standpoint, we were quickly convinced that TechDocs could match the readme.com experience. At the same time, TechDocs gave us more extensibility and the ability to tightly integrate documentation improvements into our code review processes, which provided a lot of value for the creators of the given documentation.

We went from experiment to adoption by migrating all remaining docs from readme.com to TechDocs, and declaring Backstage our go-to place for all company-internal cross-team developer documentation at Meltwater.

The Future of Backstage at Meltwater Engineering

We keep investing in Backstage, as we like the outcomes of our first experiments. But do we truly know what the future Backstage at Meltwater Engineering will be? Not yet!

However, we do know what we learned, what we liked, what we didn’t like, and what questions we have. Together with the feedback from our teams this will inform which area we invest in next.

Here a few highlights of what we have observed after using Backstage for a few months. Ask us anything about these! We may write more in-depth blog posts about some of them, should we decide to go down those routes further.

Curation > Technology

As so often, technology alone won’t solve things.

If the information provided in Backstage increases quickly and without structure, it might reach a point where it would not be useful for our developers anymore, as they won’t find essential info quickly. Preventing that will require some level of curation of the content/components that we present in Backstage.

One particular problem: Can our teams use Backstage for team-internal documentation (runbooks, etc.), without creating confusion for developers from other teams?

For example: A Meltwater developer is looking for information on how to deploy to our Kubernetes cluster. Instead of finding the relevant info from our Platform teams, they might find an internal Kubernetes runbook from some other team. That would be confusing, rather than helpful.

docs-like-code is the way to go

With the docs living in the same repo as the source code, we can tightly integrate documentation improvements into our code review processes. The teams that are practicing this docs-like-code approach are confident that it will increase the quality of our documentation and the onboarding speed of new developers.

Gotchas: TechDocs offers formatting features on top of GitHub Flavored Markdown (GFM), which many of our engineers are used to. Also there are some nuances where it actually parses the input differently. - We have a dedicated internal documentation section in our Backstage instance, where we explain how TechDocs works in detail. Hopefully this will make the migration from GFM to TechDocs less painful for our teams.
Question: How to structure documentation into larger categories when the docs for individual components always live in the respective repos? How to highlight which components belong together or depend on each other? - We have experimented with adding docs for systems/domains in Backstage but we aren’t fully convinced whether this is the best way forward.
Concern: With TechDocs, any docs change requires a PR in GitHub. Does that make the barrier to entry so high that functions like Product or Support will not contribute improvements and fixes to the documentation?

Discovery - Finding Key Documentation quickly

The default search provided by Backstage is not great. Sometimes we don’t see results that we would be expecting (a bug?), and the excerpts as result previews aren’t exactly pretty. Most importantly we don’t know how to influence the result ranking and we want to give certain docs precedence over others.

Meltwater Engineering having a lot of search knowledge should give us an edge here.

Can Templates support Paved Roads?

We have have only run tiny experiments with the Templates mechanism in Backstage so far. However we do like its potential to create easier getting-started experiences for common tasks while at the same time promoting certain standards within our Engineering org.

We call these paved roads (aka golden paths), and are looking forward to learning and sharing more about how this concept is working out for us in the future.

Keeping Backstage Up-to-date

The challenge of keeping Backstage up-to-date, is that it is meant as a template for a developer portal, supported by a platform of standardized plugins. Although it is fairly easy to bump the dependencies for the plugins, one also has to update the template itself, which becomes more and more difficult as changes are made locally.

While the official documentation recommends a way to solve this, we still found this to become messy quickly and are looking for a better approach so that we don’t fall behind on Backstage upgrades just because the upgrade is cumbersome.

In Closing

This is the beginning of our journey with Backstage. We hope that this post has been useful for you if your org is at a similar stage.

Are you also introducing Backstage? Which challenges of migration and adoption did you have to overcome in your orgs? Which other gotchas/questions/challenges with the technology did you have to solve? We would love to hear more about any and all of such first-hand experiences.

Aside: This project with Backstage, as well as this post, were done in collaboration with my fantastic Meltwater colleague Petter Remen.

Image credits:

Contribute to Grow: Joining the Board of the InnerSource Commons

2022-06-26T00:00:00+00:00

In 2022 I am serving on the Board of the InnerSource Commons Foundation. Thank you to the Members of the Foundation who voted for me! I appreciate your trust and will do my best to move this community forward.

Back in 2019 I could not have imagined contributing to an open community in any significant way. I would not even have known how to get started.

I want to help you to join an open (source) community yourself by sharing my 3 year journey in the InnerSource Commons.

You wonder: Why should I donate my time to such things? Bertrand Delacretaz answers this best in How to convince your left brain (or manager) to follow the Open Source path that your right brain desires..

Joining an Open (Source) Community

The following will be pretty specific, to let you compare your experience with mine, and evaluate whether my story is similar enough to apply to you.

My Context

I studied computer science, and had been working as an Engineering Manager for 10 years. Over time I learned various techniques to help my teams. Lean and Agile methods, how (Engineering) orgs change as they grow, and how autonomy and purpose motivate people and change the teams that they work in.

Around 2019 I had the sense that I was maxing out on my ability to make my teams better, without also changing the surrounding org. We had reduced much of the wasted cycles in my teams but were struggling to do the same in the interactions of my teams with other teams in the org.

While researching cross-team collaboration challenges in orgs that have autonomous teams, I came across the term InnerSource. The challenges that InnerSource is trying to solve were so similar to what I had experienced! I was excited. The hope was that I had found something that could help my teams and myself to increase our impact on our business and organization.

User/Reader: Joining & Lurking

Joining the InnerSource Commons community was a lucky coincidence … let’s call it fate! ;)

While reading up on the concepts on innersourcecommons.org I found a typo that I fixed. My tiny contribution was accepted, and I started talking with a community members, who invited me to join their Slack to continue the conversation.

I joined Slack, but didn’t quite know what to do there. I just lurked, and listened to the conversations for some time. Eventually I joined in on some conversations and asked questions. Still I was largely a passive member of the community.

Contributor: Learning by Doing

One of the ways that the community is sharing experiences with each other are the InnerSource Patterns. I enjoyed the practical nature of these patterns, and when I found some of the pattern authors in Slack, I was hooked.

The patterns, and the community, allowed gave me access to practitioners with similar challenges across company boundaries. Chatting under Chatham House Rule allowed us to share a great level of detail with each other, creating learning opportunities based on real-life experiences that otherwise are hard to find.

The patterns quickly became my primary vehicle to learn more about InnerSource. While reading the patterns, I started to improving the writing in some of them. That way I was getting to know other community members that were interested in similar topics, which lead to great conversations.

The idea of learning through contributing fully clicked for me, at FOSS Backstage in early 2020. Cornelius Schumacher presented how they were bringing an open-source-like culture into the corporate world. Their InnerSource License simplified the source code distribution and collaboration on a code level between the companies within their group.

After the conference I met Cornelius in the InnerSource Commons community. I started a conversation with him to learn more about their InnerSource License, which led to us writing up their approach as the InnerSource License pattern.

This was the final confirmation that I needed:

By contributing to the InnerSource Patterns, I was having conversations and learning things at a faster pace than what would have happened had I staid just within the confines of my own organization. The more I was contributing, the more I was learning. An addictive drug! :)

Motivated by what I was learning from my first contributions, I kept contributing and sharing. Mostly I was doing little things like cleanup of the Patterns repository, consolidating issues, pull request, and so forth.

In late 2020, a year after joining the community, I submitted a talk for the Fall Summit. While I was second-guessing whether I had enough relevant content to share, I presented my talk 1 year in the ISC: Getting your money’s worth. And I am happy I did!

That talk was an initial version of this blog post here. While it felt awkward to put myself out there, it helped me to introspect on how I was benefiting from doing open source work in the InnerSoure Commons. It also created new opportunities to talk to other community members, which got me even more engaged and excited about the community itself.

The InnerSource Patterns still were my primary way of learning about InnerSource. Many times I was able to use those learnings at work. Through applying those patterns, I was learning more again, which I could bring back to the InnerSource Commons community. A cycle!

A proud moment was to publish the first version of the Patterns online book to patterns.innersourcecommons.org, something that the Patterns working group had been working towards for a while.

Shortly after I was invited to the Patterns Working Group as a Trusted Committer. A confirmation that my contributions were seen and appreciated.

Member: Promoting InnerSource

Communities have a small amount (~1%) of people that contribute a lot. This is also true for the InnerSource Commons and the Patterns Working Group. But how to change that?

I had learned a lot faster by actively contributing to the InnerSource Commons. Therefore I tried to share that approach in a presentation called Level up your InnerSource through Patterns.

Together with Fei Wan from Comcast we presented this topic at the ISC APAC Summit. Our talk was streamed at offices of Tecent, where 1000+ developers were exposed to the ideas of InnerSource, many of them for the first time.

I also joined the Marketing & Outreach Working Group, to see if we could find ways to reach a larger audience. We redesigned the website, organized Community Calls, and so forth. Practicing these marketing activities at the Commons gave me useful material to promote InnerSource at work as well.

As a result of my larger involvement in the InnerSource Commons, I got invited to join as a Member in 2021.

Joining the Board: Let’s see :)

Well, I just joined the Board, so I cannot speak with much experience yet :)

However I am certain that I will learn a lot, and that serving on the Board will help me understand InnerSource and the software industry at large better.

In Summary

I went from lurker, to contributor, to trusted committer, to Member, and finally to the Board of the InnerSource Commons Foundation.

Every step of the way I was contributing more, and through that also learning more!

I hope that some of you will find this useful to find your own path in an open (source) community. There is a lot to be gained from it!

Last but not least: Thank you - to everybody in the InnerSource Commons community who has spoken to me and helped me along the way. This community is full of friendly and helpful people, and am happy to be a part of it.

Level up your InnerSource through Patterns

2020-12-02T00:00:00+00:00

I did it again. ;) After speaking at the previous InnerSource Commons Summit already, this time around I joined forces with fellow InnerSourcerer Fei Wan from Comcast to present at the ISC APAC 2020 Summit.

Together where shared how our InnerSource adoption benefitted from Patterns and how we started contributing to the InnerSource Patterns working group.

Our talk coincided with the release of the InnerSource Patterns book, which we hope will make it easier for the community to consume the InnerSource Patterns and learn from them.

You find both video and slides of the presentation below.

Video

Slides

Presenting at the InnerSource Commons 2020 Fall Summit

2020-09-15T00:00:00+00:00

Since late 2019 I am a member of the InnerSource Commons, a community of people passionate about fostering the adoption of InnerSource: the use of Open Source best practices for software development within the confines of an organization .

When I heard that they were organizing an Online Summit, I decided to throw my hat in the ring and try to get involved as a presenter. This was also motived by my experience at FOSS Backstage earlier in the year, which I had thoroughly enjoyed.

1 year in the InnerSource Commons Community: Getting your money’s worth

I got lucky, as my talk proposal got accepted. The topic of my talk was aiming at people like me that were new to a community like this, and didn’t know how to get the most out of their involvement.

Or in the words of my talk abstract:

Let me make the argument for you: Convince yourself (and your manager) of the benefits of participating in the InnerSource Commons community.

I prepared my slides and the recording for the talk, which got screened at the InnerSource Commons 2020 Fall Summit on September 15th.

Presenting at this conference was a great experience, as I got to practice my presentation skills and it allowed me to get involved even more actively in this fantastic community.

You find both video and slides of my presentation below.

Video

Slides

Biases busted at FOSS Backstage 2020

2020-04-03T00:00:00+00:00

I went to FOSS Backstage this year. What was planned to be a small conference anyways, became an even smaller event due to the protective measures against COVID-19 in Europe. Some speakers had to opt out, while others presented from remote. It looked like some registered attendees did not show up either, which led to an intimate group of 30-40 people at the event.

What is FOSS Backstage?

With so many different events around in Tech, every conference looks for its own niche. FOSS Backstage found a great one! Hear it from the organizers themselves:

[…] discussion and learning about non-technical aspects of open source development. Topics include open source project leadership best practices, community management, open source project metrics, open source strategy for enterprise, adopting open source collaboration in corporations, legal matters when dealing with open source and many more.

My Favorite Talks from FOSS Backstage 2020

Let me preface this with: All talks at FOSS Backstage 2020 were interesting in their own right! My greatest respect for anybody that gets on stage to share what they have learned in their respective field.

For my highlights I picked the ones where the concepts were the most novel to me, or where I simply found the presentation entertaining. In no particular order, these were my favorite talks.

Why the GPL is great for business

In this comparison of licenses and business models in the Open Source world, Jos Poortvliet from nextcloud.com concluded that “the best open-source business model is the subscription model”, taking the acquisition of Red Hat by IBM as proof for this claim.

While parts of this talk were hard to follow, as I don’t have enough prior knowledge about software licensing, this was an extremely interesting talk. I enjoyed the business angle of this talk. For any project to be sustainable, money needs to come from somewhere after all.

Other notes from this talk that need further unpacking (for me):

“Some of the differences between GPL and MIT disappear if you don’t enforce it.”
The AGPL is a version of GPL that was meant to be used for the Cloud. Also see GNU Licenses FAQ.

How to convince your left brain (or manager) to follow the Open Source path that your right brain desires.

Bertrand Delacretaz convinces us of the benefits of contributing to Open Source. His talk is focused on what the individual developer will get out of Open Source for their personal development.

In short, Open Source contributions lead to:

Better Programmers (important projects; visibility of your contributions; impact)
Better Communicators (collaborate asynchronously in low emotional bandwidth environments)
Better Persons (working in multi-cultural environments and “traveling the world”)

Great teaser to motivate developers to invest time in Open Source development. Also the delivery by Bertrand was fantastic!

Blending Open Source and Corporate Values

The speaker Cornelius Schumacher presented how they are bringing an open-source-like culture into the corporate world. He shared the InnerSource License they created to simplify the source code distribution and collaboration on a code level between the companies in their group.

It was fascinating to see the collaboration problems one has to solve when working in a slightly older business domain, but also in a larger group of companies.

One key recommendation:

Call it InnerSource! That does not sound threatening to business people. – see innersourcecommons.org

A community-driven approach towards open innovation for research communication

In what was the most intellectually stimulating talk of the conference for me, Emmy Tsang presented the challenges of the century-old peer review system in science. She went on to explain how eLife is aiming to improve science publishing, technology and research culture.

My Bias was Busted

While I have found the ideas of Free and open-source software (FOSS) interesting for a long time, the sometimes religiously seeming conversations about it had been a turn-off for me. When registering for this event, I was both curious but also a bit worried if I would be able to connect with the line of thinking at this conference.

My worries were completely unnecessary. I enjoyed the event from beginning to end, found a group of open and caring people, and got to learn many new angles on parts of the software industry that I am not exposed to in my day-to-day work as an Engineering Manager. For me, it was well worth moving out of my comfort zone by attending this event.

Thank you to the conferences organizers for making this such a great event, and so easy for FOSS outsiders like me to get their bias busted :)

Repo-Rot is Real at RUG::B

2019-12-05T00:00:00+00:00

I gave a presentation at the December 2019 Meetup of RUG::B (the Ruby User Group Berlin).

Topic: Repo-Rot is Real

I help to maintain a GitHub org of 350 engineers and 3100 repos. Often I wonder: What can we do to maintain some sanity with this amount of people, teams, and repos?

The longer a team exists, the more software components and repos they tend to have. How do you know, which of those are actively maintained, and which ones are not (they are rotting)?

This talk explored what I call “repo rot”, which effect it has on your team/company, and what you can do about it.

I also described an experiment that I ran with my teams, and what we learned from it.

Costs of Repo-Rot

For the Company
- maintaining documentation and security standards becomes harder
- tedious handover of components
For the Team
- Which repos matter for a new hire?
- Dead code (unused repos) makes code search on GitHub ineffective
For the Individual
- Fewer InnerSource contributions (PRs across teams)
- Broken-Window effect

Slides

No video was recorded at the meetup and I didn’t share any slides yet either. However if the topic sounds interesting to you just ask, and I am happy to share more.

A Simple Tech Stack for your Engineering Blog

2019-11-25T00:00:00+00:00

For quite a while I am maintaining the Engineering blog for Meltwater at underthehood.meltwater.com. What flew under the radar for a number of years as a side project, is now pushing out multiple blog posts every month, and is helping the company both externally and internally. The blog attracts 50k unique page views per year, so it looks like the content we share is useful for the readers.

This post shares the tech stack we use, and the editing process we apply when releasing blog posts. If you want to start an Engineering blog at your company, this post may help you.

The Tech Stack of our Blog

For our Engineering blog underthehood.meltwater.com we went a bit nerdy. We chose jekyll and GitHub Pages as our platform. More on the pros and cons of that choice shortly.

More about the stack:

CMS: We are not using a Content Management System (CMS) but instead jekyll as a static site generator. In essence you feed it markdown files, and it renders HTML out of it.
Theme: a custom theme for jekyll (that we built during a hackathon project)
Hosting: GitHub Pages for hosting the static content. It supports jekyll out of the box, and allows us to host our blog without any additional costs.
DNS setup to serve the blog at underthehood.meltwater.com

Pros and Cons of the Tech Stack

As a pro our blog loads fast, and is highly available at negligible cost. Getting the same speed and uptime with any CMS rendering the blog dynamically, and involving a database for content storage, would be a lot more work and certainly more expensive.

The con here is that we cannot do fancy things like loading dynamic content from twitter or GitHub. Quite frankly we don’t need that anyways, hence this approach was right for us.

Also as markdown and GitHub are well known by our Engineers, we don’t have to explain hardly anything new to them when they want to write a blog post.

However this can be a gatekeeper for colleagues in Product/UX or other functions in our Product & Engineering department that might not have used markdown and GitHub before. We counteract that with our Editing process (see below).

Editing and Publishing Process

Our approach is simple. We are not the New York Times, so small mistakes are acceptable! :)

the author finds a topic to write about (it often starts with nothing more than a comment in Slack, an email, or a corridor conversation)
the author writes a draft of their blog post in a Google Doc
somebody from their team or I providing feedback for the draft, and clarify the target audience and the main takeaways they author wants to get across
once the content is done, we convert the Google Doc to markdown and the author sends a Pull Request to our blog
Done! Fame and glory for the author forever :)

Why Google Docs?

Google Docs has great commenting/suggesting features, that make it easy for authors and editors to collaborate. The alternative of writing the post in markdown and then using Pull Requests on GitHub for collaboration is not attractive for us, as it does not make it easy enough to suggest changes inline in the blog post and have a conversation about the content between the authors and editors.

Additionally Google Docs is well known by everybody in our Product & Engineering department, which lowers the technical barrier of entry for people that want to write a blog post a lot.

However for the authors that don’t know the markdown format, this also means that somebody has to do the Google Doc to markdown conversion for them. No problem though, as I wrote docx2gfm, a little tool that makes this conversion faster.

Starting your own Engineering Blog

Some developers are already maintaining their personal development blogs. This is great but they often fall short on content, as you can see on my own spier.hu blog. :) So if you are an engineer, why not write a post on your company’s Engineering blog about the topics of your day-to-day work?

Many companies are already writing about how they are approaching various challenges in their Engineering department, but I think it could be more. We can all learn from each other! So the more honest stories we all put out there about the realities of the software development industry the better.

If you want to start an Engineering blog similar to underthehood.meltwater.com for your company, I hope you found this post useful. If you want to discuss how to motivate people to write, get buy-in from leadership, and set up tech and process for your blog to be successful, please reach out, I am happy to help!

PS: I also want to thank Tristan for the design, Alex for helping with the migration to GitHub Pages, Niall for co-maintaining the blog with me for the last 2 years, and the many authors that have provided invaluable feedback throughout the years. You all made this approach work!

docx2gfm - Your friendly docx to markdown converter

2018-12-23T00:00:00+00:00

I am maintaining the Engineering blog for Meltwater. Our blog uses jekyll and GitHub Pages as the publishing platform.

In our tech stack and publishing process, the authors and editors use the commenting/suggesting features of Google Docs to collaborate on their blog posts.

As we use Google Docs for crafting the posts, we convert posts from .docx to markdown format frequently. I wrote docx2gfm, to make this task less tedious and error-prone.

This post shows how to use docx2gfm to convert a document from .docx to markdown (github-flavored-markdown).

The original doc

Let’s say you have a document simple-doc.docx that looks like this:

Turning this into Markdown

With docx2gfm you can turn simple-doc.docx into markdown.

Running this command …

docx2gfm --no-jekyll -f simple-doc.docx

… will generate this markdown:

---
title: '<span id="_iq8c5s4280y5" class="anchor"></span>Super Simple Doc'
---

A short intro goes here.

## A headline looks like this

Let’s see how [docx2gfm][docx2gfm] converts links.

## Emphasize section

We **may be Bolder**, we *may be Italian* (ouch!), we ~~may be on Strike~~, or under the line.

Not all of these work in [github-flavored-markdown][github-flavored-markdown]. This is markup, not [text formatting][text-formatting].

## How about lists?

- A list ...

- … with bullets

1. Numbered lists …

2. Are great too!

## An image

<img src="media/image1.png" style="width:0.74479in;height:0.74479in" />

Von Ludwigs2 - Eigenes Werk, CC BY-SA 2.5, [https://commons.wikimedia.org/w/index.php?curid=6292191][https://commons.wikimedia.org/w/index.php?curid=6292191]

## Table Tests

| **TH 1** | **TH 2** | **TH 3** |
|----------|----------|----------|
| TD 1     | TD 2     | TD 3     |

## Quote!

Every once in a while you need a great quote. To do so, write your quote and then move the indentation level in the Google Doc to the right, as shown below:

> Blogging is great for your charma.
>
> \- Sebastian Spier (2019)

[docx2gfm]: https://github.com/spier/docx2gfm
[github-flavored-markdown]: https://help.github.com/articles/basic-writing-and-formatting-syntax/
[text-formatting]: https://softwareengineering.stackexchange.com/questions/207727/why-there-is-no-markdown-for-underline
[https://commons.wikimedia.org/w/index.php?curid=6292191]: https://commons.wikimedia.org/w/index.php?curid=6292191

Rendered Markdown (HTML)

This markdown will render to HTML similar to this:

Post Processing

You might say that the resulting markdown is not perfect. That is correct, it is not 100% the same format as the original .docx file.

Important post-processing that you still need to do:

set the title of the document. Markdown does not know what to do with the title format it gets from .docx.
move images to the correct directory in your blog and fix the image paths in the markdown

In Closing

Try out docx2gfm if you need to convert from Google Docs or .docx to markdown frequently. docx2gfm also has features to configure the markdown output e.g. to include a jekyll Front Matter.

Technically docx2gfm is a thin wrapper around pandoc. Hence big props to the amazing people that built pandoc!

Open source at it’s best :)

Go to Conferences!

2016-05-31T00:00:00+00:00

This April about 40 colleagues from Meltwater went to Craft Conference in Budapest. We did not plan for it that way, but it ended up like a mini kick-off for the Product & Engineering teams in Europe.

It turned out to be an amazing conference with many great speakers that shared their thoughts about how we can collectively improve our craft and create better products for it.

Summarizing Craft Conference in this blog post would not do it justice. Instead we want to share our thoughts on why we are sending our people to conferences in the first place.

External stimulus leads to better decisions

Making good decisions and finding creative solutions to tasks is limited by the amount of information you have readily available to you. Within a small team, typically 5-8 people in our case, it quickly happens that a clan culture develops. While strengthening collaboration, clan culture can also can lead to suboptimal decisions due to a lack of diversity in knowledge and opinion.

Sending team members to conferences to gain external stimulus is a great solution to counteract the negative effects of clan culture. Depending on the conference, team members can gain new ideas and inspiration about process, technology, and any other topics relevant for the team. Armed with these new ideas and inspiration, teams will be able to approach old situations in a new way or try experiments with new technologies.

Networking - Both external and internal

Without a doubt, conferences are a great place to recruit and make business connections. However, the emphasis is often put on the external networking benefits. Often ignored are the internal connections within the same company that can be formed through events like this. Building software is all about communication. That being said, having people who have only communicated through phone or video chat, meet face to face and share an experience, only enhances their communication when they return to their day to day.

When you are talking about a geographically dispersed organisation, like Meltwater, and sending a group of people all to the same place, something magical happens. People form long-lasting bonds and friendships, which make working together easier and more fun! By the way, this is one of our motivations to organize company wide Kick-Off events as well (see benefit 10 here).

Keeping the momentum

Another interesting effect of visiting a conference as a group are the instant feedback loops that emerge between people that are learning about new and exciting concepts together.

For instance many of us at Craft Conference were inspired by the sessions around product discovery and dual track Agile by Marty Cagan and others. As we saw these talks together, we are able to quickly and clearly communicate about these concepts with each other, while also pushing each other to learn more about these topics and try them out.

The approach of a shared conference experience is far better than coming back and trying to tell your colleagues about what you learned, because they won’t have the same inspiration that you had when hearing these things live by speakers who really know how to deliver them.

Also there is more of a drive to move things forward when you return to your work place, as you have now made plans for change together with other people and you will feel accountable to them.

When you learn things as a group, the inertia from the conference is not as easily lost.

Represent your company and pitch your product

People in Product & Engineering often have less face time with their customers than their colleagues in Sales or Marketing. While this might be a challenge in it’s own right, at conferences, you, as an Engineer, are quite obviously representing your company.

A frequent question that you get asked by other conference goers is “What does your company do?”. This is a great opportunity to pitch both your company and your product to others.

What’s even more beneficial: Since people are often from a similar background they are representing the company in a way that other engineers can understand and are interested in. If you send your sales team to have a booth at a conference and sell your product, is not at all the same thing as having an engineer passionately describe the culture of your organisation to another engineer.

In summary

But but, the costs … (the cost-aware manager says).

Looking at all the benefits, for us, investing in conferences is often a more impactful way to spend your training budget than sending people to courses etc. If someone is motivated to learn, they will find ways to do it, no matter how. These are the kind of people we have in Meltwater, and these are the kind of people we are looking for.

We believe that providing initial sparks of ‘new things’ to our employees is one of our most important investments into their development, and conferences are a great way to do this!

So our recommendation is: Go to conferences!

Craft Conference 2016 Review

2016-05-08T00:00:00+00:00

This April me and many colleagues from Meltwater went to Craft Conference in Budapest. While I was typing up my notes, I figured I might as well share them publicly.

The talks below are ordered by my preference i.e. if you trust my judgement, then start viewing them from the top.

What you might get out of this:

a list of talks that I thoroughly enjoyed (I only summarized those)
video links, twitter handles of the presenters, and slides for the talks where available
direct quotes and some of my notes and opinions about these talks
references for further reading

Some of my notes might not make sense out of context, so continue at your own risk :)

Great Product Team, Successful Product (closing keynote day 1)

Marty Cagan (Silicon Valley Product Group)

Craft Profile, Slides, Video

Marty gave an eye opening talk about what he considers key ingredients of great product teams. He summarizes them very well in this blog post “product success”, so I will only add some additional notes below.

His overall statement is that while many companies follow agile principles during the development phase, the rest of their process of building software products is still more waterfall than anything else. (I believe elsewhere I also heard Water-Scrum-Fall as a term.)

Marty emphasized the importance of connecting the team directly with the customer. Only then can the whole team help to contribute to the solution, instead of just acting like an internal agency.

To keep the team on track towards their shared goal, one needs a clear vision (inspiring) and strategy (intentional). Having this can also be a great hiring tool for a company!

Instead of pretending we know what we are building, we should focus on quick discovery, and risk assessment. Risks may come in different forms but for example:

(Photo credit: @Remarker_eu)

Some setup elements that make it easier for teams to become great:

Durable (team together a longer time and a lot)
Co-located
Chemistry/ camaraderie - we need a safe place to share ideas
Skill-set diversity

And lastly a couple of references:

“Thick” Data and Analysis: Engineering with a capital E

John Allspaw (Etsy)

Craft Profile, Slides (not published as of May 8th), Video

John took a highly data-driven approach in analysing what happens while we work as a software developer. How do we debug problems? What works, and what doesn’t? How do we learn from this?

Not everything that counts, can be counted. Not everything that can be counted, counts. (John Allspaw)

The overall point he was trying to get across: We need more people to explore what actually happens (below the cover) when we build software. More people need to write books about this topic!

His many references included:

His thesis http://bitly.com/AllspawThesis
Walter G. Vincenti - What engineers know, and how they know it
Julian Orr - Talking about machines / Review

Psychological Aspects of Estimating

Joseph Pelrine (MetaProg GmbH)

Craft Profile, Slides - requested via jpelrine@sendyourslides.com / subject: PAE, Video

Joseph delivered a rather uncommon talk for a technology conference but that is probably what made it so great. He focused on how we typically do estimations in software development teams, and which biases we have when we do this.

Even more impactful for me however was the following quote:

If we find that our language (in software) is reducing our ability to communicate with our customers, then that is the problem we should solve. (Joseph Pelrine)

This resonated a lot with me. Truly great product teams need a healthy amount of communication with customers and other business stakeholders. Hence finding a common language between both worlds is very important. Last but not least as the language we speak influences the way we think.

Embracing uncertainty: Why you should and why you won’t (Keynote)

Dan North

Craft Profile, Slides, Video

Dan is a very engaging speaker. I attended his workshop as well, and many of his ideas center around this quote, which takes quite a bit of unpacking:

Sustainably reduce lead time to business impact. (Dan North)

In his talk he focused on how uncomfortable we are with uncertainty, and how this need for creating certainty may lead to dysfunctions in our software development processes.

Most noteworthy for me, he mentioned that measuring “proxy metrics” can negatively impact the behavior of your team. Make sure you know why you are measuring what, and what you are doing with this number. (e.g. velocity is a pure capacity planning metric but has nothing to do with performance)

Deploying and Scaling Microservices

Sam Newman (ThoughtWorks)

Craft Profile, Slides, Video

Sam gave an overview talk, starting from artifacts and how you might build them and ending with deployment platforms for containers. It suspect that for people will experience in container deployment, this talk didn’t have much new. However as an overview talk into the topic I found it helpful.

He reviewed different techniques of artifact creation, along these 4 guiding principles:

Easy to create
Easy to deploy
Abstracts out the tech stack
Good for dev, good for ops

According to Sam the vendor lock-in of the deployment platforms is not that big of a problem, as the decision of where you run mesos or kubernetes is yours. (amazon ECS, Windows Azure, google cloud, …).

Some of the relevant technologies to read up on in this field are: Docker, mesos, kubernetes

Architecture without architects

Martin Fowler (ThoughtWorks) & Erik Dörnenburg (ThoughtWorks)

Craft Profile Martin Fowler, Craft Profile Erik Dörnenburg, Slides (not published as of May 8th), Video

After their talk I didn’t necessarily have an answer to how to achieve “architecture without architects” but they certainly provided interesting stimuli. Martin and Erik view the role of an architect as somebody that improves other developers, and not somebody that draws diagrams.

Can we imagine a system where adding new features becomes easier, the older the system is? If that is the case, then we have to assume that some architectural thinking was applied.

Build upon each others work, with local rules and an evolutionary architecture (Martin Fowler & Erik Dörnenburg)

One more instructive comment for me was that each microservice should contain part of UI composition and API gateway, as the team that builds this service has the required domain expertise. Interesting thought, now I am just wondering how to do this :smiley:

The Technical Debt Trap

Michael (Doc) Norton (CTO2)

Craft Profile, Slides (further references from cto2@sendyourslides.com / TechnicalDebt), Video

Doc Norton, one of the better known names in the conference circuit, gave a presentation about how to deal with evolving code bases.

Doc believes that cleaning sprint or hardening sprint don’t work. Instead we should use the boy scout rule and apply it to our code (“Always leave the campground cleaner than you found it”).

As a simple but good enough proxy metric of code complexity he suggests lines of code.

And finally this quote that will make a lot of developers happy:

Never ask for permission to do your job correctly. (Doc Norton)

Building Scalable Stateful Services

Caitie McCaffrey (Twitter)

Craft Profile, Slides, Video (not published as of May 8th)

In her talk, Caitie was able to break down a rather complex topic into digestible pieces. Her excellent slides helped with this a lot as well!

There is much more information in the slides than what I can repeat here but one important comment was that stateless services still have state (but it’s outside your service).

Scaling Engineering Culture

Raffi Krikorian (Uber)

Craft Profile, Slides (not published as of May 8th), Video

Raffi could unfortunately not be at the conference in person but even the remove presentation was good. I didn’t take that many notes but my key take-aways were:

What feedback loops do you have in place about your Engineering culture? What do they tell you?

Be careful when rewriting software:

business moves on (if it takes them too long)
engineers feel terrible, as they want to deliver but it isn’t possible

Show (to you company) what (business) impact great engineering culture has. Tell the story!

Monitoring as a Service - Modernity and Self-Service

James Turnbull (Kickstarter)

Craft Profile, Slides (not published as of May 8th), Video

James suggests to make interactions between Operations and Development happen when there is no conflict, i.e. when there is no production escalation. Sounds like solid advice :smirk:

Know that you are not the only customer of your monitoring. Find the others and understand the different needs. Then use different visualizations, e.g. consoles and graphs, for different purposes.

Toward Better Design and Code - A Case Study with Pelikan Cache

Yao Yue (Twitter)

Craft Profile, Slides, Video

Yao talked about a case study at Twitter but while doing that also explored general principles of writing and re-writing software. Her overall approach is that software should be modified by evolution, not revolution.

She finds it dangerous to aim for a better solution. According to her we cannot always afford ‘better’. Do good first, then make it better.

Yao also mentioned that their team would not have been able to design pelikan, without first using all the other technologies that pelikan is abstracting from. So in short, you cannot shortcut the learnings that need to be made. Damn!

Every Product Company needs Corporate Innovation Days!

2015-11-19T00:00:00+00:00

I am part of a small team that organizes a company internal innovation event for our 100+ man strong development organization. I have been part of this so called Innovation Day (iDay) from the very beginning, therefore I want to share how such an event can impact the culture of a company.

TL;DR If you are a product company, developing any type of custom software, and you don’t have an event that taps into the creativity of your engineers: Change that today! It is worth every penny!

Innovation Day Setup

So what is an Innovation Day? In other places these events are called hackathon or hackday. Some corporate examples are the google 20% day, zalando hack week and many others. Fundamentally they are all the same: Allow your engineers to work on projects of their own liking for a fixed amount of time, typically outside of the context of their team.

We kept the concept both simple and open:

work on a project of your own liking for 1 day (the projects typically still end up at least somewhat related to the domain in which our company operates)
submit your project if you feel you have something worthwhile submitting (otherwise you can also continue on the next iDay)
share the projects with the whole organization for public voting. this determines the overall ‘winner’ of the event
next to the public voting, a jury picks a couple of projects in different categories e.g. biggest productivity improvement, or most flashy UI

By design, we took every single iDay as an experiment. Much like in the agile development paradigm, we used the retrospective learnings from each event, to make the next event better, more fulfilling for the participants, and more beneficial for the company.

Learnings

Some of the areas that have been positively impacted by these Innovation Days:

The creativity of that many people puts a lot of interesting projects in your pipeline. Hence we had to revisit our innovation capitalization process quite a bit. Every time we were reviewing the submissions from our engineers, we were trying to figure out how to best make these projects flow into existing or new products.

Many engineers were experimenting with internal APIs and tools unrelated to their normal work responsibilities. Over time this started to foster an internal open source mentality in the company, which helped to improve the quality of API documentation and the way we think about services in general.

Communication and collaboration. We saw increased discussions between Engineering and Sales about projects coming out of iDay. New collaborations started within Engineering because people from different teams were now working on the same project for a day. The communication between Engineering and Product benefitted from these events as well, because both teams could discuss new ideas independent of any organizational boundaries.

Engineering efficiency got a boost from our iDay activities too. Tools were developed to better align the development guidelines across teams, to get more out of our communication tools (HipChat bot anybody?), or to inspect the internal data structures of our Search Engine.

Culture and Fun. iDays have helped to bring our teams together in a fun, shared global experience. They even had some ripple effects into other departments, which also started to experiment with similar concepts.

Summary

In case your company develops custom software, and you don’t have any type of event that taps into the creativity of your engineers yet, then I hope that this post has you wondering if you should change that.

I would be very happy to hear about your experiences with similar concepts at your place of work!

Photo credit: Missy Schmidt - Innovation chalkboard

BE MORE - Sebastian Spier - Careers at Meltwater

2014-12-02T00:00:00+00:00

Meltwater recently produced the BE MORE video series that portrays various employees and their different roles in our company. Sebastian Spier, Development Manager in Berlin, was also interviewed for this series, so that you can learn a bit more about him and the work he and his team do for Meltwater. The video is in German but we are sure that your German is great, right? :)

Also checkout the other videos of the BE MORE series, and our careers site.

Transcript Interview Sebastian Spier

Wie heißt du, und seit wann bist du bei Meltwater?

Ich bin Sebastian Spier und ich bin seit 2008 bei Meltwater.

Ich habe zunächst Germanistik und Sportwissenschaft studiert, weil ich Journalist werden wollte, aber dann habe ich umgesattelt und an der TU Berlin Informatik studiert.

Bereits während des Studiums habe ich als Software Entwickler in unterschiedlichen Unternehmen gearbeitet. Nach dem Studium war ich dann Wissenschaftlicher Mitarbeiter an der TU Berlin. Dabei ging es im Schwerpunkt um die Erforschung von besseren Suchalgorithmen.

Warum Meltwater?

Ich habe damals nach einer Herausforderung gesucht, die mich ins Ausland bringt. Deswegen war für mich die internationale Ausrichtung der Firma sehr wichtig. Ende 2007 bin ich dann bei einer Firmenkontaktmesse auf Meltwater aufmerksam geworden, und zwar über MEST, das Non-For-Profit Programm von Meltwater in Ghana und so habe ich dann Anfang 2008 als Teaching Fellow in Ghana angefangen.

Was ist deine jetzige Rolle?

Momentan bin ich Development Manager für Self-Service und Mobile Applications und ich leite hier zwei Entwicklungsteams in Berlin.

Die Kultur in Engineering ist sehr von unmittelbaren Zielen getrieben. Das heißt wir wollen immer versuchen die Weiterentwicklung so unmittelbar wie möglich an die Kunden weiterzugeben. Das ist für Softwareentwickler sehr befriedigend, weil wir einfach sehr kurzfristig sehen, was die Auswirkungen von unserer Arbeit bei den Kunden sind.

Nach welchen Eigenschaften suchst du beim Einstellen?

Wenn ich einstelle, dann suche ich auf der einen Seite nach Fähigkeiten, die ich konkret für mein Team brauche, auf der anderen Seite suche ich aber ganz bewusst nach Leuten die bereit sind Feedback aufzunehmen.

Also wenn ich sehe, dass jemand vielleicht schon zu sehr in seinen festen Denkmustern steckt und sich da nicht mehr rausholen lässt, dann wäre das für unsere Teams kein guter Fit.

Welche Qualitäten sind bei Meltwater gefragt?

Ich denke das Wichtigste, wenn man bei Meltwater anfängt, ist zum Teil zu vergessen, was man vorher für Erfahrungen gemacht hat und einfach wirklich offen zu sein um neue Erfahrungen reinzulassen.

Meltwater ist sehr gut darin Feedback zu geben an Menschen und auch sehr direktes. Sowohl positives, wie auch konstruktives. Ich denke, wenn man dazu bereit ist, und wirklich offen machen kann um das anzunehmen, kann man einfach in kurzer Zeit unglaublich viel lernen.

6 Good Reasons to Study Computer Science

2014-11-18T00:00:00+00:00

Software development is a great profession! One way to get a sound footing in this field is a major in computer science. Unfortunately there are various biases against both the profession and the major which is why there are way too few freshman. Therefore I am making the case for new ways to pitch the software development profession and with that motivate students to study computer science.

Disclaimer: I never thought I would ever write a “x reasons to do y” post but the computer science major needs some serious marketing!

Recently I was talking to a friend who is a doctor that had issues to get her medical studies accredited in Germany. It struck me how comparably easy things are in my profession, which is software development. While doing some further research on the subject I realized that there is an unfortunate combination of negative bias and missing education about the software development profession, which is also contributing to too few freshman starting university studies relevant for the software development industry.

As I have been working in this industry for a couple of years now, and feel strongly that this is a great field, I want to excite people about our profession, because it is a really great one, once you get to know it!

So here 6 good reasons to study computer science:

Working abroad is easy! You need to speak English, and you are pretty much good to go. In the EU you can travel freely, and fairly easy get the necessary papers to work in any of the EU countries. Non-EU citizens can get a blue card, given that they have a degree of higher education. Even getting a working visa in the US is doable, although that takes a bit more effort from what I hear.
Teams are intercultural by default. Due to shortages in qualified applicants, companies consider a wider geographic audience than in other professions. It is common to work in teams where every colleague represents another nation. That adds a lot of energy to the team dynamics. :smile:
You can switch between domains because software is now used everywhere. Education, banking, government, gaming, manufacturing … you name it. All these industries are buying and building software these days, so you can choose an industry that you are excited about. In other words, what you do now, doesn’t necessarily determine what you will do in 10 years from now.
Working from everywhere is not a far-fetched illusion in the IT field. Some companies go as far as making office work entirely optional, although I personally don’t consider that an efficient approach. In any case, working remotely for a couple of days here and there usually isn’t a problem.
Software is still a young and developing profession and you have many opportunities to actively shape this industry. Contribute to open source projects, organize conferences, and work in your local communities. Want to use your skills to find other ways to make a difference and do something good? There are an abundance of ways to do that.
Financing your studies yourself is doable. There are many student jobs that you can do, even while still in your undergrads. Especially once you have your first programming experiences, many companies have jobs that you can apply for.

Excited yet? You should be! Talk to teachers, doctors, and people in other professions and I believe you will find that the IT industry is uniquely positioned in many different ways!

One last thing: It isn’t exclusively about programming! Really, it isn’t! So if you have an interest in this field, you should get more information and explore the different opportunities that it offers. If you need it, I am happy to help.

References

In Germany there is an initiative called “Informatik studieren” that aims to motivate more students to study computer science. If you are an experienced IT professional, consider becoming an ambassador for this initiative.

Image Credit: Commodore PET computer woolamaloo_gazette @ flickr

eurucamp 2014 retrospective

2014-08-03T00:00:00+00:00

I am just coming back from eurucamp 2014, this year held at HPI in Potsdam. My head is buzzing with all the different concepts I have learnt about and things that I want to try now.

Just like last year I collected some notes on the talks I liked most, and also some thoughts about the conference in general.

Talks

There were many good talks, and if I had to pick a best overall, it would probably be Frank Webber’s Utils is a Junk Drawer! Still there were so many good talks, that I think it si best to highlight some of the good ones in different categories.

Most fun:

Internet of Dusty Things - Daniel Bovensiepen
Let’s build a game with Ruby - Rin Räuber

Most thought provoking:

Utils is a Junk Drawer! - Frank Webber
Decentralise ALL THE THINGS! - Jan Krutisch

Most surprizing:

The Scientific Method of Troubleshooting - Blithe Rocher
Take note! - Såndra Wendebørn

Last but not least, the best slide came from the Take note! talk. Too bad I cannot draw like that :)

Feedback

Great talks, great people, great food, great weather, great conference all in all.

My sincere thanks goes to the eurucamp organizers for making this event happen. I appreciate all the hard work that went into it, and I even took a membership applications for Ruby Berlin e.V., to see how I can contribute to keep this community as vibrant as it is.

I also have some constructive feedback:

For me the Pendelverkehr between the two presentation halls didn’t work that well. Especially on day 2, people moving in and out of the rooms created quite a bit of distraction.

Day 1 had an almost 5 hour long lunch and siesta break. The majority of the activities happened in that time. For me that was too long. Personally I would have preferred to pull some of the evening talks to the front and rather move the activities to the end of the day.

While I understand that the eurucamp concept is a bit different from a traditional technology conference, I still think that some more technical talks would do the conference good.

As always with feedback, this is just my personal opinion.

That said, I will certainly be coming back next eurucamp. This conference has me convinced that speaking at conferences is worthwhile and possible. So thanks to all organizers and speakers for providing that type of platform.

Scrum - The Art of the Possible

2014-05-10T00:00:00+00:00

Writing is a good way for me to digest information, so in this post I am collecting important principles and practices of Scrum. Lastly I am listing questions that I have about Scrum and references for further reading.

If you can only do one thing, watch the video Agile Product Ownership in a Nutshell!

Below you see the city of trees that we built during our two sprints at a ScrumMaster training. Impressive, huh? :-)

ScrumMaster training

This week our team attended a 2 day training by Andrea Tomasini and Niels Verdonk of agile42, with the eventual goal to become a Certified ScrumMaster. After using Scrum with my teams for the better parts of 2 years, I thought I had a pretty good idea about how Agile and Scrum work. During this training I realized that there are many more things to learn.

I am collecting bullet points of what I found most interesting, as I don’t have further explanations for most of them. Maybe these notes can still be useful for others to start thinking about Agile and Scrum in new ways.

Principles

Don’t focus on the practices. Understand the principles, so that you can adapt the practices as needed.
Scrum is a vehicle for teams to practice empirical process control
It aims for transparency, inspectability and adaptability.
Software development is not a linear activity, especially when dealing with the uncertainties of building a new product. Trying to force the development into a linear process is therefore counterproductive.
Empower the team to find a solution. Finding solutions is a creative activity for which you need to trust the team!

Practices

The Scrum framework consists of
- 3 roles (product owner, scrum master, development team)
- 3 artifacts (product backlog, sprint backlog, burndown chart)
- 3 ceremonies (sprint planning, daily scrum, sprint review & retrospective)
The ScrumMaster focuses on the efficiency of the team, not the usefulness of the product
The backlog is the product of the continuous conversation between Scrum Team and stakeholders
Items in the backlog have different granularity. Sometimes the granularity of the stories is expressed by different names like User Stories < Epics < Themes.
Depending on the main focus of the story, sometimes we differentiate between
- Feature Stories
- Technical User Story
- Spike
All stories should be as equal as possible in complexity
The question “who is doing what” should always be answered as late as possible within a Scrum team
Working on 6-10 stories per sprint tends to be a good sized 2 week sprint.
A healthy backlog should be DEO
- Detailed appropriately
- Emergent (continuous / not planned)
- Ordered
A user story needs to be integrated across the whole application layer, as it
- Forces the team to work together cross functionally.
- Prevents mocking of functionality.
- Identifies risks earlier

Questions

In Scrum we want to get feedback fast, in order to learn and adapt fast as well. At the same time we don’t want to build prototypes but always build production ready quality. Isn’t that a contradiction?
How to introduce shared responsibility in a team, while still getting a sense for the performance of an individual developer?
Scrum aims to deliver the highest possible value in each iteration. Does this always have to be business value, or is e.g. learning considered valuable?
Which type of project is Scrum not a good fit for?

References

Media Hack Day 2014 - How do hackathons work?

2014-03-27T00:00:00+00:00

I recently attended Media Hack Day 2014 at the Axel Springer Plug & Play Accelerator in Berlin. During the event a couple of interesting ideas were proposed and prototyped. You can find all of them, including the app that my team worked on, at the hackerleague site.

It was a great location, good catering, well planned by the organizes and a pretty competitive crowd. So all in all a successful hackathon I believe, although I am not sure what qualifies a successful event from the perspective of the organizer.

Afterwards I have been reflecting on the event, and I see a couple of areas where the event could have been improved. The paragraphs below summarize my thoughts.

Focus, Output, Exchange

At a hackathon, the teams are prototyping ideas, often aiming at a not so well defined goal. At Media Hack Day the mission statement was to reinvent and reengineer content for the mobile age. You can imagine that it is hard for teams to come up with ideas in a short amount of time that are anywhere close to hitting such a goal.

As the majority of participants at hackathons tends to be developers, and not people with domain expertise (journalists in this case), the ideas are often driven by the developers themselves. Consequently developers are only imagining, what type of challenges journalists, editors, and publishers might have.

One possible improvement would be to have presentations by more domain experts at the beginning of a hackathon, to foster a greater amount of direct exchange between both sides, and a more focused ideation process.

Partners, Agendas, Full Disclosure

Hackathons frequently offer special prizes for the use of certain APIs, which are generally the APIs of the partners/sponsors of the hackathon. Other hackathons are even completely focused on the use of a single API. In order for such a setup to work for both API provider and developers, I believe some prerequisites need to be met.

The API needs to be ready. Ready as in tested, documented, stable, etc. Only if that is the case, then the developers can create something useful on top of the API. If there are too many cases where the developers are facing significant challenges with the API, they will stop using it or just not produce anything interesting.

The partners need to plan carefully how to introduce the API to the developers. An unfortunate counter-example at this hackathon was gettyimages, who decided to show multiple marketing videos during their API introduction. Marketing speech in a hackathon is certainly turning developers down. Instead the partners should help developers understand the potential of their APIs e.g. show existing prototypes, or interesting features of the API, or highlight use cases that the API partner is especially interested in.

In addition, API partners and sponsors should be as transparent as possible about their own agenda and interests. It is clear that companies would not organize such events if they wouldn’t see any return on investment for them. It is totally fine that these interests exist, as the developers that attend such events have some type of agenda for themselves too - at least that is what I would expect.

Teams

Team creation is an important and sometimes complicated topic.

This hackathon had a lost souls session at the beginning, where all participants that didn’t have a team yet could pitch ideas and form teams. I thought that was a very good idea!

Other teams had already formed before the event. Some of them were even traveling to Berlin from Poland and other countries, already with an idea in their pocket. This certainly leads to a somewhat uneven setup between the different teams but as long as people don’t show up at hackathons with completely built projects this seems to be fine.

Ownership of data created

When participating in such an event, lots of data about the teams and participants is created. Plenty of pictures are taken, videos are filmed, sometimes even interviews with individual participants are made. One thing that the organizers should do - which was missed at the Media Hack Day as well - was to inform all participants what the material will be used for.

Just because a developer is signing up for a hackathon, it doesn’t mean that he wants to see his picture in any material used for marketing purposes of the hackathon organizer or the API partners. Again, full disclosure in this regards would help a lot. From talking to the camera man at the event it seems that this type of announcement was planned but I cannot recall it. (In case I was just tired and missed it, sorry :))

Summary

In my experience, developers are a fairly forgiving crowd. Often times they are honestly interested in the tools that they are using, hence they just like to build stuff with them. I have also found many to have strong opinions, not only about technology but also about other topics such as politics. Some are even willing to donate their time for a good cause.

With that in mind, and the fact that most people attending a hackathon should be able to earn more money as a freelancer than the value of the prizes that they might win at the event, it is in the responsibility of the organizer of the event that the hackathon has an as productive and focused outcome as possible.

I am looking forward to attend more such event. I am also interested to hear what experiences other participants of Media Hack Day 2014 or other hackathons have made. Please leave a comment below if you would like to share your thoughts.

eurucamp 2013 retrospective

2013-08-20T00:00:00+00:00

Wow! eurucamp 2013 is over, and I left with a lot of new impressions about ruby but even more so about the awesome ruby community in Berlin and Europe. I met rubyists from Hungary, Italy, and Spain and everybody seemed to enjoy the event.

One of the few things that a particpant of conference can give back is feedback. This is what this post is about.

Feedback for eurucamp 2013

Location. The conference location at Hotel Müggelsee was great. First I thought it might be too far on the outskirts of Berlin but actually that in itself had the benefit of keeping the conference participants together as a group. Some of the other hotel guests probably thought that a radical splitter group of the NSA took over the place, flying and hacking AR.Drones, talking about this strange thing called ruby, and all speaking English. But they will get over it :)

Timing. The conference being on the weekend made it easier for me to attend. I often see conference scheduled during weekdays, and while that might have its advantages too, many cannot afford to take vacation for going to a conference.

Affordability. The ticket price of about 120 EUR is a very affordable for a 2 1/2 day conference, especially when compared to others. In that context it is even more impressive that coffee, beverages, fruits, small snacks, and lunches were provided. Awesome!

Organization. Check-In, sessions, and breaks were all well planned. Generally there was very little confusion about the schedule or what would take place when and where. My only pliece of constructive feedback here is that speakers should stay to their allotted time, in order to respect the overall schedule as much as possible. So the organizers of Ruby Berlin e.V. could be a little bit tougher on the speakers in this regard.

Activities. The activies at eurocamp made for a nice compensation for all the sitting. They presented an easy way to get to know other conference participants, that you otherwise might not have interacted with. Thumbs up for that idea!

Talk Selection

Simply put: Good work with the talk selection! I am sure that with this variety of talks, everybody found somethink new for her/him.

Either by design or by accident the Saturday contained most of the less technical talks while Sunday’s talks tended to be more technical.

Also the topic of diversity carried us through the conference. Not only was it part of the code of conduct - which was explicitly stressed during the opening remarks - but it was also picked up by a couple of speakers afterwards. This definitely contributed to the overall success of the conference as well.

Top Talks My favorite talks were:

Once Upon a Time … ruby. by Vicent Martí - @vmg
Practice makes perfect? by Ellen König - @ellen_koenig
Debugging Production by Michael Grosser - @grosser

In Summary

All in all I had a great time at eurucamp and recommend it highly to everybody. If you get a chance to go next year, do it! For me personally it definitely triggered a greater interest in getting involved with the community of rubyists in the Berlin area. So thanks to everybody who made this event as great as it was!

anabin Meta-API

2013-04-11T00:00:00+00:00

The University Database anabin did not have an API, so I wrote a simple one to demonstrate what cool integrations could be built, if there only was an API. Uses grape for the API, swagger-ui for documentation, and heroku for hosting.

API for anabin University Database

2013-04-11T00:00:00+00:00

Recently I was using the University Database anabin which has rather good data but unfortunately no API. So I wrote a prototypic API to demonstrate what cool integrations could be built, if there only was an API.

TL;DR Jump to the demo of the anabin Meta-API :)

What is anabin?

If you want to hire candidates from outside of Germany for the German labor market, different rules apply for the Visa process depending on the country of origin. If the candidate comes from outside of the EU, one option is to apply for the so called Blue Card. One of the requirements for this card is, that the candidate needs a degree from a German university, or from an accredited foreign university.

To find out, if the university has been accredited you can use the University Database anabin. anabin, which has been built over a period of 10 years, contains information from around 180 countries, 25 institutions, and 22.000 university degrees.

(btw anabin stands for “Anerkennung und Bewertung ausländischer Bildungsnachweise”. Yes, in Germany we love accronyms!)

While browsing the database, I noticed a couple of things:

It is pretty simple to find an institution via the search functionality provided by anabin. Unfortunately there are no permantent links for the university profiles. This is rather unpractical because one cannot set direct links to these profiles, which could for example be practical in order to add it to a persons profile at say Xing or LinkedIn.
Anabin is funded by Federal Foreign Office (Auswärtiges Amt), meaning the creating of this database has essentially been funded by the tax payer. Still it has no API, so that integrating this data in other applications is not possible.

AJAX calls in anabin

While anabin has no official API, it has a site search that uses AJAX calls to dynamically pull new content from the backend to populate the search results table. For example if you search for London, the following AJAX call is triggered.

http://anabin.kmk.org/index.php
?eID=user_anabin_institutionen&conf=institutionsergebnisliste
&sEcho=20&iColumns=13&iDisplayStart=0&iDisplayLength=10
&bRegex=false&iSortingCols=1&iSortCol_0=2&sSortDir_0=asc&s1=London&iDataIds=1

The JSON structure returned by this AJAX call looks like this:

{
  "iTotalDisplayRecords": 56,
  "iTotalRecords": 27765,
  "sEcho": 20,
  "aaData": [
    [
      "<img style=\"cursor:pointer;\" src=/typo3conf/ext/user_anabin/pi1/res/details_open.png>",
      "6714",
      "Goldsmiths College / University of London",
      "London",
      "Hochschule",
      "H+",
      "Großbritannien",
      "0",
      "London",
      "<a href=\"javascript:;\" class=\"detailLink-institutionen\" data=\"1250\">University of London</a>",
      "1250",
      "",
      "3"
    ],
    ...
  ]
}

Building a Meta-API

I don’t really want to create an API for anabin, as anabin is best positioned to do that themselves. So consider the following just as a prototype of a Meta-API to anabin, which shall demonstrate what could be done if anabin had a real API.

Essentially this is what I did:

write a ruby wrapper around the AJAX calls that anabin provides (using grape)
added a user friendly API documentation (using the great swagger-ui)
hosting it on heroku, so that everybody can access it easily

So go ahead and check out the demo of the Meta-API.

While doing this I also converted the JSON structure into a bit more meaningful form, by adding more verbose identifiers.

{
  "uid": "6714",
  "name": "Goldsmiths College / University of London",
  "location": "London",
  "type": "Hochschule",
  "status": "H+",
  "country": "Großbritannien",
  "other_fields": [
    "0",
    "London",
    "<a href=\"javascript:;\" class=\"detailLink-institutionen\" data=\"1250\">University of London</a>",
    "1250",
    "",
    "3"
  ]
}

Integration of anabin into other services

So now it is up to you guys! Who integrates anabin into another service first? I would for example love to see a little icon next to every university name on LinkedIn that indicates, which blue card accreditation this university has. Just saying! :)

Now there is an API for anabin, so have fun hacking :)

P.S.: If you should need it, here is the source code of the Meta-API.

Images Credit: blue card

Technical Book Review: Universalcode

2013-03-10T00:00:00+00:00

These days I am reading the German book Universalcode - “Journalismus im digitalen Zeitalter” (journalism in a digital age). On 580 pages it contains articles by many journalists, working for some of the major German publications.

All in all it is a pretty interesting read. Especially for me, as I am dealing with online articles at work daily, so learning more about the creative process that goes into them is quite exciting.

While I don’t qualify to comment on the content on the book, I am a technologist, and therefore have a couple of ideas about how the digital edition of the book - which I bought - could be improved.

Do not use bol.de for ebooks

Unrelated to the actual technical book review, and therefore separate: Do not use bol.de for ebooks.

First of all, buying the book at bol.de and getting it to run on my iPad was a rather painful experience. The format of the ebook was .acsm, a digital rights management format by Adobe. To make that work on the iPad, I need a new reader - I chose Bluefire Reader - that can handle that format. My conclusion for the next buy: Don’t use bol.de for ebooks, checkout the iBooks bookstore first!

Question: Does anybody have the ebook from iBooks? I am wondering if my review below would actually look different, had I not bought the book from bol.de.

Technical Book Review

Now to the technical review of the digital version of this book:

The formatting of the book is rather minimal. Really just paragraphs, section titles, and bullet points. From time to time some italics, even bold or underlined text isn’t used much.
The book contains no graphics whatsoever. Meaning neither any data visualization (like charts etc) or any other sort of imagery.
I read three articles so far and have already spotted a handful of errors of various sorts. Definitely more than I would usually expect in a book, after proofreading and quality control.
The book rarely uses direct links to websites or similar. Additional multimedia material does not seem to be available either. The book’s website abzv-universalcode.de has some content but the relation between website and book isn’t quite clear.

Summary

The digital version of this book costs about 20 EUR. Almost too much, when what I am getting is more like a PDF with a list of articles.

My recommendation: Practice what you preach! Take the digital version of this book, and make it an example of great digital publishing!

I just saw that there is a crowdfunding initiative with the working title Universalcode 2 - a potential success of this book. Maybe some of these hints here are helpful for that as well.

Developer Challenges – Lessons Learned

2013-01-18T00:00:00+00:00

In 2011 I participated in the 1st ever Springer API Challenge. Here is what I learned about developer challenges in general and how they can be beneficial for both developers and companies.

What is the Springer API Challenge?

Springer is a publishing house with 5500 employees and a yearly revenue of 850 Mil EUR - in short they are not small. They are focusing on scientific publications in German speaking countries.

At the end of 2010 Springer decided to open up access to their scientific content (mainly articles and images) via an API. In order to motivate developers to create innovative applications around their services, they set up the Springer API Challenge.

Springer Quotes

Even before the Springer API Challenge, I had frequently read about similar development contests, and had been curious to find out what developers and companies get out of them. So when I saw that Springer would host such a challenge, I thought I should give it a try, to find out if I could come up with something that Springer would consider prize worthy.

The application I came up with is called Springer Quotes. Springer Quotes is a single purpose application: It lets you search articles from Springer’s open access journals. When you find a particularly interesting section in an article you can create quotes from paragraphs or individual sentences, and share these quotes with others directly via the web.

Although Springer Quotes has a rather niche use case, Springer apparently liked it, as they awarded it with the 1st prize (press release in German). Other contestants developed KontentLinks (now kleenk.com), which lets you create more meaningful links between articles, and JournalSuggest which helps scientists to publish their papers to the right Journal.

Lessons Learned

Here a couple of thoughts, after taking part in the API Challenge:

[participants] Be a doer: It is always worthwhile to invest energy and time into something that you are genuinely interested in. Even without the extra bonus of winning something, taking part in this contest taught me a lot, both in terms of development skills, and focusing on what I considered important in order to convince the Springer jury.
[companies] Know your objectives: A developer challenge like this is not the right thing for every product/service. Before setting up such a contest, companies need to make sure that they know their objectives very clearly. One important aspect is, to find ways to channel the creativity of the contestants in the right direction, in order to get the most out of the investment.
[companies] Engage with the community: During the contest the participating developers will have a lot of questions. Make sure that before starting the contest you have the tools and resources in place to keep up with the increased community communication.
[companies] Use the results: Before starting, you should know how you want to use the results of the contest. Do you want to try to integrate them into existing products, build new products out of it, or just use the contest for PR purposes?

What about Springer? What did they get out of this? I don’t know it for a fact but at least they liked the API Challenge 1.0 enough to launch the Springer API Challenge 2.0 a year later. So apparently it wasn’t that bad for them either.

So much for Springer Quotes. Unfortunately the App is no longer online but you can take a look at the source code on github.

Final remarks: If you are a developer and have some spare time, just try yourself in a developer challenge. It is a lot of fun!

Zeitly

2012-12-01T00:00:00+00:00

Weekend project that uses the ZEIT ONLINE Contet API to answer these questions: Which authors haven written for DIE ZEIT and ZEIT ONLINE in which year? How often have their articles been mentioned on Twitter, Facebook, etc?

I/O Docs for ZEIT ONLINE Content API

2012-11-25T00:00:00+00:00

An alternative to the official API Explorer of the ZEIT ONLINE Content-API Explorer, based on mashery’s I/O Docs.

Also see related blog post.

I/O Docs for ZEIT ONLINE Content API

2012-11-25T00:00:00+00:00

One of the big German newspapers ZEIT has just launched the ZEIT ONLINE Content-API. The API gives developers access to the metadata of ZEIT articles, ranging back to 1946.

I am excited to see that such a renowned newspaper is showing some guts by entering the API world. Even globally, this is still an innovative step, as ZEIT is now the first German newspaper to have an API, if I am not mistaken.

Alternative to the API Explorer

So what can I do?

The API is still in beta, so it should be expected that some bugs still need to be ironed out. This is one of the many areas in which external developers can help to improve the API, by writing many small prototypes and hence identifying these bugs early on.

ZEIT is providing an API Explorer, which makes it easier for developers to test API calls. The problem is often that such an explorer (also known as API console) is maintained separately from the documentation of the API. Due to this separation, documentation and actual implementation of the API are often out of sync.

Therefore I have implemented an alternative to the official API Explorer, based on mashery’s I/O Docs. I/O Docs brings API documentation and API console closer together, and thereby mitigates the described risks of outdated documentation.

Take a look at the I/O Docs for the ZEIT API and let me know what you think.

Further comments

For the lack of a better place to post my comments about the API, I am posting them here.

the /content endpoint misses documentation for sort and fq
the /product/{id} endpoint misses documentation for the search parameters
some endpoints that allow searches are missing the operator parameter (although it is probably allowed)
the /series/{id} endpoint does not work at all. Error: “Due to an internal error the request could not be fulfilled.”
the documentation states possible keyword types as: location, person, organisation or issue. When searching for keywords I also see subject though.
the documentation of the search query syntax (parameter q) is not sufficient. It looks like under the hood Lucene is being used but that is just a guess.
potentially the calls /author/{id}, /department/{id}, /keyword/{id}, /product/{id}, and /series/{id} should all be grouped below the content endpoint, as all of them are returning articles. The documentation is confusing as it e.g. states “Get keyword by ID”, although the API call returns articles and not keywords.
documentation of the id parameter is missing. This parameter is part of the path, and not part of the query but nevertheless it is a parameter.
what are the maximum values for limit and offset?

Conclusion

Both from private explorations as well as from my work at Meltwater - who among others offers media monitoring solutions - I know that the relationship between content providers and companies that offer services around this content can be difficult.

Also for this reason I hope that the ZEIT will reap many benefits from launching their Content API. This kind of openness will eventually benefit the API owner, the industry around it, and last but not least the content consumers i.e. readers. I am looking forward to the increased dialogue between publishers and developers and all the good things that will come out of that.

So, ZEIT ONLINE, good luck with the API :)

Guest Blog Post at the Yahoo Developer Network

2012-11-06T00:00:00+00:00

I am very excited that Yahoo gave me the opportunity to write a guest blog post about YQL at the Yahoo Developer Network. The post I cam up with is called YQL – The Swiss army knife of web services.

Also check out the geeky graphic they came up with (Photo by by Jinho.Jung).

Tesseract for Zugmonitor

2012-03-26T00:00:00+00:00

This demo applies the concept from Tesseract Airline on-time performance to the train data of Zugmonitor. Zugmonitor is an application that provides data about delays in the Germany Railways. The coordinated visualizations (built with D3) show nearly 60,000 train connections from early 2012 (data courtesy of OpenDataCity).

Writing for ProgrammableWeb

2012-02-03T00:00:00+00:00

Since January I am writing for ProgrammableWeb in my spare time. Although my main area of experience is the Tech industry, I have always enjoyed writing, so publishing articles at ProgrammableWeb gives me the opportunity to expose the content I write to a much larger audience, and get into discussions with a lot of people about topics that I am very interested in.

The first two articles I wrote for ProgrammableWeb are:

On the flip side this also means that I will have less time to write articles on spier.hu. Instead I will regularly link to my articles from here that get published on ProgrammableWeb.

Really looking forward to this new experience!

API Console with ioDocs

2011-10-23T00:00:00+00:00

Finally I had a chance to play around with Mashery’s ioDocs, which Mashery has open sourced some months back. ioDocs has a couple of prerequisites, namely Node.js, npm (Node package manager), and Redis (you can find full installation instructions on the ioDocs github project). Nevertheless, I struggled through the installation process, as I wanted to see how it compares with the API Console from Apigee which I wrote about a week ago.

I chose the USA TODAY Articles API as my guinea pig for this experiment because just yesterday I wrote a set of YQL tables for that API, so I already knew the API documentation pretty well.

After changing a couple of configuration settings (in JSON files) I had a new API Console running in ioDocs. It took me a bit of experimenting to get things running but all in all maybe 30 minutes, which is not bad at all. The image below shows how my ioDocs for the USA TODAY Articles API ended up.

Here a couple of notes I have about ioDocs:

plus
- The UI of ioDocs is really slick, thumbs up! I especially like that there is plenty of room for documentation. Bonus points for the ability to provide pre-populated drop downs via EnumeratedList":["json", "rss"]. Very cool!
- I am very surprised that USA TODAY does not have ioDocs on there Developer page. It would make experimenting with their API so much easier. So USA TODAY, if you are listening: Go and set up ioDocs!
minus
- ioDocs requires the description of an API in a JSON configuration file. I believe that choosing a standard for this purpose (like WADL or WSDL 2.0) would have been better). Although I know that Apigee is competition for Mashery, so there might be other motives here to do things differently :)
- I don’t think that the expressiveness of the current configuration files is high enough to model all APIs. Even some very commonly used URL patterns are not possible e.g. if I had an API that would require the replacement of the parameter section in the URL path like this http://someapi.com/v1/stories/{section}/?date=2011, then I don’t think that ioDocs would currently make this possible.
- documentation is key for anything that is new! Currently the documentation of ioDocs on github is a bit tricky to ready.

All in all, I like ioDocs!

Suggestion: In my mind a mixture of the advantages of Apigee To-Go and ioDocs would make for the perfect API Console.

Code snippet of the ioDocs configuration file

"MethodName":"Search",
"Synopsis":"",
"HTTPMethod":"GET",
"URI":"/articles",
"RequiresOAuth":"N",
"parameters":[
   {
			"Name":"encoding",
			"Required":"N",
			"Default":"json",
			"Type":"enumerated",
			"EnumeratedList":["json", "rss"],
			"Description":"Format in which to return the results. Either 'json' or 'rss'."
   },
   {
			"Name":"search",
			"Required":"N",
			"Default":"",
			"Type":"",
			"Description":"The search string."
   },
	 ...
]

Apigee Console for the RubyGems API

2011-10-15T00:00:00+00:00

I had heard of apigee.com before but I had not looked at their services yet. Today I took a closer look and I discovered Apigee To-Go, a way to create a simple API console for any API (including your own). After creating such an API console, you can embed it elsewhere e.g. in your the developer section of your website.

In order to experiment with Apigee To-Go I needed a guinea pig. My choice fell on the RubyGems.org API. I have not configured the console to support all API calls of RubyGems. Instead the console below only supports the calls for which no authorization is needed. Adding authorization and the corresponding HTTP verbs (POST, PUT, DELETE) would not be that difficult though.

Console for the RubyGems.org API

How to create an Apigee To-Go Console

register at apigee.com
create a WADL file (with some additional apigee-specific extensions)
upload this file to apigee
apigee will automatically create the API Console
Done!

I had the first console for just one API endpoint up and running after 5 minutes. After that it took a couple of trial and error iterations, until the console that you see above was done. apigee.com does a decent job in informing you about syntax errors in your WADL file, which helps you to fix them. (I should say though that I have worked with WADL before.)

Summary

Apigee To-Go is a pretty nice way for developers to get started, when trying to integrate with a new API. A couple of areas of Apigee To-Go could be improved but it is definitely good enough to be useable.

From a developer perspective such a console is mainly useful when trying to find out how an API works in general. For later stages of the development there is still no way around reading the full API documentation.

Op Art à la Victor Vasarely with CSS

2011-09-18T00:00:00+00:00

I recently saw the exhibition of Victor Vasarely, one of the founders of Op Art (Optical Art). In the pieces that eventually made him famous, he frequently drew different geometrical figures to create interesting visual effects. See one simple but still interesting piece called Tlinko from 1955 below.

Besides being impressed by his art, seeing his pictures made me wonder if it would be possible to re-create some of his pictures with CSS. I don’t know much about CSS but this sounds like a good chance to learn more about it’s boundaries.

The pattern-like structure reminded me of the tool patternify.com which I had seen previously but not tried out. With patternify I managed to create the below CSS code, which creates the following pattern.

<div 
	style="
	margin: auto; 
	width:220px; height:260px;
	background:url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAYAAACNMs+9AAAAIElEQVQYGWNkYGD4D8TYACOyIBMyBx97KCjE5/7BIgcApGEBDCu0cukAAAAASUVORK5CYII=) repeat;">
</div>

Not quite there yet, right? I would need a way now to intermix the standard pattern of the small black squares, with some squares that are rotated, squeezed, etc. Just I have absolutely no idea how to approach this!

For some additional inspiration, here you can see more of Vasarely’s art in black & white at Google. Maybe some expert CSS folks will find this post and can give me pointers towards whether this is doable at all with CSS. To finish off this brief description of the idea, here a quote from Vasarely himself. I am pretty sure he was not talking about CSS :)

“I have faith in the advent of a polychrome and multidimensional technological art that will radiate as the first purely visual language void of any anecdotic or literary references.”

** - Victor Vasarely (1950) -**

Guest Blog Post at PeerIndex

2011-08-13T00:00:00+00:00

Recently I wrote a guest blog post for PeerIndex, titled YQL-ing the PeerIndex API. It was a very interesting experience and it made me rethink why I like to write in general and specifically for the web. These are my thoughts on the topic of producing content for the WWW, driven by the example of the mentioned blog post.

Do something and talk about it

I used to think that people who frequently talk about what they do are just egocentric show-offs, who actually don’t really know anything. While I still generally feel that way, I made a couple of realization some time ago:

If normal people (like you and I) don’t talk about what they do, then the content on the Internet will definitely be dominated by the products of nonsensical spammers, “social media experts”, SEO cheaters, and other people of similar behavior that don’t produce value. I cannot allow this to happen, so as I consider myself a normal person, I need to contribute to useful content out here!
You need to expose yourself in order to be found. With that I mean that even if you are doing great stuff, there is a certain amount of marketing involved to enable other people to find you. How good is something really if nobody knows about it? So it is a necessity to talk about it, like it or not.
People are very tolerant when it comes to ignoring content that they are not interested in. I am in no way suggesting that you should become a spammer but there is no harm in producing a tweet, blog post, or other content from time to time. If people are not interested then they won’t be disturbed by it either.

For example the aforementioned blog post only happened because I am interested in YQL and I played around with YQL and the PeerIndex API and then posted my experiments on Twitter. This is how the PeerIndex team found me and later on asked me to write the post YQL-ing the PeerIndex API. Nothing magic about it, talk about what you do and interested people will listen.

Don’t become a Spammer

Yes, I mean you! Spammer! Don’t spam! :)

In response to my PeerIndex blog post, I saw some tweets (see below) that were so strange that I had a hard time believing that these things really exist.

So that’s what people do? For real? They pick up random tweets and then RT them from multiple accounts? For what? Are there even tools that support this kind of activity or what? Anybody knows? I really don’t get it.

Don’t worry about SEO but care about Analytics

When you start producing content, don’t worry about how to improve your ranking in the search engine rankings. If you produce valuable content, people will find you. You can still try to optimize things later on when you realize that there is a general interest in what you do but please don’t try to become a “SEO expert” before you even start - you can probably imagine how much I disapprove this term due to its frequent misuse. Posting a link to your content on Twitter is probably a good enough start to find some audience.

You can still do little things in order to find out a bit about your audience. For example if you are using bit.ly to share your links, you should check out the bitly Info Page+ for the content your produce or share. You can check out the bitly Info Page+ for my blog post as an example. This can give useful insights into which of your activities generate more interest/traction.

Contribute, contribute, contribute

Do something, and give back where you can. While this is an easier thing to do for developers due to the abundance of Open Source projects that we can contribute to, I believe it is also possible for people that don’t write software.

Give useful feedback to other people. Don’t write “that’s great” only. You can of course do that because some feedback is better than no feedback but at the end of the day, don’t we all want a real discussion about the topics that we are interested in? Instead of reading 10 x “That’s great”, wouldn’t you rather want to read one constructive comment/question? I certainly favor the latter.

So in summary. Do your thing. Talk about it. This is not for eternity, content comes and goes. Most importantly, don’t worry about what other people might think about your content. If you are neither a spammer nor a troll, then there is nothing to loose.

Springer Quotes

2011-08-12T00:00:00+00:00

With this application I won the 1. Prize at the Springer API Challenge in 2011. It was a great learning experience in terms of working with heroku, ruby, and the Springer API.

Springer Quotes let’s you search more than 80 thousand articles from Springer’s open access journals. When you find a particularly interesting article you can create quotes from paragraphs or individual sentences. These quotes can be used to point fellow resources to specific sections of articles or to reference these quotes within your own scientific work.

Was running at http://springerquotes.heroku.com, now offline.

Springer Quotes is programmableweb's Mashup of the Day

2011-08-12T00:00:00+00:00

Nice. The awesome folks from programmableweb have selected my application Springer Quotes as their Mashup of the Day. Loving it, thanks guys!

What did I win? :)

Making YQL Tables an API Query Wizard

2011-07-25T00:00:00+00:00

This idea is sitting in my draft box for some time now. The idea I am proposing:

Take a YQL table and automatically transform it into an API Query Wizard. This Wizard is a web interface that is used for querying and documenting the API wrapped by this YQL table.

So why did I get back to this idea just now? Today Mashey launched an interesting new offering called Mashery I/O Docs, which they call “the next generation of API documentation”. I recommend watching the introductory video over at their site. See below how I/O Docs looks like in action by the example of the I/O Docs for the Klout API. I think it looks pretty neat, as they do a good job in bringing together the input parameters for the API calls with their corresponding documentation.

Mashery I/O Docs for the Klout API

Now all of you who know YQL might say: “Hey, this looks much like the YQL console.” Well I would say yes and no (see screenshots of the YQl console further below). While the YQL console does a decent job in letting you create API calls without too much typing, it also has a couple of shortcomings.

These shortcomings are mainly:

too strong focus on the SQL-like typing mode instead of the Query Wizard (see below)
no documentation for the individual API parameters
visual separation of Query Wizard and data generated by the API call

YQL Console - Query Wizard

YQL Console - Results

So now back to my idea. Transforming a YQL table into an API Query Wizard. Without taking anything away from the wonderful job that Mashery did with I/O Docs, it is not all that hard, is it? All that it takes is a somewhat appealing UI, syntax highlighting for the API output (mainly JSON/XML) and of course proper documentation of the different API endpoints and query parameters.

From a technical standpoint that would be reasonably easy. I would probably use XSLT to automatically transform the XML of the YQL table into the HTML for the Wizard. Then one would be able to just point the web browser to any given YQL table on the internet, which would then get displayed as an API Query Wizard to the underlying API.

So YAHOO! / YQL Team, why not just do this? It might take a couple of minor changes to your XML schema, mainly to allow for the ability to add description to the different input parameters. Also grouping multiple YQL tables into one wizard, like Mashery does it, would be a bit trickier but still possible. I guess you can still pull it off, it is not too late :)

Robots are Fun

2011-07-09T00:00:00+00:00

Did you see ROBOHASH? I found them on Twitter, where else. Anyways, the services lets you create unique, robot images for any text you throw at it. This is so much fun, I needed to add even more randomness. So I threw together a bit of jQuery and JavaScript that adds a robot here and there in my text, generated from the paragraphs of this text itself. So if the text you just read was a robot, it would look like this:

Hello Mr. Robot! Check the source code of this page for the full code. The main JavaScript needed is this:

$('div#post p').each(function(index) {
	// only every second paragraph gets a robot
	if (index % 2 == 0) {
		// choose a set of robots
		var set_name = "set" + rand(1,3);
		// fixed size of robot image
		var size = "150x150";
		
		// put together the image URL
		var img_url = "http://robohash.org/" + escape($(this).text()) + ".png";
		img_url += "?set=" + set_name;
		img_url += "&size=" + size;
		
		// some information about the robot
		var img_alt_text = "I am a " + size + "px robot from family " + set_name + " at http://robohash.org.";

		// append the robot image to this paragraph
		$(this).append("<img src='" + img_url  + "' title='" + img_alt_text + "'/>");
	}
});

So now you will see a new set of robots appear every time you reload this page. Try it out. (Actually what you see is a random tuple out of 27 possible permutations to be more precise but this is just nerd talk.)

Why? Well first of all it is Saturday … and then most importantly this was fun. The same question could be asked for ROBOHASH as well. Why do they do what they do? They give some hints about the Why on their website but how does that really matter? It is the Internet, people do crazy things all the time and some of these things are a lot of fun :)

If you are still not happy without the Why, you can ask the author directly at @Robohash.

I have a dream: Always Fly on Time

2011-07-05T00:00:00+00:00

“Scratch your own itch” … this problem clearly falls into this category. I am living in Budapest and take frequent flights to Germany. easyJet is one of the carriers that happens to fly to the places I want to go to. While they have extremely competitive prices - yes, they are cheap - they have also some issues with scheduling their flights or whatever else but fore sure they are regularly a bit late with their departures.

It is hard to put exact numbers to this but I estimate that about 30% of their flights that I have used are late. The waiting times vary between 30 minutes to 2 hours (to be fair to them, the 2 hours were a one time only incident). Waiting for 30 minutes is quite ok you might say. True, it ain’t that much. Still, if I had the choice, I would rather not wait that extra 30, especially because I usually travel light, so I can afford to be at the airport pretty late.

Now the idea:

Track the expected delay of a plane and make this information easily accessible. (I don’t want to have to go to the website of the different airports or similar.)

I already found some websites that seem to be doing something similar, or at least provide data necessary for this:

flightradar24.com shows live airplane traffic from different parts around the world on a Google Map
data.flight24.com claims to be the largest online aviation database. One can search for airplanes, flights or airports. They belong to flighradar24.com.
flightstats.com delivers real-time and historical flight information. They have an API as a premium service but that one is pretty expensive.

I don’t have time to search further right now but I already put this out there, so that maybe somebody can point me to further resources about flight data.

Just one more note: flightstats.com also has widgets for embedding in your website. I am using the departure widget for Budapest airport as an example. Pretty neat. (Select another Time Period if you see no flights)

I won the Springer API Challenge 1.0

2011-07-01T00:00:00+00:00

It is official! I won the Springer API Challenge 1.0 with my pet project Springer Quotes. Wohoooo!:)

So go check out Springer Quotes and let me know what you think.

I want to give credit where credit is due. I specially want to thank these two people for their prior work and support:

Michael Donohoe for his awesome Emphasis Library that I adapted and reimplemented in Ruby to highlight the quotes
Pascal Bertram for the logo artwork

In addition, these are the great technologies and open source libraries that I have used to build Springer Quotes:

heroku for hosting this site
CouchDB for the data storage
Ruby and Sinatra for the implementation and server DSL
bit.ly for shortening the links
Nokogiri + XSLT for the conversion of Springer's A++ format into somewhat more readable HTML.
MathJax for beautifying mathematic formulas
'Sticky' buttons at the top of the article
Pure CSS speech bubbles for the nice speech bubbles you see on the /quotes page
jQuery as my JavaScript library of choice

Open Source Software is the thing! Build it, use it, contribute to it!

YQL Tables for Social Media Influence Rankings

2011-06-29T00:00:00+00:00

While evaluating different available services for “Social Media Influence Ranking”, I also implemented YQL data tables for them. The services I looked at were PeerIndex, Klout, and Topsy. Implementing the YQL tables has these two major benefits for me:

When creating the open data table, you have to take a detailed look at the API you are mapping to, and hence read the documentation very thoroughly. That in itself is a good exercise to really understand the nature of the input and output data.
For assessing data returned by an API, YQL is my absolute go-to-tool. It makes it really easy to do a lot of ad hoc queries and play with the returned data. Also feeding data from one API to the other is ist trivial. This way the exploration of the data is less time consuming and does not require any scripting.

PeerIndex

I started with PeerIndex. That one was pretty easy, as they have only one API endpoint so far. Also had some good twitter communication with CEO Azeem Azhar. Twitter is just awesome for this kind of stuff - in case you did not know - you just have to love it :)

So get yourself a PeerIndex API key and then try it out in the YQL Console.

A sample YQL query for PeerIndex looks like this:

// enter your API key here !!!
SET api_key="YOUR_API_KEY" ON peerindex;
SELECT * FROM peerindex.profile WHERE id='sebastianspier';

Klout

Klout has only a handful of API points, so creating the YQl tables for them did not take too much time either. You can query information about a user’s Klout score but also who he is influencing and who is influencing him.

So get yourself a Klout API key and then try it out in the YQL Console.

A sample YQL query for Klout looks like this:

// enter your API key here !!!
SET api_key="YOUR_API_KEY" ON klout;
SELECT * FROM klout.user.show WHERE users='sebastianspier'

Topsy

Topsy has way more API endpoints, so I ended up with a total of YQL 17 tables. Multiple of them have a fairly repetitive structure though, which makes using them more comfortable.

Topsy does not require an API key, so you can try it out in the YQL Console right away.

A sample YQL query for PeerIndex looks like this:

SELECT * FROM topsy.authorinfo WHERE url="http://twitter.com/barackobama"

Contributing to the YQL Community

I sent pull request for the YQL tables for all three services to the official YQL Community Tables repository on github. The PeerIndex tables already got included, which makes them available for a broader audience. Integration of Klout and Topsy are pending.

So have fun with YQL and your experiments with Social Media Influence Rankings.

Your Social Media Influence in Numbers

2011-06-28T00:00:00+00:00

More and more services are spawning these days that are claiming to measure your “social media influence”. Here I am presenting an overview of the services Klout, PeerIndex, Topsy, and Tweet Grader (plus a short bonus round with Twinfluence).

If this topic interests you, you may also want to read 3 Ways Other Than Klout to Measure Influence and The Landscape: SME’s List Of Influencer Identification Tools for an even broader (but less detailed) overview of the tools for measuring influence.

Details

Wherever possible I am using my own twitter handle “sebastianspier” to show how these services work.

##Klout

Even when you are not a Klout user but are using Twitter, you will probably already have a profile on Klout e.g. this is my Klout profile. You can see your overall Klout Score, the Klout Class assigned to you (in my case “Conversationalist”), and some people that you are apparently influencing. To get more information, you need to sign in with either your Twitter of Facebook login. (I have not done this, so cannot make any statements about the functionality gained by doing so.)

Klout - with their confident tag line “the Standard for Influence” - claims to measure over 35 variables on Facebook and Twitter for each profile. Of course it can only merge the values for Facebook and Twitter into one calculation if you have logged in into both from the Klout website. Besides the prominent Klout Score, three further metrics are measured. The Klout documentation defines them as follows:

Klout Score is a representation of how successful a person is at engaging their audience and how big of an impact their messages have on people. The scores range from 1 to 100 with higher scores representing a wider and stronger sphere of influence.
True Reach is the size of your engaged audience and is based on those of your followers and friends who actively listen and react to your messages.
Amplification Score is the likelihood that your messages will generate actions (retweets, @messages, likes and comments) and is on a scale of 1 to 100.
Network Score indicates how influential your engaged audience is and is also on a scale from 1 to 100.

The dataset returned by the Klout API looks like this:

<klout>
    <twitter_id>96320763</twitter_id>
    <twitter_screen_name>sebastianspier</twitter_screen_name>
    <score>
        <kscore>29.87</kscore>
        <slope>0.18</slope>
        <description>is influential to a tightly formed network that is growing larger</description>
        <kclass_id>0</kclass_id>
        <kclass>Conversationalist</kclass>
        <kclass_description>You love to connect and always have the inside scoop. Good conversation is not just a skill, it's an art. You might not know it, but when you are witty, your followers hang on every word.</kclass_description>
        <kscore_description>is influential to a tightly formed network that is growing larger</kscore_description>
        <network_score>39.32</network_score>
        <amplification_score>14.2</amplification_score>
        <true_reach>36</true_reach>
        <delta_1day>0.75</delta_1day>
        <delta_5day>0.87</delta_5day>
    </score>
</klout>

##PeerIndex

PeerIndex describes its score as “a relative measure of your online authority. This score reflects the impact of your online activities, and the extent to which you have built up social and reputational capital on the web.”

According to the PeerIndex documentation they use these metrics in order to calculate the PeerIndex Score:

Authority is the measure of trust; how much can you rely on that person’s recommendations and opinion on a given topic.
Audience is indication of your reach. It is not simply determined by the number of people who follow you, but instead generate from the number of people who listen and are receptive to what you are saying.
Activity is the measure of how much you do that is related to the topic area. Being to active and people will stop listening to you and if you are too inactive people will never know to listen to you. The Activity Score takes into account this behaviour.

The dataset returned by the PeerIndex API looks like this:

<peerindex>
    <name>Sebastian Spier</name>
    <twitter>sebastianspier</twitter>
    <slug>sebastianspier</slug>
    <known>1</known>
    <authority>38</authority>
    <activity>24</activity>
    <audience>30</audience>
    <peerindex>36</peerindex>
    <url>http://pi.mu/i2VL</url>
    <topics>universal mobile telecommunications system</topics>
    <topics>science</topics>
</peerindex>

Topsy

Topsy is a real-time search search engine and rather has ranking functionality as an auxiliary service among others. On 2010/08/24 Topsy stated that they are “now the largest searchable index of content posted on Twitter” with more than 5 billion tweets and 2.5 billion links in their index. In their article Influence: The Social Web Currency the Topsy team explain their definition of influence:

Roughly the top 0.2% most influential of all Twitter users are tagged “Highly Influential”, and “Influential” tags appear for the top 0.5% […]. So if Topsy says you are influential, you are part of a pretty small group!

Influence: The Social Web Currency

Besides their search service, I found the analytics functionality most interesting. For example you can check how close the Topsy Race between Dave Winer and me was (to spoil the fun: I lost!!!)

The dataset returned by the Topsy API when querying their authorinfo endpoint looks like this:

<topsy>
    <name>Sebastian Spier</name>
    <url>http://twitter.com/sebastianspier</url>
    <type>twitter</type>
    <nick>sebastianspier</nick>
    <description>MD R&amp;D: Meltwater Group. Tweets are my own.&#xd;
#tech #ruby #yql #search #noSQL #BigData #basketball &#xd;
Learning by doing!</description>
    <topsy_author_url>http://topsy.com/twitter/sebastianspier</topsy_author_url>
    <influence_level>0</influence_level>
</topsy>

Tweet Grader

The last post from the graderblog is from Nov 17, 2009, so I am not so sure how active Tweet Grader (aka Twitter Grader) is currently being developed. For the sake of comparison to the other services I am still listing them here.

What Twitter Grader is trying to measure is the power, reach and authority of a twitter account. In other words, when you tweet, what kind of an impact does it have?

Chris Keller, 2009-11-17, How Does Twitter Grader Calculate Twitter Rankings?

In his blog post Chris Keller of Tweet Grader describes the different factors that go into determining a users grade. These are Number of Followers, Power of Followers, Updates, Update Recency, Follower/Following Ratio, and Engagement. Please refer to the blog post for more details.

One final note: I don’t have a lot of confidence in the grade that Tweet Grader produces. I compared my own grade against the grade of an account that had not posted a single tweet but more followers. You already guessed it: My grade was lower. Together with my impression that Tweet Grader looks somewhat unmaintained right now, I don’t recommend Tweet Grader.

Also Twitter Grader did not answer my request for API access in time, so I cannot post a data sample from them here.

Last and Least: Twinfluence

The domain twinfluence.com is redirected to the profile of founder Guy Hagen and the last message on their twitter account is from 9 Nov 2009. So I guess it is fair to assume that Twinfluence is dead, which is confirmed by the tweet above by Guy Hagen.

Why Fork the YQL Community Tables?

2011-06-22T00:00:00+00:00

The question of this post is: Why does it make sense to have a personal fork of YQL’s Community Open Data Tables, short Community Tables?

So what again are these Open Data Tables?

Open Data Tables are XML files that can be “plugged” into the Yahoo! Query Language open platform (YQL). > These files describe how the YQL SQL-like language can be mapped onto any web service or source on internet.

Source: YQL’s Community Open Data Tables

Yahoo obviously encourages developers to contribute their own tables to the Community Tables, more specifically to the github repository. That makes perfect sense and is just how it should be. One the other hand this can be confusing for people that are starting with YQL and are searching for YQL on github, as they will find multiple forks of the Community Tables as well. Actually that is exactly what happened to me when I was looking into YQL the first time.

Some reasons for forking

It is just so easy the do a fork with github, so people do it, even if they might not need to.
People that want to have their own tables pulled into the Community Tables need to have a fork of the main repo, so that they can send a pull request via github.
Having a personal fork of the tables also has the advantage that you can keep them in a “known state”.

Why do I have a personal fork?

Initially I kept my personal fork of the Community Tables because I fixed some bugs and wanted to push my changes back to the community. Also this was the best playground to get some hands-on experience with how the creation of open data tables works. That’s how it started.

Also my own fork gives me more control over which updates I add and when I do it. This actually applies to doing updates faster or slower than they are done in the Community tables. Meaning sometimes I want to get changes merged into my own fork faster, than the central Community Tables does it. On the other hand there are also situations where I might want to keep an older version of an open data table because I have an application that uses exactly this version of the table.

The latter is actually an issue in it’s own right. YQL’s open data tables are not versioned, so they do not have an own version number themselves. They normally do map to a specific version of an API by pointing to the API endpoint that contains a specific version number. The issue is though that pretty much everything in this table could be changed without you noticing it. I will not get deeper into this issue but it is a question that has been lingering in my head for quite a while now, so if I should eventually come to a conclusion about this I might move it into an additional post.

Finally an own fork also is a good place to share extra information with others. e.g. I have added a README to me fork, which contains for example a list of issues that I have encountered with YQL that might have been solved by others already who could give me input.

Finally, sharing is everything. So if you are a YQL developer, make sure you push your updates back to the github repository of the Community Tables from time to time.

Adding Some Emphasis

2011-06-16T00:00:00+00:00

Being able to link to an article on this site is pretty cool. But there are rare occasions where it might be useful for me to be able to link to individual paragraphs or even highlight certain sentences. Some sort of “deep links” basically, just with a twisted meaning of the phrase.

This is a concept I remember being described by Dave Winer (please excuse me, I cannot find the link to the article:) ). Not sure if he was the first to do so but that does not really matter after all either. What I don’t like that much about Winer’s implementation of these deep links is that they clutter his articles with #-characters after each paragraph.

An implementation of this that I like much better has been done by Michael Donohoe for the New York Times. He implemented a javascript library called Emphasis, the source code is on github. It is pretty much a plug-and-play solution, you just have to specify with an CSS selector which paragraphs from your articles should be selectable.

How-To

Just Double-tap the Shift key and paragraph symbols ¶ will appear next to each paragraph. Click any paragraph symbol to generate a link to that paragraph, or click on sentences to highlight them. The link will appear in your browser’s address bar and you can use these modified URLs for sharing. Pretty cool!

Demo

For example you can open this article with a section being highlighted. (needs to open in a new window for it to work. sorry!) Check the URL!

I have already used this in a recent pet project springerquotes that I will probably write about more soon.

API Health Checker with YQL

2011-06-15T00:00:00+00:00

Wouln’t it be possible to build an API Health Checker with YQL?

Don’t know what YQL is? It’s the thing! I love it. Go checkout the excellent YQL documentation.

Now that you know what YQL is, you need to learn about the open data tables as well, also called community tables. They open source, every body can contribute to them, the main repository is hosted at github.

With the combination of YQL and the open data tables I am imagining the following to be possible:

One could implement a test suite, much like a continuous integration test, that runs over all YQL tables and sends sample queries to all these APIs. Then you compare the returned results to a predefined expected value - much like an assertion in a Unit Test. This way one could check if the APIs are still operating and if the mapping created by the YQL table is correct.

Building that into the YQL table DTD would result in something like this:

<meta>
	...
	<sampleQuery>
		<query>SELECT * FROM {table} WHERE id="123"</query>
		<expectedResult>SOME_JSON_STRING_HERE</expectedResult>
	</sampleQuery>
</meta>

Challenges

An incomplete list of the known challenges from the top of my head:

Many APIs require an API key. Of course one could get an API key for all these APIs but we are talking about a lot of APIs here and managing all these keys does not sound like fun (not to mention that they have to renewed etc)
Most API results are not static (Why would you call the API if you always get the exact same data anyways, right?) One could still map parts of the results against some known values but that becomes more complicated then.

So, API builders, YQL lovers, tech heads. What do you think? Is this possible, and is this a good idea at all?

Why Start from Scratch?

2011-06-14T00:00:00+00:00

Seriously! Why would I want to start a new blog? After all I already one one at blog.airness.de. So why start from scratch?

In short: I need a place where I can easily dump ideas and where people that are interested in stuff like that can find it.

If you are interested in the long story, read on …

Prior Blogging

For my prior blog at blog.airness.de I used wordpress. That worked well for me for quite a while. I even ventured a bit deeper by writing some small plugins for it. Just eventually my blog became bloated with too many sidebars and thereby also quite slow. Worst of all, everything distracts from the main thing, which is the content.

I also ended up with two handful of drafts piling up that I never finished. Of course I mainly have to blame that on myself for being lazy but I also felt that in parts the process of posting and writing was too clumsy for me.

Then I ventured into drupal for a longer while. I really loved it at still do. At some point I will for sure use it for a bigger project but for what I want to do here, drupal has just too much power. It’s like taking a sledgehammer to crack a nut, just toooooo much.

Geek? Geek!

I am a geek by heart. Took me a while to admit that but I guess it had to be said :) I love technology and trying out new stuff. So why not try jekyll?

It is plain, clean and easy. I write my posts in plain text files (that I use for all kinds of stuff anyways), and I can use markdown syntax for the formatting! Nice!

To top things, I can host the whole thing at GitHub Pages, pushing new post via git. Even nicer!

Almost Like a Longer Tweet

I find myself using twitter fairly frequently lately. It is quite a useful tool for a bunch of different purposes but I have to agree with Dave Winer when it comes to the arbitrary 140 character limit: It is useless. Sometimes you just need to say a little bit more :)

So for all things that just break the 140 characters by a bit, I will be using this blog as well, and then point people to my posts from twitter, if I consider them relevant enough.

Idea Curation

We all have ideas, all the time. Most of them pass and for most of them it is good that way. Some others stick around for a bit longer.

Now I will finally have a place where I can collect them to get input from others wrt their usefulness and support in implementing them.

Inspiration

This blog is a fork of Zach Holdman’s Blog. As you see I even kept his design because I am too lazy to change things and I just want something to get started. If you should ever find this Zach, bare with me. It’s open source after all, right? :)

I also kind of like the idea of the timeless repository. Maybe I will morph jekyll into behaving like that one day, who knows.

For now, this is just a start. I will see where this leads.

World Bank Competition - YQL Tables

2010-10-23T00:00:00+00:00

During the Apps for Development contest of the World Bank I created this YQL datatables to make working with the World Bank Data API easier. The sources are hosted at github and I also contributed them to the official YQL Community Tables.

My thoughts on MEST and explaining “the dis tin”

2010-03-09T00:00:00+00:00

Note: Cross posting from Meltwater Blog (09.03.2010)

“The dis tin,” or “the this thing”, is a phrase frequently used in Ghana to describe something that either is difficult to describe, does not have a name, or a name that the speaker has just momentary forgotten.

(Disclaimer: Please forgive me, all Ghanaians if I am translating this somewhat incorrectly.)

In 2008, I worked at Meltwater Entrepreneurial School of Technology (MEST) as a Teaching Fellow – more on that later. In January of 2010 I was fortunate enough to go back to Ghana and attend the graduation of the first class of trainees.

So what is “the dis tin” that is MEST? What does it mean to me, to Meltwater Group, to our employees, and most importantly, to the trainees in the program? Since joining MEST as a Teaching Fellow in 2008, many people have asked me these questions.

MEST is Meltwater’s non-profit program launched in Accra, Ghana two years ago. MEST follows the Meltwater Group’s deeply rooted belief that “talent is talent everywhere” and that when “talented people receive the right amount of support, guidance and encouragement, they can accomplish extraordinary things” as our CEO Jorn Lyseggen once put it.

MEST’s mission is to train young Ghanaians in order to enable them to start their own globally successful software companies.

My work at MEST

A Teaching Fellow at MEST is a hybrid between a teaching assistant at university, a coach, a technology expert and a mentor … so pretty much a jack-of-all-trades ☺

In my time at MEST my main focus was to strengthen the trainees’ understanding of current technologies and how they can be successfully leveraged for the trainees’ respective business ideas. The depth in which the different technologies were addressed was really diverse since the trainees all came from varied academic backgrounds (but all hold a bachelor’s degree). Therefore the level of support a trainee needed often was decided on a case by case basis.

Generally speaking, my experience as a Teaching Fellow was a very enjoyable one for me. Besides having the chance to work in a new cultural context and being able to broaden my horizons both professionally as well as personally, I developed a lot of new friendships with other MEST staff and of course with the trainees themselves.

My impressions when going back to MEST

Going back this January to see how MEST and the trainees had developed in the last year and also to celebrate the graduation of the first class of trainees was a great experience. I met a lot of my old friends, as former Teaching Fellows like myself flew in from all over the world to be part of this truly unique and historic occasion.

It was exciting to see how the trainees have developed. The group of eleven successful graduates have built up incredible entrepreneurial spirits and technological skills over the course of the last two years during their training at MEST. I am sure that successful businesses will be started by them one day. They may not necessarily all be in the global marketplace, but will achieve their own individual definition of “being successful”.

One of the teams was accepted into the MEST Incubator, which is phase two of the MEST program that helps MEST graduates validate their ideas and get their new businesses off the ground. Supplied with initial funding, they are already developing the first prototype streemio.com and hope to go live with it in the middle of this year. Follow Streemio closely in order not to miss what they are up to.

I am looking forward to seeing all the great talent and ideas that will come out of MEST in the future. To the trainees, I have to say: You guys rock, go for it!

So I hope I was able to tell you a little more about “the dis tin” called MEST.

In conclusion, I am proud to have been part of MEST. “Hats off!” to the Meltwater Foundation and especially to Jorn for staying true to the unique philanthropic character of MEST and for bringing to life such an extraordinary example of corporate responsibility.

Briefling

2010-01-01T00:00:00+00:00

Briefling ist a project I did together with my friend Pascal. briefling.de makes it very easy to create professionally layouted letters in PDF format. They can be directly created on the website be filling out the required data in the forms. The PDFs can then be stored locally. In the future we might extent the functionality to enable the posting of letters directly from our service.

Auf schnellem Weg nach Ghana

2008-10-13T00:00:00+00:00

Note: Cross posting from TU intern (13.10.2008)

TU-Alumnus Sebastian Spier arbeitet als Teaching Fellow in Ghana

Sebastian Spier (l.) an der Meltwater Entrepreneurial School of Technology in Ghana

Nachdem Sebastian Spier im vergangenen Jahr sein Informatikstudium an der TU Berlin abschloss, ahnte er noch nichts davon, dass er sich nur wenige Monate später in Richtung Ghana aufmachen würde. “Gerade noch zu kalt in Deutschland und auf einmal schon zu heiß in Ghana”, beschreibt er diesen schnellen Wechsel. Wie es dazu kam und womit sich Sebastian Spier heute in Ghana beschäftigt, hat er für TU intern aufgeschrieben.

Ich suchte nach dem Studienende einen Weg, meine technische Ausbildung und mein Interesse für die Lehre mit einem Auslandsaufenthalt zu verbinden, und besuchte mit diesem Ziel im Herbst 2007 die Firmenkontaktmesse „bonding“. So kam ich zu der IT-Firma Meltwater, die ein nahezu maßgeschneidertes Angebot für mich parat hatte: “Teaching Fellow” in der Meltwater Entrepreneurial School of Technology (MEST) in Ghana - und das Ganze schon ab Januar 2008. Eine Bewerbung, drei Interviews und vier Wochen später hatte ich eine Zusage. Natürlich hatte ich die gleichen Zweifel, die jeder hat, wenn er an Ghana denkt. Afrika … wie ist das da eigentlich? Nachdem ich mich über Land und Leute informiert hatte, habe ich den Rest auf mich zukommen lassen.

Was genau ist nun MEST? Es ist ein Non-Profit-Projekt, das vollständig von der Meltwater Group entwickelt, finanziert und umgesetzt wird. Ziel des Projektes ist es, junge ghanaische Absolventen im Bereich Softwareentwicklung und Entrepreneurship auszubilden. Wir begleiten sie dabei auf ihrem Weg, internationale Software-Unternehmen aufzubauen, um somit eine nachhaltige Entwicklung der Softwarebranche in Afrika zu unterstützen. Softwareentwicklung ist einer der größten, wenn nicht der größte Wachstumszweig der Informatik in den letzten zehn Jahren. Zur Entwicklung eines Programms bedarf es keiner großen Investition, sondern lediglich eines Computers, den es bereits für einige 100 Euro zu kaufen gibt. Gerade dies macht Gründungen im Softwarebereich für Länder wie Ghana so interessant.

Wir vermitteln den Teilnehmern nicht nur die Hard Skills sondern vor allem auch die Soft Skills, die den erfolgreichen Geschäftsmann ausmachen. Die Tätigkeit als Teaching Fellow ist vergleichbar mit einer Assistenzstelle an einer Uni. Ich entwickle und unterstütze die Projektarbeiten, ich halte Vorträge, bereite Übungen und Tests vor und bin beteiligt an der Pflege der IT-Infrastruktur. Insgesamt ist die Arbeitsumgebung sehr flexibel, was sehr viel Spaß und natürlich auch viel Arbeit mit sich bringt. Sie fördert und fordert ein hohes Maß an Selbstständigkeit und Eigenverantwortung, und sie bringt mich in kürzester Zeit in Kontakt mit Menschen unterschiedlichster Nationen.

Mein Engagement bei der MEST läuft noch bis Ende Dezember. Danach habe ich schon unterschiedliche Angebote, allerdings habe ich mich noch nicht entschlossen, ob ich in Deutschland, Europa oder in anderen Teilen der Welt arbeiten möchte. Sicher ist jedoch jetzt schon, dass mir die Arbeit hier eine ganz andere Sicht auf viele Dinge im Alltag gebracht hat. Nicht alles, was in Deutschland normal ist, würde man in Ghana genauso machen, und andersherum.

Wer also wissen will, was “Fufu” und “Banku” sind, wie man in Twi “Hallo” sagt, was, zur Hölle, ein “Trotro” ist und wie viele Kilo ein menschlicher Kopf wirklich tragen kann, der könnte sich als “Teaching Fellow” für das nächste Semester der MEST in Ghana bewerben.

FANtasy Rotation

2006-01-01T00:00:00+00:00

I developed FANtasy Rotation with my good friend Helge back in 2006 and we have been maintaining it ever since. It is a game for the German basketball club Alba Berlin, which allows to users to make bets on how much playing time each individual player will get for the upcoming games. The site also has a vast collection of statistics and news reaching back to 2005, which eventually turned out to be more popular than the game itself.

Kneipenguide

2003-01-01T00:00:00+00:00

In 2003 I architected and implemented a Content Management System for one of the then early location services in the German market called kneipenguide.de, again with my friend Helge. “kneipe” means pub in German, and kneipenguide.de made it easy for people to explore the best places for drinks and food in their city. On of the most favoured tools was the Happy Hour Finder, that helped the users to find an affordable way through their party night.

Sebastian Spier

InnerSource is like Sourdough

Summary

Video

Knowledge Sharing as a Catalyst for Professional Growth

Creating a Knowledge Base

What’s in it for Meltwater?

Reasons for Meltwater Engineers to Present in Public

Where Do I Find the Videos?

Centralizing Developer Docs in Backstage

Challenges of Autonomy and Growth

What Do We Want Instead?

First Experiments with Backstage

Milestone 1: Starting with TechDocs Only!

The Future of Backstage at Meltwater Engineering

Curation > Technology

docs-like-code is the way to go

Discovery - Finding Key Documentation quickly

Can Templates support Paved Roads?

Keeping Backstage Up-to-date

In Closing

Contribute to Grow: Joining the Board of the InnerSource Commons

Joining an Open (Source) Community

My Context

User/Reader: Joining & Lurking

Contributor: Learning by Doing

Trusted Committer: Keep contributing & sharing

Member: Promoting InnerSource

Joining the Board: Let’s see :)

In Summary

Level up your InnerSource through Patterns

Video

Slides

Presenting at the InnerSource Commons 2020 Fall Summit

1 year in the InnerSource Commons Community: Getting your money’s worth

Video

Slides

Biases busted at FOSS Backstage 2020

What is FOSS Backstage?

My Favorite Talks from FOSS Backstage 2020

Why the GPL is great for business

How to convince your left brain (or manager) to follow the Open Source path that your right brain desires.

Blending Open Source and Corporate Values

A community-driven approach towards open innovation for research communication

My Bias was Busted

Repo-Rot is Real at RUG::B

Topic: Repo-Rot is Real

Costs of Repo-Rot

Slides

A Simple Tech Stack for your Engineering Blog

The Tech Stack of our Blog

Pros and Cons of the Tech Stack

Editing and Publishing Process

Why Google Docs?

Starting your own Engineering Blog

docx2gfm - Your friendly docx to markdown converter

The original doc

Turning this into Markdown

Rendered Markdown (HTML)

Post Processing

In Closing

Go to Conferences!

External stimulus leads to better decisions

Networking - Both external and internal

Keeping the momentum

Represent your company and pitch your product

In summary

Craft Conference 2016 Review

Great Product Team, Successful Product (closing keynote day 1)

“Thick” Data and Analysis: Engineering with a capital E

Psychological Aspects of Estimating

Embracing uncertainty: Why you should and why you won’t (Keynote)

Deploying and Scaling Microservices

Architecture without architects

The Technical Debt Trap

Building Scalable Stateful Services

Scaling Engineering Culture

Monitoring as a Service - Modernity and Self-Service

Toward Better Design and Code - A Case Study with Pelikan Cache

Every Product Company needs Corporate Innovation Days!