Daniel Westheide on making software

Pair Programming Considered Unnecessary: The Costs of Productive Solitude

Unknown — Sun, 29 Mar 2026 00:00:00 +0000

JetBrains recently announced that they are sunsetting Code With Me</a>, their IDE feature for remote collaborative coding. They explain this decision with declining demand after the pandemic, stating that teams have found "different collaboration workflows".

As someone who has experienced the value of pair and ensemble programming, both sitting at the same computer and remotely, I find it sad to see that this kind of working is apparently in decline, or even at risk of dying out, if a product specifically developed for supporting this way of collaboration remotely is no longer economically viable.

Pair programming, and even more so ensemble programming, can be draining, yes, so I totally understand that not everyone likes this way of working. But in my experience, the shared mental model you build up over time by regularly working on a code base in a pair or ensemble is absolutely priceless. I have never experienced this level of alignment within a team with any other method of collaboration. It's hands down one of the best methods of building a shared theory of the program in a team, as described by Naur's classic paper "Programming as Theory Building". Such a shared understanding cannot be built through documentation, which, according to Naur, and confirmed by my experience, is always lossy. It cannot be built through code reviews either, but only through interaction, working together on the same task in the same code base.

I very much suspect that the declining popularity of Code With Me has a lot to do with the rising popularity of AI assisted coding and agentic software development. As I mentioned in my previous blog post, I have long tried to ignore this rising tide completely. Meanwhile, I have been using JetBrains Junie for a year or so. And I noticed repeatedly that the fact that everyone in my team has adopted an AI-assisted way of working in some way or another has reduced the amount of time we work together. I often use AI as a rubber duck and thinking aid, and it works. As a consequence, the threshold for reaching out to a colleague when struggling with a problem or having a question has quietly risen. This is not a conscious decision. It just no longer feels necessary a lot of times to ask a co-worker if your AI assistant can help you as well. We all know how difficult it is to get productive again after an interruption, there are whole studies that have examined and quantified this problem. So having fewer of those interruptions has some merit.

And no, this doesn't mean that we don't collaborate any more at all. However, a lot of the collaboration has become asynchronous and artefact-based, discussing merge requests or architecture decision records. It's a rare occasion that we come together synchronously for exploratory thinking, for example for technical refinements.

For theory building in the Naur sense, this is not great. AI doesn't accumulate understanding of the system across sessions, and if I use AI as a thinking partner, that thinking does not propagate to the team. We're as far away from the mind-meld experience of ensemble programming as you can get. While we probably get better at producing and reviewing artefacts, we become worse at building a shared understanding of the system. I also imagine that onboarding new team members will be more difficult because if you hardly ever work together on a problem, there are very few chances for transfer of tacit knowledge. Somehow, an AI-assisted team seems to have a tendency towards creating silos of knowledge.

As a team you have to find a good balance between individual efficiency (in this case, by reducing interruptions) and long-term productivity of the team (through collaborative problem solving, leading to a shared mental model of the system in the whole team). The question is whether you are aware of this shift, and whether you take a conscious decision of accepting the trade-offs or whether you take active measures against it.

Developing with AI through the Cognitive Lens

Unknown — Fri, 20 Mar 2026 00:00:00 +0000

Wow, I can't believe that this blog has been dormant for more than six years! And the last post was frankly not more than a sign of life, informing about the migration from Octopress to Zola.

It's not that I haven't had anything to say in all those years. It's just that I have exclusively published my blog posts on the INNOQ company blog</a>. While I plan to be an even more active contributor to the company blog this year, I also decided it was time to come out of hibernation on my personal blog.

In this post, I provide some meta discussion of the latest blog series I started on the INNOQ company blog, titled Developing with AI through the Cognitive Lens. Here are two interesting facts:

Firstly, even though I studied Applied Communication and Media Science, an interdisciplinary study course integrating general psychology, computer science, and sociology, with a focus in human-computer-interaction and usability, I started my career as a Java backend developer. The relentless forces of path dependency led to me to continue that road.

Secondly, I have long held a very critical view of the current LLM offerings both for ethical and environmental reasons.

That's still true. However, AI-assisted coding and fully agentic software development have become so pervasive that I found it increasingly hard to ignore them — especially as someone clients expect to be familiar with current trends.

Some time last year, I finally had time to read Felienne Hermans'</a> great book The Programmer's Brain</a>, which teaches software developers all they need to know about cognition.

Reading this book rekindled my interest in cognitive psychology. I wanted to explore what cognitive psychology can teach us about the likely effects of using AI assistants or agents in software development. Knowing how humans learn, how they encode knowledge, and about the differences in how juniors and experts encode and retrieve knowledge, should tell us something about patterns of interacting with AI tools we should avoid, and which ones may prove beneficial.

In the first post, Speed vs Skill</a>, I examine the tension between increasing short-term productivity and skill retention.

In the second post, I explain elaboration</a>, a term from the science of learning, why it's important for learning, which AI interaction patterns eliminate the need for elaboration and which ones maintain it.

In the third post, the last one so far, I apply the concept of cognitive load theory to AI-assisted coding</a>, using it to explain the limitations of what we can learn from a recent study by Anthropic</a> about the effect of AI assistance on coding skills.

So far, writing this series has been really exciting. It's immensely satisfying to finally combine my interest and knowledge in general psychology with my expertise in software development and computer science. I definitely plan to write a few more posts in the series.

Scala from Scratch: Understanding

Unknown — Tue, 25 Aug 2020 00:00:00 +0000

Scala from Scratch: Understanding is the second in a series of two books that teach you the Scala programming language. Readers that have read Scala from Scratch: Exploration or have learned some Scala trhough other means will get a deeper understanding of the language features and underlying functional programming concepts that have been explored in the first book. You will also learn about important advanced language features that play a crucial role in many real-life Scala projects. While doing so, you'll get to know best practices that have been established over the years. By the end of the book, you will have a grasp of the language, its idioms, and common tools and techniques, allowing you to be a productive member in commercial or open-source Scala projects.
Get ebook</a> Get hardcover</a> </div>

Scala from Scratch: Exploration

Unknown — Sun, 08 Dec 2019 00:00:00 +0000

Scala from Scratch: Exploration is the first in a series of two books that teach you the Scala programming language. Readers that have some experience with an object-oriented or imperative language, like Java, Ruby, or Python, will get a comprehensive overview of what Scala brings to the table. The goal is to get you excited about Scala and give you a strong foundation for writing your first applications and continuing your journey. By the end of the book, you will have a good idea of what it's like to work with Scala, and you will have written a small, but complete command-line application based on a Functional Core, Imperative Shell architecture.
Get ebook</a> Get hardcover</a> </div>

Bundle</h3>
The ebook and the hardcover book are available at two different shops. Only Leanpub allows me to create discount codes. If you would like to get both the hardcover and the ebook, here's what you can do: Send me an email</a> with a picture of your hardcover book, and I'll send you a Leanpub coupon code allowing you to get the ebook for free.

Goodbye Octopress, hello Zola!

Unknown — Mon, 23 Sep 2019 00:00:00 +0000

If you have visited my website before, you will notice that things look a bit different from what they used to now. This is because I finally took the time to get rid of Octopress</a>, the static-site generator on which my website was based. In doing so, I took the opportunity to rewrite the HTML and CSS as well. If you're interested in why I put so many hours into replacing something that seemed just fine, read on.
The joys of using unmaintained software</h2>
Octopress was great when I started my website. It provided a decent default theme and all the features I needed to get started blogging quickly. For quite a while, though, Octopress has been a project that's no longer maintained. I was in the situation that could only generate my website with a specific ancient version of Ruby and the dependencies specified in my Gemfile.lock</code> a few years ago. I couldn't update Octopress to the latest version before it stopped being maintained. Even worse, when I got a new laptop, I wasn't even able to install the version of Ruby I needed for Octopress to work. I spent a few hours trying to create a Docker image that would do the trick, but gave up when my attempts lead nowhere. This brought me to the ridiculous situation where there was only exactly one machine from which I was able to generate my website when I wrote a new article — my old Macbook Pro. I really didn't want to depend on a single piece of hardware any longer. In my job, I would hardly accept such a sitation, so why should I do so for my personal website?
I wanted to go for a static site generator that would be easier to run on future hardware, ideally something consisting of a single, self-contained binary. Since I really like Rust, I didn't go for the the obvious choice, which would have been Hugo</a>, but decided to take Zola</a> for a ride, a static-site generator written in said language.
Less is more</h2> Another thing I didn't like about the old website was that, using the default Octopress theme, it was quite bloated. My goal for the re-write was to create something that is visually based on the old look, but is a lot more minimalistic. My website now consists of a few kilobytes of HTML, plus 2KB of CSS. Part of the concept is that, in order to keep the website as lean as possible, I decided against using web fonts and went for a font stack, depending on pre-installed fonts. I don't care too much about the website looking 100% consistent across all browsers and machines. What I do care about is that my website loads fast, even when you're sitting in a train with a really bad internet connection. Naturally, without any JavaScript, there is no tracking whatsoever on my website, which is something that was very important to me. In the old version using the default Octopress theme, I was loading some JavaScript from Twitter. As a consequence of removing all JavaScript, I also had to remove Disqus. I'm still evaluating alternatives, because there were actually quite a lof of comments and discussions in there. They are actually still there, I didn't delete anything. However, I decided against embedding Disqus into my website for now. Migration anxiety</h2> Have you ever bookmarked a URL of an interesting blog article, or even linked to it from your own blog, and some time later, you noticed that the link or bookmark was no longer working? Well, it happened to me numerous times. One of my biggest fears while preparing the new website was that I might break things for people as well. Since there are actually quite a few links pointing to specific posts on my website, this would be completely unacceptable. While Zola is not perfectly flexible with regard to the URLs it produces, you can configure redirects for pages and sections. This way, everyone who has bookmarked or linked to a page on my website should still be able to find those pages.
The Complexity Trap: Think Before You Leap Unknown — Fri, 07 Dec 2018 00:00:00 +0000 This blog post is a text version of a talk</a> I gave at Lambda World Cádiz</a> on October 25, 2018, for those who couldn't attend and don't like watching video recordings</a>. Thanks a lot to Jan Stępień for encouraging me to publish a text version of my talk. </div> In this article, I want to share my thoughts on a trend I have been observing for a few years. It's a trend that probably exists in the strongly typed functional programming community at large, and to some extent, it's possibly just a specific shape of a general trend in the realm of software development. But I have mainly observed it in the Scala community, because that's where I spend most of my time. I want to talk about how we often cause more complexity than necessary with the methods we choose. In fact, it seems to me as if we sometimes cherish complexity instead of fighting it. So let's see why that's the case and what we could do about it. </div> While this blog post is based on my experience in the Scala community, you don't have to be fluent with Scala in order to take something away from it. Before we can have a conversation about what's wrong with our relationship to complexity, though, it's necessary to talk about what complexity actually is. After that, I will go into more detail about the phenomemon I call the complexity trap. </div> What is complexity?</h2> When we say that something is "complex", what it means, according to the dictionary, is that it's "hard to separate, analyze, or solve". The word originates from the Latin verb "complectere", which means "to braid". Consider a bunch of modules that are completely entangled — so much so that you can neither reason about them individually any more nor change one of them without touching the other as well. That's a pretty good example of complexity in software development. </div> But it also helps to look at the origin of the word "simple" — the opposite of "complex". "Simple" has its roots in the Latin adjective "simplex", which means "single" or "having one ingredient". In software development, we have a few principles that aim to achieve simplicity in this original sense. Two prominent examples of that are The Single Responsibility Principle and Separation of Concerns. On such separation of concerns that we really love to enforce as functional programmers is the separation between effects and business logic. If these two concerns are not separated, it makes our code harder to reason about and more difficult to test. Hence, we try to isolate business logic from effects and implement the former as pure functions. </div> In his classic paper "No Silver Bullet", Fred Brooks identified two different types of complexity. Essential complexity is inherent in the problem domain. We cannot remove it because it stems from the very problem that we are supposed to solve with our software. Accidental complexity, on the other hand, is of our own making. This is complexity that is created by software developers, and in principle, it's possible to remove complexity of this type. According to Brooks, we spend most of our time tacking essential complexity these days, and there is not much accidental complexity left. This means that reducing accidental complexity further will not yield a lot of benefit. But that statement is from 1986. Today, it's 2018, and I beg to disagree. I think we are spending a crazy amount of time managing accidental complexity, and we should put our focus much more on minimizing it. </div> Over time, various metrics for complexity have been created, especially, but not exclusively, for code complexity. The most well-known of those metrics is probably McCabe's cyclomatic complexity: The more different code paths there are in a function, the higher its complexity. So one of the things that increased a function's complexity is branching, for example by means of if expressions of pattern matching expressions. The higher the cyclomatic complexity of a function, the more test cases are necessary to achieve full path coverage. Of course, there are other relevant metrics related to complexity in software development. Complexity can be measured at different levels, not only at the code level. For example, we can measure the extent of coupling between modules. The more dependencies there are between the modules of a system, the higher its overall complexity. </div> But let's come back to cyclomatic complexity. It's certainly okay as a metric of how hard it is to test a function. However, I am actually more interested in how difficult it is to reason about what a function or a program is doing — or to reason about a system or an entire system of systems. In other words, I am interested in cognitive complexity. In the imperative version of the sumOfPrimes</code> function, a lot of complexity stems from mutable state and flow control statements. The functional solution doesn't have any of that. So you could say that we are a bit at an advantage as functional programmers. However, we shouldn't be too smug about it either. What I find interesting is that Sonarsource actually created a metric for the cognitive complexity of code. However, I think that not every aspect of what constitutes cognitive complexity can easily be captured in metrics. For example, syntax, boilerplate, indirections can all add to a program's cognitive complexity as well. </div> But why is it so important to keep complexity in check anyway? Sadly, we human beings are incredibly bad at juggling with many things at once. According to a classic study by George Miller, a cognitive psychologist, human beings can only hold seven items in working memory at the same time, plus or minus two, on average. So the more concerns are entangled, the more code paths there are, and the more syntax and boilerplate there is, the more difficult it will be for us to reason about a program. </div> In order to mitigate this, we use abstractions. Ideally, our programs are composed of many small functions, and we put related functions into the same module. In order to reduce coupling and the complexity of the whole system, we hide implementation details of our modules from others that depend on them. In functional programming, we take abstraction much further than in other paradigms. We often abstract over things like iteration or data types in a way that is rarely seen outside of functional programming. This faciliates the definition of very simple functions that can be composed in many different ways — which is exactly what allowed us to write our sumOfPrimes</code> function in a way that is much easier to reason about than the imperative version. The complexity trap and how to escape it</h2> </div> Now that we have achieved a common understanding of what I mean when I talk about complexity, let me introduce you to my notion of the complexity trap. It might be fair to say that there is not a single complexity trap but rather multiple traps, even though they are all related to each other in one way or another. </div> The first trap that I want to talk about is neglecting the costs. We often seem to ignore the costs of the approaches and technique we choose to implement our programs. Usually, the intentions are good: Often, we choose a technique because we explicitly want to reduce the complexity in our code base. But sometimes, doing so comes with a cost — and that cost can even be new complexity at a different level, which might be worst than what we had before. </div> Let me illustrate this with an example. In general, scrap your boilerplate is a very desirable principle, because the more boilerplate code we have to write, the higher the complexity of our program and the higher the chance that we introduce bugs that could easily be avoided. There is one specific way of scrapping boilerplate that is as popular in the Scala community as it is problematic. In Scala, the way that you encode values of a certain type to JSON and back is often defined via typeclasses. Instances of these typeclasses can often be derived automatically for arbitrary data and record types with a single line of code. This feature is heavily used in many Scala projects. However, people rarely seem to consider the cost-value ratio of the way they use these automatically derived JSON codecs: In almost all projects I have seen, people use automatically derived JSON codecs to expose their internal domain model to the outside world. Doing that certainly means that you scrap quite a bit of boilerplate and reduce the complexity in your code. However, if you look at the bigger picture, this is the opposite of removing complexity. Exposing your domain model like this almost always leads to strongly coupled systems that are very difficult to evolve. If you ask me, this is one of the most promising ways to create a distributed monolith. It really amazes me how many people are totally okay with simply making their internal model their public API, especially since functional programmers are usually crazy about abstractions. </div> Apparently, we are even crazier about removing boilerplate. However, there are alternatives. If you don't want to expose your domain model, but you want to avoid having to define your JSON codecs manually, another way to decouple your model and your API is to introduce a DTO layer. This is an abstraction layer that is often ridiculed these days While it would allow you to use automatically derived JSON codecs for your separate DTO type hierarchy, it would still be necessary to implement a mapping between your domain model and your DTOs. You will probably end up with as much boilerplate code as if you had manually defined your JSON codecs for your domain model types. In the end, you have to choose your poison. Simply exposing your domain model with automatically derived JSON codecs is probably the most addictive poison, because it gives you instant gratification. In the long run, however, you will almost certainly have to pay the price. </div> There is a variant of neglecting the costs that deserves to be discussed on its own. In many projects I have seen, an approach or technology is picked because it is being perceived as common courtesy. The reasoning is that "this is what you do in 2018". If you ask people why they use approach or technology X, one of the most common reactions is that they are completely baffled to even hear such a question. So, in a slightly anxious and reproachful tone, they answer with a counter question: "Well, isn't that industry standard?" </div> You know what's industry standard? Shipping containers! However, when people say "industry standard", what they actually refer to is often just the latest fad — something that will definitely not be industry standard any more in a couple of years. If we follow this approach, it means that we adopt techniques without thinking. We don't think about their costs, and we certainly don't think about whether the benefits of that technique can even be realized in our specific situation. </div> Here is a prime example that pops up in many Scala projects: Many people seem to think that they will be excommunicated from the Church of Functional Programming if they don't use free monads or tagless final. There is this mantra that you must not commit to a specific effect monad too early. So what is a Scala developer going to do? Well, one thing is for sure: They are not going to commit to anything. Instead, they are going to abstract over monads and over many other things — everywhere and all the time. </div> There are good reasons why you want to abstract over the effect monad when you develop a library. For example, when you provide an HTTP client library, you want to enable people to use it regardless of which effect monad they prefer. Also, if you really have the need to interpret some instructions in different ways in your application, free monads or tagless final are a perfectly reasonable choice. For example, you may want to define a data processing pipeline that and then be able to execute it either in a Hadoop cluster or on a single machine. </div> However, most of the time when we write applications, such a need doesn't exist. So what other reasons to abstract over the effect monad are there? Well, preparing for the ability to switch from Monix Task</code> to ZIO is a bit like staying away from Postgres-specific SQL because you might switch to MySQL one day. So the main reason that remains for people to abstract over the effect type in application development is the desire to make their effectful code unit-testable. How does this work? You have a function that mixes logic as well as effects, and in your production code, you execute it in your favourite effect monad, talking to a real external system. In unit tests, on the other hand, you execute it in the Id</code> monad, using hard-coded test data. Object-oriented programmers call this technique stubbing, and they get mocked for doing it, mostly by people like us. </div> It's important to understand and keep in mind that abstracting over effect monads comes with a cost. For one, there is quite a bit of boilerplate code involved. Some of that is just syntactic noise that is due to the fact that Scala's type inference is quite limited. But there is also the whole interpreter for your algebra, which I'm not even showing. And if your program has some conditional logic, things can get rather messy pretty quickly. Also, by their very definition, monads are strictly sequential. If you want to execute some effects in parallel, because they are actually independent from one another, you will have to abstract over applicatives as well. Getting those to actually execute in parallel in your target effect tyoe, however, is surprisingly easy to get wrong in my experience. Finally, abstracting over effect monads seems to lead people to writing big functions in which logic and effects are completely entangled, because they are still abel to unit test the whole big function using the Id</code> monad. </div> There is an alternative that seems to be almost taboo, which is to not abstract over your effect monad at all in your application code. If you're honest, when testing with the Id</code> monad, you're only testing the few pieces of application logic within your big effectful function and you're verifying that your test interpreter is working as expected. But you have no clue if your production interpreter, which is talking to an external system, is working correctly at all. So abstracting over the effect monad and using the Id</code> monad in tests doesn't even help you to write meaningful tests. If you forego this abstraction, you will be able to use your effect type in all its glory. Among other things, this makes stuff like parallel execution very straightforward. </div> If you follow this approach, you have to be very strict about separating the logic of your program from the parts involving IO or other effects. One way of doing this is to employ a Functional Core, Imperative Shell architecture. What this means is that you'll move all the logic into the purely functional core. If necessary, you will have to extract it from your big effectful functions. This way, you can minimize the branching happening in the imperative shell and, since you can use the whole functionality of your favourite effect type there. </div> With all the logic being implemented in rather small, pure, and effect-free functions, it becomes a lot easier to write unit tests for your application's logic. The imperative shell will be almost free of logic, so that it can be covered with only a small number of integration tests. This does a better job at testing what is happening in a production-like environment. Free monads or tagless final are not a great choice if you only care about testability. There are alternatives that are less complex and at least as adequate for achieveing this goal. </div> So why is it that people jump to certain solutions so quickly? I have the impression that the discourse in our community is characterized by a focus on how to use certain abstractions. At best, we find discussions of when to use one abstraction over another. But we rarely see discussions about when not using any of those abstractions might be a viable solution. </div> I think we should all talk and write more about the costs of certain abstractions and have critical discussions of the trade-offs we need to make. Abstractions are supposed to help us keep complexity in check, but using them can come with its own complexity as well. Usually, a given abstraction is a very good fit for a few particular use cases. However, there are often alternative solutions that are more adequate for other use cases. These solutions are often both less complex and more boring. </div> Unfortunately, we don't like boring. We like to take the language and the compiler to their limits. We also like to adopt the latest fad. But the software systems we work on in our jobs should not be a playground or a test bed for the latest fad. So let's talk and write more about when certain abstractions are a good fit and when they are not, and about when a more boring solution might be a better fit. And when someone in your team tells you, "This problem must be solved with free monads", don't be afraid to ask why. In general, make a habit of demanding to hear why and approach was chosen, which alternatives have been considered, and why they've been discarded. A great technique for documenting these decisions for future reference is to write Architecture Decision Records (ADR). </div> The last aspect of the complexity trap that I want to discuss is what I call maximum zoom. Have you ever used one of those coin-operated telescopes, or binoculars, in order to zoom in to some detail of a scenic view? Maybe you have also experienced how easy it is to lose your bearings and how difficult it can be to keep track of the broader context. I think the same thing happens a lot in software engineering. </div> Let me explain this with an example. These days, many teams face the challenge that the system they are working on needs to make a lot of requests to various other systems. Very often, this happens in a Netflix-style microservice architecture, where you have tons of tiny microservices and another service in front of those, orchestrating the others, aggregating their response data and doing some additional processing, for example filtering. One particular pattern in which such an orchestration service occurs is BFF (backend for frontend)</a>. Very often, having to make all these requests to multiple backend microservices leads to high response times for the end user, which is then mitigated by introducing batching and caching of previous results, and also by making requests that are independent from one another in parallel. Imagine we'd be taking care of all these different concerns in the same place. That would certainly not be simple, as in "having one ingredient" — which is exactly why there are libraries that provide abstractions over these concerns, for example Stitch, Clump, or Fetch. </div> Keep in mind that I don't know anything about the context in which these libraries were created. For all I know, following that approach may have been the only viable solution to the problem the developers were facing in their specific situation, after careful analysis. Nevertheless, in my experience from other projects, the motivation to build abstractions like these is often to solve problems of a Netflix-style microservice architecture as the one I outlined above. As such, it serves as a nice example of the phenomenon that we often implement technological solutions to problems prematurely. I think that the reason why we see this phenomenon is that we don't apply systematic approaches at improving our software systems often enough. </div> Coming back to our example, it's fair to assume that caching, batching and parallel execution of requests, all nicely abstracted away, probably works kind of okay. However, if we apply a systematic approach at improving our software, we may very well end up with a different solution. One such systematic apporach is called the Architecture Improvement Method (arc42)</a>. Using this method, you work iteratively in three phases in order to improve your software system: Analyse in order to identify issues and improvements</li> Evaluate in order to estimate the value of issues and the effort or cost associated with improvements</li> Improve by applying selected improvements</li> </ol> </div> One of the most important techniques in the analysis phase is root cause analysis. Unfortunately, it's also a tremendously underused technique. In my experience, people often jump to solutions too quickly. When we face a problem, we tend to immediately search for a solution for exactly this problem. Root cause analysis differentiates between problem and cause. The idea is to trace a problem to its origin and to identify root causes of symptoms or problems. When doing a root cause analysis, you continue to ask why until you have found the root cause in the causal chain. </div> Let's apply this to our high latency problem from our example. We could ask ourselves why our latency is so high and get to the answer that it's because of all the requests we need to make in our BFF. But why do we need to make so many requests? The answer could be that it's because we need to fetch data from many different services. And why do we need to do that? Maybe it's because all our services are designed around a single entity. And why is that? At some point, we realize that we have an architectural problem at the macro level. Maybe, if we continue to ask why long enough, we will come to the conclusion that all these services we need to communicate with exist in exactly this way because of how the company is organized. </div> In any case, high latency of our BFF is only a symptom, and the root cause, the thing we should try to solve, lies a lot deeper. By making the symptom less painful, we don't reduce complexity in the system as a whole. Think of all the complexity that could be eliminated if we changed our architecture at the macro level — for example, if instead of a service in front of all those tiny microservices, we had a few slightly bigger systems that are not just built around a single entity or aggregate, but a whole Bounded Context. The whole orchestration and aggregation layer would be gone, and browsers would talk to those bigger systems directly, because they are entirely self-contained</a>, including their own UI. </div> But the maximum zoop trap also applies to our issue with the JSON codecs. If we do a root cause analysis here, at some point, we may ask ourselves why we have a JSON API in the first place. Often, the sole consumer of such an API is a single page application (SPA) running in the browser. While there are perfectly valid use cases for SPAs, they are often chosen merely because it's "industry standard" to use them these days — which brings us back to the trap of embracing industry standards. Moreover, we are programmers, and we like to write code. Having to build an SPA allows us to write even more code. It's a new playground for doing functional programming — applicative functors in the browser, how amazing is that? </div> But imagine the unthinkable for a minute: What if we did server-side rendering of HTML instead? There wouldn't be any need to design an evolvable JSON API, which, unlike simply exposing your domain model, is not trivial, and there would definitely be no need for defining any JSON codecs. And that's just what you get on the server side. On the frontend side, you would have one piece of software less to maintain, you'd get faster load times and you could get rid of duplicated business logic. To sum it up, you could reduce the complexity of your frontend JavaScript code a lot. In the words of Jorge Manrubia, who wrote a great article</a> about how the complexity introduced by single-page applications can often be entirely avoided: "[..] what about if, instead of trying to find a solution, you put yourself in a situation where those problems don’t exist?" Eliminating problems altogether is always better than writing code to solve them, regardless of how purely functional that could is. </div> But why is it that we fall for technological solutions so often, especially ones that don't reduce complexity at all? I think that maybe we love progrmaming too much. I was recently talking to someone about Rust, and that person said, "A language without higher-kinded types is completely useless!". Yes, certain abstractions are not possible without them, but I have never seen a project fail due to a lack of higher-kinded types. Projects fail because the problem domain is not well understood, or because the architecture is not a good fit for the requirements. They also fail because of how the organisation is structured and because of politics, be it intra- or inter-organizational. Projects fail because of people, and because of people not talking to each other. </div> When we jump towards technological solutions, we often do this because we love programming. In our case, we love functional programming. We also love technological challenges. Solving a problem by writing code is our comfort zone, so we take this route even if the actual problem is not a technological one. We prefer to work around non-technical problems using technological solutions because that's what we know, and it's the more interesting way out. Conclusions</h2> </div> I work at a consultancy, and we are often hired to solve challenging technological problems. Often, it turns out that people only think that their challenges are about technology, but the important problems are not technological at all. I believe that as developers, our job is to solve problems and create value for the business and the users of the product we build. Our job is not to write code. Sometimes, doing our job means that we have to search for the root cause of a problem. Instead of solving a symptom by writing code, we have to tackle communication problems, deal with organizational issues, or improve the architecture of our system at the macro level. I think that jumping towards technological solutions for symptoms also happens because everything that is not solvable with code is often seen as a given, something you can't change anyway. I think it's worth giving it a try more often, whether you are a consultant or not. Things are not always set in stone. And if we are serious about reducing accidental complexity, we should question our self-concept, our idea of what it means to be a software developer. </div> To summarize, we have to deal with a lot of accidental complexity these days, for multiple reasons. First of all, we adopt techniques and abstractions without analying and evaluating their costs and their usefulness in our specific situation. Moreover, we love solving technological problems. So let's shift our discourse and talk more about when certain abstractions are a good fit and when they aren't, and let's bring more boring but possibly simpler solutions to the table. Also, don't be afraid to demand explanations for architectural decisions. Finally, let's slow down when facing a problem. Let's take the time to analyze its root cause and let's jump out of our comfort zone, if necessary — even if it means that we will eliminate the problem and rob ourselved of a chance to solve it. </div> Credits</h2> self-contained systems diagram</a></li> Photo by Domenico Gentile on Unsplash</a></li> Photo by drmakete lab on Unsplash</a></li> Photo by Johny Goerend on Unsplash</a></li> Photo by juan pablo rodriguez on Unsplash</a></li> Photo by Mario Azzi on Unsplash</a></li> Photo by Marissa Rodriguez on Unsplash</a></li> Photo by Verena Yunita Yapi on Unsplash</a></li> Photo by Samuel Zeller on Unsplash</a></li> Photo by Markus Spiske on Unsplash</a></li> Photo by Kelly Sikkema on Unsplash</a></li> Photo by Jaciel Melnik on Unsplash</a></li> Photo by Shawn McKay on Unsplash</a></li> Photo by Drew Graham on Unsplash</a></li> Photo by Marcos Gabarda on Unsplash</a></li> Photo by Kylli Kittus on Unsplash</a></li> </ul> Introducing kontextfrei Unknown — Tue, 31 Oct 2017 00:00:00 +0000 For the past 15 months, I have been working on a new library on and off. So far, I have been mostly silent about it, because I didn't feel like it was ready for a wider audience to use – even though we had been using it successfully in production for a while. However, since I broke my silence as long ago as April this year, when I did a talk about it at this year's ScalarConf in Warsaw, a blog post is overdue in which I explain what this library does and why I set out to write it in the first place. Last year, I was involved in a project that required my team to implement a few Spark applications. For most of them, the business logic was rather complex, so we tried to implement this business logic in a test-driven way, using property-driven tests. The pain of unit-testing Spark applications</h2> At first glance, it looks like this is a great match. When it comes down to it, a Spark application consists of IO stages (reading from and writing to data sources) and transformations of data sets. The latter constitute our business logic and are relatively easy to separate from the IO parts. They are mostly built from pure functions. Functions like these are usually a perfect fit for test-driven development as well as for property-based testing. However, all was not great. It may be old news to you if you have been working with Apache Spark for a while, but it turns out that writing real unit tests is not actually supported that well by Spark, and as a result, it can be quite painful. The thing is that in order to create an RDD</code>, we always need a SparkContext</code>, and the most light-weight mechanism for getting one is to create a local SparkContext</code>. Creating a local SparkContext</code> means that we start up a server, which takes a few seconds, and testing our properties with lots of different generated input data takes a really long time. Most certainly, we are losing the fast feedback loop we are used to from developing web applications, for example. Abstracting over RDDs with kontextfrei</h2> Now, we could confine ourselves to only unit-testing the functions that we pass to RDD</code> operators, so that our unit tests do not have any dependency on Spark and can be verified as quickly as we are used to. However, this leaves quite a lot of business logic uncovered. Instead, at a Scala hackathon last May, I started to experiment with the idea of abstracting over Spark's RDD</code>, and kontextfrei was born. The idea is the following: By abstracting over RDD</code>, we can write business logic that has no dependency on the RDD</code> type. This means that we can also write test properties that are Spark-agnostic. Any Spark-agnostic code like this can either be executed on an RDD</code> (which you would do in your actual Spark application and in your integration tests), or on a local and fast Scala collection (which is really great for unit tests that you continously run locally during development). Obtaining the library</h2> It's probably easier to show how this works than to describe it with words alone, so let's look at a really minimalistic example, the traditional word count. First, we need to add the necessary dependencies to our SBT build file. Kontextfrei consists of two different modules, kontextfrei-core</code> and kontextfrei-scalatest</code>. The former is what you need to abstract over RDD</code> in your main code base, the former to get some additional support for writing your RDD-independent tests using ScalaTest with ScalaCheck. Let's add them to our build.sbt</code> file, together with the usual Spark dependency you would need anyway: resolvers += "dwestheide" at "https://dl.bintray.com/dwestheide/maven" libraryDependencies += "com.danielwestheide" %% "kontextfrei-core-spark-2.2.0" % "0.6.0" libraryDependencies += "com.danielwestheide" %% "kontextfrei-scalatest-spark-2.2.0" % "0.6.0" % "test,it" libraryDependencies += "org.apache.spark" %% "spark-core" % "2.2.0" </code></pre> Please note that in this simple example, we create a Spark application that you can execute in a self-contained way. In the real world, you would add spark-core</code> as a provided</code> dependency and create an assembly JAR that you pass to spark-submit</code>. Implementing the business logic</h2> Now, let's see how we can implement the business logic of our word count application using kontextfrei. In our example, we define all of our business logic in a trait called WordCount</code>: package com.danielwestheide.kontextfrei.wordcount import com.danielwestheide.kontextfrei.DCollectionOps import com.danielwestheide.kontextfrei.syntax.SyntaxSupport trait WordCount extends SyntaxSupport { def counts[F[_]: DCollectionOps](text: F[String]): F[(String, Long)] = text .flatMap(line => line.split(" ")) .map(word => (word, 1L)) .reduceByKey(_ + _) .sortBy(_._2, ascending = false) def formatted[F[_]: DCollectionOps](counts: F[(String, Long)]): F[String] = counts.map { case (word, count) => s"$word,$count" } } </code></pre> The first thing you'll notice is that the implementations of counts</code> and formatted</code> look exactly the same as they would if you were programming against Spark's RDD</code> type. You could literally copy and paste RDD</code>-based code into a program written with kontextfrei. The second thing you notice is that the method signatures of counts</code> and formatted</code> contain a type constructor, declared as F[_]</code>, which is constrained by a context bound: For any concrete type constructor we pass in here, there must be an instance of kontextfrei's DCollectionOps</code> typeclass. In our business logic, we do not care what concrete type constructor is used for F</code>, as long as the operations defined in DCollectionOps</code> are supported for it. This way, we are liberating our business logic from any dependency on Spark, and specifically on the annoying SparkContext</code>. In order to be able to use the familiar syntax we know from the RDD</code> type, we mix in kontextfrei's SyntaxSupport</code> trait, but you could just as well use an import instead, if that's more to your liking. Plugging our business logic into the Spark application</h2> At the end of the day, we want to be able to have a runnable Spark application. In order to achieve that, we must plug our Spark-agnostic business logic together with the Spark-dependent IO parts of our application. Here is what this looks like in our word count example: package com.danielwestheide.kontextfrei.wordcount import com.danielwestheide.kontextfrei.rdd.RDDOpsSupport import org.apache.spark.SparkContext object Main extends App with WordCount with RDDOpsSupport { implicit val sparkContext: SparkContext = new SparkContext("local[1]", "word-count") val inputFilePath = args(0) val outputFilePath = args(1) try { val textFile = sparkContext.textFile(inputFilePath, minPartitions = 2) val wordCounts = counts(textFile) formatted(wordCounts).saveAsTextFile(outputFilePath) } finally { sparkContext.stop() } } </code></pre> Our Main</code> object mixes in our WordCount</code> trait as well as kontextfrei's RDDOpsSupport</code>, which proves to the compiler that we have an instance of the DCollectionOps</code> typeclass for the RDD</code> type constructor. In order to prove this, we also need an implicit SparkContext</code>. Again, instead of mixing in this trait, we can also use an import. Now, our Main</code> object is all about doing some IO and integrating our business logi into it. Writing Spark-agnostic tests</h2> So far so good. We have liberated our business logic from any dependency on Spark, but what do we gain from this? Well, now we are able to write our unit tests in a Spark-agnostic way as well. First, we define a BaseSpec</code> which inherits from kontextfrei's KontextfreiSpec</code> and mixes in a few other goodies from kontextfrei-scalatest and from ScalaTest itself: package com.danielwestheide.kontextfrei.wordcount import com.danielwestheide.kontextfrei.scalatest.KontextfreiSpec import com.danielwestheide.kontextfrei.syntax.DistributionSyntaxSupport import org.scalactic.anyvals.PosInt import org.scalatest.prop.GeneratorDrivenPropertyChecks import org.scalatest.{MustMatchers, PropSpecLike} trait BaseSpec[F[_]] extends KontextfreiSpec[F] with DistributionSyntaxSupport with PropSpecLike with GeneratorDrivenPropertyChecks with MustMatchers { implicit val config: PropertyCheckConfiguration = PropertyCheckConfiguration(minSuccessful = PosInt(100)) } </code></pre> BaseSpec</code>, like our WordCount</code> trait, takes a type constructor, which it simply passes along to the KontextfreiSpec</code> trait. We will get back to that one in a minute. Our actual test properties can now be implemented for any type constructor F[_]</code> for which there is an instance of DCollectionOps</code>. We define them in a trait WordCountProperties</code>, which also has to be parameterized by a type constructor: package com.danielwestheide.kontextfrei.wordcount trait WordCountProperties[F[_]] extends BaseSpec[F] with WordCount { import collection.immutable._ property("sums word counts across lines") { forAll { (wordA: String) => whenever(wordA.nonEmpty) { val wordB = wordA.reverse + wordA val result = counts(Seq(s"$wordB $wordA $wordB", wordB).distributed) .collectAsMap() assert(result(wordB) === 3) } } } property("does not have duplicate keys") { forAll { (wordA: String) => whenever(wordA.nonEmpty) { val wordB = wordA.reverse + wordA val result = counts(Seq(s"$wordA $wordB", s"$wordB $wordA").distributed) assert( result.keys.distinct().collect().toList === result.keys .collect() .toList) } } } } </code></pre> We want to be able to test our Spark-agnostic properties both against fast Scala collections as well as against RDD</code>s in a local Spark cluster. To get there, we will need to define two test classes, one in the test</code> sources directory, the other one in the it</code> sources directory. Here is the unit test: package com.danielwestheide.kontextfrei.wordcount import com.danielwestheide.kontextfrei.scalatest.StreamSpec class WordCountSpec extends BaseSpec[Stream] with StreamSpec with WordCountProperties[Stream] </code></pre> We mix in BaseSpec</code> and pass it the Stream</code> type constructor. Stream</code> has the same shape as RDD</code>, but it is a Scala collection. The KontextfreiSpec</code> trait extended by BaseSpec</code> defines an abstract implicit DCollectionOps</code> for its type constructor. By mixing in StreamSpec</code>, we get an instance of DCollectionOps</code> for Stream</code>. When we implement our business logic, we can run the WordCountSpec</code> test and get instantaneous feedback. We can use SBT's triggered execution and have it run our unit tests upon every detected source change, using ~test</code>, and it will be really fast. In order to make sure that none of the typical bugs that you would only notice in a Spark cluster have sneaked in, we also define an integration test, which tests exactly the same properties: package com.danielwestheide.kontextfrei.wordcount import com.danielwestheide.kontextfrei.scalatest.RDDSpec import org.apache.spark.rdd.RDD class WordCountIntegrationSpec extends BaseSpec[RDD] with RDDSpec with WordCountProperties[RDD] </code></pre> This time, we mix in RDDSpec</code> because we pass parameterize BaseSpec</code> with the RDD</code> type constructor. Design goals</h2> It was an explicit design goal to stick to the existing Spark API as closely as possible, allowing people with existing Spark code bases to switch to kontextfrei as smoothly as possible, or even to migrate parts of their application without too much hassle, with the benefit of now being able to cover their business logic with missing tests without the usual pain. An alternative to this, of course, would have been to build this library based on the ever popular interpreter pattern. To be honest, I wish Spark itself was using this pattern – other libraries like Apache Crunch have shown successfully that this can help tremendously with enabling developers to write tests for the business logic of their applications. If Spark was built on those very principles, there wouldn't ne any reason for kontextfrei to exist at all. Limitations</h2> kontextfrei is still a young library, and while we have been using it in production in one project, I do not know of any other adopters. One if its limitations is that it doesn't yet support all operations defined on the RDD</code> type – but we are getting closer. In addition, I have yet to find a clever way to support broadcast variables and accumulators. And of course, who is using RDD</code>s anyway in 2017? While I do think that there is still room for RDD</code>-based Spark applications, I am aware that many people have long moved on to Dataset</code>s and to Spark Streaming. It would be nice to create a similar typeclass-based abstraction for datasets and for streaming applications, but I haven't had the time to look deeper into what would be necessary to implement either of those. Summary</h2> kontextfrei is a Scala library that aims to provide developers with a faster feedback loop when developing Apache Spark applications. To achieve that, it enables you to write the business logic of your Spark application, as well as your test code, against an abstraction over Spark’s RDD. I would love to hear your thoughts on this approach. Do you think it's worth it defining the biggest typeclass ever and reimplementing the RDD</code> logic for Scala collections for test purposes? Please, if this looks interesting, do try it out. I am always interested in feedback and in contributions of all kind. Links</h2> kontextfrei project website</a></li> kontextfrei GitHub repo</a></li> kontextfrei wordcount example</a></li> </ul> The Empathic Programmer Unknown — Mon, 16 Jan 2017 00:00:00 +0000 In 1999, Andrew Hunt and Dave Thomas, in their seminal book, demanded that programmers be pragmatic</a>. Ten years later, Chad Fowler, in his excellent book on career development, asked programmers to be passionate</a>. Even today, I still consider a lot of the advice in both of these books to be incredibly valuable, especially Fowler's book that helped me a lot, personally. Nevertheless, in recent years, I have witnessed again and again that one other quality in programmers is at least as important and that it hasn't even seen a fraction of the attention it deserves. The programmer we should all strive to be is the empathic programmer. Of course, I am not the only one, let alone the first one, to realize that. For starters, in my bubble, Benjamin Reitzimmer wrote an excellent post about what he considers to be important qualities of a mature developer</a> a while ago, and empathy is one of them. I consider a lack of empathy to be the root cause for some of the biggest problems in our industry and in the tech community. In this post, I want to share some observations on how a lack of empathy leads to problems. Consider it a call to strive for more empathy. So what is empathy? Here is a definition from Merriam-Webster</a>: the action of understanding, being aware of, being sensitive to, and vicariously experiencing the feelings, thoughts, and experience of another of either the past or present without having the feelings, thoughts, and experience fully communicated in an objectively explicit manner; also: the capacity for this </blockquote> Empathy at the workplace</h2> It shouldn't come as a surprise that the ability to show empathy can come in handy in any kind of job that involves working with other people, including the job as a programmer. This is true even if you work remotely – the other messages you see in your Slack channels are not all coming from bots. There are actual human beings behind them. One of the situations where we often forget to think about that is code reviews. Just writing down what is wrong with a pull request without thinking about tone can easily lead to the creator of the pull request feeling personally offended. April Wensel has some good advice</a> on code reviews. What's crucial is to develop some sensitivity for how your words will be perceived by the receiver, which requires to put yourself into their shoes, see through their eyes and reflect how they will feel. This is easier the better you know the person, otherwise you will have to make some assumptions, but I think that's still far better than not reflecting at all on how the other person will feel. Another workplace situation where I have often seen a lack of empathy is when members of two different teams need to collaborate to solve a problem or get a feature done. In some companies, I have seen an odd, competitive "us versus them" attitude between teams. This phenomenon has been explored by social and evolutionary psychologists</a>, and while such a behaviour might still be in our nature, that doesn't mean that we cannot try to overcome it. A variant of "us versus them" is "developers versus managers". We developers have a hard time understanding why managers do what they do, but frankly, often, we don't try very hard. I have often seen developers taking on a very defensive stance against managers, and of course, the relationship between managers and developers in these cases was rather chilly. Getting to know "the other side" would certainly help to empathize with managers. Understanding why they act in a specific way is absolutely necessary in order to get to a healthy relationship with them. Empathy in the tech community</h2> Empathy is not only important at your workplace, but also very much so when you are interacting with others in our community, be it on mailing lists, conferences, or when communicating with users of your open source library, or developers of an open source library you are using. In some of these situations, a lack of empathy can strengthen exclusion, ultimately leading to a closed community that is perceived as elitist and arrogant. As a developer using an open source library, empathize with the developers of the library before you start complaining about a bug, or better yet, a missing feature. Sam Halliday wrote an interesting post called The Open Source Entitlement Complex</a>. It's hard to believe, but apparently, many users of open source libraries have this attitude that the developers of these libraries are some kind of service provider, happily working for free to do exactly what you want. This is not how it works. The same way that wording and tone are important in code reviews, try to empathize with the developers who spend their free time on this library you use. Serving you and helping you out because you didn't read the documentation is probably not their highest priority in life, so don't treat them as if it is. On the other hand, when presenting your open source library to potential users, consider how these people will feel about that presentation. Does it make them feel respected? Does it make them feel welcome? I am sorry to disappoint you, but I think that a foo bar "for reasonable people" does not have that effect. Personally, I find this to be very condescending and think it will intimidate a lot of people and turn them away. It implies that any other way than yours is not reasonable, and that, hence, people who have not used your library yet, but some different approach, are unreasonable people. As library authors, let's show some empathy as well towards our potential users. As always in tech, there is no silver bullet, and there are trade-offs. There are probably perfectly good reasons why someone has been using a different library so far, and maybe even after looking at your library, there will still be good reasons not to use yours. Even if you are convinced that your library is so much better, you aren't exactly creating an open and welcoming atmosphere by basically telling people visiting your project page that they are unreasonable for using anything else. If you are at a tech conference, and you ask women whether they are developers at the very beginning of your conversation, but don't do the same with men, you are probably not doing that out of malignity, but because you don't see many women at tech conferences who are actually developers. Nevertheless, to the receiver, this seemingly harmless and neutral question doesn't come across like that at all. She has probably heard this question many times, and constantly hearing doubts about whether you are really a programmer doesn't exactly make you feel welcome, or confident. Show some empathy when you talk to other people at tech conferences. Imagine what it would be like to constantly be doubted, for example. If you don't see a need for being inclusive, that's probably because you had no problem being included in the community. This likely means that you are a man, and probably white. Since most people around you are like you, chances are you don't even know any women or other unprivileged people who are developers. The problem of being privileged is that you don't notice it. Talk to women on conferences and let them tell you about their experiences. By showing empathy, you can create a more welcoming environment of inclusion and foster diversity. Summary</h2> These are my two cents about empathy, and lack thereof, in the tech community, and how it relates to inclusion and diversity. Empathy is important not only at the workplace, when interacting with co-workers, but also when we are participating in the tech community, as conference visitors, open source developers, and users of open source libraries. Only by showing empathy, we can create an inclusive and open community. Let's try to be more aware of the effects we have on each other, and act accordingly. Thanks! When Option is not good enough Unknown — Tue, 26 Apr 2016 00:00:00 +0000 Recently, I tweeted about an observation I have made a couple of times in various Scala projects: Observation: People are using Option too often where their business logic clearly indicates they should use their own, custom ADT.— Daniel Westheide (@kaffeecoder) April 20, 2016</a></blockquote> Since 140 characters are not nearly enough to discuss this issue appropriately and there were a few questions around this whole topic, I decided to put my thoughts down in a blog post, providing a few examples of what I actually talking about. Option is great, Option is good!</h2> As Scala programmers, we can all give ourselves a pat on the back for being upright users of the Option</code> type. Option</code> is great! It allows us to express in types that a value may or may not be there. Option is good! It allows us to work with potentially missing values in a safe and elegant way, without having to check for presence all the time or risking those dreaded NullPointerException</code>s. All hail Option</code>! The problem of overloaded semantics</h2> Option is also bad! Or, more precisely, the way that this type is being used is somewhat problematic. The semantics of the Option</code> type are pretty clear: it is about potential absence of a value. For example, if you try to get a value for a specific key from a Map</code>, the result may or may not be there: scala> Map("DE" -> "Germany", "FR" -> "France") get "DE" res0: Option[String] = Some(Germany) </code></pre> This usage of the Option</code> type is consistent with its semantics. However, sometimes, we attach different or additional meaning to this type. An example of this has been summarised by @tksfz</a> as a reply to my tweet: @kaffeecoder</a> An example: A Query type where empty means match everything. Have that in our code right now.— tksfz (@tksfz) April 20, 2016</a></blockquote> Since this is such a good example, I decided to shamelessly make use of it to explain the problem. Imagine you are developing a system that allows users to search for various offers, both by retailers and private sellers. A very simplified version of the search function could look like this: def searchOffers( productTitle: Option[String], retailer: Option[Retailer] ): Seq[Offer] = ??? </code></pre> Apparently, there are two search criteria you can provide to filter the results: the product, and the retailer offering the product. But what does this really mean? It looks like we have attached some new semantics to the Option</code> type. It seems that if productTitle</code> is None</code>, the user wants to search for all offers, regardless of the product title, which means that None</code> has the meaning of a wildcard. However, if productTitle</code> is a Some</code>, is this supposed to be an exact match or does the provided search string just have to be contained in the product title? For the retailer, the same semantics might apply if the Option</code> is undefined. Or maybe, in this case, None</code> actually means that the user wants to search for offers that are not provided by a professional retailer. Who knows? As a team member seeing this code for the first time, or coming back to it after a few months, I would probably be confused. The problem is that we are overloading the semantics of None</code> and Some</code>. The former is very similar to how null</code> in Java is sometimes used with meanings that are different from simple absence. Towards meaningful types</h2> Luckily, we can do better in Scala. Whenever someone in your team is confused about the exact meaning of an Option</code> in your code, this is a good indicator that you should introduce your own algebraic data type that captures the implicit semantics of your domain. In the example above, something like this would probably be clearer for the reader: sealed trait SearchCriteria object SearchCriteria { final case object MatchAll extends SearchCriteria final case class Contains(s: String) extends SearchCriteria } sealed trait RetailerCriteria object RetailerCriteria { final case object AnyRetailer extends RetailerCriteria final case class Only(retailer: Retailer) extends RetailerCriteria } def searchOffers( product: SearchCriteria, retailer: RetailerCriteria ): Seq[Offer] = ??? </code></pre> Introducing your own algebraic data types here allows you to use to close the gap between the language of your domain and the one used in your code, bringing you closer to a ubiquitous language, one of the core values in domain-driven design. In this example, any confusion about whether the product title is an exact search in the Some</code> case or whether None</code> is a wildcard is now eliminated. In the searchOffers</code> implementation, we can simply use pattern matching on the SearchCriteria</code> and RetailerCriteria</code>, and since they are sealed traits, we will get a warning if our pattern matching is not exhaustive, i.e. it does not cover all the cases. If you want to read more about algebraic data types and sealed traits in Scala, read this excellent blog post by Noel Welsh</a>. Now, you might say that SearchCriteria</code> and RetailerCriteria</code> have the same shape as Option</code>, apart from the fact that they are not generic. Wouldn't it be good enough to use some type aliases then in order to get the benefits of the ubiquitous language? You could certainly do that, but having your own algebraic data types is more future-proof. You can easily extend your algebraic data types to enable additional functionality. If we want to allow users to do an exact search for the product title, and to specify that they are only interested in articles that are not offered by retailers, the following will do the trick: sealed trait SearchCriteria object SearchCriteria { final case object MatchAll extends SearchCriteria final case class Contains(s: String) extends SearchCriteria final case class Exactly(s: String) extends SearchCriteria } sealed trait RetailerCriteria object RetailerCriteria { final case object AnyRetailer extends RetailerCriteria final case class Only(retailer: Retailer) extends RetailerCriteria final case object NoRetailer extends RetailerCriteria } </code></pre> Since we are already using our own algebraic data types, no big refactorings should be necessary. Of course, it is possible to encode the retailer criteria as an Either[Unit, Option[Retailer]]</code>, but can you tell me immediately what each of the possible cases is supposed to mean? More examples</h2> The previous example was mainly about Option</code> being used in function arguments. It seems like this is rarely a good idea. Here is another example where Option</code> is used as a field of a case class and as a return type of a function, with confusing semantics. Imagine you are working at Nextflix, the next big platform for watching TV series online. Things being as they are, you need to block certain TV series from users located in specific countries. To do that, you could make use of a filter chain in your web application. One filter in this scenario needs to immediately return a response if the content is blocked in the user's country, or forward to the next filter in the chain if the content is accessible from the user's country. Here is what this could look like in Scala code: def checkTerritory(request: Request): Option[Block] = { val territory = territoryFrom(request) if (accepted(territory) None else Some(Block.requestsFrom(territory)) } def contentFilter(request: Request, next: Request => Response): Response = { val blockedResponse = for { restriction <- checkTerritory(request) } yield requestBlockedResponse(restriction) blockedResponse getOrElse next(request) } </code></pre> If the Option</code> returned by checkTerritory</code> is None</code>, that's the happy path and we can call the next filter in the chain. If it is defined, however, we need to short-circuit and immediately return a response informing the user that the content is blocked. Since Option</code> is success-biased and short-circuits if it is None</code>, this doesn't look very intuitive to me. What if we had our own algebraic data type for the verdict on the territory that a request is from? Such a solution could look like this: sealed abstract class Verdict final case object Pass extends Verdict final case class Block(reason: String) extends Verdict def checkTerritory(request: Request): Verdict = { val territory = territoryFrom(request) if (accepted(territory) Pass else Block.requestsFrom(territory) } def contentFilter(request: Request, next: Request => Response): Response = { checkTerritory(request) match { case Pass => next(request) case Block(reason) => requestBlockedResponse(reason) } } </code></pre> I find that to be a lot more readable, and again, it has the advantage that it is easier to extend this in the future with additional verdict types. Summary</h2> It is often possible to express a concept from your domain as an Option</code>, or if that doesn't work, as an Either</code>. Nevertheless, it is sometimes better to not use these generic types. If your actual semantics are different from potential absence of values, don't force it, as this causes unnecessary indirection when reasoning about the code and the domain. Unsurprisingly, this can easily lead to bugs. The domain logic is easier to understand if you use your own types from the ubiquitous language. Do you have more examples from your projects where the lack of custom algebraic data types has led to bugs or code that is difficult to understand? Feel free to share those in the comments or via Twitter</a>. Put your writes where your master is: Compile-time restriction of Slick effect types Unknown — Sun, 28 Jun 2015 00:00:00 +0000 When the access patterns of a service are such that there are a lot more reads than writes, a common practice for scaling it horizontally is to have the service talk to a master database only for operations that result in state changes, while all read-only operations are performed against one or more read slaves. Those read replicas are only eventually consistent with the master. This practice is an easy solution if you don't want to or cannot go the full CQRS way. All you need to do is maintain two separate data sources and point all your write operations to the master, and all your read operations to your slave. However, how do you make sure that you're actually putting your writes where your master is? It's easy to accidentally get this wrong, and usually, you will only find out at runtime, when the read slave is denying your service to perform its writes. Wouldn't it be nice if you could already verify at compile-time that your operations are hitting the correct database? With Slick 3, the latest version of Typesafe's functional-relational mapper, this is actually very easy to do, provided you know a bit about Scala's type system and Slick's notion of database actions and effect types. In this blog post, I will explain all you need to know in order to restrict the evaluation of Slick database actions based on their effect types, resulting in the elimination of yet another source of bugs. The full example application on which the code snippets in this blog post are based is available on GitHub</a>. Database actions</h2> Whether you want to query a table, insert or update a row, or update your database schema, in Slick 3, each of those things is expressed as a DBIOAction[+R, +S <: NoStream, -E <: Effect]</code>. R</code> is the result type of the action, and S</code> indicates the result type for streaming results, where the super-type NoStream</code> refers to non-streaming database actions. For this article, those two type parameters are not of importance, however. Let's look at the third parameter instead. The Effect type</h3> E</code> is some type of Effect</code> and describes the database action's effect type. Slick 3 defines the following sub-types of Effect</code>: trait Effect object Effect { trait Read extends Effect trait Write extends Effect trait Schema extends Effect trait Transactional extends Effect trait All extends Read with Write with Schema with Transactional } </code></pre> The Effect</code> type is a so-called phantom type</a>. This means that we never create any instances of Effect</code> at runtime. Rather, the sole purpose of this type is to give additional information to the compiler, so that it can prevent certain error conditions before running the application. If you use Slick's lifted embedding syntax for creating database actions, those actions will always have the appropriate subtype of Effect</code>. For example, if you have a table statuses</code>, you might implement the following StatusRepository</code>: class StatusRepository { def save(status: Status) = statuses.insertOrUpdate(status) def forId(statusId: StatusId) = statuses.filter(_.id === statusId).result.headOption } </code></pre> Here, the type of action returned by save</code> is automatically inferred to be DBIOAction[Int, NoStream, Write]</code>. The action returned by forId</code>, on the other hand, is DBIOAction[Option[Status], NoStream, Read]</code>. Effect types of composed actions</h3> If you compose several database actions using one of Slick's provided combinators, the correct intersection type will automatically be inferred. For instance, a common approach for updating the state of an aggregate is to have an application service load the aggregate from a repository, perform some business logic on the aggregate, and then save the updated version back to the repository. The application service is providing the transactional boundary for changing the state of the aggregate. This may look similar to the following, hopefully with a more complex business logic than in this oversimplified example: def categorize(statusId: StatusId, newCategory: String) = { val actions = for { statusOpt <- statusRepository.forId(statusId) result <- statusOpt match { case Some(status) => val newStatus = status.copy(category = newCategory) statusRepository.save(newStatus).map(_ => Right(newStatus)) case None => DBIO.successful(Left("unknown status")) } } yield result actions.transactionally } </code></pre> Here, the inferred effect type of the composed action – actions.transactionally</code>, will be: DBIOAction[Either[String, Status], NoStream, Read with Write with Transactional] </code></pre> Evaluating a database action</h3> A DBIOAction</code> merely describes what you want to do – in order to actually have it executed, you need to run it like this: class StatusReadService(database: DatabaseDef) { def statusesByAuthor(author: String, offset: Int, limit: Int): Future[Seq[Status]] = { database.run { statuses .filter(_.author === author) .sortBy(_.createdAt.desc) .drop(offset) .take(limit) .result } } } </code></pre> If you haved used Slick 3 before, you will probably be familiar with this. Let's look at the signature of run</code> as defined in DatabaseDef</code>: def run[R](a: DBIOAction[R, NoStream, Nothing]): Future[R] </code></pre> Having an effect type of Nothing</code> means that Slick accepts database actions with any effect type, not caring about whether it is a Write</code>, a Read</code>, or something else. This is perfectly fine for a generic library. However, when you are working with a master and a slave database, it means that your code will look like this: class DatabaseModule { val masterDatabase: DatabaseDef = Database.forConfig("databases.master") val slaveDatabase: DatabaseDef = Database.forConfig("databases.slave") } class StatusReadService(database: DatabaseDef) { // ... } class StatusService(statusRepository: StatusRepository, database: DatabaseDef) { // ... } </code></pre> On the type level, you are not able to differentiate between your master and slave databases, so performing an undesired effect against one of your databases by accident is a very real issue. Luckily, with its Effect</code> type, Slick 3 gives you all the compile-time information you need to implement a restriction of effect types for your own application. Restricting effect types</h2> In order to restrict the effect types that are allowed for a certain database, we are going to attach a role (e.g. master or slave) to each database and associate certain privileges to each role that will then be checked when trying to run a database action – all at compile time. Roles</h3> The first thing we need to introduce is another phantom type called Role</code>: trait Role trait Master extends Role trait Slave extends Role </code></pre> Like Slick's Effect</code>, our Role</code> trait will never be instantiated in our application. Privileges</h3> Instead, Role</code> will appear as a type parameter in yet another new phantom type: @implicitNotFound("'${R}' database is not privileged to to perform effect '${E}'.") trait HasPrivilege[R <: Role, E <: Effect] </code></pre> The HasPrivilege</code> phantom type is meant to provide implicit evidence that a certain role R</code> is allowed to perform the Slick effect type E</code>. The @implicitNotFound</code> annotation allows us to provide a custom error message for the case that no implicit evidence of HasPrivilege</code> can be found, where it is required. We are going to spell out the privileges for our two roles to the compiler like this: type ReadWriteTransaction = Read with Write with Transactional implicit val slaveCanRead: Slave HasPrivilege Read = null implicit val masterCanRead: Master HasPrivilege Read = null implicit val masterCanWrite: Master HasPrivilege Write = null implicit val masterCanPerformTransactions: Master HasPrivilege ReadWriteTransaction = null </code></pre> As you can see, while we do provide evidences in terms of implicit val</code>s, we don't actually create any instances of HasPrivilege</code>. Since the code that will make use of our evidence will never work with our implicit evidences at runtime, we can safely assign null</code> here. Also, please not that Slave HasPrivilege Read</code> is just another notation for HasPrivilege[Slave, Read]</code>. Unfortunately, we have to provide an implicit evidence for every combination of effect types we want to allow. In this example, we want to allow combining reads with writes and transactions, but it's not allowed to combine reads and writes without also using a transaction. Check your privileges!</h3> Now, in order to actually restrict the database actions that can be run according to the role of the database, we need to introduce a wrapper around Slick's DatabaseDef</code> – as we saw earlier, the latter does not care about the type of effect, and of course, it doesn't know anything about our roles. Hence, we are introducing a class DB</code>: class DB[R <: Role](databaseConfiguration: DatabaseConfiguration[R]) { private val underlyingDatabase = databaseConfiguration.createDatabase() def run[A, E <: Effect](a: DBIOAction[A, NoStream, E]) (implicit p: R HasPrivilege E): Future[A] = underlyingDatabase.run(a) } </code></pre> Our new wrapper type has a type parameter R</code> that specifies its role, and it creates the underlying Slick DatabaseDef</code> from an instance of DatabaseConfiguration</code> with the same role, which we will look at in a moment. For now, the important thing is the run</code> method on our DB</code> class, which looks very similar to the Slick's run</code> method we saw earlier. The crucial difference is that our run</code> method has a second type parameter E</code> that specifies the type of effect of our action, and that our DBIOAction</code>s effect type parameter is that E</code> instead of just Nothing</code>. Moreover, our run</code> method has a second, implicit parameter list, with evidence of HasPrivilege[R, E]</code>, i.e. that our database role R</code> is privileged to execute the effect E</code>. Instead of using DatabaseDef</code> directly, we will now always make use of our role-annotated DB</code>. Providing role-annotated databases</h3> To achieve that, we introduce a type DatabaseConfiguration</code> which, like DB</code>, is annotated with a role: sealed trait DatabaseConfiguration[R <: Role] { def createDatabase(): DatabaseDef } object DatabaseConfiguration { object Master extends DatabaseConfiguration[Master] { def createDatabase() = Database.forConfig("databases.master") } object Slave extends DatabaseConfiguration[Slave] { def createDatabase() = Database.forConfig("databases.slave") } } </code></pre> The DatabaseConfiguration</code> is the one place where we interact with the untyped outside world, reading our database connection information for the respective key, databases.master</code> or databases.slave</code>. Hence, the only place where we can still get things wrong is in our application.conf</code> configuration file, if we accidentally provide the wrong database host, for example. Our database module providing the master and slave databases to our application will now look like this: class DatabaseModule { val masterDatabase: DB[Master] = new DB(DatabaseConfiguration.Master) val slaveDatabase: DB[Slave] = new DB(DatabaseConfiguration.Slave) } </code></pre> Unlike the previous version, the masterDatabase</code> and slaveDatabase</code> fields are now properly typed, and our service implementations that made use of DatabaseDef</code> before must now explicitly pick a database with the correct Role</code> for their purposes: class StatusService(statusRepository: StatusRepository, database: DB[Master]) { // ... } class StatusReadService(database: DB[Slave]) { // ... } </code></pre> Oops, I used the slave for a write...</h3> To verify that all of this has the desired effect, let's make our StatusService</code> use the slave database: class StatusService(statusRepository: StatusRepository, database: DB[Slave]) { // ... } </code></pre> When we try to compile this, we will receive this nice error message from the compiler: 'com.danielwestheide.slickeffecttypes.db.Slave' database is not privileged to perform effect 'slick.dbio.Effect.Write'. </code></pre> Plain SQL queries</h3> For plain SQL queries, of course, Slick cannot infer any effect types automatically. Hence, if you need to fall back from lifted embedding to plain SQL queries, you have to annotate the resulting database actions explicitly: def statusesByCategory(category: String, offset: Int, limit: Int): Future[Seq[Status]] = { val action: SqlStreamingAction[Seq[Status], Status, Read] = sql"""select id, created_at, author, text, category from statuses where category = $category order by created_at desc limit $limit offset $offset""".as[Status] database.run(action) } </code></pre> If you don't do provide an explicit type here, the actions's effect type parameter will be inferred to be Effect</code>, the super type of all effect types. Custom effect types</h3> Since Effect</code> is not a sealed trait, you may introduce your own effect types and prevent certain databases from performing those effects. For example, you may want to disallow certain expensive queries to be performed against the master database. To do that, you could introduce an effect type ExpensiveRead</code>, and only allow slave databases to run actions with that type: trait ExpensiveRead extends Read implicit val slaveCanDoExpensiveReads: Slave HasPrivilege ExpensiveRead = null </code></pre> Just as with plain SQL queries, you can annotate your database action explicitly to be of type ExpensiveRead</code>: def statusesByCategory(category: String, offset: Int, limit: Int): Future[Seq[Status]] = { val action: SqlStreamingAction[Seq[Status], Status, ExpensiveRead] = sql"""select id, created_at, author, text, category from statuses where category = $category order by created_at desc limit $limit offset $offset""".as[Status] database.run(action) } </code></pre> If you accidentally use a DB[Master]</code> in this read service, you will get a compile error. Of course, once you have to start annotating your database action explicitly in order to benefit from these compile-time checks, you are prone to another source of errors. Failing to annotate your actions correctly may lead to the same kinds of runtime errors we wanted to prevent. Conclusion</h2> In this article I have shown how to use Slick's Effect</code> type, together with a few other phantom types, in order to have your compiler help you verify that you run your Slick database actions against the correct database. While it's possible to use this technique with plain SQL queries and custom effect types, the greatest benefit will come in cases where you only use lifted embedding and the standard effect types. Thanks a lot to @missingfaktor</a> who collaborated with me on developing this technique on top of Slick 3 and gave a lot of valuable input. The Neophyte's Guide to Scala Part 16: Where to Go from Here Unknown — Wed, 03 Apr 2013 00:00:00 +0000 As I have already hinted at in the previous article, the Neophyte's Guide to Scala is coming to an end. Over the last five months, we have delved into numerous language and library features of Scala, hopefully deepening your understanding of those features and their underlying ideas and concepts. As such, I hope this guide has served you well as a supplement to whatever introductory resources you have been using to learn Scala, whether you attended the Scala course at Coursera</a> or learned from a book. I have tried to cover all the quirks I stumbled over back when I learned the language – things that were only mentioned briefly or not covered at all in the books and tutorials available to me – and I hope that especially those explanations were of value to you. As the series progressed, we ventured into more advanced territory, covering ideas like type classes and path-dependent types. While I could go on writing about more and more arcane features, I felt like this would go against the idea of this series, which is clearly targeted at aspiring neophytes. Hence, I will conclude this series with some suggestions of where to go from here if you want more. Rest assured that I will continue blogging about Scala, just not within the context of this series. How you want to continue your journey with Scala, of course, depends a lot on your individual preferences: Maybe you are now at a point where you would like to teach Scala to others, or maybe you are excited about Scala's type system and would like to explore some of the language's more arcane features by delving into type-level programming. More often than not, a good way to really get comfortable with a new language and its whole ecosystem of libraries is to use it for creating something useful, i.e. a real application that is more than just toying around. Personally, I have also gained a lot from contributing to open source projects early on. In the following, I will elaborate those four paths, which, of course, are not mutually exclusive, and provide you with numerous links to highly recommendable additional resources. Teaching Scala</h2> Having followed this series, you should be familiar enough with Scala to be able to teach the basics. Maybe you are in a Java or Ruby shop and want to get your coworkers excited about Scala and functional programming. Great, then why not organize a workshop? A nice way of introducing people to a new language is to not do a talk with lots of slides, but to teach by example, introducing a language in small steps by solving tiny problems together. Active participation is key! If that's something you'd like to do, the Scala community has you covered. Have a look at Scala Koans</a>, a collection of small lessons, each of which provides a problem to be solved by fixing an initially failing test. The project is inspired by the Ruby Koans</a> project and is a really good resource for teaching the language to others in small collaborative coding sessions. Another amazing project ideally suited for workshops or other events is Scalatron</a>, a game in which bots fight against each other in a virtual arena. Why not teach the language by developing such a bot together in a workshop, that will then fight against the computer? Once the participants are familiar enough with the language, organize a tournament, where each participant will develop their own bot. Mastering arcane powers</h2> We have only seen a tiny bit of what the Scala type system allows you to do. If this small hint at the high wizardry that's possible got you excited and you want to master the arcane powers of type-level programming, a good starting resource is the blog series Type-Level Programming in Scala</a> by Mark Harrah. After that, I recommend to have a look at Shapeless</a>, a library in which Miles Sabin explores the limits of the Scala language in terms of generic and polytypic programming. Creating something useful</h2> Reading books, doing tutorials and toying around with a new language is all fine to get a certain understanding of it, but in order to become really comfortable with Scala and its paradigm and learn how to think the Scala way, I highly recommend that you start creating something useful with it – something that is clearly more than a toy application (this is true for learning any language, in my opinion). By tackling a real-world problem and trying to create a useful and usable application, you will also get a good overview of the ecosystem of libraries and get a feeling for which of those can be of service to you in specific situations. In order to find relevant libraries or get updates of ones you are interested in, you should subscribe to implicit.ly</a> and regularly take a look at Scala projects on GitHub</a>. It's all about the web</h3> These days, most applications written in Scala will be some kind of server applications, often with a RESTful interface exposed via HTTP and a web frontend. If the actor model for concurrency is a good fit for your use case and you hence choose to use the Akka</a> toolkit, an excellent choice for exposing a REST API via HTTP is Spray Routing</a>. This is a great tool if you don't need a web frontend, or if you want to develop a single-page web application that will talk to your backend by means of a REST API. If you need something less minimalistic, of course, Play</a>, which is part of the Typesafe stack, is a good choice, especially if you seek something that is widely adopted and hence well supported. Living in a concurrent world</h3> If after our two parts on actors and Akka, you think that Akka is a good fit for your application, you will likely want to learn a lot more about it before getting serious with it. While the Akka documentation</a> is pretty exhaustive and thus serves well as a reference, I think that the best choice for actually learning Akka is Derek Wyatt's book Akka Concurrency</a>, a preliminary version of which is already available as a PDF. Once you have got serious with Akka, you should definitely subscribe to Let It Crash</a>, which provides you with news and advanced tips and tricks and regarding all things Akka. If actors are not your thing and you prefer a concurrency model allowing you to leverage the composability of Futures, your library of choice is probably Twitter's Finagle</a>. It allows you to modularize your application as a bunch of small remote services, with support for numerous popular protocols out of the box. Contributing</h2> Another really great way to learn a lot about Scala quickly is to start contributing to one or more open source projects – preferably to libraries you have been using while working on your own application. Of course, this is nothing that is specific to Scala, but still I think it deserves to be mentioned. If you have only just learned Scala and are not using it at your day job already, it's nearly the only choice you have to learn from other, more experienced Scala developers. It forces you to read a lot of Scala code from other people, discovering how to do things differently, possibly more idiomatically, and you can have those experienced developers review your code in pull requests. I have found the Scala community at large to be very friendly and helpful, so don't shy away from contributing, even if you think you're too much of a rookie when it comes to Scala. While some projects might have their own way of doing things, it's certainly a good idea to study the Scala Style Guide</a> to get familiar with common coding conventions. Connecting</h2> By contributing to open source projects, you have already started connecting with the Scala community. However, you may not have the time to do that, or you may prefer other ways of connecting to like-minded people. Try finding a local Scala user group or meetup. If you don't like connecting in meatspace, the scala-user mailing list/Google group</a> and the Scala Gitter channel</a> may be good alternatives. Other resources</h2> Regardless of which of the paths outlined above you follow, there are a few resources I would like to recommend: Functional Programming in Scala</a> by Paul Chiusano and Rúnar Bjarnason, which is currently available as an Early Access Edition and teaches you a lot more about functional programming and thinking about problems with a functional mindset.</li> The Scala Documentation Site</a>, which, for some reason, is not linked very prominently from the main Scala website, especially the available guides</a> and tutorials</a>.</li> </ul> Conclusion</h2> I hope you have enjoyed this series and that I could get you excited about Scala. While this series is coming to an end, I seriously hope that it's just the beginning of your journey through Scala land. Let me know in the comments how your journey went so far and where you think it will go from here. The Neophyte's Guide to Scala Part 15: Dealing with Failure in Actor Systems Unknown — Wed, 20 Mar 2013 00:00:00 +0000 In the previous part of this series, I introduced you to the second cornerstone of Scala concurrency: The actor model, which complements the model based on composable futures backed by promises. You learnt how to define and create actors, how to send messages to them and how an actor processes these messages, possibly modifying its internal state as a result or asynchronously sending a response message to the sender. While that was hopefully enough to get you interested in the actor model for concurrency, I left out some crucial concepts you will want to know about before starting to develop actor-based applications that consist of more than a simple echo actor. The actor model is meant to help you achieve a high level of fault tolerance. In this article, we are going to have a look at how to deal with failure in an actor-based application, which is fundamentally different from error handling in a traditional layered server architecture. The way you deal with failure is closely linked to some core Akka concepts and to some of the elements an actor system in Akka consists of. Hence, this article will also serve as a guide to those ideas and components. Actor hierarchies</h2> Before going into what happens when an error occurs in one of your actors, it's essential to introduce one crucial idea underlying the actor approach to concurrency – an idea that is the very foundation for allowing you to build fault-tolerant concurrent applications: Actors are organized in a hierarchy. So what does this mean? First of all, it means that every single of your actors has got a parent actor, and that each actor can create child actors. Basically, you can think of an actor system as a pyramid of actors. Parent actors watch over their children, just as in real life, taking care that they get back on their feet if they stumble. You will see shortly how exactly this is done. The guardian actor</h3> In the previous article, we only had two different actors, a Barista</code> actor and a Customer</code> actor. I will not repeat their rather trivial implementations, but focus on how we created instances of these actor types: import akka.actor.ActorSystem val system = ActorSystem("Coffeehouse") val barista = system.actorOf(Props[Barista], "Barista") val customer = system.actorOf(Props(classOf[Customer], barista), "Customer") </code></pre> As you can see, we create these two actors by calling the actorOf</code> method defined on the ActorSystem</code> type. So what is the parent of these two actors? Is it the actor system? Not quite, but close. The actor system is not an actor itself, but it has got a so-called guardian actor that serves as the parent of all root-level user actors, i.e. actors we create by calling actorOf</code> on our actor system. There shouldn't be a whole lot of actors in your system that are children of the guardian actor. It really makes more sense to have only a few top-level actors, each of them delegating most of the work to their children. Actor paths</h3> The hierarchical structure of an actor system becomes apparent when looking at the actor paths of the actors you create. These are basically URLs by which actors can be addressed. You can get an actor's path by calling path</code> on its ActorRef</code>: barista.path // => akka.actor.ActorPath = akka://Coffeehouse/user/Barista customer.path // => akka.actor.ActorPath = akka://Coffeehouse/user/Customer </code></pre> The akka</code> protocol is followed by the name of our actor system, the name of the user guardian actor and, finally, the name we gave our actor when calling actorOf</code> on the system. In the case of remote actors, running on different machines, you would additionally see a host and a port. Actor paths can be used to look up another actor. For example, instead of requiring the barista reference in its constructor, the Customer</code> actor could call the actorSelection</code> method on its ActorContext</code>, passing in a relative path to retrieve a reference to the barista: context.actorSelection("../Barista") </code></pre> However, while being able to look up an actor by its path can sometimes come in handy, it's often a much better idea to pass in dependencies in the constructor, just as we did before. Too much intimate knowledge about where your dependencies are located in the actor system makes your system more susceptible to bugs, and it will be difficult to refactor later on. An example hierarchy</h3> To illustrate how parents watch over their children and what this has got to do with keeping your system fault-tolerant, I'm going to stick to the domain of the coffeehouse. Let's give the Barista</code> actor a child actor to which it can delegate some of the work involved in running a coffeehouse. If we really were to model the work of a barista, we were likely giving them a bunch of child actors for all the various subtasks. But to keep this article focused, we have to be a little simplistic with our example. Let's assume that the barista has got a register. It processes transactions, printing appropriate receipts and incrementing the day's sales so far. Here is a first version of it: import akka.actor._ object Register { sealed trait Article case object Espresso extends Article case object Cappuccino extends Article case class Transaction(article: Article) } class Register extends Actor { import Register._ import Barista._ var revenue = 0 val prices = Map[Article, Int](Espresso -> 150, Cappuccino -> 250) def receive = { case Transaction(article) => val price = prices(article) sender ! createReceipt(price) revenue += price } def createReceipt(price: Int): Receipt = Receipt(price) } </code></pre> It contains an immutable map of the prices for each article, and an integer variable representing the revenue. Whenever it receives a Transaction</code> message, it increments the revenue accordingly and returns a printed receipt to the sender. The Register</code> actor, as already mentioned, is supposed to be a child actor of the Barista</code> actor, which means that we will not create it from our actor system, but from within our Barista</code> actor. The initial version of our actor-come-parent looks like this: object Barista { case object EspressoRequest case object ClosingTime case class EspressoCup(state: EspressoCup.State) object EspressoCup { sealed trait State case object Clean extends State case object Filled extends State case object Dirty extends State } case class Receipt(amount: Int) } class Barista extends Actor { import Barista._ import Register._ import EspressoCup._ import context.dispatcher import akka.util.Timeout import akka.pattern.ask import akka.pattern.pipe import concurrent.duration._ implicit val timeout = Timeout(4.seconds) val register = context.actorOf(Props[Register], "Register") def receive = { case EspressoRequest => val receipt = register ? Transaction(Espresso) receipt.map((EspressoCup(Filled), _)).pipeTo(sender) case ClosingTime => context.stop(self) } } </code></pre> First off, we define the message types that our Barista</code> actor is able to deal with. An EspressoCup</code> can have one out of a fixed set of states, which we ensure by using a sealed trait. The more interesting part is to be found in the implementation of the Barista</code> class. The dispatcher</code>, ask</code>, and pipe</code> imports as well as the implicit timeout are required because we make use of Akka's ask syntax and futures in our Receive</code> partial function: When we receive an EspressoRequest</code>, we ask the Register</code> actor for a Receipt</code> for our Transaction</code>. This is then combined with a filled espresso cup and piped to the sender, which will thus receive a tuple of type (EspressoCup, Receipt)</code>. This kind of delegating subtasks to child actors and then aggregating or amending their work is typical for actor-based applications. Also, note how we create our child actor by calling actorOf</code> on our ActorContext</code> instead of the ActorSystem</code>. By doing so, the actor we create becomes a child actor of the one who called this method, instead of a top-level actor whose parent is the guardian actor. Finally, here is our Customer</code> actor, which, like the Barista</code> actor, will sit at the top level, just below the guardian actor: object Customer { case object CaffeineWithdrawalWarning } class Customer(coffeeSource: ActorRef) extends Actor with ActorLogging { import Customer._ import Barista._ import EspressoCup._ def receive = { case CaffeineWithdrawalWarning => coffeeSource ! EspressoRequest case (EspressoCup(Filled), Receipt(amount)) => log.info(s"yay, caffeine for ${self}!") } } </code></pre> It is not terribly interesting for our tutorial, which focuses more on the Barista</code> actor hierarchy. What's new is the use of the ActorLogging</code> trait, which allows us to write to the log instead of printing to the console. Now, if we create our actor system and populate it with a Barista</code> and two Customer</code> actors, we can happily feed our two under-caffeinated addicts with a shot of black gold: import Customer._ val system = ActorSystem("Coffeehouse") val barista = system.actorOf(Props[Barista], "Barista") val customerJohnny = system.actorOf(Props(classOf[Customer], barista), "Johnny") val customerAlina = system.actorOf(Props(classOf[Customer], barista), "Alina") customerJohnny ! CaffeineWithdrawalWarning customerAlina ! CaffeineWithdrawalWarning </code></pre> If you try this out, you should see two log messages from happy customers. To crash or not to crash?</h2> Of course, what we are really interested in, at least in this article, is not happy customers, but the question of what happens if things go wrong. Our register is a fragile device – its printing functionality is not as reliable as it should be. Every so often, a paper jam causes it to fail. Let's add a PaperJamException</code> type to the Register</code> companion object: class PaperJamException(msg: String) extends Exception(msg) </code></pre> Then, let's change the createReceipt</code> method in our Register</code> actor accordingly: def createReceipt(price: Int): Receipt = { import util.Random if (Random.nextBoolean()) throw new PaperJamException("OMG, not again!") Receipt(price) } </code></pre> Now, when processing a Transaction</code> message, our Register</code> actor will throw a PaperJamException</code> in about half of the cases. What effect does this have on our actor system, or on our whole application? Luckily, Akka is very robust and not affected by exceptions in our code at all. What happens, though, is that the parent of the misbehaving child is notified – remember that parents are watching over their children, and this is the situation where they have to decide what to do. Supervisor strategies</h3> The whole act of being notified about exceptions in child actors, however, is not handled by the parent actor's Receive</code> partial function, as that would confound the parent actor's own behaviour with the logic for dealing with failure in its children. Instead, the two responsibilities are clearly separated. Each actor defines its own supervisor strategy, which tells Akka how to deal with certain types of errors occurring in your children. There are basically two different types of supervisor strategy, the OneForOneStrategy</code> and the AllForOneStrategy</code>. Choosing the former means that the way you want to deal with an error in one of your children will only affect the child actor from which the error originated, whereas the latter will affect all of your child actors. Which of those strategies is best depends a lot on your individual application. Regardless of which type of SupervisorStrategy</code> you choose for your actor, you will have to specify a Decider</code>, which is a PartialFunction[Throwable, Directive]</code> – this allows you to match against certain subtypes of Throwable</code> and decide for each of them what's supposed to happen to your problematic child actor (or all your child actors, if you chose the all-for-one strategy). Directives</h3> Here is a list of the available directives: sealed trait Directive case object Resume extends Directive case object Restart extends Directive case object Stop extends Directive case object Escalate extends Directive </code></pre> Resume: If you choose to Resume</code>, this probably means that you think of your child actor as a little bit of a drama queen. You decide that the exception was not so exceptional after all – the child actor or actors will simply resume processing messages as if nothing extraordinary had happened. </li> Restart: The Restart</code> directive causes Akka to create a new instance of your child actor or actors. The reasoning behind this is that you assume that the internal state of the child/children is corrupted in some way so that it can no longer process any further messages. By restarting the actor, you hope to put it into a clean state again. </li> Stop: You effectively kill the actor. It will not be restarted. </li> Escalate: If you choose to Escalate</code>, you probably don't know how to deal with the failure at hand. You delegate the decision about what to do to your own parent actor, hoping they are wiser than you. If an actor escalates, they may very well be restarted themselves by their parent, as the parent will only decide about its own child actors. </li> </ul> The default strategy</h3> You don't have to specify your own supervisor strategy in each and every actor. In fact, we haven't done that so far. This means that the default supervisor strategy will take effect. It looks like this: final val defaultStrategy: SupervisorStrategy = { def defaultDecider: Decider = { case _: ActorInitializationException ⇒ Stop case _: ActorKilledException ⇒ Stop case _: Exception ⇒ Restart } OneForOneStrategy()(defaultDecider) } </code></pre> This means that for exceptions other than ActorInitializationException</code> or ActorKilledException</code>, the respective child actor in which the exception was thrown will be restarted. Hence, when a PaperJamException</code> occurs in our Register</code> actor, the supervisor strategy of the parent actor (the barista) will cause the Register</code> to be restarted, because we haven't overridden the default strategy. If you try this out, you will likely see an exception stacktrace in the log, but nothing about the Register</code> actor being restarted. Let's verify that this is really happening. To do so, however, you will need to learn about the actor lifecycle. The actor lifecycle</h3> To understand what the directives of a supervisor strategy actually do, it's crucial to know a little bit about an actor's lifecycle. Basically, it boils down to this: when created via actorOf</code>, an actor is started. It can then be restarted an arbitrary number of times, in case there is a problem with it. Finally, an actor can be stopped, ultimately leading to its death. There are numerous lifecycle hook methods that an actor implementation can override. It's also important to know their default implementations. Let's go through them briefly: preStart: Called when an actor is started, allowing you to do some initialization logic. The default implementation is empty.</li> postStop: Empty by default, allowing you to clean up resources. Called after stop</code> has been called for the actor.</li> preRestart: Called right before a crashed actor is restarted. By default, it stops all children of that actor and then calls postStop</code> to allow cleaning up of resources.</li> postRestart: Called immediately after an actor has been restarted. Simply calls preStart</code> by default.</li> </ul> Let's see if our Register</code> gets indeed restarted upon failure by simply adding some log output to its postRestart</code> method. Make the Register</code> type extend the ActorLogging</code> trait and add the following method to it: override def postRestart(reason: Throwable) { super.postRestart(reason) log.info(s"Restarted because of ${reason.getMessage}") } </code></pre> Now, if you send the two Customer</code> actors a bunch of CaffeineWithdrawalWarning</code> messages, you should see the one or the other of those log outputs, confirming that our Register</code> actor has been restarted. Death of an actor</h3> Often, it doesn't make sense to restart an actor again and again – think of an actor that talks to some other service over the network, and that service has been unreachable for a while. In such cases, it is a very good idea to tell Akka how often to restart an actor within a certain period of time. If that limit is exceeded, the actor is instead stopped and hence dies. Such a limit can be configured in the constructor of the supervisor strategy: import scala.concurrent.duration._ import akka.actor.OneForOneStrategy import akka.actor.SupervisorStrategy.Restart OneForOneStrategy(10, 2.minutes) { case _ => Restart } </code></pre> The self-healing system?</h3> So, is our system running smoothly, healing itself whenever this damn paper jam occurs? Let's change our log output: override def postRestart(reason: Throwable) { super.postRestart(reason) log.info(s"Restarted, and revenue is $revenue cents") } </code></pre> And while we are at it, let's also add some more logging to our Receive</code> partial function, making it look like this: def receive = { case Transaction(article) => val price = prices(article) sender ! createReceipt(price) revenue += price log.info(s"Revenue incremented to $revenue cents") } </code></pre> Ouch! Something is clearly not as it should be. In the log, you will see the revenue increasing, but as soon as there is a paper jam and the Register</code> actor restarts, it is reset to 0</code>. This is because restarting indeed means that the old instance is discarded and a new one created as per the Props</code> we initially passed to actorOf</code>. Of course, we could change our supervisor strategy, so that it resumes in case of a PaperJamException</code>. We would have to add this to the Barista</code> actor: val decider: PartialFunction[Throwable, Directive] = { case _: PaperJamException => Resume } override def supervisorStrategy: SupervisorStrategy = OneForOneStrategy()(decider.orElse(SupervisorStrategy.defaultStrategy.decider)) </code></pre> Now, the actor is not restarted upon a PaperJamException</code>, so its state is not reset. Error kernel</h3> So we just found a nice solution to preserve the state of our Register</code> actor, right? Well, sometimes, simply resuming might be the best thing to do. But let's assume that we really have to restart it, because otherwise the paper jam will not disappear. We can simulate this by maintaining a boolean flag that says if we are in a paper jam situation or not. Let's change our Register</code> like so: class Register extends Actor with ActorLogging { import Register._ import Barista._ var revenue = 0 val prices = Map[Article, Int](Espresso -> 150, Cappuccino -> 250) var paperJam = false override def postRestart(reason: Throwable) { super.postRestart(reason) log.info(s"Restarted, and revenue is $revenue cents") } def receive = { case Transaction(article) => val price = prices(article) sender ! createReceipt(price) revenue += price log.info(s"Revenue incremented to $revenue cents") } def createReceipt(price: Int): Receipt = { import util.Random if (Random.nextBoolean()) paperJam = true if (paperJam) throw new PaperJamException("OMG, not again!") Receipt(price) } } </code></pre> Also remove the supervisor strategy we added to the Barista</code> actor. Now, the paper jam remains forever, until we have restarted the actor. Alas, we cannot do that without also losing important state regarding our revenue. This is where the error kernel pattern comes in. Basically, it is just a simple guideline you should always try to follow, stating that if an actor carries important internal state, then it should delegate dangerous tasks to child actors, so as to prevent the state-carrying actor from crashing. Sometimes, it may make sense to spawn a new child actor for each such task, but that's not a necessity. The essence of the pattern is to keep important state as far at the top of the actor hierarchy as possible, while pushing error-prone tasks as far to the bottom of the hierarchy as possible. Let's apply this pattern to our Register</code> actor. We will keep the revenue state in the Register</code> actor, but move the error-prone behaviour of printing the receipt to a new child actor, which we appropriately enough call ReceiptPrinter</code>. Here is the latter: object ReceiptPrinter { case class PrintJob(amount: Int) class PaperJamException(msg: String) extends Exception(msg) } class ReceiptPrinter extends Actor with ActorLogging { var paperJam = false override def postRestart(reason: Throwable) { super.postRestart(reason) log.info(s"Restarted, paper jam == $paperJam") } def receive = { case PrintJob(amount) => sender ! createReceipt(amount) } def createReceipt(price: Int): Receipt = { if (Random.nextBoolean()) paperJam = true if (paperJam) throw new PaperJamException("OMG, not again!") Receipt(price) } } </code></pre> Again, we simulate the paper jam with a boolean flag and throw an exception each time someone asks us to print a receipt while in a paper jam. Other than the new message type, PrintJob</code>, this is really just extracted from the Register</code> type. This is a good thing, not only because it moves away this dangerous operation from the stateful Register</code> actor, but it also makes our code simpler and consequently easier to reason about: The ReceiptPrinter</code> actor is responsible for exactly one thing, and the Register</code> actor has become simpler, too, now being only responsible for managing the revenue, delegating the remaining functionality to a child actor: class Register extends Actor with ActorLogging { import akka.pattern.ask import akka.pattern.pipe import context.dispatcher implicit val timeout = Timeout(4.seconds) var revenue = 0 val prices = Map[Article, Int](Espresso -> 150, Cappuccino -> 250) val printer = context.actorOf(Props[ReceiptPrinter], "Printer") override def postRestart(reason: Throwable) { super.postRestart(reason) log.info(s"Restarted, and revenue is $revenue cents") } def receive = { case Transaction(article) => val price = prices(article) val requester = sender (printer ? PrintJob(price)).map((requester, _)).pipeTo(self) case (requester: ActorRef, receipt: Receipt) => revenue += receipt.amount log.info(s"revenue is $revenue cents") requester ! receipt } } </code></pre> We don't spawn a new ReceiptPrinter</code> for each Transaction</code> message we get. Instead, we use the default supervisor strategy to have the printer actor restart upon failure. One part that merits explanation is the weird way we increment our revenue: First we ask the printer for a receipt. We map the future to a tuple containing the answer as well as the requester, which is the sender of the Transaction</code> message and pipe this to ourselves. When processing that message, we finally increment the revenue and send the receipt to the requester. The reason for that indirection is that we want to make sure that we only increment our revenue if the receipt was successfully printed. Since it is vital to never ever modify the internal state of an actor inside of a future, we have to use this level of indirection. It helps us make sure that we only change the revenue within the confines of our actor, and not on some other thread. Assigning the sender</code> to a val</code> is necessary for similar reasons: When mapping a future, we are no longer in the context of our actor either – since sender</code> is a method, it would now likely return the reference to some other actor that has sent us a message, not the one we intended. Now, our Register</code> actor is safe from constantly being restarted, yay! Of course, the very idea of having the printing of the receipt and the management of the revenue in one place is questionable. Having them together came in handy for demonstrating the error kernel pattern. Yet, it would certainly be a lot better to seperate the receipt printing from the revenue management altogether, as these are two concerns that don't really belong together. Timeouts</h3> Another thing that we may want to improve upon is the handling of timeouts. Currently, when an exception occurs in the ReceiptPrinter</code>, this leads to an AskTimeoutException</code>, which, since we are using the ask syntax, comes back to the Barista</code> actor in an unsuccessfully completed Future</code>. Since the Barista</code> actor simply maps over that future (which is success-biased) and then pipes the transformed result to the customer, the customer will also receive a Failure</code> containing an AskTimeoutException</code>. The Customer</code> didn't ask for anything, though, so it is certainly not expecting such a message, and in fact, it currently doesn't handle these messages. Let's be friendly and send customers a ComebackLater</code> message – this is a message they already understand, and it makes them try to get an espresso at a later point. This is clearly better, as the current solution means they will never know that they will not get their espresso. To achieve this, let's recover from AskTimeoutException</code> failures by mapping them to ComebackLater</code> messages. The Receive</code> partial function of our Barista</code> actor thus now looks like this: def receive = { case EspressoRequest => val receipt = register ? Transaction(Espresso) receipt.map((EspressoCup(Filled), _)).recover { case _: AskTimeoutException => ComebackLater } pipeTo(sender) case ClosingTime => context.system.shutdown() } </code></pre> Now, the Customer</code> actors know they can try their luck later, and after trying often enough, they should finally get their eagerly anticipated espresso. Death Watch</h3> Another principle that is important in order to keep your system fault-tolerant is to keep a watch on important dependencies – dependencies as opposed to children. Sometimes, you have actors that depend on other actors without the latter being their children. This means that they can't be their supervisors. Yet, it is important to keep a watch on their state and be notified if bad things happen. Think, for instance, of an actor that is responsible for database access. You will want actors that require this actor to be alive and healthy to know when that is no longer the case. Maybe you want to switch your system to a maintenance mode in such a situation. For other use cases, simply using some kind of backup actor as a replacement for the dead actor may be a viable solution. In any case, you will need to place a watch on an actor you depend on in order to get the sad news of its passing away. This is done by calling the watch</code> method defined on ActorContext</code>. To illustrate, let's have our Customer</code> actors watch the Barista</code> – they are highly addicted to caffeine, so it's fair to say they depend on the barista: class Customer(coffeeSource: ActorRef) extends Actor with ActorLogging { import context.dispatcher context.watch(coffeeSource) def receive = { case CaffeineWithdrawalWarning => coffeeSource ! EspressoRequest case (EspressoCup(Filled), Receipt(amount)) => log.info(s"yay, caffeine for ${self}!") case ComebackLater => log.info("grumble, grumble") context.system.scheduler.scheduleOnce(300.millis) { coffeeSource ! EspressoRequest } case Terminated(barista) => log.info("Oh well, let's find another coffeehouse...") } } </code></pre> We start watching our coffeeSource</code> in our constructor, and we added a new case for messages of type Terminated</code> – this is the kind of message we will receive from Akka if an actor we watch dies. Now, if we send a ClosingTime</code> to the message and the Barista</code> tells its context to stop itself, the Customer</code> actors will be notified. Give it a try, and you should see their output in the log. Instead of simply logging that we are not amused, this could just as well initiate some failover logic, for instance. Summary</h2> In this part of the series, which is the second one dealing with actors and Akka, you got to know some of the important components of an actor system, all while learning how to put the tools provided by Akka and the ideas behind it to use in order to make your system more fault-tolerant. While there is still a lot more to learn about the actor model and Akka, we shall leave it at that for now, as this would go beyond the scope of this series. In the next part, which shall bring this series to a conclusion, I will point you to a bunch of Scala resources you may want to peruse to continue your journey through Scala land, and if actors and Akka got you excited, there will be something in there for you, too. The Neophyte's Guide to Scala Part 14: The Actor approach to concurrency Unknown — Wed, 27 Feb 2013 00:00:00 +0000 After several articles about how you can leverage the Scala type system to achieve a great amount of both flexibility and compile-time safety, we are now shifting back to a topic that we already tackled previously in this series: Scala's take on concurrency. In these</a> earlier articles</a>, you learned about an approach that allows you to work asynchronously by making use of composable Futures. This approach is a very good fit for numerous problems. However, it's not the only one Scala has to offer. A second cornerstone of Scala concurrency is the Actor model. It provides an approach to concurrency that is entirely based on passing messages between processes. Actors are not a new idea – the most prominent implementation of this model can be found in Erlang. The Scala core library has had its own actors library for a long time, but it faces the destiny of deprecation in the coming Scala version 2.11, when it will ultimately be replaced by the actors implementation provided by the Akka</a> toolkit, which has been a de-facto standard for actor-based development with Scala for quite a while. In this article, you will be introduced to the rationale behind Akka's actor model and learn the basics of coding within this paradigm using the Akka toolkit. It is by no means an in-depth discussion of everything you need to know about Akka actors, and in that, it differs from most of the previous articles in this series. Rather, the intention is to familiarize you with the Akka mindset and serve as an initial spark to get you excited about it. The problems with shared mutable state</h2> The predominant approach to concurrency today is that of shared mutable state – a large number of stateful objects whose state can be changed by multiple parts of your application, each running in their own thread. Typically, the code is interspersed with read and write locks, to make sure that the state can only be changed in a controlled way and prevent multiple threads from mutating it simultaneously. At the same time, we are trying hard not to lock too big a block of code, as this can drastically slow down the application. More often than not, code like this has originally been written without having concurrency in mind at all – only to be made fit for a multi-threaded world once the need arose. While writing software without the need for concurrency like this leads to very straightforward code, adapting it to the needs of a concurrent world leads to code that is really, really difficult to read and understand. The core problem is that low-level synchronization constructs like locks and threads are very hard to reason about. As a consequence, it's very hard to get it right: If you can't easily reason about what's going on, you can be sure that nasty bugs will ensue, from race conditions to deadlocks or just strange behaviour – maybe you'll only notice after some months, long after your code has been deployed to your production servers. Also, working with these low-level constructs makes it a real challenge to achieve an acceptable performance. The Actor model</h2> The Actor programming model is aimed at avoiding all the problems described above, allowing you to write highly performant concurrent code that is easy to reason about. Unlike the widely used approach of shared mutable state, it requires you to design and write your application from the ground up with concurrency in mind – it's not really possible to add support for it later on. The idea is that your application consists of lots of light-weight entities called actors. Each of these actors is responsible for only a very small task, and is thus easy to reason about. A more complex business logic arises out of the interaction between several actors, delegating tasks to others or passing messages to collaborators for other reasons. The Actor System</h3> Actors are pitiful creatures: They cannot live on their own. Rather, each and every actor in Akka resides in and is created by an actor system. Aside from allowing you to create and find actors, an ActorSystem</code> provides for a whole bunch of additional functionality, none of which shall concern us right now. In order to try out the example code, please add the following resolver and dependency to your SBT-based Scala 2.10 project first: resolvers += "Typesafe Releases" at "http://repo.typesafe.com/typesafe/releases" libraryDependencies += "com.typesafe.akka" %% "akka-actor" % "2.2.3" </code></pre> Now, let's create an ActorSystem</code>. We'll need it as an environment for our actors: import akka.actor.ActorSystem object Barista extends App { val system = ActorSystem("Barista") system.shutdown() } </code></pre> We created a new instance of ActorSystem</code> and gave it the name "Barista"</code> – we are returning to the domain of coffee, which should be familiar from the article on composable futures</a>. Finally, we are good citizens and shut down our actor system once we no longer need it. Defining an actor</h3> Whether your application consists of a few dozen or a few million actors totally depends on your use case, but Akka is absolutely okay with a few million. You might be baffled by this insanely high number. It's important to understand that there is not a one-to-one relationship between an actor and a thread. You would soon run out of memory if that were the case. Rather, due to the non-blocking nature of actors, one thread can execute many actors – switching between them depending on which of them has messages to be processed. To understand what is actually happening, let's first create a very simple actor, a Barista</code> that can receive orders but doesn't really do anything apart from printing messages to the console: sealed trait CoffeeRequest case object CappuccinoRequest extends CoffeeRequest case object EspressoRequest extends CoffeeRequest import akka.actor.Actor class Barista extends Actor { def receive = { case CappuccinoRequest => println("I have to prepare a cappuccino!") case EspressoRequest => println("Let's prepare an espresso.") } } </code></pre> First, we define the types of messages that our actor understands. Typically, case classes are used for messages sent between actors if you need to pass along any parameters. If all the actor needs is an unparameterized message, this message is typically represented as a case object – which is exactly what we are doing here. In any case, it's crucial that your messages are immutable, or else bad things will happen. Next, let's have a look at our class Barista</code>, which is the actual actor, extending the aptly named Actor</code> trait. Said trait defines a method receive</code> which returns a value of type Receive</code>. The latter is really only a type alias for PartialFunction[Any, Unit]</code>. Processing messages</h3> So what's the meaning of this receive</code> method? The return type, PartialFunction[Any, Unit]</code> may seem strange to you in more than one respect. In a nutshell, the partial function returned by the receive</code> method is responsible for processing your messages. Whenever another part of your software – be it another actor or not – sends your actor a message, Akka will eventually let it process this message by calling the partial function returned by your actor's receive</code> method, passing it the message as an argument. Side-effecting</h4> When processing a message, an actor can do whatever you want it to, apart from returning a value. Wat!? As the return type of Unit</code> suggests, your partial function is side-effecting. This might come as a bit of a shock to you after we emphasized the usage of pure functions all the time. For a concurrent programming model, this actually makes a lot of sense. Actors are where your state is located, and having some clearly defined places where side-effects will occur in a controllable manner is totally fine – each message your actor receives is processed in isolation, one after another, so there is no need to reason about synchronization or locks. Untyped</h4> But... this partial function is not only side-effecting, it's also as untyped as you can get in Scala, expecting an argument of type Any</code>. Why is that, when we have such a powerful type system at our fingertips? This has a lot to do with some important design choices in Akka that allow you to do things like forwarding messages to other actors, installing load balancing or proxying actors without the sender having to know anything about them and so on. In practice, this is usually not a problem. With the messages themselves being strongly typed, you typically use pattern matching for processing those types of messages you are interested in, just as we did in our tiny example above. Sometimes though, the weakly typed actors can indeed lead to nasty bugs the compiler can't catch for you. If you have grown to love the benefits of a strong type system and think you don't want to go away from that at any costs for some parts of your application, you may want to look at Akka Typed</a> feature. Asynchronous and non-blocking</h4> I wrote above that Akka would let your actor eventually process a message sent to it. This is important to keep in mind: Sending a message and processing it is done in an asynchronous and non-blocking fashion. The sender will not be blocked until the message has been processed by the receiver. Instead, they can immediately continue with their own work. Maybe they expect to get a messsage from your actor in return after a while, or maybe they are not interested in hearing back from your actor at all. What really happens when some component sends a message to an actor is that this message is delivered to the actor's mailbox, which is basically a queue. Placing a message in an actor's mailbox is a non-blocking operation, i.e. the sender doesn't have to wait until the message is actually enqueued in the recipient's mailbox. The dispatcher will notice the arrival of a new message in an actor's mailbox, again asynchronously. If the actor is not already processing a previous message, it is now allocated to one of the threads available in the execution context. Once the actor is done processing any previous messages, the dispatcher sends it the next message from its mailbox for processing. The actor blocks the thread to which it is allocated for as long as it takes to process the message. While this doesn't block the sender of the message, it means that lengthy operations degrade overall performance, as all the other actors have to be scheduled for processing messages on one of the remaining threads. Hence, a core principle to follow for your Receive</code> partial functions is to spend as little time inside them as possible. Most importantly, avoid calling blocking code inside your message processing code, if possible at all. Of course, this is something you can't prevent doing completely – the majority of database drivers nowadays is still blocking, and you will want to be able to persist data or query for it from your actor-based application. There are solutions to this dilemma, but we won't cover them in this introductory article. Creating an actor</h3> Defining an actor is all well and good, but how do we actually use our Barista</code> actor in our application? To do that, we have to create a new instance of our Barista</code> actor. You might be tempted to do it the usual way, by calling its constructor like so: val barista = new Barista // will throw exception </code></pre> This will not work! Akka will thank you with an ActorInitializationException</code>. The thing is, in order for the whole actor thingie to work properly, your actors need to be managed by the ActorSystem</code> and its components. Hence, you have to ask the actor system for a new instance of your actor: import akka.actor.{ActorRef, Props} val barista: ActorRef = system.actorOf(Props[Barista], "Barista") </code></pre> The actorOf</code> method defined on ActorSystem</code> expects a Props</code> instance, which provides a means of configuring newly created actors, and, optionally, a name for your actor instance. We are using the simplest form of creating such a Props</code> instance, providing the apply</code> method of the companion object with a type parameter. Akka will then create a new instance of the actor of the given type by calling its default constructor. Be aware that the type of the object returned by actorOf</code> is not Barista</code>, but ActorRef</code>. Actors never communicate with another directly and hence there are supposed to be no direct references to actor instances. Instead, actors or other components of your application aquire references to the actors they need to send messages to. Thus, an ActorRef</code> acts as some kind of proxy to the actual actor. This is convenient because an ActorRef</code> can be serialized, allowing it to be a proxy for a remote actor on some other machine. For the component aquiring an ActorRef</code>, the location of the actor – local in the same JVM or remote on some other machine – is completely transparent. We call this property location transparency. Please note that ActorRef</code> is not parameterized by type. Any ActorRef</code> can be exchanged for another, allowing you to send arbitrary messages to any ActorRef</code>. This is by design and, as already mentioned above, allows for easily modifying the topology of your actor system wihout having to make any changes to the senders. Sending messages</h3> Now that we have created an instance of our Barista</code> actor and got an ActorRef</code> linked to it, we can send it a message. This is done by calling the !</code> method on the ActorRef</code>: barista ! CappuccinoRequest barista ! EspressoRequest println("I ordered a cappuccino and an espresso") </code></pre> Calling the !</code> is a fire-and-forget operation: You tell the Barista</code> that you want a cappuccino, but you don't wait for their response. It's the most common way in Akka for interacting with other actors. By calling this method, you tell Akka to enqueue your message in the recipient's mailbox. As described above, this doesn't block, and eventually the recipient actor will process your message. Due to the asynchronous nature, the result of the above code is not deterministic. It might look like this: I have to prepare a cappuccino! I ordered a cappuccino and an espresso Let's prepare an espresso. </code></pre> Even though we first sent the two messages to the Barista</code> actor's mailbox, between the processing of the first and second message, our own output is printed to the console. Answering to messages</h3> Sometimes, being able to tell others what to do just doesn't cut it. You would like to be able to answer by in turn sending a message to the sender of a message you got – all asynchronously of course. To enable you to do that and lots of other things that are of no concern to us right now, actors have a method called sender</code>, which returns the ActorRef</code> of the sender of the last message, i.e. the one you are currently processing. But how does it know about that sender? The answer can be found in the signature of the !</code> method, which has a second, implicit parameter list: def !(message: Any)(implicit sender: ActorRef = Actor.noSender): Unit </code></pre> When called from an actor, its ActorRef</code> is passed on as the implicit sender</code> argument. Let's change our Barista</code> so that they immediately send a Bill</code> to the sender of a CoffeeRequest</code> before printing their usual output to the console: case class Bill(cents: Int) case object ClosingTime class Barista extends Actor { def receive = { case CappuccinoRequest => sender ! Bill(250) println("I have to prepare a cappuccino!") case EspressoRequest => sender ! Bill(200) println("Let's prepare an espresso.") case ClosingTime => context.system.shutdown() } } </code></pre> While we are at it, we are introducing a new message, ClosingTime</code>. The Barista</code> reacts to it by shutting down the actor system, which they, like all actors, can access via their ActorContext</code>. Now, let's introduce a second actor representing a customer: case object CaffeineWithdrawalWarning class Customer(caffeineSource: ActorRef) extends Actor { def receive = { case CaffeineWithdrawalWarning => caffeineSource ! EspressoRequest case Bill(cents) => println(s"I have to pay $cents cents, or else!") } } </code></pre> This actor is a real coffee junkie, so it needs to be able to order new coffee. We pass it an ActorRef</code> in the constructor – for the Customer</code>, this is simply its caffeineSource</code> – it doesn't know whether this ActorRef</code> points to a Barista</code> or something else. It knows that it can send CoffeeRequest</code> messages to it, and that is all that matters to them. Finally, we need to create these two actors and send the customer a CaffeineWithdrawalWarning</code> to get things rolling: val barista = system.actorOf(Props[Barista], "Barista") val customer = system.actorOf(Props(classOf[Customer], barista), "Customer") customer ! CaffeineWithdrawalWarning barista ! ClosingTime </code></pre> Here, for the Customer</code> actor, we are using a different factory method for creating a Props</code> instance: We pass in the type of the actor we want to have instantiated as well as the constructor arguments that actor takes. We need to do this because we want to pass the ActorRef</code> of our Barista</code> actor to the constructor of the Customer</code> actor. Sending the CaffeineWithdrawalWarning</code> to the customer makes it send an EspressoRequest</code> to the barista who will then send a Bill</code> back to the customer. The output of this may look like this: Let's prepare an espresso. I have to pay 200 cents, or else! </code></pre> First, while processing the EspressoRequest</code> message, the Barista</code> sends a message to the sender of that message, the Customer</code> actor. However, this operation doesn't block until the latter processes it. The Barista</code> actor can continue processing the EspressoRequest</code> immediately, and does this by printing to the console. Shortly after, the Customer</code> starts to process the Bill</code> message and in turn prints to the console. Asking questions</h3> Sometimes, sending an actor a message and expecting a message in return at some later time isn't an option – the most common place where this is the case is in components that need to interface with actors, but are not actors themselves. Living outside of the actor world, they cannot receive messages. For situations such as these, there is Akka's ask support, which provides some sort of bridge between actor-based and future-based concurrency. From the client perspective, it works like this: import akka.pattern.ask import akka.util.Timeout import scala.concurrent.duration._ implicit val timeout = Timeout(2.second) implicit val ec = system.dispatcher val f: Future[Any] = barista2 ? CappuccinoRequest f.onSuccess { case Bill(cents) => println(s"Will pay $cents cents for a cappuccino") } </code></pre> First, you need to import support for the ask syntax and create an implicit timeout for the Future</code> returned by the ?</code> method. Also, the Future</code> needs an ExecutionContext</code>. Here, we simply use the default dispatcher of our ActorSystem</code>, which is conveniently also an ExecutionContext</code>. As you can see, the returned future is untyped – it's a Future[Any]</code>. This shouldn't come as a surprise, since it's really a received message from an actor, and those are untyped, too. For the actor that is being asked, this is actually the same as sending some message to the sender of a processed message. This is why asking our Barista</code> works out of the box without having to change anything in our Barista</code> actor. Once the actor being asked sends a message to the sender, the Promise</code> belonging to the returned Future</code> is completed. Generally, telling is preferable to asking, because it's more resource-sparing. Akka is not for polite people! However, there are situations where you really need to ask, and then it's perfectly fine to do so. Stateful actors</h3> Each actor may maintain an internal state, but that's not strictly necessary. Sometimes, a large part of the overall application state consists of the information carried by the immutable messages passed between actors. An actor only ever processes one message at a time. While doing so, it may modify its internal state. This means that there is some kind of mutable state in an actor, but since each message is processed in isolation, there is no way the internal state of our actor can get messed up due to concurrency problems. To illustrate, let's turn our stateless Barista</code> into an actor carrying state, by simply counting the number of orders: class Barista extends Actor { var cappuccinoCount = 0 var espressoCount = 0 def receive = { case CappuccinoRequest => sender ! Bill(250) cappuccinoCount += 1 println(s"I have to prepare cappuccino #$cappuccinoCount") case EspressoRequest => sender ! Bill(200) espressoCount += 1 println(s"Let's prepare espresso #$espressoCount.") case ClosingTime => context.system.shutdown() } } </code></pre> We introduced two vars</code>, cappuccinoCount</code> and espressoCount</code> that are incremented with each respective order. This is actually the first time in this series that we have used a var</code>. While to be avoided in functional programming, they are really the only way to allow your actors to carry state. Since each message is processed in isolation, our above code is similar to using AtomicInteger</code> values in a non-actor environment. Conclusion</h2> And here ends our introduction to the actor programming model for concurrency and how to work within this paradigm using Akka. While we have really only scratched the surface and have ignored some important concepts of Akka, I hope to have given enough of an insight into this approach to concurrency to give you a basic understanding and get you interested in learning more. In the coming articles, I will elaborate our little example, adding some meaningful behaviour to it while introducing more of the ideas behind Akka actors, among them the question of how errors are handled in an actor system. P.S. Please note that starting with this article I have switched to a biweekly schedule for the remaining parts of this series. The Neophyte's Guide to Scala Part 13: Path-dependent types Unknown — Wed, 13 Feb 2013 00:00:00 +0000 In last week's article, I introduced you to the idea of type classes</a> – a pattern that allows you to design your programs to be open for extension without giving up important information about concrete types. This week, I'm going to stick with Scala's type system and talk about one of its features that distinguishes it from most other mainstream programming languages: Scala's form of dependent types, in particular path-dependent types and dependent method types. One of the most widely used arguments against static typing is that "the compiler is just in the way" and that in the end, it's all only data, so why care about building up a complex hierarchy of types? In the end, having static types is all about preventing bugs by allowing the über-smart compiler to humiliate you on a regular basis, making sure you're doing the right thing before it's too late. Using path-dependent types is one powerful way to help the compiler prevent you from introducing bugs, as it places logic that is usually only available at runtime into types. Sometimes, accidentally introducing path-dependent types can lead to frustation, though, especially if you have never heard of them. Hence, it's definitely a good idea to get familiar with them, whether you decide to put them to use or not. The problem</h2> I will start by presenting a problem that path-dependent types can help us solving: In the realm of fan fiction, the most atrocious things happen – usually, the involved characters will end up making out with each other, regardless how inappropriate it is. There is even crossover fan fiction, in which two characters from different franchises are making out with each other. However, elitist fan fiction writers look down on this. Surely there is a way to prevent such wrongdoing! Here is a first version of our domain model: object Franchise { case class Character(name: String) } class Franchise(name: String) { import Franchise.Character def createFanFiction( lovestruck: Character, objectOfDesire: Character): (Character, Character) = (lovestruck, objectOfDesire) } </code></pre> Characters are represented by instances of the Character</code> case class, and the Franchise</code> class has a method to create a new piece of fan fiction about two characters. Let's create two franchises and some characters: val starTrek = new Franchise("Star Trek") val starWars = new Franchise("Star Wars") val quark = Franchise.Character("Quark") val jadzia = Franchise.Character("Jadzia Dax") val luke = Franchise.Character("Luke Skywalker") val yoda = Franchise.Character("Yoda") </code></pre> Unfortunately, at the moment we are unable to prevent bad things from happening: starTrek.createFanFiction(lovestruck = jadzia, objectOfDesire = luke) </code></pre> Horrors of horrors! Someone has created a piece of fan fiction in which Jadzia Dax is making out with Luke Skywalker. Preposterous! Clearly, we should not allow this. Your first intuition might be to somehow check at runtime that two characters making out are from the same franchise. For example, we could change the model like so: object Franchise { case class Character(name: String, franchise: Franchise) } class Franchise(name: String) { import Franchise.Character def createFanFiction( lovestruck: Character, objectOfDesire: Character): (Character, Character) = { require(lovestruck.franchise == objectOfDesire.franchise) (lovestruck, objectOfDesire) } } </code></pre> Now, the Character</code> instances have a reference to their Franchise</code>, and trying to create a fan fiction with characters from different franchises will lead to an IllegalArgumentException</code> (feel free to try this out in a REPL). Safer fiction with path-dependent types</h2> This is pretty good, isn't it? It's the kind of fail-fast behaviour we have been indoctrinated with for years. However, with Scala, we can do better. There is a way to fail even faster – not at runtime, but at compile time. To achieve that, we need to encode the connection between a Character</code> and its Franchise</code> at the type level. Luckily, the way Scala's nested types work allow us to do that. In Scala, a nested type is bound to a specific instance of the outer type, not to the outer type itself. This means that if you try to use an instance of the inner type outside of the instance of the enclosing type, you will face a compile error: class A { class B var b: Option[B] = None } val a1 = new A val a2 = new A val b1 = new a1.B val b2 = new a2.B a1.b = Some(b1) a2.b = Some(b1) // does not compile </code></pre> You cannot simply assign an instance of the B</code> that is bound to a2</code> to the field on a1</code> – the one is an a2.B</code>, the other expects an a1.B</code>. The dot syntax represents the path to the type, going along concrete instances of other types. Hence the name, path-dependent types. We can put these to use in order to prevent characters from different franchises making out with each other: class Franchise(name: String) { case class Character(name: String) def createFanFictionWith( lovestruck: Character, objectOfDesire: Character): (Character, Character) = (lovestruck, objectOfDesire) } </code></pre> Now, the type Character</code> is nested in the type Franchise</code>, which means that it is dependent on a specific enclosing instance of the Franchise</code> type. Let's create our example franchises and characters again: val starTrek = new Franchise("Star Trek") val starWars = new Franchise("Star Wars") val quark = starTrek.Character("Quark") val jadzia = starTrek.Character("Jadzia Dax") val luke = starWars.Character("Luke Skywalker") val yoda = starWars.Character("Yoda") </code></pre> You can already see in how our Character</code> instances are created that their types are bound to a specific franchise. Let's see what happens if we try to put some of these characters together: starTrek.createFanFictionWith(lovestruck = quark, objectOfDesire = jadzia) starWars.createFanFictionWith(lovestruck = luke, objectOfDesire = yoda) </code></pre> These two compile, as expected. They are tasteless, but what can we do? Now, let's see what happens if we try to create some fan fiction about Jadzia Dax and Luke Skywalker: starTrek.createFanFictionWith(lovestruck = jadzia, objectOfDesire = luke) </code></pre> Et voilà: The thing that should not be does not even compile! The compiler complains about a type mismatch: found : starWars.Character required: starTrek.Character starTrek.createFanFictionWith(lovestruck = jadzia, objectOfDesire = luke) ^ </code></pre> This technique also works if our method is not defined on the Franchise</code> class, but in some other module. In this case, we can make use of dependent method types, where the type of one parameter depends on a previous parameter: def createFanFiction(f: Franchise)(lovestruck: f.Character, objectOfDesire: f.Character) = (lovestruck, objectOfDesire) </code></pre> As you can see, the type of the lovestruck</code> and objectOfDesire</code> parameters depends on the Franchise</code> instance passed to the method. Note that this only works if the instance on which other types depend is in its own parameter list. Abstract type members</h2> Often, dependent method types are used in conjunction with abstract type members. Suppose we want to develop a hipsterrific key-value store. It will only support setting and getting the value for a key, but in a typesafe manner. Here is our oversimplified implementation: object AwesomeDB { abstract class Key(name: String) { type Value } } import AwesomeDB.Key class AwesomeDB { import collection.mutable.Map val data = Map.empty[Key, Any] def get(key: Key): Option[key.Value] = data.get(key).asInstanceOf[Option[key.Value]] def set(key: Key)(value: key.Value): Unit = data.update(key, value) } </code></pre> We have defined a class Key</code> with an abstract type member Value</code>. The methods on AwesomeDB</code> refer to that type without ever knowing or caring about the specific manifestation of this abstract type. We can now define some concrete keys that we want to use: trait IntValued extends Key { type Value = Int } trait StringValued extends Key { type Value = String } object Keys { val foo = new Key("foo") with IntValued val bar = new Key("bar") with StringValued } </code></pre> Now we can set and get key/value pairs in a typesafe manner: val dataStore = new AwesomeDB dataStore.set(Keys.foo)(23) val i: Option[Int] = dataStore.get(Keys.foo) dataStore.set(Keys.foo)("23") // does not compile </code></pre> Path-dependent types in practice</h2> While path-dependent types are not necessarily omnipresent in your typical Scala code, they do have a lot of practical value beyond modelling the domain of fan fiction. One of the most widespread uses is probably seen in combination with the cake pattern, which is a technique for composing your components and managing their dependencies, relying solely on features of the language. See the excellent articles by Debasish Ghosh</a> and Precog's Daniel Spiewak</a> to learn more about both the cake pattern and how it can be improved by incorporating path-dependent types. In general, whenever you want to make sure that objects created or managed by a specific instance of another type cannot accidentally or purposely be interchanged or mixed, path-dependent types are the way to go. Path-dependent types and dependent method types play a crucial role for attempts to encode information into types that is typically only known at runtime, for instance heterogenous lists, type-level representations of natural numbers and collections that carry their size in their type. Miles Sabin is exploring the limits of Scala's type system in this respect in his excellent library Shapeless</a>. The Neophyte's Guide to Scala Part 12: Type classes Unknown — Wed, 06 Feb 2013 00:00:00 +0000 After having discussed several functional programming techniques for keeping things DRY and flexible in the last two weeks, in particular function composition, partial function application, and currying, we are going to stick with the general notion of making your code as flexible as possible. However, this time, we are not looking so much at how you can leverage functions as first-class objects to achieve this goal. Instead, this article is all about using the type system in such a manner that it's not in the way, but rather supports you in keeping your code extensible: You're going to learn about type classes. You might think that this is some exotic idea without practical relevance, brought into the Scala community by some vocal Haskell fanatics. This is clearly not the case. Type classes have become an important part of the Scala standard library and even more so of many popular and commonly used third-party open-source libraries, so it's generally a good idea to make yourself familiar with them. I will discuss the idea of type classes, why they are useful, how to benefit from them as a client, and how to implement your own type classes and put them to use for great good. The problem</h2> Instead of starting off by giving an abstract explanation of what type classes are, let's tackle this subject by means of an – admittedly simplified, but nevertheless resonably practical – example. Imagine that we want to write a fancy statistics library. This means we want to provide a bunch of functions that operate on collections of numbers, mostly to compute some aggregate values for them. Imagine further that we are restricted to accessing an element from such a collection by index and to using the reduce</code> method defined on Scala collections. We impose this restriction on ourselves because we are going to re-implement a little bit of what the Scala standard library already provides – simply because it's a nice example without many distractions, and it's small enough for a blog post. Finally, our implementation assumes that the values we get are already sorted. We will start with a very crude implementation of median</code>, quartiles</code>, and iqr</code> numbers of type Double</code>: object Statistics { def median(xs: Vector[Double]): Double = xs(xs.size / 2) def quartiles(xs: Vector[Double]): (Double, Double, Double) = (xs(xs.size / 4), median(xs), xs(xs.size / 4 * 3)) def iqr(xs: Vector[Double]): Double = quartiles(xs) match { case (lowerQuartile, _, upperQuartile) => upperQuartile - lowerQuartile } def mean(xs: Vector[Double]): Double = { xs.reduce(_ + _) / xs.size } } </code></pre> The median cuts a data set in half, whereas the lower and upper quartile (first and third element of the tuple returned by our quartile</code> method), split the lowest and highest 25 percent of the data, respectively. Our iqr</code> method returns the interquartile range, which is the difference between the upper and lower quartile. Now, of course, we want to support more than just double numbers. So let's implement all these methods again for Int</code> numbers, right? Well, no! First of all, that would be a tiny little bit repetitious, wouldn't it? Also, in situations such as these, we quickly run into situations where we cannot overload a method without some dirty tricks, because the type parameter suffers from type erasure. If only Int</code> and Double</code> would extend from a common base class or implement a common trait like Number</code>! We could be tempted to change the type required and returned by our methods to that more general type. Our method signatures would look like this: object Statistics { def median(xs: Vector[Number]): Number = ??? def quartiles(xs: Vector[Number]): (Number, Number, Number) = ??? def iqr(xs: Vector[Number]): Number = ??? def mean(xs: Vector[Number]): Number = ??? } </code></pre> Thankfully, in this case there is no such common trait, so we aren't tempted to walk this road at all. However, in other cases, that might very well be the case – and still be a bad idea. Not only do we drop previously available type information, we also close our API against future extensions to types whose sources we don't control: We cannot make some new number type coming from a third party extend the Number</code> trait. Ruby's answer to that problem is monkey patching, polluting the global namespace with an extension to that new type, making it act like a Number</code> after all. Java developers who have got beaten up by the Gang of Four in their youth, on the other hand, will think that an Adapter may solve all of their problems: object Statistics { trait NumberLike[A] { def get: A def plus(y: NumberLike[A]): NumberLike[A] def minus(y: NumberLike[A]): NumberLike[A] def divide(y: Int): NumberLike[A] } case class NumberLikeDouble(x: Double) extends NumberLike[Double] { def get: Double = x def minus(y: NumberLike[Double]) = NumberLikeDouble(x - y.get) def plus(y: NumberLike[Double]) = NumberLikeDouble(x + y.get) def divide(y: Int) = NumberLikeDouble(x / y) } type Quartile[A] = (NumberLike[A], NumberLike[A], NumberLike[A]) def median[A](xs: Vector[NumberLike[A]]): NumberLike[A] = xs(xs.size / 2) def quartiles[A](xs: Vector[NumberLike[A]]): Quartile[A] = (xs(xs.size / 4), median(xs), xs(xs.size / 4 * 3)) def iqr[A](xs: Vector[NumberLike[A]]): NumberLike[A] = quartiles(xs) match { case (lowerQuartile, _, upperQuartile) => upperQuartile.minus(lowerQuartile) } def mean[A](xs: Vector[NumberLike[A]]): NumberLike[A] = xs.reduce(_.plus(_)).divide(xs.size) } </code></pre> Now we have solved the problem of extensibility: Users of our library can pass in a NumberLike</code> adapter for Int</code> (which we would likely provide ourselves) or for any possible type that might behave like a number, without having to recompile the module in which our statistics methods are implemented. However, always wrapping your numbers in an adapter is not only tiresome to write and read, it also means that you have to create a lot of instances of your adapter classes when interacting with our library. Type classes to the rescue!</h2> A powerful alternative to the approaches outlined so far is, of course, to define and use a type class. Type classes, one of the prominent features of the Haskell language, despite their name, haven't got anything to do with classes in object-oriented programming. A type class C</code> defines some behaviour in the form of operations that must be supported by a type T</code> for it to be a member of type class C</code>. Whether the type T</code> is a member of the type class C</code> is not inherent in the type. Rather, any developer can declare that a type is a member of a type class simply by providing implementations of the operations the type must support. Now, once T</code> is made a member of the type class C</code>, functions that have constrained one or more of their parameters to be members of C</code> can be called with arguments of type T</code>. As such, type classes allow ad-hoc and retroactive polymorphism. Code that relies on type classes is open to extension without the need to create adapter objects. Creating a type class</h3> In Scala, type classes can be implemented and used by a combination of techniques. It's a little more involved than in Haskell, but also gives developers more control. Creating a type class in Scala involves several steps. First, let's define a trait. This is the actual type class: object Math { trait NumberLike[T] { def plus(x: T, y: T): T def divide(x: T, y: Int): T def minus(x: T, y: T): T } } </code></pre> We have created a type class called NumberLike</code>. Type classes always take one or more type parameters, and they are usually designed to be stateless, i.e. the methods defined on our NumberLike</code> trait operate only on the passed in arguments. In particular, where our adapter above operated on its member of type T</code> and one argument, the methods defined for our NumberLike</code> type class take two parameters of type T</code> each – the member has become the first parameter of the operations supported by NumberLike</code>. Providing default members</h3> The second step in implementing a type class is usually to provide some default implementations of your type class trait in its companion object. We will see in a moment why this is generally a good strategy. First, however, let's do this, too, by making Double</code> and Int</code> members of our NumberLike</code> type class: object Math { trait NumberLike[T] { def plus(x: T, y: T): T def divide(x: T, y: Int): T def minus(x: T, y: T): T } object NumberLike { implicit object NumberLikeDouble extends NumberLike[Double] { def plus(x: Double, y: Double): Double = x + y def divide(x: Double, y: Int): Double = x / y def minus(x: Double, y: Double): Double = x - y } implicit object NumberLikeInt extends NumberLike[Int] { def plus(x: Int, y: Int): Int = x + y def divide(x: Int, y: Int): Int = x / y def minus(x: Int, y: Int): Int = x - y } } } </code></pre> Two things: First, you see that the two implementations are basically identical. That is not always the case when creating members of a type classes. Our NumberLike</code> type class is just a rather narrow domain. Later in the article, I will give examples of type classes where there is a lot less room for duplication when implementing them for multiple types. Second, please ignore the fact that we are losing precision in NumberLikeInt</code> by doing integer division. It's all to keep things simple for this example. As you can see, members of type classes are usually singleton objects. Also, please note the implicit</code> keyword before each of the type class implementations. This is one of the crucial elements for making type classes possible in Scala, making type class members implicitly available under certain conditions. More about that in the next section. Coding against type classes</h3> Now that we have our type class and two default implementations for common types, we want to code against this type class in our statistics module. Let's focus on the mean</code> method for now: object Statistics { import Math.NumberLike def mean[T](xs: Vector[T])(implicit ev: NumberLike[T]): T = ev.divide(xs.reduce(ev.plus(_, _)), xs.size) } </code></pre> This may look a little intimidating at first, but it's actually quite simple. Our method takes a type parameter T</code> and a single parameter of type Vector[T]</code>. The idea to constrain a parameter to types that are members of a specific type class is realized by means of the implicit</code> second parameter list. What does this mean? Basically, that a value of type NumberLike[T]</code> must be implicitly available in the current scope. This is the case if an implicit value has been declared and made available in the current scope, very often by importing the package or object in which that implicit value is defined. If and only if no other implicit value can be found, the compiler will look in the companion object of the type of the implicit parameter. Hence, as a library designer, putting your default type class implementations in the companion object of your type class trait means that users of your library can easily override these implementations with their own ones, which is exactly what you want. Users can also pass in an explicit value for an implicit parameter to override the implicit values that are in scope. Let's see if the default type class implementations can be resolved: val numbers = Vector[Double](13, 23.0, 42, 45, 61, 73, 96, 100, 199, 420, 900, 3839) println(Statistics.mean(numbers)) </code></pre> Wonderful! If we try this with a Vector[String]</code>, we get an error at compile time, stating that no implicit value could be found for parameter ev: NumberLike[String]</code>. If you don't like this error message, you can customize it by annotating your type class trait with the @implicitNotFound</code> annotation: object Math { import annotation.implicitNotFound @implicitNotFound("No member of type class NumberLike in scope for ${T}") trait NumberLike[T] { def plus(x: T, y: T): T def divide(x: T, y: Int): T def minus(x: T, y: T): T } } </code></pre> Context bounds</h4> A second, implicit parameter list on all methods that expect a member of a type class can be a little verbose. As a shortcut for implicit parameters with only one type parameter, Scala provides so-called context bounds. To show how those are used, we are going to implement our other statistics methods using those instead: object Statistics { import Math.NumberLike def mean[T](xs: Vector[T])(implicit ev: NumberLike[T]): T = ev.divide(xs.reduce(ev.plus(_, _)), xs.size) def median[T : NumberLike](xs: Vector[T]): T = xs(xs.size / 2) def quartiles[T: NumberLike](xs: Vector[T]): (T, T, T) = (xs(xs.size / 4), median(xs), xs(xs.size / 4 * 3)) def iqr[T: NumberLike](xs: Vector[T]): T = quartiles(xs) match { case (lowerQuartile, _, upperQuartile) => implicitly[NumberLike[T]].minus(upperQuartile, lowerQuartile) } } </code></pre> A context bound T : NumberLike</code> means that an implicit value of type NumberLike[T]</code> must be available, and so is really equivalent to having a second implicit parameter list with a NumberLike[T]</code> in it. If you want to access that implicitly available value, however, you need to call the implicitly</code> method, as we do in the iqr</code> method. If your type class requires more than one type parameter, you cannot use the context bound syntax. Custom type class members</h3> As a user of a library that makes use of type classes, you will sooner or later have types that you want to make members of those type classes. For instance, we might want to use the statistics library for instances of the Joda Time Duration</code> type. To do that, we need Joda Time on our classpath, of course: libraryDependencies += "joda-time" % "joda-time" % "2.1" libraryDependencies += "org.joda" % "joda-convert" % "1.3" </code></pre> Now we just have to create an implicitly available implementation of NumberLike</code> (please make sure you have Joda Time on your classpath when trying this out): object JodaImplicits { import Math.NumberLike import org.joda.time.Duration implicit object NumberLikeDuration extends NumberLike[Duration] { def plus(x: Duration, y: Duration): Duration = x.plus(y) def divide(x: Duration, y: Int): Duration = Duration.millis(x.getMillis / y) def minus(x: Duration, y: Duration): Duration = x.minus(y) } } </code></pre> If we import the package or object containing this NumberLike</code> implementation, we can now compute the mean value for a bunch of durations: import Statistics._ import JodaImplicits._ import org.joda.time.Duration._ val durations = Vector(standardSeconds(20), standardSeconds(57), standardMinutes(2), standardMinutes(17), standardMinutes(30), standardMinutes(58), standardHours(2), standardHours(5), standardHours(8), standardHours(17), standardDays(1), standardDays(4)) println(mean(durations).getStandardHours) </code></pre> Use cases</h2> Our NumberLike</code> type class was a nice exercise, but Scala already ships with the Numeric</code> type class, which allows you to call methods like sum</code> or product</code> on collections for whose type T</code> a Numeric[T]</code> is available. Another type class in the standard library that you will use a lot is Ordering</code>, which allows you to provide an implicit ordering for your own types, available to the sort</code> method on Scala's collections. There are more type classes in the standard library, but not all of them are ones you have to deal with on a regular basis as a Scala developer. A very common use case in third-party libraries is that of object serialization and deserialization, most notably to and from JSON. By making your classes members of an appropriate formatter type class, you can customize the way your classes are serialized to JSON, XML or whatever format is currently the new black. Mapping between Scala types and ones supported by your database driver is also commonly made customizable and extensible via type classes. Summary</h2> Once you start to do some serious work with Scala, you will inevitably stumble upon type classes. I hope that after reading this article, you are prepared to take advantage of this powerful technique. Scala type classes allow you to develop your Scala code in such a way that it's open for retroactive extension while retaining as much concrete type information as possible. In contrast to approaches from other languages, they give developers full control, as default type class implementations can be overridden without much hassle, and type classes implementations are not made available in the global namespace. You will see that this technique is especially useful when writing libraries intended to be used by others, but type classes also have their use in application code to decrease coupling between modules. The Neophyte's Guide to Scala Part 11: Currying and partially applied functions Unknown — Wed, 30 Jan 2013 00:00:00 +0000 Last week's article</a> was all about avoiding code duplication, either by lifting existing functions to match your new requirements or by composing them. In this article, we are going to have a look at two other mechanisms the Scala language provides in order to enable you to reuse your functions: Partial application of functions and currying. Partially applied functions</h2> Scala, like many other languages following the functional programming paradigm, allows you to apply a function partially. What this means is that, when applying the function, you do not pass in arguments for all of the parameters defined by the function, but only for some of them, leaving the remaining ones blank. What you get back is a new function whose parameter list only contains those parameters from the original function that were left blank. Do not confuse partially applied functions with partially defined functions</a>, which are represented by the PartialFunction</code> type in Scala. To illustrate how partial function application works, let's revisit our example from last week: For our imaginary free mail service, we wanted to allow the user to configure a filter so that only emails meeting certain criteria would show up in their inbox, with all others being blocked. Our Email</code> case class still looks like this: case class Email( subject: String, text: String, sender: String, recipient: String) type EmailFilter = Email => Boolean </code></pre> The criteria for filtering emails were represented by a predicate Email => Boolean</code>, which we aliased to the type EmailFilter</code>, and we were able to generate new predicates by calling appropriate factory functions. Two of the factory functions from last week's article created EmailFiter</code> instances that checked if the email text satisfied a given minimum or maximum length. This time we want to make use of partial function application to implement these factory functions. We want to have a general method sizeConstraint</code> and be able to create more specific size constraints by fixing some of its parameters. Here is our revised sizeConstraint</code> method: type IntPairPred = (Int, Int) => Boolean def sizeConstraint(pred: IntPairPred, n: Int, email: Email) = pred(email.text.size, n) </code></pre> We also define an alias IntPairPred</code> for the type of predicate that checks a pair of integers (the value n</code> and the text size of the email) and returns whether the email text size is okay, given n</code>. Note that unlike our sizeConstraint</code> function from last week, this one does not return a new EmailFilter</code> predicate, but simply evaluates all the arguments passed to it, returning a Boolean</code>. The trick is to get such a predicate of type EmailFilter</code> by partially applying sizeConstraint</code>. First, however, because we take the DRY principle very seriously, let's define all the commonly used instances of IntPairPred</code>. Now, when we call sizeConstraint</code>, we don't have to repeatedly write the same anonymous functions, but can simply pass in one of those: val gt: IntPairPred = _ > _ val ge: IntPairPred = _ >= _ val lt: IntPairPred = _ < _ val le: IntPairPred = _ <= _ val eq: IntPairPred = _ == _ </code></pre> Finally, we are ready to do some partial application of the sizeConstraint</code> function, fixing its first parameter with one of our IntPairPred</code> instances: val minimumSize: (Int, Email) => Boolean = sizeConstraint(ge, _: Int, _: Email) val maximumSize: (Int, Email) => Boolean = sizeConstraint(le, _: Int, _: Email) </code></pre> As you can see, you have to use the placeholder _</code> for all parameters not bound to an argument value. Unfortunately, you also have to specify the type of those arguments, which makes partial function application in Scala a bit tedious. The reason is that the Scala compiler cannot infer these types, at least not in all cases – think of overloaded methods where it's impossible for the compiler to know which of them you are referring to. On the other hand, this makes it possible to bind or leave out arbitrary parameters. For example, we can leave out the first one and pass in the size to be used as a constraint: val constr20: (IntPairPred, Email) => Boolean = sizeConstraint(_: IntPairPred, 20, _: Email) val constr30: (IntPairPred, Email) => Boolean = sizeConstraint(_: IntPairPred, 30, _: Email) </code></pre> Now we have two functions that take an IntPairPred</code> and an Email</code> and compare the email text size to 20</code> and 30</code>, respectively, but the comparison logic has not been specified yet, as that's exactly what the IntPairPred</code> is good for. This shows that, while quite verbose, partial function application in Scala is a little more flexible than in Clojure, for example, where you have to pass in arguments from left to right, but can't leave out any in the middle. From methods to function objects</h3> When doing partial application on a method, you can also decide to not bind any parameters whatsoever. The parameter list of the returned function object will be the same as for the method. You have effectively turned a method into a function that can be assigned to a val</code> or passed around: val sizeConstraintFn: (IntPairPred, Int, Email) => Boolean = sizeConstraint _ </code></pre> Producing those EmailFilters</h3> We still haven't got any functions that adhere to the EmailFilter</code> type or that return new predicates of that type – like sizeConstraint</code>, minimumSize</code> and maximumSize</code> don't return a new predicate, but a Boolean</code>, as their signature clearly shows. However, our email filters are just another partial function application away. By fixing the integer parameter of minimumSize</code> and maximumSize</code>, we can create new functions of type EmailFilter</code>: val min20: EmailFilter = minimumSize(20, _: Email) val max20: EmailFilter = maximumSize(20, _: Email) </code></pre> Of course, we could achieve the same by partially applying our constr20</code> we created above: val min20: EmailFilter = constr20(ge, _: Email) val max20: EmailFilter = constr20(le, _: Email) </code></pre> Spicing up your functions</h2> Maybe you find partial function application in Scala a little too verbose, or simply not very elegant to write and look at. Lucky you, because there is an alternative. As you should know by now, methods in Scala can have more than one parameter list. Let's define our sizeConstraint</code> method such that each parameter is in its own parameter list: def sizeConstraint(pred: IntPairPred)(n: Int)(email: Email): Boolean = pred(email.text.size, n) </code></pre> If we turn this into a function object that we can assign or pass around, the signature of that function looks like this: val sizeConstraintFn: IntPairPred => Int => Email => Boolean = sizeConstraint _ </code></pre> Such a chain of one-parameter functions is called a curried function, so named after Haskell Curry, who re-discovered this technique and got all the fame for it. In fact, in the Haskell programming language, all functions are in curried form by default. In our example, it takes an IntPairPred</code> and returns a function that takes an Int</code> and returns a new function. That last function, finally, takes an Email</code> and returns a Boolean</code>. Now, if we we want to bind the IntPairPred</code>, we simply apply sizeConstraintFn</code>, which takes exactly this one parameter and returns a new one-parameter function: val minSize: Int => Email => Boolean = sizeConstraint(ge) val maxSize: Int => Email => Boolean = sizeConstraint(le) </code></pre> There is no need to use any placeholders for parameters left blank, because we are in fact not doing any partial function application. We can now create the same EmailFilter</code> predicates as we did using partial function application before, by applying these curried functions: val min20: Email => Boolean = minSize(20) val max20: Email => Boolean = maxSize(20) </code></pre> Of course, it's possible to do all this in one step if you want to bind several parameters at once. It just means that you immediately apply the function that was returned from the first function application, without assigning it to a val</code> first: val min20: Email => Boolean = sizeConstraintFn(ge)(20) val max20: Email => Boolean = sizeConstraintFn(le)(20) </code></pre> Currying existing functions</h3> It's not always the case the you know beforehand whether writing your functions in curried form makes sense or not – after all, the usual function application looks a little more verbose than for functions that have only declared a single parameter list for all their parameters. Also, you'll sometimes want to work with third-party functions in curried form, when they are written with a single parameter list. Transforming a function with multiple parameters in one list to curried form is, of course, just another higher-order function, generating a new function from an existing one. This transformation logic is available in form of the curried</code> method on functions with more than one parameter. Hence, if we have a function sum</code>, taking two parameters, we can get the curried version simply calling calling its curried</code> method: val sum: (Int, Int) => Int = _ + _ val sumCurried: Int => Int => Int = sum.curried </code></pre> If you need to do the reverse, you have Function.uncurried</code> at your fingertips, which you need to pass the curried function to get it back in uncurried form. Injecting your dependencies the functional way</h3> To close this article, let's have a look at what role curried functions can play in the large. If you come from the enterprisy Java or .NET world, you will be very familiar with the necessity to use more or less fancy dependency injection containers that take the heavy burden of providing all your objects with their respective dependencies off you. In Scala, you don't really need any external tool for that, as the language already provides numerous features that make it much less of a pain to this all on your own. When programming in a very functional way, you will notice that there is still a need to inject dependencies: Your functions residing in the higher layers of your application will have to make calls to other functions. Simply hard-coding the functions to call in the body of your functions makes it difficult to test them in isolation. Hence, you will need to pass the functions your higher-layer function depends on as arguments to that function. It would not be DRY at all to always pass the same dependencies to your function when calling it, would it? Curried functions to the rescue! Currying and partial function application are one of several ways of injecting dependencies in functional programming. The following, very simplified example illustrates this technique: case class User(name: String) trait EmailRepository { def getMails(user: User, unread: Boolean): Seq[Email] } trait FilterRepository { def getEmailFilter(user: User): EmailFilter } trait MailboxService { def getNewMails(emailRepo: EmailRepository)(filterRepo: FilterRepository)(user: User) = emailRepo.getMails(user, true).filter(filterRepo.getEmailFilter(user)) val newMails: User => Seq[Email] } </code></pre> We have a service that depends on two different repositories, These dependencies are declared as parameters to the getNewMails</code> method, each in their own parameter list. The MailboxService</code> already has a concrete implementation of that method, but is lacking one for the newMails</code> field. The type of that field is User => Seq[Email]</code> – that's the function that components depending on the MailboxService</code> will call. We need an object that extends MailboxService</code>. The idea is to implement newMails</code> by currying the getNewMails</code> method and fixing it with concrete implementations of the dependencies, EmailRepository</code> and FilterRepository</code>: object MockEmailRepository extends EmailRepository { def getMails(user: User, unread: Boolean): Seq[Email] = Nil } object MockFilterRepository extends FilterRepository { def getEmailFilter(user: User): EmailFilter = _ => true } object MailboxServiceWithMockDeps extends MailboxService { val newMails: (User) => Seq[Email] = getNewMails(MockEmailRepository)(MockFilterRepository) _ } </code></pre> We can now call MailboxServiceWithMoxDeps.newMails(User("daniel"))</code> without having to specify the two repositories to be used. In a real application, of course, we would very likely not use a direct reference to a concrete implementation of the service, but have this injected, too. This is probably not the most powerful and scaleable way of injecting your dependencies in Scala, but it's good to have this in your tool belt, and it's a very good example of the benefits that partial function application and currying can provide in the wild. If you want know more about this, I recommend to have a look at the excellent slides for the presentation "Dependency Injection in Scala</a>" by Debasish Ghosh, which is also where I first came across this technique. Summary</h2> In this article, we discussed two additional functional programming techniques that help you keep your code free of duplication and, on top of that, give you a lot of flexibility, allowing you to reuse your functions in manifold ways. Partial function application and currying have more or less the same effect, but sometimes one of them is the more elegant solution. In the next part of this series, we will continue to look at ways to keep things flexible, discussing the what and how of type classes in Scala. The Neophyte's Guide to Scala Part 10: Staying DRY with higher-order functions Unknown — Wed, 23 Jan 2013 00:00:00 +0000 In the previous articles, I discussed the composable nature of Scala's container types. As it turns out, being composable is a quality that you will not only find in Future</code>, Try</code>, and other container types, but also in functions, which are first class citizens in the Scala language. Composability naturally entails reusability. While the latter has often been claimed to be one of the big advantages of object-oriented programming, it's a trait that is definitely true for pure functions, i.e. functions that do not have any side-effects and are referentially transparent. One obvious way is to implement a new function by calling already existing functions in its body. However, there are other ways to reuse existing functions: In this blog post, I will discuss some fundementals of functional programming that we have been missing out on so far. You will learn how to follow the DRY</a> principle by leveraging higher-order functions in order to reuse existing code in new contexts. On higher-order functions</h2> A higher-order function, as opposed to a first-order function, can have one of three forms: One or more of its parameters is a function, and it returns some value.</li> It returns a function, but none of its parameters is a function.</li> Both of the above: One or more of its parameters is a function, and it returns a function.</li> </ol> If you have followed this series, you have seen a lot of usages of higher-order functions of the first type: We called methods like map</code>, filter</code>, or flatMap</code> and passed a function to it that was used to transform or filter a collection in some way. Very often, the functions we passed to these methods were anonymous functions, sometimes involving a little bit of duplication. In this article, we are only concerned with what the other two types of higher-order functions can do for us: The first of them allows us to produce new functions based on some input data, whereas the other gives us the power and flexibility to compose new functions that are somehow based on existing functions. In both cases, we can eliminate code duplication. And out of nowhere, a function was born</h2> You might think that the ability to create new functions based on some input data is not terribly useful. While we mainly want to deal with how to compose new functions based on existing ones, let's first have a look at how a function that produces new functions may be used. Let's assume we are implementing a freemail service where users should be able to configure when an email is supposed to be blocked. We are representing emails as instances of a simple case class: case class Email( subject: String, text: String, sender: String, recipient: String) </code></pre> We want to be able to filter new emails by the criteria specified by the user, so we have a filtering function that makes use of a predicate, a function of type Email => Boolean</code> to determine whether the email is to be blocked. If the predicate is true</code>, the email is accepted, otherwise it will be blocked: type EmailFilter = Email => Boolean def newMailsForUser(mails: Seq[Email], f: EmailFilter) = mails.filter(f) </code></pre> Note that we are using a type alias for our function, so that we can work with more meaningful names in our code. Now, in order to allow the user to configure their email filter, we can implement some factory functions that produce EmailFilter</code> functions configured to the user's liking: val sentByOneOf: Set[String] => EmailFilter = senders => email => senders.contains(email.sender) val notSentByAnyOf: Set[String] => EmailFilter = senders => email => !senders.contains(email.sender) val minimumSize: Int => EmailFilter = n => email => email.text.size >= n val maximumSize: Int => EmailFilter = n => email => email.text.size <= n </code></pre> Each of these four vals</code> is a function that returns an EmailFilter</code>, the first two taking as input a Set[String]</code> representing senders, the other two an Int</code> representing the length of the email body. We can use any of these functions to create a new EmailFilter</code> that we can pass to the newMailsForUser</code> function: val emailFilter: EmailFilter = notSentByAnyOf(Set("johndoe@example.com")) val mails = Email( subject = "It's me again, your stalker friend!", text = "Hello my friend! How are you?", sender = "johndoe@example.com", recipient = "me@example.com") :: Nil newMailsForUser(mails, emailFilter) // returns an empty list </code></pre> This filter removes the one mail in the list because our user decided to put the sender on their black list. We can use our factory functions to create arbitrary EmailFilter</code> functions, depending on the user's requirements. Reusing existing functions</h2> There are two problems with the current solution. First of all, there is quite a bit of duplication in the predicate factory functions above, when initially I told you that the composable nature of functions made it easy to stick to the DRY principle. So let's get rid of the duplication. To do that for the minimumSize</code> and maximumSize</code>, we introduce a function sizeConstraint</code> that takes a predicate that checks if the size of the email body is okay. That size will be passed to the predicate by the sizeConstraint</code> function: type SizeChecker = Int => Boolean val sizeConstraint: SizeChecker => EmailFilter = f => email => f(email.text.size) </code></pre> Now we can express minimumSize</code> and maximumSize</code> in terms of sizeConstraint</code>: val minimumSize: Int => EmailFilter = n => sizeConstraint(_ >= n) val maximumSize: Int => EmailFilter = n => sizeConstraint(_ <= n) </code></pre> Function composition</h2> For the other two predicates, sentByOneOf</code> and notSentByAnyOf</code>, we are going to introduce a very generic higher-order function that allows us to express one of the two functions in terms of the other. Let's implement a function complement</code> that takes a predicate A => Boolean</code> and returns a new function that always returns the opposite of the given predicate: def complement[A](predicate: A => Boolean) = (a: A) => !predicate(a) </code></pre> Now, for an existing predicate p</code> we could get the complement by calling complement(p)</code>. However, sentByAnyOf</code> is not a predicate, but it returns one, namely an EmailFilter</code>. Scala functions provide two composing functions that will help us here: Given two functions f</code> and g</code>, f.compose(g)</code> returns a new function that, when called, will first call g</code> and then apply f</code> on the result of it. Similarly, f.andThen(g)</code> returns a new function that, when called, will apply g</code> to the result of f</code>. We can put this to use to create our notSentByAnyOf</code> predicate without code duplication: val notSentByAnyOf = sentByOneOf andThen(g => complement(g)) </code></pre> What this means is that we ask for a new function that first applies the sentByOneOf</code> function to its arguments (a Set[String]</code>) and then applies the complement</code> function to the EmailFilter</code> predicate returned by the former function. Using Scala's placeholder syntax for anonymous functions, we could write this more concisely as: val notSentByAnyOf = sentByOneOf andThen(complement(_)) </code></pre> Of course, you will now have noticed that, given a complement</code> function, you could also implement the maximumSize</code> predicate in terms of minimumSize</code> instead of extracting a sizeConstraint</code> function. However, the latter is more flexible, allowing you to specify arbitrary checks on the size of the mail body. Composing predicates</h3> Another problem with our email filters is that we can currently only pass a single EmailFilter</code> to our newMailsForUser</code> function. Certainly, our users want to configure multiple criteria. We need a way to create a composite predicate that returns true</code> if either any, none or all of the predicates it consists of return true</code>. Here is one way to implement these functions: def any[A](predicates: (A => Boolean)*): A => Boolean = a => predicates.exists(pred => pred(a)) def none[A](predicates: (A => Boolean)*) = complement(any(predicates: _*)) def every[A](predicates: (A => Boolean)*) = none(predicates.view.map(complement(_)): _*) </code></pre> The any</code> function returns a new predicate that, when called with an input a</code>, checks if at least one of its predicates holds true for the value a</code>. Our none</code> function simply returns the complement of the predicate returned by any</code> – if at least one predicate holds true, the condition for none</code> is not satisfied. Finally, our every</code> function works by checking that none of the complements to the predicates passed to it holds true. We can now use this to create a composite EmailFilter</code> that represents the user's configuration: val filter: EmailFilter = every( notSentByAnyOf(Set("johndoe@example.com")), minimumSize(100), maximumSize(10000) ) </code></pre> Composing a transformation pipeline</h3> As another example of function composition, consider our example scenario again. As a freemail provider, we want not only to allow user's to configure their email filter, but also do some processing on emails sent by our users. These are simple functions Email => Email</code>. Some possible transformations are the following: val addMissingSubject = (email: Email) => if (email.subject.isEmpty) email.copy(subject = "No subject") else email val checkSpelling = (email: Email) => email.copy(text = email.text.replaceAll("your", "you're")) val removeInappropriateLanguage = (email: Email) => email.copy(text = email.text.replaceAll("dynamic typing", "**CENSORED**")) val addAdvertismentToFooter = (email: Email) => email.copy(text = email.text + "\nThis mail sent via Super Awesome Free Mail") </code></pre> Now, depending on the weather and the mood of our boss, we can configure our pipeline as required, either by multiple andThen</code> calls, or, having the same effect, by using the chain</code> method defined on the Function</code> companion object: val pipeline = Function.chain(Seq( addMissingSubject, checkSpelling, removeInappropriateLanguage, addAdvertismentToFooter)) </code></pre> Higher-order functions and partial functions</h2> I won't go into detail here, but now that you know more about how you can compose or reuse functions by means of higher-order functions, you might want to have a look at partial functions again. Chaining partial functions</h3> In the article on pattern matching anonymous functions</a>, I mentioned that partial functions can be used to create a nice alternative to the chain of responsibility pattern: The orElse</code> method defined on the PartialFunction</code> trait allows you to chain an arbitrary number of partial functions, creating a composite partial function. The first one, however, will only pass on to the next one if it isn't defined for the given input. Hence, you can do something like this: val handler = fooHandler orElse barHandler orElse bazHandler </code></pre> Lifting partial functions</h3> Also, sometimes a PartialFunction</code> is not what you need. If you think about it, another way to represent the fact that a function is not defined for all input values is to have a standard function whose return type is an Option[A]</code> – if the function is not defined for an input value, it will return None</code>, otherwise a Some[A]</code>. If that's what you need in a certain context, given a PartialFunction</code> named pf</code>, you can call pf.lift</code> to get the normal function returning an Option</code>. If you have one of the latter and require a partial function, call Function.unlift(f)</code>. Summary</h2> In this article, we have seen the value of higher-order functions, which allow you to reuse existing functions in new, unforeseen contexts and compose them in a very flexible way. While in the examples, you didn't save much in terms of lines of code, because the functions I showed were rather tiny, the real point is to illustrate the increase in flexibility. Also, composing and reusing functions is something that has benefits not only for small functions, but also on an architectural level. In the next article, we will continue to examine ways to combine functions by means of partial function application and currying. The Neophyte's Guide to Scala Part 9: Promises and Futures in practice Unknown — Wed, 16 Jan 2013 00:00:00 +0000 In the previous article in this series</a>, I introduced you to the Future</code> type, its underlying paradigm, and how to put it to use to write highly readable and composable asynchronously executing code. In that article, I also mentioned that Future</code> is really only one piece of the puzzle: It's a read-only type that allows you to work with the values it will compute and handle failure to do so in an elegant way. In order for you to be able to read a computed value from a Future</code>, however, there must be a way for some other part of your software to put that value there. In this post, I will show you how this is done by means of the Promise</code> type, followed by some guidance on how to use futures and promises in practice. Promises</h2> In the previous article on futures, we had a sequential block of code that we passed to the apply</code> method of the Future</code> companion object, and, given an ExecutionContext</code> was in scope, it magically executed that code block asynchronously, returning its result as a Future</code>. While this is an easy way to get a Future</code> when you want one, there is an alternative way to create Future</code> instances and have them complete with a success or failure. Where Future</code> provides an interface exclusively for querying, Promise</code> is a companion type that allows you to complete a Future</code> by putting a value into it. This can be done exactly once. Once a Promise</code> has been completed, it's not possible to change it any more. A Promise</code> instance is always linked to exactly one instance of Future</code>. If you call the apply</code> method of Future</code> again in the REPL, you will indeed notice that the Future</code> returned is a Promise</code>, too: import concurrent.Future import concurrent.ExecutionContext.Implicits.global val f: Future[String] = Future { "Hello world!" } // REPL output: // f: scala.concurrent.Future[String] = scala.concurrent.impl.Promise$DefaultPromise@793e6657 </code></pre> The object you get back is a DefaultPromise</code>, which implements both Future</code> and Promise</code>. This is an implementation detail, however. The Future</code> and the Promise</code> to which it belongs may very well be separate objects. What this little example shows is that there is obviously no way to complete a Future</code> other than through a Promise</code> – the apply</code> method on Future</code> is just a nice helper function that shields you from this. Now, let's see how we can get our hands dirty and work with the Promise</code> type directly. Promising a rosy future</h3> When talking about promises that may be fulfilled or not, an obvious example domain is that of politicians, elections, campaign pledges, and legislative periods. Suppose the politicians that then got elected into office promised their voters a tax cut. This can be represented as a Promise[TaxCut]</code>, which you can create by calling the apply</code> method on the Promise</code> companion object, like so: import concurrent.Promise case class TaxCut(reduction: Int) // either give the type as a type parameter to the factory method: val taxcut = Promise[TaxCut]() // or give the compiler a hint by specifying the type of your val: val taxcut2: Promise[TaxCut] = Promise() </code></pre> Once you have created a Promise</code>, you can get the Future</code> belonging to it by calling the future</code> method on the Promise</code> instance: val taxcutF: Future[TaxCut] = taxcut.future </code></pre> The returned Future</code> might not be the same object as the Promise</code>, but calling the future</code> method of a Promise</code> multiple times will definitely always return the same object to make sure the one-to-one relationship between a Promise</code> and its Future</code> is preserved. Completing a Promise</h3> Once you have made a Promise</code> and told the world that you will deliver on it in the forseeable Future</code>, you better do your very best to make it happen. In Scala, you can complete a Promise</code> either with a success or a failure. Delivering on your Promise</h4> To complete a Promise</code> with a success, you call its success</code> method, passing it the value that the Future</code> associated with it is supposed to have: taxcut.success(TaxCut(20)) </code></pre> Once you have done this, that Promise</code> instance is no longer writable, and future attempts to do so will lead to an exception. Also, completing your Promise</code> like this leads to the successful completion of the associated Future</code>. Any success or completion handlers on that future will now be called, or if, for instance, you are mapping that future, the mapping function will now be executed. Usually, the completion of the Promise</code> and the processing of the completed Future</code> will not happen in the same thread. It's more likely that you create your Promise</code>, start computing its value in another thread and immediately return the uncompleted Future</code> to the caller. To illustrate, let's do something like that for our taxcut promise: object Government { def redeemCampaignPledge(): Future[TaxCut] = { val p = Promise[TaxCut]() Future { println("Starting the new legislative period.") Thread.sleep(2000) p.success(TaxCut(20)) println("We reduced the taxes! You must reelect us!!!!1111") } p.future } } </code></pre> Please don't get confused by the usage of the apply</code> method of the Future</code> companion object in this example. I'm just using it because it is so convenient for executing a block of code asynchronously. I could just as well have implemented the computation of the result (which involves a lot of sleeping) in a Runnable</code> that is executed asynchrously by an ExecutorService</code>, with a lot more boilerplate code. The point is that the Promise</code> is not completed in the caller thread. Let's redeem our campaign pledge then and add an onComplete</code> callback function to our future: import scala.util.{Success, Failure} val taxCutF: Future[TaxCut] = Government.redeemCampaignPledge() println("Now that they're elected, let's see if they remember their promises...") taxCutF.onComplete { case Success(TaxCut(reduction)) => println(s"A miracle! They really cut our taxes by $reduction percentage points!") case Failure(ex) => println(s"They broke their promises! Again! Because of a ${ex.getMessage}") } </code></pre> If you try this out multiple times, you will see that the order of the console output is not deterministic. Eventually, the completion handler will be executed and run into the success case. Breaking Promises like a sir</h4> As a politician, you are pretty much used to not keeping your promises. As a Scala developer, you sometimes have no other choice, either. If that happens, you can still complete your Promise</code> instance gracefully, by calling its failure</code> method and passing it an exception: case class LameExcuse(msg: String) extends Exception(msg) object Government { def redeemCampaignPledge(): Future[TaxCut] = { val p = Promise[TaxCut]() Future { println("Starting the new legislative period.") Thread.sleep(2000) p.failure(LameExcuse("global economy crisis")) println("We didn't fulfill our promises, but surely they'll understand.") } p.future } } </code></pre> This implementation of the redeemCampaignPledge()</code> method will to lots of broken promises. Once you have completed a Promise</code> with the failure</code> method, it is no longer writable, just as is the case with the success</code> method. The associated Future</code> will now be completed with a Failure</code>, too, so the callback function above would run into the failure case. If you already have a Try</code>, you can also complete a Promise</code> by calling its complete</code> method. If the Try</code> is a Success</code>, the associated Future</code> will be completed successfully, with the value inside the Success</code>. If it's a Failure</code>, the Future</code> will completed with that failure. Future-based programming in practice</h2> If you want to make use of the future-based paradigm in order to increase the scalability of your application, you have to design your application to be non-blocking from the ground-up, which basically means that the functions in all your application layers are asynchronous and return futures. A likely use case these days is that of developing a web application. If you are using a modern Scala web framework, it will allow you to return your responses as something like a Future[Response]</code> instead of blocking and then returning your finished Response</code>. This is important since it allows your web server to handle a huge number of open connections with a relatively low number of threads. By always giving your web server Future[Response]</code>, you maximize the utilization of the web server's dedicated thread pool. In the end, a service in your application might make multiple calls to your database layer and/or some external web service, receiving multiple futures, and then compose them to return a new Future</code>, all in a very readable for comprehension, such as the one you saw in the previous article. The web layer will turn such a Future</code> into a Future[Response]</code>. However, how do you implement this in practice? There are are three different cases you have to consider: Non-blocking IO</h3> Your application will most certainly involve a lot of IO. For example, your web application will have to talk to a database, and it might act as a client that is calling other web services. If at all possible, make use of libraries that are based on Java's non-blocking IO capabilities, either by using Java's NIO API directly or through a library like Netty. Such libraries, too, can serve many connections with a reasonably-sized dedicated thread pool. Developing such a library yourself is one of the few places where directly working with the Promise</code> type makes a lot of sense. Blocking IO</h3> Sometimes, there is no NIO-based library available. For instance, most database drivers you'll find in the Java world nowadays are using blocking IO. If you made a query to your database with such a driver in order to respond to a HTTP request, that call would be made on a web server thread. To avoid that, place all the code talking to the database inside a future</code> block, like so: // get back a Future[ResultSet] or something similar: Future { queryDB(query) } </code></pre> So far, we always used the implicitly available global ExecutionContext</code> to execute such future blocks. It's probably a good idea to create a dedicated ExecutionContext</code> that you will have in scope in your database layer. You can create an ExecutionContext</code> from a Java ExecutorService</code>, which means you will be able to tune the thread pool for executing your database calls asynchronously independently from the rest of your application: import java.util.concurrent.Executors import concurrent.ExecutionContext val executorService = Executors.newFixedThreadPool(4) val executionContext = ExecutionContext.fromExecutorService(executorService) </code></pre> Long-running computations</h3> Depending on the nature of your application, it will occasionally have to call long-running tasks that don't involve any IO at all, which means they are CPU-bound. These, too, should not be executed by a web server thread. Hence, you should turn them into Futures</code>, too: Future { longRunningComputation(data, moreData) } </code></pre> Again, if you have long-running computations, having them run in a separate ExecutionContext</code> for CPU-bound tasks is a good idea. How to tune your various thread pools is highly dependent on your individual application and beyond the scope of this article. Summary</h2> In this article, we looked at promises, the writable part of the future-based concurrency paradigm, and how to use them to complete a Future</code>, followed by some advice on how to put futures to use in practice. In the next part of this series, we are taking a step back from concurrency issues and examine how functional programming in Scala can help you to make your code more reusable, a claim that has long been associated with object-oriented programming. The Neophyte's Guide to Scala Part 8: Welcome to the Future Unknown — Wed, 09 Jan 2013 00:00:00 +0000 As an aspiring and enthusiastic Scala developer, you will likely have heard of Scala's approach at dealing with concurrency – or maybe that was even what attracted you in the first place. Said approach makes reasoning about concurrency and writing well-behaved concurrent programs a lot easier than the rather low-level concurrency APIs you are confronted with in most other languages. One of the two cornerstones of this approach is the Future, the other being the Actor. The former shall be the subject of this article. I will explain what futures are good for and how you can make use of them in a functional way. Please make sure that you have version 2.9.3 or later if you want to get your hands dirty and try out the examples yourself. The futures we are discussing here were only incorporated into the Scala core distribution with the 2.10.0 release and later backported to Scala 2.9.3. Originally, with a slightly different API, they were part of the Akka concurrency toolkit. Why sequential code can be bad</h2> Suppose you want to prepare a cappuccino. You could simply execute the following steps, one after another: Grind the required coffee beans</li> Heat some water</li> Brew an espresso using the ground coffee and the heated water</li> Froth some milk</li> Combine the espresso and the frothed milk to a cappuccino</li> </ol> Translated to Scala code, you would do something like this: import scala.util.Try // Some type aliases, just for getting more meaningful method signatures: type CoffeeBeans = String type GroundCoffee = String case class Water(temperature: Int) type Milk = String type FrothedMilk = String type Espresso = String type Cappuccino = String // dummy implementations of the individual steps: def grind(beans: CoffeeBeans): GroundCoffee = s"ground coffee of $beans" def heatWater(water: Water): Water = water.copy(temperature = 85) def frothMilk(milk: Milk): FrothedMilk = s"frothed $milk" def brew(coffee: GroundCoffee, heatedWater: Water): Espresso = "espresso" def combine(espresso: Espresso, frothedMilk: FrothedMilk): Cappuccino = "cappuccino" // some exceptions for things that might go wrong in the individual steps // (we'll need some of them later, use the others when experimenting // with the code): case class GrindingException(msg: String) extends Exception(msg) case class FrothingException(msg: String) extends Exception(msg) case class WaterBoilingException(msg: String) extends Exception(msg) case class BrewingException(msg: String) extends Exception(msg) // going through these steps sequentially: def prepareCappuccino(): Try[Cappuccino] = for { ground <- Try(grind("arabica beans")) water <- Try(heatWater(Water(25))) espresso <- Try(brew(ground, water)) foam <- Try(frothMilk("milk")) } yield combine(espresso, foam) </code></pre> Doing it like this has several advantages: You get a very readable step-by-step instruction of what to do. Moreover, you will likely not get confused while preparing the cappuccino this way, since you are avoiding context switches. On the downside, preparing your cappuccino in such a step-by-step manner means that your brain and body are on wait during large parts of the whole process. While waiting for the ground coffee, you are effectively blocked. Only when that's finished, you're able to start heating some water, and so on. This is clearly a waste of valuable resources. It's very likely that you would want to initiate multiple steps and have them execute concurrently. Once you see that the water and the ground coffee is ready, you'd start brewing the espresso, in the meantime already starting the process of frothing the milk. It's really no different when writing a piece of software. A web server only has so many threads for processing requests and creating appropriate responses. You don't want to block these valuable threads by waiting for the results of a database query or a call to another HTTP service. Instead, you want an asynchronous programming model and non-blocking IO, so that, while the processing of one request is waiting for the response from a database, the web server thread handling that request can serve the needs of some other request instead of idling along. "I heard you like callbacks, so I put a callback in your callback!" </blockquote> Of course, you already knew all that - what with Node.js being all the rage among the cool kids for a while now. The approach used by Node.js and some others is to communicate via callbacks, exclusively. Unfortunately, this can very easily lead to a convoluted mess of callbacks within callbacks within callbacks, making your code hard to read and debug. Scala's Future</code> allows callbacks, too, as you will see very shortly, but it provides much better alternatives, so it's likely you won't need them a lot. "I know Futures, and they are completely useless!" </blockquote> You might also be familiar with other Future</code> implementations, most notably the one provided by Java. There is not really much you can do with a Java future other than checking if it's completed or simply blocking until it is completed. In short, they are nearly useless and definitely not a joy to work with. If you think that Scala's futures are anything like that, get ready for a surprise. Here we go! Semantics of Future</h2> Scala's Future[T]</code>, residing in the scala.concurrent</code> package, is a container type, representing a computation that is supposed to eventually result in a value of type T</code>. Alas, the computation might go wrong or time out, so when the future is completed, it may not have been successful after all, in which case it contains an exception instead. Future</code> is a write-once container – after a future has been completed, it is effectively immutable. Also, the Future</code> type only provides an interface for reading the value to be computed. The task of writing the computed value is achieved via a Promise</code>. Hence, there is a clear separation of concerns in the API design. In this post, we are focussing on the former, postponing the use of the Promise</code> type to the next article in this series. Working with Futures</h2> There are several ways you can work with Scala futures, which we are going to examine by rewriting our cappuccino example to make use of the Future</code> type. First, we need to rewrite all of the functions that can be executed concurrently so that they immediately return a Future</code> instead of computing their result in a blocking way: import scala.concurrent.future import scala.concurrent.Future import scala.concurrent.ExecutionContext.Implicits.global import scala.concurrent.duration._ import scala.util.Random def grind(beans: CoffeeBeans): Future[GroundCoffee] = Future { println("start grinding...") Thread.sleep(Random.nextInt(2000)) if (beans == "baked beans") throw GrindingException("are you joking?") println("finished grinding...") s"ground coffee of $beans" } def heatWater(water: Water): Future[Water] = Future { println("heating the water now") Thread.sleep(Random.nextInt(2000)) println("hot, it's hot!") water.copy(temperature = 85) } def frothMilk(milk: Milk): Future[FrothedMilk] = Future { println("milk frothing system engaged!") Thread.sleep(Random.nextInt(2000)) println("shutting down milk frothing system") s"frothed $milk" } def brew(coffee: GroundCoffee, heatedWater: Water): Future[Espresso] = Future { println("happy brewing :)") Thread.sleep(Random.nextInt(2000)) println("it's brewed!") "espresso" } </code></pre> There are several things that require an explanation here. First off, there is the apply</code> method on the Future</code> companion object, that requires two arguments: object Future { def apply[T](body: => T)(implicit execctx: ExecutionContext): Future[T] } </code></pre> The computation to be computed asynchronously is passed in as the body</code> by-name parameter. The second argument, in its own argument list, is an implicit one, which means we don't have to specify one if a matching implicit value is defined somewhere in scope. We make sure this is the case by importing the global execution context. An ExecutionContext</code> is something that can execute our future, and you can think of it as something like a thread pool. Since the ExecutionContext</code> is available implicitly, we only have a single one-element argument list remaining. Single-argument lists can be enclosed with curly braces instead of parentheses. People often make use of this when calling the future</code> method, making it look a little bit like we are using a feature of the language and not calling an ordinary method. The ExecutionContext</code> is an implicit parameter for virtually all of the Future</code> API. Furthermore, of course, in this simple example, we don't actually compute anything, which is why we are putting in some random sleep, simply for demonstration purposes. We also print to the console before and after our "computation" to make the non-deterministic and concurrent nature of our code clearer when trying it out. The computation of the value to be returned by a Future</code> will start at some non-deterministic time after that Future</code> instance has been created, by some thread assigned to it by the ExecutionContext</code>. Callbacks</h3> Sometimes, when things are simple, using a callback can be perfectly fine. Callbacks for futures are partial functions. You can pass a callback to the onSuccess</code> method. It will only be called if the Future</code> completes successfully, and if so, it receives the computed value as its input: grind("arabica beans").onSuccess { case ground => println("okay, got my ground coffee") } </code></pre> Similarly, you could register a failure callback with the onFailure</code> method. Your callback will receive a Throwable</code>, but it will only be called if the Future</code> did not complete successfully. Usually, it's better to combine these two and register a completion callback that will handle both cases. The input parameter for that callback is a Try</code>: import scala.util.{Success, Failure} grind("baked beans").onComplete { case Success(ground) => println(s"got my $ground") case Failure(ex) => println("This grinder needs a replacement, seriously!") } </code></pre> Since we are passing in baked beans, an exception occurs in the grind</code> method, leading to the Future</code> completing with a Failure</code>. Composing futures</h3> Using callbacks can be quite painful if you have to start nesting callbacks. Thankfully, you don't have to do that! The real power of the Scala futures is that they are composable. If you have followed this series, you will have noticed that all the container types we discussed made it possible for you to map them, flat map them, or use them in for comprehensions and that I mentioned that Future</code> is a container type, too. Hence, the fact that Scala's Future</code> type allows you to do all that will not come as a surprise at all. The real question is: What does it really mean to perform these operations on something that hasn't even finished computing yet? Mapping the future</h4> Haven't you always wanted to be a traveller in time who sets out to map the future? As a Scala developer you can do exactly that! Suppose that once your water has heated you want to check if its temperature is okay. You can do so by mapping your Future[Water]</code> to a Future[Boolean]</code>: val temperatureOkay: Future[Boolean] = heatWater(Water(25)).map { water => println("we're in the future!") (80 to 85).contains(water.temperature) } </code></pre> The Future[Boolean]</code> assigned to temperatureOkay</code> will eventually contain the successfully computed boolean value. Go change the implementation of heatWater</code> so that it throws an exception (maybe because your water heater explodes or something) and watch how we're in the future</code> will never be printed to the console. When you are writing the function you pass to map</code>, you're in the future, or rather in a possible future. That mapping function gets executed as soon as your Future[Water]</code> instance has completed successfully. However, the timeline in which that happens might not be the one you live in. If your instance of Future[Water]</code> fails, what's taking place in the function you passed to map</code> will never happen. Instead, the result of calling map</code> will be a Future[Boolean]</code> containing a Failure</code>. Keeping the future flat</h4> If the computation of one Future</code> depends on the result of another, you'll likely want to resort to flatMap</code> to avoid a deeply nested structure of futures. For example, let's assume that the process of actually measuring the temperature takes a while, so you want to determine whether the temperature is okay asynchronously, too. You have a function that takes an instance of Water</code> and returns a Future[Boolean]</code>: def temperatureOkay(water: Water): Future[Boolean] = Future { (80 to 85).contains(water.temperature) } </code></pre> Use flatMap</code> instead of map</code> in order to get a Future[Boolean]</code> instead of a Future[Future[Boolean]]</code>: val nestedFuture: Future[Future[Boolean]] = heatWater(Water(25)).map { water => temperatureOkay(water) } val flatFuture: Future[Boolean] = heatWater(Water(25)).flatMap { water => temperatureOkay(water) } </code></pre> Again, the mapping function is only executed after (and if) the Future[Water]</code> instance has been completed successfully, hopefully with an acceptable temperature. For comprehensions</h4> Instead of calling flatMap</code>, you'll usually want to write a for comprehension, which is essentially the same, but reads a lot clearer. Our example above could be rewritten like this: val acceptable: Future[Boolean] = for { heatedWater <- heatWater(Water(25)) okay <- temperatureOkay(heatedWater) } yield okay </code></pre> If you have multiple computations that can be computed in parallel, you need to take care that you already create the corresponding Future</code> instances outside of the for comprehension. def prepareCappuccinoSequentially(): Future[Cappuccino] = { for { ground <- grind("arabica beans") water <- heatWater(Water(20)) foam <- frothMilk("milk") espresso <- brew(ground, water) } yield combine(espresso, foam) } </code></pre> This reads nicely, but since a for comprehension is just another representation for nested flatMap</code> calls, this means that the Future[Water]</code> created in heatWater</code> is only really instantiated after the Future[GroundCoffee]</code> has completed successfully. You can check this by watching the sequential console output coming from the functions we implemented above. Hence, make sure to instantiate all your independent futures before the for comprehension: def prepareCappuccino(): Future[Cappuccino] = { val groundCoffee = grind("arabica beans") val heatedWater = heatWater(Water(20)) val frothedMilk = frothMilk("milk") for { ground <- groundCoffee water <- heatedWater foam <- frothedMilk espresso <- brew(ground, water) } yield combine(espresso, foam) } </code></pre> Now, the three futures we create before the for comprehension start being completed immediately and execute concurrently. If you watch the console output, you will see that it's non-deterministic. The only thing that's certain is that the "happy brewing"</code> output will come last. Since the method in which it is called requires the values coming from two other futures, it is only created inside our for comprehension, i.e. after those futures have completed successfully. Failure projections</h4> You will have noticed that Future[T]</code> is success-biased, allowing you to use map</code>, flatMap</code>, filter</code> etc. under the assumption that it will complete successfully. Sometimes, you may want to be able to work in this nice functional way for the timeline in which things go wrong. By calling the failed</code> method on an instance of Future[T]</code>, you get a failure projection of it, which is a Future[Throwable]</code>. Now you can map</code> that Future[Throwable]</code>, for example, and your mapping function will only be executed if the original Future[T]</code> has completed with a failure. Outlook</h2> You have seen the Future</code>, and it looks bright! The fact that it's just another container type that can be composed and used in a functional way makes working with it very pleasant. Making blocking code concurrent can be pretty easy by wrapping it in a call to future</code>. However, it's better to be non-blocking in the first place. To achieve this, one has to make a Promise</code> to complete a Future</code>. This and using the futures in practice will be the topic of the next part of this series. The Neophyte's Guide to Scala Part 7: The Either type Unknown — Wed, 02 Jan 2013 00:00:00 +0000 In the previous article, I discussed functional error handling using Try</a>, which was introduced in Scala 2.10. I also mentioned the existence of another, somewhat similar type called Either</code>, which is the subject of this article. You will learn how to use it, when to use it, and what its particular pitfalls are. Speaking of which, at least at the time of this writing, Either</code> has some serious design flaws you need to be aware of, so much so that one might argue about whether to use it at all. So why then should you learn about Either</code> at all? For one, people will not all migrate their existing code bases to use Try</code> for dealing with exceptions, so it is good to be able to understand the intricacies of this type, too. Moreover, Try</code> is not really an all-out replacement for Either</code>, only for one particular usage of it, namely handling exceptions in a functional way. As it stands, Try</code> and Either</code> really complement each other, each covering different use cases. And, as flawed as Either</code> may be, in certain situations it will still be a very good fit. The semantics</h2> Like Option</code> and Try</code>, Either</code> is a container type. Unlike the aforementioned types, it takes not only one, but two type parameters: An Either[A, B]</code> instance can contain either an instance of A</code>, or an instance of B</code>. This is different from a Tuple2[A, B]</code>, which contains both an A</code> and a B</code> instance. Either</code> has exactly two sub types, Left</code> and Right</code>. If an Either[A, B]</code> object contains an instance of A</code>, then the Either</code> is a Left</code>. Otherwise it contains an instance of B</code> and is a Right</code>. There is nothing in the semantics of this type that specifies one or the other sub type to represent an error or a success, respectively. In fact, Either</code> is a general-purpose type for use whenever you need to deal with situations where the result can be of one of two possible types. Nevertheless, error handling is a popular use case for it, and by convention, when using it that way, the Left</code> represents the error case, whereas the Right</code> contains the success value. Creating an Either</h2> Creating an instance of Either</code> is trivial. Both Left</code> and Right</code> are case classes, so if we want to implement a rock-solid internet censorship feature, we can just do the following: import scala.io.Source import java.net.URL def getContent(url: URL): Either[String, Source] = if (url.getHost.contains("google")) Left("Requested URL is blocked for the good of the people!") else Right(Source.fromURL(url)) </code></pre> Now, if we call getContent(new URL("http://danielwestheide.com"))</code>, we will get a scala.io.Source</code> wrapped in a Right</code>. If we pass in new URL("https://plus.google.com")</code>, the return value will be a Left</code> containing a String</code>. Working with Either values</h2> Some of the very basic stuff works just as you know from Option</code> or Try</code>: You can ask an instance of Either</code> if it isLeft</code> or isRight</code>. You can also do pattern matching on it, which is one of the most familiar and convenient ways of working with objects of this type: getContent(new URL("http://google.com")) match { case Left(msg) => println(msg) case Right(source) => source.getLines.foreach(println) } </code></pre> Projections</h3> You cannot, at least not directly, use an Either</code> instance like a collection, the way you are familiar with from Option</code> and Try</code>. This is because Either</code> is designed to be unbiased. Try</code> is success-biased: it offers you map</code>, flatMap</code> and other methods that all work under the assumption that the Try</code> is a Success</code>, and if that's not the case, they effectively don't do anything, returning the Failure</code> as-is. The fact that Either</code> is unbiased means that you first have to choose whether you want to work under the assumption that it is a Left</code> or a Right</code>. By calling left</code> or right</code> on an Either</code> value, you get a LeftProjection</code> or RightProjection</code>, respectively, which are basically left- or right-biased wrappers for the Either</code>. Mapping</h4> Once you have a projection, you can call map</code> on it: val content: Either[String, Iterator[String]] = getContent(new URL("http://danielwestheide.com")).right.map(_.getLines()) // content is a Right containing the lines from the Source returned by getContent val moreContent: Either[String, Iterator[String]] = getContent(new URL("http://google.com")).right.map(_.getLines) // moreContent is a Left, as already returned by getContent </code></pre> Regardless of whether the Either[String, Source]</code> in this example is a Left</code> or a Right</code>, it will be mapped to an Either[String, Iterator[String]]</code>. If it's called on a Right</code>, the value inside it will be transformed. If it's a Left</code>, that will be returned unchanged. We can do the same with a LeftProjection</code>, of course: val content: Either[Iterator[String], Source] = getContent(new URL("http://danielwestheide.com")).left.map(Iterator(_)) // content is the Right containing a Source, as already returned by getContent val moreContent: Either[Iterator[String], Source] = getContent(new URL("http://google.com")).left.map(Iterator(_)) // moreContent is a Left containing the msg returned by getContent in an Iterator </code></pre> Now, if the Either</code> is a Left</code>, its wrapped value is transformed, whereas a Right</code> would be returned unchanged. Either way, the result is of type Either[Iterator[String], Source]</code>. Please note that the map</code> method is defined on the projection types, not on Either</code>, but it does return a value of type Either</code>, not a projection. In this, Either</code> deviates from the other container types you know. The reason for this has to do with Either</code> being unbiased, but as you will see, this can lead to some very unpleasant problems in certain cases. It also means that if you want to chain multiple calls to map</code>, flatMap</code> and the like, you always have to ask for your desired projection again in between. Flat mapping</h4> Projections also support flat mapping, avoiding the common problem of creating a convoluted structure of multiple inner and outer Either</code> types that you will end up with if you nest multiple calls to map</code>. I'm putting very high requirements on your suspension of disbelief now, coming up with a completely contrived example. Let's say we want to calculate the average number of lines of two of my articles. You've always wanted to do that, right? Here's how we could solve this challenging problem: val part5 = new URL("http://t.co/UR1aalX4") val part6 = new URL("http://t.co/6wlKwTmu") val content = getContent(part5).right.map(a => getContent(part6).right.map(b => (a.getLines().size + b.getLines().size) / 2)) </code></pre> What we'll end up with is an Either[String, Either[String, Int]]</code>. Now, content</code> being a nested structure of Right</code>s, we could flatten it by calling the joinRight</code> method on it (you also have joinLeft</code> available to flatten a nested structure of Left</code>s). However, we can avoid creating this nested structure altogether. If we flatMap</code> on our outer RightProjection</code>, we get a more pleasant result type, unpacking the Right</code> of the inner Either</code>: val content = getContent(part5).right.flatMap(a => getContent(part6).right.map(b => (a.getLines().size + b.getLines().size) / 2)) </code></pre> Now content</code> is a flat Either[String, Int]</code>, which makes it a lot nicer to work with, for example using pattern matching. For comprehensions</h4> By now, you have probably learned to love working with for comprehensions in a consistent way on various different data types. You can do that, too, with projections of Either</code>, but the sad truth is that it's not quite as nice, and there are things that you won't be able to do without resorting to ugly workarounds, out of the box. Let's rewrite our flatMap</code> example, making use of for comprehensions instead: def averageLineCount(url1: URL, url2: URL): Either[String, Int] = for { source1 <- getContent(url1).right source2 <- getContent(url2).right } yield (source1.getLines().size + source2.getLines().size) / 2 </code></pre> This is not too bad. Note that we have to call right</code> on each Either</code> we use in our generators, of course. Now, let's try to refactor this for comprehension – since the yield expression is a little too involved, we want to extract some parts of it into value definitions inside our for comprehension: def averageLineCountWontCompile(url1: URL, url2: URL): Either[String, Int] = for { source1 <- getContent(url1).right source2 <- getContent(url2).right lines1 = source1.getLines().size lines2 = source2.getLines().size } yield (lines1 + lines2) / 2 </code></pre> This won't compile! The reason will become clearer if we examine what this for comprehension corresponds to, if you take away the sugar. It translates to something that is similar to the following, albeit much less readable: def averageLineCountDesugaredWontCompile(url1: URL, url2: URL): Either[String, Int] = getContent(url1).right.flatMap { source1 => getContent(url2).right.map { source2 => val lines1 = source1.getLines().size val lines2 = source2.getLines().size (lines1, lines2) }.map { case (x, y) => x + y / 2 } } </code></pre> The problem is that by including a value definition in our for comprehension, a new call to map</code> is introduced automatically – on the result of the previous call to map</code>, which has returned an Either</code>, not a RightProjection</code>. As you know, Either</code> doesn't define a map</code> method, making the compiler a little bit grumpy. This is where Either</code> shows us its ugly trollface. In this example, the value definitions are not strictly necessary. If they are, you can work around this problem, replacing any value definitions by generators, like this: def averageLineCount(url1: URL, url2: URL): Either[String, Int] = for { source1 <- getContent(url1).right source2 <- getContent(url2).right lines1 <- Right(source1.getLines().size).right lines2 <- Right(source2.getLines().size).right } yield (lines1 + lines2) / 2 </code></pre> It's important to be aware of these design flaws. They don't make Either</code> unusable, but can lead to serious headaches if you don't have a clue what's going on. Other methods</h4> The projection types have some other useful methods: You can convert your Either</code> instance to an Option</code> by calling toOption</code> on one of its projections. For example, if you have an e</code> of type Either[A, B]</code>, e.right.toOption</code> will return an Option[B]</code>. If your Either[A, B]</code> instance is a Right</code>, that Option[B]</code> will be a Some</code>. If it's a Left</code>, it will be None</code>. The reverse behaviour can be achieved, of course, when calling toOption</code> on the LeftProjection</code> of your Either[A, B]</code>. If you need a sequence of either one value or none, use toSeq</code> instead. Folding</h3> If you want to transform an Either</code> value regardless of whether it is a Left</code> or a Right</code>, you can do so by means of the fold</code> method that is defined on Either</code>, expecting two transform functions with the same result type, the first one being called if the Either</code> is a Left</code>, the second one if it's a Right</code>. To demonstrate this, let's combine the two mapping operations we implemented on the LeftProjection</code> and the RightProjection</code> above: val content: Iterator[String] = getContent(new URL("http://danielwestheide.com")).fold(Iterator(_), _.getLines()) val moreContent: Iterator[String] = getContent(new URL("http://google.com")).fold(Iterator(_), _.getLines()) </code></pre> In this example, we are transforming our Either[String, Source]</code> into an Iterator[String]</code>, no matter if it's a Left</code> or a Right</code>. You could just as well return a new Either</code> again or execute side-effects and return Unit</code> from your two functions. As such, calling fold</code> provides a nice alternative to pattern matching. When to use Either</h2> Now that you have seen how to work with Either</code> values and what you have to take care of, let's move on to some specific use cases. Error handling</h3> You can use Either</code> for exception handling very much like Try</code>. Either</code> has one advantage over Try</code>: you can have more specific error types at compile time, while Try</code> uses Throwable</code> all the time. This means that Either</code> can be a good choice for expected errors. You'd have to implement a method like this, delegating to the very useful Exception</code> object from the scala.util.control</code> package: import scala.util.control.Exception.catching def handling[Ex <: Throwable, T](exType: Class[Ex])(block: => T): Either[Ex, T] = catching(exType).either(block).asInstanceOf[Either[Ex, T]] </code></pre> The reason you might want to do that is because while the methods provided by scala.util.Exception</code> allow you to catch only certain types of exceptions, the resulting compile-time error type is always Throwable</code>. With this method at hand, you can pass along expected exceptions in an Either</code>: import java.net.MalformedURLException def parseURL(url: String): Either[MalformedURLException, URL] = handling(classOf[MalformedURLException])(new URL(url)) </code></pre> You will have other expected error conditions, and not all of them result in third-party code throwing an exception you need to handle, as in the example above. In these cases, there is really no need to throw an exception yourself, only to catch it and wrap it in a Left</code>. Instead, simply define your own error type, preferably as a case class, and return a Left</code> wrapping an instance of that error type in your expected error condition occurs. Here is an example: case class Customer(age: Int) class Cigarettes case class UnderAgeFailure(age: Int, required: Int) def buyCigarettes(customer: Customer): Either[UnderAgeFailure, Cigarettes] = if (customer.age < 16) Left(UnderAgeFailure(customer.age, 16)) else Right(new Cigarettes) </code></pre> You should avoid using Either</code> for wrapping unexpected exceptions. Try</code> does that better, without all the flaws you have to deal with when working with Either</code>. Processing collections</h3> Generally, Either</code> is a pretty good fit if you want to process a collection, where for some items in that collection, this might result in a condition that is problematic, but should not directly result in an exception, which would result in aborting the processing of the rest of the collection. Let's assume that for our industry-standard web censorship system, we are using some kind of black list: type Citizen = String case class BlackListedResource(url: URL, visitors: Set[Citizen]) val blacklist = List( BlackListedResource(new URL("https://google.com"), Set("John Doe", "Johanna Doe")), BlackListedResource(new URL("http://yahoo.com"), Set.empty), BlackListedResource(new URL("https://maps.google.com"), Set("John Doe")), BlackListedResource(new URL("http://plus.google.com"), Set.empty) ) </code></pre> A BlackListedResource</code> represents the URL of a black-listed web page plus the citizens who have tried to visit that page. Now we want to process this black list, where our main purpose is to identify problematic citizens, i.e. those that have tried to visit blocked pages. At the same time, we want to identify suspicous web pages – if not a single citizen has tried to visit a black-listed page, we must assume that our subjects are bypassing our filter somehow, and we need to investigate that. Here is how we can process our black list: val checkedBlacklist: List[Either[URL, Set[Citizen]]] = blacklist.map(resource => if (resource.visitors.isEmpty) Left(resource.url) else Right(resource.visitors)) </code></pre> We have created a sequence of Either</code> values, with the Left</code> instances representing suspicious URLs and the Right</code> ones containing sets of problem citizens. This makes it almost a breeze to identify both our problem citizens and our suspicious web pages: val suspiciousResources = checkedBlacklist.flatMap(_.left.toOption) val problemCitizens = checkedBlacklist.flatMap(_.right.toOption).flatten.toSet </code></pre> These more general use cases beyond exception handling are where Either</code> really shines. Summary</h2> You have learned how to make use of Either</code>, what its pitfalls are, and when to put it to use in your code. It's a type that is not without flaws, and whether you want to have to deal with them and incorporate it in your own code is ultimately up to you. In practice, you will notice that, now that we have Try</code> at our hands, there won't be terribly many use cases for it. Nevertheless, it's good to know about it, both for those situations where it will be your perfect tool and to understand pre-2.10 Scala code you come across where it's used for error handling. The Neophyte's Guide to Scala Unknown — Tue, 01 Jan 2013 00:00:00 +0000 From November 2012 to April 2013, I created and published a blog series called The Neophyte's Guide to Scala, targeted at aspiring Scala enthusiasts who have already made their first steps with the language and are looking for more detailed explanations. Later in 2013, I published an ebook version (PDF, epub, Kindle) at Leanpub</a> using a pay-as-much-as-you-want price model. Get ebook</a> </div> If you prefer an HTML version, here are the links to all its 16 parts: </div> Part 1: Extractors</a></li> Part 2: Extracting Sequences</a></li> Part 3: Patterns Everywhere</a></li> Part 4: Pattern Matching Anonymous Functions</a></li> Part 5: The Option type</a></li> Part 6: Error handling with Try</a></li> Part 7: The Either type</a></li> Part 8: Welcome to the Future</a></li> Part 9: Promises and Futures in practice</a></li> Part 10: Staying DRY with higher-order functions</a></li> Part 11: Currying and partially applied functions</a></li> Part 12: Type classes</a></li> Part 13: Path-dependent types</a></li> Part 14: The Actor approach to concurrency</a></li> Part 15: Dealing with Failure in Actor Systems</a></li> Part 16: Where to go from here</a></li> </ul> The Neophyte's Guide to Scala Part 6: Error handling with Try Unknown — Wed, 26 Dec 2012 00:00:00 +0000 When just playing around with a new language, you might get away with simply ignoring the fact that something might go wrong. As soon you want to create anything serious, though, you can no longer run away from handling errors and exceptions in your code. The importance of how well a language supports you in doing so is often underestimated, for some reason or another. Scala, as it turns out, is pretty well positioned when it comes to dealing with error conditions in an elegant way. In this article, I'm going to present Scala's approach to dealing with errors, based on the Try</code> type, and the rationale behind it. I'm using features introduced with Scala 2.10 and ported back to Scala 2.9.3, so make sure your Scala version in SBT is at 2.9.3 or later. Throwing and catching exceptions</h2> Before going straight to Scala's idiomatic approach at error handling, let's first have a look at an approach that is more akin to how you are used to working with error conditions if you come from languages like Java or Ruby. Like these languages, Scala allows you to throw an exception: case class Customer(age: Int) class Cigarettes case class UnderAgeException(message: String) extends Exception(message) def buyCigarettes(customer: Customer): Cigarettes = if (customer.age < 16) throw UnderAgeException(s"Customer must be older than 16 but was ${customer.age}") else new Cigarettes </code></pre> Thrown exceptions can be caught and dealt with very similarly to Java, albeit using a partial function to specify the exceptions we want to deal with. Also, Scala's try</code>/catch</code> is an expression, so the following code returns the message of the exception: val youngCustomer = Customer(15) try { buyCigarettes(youngCustomer) "Yo, here are your cancer sticks! Happy smokin'!" } catch { case UnderAgeException(msg) => msg } </code></pre> Error handling, the functional way</h2> Now, having this kind of exception handling code all over your code base can become ugly very quickly and doesn't really go well with functional programming. It's also a rather bad solution for applications with a lot of concurrency. For instance, if you need to deal with an exception thrown by an Actor that is executed on some other thread, you obviously cannot do that by catching that exception – you will want a possibility to receive a message denoting the error condition. Hence, in Scala, it's usually preferred to signify that an error has occurred by returning an appropriate value from your function. Don't worry, we are not going back to C-style error handling, using error codes that we need to check for by convention. Rather, in Scala, we are using a specific type that represents computations that may result in an exception. In this article, we are confining ourselves to the Try</code> type that was introduced in Scala 2.10 and later backported to Scala 2.9.3. There is also a similar type, called Either</code>, which, even after the introduction of Try</code>, can still be very useful, but is more general. The semantics of Try</h3> The semantics of Try</code> are best explained by comparing them to those of the Option</code> type that was the topic of the previous part of this series</a>. Where Option[A]</code> is a container for a value of type A</code> that may be present or not, Try[A]</code> represents a computation that may result in a value of type A</code>, if it is successful, or in some Throwable</code> if something has gone wrong. Instances of such a container type for possible errors can easily be passed around between concurrently executing parts of your application. There are two different types of Try</code>: If an instance of Try[A]</code> represents a successful computation, it is an instance of Success[A]</code>, simply wrapping a value of type A</code>. If, on the other hand, it represents a computation in which an error has occurred, it is an instance of Failure[A]</code>, wrapping a Throwable</code>, i.e. an exception or other kind of error. If we know that a computation may result in an error, we can simply use Try[A]</code> as the return type of our function. This makes the possibility explicit and forces clients of our function to deal with the possibility of an error in some way. For example, let's assume we want to write a simple web page fetcher. The user will be able to enter the URL of the web page they want to fetch. One part of our application will be a function that parses the entered URL and creates a java.net.URL</code> from it: import scala.util.Try import java.net.URL def parseURL(url: String): Try[URL] = Try(new URL(url)) </code></pre> As you can see, we return a value of type Try[URL]</code>. If the given url</code> is syntactically correct, this will be a Success[URL]</code>. If the URL</code> constructor throws a MalformedURLException</code>, however, it will be a Failure[URL]</code>. To achieve this, we are using the apply</code> factory method on the Try</code> companion object. This method expects a by-name parameter of type A</code> (here, URL</code>). For our example, this means that the new URL(url)</code> is executed inside the apply</code> method of the Try</code> object. Inside that method, non-fatal exceptions are caught, returning a Failure</code> containing the respective exception. Hence, parseURL("http://danielwestheide.com")</code> will result in a Success[URL]</code> containing the created URL, whereas parseURL("garbage")</code> will result in a Failure[URL]</code> containing a MalformedURLException</code>. Working with Try values</h3> Working with Try</code> instances is actually very similar to working with Option</code> values, so you won't see many surprises here. You can check if a Try</code> is a success by calling isSuccess</code> on it and then conditionally retrieve the wrapped value by calling get</code> on it. But believe me, there aren't many situations where you will want to do that. It's also possible to use getOrElse</code> to pass in a default value to be returned if the Try</code> is a Failure</code>: val url = parseURL(Console.readLine("URL: ")) getOrElse new URL("http://duckduckgo.com") </code></pre> If the URL given by the user is malformed, we use the URL of DuckDuckGo as a fallback. Chaining operations</h3> One of the most important characteristics of the Try</code> type is that, like Option</code>, it supports all the higher-order methods you know from other types of collections. As you will see in the examples to follow, this allows you to chain operations on Try</code> values and catch any exceptions that might occur, and all that in a very readable manner. Mapping and flat mapping</h4> Mapping a Try[A]</code> that is a Success[A]</code> to a Try[B]</code> results in a Success[B]</code>. If it's a Failure[A]</code>, the resulting Try[B]</code> will be a Failure[B]</code>, on the other hand, containing the same exception as the Failure[A]</code>: parseURL("http://danielwestheide.com").map(_.getProtocol) // results in Success("http") parseURL("garbage").map(_.getProtocol) // results in Failure(java.net.MalformedURLException: no protocol: garbage) </code></pre> If you chain multiple map</code> operations, this will result in a nested Try</code> structure, which is usually not what you want. Consider this method that returns an input stream for a given URL: import java.io.InputStream def inputStreamForURL(url: String): Try[Try[Try[InputStream]]] = parseURL(url).map { u => Try(u.openConnection()).map(conn => Try(conn.getInputStream)) } </code></pre> Since the anonymous functions passed to the two map</code> calls each return a Try</code>, the return type is a Try[Try[Try[InputStream]]]</code>. This is where the fact that you can flatMap</code> a Try</code> comes in handy. The flatMap</code> method on a Try[A]</code> expects to be passed a function that receives an A</code> and returns a Try[B]</code>. If our Try[A]</code> instance is already a Failure[A]</code>, that failure is returned as a Failure[B]</code>, simply passing along the wrapped exception along the chain. If our Try[A]</code> is a Success[A]</code>, flatMap</code> unpacks the A</code> value in it and maps it to a Try[B]</code> by passing this value to the mapping function. This means that we can basically create a pipeline of operations that require the values carried over in Success</code> instances by chaining an arbitrary number of flatMap</code> calls. Any exceptions that happen along the way are wrapped in a Failure</code>, which means that the end result of the chain of operations is a Failure</code>, too. Let's rewrite the inputStreamForURL</code> method from the previous example, this time resorting to flatMap</code>: def inputStreamForURL(url: String): Try[InputStream] = parseURL(url).flatMap { u => Try(u.openConnection()).flatMap(conn => Try(conn.getInputStream)) } </code></pre> Now we get a Try[InputStream]</code>, which can be a Failure</code> wrapping an exception from any of the stages in which one may be thrown, or a Success</code> that directly wraps the InputStream</code>, the final result of our chain of operations. Filter and foreach</h4> Of course, you can also filter a Try</code> or call foreach</code> on it. Both work exactly as you would expect after having learned about Option</code>. The filter</code> method returns a Failure</code> if the Try</code> on which it is called is already a Failure</code> or if the predicate passed to it returns false</code> (in which case the wrapped exception is a NoSuchElementException</code>). If the Try</code> on which it is called is a Success</code> and the predicate returns true</code>, that Succcess</code> instance is returned unchanged: def parseHttpURL(url: String) = parseURL(url).filter(_.getProtocol == "http") parseHttpURL("http://apache.openmirror.de") // results in a Success[URL] parseHttpURL("ftp://mirror.netcologne.de/apache.org") // results in a Failure[URL] </code></pre> The function passed to foreach</code> is executed only if the Try</code> is a Success</code>, which allows you to execute a side-effect. The function passed to foreach</code> is executed exactly once in that case, being passed the value wrapped by the Success</code>: parseHttpURL("http://danielwestheide.com").foreach(println) </code></pre> For comprehensions</h4> The support for flatMap</code>, map</code> and filter</code> means that you can also use for comprehensions in order to chain operations on Try</code> instances. Usually, this results in more readable code. To demonstrate this, let's implement a method that returns the content of a web page with a given URL using for comprehensions. import scala.io.Source def getURLContent(url: String): Try[Iterator[String]] = for { url <- parseURL(url) connection <- Try(url.openConnection()) is <- Try(connection.getInputStream) source = Source.fromInputStream(is) } yield source.getLines() </code></pre> There are three places where things can go wrong, all of them covered by usage of the Try</code> type. First, the already implemented parseURL</code> method returns a Try[URL]</code>. Only if this is a Success[URL]</code>, we will try to open a connection and create a new input stream from it. If opening the connection and creating the input stream succeeds, we continue, finally yielding the lines of the web page. Since we effectively chain multiple flatMap</code> calls in this for comprehension, the result type is a flat Try[Iterator[String]]</code>. Please note that this could be simplified using Source#fromURL</code> and that we fail to close our input stream at the end, both of which are due to my decision to keep the example focussed on getting across the subject matter at hand. Pattern Matching</h3> At some point in your code, you will often want to know whether a Try</code> instance you have received as the result of some computation represents a success or not and execute different code branches depending on the result. Usually, this is where you will make use of pattern matching. This is easily possible because both Success</code> and Failure</code> are case classes. We want to render the requested page if it could be retrieved, or print an error message if that was not possible: import scala.util.Success import scala.util.Failure getURLContent("http://danielwestheide.com/foobar") match { case Success(lines) => lines.foreach(println) case Failure(ex) => println(s"Problem rendering URL content: ${ex.getMessage}") } </code></pre> Recovering from a Failure</h3> If you want to establish some kind of default behaviour in the case of a Failure</code>, you don't have to use getOrElse</code>. An alternative is recover</code>, which expects a partial function and returns another Try</code>. If recover</code> is called on a Success</code> instance, that instance is returned as is. Otherwise, if the partial function is defined for the given Failure</code> instance, its result is returned as a Success</code>. Let's put this to use in order to print a different message depending on the type of the wrapped exception: import java.net.MalformedURLException import java.io.FileNotFoundException val content = getURLContent("garbage") recover { case e: FileNotFoundException => Iterator("Requested page does not exist") case e: MalformedURLException => Iterator("Please make sure to enter a valid URL") case _ => Iterator("An unexpected error has occurred. We are so sorry!") } </code></pre> We could now safely get</code> the wrapped value on the Try[Iterator[String]]</code> that we assigned to content</code>, because we know that it must be a Success</code>. Calling content.get.foreach(println)</code> would result in Please make sure to enter a valid URL</code> being printed to the console. Conclusion</h2> Idiomatic error handling in Scala is quite different from the paradigm known from languages like Java or Ruby. The Try</code> type allows you to encapsulate computations that result in errors in a container and to chain operations on the computed values in a very elegant way. You can transfer what you know from working with collections and with Option</code> values to how you deal with code that may result in errors – all in a uniform way. To keep this article at a reasonable length, I haven't explained all of the methods available on Try</code>. Like Option</code>, Try</code> supports the orElse</code> method. The transform</code> and recoverWith</code> methods are also worth having a look at, and I encourage you to do so. In the next part we are going to deal with Either</code>, an alternative type for representing computations that may result in errors, but with a wider scope of application that goes beyond error handling. The Neophyte's Guide to Scala Part 5: The Option type Unknown — Wed, 19 Dec 2012 00:00:00 +0000 For the last couple of weeks, we have pressed ahead and covered a lot of ground concerning some rather advanced techniques, particularly ones related to pattern matching and extractors. Time to shift down a gear and look at one of the more fundamental idiosyncrasies of Scala: the Option</code> type. If you have participated in the Scala course at Coursera, you have already received a brief introduction to this type and seen it in use in the Map</code> API. In this series, we have also used it when implementing our own extractors</a>. And yet, there is still a lot left to be explained about it. You may have wondered what all the fuss is about, what is so much better about options than other ways of dealing with absent values. You might also be at a loss how to actually work with the Option</code> type in your own code. The goal of this part of the series is to do away with all these question marks and teach you all you really need to know about Option</code> as an aspiring Scala novice. The basic idea</h2> If you have worked with Java at all in the past, it is very likely that you have come across a NullPointerException</code> at some time (other languages will throw similarly named errors in such a case). Usually this happens because some method returns null</code> when you were not expecting it and thus not dealing with that possibility in your client code. A value of null</code> is often abused to represent an absent optional value. Some languages treat null</code> values in a special way or allow you to work safely with values that might be null</code>. For instance, Groovy has the null-safe operator for accessing properties, so that foo?.bar?.baz</code> will not throw an exception if either foo</code> or its bar</code> property is null</code>, instead directly returning null</code>. However, you are screwed if you forget to use this operator, and nothing forces you to do so. Clojure basically treats its nil</code> value like an empty thing, i.e. like an empty list if accessed like a list, or like an empty map if accessed like a map. This means that the nil</code> value is bubbling up the call hierarchy. Very often this is okay, but sometimes this just leads to an exception much higher in the call hierchary, where some piece of code isn't that nil-friendly after all. Scala tries to solve the problem by getting rid of null</code> values altogether and providing its own type for representing optional values, i.e. values that may be present or not: the Option[A]</code> trait. Option[A]</code> is a container for an optional value of type A</code>. If the value of type A</code> is present, the Option[A]</code> is an instance of Some[A]</code>, containing the present value of type A.</code> If the value is absent, the Option[A]</code> is the object None</code>. By stating that a value may or may not be present on the type level, you and any other developers who work with your code are forced by the compiler to deal with this possibility. There is no way you may accidentally rely on the presence of a value that is really optional. Option</code> is mandatory! Do not use null</code> to denote that an optional value is absent. Creating an option</h2> Usually, you can simply create an Option[A]</code> for a present value by directly instantiating the Some</code> case class: val greeting: Option[String] = Some("Hello world") </code></pre> Or, if you know that the value is absent, you simply assign or return the None</code> object: val greeting: Option[String] = None </code></pre> However, time and again you will need to interoperate with Java libraries or code in other JVM languages that happily make use of null</code> to denote absent values. For this reason, the Option</code> companion object provides a factory method that creates None</code> if the given parameter is null</code>, otherwise the parameter wrapped in a Some</code>: val absentGreeting: Option[String] = Option(null) // absentGreeting will be None val presentGreeting: Option[String] = Option("Hello!") // presentGreeting will be Some("Hello!") </code></pre> Working with optional values</h2> This is all pretty neat, but how do you actually work with optional values? It's time for an example. Let's do something boring, so we can focus on the important stuff. Imagine you are working for one of those hipsterrific startups, and one of the first things you need to implement is a repository of users. We need to be able to find a user by their unique id. Sometimes, requests come in with bogus ids. This calls for a return type of Option[User]</code> for our finder method. A dummy implementation of our user repository might look like this: case class User( id: Int, firstName: String, lastName: String, age: Int, gender: Option[String]) object UserRepository { private val users = Map(1 -> User(1, "John", "Doe", 32, Some("male")), 2 -> User(2, "Johanna", "Doe", 30, None)) def findById(id: Int): Option[User] = users.get(id) def findAll = users.values } </code></pre> Now, if you received an instance of Option[User]</code> from the UserRepository</code> and need to do something with it, how do you do that? One way would be to check if a value is present by means of the isDefined</code> method of your option, and, if that is the case, get that value via its get</code> method: val user1 = UserRepository.findById(1) if (user1.isDefined) { println(user1.get.firstName) } // will print "John" </code></pre> This is very similar to how the Optional</code> type in the Guava library is used in Java. If you think this is clunky and expect something more elegant from Scala, you're on the right track. More importantly, if you use get</code>, you might forget about checking with isDefined</code> before, leading to an exception at runtime, so you haven't gained a lot over using null</code>. You should stay away from this way of accessing options whenever possible! Providing a default value</h2> Very often, you want to work with a fallback or default value in case an optional value is absent. This use case is covered pretty well by the getOrElse</code> method defined on Option</code>: val user = User(2, "Johanna", "Doe", 30, None) println("Gender: " + user.gender.getOrElse("not specified")) // will print "not specified" </code></pre> Please note that the default value you can specify as a parameter to the getOrElse</code> method is a by-name parameter, which means that it is only evaluated if the option on which you invoke getOrElse</code> is indeed None</code>. Hence, there is no need to worry if creating the default value is costly for some reason or another – this will only happen if the default value is actually required. Pattern matching</h2> Some</code> is a case class, so it is perfectly possible to use it in a pattern, be it in a regular pattern matching expression or in some other place where patterns are allowed. Let's rewrite the example above using pattern matching: val user = User(2, "Johanna", "Doe", 30, None) user.gender match { case Some(gender) => println("Gender: " + gender) case None => println("Gender: not specified") } </code></pre> Or, if you want to remove the duplicated println</code> statement and make use of the fact that you are working with a pattern matching expression: val user = User(2, "Johanna", "Doe", 30, None) val gender = user.gender match { case Some(gender) => gender case None => "not specified" } println("Gender: " + gender) </code></pre> You will hopefully have noticed that pattern matching on an Option</code> instance is rather verbose, which is also why it is usually not idiomatic to process options this way. So, even if you are all excited about pattern matching, try to use the alternatives when working with options. There is one quite elegant way of using patterns with options, which you will learn about in the section on for comprehensions, below. Options can be viewed as collections</h2> So far you haven't seen a lot of elegant or idiomatic ways of working with options. We are coming to that now. I already mentioned that Option[A]</code> is a container for a value of type A</code>. More precisely, you may think of it as some kind of collection — some special snowflake of a collection that contains either zero elements or exactly one element of type A</code>. This is a very powerful idea! Even though on the type level, Option</code> is not a collection type in Scala, options come with all the goodness you have come to appreciate about Scala collections like List</code>, Set</code> etc — and if you really need to, you can even transform an option into a List</code>, for instance. So what does this allow you to do? Performing a side-effect if a value is present</h3> If you need to perform some side-effect only if a specific optional value is present, the foreach</code> method you know from Scala's collections comes in handy: UserRepository.findById(2).foreach(user => println(user.firstName)) // prints "Johanna" </code></pre> The function passed to foreach</code> will be called exactly once, if the Option</code> is a Some</code>, or never, if it is None</code>. Mapping an option</h3> The really good thing about options behaving like a collection is that you can work with them in a very functional way, and the way you do that is exactly the same as for lists, sets etc. Just as you can map a List[A]</code> to a List[B]</code>, you can map an Option[A]</code> to an Option[B]</code>. This means that if your instance of Option[A]</code> is defined, i.e. it is Some[A]</code>, the result is Some[B]</code>, otherwise it is None</code>. If you compare Option</code> to List</code>, None</code> is the equivalent of an empty list: when you map an empty List[A]</code>, you get an empty List[B]</code>, and when you map an Option[A]</code> that is None</code>, you get an Option[B]</code> that is None</code>. Let's get the age of an optional user: val age = UserRepository.findById(1).map(_.age) // age is Some(32) </code></pre> flatMap and options</h3> Let's do the same for the gender: val gender = UserRepository.findById(1).map(_.gender) // gender is an Option[Option[String]] </code></pre> The type of the resulting gender</code> is Option[Option[String]]</code>. Why is that? Think of it like this: You have an Option</code> container for a User</code>, and inside that container you are mapping the User</code> instance to an Option[String]</code>, since that is the type of the gender</code> property on our User</code> class. These nested options are a nuisance? Why, no problem, like all collections, Option</code> also provides a flatMap</code> method. Just like you can flatMap</code> a List[List[A]]</code> to a List[B]</code>, you can do the same for an Option[Option[A]]</code>: val gender1 = UserRepository.findById(1).flatMap(_.gender) // gender is Some("male") val gender2 = UserRepository.findById(2).flatMap(_.gender) // gender is None val gender3 = UserRepository.findById(3).flatMap(_.gender) // gender is None </code></pre> The result type is now Option[String]</code>. If the user is defined and its gender is defined, we get it as a flattened Some</code>. If either the use or its gender is undefined, we get a None</code>. To understand how this works, let's have a look at what happens when flat mapping a list of lists of strings, always keeping in mind that an Option</code> is just a collection, too, like a List</code>: val names: List[List[String]] = List(List("John", "Johanna", "Daniel"), List(), List("Doe", "Westheide")) names.map(_.map(_.toUpperCase)) // results in List(List("JOHN", "JOHANNA", "DANIEL"), List(), List("DOE", "WESTHEIDE")) names.flatMap(_.map(_.toUpperCase)) // results in List("JOHN", "JOHANNA", "DANIEL", "DOE", "WESTHEIDE") </code></pre> If we use flatMap</code>, the mapped elements of the inner lists are converted into a single flat list of strings. Obviously, nothing will remain of any empty inner lists. To lead us back to the Option</code> type, consider what happens if you map a list of options of strings: val names: List[Option[String]] = List(Some("Johanna"), None, Some("Daniel")) names.map(_.map(_.toUpperCase)) // List(Some("JOHANNA"), None, Some("DANIEL")) names.flatMap(xs => xs.map(_.toUpperCase)) // List("JOHANNA", "DANIEL") </code></pre> If you just map over the list of options, the result type stays List[Option[String]]</code>. Using flatMap</code>, all elements of the inner collections are put into a flat list: The one element of any Some[String]</code> in the original list is unwrapped and put into the result list, whereas any None</code> value in the original list does not contain any element to be unwrapped. Hence, None</code> values are effectively filtered out. With this in mind, have a look again at what flatMap</code> does on the Option</code> type. Filtering an option</h3> You can filter an option just like you can filter a list. If the instance of Option[A]</code> is defined, i.e. it is a Some[A]</code>, and the predicate passed to filter</code> returns true</code> for the wrapped value of type A</code>, the Some</code> instance is returned. If the Option</code> instance is already None</code> or the predicate returns false</code> for the value inside the Some</code>, the result is None</code>: UserRepository.findById(1).filter(_.age > 30) // Some(user), because age is > 30 UserRepository.findById(2).filter(_.age > 30) // None, because age is <= 30 UserRepository.findById(3).filter(_.age > 30) // None, because user is already None </code></pre> For comprehensions</h2> Now that you know that an Option</code> can be treated as a collection and provides map</code>, flatMap</code>, filter</code> and other methods you know from collections, you will probably already suspect that options can be used in for comprehensions. Often, this is the most readable way of working with options, especially if you have to chain a lot of map</code>, flatMap</code> and filter</code> invocations. If it's just a single map</code>, that may often be preferrable, as it is a little less verbose. If we want to get the gender for a single user, we can apply the following for comprehension: for { user <- UserRepository.findById(1) gender <- user.gender } yield gender // results in Some("male") </code></pre> As you may know from working with lists, this is equivalent to nested invocations of flatMap</code>. If the UserRepository</code> already returns None</code> or the Gender</code> is None</code>, the result of the for comprehension is None</code>. For the user in the example, a gender is defined, so it is returned in a Some</code>. If we wanted to retrieve the genders of all users that have specified it, we could iterate all users, and for each of them yield a gender, if it is defined: for { user <- UserRepository.findAll gender <- user.gender } yield gender </code></pre> Since we are effectively flat mapping, the result type is List[String]</code>, and the resulting list is List("male")</code>, because gender</code> is only defined for the first user. Usage in the left side of a generator</h3> Maybe you remember from part three of this series that the left side of a generator in a for comprehension is a pattern</a>. This means that you can also patterns involving options in for comprehensions. We could rewrite the previous example as follows: for { User(_, _, _, _, Some(gender)) <- UserRepository.findAll } yield gender </code></pre> Using a Some</code> pattern in the left side of a generator has the effect of removing all elements from the result collection for which the respective value is None</code>. Chaining options</h2> Options can also be chained, which is a little similar to chaining partial functions. To do this, you call orElse</code> on an Option</code> instance, and pass in another Option</code> instance as a by-name parameter. If the former is None</code>, orElse</code> returns the option passed to it, otherwise it returns the one on which it was called. A good use case for this is finding a resource, when you have several different locations to search for it and an order of preference. In our example, we prefer the resource to be found in the config dir, so we call orElse</code> on it, passing in an alternative option: case class Resource(content: String) val resourceFromConfigDir: Option[Resource] = None val resourceFromClasspath: Option[Resource] = Some(Resource("I was found on the classpath")) val resource = resourceFromConfigDir orElse resourceFromClasspath </code></pre> This is usually a good fit if you want to chain more than just two options – if you simply want to provide a default value in case a given option is absent, the getOrElse</code> method may be a better idea. Summary</h2> In this article, I hope to have given you everything you need to know about the Option</code> type in order to use it for your benefit, to understand other people's Scala code and write more readable, functional code. The most important insight to take away from this post is that there is a very basic idea that is common to lists, sets, maps, options, and, as you will see in a future post, other data types, and that there is a uniform way of using these types, which is both elegant and very powerful. In the following part of this series I am going to deal with idiomatic, functional error handling in Scala. The Neophyte's Guide to Scala Part 4: Pattern Matching Anonymous Functions Unknown — Wed, 12 Dec 2012 00:00:00 +0000 In the previous part of this series</a>, I gave an overview of the various ways in which patterns can be used in Scala, concluding with a brief mention of anonymous functions as another place in which patterns can be put to use. In this post, we are going to take a detailed look at the possibilities opened up by being able to define anonymous functions in this way. If you have participated in the Scala course at Coursera</a>, or have coded in Scala for a while, you will likely have written anonymous functions on a regular basis. For example, given a list of song titles which you want to transform to lower case for your search index, you might make want to define an anonymous function that you pass to the map</code> method, like this: val songTitles = List("The White Hare", "Childe the Hunter", "Take no Rogues") songTitles.map(t => t.toLowerCase) </code></pre> Or, if you like it even shorter, of course, you will probably write the normalize function like this, making use of Scala's placeholder syntax: songTitles.map(_.toLowerCase) </code></pre> So far so good. However, let's see how this syntax performs for a slightly different example: We have a sequence of pairs, each representing a word and its frequency in some text. Our goal is to filter out those pairs whose frequency is below or above a certain threshold, and then only return the remaining words, without their respective frequencies. We need to write a function wordsWithoutOutliers(wordFrequencies: Seq[(String, Int)]): Seq[String]</code>. Our initial solution makes use of the filter</code> and map</code> methods, passing anonymous functions to them using our familiar syntax: val wordFrequencies = ("habitual", 6) :: ("and", 56) :: ("consuetudinary", 2) :: ("additionally", 27) :: ("homely", 5) :: ("society", 13) :: Nil def wordsWithoutOutliers(wordFrequencies: Seq[(String, Int)]): Seq[String] = wordFrequencies.filter(wf => wf._2 > 3 && wf._2 < 25).map(_._1) wordsWithoutOutliers(wordFrequencies) // List("habitual", "homely", "society") </code></pre> This solution has several problems. The first one is only an aesthetic one – accessing the fields of the tuple looks pretty ugly to me. If only we could destructure the pair, we could make this code a little more pleasant and probably also more readable. Thankfully, Scala provides an alternative way of writing anonymous functions: A pattern matching anonymous function is an anonymous function that is defined as a block consisting of a sequence of cases, surrounded as usual by curly braces, but without a match</code> keyword before the block. Let's rewrite our function, making use of this notation: def wordsWithoutOutliers(wordFrequencies: Seq[(String, Int)]): Seq[String] = wordFrequencies.filter { case (_, f) => f > 3 && f < 25 } map { case (w, _) => w } </code></pre> In this example, we have only used a single case in each of our anonymous functions, because we know that this case always matches – we are simply decomposing a data structure whose type we already know at compile time, so nothing can go wrong here. This is a very common way of using pattern matching anonymous functions. If you try to assign these anonymous functions to values, you will see that they have the expected type: val predicate: ((String, Int)) => Boolean = { case (_, f) => f > 3 && f < 25 } val transformFn: ((String, Int)) => String = { case (w, _) => w } </code></pre> Please note that you have to specify the type of the value here, the Scala compiler cannot infer it for pattern matching anonymous functions. Nothing prevents you from defining a more complex sequence of cases, of course. However, if you define an anonymous function this way and want to pass it to some other function, such as the ones in our example, you have to make sure that for all possible inputs, one of your cases matches so that your anonymous function always returns a value. Otherwise, you will risk a MatchError</code> at runtime. Partial functions</h2> Sometimes, however, a function that is only defined for specific input values is exactly what you want. In fact, such a function can help us get rid of another problem that we haven't solved yet with our current implementation of the wordsWithoutOutliers</code> function: We first filter the given sequence and then map the remaining elements. If we can boil this down to a solution that only has to iterate over the given sequence once, this would not only need fewer CPU cycles but would also make our code shorter and, ultimately, more readable. If you browse through Scala's collections API, you will notice a method called collect</code>, which, for a Seq[A]</code>, has the following signature: def collect[B](pf: PartialFunction[A, B]) </code></pre> This method returns a new sequence by applying the given partial function to all of its elements – the partial function both filters and maps the sequence. So what is a partial function? In short, it's a unary function that is known to be defined only for certain input values and that allows clients to check whether it is defined for a specific input value. To this end, the PartialFunction</code> trait provides an isDefinedAt</code> method. As a matter of fact, the PartialFunction[-A, +B]</code> type extends the type (A) => B</code> (which can also be written as Function1[A, B]</code>), and a pattern matching anonymous function is always of type PartialFunction</code>. Due to this inheritance hierarchy, passing a pattern matching anonymous function to a method that expects a Function1</code>, like map</code> or filter</code>, is perfectly fine, as long as that function is defined for all input values, i.e. there is always a matching case. The collect</code> method, however, specifically expects a PartialFunction[A, B]</code> that may not be defined for all input values and knows exactly how to deal with that case. For each element in the sequence, it first checks if the partial function is defined for it by calling isDefinedAt</code> on the partial function. If this returns false</code>, the element is ignored. Otherwise, the result of applying the partial function to the element is added to the result sequence. Let's first define a partial function that we want to use for refactoring our wordsWithoutOutliers</code> function to make use of collect</code>: val pf: PartialFunction[(String, Int), String] = { case (word, freq) if freq > 3 && freq < 25 => word } </code></pre> We added a guard clause to our case, so that this function will not be defined for word/frequency pairs whose frequency is not within the required range. Instead of using the syntax for pattern matching anonymous functions, we could have defined this partial function by explicitly extending the PartialFunction</code> trait: val pf = new PartialFunction[(String, Int), String] { def apply(wordFrequency: (String, Int)) = wordFrequency match { case (word, freq) if freq > 3 && freq < 25 => word } def isDefinedAt(wordFrequency: (String, Int)) = wordFrequency match { case (word, freq) if freq > 3 && freq < 25 => true case _ => false } } </code></pre> Usually, however, you will want to use the much more concise anonymous function syntax. Now, if we passed our partial function to the map</code> method, this would compile just fine, but result in a MatchError</code> at runtime, because our partial function is not defined for all possible input values, thanks to the added guard clause: wordFrequencies.map(pf) // will throw a MatchError </code></pre> However, we can pass this partial function to the collect</code> method, and it will filter and map the sequence as expected: wordFrequencies.collect(pf) // List("habitual", "homely", "society") </code></pre> The result of this is the same as that of our current implementation of wordsWithoutOutliers</code> when passing our dummy wordFrequencies</code> sequence to it. So let's rewrite that function: def wordsWithoutOutliers(wordFrequencies: Seq[(String, Int)]): Seq[String] = wordFrequencies.collect { case (word, freq) if freq > 3 && freq < 25 => word } </code></pre> Partial functions have some other very useful properties. For example, they provide the means to be chained, allowing for a neat functional alternative to the chain of responsibility pattern</a> known from object-oriented programming. This, however, will have to be the subject of a future post in this series, when I am going to address the issue of functional composability. Partial functions are also a crucial element of many Scala libraries and APIs. For example, the way an Akka</a> actor processes messages sent to it is defined in terms of a partial function. Hence, it's quite important to know and understand this concept. Summary</h2> In this part, we examined an alternative way of defining anonymous functions, namely as a sequence of cases, which opens up some nice destructuring possibilities in a rather concise way. Moreover, we delved into the topic of partial functions, demonstrating their usefulness by means of a simple use case. In the next article, I am going to dig deeper into the ever-present Option</code> type, explaining the reasoning behind its existence and how best to make use of it. Please let me know if you have any questions or feedback. Is there any particular topic you would like to see covered in an article? The Neophyte's Guide to Scala Part 3: Patterns Everywhere Unknown — Wed, 05 Dec 2012 00:00:00 +0000 In the first two parts of this series, I spent quite some time explaining what's actually happening when you destructure an instance of a case class in a pattern, and how to write your own extractors, allowing you to destructure any types of objects in any way you desire. Now it is time to get an overview of where patterns can actually be used in your Scala code, because so far you have only seen one of the different possible ways to make use of patterns. Here we go! Pattern matching expressions</h2> One place in which patterns can appear is inside of a pattern matching expression. This way of using patterns should be very familiar to you after attending the Scala course at Coursera and following along in this series. You have some expression e</code>, followed by the match</code> keyword and a block, which can contain any number of cases. A case, in turn, consists of the case</code> keyword followed by a pattern and, optionally, a guard clause on the left side, plus a block on the right side, to be executed if this pattern matches. Here is a simple example, making use of patterns and, in one of the cases, a guard clause: case class Player(name: String, score: Int) def printMessage(player: Player) = player match { case Player(_, score) if score > 100000 => println("Get a job, dude!") case Player(name, _) => println("Hey " + name + ", nice to see you again!") } </code></pre> The printMessage</code> method has a return type of Unit</code>, its sole purpose is to perform a side effect, namely printing a message. It is important to remember that you don't have to use pattern matching as you would use switch statements in languages like Java. What we are using here is called a pattern matching expression for a reason. Their return value is what is returned by the block belonging to the first matched pattern. Usually, it's a good idea to take advantage of this, as it allows you to decouple two things that do not really belong together, making it easier to test your code, too. We could rewrite the example above as follows: def message(player: Player) = player match { case Player(_, score) if score > 100000 => "Get a job, dude!" case Player(name, _) => "Hey " + name + ", nice to see you again!" } def printMessage(player: Player) = println(message(player)) </code></pre> Now, we have a separate message</code> method whose return type is String</code>. This is essentialy a pure function, returning the result of a pattern matching expression. You could also store the result of such a pattern matching expression as a value or assign it to a variable, of course. Patterns in value definitions</h2> Another place in which a pattern can occur in Scala is in the left side of a value definition (and in a variable definition, for that matter, but we want write our Scala code in a functional style, so you won't see a lot of usage of variables in this series). Let's assume we have a method that returns our current player. We will use a dummy implementation that always returns the same player: def currentPlayer(): Player = Player("Daniel", 3500) </code></pre> Your usual value definition looks like this: val player = currentPlayer() doSomethingWithTheName(player.name) </code></pre> If you know Python, you are probably familiar with a feature called sequence unpacking. The fact that you can use any pattern in the left side of a value definition or variable definition lets you write your Scala code in a similar style. We could change our above code and destructure the given current player while assigning it to the left side: val Player(name, _) = currentPlayer() doSomethingWithTheName(name) </code></pre> You can do this with any pattern, but generally, it is a good idea to make sure that your pattern always matches. Otherwise, you will be the witness of an exception at runtime. For instance, the following code is problematic. scores</code> is a method returning a list of scores. In our code below, this method simply returns an empty list to illustrate the problem. def scores: List[Int] = List() val best :: rest = scores println("The score of our champion is " + best) </code></pre> Oops, we've got a MatchError</code>. It seems like our game is not that successful after all, having no scores whatsoever. A safe and very handy way of using patterns in this way is for destructuring case classes whose type you know at compile time. Also, when working with tuples, this makes your code a lot more readable. Let's say we have a function that returns the name of a player and their score as a tuple, not using the Player</code> class we have used so far: def gameResult(): (String, Int) = ("Daniel", 3500) </code></pre> Accessing the fields of a tuple always feels very awkward: val result = gameResult() println(result._1 + ": " + result._2) </code></pre> It's safe to destructure our tuple in the value definition, as we know we are dealing with a Tuple2</code>: val (name, score) = gameResult() println(name + ": " + score) </code></pre> This is much more readable, isn't it? Patterns in for comprehensions</h2> Patterns also have a very valuable place in for comprehensions. For one, a for comprehension can also contain value definitions. And everything you learnt about the usage of patterns in the left side of value definitions holds true for value definitions in for comprehensions. So if we have a collection of results and want to determine the hall of fame, which in our game is simply a collection of the names of players that have trespassed a certain score threshold, we could do that in a very readable way with a for comprehension: def gameResults(): Seq[(String, Int)] = ("Daniel", 3500) :: ("Melissa", 13000) :: ("John", 7000) :: Nil def hallOfFame = for { result <- gameResults() (name, score) = result if (score > 5000) } yield name </code></pre> The result is List("Melissa", "John")</code>, since the first player does not meet the condition of the guard clause. This can be written even more concisely, because in for comprehensions, the left side of a generator is also a pattern. So, instead of first assigning each game result to result</code>, we can directly destructure the result in the left side of the generator: def hallOfFame = for { (name, score) <- gameResults() if (score > 5000) } yield name </code></pre> In this example, the pattern (name, score)</code> always matches, so if it were not for the guard clause, if (score > 5000)</code>, the for comprehension would be equivalent to simply mapping from the tuples to the player names, without filtering anything. It is important to know that patterns in the left side of generators can already be used for filtering purposes – if a pattern on the left side of a generator does not match, the respective element is filtered out. To illustrate, let's say we have a sequence of lists, and we want to return the sizes of all non-empty lists. This means we have to filter out all empty lists and then return the sizes of the ones remaining. Here is one solution: val lists = List(1, 2, 3) :: List.empty :: List(5, 3) :: Nil for { list @ head :: _ <- lists } yield list.size </code></pre> The pattern on the left side of the generator does not match for empty lists. This will not throw a MatchError</code>, but result in any empty list being removed. Hence, we get back List(3, 2)</code>. Patterns and for comprehensions are a very natural and powerful combination, and if you work with Scala for some time, you will see that you'll be using them a lot. Anonymous functions</h2> Finally, patterns can be used for defining anonymous functions. If you have ever used a catch</code> block in order to deal with an exception in Scala, then you have made used of this feature. Pattern matching anonymous functions is a subject that warrants its own blog post, because there is a lot to be said about it. Hence, I will refrain from delving into this usage of patterns in this article, instead leaving you with the promise of dealing with it in the next part of the series. Update: Fixed a mistake in the expected result of the hallOfFame</code> for comprehension. Thanks to Rajiv for pointing it out. The Neophyte's Guide to Scala Part 2: Extracting Sequences Unknown — Wed, 28 Nov 2012 00:00:00 +0000 In the first part of this series, we learned how to implement our own extractors</a> and how these extractors can be used for pattern matching. However, we only discussed extractors that allow you to destructure a given object into a fixed number of parameters. Yet, for certain kinds of data structures, Scala allows you to do pattern matching expecting an arbitrary number of extracted parameters. For example, you can use a pattern that only matches a list of exactly two elements, or a list of exactly three elements: val xs = 3 :: 6 :: 12 :: Nil xs match { case List(a, b) => a * b case List(a, b, c) => a + b + c case _ => 0 } </code></pre> What's more, if you want to match lists the exact length of which you don't care about, you can use a wildcard operator, _*</code>: val xs = 3 :: 6 :: 12 :: 24 :: Nil xs match { case List(a, b, _*) => a * b case _ => 0 } </code></pre> Here, the first pattern matches, binding the first two elements to the variables a</code> and b</code>, while simply ignoring the rest of the list, regardless how many remaining elements there are. Clearly, extractors for these kinds of patterns cannot be implemented with the means I introduced in the first article. We need a way to specify that an extractor takes an object of a certain type and destructures it into a sequence of extracted values, where the length of that sequence is unknown at compile time. Enter unapplySeq</code>, an extractor method that allows for doing exactly that. Let's take a look at one of its possible method signatures: def unapplySeq(object: S): Option[Seq[T]] </code></pre> It expects an object of type S</code> and returns either None</code>, if the object does not match at all, or a sequence of extracted values of type T</code>, wrapped in a Some</code>. Example: Extracting given names</h2> Let's make use of this kind of extractor method in an admittedly contrived example. Let's say that in some piece of our application, we are receiving a person's given name as a String</code>. This string can contain the person's second or third name, if that person has more than one given name. Hence, possible values could be "Daniel"</code>, or "Catherina Johanna"</code>, or "Matthew John Michael"</code>. We want to be able to match against these names, extracting and binding the individual given names. Here is a very simple extractor implementation by means of the unapplySeq</code> method that will allow us to do that: object GivenNames { def unapplySeq(name: String): Option[Seq[String]] = { val names = name.trim.split(" ") if (names.forall(_.isEmpty)) None else Some(names) } } </code></pre> Given a String</code> containing one or more given names, it will extract those as a sequence. If the input name does not contain at least one given name, this extractor will return None</code>, and thus, a pattern in which this extractor is used will not match such a string. We can now put our new extractor to test: def greetWithFirstName(name: String) = name match { case GivenNames(firstName, _*) => "Good morning, " + firstName + "!" case _ => "Welcome! Please make sure to fill in your name!" } </code></pre> This nifty little method returns a greeting for a given name, ignoring everything but the first name. greetWithFirstName("Daniel")</code> will return "Good morning, Daniel!"</code>, while greetWithFirstName("Catherina Johanna")</code> will return "Good morning, Catherina!"</code> Combining fixed and variable parameter extraction</h2> Sometimes, you have certain fixed values to be extracted that you know about at compile time, plus an additional optional sequence of values. Let's assume that in our example, the input name contains the person's complete name, not only the given name. Possible values might be "John Doe"</code> or "Catherina Johanna Peterson"</code>. We want to be able to match against such strings using a pattern that always binds the person's last name to the first variable in the pattern and the first name to the second variable, followed by an arbitrary number of additional given names. This can be achieved by means of a slight modification of our unapplySeq</code> method, using a different method signature: def unapplySeq(object: S): Option[(T1, .., Tn-1, Seq[T])] </code></pre> As you can see, unapplySeq</code> can also return an Option</code> of a TupleN</code>, where the last element of the tuple must be the sequence containing the variable parts of the extracted values. This method signature should be somewhat familiar, as it is similar to one of the possible signatures of the unapply</code> method that I introduced last week. Here is an extractor making use of this: object Names { def unapplySeq(name: String): Option[(String, String, Seq[String])] = { val names = name.trim.split(" ") if (names.size < 2) None else Some((names.last, names.head, names.drop(1).dropRight(1))) } } </code></pre> Have a close look at the return type and the construction of the Some</code>. Our method returns an Option</code> of Tuple3</code>. That tuple is created with Scala's syntax for tuple literals by just putting the three elements – the last name, the first name, and the sequence of additional given names – in a pair of parentheses. If this extractor is used in a pattern, the pattern will only match if at least a first and last name is contained in the given input string. The sequence of additional given names is created by dropping the first and the last element from the sequence of names. We can use this extractor to implement an alternative greeting method: def greet(fullName: String) = fullName match { case Names(lastName, firstName, _*) => "Good morning, " + firstName + " " + lastName + "!" case _ => "Welcome! Please make sure to fill in your name!" } </code></pre> Feel free to play around with this in the REPL or a worksheet. Summary</h2> In this article, we learned how to implement and use extractors that return variable-length sequences of extracted values. Extractors are a pretty powerful mechanism. They can often be re-used in flexible ways and provide a powerful way to extend the kinds of patterns you can match against. We will revisit extractors in a case study towards the end of this series. In the next part, however, I will give an overview of the different ways in which patterns can be applied in Scala code – there is more to it than just the pattern matching you have seen in the examples so far. Update, 24.01.2013: I updated the code example implementing the GivenNames extractor. Thanks to Christophe Bliard for pointing out a mistake in there. The Neophyte's Guide to Scala Part 1: Extractors Unknown — Wed, 21 Nov 2012 00:00:00 +0000 More than 50,000 people signed up for Martin Odersky's course "Functional Programming Principles in Scala"</a> at Coursera. That's a huge number of developers for whom this might have been the first contact with Scala, functional programming, or both. If you are reading this, maybe you are one of them, or maybe you have started to learn Scala by some other means. In any case, if you have started to learn Scala, you are excited to delve deeper into this beautiful language, but it all still feels a little exotic or foggy to you, then the series of articles that is beginning with this one is for you. Even though the Coursera course covered quite a lot of what you need to know about Scala, the given time constraints made it impossible to explain everything in detail. As a result, some Scala features might seem like magic to you if you are new to the language. You are able to use them somehow, but you haven't fully grasped how they work and, more importantly, why they work as they do. In this article and the ones following in the coming weeks, I would like to clear things up and remove those question marks. I will also explain some of the features of the Scala language and library that I had trouble with when I started learning the language, partially because I didn't find any good explanations for them, but instead just stumbled upon them in the wild. Where appropriate, I will also try to give guidance on how to use these features in an idiomatic™ way. Enough of the introductions. Before I begin, keep in mind that, while having attended the Coursera course is not a prerequisite for following this series, having roughly the knowledge of Scala as can be acquired in that course is definitely helpful, and I will sometimes refer to the course. So how does this pattern matching thingie actually work?</h2> In the Coursera course, you came across one very powerful language feature of Scala: Pattern matching</a>. It allows you to decompose a given data structure, binding the values it was constructed from to variables. It's not an idea that is unique to Scala, though. Other prominent languages in which pattern matching plays an important role are Haskell and Erlang, for instance. If you followed the video lectures, you saw that you can decompose various kinds of data structures using pattern matching, among them lists, streams, and any instances of case classes. So is this list of data structures that can be destructured fixed, or can you extend it somehow? And first of all, how does this actually work? Is there some kind of magic involved that allows you to write things like the following? case class User(firstName: String, lastName: String, score: Int) def advance(xs: List[User]) = xs match { case User(_, _, score1) :: User(_, _, score2) :: _ => score1 - score2 case _ => 0 } </code></pre> As it turns out, there isn't. At least not much. The reason why you are able to write the above code (no matter how little sense this particular example makes) is the existence of so-called extractors</a>. In its most widely applied form, an extractor has the opposite role of a constructor: While the latter creates an object from a given list of parameters, an extractor extracts the parameters from which an object passed to it was created. The Scala library contains some predefined extractors, and we will have a look at one of them shortly. Case classes are special because Scala automatically creates a companion object for them: a singleton object that contains not only an apply</code> method for creating new instances of the case class, but also an unapply</code> method – the method that needs to be implemented by an object in order for it to be an extractor. Our first extractor, yay!</h3> There is more than one possible signature for a valid unapply</code> method, but we will start with the ones that are most widely used. Let's pretend that our User</code> class is not a case class after all, but instead a trait, with two classes extending it, and for the moment, it only contains a single field: trait User { def name: String } class FreeUser(val name: String) extends User class PremiumUser(val name: String) extends User </code></pre> We want to implement extractors for the FreeUser</code> and PremiumUser</code> classes in respective companion objects, just as Scala would have done were these case classes. If your extractor is supposed to only extract a single parameter from a given object, the signature of an unapply</code> method looks like this: def unapply(object: S): Option[T] </code></pre> The method expects some object of type S</code> and returns an Option</code> of type T</code>, which is the type of the parameter it extracts. Remember that Option</code> is Scala's safe alternative to the existence of null</code> values. There will be a separate article about it, but for now, it's enough to know that the unapply</code> method returns either Some[T]</code> (if it could successfully extract the parameter from the given object) or None</code>, which means that the parameters could not be extracted, as per the rules determined by the extractor implementation. Here are our extractors: trait User { def name: String } class FreeUser(val name: String) extends User class PremiumUser(val name: String) extends User object FreeUser { def unapply(user: FreeUser): Option[String] = Some(user.name) } object PremiumUser { def unapply(user: PremiumUser): Option[String] = Some(user.name) } </code></pre> We can now use this in the REPL: scala> FreeUser.unapply(new FreeUser("Daniel")) res0: Option[String] = Some(Daniel) </code></pre> But you wouldn't usually call this method directly. Scala calls an extractor's unapply</code> method if the extractor is used as an extractor pattern. If the result of calling unapply</code> is Some[T]</code>, this means that the pattern matches, and the extracted value is bound to the variable declared in the pattern. If it is None</code>, this means that the pattern doesn't match and the next case statement is tested. Let's use our extractors for pattern matching: val user: User = new PremiumUser("Daniel") user match { case FreeUser(name) => "Hello " + name case PremiumUser(name) => "Welcome back, dear " + name } </code></pre> As you will already have noticed, our two extractors never return None</code>. The example shows that this makes more sense than it might seem at first. If you have an object that could be of some type or another, you can check its type and destructure it at the same time. In the example, the FreeUser</code> pattern will not match because it expects an object of a different type than we pass it. Since it wants an object of type FreeUser</code>, not one of type PremiumUser</code>, this extractor is never even called. Hence, the user</code> value is now passed to the unapply</code> method of the PremiumUser</code> companion object, as that extractor is used in the second pattern. This pattern will match, and the returned value is bound to the name</code> parameter. Later in this article, we will see an example of an extractor that does not always return Some[T]</code>. Extracting several values</h3> Now, let's assume that our classes against which we want to match have some more fields: trait User { def name: String def score: Int } class FreeUser(val name: String, val score: Int, val upgradeProbability: Double) extends User class PremiumUser(val name: String, val score: Int) extends User </code></pre> If an extractor pattern is supposed to decompose a given data structure into more than one parameter, the signature of the extractor's unapply</code> method looks like this: def unapply(object: S): Option[(T1, ..., Tn)] </code></pre> The method expects some object of type S</code> and returns an Option</code> of type TupleN</code>, where N</code> is the number of parameters to extract. Let's adapt our extractors to the modified classes: trait User { def name: String def score: Int } class FreeUser(val name: String, val score: Int, val upgradeProbability: Double) extends User class PremiumUser(val name: String, val score: Int) extends User object FreeUser { def unapply(user: FreeUser): Option[(String, Int, Double)] = Some((user.name, user.score, user.upgradeProbability)) } object PremiumUser { def unapply(user: PremiumUser): Option[(String, Int)] = Some((user.name, user.score)) } </code></pre> We can now use this extractor for pattern matching, just like we did with the previous version: val user: User = new FreeUser("Daniel", 3000, 0.7d) user match { case FreeUser(name, _, p) => if (p > 0.75) name + ", what can we do for you today?" else "Hello " + name case PremiumUser(name, _) => "Welcome back, dear " + name } </code></pre> A Boolean extractor</h3> Sometimes, you don't really have the need to extract parameters from a data structure against which you want to match – instead, you just want to do a simple boolean check. In this case, the third and last of the available unapply</code> method signatures comes in handy, which expects a value of type S</code> and returns a Boolean</code>: def unapply(object: S): Boolean </code></pre> Used in a pattern, the pattern will match if the extractor returns true</code>. Otherwise the next case, if available, is tried. In the previous example, we had some logic that checks whether a free user is likely to be susceptible to being persuaded to upgrade their account. Let's place this logic in its own boolean extractor: object premiumCandidate { def unapply(user: FreeUser): Boolean = user.upgradeProbability > 0.75 } </code></pre> As you can see here, it is not necessary for an extractor to reside in the companion object of the class for which it is applicable. Using such a boolean extractor is as simple as this: val user: User = new FreeUser("Daniel", 2500, 0.8d) user match { case freeUser @ premiumCandidate() => initiateSpamProgram(freeUser) case _ => sendRegularNewsletter(user) } </code></pre> This example shows that a boolean extractor is used by just passing it an empty parameter list, which makes sense because it doesn't really extract any parameters to be bound to variables. There is one other peculiarity in this example: I am pretending that our fictional initiateSpamProgram</code> function expects an instance of FreeUser</code> because premium users are never to be spammed. Our pattern matching is against any type of User</code>, though, so I cannot pass user</code> to the initiateSpamProgram</code> function – not without ugly type casting anyway. Luckily, Scala's pattern matching allows to bind the value that is matched to a variable, too, using the type that the used extractor expects. This is done using the @</code> operator. Since our premiumCandidate</code> extractor expects an instance of FreeUser</code>, we have therefore bound the matched value to a variable freeUser</code> of type FreeUser</code>. Personally, I haven't used boolean extractors that much, but it's good to know they exist, as sooner or later you will probably find yourself in a situation where they come in handy. Infix operation patterns</h3> If you followed the Scala course at Coursera, you learned that you can destructure lists and streams in a way that is akin to one of the ways you can create them, using the cons operator, ::</code> or #::</code>, respectively: val xs = 58 #:: 43 #:: 93 #:: Stream.empty xs match { case first #:: second #:: _ => first - second case _ => -1 } </code></pre> Maybe you have wondered why that is possible. The answer is that as an alternative to the extractor pattern notation we have seen so far, Scala also allows extractors to be used in an infix notation. So, instead of writing e(p1, p2)</code>, where e</code> is the extractor and p1</code> and p2</code> are the parameters to be extracted from a given data structure, it's always possible to write p1 e p2</code>. Hence, the infix operation pattern head #:: tail</code> could also be written as #::(head, tail)</code>, and our PremiumUser</code> extractor could also be used in a pattern that reads name PremiumUser score</code>. However, this is not something you would do in practice. Usage of infix operation patterns is only recommended for extractors that indeed are supposed to read like operators, which is true for the cons operators of List</code> and Stream</code>, but certainly not for our PremiumUser</code> extractor. A closer look at the Stream extractor</h3> Even though there is nothing special about how the #::</code> extractor can be used in pattern matching, let's take a look at it, to better understand what is going on in our pattern matching code above. Also, this is a good example of an extractor that, depending on the state of the passed in data structure, may return None</code> and thus not match. Here is the complete extractor, taken from the sources of Scala 2.9.2: object #:: { def unapply[A](xs: Stream[A]): Option[(A, Stream[A])] = if (xs.isEmpty) None else Some((xs.head, xs.tail)) } </code></pre> If the given Stream</code> instance is empty, it just returns None</code>. Thus, case head #:: tail</code> will not match for an empty stream. Otherwise, a Tuple2</code> is returned, the first element of which is the head of the stream, while the second element of the tuple is the tail, which is itself a Stream</code> again. Hence, case head #:: tail</code> will match for a stream of one or more elements. If it has only one element, tail</code> will be bound to the empty stream. To understand how this extractor works for our pattern matching example, let's rewrite that example, going from infix operation patterns to the usual extractor pattern notation: val xs = 58 #:: 43 #:: 93 #:: Stream.empty xs match { case #::(first, #::(second, _)) => first - second case _ => -1 } </code></pre> First, the extractor is called for the intitial stream xs</code> that is passed to the pattern matching block. The extractor returns Some((xs.head, xs.tail))</code>, so first</code> is bound to 58</code>, while the tail of xs</code> is passed to the extractor again, which is used again inside of the first one. Again, it returns the head and and tail as a Tuple2</code> wrapped in a Some</code>, so that second</code> is bound to the value 43</code>, while the tail is bound to the wildcard _</code> and thus thrown away. Using extractors</h3> So when and how should you actually make use of custom extractors, especially considering that you can get some useful extractors for free if you make use of case classes? While some people point out that using case classes and pattern matching against them breaks encapsulation, coupling the way you match against data with its concrete representation, this criticism usually stems from an object-oriented point of view. It's a good idea, if you want to do functional programming in Scala, to use case classes as algebraic data types (ADTs)</a> that contain pure data and no behaviour whatsoever. Usually, implementing your own extractors is only necessary if you want to extract something from a type you have no control over, or if you need additional ways of pattern matching against certain data. For example, a common usage of extractors is to extract meaningful values from some string. As an exercise, think about how you would implement and use a URLExtractor</code> that takes String</code> representations of URLs. Conclusion</h3> In this first part of the series, we have examined extractors, the workhorse behind pattern matching in Scala. You have learned how to implement your own extractors and how the implementation of an extractor relates to its usage in a pattern. We haven't covered all there is to say about extractors, because this article is already long enough as it is. In the next part of this series, I am going to revisit extractors, covering how to implement them if you you want to bind a variable number of extracted parameters in a pattern. Please do let me know if this article was helpful to you or if something is not clear to you. Functional Programming Principles in scala: A Wrap-up Unknown — Sun, 04 Nov 2012 00:00:00 +0000 For the past seven weeks, I have been participating in the online class "Functional Programming Principles in Scala"</a> provided by Martin Odersky, the EPFL, and Coursera. This has not only reignited my more or less dormant passion for the Scala language, but has also been tremendously fun and instructive at the same time. Having just finished the final assignment, let me tell you why. First of all, of course, the chance of attending a class on a programming language and paradigm that is conducted by the very designer of that language, a distinguished expert in their field, is all kinds of amazing in itself. It's great to live in a time in which attending such a class online and for free is possible for thousands of interested students who would otherwise not have had the opportunity at all, or who otherwise have to make do with the often times mediocre lecturers at their own university. This was my first ever online class, and I must say I didn't really miss the physical presence of a meatspace university class at all. In each of the seven weeks, the programme consisted of numerous video lectures, sessions of five to 25 minutes, altogether between one and two hours a week. These video lectures were accompanied by weekly (or sometimes biweekly) programming assignments in which students had to transfer and apply what they learned to a new problem. Usually, this meant to implement several pre-defined functions, with the groundwork already laid for you. During the first couple of weeks, the course was focused a lot on the basic principles of functional programming, while keeping the amount of Scala syntax and library code used to an absolute minimum. This changed in the second half, when a bunch of important Scala concepts were introduced, like for comprehensions and pattern matching. I found the course to be challenging at just the right level, and the assignments had just about the right mixture of highly abstract problems and ones with interesting practical relevance, like analyzing a feed of tweets or implementing a solver for an interesting Flash game. I have already had my share of experience with functional programming in general, and Scala in particular, so some small parts of assignments were just a finger exercise, and I assume I haven't had to think about getting my Scala code syntactically right or more idiomatic as much as students new to the language. Thankfully, such experience didn't make solving the assignments a cakewalk altogether. I noticed that I wasn't used to thinking in the abstract any more as much as I had thought, so having to reflect on these problems to come up with a solution was not always easy and took a while, but was also one of most important profits I drew from the class. I literally felt like a student again, back in the days, and enjoyed this very much. The Coursera Scala course is highly recommended if you want to delve into the world of functional programming or you want to learn Scala. I don't know any other up-and-coming language with such high-quality, educational and fun learning material, so I can imagine Scala gaining some traction from it. Have you attended the course, too? If so, tell me about your experience! Bonn Agile August 2012: CQRS-based Architectures Unknown — Sun, 26 Aug 2012 00:00:00 +0000 Last Monday, at the August meetup of Bonn Agile</a>, I did a talk about architectures based on the CQRS pattern. I presented various opportunities enabled by this pattern, specifically asymmetrical scaling, separating the models for reading and writing, eventual consistency, and event sourcing. I also put up some very crude demo code in Scala</a> to show how one might implement a command handler and a write model handler, and how easy it is to build the current state of an aggregate from the events stored for it. Although the pattern and the architectural decisions it enables haven't made their way into the mainstream yet, a couple of people had at least heard of them. Since I was speaking to an audience of agilists, I tried to bring two questions into the focus of the discussion: How can such an architecture help us be more agile?</li> When does it rather hinder us from being agile?</li> </ol> I might put my thoughts on these topics into a blog post if there is sufficient interest in that. What I really found interesting about the evolving discussion at the end of my talk was the fact that some people have built applications in which events are actually an almost natural part of the domain model. Apparently, they implemented some of the ideas behind event sourcing, but without really knowing about it, so events did not play the first class role that they have in an event-sourced architecture. This suggests that currently, CQRS-based architectures, and especially event sourcing, are indeed under-used. Link: Slides</a> Book Review: Team Geek Unknown — Mon, 13 Aug 2012 00:00:00 +0000 Many books have been published about the technical aspects of software development, and, granted, a lot has been written about how to successfully manage IT projects and about why and how to apply certain agile methodologies. Yet, you hardly find any literature dealing with the human factors of software development, neither does it play a significant role on any of the computer science curricula that I know of. This is is especially strange because, based on my own experience and my conversations with other developers, human problems among team members as well as with other stakeholders, be it managers or users, seem to be very common and can have more serious consequences for the success of a project than is apparent at first glance. Behold, though, fellow developers, you no longer need to shed lonely tears! "Team Geek"</a> is a completely non-technical book by Google's Brian Fitzpatrick and Ben Collins-Sussman that, while also containing some crucial lessons for management people, is targeted explicitly at software developers. As a developer, you are nearly always in the position of being a member of a team, and all of a sudden, sooner than you know, you may even find yourself leading a team, even if it's only temporary. This book teaches you a couple of important things: How to be a great team member</li> How to be a good team leader</li> How to recognize and deal with poisonous team members of various categories</li> How to deal with poisonous people from outside your team</li> How to exert influence in your company or other organization</li> </ul> In each of the chapters of the book, Fitzpatrick and Collins-Sussman describe certain problems that are likely to occur in software development teams or projects sooner or later, often anecdotally, sometimes also as a collection of anti-patterns. Most of the problems can occur both in a company environment as well as in medium-sized or big open source projects. However, some of them are very specific to open-source projects and stem directly from the authors' experience with the Subversion project. All this is written in a very entertaining and engaging way, and unless you are an extremely lucky person or someone who is only about to start their career as a software developer, you will often nod in agreement, hopefully smiling, thinking of specific instances of the described problems that you have experienced first-hand. The text is far from being dry, and so are the humourous illustrations that clarify the authors' respective messages. While the price of the book is worth paying for the entertainment value of the described problems alone, what makes this book really valuable, of course, are the proposed ways of dealing with each of the described problems, which are based on many years of experience by the two authors. In the end, Fitzpatrick and Collins-Sussman always come back to the trinity of humility, respect, and trust (HRT) as the core of nearly all their problem solutions, and to building and keeping intact a specific team culture. Of course, there are more specific recommendations than that, but they are mostly based on one or more of those three principles. While some of the solutions are rather obvious, and some very few I do not wholeheartedly agree with, all in all I found this book to be a real treasure, giving me new insight, new perspectives, and also more confidence in dealing with the addressed human problems of software development. An especially inspirational chapter was the one on "The Art of Organizational Manipulation". It demonstrates that fighting the ways an organization works, no matter how pointless its rules are, only brings you so far. Instead, understand the rules, play by them, and the chances are usually much better that you will be able to change it to the better, thereby essentially hacking the organization. TL;DR: Being a great developer also comprises being a great team player. "Team Geek" teaches you exactly that and should be required reading for prospective and experienced software developers alike. Even if you don't have any of the covered problems in your current team, you will want to be prepared for it. Team Geek: A Software Developer's Guide to Working Well with Others By Brian W. Fitzpatrick and Ben Collins-Sussman Publisher: O'Reilly Released: July 2012 Pages: 194 Link: "Team Geek" at the O'Reilly shop</a> Advance Hackathon 2012: A Retrospective Unknown — Tue, 01 May 2012 00:00:00 +0000 Last weekend, I attended the Advance Hackathon 2012, which was hosted at the Coworking Space Gasmotorenfabrik in Cologne. Even though I had been intending to participate at such an event for a long time, this was indeed my first hackathon. There were 80-90 people working on 20 projects from Saturday, 10am to Sunday 4pm. All of the projects were very interesting, and most of them managed to make a lot of progress in the course of the weekend. The project I was involved in is called lunjure</a>. Our goal was to create a lunch planner as a single-page HTML5 web application that is written in pure Clojure and ClojureScript, making use of web sockets for most of the communication between the client and the server. While we didn't finish this project in the short time of the hackathon, we managed to implement all of the core functionality and will probably continue working on it. If nothing else, it is certainly a nice demo project showcasing the usage of the aforementioned technologies. The only minor complaint I have to make is this: Everybody who had submitted a project idea was to pitch it on Saturday morning in order to find interested developers. However, my impression was that least 80 percent of the participants had already formed teams before the hackathon, and that for the most part, people who already knew each other worked together in a team. This is okay, of course (and maybe my impression is completely wrong anyway), but it made the whole pitching process somewhat redundant. Nevertheless, it was an amazing weekend, and all the people involved in the organization did an outstanding and highly professional job, and what little need for improvisation arose once in a while was handled very gracefully. Also, the whole atmosphere was very friendly and cooperative. I would definitely like to see an Advance Hackathon 2013. Mission Statement Unknown — Fri, 01 Jul 2011 00:00:00 +0000 I have been intending to start blogging again for a long time. Now I have finally managed to put this into practice. I am a maker of software, and my plan is to primarily blog about my experience as well as opinions on various aspects of the software development process. But from time to time, I will certainly put on the user hat and write about the intricacies of human computer interaction from that very different perspective. Why do I do all that? I am humble enough to be convinced that the majority of the world will not care about what I write here, will not even read it. But I like to write, it helps me to get a clearer understanding by forcing me to organize my thoughts. Also, I have found other people's blogs to be not only interesting, but also very useful in certain situations, and I do think that my contributions might be helpful to the one or the other person, too, or at least entertaining. For those interested, I decided to stay away from hosted blogging systems and instead make use of Jekyll</a>, a very nice, blog-aware static-site generator written in Ruby.

`Links</h2> kontextfrei project website</a></li> kontextfrei GitHub repo</a></li> kontextfrei wordcount example</a></li> </ul>`

Higher-order functions and partial functions</h2>
I won't go into detail here, but now that you know more about how you can compose or reuse functions by means of higher-order functions, you might want to have a look at partial functions again.</p>

Daniel Westheide on making software

Pair Programming Considered Unnecessary: The Costs of Productive Solitude

Developing with AI through the Cognitive Lens

Scala from Scratch: Understanding

Scala from Scratch: Exploration

Goodbye Octopress, hello Zola!

The Complexity Trap: Think Before You Leap

Introducing kontextfrei

The Empathic Programmer

When Option is not good enough

Put your writes where your master is: Compile-time restriction of Slick effect types

The Neophyte's Guide to Scala Part 16: Where to Go from Here

The Neophyte's Guide to Scala Part 15: Dealing with Failure in Actor Systems

The Neophyte's Guide to Scala Part 14: The Actor approach to concurrency

The Neophyte's Guide to Scala Part 13: Path-dependent types

The Neophyte's Guide to Scala Part 12: Type classes

The Neophyte's Guide to Scala Part 11: Currying and partially applied functions

The Neophyte's Guide to Scala Part 10: Staying DRY with higher-order functions

The Neophyte's Guide to Scala Part 9: Promises and Futures in practice

The Neophyte's Guide to Scala Part 8: Welcome to the Future

The Neophyte's Guide to Scala Part 7: The Either type

The Neophyte's Guide to Scala

The Neophyte's Guide to Scala Part 6: Error handling with Try

The Neophyte's Guide to Scala Part 5: The Option type

The Neophyte's Guide to Scala Part 4: Pattern Matching Anonymous Functions

The Neophyte's Guide to Scala Part 3: Patterns Everywhere

The Neophyte's Guide to Scala Part 2: Extracting Sequences

The Neophyte's Guide to Scala Part 1: Extractors

Functional Programming Principles in scala: A Wrap-up

Bonn Agile August 2012: CQRS-based Architectures

Book Review: Team Geek

Advance Hackathon 2012: A Retrospective

Mission Statement

Daniel Westheide on making software

Pair Programming Considered Unnecessary: The Costs of Productive Solitude

Developing with AI through the Cognitive Lens

Scala from Scratch: Understanding

Scala from Scratch: Exploration

Goodbye Octopress, hello Zola!

The Complexity Trap: Think Before You Leap

Introducing kontextfrei

Links</h2> kontextfrei project website</a></li> kontextfrei GitHub repo</a></li> kontextfrei wordcount example</a></li> </ul>

The Empathic Programmer

When Option is not good enough

Put your writes where your master is: Compile-time restriction of Slick effect types

Evaluating a database action</h3> A DBIOAction</code> merely describes what you want to do – in order to actually have it executed, you need to run it like this:</p>

The Neophyte's Guide to Scala Part 16: Where to Go from Here

The Neophyte's Guide to Scala Part 15: Dealing with Failure in Actor Systems

The Neophyte's Guide to Scala Part 14: The Actor approach to concurrency

The Neophyte's Guide to Scala Part 13: Path-dependent types

The Neophyte's Guide to Scala Part 12: Type classes

The Neophyte's Guide to Scala Part 11: Currying and partially applied functions

The Neophyte's Guide to Scala Part 10: Staying DRY with higher-order functions

On higher-order functions</h2> A higher-order function, as opposed to a first-order function, can have one of three forms:</p> One or more of its parameters is a function, and it returns some value.</li> It returns a function, but none of its parameters is a function.</li>

Higher-order functions and partial functions</h2> I won't go into detail here, but now that you know more about how you can compose or reuse functions by means of higher-order functions, you might want to have a look at partial functions again.</p>

The Neophyte's Guide to Scala Part 9: Promises and Futures in practice

The Neophyte's Guide to Scala Part 8: Welcome to the Future

The Neophyte's Guide to Scala Part 7: The Either type

`Links</h2> kontextfrei project website</a></li> kontextfrei GitHub repo</a></li> kontextfrei wordcount example</a></li> </ul>`

Higher-order functions and partial functions</h2>
I won't go into detail here, but now that you know more about how you can compose or reuse functions by means of higher-order functions, you might want to have a look at partial functions again.</p>