• Spark SQL = a new SQL engine for Spark
• compatibility with Shark/Hive
• the new Hive on Spark effort – https://issues.apache.org/jira/browse/HIVE-7292
• ending development of Shark – all hands on Spark SQL = moving all Databricks’ development resources to Spark SQL
• a superset of Shark’s features for existing Shark users
• designed from ground-up to leverage the power of Spark
• upgrade path from Shark 0.9 to Spark SQL to help migrate people from Hive to Spark
• Hive (on MapReduce) was the only choice for SQL on Hadoop 3 years ago
• Hive compiled SQL into scalable MapReduce jobs and could work with a variety of formats (through its SerDes)
  • performance not satisfactory though = deficiencies that made Hive slow were fundamental
  • no interactive queries
• Shark became one of the first interactive SQL on Hadoop systems
  • the only one built on top of a general runtime (Spark)
• Shark built on the Hive codebase and achieved performance improvements by swapping out the physical execution engine part of Hive
• Shark inherited a large, complicated code base from Hive that made it hard to optimize and maintain.
  • constrained by the legacy that was designed for MapReduce.
• Spark SQL supports all existing Hive data formats, user-defined functions (UDF), and the Hive megastore.
• Spark SQL becomes the narrow-waist for manipulating (semi-) structured data as well as ingesting data from sources that provide schema, such as JSON, Parquet, Hive, or EDWs (Enterprise Data Warehourses)
• unifies SQL and sophisticated analysis, allowing users to mix and match SQL and more imperative programming APIs for advanced analytics.
• Spark SQL proposes a novel, elegant way of building query planners
  • easy to add new optimisations
• Spark SQL is becoming the standard for SQL on Spark
• Spark as an alternative execution engine to Hive = to migrate the execution to Spark
• Spark SQL will be the future of not only SQL, but also structured data processing on Spark.

BTW, the examples in Spark SQL Programming Guide are working like a charm. They’re so easy to run and comprehend that I couldn’t understand at first what the fuss is about.

The following book review is about the Early Release (Raw & Unedited) version of the book Learning Spark: Lightning-Fast Big Data Analytics from O’Reilly. I was reading the book in the mobi format on Kindle Paperwhite. It was a review copy from O’Reilly.

tl;dr Read the book if you’re curious about Apache Spark and are on the lookout for a more systematic approach to learn its features. Even at this writing stage can the book be very useful for newcomers to the field as well as people more experienced.

I have never worked commercially or as a hobby with any project that would claim to be some sort of Big Data solution, and Apache Spark was no exception. The reason for more interest in Spark was due to the Scala language the project is developed in (reviewing the source code’s changes) and at some point in the project’s history they were more convinced to use sbt that I hoped to have helped myself and them to comprehend. Later on, it turned out that not only could I learn Scala and sbt, but also Akka and the concept of data stream mining with Spark Streaming (I’ve also been considering Storm, but it’s in Clojure that I once left aside for Scala). Spark shell (based on Scala REPL) made the learning so much easier that I was sure I’m with Spark for longer. I also happened to have developed an Activator template to let others get started with Spark Streaming – Spark Streaming with Scala and Akka (spark-streaming-scala-akka).

With a few weeks of learning Spark under my belt I needed a book to overcome initial hurdles and reach higher level of confidence in applying Spark where it’d fit well. I simply needed a mentor who’d guide me through “what, when, how” of Spark and the book did that far beyond my expectations.

There are already 5 chapters of quality that I didn’t expect from a book in an early release – I must admit that the content’s polished and after having read the chapters I need more of it. The book’s written by people who are the committers of the project and their writing style is very engaging with enough theory and code samples in Java, Scala and Python. There are many use cases for which Spark is a valid software offering and I’m in no way to imagine how my Spark skills will have grown up after the other chapters yet to come like Advanced Programming with RDDs, Spark Architecture and Deploying Spark. It’s undoubtedly going to be a painful experience waiting for them to show up.

If the 5 chapters (out of 13 planned) were any indication of what the book’s going to look like in the final version, I’m fully confident of its success – it’s going to be the bestseller in the area of Big Data Analytics. No programming language – out of Java, Scala or Python – is favoured. As the authors pointed out in the initial pages, they’re going to show examples of using Spark in the three programming languages and they’re doing it for each and every use case. That’s also one of the selling points of Spark that the book highlights very well – the samples are simple to comprehend, almost no-brainers, and can easily fit a page, even in all three languages. Without Spark the samples would not have been so easy to implement and would’ve required much more from the implementer, be it an engineer or data analyst. The book demonstrates it well.

As we’re at it, the two job titles – a software engineer and a data analyst – are the people the book targets. It’s just this book that has helped me to notice the difference between them and how Spark blends their needs into a single software offering. After the 5 chapters Spark appears so simple that I doubt there’s anything that can surprise me that would not be a bug or an intended (yet surprising) feature. The book has helped me to build confidence in understanding the benefits of using Spark in my project and I’m really looking forward to reading the remaining chapters. I’m hoping that the authors and the publisher won’t let me wait too long.

Thanks Andriy Andrunevchyn and Andriy Pashkevych for the fabulous JDay 2014 conference in Lviv, Ukraine. The city, the venue, the people and the temperature outside made the conference such a fantastic event. Thanks a bunch! I owe you a lot for having been invited to speak on the conference.

My talk “#Be #social #FTW aka Your #Professional #Development with #StackOverflow #GitHub #twitter et al” went pretty well. I was speaking about my own experience using social development tools like StackOverflow, GitHub, twitter et al. to develop professional skills I care about the most – being in touch with people (I might never ever meet in my life in person) about technical stuff I like spending my time on these days.

Honestly, what could’ve gone wrong since I was speaking about my own reasons for the tools I picked myself and how it’s going?! It’s going well so it was hardly possible to spoil the talk, really (except for my English, speaking skills and the jokes, perhaps). There could’ve been too many I‘s and me‘s, that could not have been well received, but since I was speaking about my own experience I couldn’t have done it otherwise. I do hope it was not an issue whatsoever.

Watch the slides #Be #social #FTW aka Your #Professional #Development with #StackOverflow #GitHub #twitter et al at SlideShare.

I’m tempted to think I managed to spur some activity in using these tools as a means of personal professional development (my followers list grew up with additional 15 people after the talk that amounts to ca 10% of all the attendees so I think I can claim some success).

When I asked a few questions about how many people are using StackOverflow, GitHub and twitter the number of hands were matching the order I mentioned the tools. There was one exception with coursera that I only mentioned for a moment that turned out as much widely recognised and used as GitHub if not StackOverflow. It was nice to have talked to people who want to learn from the speaker and very actively participated in conversations. I once or twice asked questions about who doesn’t know a thing and there were few brave souls who raised their hands to tell me that I need to explain what I thought would not have been needed at all.

Lviv is a nice city with lots of nice-looking people on the streets. It was surely more crowded during the weekend than in Warsaw, but it could’ve also been that I don’t spend much time walking in the centre of Warsaw so it might be just me who could be surprised. I was indeed. The sun was shining so warmly. A t-shirt was all you needed to have a walk any time, even around the midnight – the night of the conference – after the after-party.

I was pleasantly surprised twice about the people living in Lviv when I was asked (in a way I’d not be afraid calling a request) to speak Polish since I’m Polish. The first time was at the customs after arriving to Lviv and another when I needed some guidance from a 25ish-year-old man on how to get to the conference venue. It turned out that the young man had his grandparents Polish. As a matter of fact, his Polish language was far better at times than mine since he was using words I simply forgot they existed. I even thought that the Polish language in Lviv might’ve been a bit purer since the language could’ve been specially treated for it not to be forgotten and couldn’t have had that much changed by new words coming in from different parts of Poland and abroad. After all, it’s not very long – about 70 years – when Lviv was Polish (amazingly, the history of Lviv during and after the World War II resembles what’s going on with the eastern part of Ukraine today, and how the Europe and US leaders agreed on Soviet re-occupation).

Thanks Andriy’s for a lovely conference in Lviv, Ukraine. I very much enjoyed speaking at the conference and wish myself I could be invited again the next year. I’m looking forward to it. See you!

The slide deck was exactly what I needed – fully packed with enough in-depth information on Angular components: controllers, services, directives, scopes, filters, and modules.

REMEMBER Never ever manipulate the DOM from the controller

Directives

• Declarative HTML
• Use the data available in the scope
• Create DOM on the fly
• ng-repeat
• ng-switch

(bidirectional) Data binding

• connects models to views and vice versa
• ngModel – new bindings

filters

• {{ expression | filter }}
• uppercase filter

partial views

• history management
• bind views to routes
• ng-view

Components of Angular application

• module
• controllers
• services
• filters
• directives

$routeProvider – configures routes in your module

Controllers

• data provider for views
• angularModule.controller

Scope

• link between controllers and the views they’re bound to
• controller adds data and functions so they’re visible in views
• child scopes
• directives can manage their own scope

Scope $watch

• observe changes on the data of the scope in the controller

Scope $broadcast and $on

• access to system of events and listeners on the scope
• $broadcast to fire an event
• $on to receive an event
• Use $rootScope to send events to the whole application

Services

• utility components of your application
• controllers manage data for views
• Use services to separate code to be reused in controllers
• angularModule.factory
• services can be injected into other Angular components – services, filters, controllers, directives

Directives

• today’s HTML components
• angularModule.directive
• <div nameOfTheDirective></div>

REMEMBER Never ever manipulate the DOM from the controller

Here are the slides. They’re worth the time!

Today I watched the slides Advanced Tips & Tricks for using Angular JS and here are the main points I wrote down to remember:

• There’s more than controllers in AngularJS. Use services and directives early and often.
• Directives:
  • make markup declarative
  • support templates
  • promote re-use
  • prevent lots of JavaScripts
• slide 42: avoid console.log. Use $log instead.
• AngularJS Batarang
• UI Bootstrap – Bootstrap components written in pure AngularJS by the AngularUI Team
• slide 53: Controllers should never refer to DOM elements
• slide 53: Controllers should use services for business logic
• $scope is glue between controller and model (accessed through services)
• Services are singletons
• Use watch collection to calculate required elements at the first invocation of a controller as follows:
  $scope.$watchCollection(‘items’, function(newItems) {
    for(var i = 0; i < newItems.length; i++) {
      newItems[i].price = getPrice(newItems[i].id);
    }
    $scope.items = newItems;
  });

What made the day even more worthwhile to remember was the decision of Citi and mine to terminate the employment contract, effective today. After a year and two months our vision on what and how to deliver no longer converged and so there was no point pretending otherwise. It just didn’t work out.

Below is the farewell letter I sent to the team to thank them for their support. I now consider myself unemployed and am looking for greener pastures elsewhere.

Ping me at jacek@japila.pl when you’ve got a job contract for a person who often voices his thoughts, can build and lead technical teams, develops code, is passionate about delivering high-quality software, can be a liaison between technical and business teams, is open-minded and moreover always keeps pushing own knowledge boundaries. I’m open for crazy ideas and can offer some myself, too.

From: Jacek Laskowski
Subject: Farewell
Date: August 13th, 2014 5:00 pm

As you may’ve heard I’m leaving the company to seek greener pastures elsewhere. This is to say BIG THANK YOU for your patience and bearing with my crazy ideas. There were way too many to keep sane. I’m sure of it.

I joined Citi a year ago and worked directly or indirectly with almost all of you that makes my leaving so painful. Anyway, life goes on and so our careers. Thanks for your support, patience, willingness, openness and so many opportunities that offered venues to enormously improve my technical and interpersonal skills. You’ve been stretching my skills in all possible directions. I do consider no day in Citi a waste of my time and will miss y’a, folks!

Get in touch with me on LinkedIn (http://pl.linkedin.com/in/jaceklaskowski), twitter (https://twitter.com/jaceklaskowski), blogs, stackoverflow (http://stackoverflow.com/users/1305344), facebook (https://www.facebook.com/laskowski.jacek), you name it. My email’s jacek@japila.pl and should you have a work you don’t want to do I’ll be waiting for your email to offer my support! I’ll be round the corner. Thanks!

Best wishes for the future! See you later.

As the result of the Scala hackathon we created a change that’s the following snippet (that ultimately may become part of sbt.EvaluateConfigurations object):

It’s surely going to take more time to polish the changes until it’s merged to sbt however the meetup showed us how much we can hack together when the topic under discussion is well-presented and the meeting led by highly-skilled developers. Kudos to Grzegorz for leading the meetup! I can’t wait to the next Scala shackathons.

This time we were discussing the (in)famous blank lines: why, what do they do and can we make them a happy memory of the past? As the result we’re going to host a mini Scala hackathon in two weeks in which we’re going to hack the changes needed for the issue to go away (or at the very least an action plan to address the issue). That’s going to be our first Scala hack-a-thon in the group that should give us experience in hosting larger ones for other challenging issues in Scala tools. We’re not going to be limited to sbt only (yet that’s where we’ve got quite impressive bunch of people who know about sbt quite enough – @gkossakowski from Typesafe, @lpiepiora, @jozwikandrzej and @jaceklaskowski). I know about people’s wishes to hack the other well-known tools like Akka, Spray, Play to name a few.

I’m sure I’m going to say what the Scala community in Warszawa, Poland is saying – with Grzegorz we’re going to make the change in Scala! And it’s going to be soon. Stay tuned!

Thanks @javeo_eu for offering us the space for the meetups, Grzegorz for the efforts to shorten the time to dig deeper into Scala, and the Scala community in Poland for the environment to have such ideas flourish! I’m deeply indebted to you for being part of the group. Thanks!

See you in two weeks during the mini-Scala-thon. That’s gonna be hacking fun! Follow us on @WarszawScaLa twitter to get notified about our meetups or join the WarszawScaLa User Group meetup.

I listened to a talk about java.time by Tomasz Nurkiewicz, JSR 356 Java API for WebSocket by Andrzej Golawski, the latest changes in Map by Michal Lewandowski, Java 8 on Android by Krzysztof Siejkowski, java.util.stream by Pawel Szklarz, Optional by Piotr Trzpil, HashMap’s performance improvements in Java 8 by Tomasz Nurkiewicz and JSR 352 Batch Applications by Krzysztof Miksa.

I’m deeply sorry for asking so many questions, but the topics were so interesting that I couldn’t resist learning more.

The number of the flash-talks would attract many hard-core, full-blown Java conferences, but it would not be able to attend all of them when they’re 30- or 45-minute presentations given time constraints. With 5 minutes each I’d an excellent introduction to the many latest additions in Java 8 and Java EE 7. That’s what I liked the most about the meetup.

I enjoyed hanging out with people during and after the meetup. There were many people from Java EE 7, Java 8, Android and Scala camps. Many people share their interests between the areas and it’s no surprise to have seen so many people in the room as the topics were very well selected to attract attention. Being able to listen to the superb speaker line-up talking about the many interesting things in Java space is rarely possible. And all in my town! I could simply not miss the meetup!

After the meetup, I hung out with around 20 people who celebrated the release of Scala 2.11 as well as to continue talking about future project ideas with what we could learn during the meeting. Very lovely atmosphere. Wish you were with us. It’s the meeting I surely never forget that gave me a fast-paced introduction to Java 8. Thanks.

Once I reached a certain level of confidence in my understanding of sbt, I’ve looked around for open source projects that use sbt as their build tool in order to get more insight on how much we could help each other. I’ve found quite a few projects that made me sad how far their builds are from what I’ve been learning about sbt for the past months. That is not to say it’s these projects’ fault. Quite the contrary, sbt was once meant to be simple (I kept hearing it here and there for quite some time before I jumped on the sbt bandwagon), however expectations seemed to have been too high on both sides.

It should come as no surprise that people are bashing sbt for its name that once meant to have reflected simplicity (as in “simple build tool”), but finally got touted as THE “Scala build tool” to better reflect its focus on Scala as the build language (that might be a reason for losing sight of simplicity during the “transformation” to Scala as the language is far from simple for some). I think it needs to change and begs for more effort from the sbt community.

As Josh Suereth has said during his talk about sbt.next – “Lot of investments going on in the documentation.” That’s quite too often a sparse resource.

Once burnt, twice shy?

Given how much sbt has changed with 0.13 and the plans ahead, it’s now the sbt community’s turn to take the stage and do the due diligence which is to help these open source projects getting better in their use of sbt (that should ultimately help both sides – the sbt community as a whole with more appreciation from the projects for how much their builds got easier with sbt). sbt 0.12.x and previous releases should be deemed highly poisonous with sbt 0.13.x the recipe.

Let’s treat it a call for action for the sbt community (or we’re going to end up as endangered species very soon with our hope and love for sbt). I think it’s about time!

Akka and the sbt build

Konrad Malawski has recently joined the Akka team (that with Grzegorz Kossakowski gives two Poles in Typesafe! Woohoo!), and I decided to celebrate the (not-so-small) change in Konrad’s career by embracing Akka in my tooling. I’m sure Konrad will like it (don’t you?)

I usually get myself engaged in an open source project by the sources (starting with compilation and reviewing the docs) and books (if they’re available). Akka was no exception and since I’m interested in sbt I couldn’t resist doing both – reviewing the Akka sources with their sbt build as the first gig. I’ve also got three books about Akka on my shelf – Akka Concurrency, Effective Akka, and Developing an Akka Edge – and there are quite a few coming (to the press or my shelf) – Akka in Action and Akka Essentials (let me know if I’ve missed any).

To make the story short – there’s room for improvement in Akka build- and documentation-wise (which again should come as no surprise as it’s a sort of fate of open source projects that usually focus on the code with community and documentation as a second thought). I know far too many people who’d like to get into Scala and one of the big open source projects, and think with Konrad aboard, Akka might be a good fit (and since there’s the sbt build they’ll learn the tool, too).

See you on GitHub!