mishadoff thoughts

Architecture Components

2025-05-30T10:49:00+00:00

Thoughts on architecture components and software stability.
In this article we also discuss one approach to evaluating architecture quality.

Software Architecture is about drawing boxes and arrows.
– Some architect

Architecture Quality Challenge

Evaluating software architecture is difficult. Unlike performance or test coverage, architecture lacks clear, objective metrics. Most assessments rely on expert judgment, which are often subjective and inconsistent.

This is a problem because architecture has a huge impact on maintainability, scalability, and delivery speed. To move beyond intuition, we need some indicators to measure architecture based on component relationships, dependencies, and change impact. This article explores one approach, using graph-based analysis to uncover hidden complexity and assess architectural quality.

Architecture Component (box)

rectangle "component"

An architecture component is a unit of software with a clearly defined responsibility and interface.
In diagrams, it is “a box” - but its physical form can vary depending on the architecture style and point of view. It might be a folder in a monorepo, a class module, a shared library, or a containerized microservice with its own database.

Regardless of form, a component has two essential traits:

Encapsulates functionality
Exposes an interface

Well-designed components also tend to be:

Reusable in different contexts
Interdependent, working with other components to fulfill business scenarios

In other words, components divide complexity and create boundaries that make the system understandable and maintainable. They are the atoms of architecture.

Dependency (arrow)

Components rarely operate alone. They collaborate by calling each other’s interfaces, forming dependencies. In architectural diagrams, these relationships are shown as arrows: if component A uses component B, we draw an arrow from A to B.

rectangle "A"
rectangle "B"

A -> B

The direction of the arrow is crucial. It tells us who depends on whom.

If we change A (e.g. adjust its logic or interface usage), only A needs to be tested.

rectangle "A" #red
rectangle "B"

A -> B

But if we change B (e.g. modify its interface or data model), this may affect all components that depend on it, including A:

rectangle "A" #pink
rectangle "B" #red

A -> B

This makes B more critical in the architecture - it sits upstream, and changes to it have a wider impact. Components with many inbound arrows often need stronger contracts, better test coverage, and more cautious deployment strategies.

Transitive Dependency

Dependencies can be transitive. If A depends on B, and B depends on C, then A indirectly depends on C

rectangle "A" #pink
rectangle "B" #pink
rectangle "C" #red

A -> B
B -> C

In this setup, a change in C can go through B and break A, even if A has no direct connection to C. These chains of dependencies increase the system’s fragility and make reasoning about change more complex.

Circular Dependency

Circular dependencies are even more sneakier. If components form a loop A -> B -> C -> A - then each one indirectly depends on itself.

rectangle "A" #red 
rectangle "B" #red
rectangle "C" #red

A -> B
B -> C
C -> A

Just think how fragile it is. This breaks modularity. A change in any component affects the entire cycle. Worse, the cycle hides these effects, making them difficult to test or isolate. Circular dependencies should be avoided or broken explicitly through architectural refactoring.

Case Study: Website “Go Bananas”

Let’s apply our ideas to a concrete example. Imagine a grocery delivery website called “Go Bananas”. From a user’s perspective, it’s simple: they open the site, find a product, check availability, add it to the cart, place an order, pay, and wait for delivery.

But architecture is about perspective. For the end user, it’s just one system. For a UI/UX designer, it might break down into two broad areas: the UI and “everything else.”

A software architect, however, sees a more complex structure - a system made up of distinct components, each responsible for a part of the functionality. Let’s break the requirements down and map them to functional units:

“User opens website” — There needs to be a frontend that displays product information. This is typically called the ecommerce or website component.
“Find a product” — We need a list of products, so we introduce a products component.
“Check availability” — Knowing how many bananas are left in stock is different from knowing what types of bananas we sell. Availability changes frequently, while product definitions do not. This leads us to split out an assortment component, responsible for stock levels.
“Place an order” — Requires an orders component.
“Pay for it” — This process is handled by some payments component.
“Wait to be delivered” — Managed by a delivery component.

From a single user actions sequence, we now have a system composed of six components:

rectangle "ecommerce"
rectangle "products"
rectangle "assortment"
rectangle "orders"
rectangle "payments"
rectangle "delivery"

Component diagram (boxes and arrows)

A component diagram is a visual representation of a system’s structure. It uses the basic architectural building blocks we’ve discussed: boxes for components and arrows for dependencies. It shows how different parts of the system interact and rely on each other to fulfill business processes. Such diagram is a key tool for analyzing architecture - it reveals dependency chains, coupling, and potential hotspots that affect maintainability.

Now let’s return to our components in the “Go Bananas” system and connect them with arrows to form a component diagram.

User -> Ecommerce
The user “depends” on the website. If the site changes (e.g. new layout or navigation), the user experience is affected and may require adjustment or relearning.
Ecommerce -> Products, Assortment
To render the UI, the ecommerce component pulls product information (names, images, prices) from the products component, and stock availability from assortment.
Ecommerce -> Orders
Once the user builds their cart, the selected items are sent to the orders component to create a new order.
Assortment -> Products
Stock information in assortment must be linked to specific products. To register an item in stock, we need to know what that item is.
Orders -> Products, Assortment
The orders component validates the incoming cart - checking that all products exist and that enough stock is available.
Orders -> Payments
Once the order is created, the user is directed to the payments component to complete the transaction.
Orders -> Delivery
After payment and confirmation, the orders component schedules delivery via the delivery component.

Here is the complete diagram:

actor "user"
rectangle "ecommerce"
rectangle "products"
rectangle "assortment"
rectangle "orders"
rectangle "payments"
rectangle "delivery"

user -> ecommerce
ecommerce -down-> products
ecommerce -down-> assortment

assortment -left-> products 

ecommerce -down--> orders

orders -up-> products
orders -up-> assortment

orders -left-> payments
orders -right-> delivery

This diagram captures both component structure and data flow, which are key to understanding how the architecture behaves during change. At this stage, we’re not judging whether this is “good” or “bad” architecture - we’re focused on mapping the relationships clearly. This sets the foundation for analyzing system maintainability.

Maintainability Use Cases

What happens if we change something?

We’ll walk through a few scenarios and highlight all components affected by the change, directly or transitively.

Case A: Change in `ecommerce`

Let’s say we update the UI layout, add a filter, or redesign how items appear to the user. This affects only the ecommerce component. Since it consumes data from other components but doesn’t expose its own interface, the change is isolated.

actor "user"
rectangle "ecommerce" #red
rectangle "products"
rectangle "assortment"
rectangle "orders"
rectangle "payments"
rectangle "delivery"

user -> ecommerce
ecommerce -down-> products
ecommerce -down-> assortment
assortment -left-> products 
ecommerce -down--> orders
orders -up-> products
orders -up-> assortment
orders -left-> payments
orders -right-> delivery

Impact: Low. ecommerce can evolve independently as long as its usage of other components remains unchanged.

Case B: Change in `products`

Now imagine we change the product schema - rename fields, restructure data, or split categories. This breaks consumers of the products component, namely ecommerce, orders, and assortment.

actor "user"
rectangle "ecommerce" #pink
rectangle "products" #red
rectangle "assortment" #pink
rectangle "orders" #pink
rectangle "payments"
rectangle "delivery"

user -> ecommerce
ecommerce -down-> products
ecommerce -down-> assortment
assortment -left-> products 
ecommerce -down--> orders
orders -up-> products
orders -up-> assortment
orders -left-> payments
orders -right-> delivery

Impact: High. products is upstream of multiple components. Any changes must be coordinated and tested across the system.

Case C: Change in `orders`

Suppose we redesign the order lifecycle or adjust how order validation works. This affects only ecommerce (which sends orders).

actor "user"
rectangle "ecommerce" #pink
rectangle "products"
rectangle "assortment"
rectangle "orders" #red
rectangle "payments"
rectangle "delivery"

user -> ecommerce
ecommerce -down-> products
ecommerce -down-> assortment
assortment -left-> products 
ecommerce -down--> orders
orders -up-> products
orders -up-> assortment
orders -left-> payments
orders -right-> delivery

Impact: Medium. Despite orders is a central component with many inbound and outbound connections, the way we designed it has pretty isolated changes.

Case D: Change in `payments`

We might update payment provider integration, add fraud detection, or restructure transaction handling. Since payments is used only by orders, the change is relatively contained.

actor "user"
rectangle "ecommerce" #pink
rectangle "products"
rectangle "assortment"
rectangle "orders" #pink
rectangle "payments" #red
rectangle "delivery"

user -> ecommerce
ecommerce -down-> products
ecommerce -down-> assortment
assortment -left-> products 
ecommerce -down--> orders
orders -up-> products
orders -up-> assortment
orders -left-> payments
orders -right-> delivery

Impact: Medium. Changes in payments may require minor adjustments in orders, but are unlikely to cascade further.

This type of impact analysis is a powerful tool for objectively evaluating maintainability. The more isolated changes can be, the more maintainable the architecture is. Components with high fan-in or fan-out are more expensive to modify and should be designed and managed with extra care.

“Hot” Components

Now that we’ve explored how changes in one component can affect others, let’s look at the architecture through a more quantitative lens. Some components are hotspots - they are connected to many others and become bottlenecks for development and maintenance. The more central a component is in the dependency graph, the more “expensive” it is to change.

We can start measuring this by calculating three metrics for each component:

IC (Number of Inbound Components) How many other components directly depend on this one.
OC (Number of Outbound Components) How many other components this one depends on.

These metrics help identify overloaded components - ones with high inbound or outbound coupling - which can signal risk, fragility, or the need for architectural refactoring.

Metrics for “Go Bananas”

Let’s apply this to our current component diagram:

actor "user"
rectangle "ecommerce"
rectangle "products"
rectangle "assortment"
rectangle "orders"
rectangle "payments"
rectangle "delivery"

user -> ecommerce
ecommerce -down-> products
ecommerce -down-> assortment
assortment -left-> products 
ecommerce -down--> orders
orders -up-> products
orders -up-> assortment
orders -left-> payments
orders -right-> delivery

Component     | IC (Inbound)                         | OC (Outbound)
--------------|--------------------------------------|-----------------------------------------------
ecommerce     | 1 (user)                             | 3 (products, assortment, orders)
products      | 3 (ecommerce, assortment, orders)    | 0
assortment    | 2 (ecommerce, orders)                | 1 (products)
orders        | 1 (ecommerce)                        | 4 (products, assortment, payments, delivery)
payments      | 1 (orders)                           | 0
delivery      | 1 (orders)                           | 0

Observations

orders is the most connected component. It has the highest outbound count (OC = 4), depending on products, assortment, payments, and delivery. It also has inbound connections from ecommerce. This makes orders a clear hotspot - any change to it risks impacting multiple other components.
products has the highest inbound count (IC = 3), meaning it is relied on by ecommerce, assortment, and orders. While it doesn’t depend on anything itself, its central position makes it fragile: changing it can break multiple downstream components.
assortment plays a mid role - used by ecommerce and orders, while depending on products. It links catalog data to inventory and order logic, making it moderately sensitive to change.
ecommerce, payments, and delivery are more isolated. They either consume other services or are used in a limited way. As a result, they’re generally safer to change without wide system impact.

Why This Matters

Overloaded components can slow down teams. A single change may require coordination with multiple services, broader test coverage, and longer release cycles. These components tend to accumulate complex logic, become harder to reason about, and resist refactoring.

By identifying such components early using simple dependency metrics like Inbound (IC) and Outbound (OC), we can:

Split responsibilities into smaller, clearer modules
Introduce anti-corruption layers to isolate fragile contracts
Improve testability by decoupling logic
Reduce unexpected ripple effects during change

In short, structural metrics turn architectural intuition into actionable insight - giving teams a way to spot problems before they become blockers.

Mnemonics-based keymap for IntelliJ IDEA

2018-09-25T14:00:00+00:00

Emacs is great for editing text, IntelliJ IDEA is great for navigation.
Let’s take best from both worlds into one editor.

Rationale

If you can configure it, you will configure it
- Anonymous, about Linux

Emacs is keyboard-driven editor, meaning all actions you can do without mouse or touchpad. It is a good part. The bad part is it lacks rich navigation capabilities, integration with external tools, code aware completion, etc. Well, you still can try hundred of plugins and configure exactly what you need, or try tens of prebuild emacs-based distributions with batteries included, but IntelliJ IDEA just works out of the box.

IDEA has most things that missing in Emacs and still considered among many as the best IDE for Java.
Even more, Cursive extension for Clojure works really well.

The most annoying things in IDEA is an editor and interaction with tool windows, I mean you need mouse. Even using emacs-based keymap, you will click mouse/touchpad extensively. Your fingers will be screaming. Especially, if you work on the latest MacBook Pro.

This is an attempt to develop mnemonics-based keymap (a.k.a Emacs style) for IntelliJ, to reduce mouse interaction to minimum, while providing a simple way to remember the hotkeys.

Preparation

I’m a happy user of MacBook Pro 13’’ (2017). Really, I find MacOS still the best system for people and devs, but this new keyboard drive me crazy.

Ironically, Escape button escaped to virtual world, so you can’t get tactile feedback from it. Apple provided a discount, so you got four arrows in the ~~price~~ place of three. Greedy backspace, which steal all your home row attention if you want just delete a char. Small vertical enter.

I can continue ranting about non-consistent sound from different keypresses, but thanks to this blog I remapped most of this keys, and happy so far.

Shortly, CapsLock remapped to Control (do you know people who use CapsLock?), then Control-IJKL become normal arrows, Control-H as a backspace (hello Emacs users), Control-M and Control-N work in pair as a horizontal Enter, and the same way Control-[ and Control-] work like Escape. Also, I don’t use right Command, so it also became Control, now you can use arrows with one hand while… eating or so.

This may sound strange configuration, but for me it works like a charm.
It took approximately two weeks to get used to new keyboard mapping.
Below, I list all hotkeys.

Basic hotkeys (C- prefix)

Basic hotkeys triggered with Control prefix. (C-a means press “A” while holding Control key)
This category exposes basic keybindings for navigating and editing text.

`C-a` - move cursor to the start of line

“A” is always beginning of something, line in this case.

`C-b` - move cursor one word backward

“B” for backward. For long sentences it’s faster than arrows.

`C-c` - prefix, reserved for Code mode

`C-d` - delete character (forward)

“D” for delete. Because, my MacBook keyboard doesn’t have delete button, only backspace.

`C-e` - move cursor to the end of line

“E” for end. Opposite to “A”.

`C-f` - move cursor one word forward

“F” for forward. Opposite to “B”.

`C-g` - IDEA Escape

IDEA Escape used in different context, mostly meaning stop input or action.
Stop selection, stop search, close popups, etc.
Mnemonic is stop looking for “G” spot.

`C-h` - delete character (backward, a.k.a Backspace)

It was a hotkey for backspace in emacs, don’t know why…
In our case it’s close to J key, which represent left arrow, so instead of moving cursor left, you slide index finger to the left, meaning, move cursor and delete the char.

`C-i`/`C-j`/`C-k`/`C-l` - arrows up/left/down/right

Look at the keyboard, how nicely ijkl combination represent the arrows, like wasd for gamers.
And almost like hjkl for vimers.

`C-m`/`C-n` - enter or return

In home row position your right index finger on J button, just slide a little down and you will get to M. Or N. Works for both cases.

`C-r` - replace in current file

“R” for replace. Also note, that while replace top dialog is available you have additional contextual hotkeys

C-m - (enter) replace occurence and go to next
C-k - (arrow down) skip and go to next occurence
C-i - (arrow up) skip and go to previous occurence
C-g - (escape) exit from replace dialog

`C-s` - search in current file

“S” for Search (and Solr). Similar to replace dialog, some contextual hotkeys available

C-m or C-k - (enter or arrow down) go to next occurence
C-i - (arrow up) skip and go to previous occurence
C-g - (escape) exit from replace dialog

`C-t` - prefix for Tool Mode

`C-w` - cut

Like C-x for normal people. I don’t how what mnemonic here, just used it since emacs times.
Maybe, “W” looks like a saw. Or two pairs of scissors.

`C-x` - prefix for Execute mode

`C-y` - paste

Like C-v for normal people. “Y” stands for “yanking” in emacs. Don’t know what it means.

`C-z` - undo

Good old C-z. It’s like time machine, if everything is fucked up, you press it to go back in time.

`C-=` - extend selection

= located on the same key, where + is. Plus means to enlarge something.
Smart selection is my #1 used feature in IDEA.

`C--` - shrink selection

- is an opposite to +, so if there is a way to enlarge something, should be an option to shrink it.
Not as popular as extend selection, but if you did one extra selection, there is a way to go back, rather than starting from beginning.

`C-] / C-[` - escape

IDEA has it’s own escape C-g but it doesn’t work for all cases.
You still need real global escape. And better if you can touch it.

`C-.` - completion

For me as for java-programmer-in-the-past, . has a special meaning - access members of object. IDEA often in this case automatically trigger code completion. So yes, . is for completion.

`C-SPACE` - start/end sticky selection

Alternative Keys (C-M-)

This category slightly changes the functionality of basic hotkeys.
Hold both Control and Meta key (Command for MacOS) and you have new set of keybindings.
Semantic meaning of such hotkeys stays the same, but adds some features or works in a different context.

`C-M-a` - move cursor to the start of the file

C-a move cursor to start of the line, with meta it moves to the start of the file.

`C-M-b` - move cursor one function backward

C-b move cursor one word backward, with meta it moves one function backward.

`C-M-d` - delete line from cursor to the end of line

C-d delete one character backward, with meta it deletes the rest of the line in the same direction.
Note: for most programs (including terminal) this functionality available with hotkey C-k, but C-k is global arrow down for us. We resolved this conflict by global remapping C-M-d to C-k. You can use C-M-d across all applications which support C-k.

`C-M-e` - move cursor to the end of the file

C-e move cursor to end of the line, with meta it moves to the end of the file.

`C-M-f` - move cursor one function forward

C-f move cursor one word forward, with meta it moves one function forward.

`C-M-h` - delete line from the cursor to the start of line

C-h delete one character forward, with meta it deletes the rest of the line in the same direction.
Never used.

`C-M-i`/`C-M-j`/`C-M-k`/`C-M-l` - text movement

C-ijkl move cursor in four direction, with meta it moves line or selection in four directions. Left/right directions works like a Tab/Shift-Tab (indent/unindent)

`C-M-m`/`C-M-n` - alternative enter

C-m/C-n powered by Karabiner and replace default enter fucntionality.

Alternative enter available in Idea in various context. For example, pressing enter on project view file, open this file in editor, but doesn’t switch to it. Alternative enter switch to the file. Also, pressing enter on unfinished input in REPL, adds newline. If you press alternative enter instead, whole form will be sent for evaluation.

`C-M-r` - replace globally

C-r replace occurrences in current file, with meta it replace across whole project/module. While in this dialog use arrows to navigate through occurences and press Opt-R ro replace. (Don’t like Opt-R, but there is no way to override it in Idea)

`C-M-s` - search globally

C-s search for occurrence in current file, with meta it search across whole project/module.

`C-M-z` - redo

C-z undo the action, with meta you can redo it it.

`C-M-[`/`C-M-]` - escape from tool window and close it

One usage of escape button is to switch from the tool window back to editor. Tool window stil remains visible. If you want to hide tool window press extra meta key (will be covered more in Tool Mode section)

Code Mode (C-c prefix)

Code Mode provide hotkeys (prefixed with C-c) to work with the code.

`C-c a` - git annotate

When you find shitty piece of code, you want to know who added it. Blame him.
Idea call it A - Annotate.

`C-c b` - move cursor to matching brace

B for Brace. Navigating to matching brace helps you to add something before or after form.
Also, nice to see your parens are balanced.

`C-c c` - comment

Most of the time is simpler to comment by adding ; in front of the line, or #_ if you want to comment whole form. But when cursor is somewhere in the middle of line, this hotkey, comment the line and jumps to the next one. Handy. C obviously for comment.

`C-c d` - duplicate line

DRY principle stands for Don’t Repeat Yourself. I doubt it is true for developers, copy/paste it’s all we have.
D for Duplicate or Do Repeat Yourself.

`C-c e` - jump to next error

Programming without errors is not a programming. Jump to next error.

`C-c C-e` - error description

After you jumped to error, you need to get description of what’s wrong. Usually people do this by hovering mouse over the error, but we have a keyboard.

`C-c f` - find current file in project view

To see context of directory or package you are working in, reveal the file in project view.

`C-c h` - inline help

Call inline documentation of the function, if you forgot the order of arguments or want to make sure it does exactly what you need. H for Help. Escape closes popup.

`C-c i` - indent code

Copypasted code from somewhere else and got broken indentation? Press “I” to indent.

`C-c j` - ace jump

Jump. It’s a special keybinding prefix powered by acejump plugin. Type this prefix, type some char and acejump will highlight all occurences of this char in the visible editor. Use hints to move to specific place.
Poor man’s replacement for the mouse “click here”.

`C-c k` - kill whole line

Kill current line, no matter where you at.

`C-c l` - go to line

Move to the line. Generally used if you got an exception with extra on what line error occurs.

`C-c m` - list methods

Correct name for this command is “show file structure”, but you mostly working with the code, so file structure are methods and variables. M methods.

`C-c C-m`/`C-c C-n` - IDEA show intentions

Ideal way to perform imports, requires and other contextual suggestions.
Mnemonic is “Code (C-c) Enter (C-m, C-n)”.

`C-c r` - refactor menu

R for refactor, very powerful IDEA feature, but limited in clojure. Hopefully, could be improved to contain more refactoring items.

`C-c s` - go to source

Exciting code navigating feature. Press “S” (source), to go to the function definition, either current project or external library.

`C-c t` - transpose forms

If you misplaced the arguments to the function, press “T” (transpose) to swap them. Works on forms as well, current form will be swapped with the left.

`C-c u` - find usages

U for usages. Prior to changing function or its implementation you want to know what pieces it affects. Must have feature for any editor.

`C-c w` - wrap selection

Select piece of code, and wrap it to something. try-catch block is the most popular choice, but you can add your own snippets.

`C-c -` - collapse

Long function can be collapsed into oneliner. Minus sign means reduce visible part of text.

`C-c =` - expand

Collapsed function, can be expanded to normal view again. As already mentioned, = sign located on the button with +, which means enlarge.

`C-c C--` - collapse all

Alternatively to collapse, there is posibility to collapse all functions. Hide all, open one and focus on it. But if your namespace overloaded with a lot of functions think about better way to focus on one function.

`C-c C-=` - expand all

Similarly, there is expand all keyboard, bring everything to normal.

Execute mode (C-x prefix)

Execute mode provide hotkeys to interact with REPL and editor window.

`C-x b` - set bookmark

Save frequently-visited location in the code to bookmark.

`C-x C-b` - show bookmarks

Show all saved bookmarks and navigate/edit them.

`C-x d` - show debug menu

D for Debug. But remember, best debug are print statements.

`C-x e` - evaluate form in REPL

Eval is most important hotkey for REPL driven development.
So important so C-x C-e does the same.

`C-x f` - go to file

Old school emacs hotkey to open file, C-x C-f does the same.

`C-x g` - show git dialog

Opens git dialog. Not keyboard friendly and not as powerful as magit, but just in case you use IntelliJ git client. I am totally fine with console tig tool, though simple magit port for IDEA would be good.

`C-x C-h` - Clear REPL output

C-h is backspace in our world, and backspace means to clear.

`C-x i` - interrupt

If function runs too long, press I to interrupt.

`C-x k` - kill window

Kill editor window, when it’s not needed anymore.

`C-x l` - load file in REPL

Load file in REPL. Sync all forms between file and REPL environment.
Very important file of the flow: change file -> load to REPL -> repeat

`C-x m` - start/stop macro recording

Not as powerful macro capabilities as in emacs, but still there is a way to record routine actions.
The same hotkeys start and stops macro recording. After macro is recorded, you can provide a name for it, to make it persistent.
M-m plays the last macro.

`C-x n` - set current file to namespace in REPL

Working in user namespace is often error-prone, when loaded function names mixed in one environment.
Use namespaces for REPL instead. The same as (in-ns)

`C-x o` - go to next split window

Emacs allows to open and arrange several buffers in one screen. Similarly, IDEA allows to view multiple editors at once, C-x o cycles to next visible editor. I always working in one editor, sometimes open two to copy chunks or compare files. Therefore, O means other.

`C-x C-o` - Open file…

IDEA is project-based editor, but there is a way to open external files for quick edit. Don’t know why.

`C-x q` - stop REPL

Sometimes, you will need to restart REPL to refresh dependenies, cleanup environment, etc.
Just quit current REPL and start new one using C-x x.

`C-x s` - search REPL history

Just search.

`C-x t` - Run test under cursor

TDD adepts will love that.

`C-x C-t` - Run all tests in current namespace

When feature is done, run all tests for it.

`C-x C-j` - navigate backward

Go through history of files and places which has been edited.
C-j is a left arrow, remember?

`C-x C-l` - navigate forward

Similarly, go forward through history of edited files.

`C-x x` - show run menu

X for execute. Double X for Execute program using execute mode.

`C-x 0` - close current split editor

The same as C-x k, good to have consistency there. 0 means nothing.

`C-x 1` - close all split editors except current

Leave only current editor. 1 and only one.

`C-x 2` - split editor verically

A way to add more editors to the view.

`C-x 3` - split editor horizontally

Another way to add more editors to the view.

`C-x C-=` - Increase font size

Again, = located on the same button as +.

`C-x C--` - Decrease font size

Decrease font size. - opposite to +.

`C-x C-0` - Set font size to default

Good to have default one.

Tool mode (C-t prefix)

Tool mode allows to manage IDEA tool windows.

`C-t c` - open TERMINAL tool window

C for console. Generally not used, but when you switch often between terminal and edited file, could be handy.

`C-t d` - open DEBUG tool window

IDEA Debug has a lot of features, when you debugging Java code, which could be another topic.
I use sometimes Opt-I (step in), Opt-O (step-out), Opt-N (next statement) and Opt-E (valuate expression). Because keys like F8 are completely meaningless (and not available in MacBook Pro)

`C-t e` - open EVENT LOG tool window

Event log holds some information about IDEA system exceptions and compilation.
Not really useful when you have REPL.

`C-t f` - maximize tool window

F for Fullscreen.

`C-t g` - open GIT tool window

Another place for Git. Quick look at what has been edited.

`C-t h` - Hide all windows (except editor)

H for Hide. Also C-h is backspace which has meaning delete, tool windows in this context.

`C-t C-i/C-j/C-k/C-l` - Adjust Size of tool window

Arrows naturally shows the direction in which tool window will be extended.

`C-t l` - open LEININGEN tool window

L is lein. Shows dependency tree and tasks available. Also lein tasks could be run from this view, what some people find handy.
I still prefer using lein from terminal.

`C-t m` - open MAVEN tool window

The same for Maven.

`C-t o` - open REPL OUTPUT tool window

Jump to the REPL output.

`C-t p` - open PROJECT tool window

P. Open project view.

`C-t q` - Quit tool window

Tool windows which are not needed anymore should be closed to free space.

`C-t r` - open REPL tool window

All you need is REPL. REPL is all you need.

`C-t s` - open SEARCH tool window

Back to search view, where find usages C-c u was submitted.

`C-t t` - open TODO tool window

Who else uses TODO list to organize their work?

`C-t x` - open RUN tool window

Mostly used in Java. When you executed program, shows it’s output.

Popular keybindings (M-)

Popular hotkeys has no specific topic, but due to its frequent usage they belong to one group. M- does not require prefix, which means, they will be even faster than other hotkeys.

`M-a` - select all

Select all is a frequent action, for example copy configuration from one file to another.

`M-b` - Paredit Barf

Structural editing for Clojure, moves last form out of the current form.

`M-c` - Copy

Just plain copy.

`M-d` - Delete whitespaces

Delete whitespaces (including newlines). Like C-d but applied continuosly to the first non-whitespace symbol.

`M-e` - Recently edited files

Shows menu with recently edited files and tool windows.
Poor replacement for emacs ibuffer.

`M-f` - Move forward into SEXP on the same level

Structural movement on the forms at one level. Kinda useful.

`M-i` - Interrupt evaluation

Already discussed, available at hotkey C-x i. I so often evaluate lazy sequence in REPL, so hotkey promoted to the popular ones.

`M-j` - Jump

The same, available in code mode C-c j, but if used extensively extra prefix makes it slower.

`M-k` - open symbol

If you remember name of the function or var, but forgot namespace it belongs to, use this. K has no special meaning here, just available no prefix hotkey.

`M-l` - Load file into REPL

Promoted from C-x l

`M-m` - Play last macro

We have already seen how you can record a macro using C-x m. Playback just applies it. Unfortunately, there is no C-u modifier like in emacs to repeat macro n times. But you can press M-m n times.

`M-n` - open namespace

Used mostly when working with clojure code, otherwise could be superceded by open file C-x C-f.

`M-o` - REPL output

Promoted from C-t o

`M-p` - project view

The same as C-t p

`M-r` - REPL

No questions why REPL is so popular, promoted C-t r.

`M-s` - Paredit Slurp

Structural editing for Clojure. Awesome feature.

`M-t` - Run Tests

Promoted from C-x C-t

`M-u` - Uber menu

Open context menu. If some action can not be achieved using hotkey, you still have the context menu with all options available.

`M-v`/`M-w` - Paste

One from MacOS, one from emacs, so I added them both.

`M-x` - Enter action

Honestly, there is no need to have a hotkey for every action. It makes sense to add hotkey if it has mnemonic, so you do not spend extra cost remebering it, or its really popular. Otherwise all IDEA actions available to you under M-x, execute action.

`M-,` - IDEA Settings

Open IDEA settings. I doubt you can do something there without a mouse.

Conclusion

I guess everyone has its own way of coding, favorite editor and set of plugins. That is fine, and I am not going to convince you this is better way of doing that yours. All I wanted to share, that despite many years of clojure in emacs, I found IDEA with Cursive smart and very promising development environment for clojurists. (maybe, not so extensible)

I still use emacs and cider sometimes for simple clojure projects, org-mode, markdown editor, notes, default terminal editor and ocassional development on other languages.

And lastly, in case if you interested in the same settings, idea_keys.jar

Evolville 0.0.2: Life and Death

2018-03-30T23:43:00+00:00

Evolville is an imaginary computer world, inhabited by creatures.
They live, move, eat, communicate and evolve.

In previous post we’ve seen the origins of the World.

It looks pretty simple: white space inhabited by some creatures. All those creatures just moving in random directions causing world look like boiling soup. Even it seems that some creatures exist, there is no life yet.

God of Setup was responsible for adding creatures to the world initially, but they can’t produce new creatures by themselves. Also, there was no death, once creature appeared in the world it was living there till the end of time.

There is an awesome example of simulation game, called Conway’s Game of Life. Life and Death defined with just four simple rules, but even this simplicity cause to observe very interesting things, like viruses, stable population and replicants.

We’re going to define similar rules for our 2D-world, so let’s start.

Living Creatures

Our world had only list of unknown creatures so far.

(def world {:creatures [{:loc [200 200] :size 10 :speed 1 :dir 45}
                        {:loc [100 300] :size 20 :speed 4 :dir 135}
                        {:loc [350 150] :size 30 :speed 2 :dir 60}]})

This is ok for soup, but not ok for living creatures.
Every creature is unique, so it should have a name. Let’s change our structure a bit.

(def world {:creatures {"A" {:loc [200 200] :size 10 :speed 1 :dir 45}
                        "B" {:loc [100 300] :size 20 :speed 4 :dir 135}
                        "C" {:loc [350 150] :size 30 :speed 2 :dir 60}}})

Now, we can uniquely identify each creature by name.

Life

We define Life as an ability of creatures to give birth to other creatures.

One of the simplest form of life is cell division.

Since our world not aware about energy, dna and chemical processes yet, we say, one creature can spawn another creature. Then each of those creatures spawn another creature and so on. Exponentially.

Spawning takes some time, so let’s say our creatures will be spawning once per 3 seconds.

(def spawn-rate 3000) ;; new creature will appear every 3 seconds 

(defn spawn [world [id creature]]
  (let [now (System/currentTimeMillis)
        last-spawn-ts (some-> creature :spawn :ts)]
    (cond
      ;; initially make creature aware about spawning
      (nil? last-spawn-ts) (assoc-in world [:creatures id :spawn :ts] now)

      ;; if time has passed creature can spawn
      (< (+ last-spawn-ts spawn-rate) now)
      (let [[child-name child-creature] (creature/random world)] ;; generate random creature
        (-> world
            (assoc-in [:creatures child-name] child-creature)       ;; add it to the world 
            (assoc-in [:creatures child-name :loc] (:loc creature)) ;; give it parent location
            (assoc-in [:creatures id :spawn :ts] now)))             ;; update parent spawn ts

      ;; nothing changes, leave the world as is
      :else world)))

Our spawn function takes the world, and a creature. It could return the unchanged world state, or add creature to it.

This convention is kinda useful so far, so let’s do a utility world function which changes the world by appling function to all creatures.

(defn for-each-creature [world f]
  (reduce f world (->> world :creatures seq)))

Global spawning for all creatures will look like this

(defn spawn-creatures [world]
  (w/for-each-creature world spawn))

Meanwhile, God of Update hired an assistant Action to help him with programming. Action refactored world updates and made them structured set of actions.
All of our creatures can move and spawn.

(defn act [world]
  (-> world
      (move-creatures)
      (spawn-creatures)))

Spawners

Spawning is the uncontrolled form of Life. No matter what happens, each spawner produce another spawner every three seconds.

As soon as the world starts with at least one spawner, it will be flooded with living creatures soon.

Death

To make sure the world is balanced, we need death. There are lot of strategies could be implemented to cause death, but let’s start with the simplest one - ageing.

Opposite to spawning, ageing is an uncontrolled form of death. No matter what spawner is doing, it will be dead at some point by ageing. It’s just a matter of time.

Since ageing is not controlled, it is not a complex process either. Creatures will lose some part of their size on every world update. If size becomes less than zero the creature will die and lost for our world, forever.

(def age-rate 0.05)

(defn age [world [id creature]]
  (let [newsize (- (get-in creature [:size]) age-rate)]
    (cond (<= newsize 0)  ;; remove dead creatures
          (u/dissoc-in world [:creatures id]) 
          :else (assoc-in world [:creatures id :size] newsize))))

(defn age-creatures [world]
  (w/for-each-creature world age))

Note: clojure lacks of dissoc-in functions, so we took it from there

Warning: Spawning works in terms of time (e.g every three seconds), ageing works in terms of frames (e.g every frame creature lose 0.05 in size). These values are not always related with the same ratio, therefore there is no simple conversion between frames per second and native clock so far. Accept the fact, that our world is not ideal at all, we will try to solve the issue later.

After adding ageing to our world actions structure, we can observe this picture.

Two things here. Creature spawn another creature periodically and creature lose its size with the time, therefore dying. Those are two opposite processes, so you need to work really hard to balance them. If spawn rate is greater than age rate, the world become overpopulated, otherwise all life forms will die.

That’s educational for us, but let’s move further and develop more complex rules for life and death.

Breeding

Similar to the reproduction process, when two parents produce child, we define breeding rule in our world. If two creatures collide, they will produce another creature (omit gender for now).

To avoid fast overpopulation, let’s say creatures can breed only once per three seconds.

(def breed-delay 3000)

(defn breed [world [id creature]]
  (cond
    ;; if creature can not breed leave the world unchanged
    (not (can-breed? creature)) world
    :else
    (let [mate-creature (->> (w/creatures world)
                             (filter (fn [[_id mate]]
                                       (and (not= id _id) ;; exclude breeding with itself, lol
                                            (can-breed? mate)
                                            (collide? creature mate))))
                             (first))]
      (cond (nil? mate-creature) world
            :else
            (let [now (System/currentTimeMillis)
                  [_id mate] mate-creature
                  [cid child]
                  [(ec/uuid)
                   {:loc (mapv #(/ (+ %1 %2) 2.0) (:loc creature) (:loc mate))
                    :size (random/random 10 20)
                    :speed (random/random 2 4)
                    :dir (/ (+ (:dir creature) (:dir mate)) 2.0)
                    :breed {:available-at (+ now breed-delay)}}]]
              (-> world
                  (assoc-in [:creatures id :breed :available-at] (+ now breed-delay))
                  (assoc-in [:creatures _id :breed :available-at] (+ now breed-delay))
                  (assoc-in [:creatures cid] child)))))))
                  
(defn breeding [world]
  (w/for-each-creature world breed))                  

Though this code may look a bit clumsy, it adds breeding to our world.

The logic is following, if some creature can breed and there is at least one another creature who can breed and both those creatures collide, there is new creature born at their location. Newborn creature gets random speed, random size, but inherits average direction from the parents. Both parents got breed delay as well as new born child can not breed for 3 seconds.

Example: If parent A moves to the right (angle 0), and parent B moves to the left (angle 180), their child will get direction (0 + 180)/2 = 90 and will go up. It could go down as well, but let’s stick to the deterministic average rule.

Direction inheritance is the first example in our world, how parents affect their children.

We disable spawning and promote breeding as a new form of Life. Ageing is still active.

Having such simple rule in our imaginary world we can observe interesting patterns, which are similar to our real world.

Loneliness

One creature can not breed, so no life appears in the world.
Lonely creature will occasionally die because of ageing.

Forever Alone

Even if there is another creature in the world, but their paths never cross, there is no chance for life in the world. It’s like if you can not meet girl in Virginia, you can’t make children with her. This is, probably, why a lot of people like to travel.

Just a bad timing

If two creatures have crossing paths, they can make child theoretically. But bad timing is not on their side. Girl and boy live in the same town and visiting the same places. But they can never meet, as I said bad timing.

It’s not you, It’s me

Sometimes you know the girl, your paths often crosses, you even like it, but haven’t enough resources (money, car, house) to make her happy. The same here, one of those creature follows another, but has not enough speed to make breeding. World will collapse.

King’s Family

Considering the previous example, if one creature step back (reduce speed) or push more (gain more speed), the breeding could happen and they could produce a lot of children. The only issue is such population will contain very limited creatures. All of them will be moving in one direction, inherited from both parents. There is rare chance for other creatures to affect their direction. Like king’s family. Princess is not for you.

Rich and Dirty

Another problem with King’s Family, it doesn’t give a chances for other creatures in the world. You can see the Kingdom growing, but some lonely creature slowly dying near the gate.

Love

Much better when two people have different interests and love each other. For example, two creatures have opposite directions, but move to each other. They produce creatures with variable directions, causing more chances to breed with other creatures in the world.

It may seem first there is the same limited vertical area like in King’s Family, but it’s only at the beginning. At over 100000 creatures, it will cover almost whole world.

The true power of Love!

Broken Love

Sometimes love can be broken. Consider pair of creatures who moves towards each other. They slowly produce a population which is able to survive.

Now we start with the same configuration, but there is a third creature which is breeding faster. This creature blocks left creature from breeding, causing death to overall population. Don’t hang with losers.

Queen of Ants

You can mimic basic behaviour not only from human population. Ants are awesome creatures for evolution algorithms, so let’s do something from ants world. Consider one big ant (mother of all ants, called queen) which doesn’t move, but able to breed with other ants moving towards her. That could be a rise of Ants Empire.

There are plenty of other analogies you can see in the world simulated using breeding and ageing rules, but let’s move forward.

We’ve seen uncontrolled and controlled life (spawning, breeding) and uncontrolled death (ageing). Even ageing is works for small populations, most of the time simulations end with a lot of creatures in the world. Just because age rate is constant, and growth is not.

Seems like we need another form of death, let’s call it overpopulation.

Overpopulation

Overpopulation is an effective way to control the population size in the nature. If we have a forest with a lot of wolves and few rabbits, obviously most of the wolves will die, because there is not enough resources for them for a living. The same applies for humans and most of the other creatures.

Based on this we define an overpopulation rule for our creatures. If creature collides with two or more creatures, it dies.

(defn- overpopulated [world [id creature]]
  (let [collisions (->> (w/creatures world)
                        (filter (fn [[_id _creature]]
                                  (and (not= id _id)
                                       (c/collide? creature _creature))))
                        (count))]
    (if (>= collisions 2)
      world
      (assoc-in world [:creatures-new id] creature))))
      
(defn overpopulation [world]
  (let [w (w/for-each-creature world overpopulated)]
    (-> w
        (assoc :creatures (or (:creatures-new w) {}))
        (dissoc :creatures-new))))

Sounds simple, huh?

Here we just use temporary :creatures-new key, to keep track of living creatures, so we don’t modify our world before we check all collisions.

Something goes wrong. First, left and right creatures kills the middle one, what is correct. But then they produce a child which cause mutual collision for all three and therefore everyone dies.

Well, let’s assume that child and mate collisions doesn’t count. If parent produce a child, child can’t kill them. The same for pairs. If two creatures breeds, they can’t kill each other, we’re not Mantis creatures.

Just track who is creature parents and who breeds and then remove them from collisions list.

(defn- parent? [id creature]
  (some-> creature :parents (get id)))

(defn mate? [w aid bid]
  (or (= (get-in w [:creatures aid :mate]) bid)
      (= (get-in w [:creatures bid :mate]) aid)))

Looks like it works.

As a breeding, overpopulation has also some interesting patterns.

We don’t discuss all of them, but take a look at one which is really nice.

Balance

As we have seen, if three creature collides, one of them dies, and the rest two continues breeding for a while. But what if four creatures collide at the same time.

Unfortunately, everyone dies. Look at this like a situation, where everyone pushes, but no one wins. Like war. Or beer pong.

If even one creature could relax a bit, everything will be fine. Here we reduced speed of top left creature by 0.1

Consider this like life advice, sometimes you need to relax.

And finally, let’s go back to the boiling soup, from the previous post. Remember the 300 creatures were moving constantly? Now we try to apply to this boiling soup rules of life (breeding) and death (ageing and overpopulation).

Everything happens in the matter of second, so we added some delay and extra statistics for you.

Finally, our world looks more interesting. Some creatures are born, some dies. If breed rate becomes too large, overpopulation fixes that. If there are small amount of creatures, much easier for them to breed.

Life and Death is all about balance.

To be continued…

Source

Evolville 0.0.1: The Beginning

2018-03-19T14:48:00+00:00

Evolville is an imaginary computer world, inhabited by creatures.
They live, move, eat, communicate and evolve.

This is going to be series of posts for simulation, starting from scratch and test various evolution strategies. Our world will evolve, our creatures will evolve, our code will evolve.

Welcome to Evolville!

The Beginning

At the beginning there was nothing.
No big boom, no hydrogen, no neutrinos, nothing.
Only empty white world in Quil.

(require '[quil.core :as q])

(q/sketch
  :title "Evolville 0.0.1"
  :size [400 400]
  :setup #(q/background 255))

The Gods

The World has nothing, but it wasn’t empty.

The God of Setup managed to create world as empty map, so in future it can get terrain and be inhabited by creatures. Map is the best data structure, you know.

The world was not static either and I’m not talking about space extension. The world already had some fuzzy notion of time and was constantly updated by The God of Update couple of times per second.

The God of Draw was just sleeping. Why care if the world is empty?

(require '[quil.middleware :as m])

(def world {})
(def nothing (q/background 255))

(defn setup-world []
  (nothing)
  world)

(defn draw-world [world]
  (nothing))

(defn update-world [world] 
  world)

(q/sketch
  :title "Evolville 0.0.1"
  :size [400 400]
  :setup setup-world
  :draw draw-world
  :update update-world
  :middleware [m/fun-mode])

Obviosly, nothing happened so far.

First Creature

God of Setup was not satisfied with inactivity of the God of Draw.
He created a small creature in the middle of the world. So that God of Draw has no free time anymore.

(def world {:creatures [{:loc [200 200] :size 10}]})

(defn draw-world [world]
  (nothing)
  (doseq [creature (:creatures world)]
    (let [{:keys [loc size]} creature
          [x y] loc]
      (q/ellipse x y size size))))

Movement

Created creature seems dead, and the God of Setup gave it speed and direction to move (angle)

(def world {:creatures [{:loc [200 200] :size 10 :speed 1 :dir 45}]})

God of Update took this under his responsibility.

(defn move [creature]
  (let [{:keys [loc speed dir]} creature
        [x y] loc
        delta-x (* speed (q/cos (q/radians (- 360 dir))))
        delta-y (* speed (q/sin (q/radians (- 360 dir))))]
    (assoc creature :loc [(+ x delta-x) (+ y delta-y)])))

(defn move-creatures [world]
  (update world :creatures #(mapv move %)))

(defn update-world [world]
  (-> world
      (move-creatures)))

Unfortunately, our eyes can’t process the infinite world, so God of Update made a trick. Each time creature appears outside of the visible part of the world, it appears on the other side of universe.

Finite, but infinite world.

(defn- overflow-x [x]
  (cond (> x (q/width)) (overflow-x (- x (q/width)))
        (< x 0) (overflow-x (+ x (q/width)))
        :else x))

(defn- overflow-y [y]
  (cond (> y (q/height)) (overflow-y (- y (q/height)))
        (< y 0) (overflow-y (+ y (q/height)))
        :else y))

(defn move [creature]
  (let [{:keys [loc speed dir]} creature
        [x y] loc
        delta-x (* speed (q/cos (q/radians (- 360 dir))))
        delta-y (* speed (q/sin (q/radians (- 360 dir))))]
    (assoc creature :loc [(overflow-x (+ x delta-x)) 
                          (overflow-y (+ y delta-y))])))

Pack of them

In the next few million years, Gods observed how lonely was the single creature in the world. To make it happier, they created 300 random creatures. Each creature was born in random place, have random size, random speed and moves in random direction.

(defn make-creatures [n]
  (repeatedly n (fn []
                  {:loc [(q/random (q/width)) (q/random (q/height))]
                   :size (q/random 5 15)
                   :speed (q/random 1 3)
                   :dir (q/random 360)})))

(def world {:creatures (make-creatures 300)})

Now the world looks like prebiotic soup, where real things may happen.

To be continued…

Source

DFS on Binary Tree Array

2017-09-10T11:52:00+00:00

Implementing Depth-First Search for the Binary Tree without stack and recursion.

Binary Tree Array

This is binary tree.

0 is a root node.
0 has two children: left 1 and right: 2.
Each of its children have their children and so on.
The nodes without children are leaf nodes (3,4,5,6).

The number of edges between root and leaf nodes define tree height. The tree above has the height 2.

Standard approach to implement binary tree is to define class called BinaryNode

public class BinaryNode<T> {
    public T data;
    public BinaryNode<T> left;
    public BinaryNode<T> right;
}	

However, there is a trick to implement binary tree using only static arrays. If you enumerate all nodes in binary tree by levels starting from zero you can pack it into array.

The relationship between parent index and child index, allow us to move from the parent to child, as well as from the child to the parent

left_child(i)  = 2 * i + 1
right_child(i) = 2 * i + 2
parent(i)      = (i - 1) / 2

Important note that the tree should be complete, what means for the specific height it should contain exactly $2^{height + 1} - 1$ elements.

You can avoid such requirement by placing null instead of missing elements. The enumeration still applies for empty nodes as well. This should cover any possible binary tree.

Such array representation for binary tree we will call Binary Tree Array. We always assume it will contain $2^{n} - 1$ elements.

BFS vs DFS

There are two search algorithms exist for binary tree: breadth-first search (BFS) and depth-first search (DFS).

The best way to understand them is visually

BFS search nodes level by level, starting from the root node. Then checking its children. Then children for children and so on. Until all nodes are processed or node which satisfies search condition is found.

DFS behave differently. It checks all nodes from leftmost path from the root to the leaf, then jumps up and check right node and so on.

For classic binary trees BFS implemented using Queue data structure (LinkedList implements Queue)

public BinaryNode<T> bfs(Predicate<T> predicate) {
    Queue<BinaryNode<T>> queue = new LinkedList<>();
    queue.offer(this);
    while (!queue.isEmpty()) {
        BinaryNode<T> node = queue.poll();
        if (predicate.test(node.data)) return node;
        if (node.left != null) queue.offer(node.left);
        if (node.right != null) queue.offer(node.right);
    }
    return null;
}

If you replace queue with the stack you will get DFS implementation.

public BinaryNode<T> dfs(Predicate<T> predicate) {
    Stack<BinaryNode<T>> stack = new Stack<>();
    stack.push(this);
    while (!stack.isEmpty()) {
        BinaryNode<T> node = stack.pop();
        if (predicate.test(node.data)) return node;
        if (node.right != null) stack.push(node.right);
        if (node.left != null) stack.push(node.left);
    }
    return null;
}

For example, assume you need to find node with value 1 in the following tree.

Yellow cell are cells which are tested in search algorithm before needed node found. Take into account that for some cases DFS require less nodes for processing. For some cases it requires more.

BFS and DFS for Binary Tree Array

Breadth-first search for binary tree array is trivial. Since elements in binary tree array are placed level by level and bfs is also checking element level by level, bfs for binary tree array is just linear search

public int bfs(Predicate<T> predicate) {
    for (int i = 0; i < array.length; i++) {
        if (array[i] != null && predicate.test(array[i])) return i;
    }
    return -1; // not found
}

Depth-first search is a bit harder.

Using stack data structure it could be implemented the same way as for classic binary tree, just put indices into the stack.

From this point recursion is not different at all, you just use implicit method call stack instead of data structure stack.

What if we could implement DFS without stack and recursion. In the end is just a selecting correct index on each step?

Well, let’s try.

DFS for Binary Tree Array (without stack and recursion)

As in BFS we start from the root index 0.
Then we need to inspect its left child, which index could be calculated by formula 2*i+1.
Let’s call process of going to next left child doubling.

We continue doubling the index until we don’t stop at the leaf node.

So how can we understand it’s a leaf node?

If next left child produce index out of array bounds exception we need to stop doubling and jump to the right child.

That’s a valid solution, but we can avoid unnesessary doubling. The observation here is the number of elements in the tree of some height is two times larger (and plus one) than the tree with height - 1.

For tree with height = 2 we have 7 elements, for tree with height = 3 we have 7*2+1 = 15 elements.

Using this observation we build simple condition

i < array.length / 2

You can check this condition is valid for any non-leaf node in the tree.

Nice so far. We can run doubling, and we even know when to stop. But this way we can check only left children of the nodes.

When we stuck at the leaf node and can not apply doubling anymore, we perform jump operation. Basically, if doubling is go to left child, jump is go to parent and go to right child.

Such jump operation can be implemented by just incrementing the leaf index. But when we stuck at the next leaf node, increment doesn’t move node to the next node, required for DFS. So we need to do double jump

Make it more generic, for our algorithm we could define operation jump N, which specifies how many jumps to the parent it requires and always finishes with one jump to the right child.

For the tree with height 1, we need one jump, jump 1

For the tree with height 2, we need one jump, one double jump, and than one jump again

For the tree with height 3, you can apply the same rules and get 7 jumps.

See the pattern?

\[Jumps_{height} = Jumps_{height-1} \space{Height}\space Jumps_{height-1}\]

If you apply the same rule to the tree of height 4 and write down only jump values you will get next sequence number

1 2 1 3 1 2 1 4 1 2 1 3 1 2 1

This sequence known as Zimin words and it has recursive definiton, which is not great, because we not allowed to use recursion.

Let’s find some another way to calculate it.

If you look carefully at the jump sequence, you can see non recursive pattern.

We can track index for current leaf j, which corresponds to the index in sequence array above.

To verify statement like “Every 8th element starting from 3” we just check j % 8 == 3, if it’s true three jumps to the parent node required.

Iterative jumping process will be following

Start with k = 1
Jump once to the parent by applying i = (i - 1)/2
If leaf % 2*k == k - 1 all jumps performed, exit
Otherwise, set k to 2*k and repeat.

If you putting all together what we’ve discussed

public int dfs(Predicate<T> predicate) {
    int i = 0;
    int leaf = 0;
    while (i < array.length) {
        if (array[i] != null && predicate.test(array[i])) return i; // node found
        if (i < array.length / 2) { // not leaf node, can be advanced
            i = 2 * i + 1; // try left child
        } else { // leaf node, jump
            int k = 1;
            while (true) {
                i = (i - 1) / 2; // jump to the parent
                int p = k * 2;
                if (leaf % p == k - 1) break; // correct number of jumps found
                k = p;
            }
            // after we jumped to the parent, go to the right child
            i = 2 * i + 2;
            leaf++; // next leaf, please
        }
    }
    return -1;
}

Done. DFS for binary tree array without stack and recursion

Complexity

The complexity looks like quadratic because we have loop in a loop, but let’s see what reality is.

For half of the total nodes (n/2) we use doubling strategy, which is constant O(1)
For other half of the total nodes (n/2) we use leaf jumping strategy.
Half of the leaf jumping nodes (n/4) require only one parent jump, 1 iteration. O(1)
Half from this half (n/8) require two jumps, or 2 iterations. O(1)
…
Only one leaf jumping node require full jump to the root, which is O(logN)
So, if all cases has O(1) except one which is O(logN), we can say the average for jump is O(1)
Total complexity is O(n)

Performance

Although theory is good, let’s see how this algorithm works in practice, comparing to others.

We test 5 algorithms, using JMH.

BFS on Binary Tree (linked list as queue)
BFS on Binary Tree Array (linear search)
DFS on Binary Tree (stack)
DFS on Binary Tree Array (stack)
Iterative DFS on Binary Tree Array

We’ve precreated a tree and using predicate to match the last node, to make sure we always hit the same number of nodes for all algorithms.

So, results

Benchmark                                Mode  Cnt         Score        Error  Units
------------------------------------------------------------------------------------
BinaryTreeBenchmarks.bfsOnArray         thrpt  200  91378159.672 ± 625885.079  ops/s
BinaryTreeBenchmarks.bfsOnBinaryTree    thrpt  200   7892439.856 ±  84124.694  ops/s
BinaryTreeBenchmarks.dfsOnArray         thrpt  200  19980835.346 ± 209143.115  ops/s
BinaryTreeBenchmarks.dfsOnBinaryTree    thrpt  200  13615106.113 ±  91863.833  ops/s
BinaryTreeBenchmarks.dfsOnStackOnArray  thrpt  200  11643533.145 ±  87018.132  ops/s

Obviously, BFS on array wins. It’s just a linear search, so if you can represent binary tree as array, do it.

Iterative DFS, which we just described in the article, took 2nd place and 4.5 times slower then linear search (BFS on array)

The rest three algorithms even slower then iterative DFS, probably, because they involve additional data structures, extra space and memory allocation.

And the worst case is BFS on Binary Tree. I suspect here the linked list is a problem.

Conclusion

Binary Tree Array is a nice trick to represent hierarchical data structure as a static array (contiguous region in the memory). It defines a simple index relationship for moving from parent to child, as well as moving from child to parent, which is not possible with classic binary array representation. Some drawbacks, like slow insertion/deletion exists, but I’m sure there are plenty applications for binary tree array.

As we have shown, search algorithms, BFS and DFS implemented on arrays are much faster then their brothers on a reference based structure. For real world problems it’s not so important, and, probably, I still will use the reference based binary array.

Anyway, it was a nice excercise. Expect question “DFS on binary array without recursion and stack” on your next interview.

Clojure Euler: Problem 026

2017-05-09T12:19:00+00:00

Find the value of d < 1000 for which 1/d contains the longest recurring cycle in its decimal fraction part.

Permalink: https://projecteuler.net/problem=26

1/2 = 0.5
1/3 = 0.(3)
1/4 = 0.25
1/5 = 0.2
1/6 = 0.1(6)
1/7 = 0.(142857)
1/8 = 0.125
1/9 = 0.(1)
1/10 = 0.1

As you see for d from 2 to 10, the longest cycle length is 6, for d=7.

Let’s start with a function, which finds the cycle length for given d

(defn unit-fraction [denom]
  (loop [numer 1 i 1 known {}]
    (let [r (rem (* 10 numer) denom)]
      (cond (zero? r) 0
            (get known r) (- i (get known r))
            :else (recur r (inc i) (assoc known r i))))))

Optimal algorithm is a little bit tricky.

In simplest case, we need to find a cycle for 1/4. Calculate next decimal digit (we index digits number with i variable), by finding quotient. 1/4 does not produce integer number, so advance decimal index (result becomes 0. something) and multiply 1 by 10. Quotient 10/4=2 and remainder is 2 (result becomes 0.2 something). Again 2/4 does not produce integer number, so advance index and multiply by 10. Quotient of 20/4=5, without remainder, so whole result is equal to 0.25, no cycle detected.

For d, which contain cycles we introduce known map to store all previous numbers. Assume d is 11, starting with remainder 1 we don’t have integer result (result so far 0.) and we store the fact, for remainder=1, index=1. Proceeding, we multiple remainder by 10, but 10/11 does not produce integer result as well (result so far 0.0) and store remainder=10, index=2. Proceeding, For third index 100/11=9, with remainder 1. But remainder 1 has been seen before, at index 1, so all subsequent calculation will produce the cycle. And cycle length is difference between current index and index seen before, 3-1=2, and as result we have 0.(09) with cycle length 2.

Finally, just map this function to all denominators and find the one which will produce max result.

(->> (range 1 1000)
     (map #(vec [% (unit-fraction %)]))
     (apply max-key second)
     (first))

Easy.

Code

Code Golf: New Year Tree

2016-01-07T14:05:00+00:00

Let’s celebrate geeky New Year by drawing New Year Tree using Clojure!

Problem

Write a program which prints New Year Tree to console.
Source code should take as small number of characters as possible.

I came up with the following tree rules:

Star. There is should be a star on top of the tree.
Triangles. Tree body is a set of triangle levels. Let the number of levels will be an input argument to the program. Next level height is a previous level height plus one.
Decorations. Yellow squares is a possible place for decorations (place ‘o’ instead of empty space). Let them appear randomly with the specified probability ([0, 1])
Stem. And finally, every tree should have a stem at the bottom.

So, the function signature is following:

(fn [level ratio])

Code

(fn[l r](let[p print n println e range m(reduce + 2(e(+ l 1)))
f(fn[w n](dotimes[i n](p(if(fn? w)(w)w))))d #(f(fn[]
(if(<(rand)r)\o\ ))%)](f" "m)(n\★)(doseq[b(e l)h(e(+ b 3))
:let[u(quot(*(+ b 1)b)2)a(=(+ b 2)h)t(+ h h u u 1)]]
(f\ (- m h u 1))(p\/)(cond(and(=(+ b 1)l)a)(f"_"t)
a(do(p\_)(d(- t 2))(p\_))1(d t))(n\\))
(dotimes[i 2](f\ (- m 1))(n"| |"))))

348 characters in Clojure.

Demo

P.S. Later I realized we could reuse symmetry property of the tree to make program more compact.

Clojure Design Patterns

2015-12-18T14:20:00+00:00

Quick overview of the classic Design Patterns in Clojure

Disclaimer: Most patterns are easy to implement because we use dynamic typing, functional programming and, of course, Clojure. Some of them look wrong and ugly. It’s okay. All characters are fake, coincidences are accidental.

Index

Intro
Episode 1. Command
Episode 2. Strategy
Episode 3. State
Episode 4. Visitor
Episode 5. Template Method
Episode 6. Iterator
Episode 7. Memento
Episode 8. Prototype
Episode 9. Mediator
Episode 10. Observer
Episode 11. Interpreter
Episode 12. Flyweight
Episode 13. Builder
Episode 14. Facade
Episode 15. Singleton
Episode 16. Chain of Responsibility
Episode 17. Composite
Episode 18. Factory Method
Episode 19. Abstract Factory
Episode 20. Adapter
Episode 21. Decorator
Episode 22. Proxy
Episode 23. Bridge
Cheatsheet
Cast

Intro

Our programming language is fucked up.
That’s why we need design patterns.

– Anonymous

Two modest programmers Pedro Veel and Eve Dopler are solving common software engineering problems and applying design patterns.

Episode 1. Command

Leading IT service provider “Serpent Hill & R.E.E” acquired new project for USA customer. First delivery is a register, login and logout functionality for their brand new site.

Pedro: Oh, that’s easy. You just need a Command interface…

interface Command {
  void execute();
}

Pedro: Every action should implement this interface and define specific execute behaviour.

public class LoginCommand implements Command {

  private String user;
  private String password;

  public LoginCommand(String user, String password) {
    this.user = user;
    this.password = password;
  }

  @Override
  public void execute() {
    DB.login(user, password);
  }
}

public class LogoutCommand implements Command {

  private String user;

  public LogoutCommand(String user) {
    this.user = user;
  }

  @Override
  public void execute() {
    DB.logout(user);
  }
}

Pedro: Usage is simple as well.

(new LoginCommand("django", "unCh@1ned")).execute();
(new LogoutCommand("django")).execute();

Pedro: What do you think, Eve?
Eve: Why are you using redundant wrapping into LoginCommand and just don’t call DB.login?
Pedro: It’s important to wrap here, because now we can operate on generic Command objects.
Eve: For what purpose?
Pedro: Delayed call, logging, history tracking, caching, plenty of usages.
Eve: Ok, how about that?

(defn execute [command]
  (command))

(execute #(db/login "django" "unCh@1ned"))
(execute #(db/logout "django"))

Pedro: What the hell this hash sign?
Eve: A shortcut for javaish

new SomeInterfaceWithOneMethod() {
  @Override
  public void execute() {
    // do
  }
};

Pedro: Just like Command interface…
Eve: Or if you want - no-hash solution.

(defn execute [command & args]
  (apply command args))

(execute db/login "django" "unCh@1ned")

Pedro: And how do you save function for delayed call in that case?
Eve: Answer yourself. What do you need to call a function?
Pedro: Its name…
Eve: And?
Pedro: …arguments.
Eve: Bingo. All you do is saving a pair (function-name, arguments) and call it whenever you want using (apply function-name arguments)
Pedro: Hmm… Looks simple.
Eve: Definitely, Command is just a function.

Episode 2. Strategy

Sven Tori pays a lot of money to see a page with list of users. But users must be sorted by name and users with subscription must appear before all other users. Obviously, because they pay. Reverse sorting should keep subscripted users on top.

Pedro: Ha, just call Collections.sort(users, comparator) with custom comparator.
Eve: How would you implement custom comparator?
Pedro: You need to take Comparator interface and provide implementation for compare(Object o1, Object o2) method. Also you need another implementation for ReverseComparator
Eve: Stop talking, show me the code!

class SubsComparator implements Comparator<User> {

  @Override
  public int compare(User u1, User u2) {
    if (u1.isSubscription() == u2.isSubscription()) {
      return u1.getName().compareTo(u2.getName());
    } else if (u1.isSubscription()) {
      return -1;
    } else {
      return 1;
    }
  }
}

class ReverseSubsComparator implements Comparator<User> {

  @Override
  public int compare(User u1, User u2) {
    if (u1.isSubscription() == u2.isSubscription()) {
      return u2.getName().compareTo(u1.getName());
    } else if (u1.isSubscription()) {
      return -1;
    } else {
      return 1;
    }
  }
}

// forward sort
Collections.sort(users, new SubsComparator());

// reverse sort
Collections.sort(users, new ReverseSubsComparator());

Pedro: Could you do the same?
Eve: Yeah, something like that

(sort (comparator 
       (fn [u1 u2]
         (cond
          (= (:subscription u1)
             (:subscription u2)) (neg? (compare (:name u1)
                                                (:name u2)))
             (:subscription u1) true
             :else false))) users)

Pedro: Pretty similar.
Eve: But we can do it better

;; forward sort
(sort-by (juxt (complement :subscription) :name) users)

;; reverse sort
(sort-by (juxt :subscription :name) #(compare %2 %1) users)

Pedro: Oh my gut! Monstrous oneliners.
Eve: Functions, you know.
Pedro: Whatever, it’s very hard to understand what’s happening there.

Eve explains juxt, complement and sort-by
10 minutes later

Pedro: Very doubtful approach to pass strategy.
Eve: I don’t care, because Strategy is just a function passed to another function.

Episode 3. State

Sales person Karmen Git investigated the market and decided to provide user-specific functionality.

Pedro: Smooth requirements.
Eve: Let’s clarify them.

If user has subscription show him all news in a feed
Otherwise, show him only recent 10 news
If he pays money, add the amount to his account balance
If user doesn’t have subscription and there is enough money to buy subscription, change his state to…

Pedro: State! Awesome pattern. First we make a user state enum

public enum UserState {
  SUBSCRIPTION(Integer.MAX_VALUE),
  NO_SUBSCRIPTION(10);

  private int newsLimit;

  UserState(int newsLimit) {
    this.newsLimit = newsLimit;
  }

  public int getNewsLimit() {
    return newsLimit;
  }
}

Pedro: User logic is following

public class User {
  private int money = 0;
  private UserState state = UserState.NO_SUBSCRIPTION;
  private final static int SUBSCRIPTION_COST = 30;

  public List<News> newsFeed() {
    return DB.getNews(state.getNewsLimit());
  }

  public void pay(int money) {
    this.money += money;
    if (state == UserState.NO_SUBSCRIPTION
	    && this.money >= SUBSCRIPTION_COST) {
      // buy subscription
      state = UserState.SUBSCRIPTION;
      this.money -= SUBSCRIPTION_COST;
    }
  }
}

Pedro: Lets call it

User user = new User(); // create default user
user.newsFeed(); // show him top 10 news
user.pay(10); // balance changed, not enough for subs
user.newsFeed(); // still top 10
user.pay(25); // balance enough to apply subscription
user.newsFeed(); // show him all news

Eve: You just hide value that affects behaviour inside User object. We could use strategy to pass it directly user.newsFeed(subscriptionType).
Pedro: Agreed, State is very close to the Strategy. They even have the same UML diagrams. but we encapsulate balance and bind it to user.
Eve: I think it achieves the same goal using another mechanism. Instead of providing strategy explicitly, it depends on some state. From clojure perspective it can be implemented the same way as strategy pattern.
Pedro: But successive calls can change object’s state.
Eve: Correct, but it has nothing to do with Strategy it is just implementation detail.
Pedro: What about “another mechanism”?
Eve: Multimethods.
Pedro: Multi what?
Eve: Look at this

(defmulti news-feed :user-state)

(defmethod news-feed :subscription [user]
  (db/news-feed))

(defmethod news-feed :no-subscription [user]
  (take 10 (db/news-feed)))

Eve: And pay function it’s just a plain function, which changes state of object. We don’t like state too much in clojure, but if you wish.

(def user (atom {:name "Jackie Brown"
                 :balance 0
                 :user-state :no-subscription}))
				 
(def ^:const SUBSCRIPTION_COST 30)

(defn pay [user amount]
  (swap! user update-in [:balance] + amount)
  (when (and (>= (:balance @user) SUBSCRIPTION_COST)
             (= :no-subscription (:user-state @user)))
    (swap! user assoc :user-state :subscription)
    (swap! user update-in [:balance] - SUBSCRIPTION_COST)))

(news-feed @user) ;; top 10
(pay user 10)
(news-feed @user) ;; top 10
(pay user 25)
(news-feed @user) ;; all news

Pedro: Is dispatching by multimethods better than dispatching by enum?
Eve: No, in this particlular case, but in general yes.
Pedro: Explain, please
Eve: Do you know what double dispatch is?
Pedro: Not sure.
Eve: Well, it is topic for Visitor pattern.

Episode 4. Visitor

Natanius S. Selbys suggested to implement functionality which allows users export their messages, activities and achievements in different formats.

Eve: So, how do you plan to do it?
Pedro: We have one hierarchy for item types (Message, Activity) and another for file formats (PDF, XML)

abstract class Format { }
class PDF extends Format { }
class XML extends Format { }

public abstract class Item {
  void export(Format f) {
    throw new UnknownFormatException(f);
  }
  abstract void export(PDF pdf);
  abstract void export(XML xml);
}

class Message extends Item {
  @Override
  void export(PDF f) {
    PDFExporter.export(this);
  }

  @Override
  void export(XML xml) {
    XMLExporter.export(this);
  }
}

class Activity extends Item {
  @Override
  void export(PDF pdf) {
    PDFExporter.export(this);
  }

  @Override
  void export(XML xml) {
    XMLExporter.export(this);
  }
}

Pedro: That’s all.
Eve: Nice, but how do you dispatch on argument type?
Pedro: What the problem?
Eve: Consider this snippet

Item i = new Activity();
Format f = new PDF();
i.export(f);

Pedro: Nothing suspicious here.
Eve: Actually, if you run this code you get UnknownFormatException
Pedro: Wait…Really?
Eve: In java you can use only single dispatch. That means if you call i.export(f) you dispatches on the actual type of i, not f.
Pedro: I’m surprised. So, there is no dispatch on argument type?
Eve: That’s what visitor hack for. After you got a dispatch on i type, you additionally call f.someMethod(i) and dispatched on f type.
Pedro: How that looks in code?
Eve: You separately define export operations for all types as a Visitor

public interface Visitor {
  void visit(Activity a);
  void visit(Message m);
}

public class PDFVisitor implements Visitor {
  @Override
  public void visit(Activity a) {
    PDFExporter.export(a);
  }

  @Override
  public void visit(Message m) {
    PDFExporter.export(m);
  }
}

Eve: Your items change signature to accept different visitors.

public abstract class Item {
  abstract void accept(Visitor v);
}

class Message extends Item {
  @Override
  void accept(Visitor v) {
    v.visit(this);
  }
}

class Activity extends Item {
  @Override
  void accept(Visitor v) {
    v.visit(this);
  }
}

Eve: To use it you may call

Item i = new Message();
Visitor v = new PDFVisitor(); 
i.accept(v);

Eve: And everything works fine. Moreover, you can add new operations for activities and messages by just defining new visitors and without changing their code.
Pedro: That’s really useful. But implementation is tough, it is the same for clojure?
Eve: Not really, clojure supports it natively via multimethods
Pedro: Multi what?
Eve: Just follow the code… First we define dispatcher function

(defmulti export
  (fn [item format] [(:type item) format]))

Eve: It accepts item and format to be exported. Examples:

;; Message
{:type :message :content "Say what again!"}
;; Activity
{:type :activity :content "Quoting Ezekiel 25:17"}
;; Formats
:pdf, :xml

Eve: And now you just provide a functions for different combinatations, and dispatcher decide which one to call.

(defmethod export [:activity :pdf] [item format]
  (exporter/activity->pdf item))

(defmethod export [:activity :xml] [item format]
  (exporter/activity->xml item))

(defmethod export [:message :pdf] [item format]
  (exporter/message->pdf item))

(defmethod export [:message :xml] [item format]
  (exporter/message->xml item))

Pedro: What if unknown format passed?
Eve: We could specify default dipatcher function.

(defmethod export :default [item format]
  (throw (IllegalArgumentException. "not supported")))

Pedro: Ok, but there is no hierarchy for :pdf and :xml. They are just keywords?
Eve: Correct, simple problem - simple solution. If you need advanced features, you could use adhoc hierarchies or dispatch by class.

(derive ::pdf ::format)
(derive ::xml ::format)

Pedro: Quadrocolons?!
Eve: Assume they are just keywords.
Pedro: Ok.
Eve: Then you add functions for every dispatch type ::pdf, ::xml and ::format

(defmethod export [:activity ::pdf])
(defmethod export [:activity ::xml])
(defmethod export [:activity ::format])

Eve: If some new format (i.e. csv) appears in the system

(derive ::csv ::format)

Eve: It will be dispatched to ::format function, until you add a separate ::csv function.
Pedro: Seems good.
Eve: Definitely, much easier.
Pedro: So, basically, if a language support multiple dispatch, you don’t need Visitor pattern?
Eve: Exactly.

Episode 5. Template Method

MMORPG Mech Dominore Fight Saga requested to implement a game bot for their VIP users. Not fair.

Pedro: First, we must decide what actions should be automated with bot.
Eve: Have you ever played RPG?
Pedro: Fortunately, no
Eve: Oh my… Let’s go, I’ll show you…

2 weeks later

Pedro: …fantastic, I found epic sword, which has +100 attack.
Eve: Unbelievable. But now, it’s time for bot.
Pedro: Easy-peasy. We could select following events

Battle
Quest
Open Chest

Pedro: Characters behave differently in different events, for example mages cast spells in battle, but rogues prefer silent melee combat; locked chests are skipped by most characters, but rogues can unlock them, etc.
Eve: Looks like ideal candidate for Template Method?
Pedro: Yes. We define abstract algorithm, and then specify differences in subclasses.

public abstract class Character {
  void moveTo(Location loc) {
    if (loc.isQuestAvailable()) {
      Journal.addQuest(loc.getQuest());
    } else if (loc.containsChest()) {
      handleChest(loc.getChest());
    } else if (loc.hasEnemies()) {
      attack(loc.getEnemies());
    }
    moveTo(loc.getNextLocation());
  }

  private void handleChest(Chest chest) {
    if (!chest.isLocked()) {
      chest.open();
    } else {
      handleLockedChest(chest);
    }
  }

  abstract void handleLockedChest(Chest chest);
  abstract void attack(List<Enemy> enemies);
}

Pedro: We’ve separated to Character class everything common to all characters. Now we can create subclasses, that define how character should behave in specific situation. In out case: handling locked chests and attacking enemies.
Eve: Let’s start with a Mage class.
Pedro: Mage? Okay. He can’t open locked chest, so implementation is just do nothing. And if he is attacking enemies, if there are more than 10 enemies, freeze them, and cast teleport to run away. If there are 10 enemies or less cast fireball on each of them.

public class MageCharacter extends Character {
  @Override
  void handleLockedChest(Chest chest) {
    // do nothing
  }

  @Override
  void attack(List<Enemy> enemies) {
    if (enemies.size() > 10) {
      castSpell("Freeze Nova");
      castSpell("Teleport");
    } else {
      for (Enemy e : enemies) {
        castSpell("Fireball", e);
      }
    }
  }
}

Eve: Excellent, what about Rogue class?
Pedro: Easy as well, rogues can unlock chests and prefer silent combat, handle enemies one by one.

public class RogueCharacter extends Character {
  @Override
  void handleLockedChest(Chest chest) {
    chest.unlock();
  }

  @Override
  void attack(List<Enemy> enemies) {
    for (Enemy e : enemies) {
      invisibility();
	  attack("backstab", e);
    }
  }
}

Eve: Excellent. But how this approach is differrent from Strategy?
Pedro: What?
Eve: I mean, you redefined behaviour by using subclasses, but in Strategy pattern you did the same: redefined behaviour by using functions.
Pedro: Well, another approach.
Eve: State was handled with another approach as well.
Pedro: What are you trying to say?
Eve: You are solving the same kind of problem, but change the approach to it.
Pedro: How do yo solve this problem using strategy in clojure?
Eve: Just pass a set of specific functions for each character. For example, your abstract move may look like:

(defn move-to [character location]
  (cond
   (quest? location)
   (journal/add-quest (:quest location))

   (chest? location)
   (handle-chest (:chest location))

   (enemies? location)
   (attack (:enemies location)))
  (move-to character (:next-location location)))

Eve: To add character-specific implementation of methods handle-chest and attack, implement them and pass as an argument.

;; Mage-specific actions
(defn mage-handle-chest [chest])

(defn mage-attack [enemies]
  (if (> (count enemies) 10)
    (do (cast-spell "Freeze Nova")
        (cast-spell "Teleport"))
    ;; otherwise
    (doseq [e enemies]
      (cast-spell "Fireball" e))))

;; Signature of move-to will change to

(defn move-to [character location
               & {:keys [handle-chest attack]
                  :or {handle-chest (fn [chest])
                       attack (fn [enemies] (run-away))}}]
  ;; previous implementation
)

Pedro: OMG, what’s happening there?
Eve: We changed signature of move-to to accept handle-chest and attack functions.
Think of them like optional parameters.

(move-to character location
  :handle-chest mage-handle-chest
  :attack       mage-attack)

Eve: Keep in mind that if these functions are not provided we use default behavior: do nothing for handle-chest and run away from enemies in attack
Pedro: Fine, but is this better than approach by subclassing? Seems that we have a lot of redundant information in move-to call.
Eve: It’s fixable, just define this call once, and give it alias

(defn mage-move [character location]
  (move-to character location
    :handle-chest mage-handle-chest
    :attack       mage-attack))

Eve: Or use multimethods, it’s even better.

(defmulti move 
  (fn [character location] (:class character)))

(defmethod move :mage [character location]
  (move-to character location
    :handle-chest mage-handle-chest
    :attack       mage-attack))

Pedro: I understand. But why do you think pass as argument is better than subclassing?
Eve: You can change behaviour dynamically. Assume your mage has no mana, so instead of trying to cast fireballs, he can just teleport and run away, you just provide new function.
Pedro: Makes sense. Functions everywhere.

Episode 6. Iterator

Technical consultant Kent Podiololis complains for C-style loops usage.

“Are we in 1980 or what?” – Kent

Pedro: We definitely should use pattern Iterator from java.
Eve: Don’t be fool, nobody’s using java.util.Iterator
Pedro: Everybody use it implicitly in for-each loop. It’s a good way to traverse a container.
Eve: What does it mean “to traverse a container”?
Pedro: Formally, the container should provide two methods for you:
next() to return next element and hasNext() to return true if container has more elements.
Eve: Ok. Do you know what linked list is?
Pedro: Singly linked list?
Eve: Singly linked list.
Pedro: Sure. It is a container consists of nodes. Each node has a data value and reference to the next node. And null value if there is no next node.
Eve: Correct. Now tell me how traversing such list is differ from traversing via iterator?
Pedro: Emmm…

Pedro wrote two traversing snippets:

Traversing using iterator

Iterator i;
while (i.hasNext()) {
  i.next();
}

Traversing using linked list

Node next = root;
while (next != null) {
  next = next.next;
}

Pedro: They are pretty similar…What is analogue of Iterator in clojure?
Eve: seq function.

(seq [1 2 3])       => (1 2 3)
(seq (list 4 5 6))  => (4 5 6)
(seq #{7 8 9})      => (7 8 9)
(seq (int-array 3)) => (0 0 0)
(seq "abc")         => (\a \b \c)

Pedro: It returns a list…
Eve: Sequence, because Iterator is just a sequence
Pedro: Is it possible to make seq works on custom datastructures?
Eve: Implement clojure.lang.Seqable interface

(deftype RedGreenBlackTree [& elems]
  clojure.lang.Seqable
  (seq [self]
	;; traverse element in needed order
	))

Pedro: Fine then. But I’ve heard iterator is often used to achive laziness, for example to calculate value only during getNext() call, how list handle that?
Eve: List can be lazy as well, clojure calls such list “lazy sequence”.

(def natural-numbers (iterate inc 1))

Eve: We defined thing to represent ALL natural numbers, but we haven’t got OutOfMemory yet, because we haven’t requested any value. It’s lazy.
Pedro: Could you explain more?
Eve: Unfortunately, I am too lazy for that.
Pedro: I will remember that!

Episode 7: Memento

User Chad Bogue lost the message he was writing for two days. Implement save button for him.

Pedro: I don’t believe there are people who can type in textbox for two days. Two. Days.
Eve: Let’s save him.
Pedro: I googled this problem. Most popular approach to implement save button is Memento pattern. You need originator, caretaker and memento objects.
Eve: What’s that?
Pedro: Originator is just an object or state that we want to preserve. (text inside a textbox), caretaker is responsible to save state (save button) and memento is just an object to encapsulate state.

public class TextBox {
  // state for memento
  private String text = "";

  // state not handled by memento
  private int width = 100;
  private Color textColor = Color.BLACK;

  public void type(String s) {
    text += s;
  }

  public Memento save() {
    return new Memento(text);
  }

  public void restore(Memento m) {
    this.text = m.getText();
  }

  @Override
  public String toString() {
    return "[" + text + "]";
  }
}

Pedro: Memento is just an immutable object

public final class Memento {
  private final String text;

  public Memento(String text) {
    this.text = text;
  }

  public String getText() {
    return text;
  }
}

Pedro: And caretaker is a demo

// open browser, init empty textbox
TextBox textbox = new TextBox();

// type something into it
textbox.type("Dear, Madonna\n");
textbox.type("Let me tell you what ");

// press button save
Memento checkpoint1 = textbox.save();

// type again
textbox.type("song 'Like A Virgin' is about. ");
textbox.type("It's all about a girl...");

// suddenly browser crashed, restart it, reinit textbox
textbox = new TextBox();

// but it's empty! All work is gone!
// not really, you rollback to last checkpoint
textbox.restore(checkpoint1);

Pedro: Just a note if you want a multiple checkpoints, save memento’s to the list.
Eve: Originator, caretaker, memento - looks as a bunch of nouns, but actually it’s all about two functions save and restore.

(def textbox (atom {}))

(defn init-textbox [] 
 (reset! textbox {:text ""
                  :color :BLACK
                  :width 100}))

(def memento (atom nil))

(defn type-text [text]
  (swap! textbox
    (fn [m]
      (update-in m [:text] (fn [s] (str s text))))))

(defn save []
  (reset! memento (:text @textbox)))

(defn restore []
  (swap! textbox assoc :text @memento))

Eve: And demo as well.

(init-textbox)
(type-text "'Like A Virgin' ")
(type-text "it's not about this sensitive girl ")
(save)
(type-text "who meets nice fella")
;; crash
(init-textbox)
(restore)

Pedro: It’s pretty the same code.
Eve: Yes, but you must care about memento immutability
Pedro: What does it mean?
Eve: You are lucky, that you got String object in this example, String is immutable. But if you have something, that may change its internal state, you need to perform deep copy of this object for memento.
Pedro: Oh, right. It’s just a recursive clone() calls to obtain prototype.
Eve: We will talk about Prototype in a minute, but just remember that Memento is not about caretaker and originator, it is about save and restore.

Episode 8: Prototype

Dex Ringeus detected that users feel uncomfortable with registration form. Make it more usable.

Pedro: So, what’s the problem with the registration?
Eve: There are lot of fields users bored to type in.
Pedro: For example?
Eve: For example, weight. Having such field scares 90% of female users.
Pedro: But this field is important for our analytics system, we make food and clothes recomendations based on that field.
Eve: Then, make it optional, and if it is not provided, take some default value.
Pedro: 60 kg is ok?
Eve: I think so.
Pedro: Ok, give me two minutes.

2 hours later

Pedro: I suggest to use some registration prototype which has all fields are filled with default values. After user completes the form we modify filled values.
Eve: Sounds great.
Pedro: Here it is our standard registration form, with prototype in clone() method.

public class RegistrationForm implements Cloneable {
  private String name = "Zed";
  private String email = "zzzed@gmail.com";
  private Date dateOfBirth = new Date(1970, 1, 1);
  private int weight = 60;
  private Gender gender = Gender.MALE;
  private Status status = Status.SINGLE;
  private List<Child> children = Arrays.asList(new Child(Gender.FEMALE));
  private double monthSalary = 1000;
  private List<Brand> favouriteBrands = Arrays.asList("Adidas", "GAP");
  // few hundreds more properties

  @Override
  protected RegistrationForm clone() throws CloneNotSupportedException {
    RegistrationForm prototyped = new RegistrationForm();
      prototyped.name = name;
      prototyped.email = email;
      prototyped.dateOfBirth = (Date)dateOfBirth.clone();
      prototyped.weight = weight;
      prototyped.status = status;
      List<Child> childrenCopy = new ArrayList<Child>();
      for (Child c : children) {
        childrenCopy.add(c.clone());
      }
      prototyped.children = childrenCopy;
      prototyped.monthSalary = monthSalary;
      List<String> brandsCopy = new ArrayList<String>();
      for (String s : favouriteBrands) {
        brandsCopy.add(s);
      }
      prototyped.favouriteBrands = brandsCopy;
    return  prototyped;
  }
}

Pedro: Every time we create a user, call clone() and then override needed properties.
Eve: Awful! In mutable world clone() is needed to create new object with the same properties. The hard part is the copy must be deep, i.e. instead of copying reference you need recursively clone() other objects, and what if one of them doesn’t have clone()…
Pedro: That’s the problem and this pattern solves it.
Eve: I don’t think it is a solution if you need to implement clone every time you adding new object.
Pedro: How clojure avoid this?
Eve: Clojure has immutable data structures. That’s all.
Pedro: How does it solve prototype problem?
Eve: Every time you modify object, you get a fresh new immutable copy of your data, and old one is not changed. Prototype is not needed in immutable world

(def registration-prototype
	 {:name          "Zed"
	  :email         "zzzed@gmail.com"
	  :date-of-birth "1970-01-01"
	  :weight        60
	  :gender        :male
	  :status        :single
	  :children      [{:gender :female}]
	  :month-salary  1000
	  :brands        ["Adidas" "GAP"]})

;; return new object
(assoc registration-prototype 
	 :name "Mia Vallace"
	 :email "tomato@gmail.com"
	 :weight 52
	 :gender :female
	 :month-salary 0)

Pedro: Great! But how this affects performance?
Copying million rows each time you adding new value seems time consuming operation.
Eve: No, it is not. Go to Google and search for persistent data structures and structural sharing
Pedro: Thanks a lot.

Episode 9: Mediator

Recently performed external code review shows a lot of issues with current codebase. Veerco Wierde emphasizes tight coupling in chat application.

Eve: What is tight coupling?
Pedro: It’s the problem when objects know too much about each other.
Eve: Could you be more specific?
Pedro: Look at the current chat implementation

public class User {
  private String name;
  List<User> users = new ArrayList<User>();

  public User(String name) {
    this.name = name;
  }

  public void addUser(User u) {
    users.add(u);
  }

  void sendMessage(String message) {
    String text = String.format("%s: %s\n", name, message);
    for (User u : users) {
      u.receive(text);
    }
  }

  private void receive(String message) {
    // process message
  }
}

Pedro: The problem here is the user knows everything about other users. It is very hard to use and maintain such code. When new user connects to the chat, you must add a reference to him via addUser for every existing user.
Eve: So, we just move one piece of responsibility to another class?
Pedro: Yes, kind of. We create mega-aware class, called mediator, that binds all parts together. Obviously, each part knows only about mediator.

public class User {
  String name;
  private Mediator m;

  public User(String name, Mediator m) {
    this.name = name;
    this.m = m;
  }

  public void sendMessage(String text) {
    m.sendMessage(this, text);
  }

  public void receive(String text) {
    // process message
  }
}

public class Mediator {

  List<User> users = new ArrayList<User>();

  public void addUser(User u) {
    users.add(u);
  }

  public void sendMessage(User u, String text) {
    for (User user : users) {
      u.receive(text);
    }
  }
}

Eve: That seems like a simple refactoring problem.
Pedro: Profit may be underestimated, but if you have hundreds of mutually connected components (UI for example) mediator is really a savior.
Eve: Agreed.
Pedro: Now the clojure turn.
Eve: Ok…let’s look…your mediator is responsible for saving users and sending messages

(def mediator
  (atom {:users []
         :send (fn [users text]
			     (map #(receive % text) users))}))

(defn add-user [u]
	(swap! mediator 
	  (fn [m]
	    (update-in m [:users] conj u))))

(defn send-message [u text]
	(let [send-fn (:send @mediator)
	      users (:users @mediator)]
	  (send-fn users (format "%s: %s\n" (:name u) text))))

(add-user {:name "Mister White"})
(add-user {:name "Mister Pink"})
(send-message {:name "Joe"} "Toby?")

Pedro: Good enough.
Eve: Nothing interesing here, because it is just a one approach to reduce coupling.

Episode 10: Observer

Independent security commision detected hacker Dartee Hebl has over a billion dollars balance on his account. Track big transactions to the accounts.

Pedro: Are we Holmses?
Eve: No, but there is a lack of logging in the system, need to find a way to track all changes to balance.
Pedro: We could add observers. Every time we modify balance, if change is big enough, notify about this and track the reason. We need Observer interface

public interface Observer {
  void notify(User u);
}

Pedro: And two specific observers

class MailObserver implements Observer {
  @Override
  public void notify(User user) {
    MailService.sendToFBI(user);
  }
}

class BlockObserver implements Observer {
  @Override
  public void notify(User u) {
    DB.blockUser(u);
  }
}

Pedro: Tracker class would be responsible for managing observers.

public class Tracker {
  private Set<Observer> observers = new HashSet<Observer>();

  public void add(Observer o) {
    observers.add(o);
  }

  public void update(User u) {
    for (Observer o : observers) {
      o.notify(u);
    }
  }
}

Pedro: And the last part: init tracker with the user and modify its addMoney method. If transcation amount is greaterthan 100$, notify FBI and block this user.

public class User {
  String name;
  double balance;
  Tracker tracker;

  public User() {
    initTracker();
  }

  private void initTracker() {
    tracker = new Tracker();
    tracker.add(new MailObserver());
    tracker.add(new BlockObserver());
  }


  public void addMoney(double amount) {
    balance += amount;
    if (amount > 100) {
      tracker.update(this);
    }
  }
}

Eve: Why are you created two separate observers? You could use it in a one.

class MailAndBlock implements Observer {
  @Override
  public void notify(User u) {
    MailService.sendToFBI(u);
    DB.blockUser(u);
  }
}

Pedro: Single responsibility principle.
Eve: Oh, yeah.
Pedro: And you can compose observer functionality dynamically.
Eve: I see your point.

;; Tracker

(def observers (atom #{}))

(defn add [observer]
  (swap! observers conj observer))

(defn notify [user]
  (map #(apply % user) @observers))

;; Fill Observers

(add (fn [u] (mail-service/send-to-fbi u)))
(add (fn [u] (db/block-user u)))

;; User

(defn add-money [user amount]
  (swap! user
    (fn [m]
      (update-in m [:balance] + amount)))
  ;; tracking
  (if (> amount 100) (notify)))

Pedro: It’s a pretty the same way?
Eve: Yeah, in fact observer is just a way to register function, which will be called after another function.
Pedro: It is still the pattern.
Eve Sure, but we can improve solution a bit using clojure watches.

(add-watch
  user
  :money-tracker 
  (fn [k r os ns] 
    (if (< 100 (- (:balance ns) (:balance os)))
      (notify))))

Pedro: Why is that better?
Eve: First of all, our add-money function is clean, it just adds money. Also, watcher tracks every change to the state, not the ones we handle in mutator functions, like add-money
Pedro: Explain please.
Eve: If there is provided another secred method secret-add-money for changing balance, watchers will handle that as well.
Pedro: That’s awesome!

Episode 11: Interpreter

Bertie Prayc stole important data from our server and shared it via BitTorrent system. Create a fake account for Bertie to discredit him.

Pedro: BitTorrent system based on .torrent files. We need to write Bencode encoder.
Eve: Yes, but first let’s agree on the format spec

Bencode encoding rules:

Two datatypes are supported:
- Integer N is encoded as i<N>e. (42 = i42e)
- String S is encoded as <length>:<contents> (hello = 5:hello)
Two containers are supported:
- List of values is encoded as l<contents>e ([1, “Bye”] = li1e3:Byee)
- Map of values is encoded as d<contents>e ({“R” 2, “D” 2} = d1:Ri2e1:Di2ee)
  - Keys are just strings, Values are any bencode element

Pedro: Seems easy.
Eve: Maybe, but take into account that values may be nested, list inside list, etc.
Pedro: Sure. I think we can use Interpreter pattern for bencode encoding.
Eve: Try it.
Pedro: We start from interface for all bencode elements

interface BencodeElement {
  String interpret();
}

Pedro: Then we provide implementation for each datatype and datacontainer

class IntegerElement implements BencodeElement {
  private int value;

  public IntegerElement(int value) {
    this.value = value;
  }

  @Override
  public String interpret() {
    return "i" + value + "e";
  }
}

class StringElement implements BencodeElement {
  private String value;

  StringElement(String value) {
    this.value = value;
  }

  @Override
  public String interpret() {
    return value.length() + ":" + value;
  }
}

class ListElement implements BencodeElement {
  private List<? extends BencodeElement> list;

  ListElement(List<? extends BencodeElement> list) {
    this.list = list;
  }

  @Override
  public String interpret() {
    String content = "";
    for (BencodeElement e : list) {
      content += e.interpret();
    }
    return "l" + content + "e";
  }
}

class DictionaryElement implements BencodeElement {
  private Map<StringElement, BencodeElement> map;

  DictionaryElement(Map<StringElement, BencodeElement> map) {
    this.map = map;
  }

  @Override
  public String interpret() {
    String content = "";
    for (Map.Entry<StringElement, BencodeElement> kv : map.entrySet()) {
      content += kv.getKey().interpret() + kv.getValue().interpret();
    }
    return "d" + content + "e";
  }
}

Pedro: And finally, our bencoded string can be constructed from common datastructures programmatically

// discredit user
Map<StringElement, BencodeElement> mainStructure = new HashMap<StringElement, BencodeElement>();
// our victim
mainStructure.put(new StringElement("user"), new StringElement("Bertie"));
// just downloads files
mainStructure.put(new StringElement("number_of_downloaded_torrents"), new IntegerElement(623));
// and nothing uploads
mainStructure.put(new StringElement("number_of_uploaded_torrents"), new IntegerElement(0));
// and nothing donates
mainStructure.put(new StringElement("donation_in_dollars"), new IntegerElement(0));
// prefer dirty categories
mainStructure.put(new StringElement("preffered_categories"),
                      new ListElement(Arrays.asList(
                          new StringElement("porn"),
                          new StringElement("murder"),
                          new StringElement("scala"),
                          new StringElement("pokemons")
                      )));
BencodeElement top = new DictionaryElement(mainStructure);

// let's totally discredit him
String bencodedString = top.interpret();
BitTorrent.send(bencodedString);

Eve: Interesting, but that is ton of code!
Pedro: We pay readability for capabilities.
Eve: I suppose you’ve heard concept Code is Data, that’s a lot easier in clojure

;; multimethod to handle bencode structure
(defmulti interpret class)

;; implementation of bencode handler for each type
(defmethod interpret java.lang.Long [n]
  (str "i" n "e"))

(defmethod interpret java.lang.String [s]
  (str (count s) ":" s))

(defmethod interpret clojure.lang.PersistentVector [v]
  (str "l" 
       (apply str (map interpret v))
	   "e"))

(defmethod interpret clojure.lang.PersistentArrayMap [m]
  (str "d" 
       (apply str (map (fn [[k v]] 
	                     (str (interpret k)
                              (interpret v))) m))
       "e"))

;; usage
(interpret {"user" "Bertie"
            "number_of_downloaded_torrents" 623
			"number_of_uploaded_torrent" 0
			"donation_in_dollars" 0
			"preffered_categories" ["porn"
                                    "murder"
			                        "scala"
									"pokemons"]})

Eve: You see how it is much easier define specific data?
Pedro: Sure, and interpret it’s just a function per bencode type, instead of separate class.
Eve: Correct, interpreter is nothing but a set of functions to process a tree.

Episode 12: Flyweight

Administrators of the lawyer firm Cristopher, Matton & Pharts detected that reporting system consumes a lot of memory and garbage collector continiously hangs system for seconds. Fix that.

Pedro: I’ve seen this issue before.
Eve: What’s wrong?
Pedro: They have realtime charts with a lot of different points. It’s a huge amount of memory. As a result garbage collector stops the system.
Eve: Hmmm, what we can do?
Pedro: Not much, caching does not help us because points are…
Eve: Wait!
Pedro: What?
Eve: They use age values for points, why not precompute these points for most common ages? Say for age [0, 100]
Pedro: You mean use Flyweight pattern?
Eve: I mean reuse objects.

class Point {
  int x;
  int y;

  /* some other properties*/

  // precompute 10000 point values at class loading time
  private static Point[][] CACHED;
  static {
    CACHED = new Point[100][];
    for (int i = 0; i < 100; i++) {
      CACHED[i] = new Point[100];
      for (int j = 0; j < 100; j++) {
        CACHED[i][j] = new Point(i, j);
      }
    }
  }

  Point(int x, int y) {
    this.x = x;
    this.y = y;
  }

  static Point makePoint(int x, int y) {
    if (x >= 0 && x < 100 &&
        y >= 0 && y < 100) {
      return CACHED[x][y];
    } else {
      return new Point(x, y);
    }
  }
}

Pedro: For this pattern we need two things: precompute most used points at a startup time, and use static factory method instead of constructor to return cached object.
Eve: Have you tested it?
Pedro: Sure, the system works like a clock.
Eve: Excellent, here is my version

(defn make-point [x y]
  [x y {:some "Important Properties"}])

(def CACHE
  (let [cache-keys (for [i (range 100) j (range 100)] [i j])]
	  (zipmap cache-keys (map #(apply make-point %) cache-keys))))

(defn make-point-cached [x y]
  (let [result (get CACHE [x y])]
    (if result
	  result
	  (make-point x y))))

Eve: It creates a flat map with pair [x, y] as a key, instead of two-dimensional array.
Pedro: Pretty the same.
Eve: No, it is much flexible, you can’t use two-dimensional array if you need to cache three points or non-integer values.
Pedro: Oh, got it.
Eve: Even better, in clojure you can just use memoize function to cache calls to factory function make-point

(def make-point-memoize (memoize make-point))

Eve: Every call (except first one) with the same parameters return cached value.
Pedro: That’s awesome!
Eve: Of course, but remember if your function has side-effects, memoization is bad idea.

Episode 13: Builder

Tuck Brass complains that his old automatic coffee-making system is very slow in usage. Customers can’t wait long enough and going away.

Pedro: Need to understand what’s the problem exactly?
Eve: I’ve researched, system is old, written in COBOL and built around expert system question-answer. They were very popular long time ago.
Pedro: What do you mean by “question-answer”
Eve: There is operator in front of terminal. System asks: “Do yo want to add water?”, operator answers “Yes”. Then system asks again: “Do you want to add coffee”, operator answers “Yes” and so forth.
Pedro: It’s a nightmare, I just want coffee with milk. Why they don’t use predefined options: coffee with milk, coffee with sugar and so on.
Eve: Because it is the raisin of the system: customer can make coffee with any set of ingridients by themselves
Pedro: Okay, let’s fix it with Builder pattern.

public class Coffee {
  private String coffeeName; // required
  private double amountOfCoffee; // required
  private double water; // required
  private double milk; // optional
  private double sugar; // optional
  private double cinnamon; // optional

  private Coffee() { }

  public static class Builder {
    private String builderCoffeeName;
    private double builderAmountOfCoffee; // required
    private double builderWater; // required
    private double builderMilk; // optional
    private double builderSugar; // optional
    private double builderCinnamon; // optional

    public Builder() { }

    public Builder setCoffeeName(String name) {
      this.builderCoffeeName = name;
      return this;
    }

    public Builder setCoffee(double coffee) {
      this.builderAmountOfCoffee = coffee;
      return this;
    }

    public Builder setWater(double water) {
      this.builderWater = water;
      return this;
    }

    public Builder setMilk(double milk) {
      this.builderMilk = milk;
      return this;
    }

    public Builder setSugar(double sugar) {
      this.builderSugar = sugar;
      return this;
    }

    public Builder setCinnamon(double cinnamon) {
      this.builderCinnamon = cinnamon;
      return this;
    }

    public Coffee make() {
      Coffee c = new Coffee();
        c.coffeeName = builderCoffeeName;
        c.amountOfCoffee = builderAmountOfCoffee;
        c.water = builderWater;
        c.milk = builderMilk;
        c.sugar = builderSugar;
        c.cinnamon = builderCinnamon;

      // check required parameters and invariants
      if (c.coffeeName == null || c.coffeeName.equals("") ||
          c.amountOfCoffee <= 0 || c.water <= 0) {
        throw new IllegalArgumentException("Provide required parameters");
      }

      return c;
    }
  }
}

Pedro: As you see, you can’t instantiate Coffee class easily, you need to set parameters with nested Builder class

Coffee c = new Coffee.Builder()
        .setCoffeeName("Royale Coffee")
        .setCoffee(15)
        .setWater(100)
        .setMilk(10)
        .setCinnamon(3)
        .make();

Pedro: Calling to method make checks all required parameters, and could validate and throw an exception if object is in inconsistent state.
Eve: Awesome functionality, but why so verbose?
Pedro: Beat it.
Eve: A piece of cake, clojure supports optional arguments, everything what builder pattern is about.

(defn make-coffee [name amount water
                   & {:keys [milk sugar cinnamon]
                      :or {milk 0 sugar 0 cinnamon 0}}]
  ;; definition goes here
  )

(make-coffee "Royale Coffee" 15 100
             :milk 10
             :cinnamon 3)

Pedro: Aha, you have three required parameters and three optionals, but required parameters still without names.
Eve: What do you mean?
Pedro: From the client call I see number 15 but I have no idea what it might be.
Eve: Agreed. Then, let’s make all parameters are named and add precondition for required, the same way you do with the builder.

(defn make-coffee
  [& {:keys [name amount water milk sugar cinnamon]
      :or {name "" amount 0 water 0 milk 0 sugar 0 cinnamon 0}}]
  {:pre [(not (empty? name))
         (> amount 0)
         (> water 0)]}
  ;; definition goes here		 
  )

(make-coffee :name "Royale Coffee"
             :amount 15
             :water 100
             :milk 10
             :cinnamon 3)

Eve: As you see all parameters are named and all required params are checked in :pre constraint. If constraints are violated AssertionError is thrown.
Pedro: Interesting, :pre is a part of a language?
Eve: Sure, it’s just a simple assertion. There is also :post constraint, with the similar effect.
Pedro: Hm, okay. But as you know Builder pattern often used as a mutable datastucture, StringBuilder for example.
Eve: It’s not a part of clojure philosophy to use mutables, but if you really want, no problem. Just create a new class with deftype and do not forget to use volatile-mutable on the properties you want to mutate.
Pedro: Where is the code?
Eve: Here is example of custom implementation of mutable StringBuilder in clojure. It has a lot of drawbacks and limitations but you’ve got the idea.

;; interface
(defprotocol IStringBuilder
	(append [this s])
	(to-string [this]))

;; implementation
(deftype ClojureStringBuilder [charray ^:volatile-mutable last-pos]
	IStringBuilder
	(append [this s] 
	  (let [cs (char-array s)]
	    (doseq [i (range (count cs))]
	      (aset charray (+ last-pos i) (aget cs i))))
	  (set! last-pos (+ last-pos (count s))))
	(to-string [this] (apply str (take last-pos charray))))

;; clojure binding
(defn new-string-builder []
  (ClojureStringBuilder. (char-array 100) 0))

;; usage
(def sb (new-string-builder))
(append sb "Toby Wong")
(to-string sb) => "Toby Wong"
(append sb " ")
(append sb "Toby Chung") => "Toby Wang Toby Chung"

Pedro: Not as hard as I thought.

Episode 14: Facade

Our new member Eugenio Reinn Jr. commited file with diff in 134 lines to processing servlet. But actual work is just to process request. All other code injection and imports. It MUST be one line commit.

Pedro: Who cares how many lines are committed?
Eve: Someone cares.
Pedro: Let’s see where is the problem

class OldServlet {
  @Autowired
  RequestExtractorService requestExtractorService;
  @Autowired
  RequestValidatorService requestValidatorService;
  @Autowired
  TransformerService transformerService;
  @Autowired
  ResponseBuilderService responseBuilderService;

  public Response service(Request request) {
    RequestRaw rawRequest = requestExtractorService.extract(request);
    RequestRaw validated = requestValidatorService.validate(rawRequest);
    RequestRaw transformed = transformerService.transform(validated);
    Response response = responseBuilderService.buildResponse(transformed);
    return response;
  }
}

Eve: Oh shi…
Pedro: That’s our internal API for developers, every time they need to process request, inject 4 services, include all imports, and write this code.
Eve: Let’s refactor it with…
Pedro: …Facade pattern. We resolve all dependencies to a single point of access and simplify API usage.

public class FacadeService {
  @Autowired
  RequestExtractorService requestExtractorService;
  @Autowired
  RequestValidatorService requestValidatorService;
  @Autowired
  TransformerService transformerService;
  @Autowired
  ResponseBuilderService responseBuilderService;

  RequestRaw extractRequest(Request req) {
    return requestExtractorService.extract(req);
  } 
  
  RequestRaw validateRequest(RequestRaw raw) {
    return requestValidatorService.validate(raw);
  }
  
  RequestRaw transformRequest(RequestRaw raw) {
    return transformerService.transform(raw);
  }
  
  Response buildResponse(RequestRaw raw) {
    return responseBuilderService.buildResponse(raw);
  }
}

Pedro: Then if you need any service or set of services in the code you just injecting facade to your code

class NewServlet {
  @Autowired
  FacadeService facadeService;

  Response service(Request request) {
    RequestRaw rawRequest = facadeService.extractRequest(request);
    RequestRaw validated = facadeService.validateRequest(rawRequest);
    RequestRaw transformed = facadeService.transformRequest(validated);
    Response response = facadeService.buildResponse(transformed);
    return response;
  }
}

Eve: Wait, you’ve just moved all dependencies to one and everytime using this one, correct?
Pedro: Yes, now everytime some functionality is needed, use FacadeService. Dependency is already there.
Eve: But we did the same in Mediator pattern?
Pedro: Mediator is behavioral pattern. We resolved all dependency to Mediator and added new behavior to it.
Eve: And facade?
Pedro: Facade is structural, we don’t add new functionality, we just expose existing functionality with facade.
Eve: Got it. But seems that pattern very loud word for such little tweak.
Pedro: Maybe.
Eve: Here is clojure version using structure by namespaces

(ns application.old-servlet
  (:require [application.request-extractor :as re])
  (:require [application.request-validator :as rv])
  (:require [application.transformer :as t])
  (:require [application.response-builder :as rb]))

(defn service [request]
  (-> request
      (re/extract)
      (rv/validate)
	  (t/transform)
	  (rb/build)))

Eve: Exposing all services via facade.

(ns application.facade
  (:require [application.request-extractor :as re])
  (:require [application.request-validator :as rv])
  (:require [application.transformer :as t])
  (:require [application.response-builder :as rb]))

(defn request-extract [request]
  (re/extract request))

(defn request-validate [request]
  (rv/validate request))

(defn request-transform [request]
  (t/transform request))

(defn response-build [request]
  (rb/build request))

Eve: And use it.

(ns application.old-servlet
  (:use [application.facade]))

(defn service [request]
  (-> request
      (request-extract)
      (request-validate)
	  (request-transform)
	  (request-build)))

Pedro: What the difference between :use and :require?
Eve: They are almost similar, but with :require you expose functionality via namespace qualificator (namespace/function) where with :use you can refer to it directly (function)
Pedro: So, :use is better.
Eve: No, be aware of :use because it can conflict with existing names in your namespace.
Pedro: Oh, I see your point. And every time you call (:use [application.facade]) in some namespace all existing functionality from facade is available?
Eve: Yes.
Pedro: Pretty the same.

Episode 15: Singleton

Feverro O’Neal complains that we have a lot of different styles for UI.
Force one per application UI configuration.

Pedro: But wait, there was requirement to save UI style per user.
Eve: Probably it was changed.
Pedro: Ok, then we should just save configuration to Singleton and use it from all the places.

public final class UIConfiguration {
  public static final UIConfiguration INSTANCE = new UIConfiguration("ui.config");

  private String backgroundStyle;
  private String fontStyle;
  /* other UI properties */

  private UIConfiguration(String configFile) {
    loadConfig(configFile);
  }

  private static void loadConfig(String file) {
    // process file and fill UI properties
	INSTANCE.backgroundStyle = "black";
    INSTANCE.fontStyle = "Arial";
  }

  public String getBackgroundStyle() {
    return backgroundStyle;
  }

  public String getFontStyle() {
    return fontStyle;
  }
}

Pedro: That way all configuration will be shared across the UIs.
Eve: Yes, but…why so much code?
Pedro: We guarantee that only one instance of UIConfiguration will exist.
Eve: Let me ask you: what’s the difference between singleton and global varaible.
Pedro: What?
Eve: …the difference between singleton and global variable.
Pedro: Java does not support global variables.
Eve: But UIConfiguration.INSTANCE is global variable.
Pedro: Well, sort of.
Eve: Your code is just simple def in clojure.

(def ui-config (load-config "ui.config"))

(defn load-config [config-file]
  ;; process config file and return map with configuratios
  {:bg-style "black" :font-style "Arial"})

Pedro: But, how do you change the style?
Eve: The same way you will change it in your code.
Pedro: Uhm… Ok, we need a simple tweak. Make UIConfiguration.loadConfig is public and call it when the configuration changes.
Eve: Then we make ui-config an atom and call swap! when configuration changes.
Pedro: But atoms are useful only in concurrent environment.
Eve: First, yes, they useful, but NOT only in concurrent environment. Second, atom read is not as slow as you think. Third, it changes the state of UI configuration atomically
Pedro: It is redundant for such simple example.
Eve: No, it is not. There is a posibility that UI configuration changes and some renders read new backgroundStyle, but old fontStyle
Pedro: Ok, use synchronized for loadConfig
Eve: Then you must use synchonized on getters as well, it is slow.
Pedro: There is still Double-Checked Locking idiom
Eve: Double-checked locking is clever but broken
Pedro: Ok, I give up, you won.

Episode 16: Chain Of Responsibility

New York marketing organization “A Profit NY” opened request to filter profanity words from their public chat system.

Pedro: Fuck, they don’t like the word “fuck”?
Eve: It is profit organization, they lose money if someone use profanity words in public chat.
Pedro: Who defined profanity words list?
Eve: George Carlin

Watching and laughing

Pedro: Ok, so let’s just add a filter to replace these rude words with the asterisks.
Eve: Make sure your solution is extendable, other filters could be applied.
Pedro: Chain of Responisibility seems like a good pattern candidate for that. First of all we make some abstract filter.

public abstract class Filter {
  protected Filter nextFilter;

  abstract void process(String message);

  public void setNextFilter(Filter nextFilter) {
    this.nextFilter = nextFilter;
  }
}

Pedro: Then, provide implementation for each specific filter you want to apply

class LogFilter extends Filter {
  @Override
  void process(String message) {
    Logger.info(message);
    if (nextFilter != null) nextFilter.process(message);
  }
}

class ProfanityFilter extends Filter {
  @Override
  void process(String message) {
    String newMessage = message.replaceAll("fuck", "f*ck");
    if (nextFilter != null) nextFilter.process(newMessage);
  }
}

class RejectFilter extends Filter {
  @Override
  void process(String message) {
    System.out.println("RejectFilter");
    if (message.startsWith("[A PROFIT NY]")) {
      if (nextFilter != null) nextFilter.process(message);
    } else {
      // reject message - do not propagate processing
    }
  }
}

class StatisticsFilter extends Filter {
  @Override
  void process(String message) {
    Statistics.addUsedChars(message.length());
    if (nextFilter != null) nextFilter.process(message);
  }
}

Pedro: And finally build a chain of filters which defines an order how message will be processed.

Filter rejectFilter = new RejectFilter();
Filter logFilter = new LogFilter();
Filter profanityFilter = new ProfanityFilter();
Filter statsFilter = new StatisticsFilter();

rejectFilter.setNextFilter(logFilter);
logFilter.setNextFilter(profanityFilter);
profanityFilter.setNextFilter(statsFilter);

String message = "[A PROFIT NY] What the fuck?";
rejectFilter.process(message);

Eve: Ok, now clojure turn. Just define each filter as a function.

;; define filters

(defn log-filter [message]
  (logger/log message)
  message)

(defn stats-filter [message]
  (stats/add-used-chars (count message))
  message)

(defn profanity-filter [message]
  (clojure.string/replace message "fuck" "f*ck"))

(defn reject-filter [message]
  (if (.startsWith message "[A Profit NY]")
    message))

Eve: And use some-> macro to chain filters

(defn chain [message]
  (some-> message
          reject-filter
          log-filter
          stats-filter
          profanity-filter))

Eve: You see how much it is easier, you don’t need every-time call if (nextFilter != null) nextFilter.process(), because it’s natural. The next filter defined at the some-> level naturally, instead of calling manuall setNext.
Pedro: That’s definitely better for composability, but why did you use some-> instead of ->?
Eve: Just for reject-filter. It could stop further processing, so some-> returns nil as soon as nil encountered as a filter
Pedro: Could you explain more?
Eve: Look at the usage

(chain "fuck") => nil
(chain "[A Profit NY] fuck") => "f*ck"

Pedro: Understood.
Eve: Chain of Responsibility just an approach to function composition

Episode 17: Composite

Actress Bella Hock don’t see user avatars in our social network on her computer.

“Everything is black. Is it black holes?”

Pedro: It is black squares.
Eve: Hmm, the same problem on our side.
Pedro: Seems, that latest feature broke user avatars.
Eve: Strange, because avatars rendered the same way as other elements, but they are visible.
Pedro: Are you sure it rendered the same way?
Eve: Well…no

Digging code

Pedro: What the hell is going on here?
Eve: Someone copypasted code, but forgot to reflect changes in avatars.
Pedro: Let’s blame him, git-blame
Eve: Blame is good, but we need to fix the problem
Pedro: It’s simple just additional line here.
Eve: I mean, really solve the problem. Why do we need two similar snippets of code to process the same blocks?
Pedro: Correct, I think we could use Composite pattern to handle rendering of the whole page. Our smallest element to render is block.

public interface Block {
  void addBlock(Block b);
  List<Block> getChildren();
  
  void render();
}

Pedro: Obviously blocks may contain other blocks, that’s the point of Composite pattern. We may create some specific blocks.

public class Page implements Block { }
public class Header implements Block { }
public class Body implements Block { }
public class HeaderTitle implements Block { }
public class UserAvatar implements Block { }

Pedro: And treat every specific element as a Block

Block page = new Page();
Block header = new Header();
Block body = new Body();
Block title = new HeaderTitle();
Block avatar = new UserAvatar();

page.addBlock(header);
page.addBlock(body);
header.addBlock(title);
header.addBlock(avatar);

page.render();

Pedro: This is a structural pattern, a good way to compose objects. That’s why it is called composite
Eve: Hey, composite is a simple tree structure.
Pedro: Yes.
Eve: There is pattern for every datastructure?
Pedro: No, just for list and tree.
Eve: Actually, tree can be represented as a list
Pedro: How?
Eve: First element of a list is a node value, next elements are children, each of them is…
Pedro: I understand.
Eve: To be specific, here is the tree

        A
     /  |  \
    B   C   D
    |   |  / \
	E   H J   K
   / \       /|\
  F   G     L M N

Eve: And here is the list represents this tree

(def tree
  '(A (B (E (F) (G))) (C (H)) (D (J) (K (L) (M) (N)))))

Pedro: It’s a plenty of parentheses!
Eve: They define structure, you know
Pedro: But it’s hard to understand
Eve: It’s easy for machine, there is awesome tree-seq function to process the tree.

(map first (tree-seq next rest tree)) => (A B E F G C H D J K L M N)

Eve: If you need more advanced traversals, use clojure.walk
Pedro: I don’t know, everything seems just a bit harder.
Eve: No, you define the whole tree with one datastructure and use one function to operate on it.
Pedro: What this function will do?
Eve: It traverses the tree and applies to every node, so in our case it can render each component.
Pedro: I don’t know, maybe I am too young for the trees, let’s move forward.

Episode 18. Factory Method

Sir Dry Bang suggest to create new levels for their popular game. More levels - more money.

Pedro: How can we create new levels?
Eve: Just change assets and add new blocks: paper, wood, iron…
Pedro: It’s too silly, isn’t it?
Eve: The whole game is too silly. If user pay for color hats for their characters, then they will pay for wooden blocks as well.
Pedro: I think it’s a crap, but anyway, let’s make MazeBuilder generic and add specific builder for each type of the block. It’s a Factory Method pattern.

class Maze { }
class WoodMaze extends Maze { }
class IronMaze extends Maze { }

interface MazeBuilder {
  Maze build();
}

class WoodMazeBuilder {
  @Override
  Maze build() {
    return new WoodMaze();
  }
}

class IronMazeBuilder {
  @Override
  Maze build() {
    return new IronMaze();
  }
}

Eve: Isn’t it obvious that IronMazeBuilder will return IronMazes?
Pedro: Not for program. But see, to make a maze from another blocks we just change implementation responsible for block creation.

MazeBuilder builder = new WoodMazeBuilder();
Maze maze = builder.build();

Eve: I’ve seen something similar before.
Pedro: What exactly?
Eve: For me it seems like a strategy or state pattern.
Pedro: No way! Strategy is about performing specific operations and factory is for creating specific object.
Eve: But create is an operation as well.

(defn maze-builder [maze-fn])

(defn make-wood-maze [])
(defn make-iron-maze [])

(def wood-maze-builder (partial maze-builder make-wood-maze))
(def iron-maze-builder (partial maze-builder make-iron-maze))

Pedro: Hm, seems similar.
Eve: Think about it.
Pedro: Any usage examples?
Eve: No, everything is obvious here, just re-read Strategy, State or TemplateMethod episodes.

Episode 19: Abstract Factory

Users are not buying new levels in the game. Saimank Gerr build a complains cloud and the most popular negative feedback words are: “ugly”, “crap” and “shit”.

Improve levels-building system.

Pedro: I said this is crap.
Eve: Sure, snow background with wooden walls, space invaders with wooden walls every setting with wooden walls.
Pedro: Then we must separate game worlds and build a set of specific objects for particular world.
Eve: Explain.
Pedro: Instead of using Factory Method for building specific blocks, we use Abstract Factory to build a set of related objects, to make a level look less crappy.
Eve: Example would be good.
Pedro: Code is my example. First we define abstract behaviour of level factory

public interface LevelFactory {
  Wall buildWall();
  Back buildBack();
  Enemy buildEnemy();
}

Pedro: Then we have a hierarchy of objects that level consists of

class Wall {}
class PlasmaWall extends Wall {}
class StoneWall extends Wall {}

class Back {}
class StarsBack extends Back {}
class EarthBack extends Back {}

class Enemy {}
class UFOSoldier extends Enemy {}
class WormScout extends Enemy {}

Pedro: See? We have a specific object for each level, let’s create factory for them.

class SpaceLevelFactory implements LevelFactory {
  @Override
  public Wall buildWall() {
    return new PlasmaWall();
  }

  @Override
  public Back buildBack() {
    return new StarsBack();
  }

  @Override
  public Enemy buildEnemy() {
    return new UFOSoldier();
  }
}

class UndergroundLevelFactory implements LevelFactory {
  @Override
  public Wall buildWall() {
    return new StoneWall();
  }

  @Override
  public Back buildBack() {
    return new EarthBack();
  }

  @Override
  public Enemy buildEnemy() {
    return new WormScout();
  }
}

Pedro: Each implementation of level factory creates related objects for level. Levels will look less crappy for sure.
Eve: Let me understand. I really can’t spot the difference.
Pedro: Factory Method defers object creation to a subclasses, Abstract Factory do the same but for a set of related object.
Eve: Aha, that means I need to pass set of related functions to abstract builder

(defn level-factory [wall-fn back-fn enemy-fn])

(defn make-stone-wall [])
(defn make-plasma-wall [])

(defn make-earth-back [])
(defn make-stars-back [])

(defn make-worm-scout [])
(defn make-ufo-soldier [])

(def underground-level-factory
  (partial level-factory
           make-stone-wall
           make-earth-back
           make-worm-scout))

(def space-level-factory
  (partial level-factory
           make-plasma-wall
           make-stars-back
           make-ufo-soldier))

Pedro: I knew.
Eve: Everything is fair. Your lovely “set of related Xs”, where X is a function
Pedro: Yes, clarify, what partial is.
Eve: Provide some parameters for function. So, underground-level-factory knows how to construct walls, backs and enemies. Everything other inherited from abstract level-factory function.
Pedro: Handy.

Episode 20: Adapter

Deam Evil conducts a medieval tournament for knights. The prize is $100.000

I’ll pay you the half if you break the system and allow my armed commando to take part in competition.

Pedro: Finally we’ve got interesting work.
Eve: Funny to see the competition. Especially, M16 vs Iron Sword part.
Pedro: Knights have a good armor.
Eve: F1 grenade does not care about armor.
Pedro: Nevermind, we do the work, we get the money.
Eve: Fifty grands - nice compensation
Pedro: Yes, look at this, I’ve stolen sources of the competition system, though it is not possible to modify their sources, we can find some vulneravility.
Eve: Here it is

public interface Tournament {
  void accept(Knight knight);
}

Pedro: Aha! System validates only incoming types via Knight interface. All we need to do is to adapt commando to be a knight. Let’s see how knight look like

interface Knight {
  void attackWithSword();
  void attackWithBow();
  void blockWithShield();
}

class Galahad implements Knight {
  @Override
  public void blockWithShield() {
    winkToQueen();
    take(shield);
    block();
  }

  @Override
  public void attackWithBow() {
    winkToQueen();
    take(bow);
    attack();
  }

  @Override
  public void attackWithSword() {
    winkToQueen();
    take(sword);
    attack();
  }
}

Pedro: To accept the commando let’s take an old implementation

class Commando {
    void throwGrenade(String grenade) { }
    shot(String rifleType) { }
}

Pedro: And adapt it.

class Commando implements Knight {
  @Override
  public void blockWithShield() {
    // commando don't block
  }

  @Override
  public void attackWithBow() {
    throwGrenade("F1");
  }

  @Override
  public void attackWithSword() {
    shotWithRifle("M16");
  }
}

Pedro: That’s it.
Eve: It’s simpler in clojure.
Pedro: Really?
Eve: We don’t love types so their validation won’t work at all
Pedro: So, how do you replace knight with commando?
Eve: Basically, what knight is? It’s a map, consists of data and behaviour

{:name "Lancelot"
 :speed 1.0
 :attack-bow-fn attack-with-bow
 :attack-sword-fn attack-with-sword
 :block-fn block-with-shield}

Eve: To adapt commando, just pass his functions instead original ones

{:name "Commando"
 :speed 5.0
 :attack-bow-fn (partial throw-grenade "F1")
 :attack-sword-fn (partial shot "M16")
 :block-fn nil}

Pedro: How did we share money?
Eve: 50/50
Pedro: I wrote more code, I want 70
Eve: Ok, 70/70
Pedro: Deal.

Episode 21: Decorator

Podrea Vesper caught us on cheating for the tournament. We have a choice: to be busted by police or to help his super knight to take part in competition

Pedro: I don’t wanna go to prison.
Eve: Me either.
Pedro: Let’s cheat for him one more time.
Eve: This is the same solution, isn’t it?
Pedro Similar, but not the same. Commando was a soldier and they are not allowed on the tournament. We adapted it. But knight is allowed for tournament we don’t need to adapt it, we must add functionality to existing object.
Eve: Inheritance or composition?
Pedro: Composition, the main goal of decorator to change behaviour at a runtime
Eve: So how we deal with this super knight?
Pedro: They plan to use Galahad knight and decorate it with more HP and Power Armor
Eve: Heh, funny that cops are playing Fallout
Pedro: Yeah, let’s make knight an abstract class

public class Knight {
    protected int hp;
    private Knight decorated;

    public Knight() { }

    public Knight(Knight decorated) {
        this.decorated = decorated;
    }

    public void attackWithSword() {
        if (decorated != null) decorated.attackWithSword();
    }

    public void attackWithBow() {
        if (decorated != null) decorated.attackWithBow();
    }

    public void blockWithShield() {
        if (decorated != null) decorated.blockWithShield();
    }
}

Eve: So what we improved there?
Pedro: First af all we make class Knight instead of interface to have access to hit points. Then we provide two diferent constructors, default for standard behaviour, and decorated, which delegates call to decorated object.
Eve: Is it fair to use abstract class instead of interface?
Pedro: No, but we avoid two classes with similar behaviour and fill each decorated object with default implementation, instead of forcing to implement each of the methods.
Eve: Ok, what’s about Power Armor?
Pedro: Easy as well

public class KnightWithPowerArmor extends Knight {
    public KnightWithPowerArmor(Knight decorated) {
        super(decorated);
    }

    @Override
    public void blockWithShield() {
        super.blockWithShield();
        Armor armor = new PowerArmor();
        armor.block();
    }
}

public class KnightWithAdditionalHP extends Knight {
    public KnightWithAdditionalHP(Knight decorated) {
        super(decorated);
        this.hp += 50;
    }
}

Pedro: Two decorators that fulfils FBI requirements, and we able to create super knight, with behaviour like Galahad, but super armor and 50 more hit points.

Knight superKnight =
     new KnightWithAdditionalHP(
     new KnightWithPowerArmor(
	 new Galahad()));

Eve: Nice trick!
Pedro: You are welcome to show the similar behaviour in clojure
Eve: Here it is

(def galahad {:name "Galahad"
              :speed 1.0
              :hp 100
              :attack-bow-fn attack-with-bow
              :attack-sword-fn attack-with-sword
              :block-fn block-with-shield})

(defn make-knight-with-more-hp [knight]
  (update-in knight [:hp] + 50))

(defn make-knight-with-power-armor [knight]
  (update-in knight [:block-fn]
             (fn [block-fn]
               (fn []
                 (block-fn)
                 (block-with-power-armor)))))

;; create the knight
(def superknight (-> galahad
                     make-knight-with-power-armor
                     make-knight-with-more-hp)

Pedro: The same functionality.
Eve: Yes, just pay attention to power armor decorator.

Episode 22: Proxy

Deren Bart manages the system for making mixed drinks. It is rigid system, because after making a drink Bart must manually subtract used ingredients from the bar. Do this automatically.

Pedro: Can we get access to his codebase?
Eve: No, but he sent some APIs.

interface IBar {
    void makeDrink(Drink drink);
}

interface Drink {
    List<Ingredient> getIngredients();
}
    
interface Ingredient {
    String getName();
    double getAmount();
}

Pedro: Bart doesn’t want us to modify sources, instead we need to provide some additional implementation for IBar interface with autosubtracting used ingredients.
Eve: And what are we responsible for?
Pedro: Implementing Proxy Pattern, I’ve read about it some times ago.
Eve: I’m all listening.
Pedro: Basically we delegate all existing functionality to standard IBar implementation and provide new functionality inside ProxiedBar

class ProxiedBar implements IBar {
    BarDatabase bar;
    IBar standardBar;

    public void makeDrink(Drink drink) {
       standardBar.makeDrink(drink);
       for (Ingredient i : drink.getIngredients()) {
           bar.subtract(i);
       }
    }
}

Pedro: They need to replace StandardBar implementation with our ProxiedBar.
Eve: Seems super easy.
Pedro: Yes, additional plus that we don’t break existing functionality.
Eve: Are you sure? We didn’t run regression tests.
Pedro: Everything we do is delegating functionality to already tested StandardBar
Eve: But you also substracts used ingredients from BarDatabase
Pedro: We assume, they are decoupled.
Eve: Oh…
Pedro: Does clojure have some alternative?
Eve: Well, I don’t know. What I see here you are using function composition.
Pedro: Explain.
Eve: IBar implementation is a set of functions, and another IBar is another set of functions. Everything you talking about additional implementation could be covered by function composition. It’s like make-drink and after that subtract-ingredients from bar.
Pedro: Maybe code is more clear?
Eve: Yes, but I don’t think something special here

;; interface
(defprotocol IBar
  (make-drink [this drink]))

;; Bart's implementation
(deftype StandardBar []
  IBar
  (make-drink [this drink]
    (println "Making drink " drink)
    :ok))

;; our implementation
(deftype ProxiedBar [db ibar]
  IBar
  (make-drink [this drink]
    (make-drink ibar drink)
    (subtract-ingredients db drink)))

;; this how it was before
(make-drink (StandardBar.)
	{:name "Manhattan"
	 :ingredients [["Bourbon" 75] ["Sweet Vermouth" 25] ["Angostura" 5]]})

;; this how it becomes now
(make-drink (ProxiedBar. {:db 1} (StandardBar.))
	{:name "Manhattan"
     :ingredients [["Bourbon" 75] ["Sweet Vermouth" 25] ["Angostura" 5]]})

Eve: We could leverage protocol and types to group set of functions as a single object.
Pedro: Looks like clojure has object-oriented capabilities as well.
Eve: Correct, moreover it has a reify function, which allow you to create proxies in a runtime
Pedro: Like class in a runtime?
Eve: Sort of.

(reify IBar
  (make-drink [this drink]
    ;; implementation goes here
  ))

Pedro: Looks handy.
Eve: Yes, but I still don’t understand how it differs from Decorator.
Pedro: They are completely different.
Eve: Decorator adds functionality to the same interface, and so does Proxy.
Pedro: Well, but Proxy is…
Eve: Even more, Adapter is not very different as well.
Pedro: It uses another interface.
Eve: But from implementation perspective all these pattern are the same, wrap something and delegate calls to wrapper. “Wrapper” could be a good name for these patterns.

Episode 23: Bridge

Girls from HR agency “Hurece’s Sour Man” trying to identify candidates to their open job positions. The problem is jobs often created by customers, but requirements to the jobs are developed by HR. Provide them with a flexible way of collaborating.

Eve: I don’t understand the problem.
Pedro: I have a bit of background. They have a very strange system which defines job requirements as an interface.

interface JobRequirement {
    boolean accept(Candidate c);
}

Pedro: Every specific requirement implemented as a new subclass of this.

class JavaRequirement implements JobRequirement {
    public boolean accept(Candidate c) {
        return c.hasSkill("Java");
    }
}

class Experience10YearsRequirement implements JobRequirement {
    public boolean accept(Candidate c) {
        return c.getExperience() >= 10;
    }
}

Eve: I’ve got an idea.
Pedro: Take into account, this requirement hierarchy is designed by HR Department.
Eve: Ok.
Pedro: And they have a Job hierarchy, where each specific job is subclass as well.
Eve: Why do they need a class for each job? It should be an object.
Pedro: The system was designed when classes were more popular than objects, so live with this.
Eve: Class were popular than objects?!
Pedro: Yes, listen and don’t interrupt me. Jobs with requirements is completely separate hierarchy, and it is developed by customers. We introduce pattern Bridge to separate these two hierarchies and allow them live independently.

abstract class Job {
    protected List<? extends JobRequirement> requirements;

    public Job(List<? extends JobRequirement> requirements) {
        this.requirements = requirements;
    }

	protected boolean accept(Candidate c) {
        for (JobRequirement j : requirements) {
            if (!j.accept(c)) {
                return false;
			}
        }
        return true;
    }
}

class CognitectClojureDeveloper extends Job {
    public CognitectClojureDeveloper() {
        super(Arrays.asList(
                  new ClojureJobRequirement(),
                  new Experience10YearsRequirement()
        ));
    }
}

Eve: So where is the bridge?
Pedro: JobRequirement, JavaRequirement, ExperienceRequirement is one hierarchy, yes?
Eve: Yes.
Pedro: Job, CongnitectClojureDeveloperJob, OracleJavaDeveloperJob is another hierarchy.
Eve: Oh, now I see. A link from Job to JobRequirement is a bridge.
Pedro: Exactly! This is how HR can use this system to find candidate matches.

Candidate joshuaBloch = new Candidate();
(new CognitectClojureDeveloper()).accept(joshuaBloch);
(new OracleSeniorJavaDeveloper()).accept(joshuaBloch);

Pedro: Here is the point. Customers use Job as abstraction and JobRequirement as an implementation. They just create a job with descriptions or so, and HR is responsible to convert these descriptions into specific set of JobRequirement objects.
Eve: Got it.
Pedro: So as far as I understand, clojure could mimic this pattern by using defprotocol and defrecord?
Eve: Yes, but I want to revisit the problem.
Pedro: What’s wrong?
Eve: Here is we have a static flow: customer create job positions, human resources convert job position to a set of requirements and run a script against their candidates database to find a matches.
Pedro: Correct.
Eve: So there is already dependency, HR can’t do anything without open job positions.
Pedro: Well, yes. But they can develop a set of structured requirements without knowing what open positions might be.
Eve: For what purpose?
Pedro: Later this can be reused by Job creators, so HR avoid doing the same work twice.
Eve: Okay, got it, but this problem is artificial. Basically what we need is a way to colaborate between abstraction and implementation.
Pedro: Maybe, but I want to see your clojure way solution for that specific problem using bridge pattern.
Eve: Easy. Let’s use adhoc hierarchies.
Pedro: For abstractions?
Eve: Yes, jobs hierarchy is abstraction, and people need just to enhance hierarchy.

;; abstraction

(derive ::clojure-job ::job)
(derive ::java-job ::job)
(derive ::senior-clojure-job ::clojure-job)
(derive ::senior-java-job    ::java-job)

Eve: HR department are like developers, they provide implementation for this abstraction.

;; implementation
(defmulti accept :job)

(defmethod accept :java [candidate]
  (and (some #{:java} (:skills candidate))
       (> (:experience candidate) 1)))

Eve: Later, when new jobs are created, but requirements are not yet developed and no accept method implementation for this type of job, we fallback using adhoc hierarchy.
Pedro: Hm?
Eve: Assume someone created a new ::senior-java as a child of ::java job.
Pedro: Oh, and if HR not provided accept implementation for dispatch value ::senior-java, method with dispatch value ::java will be called, yes?
Eve: You learning so fast.
Pedro: But is it real bridge pattern?
Eve: There is no bridge here, but abstraction and implementation can live independently.

The End.

Cheatsheet

It is very confusing to understand patterns, which often presented in object-oriented way with bunch of UML diagrams, fancy nouns and exist to solve language-specific problems, so here is a revisited mini cheatsheet, which helps you to understand patterns by analogy.

Command - function
Strategy - function, which accepts function
State - strategy, depends on state
Visitor - multiple dispatch
Template Method - strategy with defaults
Iterator - sequence
Memento - save and restore
Prototype - immutability
Mediator - reduce coupling
Observer - function, which calls after another function
Interpreter - set of functions to process a tree
Flyweight - cache
Builder - optional arguments
Facade - single point of access
Singleton - global variable
Chain of Responsibility - function composition
Composite - tree
Factory Method - strategy for creating objects
Abstract Factory - strategy for creating set of related objects
Adapter - wrapper, same functionality, different type
Decorator - wrapper, same type, new functionality
Proxy - wrapper, function composition
Bridge - separate abstraction and implementation

Cast

A long time ago in a galaxy far, far away…

With the lack of imagination all characters and names are just anagrams.

Pedro Veel - Developer
Eve Dopler - Developer
Serpent Hill & R.E.E. - Enterprise Hell
Sven Tori - Investor
Karmen Git - Marketing
Natanius S. Selbys - Business Analyst
Mech Dominore Fight Saga - Heroes of Might and Magic
Kent Podiololis - I don’t like loops
Chad Bogue - Douchebag
Dex Ringeus - UX Designer
Veerco Wierde - Code Review
Dartee Hebl - Heartbleed
Bertie Prayc - Cyber Pirate
Cristopher, Matton & Pharts - Important Charts & Reports
Tuck Brass - Starbucks
Eugenio Reinn Jr. - Junior Engineer
Feverro O’Neal - Forever Alone
A Profit NY - Profanity
Bella Hock - Black Hole
Sir Dry Bang - Angry Birds
Saimank Gerr - Risk Manager
Deam Evil - Medieval
Podrea Vesper - Eavesdropper
Deren Bart - Bartender
Hurece’s Sour Man - Human Resources

P.S. I’ve started to write this article more than 2 years ago. Time has passed, things have changed and even Java 8 was released.

Littlepot: Autofilled Cache

2015-10-04T14:22:00+00:00

Cook, little pot, cook!

– The Magic Porridge Pot, Brothers Grimm

I’ve just released littlepot, tiny library devoted to transform batched request into single element request.

The Problem

Imagine you develop a service which return one random image and some witty text below. As an image provider, you have chosen some http://image.provider.com, because they have a free limited API. Their endpoint http://image.provider.com/api/random is exactly what you need.

You wrote a function to get batched data.

(def get-batch []
  (-> api-endpoint
      http/get
      json/parse))

But your service need to return one image (get-image)

Wasting Requests

You lazy enough, so you just return random image from fresh batch.

(defn get-image []
  (rand-nth (get-batch)))

Simple enough, unless you’ll encounter RateLimitException because guys from image.provider.com too greedy and allow only 100 requests per hour, but you just utilize 2% of all images.

Storage

Obvious solution: save images somewhere and return from there.

Congratulations, you discovered state, your life will never be the same.

Assume you introduced atom over list to save the data from the batch. Now every time you retrieve image, you need to remove it from cached batch.

(defn get-image []
  (let [element (first @batch)]
    (swap! batch rest)
    element))

Refill

Good enough, but the cache is not infinite, you need to refill it when data is over.

(defn get-image []
  (when (empty? @batch)
        (fill batch))
  (let [element (first @batch)]
    (swap! batch rest)
    element))

All clients are blocked until you cache is filled and see little latency.

What if we could send request in background, allowing clients still retrieve the data?

Littlepot

Meet littlepot, solutions to some of these problems.

Storage. It is backed by clojure.lang.PersistentQueue, clojure queue implementation, so you don’t need to care about efficient storage.

Autofill. It sends request for next batch in a background, when your cached data is close to exhaustion, so the process of filling cache goes automatically and silently.

Non-blocking. You do not need to wait when data appears in cache; if something there, return it, if not, return :no-data.

Composable. Having function to retrieve single element (get-one) you can easily get fifty elements by calling (take 50 (repeatedly get-one)).

Concurrency. It encapsulates whole state in ref and uses STM, so multiple consumers allowed. Also, guaranteed that at most one batch will be in progress.

Getting Started Guide or some Advanced Usages could be found on github.

Spark in Clojure

2015-06-14T19:42:00+00:00

Apache Spark is a fast and general engine for large-scale data processing.

100 times faster than Hadoop.

Everyone knows SQL. But traditional databases are not good in hadling big amount of data. Nevertheless, SQL is a good DSL for data processing and it is much easier to understand Spark if you have similar query implemented in SQL.

This article shows how common SQL queries implemented in Spark.

Spark Intro

Few words about Spark.

It is written in Scala, but contains bindings for Java, Python and R.

Other guys created bindings for clojure, it is not the official one, but the one we will use.

The main Spark abstraction is RDD (Resilient Distributed Dataset), in clojure terms it is a lazy sequence distributed over machines.

You can apply transformations to RDD. Something, like map and filter, they are lazy, the same as analogues in clojure.

You can perform actions on RDD, count, reduce, etc. They are not lazy, and this time calculation happens.

Prepare Data

I like drinks and as example dataset we will take Iowa Liquor Sales dataset.

Download (~800Mb)

curl https://data.iowa.gov/api/views/m3tr-qhgy/rows.csv\?accessType\=DOWNLOAD -o iowa-liquor.csv

Preprocess

cat iowa-liquor.csv \
    | awk '(NR-1)%2{$1=$1}1' RS=\" ORS=\" \
    | sed -E "s#([0-9]{2})/([0-9]{2})/([0-9]{4})#\3-\1-\2#" \
	| tr -d '$' \
	| sed '$ d' \
	| tail -n +2 \
	> iowa_liquor_sales_fixed.csv

This preprocessing step is just for the sake of simplicity and avoid a lot of hell dealing with data. Magic drops the header, redundant newline characters (which is inside double quotes), convert dates and removes dollar sign from money fields.

Import to Postgres

Create table structure

CREATE TABLE iowa_liquor_sales (
  "DATE" date,
  "CONVENIENCE STORE" varchar,
  "STORE" varchar,
  "NAME" varchar,
  "ADDRESS" varchar,
  "CITY" varchar,
  "ZIPCODE" varchar,
  "STORE LOCATION" varchar,
  "COUNTY NUMBER" varchar,
  "COUNTY" varchar,
  "CATEGORY" varchar,
  "CATEGORY NAME" varchar,
  "VENDOR NO" varchar,
  "VENDOR" varchar,
  "ITEM" varchar,
  "DESCRIPTION" varchar,
  "PACK" numeric,
  "LITER SIZE" numeric,
  "STATE BTL COST" numeric,
  "BTL PRICE" numeric,
  "BOTTLE QTY" numeric,
  "TOTAL" numeric
)

and import from CSV

COPY iowa_liquor_sales 
FROM '/absolute/path/to/iowa_liquor_sales_fixed.csv' 
DELIMITER AS ',' CSV;

Be patient, it will take a minute.

Upload to HDFS

hadoop fs -put iowa_liquor_sales_fixed.csv hdfs://localhost:9000/data/iowa_liquor_sales_fixed.csv

Configure Spark

Setup spark-context, it’s our working horse.

(defonce sc (-> (conf/spark-conf)
                (conf/app-name "SQL vs Spark")
                (conf/master "local")
                (conf/set "spark.driver.allowMultipleContexts" "true")
                (spark/spark-context)))

Specify path to CSV on HDFS

(def iowa-liquor-sales "hdfs://localhost:9000/data/iowa_liquor_sales.csv")

From spark perspective, csv is just a plain file, and we need additional mapping to refer column by names

(def name->index
  {"DATE"                0
   "CONVENIENCE STORE"   1
   "STORE"               2
   "NAME"                3
   "ADDRESS"             4
   "CITY"                5
   "ZIPCODE"             6
   "STORE LOCATION"      7
   "COUNTY NUMBER"       8
   "COUNTY"              9
   "CATEGORY"            10
   "CATEGORY NAME"       11
   "VENDOR NO"           12
   "VENDOR"              13
   "ITEM"                14
   "DESCRIPTION"         15
   "PACK"                16
   "LITER SIZE"          17
   "STATE BTL COST"      18
   "BTL PRICE"           19
   "BOTTLE QTY"          20
   "TOTAL"               21})

Note: We can keep header in file on HDFS and build this mapping in runtime, but keeping data and metadata separatly is a good practice and this way we avoid a lot of pain.

And the last piece, we want a function to parse one csv line into vector

(defn csv-read-line [line]
  (first (csv/read-csv line)))

Let’s start.

SQL vs Spark

1. Select Star

Get a sense what data are you working with.

SQL

select * from iowa_liquor_sales

Spark

(->> (spark/text-file sc iowa-liquor-sales)
     (spark/map csv-read-line)
     (spark/collect))

Result

Query is running...

2. Count

Ouch, the query hangs. Seems we have a lot of data. Return the total number of rows.

SQL

select count(*) from iowa_liquor_sales

Spark

(->> (spark/text-file sc iowa-liquor-sales)
     (spark/count))

Result

3. Limit

There are over 3 million rows, inspect only 5 of them.

SQL

select * from iowa_liquor_sales limit 5

Spark

(->> (spark/text-file sc iowa-liquor-sales)
     (spark/map csv-read-line)
     (spark/take 5))

Result

╔════════════╦══╦══════╦═══════════════════════════════════╦════════════════╦════════════╦═══════╦═══════════════════════════════════════════════════════════════════════════╦════╦═════════════╦═════════╦═════════════════╦═════╦════════════════════════════╦═══════╦════════════════════════╦════╦══════╦═══════╦═══════╦════╦════════╗
║ 2015-02-26 ║  ║ 2515 ║ Hy-Vee Food Store #1 / Mason City ║ 2400 4TH ST SW ║ MASON CITY ║ 50401 ║ 2400 4TH ST SW MASON CITY 50401 (43.148463097000047, -93.236272961999987) ║ 17 ║ Cerro Gordo ║ 1022100 ║ TEQUILA         ║ 434 ║ Luxco-St Louis             ║ 87937 ║ Juarez Tequila Silver  ║ 12 ║ 1000 ║ 6.92  ║ 10.38 ║ 48 ║ 498.24 ║
║ 2015-02-26 ║  ║ 2515 ║ Hy-Vee Food Store #1 / Mason City ║ 2400 4TH ST SW ║ MASON CITY ║ 50401 ║ 2400 4TH ST SW MASON CITY 50401 (43.148463097000047, -93.236272961999987) ║ 17 ║ Cerro Gordo ║ 1022100 ║ TEQUILA         ║ 434 ║ Luxco-St Louis             ║ 87938 ║ Juarez Tequila Silver  ║  6 ║ 1750 ║ 13.50 ║ 20.25 ║  6 ║ 121.50 ║
║ 2015-02-26 ║  ║ 2515 ║ Hy-Vee Food Store #1 / Mason City ║ 2400 4TH ST SW ║ MASON CITY ║ 50401 ║ 2400 4TH ST SW MASON CITY 50401 (43.148463097000047, -93.236272961999987) ║ 17 ║ Cerro Gordo ║ 1012200 ║ SCOTCH WHISKIES ║  55 ║ Sazerac North America      ║  8824 ║ Lauder's               ║ 24 ║  375 ║ 3.21  ║ 4.82  ║  3 ║ 14.46  ║
║ 2015-02-26 ║  ║ 2515 ║ Hy-Vee Food Store #1 / Mason City ║ 2400 4TH ST SW ║ MASON CITY ║ 50401 ║ 2400 4TH ST SW MASON CITY 50401 (43.148463097000047, -93.236272961999987) ║ 17 ║ Cerro Gordo ║ 1022100 ║ TEQUILA         ║ 410 ║ The Patron Spirits Company ║ 88294 ║ Patron Silver Tequila  ║ 12 ║  375 ║ 14.00 ║ 21.00 ║  2 ║ 42.00  ║
║ 2015-02-26 ║  ║ 2515 ║ Hy-Vee Food Store #1 / Mason City ║ 2400 4TH ST SW ║ MASON CITY ║ 50401 ║ 2400 4TH ST SW MASON CITY 50401 (43.148463097000047, -93.236272961999987) ║ 17 ║ Cerro Gordo ║ 1022100 ║ TEQUILA         ║ 421 ║ Sazerac Co., Inc.          ║ 88767 ║ Tortilla White Tequila ║ 12 ║ 1000 ║ 6.64  ║ 9.96  ║ 24 ║ 239.04 ║
╚════════════╩══╩══════╩═══════════════════════════════════╩════════════════╩════════════╩═══════╩═══════════════════════════════════════════════════════════════════════════╩════╩═════════════╩═════════╩═════════════════╩═════╩════════════════════════════╩═══════╩════════════════════════╩════╩══════╩═══════╩═══════╩════╩════════╝

4. Select Fields

Document is too wide to concentrate, select specific columns.

SQL

select "DATE", "CATEGORY NAME", "DESCRIPTION", "TOTAL"
from iowa_liquor_sales limit 5

Spark

(->> (spark/text-file sc iowa-liquor-sales)
     (spark/map csv-read-line)
     (spark/map
	   (fn [row]
         (let [indices (->> ["DATE" "CATEGORY NAME" "DESCRIPTION" "TOTAL"]
                            (mapv name->index)
                            (set))]
           (keep-indexed (fn [i v] (if (indices i) v)) row))))
     (spark/take 5))

Ugh, so ugly.
Let’s extract select columns to the function and make sure we never see it again

(->> (spark/text-file sc iowa-liquor-sales)
     (spark/map csv-read-line)
     (columns ["DATE" "CATEGORY NAME" "DESCRIPTION" "TOTAL"])
     (spark/take 5))

If you go back and review previous examples, we almost always use the pair

(spark/text-file sc iowa-liquor-sales) ;; read file from HDFS
(spark/map csv-read-line)              ;; parse each csv line into vector 

Extract it to function as well. Functions everywhere!

Not it looks almost like SQL

(defn select-fields []
  (->> (read-csv sc iowa-liquor-sales)
       (columns ["DATE" "CATEGORY NAME" "DESCRIPTION" "TOTAL"])
       (spark/take 5)))

Result

╔════════════╦═════════════════╦════════════════════════╦════════╗
║ 2015-02-26 ║ TEQUILA         ║ Juarez Tequila Silver  ║ 498.24 ║
║ 2015-02-26 ║ TEQUILA         ║ Juarez Tequila Silver  ║ 121.50 ║
║ 2015-02-26 ║ SCOTCH WHISKIES ║ Lauder's               ║ 14.46  ║
║ 2015-02-26 ║ TEQUILA         ║ Patron Silver Tequila  ║ 42.00  ║
║ 2015-02-26 ║ TEQUILA         ║ Tortilla White Tequila ║ 239.04 ║
╚════════════╩═════════════════╩════════════════════════╩════════╝

5. Offset

Get another set of 5 rows somewhere in the middle.

SQL

select "DATE", "CATEGORY NAME", "DESCRIPTION", "TOTAL"
from iowa_liquor_sales limit 5 offset 1e6

Spark

(->> (read-csv sc iowa-liquor-sales)
     (columns ["DATE" "CATEGORY NAME" "DESCRIPTION" "TOTAL"])
     (spark/repartition 1)
     (spark/map-partition-with-index
       (fn [idx iter]
         (when (zero? idx)
           (dotimes [_ 1e6] (.next iter)))
         iter))
     (spark/take 5))

Spark does not have a method for skipping rows. In most cases it would be inefficient solution, because we don’t know how many data we have on each partition.

So the hack is simple:

Repartition all data into one partition (Ouch.)

(spark/repartition 1)

Apply skipping function at the start of this partition

(when (zero? idx)
  (dotimes [_ 1e6] (.next iter)))

We use .next to access native java.util.Iterator method.

Hackish, but works.

Result

╔════════════╦═════════════════════════╦════════════════════════════════════╦════════╗
║ 2014-08-13 ║ TEQUILA                 ║ Hornitos Lime Shot                 ║ 236.16 ║
║ 2014-08-13 ║ AMERICAN GRAPE BRANDIES ║ E & J Vs                           ║ 106.56 ║
║ 2014-08-13 ║ IMPORTED SCHNAPPS       ║ Dr. Mcgillicuddy's Cherry Schnapps ║ 312.24 ║
║ 2014-08-13 ║ IMPORTED DRY GINS       ║ Bombay Sapphire Gin                ║ 245.52 ║
║ 2014-08-13 ║ SCOTCH WHISKIES         ║ Scoresby Rare Scotch               ║ 99.00  ║
╚════════════╩═════════════════════════╩════════════════════════════════════╩════════╝

6. Sample

More efficient way to inspect random data is to use sample

SQL

select "DATE", "CATEGORY NAME", "DESCRIPTION", "TOTAL"
from iowa_liquor_sales
order by random()
limit 10

Spark

(->> (read-csv sc iowa-liquor-sales)
     (columns ["DATE" "CATEGORY NAME" "DESCRIPTION" "TOTAL"])
     (spark/sample true 5E-6 (System/currentTimeMillis))
     (spark/take 10))

Spark has a capability to sample RDD. All you need is to specify fraction of data, seed and flag which allows replacements.

Result

╔════════════╦═══════════════════════════╦══════════════════════════════════════════╦════════╗
║ 2015-01-07 ║ STRAIGHT BOURBON WHISKIES ║ Four Roses Yellow Label                  ║ 64.88  ║
║ 2014-10-22 ║ 80 PROOF VODKA            ║ Five O'clock Vodka                       ║ 10.12  ║
║ 2014-10-09 ║ IMPORTED SCHNAPPS         ║ Dr. Mcgillicuddy's Cherry Schnapps       ║ 198.00 ║
║ 2014-09-02 ║ CANADIAN WHISKIES         ║ Black Velvet                             ║ 94.20  ║
║ 2014-06-30 ║ TEQUILA                   ║ El Jimador Reposado Tequila              ║ 16.73  ║
║ 2014-06-18 ║ 80 PROOF VODKA            ║ Skol Vodka                               ║ 42.68  ║
║ 2014-05-05 ║ FLAVORED RUM              ║ Malibu Coconut Rum                       ║ 134.88 ║
║ 2014-04-23 ║ SCOTCH WHISKIES           ║ Lauder's                                 ║ 110.52 ║
║ 2014-04-08 ║ SCOTCH WHISKIES           ║ Cutty Sark Scotch                        ║ 71.70  ║
║ 2014-03-19 ║ WHISKEY LIQUEUR           ║ Fireball Cinnamon Whiskey Mini Dispenser ║ 133.74 ║
╚════════════╩═══════════════════════════╩══════════════════════════════════════════╩════════╝

7. Distinct

What alcohol categories do we have?

SQL

select distinct "CATEGORY NAME"
from iowa_liquor_sales

Spark

(->> (read-csv sc iowa-liquor-sales)
     (columns ["CATEGORY NAME"])
     (spark/distinct)
     (spark/collect))

Result

╔════════════════════════════════════╗
║ SINGLE BARREL BOURBON WHISKIES     ║
║ PEPPERMINT SCHNAPPS                ║
║ WHITE CREME DE CACAO               ║
║ IMPORTED VODKA                     ║
║ ROCK & RYE                         ║
║ AMERICAN SLOE GINS                 ║
║ WHISKEY LIQUEUR                    ║
║ BARBADOS RUM                       ║
║ IMPORTED GRAPE BRANDIES            ║
║ TENNESSEE WHISKIES                 ║
║ CREME DE ALMOND                    ║
║ IMPORTED SCHNAPPS                  ║
║ SCOTCH WHISKIES                    ║
║ APPLE SCHNAPPS                     ║
║ AMERICAN AMARETTO                  ║
║ IMPORTED DRY GINS                  ║
║ JAMAICA RUM                        ║
║ TRIPLE SEC                         ║
║ SPEARMINT SCHNAPPS                 ║
║ AMERICAN DRY GINS                  ║
║ BLENDED WHISKIES                   ║
║ DECANTERS & SPECIALTY PACKAGES     ║
║ HIGH PROOF BEER                    ║
║ BLACKBERRY BRANDIES                ║
║ DARK CREME DE CACAO                ║
║ DISTILLED SPIRITS SPECIALTY        ║
║ TEQUILA                            ║
║ OTHER PROOF VODKA                  ║
║ PEACH SCHNAPPS                     ║
║ MISC. IMPORTED CORDIALS & LIQUEURS ║
║ TROPICAL FRUIT SCHNAPPS            ║
║ CREAM LIQUEURS                     ║
║ LOW PROOF VODKA                    ║
║ CHERRY BRANDIES                    ║
║ WHITE CREME DE MENTHE              ║
║ RASPBERRY SCHNAPPS                 ║
║ CINNAMON SCHNAPPS                  ║
║ IMPORTED AMARETTO                  ║
║ SINGLE MALT SCOTCH                 ║
║ GREEN CREME DE MENTHE              ║
║ MISCELLANEOUS BRANDIES             ║
║ AMERICAN ALCOHOL                   ║
║ AMERICAN GRAPE BRANDIES            ║
║ FLAVORED VODKA                     ║
║ COFFEE LIQUEURS                    ║
║ IRISH WHISKIES                     ║
║ SPICED RUM                         ║
║ GRAPE SCHNAPPS                     ║
║ 100 PROOF VODKA                    ║
║ ANISETTE                           ║
║ STRAWBERRY SCHNAPPS                ║
║ AMERICAN COCKTAILS                 ║
║ 80 PROOF VODKA                     ║
║ FLAVORED GIN                       ║
║ MISC. AMERICAN CORDIALS & LIQUEURS ║
║ BUTTERSCOTCH SCHNAPPS              ║
║ PUERTO RICO & VIRGIN ISLANDS RUM   ║
║ PEACH BRANDIES                     ║
║ WATERMELON SCHNAPPS                ║
║ BOTTLED IN BOND BOURBON            ║
║ APRICOT BRANDIES                   ║
║ FLAVORED RUM                       ║
║ MISCELLANEOUS SCHNAPPS             ║
║ ROOT BEER SCHNAPPS                 ║
║ STRAIGHT BOURBON WHISKIES          ║
║ STRAIGHT RYE WHISKIES              ║
║ IMPORTED VODKA - MISC              ║
║ CANADIAN WHISKIES                  ║
╚════════════════════════════════════╝

8. Filters

Ok, we’ve got a sense what type of drinks people in Iowa like.
Let’s now select some bourbon and whiskey related categories.

SQL

select "CITY", "CATEGORY NAME", "DESCRIPTION", "TOTAL" 
from iowa_liquor_sales
where "CATEGORY NAME" ilike '%bourbon%' or
      "CATEGORY NAME" ilike '%whisk%' 
limit 10;

Spark

(->> (read-csv sc iowa-liquor-sales)
     (columns ["CITY" "CATEGORY NAME" "DESCRIPTION" "TOTAL"])
     (spark/filter (fn [[_ cat _ _]] (re-find #"(?i)(bourbon|whisk)" cat)))
     (spark/take 10))

Result

╔════════════╦═══════════════════════════╦═══════════════════════════════╦════════╗
║ MASON CITY ║ SCOTCH WHISKIES           ║ Lauder's                      ║ 14.46  ║
║ DES MOINES ║ CANADIAN WHISKIES         ║ Black Velvet Toasted Caramel  ║ 121.56 ║
║ DES MOINES ║ CANADIAN WHISKIES         ║ Black Velvet Cinnamon Rush    ║ 121.56 ║
║ DES MOINES ║ CANADIAN WHISKIES         ║ Black Velvet                  ║ 188.40 ║
║ DES MOINES ║ CANADIAN WHISKIES         ║ Black Velvet                  ║ 376.32 ║
║ DES MOINES ║ CANADIAN WHISKIES         ║ Windsor Canadian Pet          ║ 82.62  ║
║ DES MOINES ║ BOTTLED IN BOND BOURBON   ║ Evan Williams Bottled in Bond ║ 152.52 ║
║ DES MOINES ║ STRAIGHT BOURBON WHISKIES ║ Elijah Craig                  ║ 42.12  ║
║ DES MOINES ║ STRAIGHT BOURBON WHISKIES ║ Evan Williams Str Bourbon     ║ 126.60 ║
║ DES MOINES ║ STRAIGHT BOURBON WHISKIES ║ Evan Williams Green Label     ║ 44.46  ║
╚════════════╩═══════════════════════════╩═══════════════════════════════╩════════╝

9. Filters, Part II

We are not interested in sales lower than $100

SQL

select "CITY", "CATEGORY NAME", "DESCRIPTION", "TOTAL" 
from iowa_liquor_sales
where ("CATEGORY NAME" ilike '%bourbon%' or
      "CATEGORY NAME" ilike '%whisk%') and
      "TOTAL" > 100.0
limit 10;

Spark

(->> (read-csv sc iowa-liquor-sales)
     (columns ["CITY" "CATEGORY NAME" "DESCRIPTION" "TOTAL"])
     (spark/filter (fn [[_ cat _ _]] (re-find #"(?i)(bourbon|whisk)" cat)))
     (spark/filter (fn [[_ _ _ total]] (> (Double/parseDouble total) 100)))
     (spark/take 10))

Few things here.
We used two different filters here instead of one for better composability.
For “total”-filter we used Double/parseDouble function, because we have all data in raw string representation. If we use several filters at once or perform aggregation (later) we need to have typed data, so, provide additional mapping for types.

Mapping would change to this structure

{ "DATE"                {:index 0 :type :date :parse-fn #(.parse (java.text.SimpleDateFormat. "yyyy-MM-dd") %)}
  "CITY"                {:index 5 :type :string :parse-fn idenity}
  "CATEGORY"            {:index 10 :type :string :parse-fn identity}
  "DESCRIPTION"         {:index 15 :type :string :parse-fn identity}
  "TOTAL"               {:index 21 :type :string :parse-fn #(Double/parseDouble %)}

Then apply it, during columns retrieval.

(defn- columns [columns rdd]
  (spark/map 
   (fn [row]
     (let [indices (->> columns
                        (map #(get-in name->index [% :index]))
                        (set))]
       (->> row
            (keep-indexed (fn [i v] (if (indices i) v)))
	        ;; this part
            (map-indexed (fn [idx elem]
                           (if (clojure.string/blank? elem)
                             nil
                             (let [column (nth columns idx)
                                   parsefn (get-in name->index [column :parse-fn])]
                               (parsefn elem)))))
            (vec))))
   rdd))

Do not forget to filter out nulls.
Final solution.

(->> (read-csv sc iowa-liquor-sales)
       (columns ["CITY" "CATEGORY NAME" "DESCRIPTION" "TOTAL"])
       (spark/filter (fn [[_ cat _ _]] (and cat (re-find #"(?i)(bourbon|whisk)" cat))))
       (spark/filter (fn [[_ _ _ total]] (and total (> total 100))))
       (spark/take 10))

Result

╔════════════╦═══════════════════════════╦═══════════════════════════════╦════════╗
║ DES MOINES ║ CANADIAN WHISKIES         ║ Black Velvet Toasted Caramel  ║ 121.56 ║
║ DES MOINES ║ CANADIAN WHISKIES         ║ Black Velvet Cinnamon Rush    ║ 121.56 ║
║ DES MOINES ║ CANADIAN WHISKIES         ║ Black Velvet                  ║ 188.40 ║
║ DES MOINES ║ CANADIAN WHISKIES         ║ Black Velvet                  ║ 376.32 ║
║ DES MOINES ║ BOTTLED IN BOND BOURBON   ║ Evan Williams Bottled in Bond ║ 152.52 ║
║ DES MOINES ║ STRAIGHT BOURBON WHISKIES ║ Evan Williams Str Bourbon     ║ 126.60 ║
║ DES MOINES ║ STRAIGHT BOURBON WHISKIES ║ Jim Beam                      ║ 188.88 ║
║ DES MOINES ║ STRAIGHT BOURBON WHISKIES ║ Maker's Mark                  ║ 311.76 ║
║ DES MOINES ║ STRAIGHT BOURBON WHISKIES ║ Wild Turkey 101               ║ 243.00 ║
║ DES MOINES ║ STRAIGHT BOURBON WHISKIES ║ Woodford Reserve Bourbon      ║ 189.18 ║
╚════════════╩═══════════════════════════╩═══════════════════════════════╩════════╝

10. Order

Find out the most expensive deal.

SQL

select "CITY", "CATEGORY NAME", "DESCRIPTION", "TOTAL" 
from iowa_liquor_sales
where ("CATEGORY NAME" ilike '%bourbon%' or
      "CATEGORY NAME" ilike '%whisk%')
order by "TOTAL" desc
limit 1;

Spark

(->> (read-csv sc iowa-liquor-sales)
     (columns ["CATEGORY NAME" "DESCRIPTION" "TOTAL" "BOTTLE QTY"])
     (spark/filter (fn [[cat _ _ _]] (and cat (re-find #"(?i)(bourbon|whisk)" cat))))
     (spark/map-to-pair
      (fn [[_ _ total _ :as row]]
        (spark/tuple total row)))
     (spark/sort-by-key > false)
     (spark/values)
     (spark/take 1))

Spark can sort key-value pair by key. To use this functionality we need to convert our row into key value pair, where key is total and value is whole row.

After sorting, we just need our values back.

Result

╔═══════════════════╦═════════════════════════════╦════════╦══════════╗
║ CANADIAN WHISKIES ║ Crown Royal Canadian Whisky ║ 3456.0 ║ 94590.72 ║
╚═══════════════════╩═════════════════════════════╩════════╩══════════╝

OMG, someone spent almost $100000 on whiskey! 3456 bottles, nice party!

11. Aggregation

We also have a party soon, and we need some really cheap bourbon.

Note: Make sure to take bottle volume into account. Calculate price per liter.

SQL

select "DESCRIPTION", 
       avg("BTL PRICE" * 1000 / "LITER SIZE") as price_per_liter  
from iowa_liquor_sales
where "LITER SIZE" > 0 and
      "CATEGORY NAME" ilike '%bourbon%'
group by "DESCRIPTION"      
order by price_per_liter asc
limit 10;

Spark

(->> (read-csv sc iowa-liquor-sales)
     (columns ["CATEGORY NAME" "DESCRIPTION" "LITER SIZE" "BTL PRICE"])
     (spark/filter (fn [[cat _ _ _]] (and cat (re-find #"(?i)bourbon" cat))))
     (spark/filter (fn [[_ _ liter _]] (and liter (pos? liter))))
     (spark/map (fn [[_ _ liter price :as row]] (conj row (/ (* 1000 price) liter))))
     (spark/map-to-pair (fn [[_ desc _ _ _ :as row]] (spark/tuple desc row)))
     (spark/combine-by-key
      (fn [row] (conj row 1))
        (fn [acc row] (-> acc
                          (update-in [4] + (nth row 4))
                          (update-in [5] inc)))
        (fn [acc1 acc2] (-> acc1
                            (update-in [4] + (nth acc2 4))
                            (update-in [5] + (nth acc2 5)))))
     (spark/map-values
       (fn [[cat desc price liter total-price-per-liter cnt]]
         [cat desc price liter (/ total-price-per-liter cnt)]))
     (spark/values)
     (spark/map-to-pair
       (fn [[_ _ _ _ avg-price-per-liter :as row]]
         (spark/tuple avg-price-per-liter row)))
     (spark/sort-by-key < false)
     (spark/values)
     (spark/take 1))

Are you kidding me?
So much code to calculate an average?

Result

╔═══════════════════════════╦════════════════════════════╦════════╦══════╦══════╗
║ STRAIGHT BOURBON WHISKIES ║ Kentucky Gentleman Bourbon ║ 1000.0 ║ 6.74 ║ 6.74 ║
╚═══════════════════════════╩════════════════════════════╩════════╩══════╩══════╝

12. Aggregation, Part II

“Kentucky Gentleman Bourbon” only for $6.74/liter

Seems, not a very popular bourbon.
Let’s better calculate score for each bourbon which depends on total liters sold and average price per liter.

Formula is simple:

More liters sold, more popular, more score.
Bigger average price, less score.

Find the top 3 bourbons by score.

SQL

with rating as (
  select "CATEGORY NAME", "DESCRIPTION",
         sum("BTL PRICE" * "BOTTLE QTY") as liters_sold,
         avg(("BTL PRICE" * 1000) / "LITER SIZE") as price_per_liter
  from iowa_liquor_sales
  where "LITER SIZE" > 0 and
        "CATEGORY NAME" ilike '%bourbon%'
  group by "CATEGORY NAME", "DESCRIPTION"
)
select "DESCRIPTION", liters_sold/price_per_liter as score
from rating
order by score desc
limit 3

Spark

(->> (read-csv sc iowa-liquor-sales)
     (columns ["CATEGORY NAME" "DESCRIPTION" "LITER SIZE" "BTL PRICE" "BOTTLE QTY"])
     (spark/filter (fn [[cat _ _ _]] (and cat (re-find #"(?i)bourbon" cat))))
     (spark/filter (fn [[_ _ liter _]] (and liter (pos? liter))))
     (spark/map (fn [[_ _ liter price _ :as row]] (conj row (/ (* 1000 price) liter))))
     (spark/map (fn [[_ _ _ price qty _ :as row]] (conj row (* price qty))))
     (spark/map-to-pair (fn [[_ desc _ _ _ _ _ :as row]] (spark/tuple desc row)))
     (spark/combine-by-key
       (fn [row] (conj row 1))
       (fn [acc row] (-> acc
                         (update-in [5] + (nth row 5)) ;; sum of price per liters
                         (update-in [6] + (nth row 6)) ;; sum of liters sold
                         (update-in [7] inc)           ;; count 
                      ))
       (fn [acc1 acc2] (-> acc1
                           (update-in [5] + (nth acc2 5))
                           (update-in [6] + (nth acc2 6))
                           (update-in [7] + (nth acc2 7)))))
     (spark/map-values
       (fn [[cat desc price liter total-ppl liters-sold cnt]]
         [desc (/ liters-sold (/ total-ppl cnt))]))
     (spark/values)
     (spark/map-to-pair
       (fn [[desc score :as row]]
         (spark/tuple score row)))
     (spark/sort-by-key < false)
     (spark/values)
     (spark/take 3))

Result

╔══════════╦════════════════════╗
║ Jim Beam ║ 267415.26103858394 ║
║ Ten High ║ 181012.79177772935 ║
║ Old Crow ║ 102744.61073467274 ║
╚══════════╩════════════════════╝

Congratulations, we’ve found top 3 bourbons in Iowa using Clojure and Spark.

Interesting that top 3 bourbons have similar names.
Two words, two syllables.

Jim Beam, Ten High, Old Crow, Face Book, Goo Gle, Net Flix, Ya Hoo…

Their marketing department know something.

Conclusion

We’ve seen Spark has a good functional style API, but simple tasks are not so obvious as in SQL.

Anyway, this was just a practice and there is an awesome SparkSQL

Emacs: org mode

2015-03-27T14:30:00+00:00

Emacs org-mode use cases, for non-programmers.

org-mode

Org-mode is what makes emacs popular among non-programmers community. It could replace your Notes, TODO, Calendar, Agenda, Project Management, Time Tracking, Spreadsheets and other apps.

But seems like most of org-mode tutorials are dedicated to hardcore emacs users, these tutorials are hard to follow as they are exposing all-the-power of org-mode.

Instead of drinking from the firehose, let’s focus on some basic functionality, and you can incrementally improve it based on own needs. Just skip functionality you don’t care.

Prerequisites: emacs novice
Keybindings: based on my configuration

Index

Notes
- New File
- Markup
TODO
- List
- Nested Lists
- Folding
- Search
- Manage Nesting
- Parent
- Narrow
- Delete
- Completion
- Description
- Sparse Tree
- Statistics
- Local References
- File References

<div id="notes"/>Notes

<div id="notes_new_file"/>New File

Create a plain text file in emacs and name it notes.org. Emacs looks at .org extension and automatically enables org-mode
This is where journey begins!

C-x C-f - create file
C-x C-s - save file

<div id="notes_markup"/>Markup

Plain text is the best format ever. Both Human and Computers can read it. Org-mode improves it by providing a simple markup syntax to emphasize some points.

<div id="todo"/>TODO

<div id="todo_list"/> List

Plain text is not well suited for planning, Lists are. To make a list item in org mode, just put a * before entry title.

Make a plan for birthday celebration.

<div id="todo_nested_lists"/>Nested Lists

You can split each task into subtasks by adding a nested list.
Just one more * introduce nested list.

Add more details to plan

<div id="todo_folding"/>Folding

So far, we have seen functionality available in every plain text editor, where is the power of org-mode?
Press TAB on list item and it will collapse/expand all the subitems in a tree.

Hide all details

As you noticed in minibuffer tree different messages appear

FOLDED fold all children and show only top-level item
CHILDREN show current item and all its direct children
SUBTREE expand whole tree for current item, includes deep nesting as well

<div id="todo_search"/>Search

Even if your items are folded, you still have the power of search.

Make sure pork is in the list.

C-s - search

<div id="todo_move_items"/>Move Items

Having tree of tasks it’s easy to mimic priorities and move most important items to the top of list.

Call friends first, maybe they are busy and you will celebrate birthday alone

C-S-up - move subtree up
C-S-down - move subtree down

Have you noticed C-up/down is used to jump items with the same nesting level?

<div id="todo_manage_nesting"/>Manage Nesting

Buying food and buying drinks are actually one activity - going to shop. Let’s fix that.

M-RET - create item with the same nesting level
M-right/left - add/remove one nesting level for item
C-M-right/left - add/remove one nesting level for subtree

<div id="todo_parent"/>Parent

Having lot of items and deep nesting makes easy to get lost.
M-<up> is your Ariadne’s thread.

<div id="todo_narrow"/>Narrow

Other approach to limit a visible part is narrowing.

Let’s focus on syrups for now

C-o n - narrow to subtree
C-o w - widen to whole file

Note, there is a Narrow text indicates that you are in narrowed mode.

<div id="todo_delete"/>Delete

Narrow mode is great feature to focus. But the best way to focus is to cleanup obsolete items.

Review the tasks and clean obsolete

C-o k - kill subtree

<div id="todo_completion"/>Completion

The most important feature of todo lists is tracking completion. Sure you can delete completed tasks, but how to report later?

Meet special markers TODO/DONE

Review the tasks and mark what is TODO and what is already DONE

S-right - cycle item through states {EMPTY, TODO, DONE}

<div id="todo_description"/>Description

Avoid long names for titles. Give title a self-contained short name and provide detailed desription inside.

Describe what tickets section mean

You still can navigate items, skipping description.
M-down - go to next visible item

<div id="todo_sparse_tree"/>Sparse Tree

Sparse trees are used if you need find and highlight all occurences of some item, and unfold tree for matching clauses.

List all TODO items

C-o s - sparse tree
C-o s t - sparse tree of TODO items
C-o e - next highlighted element
C-o r - previous highlighted item
C-c C-c - (sparse tree context) clear highlighting

<div id="todo_statistics"/>Statistics

Often you need to get a sense how many TODO items in subtree.
Type [/] (total number of todo items) or [%] (completion percentage) in the item heading. Stats are automatically updated if subitem status changes.

C-c C-c - (statisics context) force refresh statistics

If all subtasks are DONE, than the parent task considered DONE as well.

<div id="todo_local_refs"/>Local References

Information can be in the same document but in another place. Use local references.

<<ref>> - create a reference
[[ref][Ref Title]] - creates a link that follows to ref

Add responsible for tickets design

<div id="todo_file_refs"/>File References

If additional information is in another file, instead of copying content you can refer it as well. Just another syntax.

[[file:~/games.txt::27]] - creates a link to external file games.txt, line:22

Provide links for games

I plan to update this post later with exposing more advanced functionality of org-mode: item properties, estimates, project planning, calendar and spreadsheets. If you don’t want to miss an update subscribe to rss or follow me on twitter

Code Golf: Calculating PI

2015-03-14T09:56:00+00:00

Let’s celebrate Pi day in clojure!

Exactly 140 characters to print 9 digits of $\pi$ (3.141592654)

(let[r reduce N(rest(range))t take](clojure.pprint/cl-format nil"~11f"
(r +' -2(map #(/(r *' 2(t % N))(r *'(t %(take-nth 2 N))))(t 99 N)))9))

“Sugared” version

There are two parts of a problem:

Calculate rational for $\pi$ using arctangent approximation

\[\pi = 2 \cdot (1 + \frac{1}{3} + \frac{1 \cdot 2}{3 \cdot 5} + \frac{1 \cdot 2 \cdot 3}{3 \cdot 5 \cdot 7} + ...)\]

(defn pi 
  "Calculate PI by arctangent algorithm"
  [& {:keys [iterations limit] 
      :or {iterations 99 limit 9}}]
  (->> (range 1 (inc iterations))
       (map #(/ (reduce *' (take % (iterate inc 1)))
                (reduce *' (take % (filter odd? (iterate inc 1))))))
       (reduce +')
       (* 2)
       (dec)
       (dec)))

Convert rational to decimal. Handled automatically by awesome clojure.pprint/cl-format

(clojure.pprint/cl-format nil "~4f" 22/7) ;;=> "3.14"

Tweaks

To achieve 140 characters barrier, some functionality was relaxed

Tweak number 99 to control number of iterations for $\pi$ calculation (if you have a high number of iterations, be patient - wait)
Tweak number 9 to increase number of digits (that makes sense if you improved number of iterations)

P.S. Calculating $\pi$ and some other irrationals is available in numberto

Java Magic. Part 5: SecurityManager

2014-11-18T15:27:00+00:00

Pouring a bit light on SecurityManager and its use cases.

Intro

You can do a terrible things in java using sun.misc.Unsafe class. Some really creepy examples were discussed in Java Magic. Part 5: sun.misc.Unsafe

SecurityManager is a guard, which could help to prevent some sensitive actions (io, net, reflection, access etc.)

SecurityManager manager = System.getSecurityManager();
if (manager != null) {
    manager.checkAction(action);
}

If action is not allowed SecurityExeption occurs.

Use Case

Now its a time to write some code.

Assume we are building online grader, a system which accepts some java code, runs it, gets results and verify that results are correct. Such graders are useful for computer science courses in MOOC platforms like coursera, udacity, etc.

Obviously, running untrusted code is unsafe, so we need to make sure code submitter does not break/compromise whole grader system. For example such sumbitter could read passwords and modify grading entry in database. Or even worse, it could fill out the whole file system, RAM or consume all threads and prevent grading for other submitters.

SecurityManager solves these issues.
Extend it and override needed policies, specifying what is allowed and what is not.

class MySecurityManager extends SecurityManager {
    @Override
    public void checkRead(FileDescriptor fd) {
        throw new SecurityException("File reading is not allowed");
    }

    @Override
    public void checkWrite(FileDescriptor fd) {
        throw new SecurityException("File writing is not allowed");
    }

    @Override
    public void checkConnect(String host, int port) {
        throw new SecurityException("Socket connections are not allowed");
    }
}

You can set such security manager in runtime using:

System.setSecurityManager(new MySecurityManager());

Note: setSecurityManager is controlled by security manager as well.

If some restricted action is executed, SecurityException occurs.

Inspect methods from SecurityManager which starts with check prefix. There are plenty of checks JVM may run before your code.

Though, security manager is useful tool for configuring access to subsystems and prevent untrusted code from doing a terrible things, some actions are not controlled by security manager.

Memory Allocation

(un)fortunately, memory allocation is not controlled by a programmer and the same way SecurityManager can’t restrict object creation. If you need validate that untrusted code fulfills memory requirements, execute it in a separate JVM and give it maximum amount of memory java -Xmx128m. If memory requirements are broken OutOfMemory occurs, but as long as this was executed in another JVM, this won’t affect grader.

For more accurate memory management you need to attach instrumentation agent to a java process.

Threads

There is no way to limit number of threads spawned by a java process. If only ExecutorService responsible for thread creation, then introduce limit by using ExecutorService.newFixedThreadPool(limit) inside the code and make this as a convention.

Otherwise you need to write a custom agent that tracks number of active threads. Such functionality available in some proprietary java agents.

Timeouts

To make sure that program finishes in specific time period, use some external tool for setting a timeout for it. For linux it’s a coreutils project, see a related discussion on stackoverflow

Libraries

You can prevent whole package usage by SecurityManager.checkPackageAccess. The same way you can prevent usage of some external libraries or products from whole organisations. But if you want prevent usage of some specific method, like java.lang.Math.min(), you probably need to manually scan java source file and detect such call.

ClojureCup 2014: Impressions

2014-10-01T00:30:00+00:00

Clojure Cup 2014 has finished and we’ve build clojure app in two days!
We are proud to announce funstructor (pre-alpha)

Funstructor

Funstructor is a turn-based multiplayer card game for clojurists. Your goal is to make some clojure code snippet, function or expression, but instead of typing characters you must use cards.

Every card has unique behaviour and I suggest actually play game, instead of reading what this game is about.

Tutorial might help to understand the basics.

split-brain team

(sort-by count team)

Max (frontend, javascript, design)
Dima (frontend, backend, clojure, clojurescript, ops)
Oleg (backend, clojure, core.async, websockets)
Misha (backend, clojure, game logic, art)

Good Parts

In one place

We were working in one office, so it was easy for us to communicate and solve problems. Hackathon is not solely about 48 hours coding, it is more about learning new things while having fun. I suppose, sitting two days at home alone is not much fun, even if you are hacking Clojure.

Protocols and API

At the start of the hackathon we’ve made some planning and designed protocols and API for communication. Obviously, this API was improving regularly, but the main profit is we could work independently. Even more, we had nothing interesting to show after first day of coding. At the next day, when we’ve connected our parts together, magic things happened. It worked.

Client was communicating with server via WebSockets by sending messages like below

{:type :game-request
 :data {:player-name "Rich Hickey"}}

{:type :action
 :data {:player-name "Rich Hickey"
        :card-position 3
		:target :self
		:args [2]}}

And server was using game logic API with small set of functions

(make-game)
(end-turn-for-player)
(end-turn)
(use-card)

REPL

Actually, all worked because of REPL. No unit-testing, just REPL. Write some small function, test it locally on the game state, handle corner-cases, test again and commit.

WebSocket

Websockets with chord and core.async as easy as pie.

(defn ws-handler [{:keys [ws-channel] :as req}]
  (go-loop []
    (let [{:keys [message error] :as msg}
	      (async/<! ws-channel)] ;; reading from ws channel
	  (process-message message)
      (recur))))

(defroutes app-routes
  (GET "/ws" []
    (-> ws-handler
        (wrap-websocket-handler {:format :str
                                 :read-ch (chan)
                                 :write-ch (chan)}))))

State? You’re doing it wrong

Another good part that game logic was coded without atoms and refs. We have map that represent game state and we have bunch of pure functions which make some operations and return new map.

Server code that make update to atom, uses swap! in one place. What can be better?

Looks like good thought is to replace global state with flow that core.async provides. Instead of swap!ping our global atom with new updated state, server side can be decomposed into several processes:

handshake-process: accepts websocket channel and waits for game-request. When player requests for game it sends websocket and newly created player id to pending-checker-process
pending-checker-process: accumulates all pending players and initiates game-process with random pair of players.
game-process: accepts initial game state, processes actions from players and sends updates. State is updated locally inside go-loop

Multimethods

Multimethods are awesome. We need to handle lot of cards in a similar way, cond would be a total mess. Instead we just add multimethod handles specific card.

(defmethod apply-card
  :mutator-right-gap
  [game-map player-key card & args]
  ((apply-to-funstruct game-map player-key)
   (fn [funstruct]
     (conj funstruct (gap)))))

Adhoc Hierarchies

Lot of cards are handled the same way. For example terminal cards are separated token cards with different keys, but have the same implementation. Adhoc hierarchies handle that very well.

;; build hierarchy
(def terminal-hierarchy
  (-> (make-hierarchy)
      (derive :terminal-left-paren :terminal)
      (derive :terminal-right-paren :terminal)
      (derive :terminal-left-square :terminal)
      (derive :terminal-right-square :terminal)))

;; add hierarchy to multimethod
(defmulti apply-card
  "Modify game-state map and return this map"
  (fn [m p card & args] card)
   :hierarchy #'terminal-hierarchy)

;; add parent implementation
(defmethod apply-card
  :terminal
  [game-map player-key card & args]
  (let [[pos & _] args]
    ((apply-to-funstruct game-map player-key)
     (fn [funstruct]
       (assoc funstruct pos
              {:terminal
               (get-in c/cards [card :value])})))))

Bad Parts

Confusion

Our bad we didn’t provide instructions for people how to play, we just assumed everyone knows.

First time user press “Play”, white rectangle is rolling and seems that nothing works. Actually, the game is multiplayer and it seeks for an opponent. Our game is not popular yet, so ask your friend to play with you or open game in two different browser tabs.

(Clojure/Java)Script

In any way we don’t want to say here that ClojureScript is bad or something like that. We didn’t actually had anyone in our team who had prior experience with ClojureScript. And it didn’t feel like you can start hacking ClojureScript from the first day even if you have experience with JavaScript.

Front-end in ClojureScript was the highest risk in our application and we couldn’t manage to implement everything in it in time. So we had to fallback to JavaScript and React.js in order to deliver a playable application. That said, learning curve with ClojureScript is steep and we should have started learning it long before the hackathon.

Try Catch

Suddenly, client sends a corrupted data to a server, using card that needs target position, without position. To save time we use straightforward approach: wrapped call to use-card in try-catch block, logged the wrong parameters and returned game map without modifications.

(catch Exception e
  (do
    (u/log
      (.getMessage e)
      "\n"
      "Critical Error:\n\n"
      "Gamestate: " game-map "\n"
      "Player:"     player-key "\n"
      "Card"        (get-card game-map player-key card-pos) "\n"
      "Args[]:"     args "\n")
      game-map))

From the client perspective it looks like the actions are not applied and he did something wrong, so he’ll try again. But, we are pretty sure there is more elegant solution.

Losing the type

Pay attention to the datastructures and operations you apply on them. Consider function which adds card to the tail of vector

(defn add-card [cards card]
  (conj cards card))

This function remove all the gaps from the vector

(defn remove-gaps [cards]
  (remove #(= :gap %) cards))

But! remove transformed cards vector to a sequence, and next call to add-card adds it to the head of the list. So instead remove use filterv

(defn remove-gaps [cards]
  (filterv #(not= :gap %) cards))

Multimethod Unmap

This is annoying one, we wasted some time to understand why we getting some strange error. The problem is if you change number of arguments of your multimethod, it is NOT updated in namespace. Do ns-unmap before.

Private methods

Creating function with defn makes hard to refactor namespace and move some functions to another. You can’t be sure nobody at external namespace references this function.

Create all functions with defn- by default and expose it only if needed.

core.async error handling

It was really hard to figure out where exactly some exception occured inside go block. Stacktrace was not printed, you had only exception name and guesses what happened.

Appears that chan function can accept optional exception handler which is executed when something bad happens during execution of put or take callbacks. So we were creating channels as follows:

(chan nil nil #(.printStackTrace %))

Also make sure that you don’t return anything in exception handler function, because all non-nil return values will be placed on channel.

Future

We plan to continue working on the game in free time and our first priority improvements are:

Nice tutorial to help people understand the game
Rewrite UI from JavaScript/React.js to ClojureScript/Reagent
Add timeout for the game turn
Private battles
Play against computer
More cards, good art & game balance

Conclusion

We really enjoyed coding for ClojureCup this year and will definetely participate next year. ClojureCup and hackathons like that is a great way to apply knowledge that you accumulate throughout the year by doing something cool and learning from other teams.

We are a bit saddened that we couldn’t deliver everything that we wanted to accomplish in our application but it was educational and fun! Huge thanks to ClojureCup organizers for all the work that was done to make it happen! Also, kudos to all the teams who participated, you’re all super creative and we learned a lot from you! See you next year at ClojureCup 2015!

Final Notes

If you interested in code check out repo

And finally, if you liked our app, please, vote!

Clojure Euler: Problem 025

2014-09-09T00:51:00+00:00

What is the first term in the Fibonacci sequence to contain 1000 digits?

Permalink: https://projecteuler.net/problem=25

We’ve introduced Fibonacci sequence at Clojure Euler: Problem 002

Lazy sequence generates fibonacci numbers is dead simple:

(defn fibonacci []
  (->> [0 1]
       (iterate (fn [[a b]] [b (+' a b)]))
       (map first)))

We also can count the length of number in digits. No decimal arithmetic. Cast number to string, and take length of a string

(defn num-of-digits [n]
  (count (str n)))

To solve the problem use the most straightforward approach and you’ll find the answer in less than 1 second.

(->> (fibonacci)
     (map-indexed (fn [i n] [i (num-of-digits n)]))
     (drop-while #(< (second %) 1000))
     (first)
     (first))

Code

P.S. If you followed all posts upto this one, then you have 25 problems solved, level 1 and nice badge

Clojure Euler: Problem 024

2014-08-28T00:43:00+00:00

A permutation is an ordered arrangement of objects. For example, 3124 is one possible permutation of the digits 1, 2, 3 and 4. If all of the permutations are listed numerically or alphabetically, we call it lexicographic order. The lexicographic permutations of 0, 1 and 2 are:

012 021 102 120 201 210

What is the millionth lexicographic permutation of the digits 0, 1, 2, 3, 4, 5, 6, 7, 8 and 9?

Permalink: http://projecteuler.net/problem=24

Instead of developing permutation function we use math.combinatorics

permutations function does the trick

(->> (range 10)
     (comb/permutations)
     (drop (dec 1000000))
     (first)
     (apply str)))

That was easy! It returns result in just 3 seconds, what is acceptable.

If you need to solve this problem faster, observe the interesting properties:

1      - 0123456789
2      - 0123456798
...
9!     - 0987654321
9!+1   - 1023456789
...
2*9!   - 1987654320
2*9!+1 - 2013456789
...
3*9!   - 2987654310
3*9!+1 - 3012456789

3*9! is greater than 1000000, so we stop on the 2*9! permuations and guessed first number, it’s 2

Proceed the same way and you’ve got the idea. On each factorial permutation step, when you guessed the number remove it from further permutations and decrement the factorial

(let [! (fn [n] (reduce * (range 1 (inc n))))]
  (loop [available-digits (range 10)
         num 1000000
         current-digit 0
         init 9
         result []]
    (let [f (! init)]
      (cond
        (= 0 init)
        (apply str (concat result available-digits))
         
        (< f num) (recur available-digits
                         (- num f)
                         (inc current-digit)
                         init
                         result)

        :else (recur (concat (take current-digit available-digits)
                             (drop (inc current-digit) available-digits))
                     num
                     0
                     (dec init)
                     (conj result (nth available-digits current-digit)))))))

Solution is less readable, but time is just 5 msecs. There is always tradeoff.

Code

Effective Java

2014-08-07T13:54:00+00:00

Recently, I’ve re-read awesome java book Effective Java by Joshua Bloch. The book contains 78 independent items, discussing various aspects of programming in java. Something like mini-design patterns with emphasis on their pros and cons.

Few notes from each item as a refresher.

Item 1: Consider static factory methods instead of constructors

Static factory methods have more informative names than constructors
Same parameters list could be applied
Not required to create new objects, could return cached instance
Static factory methods could return object subtype
Reduced verbosity for generics due to type inference
Classes without public/private constructor can’t be subclassed, but it is good, because it enforces to “favor composition over inheritance”
Hard to distinguish from other static methods. To avoid confusion use common names like newIntance, valueOf, etc.

Item 2: Consider a builder when faced with many constructor parameters

Telescope Constructor causes verbosity
JavaBeans may cause inconsistent state, no possibbility to make a class immutable
Builder Pattern is flexible and right way to handle optional parameters

Item 3: Enforce the singleton property with a private constructor or an enum type

Caution: Discussed singleton without lazy initialization

Throw an exception in a private constructor to avoid reflection call to constructor
If standard serialization is needed make all fields transient and override readResolve method
Best way to use single element enum as a singleton

enum Singleton {
  INSTANCE
}

Item 4: Enforce noninstantiability with a private constructor

Include a single private constructor to a class to prevent it from instantiation
Throw an exception in constructor if it is called
Almost always used technique for utility classes

Item 5: Avoid creating unnecessary objects

"hello" is better than new String("hello")
Boolean.valueOf("true") is better than new Boolean("true")
Immutable objects could be reused for free
Mutable objects could be reused if you really sure, they won’t be modified
prefer primitives to boxed primitives
watch out for hidden autoboxing

Item 6: Eliminate obsolete object references

Garbage collector is not savior from memory leaks
Nullify obsolete references
Invalidate cache periodically
Deregister outdated listeners and callbacks

Item 7: Avoid finalizers

Finalizers are not destructors
No guarantee finalizers will be executed promptly
No guarantee finalizers will be executed at all
System.gc just a hint, not a gc call
Finalizers cause severe performance penalty
Use own explicit methods for finalization like close()

Item 8: Obey the general contract when overriding equals

Overridden equals should follow equivalence relation
- Reflexive, x.equals(x) == true
- Symmetric, x.equals(y) == y.equals(x)
- Transitive, x.equals(y) and y.equals(z) == x.equals(z)
- Consistent, consequent calls x.equals(y) should produce the same value
- x.equals(null) == false if x not null
There is no way to extend an instantiable class and add a value component while preserving the equals contract
Do not write an equals method that depends on unreliable resources. This was discussed at the Java Magic: Part I
Always override hashCode when you override equals
Don’t substitute param type in equals, that cause method overload instead of override. Use @Override annotation to be safe.

Item 9: Always override hashCode when you override equals

Equal objects must have equal hashcodes
Unequal objects could have equal hashcodes
Missing hashCode implementation breaks functionality of hash-based collections
The worst possible legal hash function return 42
Bad hashcode could degrade performance in hash-based collections
Hashcode could be cached for immutable classes
Do not try to develop your own state-of-the-art hash function unless you are a mathematician

Item 10: Always override toString

Actually, not required
Easier to inspect objects
Include all needed info to toString
Document what exactly toString returns and in which format
Provide programmatic access to all of the information contained in the value returned by toString

Item 11: Override clone judiciously

Cloneable is broken
If you override the clone method in a nonfinal class, you should return an object obtained by invoking super.clone
Class that implements Cloneable is expected to provide a properly functioning public clone method
clone should not corrupt original object
Pay attention to deep and shallow copy
Provide copy constructor or copy factory instead of implementing clone

Item 12: Consider implementing Comparable

Implementing Comparable indicates that objects have natural ordering
Comparable allow to use your class in many generic algorithms: search, sorting, etc.
compareTo should be consistent with equals
Implement compareTo that returns -1, 0 and 1 and do not cause integer overflow
For non-natural ordering or inability to implement Comparable use Comparator

Item 13: Minimize the accessibility of classes and members

Make each class or member as inaccessible as possible
If a package-private top-level class is used by only one class, consider making the top-level class a private nested class of the sole class that uses it
If a method overrides a superclass method, it is not permitted to have a lower access level in the subclass than it does in the superclass
Instance fields should never be public
Classes with public mutable fields are not thread-safe
public static final arrays are mutable

Item 14: In public classes, use accessor methods, not public fields

public fields are acceptable if class is not public
if a class is accessible outside its package, provide accessor methods

Item 15: Minimize mutability

Immutable classes are easier to design, implement and use. They are less error-prone and more secure
To make a class immutable follow the rules
- Don’t provide any mutators
- Ensure that the class can’t be extended
- Make all fields final
- Make all fields private
- Ensure exclusive access to any mutable components. Return defensive copies
Immutable objects are thread-safe
Immutable objects are shared freely
Not only can you share immutable objects, but you can share their internals
The only real disadvantage of immutable classes is that they require a separate object for each distinct value
Classes should be immutable unless there’s a very good reason to make them mutable.
If a class cannot be made immutable, limit its mutability as much as possible.
Make every field final unless there is a compelling reason to make it nonfinal

Item 16: Favor composition over inheritance

Unlike method invocation, inheritance violates encapsulation
Subclasses depend on their superclasses, which could be changed and as result broken functionality in subclasses
Use composition and forwarding instead of inheritance, especially if an appropriate interface to implement a wrapper class exists.
Use inheritance when class is designed for inheritance

Item 17: Design and document for inheritance or else prohibit it

Class must document its self-use of overridable methods
Good API documentation for inheritance should describe what a given method does and how it does it.
The only way to test a class designed for inheritance is to write subclasses
Constructors must not invoke overridable methods
If supperclass implements Cloneable or Serializable neither clone nor readObject may invoke an overridable method, directly or indirectly
Prohibit subclassing in classes that are not designed and documented to be safely subclassed
Prohibit subclassing by making class final

Item 18: Prefer interfaces to abstract classes

Existing classes can be easily retrofitted to implement a new interface
Interfaces are ideal for defining mixins
Interfaces allow the construction of nonhierarchical type frameworks
Interfaces enable safe, powerful functionality enhancements via wrapper classes
Abstract classes are useful for sceletal implementation
You could safely add a method to abstract class with default implementation (the same applies to interfaces since Java 8 release, with help of default methods)
Once an interface is released and widely implemented, it is almost impossible to change

Item 19: Use interfaces only to define types

Do not use interface for defining constants
If in a future release the class is modified so that it no longer needs to use the constants, it still must implement the interface to ensure binary compatibility
If a nonfinal class implements a constant interface, all of its subclasses will have their namespaces polluted by the constants in the interface
Add constant to class if they are strongly tied to it
Make constants as enum or noninstantiable utility classes

Item 20: Prefer class hierarchies to tagged classes

Tagged class use internal state to indicate its type
Tagged classes are verbose, error-prone, and memory inefficient
Hierarchy classes provide more compile time checks

Item 21: Use function objects to represent strategies

Function objects are simulate functions in OOP
Function objects should be stateless
Primary use of function objects is to implement the Strategy pattern

Item 22: Favor static member classes over nonstatic

A nested class should exist only to serve its enclosing class
There are four kinds of nested classes
- Static member classes
- Nonstatic member classes
- Anonymous classes
- Local classes
Static member classes could exist without enclosing instance
If you declare a member class that does not require access to an enclosing instance, always put the static modifier in its declaration
Anonymous classes could be instantiated only at the point they are declared
Anonymous classes have enclosing instances if they are defined ina nonstatic context
Local classes can be declared anywhere a local variable can be declared and have the same scoping rules

Item 23: Don’t use raw types in new code

Generic types provide compile-time checking for incompatible types
Not needed manual cast to type when you retrieve element from collections
Raw types exists only for backward compatibility
You lose type safety if you use a raw type
Raw types could be used with instanceof operator

Item 24: Eliminate unchecked warnings

Eliminate every unchecked warning that you can, that means your code is typesafe
Use @SuppressWarnings("unchecked") only if you can prove the code is typesafe
Always use the @SuppressWarnings annotation on the smallest scope possible
Every time you use an @SuppressWarnings annotation, add a comment saying why it’s safe to do so
Every unchecked warning represents the potential for a ClassCastException at runtime. Do not ignore them blindly

Item 25: Prefer lists to arrays

Arrays are covariant (if Sub is subtype of Super, then Sub[] is a subtype of Super[])
Generics are invariant (List<Sub> is not a subtype of List<Super>)
Arrays are reified (enforce their element types at runtime)
Generics are non-reified and implemented by erasure (enforce types at a compile time, but erased at a runtime)
Generic array creation errors at compile time (List<E>[])
Array of non-reified types can not be created

Item 26: Favor generic types

Object type in collections are good candidate to replace with generic types
new E[] cause compile time error, use (E[]) new Object[] instead

Item 27: Favor generic methods

Generic type parameter list, which declares the type parameter, goes between the method’s modifiers and its return type (public static <T> void method())
Generic methods could infere type of arguments

Item 28: Use bounded wildcards to increase API flexibility

Generics are invariant (List<Integer> is not a subtype of List<Number>)
For maximum flexibility, use wildcard types on input parameters that represent producers or consumers
PECS: Producer - Extends , Consumer - Super
Producer: add(List<? extends Number>)
Consumer: get(List<? super Number>)
Comparable and Comparator are consumers
Do not use wildcard types as return types, it would force use wildcards in the client code
Use explicit types if compiler can’t infere them Union<Number>.union()
if a type parameter appears only once in a method declaration, replace it with a wildcard

Item 29: Consider typesafe heterogeneous containers

Single-element containers could be parametrized (ThreadLocal, AtomicReference)
String.class is of type Class<String>
Typesafe heterogeneous container pattern

public class Favorites {
  public <T> void putFavorite(Class<T> type, T instance);
  public <T> T getFavorite(Class<T> type);
}

String s = f.getFavorite(String.class) is typesafe
You can use Class objects as keys for typesafe heterogeneous containers.

Item 30: Use enums instead of int constants

int enum pattern just a class with int constants
Compiler won’t complain if you pass one int constant, where another expected
If int constant number is changed, clients should be recompiled
There is no easy way to translate int enum constants into printable strings
There is no reliable way to obtain size or iterate over all the int enum constants in a group
String enum pattern is even worse
String comparisons is expensive
Error is string constant lead to runtime error
Use enums!
Each enum internally is public static final int field
Enums provide compile-time type safety
Enum types with identically named constants coexist peacefully because each type has its own namespace
You can add or reorder constants in an enum type without recompiling clients
Translate enums into printable strings by calling their toString method
Enum types let you add arbitrary methods and fields and implement arbitrary interfaces
If an enum is generally useful, it should be a top-level class; if its use is tied to a specific top-level class, it should be a member class of that top-level class
To avoid switch on enum constant use constant-specific method implementations. Add abstract method to enum type, and override that method for each constant
Enums have auto-generated methods valueOf(String), values()
Switches on enums are good if you are client user of that enum

Item 31: Use instance fields instead of ordinals

Every enum constant associated with int value via ordinal() method
Reordering, adding or deleting enum constant cause problems if you depend on ordinal()
Use instance fields for enum (APPLE(1) instead of APPLE.ordinal())

Item 32: Use EnumSet instead of bit fields

Bit int enum pattern use int constants as a power of two (1,2,4,8,…) This lets you to perform union and intersection with bitwise operations efficiently (apply(STYLE_ITALIC | STYLE_BOLD))
Hard to interpret bit int constants
Hard to iterate over bit int constants
Just because an enumerated type will be used in sets, there is no reason to represent it with bit fields
Use EnumSet instead (apply(EnumSet.of(Style.ITALIC, Style.BOLD)))

Item 33: Use EnumMap instead of ordinal indexing

ordinal() for enums cause a lot of problems in array indexing
ordinal indexing is not typesafe, may cause wrong associations or IndexOutOfBoundsException
Use EnumMap.get(APPLE) instead of array[APPLE.ordinal()]
If the relationship that you are representing is multidimensional, use EnumMap<..., EnumMap<...>>

Item 34: Emulate extensible enums with interfaces

There is no much useful use cases to extend enum functionality
enum could inplement interfaces, therefore allow extensibility
While you cannot write an extensible enum type, you can emulate it by writing an interface to go with a basic enum type that implements the interface

Item 35: Prefer annotations to naming patterns

Prior to release 1.5, it was common to use naming patterns to indicate that some program elements demanded special treatment by a tool or framework (name test methods beginning with test for JUnit)
No warning about typos, no control over program elements, ugly and fragile approach
Annotations solve naming patterns problems
Define annotation Test public @interface Test
@Retention(RetentionPolicy.RUNTIME) meta-annotation indicates that Test annotations should be retained at runtime
@Target(ElementType.METHOD) meta-annotation indicates that the Test annotation is legal only on method declarations
Process marker annotations Method.isAnnotationPresent(Test.class)
With the exception of toolsmiths, most programmers will have no need to define annotation types
Consider using any annotations provided by your IDE or static analysis tools

Item 36: Consistently use the Override annotation

@Override can only be used on method declarations
@Override indicates that the annotated method declaration overrides a declaration in a supertype
@Override helps to catch tricky bugs (overloaded equals, hashcode)
Use the @Override annotation on every method declaration that you believe to override a superclass declaration

Item 37: Use marker interfaces to define types

A marker interface is an interface that contains no method declarations (Serializable, Cloneable)
Marker interfaces are not marker annotations
Marker interfaces define a type that is implemented by instances of the marked class; marker annotations do not
Marker interfaces can be targeted more precisely by extending that interface
Set is restricted marker interface. It extends Collection but does not add new methods. It only refines contract for some methods to be more targeted.
The chief advantage of marker annotations over marker interfaces is that it is possible to add more information to an annotation type after it is already in use, by adding one or more annotation type elements with defaults (Java8 default methods)
If you find yourself writing a marker annotation type whose target is ElementType.TYPE, take the time to figure out whether it really should be an annotation type, or whether a marker interface would be more appropriate.

Item 38: Check parameters for validity

Detect errors and wrong values as soon as possible
For public methods, use the Javadoc @throws tag to document the exception that will be thrown if a restriction on parameter values is violated
nonpublic methods should generally check their parameters using assertions
Unlike normal validity checks, they have no effect and essentially no cost unless you enable them, which you do by passing the -ea (or -enableassertions) flag to the java interpreter
Do not use validity check if it is impractical or performed implicitly in the process of doing the computation

Item 39: Make defensive copies when needed

You must program defensively, with the assumption that clients of your class will do their best to destroy its invariants
If you return mutable reference from class, then class is also mutable
To make immutable class, make a defensive copy of each mutable parameter
Defensive copies are made before checking the validity of the parameters and the validity check is performed on the copies rather than on the originals (protect against changes from another thread, TOCTOU time-of-check/time-of-use attack)
Do not use the clone method to make a defensive copy of a parameter whose type is subclassable by untrusted parties
Defensive copying of parameters is not just for immutable classes, think twice before returning a reference
Arrays are always mutable
Defensive copying can have a performance penalty associated with it and isn’t always justified
If the cost of the defensive copy would be prohibitive and the class trusts its clients not to modify the components inappropriately, then the defensive copy may be replaced by documentation outlining the client’s responsibility not to modify the affected components

Item 40: Design method signatures carefully

Choose method names carefully
Follow the code conventions
Too many methods make a class difficult to learn, use, document, test, and maintain
Avoid long parameter lists (four parameters or fewer)
Use builder pattern, helper classes or helper methods to avoid long parameter lists
For parameter types, favor interfaces over classes
Prefer two-element enum types to boolean parameters

Item 41: Use overloading judiciously

The choice of which overloaded method to invoke is made at compile time
Selection among overloaded methods is static, while selection among overridden methods is dynamic
A safe, conservative policy is never to export two overloadings with the same number of parameters
The rules that determine which overloading is selected are extremely complex. They take up thirty-three pages in the language specification [JLS, 15.12.1-3]

Item 42: Use varargs judiciously

Use call(int...) when you need zero or more arguments
Use call(int, int...) when you need one or more arguments
Don’t use varargs for every method that has a final array parameter; use varargs only when a call really operates on a variable-length sequence of values
Every invocation of a varargs method causes an array allocation and initialization
Use overloaded methods with 2, 3, 4 params to cover most use-cases, otherwise use varargs

Item 43: Return empty arrays or collections, not nulls

Do not return nulls!
Return empty collection (Collections.emptyList()), or zero-length array (new int[0]) instead of nulls
Zero-length arrays and empty collections are not performance problems, because they are immutable and only one instance could be used

Item 44: Write doc comments for all exposed API elements

Document API with the javadoc utility
To document your API properly, you must precede every exported class, interface, constructor, method, and field declaration with a doc comment
If a class is serializable, you should also document its serialized form
The doc comment for a method should describe succinctly the contract between the method and its client
With the exception of methods in classes designed for inheritance, the contract should say what the method does rather than how it does its job.
Methods should document pre- and postconditions, side effects, thread safety, exceptions
The first “sentence” of each doc comment should be the summary description
When documenting a generic type or method, be sure to document all type parameters
When documenting an enum type, be sure to document the constants
When documenting an annotation type, be sure to document any members

Item 45: Minimize the scope of local variables

The most powerful technique for minimizing the scope of a local variable is to declare it where it is first used
Nearly every local variable declaration should contain an initializer (try-catch block is an exception)
for loop allows to declare loop variable, prefer it over while
Keep methods small and focused

Item 46: Prefer for-each loops to traditional for loops

foreach saves from subtle bugs, copy-paste errors, improves readability and maintainability
foreach introduces no performance penalty
Implement Iterable interface to use custom class in foreach loop
foreach is not used in
- filtering (no access to iterator to call remove)
- transforming (no access to index element to apply change)
- parallel iteration (needed few iterators and control locks)

Item 47: Know and use the libraries

By using a standard library, you take advantage of the knowledge of the experts who wrote it and the experience of those who used it before you
If a flaw were to be discovered, it would be fixed in the next release
With using libraries you don’t have to waste your time writing ad hoc solutions to problems that are only marginally related to your work
Performance of standard libraries tends to improve over time, with no effort on your part
Libraries tend to gain new functionality over time

Item 48: Avoid float and double if exact answers are required

The float and double types are not suited for monetary calculations because it is impossible to represent 0.1 (or any other negative power of ten) as a float or double exactly
Use BigDecimal, int, or long for monetary calculations
BigDecimal has full control over rounding
If performance is crucial, you don’t mind keeping track of the decimal point yourself, and the quantities aren’t too big, use int or long

Item 49: Prefer primitive types to boxed primitives

Primitives have only their values, whereas boxed primitives have identities distinct from their values.
Boxed primitive may have null value
Primitives more time and space-efficient than boxed primitives
Applying the == operator to boxed primitives is almost always wrong
When you mix primitives and boxed primitives in a single operation, the boxed primitive is auto-unboxed,
Use boxed primitives as type parameters in parameterized types
Use boxed primitives when making reflective method invocations

Item 50: Avoid strings where other types are more appropriate

Strings are poor substitutes for other value types (5 is better than "5")
Strings are poor substitutes for enum types
Strings are poor substitutes for aggregate types; to access individual fields you must parse string

Item 51: Beware the performance of string concatenation

Using the string concatenation operator repeatedly to concatenate n strings requires O(n^2) time
To achieve acceptable performance, use a StringBuilder instead

Item 52: Refer to objects by their interfaces

If appropriate interface types exist, then parameters, return values, variables, and fields should all be declared using interface types
If you depend on any special properties of an implementation, document these requirements where you declare the variable

Item 53: Prefer interfaces to reflection

Reflection provides programmatic access to the class’s member names, field types, method signatures, etc.
Using reflection have disadvantages
- No compile-time type checking
- Code verbosity
- Performance suffers
As a rule, objects should not be accessed reflectively in normal applications at runtime
Create instances reflectively and access them normally via their interface or superclass
Using reflection cause compiler warnings

Item 54: Use native methods judiciously

The Java Native Interface (JNI) allows Java applications to call native methods, which are special methods written in native programming languages such as C or C++
JNI has three main uses
- Access to platform-specific facilities such as registries and file locks
- Access to libraries of legacy code, which could in turn provide access to legacy data
- Used to write performance-critical parts of applications
Do not use native methods for improved performance
Applications using native methods have disadvantages
- programs are not immune to memory corruption errors
- less portable
- difficult to debug

Item 55: Optimize judiciously

Premature optimization is the root of all evil
Strive to write good programs rather than fast ones
Strive to avoid design decisions that limit performance
Consider the performance consequences of your API design decisions
Measure performance before and after each attempted optimization

Item 56: Adhere to generally accepted naming conventions

Typographical
- Package: com.google.inject, org.joda.time.format
- Class/Interface: Timer, FutureTask, LinkedHashMap, HttpServlet
- Method/Field: remove, ensureCapacity, getSrc
- Constants: MIN_VALUE, NEGATIVE_INFINITY
- Local variable: i, href, houseNumber
- Type parameter: T, E, K, V, T1, T2
Grammatical
- Class - noun: Timer, Task
- Interface - noun, adjective ends with able: Comparator, Comparable
- Annotation - noun, verb, preposition, adjective: @Test, @Autowired, @ImplementedBy, @ThreadSafe
- Method - verb, rarely noun: drawImage, getDimension, isInterrupted, size
- Field - noun: height, capacity
Conventions should not be followed blinfly if long-held conventional usage dictates otherwise

Item 57: Use exceptions only for exceptional conditions

Exceptions slower than normal checks
Placing code inside a try-catch block inhibits certain optimizations that modern JVM implementations might otherwise perform
A well-designed API must not force its clients to use exceptions for ordinary control flow

Item 58: Use checked exceptions for recoverable conditions and runtime exceptions for programming errors

Java provides three kinds of throwables: checked exceptions, runtime exceptions, and errors
Checked exceptions force the caller to handle them
Use checked exceptions for conditions from which the caller can reasonably be expected to recover
Unchecked exceptions indicate programming error and not needed to be handled almost always
Errors indicated JVM problems and not needed to be handled at all

Item 59: Avoid unnecessary use of checked exceptions

Checked exceptions force the caller to handle them in try-catch block, or propagate outward
If catching exception provide no benefit (recovery, logging) use unchecked exception
One technique for turning a checked exception into an unchecked exception is to break the method that throws the exception into two methods, additional method returns a boolean that indicates whether the exception would be thrown

Item 60: Favor the use of standard exceptions

IllegalArgumentException caller passes in an argument whose value is inappropriate (e.g. negative value for square root)
IllegalStateException invocation is illegal because of the state of the receiving object (e.g. partially initialized object)
NullPointerException and IndexOutOfBoundsException are more specific versions of IllegalArgumentException
ConcurrentModificationException object that was designed for use by a single thread or with external synchronization detects that it is being (or has been) concurrently modified.
UnsupportedOperationException used by implementations that fail to implement one or more optional operations defined by an interface

Item 61: Throw exceptions appropriate to the abstraction

Exception translation. Higher layers should catch lower-level exceptions and, in their place, throw exceptions that can be explained in terms of the higher-level abstraction

try {
  // Use lower-level abstraction to do our bidding
} catch(LowerLevelException e) {
  throw new HigherLevelException();
}

Exception chaining. Higher-level exception could contain reference to lower-level exception (e.g for debugging)
While exception translation is superior to mindless propagation of exceptions from lower layers, it should not be overused

Item 62: Document all exceptions thrown by each method

Always document checked exceptions with javadoc @throws tag
Do not document multiple excetions by their common superclass (@throws Exception is bad)
Document expected unchecked exceptions with javadoc @throws tag
Do not include unchecked exceptions in method throws declaration
If an exception is thrown by many methods in a class for the same reason, it is acceptable to document the exception in the class’s documentation comment

Item 63: Include failure-capture information in detail messages

To capture the failure, the detail message of an exception should contain the values of all parameters and fields that “contributed to the exception”
To avoid verbosity, include only useful information to exception message
Exception detail message for programmers, not for users

Item 64: Strive for failure atomicity

Failure atomicity (failed method invocation should leave the object in the state that it was in prior to the invocation)
If an object is immutable, failure atomicity is free
If ab object is mutable
- Check parameters for validity before performing the operation
- Perform partial computations and then check for validity
- Write recovery code to return object to its original state after exception
- Make temporary code, apply changes and then replace original object if no exceptions are thrown
Failure atomicity is not always desirable (implementation complexity, performance)
If method is not failure atomic, reflect that in documentation

Item 65: Don’t ignore exceptions

Don’t ignore exceptions!
An empty catch block defeats the purpose of exceptions
At the very least, the catch block should contain a comment explaining why it is appropriate to ignore the exception

Item 66: Synchronize access to shared mutable data

Synchronization prevent a thread from observing an object in an inconsistent state
Synchronization ensures that each thread entering a synchronized method or block sees the effects of all previous modifications that were guarded by the same lock
Reading or writing a variable is atomic unless the variable is of type long or double
Do not use Thread.stop
A recommended way to stop one thread from another is to have the first thread poll a boolean field that is initially false but can be set to true by the second thread to indicate that the first thread is to stop itself
Liveness failure - the program fails to make progress
Synchronization has no effect unless both read and write operations are synchronized
Increment operator (++) is not atomic
Safety failure - the program computes the wrong results
Strive to assign mutable data to a single thread

Item 67: Avoid excessive synchronization

To avoid liveness and safety failures, never give control to the client within a synchronized method or block
Alien methods may cause data corruption, deadlocks
Java locks are reentrant
CopyOnWriteArrayList is a variant of ArrayList in which all write operations are implemented by making a fresh copy of the entire underlying array
Do as little work as possible inside synchronized regions
When in doubt, do not synchronize your class, but document that it is not thread-safe

Item 68: Prefer executors and tasks to threads

Executor framework separates unit of work (task) and mechanism (thread creation)
Thread is no longer a key abstraction, use Runnable or Callable
Executors.newCachedThreadPool good choice for lightly-loaded server, if no threads available for submitted task, new one will be created
Executors.newFixedThreadPool(n) good choice for heavily-loaded server, Only n threads will be created

Item 69: Prefer concurrency utilities to wait and notify

Prefer higher-level concurrency utilities (Executor Framework, concurrent collections and synchronizers) to wait and notify
ConcurrentHashMap is optimized for retrieval operations
Use ConcurrentHashMap in preference to Collections.synchronizedMap or Hashtable
BlockingQueue used for producer-consumer queues
CountdownLatch is a single-use barrier that allow one or more threads to wait for one or more other threads to do something
For interval timing, always use System.nanoTime in preference to System.currentTimeMillis. System.nanoTime is both more accurate and more precise, and it is not affected by adjustments to the system’s real-time clock.
CyclicBarrier could be used if you need multiple CountDownLatch objects
Always use the wait loop idiom to invoke the wait method; never invoke it outside of a loop

Item 70: Document thread safety

The presence of the synchronized modifier in a method declaration is an implementation detail, not a part of its exported API
To enable safe concurrent use, a class must clearly document what level of thread safety it supports
- immutable - Instances of this class appear constant. No external synchronization is necessary
- unconditionally thread-safe - Instances of this class are mutable, but the class has sufficient internal synchronization that its instances can be used concurrently without the need for any external synchronization
- conditionally thread-safe - Like unconditionally thread-safe, except that some methods require external synchronization for safe concurrent use. Examples include the collections returned by the Collections.synchronized wrappers, whose iterators require external synchronization
- not thread-safe - Instances of this class are mutable. To use them concurrently, clients must surround each method invocation (or invocation sequence) with external synchronization of the clients’ choosing
- thread-hostile - This class is not safe for concurrent use even if all method invocations are surrounded by external synchronization. Thread hostility usually results from modifying static data without synchronization. No one writes a thread-hostile class on purpose; such classes result from the failure to consider concurrency
To prevent DOS attack, you can use a private lock object instead of using synchronized methods

Item 71: Use lazy initialization judiciously

Best advice for lazy initialization is “don’t do it unless you need to”
If you use lazy initialization to break an initialization circularity, use a synchronized accessor
If you need to use lazy initialization for performance on a static field, use the lazy initialization holder class idiom
If you need to use lazy initialization for performance on an instance field, use the double-check idiom

Item 72: Don’t depend on the thread scheduler

Any program that relies on the thread scheduler for correctness or performance is likely to be nonportable.
The best way to write a robust, responsive, portable program is to ensure that the average number of runnable threads is not significantly greater than the number of processors
Do not rely on Thread.yield because it has no testable semantics
Use Thread.sleep(1) instead of Thread.yield() for concurrency testing
Thread priorities are among the least portable features of the Java platform

Item 73: Avoid thread groups

Thread groups do not provide any security functionality
Thread API is weak
Prior to release 1.5, there was one small piece of functionality that was available only with the ThreadGroup API the ThreadGroup.uncaughtException method was the only way to gain control when a thread threw an uncaught exception. As of release 1.5, however, the same functionality is available with Thread.setUncaughtExceptionHandler method.

Item 74: Implement Serializable judiciously

Implementing Serializable decreases the flexibility to change a class’s implementation once it has been released
- private and package-private fields become part of its exported API
- incompatible changes after deserialization lead to failure
Define serialVersionUID to avoid InvalidClassException
Because there is no explicit constructor associated with deserialization, it is easy to forget that you must ensure that it guarantees all of the invariants established by the constructors
Releasing new version of serialized class greatly improves number of test-cases need to be verified
Classes designed for inheritance should rarely implement Serializable
You should consider providing a parameterless constructor on nonserializable classes designed for inheritance
Use thread-safe state machine pattern (atomic-reference to enum) to implement a serializable superclass
Inner classes should not implement Serializable

Item 75: Consider using a custom serialized form

Do not accept the default serialized form without first considering whether it is appropriate
The default serialized form is likely to be appropriate if an object’s physical representation is identical to its logical content
Even if you decide that the default serialized form is appropriate, you often must provide a readObject method to ensure invariants and security
Before deciding to make a field nontransient, convince yourself that its value is part of the logical state of the object
Declare an explicit serial version UID in every serializable class you write

Item 76: Write readObject methods defensively

For classes with object reference fields that must remain private, defensively copy each object in such a field. Mutable components of immutable classes fall into this category.
Check any invariants and throw an InvalidObjectException if a check fails. The checks should follow any defensive copying.
If an entire object graph must be validated after it is deserialized, use the ObjectInputValidation interface
Do not invoke any overridable methods in the class, directly or indirectly.

Item 77: For instance control, prefer enum types to readResolve

To satisfy singleton property for serializable object, implement readResolve
If you depend on readResolve for instance control, all instance fields with object reference types must be declared transient
readResolve is not obsolete. It is needed for writing a serializable instance-controlled class whose instances are not known at compile time
Use enum types to enforce instance control invariants wherever possible

Item 78: Consider serialization proxies instead of serialized instances

Serialization proxy is a private static inner class implements Serializable and reflects serializable data for original object
Add writeReplace method to proxy class. Serialization system emits a proxy instance instead of an instance of the enclosing class.
Add readObject method to proxy class. Attacker wouldn’t be able to violate class invariants.
Add readResolve method to proxy class that returns logically equialent instance of the enclosing class.
The serialization proxy pattern has some limitations:
- It is not compatible with classes that are extendable by their clients
- It is not compatible with some classes whose object graphs contain circularities
- Serialization is slower than standard approach

Programming Digest 5

2014-06-06T13:31:00+00:00

Programming Sucks.

1. Programming Sucks

…many of the programs you depend on are written by dicks and idiots

Funny ranting about crazy programming life.

Every programmer occasionally, when nobody’s home, turns off the lights, pours a glass of scotch, puts on some light German electronica, and opens up a file on their computer…This file is Good Code.

2. Functional Programming For The Rest Of Us

There are two types of people in functional programming world: those who use monads and those who don’t know they use monads.

This is a good overview that helps to convince your colleague to use fp without monads, applicative functors and other theoretical shit.

3. Game Programming Patterns

A set of patterns and techniques applicable to game development. Probably some of them you’ve already used. Unfortunately, examples in C++.

4. A Hacker’s Guide to Git

Pour some light on git internals. Seems like almost everything in git is just a hashcode.

5. Is TDD Dead?

Hangouts with Martin Fowler, Kent Beck and David Heinemeier Hansson about TDD. Do not test everything.

6. Character Encodings

A history and caveats of encodings. Windows sucks.

7. Principles Of High Performance Programs

Generic useful rules you may consider when building high-performance applications. Memory latency, context switch, granularity and other things explained.

8. Software Talks for the Ages

Meta one. Good set of programming talks. 50 in 50 is a pure gold.

9. How the CPU works

This is quite interesting.

10. 2 + 2 = 5

Stackoverflow discussion with examples how to write a program that makes 2+2=5.

That. Is. Evil.

Clojure Pitfalls: Part 1

2014-05-22T01:48:00+00:00

Clojure is a great language, and as every language, it has some unobvious and strange things, which seem confusing.

Some of such things are gathered here. Most of them encountered by stupidity, docs misunderstanding, clojure specifics and procrastination.

1. Anybody home?

(contains? [10 20 30] 20) => false

Surprised?

The issue here is contains? function checks if key (not a value!) is present in collection. For vector - key is index, and obviously there is no 20th element.

If you need to check if value is present in collection, use some predicate:

(some #(= % 20) [10 20 30]) => true

Keep in mind this function works in linear time. If you need to call it often, build a set and use it as a function.

(def numbers (set [10 20 30]))
(numbers 20) => 20 ;; treated as true
(numbers 1) => nil ;; treated as false

Note: if set contains nil or false, it won’t work. More discussion here.

2. Breaking the Law

You need stack. In clojure you can treat list like a stack.

;; take stack head
(peek '(1 2 3)) => 1

;; pop stack
(pop '(1 2 3)) => (2 3)

;; push item into stack
(cons 0 '(1 2 3)) => (0 1 2 3)

Everything is fine. But if you call pop after cons…

ClassCastException clojure.lang.Cons cannot be cast to clojure.lang.IPersistentStack

The problem is cons is type destructive operation.

(type '(1 2 3))            => clojure.lang.PersistentList
(type (cons 0 '(1 2 3)))   => clojure.lang.Cons

And clearly, there is no pop defined for Cons type.

To preserve collection type, use conj function.

(type '(1 2 3))            => clojure.lang.PersistentList
(type (conj '(1 2 3) 0))   => clojure.lang.PersistentList

3. Apples or Oranges? Part I

What’s the difference between (last [1 2 3]) and (last '(1 2 3))?

Functionally there is no difference, both snippets return 3

Moreover, from performance view there is no difference as well. Both snippets work in linear time. Linear. O(n). Even for vector.

What’s the difference between (count [1 2 3]) and (count '(1 2 3))?

Much better here, count works in constant time for vector and linear for list. So, if you really need fast last for vector, use count.

;; 10M size vector
(def v (vec (range 1e7)))

(time (last v)) => 9999999
;; "Elapsed time: 1704.172859 msecs"

(time (get v (dec (count v)))) => 9999999
;; "Elapsed time: 0.087302 msecs"

Huge difference.

Do not use vector specific functions to list. Also, consider using mapv instead of map when processing vectors.

4. Apples or Oranges? Part II

Everybody knows reduce and apply functions. Sometimes using choosing between them does not make any difference.

;; 100K numbers
(def N (repeatedly 1e6 #(rand-int 100)))

(time (reduce + N)) => 49503683
;; "Elapsed time: 311.496332 msecs"

(time (apply + N)) => 49503683
;; "Elapsed time: 339.624222 msecs"

Sometimes it does.

;; 10K strings
(def S (repeatedly 1e5 (fn [] "1")))

(time (count(apply str S))) => 100000
;; "Elapsed time: 35.027218 msecs"

(time (count(reduce str S))) => 100000
;; "Elapsed time: 24249.845948 msecs"

5. Refactor

Conditionals are horses of programming language.

(def status 500)
(cond (= status 403)    "login.html"
      (= status 404)    "not_found.html"
      (= status 500)    "server_fault.html"
      (= status 200)    "index.html"
      :else             "something_wrong.html")

Pretty straightforward, but a piece (= status %) look ugly. Let’s refactor it a bit and avoid repetition with condp

(def status 500)
(condp = status
       403    "login.html"
       404    "not_found.html"
       500    "server_fault.html"
       200    "index.html"
       :else  "something_wrong.html")

Much better. Except we broke previous functionality. If pass invalid status (let’s say 201) we get an exception:

IllegalArgumentException No matching clause: 201

And that’s right: sugared keyword :else in cond behaves like regular clause in condp To make working solution, just remove :else.

(condp = status
       403    "login.html"
       404    "not_found.html"
       500    "server_fault.html"
       200    "index.html"
       "something_wrong.html")

6. Don’t trust your eyes

Get this portable, pure, thread-safe, well-documented, so beatiful and very awesome function:

(defn add [a b]
  "Performs addition operation.
   More about addition here: http://en.wikipedia.org/wiki/Addition"
  (+ a b))

What’s wrong here? Is it really well-documented?

If you have sources, then yes, you see all documentation. But if you are library user you have no idea what function add might do.

(doc add)
; -------------------------
; user/add
; ([a b])
;   nil

Misplaced docstring. Transpose it with arglist.

(defn add 
  "Performs addition operation.
  More about addition here: http://en.wikipedia.org/wiki/Addition"
  [a b]
  (+ a b))

(doc add)
; -------------------------
; user/add
; ([a b])
;   Performs addition operation.
;   More about addition here: http://en.wikipedia.org/wiki/Addition

Note: clojure lint tool eastwood is able to find this mistake.

7. It depends

There is less known, but useful feature dynamic binding

(def ^:dynamic *DEBUG* false)

If your code depends on *DEBUG* binding, its behaviour can be modified without changing function definition

(defn square [a]
  (when *DEBUG* (println "call square"))
  (* a a))
  
(square 10) => 100

(binding [*DEBUG* true]
  (square 10))
;; call square
=> 100

If you have another value, that depends on dynamic binding, it is not dynamic.

(def ^:dynamic *DEBUG_PREFIX*
  (if *DEBUG* "DEBUG=TRUE" "DEBUG=FALSE"))

(binding [*DEBUG* true]
  *DEBUG_PREFIX*)
=> "DEBUG=FALSE"

To fix that make it as a no-arg function. It will obtain dynamic binding right after the call.

(defn- debug-prefix []
  (if *DEBUG* "DEBUG=TRUE" "DEBUG=FALSE"))

(binding [*DEBUG* true]
  (debug-prefix))
=> "DEBUG=TRUE"

8. MAP: Access Denied

There are three ways to access map value

(def m {:name "Quentin Tarantino"})

;; 1. By keyword
(:name m) => "Quentin Tarantino"

;; 2. By map as a function
(m :name) => "Quentin Tarantino"

;; 3. Using get
(get m :name) => "Quentin Tarantino"

What version to use is a matter of style, because they are almost always interchangeble, but keep in mind some caveats.

First approach is bad, when your keys are strings or whatever-not-a-function.

(def m {"name" "Quentin Tarantino"})

("name" m) => ClassCastException java.lang.String cannot be cast to clojure.lang.IFn

Second one is able to handle that, but instead it can throw NPE, when map is nil

(def m nil)

(m :name) => NullPointerException

Using get we avoid these pros, and access become more obvious.

I prefer first version when using keyword explicitly, otherwise access via get

9. Thread Hack

Thread macro is a convenient way to implement sequential processing.

Thread First -> inserts previous result as a first argument for the next expression.

(-> (http/get URL)
    (validate-status)
    :body
    :data
    (first))

Thread Last ->> does the same, but insert previous result as a last argument

(->> (range 100)
     (map square)
     (filter even?)
     (reduce +))

If you need to insert previous result into specific position, wrap next expression to one-arg anonymous functions.

(->> (http/get URL)
     (#(validate-status % :throw-exception))
     :body
     :data
     (#(conj % [:guard])))

10. Just Words

This one is not related to clojure, but interesting. Consider problem:

Find all words in a string

Pretty obvious, huh?

(re-seq #"\w+" "Hello, world!") =>
("Hello" "world")

We don’t want numbers and underscore appear inside word

(re-seq #"\w+" "Hello, w_orld 1984!") =>
("Hello" "w_orld" "1984")

Okay, use just letters:

(re-seq #"[a-zA-Z]+" "Hello, w_orld 1984!") =>
("Hello" "w" "orld")

No, we don’t want broken words. Use \b word boundary regex.

(re-seq #"\b[a-zA-Z]+\b" "Hello, w_orld 1984!") =>
("Hello")

Solved?

Not for ukrainian

(re-seq #"\b[a-zA-Z]+\b" "Привіт, світ 1984!") =>
nil

And for every language which contains non-ascii letters in alphabet. Use unicode letter regexp \p{L} instead.

(re-seq #"\b\p{L}+\b" "Привіт, світ 1984!") =>
("Привіт" "світ")

Clojure Euler: Problem 023

2014-05-10T18:11:00+00:00

Find the sum of all the positive integers which cannot be written as the sum of two abundant numbers.

Permalink: http://projecteuler.net/problem=23

To define abundant number let’s revisit sum-of-proper-divisors from the Clojure Euler: Problem 021

(defn sum-of-proper-divisors [n]
  (let [base (filter #(zero? (mod n %)) (range 2 (Math/sqrt n)))]
    (reduce + 1 (concat (map #(/ n %) base) base))))

Using that function, abundant? predicate just follows definition.

(defn abundant? [n]
  (> (sum-of-proper-divisors n) n))

If you look carefully at sum-of-proper-divisors it is buggy.

(sum-of-proper-divisors 25) => 1 ;; instead of 6

Square numbers are not handled correctly. To fix that we add one more list to concat.

(defn sum-of-proper-divisors [n]
  (let [divs (filter #(zero? (mod n %)) (range 2 (Math/sqrt n)))]
    (reduce + 1 (set (concat
                      (let [isq (int (Math/sqrt n))]      ;; added
                        (if (= n (* isq isq)) [isq] []))  ;; added
                      divs
                      (map #(/ n %) divs))))))

Double check

(sum-of-proper-divisors 25) => 6

Great! Next step is to find all integer numbers which can not be written as the sum of two abundant numbers.

Precalculate all abundant numbers:

(def abundant (filter abundant? (range 12 28124)))

Now we can build all numbers which can be written as two abundant numbers and find their sum.

(->> (for [i abundant j abundant
           :let [num (+ i j)]
           :when (< num 28124)]
       num)
     (distinct)
     (reduce +)
     (- (reduce + (range 1 28124))))

This solution works ~44 sec, too much.

More clever approach is to iterate from all integers upto 28124 and check if some number can be written as a sum of two abundant numbers.

The trick is simple: we building sorted set of abundant numbers

(def abundant (into (sorted-set) (filter abundant? (range 12 28124))))

For every integer number n we subtract one abundant number and check if abundant set contains result. There is a good function some that makes our life easier.

(defn abundant-sum? [n abundant]
  (some #(abundant (- n %)) abundant))

Finding the sum

(->> (range 1 28124)
     (remove #(abundant-sum? % abundant))
     (reduce +))

It works ~26 sec.

Another optimization is to limit number of abundants in abundant-sum function.

(defn abundant-sum? [n abundant]
  (some #(abundant (- n %))
         (take-while #(< % n) abundant)))

Time reduced to ~14 sec. Pretty good.

Code

P.S. Buggy version of sum-of-proper-divisor solved the previous problem correctly, but it was just a luck. Make sure you test your functions.

Numberto: Expressions

2014-03-29T00:25:00+00:00

Numberto has new features!

In previous post I wrote about simple clojure library numberto for experiments with numbers.

New version of numberto has a bunch of new features.

Expressions

Expressions package provides two functions: eval-infix to evaluate infix expression and infix->prefix to build prefix lisp-style expression from mathematical notation.

Let’s give it alias for simplicity

(def e eval-infix)

Evaluate simple math expression

(e "1+2") => 3

or more complex

(e "1+2*(3-4/2)") => 3

handle priorities

(e "2+2*2") => 6

and left/right associativity

(e "1024/2/2/2/2") => 64
(e "2^3^4") => 2417851639229258349412352N

Oh, what’s this? Long numbers? Sure, ratios and floats supported as well

(e "1/3") => 1/3
(e "1.1/0.9") => 1.2222222222222223

Unary operations

(e "(-1)^100") => 1

functions and symbols

(e "sin(e) + sqrt(pi)") => 2.183235141408425

vararg functions

(e "sum(1,2,3,sum())/max(1,2)") => 3

You can also provide bindings for unknown functions and symbols

(e "factorial(n)/20"
   {:bindings
     {"factorial" #(reduce *' (range 1 (inc %)))
      "n" 10}})
=> 181440

Worth to mention that you can easily redefine existing or define your own new unary, binary operations, functions and symbols. Just add additional properties to eval-infix

;; return current time in millis
(e "now()" {:bindings {"now" #(.getTime (java.util.Date.))}}) => some long number
;; override priorities
(e "1+2*3" {:binary-ops {"+" {:function + :priority 100}}}) => 9

infix->prefix has exactly the same functionality, but it builds prefix expression instead.

(infix->prefix "(1+2)*3-(4/avg(3,5)-sum(1))")
=>
"(- (* (+ 1 2) 3) (- (/ 4 (avg 3 5)) (sum 1)))"

It can be useful if you googled some formula but bored to translate it manually to clojure.

For example, take the Simpson’s rule

(infix->prefix "(b-a)/6*(f(a)+4*f((a+b)/2)+f(b))")
=>
"(* (/ (- b a) 6) (+ (+ (f a) (* 4 (f (/ (+ a b) 2)))) (f b)))"

Implementation

Would be good to try instaparse for such purpose, but I decided to use custom implementation using standard Shunting-yard algorithm. Just couple of hacks added to handle unaries and vararg functions. Code is awful. If you really want to dig in - run debug mode.

(binding [*DEBUG* true]
  (e "1+2"))

Limitations

No way to define non-prefix unary operation, like “10!” or “ 10 ”
Can not handle simplified math form, like “2x^2 + 3^x + 5”. Multiplication sign should be declared explicitly
Not able to simplify ops in lisp-style expression. (* (* 1 2) 3) instead of (* 1 2 3)
False positives handling. Some stupid expressions indicates about error (e "1+"), some not (e "1+()1). I still working on this, but it is not critical.

Solvers

Here is the puzzle:

You have four numbers [3, 4, 5, 6].
You have four binary operations [+, -, *, /] and parentheses ()

How to insert operations between numbers to get number 42?

Hah, that simple 3*4 + 5*6 = 42

Ok, get 42, but you forced to use one division /.

Not so obvious?

(solve-insert-ops-num [3 4 5 6] 42) =>
([42N "3+45-6"] [42N "3/4*56"] [42N "3*4+5*6"])

If you use solve-insert-ops function it gives all possible values can be obtained by inserting operations between numbers.

(solve-insert-ops [3 4 5 6]) => ;; long list

Default implementation uses 4 basic operations, no parens and no restrictions. Instead, you can override options

to use parens, specify level

(solve-insert-ops-num [3 4 5 6] 42 {:parens 1}) =>
([42N "3+45-6"] [42N "(3+45)-6"] [42N "3+(45-6)"]
 [42N "3/4*56"] [42N "(3/4)*56"] [42N "3/(4/56)"]
 [42N "3*4+5*6"] [42N "(3*4)+5*6"] [42N "3*4+(5*6)"])

limit some operations

(solve-insert-ops-num [3 4 5 6] 42 {:rules [[:max "*" 1]]}) =>
([42N "3+45-6"] [42N "3/4*56"])

:max, :min, :max-in-a-row, :min-in-a-row options are supported.

Add new operations (supported by expressions package)

(solve-insert-ops-num [3 4 5 6] 80
                      {:ops ["+" "-" "*" "/" "^"]
                       :rules [[:max "^" 1]]}) =>
([80N "3^4+5-6"])

Keep in mind, always limit time consuming operations (like ^) as it builds all possible permutations and you can wait your answer forever.

There are also couple of new interesting things, like getting digits of pi, e, sqrt(n), ratio numbers up to desired level and other. Check it out

Programming Digest 4

2013-11-25T22:50:00+00:00

How to live in the Kingdom of Nouns, why mathematicians cry, filling gaps in algorithms, portion of clojure, STM, HTTP and more.

1. Execution in the Kingdom of Nouns

Another great Steve Yegge’s post. Not a post, a tale about Javaland, a place where all verbs are owned by nouns. Accept it like an analogy between OOP and FP.

Verbs in Javaland are responsible for all the work, but as they are held in contempt by all, no Verb is ever permitted to wander about freely. If a Verb is to be seen in public at all, it must be escorted at all times by a Noun.

Of course to be escorted must be replaced by VerbEscorter.

The Verb “execute”, and its synonymous cousins “run”, “start”, “go”, “justDoIt”, “makeItSo”, and the like, can perform the work of any other Verb by replacing it with an appropriate Executioner and a call to execute(). Need to wait? Waiter.execute(). Brush your teeth? ToothBrusher(myTeeth).go(). Take out the garbage? TrashDisposalPlanExecutor.doIt(). No Verb is safe; all can be replaced by a Noun on the run.

You are not forced to use functions everywhere, but use them where it is appropriate to simplify solution and avoid enterprise hell. You know, not everything is an object. The same way not everything is a function.

2. A Mathematician’s Lament

Long paper by Paul Lockhart about problems in mathematical education.

Math class is stupid and boring. - Students.

Author claims that mathematics is an art, instead of some practical thing. Of course, it can be applied in real world and it is, but it’s not primary goal of math.

A mathematician, like a painter or poet is a maker of patterns. If his patterns are more permanent than theirs, it is because they are made wit ideas.

Very exciting reading.

3. Algorithms

Algorithms are not just devils you need to learn for CS course or to prepare for interview. You can apply them to solve real-world problems. This SO answer provides links and some explanations where specific algorithm is used in linux kernel, chromium browser and some utils.

… Red-Black trees are used for scheduling, virtual memory management, to track file descriptors and directory entries, etc.

grep, implements the Boyer-Moore algorithm

fgrep implements the Aho-Corasick string matching algorithm. …

Even Bubblesort has its application!

4. Rich Hickey’s Greatest Hits

I’m huge Rich Hickey’s fan. It’s just a list of his most interesting presentations. Definitely, must see.

5. Clojure Persistent Vector

Two parts explanation how clojure vector works.

Yeah, simple square brackets [1 2 3] in clojure have a lot complex things under the hood. Moreover, it gives understanding not only how vectors work, but how all other persistent structures can be implemeted in efficient manner.

6. Software Transactional Memory

Have you heard about STM?

It’s a good model for concurrency control, like database transactions, but on memory. Much simpler and less error-prone to use. This article explain STM basics with some implementation details.

For my attempt to implement STM refer presentation

7. A Software Developer’s Huide to HTTP

Five Parts article of looking at HTTP from developer’s perspective. Worth reading to dig into some details.

8. 100 Ideas for Computing

Great list, perhaps most ideas are already partially implemented.

This is a vision for how computing could be. I began writing ideas down a number of years ago, this is a selection of ideas old and new.

Some of them are great:

Task management between computers is an unsolved problem in desktop environments. Many users have multiple devices. Sending tasks, inputs or outputs to different computers is painful. Usually sending audio to be played through another computer, taking control of an application running on another computer, sharing files or streaming data to other machines requires special software.

It would be great to have the same in reverse order also. I get a phone call, but phone is far away. Much better to click on current machine Accept Call and to talk. Security can be a challenging.

Some problematic:

Keyboard shortcuts are configured differently in each program. Shortcut configuration should be in a standard format. This way you can load them and share them between operating systems and share configurations between applications.

This reminds me of xkcd comics:

Some ideas are not so great:

Mounting source code does not solve a problem at all, it’s just adds a bit of sugar to project structure and adds more pain to build tools.

9. Top 10 Lists Common Java Mistakes

Metalist of Top 10 items of Top 10 java pitfalls.

Hundred times discussed, but if you are java programmer, you may refresh your knowledge. List of common java pitfalls includes:

Using Singletons

The GOF religion

Return null from API methods

By the way, this list contains some other useful java things, like top 10 java people and top 15 worst Computer Software blunders. Not worst, but definitely bad.

10. Rare Are GC Talks

Different Garbage Collector algorithms briefly explained. Bookshelf used as an analogy:

Bookshelf - Memory, Books - Objects.

Ruby GC implementation discussed, but you are not forced to know ruby to understand that. By the way, article contains a lot of links to more detailed GC algorithms explanation.

Clojure Euler: Problem 022

2013-10-20T13:22:00+00:00

Using names.txt a 46K text file containing over five-thousand first names, begin by sorting it into alphabetical order. Then working out the alphabetical value for each name, multiply this value by its alphabetical position in the list to obtain a name score.

For example, when the list is sorted into alphabetical order, COLIN, which is worth 3 + 15 + 12 + 9 + 14 = 53, is the 938th name in the list. So, COLIN would obtain a score of 938 x 53 = 49714.

What is the total of all the name scores in the file?

Permalink: http://projecteuler.net/problem=22

Problem is very easy, assuming you know how to work with files.

First of all read the file, clojure has awesome simple function slurp, discussed in Clojure Euler: Problem 008.

(slurp "names.txt")

Then, you need to select all words in that file:

(re-seq #"\w+" (slurp "names.txt"))

It return a list with the names. How that simple, yeah?

Next two steps are:

sort list in ascending order, what is done by calling sort function
calculate score for each word

By the way, score consists of two parts. First of all, you must calculate natural score of the word, without its actual position in sorted list, and then you must multiply that score on its position.

First part looks like this:

(defn score [string]
  (reduce + (map #(- (int %) 64) string)))

Convert each character to its positional number in alphabet. For example the character “A” (capitalized) has ascii value of 65. If we substract 64 it becomes 1, what means “A” is a first letter in alphabet and so on.

To track positions in list we use map-indexed function. It works almost the same way as map, except it accept function of two arguments, index of item in current list, and item itself.

For example:

(map-indexed #(vec [%1 %2]) ["a" "b" "c"]) => ([0 "a"] [1 "b"] [2 "c"])

As indices start from zero, do not forget increment!

Bind all together

(reduce +
        (map-indexed #(* (inc %1) (score %2)) 
                     (sort (re-seq #"\w+" (slurp "names.txt"))))))

Solved! Code is here

P.S. Actually, this code is not elegant. We need to read from the end to the beginning of expression. And it can be solved by Threading Macro. Read the nice Fogus explanation

Our last solution, becomes transformed to:

(->> "names.txt"
	(slurp)
	(re-seq #"\w+")
	(sort)
	(map-indexed #(* (inc %1) (score %2)))
	(reduce +))

Way more readable, huh?

Clojure Euler: Problem 021

2013-10-05T15:24:00+00:00

Evaluate the sum of all the amicable numbers under 10000.

Permalink: http://projecteuler.net/problem=21

Let d(n) be defined as the sum of proper divisors of n (numbers less than n which divide evenly into n).

If d(a) = b and d(b) = a, where a != b, then a and b are an amicable pair and each of a and b are called amicable numbers.

For example, the proper divisors of 220 are 1, 2, 4, 5, 10, 11, 20, 22, 44, 55 and 110; therefore d(220) = 284. The proper divisors of 284 are 1, 2, 4, 71 and 142; so d(284) = 220.

We can do it bruteforcely, but we won’t.

Let start with the function d. We call it sum-of-proper-divisors

(defn sum-of-proper-divisors [n]
  (let [base (filter #(zero? (mod n %)) (range 2 (Math/sqrt n)))]
    (reduce + 1 (concat (map #(/ n %) base) base))))

Bad approach to find all divisors: to iterate on all numbers from 1 to n and check if it is divisor or not. Instead, we iterate from 1 to sqrt(n), find one divisor and calculate another symmetric divisor.

For example, take number 10. We iterating from 2 to (sqrt 10). Number 2 is divisor, so we divide (/ 10 2), get 5 which is also divisor. And so on.

This is predicate checks where two numbers amicable or not:

(defn amicable? [a b]
  (and (not (= a b))
       (= a (sum-of-proper-divisors b))
       (= b (sum-of-proper-divisors a))))

Obvious enough, just consider that we skip case where (= a b).

The last part is to use our functions to obtain result.

(reduce +
  (let [sums (vec (map sum-of-proper-divisors (range 1 10000)))]
    (for [i (range 1 10000)]
      (if (amicable? i (nth sums (dec i))) i 0))))

Just generate all sums, and check if another number i produce the amicable pair. Done!

Code

P.S. Take a look at the last snippet. vec is really important there.

Announcing numberto

2013-09-01T04:23:00+00:00

All you need to play with numbers!

numberto

simple clojure library with a bunch of functions operating on numbers.

I created this project for two reasons:

Solving problems on Project Euler, 4Clojure and just playground with numbers needed common functions. I just extracted them to separate util library.
Practice to develop clojure project with unit tests, continuos integration, documentation and publishing to clojars.

Get it

Current version is 0.0.2 available to download on Clojars or as lein dependency [numberto "0.0.2"]

Example usage

(number->roman 19) => "XIX"
(take 10 fibonacci) => (1 1 2 3 5 8 13 21 34 55)
(shift-right 123456789 3) => 789123456
(count-digits (! 100000)) => 456574
(factorize 234) => [2 3 3 13]

Can you read long numbers?

(number-name 16532561257523723757234781264) =>
"sixteen octillion five hundred thirty two septillion five hundred sixty one sextillion
two hundred fifty seven quintillion five hundred twenty three quadrillion seven hundred
twenty three trillion seven hundred fifty seven billion two hundred thirty four million
seven hundred eighty one thousand two hundred sixty four"

Structure

Each logical set of functions moved to separate namespace. Here is the list of namespaces:

Converters. number to digits, char to digit, roman to arabic
Math. Often used functions, such as count-digits, shuffle-num, reverse-num and some from common math abs, gcd, lcm
Seqs. Various lazy sequences: fibonacci, farey, palindromes.
Primes. Prime numbers lazy sequence, factorization and totient function
Factorial. Standard and Improved version of factorial
Generators. Generate random numbers
Printers. Pretty print for long numbers

Check out detailed intro

P.S. I still experimenting, so expect a lot of changes between versions until make something stable. Comments and advices are welcome!

Code Golf: Brainfuck Interpreter

2013-08-09T17:44:00+00:00

Brainfuck Interpreter in two tweets.

Previous article Code Golf: Game of Life raised some interest, and I decided to proceed. Today’s problem is a Brainfuck Interpreter.

Brainfuck is an esoteric programming language, famous because of its small command set. It is based on array of cells (Turing Tape) and pointer to this array. There are only 8 commands:

> move to the next cell
< move to the previous cell
+ increment value in the current cell
- decrement value in the current cell
. print char with ascii value of current cell
, read ascii value for input char to the current cell
[ start loop until value of pointer is not zero
] finish loop

That’s all. Brainfuck is Turing Complete language, that means it capable to implement any program. If you crazy, of course.

Final version took 280 characters in Clojure:

(fn[a p k c](let[h #(nth %1@%2)e #(h a p)s #(swap! %1%2 1)t
#(aset a@p(%(e)1))l #(do(s k %)(case(h c k)\]()\[()(recur %)
))](while(>(count c)@k)(do(case(h c k)\>(s p +)\<(s p -)\+
(t +)\-(t -)\.(print(char(e)))\,(aset a@p(.read *in*))\[(if
(=(e)0)(l +))\](if(>(e)0)(l -)))(s k +)))))

Exactly 2 tweets.

“Sugared” version

Translating to more readable code:

(defn parse-internal [a pt pc cs]
  (letfn [(act []
            (case (nth cs @pc)
              \> (swap! pt inc)
              \< (swap! pt dec)
              \+ (aset a @pt (inc (nth a @pt))) 
              \- (aset a @pt (dec (nth a @pt)))
              \. (print (char (nth a @pt)))
              \, (aset a @pt (.read *in*))
              \[ (if (zero? (nth a @pt)) (loop- inc))
              \] (if-not (zero? (nth a @pt)) (loop- dec))))
          (loop- [f]
            (do (swap! pc f)
                (case (nth cs @pc)
                  \[ ()
                  \] () 
                  (recur f))))]
    (while (not= (count cs) @pc)
      (do
        (act)
        (swap! pc inc)))))

So, what’s happening there?

Function arguments are parameters of our tape and brainfuck program.

a is an array represents finite tape e.g (int-array 100)
pt is an atom - pointer to the tape
pc is an atom - pointer to the command list
cs command list

Function act decides which action to perform depending on current command, loop allows us to move command pointer inside a loop, and main while-do loop executes commands until they exhausted. Simple enough.

To make our interpreter more friendly we create function parse that accepts string - program, written in brainfuck.

(defn parse [s]
  (let [a (int-array 100)  ;; Turing Tape
        p (atom 0)         ;; Pointer to Tape
        k (atom 0)         ;; Pointer to Command
        c (vec (seq s))]   ;; Vector of Commands
    (parse-internal a p k c)))

Testing

Print “Hello, world”

(parse "++++++++++[>+++++++>++++++++++>+++>+<<<<-]
        >++.>+.+++++++..+++.>++.<<+++++++++++++++.>
        .+++.------.--------.>+.>.") =>
Hello World!

Input 5 characters and reverse print them

(parse ",>,>,>,>,.<.<.<.<.") => <wait input "hello">
olleh

More complex program need nested loops, which is not supported by this version (for the sake of small size!)

History of implementation, nested loops and more available here

P.S. This version is not “fully-featured” brainfuck interpreter.

It does not work for inifinite tape
Cell datatype is fixed and not assumed to be cyclic. For example, if cell type is unsigned byte [0..255], then 255 + 1 != 0 and break interpreter
Only unsigned types supported
Nested loops are not allowed
No error handling

But, you are welcome to improve it!

Programming Digest 3

2013-07-22T20:33:00+00:00

Why your software sucks and what is the “Secret Weapon”. Also, algorithm complexities refresher, functional programming, including Scala and Clojure, concurrency and lot of humor.

1. Big Ball of Mud

What does your programming day look like?

The article describes problems in software design and explains why almost every system nowadays look like “Big Ball of Mud”.

There are seven real patterns in software design:

BIG BALL OF MUD
THROWAWAY CODE
PIECEMEAL GROWTH
KEEP IT WORKING
SHEARING LAYERS
SWEEPING IT UNDER THE RUG
RECONSTRUCTION

caps preserved due to original article

And our favorite phrase:

It works, so why fix it?

2. No Silver Bullet

Classic article by Frederick Brooks, Jr.

Object-oriented programming? Artificial Intelligence? Expert systems? Time-sharing? Graphical or Automated programming?

Nothing helps you.

The most radical possible solution for constructing software is not to construct it at all.

Demotivating, but helpful.

3. Beating the Averages

This is inspiring one.

Brooks told “There is no Silver Bullet”, but Paul Graham reveals “The Secret Weapon”.

The article is success story of startup called Viaweb. Chosen technologies were so good, that allowed Viaweb to compete with all similar startups.

Our secret weapon was similar. We wrote our software in a weird AI language, with a bizarre syntax full of parentheses.

As you probably understood, it was Lisp.

4. Know Thy Complexities!

Plain algorithm complexity cheatsheet. Contains time and space complexities for common algorithms and datastructures.

5. Adventures in Functional Programming

Did you know that recursion is just syntactic sugar over higher-order functions?

Awesome screencast by Jim Weirich about Y-Combinator in JavaScript.

6. 99 Scala Problems

I am not using Scala and prefer Clojure for rapid protyping, but find such articles extremely useful when learning new language. Just exploring the code shows that Scala is the powerful and expressive language.

P.S. I am on half way to publish “99 Clojure Problems”.

7. Locks, Actors, And STM In Pictures

Great pictures. This is nice intro to concurrency and overview of some common cocurrency problems.

8. Understanding Clojure Concurrency

Clojure is exciting language with great concurrency capabilities: default immutability, atoms, refs, agents, promises, futures and software transactional memory.

This article is an introduction to them. Check out the 2nd part

9. Wat

This is the video screencast of Gary Bernhardt’s talk. Sarcastic and great presentation of weird Ruby and JavaScript behaviour.

An object plus an object is actually…. Not A Number!

10. What is the most intellectual joke you know

We touch humorous side and this article (reddit thread) not about programming, just jokes.

Q: What does the “B” in Benoit B. Mandelbrot stand for?

A: Benoit B. Mandelbrot.

In fact, some of the jokes related to programming.

There are only two hard things in Computer Science: cache invalidation, naming things, and off-by-one errors.

Some of them a bit stupid, but still fun.

How many surrealists does it take to screw in a light bulb?

A fish.

Definitely, worth to read!

Clojure Euler: Problem 020

2013-07-21T13:39:00+00:00

Find the sum of the digits in the number 100!

Permalink: http://projecteuler.net/problem=20

First of all we need to calculate factorial. Without integer overflows, stack overflows and other caveats.

Previous article Fast Factorial has working solution that we might use.

(defn ! [n]
  (reduce *' (range 1 (inc n))))

Now, calculate the sum of digits, the same function that we used in Clojure Euler: Problem 008 and Clojure Euler: Problem 016

(defn sum-of-digits [n]
  (reduce + (map #(- (int %) 48) (seq (str n)))))

And the final line

(sum-of-digits (! 100))

Simple enough, huh?

Code

P.S. The functions sum-of-digits, count-digits and similar used often and look very ugly, so I decided to create a separate set of functions (library?) which I can refer. It should be a set of common mathematic functions, including different sequences, sums, converters, interesting numbers. You might use it too. Welcome to numberto!

Clojure Euler: Problem 019

2013-07-10T01:59:00+00:00

How many Sundays fell on the first of the month during the twentieth century (1 Jan 1901 to 31 Dec 2000)?

Permalink: http://projecteuler.net/problem=19

Project Euler also gives following information:

1 Jan 1900 was a Monday.
A poem

Thirty days has September,
April, June and November.
All the rest have thirty-one,
Saving February alone,
Which has twenty-eight, rain or shine.
And on leap years, twenty-nine.
A leap year occurs on any year evenly divisible by 4, but not on a century unless it is divisible by 400.

Implementin calendars logic is great, but java interoperability is much better!

First of all, take standard java GregorianCalendar class. We can import it almost the same way as we use require

(import [java.util GregorianCalendar])

To create an object use function new

(new GregorianCalendar)

But it is more idiomatic way to create java objects with special syntax: period after class name

(GregorianCalendar.)

To configure calendar object to specific date we need to set some its properties. Setters syntax is following (assuming calendar is an object of calendar):

Java

calendar.set(GregorianCalendar.YEAR, 2013);

Clojure

(.set calendar GregorianCalendar/YEAR 2013)

Executing lot of setters is the same boilerplate as in java:

(do
  (.set calendar GregorianCalendar/YEAR 2013)
  (.set calendar GregorianCalendar/MONTH 11)
  (.set calendar GregorianCalendar/DAY_OF_MONTH 1))

To avoid repetition of object name we can use doto macro:

(doto (GregorianCalendar.)
  (.set GregorianCalendar/YEAR 2013)
  (.set GregorianCalendar/MONTH 11)
  (.set GregorianCalendar/DAY_OF_MONTH 1))

This macro magically performs the same set of operations as above, but in more readable way.

Now we can create calendar objects for the first of the specific month of the specific year

(defn calendar-for [year month]
  (doto (GregorianCalendar.)
    (.set GregorianCalendar/YEAR year)
    (.set GregorianCalendar/MONTH month)
    (.set GregorianCalendar/DAY_OF_MONTH 1)))

Just iterate through all such objects in date range given in problem definition and count Sundays.

(reduce +
  (for [year (range 1901 (inc 2000)) month (range 1 (inc 12))]
    (let [c (calendar-for year month)]
          (if (= GregorianCalendar/SUNDAY 
                 (.get c GregorianCalendar/DAY_OF_WEEK)) 1 0))))

Problem solved!

P.S. Java interoperability is ugly but still useful.

randomorg-0.1

2013-07-06T15:33:00+00:00

randomorg-0.1 released!

If you use random numbers in your software you might be interested in better numbers distribution than Random.nextInt().

Someone solves this problem with hardware generators, few entropy sources and even books.

By the way, there is great service random.org which allows to generate random numbers via atmospheric noise.

randomorg is a small java library for random.org API.

Check project page for usage and documentation!

Fast Factorial

2013-06-07T01:08:00+00:00

Factorial function is simple enough. But there is still some fun about it. In Stirling’s Approximation article we’ve seen how to calculate good factorial approximation faster than exact value. By the way, there is an algorithm to calculate exact value of factorial faster than “by definition”.

Factorials

Most of programming languages tutorials shows the following approach to calculate factorial (transformed to clojure):

(defn factorial [n]
  (if (zero? n) 1
      (* n (factorial (dec n)))))

Simple recursive solution.

Note: Since clojure allows using ! character in function name, it is good name for factorial.

(! 10) works, (! 20) works, (! 30)…

> ArithmeticException integer overflow

Oops.

Default clojure integer type is 64 bit (Long). It does not fit well for such large input, so we rewrite function.

(defn ! [n]
  (if (zero? n) 1
      (*' n (! (dec n)))))

Just one character added, * became *'. It is automatic type promotion. If type can not hold the value without losing precision, it automatically extends. In this case, Long becames BigInteger and all works fine.

(! 30) works, (! 300) works, (! 3000) works, (! 30000)…

> StackOverflowError

Yet another overflow.

Well, stack size is limited. We don’t need to consume stackframes and will use loop/recur approach.

(defn ! [n]
  (loop [cur n acc 1]
    (if (zero? cur) acc
        (recur (dec cur) (*' cur acc)))))

It is also recursive approach, but instead of saving so far calculated value in stack, we pass it to the function as a parameter. Clojure optimize tail recursion to iterative version and does not consume stack. It succesfully calculates factorial for 30000, and for any other value. Great!

If we look at the factorial as a calculation process, not definition, it means “product of all numbers between 1 and n”. It is very concise for functional programming:

(defn ! [n]
  (reduce *' (range 1 (inc n))))

Does not consume stack, no number overflow, but works ~2 times slower than loop/recur version.

I don’t know the reason, but it seems time wasted by reduce for creating intermediate results. So we choose previous loop/recur function as baseline factorial.

Fast Factorial

Here is another way to calculate factorial.

Let’s consider example 12!.

Factorial is a product of numbers 12! = 1 * 2 * 3 * ... * 11 * 12. Perform prime factorization of each number. For example, prime factorization of 12 is 2 * 2 * 3.

After that we get a factorization of the 12!, which contains only prime numbers. To find factorial value we can just multiply them all. But we do clever trick instead.

Group every prime number and its power. E.g. if we have factorization 2 * 2 * 2 * 3 * 5 it becomes vector of pairs [[2 3] [3 1] [5 1]].

To calculate power we have simple power implementation:

(defn power [x n]
  (reduce *' (repeat n x)))

Trick is using exponentiation by squaring, which reduce exponentiation complexity from O(n) to O(log n).

(defn power [x n]
  (cond (= 0 n) 1
        (= 1 n) x
        (even? n) (power* (*' x x) (/ n 2))
        (odd? n) (*' x (power* (*' x x) (/ (dec n) 2)))))

It is not tail-recursive solution and theoretically may cause stackoverflow, but its not critical. It works well for large numbers.

Actually, we do not perform factorization. We just know that factorization of n! contains all prime numbers below or equal n and every number have some multiplicity.

The function calculates how many times prime number k occurs in factorial factorization for n

(defn- find-power [n k]
  (loop [total n sum 0]
    (let [i (int (/ total k))]
      (if (zero? i) sum
          (recur i (+ sum i))))))

Binding all together:

(defn !! [n]
  (loop [[h & t] 
         (map #(power % (find-power n %))
              (take-while #(<= % n) (primes)))
         acc 1]
    (if h (recur t (*' h acc)) acc)))

Code available here

Note: primes is a function generates lazy-sequence of prime numbers. It can be taken from clojure.contrib.lazy-seqs.

Testing

Anyway, why do you think it is faster? You performing more calculations than just multiplying numbers.

Let’s see.

(do (println "== 10 ==")
    (time (! 10))
    (time (!! 10))
    (println "== 100 ==")
    (time (! 100))
    (time (!! 100))
    (println "== 1000 ==")
    (time (! 1000))
    (time (!! 1000))
    (println "== 10000 ==")
    (time (! 10000))
    (time (!! 10000))
    (println "== 100000 ==")
    (time (! 100000))
    (time (!! 100000))
    (println "== 1000000 ==")
    (time (! 1000000))
    (time (!! 1000000)) nil)

== 10 ==
"Elapsed time: 0.051054 msecs"
"Elapsed time: 0.137587 msecs"
== 100 ==
"Elapsed time: 0.081365 msecs"
"Elapsed time: 0.376653 msecs"
== 1000 ==
"Elapsed time: 1.59252 msecs"
"Elapsed time: 3.212842 msecs"
== 10000 ==
"Elapsed time: 161.856965 msecs"
"Elapsed time: 76.452529 msecs"
== 100000 ==
"Elapsed time: 19403.375319 msecs"
"Elapsed time: 6372.35266 msecs"
== 1000000 ==
"Elapsed time: 2863893.471718 msecs"
"Elapsed time: 1079632.255919 msecs"
nil

For small factorials (< 1000) improved version works ~2 times slower. Around the thousand it has the same performance as standard version. And, finally, some win (up to 3 times) for larger numbers.

Theoretically, you can implement generalized factorial with these two algorithms and switch between them, depending on input. But who really need it?

Question: What the complexity of this algorithm?

P.S. Enterprise lovers would say factorial complexity is O(1). Just precompute all values and save them to database.

Clojure Euler: Problem 018

2013-05-17T00:26:00+00:00

By starting at the top of the triangle below and moving to adjacent numbers on the row below, the maximum total from top to bottom is 23.

That is, 3 + 7 + 4 + 9 = 23.

Find the maximum total from top to bottom of the triangle below:

[Check out big triangle in original link]

NOTE: As there are only 16384 routes, it is possible to solve this problem by trying every route. However, Problem 67, is the same challenge with a triangle containing one-hundred rows; it cannot be solved by brute force, and requires a clever method! ;o)

Permalink: http://projecteuler.net/problem=18

As note says we can try every route and it will be just good. Instead of that, we can behave much better with clever method.

(Some time ago, I gave basic explanation on my approach in StackOverflow question)

We have example triangle:

Instead of testing each route from top to the bottom, we moving from bottom to the top and always select the best choice. As we want to find the max sum, we perform addition im place.

Assuming we on the 3rd line (2 4 6).

From 2 we can go either 8 or 5, 8 is better (maximize your sum), then you calculate first sum 8 + 2 = 10.

From 4 we can go either 5 or 9, 9 is better, 4 + 9 = 13.

From 6 we can go either 9 or 3, 9 is better, 9 + 6 = 15.

Now, we removing last row, because we processed all routes and selected best ones, and replacing 3rd row with calculated sums. Current triangle looks like:

        3
      7  4
    10 13 15

Repeat the same action until we get one number, which will be solution.

Let’s transform this idea to clojure.

First of all we need datastructure to represent triangle. List of lists would be good:

(def triangle '((3) (7 4) (2 4 6) (8 5 9 3)))

Now we want to know, what direction generates the best sum.

(defn max-row [lst]
  (map #(reduce max %) (partition 2 1 lst)))

It breaks list to pairs, and select max from each pair. For example, [8 5 9 3] we split to [[8 5] [5 9] [9 3]] and select max from each sublist [8 9 9].

Just add this list to previous one

(defn step-max [lst1 lst2]
  (map + (max-row lst1) lst2))

One iteration of step-max creates smaller triangle with sums on the bottom row. But we need only one number.

Correct, reduce is ideal!

(reduce step-max (reverse triangle))

That’s it.

code

P.S. Problem 067 can be solved with the same code.

Programming Digest 2

2013-04-22T14:43:00+00:00

Why math is needed for developers, what is REST, how to become experienced developer…in 10 years, “coding it’s just writing” and some practice: Naive Bayes Classifier, functional programming and Java 8.

1. Math For Programmers

Another great post by Steve Yegge about math. Math for developers. As everyone might to retort, knowing math is not mandatory for software engineering field, but you know:

Math is… ummm, please don’t tell anyone I said this; I’ll never get invited to another party as long as I live. But math, well… I’d better whisper this, so listen up: (it’s actually kinda fun.)

I totally agree. It’s a lot of fun. Just read subreddit math, even if you don’t understand anything.

Pick your favourite math subject and try to dig more. Statistics, Linear Algebra, Discrete Math, Combinatorics, Operation Research, Group Theory and more; one of them can be a good start.

Does that really helps? Well, maybe. But it’s definitely improve your problem solving skill.

And believe. Math makes you a better programmer.

2. How I Explained REST to My Wife

Good article about how web works. I found it very useful for understanding some parts I thought I understood. By the way, great examples.

Ryan: I can’t drive a cup and I can’t drink a car. But some verbs are almost universal like GET, PUT, and DELETE.

Wife: You can’t DELETE a cup.

Ryan: Well, okay, but you can throw it away.

Unfortunately, author of this article take it down due to lot of complaints about article gender-oriented nature. I don’t think it’s offensive and here is the link with copy from web archive

3. Programmer Competency Matrix

Do you think you are great programmer? Test yourself.

Algorithms

2^n (Level 0). Unable to find the average of numbers in an array (It’s hard to believe but I’ve interviewed such candidates)
n^2 (Level 1). Basic sorting, searching and data structure traversal and retrieval algorithms
n (Level 2). Tree, Graph, simple greedy and divide and conquer algorithms, is able to understand the relevance of the levels of this matrix.
log(n) (Level 3). Able to recognize and code dynamic programming solutions, good knowledge of graph algorithms, good knowledge of numerical computation algorithms, able to identify NP problems etc.

In any case, this test helps you to build plan for programming related skills and gives understanding what you need to know to become an expert.

4. Teach Yourself Programming in Ten Years

Article by Peter Norvig. Great guy, I got to know from Artificial Intelligence course.

His writing is critique of popular books, like “Learn Java in 21 Days”. Peter explains that it is impossible to be a great engineer in such short period, and to become an expert you need aproximately 10 years.

It reminds me a picture “How to become a C++ Developer in 21 Days”:

If you totally disagree with his idea, just test yourself in previous article.

5. On Being A Senior Engineer

And logically continue. What behaviour differentiate senior engineer from “not senior” one?

Mature engineers seek out constructive criticism of their designs.

Actually you could find more useful rules there and list of The Ten Commandments of Egoless Programming.

Understand and accept that you will make mistakes.

You are not your code.

No matter how much “karate” you know, someone else will always know more.

Don’t rewrite code without consultation.

Treat people who know less than you with respect, deference, and patience.

The only constant in the world is change. Be open to it and accept it with a smile.

The only true authority stems from knowledge, not from position.

Fight for what you believe, but gracefully accept defeat.

Don’t be “the coder in the corner.”

Critique code instead of people – be kind to the coder, not to the code.

Note, that article touch some psychological aspect of person. So I don’t think it can be 100% applied to everyone, but still worth to read.

6. Coding: It’s Just Writing

Jeff Atwood explais basic the idea of:

The unexpected relationship between writing code and writing.

To write good code, you must write good.

Article contains a reference to the book “The Elements of Style” with some quotes.

Vigorous writing is concise. A sentence should contain no unnecessary words, a paragraph no unnecessary sentences, for the same reason that a drawing should have no unnecessary lines and a machine no unnecessary parts. This requires not that the writer make all his sentences short, or that he avoid all detail and treat his subjects only in outline, but that every word tell.

Read this book. Perhaps it is boring, it contains some advices to improve you writing skills.

7. How To Build a Naive Bayes Classifier

Now some practice.

Article is great explanation of how Naive Bayes Classifier work. With pictures, code examples, common pitfalls and good sense of humour.

Although, it’s one of the simplest classification algorithms, but with some tweeks and good train set it can give good results.

Here is the place where the math is needed. But article is so great, that even monkey with banana (and degree in math) can understand how it works.

8. Functors, Applicatives, And Monads In Pictures

Number of articles that explain what is monad grow fast. Just another one.

Even if you don’t understand what Haskell and monad is, look at the cool pictures.

9. List Out Of Lambda

Good introduction to functional programming. Not for beginners, I think.

This blog post is a thought exercise. It’s not something you’d ever use for real code. But just like a guitarist practices scales that she won’t ever play in a song, we programmers should be exercising our brains every so often.

How to implement fancy list, map, filter features with just first-class functions. Basic operations on numbers included. JavaScript.

10. Everything about Java 8

Let’s review again what will be in Java 8. Most wanted release ever.

In this overview you will test other important changes, not only lambdas.

Sad, that release postponed to early 2014 due to security issues. But more time to play with developer preview features.

Clojure Euler: Problem 017

2013-04-09T21:13:00+00:00

If the numbers 1 to 5 are written out in words: one, two, three, four, five, then there are 3 + 3 + 5 + 4 + 4 = 19 letters used in total.

If all the numbers from 1 to 1000 (one thousand) inclusive were written out in words, how many letters would be used?

Permalink: http://projecteuler.net/problem=17

To solve this problem we need some sort of mapping that defines association between number and its string representation, e.g 123 = one hundred and twenty three.

This can be done with maps in clojure in form {:1 "one" :2 "two"}. But if pay more atention to this map, we see that key is a numeric value. Exactly what plain arrays do.

Then we just define all mappings, obtain all arrays string elements, find their lengths and sum them.

One problem that providing mapping for each number between 1 and 1000 is a “bit” routine task, so we need some formula to calculate representation instead of hardcoding it. We come to defining groups.

First group is just first 20 numbers:

(def first20 ["one" "two" "three" "four" "five"
              "six" "seven" "eight" "nine" "ten"
              "eleven" "twelve" "thirteen" "fourteen" "fifteen"
              "sixteen" "seventeen" "eighteen" "nineteen"])

Note: there is no “zero” as we don’t use it in spoken language.

Second group is numbers that divides 10:

(def decas ["" "ten" "twenty" "thirty" "forty" "fifty" "sixty" "seventy" "eighty" "ninety"])

Note: empty first element here is used to reduce additional check. Its length is 0, that has no effect at all.

And the last ones are three used values:

(def h "hundred")
(def t "thousand")
(def a "and")

The basic function will calculate the length string representation of some number. Let’s call it word-length.

(defn word-length [n]
  (cond (= n 1000) (+ (count t) (count (nth first20 1)))
        (< n 100)
        (let [q (quot n 10) m (mod n 10)
              d (count (nth decas q))]
          (if (zero? m) d
              (if (< q 2) (count (nth first20 (dec (+ m (* 10 q)))))
                  (+ d (count (nth first20 (dec m)))))))
        (< n 1000)
        (let [q (quot n 100) m (mod n 100)]
          (if (zero? m) (+ (word-length q) (count h))
              (+ (count a) (count h) (word-length q) (word-length m))))))

Little explanation:

If number is 1000 then the length of string “one” + “thousand”, 11.
If number is lower than 100 then extract its decade part. If there is no reminder for division by 10, the length is length of that decade part.
Otherwise, if number in first20, length taken from array first20.
Otherwise the length is decade part plus remainder.
If number is lower than 1000 then extract hundred part. If there is no reminder for division by 100, the length is “hundred” plus number of hundreds.
Otherwise, the length is “hundred” plus number of hundreds, plus the word “and” and recursively calculate the length of remainder, which is lower than 100.

Awful word-presentation of algorithm, because of… Reading code is much simpler, I know.

And the last part just sum up all word-lengths:

(reduce + (map word-length (range 1 1001)))

Check the code here.

P.S. In fact algorithm is very simple, but needs some time to understand few branches. By the way, the problem is little boring.

Code Golf: Game of Life

2013-03-07T17:59:00+00:00

Conway’s Game of Life in a tweet.

Recently, my friend pointed out me an article Life in a tweet where Game of Life was implemented in one tweet (less than 140 characters) in Ruby and F#. I took this challenge.

Here is the final version in 137 characters of Clojure:

(fn[g r](reduce(fn[i j](update-in i j(fn[v](get[v 1]
(-(apply +(map #(get-in g% 0)(for[a[-1 0 1]b[-1 0 1]]
(map + j[a b]))))v 2)0))))g r))

“Sugared” version

Translate tweet to more readable code:

(defn step-sugar [grid range]
  (let [count-neighbours ;; determine number of neighbours
        (fn[[i j]]
          (reduce + (map #(get-in grid % 0)
                         (for[a [-1 0 1] b [-1 0 1]]
                           [(+ i a) (+ j b)]))))
        new-value  ;; calculate new value for cell
        (fn [v [i j]]
          (let [c (- (count-neighbours [i j]) v)]
            (cond (= 3 c) 1
                  (= 2 c) v
                  :else 0)))
        evolve-cell ;; update cell in a grid
        (fn[g i] (update-in g i #(new-value % i)))]
    (reduce evolve-cell grid range)))

Actually, this version is cheating, due to passing one extra-parameter range that contains all indices need to be updated in form [[0 1] [1 1]]...

On the other side it gives us two features:

Works for rectangular grid. Although, initial implementation assumes the world is square.
Specific part of world can be updated. It gives possibility to update specific area in the world, for example west organisms evolve faster.

To use this function you need:

World

(def grid [[0 0 0]
           [1 1 1]
           [0 0 0]])

Evolution

(defn evolution [g]
  (let [p (count g) q (count (get g 0))
        range (for [i (range p) j (range q)][i j])]
    (iterate #(step-sugar % range) g)))

Visualization

(defn print-grid [g]
  (doseq [s (map #(apply str (replace {0 "." 1 "⚫"} %)) g)]
    (println s)))

Testing

To make sure results are correct use the following run function and compare results to some common examples.

(defn run [grid]
  (doseq [g (evolution grid)]
    (print-grid g)
    (Thread/sleep 1000)))

Pattern examples:

(def block [[0 0 0 0]
            [0 1 1 0]
            [0 1 1 0]
            [0 0 0 0]])

(def glider [[0 0 1 0 0 0 0 0]
             [1 0 1 0 0 0 0 0]
             [0 1 1 0 0 0 0 0]
             [0 0 0 0 0 0 0 0]
             [0 0 0 0 0 0 0 0]
             [0 0 0 0 0 0 0 0]])

Full history of implementation and more life patterns available here

Enjoy!

Clojure Euler: Problem 016

2013-03-05T01:31:00+00:00

2^15 = 32768 and the sum of its digits is 3 + 2 + 7 + 6 + 8 = 26.

What is the sum of the digits of the number 2^1000 ?

Permalink: http://projecteuler.net/problem=16

Are you kidding me?

Find 1000th power of two
Sum its digits

In Clojure Euler: Problem 008 we learned how to sum digits in the number. Just gentle reminder:

(defn sum-of-digits [n]
  (reduce + (map #(- (int %) 48) (seq (str n)))))

Now, let’s create sequence of powers of two:

(defn powers-of-2 []
  (iterate (partial * 2) 1))

Unfortunately, this sequence throws integer overflow on the 64th element. We can fix that using long arithmetics, which known as BigInteger in Java. Change 1 to 1N.

(iterate (partial * 2) 1N)

Another way is to use automatic promotion operator (+', *'). If result of some operation is not suitable for some type, instead of invalid computation and runtime exception, clojure automatically promotes the type to suitable one (for example Long.MAX_VALUE +' 1 works fine and produces correct result with type of BigInteger):

(iterate (partial *' 2) 1)

Choose powers-of-2 that you prefer and final result will look like this:

(sum-of-digits (last (take 1001 (powers-of-2))))

github

P.S. Automatic promotion is a beatiful thing. But be aware about losing in speed of calculations. Also, no way back. If promotion happened, depromotion won’t.

Java Magic. Part 4: sun.misc.Unsafe

2013-02-26T02:37:00+00:00

Java is a safe programming language and prevents programmer from doing a lot of stupid mistakes, most of which based on memory management. But, there is a way to do such mistakes intentionally, using Unsafe class.

This article is a quick overview of sun.misc.Unsafe public API and few interesting cases of its usage.

Unsafe instantiation

Before usage, we need to create instance of Unsafe object. There is no simple way to do it like Unsafe unsafe = new Unsafe(), because Unsafe class has private constructor. It also has static getUnsafe() method, but if you naively try to call Unsafe.getUnsafe() you, probably, get SecurityException. Using this method available only from trusted code.

public static Unsafe getUnsafe() {
    Class cc = sun.reflect.Reflection.getCallerClass(2);
    if (cc.getClassLoader() != null)
        throw new SecurityException("Unsafe");
    return theUnsafe;
}

This is how java validates if code is trusted. It is just checking that our code was loaded with primary classloader.

We can make our code “trusted”. Use option bootclasspath when running your program and specify path to system classes plus your one that will use Unsafe.

java -Xbootclasspath:/usr/jdk1.7.0/jre/lib/rt.jar:. com.mishadoff.magic.UnsafeClient

But it’s too hard.

Unsafe class contains its instance called theUnsafe, which marked as private. We can steal that variable via java reflection.

Field f = Unsafe.class.getDeclaredField("theUnsafe");
f.setAccessible(true);
Unsafe unsafe = (Unsafe) f.get(null);

Note: Ignore your IDE. For example, eclipse show error “Access restriction…” but if you run code, all works just fine. If the error is annoying, ignore errors on Unsafe usage in:

Preferences -> Java -> Compiler -> Errors/Warnings ->
Deprecated and restricted API -> Forbidden reference -> Warning

Unsafe API

Class sun.misc.Unsafe consists of 105 methods. There are, actually, few groups of important methods for manipulating with various entities. Here is some of them:

Info. Just returns some low-level memory information.
- addressSize
- pageSize
Objects. Provides methods for object and its fields manipulation.
- allocateInstance
- objectFieldOffset
Classes. Provides methods for classes and static fields manipulation.
- staticFieldOffset
- defineClass
- defineAnonymousClass
- ensureClassInitialized
Arrays. Arrays manipulation.
- arrayBaseOffset
- arrayIndexScale
Synchronization. Low level primitives for synchronization.
- monitorEnter
- tryMonitorEnter
- monitorExit
- compareAndSwapInt
- putOrderedInt
Memory. Direct memory access methods.
- allocateMemory
- copyMemory
- freeMemory
- getAddress
- getInt
- putInt

Interesting use cases

Avoid initialization

allocateInstance method can be useful when you need to skip object initialization phase or bypass security checks in constructor or you want instance of that class but don’t have any public constructor. Consider following class:

class A {
    private long a; // not initialized value

    public A() {
        this.a = 1; // initialization
    }

    public long a() { return this.a; }
}

Instantiating it using constructor, reflection and unsafe gives different results.

A o1 = new A(); // constructor
o1.a(); // prints 1

A o2 = A.class.newInstance(); // reflection
o2.a(); // prints 1

A o3 = (A) unsafe.allocateInstance(A.class); // unsafe
o3.a(); // prints 0

Just think what happens to all your Singletons.

Memory corruption

This one is usual for every C programmer. By the way, its common technique for security bypass.

Consider some simple class that check access rules:

class Guard {
    private int ACCESS_ALLOWED = 1;

    public boolean giveAccess() {
        return 42 == ACCESS_ALLOWED;
    }
}

The client code is very secure and calls giveAccess() to check access rules. Unfortunately, for clients, it always returns false. Only privileged users somehow can change value of ACCESS_ALLOWED constant and get access.

In fact, it’s not true. Here is the code demostrates it:

Guard guard = new Guard();
guard.giveAccess();   // false, no access

// bypass
Unsafe unsafe = getUnsafe();
Field f = guard.getClass().getDeclaredField("ACCESS_ALLOWED");
unsafe.putInt(guard, unsafe.objectFieldOffset(f), 42); // memory corruption

guard.giveAccess(); // true, access granted

Now all clients will get unlimited access.

Actually, the same functionality can be achieved by reflection. But interesting, that we can modify any object, even ones that we do not have references to.

For example, there is another Guard object in memory located next to current guard object. We can modify its ACCESS_ALLOWED field with the following code

unsafe.putInt(guard, 16 + unsafe.objectFieldOffset(f), 42); // memory corruption

Note, we didn’t use any reference to this object. 16 is size of Guard object in 32 bit architecture. We can calculate it manually or use sizeOf method, that defined… right now.

sizeOf

Using objectFieldOffset method we can implement C-style sizeof function. This implementation returns shallow size of object:

public static long sizeOf(Object o) {
    Unsafe u = getUnsafe();
    HashSet<Field> fields = new HashSet<Field>();
    Class c = o.getClass();
    while (c != Object.class) {
        for (Field f : c.getDeclaredFields()) {
            if ((f.getModifiers() & Modifier.STATIC) == 0) {
                fields.add(f);
            }
        }
        c = c.getSuperclass();
    }

    // get offset
    long maxSize = 0;
    for (Field f : fields) {
        long offset = u.objectFieldOffset(f);
        if (offset > maxSize) {
            maxSize = offset;
        }
    }

    return ((maxSize/8) + 1) * 8;   // padding
}

Algorithm is the following: go through all non-static fields including all superclases, get offset for each field, find maximum and add padding. Probably, I missed something, but idea is clear.

Much simpler sizeOf can be achieved if we just read size value from the class struct for this object, which located with offset 12 in JVM 1.7 32 bit.

public static long sizeOf(Object object){
    return getUnsafe().getAddress(
        normalize(getUnsafe().getInt(object, 4L)) + 12L);
}

normalize is a method for casting signed int to unsigned long, for correct address usage.

private static long normalize(int value) {
    if(value >= 0) return value;
    return (~0L >>> 32) & value;
}

Awesome, this method returns the same result as our previous sizeof function.

In fact, for good, safe and accurate sizeof function better to use java.lang.instrument package, but it requires specifyng agent option in your JVM.

Shallow copy

Having implementation of calculating shallow object size, we can simply add function that copy objects. Standard solution need modify your code with Cloneable, or you can implement custom copy function in your object, but it won’t be multipurpose function.

Shallow copy:

static Object shallowCopy(Object obj) {
    long size = sizeOf(obj);
    long start = toAddress(obj);
    long address = getUnsafe().allocateMemory(size);
    getUnsafe().copyMemory(start, address, size);
    return fromAddress(address);
}

toAddress and fromAddress convert object to its address in memory and vice versa.

static long toAddress(Object obj) {
    Object[] array = new Object[] {obj};
    long baseOffset = getUnsafe().arrayBaseOffset(Object[].class);
    return normalize(getUnsafe().getInt(array, baseOffset));
}

static Object fromAddress(long address) {
    Object[] array = new Object[] {null};
    long baseOffset = getUnsafe().arrayBaseOffset(Object[].class);
    getUnsafe().putLong(array, baseOffset, address);
    return array[0];
}

This copy function can be used to copy object of any type, its size will be calculated dynamically. Note that after copying you need to cast object to specific type.

Hide Password

One more interesting usage of direct memory access in Unsafe is removing unwanted objects from memory.

Most of the APIs for retrieving user’s password, have signature as byte[] or char[]. Why arrays?

It is completely for security reason, because we can nullify array elements after we don’t need them. If we retrieve password as String it can be saved like an object in memory and nullifying that string just perform dereference operation. This object still in memory by the time GC decide to perform cleanup.

This trick creates fake String object with the same size and replace original one in memory:

String password = new String("l00k@myHor$e");
String fake = new String(password.replaceAll(".", "?"));
System.out.println(password); // l00k@myHor$e
System.out.println(fake); // ????????????

getUnsafe().copyMemory(
          fake, 0L, null, toAddress(password), sizeOf(password));

System.out.println(password); // ????????????
System.out.println(fake); // ????????????

Feel safe.

UPDATE: That way is not really safe. For real safety we need to nullify backed char array via reflection:

Field stringValue = String.class.getDeclaredField("value");
stringValue.setAccessible(true);
char[] mem = (char[]) stringValue.get(password);
for (int i=0; i < mem.length; i++) {
  mem[i] = '?';
}

Thanks to Peter Verhas for pointing out that.

Multiple Inheritance

There is no multiple inheritance in java.

Correct, except we can cast every type to every another one, if we want.

long intClassAddress = normalize(getUnsafe().getInt(new Integer(0), 4L));
long strClassAddress = normalize(getUnsafe().getInt("", 4L));
getUnsafe().putAddress(intClassAddress + 36, strClassAddress);

This snippet adds String class to Integer superclasses, so we can cast without runtime exception.

(String) (Object) (new Integer(666))

One problem that we must do it with pre-casting to object. To cheat compiler.

Dynamic classes

We can create classes in runtime, for example from compiled .class file. To perform that read class contents to byte array and pass it properly to defineClass method.

byte[] classContents = getClassContent();
Class c = getUnsafe().defineClass(
              null, classContents, 0, classContents.length);
    c.getMethod("a").invoke(c.newInstance(), null); // 1

And reading from file defined as:

private static byte[] getClassContent() throws Exception {
    File f = new File("/home/mishadoff/tmp/A.class");
    FileInputStream input = new FileInputStream(f);
    byte[] content = new byte[(int)f.length()];
    input.read(content);
    input.close();
    return content;
}

This can be useful, when you must create classes dynamically, some proxies or aspects for existing code.

Throw an Exception

Don’t like checked exceptions? Not a problem.

getUnsafe().throwException(new IOException());

This method throws checked exception, but your code not forced to catch or rethrow it. Just like runtime exception.

Fast Serialization

This one is more practical.

Everyone knows that standard java Serializable capability to perform serialization is very slow. It also require class to have public non-argument constructor.

Externalizable is better, but it needs to define schema for class to be serialized.

Popular high-performance libraries, like kryo have dependencies, which can be unacceptable with low-memory requirements.

But full serialization cycle can be easily achieved with unsafe class.

Serialization:

Build schema for object using reflection. It can be done once for class.
Use Unsafe methods getLong, getInt, getObject, etc. to retrieve actual field values.
Add class identifier to have capability restore this object.
Write them to the file or any output.

You can also add compression to save space.

Deserialization:

Create instance of serialized class. allocateInstance helps, because does not require any constructor.
Build schema. The same as 1 step in serialization.
Read all fields from file or any input.
Use Unsafe methods putLong, putInt, putObject, etc. to fill the object.

Actually, there are much more details in correct inplementation, but intuition is clear.

This serialization will be really fast.

By the way, there are some attempts in kryo to use Unsafe http://code.google.com/p/kryo/issues/detail?id=75

Big Arrays

As you know Integer.MAX_VALUE constant is a max size of java array. Using direct memory allocation we can create arrays with size limited by only heap size.

Here is SuperArray implementation:

class SuperArray {
    private final static int BYTE = 1;

    private long size;
    private long address;

    public SuperArray(long size) {
        this.size = size;
        address = getUnsafe().allocateMemory(size * BYTE);
    }

    public void set(long i, byte value) {
        getUnsafe().putByte(address + i * BYTE, value);
    }

    public int get(long idx) {
        return getUnsafe().getByte(address + idx * BYTE);
    }

    public long size() {
        return size;
    }
}

And sample usage:

long SUPER_SIZE = (long)Integer.MAX_VALUE * 2;
SuperArray array = new SuperArray(SUPER_SIZE);
System.out.println("Array size:" + array.size()); // 4294967294
for (int i = 0; i < 100; i++) {
    array.set((long)Integer.MAX_VALUE + i, (byte)3);
    sum += array.get((long)Integer.MAX_VALUE + i);
}
System.out.println("Sum of 100 elements:" + sum);  // 300

In fact, this technique uses off-heap memory and partially available in java.nio package.

Memory allocated this way not located in the heap and not under GC management, so take care of it using Unsafe.freeMemory(). It also does not perform any boundary checks, so any illegal access may cause JVM crash.

It can be useful for math computations, where code can operate with large arrays of data. Also, it can be interesting for realtime programmers, where GC delays on large arrays can break the limits.

Concurrency

And few words about concurrency with Unsafe. compareAndSwap methods are atomic and can be used to implement high-performance lock-free data structures.

For example, consider the problem to increment value in the shared object using lot of threads.

First we define simple interface Counter:

interface Counter {
    void increment();
    long getCounter();
}

Then we define worker thread CounterClient, that uses Counter:

class CounterClient implements Runnable {
    private Counter c;
    private int num;

    public CounterClient(Counter c, int num) {
        this.c = c;
        this.num = num;
    }

    @Override
    public void run() {
        for (int i = 0; i < num; i++) {
            c.increment();
        }
    }
}

And this is testing code:

int NUM_OF_THREADS = 1000;
int NUM_OF_INCREMENTS = 100000;
ExecutorService service = Executors.newFixedThreadPool(NUM_OF_THREADS);
Counter counter = ... // creating instance of specific counter
long before = System.currentTimeMillis();
for (int i = 0; i < NUM_OF_THREADS; i++) {
    service.submit(new CounterClient(counter, NUM_OF_INCREMENTS));
}
service.shutdown();
service.awaitTermination(1, TimeUnit.MINUTES);
long after = System.currentTimeMillis();
System.out.println("Counter result: " + c.getCounter());
System.out.println("Time passed in ms:" + (after - before));

First implementation is not-synchronized counter:

class StupidCounter implements Counter {
    private long counter = 0;

    @Override
    public void increment() {
        counter++;
    }

    @Override
    public long getCounter() {
        return counter;
    }
}

Output:

Counter result: 99542945
Time passed in ms: 679

Working fast, but no threads management at all, so result is inaccurate. Second attempt, add easiest java-way synchronization:

class SyncCounter implements Counter {
    private long counter = 0;

    @Override
    public synchronized void increment() {
        counter++;
    }

    @Override
    public long getCounter() {
        return counter;
    }
}

Output:

Counter result: 100000000
Time passed in ms: 10136

Radical synchronization always work. But timings is awful. Let’s try ReentrantReadWriteLock:

class LockCounter implements Counter {
    private long counter = 0;
    private WriteLock lock = new ReentrantReadWriteLock().writeLock();

    @Override
    public void increment() {
        lock.lock();
        counter++;
        lock.unlock();
    }

    @Override
    public long getCounter() {
        return counter;
    }
}

Output:

Counter result: 100000000
Time passed in ms: 8065

Still correct, and timings are better. What about atomics?

class AtomicCounter implements Counter {
    AtomicLong counter = new AtomicLong(0);

    @Override
    public void increment() {
        counter.incrementAndGet();
    }

    @Override
    public long getCounter() {
        return counter.get();
    }
}

Output:

Counter result: 100000000
Time passed in ms: 6552

AtomicCounter is even better. Finally, try Unsafe primitive compareAndSwapLong to see if it is really privilegy to use it.

class CASCounter implements Counter {
    private volatile long counter = 0;
    private Unsafe unsafe;
    private long offset;

    public CASCounter() throws Exception {
        unsafe = getUnsafe();
        offset = unsafe.objectFieldOffset(CASCounter.class.getDeclaredField("counter"));
    }

    @Override
    public void increment() {
        long before = counter;
        while (!unsafe.compareAndSwapLong(this, offset, before, before + 1)) {
            before = counter;
        }
    }

    @Override
    public long getCounter() {
        return counter;
    }

Output:

Counter result: 100000000
Time passed in ms: 6454

Hmm, seems equal to atomics. Maybe atomics use Unsafe? (YES)

In fact this example is easy enough, but it shows some power of Unsafe.

As I said, CAS primitive can be used to implement lock-free data structures. The intuition behind this is simple:

Have some state
Create a copy of it
Modify it
Perform CAS
Repeat if it fails

Actually, in real it is more hard than you can imagine. There are a lot of problems like ABA Problem, instructions reordering, etc.

If you really interested, you can refer to the awesome presentation about lock-free HashMap

UPDATE: Added volatile keyword to counter variable to avoid risk of infinite loop.

Kudos to Nitsan Wakart

Bonus

Documentation for park method from Unsafe class contains longest English sentence I’ve ever seen:

Block current thread, returning when a balancing unpark occurs, or a balancing unpark has already occurred, or the thread is interrupted, or, if not absolute and time is not zero, the given time nanoseconds have elapsed, or if absolute, the given deadline in milliseconds since Epoch has passed, or spuriously (i.e., returning for no “reason”). Note: This operation is in the Unsafe class only because unpark is, so it would be strange to place it elsewhere.

Conclusion

Although, Unsafe has a bunch of useful applications, never use it.

Elephant-Oriented Programming

2013-02-26T00:58:00+00:00

Have you heard about Elephant-Oriented Programming?

1. Read very clever book with an elephant on cover.

2. I bet you didn’t understand anything, so choose easy programming language with an elephant.

3. What is IDE? We already have elephant text editor.

4. For real life programming tasks we need storage. PostgreSQL database is good. Because of elephant.

5. And finally, fancy word Big Data. Did you say “big”? Elephants are big.

Stirling's Approximation

2013-02-23T01:38:00+00:00

Do you like factorials?

Probably, you do. If no, whatever, read this post to know how get rid of factorials.

TL;DR

\[ln(n!) = n \cdot ln(n) - n + 1\]

It is Stirling’s approximation or just Stirling’s formula. It allows to replace factorials with their approximation. If you not interested in math, skip to formula usage

Proof

Replace factorial with its definition:

\[ln(n!) = ln(1 \cdot 2 \cdot 3 ...)\]

Transform logarithm of multiplication to sum of logarithms:

\[ln(1 \cdot 2 \cdot 3 ...) = ln(1) + ln(2) + ln(3) + ... + ln(n)\]

Take a look at the integral $\int_a^b f(x) dx$ Replace $f(x)$ with $ln(x)$, $a$ with $1$, and $b$ with $n$. Then use numerical integration with step 1 and right rectangle rule

\[\int_1^n ln(x) dx \approx ln(2) + ln(3) + ... + ln(n)\]

Right-hand parts of two previous formulas are equal, with respect $ln(1)=0$. Then:

\[ln(1 \cdot 2 \cdot 3 ...) \approx \int_1^n ln(x) dx\]

Calculate integral using integration by parts:

\[\int ln(x) dx = x \cdot ln(x) - \int x d(ln(x)) = x \cdot ln(x) - \int dt = x \cdot ln(x) - x + C\]

Applying boundaries [1..n]

\[n \cdot ln(n) - n + C - 1 \cdot ln(1) + 1 - C = n \cdot ln(n) - n + 1\]

yields final result:

\[ln(n!) = n \cdot ln(n) - n + 1\]

Exactly what on the top.

Developer Proof

It is much simpler proof, with lose of accuracy, but still valid for partial applications.

Replace factorial with its definition:

\[ln(n!) = ln(1 \cdot 2 \cdot 3 ...)\]

Transform logarithm of multiplication to sum of logarithms:

\[ln(1 \cdot 2 \cdot 3 ...) = ln(1) + ln(2) + ln(3) + ... = \sum_{i=1}^n ln(i)\]

From monotonic property of logarithms $a \leq b \rightarrow ln(a) \leq ln(b)$ take upper bound for each $ln(i)$:

\[\forall i \; ln(i) \leq ln(n)\]

Then, using worst case substitution, sum can be rewritten as:

\[\sum_{i=1}^n ln(i) \leq \sum_{i=1}^N ln(n)\]

But $ln(n)$ does not depend on i, so get rid of sum:

\[\sum_{i=1}^n ln(n) = n \cdot ln(n)\]

Done.

Usage

Algorithm complexity

No one cares about O(log(n!)) of your algorithm, but everybody knows that O(n log n), linearithmic complexity, is very good property of algorithm. By the way, do not scare people with factorial sign.

Overflow prevention

If you try to calculate ln(n!) with most straightforward implementation without suitable datatype, you’ll likely get the number overflow exception, in spite of result value is not very large. For example for n=100 overall result is approximately 363 (Stirling’s approximation gives 361) where factorial value is $10^{154}$.

Speedup

As far as I know, calculating factorial is O(n) complexity algorithm, because we need n multiplications. Using Stirling’s approximation we need to calculate $n^n$ term, that can be calculated in O(log n) with clever trick

Gamma function

It can be surprising, but factorials defined not only for positive integers. So, for example 2.5! is completely valid expression. -6! also. They can be calculated using Gamma-functions for real numbers. Unfortunately, they are pretty complicated, so people use Stirling’s formula instead.

Partial applications

Almost everything that using factorials can use Stirling’s approximation. Permutations, binomial coefficients, fractals, number theory and a huge amount of topics I’m not aware of.

P.S. Stirling’s formula presented here is rough enough. More accurate formula is $n! = \sqrt{2\pi} n^{n + 0.5}e^{-n}$. Check its proof here.

Heisenberg Principle

2013-02-21T00:24:00+00:00

It is so obvious, so I decided to write about it.

Heisenberg principle came from quantum mechanics and means that you can’t know exactly both coordinates and speed of particle. If you improve accuracy of detecting position, you lose accuracy in detecting speed and vice versa. Uncertain stuff.

In human-readable form it sounds like

Either one or the other

The same in every field, even in software engineering. If you implementing something or fixing bugs and your boss yelling: “you must work better and faster” tell him either one or the other.

P.S. Actually, you can say “go fuck yourself”, but it won’t be polite.

Clojure Euler: Problem 015

2013-02-19T00:38:00+00:00

Starting in the top left corner of a 2 x 2 grid, there are 6 routes (without backtracking) to the bottom right corner.

How many routes are there through a 20 x 20 grid?

Permalink: http://projecteuler.net/problem=15

Picture is always a huge help to find correct the solution. Let’s see the pattern here, with another picture

Warning: If you don’t understand art, don’t look at that picture.

It’s a solution for 3 x 3 grid. We trying to process it by steps:

At the start we located at the top left corner [0, 0]. There is just one case to be at the start so we put 1 to that point.
Then we move either down [1, 0] or right [0, 1], and we put 1 on the edges of the second diagonal line.
Obviously, the point [1, 1] we can reach either from [0, 1] or from [1, 0], so there are two routes, and so on.

If you just output the numbers from each diagonal line you’ll get:

It’s famous Pascal’s triangle. Each element in this triangle obtained by sum of two closest elements from the upper row.

To get desired result we need to sum these numbers again, until we get one number. But it’s also numbers from Pascal’s triangle.

    1   3   3   1
      4   6   4
       10  10
         20

We can treat Pascal’s triangle as a sequence, and each row is an element. Then the following function calculates next row:

(defn routes-extend [lst]
  (let [size (count lst)]
    (for [i (range (inc size))]
      (if (or (= 0 i) (= size i)) 1
        (+ (nth lst (dec i)) (nth lst i))))))

Usage:

(routes-extend [1]) => [1 1]
(routes-extend [1 1]) => [1 2 1]

Now we can build infinite sequence of Pascal’s triangle rows:

(iterate routes-extend [1])

For 3 x 3 grid we had 7 diagonals, each of them represented one row. Take a guess. For n x n grid we need to get 2 * n + 1 row. And take it middle element, which, of course, will be with the index n.

That’s the final solution:

(let [n 20 d (inc (* n 2))]
  (nth (last (take d (iterate routes-extend [1]))) n))

Problem solved.

Code

P.S Two points I want to admit here. First of all, drawing pictures and trying to understand problem with pictures is a powerful technique that must be used for every problem.

Second of all, you can see that our algorithm has complexity of O(n^2). But ideal solution is O(1) with usage binomial coefficient (2*n, n).

Standard ML

2013-02-16T13:33:00+00:00

Last weekend I was hacking Standard ML a bit.

Coursera

There are excellent course on coursera called Programming Languages

Currently, the first half is available, that explains basic and advanced functional programming constructions with Standard ML as primary language. Later some idioms will be presented using Racket and Ruby (maybe, I will finally understand ruby) so you can treat this course as introduction to these languages. In fact, it teaches much more than “how to code in XXX”.

Tools

For playing with ML you need SML/NJ compiler and sml-mode for emacs.

If you don’t have emacs installed, do it right now!

Play

Following snippets give you basic understanding of ML or refresh your knowledge for syntax.

Print “Hello, world!”:

fun print_hello_world () = print "Hello, world\n";

Variables (in fact, values):

val a = 38;
val b = a + 3;
val c = b + 1;

Conditionals:

val cond = if 5 > 4 then 5 else 4

Boolean operators have terrible names:

val cond2 = if 5 > 4 andalso 4 > 5 then 5 else 7

Function with one parameter:

fun square x = x * x

Function with more than one parameter:

fun sum (x, y, z) = x + y + z

Actually, it’s not three parameters, it’s one parameter - tuple:

fun swap (x, y) = (y, x)

To access tuple elements use hash-number, note you can specify tuple type:

fun get_month (date : int * string * int) : string= (#2 date)
val month1 = get_month(2012, "January", 21);

You can specify input types and return type, but ML is very clever. Almost always it can detect correct type using type inference without explicit definition.

I don’t understand it! Explain, please. Comments:

(* Operation '*' defined for int types, so this function has type int -> int *)
fun cube x = x * x * x

There is records. It’s like named tuples.

fun get_month_real ({year, month, day}) = month
val month2 = get_month_real({year=2012, month="January", day=21});

Using let form we can define new lexical scope:

fun add3 x =
    let
        val num = 3
    in
        x + 3
    end

This leads to closures (function + context):

fun add3 x =
    let
        val num = 3
        fun add_to x = x + num
    in
        add_to x
    end

Function add_to capture num value from context.

By the way functions are first-class objects and values also, so we can assign one function to another:

val another_cube = cube

Function composition becomes a piece of cake:

val cube_and_cube = cube o cube

We even can pass function as a parameter:

fun binary_apply (f, x, y) = f (x, y)
binary_apply(swap, 1, 2);

We don’t forced to provide function name, we can use lambdas:

binary_apply(fn (x, y) => x * y, 23, 46)

This helps us easily implement such useful functions as map, reduce and filter. But they are already available in Standard ML Basis Library

Another great feature is pattern-matching. Here is factorial function

fun factorial x =
    case x of
        0 => 1
      | _ => x * factorial(x - 1)

or list concatenation

fun concat (x, y) =
    case x of
        [] => y
      | x::xs => x::concat(xs, y)

Honestly, it’s just baby examples of pattern matching. In fact, it is much more powerful.

And the last thing I wanted to show is datatypes:

To define a list:

datatype 'a list = EMPTY | CONS of 'a * 'a list

To create that list:

val list = CONS(1, CONS(2, CONS(3, EMPTY)))

That’s very small part of all that can be done in ML. There are also a lot of useful constructs and idioms that not covered: type synonyms, polymorphic types, equality types, tail recursion, mutual recursion, function wrapping, currying, partial application, advanced pattern matching, signatures, modules and lot more.

If you really interested you will find a lot of these terms exciting.

Full code is available on github

Impressions

ML is a great language. Perhaps, it is not so great as Java in enterprise, and not so great as Clojure in rapid prototyping, but it’s worth to devote few days for ML basics. It really improve your dev skills and maybe made your Factories, Bridges and Proxies better.

Clojure Euler: Problem 014

2013-02-16T01:08:00+00:00

Which starting number, under one million, produces the longest Collatz chain?

Permalink: http://projecteuler.net/problem=14

The following iterative sequence is defined for the set of positive integers:

n → n/2 (n is even)

n → 3n + 1 (n is odd)

Using the rule above and starting with 13, we generate the following sequence:

13 → 40 → 20 → 10 → 5 → 16 → 8 → 4 → 2 → 1

It can be seen that this sequence (starting at 13 and finishing at 1) contains 10 terms. Although it has not been proved yet (Collatz Problem), it is thought that all starting numbers finish at 1.

Which starting number, under one million, produces the longest chain?

NOTE: Once the chain starts the terms are allowed to go above one million.

Sequences again. So, first of all we need function to detect next collatz number. It is pretty straightforward:

(defn collatz-next [n]
  (if (even? n) (/ n 2) (inc (* n 3))))

Using this next-element-generator we can generate infinite collatz sequence, starting from number n:

(iterate collatz-next n)

But instead of infinite sequence, we stop processing when we reach 1 and calculate count of collatz chain sequence:

(defn collatz-chain-length [n]
  (inc (count (take-while #(> % 1) (iterate collatz-next n)))))

Now we have almost all code logic and just use it: calculate collatz chain for all numbers in range [1..1000000], find maximum and output its index:

(first (reduce #(if (> (second %1) (second %2)) %1 %2)
               (map #(list % (collatz-chain-length %)) (range 1 1000000))))

Run it. Wait a minute. I mean minute literally.

The time of caclulation is slightly greater than one minute on my machine, and I have no idea how to solve it another way.

I am sure there is a better algorithm than mine. But I don’t know it. What I know is try to re-read problem definion if you are stuck. Especially that part:

13 → 40 → 20 → 10 → 5 → 16 → 8 → 4 → 2 → 1

So, for 13 the chain length is 10, for 40 is 9, for 20 is 8,…

Got it?

Each time we calculate length we perform repeated calculation. If we know chain length of 13, and whole its path to 1 we don’t need to calculate length for 40, 20 and so on.

So the optimization is somehow to cache the results. But at first we need to rewrite it to have recursive solution.

Here the first (and final) version:

(defn collatz-chain-recursive [n]
  (if (= n 1) 1
      (inc (collatz-chain-recursive (collatz-next n)))))

Recursion is impressive at the point that its implementation almost exactly follows the problem definition. It is easy to understand.

But, as you may notice our implementation not tail-recursive so we can have troubles. Do not think about it unless we actually have them.

For the sake of fun let’s run this function instead of previous to obtain the results.

20 seconds, it is great. I even don’t think that this simple recursive function gives such boost in performance.

One little tweak in result function:

(first
  (apply max-key second
         (map #(list % (collatz-chain-recursive %)) (range 1 1000000))))

And we have ~12 seconds.

Unexpected “The End”. Yeah, I love The Doors.

code-code

P.S. I really wanted to explain technique called memoization and simple clojure function memoize, but seems that caching result doesn’t have much sense here because 12 seconds is acceptable. We will have chance to introduce memoization later, where it will be more useful. By the way, I am lazy.

Programming Digest 1

2013-02-12T02:00:00+00:00

How to get job at Google, estimate complexity of your algorithm, what is Structural Immutability, why PHP is awful and Java is awesome, best programming jokes and lot more.

1. Get that job at Google

…it’s highly likely that someone on the loop will be unimpressed with you, even if you are Alan Turing. Especially if you’re Alan Turing, in fact, since it means you obviously don’t know C++

The quote accurately explains what this post about. The author gives great mental and technical tips about interviewing process (not only in Google). Big portion of motivation guaranteed, perhaps you man feel dumber after reading.

In few words, you must know: algorithm complexity, sorting, hashtables, trees, graphs, other data structures, math, operating systems and at least one programming language very well. Not bad, right? After reading that article I’ve build long-term learning plan for myself, which is still not succesful. Great article.

2. A Gentle Introduction to Algorithm Complexity Analysis

This article helps you partially cover first thing from previous article: algorithm complexity. By the way, it is written with understandable (for programmers) language, examples, pictures, excerpts - I like such style.

After reading, you’ll understand what time/space complexity is, difference between Theta and Big-O notations, recursive functions complexity and lot more. By the way, you will learn basic classes of time complexities, and get skill to classify that complexities on O(n), O(n^2 ), O(n*logn), etc. Note, the articles consists of some excercises, which immediately followed with solutions. It is more productive to solve this exercises by yourselves and then compare to solutions.

If you already know all that, don’t skip! It also helpful to fresh you knowledge.

3. On Structural Immutability

What is Structural Immutability?

We know difference between mutable and immutable data structures and their pros and cons. Structural immutability means that you can change the contents of you data, but can not change their structure. It gives better performance than plain immutable objects, but also keep your concurrent programs easier than with full mutability. Plain old array is an example. Read it to know more.

4. PHP: a fractal of bad design

Virtually every feature in PHP is broken somehow

Whole article is a criticism of bad PHP language design.

PHP was originally designed explicitly for non-programmers (and, reading between the lines, non-programs)

Total destroy. I mean it - total.

Array… This one datatype acts as a list, ordered hash, ordered set, sparse list, and occasionally some strange combination of those. How does it perform? What kind of memory use will there be? Who knows? Not like I have other options, anyway.

As I never know PHP, and hope won’t, that article was one of the funniest things I ever read.

equality operator is not transitive…comparison is not even consistent

If you PHPist don’t go away, just read it. Maybe it will change your life.

Genericism… There is none. If a function might need to do two slightly different things, PHP just has two functions.

Stop quoting. Just read it.

5. Do Not Pass This Way Again

Considering MySQL? Use something else. Already on MySQL? Migrate.

This article logically continues previous one, but for now critisism on MySQL. Perhaps, it is not powerful as PHP version, but still worth to read.

6. Big arrays in Java

Two articles of hate in a row, and currently Java? No way. How much we hate Java so we love it. By the way, JDK8 is soon.

Maybe this post not so expressive as previous ones, but it reveals capability I never knew of. How to create array in Java with size greater than Integer.MAX_VALUE. Use the undocumented class sun.misc.Unsafe. Just note this only works in SunJDK and OpenJDK.

7. State of the Lambda

If we started about JDK8…

The most interesting feature set in expected JDK8 is Project Lambda.

Functional interfaces and functions without all that finals in lexical scope and anonymous classes, common patterns for list processing like map, reduce and filter, parallelism, streams, method references and lot more comming soon.

What is the default method implementation in the interface?!

8. 40+ Fantastic Game Development Tutorials From Across the Web

Have you ever had the passion to write your own game?

I had. Because I like games and I even tried. Unfortunately, not so succesful as expected, but anyway.

We need to go deeper

Honestly, it’s not a joke that I post link to another digest. It gathers a lot of gamedev tutorials and if you really interested in gamedev, you’ll find a lot of there including game algorithms, popular engine tutorials, math behind the games, design questions and more useful tips. Great stuff.

9. How to pair socks from a pile efficiently?

Bored to pair your socks every time? Learn how to do it efficiently.

At first glance it is a humorous question, indeed it is, but also it is an interesting real-world problem with variety of algorithms can be applied. I think all algorithm courses should be teached that way.

10. What is your best programming joke

If you put a million monkeys at a million keyboards, one of them will eventually write a Java program.

The rest of them will write Perl programs.

If you have an hour or two reserved for laughing, just read this. A lot of crazy programming jokes. Stackoverflow community is amazing.

Programming Digest Announcement

2013-02-11T23:00:00+00:00

Recently, I come up with idea to share my favorite articles I found on the internet and give basic description for them.

I do not plan this digest as periodical thing, so the first pattern I found appropriate is to publish this as soon as I have ten articles marked as favorite somewhere.

That’s the rule: each post - 10 articles.

Favorite means really favorite, so I do not crosspost every article I’ve read.

Hard to say what topics will be covered. I call it programming digest because most of articles will be somehow related to programming, but do not blame me if you see articles with another topic (math or humour), believe me they can be interesting also.

By the way, a lot of articles may be old, even very old (don’t call me Necromancer) but if they appear here, I found them somehow relevant and important to post.

Hope, you’ll find such sort of digest is interesting for reading and I like if you share your favorite programming-related articles with me.

Have fun!

Data Analysis: Facebook Hacker Cup 2013

2013-02-01T12:42:00+00:00

What programming language is most used?

To answer this question we use results from the qualification round of Facebook Hacker Cup 2013

If you only interested in statistics, skip!

0. Intro

As you may know, Facebook Hacker Cup is a programming contest. You have programming problem and you need to solve it, in most cases in efficient way. You get input file and you need to submit the output file in less than 6 minutes. Pretty simple. The interesting part you need to submit your source code, that can be reviewed by any of contestants after round.

These sources we use to get programming language statistics.

1. Data

Primary resource in statistics is data.

We have scoreboard page (You must be logged to FB to see results) and interested in all links named source. Open every link manually takes much time, so we will use automated approach and will write program. Yes, in clojure.

Dependencies

For facebook authentification we can use:

I didn’t think too much about alternatives, because I wanted to try selenium in clojure. It’s a time!

In few words, selenium provides capability to perform programmatically browser events. Often used in automation. To use selenium in clojure program just add [clj-webdriver "0.6.0-beta2"] to your project dependencies.

To simplify HTTP GET access we use library [clj-http "0.6.3"] and for additional help [org.clojure/clojure-contrib "1.2.0"].

In code we use following requires

(:require [clj-http.client :as http])
(:require [clj-webdriver.taxi :as web])
(:require [clj-webdriver.core :as c])
(:require [clojure.contrib.math :as math])

Automation

Before we will automate actions to gather sources we need to decide what these actions are?

Run browser
Open facebook url
Login to facebook
- Enter login
- Enter password
- Press submit
Go to scoreboard page
Process each page loop [1..n]
- Go to page n
- Get all links
- Filter links with sources
- Process each link loop [1..k]
  - Go to link url k
  - Get all page source
  - Filter out source code
  - Generate file name
  - Write source to file

Pretty clear, right? Let’s code it sequentially.

Implementation

Run browser and Open facebook url

(web/set-driver! {:browser :firefox} "http://facebook.com")

Login to facebook

(web/input-text "#email" username)
(web/input-text "#pass" password)
(web/submit "#pass")

Go to scoreboard page

(web/to "https://www.facebook.com/hackercup/scoreboard?round=185564241586420")

Process each page

(doseq [page-num (range 1 (inc 114))]
  (process-page page-num))

Number of pages is hardcoded. It’s ok. What is process-page method?

(defn process-page [n]
  (let [url (str "https://www.facebook.com/hackercup/scoreboard?round=185564241586420&page=" n)]
    (web/to url)
    (doseq [e (web/find-elements {:css "a"})
            :let [url (c/attribute e "href")]
            :when (and url (.startsWith url "https://fbcdn-dragon-a.akamaihd.net/"))]
      (process-url url))))

First of all, we concat scoreboard link with page number to get actual link to each page. Then we go to that page obtain all a elements, get their href values and filter to save only ones that contain source code.

Process each url

(defn process-url [url]
  (let [source-code (:body (http/get url))
        file-name (generate-filename url)]
    (spit file-name source-code)))

In this part we obtain html source of url and get its :body tag. As all source urls contain just plain text, we don’t need additional filtering. Just save it to file.

Generate file name

I don’t want to name file as url. That’s why for naming we use following function: timestamp + underscore + absolute value of url hashcode

(defn generate-filename [url]
  (str DIRECTORY (System/currentTimeMillis) "_" (math/abs (hash url)) ".txt"))

DIRECTORY is just def for folder where you want to place all sources.

Whole script source available here

Execution

To run that script you need to write in clojure REPL (run "username" "password"), with correct values for username and password, obviously.

It works some time. Some time equals to eight hours on my machine. Long enough. But it’s ideal time for night crawling!

In the morning I had all work done.

ls -1 | wc -l && du

prints

20348
291120  .

Good sign. We have more than 20K of source codes with total size almost 300Mb.

2. Analysis

Data is good. But no one interested in raw data, so we need process it.

Basically, we need to detect programming language by source file. No extensions.

You can write your own classifier or use some existing tool.

I did few-minutes research on this topic and found linguist project. It is written in Ruby and used in Gist to detect snippet language. Exactly what we need!

Unfortunately, I do not know ruby. I even could not build and run linguist classifier to detect language in my files. Rvm, gems and modules driving me crazy. I surrendered.

Another solution to use javascript library Highlight.js. It is used in syntax coloring, but have automatic language detection. Again, javascript and reading files from filesystem… Don’t tell me about Node.js

I decided to write my own “classifier”. Honestly, it’s just regexp matching mechanism on common language constructions: keywords, imports, most used functions, etc.

Iterative approach has been used.

Select some popular language construction say #include <iostream> and filter it as C. After filtering we detect some subset of C language, remove them from all files list and repeat again with another construction.

I don’t know how C++ different from C, so I accept them both as interchangeable languages but call it C/C++. By the way, C - C++ = 0, so we assume they are equal.

I developed some number of patterns (they can be reviewed here) and processed all gathered source files.

Unfortunately, not all files were processed succesfully. I reviewed approximately 2000 files manually, few new languages were detected but big amount of them were the crap: input data, binary files, some text information.

I think we don’t lose too much if we say only 99% of files were processed.

3. Statistics

If you don’t like this visualization, you can create your own. Here is data:

C/C++          10524
Java            3117
Python          3102
C#              1233
PHP              821
Ruby             488
Perl             142
Pascal/Delphi    136
Javascript       109
Haskell           85
Scala             72
Clojure           29
Go                28
Visual Basic      19
F#                12
Scheme             8
OCaml              7
Common Lisp        6
Lua                5
Matlab             4
Cocoa              3
Groovy             2
Dart               2
awk                1
Powershell         1
bash               1
Kotlin             1
ActionScript       1
Dylan              1
--------------------
crap             192
not detected     196
--------------------
TOTAL          20348

4. Conclusion

Top 5 was predicted but it’s extremely crazy that C/C++ occur more than 50%
Python has almost the same amount as Java
Good occurence of functional languages: Haskell, F#, Scala, Clojure, Ocaml
29 Clojure files that means there ~10 clojurists.
Two languages I never heard of: Dart and Dylan
Parenthesis languages still used: Common Lisp, Scheme and Clojure (of course)
Respect to matlabists
Few exotic languages used: awk, powershell, bash
Kotlin becomes popular!

Warning: Do not treat this statistics as real-world pattern. It is a programming competition with a lot of geeks, they can use all they want. Also do not blame their code for quality. It was created just for correctness and speed. Pay attention to code quality in production, but always remember:

Your code may be elegant, by mine fucking works.

– Leon Fayer

P.S. First of all, it is not high-accurate statistics. As I am not programming guru, I don’t know all possible languages’ constructions so it is likely that one construction overlapped with another. In that case only first one will be checked. Ideally, would be good to have language detecting library in clojure for future analysis. Maybe, I will do it. Second of all, code presented here is not beautiful, not optimized, have a lot of hardcode, but it works. Just in the spirit of Facebook Hacker Cup.

Clojure Euler: Problem 013

2013-01-25T16:33:00+00:00

Work out the first ten digits of the sum of the following one-hundred 50-digit numbers.

Permalink: http://projecteuler.net/problem=13

Assume, here is a huge amount of digits. In any case, you can always find them in permalink.

What is the problem to sum one-hundred numbers?

(reduce + numbers)

Done.

Are you kidding me? Not so fast. The question is what type of number you must choose to sum all these numbers. int? long? looooong? super long long?

Technique to sum very long numbers called Bignum arithmetic. To solve that problem you just need to implement this technique… or use already implemented.

Java (yeah we talking about it) support long arithmetics with BigInteger. And clojure does.

(reduce + (map bigint numbers))

With that function you’ll get the big number. Just truncate first 10 digits and you are good.

(apply str (take 10 (str (reduce + (map bigint numbers)))))

(for [i “GitHub”] i)

P.S. Ideally, last line should be wrapped with (read-string ...) to produce number type. But we using manual submission, so it is fair.

Clojure Euler: Problem 012

2013-01-22T19:45:00+00:00

What is the value of the first triangle number to have over five hundred divisors?

Permalink: http://projecteuler.net/problem=12

The sequence of triangle numbers is generated by adding the natural numbers. So the 7th triangle number would be 1 + 2 + 3 + 4 + 5 + 6 + 7 = 28.

The first ten terms would be:

1, 3, 6, 10, 15, 21, 28, 36, 45, 55, …

Let us list the factors of the first seven triangle numbers:

1
1,3
1,2,3,6
1,2,5,10
1,3,5,15
1,3,7,21
1,2,4,7,14,28

We can see that 28 is the first triangle number to have over five divisors.

What is the value of the first triangle number to have over five hundred divisors?

When we see infinite sequence of some numbers, first thing that we do is implement it with using lazy sequences. Simple explanation and example for Fibonacci numbers can be found in Clojure Euler: Problem 002.

We have an easy case, so we implement triangle number as sum of all integers below:

(defn triangle-number [n]
  (reduce + (range 1 (inc n))))

Don’t miss the possibility reduce complexity from O(n) to O(1). Just apply arithmetic progression formula.

(defn triangle-number [n]
  (* n (/ (+ n 1) 2)))

Okey, we did small optimization (not so small) here and we have a function that calculates triangle number for n. Let’s find all of them!

(def triangles (map triangle-number (iterate inc 1)))

triangles refers to lazy sequence of triangle numbers.

Do not evaluate lazy seqs!

Use limit functions as take, take-while, drop, drop-while to test the sequence values, or build other lazy seqs with filter, map, etc.

Now, we need a function to calculate number of divisors.

(defn num-of-divisors [n]
  (* 2 (count (filter #(= (mod n %) 0) (range 2 (inc (int (sqrt n))))))))

If we remember Clojure Euler: Problem 003 then we know that using sqrt(n) instead of n as upper bound for divisors saves much time. Don’t forget mutiply that value by two, as you skip number after sqrt(n).

Last step: to calculate number over 500 divisors

(first (drop-while #(< (num-of-divisors %) 500) triangles))

We got the result, but spent ~11 secs. Too much.

How can we improve our solution?

There is theorem about Prime Factorization states:

Every positive integer has a unique prime factorization

For example: 15 = 3^1 * 5^1, 18 = 2^1 * 3^2 and so on.

Not hard to see that number of divisors from such factorization can be obtained by multiplying all prime powers incremented by one.

For example number 18 have 6 divisors (1, 2, 3, 6, 9, 18). If we take factorization 18 = 3^1 * 5^1, then number of divisors is equal to (1 + 1) * (2 + 1) = 6. You see, the same. Not hard to prove this theorem or read the proof.

Stop math, we need to code, so let’s code that.

First of all, we need factorization method:

(defn factorize [n]
  (loop [x n [p & ps] primes factors []]
    (cond (= 1 x) factors
          (zero? (mod x p)) (recur (/ x p) primes (conj factors p))
          :else (recur x ps factors))))

It uses primes from clojure.contrib.lazy-seqs. This method prints all factors (including duplicates) for n.

For example:

(factorize 18) => [2 3 3]

But instead of actual values of divisors, we just need their count.

(defn factorize-count [n]
  (reduce * (map (comp inc count) (vals (group-by identity (factorize n))))))

Calculate the result again:

(first (drop-while #(< (factorize-count %) 500) triangles))

Bingo! It gives the correct result in less than 3 seconds. Not bad as improvement.

P.S. If you test-addicted person, you, probably, point out that function num-of-divisors yields incorrect result for input 1. We can live with that, because our needed number is much greater than 1.

Critical Software Bugs: Part 1

2013-01-20T18:48:00+00:00

Little journey through the history of most critical software bugs with some code examples.

Do not try to reproduce any of these!

Intro

Every last bug is the last but one.

Bugs are bad. Some of them cause uncomfortable work, more actions than expected, inconsistency, layout issues, etc. Some of them “not a bug” at all.

Bad bugs much worse. They cause data corruption, invalid data representation, unavailability to perform action, losing integrity and so on.

There are critical bugs with painful consequences. They can damage the health, nature, people break buildings, technics. Theoretically, they can lead to World War III. Hope, it’s just theoretically.

If you are software engineer, you definitely will commit the bugs. No panacea. But if we aware about common mistakes, then less likely we make that mistakes.

This article is review of famous most critical software bugs including brief history, consequences, root causes, possible solutions, code examples and some advices.

Mariner 1 (1962)

History: In 1962, spacecraft Mariner 1 was guided by computer program with error causes racket do not follow its expected trajectory. To prevent tragic consequences, racket was destoyed immediately.

Consequences: Money, $18 million.

Reason: There are tons of versions about what type of error it was, but official version is missing hyphen (‘-‘) in program. Maybe it’s just a myth for publicity, but, anyway, if the reason of bug is incorrect handwritten formula transcription from paper to computer, it’s sad.

Probably, the root cause of that is writer and computer operator were different people. It’s common case for that times. Programming was synonym to the scientific calculations not an enterprise, as nowadays. That’s why scientist, programmer, and operator could be different people.

Related story, it’s why zero symbol has a period or diagonal line inside - to prevent confusing with “big O” for “typer” person. Just compare 0 and O.

Solution: It’s hardly can be a problem today (as it was 50 years ago), but if you write for somebody on the paper, write it accurate. If you type into computer from someone’s paper sheet, check, re-check and double re-check what you typing. Ask if you have concerns. Especially, if it is the program for spaceship.

In very far-fetched example missing minus sign can result into invalid computation:

double getDiscountPercent() {
    return 0.25; // instead of return -0.25;
}

double calculatePrice(double initial) {
    return initial * (1.0 + getDiscountPercent());
}

That way instead of paying $75, you pay $125.

Lesson: Always check your manual rewriting actions.

Siberian pipeline sabotage (1982)

History: Cold war. Soviet Union steal technology for managing gas pipelines from Canada. This technology intentionally contained the bug (Logic Bomb), developed by CIA for sabotage purposes. Bomb triggered.

Consequences: The greatest non-nuclear explosion in the world.

Reason: Every microship from stolen technology would work fine just for 10 million cycles, after that switches to another mode.

They used someone’s code.

Solution:

You shall not steal

– Ten Commandments

Honestly, if someone’s technology used for critical areas, like gas transportation, it must be reviewed and tested. 100 times.

Story can be translated as:

I found memory card on the city dump. Card contained one shell script please-execute-me.sh. I executed it. System crashed. Who’s guilty? Just me.

Use software from trusted sources.

In modern software world, we often use someone’s libraries, frameworks, APIs to build our own applications. We expect that code is reviewed and tested. If not, we can help to do it. How we can be sure these libs without logic bombs?

Trust or review the sources. That’s why I love open source software.

Logic bomb detection is not that hard. Just find the code that causes error and see the reason of that error. Again, code review helps to prevent the logic bomb instead of fixing it. Build application to production only from version control system (which is controlled, right?)

Just for educational purposes: simple example of time-triggered logic bomb

public class LogicBombRunner {
        public static void main(String[] args) {
                Thread thread = new Thread(new LogicBomb());
                thread.start();
        }
}

class LogicBomb implements Runnable {

        private long toStop;
        private final long TIME_TO_TRIGGER = 1000 * 60 * 60 * 24 * 100;

        public LogicBomb() {
                toStop = System.currentTimeMillis() + TIME_TO_TRIGGER;
        }

        @Override
        public void run() {
                while (true) {
                        if (System.currentTimeMillis() < toStop) {
                                System.out.println("System works fine.");
                                try {
                                        Thread.sleep(1000);
                                } catch (InterruptedException e) {
                                        e.printStackTrace();
                                }
                        } else {
                                System.out.println("Logic bomb triggered.");
                                throw new RuntimeException();
                        }
                }
        }
}

Yes. It throws RuntimeException after approximately 100 days from LogicBomb object creation.

Lesson: Do not steal. Trust to very limited circle of people.

Therac-25 (1985 - 1987)

History: Therac-25 was a medical device for radiation therapy. It could deliver either beta-particles or X-rays. Unfortunately, operating system for controling the device was written with error, that causes sad results.

Consequences: At least 2 patients died, 4 patients were given overdose, approximately 100 times intended dose.

Reason: Investigation concluded the common reason was a bad software design, development practices and a set of bugs instead of some particular critical bug.

These bad practices and bugs include:

No code review
Poor exception handling
No integration testing until device was assembled in the hospital
Poor hardware design
Bad code reuse
Concurrency issues

Solution: Code review its a must. Even if you working alone in the team, show sources to some your friend (programmer). Two pair of eyes can spot the mistake two times more.

Poor exception handling. If failure occurs, no one knows about that failure. Never handle exception like that:

catch (SomeException e) {

}

In 99% there are must be a proper handling. If you don’t know how to handle, rethrow it, possible with wrapping:

catch (SomeException e) {
   throw new SpecificException(e);
}

Much better to throw responsibility for handling one level up in method signature:

void method() throws SomeException

If your real intention to left catch block empty, say this:

catch (SomeExcpetion e) {
  // swallowing this exception is intended
}

Bad code reuse. Code reuse is a good thing. Except the case when you reuse something with errors.

For example, I have following method for testing if number is even in some of my previous projects (yes, look at your code that was written few years ago):

static boolean isEven(int num) {
    return num % 2 != 1;
}

This method returns incorrect result true for all negative odd numbers. Just because % operator returns -1 except 1 for negative numbers. Better do NOT reuse such method. Just write new. And replace old method immediately.

static boolean isEven(int num) {
    return num % 2 == 0;
}

Concurrency. Software world hardest issues are concurrency issues.

You have a problem and decide to use threads. Two now problems have you.

– Some guy from internet

You have a problem and decide to use locks. Now you have

– Some another guy from internet

If you know that some object will be shared between threads, one solution to add synchronized (Monitor object) to each read/write state method. It will slow your work with that object but prevent a lot of unexpected errors.

Although, there are much flexible synchronization mechanisms in java that plain old synchronized as semaphores, mutexes, read-write locks.

I reccomend to use java package java.util.concurrent. It consists of a lot of concurrency techniques, like Synchronizers, Executors, Atomics, Concurrent Collections, Futures, Callbacks etc.

As a bonus: Good books about concurrency in Java:

Lesson: Always handle exceptions. Pay more attention to concurrency issues.

Patriot Missile (1991)

History: In 1991, American battery fired Patriot Missile to intercept Iraq missile. Interception failed because of system bug in time step calculation. Iraq missile destroyed an American Army barracks.

Consequences: 28 soldiers died, 100 injured.

Reason: Floating point rounding error. System’s internal clock was calculating time by 0.1 seconds step (0.1, 0.2, 0.3, …). As 0.1 = 1/10 have infinite binary representation 0.0001100110011001100110011001100110011001100110011001100.. and for this calculation 24-bit register was used, register hold just 0.00011001100110011001100 introducing truncating error 0.00000000000000000000000110011 what is approximately 0.0000001 in decimal.

Small enough, huh?

Not really, Multiplying this number for 100 hours gives:

0.0000001 * 100 * 60 * 60 * 10 = 0.36 sec

During that time Iraq missile travels a half kilometer, and was out of tracking radius for Patriot Missile. Bad things happen.

Solution: Floats are very error-prone if we don’t handle them correctly.

Predict the output of the following code:

for (double d = 0.0; d != 1.0; d = d + 0.1) {
    System.out.println("Iteration");
}

Infinite loop. The same problem as described above.

One solution is to use float that have exact binary representation. 0.125, 0.25, 0.5 for example. But this greatly reduces the space of allowed values and give less flexibility.

Another solution to use big decimal or ratio types. Java, as example, have out-of-the-box BigDecimal class. Loop above rewritten with that class works as expected:

BigDecimal init = new BigDecimal("0.0");
BigDecimal bound = new BigDecimal("1.0");
BigDecimal step = new BigDecimal("0.1")

for (BigDecimal d = init; !d.equals(bound); d = d.add(step)) {
        System.out.println(d);
}

One disadvantage it has slower execution than with floats. Consider trade-off between time and accuracy. If human lives on the stake, it’s not relevant choice.

Ratio class can be defined without any problems and used similar way.

Lesson: If you need very accurate calculations use big decimals or ratios.

Mars Climate Orbiter (1998)

History: Again about space. In 1998, communication with Mars Climate Orbiter was lost, because of changed trajectory. Space probe was crashed near Mars.

Consequences: Money, $327 million

Reason: Navigation system design was developed by another team. To represent Force they used pound-force from Imperial units instead of Newton from metric system.

1 Newton = 0.22 pound-force

Without provided conversion between them system failed.

Issue resemble to using dates in different timezones, but much more critical.

Solution: First of all, at least one science-aware person must perform code review. At least for code blocks with formulas and calculations. If there are comments about using pound-force, the problem easy spotted. Second of all, received code must be tested on model instead of physical machine. This is also, likely, detects the issue. Developer who wrote that less guilty, but still guilty.

There are different approaches to implement this in java:

very bad approach, we don’t know what units are used:

new Force(1.533);

better (still bad) approach to indicate with comment:

// force in Newtons!
new Force(1.533);

better (still bad) approach to indicate different constructors for different units:

new ForceInNewtons(1.533);

I prefer approach where you specifies a value and unit in the constructor, and they converted to something common unit (metric) convenient for use in internal representation:

enum ForceUnit {
  NEWTON,
  POUND_FORCE
}

new Force(1.533, ForceUnit.NEWTON);

A bit more code, but it is understandable and easy to use for different unit systems. Unit conversion method can be defined either at the enum ForceUnit or class Force level.

Lesson: Always explicitly indicate what units used in your code.

To be continued.

Clojure Euler: Problem 011

2013-01-18T13:45:00+00:00

What is the greatest product of four adjacent numbers in the same direction (up, down, left, right, or diagonally) in the 20 x 20 grid?

Permalink: http://projecteuler.net/problem=11

In the 20 x 20 grid below, four numbers along a diagonal line have been enclosed into square brackets

02 22 97 38 15 00 40 00 75 04 05 07 78 52 12 50 77 91 08
49 99 40 17 81 18 57 60 87 17 40 98 43 69 48 04 56 62 00
49 31 73 55 79 14 29 93 71 40 67 53 88 30 03 49 13 36 65
70 95 23 04 60 11 42 69 24 68 56 01 32 56 71 37 02 36 91
31 16 71 51 67 63 89 41 92 36 54 22 40 40 28 66 33 13 80
47 32 60 99 03 45 02 44 75 33 53 78 36 84 20 35 17 12 50
98 81 28 64 23 67 10[26]38 40 67 59 54 70 66 18 38 64 70
26 20 68 02 62 12 20 95[63]94 39 63 08 40 91 66 49 94 21
55 58 05 66 73 99 26 97 17[78]78 96 83 14 88 34 89 63 72
36 23 09 75 00 76 44 20 45 35[14]00 61 33 97 34 31 33 95
17 53 28 22 75 31 67 15 94 03 80 04 62 16 14 09 53 56 92
39 05 42 96 35 31 47 55 58 88 24 00 17 54 24 36 29 85 57
56 00 48 35 71 89 07 05 44 44 37 44 60 21 58 51 54 17 58
80 81 68 05 94 47 69 28 73 92 13 86 52 17 77 04 89 55 40
52 08 83 97 35 99 16 07 97 57 32 16 26 26 79 33 27 98 66
36 68 87 57 62 20 72 03 46 33 67 46 55 12 32 63 93 53 69
42 16 73 38 25 39 11 24 94 72 18 08 46 29 32 40 62 76 36
69 36 41 72 30 23 88 34 62 99 69 82 67 59 85 74 04 36 16
73 35 29 78 31 90 01 74 31 49 71 48 86 81 16 23 57 05 54
70 54 71 83 51 54 69 16 92 33 48 61 43 52 01 89 19 67 48

The product of these numbers is 26 * 63 * 78 * 14 = 1788696.

What is the greatest product of four adjacent numbers in the same direction (up, down, left, right, or diagonally) in the 20 x 20 grid?

First step is pretty routine, we need to create such grid representation in our program from file. For the way how we read files refer Clojure Euler: Problem 008, we have similar problem there.

(defn get-matrix []
  (map #(Integer/parseInt (apply str %))
       (partition 2 2 (remove #(or (= \newline %) (= \ %))
                              (seq (slurp "res/problem011.txt"))))))

Idea is following: read the file, drop the garbage, take all two-digit combinations, concat each and transform to the integer.

To make things simple we reading the numbers in one-dimensional list of size 20 x 20 = 400. We always can use formula to represent two dimensional indexing for one-dimensional array and vice versa.

Example: suppose you need a 3 x 3 matrix A

  2    4
  1    7
  3    8

You can flat it into 1D array B

9 2 4 6 1 7 5 3 8

Now, when you want A[i][j] use the formula i * size_j + j, where size_j is second matrix dimension. Therefore A[1][2] = B[1 * 3 + 2] = B[5] = 7, what is correct.

Implementing that in clojure gives one indexing function

(defn get-at [i j matrix]
  (if (and (>= i 0) (< i 20) (>= j 0) (< j 20))
    (nth matrix (+ j (* i 20))) 0))

We check the boundaries of array inside this function, which can be inappropriate in many cases, but it is good as we use it in one place.

Ok. We have data, we have methods operate this data, now let’s code logic. Our good friend - bruteforce.

(let [matrix (get-matrix)
      ways (for [i (range 20) j (range 20)]
             [(map #(get-at i (+ % j) matrix) (range 4))
              (map #(get-at (+ % i) j matrix) (range 4))
              (map #(get-at (+ % i) (+ % j) matrix) (range 4))
              (map #(get-at (+ % i) (- j %) matrix) (range 4))])]
    (reduce max (map #(reduce * %) (reduce concat ways))))

We iterate on all matrix elements as start of 4-element range.

We have 4 conditions, each of them represents some direction. All these 4 directions represent all possible direction in grid.

NORTH -> SOUTH (Up & Down covered)
WEST -> EAST (Left & Right covered)
NW -> SE (backslash diagonals covered)
NE -> SW (slash diagonals covered)

    S      S * * *         S                  S
    *                       *                *
    *                        *              *
    *                         *            *

S is for start

If we obtain all possible 4-element ranges for matrix, just find maximum of product.

Problem solved!

github

P.S. Boring problem. I think it just for introducing matrices computation.

Eclipse Speedup

2012-12-23T18:00:00+00:00

Have you been in situation where most of the time in eclipse you were waiting for some operation to be finished?

If you here, probably, the answer is yes. You are bored with that slow functionality and want to increase eclipse performance to make your development work more comfortable. It is the right thing to do.

Warning: Most frequent advice I heard to add more RAM. Probably, you can buy better workstation with better CPU. Add SSD to make filesystem operations faster. We assume you can’t afford all that. All you can is setup your eclipse instance. All preferences based on Eclipse SDK Version: 4.3.0 Build id: I20121031-2000 for ubuntu, but almost everything is identical in other versions for other platforms.

Eclipse tweaks

Plugins

First time I found the power of plugins I was excited. I installed them more and more until I couldn’t work comfortable with eclipse. Probably, you have a lot of plugins, just disable on startup ones that not used. It is not deleting, you always can enable it back.

Window -> Preferences -> General -> Startup and Shutdown

By the way, some plugins appeared due to your experimenting and not used anymore. Delete redundant plugins if they not used at all.

Help -> About Eclipse SDK -> Instalation Details -> <Select plugin> -> Uninstall

eclipse.ini

Following tweaks based on editing eclipse.ini file located in root folder of your eclipse instalation.

Use specific JVM to run eclipse. This allows do not take into account system properties which alters your JDK.

-vm
/path/to/your/java

Use latest JDK to run eclipse. Much better if you point your eclipse to the latest JDK. Rumors said it has better performance.
Use Sun JDK to run eclipse. The reason behind this has the same explanation.
Configure VM arguments. You can set your own values for virtual machine if you think they are appropriate (Performance related options) I use following settings to increase heap size (objects) to 768Mb and permgen (classes) to 256 Mb on my 3Gb RAM machine.

-vmargs
-Xms768m
-Xmx768m
-XX:PermSize=256m
-XX:MaxPermSize=256m

You can also add -Xverify:none that skips validation of loaded clases to JVM. Due to fact it skips one step of JVM loading you will get performance boost. But it’s very risky change.

You can also test your eclipse performance with different Garbage collector strategies, server options, experimental VM options like (some of these):

-server
-XX:+UnlockExperimentalVMOptions
-XX:+UseG1GC
-XX:+UseParallelGC
-XX:+UseFastAccessorMethods
-Xss2m

Check all Eclipse runtime options for reference and choose ones that appropriate for you.

Disable animations

Animations it’s cool. But I always disable animations if it possible in all tools. Also, classic theme is most usable for me.

Window -> Preferences -> General -> Appearance -> Uncheck 'Enable animations'

Disable label decorations

Label decorations are small icons on project, file, class levels that helps visually indicate some state of file. Like commited it to git or no. They are provided by various plugins and rarely useful. Just left ones that you want.

Window -> Preferences -> General -> Appearance -> Label Decorations

Autocompletion

Sometimes on slow machines or when you have a lot of classes in your classpath autocompletion performance is terrible. Small improvement you can get by reducing group of possible proposals. I left only Java Proposals and Template Proposals:

Window -> Preferences -> Java -> Editor -> Content Assist -> Advanced

Uncheck validators

If you are sure in your competence just suspend all validators. If problem appears, you always can identify the reason, but you save your development time.

Window -> Preferences -> Validation -> Suspend All Validators

Close unrelated projects

If you working in context of few projects, better to have all unrelated projects closed. They won’t appear in index. You can manually Close unrelated projects in workspace. But I recommend to use Working Set. You can add any number of projects to one working set, and quickly switch between different working sets in one workspace.

Right Click on Project -> Assign Working Sets..

Close unused tabs in editor

Big number of tabs in editor can greatly slow down your performance. Just limit them.

Window -> Preferences -> General -> Editors

Check Close editors automatically and set Number of opened tabs to 10.

Disable spell check

Are you developer or what? I don’t see any reason to have spell check enabled.

Window -> Preferences -> General -> Editors -> Text Editors -> Spelling -> Uncheck 'Enable spell checking'

Disable autobuild

Probably, you aware about when you want to build your project and when not.

Project -> Uncheck 'Build Automatically'
Window -> Preferences -> Java -> Compiler -> Building -> Uncheck 'Scrub output folders when cleaning'
Window -> Preferences -> Java -> Compiler -> Building -> Uncheck 'Rebuild class files modified by others'

Hotkeys

It is human-related advice. Even if you working in super-fast IDE and spend 10 actions to perform single operation, your development process won’t be fast. Configure most-often used actions to hotkeys and learn them. You will get a positive feedback after a week of usage.

Windows -> Preferences -> General -> Keys

To force myself learn all hotkeys I just disabled toolbar.

Window -> Hide Toolbar

Useful Links

P.S. This article planned to be updatable, other tweaks are welcome!

Clojure Euler: Problem 010

2012-12-15T00:23:00+00:00

The sum of the primes below 10 is 2 + 3 + 5 + 7 = 17.

Find the sum of all the primes below two million.

Permalink: http://projecteuler.net/problem=10

We have already worked with prime numbers in Problem 003 and Problem 007. We also decided that the best way to work with prime numbers in clojure to use primes lazy-seq from clojure.contrib.lazy-seqs.

Gentle reminder: Just add proper :use to your file:

(:use [clojure.contrib.lazy-seqs :only (primes)])

Then, the simplest problem solution to use take-while stream and reduce for sum:

(reduce + (take-while #(< % 2000000) primes))

Again, one-line solution. Let’s run it.

Finding result on my nachine takes ~12 seconds. Small enough, but maybe somehow it can be improved?

Sieve of Eratosthenes

Sieve of Eratosthenes - ancient algorithm for finding prime numbers. The best explanation how it works with picture from wiki:

Our first naive implementation:

(defn sieve-1 []
  (loop [nums (set (cons 2 (range 3 2000000 2))) n 3]
    (if (> n 2000000) (reduce + nums)
        (recur (clojure.set/difference nums (set (range (* n n) 2000000 n))) (inc n)))))

Note: clojure.set/difference function find the difference between two sets.

This recursive implementation is even worse than our oneliner, it calculates the result in ~24 seconds.

Primary optimisation is to use (* n n) as recursion base instead of n. Because of there are no prime numbers if we crossed greater than square root of maximum. Problem evolves.

(defn sieve-2 []
  (loop [nums (set (cons 2 (range 3 2000000 2))) n 3]
    (if (> (* n n) 2000000) (reduce + nums)
        (recur (clojure.set/difference nums (set (range (* n n) 2000000 n))) (inc n)))))

~21 seconds. Still worse.

Why we increment by 1 on recursion call? If we starting our n value from 3 we don’t need to go over even numbers.

(defn sieve-3 []
  (loop [nums (set (cons 2 (range 3 2000000 2))) n 3]
    (if (> (* n n) 2000000) (reduce + nums)
        (recur (clojure.set/difference nums (set (range (* n n) 2000000 n))) (+ n 2)))))

~13 seconds. Much better… than previous, still worse than first version.

If you looked at difference documentation, take a look at its implementation under “Source”. Nothing prevent us to pass sequence as second argument instead of set. Just removing casting to set.

(defn sieve-4 []
  (loop [nums (set (cons 2 (range 3 2000000 2))) n 3]
    (if (> (* n n) 2000000) (reduce + nums)
        (recur (clojure.set/difference nums (range (* n n) 2000000 n)) (+ n 2)))))

~6 seconds. Wow! It’s better than original solution. That means our efforts were not wasteful.

Unfortuantely, we stopped improve our function. But if you want more, other major improvements can be done in the following areas:

Use non-persistent set to save prime numbers. It prevent overhead on new objects.
Use “wheel trick” that can be found in the sources of primes from clojure.contrib.lazy-seqs.
Use dynamic step of iteration. We changed (inc n) to (+ n 2), but probably there is smarter solution.
Do not use (reduce +) because it is linear algorithm. We init nums sequence with some arithmetic progression. Its sum can be calculated in O(1) by formula. When we compose another sequence (second argument for difference) its also an arithmetic progression and sum can be calculated in O(1). When doing differencem just subtract second sum from first sum, and it will be current nums sum.

There are much more optimisations can be done to improve prime numbers performace. If you interested, read nice Christophe Grand’s post Everybody Loves The Sieve Of Erathosthenes

(str “Git” “Hub”)

P.S Honestly, I am OK with 12 seconds. But 6 seconds is better. We performed improvements just to show the point if you want to improve something, probably, you can do it.

Clojure Euler: Problem 009

2012-12-11T17:41:00+00:00

A Pythagorean triplet is a set of three natural numbers, a < b < c, for which, a^2 + b^2 = c^2

For example, 3^2 + 4^2 = 9 + 16 = 25 = 5^2

There exists exactly one Pythagorean triplet for which a + b + c = 1000. Find the product abc.

Permalink: http://projecteuler.net/problem=9

Thinking about clever way to solve it…

Nothing good came up in your mind? Try bruteforce.

Again, the idea behind bruteforce for this problem to iterate over all possible values for a, b and c, that sums to 1000 and compose Pythagorean triplet. That’s why we need predicate to test if three numbers compose triplet:

(defn is-triplet? [a b c]
  (= (+ (* a a) (* b b)) (* c c)))

Then our bruteforce solution looks like this:

(first (for [a (range 1 1000) b (range 1 1000) c (range 1 1000)
             :when (and (is-triplet? a b c) (= (+ a b c) 1000))] (* a b c)))

It finds the right solution, in ~45 seconds. Bad enough.

Trick #1

Do not iterate on c variable, because knowing a and b we always can calculate c = 1000 - a - b.

Problem solution transformed into next one:

(first (for [a (range 1 1000) b (range 1 1000)
             :let [c (- 1000 a b)]
             :when (is-triplet? a b c)] (* a b c)))

Trick #2

Use rule a < b < c from problem definition. Then our iteration will be:

(for [a (range 1 1000) b (range a (- 1000 a))])

Trick #3

is-triplet? predicate use 3 multiplications, addition and comparison. We can add to :when section, predicate that compare c to b. This is also slightly reduce number of is-triplet? execution.

:when (and (> c b) (is-triplet? a b c))

Now solution found in ~45 msecs. 1000 times faster. Not bad.

(apply str (reverse “buHtiG”))

P.S. Unfortunately, there was nothing in this problem about new clojure functions, programming and even problem was not challenging. But you see the way how we solve it. We tried “bad” approach with knowing about all its disadvantages. One more step and we improved it and got right solution. So it is not that bad approach to try simple solution, even if it is wrong.

Clojure Euler: Problem 008

2012-11-27T22:53:00+00:00

Find the greatest product of five consecutive digits in the 1000-digit number.

Permalink: http://projecteuler.net/problem=8

Large number under the cut.

73167176531330624919225119674426574742355349194934
96983520312774506326239578318016984801869478851843
85861560789112949495459501737958331952853208805511
12540698747158523863050715693290963295227443043557
66896648950445244523161731856403098711121722383113
62229893423380308135336276614282806444486645238749
30358907296290491560440772390713810515859307960866
70172427121883998797908792274921901699720888093776
65727333001053367881220235421809751254540594752243
52584907711670556013604839586446706324415722155397
53697817977846174064955149290862569321978468622482
83972241375657056057490261407972968652414535100474
82166370484403199890008895243450658541227588666881
16427171479924442928230863465674813919123162824586
17866458359124566529476545682848912883142607690042
24219022671055626321111109370544217506941658960408
07198403850962455444362981230987879927244284909188
84580156166097919133875499200524063689912560717606
05886116467109405077541002256983155200055935729725
71636269561882670428252483600823257530420752963450

Idea behind this problem is pretty straightforward:

Init maximum with some small value, zero would be good.
Take first 5 digits, calculate the product.
If product is greater than our current maximum, reassign it.
Shift, to take other 5 digits, and back to the step 1.

A simple optimization can be applied: if our last element of 5 digit group is zero, then skip 5 shifts. But it’s details.

Let’s make our problem little harder to get more from clojure.

Assume that these digits stored in the file. First thing we need: to read the file.

(slurp "path/to/the/file")

slurp is the simplest function to read file content (In fact, not only files). It just returns string representation of the whole file.

Then we need to break one large string into symbols.

(seq "helloyo") => (\h \e \l \l \o \y \o)

seq return a sequence with linear access to collection. If applied to string, return sequence of characters.

But, there are some garbage in the file we don’t want, new lines for example.

(remove #(= \newline %) [\h \e \l \l \o \newline \y \o])

(slash) + something - represents character in clojure. \newline is a new line, known as \n in other languages.

Now we have long sequence with all digits from file. Need to get all consecutive groups with 5 letters.

(partition 5 1 [\h \e \l \l \o \y \o]) => [[\h \e \l \l \o] [\e \l \l \o \y] ... ]

partition is an excelent function for such tasks. We specify 5 - size of partition, and 1 shift size. It generates all consecutive groups with 5 elements till the end.

Calulating the product for each group gives us the solution.

(reduce * [1 2 3 4])

Not so fast.

You, probably, get ClassCastException.

After reading the file, we just get bunch of characters, instead of digits in mathematical sense. And applied mathematical operation * to them. So, simply speaking, \1 is not 1.

That’s why we need to transform every digit character to it’s actual digit value.

(map #(- (int %) 48) [\1 \2 \3]) => [1 2 3]

This awful implementation do the work.

Cast each character to int, get the ASCII representation for digit.
From each ASCII value we subtract 48 and get digit value.

Applying product of digits defined earlier we get solution to this problem.

Again, full code on GitHub.

P.S. slurp function very powerful and has may applications. You can specify encoding for reading as you want or read whole html page by url. But it is also very dangerous function, because reads all file contents to the memory and can take considerable amount of time. If you have a large file, say 15GB, read it line by line with buffered reader. Also, note spit function. It is opposite to slurp - writes string to the file.

Clojure Euler: Problem 007

2012-11-20T15:37:00+00:00

By listing the first six prime numbers: 2, 3, 5, 7, 11, and 13, we can see that the 6th prime is 13.

What is the 10001st prime number?

Permalink: http://projecteuler.net/problem=7

It’s also pretty clear problem to make you more familiar with prime numbers. As I said in Clojure Euler: Problem 003, big amount of Project Euler problems devoted to prime numbers so it is MUST HAVE (I don’t like this phrase either)

You need to implement by yourself effective prime number algorithm. And only after that use one from library. As we do.

(:use [clojure.contrib.lazy-seqs :only (primes)])

This line import just one symbol primes to your namespace. It’s a prime numbers lazy sequence. Lazy sequences briefly covered in Clojure Euler: Problem 002 or Wikipedia article, that’s enough for now.

Return to question.

What is the 10001st prime number?

No problem.

(last (take 10001 primes))

Two lines of code: one for import, one for solution. What can be simpler?

Note, that last return last element of the sequence and works in linear time.

Another one problem solved in few ~~seconds~~ minutes.

GitHub

P.S. Actually, I lied. In linked solution I use 6 lines of code: namespace declaration, use, empty line, comment with executing time, function declaration and, finally, solution.

Java Magic. Part 3: Finally

2012-11-18T19:39:00+00:00

Every experienced java programmer should know that finally block always executed. But is it true?

It depends on our definition of program execution. But, generally speaking, yes.

Normal program execution

Aha, look at this, someone might retort:

try {
  System.exit(1);
} finally {
  System.out.println("I'm here, man");
}

You just said, finally block always executed?

Well. In that case no, because we speaking about normal flow of program execution. This is abnormal.

From the official tutorial

Note: If the JVM exits while the try or catch code is being executed, then the finally block may not execute.

Your counter question might be: If second line of that code always exectued?

System.out.println("Line 1");
System.out.println("Line 2");
System.out.println("Line 3");

Sure, because it’s linear flow. Nothing can break…BANG…Electricity unavailable. Program stopped.

What about this? It’s also abnormal program execution, and we can’t guarantee anything for 100%. In fact, this is the same as System.exit(1) or reset button on your computer or whatever.

That’s why, we are talking about normal program execution. Only normal.

I said yes? I meant no!

Perpetuum Mobile

Consider the following code:

try {
  while (true) {
    System.out.println("I print here some useful information");
  }
} finally {
  System.out.println("Let me run");
}

Will be the line “Let me run” printed? Maybe yes, if printing error to standard output appear. Almost always the answer is no.

In that case, there are no difference between simple statement and finally block. None of them will be executed, throw this example away.

Threads

What about threads? We know that execution flow controlled by threads and they can be interrupted.

Assume that we have thread that perform some work, and other thread kills first one right before finally block. Finally wasn’t executed.

Assume that we have deadlock between two threads, right before finally line. The same thing.

From the same tutorial

…if the thread executing the try or catch code is interrupted or killed, the finally block may not execute even though the application as a whole continues.

So, we can treat thread like a program, and make one effective rule:

Rule 1. Finally executes always, except the case where controlling program or thread was aborted.

Finally we return

Ok, now we know when finally is not executed. But do we know when finally is executed?

Consider the example:

int someFunc() {
  try {
    return 0;
  } finally {
    return 1;
  }
}

The result is obviously 1. Just because finally is always executed.

Consider, next example:

int someFunc() {
  try {
    throw new RuntimeException();
  } finally {
    return 1;
  }
}

The result is 1 again. And it’s a problem. We just lose the exception. Such issue known as exception swallowing. It is very dangerous, because client’s code expect either exception or some value, but it always get only value.

One more less imaginary example.

String deposit(int amount) throws DAOException {
  try {
    return dao.deposit(amount);
  } finally {
    return "OK";
  }
}

The logic behind finally is to have some default value, and our deposit method throws the DAOException where client code is responsible for its handling. Unfortunately, perhaps compiler forces you to handle this DAOException it never occurs. And string “OK” will be returned.

Rule 2. Never use return from finally block.

Instead of conclusion

A lot of programmers are aware about this common mistake. But some are not. Maybe these two simple rules give you a tiny light on coward finally.

Clojure Euler: Problem 006

2012-11-17T14:05:00+00:00

The sum of the squares of the first ten natural numbers is,

1^2 + 2^2 + … + 10^2 = 385

The square of the sum of the first ten natural numbers is,

(1 + 2 + … + 10)^2 = 55^2 = 3025

Hence the difference between the sum of the squares of the first ten natural numbers and the square of the sum is 3025 − 385 = 2640.

Find the difference between the sum of the squares of the first one hundred natural numbers and the square of the sum.

Permalink: http://projecteuler.net/problem=6

This problem is very simple, even simpler than previous one, that confirms fact that all Project Euler problems are NOT in increasing complexity order. Ok, let’s split our problem into two subproblems.

Sum of the squares

First, we need a square function:

(defn sqr [n] (* n n))

Then, sum of squares of the first one hundred natural numbers we calculate with common (reduce + list) idiom:

(reduce + (map sqr (range 1 101)))

Square of the sum

Just take a sum, and square it:

(sqr (reduce + (range 1 101)))

Bind all together

We splitted our problem into two smaller, now is time to bind all. We can just subtract results obtained but you see, that both of smaller solutions uses common piece of code (range 1 101). Move it to let form.

(let [rn (range 1 101)]
  (- (sqr (reduce + rn)) (reduce + (map sqr rn)))))

That’ all. I’ve got result in less than millisecond. Maybe there are some pitfalls in this problem (e.g. integer overflow for larger ranges) but this most straightforward solution works fine.

Congratulations!

GitHub

P.S. First time I solved this problem and was confused: Where is the trick? Seems nowhere. That’s why using most straightforward solution is often the best choice (excluding obvious cases, of course). It has fast implementation, and if it fails, it fails fast.

Fail Fast, Succeed Faster

Clojure Euler: Problem 005

2012-11-09T19:01:00+00:00

2520 is the smallest number that can be divided by each of the numbers from 1 to 10 without any remainder.

What is the smallest positive number that is evenly divisible by all of the numbers from 1 to 20?

Permalink: http://projecteuler.net/problem=5

It’s a tricky problem. First look gives us straightforward solution: Take a number, test if it evenly divisible by all of needed numbers, if yes - we good, otherwise increment number and test again, and again, and again…

No, really, assume it’s our function to test divisibility by all 20 numbers:

(defn evenly-divisible-by-20? [n]
  (zero? (reduce +
    (for [i (range 2 21)]
      (mod n i)))))

Awful implementation, but keeps the idea. If sum of all remainders equals to zero, then every remainder equals to zero. Now, using this function we write naive implementation:

    (first
      (drop-while #(not (evenly-divisible-by-20? %)) (iterate inc 1)))

Note, function drop-while similar to take-while, but skips values until they meet predicate condition.

Waiting for result? Take a cup of tea and wait a little. Still waiting? Wait a little more…

I did not post exact answer here, but result greater than 100 millions. And obviously this linear algorithm will work considerable amount of time. So, stop the execution and let’s see at the problem from the other end.

What is the smallest number that evenly divisible by 1 and 2? Obviously 2.

What is the smallest number that evenly divisible by 2 and 3? A bit harder, but answer is 6.

What is the smallest number that evenly divisible by 4 and 5? 20.

Oh, it’s just multiplication

Unfortunately, no. Take numbers 4 and 6, multiplication gives us 24 and it will be evenly divisible by 4 and 6, but this number is not the smallest. Snmallest number for this case will be 12.

There is a function, that finds smallest number divisible by 2 different numbers. This function called Least Common Multiple (lcm). There are lot of different methods to calculate it, but we will use one that uses Greatest Common Divisor (gcd)

Where gcd calculated according to Euclid’s algorithm

(defn gcd [n k]
  (loop [a n b k]
    (if (zero? b) a (recur b (mod a b)))))

Using formula above, our lcm function will look like this:

(defn lcm [n k]
  (/ (abs (* n k)) (gcd n k)))

You see, here we need abs function. Nothing hard to implement it, but if we remember previous problem there are was a reference to clojure.contrib.math library. abs there and we can use it, just add to your namespace definition

(:use [clojure.contrib.math])

Oh look, even lcm and gcd functions there. Excellent.

Now we can solve the problem. Remember the reduce function?

(reduce lcm (range 1 21))

Actually, this one-liner is solution. And it gives us correct result.

What happens here? We iteratively (with using reduce) find the lcm for 1 and 2, then for result and 3, for result and 4 and so on.

Great! One more problem solved!

GitHub

P.S. One lesson we learned here. Be aware about library functionality. It helps you to solve problems and avoid coward mistakes. For example, we have incorrect above implementation for lcm function. If we call (lcm 0 0) program fails with DivideByZero exception. It still solve our problem, but in future uses can be an additional problem. But in lib implementation this error is covered. And we can be sure that almost always libs contain much less errors that your new fancy wheel.

What is greater: e^pi or pi^e?

2012-11-06T16:00:00+00:00

You can answer almost immediately using calculator, but what if you don’t have it?

I don’t remember the exact origin of this meme, but this situation was almost every time when we played with numbers.

– Propose a number.

– e^π.

– Integer, shithead!

This even might appear in usual conversations:

– What is her size?

– e^π

– So small. I prefer π^e.

And a following arguing about what is greater? Until we simply compare results using calculator.

Let’s see how this problem can be solved without using a calc.

Warning: Goal of this article not only to show correct solution to the problem, but provide other solutions (even incorrect!) that can convince somebody.

Origins

π is the mathematical constant. It’s just a number that represents ratio between circle circumference and diameter C/d. Approximately, equal to 3.1415. Get more decimal places for FREE!

e is the harder one. If you sum infinite sequence below, you’ll get e. Approximately, equal to 2.7183. FREE decimal places without registration!

I’m pretty sure that’s more than enough you need to know to solve the problem.

Google It

Get answer from google

Rough estimate

We can use exponent function on integers, so let’s just round the numbers. 3.1415 becomes 3, 2.7183 becomes… 3 also.

3^3 vs 3^3
27 = 27

They are equal. Wrong answer.

Let’s use floor and ceiling functions.

Floor: floor(e) = 2, floor(π) = 3

2^3 vs 3^2
8 < 9

Ceiling: ceil(e) = 3, ceil(π) = 4

3^4 vs 4^3
81 > 64

Two different results depend on round function that we use. Can’t be sure. Need another method for verification.

Calculus

This needs basic understanding of logarithm and derivative

e^π vs π^e

Use logarithm property a < b => log(a) < log(b), and take log with base e (ln). Honestly, it’s not logarithm property, but monotonic function property. Who cares?

ln(e^π) vs ln(π^e)

Use logarithm property log_a(a^b) = b

π vs ln(π^e)

Use logarithm property log_a(b^c) = c * log_a(b)

π vs e * ln(π)

Now, people who know value of ln(π) can answer the question. We don’t know. Move ln(π) to the left side

π/ln(π) vs e

Take a look at expression π/ln(π). Change π to x. Got x/ln(x). What is it? It’s a function!

Let’s find extrema of this function f(x). Take first derivative.

f'(x) = (ln(x) - 1)/(ln^2(x))

We get extrema at the points where derivative equals to zero

(ln(x) - 1)/(ln^2(x)) = 0

Ratio number equals to zero if its numerator equals to zero.

ln(x) - 1 = 0

Move 1 to the right

ln(x) = 1

Logarithm equals to 1 if base equals to argument. Therefore

x = e

And if you know 1/ln(x) behavior, you get that extremum is a minimum of function f(x). Substitute x with e

e/ln(e) = e

Function minimum is e where x = e. If x = π, due to minimum, function value will be greater than e. Remember that we do this to get estimate of π/ln(π) which was on the left side of expression π/ln(π) vs e and if its greater than e, e^π greater than π^e. Huh. Done.

Note: We skip mathematical proof that extrema x = e is minimum. Not hard to show.

Manual computations

Why don’t we perform calculation manually?

Ok, we have π, which is 3.1415 or 22/7 its famous simplest ratio approximation. On the other side we have e, which is 2.7183. But I’m not aware of any e approximation. Let’s create one from the definition of e. Take and sum first 5 elements.

1 + 1 + 1/2 + 1/6 + 1/24 = 65/24

Excellent. Now we need just compare 2.7183^(22/7) and 3.1415^(65/24). What can be simpler?

Rational exponent can be expanded as nth-root of some power:

To take n-th root manually we use a special case of Newton’s method. Just one iteration:

Estimate for e^π:

2.7183^(22/7)
sqrt_7(2.7183^22)

…Multiplying 22 times…

In fact, not all so bad. To calculate a^22 we need only 6 multiplications:

a^1 * a^1 = a^2
a^2 * a^2 = a^4
a^4 * a^4 = a^8
a^8 * a^8 = a^16
a^16 * a^4 = a^20
a^20 * a^2 = a^22

Nth-root we obtain (decimal dropped):

sqrt_7(3585440111)

Applying Newton’s method. Select closest root:

1^7 = 7 < 3585440111
2^7 = 128 < 3585440111
3^7 = 2187 < 3585440111
...
23^7 = 3404825447 < 3585440111
24^7 = 4586471424 > 3585440111

Good. Most closest value is 23. Then

sqrt_7(3585440111.25)
sqrt_7(23^7 + 180614664)
23 + 180614664/(7 * 23^6)
23 + 180614664/1036251223
23 + 0.17
23.17

And the same for π^e:

3.1415^(65/24)
sqrt_24(3.1415^65)

This power 65 much larger than 22. But it needs just 7 multiplications.

We can cheat here. We get rough result 2 * 10^32. Now we test 23^24 (most closest root from previous) and its value 4.8 * 10^32. So, definitely nth-root will be lower than 23. Solved.

Don’t ask how many basic arithmetic operations we performed. A lot.

Liar approach

Actually, it’s offensive to use the word “liar”. There is a whole technique called Mathematical Fallacy. It based on some illegal math transformation (division by zero, roots from negative numbers, etc.) to get any desired result.

This is powerful instrument for mathematicians to show magic for not mathematicians.

Suppose, we don’t know exact value of π. Not a problem, but we know complex numbers. And also the fact that sqrt(-1) = i. Where i is imaginary unit

We have bad memory and can’t remember π value, but we know Euler’s formula

e^(i * x) = cos(x) + i * sin(x)

x is a variable. Take x = 2π:

e^(2π * i) = cos(2π) + i * sin(2π)

From the school math course we know: cos(2π) = 1, sin(2π) = 0. Just substitute:

e^(2π * i) = 1 + i * 0

Everything multiplied by 0 is 0. Next step:

e^(2π * i) = 1

a^b gives 1 if b = 0. That we also know from logarithms.

2π * i = 0

It means either 2 or π or i is zero. 2 is not zero. i is not zero. π we don’t know. Ok, that means π equals to 0.

Returning to comparing our constants:

e^0 vs 0^e

Left term is 1, right is 0. That means e^π greater than π^e.

The trick that we used hard to spot. Look carefully again through the proof.

The fallacy was where we implicitly took a logarithm and said that 2π * i = 0. The number e^(2π * i) is not real and we not allowed to logarithm it. But if person who listens you don’t know this fact… This is an excellent proof.

Programming approach

You not allowed to use calculator. Nothing said about programming language.

One-liner in Java:

System.out.println(Math.pow(Math.E, Math.PI) > Math.pow(Math.PI, Math.E));

Fun approach

Derived from joke:

Scientific lab wasted a lot of money for calculating decimal places of π. They come up with idea, let’s π will be equal to 4, to save money.

– But why 4 instead of 3?

– Because 3 reserved for e.

That way e^π equal to 81, and π^e equal to 64. 81 > 64. Simple, concise way.

Conclusion

There are much more fun about these irrational bastards. Math is fun.

If you were asked question of such type on interview, 99% that interviewer is joking and want to test your reaction. Last solution would be good. But if he or she really expect solution with using calculus, nothing to lose.

Java Magic. Part 2: 0xCAFEBABE

2012-10-31T19:42:00+00:00

Did you know that all java classes start with the same 4 bytes. In hex it’s CAFEBABE.

To make sure, create simple java file, call it Hello.java:

public class Hello {
    public static void main(String[] args) {
        System.out.println("Hell, O'World!");
    }
}

Compile it by javac Hello.java, and open Hello.class with hex-editor (M-x hexl-mode for emacs)

James Gosling explained this:

We used to go to lunch at a place called St Michael’s Alley. According to local legend, in the deep dark past, the Grateful Dead used to perform there before they made it big. It was a pretty funky place that was definitely a Grateful Dead Kinda Place. When Jerry died, they even put up a little Buddhist-esque shrine. When we used to go there, we referred to the place as Cafe Dead. Somewhere along the line it was noticed that this was a HEX number. I was re-vamping some file format code and needed a couple of magic numbers: one for the persistent object file, and one for classes. I used CAFEDEAD for the object file format, and in grepping for 4 character hex words that fit after “CAFE” (it seemed to be a good theme) I hit on BABE and decided to use it. At that time, it didn’t seem terribly important or destined to go anywhere but the trash-can of history. So CAFEBABE became the class file format, and CAFEDEAD was the persistent object format. But the persistent object facility went away, and along with it went the use of CAFEDEAD - it was eventually replaced by RMI.

0xCAFEBABE is 3405691582 in decimal. If we sum all digits we get 43. One more than 42 - Ultimate Answer to the Life, the Universe, and Everything. By the way, 43 is a prime number. You see, magic everywhere. Even in the last sentence.

Clojure Euler: Problem 004

2012-10-31T19:30:00+00:00

A palindromic number reads the same both ways. The largest palindrome made from the product of two 2-digit numbers is 9009 = 91 x 99.

Find the largest palindrome made from the product of two 3-digit numbers.

Permalink: http://projecteuler.net/problem=4

Palindrome is very interesting thing. It’s a sequence (of letters, numbers, words) that reads the same backwards. Examples: MADAM, DID, 2002. There are even some fun to compose palindrome sentences, like He Goddam Mad Dog, eh?

Function to check if sequence is palindromic is very simple. Compare first element with the last, second with the penultimate and so on. It needs just n/2 comparisons, so complexity of that algorithm is O(n). Here is recursive solution:

(defn palindrome-recursive? [s]
  (if (empty? s) true
    (let [a (first s) b (last s)]
      (if (= a b) (recur (rest (butlast s))) false))))

Note two functions: function last - gets last element of collection, butlast - drops last element of collection. Both they work in linear time, so you might assume overall complexity is O(n^2). Using vectors we can get index access, and perform test much faster, but this detail is not important for now.

Using more cleaner way (obviously not the best) to implement this, which in particular I prefer, is the following:

(defn palindrome? [s]
  (= s (reverse s)))

Function reverse, returns given sequence in reverse order in linear time. Overall complexity is also O(n).

To test whether number is palindrome or not, we just cast it to string, then cast string to sequence of chars, and using above function:

(defn palindrome-number? [n]
  (palindrome? (seq (str n))))

Final solution is following:

(reduce max (filter palindrome-number?
                      (for [i (range 100 1000) j (range i 1000)] (* i j))))

We just generate all combinations of 3-digit numbers multiplication, filter palindromes, and find maximum. Done.

This technique (wow, is this technique?) called brute-force. It’s very consumable, because test each possible combination. But it’s common algorithm to such sort of problems. There are lot of clever brute force optimization to reduce set of possible values, but it is still brute force. As you see, in the code above, we also used small optimization. We iterate first multiplicand i from 100 to 1000 to cover all 3-digits numbers, and j iterated not from 100 but from i. This is because multiplication 375 * 468 gives the same result as 468 * 375. Just because commutative property. Roughly, this reduces number of iterations by factor of 2. Obviously, there are other optimizations, but they are not so value and reduces readability (If first number ends with 5 and second is even, then result ends with 0, and we skip this because in palindromic value 0 should be at the beginning).

Congratulations! 4 problems solved. And we know “so much” about clojure.

GitHub.

P.S. We create two separate functions for palindromes: one test sequence for palindrome, second test number for palindrome and reuse first implementation. Some can say it’s not good, because we write functionality that we not needed (function palindrome?). Particularly, I agree. We could hide palindrome? implementation inside palindrome-number? or make it private. But we haven’t did this and have function palindrome? that we can reuse in other problems. Also it shows important concept, how we can reuse existing implementations to get new functionality. And again: maybe there is (I’m sure there is) another solution which is more clever than bruteforce. But this solution works only 0.2 sec. on my machine, so this is ok.

Clojure Euler: Problem 003

2012-10-30T19:12:00+00:00

The prime factors of 13195 are 5, 7, 13 and 29.

What is the largest prime factor of the number 600851475143?

Permalink: http://projecteuler.net/problem=3

If you familiar with prime numbers, you apparently know basic algorithms to find them, if not, I advice to get familiar with it. There are lot of problems devoted to prime numbers on Project Euler. So, read 10 minutes about prime numbers, and go on.

In few words, prime number is a number starting from 2 that has no divisors except 1 and itself. Exposing this logic on all numbers, we can simply find first prime numbers: 2, 3, 5, 7, 11, ...

To implement function that search for prime numbers we can implement predicate, that test whether number is prime or not:

(defn prime? [n]
   (= 2 (reduce +
                (for [i (range 1 (inc n))]
                  (if (= 0 (mod n i)) 1 0)))))

We iterate from 1 up to number and calculate reminder of division number by iteration counter. If result is zero we return 1, otherwise we return 0. For each number we get list with length of that number that marked with 1 for exact divisors. For example for 6 list will be (1 1 1 0 0 1), and for 7 list will be (1 0 0 0 0 0 0 1). Than we calculate sum of this list, which represents number of divisors. If count is two, that means number is prime. This is simplest algorithm for testing prime number, built from prime number definition.

Performing small optimization you can see that in lists above, first and last elements are always equals to 1. We can just drop them. Another optimization is we don’t need go on when we found at least one remainder. And I’m pretty sure you can find lot of flaws in this implementation.

But the most useful optimization is assumption that we don’t need iterate to n, we can iterate to sqrt(n). This is based on fact that if we found some number a that is divisor of n, then there is another number b = n / a. And b is also divisior of n. Little transformation gives us a * b = n. Assume a < b, so if we found a we don’t need to seek for b. Number n is not prime. Maximum value of a achieved when a equals to b, a^2 = n, a = sqrt(n). This is reduce complexity of algorithm from O(n) to O(sqrt(n)).

In previous problem we mentioned clojure.contrib.lazy-seqs package, where we have primes lazy sequence. Let’s use it!

Algorithm for solving our problem is following:

Iterate through all prime number that less than square root of number
If this number is divisor add it to list
Find maximum from that list

To transform statements above into clojure, we get:

(defn greatest-prime-of [number]
  (reduce max (filter #(zero? (mod number %))
                      (take-while #(< % (sqrt number)) primes))))

Something new here:

max - function that returns the greatest number of its arguments.
zero? - predicate that test whether number is equal to zero or not. The same as #(= 0 %).
sqrt - function from clojure.contrib.math. Name is obvious.

Now, we want to find result. Run the line

(greatest-prime-of 600851475143))

Congratulations! Third problem solved.

GitHub for lazy!

P.S. Instead of using clojure.contrib.math for square root we could just write (Math/sqrt n) and got rid of dependency. But for me is better to reuse something that was implemented in library. It’s tested and worked, where your implementation can miss some point. If you have time, dig more into clojure.contrib.math. It needs about 9 minutes. There are only 9 functions, but you will probably reinvent them all the time: expt, abs, sqrt, round and others.

Clojure Euler: Problem 002

2012-10-15T18:46:00+00:00

Each new term in the Fibonacci sequence is generated by adding the previous two terms. By starting with 1 and 2, the first 10 terms will be:

1, 2, 3, 5, 8, 13, 21, 34, 55, 89, …

By considering the terms in the Fibonacci sequence whose values do not exceed four million, find the sum of the even-valued terms.

Permalink: http://projecteuler.net/problem=2

This is not hard problem with objective to make you familliar with Fibonacci numbers (this numbers will appear in a lot of next problems). It can be implemented with the straightforward manner with the following rules applied:

F(0) = 1
F(1) = 1
F(n) = F(n-1) + F(n-2)

So, we can define simple recursive function to calculate n-th number of Fibonacci sequence in clojure:

(defn fib [n]
  (if (or (= n 1) (= n 0)) 1
    (+ (fib (- n 1)) (fib (- n 2)))))

It’s probably not the best implementation of Fibonacci numbers, but it shows the idea. We can check that it works correctly by calling (map fib (range 10)) that produces sequence (1 1 2 3 5 8 13 21 34 55). Problem appears when we try to calculate 40th number. Calling that function on my machine takes ~20 seconds. Not good.

The reason of such bad performance is repeated calculations. Let’s see how expanded call (fib 40). It produces (+ (fib 39) (fib 38)), where (fib 39) expanded to (+ (fib 38) (fib 37)) and (fib 38) to (+ (fib 37) (fib 36)) etc. The growth with factor of 2. And with these two expansions we calculated 2 values that we calculated before, 37th and 38th.

We can make better if we producing our sequence in direct order, instead of reverse. This way function will be more complex than previous:

(defn fib-seq [n]
  ((fn [a b c seq]
    (cond (= 1 n) [1]
          (= c n) seq
          :else (recur b (+ a b) (inc c) (conj seq (+ a b)))))
   1 1 2 [1 1]))

Let’s see what’s new clojure features we used:

We define function inside function. Why we did this? All is because we used “helper”-function for accumulate current Fibonacci sequence. It used only in function fib-seq, so we don’t need to define it at the root level. It closed for using for everyone, except fib-seq.
Next thing we define anonymous function by keyword fn. Basically it’s the right thing to define functions. defn just shortcut for (def (fn [] )), and..
..we pass default (initialization) values for helper function 1 1 2 [1 1].
I skip logic of helper function here. It is pretty straightforward, just notice two new clojure functions that we used.
cond - similar to switch-case structure in C-style languages.
conj - add value to collection. Dependant on collection type, insert new value in different places. For our case we using vector, so we add this value to the end of vector.

Now we can call (fib-seq 40) and it calculates all 40 values very quickly. Nice.

But let’s return to problem description “… whose values do not exceed 4 million…“. How do we know how many values we need to take that do not exceed 4 million? 50? 100? 112? This is drawback of our fib-seq function and we need to rewrite it…or take a look at clojure mechanism called lazy sequences.

Lazy sequences

In few words, lazy sequence is infinite sequence of some values, that calculates due to some expression. The main idea here is the word lazy that means we evaluate expressions when they are needed. For example, sequence of natural numbers is a lazy sequence. Clojure supports lazy sequences along with operations on them. So, we can combine sequences, limit, filter etc. Java do not support lazy sequences but similar functionality can be implemented with generators concept. Generator interface provides method next() to retrieve next value in the sequence, and that method evaluates value. But this approach is very poor with comparison to clojure lazy sequences.

Lazy sequences in clojure can be treated like ordinary sequences. Obviously some methods make no sense, due to infinity of sequence (count, last, etc.) There are way to define lazy sequences by using macro lazy-seq. But most common way to do it with the function iterate. It takes function f, and initial value a, and produces lazy sequence (a, f(a), f(f(a)),...). For example lazy sequence of natural numbers we can define as following: (iterate inc 1). Then we can play with this sequence as we want.

WARNING: Never call lazy sequence without limiting functions. It tries to evaluate all, and… You know, never call.

Now, we have a little understanding what lazy sequence is, and can implement Fibonacci lazy sequence.

(defn fib-seq-lazy []
  (map first (iterate (fn [[a b]] [b (+ a b)]) [1 1])))

Ok, What’s here:

Read from the end. We apply anonymous function to vector [1 1] and produce lazy sequence with iterate.
Note double square brackets in anonymous function definition. Outer brackets indicate function variables. Inner brackets indicate that we take one parameter to function as argument. This parameter is a sequence and we map to a and b, first and second elements in that sequence. For example for first call, a bound to 1, b bound to 1.
Our iterate produces lazy sequence with following format ([1 1] [1 2] [2 3] [3 5] [5 8] ...). Our Fibonacci sequence is just first value of each pair, and we using…
map - takes function and applies it to each element of collection. This function also produces lazy sequence.
To gather all in one we get lazy sequence that represent Fibonacci numbers (1 1 2 3 5 8 ...).

Now we have to implement functionality that in problem description. Immediately, code:

(reduce +
  (filter even? (take-while #(< % 4000000) (fib-seq-lazy))))

Another one(two)-liner. Good.

take-while - takes values from sequence (includes lazy sequences) while condition is true.
#(< % 4000000) - #() it’s shorter shortcut for anonymous function. In our case predicate return true if value lower than 4 millions. Exacly what we need.
filter - return all values for which predicate return true. Instead of take-while it proceed all sequence till the end, and, obviously, do not accept lazy sequences.
even? - predicate that return true, if number is even. Note: this is common way to append function name with question mark to indicate that this function is predicate (return true or false). Just convention.
And, finally, sum all of them.

Congratulations! We just solved Project Euler Problem 2.

There are also few optimisations to get result faster, but KISS.

GitHub for lazy!

P.S. We could use clojure-contrib library, which is often deployed with standard clojure-core. clojure-contrib.lazy-seqs library contains method (fibs) that also produces lazy Fibonacci sequence. And interestingly it implemented the same way as we did. There are two more lazy sequences: prime numbers and powers of 2. It’s good practice to use existing functionality and not invent vehicle.

Have fun!

Clojure + Emacs + Leiningen + nREPL

2012-10-13T18:19:00+00:00

This article provides step-by-step instruction how to configure Clojure environment.

Little history

Long time I was using swank-clojure as connector to my Clojure projects from Emacs SLIME. But, suddenly, I lost all my emacs preferences and needed to configure it again (yeah, it’s my fault, this time I’ve created configuration on GitHub). So, when I entered to swank-clojure project page, I was a bit frustrated by the following message: “This project is no longer under active development. New users are strongly encouraged to try out nrepl.el instead”. Ok, let’s see what nrepl is.

Why emacs?

I don’t know. I like lisp, and emacs created in (e)lisp. Strange hotkeys like C-x C-s. Huge functionality, that you can configure all days. Internal package manager. Plenty of modes. Mail client. Games. Whatever. Oh wait, I forgot text editor. I really, can’t find any great reason that convince somebody to use emacs as IDE. But I like it.

There is a good xkcd comics about emacs.

By the way, there are lot of IDEs that can be configured much simpler way than described in this article:

Clojure Box - emacs install, preconfigured for clojure. Only for Windows
Counter Clockwise - eclipse plugin
Enclojure - netbeans plugin
La Clojure - IDEA plugin
VimClojure - vim plugin
Light Table - new generation of IDEs?

So, before reading, ask yourself, Do I really need emacs?

Emacs

First of all you need to install emacs, to get your future clojure coding environment. I’m using ubuntu, so I just type:

sudo apt-get install emacs

and it install emacs 23. If you using another OS, or do not trust package manager, check GNU Emacs official website for installation instructions.

Also, you need to install emacs package-manager.

Copy file package.el to your ~/emacs.d/ folder.
In your ~/.emacs file add folowing lines:

(add-to-list 'load-path "~/.emacs.d/")
(load "package")
(require 'package)
(add-to-list 'package-archives
    '("marmalade" .
      "http://marmalade-repo.org/packages/"))
(package-initialize)

These lines will be loaded after emacs restarted and install package manager. To check if it works, type M-x package-list-packages and this show you list of emacs plugins, hope you will find something useful there.

Leiningen

Leiningen is a build tool for clojure. It also controls dependency management, automation, tests, etc. Pretty like maven for Java.

To install lein 2 (it’s shorter name for Leiningen) do the following:

Download lein script
Place it on your $PATH
Make it executable

On ubuntu this can be performed with the following commands:

cd ~/
mkdir bin

Add line export PATH=~/bin:$PATH to your ~/.bashrc file

cd ~/bin
wget https://raw.github.com/technomancy/leiningen/preview/bin/lein
chmod +x lein

Execute lein, for first time it will download default lein package. To verify installation run lein version. On my machine for the post date it prints: Leiningen 2.0.0-preview10 on Java 1.6.0_24 OpenJDK 64-Bit Server VM

nREPL

We have already installed emacs package manager, so we can get nrepl from there. It also needs emacs major mode clojure-mode, so type the following commands in emacs to install needed packages:

M-x package-install clojure-mode
M-x package-install nrepl

Try M-x nrepl-jack-in to execute REPL, where you can play with clojure commands.

Bind all together

We have fully configured environment for clojure and how to use it?

Create new clojure project using lein:

lein new helloworld

Start nREPL server in emacs:

M-x nrepl-jack-in

Open core project file in emacs:

C-f helloworld/src/helloworld/core.clj

Write simple function:

(ns helloworld.core)
(defn hello []
  (println "Hell, O'World!"))

Compile it:

C-c C-k

Switch to nREPL buffer, change namespace and run hello function:

(ns helloworld.core)
(hello)

Probably, you’ll get output:

Hell, O'World!
nil

Congratulations! You just created working clojure project.

Note: this configuration covers only basic clojure+emacs environment. There are plenty plugins to work with parenthesis, documentation, tests, auto-completion and lot more. I’ll do my best to cover such plugins in the next sections. Also, emacs23 is an old version of emacs, verison 24 is currently available.

Clojure Euler: Problem 001

2012-10-12T17:38:00+00:00

If we list all the natural numbers below 10 that are multiples of 3 or 5, we get 3, 5, 6 and 9. The sum of these multiples is 23.

Find the sum of all the multiples of 3 or 5 below 1000.

Permalink: http://projecteuler.net/problem=1

The simplest problem and nothing hard to solve it in standard iterative way (C-style pseudocode):

int s = 0;
for (int i=0; i < 1000; i++) {
  if ((i % 3 == 0) || (i % 5 == 0)) s+=i;
}

We get result immediately. But loop that changes value of some variable it’s not a functional approach. Let’s rewrite this solution in recursive way in clojure:

(loop [i 0 s 0]
  (if (= i 1000) s
    (if (or (= 0 (mod i 3))
            (= 0 (mod i 5))) (recur (inc i) (+ s i))
                             (recur (inc i) s))))

Just few words what we are doing here:

loop - create binding between names and values in scope of loop body. For now, think about loop like a function definition with two variables i and s along with calling such function with arguments i=0, s=0. i is a loop counter, s is an accumulator variable that holds current sum.
mod - calculates remainder of division two numbers. inc - increment value, similar to (+ i 1).
Recursion base case: if we have i value equals to 1000, then we finished our work and return current accumulator value s. Otherwise,
Recursion step: if current i value is multiply of 3 or multiply of 5, we add counter value to accumulator, increment loop counter and call this function with these new arguments. Otherwise, we just increment value and call function. This way we recursively go back to the first line of function with changed values until we get recursion base condition as true and return accumulator. Recursion call performed by recur.

This can be a bit harder to understand than iterative style, but we have a couple of pros there.

There are no variables. There are values and names bound to that values. That means that we can’t change one value to another inside loop. That’s why we recursively call this function with changed values. Such trick helps to avoid hard-to-spot error-prone situations where iteration counter changed inside the loop.
This is more flexible way than iteration. We can pass as much arguments as we want, move forward-backward, select dynamic iterative step and even select any stop condition.
Safety. We are completely safe that our loop is pure local. It can not change anything outside. All garbage he left was cleared after returning value (Ecologists are very happy!)

Congratulations! Problem solved, but let’s find more elegant way.

We can split our problem into two simpler problems:

Find sum of all multiples of 3 below 1000
Find sum of all multiples of 5 below 1000
Add these two sums

Stop, stop! What about numbers that multiples of 3 and multiples of 5, like 15, 30, 45,...? We add them twice. Good catch. Let’s just subtract them once from total sum:

Find sum of all multiples of 3 below 1000
Find sum of all multiples of 5 below 1000
Find sum of all multiples of 15 below 1000
Add sum 1 to sum 2 and subtract sum 3.

You see the repeated “Find sum of all…“. It’s clear definition of function. Let’s implement it.

(defn sum-of [d] (reduce + (range 0 1000 d)))

One line! Excited? Brief description, what we wrote:

defn indicates function definition. We called function sum-of, it accepts one argument d (multiple of what?). As you, probably notice, we do not enforced to specify type of function argument, as well as function returning type. All it’s because clojure is dynamic language. Dynamic languages have pros and cons over static, but we omit their differences. For now.
reduce takes two parameters: function of two arguments and collection, and applies that function to first two values, than to result and next value, again, again, until the end of collection. So, for example (reduce * [2 3 4 5]) calculates (* 2 3) that is 6, calls (reduce * [6 4 5]), then calls (reduce * [24 5]) and finally returns 120.
range function generates list. As you see it takes three parameters, but there are overloaded versions that takes 2, 1 or even 0 values. (range 0 1000 d) start generation from 0, next value will be previous plus d, until we exceed 1000. Exclusively. So, (range 0 10 3) returns (0 3 6 9).

Ok. We have function “Find sum of all…” and almost all work is done. Just call this function with proper arguments and you’ll get the result:

(+ (sum-of 3) (sum-of 5) (- (sum-of 15)))

Done. In functional, clear, concise way in two lines of code.

Note: our algorithm complexity is O(n). If we remember school course of math, we realize that all our sequences are arithmetic progressions. That means for caluclating sum of sequence we don’t need count sum in usual way, we just can use formula:

n is a count of numbers in a sequence, a1 - first value, an - last value. Now we can reimplement our sum-of function next way:

(defn sum-of [d]
  (let [a1 d
        an (- 999 (mod 999 d))
        n (quot 999 d)]
    (/ (* cnt (+ a1 an)) 2)))

More harder to understand, but doable. New language syntax appears (999 used as we have “below 1000” in problem description):

let - creates a lexical context for expression. In fact, it’s just separate block of code, with names, that exist only in scope of that block. After let we evaluates some expression and gives some values to the result.
a1 - first element of sequence, just step value d.
an - last element of sequence. We don’t know last element, so we need to calculate it. Little trick: find the remainder of division 999 by d, and subtract it from 999.
n - count of numbers in a sequence. We also don’t know this value. It just will be integer number of division 999 by d. Exactly what quot function does.
And finally, the expression - formula for sum of arithmetic progression.

Actually, we reduced readability of our code, but improved algorithmic complexity from O(n) to O(1).

GitHub for lazy.

Try clojure online!

P.S. Programming not tied to some problem is nothing. That’s why, in first problem I prefer O(n) solution. It still solves problem in appropriate time and have more readable code. If n grow, I’ll think about switching to O(1) strategy, but not for now.

Have fun!

Clojure Euler

2012-10-11T16:51:00+00:00

Finally decided to post here my solutions to Project Euler implemented in Clojure.

As all in my life, frequency of posts with solutions will be mood-dependent.

The main reason for this series to provide a little introduction to clojure programming language during the solving interesting problems.

I’ll also try do my best to explain some general terms, why this approach better or worse than others, suggest alternative solutions or thoughts on problem and provide code. You allowed to copy that code, calculate the result and past answer directly in Project Euler site, but I don’t see any sense to do this. Problem solving process gives more fun, than result. Approximately 2.718282 times more.

Java Magic. Part 1: java.net.URL

2012-10-11T16:30:00+00:00

Recently, I found on reddit very interesting Java code snippet (slightly modified):

HashSet set = new HashSet();
set.add(new URL("http://google.com"));
set.contains(new URL("http://google.com"));
Thread.sleep(60000);
set.contains(new URL("http://google.com"));

What do you think the output for lines 3 and 5 will be?

Definitely not true, true if the question was asked. Think for two minutes.

Ok. In most cases it will be true, false because you have internet connection (How else you can read this?). Turn off your network cable or wi-fi, and you’ll get true, true.

The reason is in implementation of hashCode() and equals() methods for URL class.

Let’s see how hashCode calculated:

public synchronized int hashCode() {
  if (hashCode != -1)
    return hashCode;
  hashCode = handler.hashCode(this);
  return hashCode;
}

We can see hashCode is an instance variable that calculates once. Makes sense, URL is immutable. What is handler? It’s an instance of one of URLStreamHandler subclasses, depends on protocol type (file, http, ftp), that have helper hashCode implementation. Just look at URL.hashCode() javadoc:

The hash code is based upon all the URL components relevant for URL comparison. As such, this operation is a blocking operation.

Stop! BLOCKING OPERATION?!

- Sorry, I couldn’t check email yesterday due to hashCode calculation.

or even better

- No, mom, I can’t watch porn, It’s hashCode, you know.

Ok, let it be blocking. Another exciting part, that handler resolves host IP address for hashCode calculation. Tries to resolve, to be honest. If it can not do this, it calculates hashCode based on host, which is google.com for our example. Shit happens when IP is dynamic, or host have request balancer that also changes host IP. In that case we got different hashCodes for one host name, and will have two (or even more) instances in HashSet. Not good at all. By the way, hashCode and equals performance is terrible because of URLStreamHandler opens URLConnection. But it’s another topic.

How to avoid this?

Use java.net.URI instead of java.net.URL. It’s not the best choice though, but have deterministic hashCode implementation.
Do not use java.net.URL in collections. Good option to maintain collection of String objects (that represent host name) and get URL when needed.
Disable your network adapter during hashCode calculation. (It’s a joke, but it helps)
Use your own subclass of URLStreamHandler with proper implementation of hashCode.

Finally, I’m pretty sure java.net.URL class has lot of useful applications. But not that way.

Hello, world!

2012-10-10T16:10:00+00:00

This is my first post. It is always hard to begin something, but I’ll try.

Why?

I had blog earlier on LiveJournal, it was basically in Ukrainian, my native language. But more time I was working on something, less time I had to write something. Honestly, it even was not programming specific blog, just random thoughts, xposts, pictures, etc. You know this shit. Also, it was very hard to maintain some posts, especially about programming, and I gave up. Excellent proof is my failed attempt to share and explain Clojure solutions on Project Euler. Everything can be a reason: lack of readers, lack of feedback, not interesting topic, very specific topic, bad writing style, terrible livejournal layout and so on. Probably, these are all my excuses or maybe I just lazy.

Now I’ve got something in my mind and have passion to share it. No matter if somebody like it or not. I really believe that if you share something for this world, sooner or later somebody point you on your mistakes. And it’s a win. You got knowledge that didn’t know earlier and can share it to someone else. Simple loop.

What?

I plan to be specific in this blog, so a big part will be about programming, especially Java and Clojure. I also love puzzles and stupid self-made pictures (O_o) and they have reserved place in my blog. Time to time can appear posts with reference to some clever people, if they have clever thoughts, of course. Event reports, some critic, books, movies, music, games, humour. Only God knows what else can appear in my mind and therefore in this blog, but I’ll try to be specific as much as possible. Really.

English?

As you already understand I’m not native English speaker or writer or whatever. How many mistakes did you count?

I need some effort even for writing simple sentence in English (Kiev is the capital of Ukraine - not this one). And no matter how long I check my grammar or spelling I will have some mistakes. Big deal. Improve English is one of my goals, so it’s ok. adding english in todo list

I also don’t exclude option, that some posts will be in Ukrainian or even in Russian.

What else?

I spent half an hour writing this first post and believe that in near future I have more structured thoughts than this bag of words above. Good luck and welcome to my blog!

mishadoff thoughts

Architecture Components

Architecture Quality Challenge

Architecture Component (box)

Dependency (arrow)

Transitive Dependency

Circular Dependency

Case Study: Website “Go Bananas”

Component diagram (boxes and arrows)

Maintainability Use Cases

Case A: Change in ecommerce

Case B: Change in products

Case C: Change in orders

Case D: Change in payments

“Hot” Components

Metrics for “Go Bananas”

Observations

Why This Matters

Mnemonics-based keymap for IntelliJ IDEA

Rationale

Preparation

Basic hotkeys (C- prefix)

C-a - move cursor to the start of line

C-b - move cursor one word backward

C-c - prefix, reserved for Code mode

C-d - delete character (forward)

C-e - move cursor to the end of line

C-f - move cursor one word forward

C-g - IDEA Escape

C-h - delete character (backward, a.k.a Backspace)

C-i/C-j/C-k/C-l - arrows up/left/down/right

C-m/C-n - enter or return

C-r - replace in current file

C-s - search in current file

C-t - prefix for Tool Mode

C-w - cut

C-x - prefix for Execute mode

C-y - paste

C-z - undo

C-= - extend selection

C-- - shrink selection

C-] / C-[ - escape

C-. - completion

C-SPACE - start/end sticky selection

Alternative Keys (C-M-)

C-M-a - move cursor to the start of the file

C-M-b - move cursor one function backward

C-M-d - delete line from cursor to the end of line

C-M-e - move cursor to the end of the file

C-M-f - move cursor one function forward

C-M-h - delete line from the cursor to the start of line

C-M-i/C-M-j/C-M-k/C-M-l - text movement

C-M-m/C-M-n - alternative enter

C-M-r - replace globally

C-M-s - search globally

C-M-z - redo

C-M-[/C-M-] - escape from tool window and close it

Code Mode (C-c prefix)

C-c a - git annotate

C-c b - move cursor to matching brace

C-c c - comment

C-c d - duplicate line

C-c e - jump to next error

C-c C-e - error description

C-c f - find current file in project view

C-c h - inline help

C-c i - indent code

C-c j - ace jump

C-c k - kill whole line

C-c l - go to line

C-c m - list methods

C-c C-m/C-c C-n - IDEA show intentions

C-c r - refactor menu

C-c s - go to source

C-c t - transpose forms

C-c u - find usages

C-c w - wrap selection

C-c - - collapse

C-c = - expand

C-c C-- - collapse all

Case A: Change in `ecommerce`

Case B: Change in `products`

Case C: Change in `orders`

Case D: Change in `payments`

`C-a` - move cursor to the start of line

`C-b` - move cursor one word backward

`C-c` - prefix, reserved for Code mode

`C-d` - delete character (forward)

`C-e` - move cursor to the end of line

`C-f` - move cursor one word forward

`C-g` - IDEA Escape

`C-h` - delete character (backward, a.k.a Backspace)

`C-i`/`C-j`/`C-k`/`C-l` - arrows up/left/down/right

`C-m`/`C-n` - enter or return

`C-r` - replace in current file

`C-s` - search in current file

`C-t` - prefix for Tool Mode

`C-w` - cut

`C-x` - prefix for Execute mode

`C-y` - paste

`C-z` - undo

`C-=` - extend selection

`C--` - shrink selection

`C-] / C-[` - escape

`C-.` - completion

`C-SPACE` - start/end sticky selection

`C-M-a` - move cursor to the start of the file

`C-M-b` - move cursor one function backward

`C-M-d` - delete line from cursor to the end of line

`C-M-e` - move cursor to the end of the file

`C-M-f` - move cursor one function forward

`C-M-h` - delete line from the cursor to the start of line

`C-M-i`/`C-M-j`/`C-M-k`/`C-M-l` - text movement

`C-M-m`/`C-M-n` - alternative enter

`C-M-r` - replace globally

`C-M-s` - search globally

`C-M-z` - redo

`C-M-[`/`C-M-]` - escape from tool window and close it

`C-c a` - git annotate

`C-c b` - move cursor to matching brace

`C-c c` - comment

`C-c d` - duplicate line

`C-c e` - jump to next error

`C-c C-e` - error description

`C-c f` - find current file in project view

`C-c h` - inline help

`C-c i` - indent code

`C-c j` - ace jump

`C-c k` - kill whole line

`C-c l` - go to line

`C-c m` - list methods

`C-c C-m`/`C-c C-n` - IDEA show intentions

`C-c r` - refactor menu

`C-c s` - go to source

`C-c t` - transpose forms

`C-c u` - find usages

`C-c w` - wrap selection

`C-c -` - collapse

`C-c =` - expand

`C-c C--` - collapse all

`C-c C-=` - expand all

`C-x b` - set bookmark

`C-x C-b` - show bookmarks

`C-x d` - show debug menu

`C-x e` - evaluate form in REPL

`C-x f` - go to file

`C-x g` - show git dialog

`C-x C-h` - Clear REPL output

`C-x i` - interrupt

`C-x k` - kill window

`C-x l` - load file in REPL

`C-x m` - start/stop macro recording

`C-x n` - set current file to namespace in REPL

`C-x o` - go to next split window

`C-x C-o` - Open file…

`C-x q` - stop REPL

`C-x s` - search REPL history

`C-x t` - Run test under cursor

`C-x C-t` - Run all tests in current namespace

`C-x C-j` - navigate backward