Alex Castaño

Why and how a GenServer stops?

2020-08-23T00:00:00+02:00

The last day, I was creating a GenServer and I realized that I didn’t know the proper way to stop it. I had a vague idea about the process, but I needed fine-grained details. This post is the result of my little research about it.

I think this is a very important matter, however, I sometimes see a lot of people that don’t pay enough attention to it. If you want to create a resilient Elixir application, the startup and shutdown process must be properly implemented. The application will behave much better even in the worst situations.

Thanks to the Erlang supervision tree, if you have a bug or something unexpected happens, we are giving a chance to the application to heal itself. An extraordinary feature in my opinion.

It is also important because we are living in the Kubernetes era, where servers are continuously created and destroyed. For this reason, the shutdown process must be under control at all levels. Today we focus on how a GenServer stops.

The terminate/2 callback

Before we look at the reasons why GenServer stops, let’s see the terminate/2 callback. It is not a mandatory one, we can implement only when we need it, but it can help us if we want to do something before stopping. Besides, this is our last chance. After the execution of this function, we will lose the state of the GenServer forever.

It receives two arguments, the first one is the reason for shutting down, and the second is the last state of the process. We will see more about them in the following examples.

How a GenServer can stop itself

We have two contexts from which to stop GenServer, from the GenServer process itself, and from any other process. Let’s start with the first one.

A bug in the code

I don’t know about you, but the most common way I see a GenServer stop is a bug in my code. This is how it behaves when this happens:

defmodule Buggy do
  use GenServer

  @impl true
  def init(state) do
    {:ok, state}
  end

  @impl true
  def handle_info({:div, num}, state) do
    state = state + 1
    IO.inspect(":div received, updating state to: #{state}")
    IO.inspect("Divission = 100 / #{num} = #{100 / num}")
    {:noreply, state}
  end

  @impl true
  def terminate(reason, state) do
    IO.inspect("terminate/2 callback")
    IO.inspect({:reason, reason})
    IO.inspect({:state, state})
  end
end

iex> {:ok, pid} = GenServer.start(Buggy, 0)
{:ok, #PID<0.139.0>}

iex> send(pid, {:div, 10})
{:div, 10}
":div received, updating state to: 1"
"Divission = 100 / 10 = 10.0"

iex> send(pid, {:div, 0})
{:div, 0}
":div received, updating state to: 2"
"terminate/2 callback"
{:reason,
 {:badarith,
  [
    {Buggy, :handle_info, 2, [file: 'buggy.ex', line: 13]},
    {:gen_server, :try_dispatch, 4, [file: 'gen_server.erl', line: 680]},
    {:gen_server, :handle_msg, 6, [file: 'gen_server.erl', line: 756]},
    {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 226]}
  ]}}
{:state, 1}

## Error log
"""
18:00:28.809 [error] GenServer #PID<0.139.0> terminating
** (ArithmeticError) bad argument in arithmetic expression
    buggy.ex:13: Buggy.handle_info/2
    (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
    (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
    (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Last message: {:div, 0}
State: 1
"""

It is interesting to see that the state received by the terminate/2 callback is the last one before the last handle_call/2 callback was called. We didn’t have the chance to return the updated state, so this is the best the GenServer behavior can do. We will see this pattern more times, so keep it in mind when developing the terminate/2 callback.

We can also see that we received a lot of details about the crash in the reason parameter. It can be interesting to report an error if it stops for an unexpected reason.

Raise an exception

Something very similar will happen if we manually raise an exception:

defmodule Raising do
  use GenServer

  @impl true
  def init(state) do
    {:ok, state}
  end

  @impl true
  def handle_info(:raising, state) do
    state = state + 1
    IO.inspect(":raising received, updating state to: #{state}")
    raise "Raising!"
    # Never return
    # {:noreply, state}
  end

  @impl true
  def terminate(reason, state) do
    IO.inspect("terminate/2 callback")
    IO.inspect({:reason, reason})
    IO.inspect({:state, state})
  end
end

iex> {:ok, pid} = GenServer.start(Raising, 0)
{:ok, #PID<0.150.0>}

iex> send(pid, :raising)
":raising received, updating state to: 1"
:raising
"terminate/2 callback"
{:reason,
 { %RuntimeError{message: "Raising!"},
  [
    {Raising, :handle_info, 2, [file: 'raising.ex', line: 13]},
    {:gen_server, :try_dispatch, 4, [file: 'gen_server.erl', line: 680]},
    {:gen_server, :handle_msg, 6, [file: 'gen_server.erl', line: 756]},
    {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 226]}
  ]}}
{:state, 0}
## Error log
"""
18:08:25.685 [error] GenServer #PID<0.150.0> terminating
** (RuntimeError) Raising!
    raising.ex:13: Raising.handle_info/2
    (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
    (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
    (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Last message: :raising
State: 0
"""

Return :stop in a callback

However, the most appropriate way to stop it, in most cases, is to return the tuple {:stop, reason, state} in almost any callback:

defmodule Stop do
  use GenServer

  @impl true
  def init(state) do
    {:ok, state}
  end

  @impl true
  def handle_info({:quit, reason}, state) do
    state = state + 1
    IO.inspect(":quit received, updating state to: #{state}")
    {:stop, reason, state}
  end

  @impl true
  def terminate(reason, state) do
    IO.inspect("terminate/2 callback")
    IO.inspect({:reason, reason})
    IO.inspect({:state, state})
  end
end

iex> {:ok, pid} = GenServer.start(Stop, 0)
{:ok, #PID<0.160.0>}

iex> send(pid, {:quit, :normal})
{:quit, :normal}
":quit received, updating state to: 1"
"terminate/2 callback"
{:reason, :normal}
{:state, 1}

As we can see, the state can be updated this time before calling the callback.

Kernel.exit/1

It has very similar behaviour to the previous one. The function sends an exit signal to the current process, in our case, to the GenServer:

defmodule Exit do
  use GenServer

  @impl true
  def init(state) do
    {:ok, state}
  end

  @impl true
  def handle_info({:exit, reason}, _state) do
    IO.inspect(":exit received")
    exit(reason)
  end

  @impl true
  def terminate(reason, state) do
    IO.inspect("terminate/2 callback")
    IO.inspect({:reason, reason})
    IO.inspect({:state, state})
  end
end

iex> {:ok, pid} = GenServer.start(Exit, 0)
{:ok, #PID<0.192.0>}

iex> send(pid, {:exit, :normal})
{:exit, :normal}
":exit received"
"terminate/2 callback"
{:reason, :normal}
{:state, 0}

Important note about the “reason”

In our two last example we could manipulate the “reason” to stop a GenServer. This reason has an extra meaning for the OTP. There are three values considered “normal” reason values:

:normal
:shutdown
{:shutdown, term}

What happens when the reason is different from these three values? From the documentation:

Exiting with any other reason is considered abnormal and treated like a crash. This means the default supervisor behaviour kicks in, error reports are emitted, and so forth.

For example, we can see that an error log is emitted with custom exit signal:

iex> {:ok, pid} = GenServer.start(Exit, 0)
{:ok, #PID<0.204.0>}

iex> send(pid, {:exit, :my_own_reason})
{:exit, :my_own_reason}
":exit received"
"terminate/2 callback"
{:reason, :my_own_reason}
{:state, 0}
## Error log
"""
18:34:29.527 [error] GenServer #PID<0.204.0> terminating
** (stop) :my_own_reason
    exit.ex:12: Exit.handle_info/2
    (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
    (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
    (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Last message: {:exit, :my_own_reason}
State: 0
"""

Similar when we use {:stop, :my_own_reason, state}:

iex> {:ok, pid} = GenServer.start(Stop, 0)
{:ok, #PID<0.196.0>}

iex> send(pid, {:quit, :my_own_reason})
{:quit, :my_own_reason}
":quit received, updating state to: 1"
"terminate/2 callback"
{:reason, :my_own_reason}
{:state, 1}
## Error log
"""
18:31:02.909 [error] GenServer #PID<0.196.0> terminating
** (stop) :my_own_reason
Last message: {:quit, :my_own_reason}
State: 1
"""

We can use the {:shutdown, term} to add a custom value to the reason without considering it a crash:

iex> {:ok, pid} = GenServer.start(Stop, 0)
{:ok, #PID<0.201.0>}

iex> send(pid, {:quit, {:shutdown, :my_own_reason}})
{:quit, {:shutdown, :my_own_reason}}
":quit received, updating state to: 1"
"terminate/2 callback"
{:reason, {:shutdown, :my_own_reason}}
{:state, 1}
## No error log

How a GenServer can be stopped externally

We’ve already seen how a GenServer can stop itself. Now, we will see how other processes can do the same thing.

GenServer.stop/3

This function receives the PID—or any value that can be considered a GenServer name—the reason to stop it and a timeout. As you can imagine due to the timeout parameter, this function is synchronous, it won’t continue until ensuring that the GenServer completely stops:

defmodule GenServerStop do
  use GenServer

  @impl true
  def init(state) do
    {:ok, state}
  end

  @impl true
  def terminate(reason, state) do
    IO.inspect("terminate/2 callback")
    IO.inspect({:reason, reason})
    IO.inspect({:state, state})
    IO.inspect("sleep before terminating")
    Process.sleep(1_000)
    IO.inspect("really terminating")
  end
end

iex> {:ok, pid} = GenServer.start(GenServerStop, 0)
{:ok, #PID<0.127.0>}
iex> GenServer.stop(pid)
"terminate/2 callback"
{:reason, :normal}
{:state, 0}
"sleep before terminating"
"really terminating"
:ok

The default value for reason is :normal and for timeout is 5_000. Again, if the reason is a custom one, it will be logged:

iex> {:ok, pid} = GenServer.start(GenServerStop, 0)
{:ok, #PID<0.131.0>}
iex> GenServer.stop(pid, :abnormal)
"terminate/2 callback"
{:reason, :abnormal}
{:state, 0}
"sleep before terminating"
"really terminating"

# Error log
"""
09:35:49.180 [error] GenServer #PID<0.131.0> terminating
** (stop) :abnormal
Last message: []
State: 0
"""
:ok

If we don’t give enough time to terminate, the GenServer.stop/3 function won’t return, it will crash before returning. However, the terminate/2 will continue finishing:

iex> {:ok, pid} = GenServer.start(GenServerStop, 0)
{:ok, #PID<0.135.0>}
iex> GenServer.stop(pid, :small_timeout, 100)
"terminate/2 callback"
{:reason, :small_timeout}
{:state, 0}
"sleep before terminating"
# Our current iex process exit because timeout, Error log:
"""
** (exit) exited in: GenServer.stop(#PID<0.135.0>, :small_timeout, 100)
    ** (EXIT) time out
    (elixir 1.10.3) lib/gen_server.ex:981: GenServer.stop/3
"""
# The GenServer process finishes anyway some milliseconds later
"really terminating"
# Error log because is not a normal reason:
"""
09:38:39.020 [error] GenServer #PID<0.135.0> terminating
** (stop) :small_timeout
Last message: []
State: 0
"""

Link to another process

When two processes are linked, if one dies the other dies too. But how exactly does this process work? Let’s start with a simple example:

defmodule Link do
  use GenServer

  @impl true
  def init(state) do
    IO.inspect({:init, self()})
    {:ok, state}
  end

  @impl true
  def handle_info({:link, link}, state) do
    Process.link(link)
    {:noreply, state}
  end

  @impl true
  def terminate(reason, state) do
    IO.inspect("terminate/2 callback in pid #{inspect(self())}")
    IO.inspect({:reason, reason})
    IO.inspect({:state, state})
  end
end

iex> {:ok, pid_1} = GenServer.start(Link, 0)
{:init, #PID<0.120.0>}
{:ok, #PID<0.120.0>}
iex> {:ok, pid_2} = GenServer.start(Link, 0)
{:init, #PID<0.122.0>}
{:ok, #PID<0.122.0>}
iex> send(pid_1, {:link, pid_2})
{:link, #PID<0.122.0>}
iex> GenServer.stop(pid_2, :my_own_reason)
terminate/2 callback in pid #PID<0.122.0>
{:reason, :my_own_reason}
{:state, 0}

# Error log from pid 2:
"""
18:57:10.499 [error] GenServer #PID<0.122.0> terminating
** (stop) :my_own_reason
Last message: []
State: 0
"""
:ok
iex> Process.alive?(pid_1)
false

We stopped the pid_2 and we see it executes the terminate/2 callback. The same is not true for the pid_1. The GenServer running with the pid_1 died because it was linked to pid_2. Interestingly, the terminate/2 callback has not been called at all for the pid_1.

Trapping exit signals

One way we can modify this default behaviour is trapping exit signals. When a process died, the linked processes receives a exit signal. There are two possibilities then:

The process doesn’t trap the exit signal, so it dies with the same reason.
The process traps the exit signal, so it is converted in a regular message.

An example of the last case would be:

defmodule TrapLink do
  use GenServer

  @impl true
  def init(state) do
    # Trap exits
    Process.flag(:trap_exit, true)
    {:ok, state}
  end

  @impl true
  def handle_info({:link, link}, state) do
    Process.link(link)
    {:noreply, state}
  end

  def handle_info(msg, state) do
    IO.inspect({:handle_info, msg, state})
    {:noreply, state}
  end

  @impl true
  def terminate(reason, state) do
    IO.inspect("terminate/2 callback in pid #{inspect(self())}")
    IO.inspect({:reason, reason})
    IO.inspect({:state, state})
  end
end

iex> {:ok, pid_1} = GenServer.start(TrapLink, 0)
{:ok, #PID<0.187.0>}
iex> {:ok, pid_2} = GenServer.start(TrapLink, 0)
{:ok, #PID<0.189.0>}
iex> send(pid_1, {:link, pid_2})
{:link, #PID<0.189.0>}
iex> GenServer.stop(pid_2, :normal)
"terminate/2 callback in pid #PID<0.189.0>"
{:reason, :normal}
{:state, 0}
# This message is received by the pid_1
{:handle_info, {:EXIT, #PID<0.189.0>, :normal}, 0}
:ok
iex> Process.alive?(pid_1)
true

So the GenServer running in the pid_1 survived this time. Because it didn’t die so the terminate/2 callback it wasn’t called either. Instead, it received the {:EXIT, #PID<0.189.0>, :normal} message in the handle_info/2 callback. The same behaviour happens with not “normal” reasons:

iex> {:ok, pid_1} = GenServer.start(TrapLink, 0)
{:ok, #PID<0.198.0>}
iex> {:ok, pid_2} = GenServer.start(TrapLink, 0)
{:ok, #PID<0.200.0>}
iex> send(pid_1, {:link, pid_2})
{:link, #PID<0.200.0>}
iex> GenServer.stop(pid_2, :abnormal)
"terminate/2 callback in pid #PID<0.200.0>"
{:reason, :abnormal}
{:state, 0}

# Error log from pid_2:
"""
19:28:14.982 [error] GenServer #PID<0.200.0> terminating
** (stop) :abnormal
Last message: []
State: 0
"""
# This message is received by the pid_1
{:handle_info, {:EXIT, #PID<0.200.0>, :abnormal}, 0}
:ok

Of course, in the handle_info/2 callback, we could have decided to stop the GenServer using one of the different methods we previously saw. The main point, however, it’s that the GenServer doesn’t die directly if it traps exit signals when a linked process dies. We should do this manually.

Process.exit/2

We learned about the exit signals. Any process can send an exit signal to any other process. To do it we can use the Process.exit/2 function. As in the previous example, the consequences of calling this function will depend on whether or not the signals are trapped:

defmodule ExternalExit do
  use GenServer

  @impl true
  def init(state) do
    {:ok, state}
  end

  @impl true
  def handle_info(:trap, state) do
    Process.flag(:trap_exit, true)
    {:noreply, state}
  end

  def handle_info(msg, state) do
    IO.inspect({:info, msg, state})
    {:noreply, state}
  end

  @impl true
  def terminate(reason, state) do
    IO.inspect("terminate/2 callback")
    IO.inspect({:reason, reason})
    IO.inspect({:state, state})
  end
end

iex> {:ok, pid} = GenServer.start(ExternalExit, 0)
{:ok, #PID<0.156.0>}
iex> Process.exit(pid, :abnormal)
true
iex> Process.alive?(pid)
false
iex> {:ok, pid} = GenServer.start(ExternalExit, 0)
{:ok, #PID<0.160.0>}
iex> Process.exit(pid, :shutdown)
true
iex> Process.alive?(pid)
false

And again, the GenServer doesn’t execute the terminate/2 callback with an exit signal. It dies as soon as the signal is received.

On the other side, when the signals are trapped, it doesn’t die unless we manually want it:

iex> {:ok, pid} = GenServer.start(ExternalExit, 0)
{:ok, #PID<0.164.0>}
iex> send(pid, :trap)
:trap
iex> Process.exit(pid, :abnormal)
{:info, {:EXIT, #PID<0.111.0>, :abnormal}, 0}
true
iex> Process.alive?(pid)
true

When using the function Process.exit/2, there are two exceptions to the previous behaviour. First, when we use :normal reason. For this case, the documentation says:

If reason is the atom :normal, pid will not exit (unless pid is the calling process, in which case it will exit with the reason :normal). If it is trapping exits, the exit signal is transformed into a message {:EXIT, from, :normal} and delivered to its message queue.

iex> {:ok, pid} = GenServer.start(ExternalExit, 0)
{:ok, #PID<0.169.0>}
iex> Process.exit(pid, :normal)
true
iex> Process.alive?(pid)
true
iex> send(pid, :trap)
:trap
iex> Process.exit(pid, :normal)
{:info, {:EXIT, #PID<0.111.0>, :normal}, 0}
true
iex> Process.alive?(pid)
true

We can see the process didn’t die, even when it wasn’t trapping the exit signals!

The other exception is when we use with the :kill reason:

If reason is the atom :kill, that is if Process.exit(pid, :kill) is called, an untrappable exit signal is sent to pid which will unconditionally exit with reason :killed.

iex> {:ok, pid} = GenServer.start(ExternalExit, 0)
{:ok, #PID<0.185.0>}
iex> send(pid, :trap)
:trap
iex> Process.exit(pid, :kill)
true
iex> Process.alive?(pid)
false

We couldn’t trap the :kill signal, we sent the message: “You must die now!” :)

Supervisor stops it

In a regular application, a GenServer is usually supervised by a Supervisor. The Supervisor is in charge of stopping its GenServer children in the shutdown process. When the Supervisor wants to stop a child, it sends an exit signal too, in a very similar way to the previous example did.

We will use Supervisor.stop/3, analogous to GenServer.stop/3 but for Supervisors:

defmodule ParentStop do
  use GenServer

  def start_link(_) do
    GenServer.start(__MODULE__, 0)
  end

  @impl true
  def init(state) do
    {:ok, state}
  end

  @impl true
  def handle_info(:trap, state) do
    Process.flag(:trap_exit, true)
    {:noreply, state}
  end

  def handle_info(msg, state) do
    IO.inspect({:info, msg, state})
    {:noreply, state}
  end

  @impl true
  def terminate(reason, state) do
    IO.inspect("terminate/2 callback")
    IO.inspect({:reason, reason})
    IO.inspect({:state, state})
  end
end

iex> {:ok, pid} = Supervisor.start_link([ParentStop], strategy: :one_for_one)
{:ok, #PID<0.120.0>}
iex> Supervisor.stop(pid, :normal, 5_000)
:ok

In this case, the terminate/2 callback is not called. Let’s try with another reason:

iex> {:ok, pid} = Supervisor.start_link([ParentStop], strategy: :one_for_one)
{:ok, #PID<0.131.0>}
iex> Supervisor.stop(pid, :abnormal, 5_000)

# Error log from the GenServer:
"""
19:57:36.026 [error] GenServer #PID<0.131.0> terminating
** (stop) :abnormal
Last message: []
State: {:state, {#PID<0.131.0>, Supervisor.Default}, :one_for_one, {[ParentStop],
%{ParentStop => {:child, #PID<0.132.0>, ParentStop,
{ParentStop, :start_link, [[]]}, :permanent, 5000, :worker, [ParentStop]}}},
:undefined, 3, 5, [], 0, Supervisor.Default, {:ok, { %{intensity: 3, period: 5, strategy: :one_for_one},
[%{id: ParentStop, start: {ParentStop, :start_link, [[]]}}]}}}
"""

# The iex process dies too because it was linked to the Supervisor which dies with :abnormal reason:
"""
** (EXIT from #PID<0.129.0>) shell process exited with reason: :abnormal
"""

Same behaviour! Definitely, when an exit signal is received on a GenServer, it never runs terminate/2.

In the next example we will trap exit signals to change the behaviour:

iex> {:ok, pid} = Supervisor.start_link([TrapParentStop], strategy: :one_for_one)
{:ok, #PID<0.149.0>}
iex> Supervisor.stop(pid, :normal, 5_000)
{:info, {:EXIT, #PID<0.149.0>, :shutdown}, 0}

# Error log after 5 seconds:
"""
** (exit) exited in: GenServer.stop(#PID<0.149.0>, :normal, 5000)
    ** (EXIT) time out
    (elixir 1.10.3) lib/gen_server.ex:981: GenServer.stop/3
"""

Well, this time we received the exit signal. We had the chance to react in the handle_info/2 callback. Although, we decided that the GenServer should continue returning {:noreply, state}. After the five seconds of timeout, the Supervisor killed our GenServer because it didn’t kill itself.

Conclusion

A GenServer has 100 different ways to die. We didn’t explore all of them, but the most common ones. Try to use the one that best suits your situation.

As the last recommendation, don’t blindly trust the terminate/2 callback, as it doesn’t always run. If you really need to execute code when a GenServer dies, it is best to create another linked process that captures the exit signals. The code would be very similar to our “Link process” example.

Happy coding!

What is ActivityPub?

2018-12-07T00:00:00+01:00

This was my first question when I joined to MoodleNet team. Now, after studying it deeply for a few weeks, I feel able to give a proper answer. So I thought of writing a blog post, with my goal being to offer a nice introduction to ActivityPub for technical people. If you’re interested in my more general impressions on the fediverse and our approach to building a federated app, I recommend you to read my previous blog post about our goals.

ActivityPub is a protocol for decentralized social networks, which can also be extended to create all kinds of federated apps. So if you are thinking of building one it is very interesting to know how this standard works. It is important to notice it has two different parts:

Client to Server API
Server to Server API

A service can decide if it wants to implement only one or both of them. This decision just depends on the needs of the project. If you’re a frontend developer, you might use the Client to Server API. You will be focused on the Server to Server API when you only care about the federation part though.

ActivityStreams 2.0

The ActivityPub message format is JSON-LD, which can be used just like regular JSON and we’ll do this for this blog post. The vocabulary used in this protocol is defined by ActivityStreams 2.0 and ActivityPub just adds a couple of requirements more on top of it. In this vocabulary only exists two kind of entities: Link and Object. Here we will focus on Object and its subtypes. An Object has several fields and only two of them are required by ActivityPub: id and type. The most important properties are:

type: which defines the type of object. It can be a string or an array if the object implements more than a type. This is mandatory in ActivityPub.
id: a unique identifier for the object. For ActivityPub this field should be a URL. The interesting part is that you can retrieve the full object information making a GET request to this URL with the ActivityStreams accept header. This allows us to send just the id instead the full object information in many cases.
content: The content or textual representation of the Object, usually an HTML string.
name: A simple, human-readable, plain-text name for the object.
summary: A natural language summary of the object encoded as HTML.
image: A link to an image that represents an object.
@context: This is the only necessary property to accomplish with JSON-LD specification. In our examples we just set to "https://www.w3.org/ns/activitystreams".

Natural Language Values

So any object can have those fields. It is interesting that any field likely to be translated, like content, name or summary, can be renamed to a “map variant” to include different languages, ie:

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "type": "Note",
  "id": "https://mycoolserver.com/notes/1",
  "content": "This is a translated note"
}

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "type": "Note",
  "id": "https://mycoolserver.com/notes/1",
  "contentMap": {
     "es": "Esto es una nota traducida",
     "en": "This is a translated note"
  }
}

“Data Container” Objects

You may have noticed that the type of the previous object is "Note" instead of "Object". A Note is a subtype of object defined by ActivityStreams too, and it is used to represent any short written work. A subtype means that share the same fields, but it may define more if needed.

There are some more defined types like Article, Image, Page, Document, Video, and more. While it may feel strange to not see something like comment or similar, ActivityStreams is very generic and we have to keep this in mind. The recommendation is to use Note because it “Represents a short written work”. Those objects may be considerated “data container” objects. They don’t define more fields and they are used to transmit information.

Activity

In addition to those “data container” objects, ActivityStream defines Activity objects. The type of those objects may be more intuitive:

Create
Delete
Update
Add
Remove
Join
Follow
etc

They are the type of actions that you can expect from a social network. Special attention should be paid to the Announce activity, as it used to call attention to particular objects or notifications. It can be understood like a “retweet” on Twitter, or a simple way to share an already created object in your social network.

Activities may have 6 additional properties compared to a simple object:

object
actor
target
origin
result
instrument

Of course, not all of them make sense in every type of activities, so they are optional. So imagine you want to represent that Luke created the previous note, you can write:

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "type": "Create",
  "id": "https://mycoolserver.com/activity/1",
  "actor": "https://mycoolserver.com/actor/luke",
  "object": "https://mycoolserver.com/note/1"
}

Here, you only used actor and object. The more common ones. You don’t need to use the rest of them if they do not make sense in your activity.

One important thing to keep in mind is that the ids of the objects will not always appear, and the complete specification of them may appear instead. The previous example could be written in the following way, both being completely valid:

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "type": "Create",
  "id": "https://mycoolserver.com/activity/1",
  "actor": {
     "type": "Person",
     "id": "https://mycoolserver.com/actor/luke",
     "name": "Luke"
  }
  "object": {
    "type": "Note",
    "id": "https://mycoolserver.com/notes/1",
    "contentMap": {
       "es": "Esto es una nota traducida",
     "en": "This is a translated note"
  }
}

Actor

If you take a close look at the previous example you will see that Luke is a Person, this is a type of Object we haven’t yet mentioned. Person is one of the 5 types of actors that ActivityStreams defines:

Person
Group
Organization
Application
Service

Actors are objects that may execute the different activities, this means they can be used as the actor field on an Activity. An actor has two mandatory properties as well:

inbox
outbox

The previous two properties are mandatory for any actor. The following properties are not but they are recommended to be implemented:

following
follower
liked

Collection

All these five properties are from a core type we haven’t talked about yet: Collections. As you’d expect, a Collection is simply a group of objects. They also have additional properties, ie: totalItems with the number of elements in the Collection or items where you’ll find the objects in the collection.

So this means that the inbox of an actor is just a Collection containing all the activities received by the actor. The outbox contains all the activities sent by the actor. Any time an actor likes an object, this is added to their liked collection.

Obviously, after a while, those collections cannot be managed with just an array. It will be impractical to manage all the followers of a popular person in a simple array. For this reason, we have the last core type of ActivityStreams: the CollectionPage which should be used to paginate a full Collection.

Core types

So those are the 5 core object types that ActivityStreams defines:

Object
Actor
Activity
Collection
CollectionPage

All of them are also an Object, so they share most of the fields. Once you understand this vocabulary it becomes easier to work with the ActivityPub protocol.

Targeting

A big difference from working with other private social networks is the “Audience Targeting”. All objects can use 5 optional fields for this:

to
bto
cc
bcc
audience

Not only that, but it is the client application that fills in these fields. It can be just an actor, but maybe is your followers collection or it is just “public”. This can be better understood with the following examples.

Examples

Sending a private message

First of all, messages are always activities in ActivityPub. So if you want to send a private message to Ben, you have to build a Create activity with a Note. It is equally important to set the field to to only Ben’s id, because is a private message between Alyssa and Ben. You don’t want to make it public or to share with your followers. The JSON could be something like the following:

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "type": "Create",
  "id": "https://social.example/alyssa/posts/a29a6843-9feb-4c74-a7f7-081b9c9201d3",
  "to": [
    "https://chatty.example/ben/"
  ],
  "actor": "https://social.example/alyssa/",
  "object": {
    "type": "Note",
    "id": "https://social.example/alyssa/posts/49e2d03d-b53a-4c4c-a95c-94a6abf45a19",
    "attributedTo": "https://social.example/alyssa/",
    "to": [
      "https://chatty.example/ben/"
    ],
    "content": "Say, did you finish reading that book I lent you?"
  }
}

This is a Create activity because, of course, the message didn’t already exist. As you can see, the actor property is just a Link with Alyssa’s ID. Not need to send the full object. Similarly with the property to with Ben’s ID, but in this case in an array.

ActivityPub defines that a client should always create new activities talking to the server where the user’s account resides. In this example, Alyssa account resides at “https://social.example/”. Once the message is received by the server it is persisted so it can be requested at any time, but also, it detects that the recipient of the message does not belong to this server. This means that it’s now the server which should communicate this message to Ben’s server, in this case: “https://chatty.example/ben/”. It will send a post request to Ben’s inbox. It should then be Ben’s server that notifies Ben that he has a new message.

Add an image to an album

In this example we are working with images. The user Alyssa just move an image from her “Camera Roll” album to “My Cat Pictures”. The client application generates the following activity to represent the action that just happened:

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "id": "https://social.example/alyssa/activities/10239"
  "summary": "Alyssa added a picture of her cat to her cat picture collection",
  "type": "Add",
  "actor": "https://social.example/alyssa",
  "to": "https://social.example/alyssa/followers",
  "object": {
    "type": "Image",
    "name": "A picture of my cat",
    "id": "http://example.org/img/cat.png"
  },
  "origin": {
    "type": "Collection",
    "name": "Camera Roll"
    "id": "https://social.example/alyssa/collection/1"
  },
  "target": {
    "type": "Collection",
    "name": "My Cat Pictures",
    "id": "https://social.example/alyssa/collection/278"
  }
}

This activity is public only for the Alyssa’s followers. The client application send this activity to the Alyssa’s server. Now it is the server which has to locate all her followers and make sure that they receive this activity in their inbox. Maybe they are all locals, they resides on the same server, or maybe it has to send this activity to many external servers.

Follow request

In this scenario, Sally asks John to be friends in our application:

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "id": "http://example.org/connection-requests/123",
  "summary": "Sally requested to be a friend of John",
  "type": "Offer",
  "actor": "https://sally.example.org",
  "object": {
    "summary": "Sally and John's friendship",
    "id": "http://example.org/connections/123",
    "type": "Relationship",
    "subject": "https://sally.example.org",
    "relationship": "http://purl.org/vocab/relationship/friendOf",
    "object": "https://john.example.org"
  },
  "target": "https://john.example.org",
  "to": "https://john.example.org",
  "cc": "https://www.w3.org/ns/activitystreams#Public"
}

We make this request using the Offer activity type with a Relationship object and with the target John. Now John can accept the offering with:

{
     "@context": "https://www.w3.org/ns/activitystreams",
      "summary": "John accepted Sally's friend request",
      "id": "http://example.org/activities/122",
      "type": "Accept",
      "actor": "https://john.example.org",
      "object": "http://example.org/connection-requests/123",
      "inReplyTo": "http://example.org/connection-requests/123",
      "context": "http://example.org/connections/123",
      "to": "https://sally.example.org/",
      "cc": "https://www.w3.org/ns/activitystreams#Public"
}

It is interesting to see that in this application, the creator decided to make public this kind of interactions. To make public an object we have to add to any “targeting property” the value "https://www.w3.org/ns/activitystreams#Public", in this case in the cc property.

Conclusions

It is important to remember that ActivityPub is a protocol. It defines the expected behaviour but never the implementation details behind it. The client sends activities to the server. The server federates the same activities to other servers. The client can fetch for activities and the server should provide them. The way you do it is completely up to you.

There are a lot more things around ActivityPub, here we only scratched the surface. There are more types in the vocabulary, the possibility of extending it with your own types and properties, security recommendations, side effects, client and server responsibilities, etc. The goal of this article was to give a small introduction without getting lost in technical formalities. I hope you enjoyed.

This post was also published in MoodleNet blog.

Photo by Marius Christensen

MoodleNet: more than a technical project

2018-10-11T00:00:00+02:00

As many of you already know, I have been hired by Moodle to work on the MoodleNet project. Moodle is the world’s most popular learning platform with more than 120 million users around the world. It is used by many universities on every continent. In fact, I used it myself during my time as a student of Computer Engineering. And, of course, it’s an open source project!

Live, Work, Create

The MoodleNet project is a little bit different, but of course, related to the Moodle main project. Our mission is to empower communities of educators to share and learn from each other to improve the quality of education. We are building a new open social media platform for educators, focused on professional development and open content.

For now, I have only worked on the project for two weeks, but I have already noticed big differences from previous clients and jobs. So I’d like to use this post to explain these differences and how they influence the technical aspects of the project.

“Make the world a better place” is more than a slogan

A lot of workplaces write these kind of slogans on their walls. They think it is motivating for their workers. But do you know what is really motivating? Actually doing it. It is easier to feel like that when you write open source code, but we are pushing this mantra even further.

Do something great

We must have a prototype of the MoodleNet project ready in January. For this very reason, I just wanted to start coding immediately to make that possible. You know, if there is something that you learn in the world of technology, it is “build fast, fail fast”. This is applied almost everywhere for MVPs, proof of concepts, prototypes, etc. However, things are a little different with MoodleNet. We’re not just building for one company, but for a whole community.

I have been learning about ActivityPub, which is an open protocol developed by the W3C for social networks. We are not just implementing this protocol for MoodleNet, we want to create an ActivityPub generic implementation for the community. In this way, the future developers will be able to create their own open social network more easily. How cool is that?

This way, we cannot fail. Even if the MoodleNet project does not achieve all our goals, we’ll leave a legacy and we’ll collaborate with other open social networks. This is very important and brings us to the next point.

Philosophy in practice

We can architect MoodleNet in many possible ways. We could just create a centralized service where people log in and share their content, just like Facebook or Twitter do. This is much easier to code and to maintain. We didn’t choose this path because of our philosophy.

We are not building just a simple project, we are building a more democratic world. Learning is an essential value in any society. We want to make MoodleNet in the same way, and as democratically as possible.

This is the reason we bet on open and federated social networks, sometimes called just the “Fediverse” (Federation + Universe). This allows interacting with users that are hosted in other servers in the same way you can do it with users which are in the same server.

A great example is the email service. We all use it on a daily basis and it is a federated protocol. When you send an email from your ProtonMail address I can receive it in my Fastmail. And they are not the same server, but it works! Now, it is the same with the social networks, you can follow me from your ProtonSocial server, you can receive my messages from your ProtonSocial server and you can like my posts from your ProtonSocial server. You don’t have to worry about if I’m using FastSocial, or ProtonSocial or anything else.

Our goal is you can set up your own server, so you own your data. When you use Facebook you don’t own your data. If tomorrow Google+ decided to close its social network, all its users would lose their data. They would not be able to keep using even if they want it.

In an open federated social network like MoodleNet, you will have the code and you can install it on your own servers. So, you can always keep using it with all the other users which are using your server, and with all the other users in other servers around the world. You don’t depend on a big organization anymore! You are absolutely free! And all of this without the need to data-mine and sell your data :)

You have a voice

Another big problem for the democracy that is solved by the open social networks is censorship. Corporate owners of social networks decide what should and what should not go inside of their platforms. Most of them motivated by economic reasons such as providing shareholder value and you cannot do anything about it.

Well, here, in the fediverse, the rules change. It is more, “your server your rules”. So, if you don’t like a server you can move to another. If in the future they change their rules, you can move to another one again. Finally, if you really need it, you can create your own server only by yourself. You can make censorship a thing of the past. A Pluriform Social Network

Right now, we go to Youtube if we want to see videos, to Instagram to see photos and Medium to read something. While there is nothing wrong with have multiple accounts in different platforms, it may be a fragmented and frustrating. experience.

Thanks to the ActivityPub protocol, we have many more possibilities – more than could have imagined before I started this project! ActivityPub defines several entities, ie: “actors”, “follower”, “likes”, “notes”, “article”, “videos”, “photos”, etc. If two servers implement ActivityPub, they know how to speak each other even if they are from different platforms!

This can be explained better with an example. Imagine you have an account in Mastodon, which is best described as a kind of “Twitter clone” in the Fediverse and you love the shows of one guy on Peertube (a Youtube clone). The ActivityPub protocol means you can just follow him from your Mastodon account, instead of having to create a separate PeerTube account. When your Mastodon server receives a “video” from Peertube, it may not only notify you, but actually present the message with an embedded video! The same is true of PixelFed (an Instagram clone), if you follow people there, any new photos can appear in your Mastodon timeline. Or imagine a blog which implements ActivityPub. Blog posts can appear as a regular message in Mastodon with the introduction and a link to read more! The same data can be presented in so many different ways! I have given the example of everything appearing in Mastodon here, but of course the reverse is true.

The great thing about Open Source is that you always have a multiple options. Let’s say that the Mastodon interface is not for you. You can use another client, which implements ActivityPub, to read your data and updates. Even more, you can actually use different ones. You can have one that is very useful for community managers, another focus on a very nice UX for reading and the last one is just useful to listen to your favorite podcasts. If you are interested, you can find an alternative interface for Mastodon in Pinafore.

MoodleNet wants to be part of this Fediverse and we are going to support it creating as many tools as possible to encourage people to join us. For this very reason, we are doing the other way around, we are going to build a nice and generic platform for ActivityPub and later build, on top of it, our MoodleNet project.

Work openly

This is something I liked from the very beginning. Even before I was hired! When I was in the selection process, I did a little research about the company and the project. I was surprised they had a blog, and actually, it was regularly updated. So I saw many videos about the UX for the future prototypes. Then I could read about why they chose Elixir for the project and the next steps for the future backend developer, including the open source project they wanted to reuse. In addition, there were details about frontend technologies, why some decisions were made, diagrams, a lot of interesting stuff.

Together, We Create

I didn’t realize but I spent like 3 hours reading content about the project. I feel like I knew everything about it! It was funny then, in the interview, because I had to explain that I didn’t have questions, not because I wasn’t interested in the job, but because I had already done super research as a result of the team working openly.

What I mean is I felt like I was in the team and I understood why some decisions were made. This is a really important thing and I’ve missed in many open source projects that don’t share this knowledge with the rest of the world. Sometimes I don’t understand why some projects took one direction instead of another. After all, we can learn a lot from other project experiences, technical and not technical things.

Now that I’m on the other side, I see it is not easy at all. For example, I don’t know a lot of stuff about the Fediverse but I have to share with the world that I actually don’t know what I’m doing! Or maybe, I changed my opinion about something and I have to publish in the public issue tracker that my first decision was completely wrong.

It’s not easy to work so openly, especially at the beginning. You’re exposing yourself to the world. On the other hand, I see how the people actually understand your process and they empathize with you. It’s also cool to see how some people, which don’t work in Moodle, are interested in the project and collaborates with you. They give us their perspective and it is very enlightening, to see their perspective at the beginning of building a new project. So it is hard but it’s worth.

Conclusion

So, as you can see, Moodle’s core values affect our development process deeply. It is more difficult to code a federated system that a centralized one, but at the same time, makes it more exciting, and more importantly, we can have a broader impact. We are creating a tool to empowering teachers to share content, impossible a better goal. Besides this, we are collaborating with open source community exposing our code and creating new open source libraries. Equally important, we are promoting technologies and protocols to make the world a better and more democratic place. All of this in just one project. And this is one of the reason because I love this job.

If you are interested in our project you can also collaborate with us, although a simple comment is more than enough.

This post was also published in MoodleNet blog.

Photos by Jon Tyson, Clark Tibbs, Ilyass SEDDOUG and My Life Through A Lens

Everything you should know about Ruby Splats

2017-05-24T00:00:00+02:00

I’m working on a new project playing with new kids in town: Rails 5.1, Webpack(er), Bootstrap 4, React, etc. I was working with forms when I realized that Bootstrap 4 needs a lot of conditional CSS class to show states. Concatenating strings in a view it does not produce a very nice code.

So I thought I could create a similar behavior replicating the classNames node package:

model.valid?
=> true

class_string(:row, :promo, good: model.valid?, bad: !model.valid?)
=> "row promo good"

It concatenates all the hash’s keys whose value is truthy. For convenience, it also allows receiving an array for CSS classes that are always included. This is my helper’s implementation:

def class_string(*arr, **hash)
  hash
    .map { |css, bool| css if bool }
    .compact
    .concat(arr)
    .map(&:to_s)
    .join(" ")
end

Later, I found a bug in this method that amazed me. I believed I understood completely how the splats work in ruby, but his edge case it astonished me:

def splat_test(*array, **hash)
  puts "array: #{array} ; hash: #{hash}"
end

splat_test(42, 'foo' => 'foo', baz: :baz, String => String, bar: :bar)
# array: [42, {"foo"=>"foo", String=>String}] ; hash: {:baz=>:baz, :bar=>:bar}

Bazinga splat!

And this is the reason why I wrote this post. I have been investigating how the splats work in depth after discovering this code and now I want to share with you.

First I’ll explain the simple splat in detail, ~~including an unexpected benchmark result~~ updated: thanks to some comments I realized that the benchmark was not doing what I expected. I will continue with every case that has come to my mind with the double splat and the keyword arguments, to see later why they aren’t the same as a hash argument. Finally, I’ll try to explain why this code works in this way.

I will use several snippets from pry sessions. For me, it is the best way to learn ruby, hopefully, it is useful for you too.

The simple *splat

The simple splat works with arrays.

Ruby splat with methods

One of the main uses of the splats is to receive an undefined number of arguments:

def splat_arguments(*args)
  args
end

splat_arguments(:linux, "ms", String)
=> [:linux, "ms", String]

splat_arguments(linux: :cool, arch: :very_cool)
=> [{:linux=>:cool, :arch=>:very_cool}]

It works with more arguments as well:

def splat_more_arguments(a, *args, c)
  puts "a #{a} ; args #{args} ; c #{c}"
end

splat_more_arguments(:archlinux, :elementaryos, :gentoo, :ubuntu, :centos)
# a: archlinux ; args: [:elementaryos, :gentoo, :ubuntu] ; c: centos

splat_more_arguments(:archlinux, :centos)
# a: archlinux ; args: [] ; c centos

splat_more_arguments(:archlinux, [:elementaryos, :gentoo], :ubuntu, :centos)
# a: archlinux ; args: [[:elementaryos, :gentoo], :ubuntu] ; c: centos

splat_more_arguments(:archlinux)
# ArgumentError: wrong number of arguments (given 1, expected 2+)

You can avoid giving a name to the splat variable. This way you show your intention of not using those variables and it won’t be confused with a possible bug in the future:

def anonymous_splat_arguments(first, *, last)
  [first, last]
end

anonymous_splat_arguments(:archlinux, :elementaryos, :gentoo, :ubuntu, :centos)
=> [:archlinux, :centos]

# Other interesting use
def inspected_method(*)
  puts "before calling inspected_method"
  super
  puts "after calling inspected_method"
end

Ruby splat as parameters

It can be used to pass parameters from an array:

def splat_parameters(a, b, c)
  puts "a: #{a} ; b: #{b} ; c: #{c} "
end

distros = [:archlinux, :debian, :redhat]

splat_parameters(distros[0], distros[1], distros[2])
# a: archlinux ; b: debian ; c: redhat

# Equivalent but easier to write
splat_parameters(*distros)
# a: archlinux ; b: debian ; c: redhat

splat_parameters(:archlinux, *[:debian, :redhat])
# a: archlinux ; b: debian ; c: redhat

Using splats with blocks

The previous points are also true for blocks:

def block_splat
  yield(:linux, "ms", String)
end

block_splat do |*args|
  p args
end
# [:linux, "ms", String]

def splatted_block
  distros = [:archlinux, :elementaryos, :gentoo, :ubuntu, :centos]
  yield *distros
end

splatted_block do |best, better, good, common, secure|
  puts "#{best}, #{better}, #{good}, #{common}, #{secure}"
end
# archlinux, elementaryos, gentoo, ubuntu, centos

As you can see it works exactly in the same way.

Array destructuring with splats

The simple splat can be used outside the method definitions and calls. It can be used in assignments in a very analogous way:

distros = [:archlinux, :elementaryos, :gentoo, :centos, :ubuntu]
best, *rest, last = distros
puts "best: #{best} ; rest: #{rest} ; last: #{last}"
# first: archlinux ; rest: [:elementaryos, :gentoo, :centos] ; last: ubuntu

Updated: In a previous version of the article, I showed a “wrong” benchmark. Splat is not very performant technique. See comments for more information.

An astute reader may have realized that it is not exactly the same as using first and last methods, or [1], [2], [-1], [-2], etc. For example:

first, *, last = [:a]
first
=> :a
last
=> nil

first = [:a].first
first
=> :a
last = [:a].last
last
=> :a

first and last are not always the same that using splats!

Sometimes using splats is cleaner and better to show your real intentions. Just remember to no name the splat variable, like the above example, for better performance if you don’t need it.

Flatten arrays with splats

Flatten splat by SidewaysSarah

I think this example is self-explanatory:

distros = [:archlinux, :elementaryos, :gentoo, :centos, :ubuntu]
more_distros = [:debian, *distros, :fedora]
=> [:debian, :archlinux, :elementaryos, :gentoo, :centos, :ubuntu, :fedora]

Wrap in an array with splats

Last but not least, did you know splats can also replace the Array() method?

Array("string")
=> ["string"]

wrap = *"string"
=> ["string"]

wrap = *1
=> [1]

Array(nil)
=> []

wrap = *nil
=> []

var = "string"
wrap = *var
=> ["string"]

var = 1
wrap = *var
=> [1]

var = [1]
wrap = *var
=> [1]

var = nil
wrap = *var
=> []

But maybe it doesn’t do what you expected with a hash:

var = {a: :a}
wrap = *var
=> [[:a, :a]]

And with this wrap, we finished the simple splat. Let’s take a look at the double one!

The double **splat operator in Ruby

The double splat in Ruby works with hashes instead of arrays, but not with all of them like I thought.

Double Splat by Epic Fireworks

Let’s see some examples:

def double_splat(**hash)
  hash
end

double_splat(a: :a, b: :b)
=> {:a=>:a, :b=>:b}

hash = { a: :a }
double_splat(hash)
=> {:a=>:a}

Keyword arguments

This feature introduced in Ruby 2.0 is related to the double splat operator. You can define a method with keyword arguments, this means the parameters have a name. When you want to call a keyword argument method you have to name each argument:

def keyword_arguments(key_a: :default_a, key_b: :default_b)
  [key_a, key_b]
end

keyword_arguments(key_a: :a, key_b: :b)
=> [:a, :b]

keyword_arguments(key_b: :b, key_a: :a)
=> [:a, :b]

As you can see, the order of the arguments does not matter, but calling this kind of methods is cumbersome. It may be nice for methods with a long list of parameters, however, this is usually considered a code smell.

But this is not the only feature. They were declared with a default value, you can use them:

keyword_arguments(key_b: :b)
=> [:default_a, :b]

keyword_arguments
=> [:default_a, :default_b]

You can use a hash with all the parameters:

hash = { key_a: :a, key_b: :b }
keyword_arguments(hash)
=> [:a, :b]

If you forget to name the parameters it will fail:

keyword_arguments(:a, :b)
# ArgumentError: wrong number of arguments (given 2, expected 0)

The errors are very difficult to understand, so they deserve their own section…

Double splat and Keyword argument Errors

The last error it seems very strange:

ArgumentError: wrong number of arguments (given 2, expected 0)

Why did it expect 0 when 2 arguments were defined?

The double splat argument and keyword arguments are special arguments in Ruby. You can check it with your ruby console and a little of introspection:

def simple_method(value)
  value
end
Object.instance_method(:simple_method).parameters
=> [[:req, :value]]

def optional_method(value=3)
  value
end
Object.instance_method(:optional_method).parameters
=> [[:opt, :value]]

def splat_method(*args)
  args
end
Object.instance_method(:splat_method).parameters
=> [[:rest, :args]]

def double_splat_method(**args)
  args
end
Object.instance_method(:double_splat_method).parameters
=> [[:key_rest, :args]]

def keyword_method(a_key: :value)
  a_key
end
Object.instance_method(:keyword_method).parameters
=> [[:key, :a_key]]

def block_method(&a_block)
  a_block
end
Object.instance_method(:block_method).parameters
=> [[:block, :a_block]]

The error message is confused because you don’t expect that a keyword argument isn’t an argument, but for Ruby, they are completely different kind of arguments.

Thus if you use regular values for double splat and keyword arguments Ruby complains:

double_splat_method(42)
# ArgumentError: wrong number of arguments (given 1, expected 0)

keyword_method("4, 8, 15, 16, 23, 42")
# ArgumentError: wrong number of arguments (given 1, expected 0)

Hash keys MUST be symbols!

I didn’t know this before creating this post and it was part of my initial confusion. If you use a hash with no symbol keys, they aren’t used by the double splat and keyword arguments:

keyword_method("a_key" => "value")
# ArgumentError: wrong number of arguments (given 1, expected 0)

keyword_method(a_key: "value")
=> "value"

double_splat_method("a_key" => "value")
# ArgumentError: wrong number of arguments (given 1, expected 0)

double_splat_method(a_key: "value")
=> {:a_key=>"value"}

Hash MUST contain exactly the same keys

If you pass more keys than the defined in a keyword arguments method Ruby fails again:

keyword_method(a_key: "value")
=> "value"

keyword_method(a_key: "value", other_key: "666")
# ArgumentError: unknown keyword: other_key

hash = { a_key: "value", other_key: "666" }
keyword_method(hash)
# ArgumentError: unknown keyword: other_key

To avoid this kind of errors, you can use the anonymous double splat operator:

def keyword_arguments_no_crash(key_a: :default_a, key_b: :default_b, **)
  [key_a, key_b]
end

invalid_hash = { key_a: :hash_a, key_b: :hash_b, key_c: :hash_c }
keyword_arguments_no_crash(invalid_hash)
=> [:hash_a, :hash_b]

In fact, you can give it a name and use it in the method:

def keyword_arguments_plus(key_a: :default_a, key_b: :default_b, **plus)
  [key_a, key_b, plus]
end

invalid_hash = { key_a: :hash_a, key_b: :hash_b, key_c: :hash_c }
keyword_arguments_plus(invalid_hash)
=> [:hash_a, :hash_b, {:key_c=>:hash_c}]

Mandatory keyword arguments

Since Ruby 2.1, it is allowed to not add a default value to the arguments. In this case, the argument is mandatory:

def mandatory_arguments(mandatory:)
  mandatory
end

mandatory_arguments
# ArgumentError: missing keyword: mandatory

mandatory_arguments(mandatory: :learn)
=> :learn

It can be combined with the double splat to make some arguments optional and another mandatory:

def configuration(db:, **other_options)
  puts "db: #{db}, other_options: #{other_options}"
end

configuration(db: :postgres)
# db: postgres, other_options: {}

configuration(pool: 5)
# ArgumentError: missing keyword: db

configuration(db: :postgres, pool: 5)
# db: postgres, other_options: {:pool=>5}

conf = { pool: 5, db: :postgres }
configuration(conf)
# db: postgres, other_options: {:pool=>5}

As you can see, Ruby splits the hash for you, making the code easier to read and to work with.

Splat everywhere by Dimitri Popov

The double splat operator & destructuring hashes

Finally, you can combine keyword arguments and hashes using the double splat operator. It will be similar to use the merge method:

my_options = { logger: :stdout }

configuration(my_options.merge(db: :mongodb))
# db: mongodb, other_options: {:logger=>:stdout}

configuration(db: :mongodb, **my_options)
# db: mongodb, other_options: {:logger=>:stdout}

Using double splats with blocks

All the previous points can also be applied to the blocks in Ruby:

def block_double_splat
  yield(kernel: :panic)
end

block_double_splat do |**args|
  p args
end
# {:kernel=>:panic}

def double_splatted_block
  distros = { best: :archlinux, common: :ubuntu }
  yield distros
end

double_splatted_block do |best:, **|
  puts "The best is #{best}"
end
# The best is archlinux

Why double splat when the last argument could be a hash?

Do you remember that the double splat argument is not a regular argument? This causes strange message errors when the arguments are not expected, but you can also use it to take advantage of it:

def using_double_splat(*args, **hash_args)
  hash_args
end

def no_using_double_splat(*args, hash_args)
  hash_args
end

using_double_splat(7, 13, daenerys: :targaryen)
 => {daenerys: :targaryen}
no_using_double_splat(7, 13, daenerys: :targaryen)
 => {daenerys: :targaryen}

This seems to be the same behavior, but it is different when you don’t pass a hash as the last argument:

using_double_splat(7, 13)
=> {}
no_using_double_splat(7, 13)
=> 13

So, if you need the last argument to always be a hash, the double splat operator in the definition is the way to go.

The strange behavior of the beginning

This was the code:

def splat_test(*array, **hash)
  puts "array: #{array} ; hash: #{hash}"
end

pry(main)> splat_test(42, 'foo' => 'foo', baz: :baz, String => String, bar: :bar)
array: [42, {"foo"=>"foo", String=>String}] ; hash: {:baz=>:baz, :bar=>:bar}

Fighting with splats by Ixis IT

My first mistake was thinking that the double splat operator captures all pairs key-value of the last hash. However, as we have seen before double splat operator only captures symbol keys!

Ruby interprets the call of the method more or less in this way:

42: is a number and it will be the first element of the splat *array.
'foo' => 'foo': the key is not a symbol, it is a string, this means it cannot be added to the keyword hash. Hence it adds the couple key-value to the second element of the *array. This second element is, of course, an array.
:baz => :baz: ruby finds a new couple key-value. This time the key is a symbol (:baz) and it is added to the keyword hash
String => String: the key is not a symbol, it is a class. It cannot be added to the keyword hash. So ruby instead of complaining, cleverly (maybe too cleverly for my taste), it adds it to the splatted array. Instead of being added to a new hash at the third position, it is merged with the second and last element in this array. For me, it is clearly an unexpected behavior.
:bar => bar: This time the key is a symbol. It is added to the keyword hash.

Ruby does not oppose to this bizarre syntax, it allows you mix keys of all types and they are added to the “right” variable on the fly. Ruby is literally splitting our last hash argument into two different hashes, depending on the type of the key.

Conclusions

Congratulations if you reach until here :) This was a big review of splat features in Ruby. Hopefully, you learned something new among all the code snippets.

Double Splat by Reuben Whitehouse

The last part of this post, Ruby splitting the hash, is a curious feature. I can see the point because the keyword argument must have symbol keys, and it can be very useful as last optional argument. On the other hand, it is not very intuitive and it has to be used with caution. A future programmer may not know all the difficulties of splats in Ruby and bugs can easily introduce.

And what do you think about splat? Did you see something that surprised you?

The hidden cost of the invisible queries in Rails

2017-05-11T00:00:00+02:00

Ruby on Rails is awesome, it is really awesome. It allows us to create web applications without knowing every involved technology. After all, Rails is optimized for programmers happiness and it does a terrific job.

One of those technologies where Rails is really helpful is SQL. It removes the complexity of creating manual SQL queries using ActiveRecord. However, it is not only for our own good. Today we will see some of the drawbacks and how they affect the Rails application development.

What the heck am I doing?

Yes, we know we are creating a Ruby on Rails application, but…

Do you know exactly what are you doing on each line of your code?

This is not a trap question, it is a very important one, and it should be asked continuously while we are developing. We are better coders when we learn new implications of our code.

Rails helps us to write nice code, although the less experimented developers forget that behind a beautiful method call there are queries to the database, email sending, calls to remote API or messages to Redis. For example:

# We are doing a query to the table movies
# where actor_id == actor.id
actor.movies.each do |movie|
  ...
end

# Sends an email
UserMailer.welcome_email(@user).deliver_now

# Here we are doing 3 different API calls to Stripe
customer = Stripe::Customer.retrieve("cus_AcnQorgF0BIeBt")
card = customer.sources.retrieve("card_1AHXpr2eZvKYlo2CtTq8IsO5")
card.name = "Noah Moore"
card.save

# We are sending a message to a Redis server
# with the needed data to perform a background task.
# Later a Sidekiq server will read it and it will be executed
MyWorker.perform_async(1, 2, 3)

All those operations have something in common: they have to communicate with external processes. Maybe with a process on the same computer, so they are faster, or maybe on a remote server.

Communication with an external process is slow

Anyway, the point is those communications take time, much more than meaning_of_life = 21 * 2. So we have to be careful when using them. With a deeper knowledge about the framework we are using, we start to write better code and do fewer beginners mistakes.

Let’s see some of those mistakes that I personally did or I found in candidates’ code Deverify submissions

The super infamous 1+N

The most common beginner mistake in Ruby on Rails:

# This is a query
actor = Actor.find(actor_id)

actor.movies.each do |movie|
  # for each movie we are doing another query to get the director
  # so it is not very efficient
  puts movie.director.name
end

A good solution is to make only two queries:

Get all the movies from the actors.
Get all the directors using the already loaded movies.

This can be done in a convenient way with the includes method:

# This is a query
actor = Actor.find(actor_id)
# This makes two queries, one for the movies
# and another for all the directors of the movies
actor.movies.includes(:director).each do |movie|
  # No query is performed here
  puts movie.director.name
end

After the includes(:director), all the movies have the associated director model loaded in memory. Another way to get the same result is:

# It makes the three queries here: actor, movies and directors
actor = Actor.find(actor_id).includes(movies: :director)

actor.movies.each do |movie|
  # No query is performed here
  puts movie.director.name
end

We have also the fantastic gem Bullet that warns us if it detects a 1+N query. Take a look. It is a very good one.

Finally, we have this fantastic Arkency post to learn more about preloading associations in Rails.

How far do you plan to count?

Another poor performance statement using ActiveRecord is:

if Actor.count == 0
  puts "No actors"
end

This code performs a count query on the SQL server. However, we don’t need to count all the actors on the table. The only thing we want is to know if there is at least one.

Counting fast by Loic Djim

The SQL count query can be slower than we think. It possibly makes a sequential scan of the table. This means the database will go through each row, until the very last one, just to return the number of actors. A better approach is:

if Actor.limit(1).count == 0
  puts "No actors"
end

Here we are telling to the database to stop the query after finding the first one and return the result immediately. It’s much more performant this way!

Count only until you need

It’s also applicable to compare the count with other numbers:

if Actor.limit(2).count > 1
  puts "Many actors"
end

if Actor.limit(101).count > 100
  puts "More than 10 pages of 10 actors"
end

Some Rails query methods seem not optimized

When I was writing this post, I was testing alternatives methods in Rails to do the same. It surprised me that the common methods like any?, many? and empty? don’t use the simple optimization I recommended before. Maybe there is a reason for this behavior, but I don’t know.

Let me show some examples:

pry(main)> Actor.count
   (3.4ms)  SELECT COUNT(*) FROM "actors"
=> 9211
pry(main)> Actor.all.empty?
   (3.6ms)  SELECT COUNT(*) FROM "actors"
=> false
pry(main)> Actor.any?
   (3.3ms)  SELECT COUNT(*) FROM "actors"
=> true
pry(main)> Actor.many?
   (3.1ms)  SELECT COUNT(*) FROM "actors"
=> true
pry(main)> Actor.all.limit(1).count
   (0.8ms)  SELECT COUNT(count_column) FROM (SELECT  1 AS count_column FROM "actors" LIMIT $1) subquery_for_count  [["LIMIT", 1]]
=> 1
pry(main)> Actor.all.limit(2).count
   (0.9ms)  SELECT COUNT(count_column) FROM (SELECT  1 AS count_column FROM "actors" LIMIT $1) subquery_for_count  [["LIMIT", 2]]
=> 2

And a simple benchmark:

Benchmark.ips do |x|
  x.report("any?") { Actor.any? }
  x.report("limit(1)") { Actor.limit(1).count }
  x.compare!
end

Comparison:
  limit(1):     1213.5 i/s
      any?:      597.1 i/s - 2.03x  slower

As we can see, only with a table of 9.000 rows the difference is noticeable. With 100.000.000 rows the difference will be huge.

If someone knows the reason, please leave a comment about this. Thank you.

Exists? method is good (Updated)

KD made a comment to add that the method exists? works as expected.

2.2.4 :007 >User.exists?
User Exists (3.0ms) SELECT 1 AS one FROM "users" LIMIT 1
=> true

It uses the limit 1, so it is efficient. Nevertheless, this method won’t help us when we want to know if there are 2, 3, 4 or more rows in the database.

From wave-particle duality to relation-array duality on Rails

A has_many association in Ruby on Rails is not an array:

pry(main)> actor.movies.class
=> Movies::ActiveRecord_Associations_CollectionProxy

But it can behave like one:

pry(main)> Array.wrap(actor.movies)
=> [#<Movie:0x00000004a77660...

The ActiveRecord_Associations_CollectionProxy is a subclass of ActiveRecord::Relation (see the code on Github).

ActiveRecord::Relation is like a query proxy object and it allows us to create pretty queries:

User.where(surname: 'Stallman') # this returns an ActiveRecord::Relation
User.where(surname: 'Stallman').where(name: 'Richard') # and this one too

And the ActiveRecord::Relation class includes the module Enumerable (see the code on Github).

It’s an array, it’s a relation, it’s RAILS!

So there are methods in the ActiveRecord_Associations_CollectionProxy that work like a SQL query and others work like an array. It is really important to see the difference and take advantage of it when it is possible.

Let’s see some examples.

When once is better than twice

Imagine we want to print two movie lists, one with the published movies and another with the unpublished movies. If we do:

puts "Published movies"
# First query
director.movies.where(published: true).each do |movie|
  puts movie.name
end

puts "Unpublished movies"
# Second query
director.movies.where(published: false).each do |movie|
  puts movie.name
end

We are doing two queries, but we are loading all the movies for this director anyway. It has better performance if we just load all movies at once and then we use them like an array:

# This is an array now
movies = director.movies.to_a

puts "Published movies"
movies.select { |m| m.published }.each do |movie|
  puts movie.name
end

puts "Unpublished movies"
movies.reject { |m| m.published }.each do |movie|
  puts movie.name
end

There are several situations where doing a select in plain ruby is faster than doing it in the SQL server. This isn’t a general rule, we should considerate each case independently. For example, if we are not showing all the models, we are paginating them, to do the select in plain ruby is slower because we are requesting and instantiating unused models.

Where is my movie??

Does this code surprise you?

pry(main)> director.movies.build(name: "Gone with the Rails")
=> #<Movie:0x00000007be57b0
 id: nil,
 name: "Gone with the Rails">

pry(main)> director.movies.size
=> 1

pry(main)> director.movies.count
   (1.4ms)  SELECT COUNT(*) FROM "movies" WHERE "movies"."director_id" = $1  [["director_id", 1]]
=> 0

It is the array-relation duality in action again! First of all, we create a movie associated with the director; but this model is not saved in the database because we used the build method instead of the create method.

The method ActiveRecord::Relation#size is the “array method” and the ActiveRecord::Relation#count is the “relation/SQL method”. The count method will always perform a query. Our new movie is not in the database so it is impossible to be counted.

On the other hand, size is taking care of the models in memory (in the array), and it will make the query only if it is really needed.

This is not the only case, though, it can also happen in a very different situation:

pry(main)> director.movies.build(name: "Gone with the Rails")
=> #<Movie:0x00000007be57b0
 id: nil,
 name: "Gone with the Rails">

pry(main)> director.movies.where(name: "Gone with the Rails").first
  Movie Load (1.1ms)  SELECT  "movies".* FROM "movies" WHERE "movies"."director_id" = $1 AND "movies"."name" = $2 ORDER BY "movies"."id" ASC LIMIT $3  [["director_id", 1], ["name", "Gone with the Rails"], ["LIMIT", 1]]
=> nil

pry(main)> director.movies.select { |m| m.name == "Gone with the Rails" }.first
  Movie Load (1.0ms)  SELECT "movies".* FROM "movies" WHERE "movies"."director_id" = $1  [["director_id", 1]]
=> #<Movie:0x00000007be57b0
 id: nil,
 name: "Gone with the Rails">

Again, the where method just performs the query and the select method uses in memory models as well.

Conclusion

So nice as the invisible queries in Rails

At the end, this is not a critic to Ruby on Rails. It is an awesome technology that I love, and beginners have the opportunity to learn it fast. As we learn we have to take into account this kind of peculiarities, though.

Specially if your application, with thousands of active users, is down because a query is really really slow; and the problem is this harmless line:

if notes.count > 0

Oh my …! Believe me! You learn faster! True story.

So be aware of those issues and use it to your advantages to improve as programmer! And RTFM! :wink:

What about you? What unexpected behaviour did you find developing your app? Do you find bad performance issues due to invisible queries?

Username generation in Ruby on Rails

2017-04-25T00:00:00+02:00

Today we are going to create a unique username automatically using the user’s full name in a typical Ruby on Rails application, i.e.,

"Alex Castaño" => "alex_castano"

It can be very useful when creating a sign-up form, as we want it to be as simple as possible, like our Deverify register form.

We don’t want to bother our potential user with another field. Furthermore, it has to be unique in our database; it could be a pain for our user to find an available one. If the username is very visible in the application, we could give the user the ability to change it if he doesn’t like it.

The username conventions

Our username can only contain lower letters and underscores: /[a-z_]+/. We will use the full name given by the user to generate this username. However, the full name can contain capital letters, spaces, accents, etc. So our first step is to turn our full name in a valid username, keeping it as similar as possible:

  def generate_username(fullname)
    ActiveSupport::Inflector.transliterate(fullname) # change ñ => n
      .downcase              # only lower case
      .strip                 # remove spaces around the string
      .gsub(/[^a-z]/, '_')   # any character that is not a letter or a number will be _
      .gsub(/\A_+/, '')      # remove underscores at the beginning
      .gsub(/_+\Z/, '')      # remove underscores at the end
      .gsub(/_+/, '_')       # maximum an underscore in a row
  end

In addition to the rules above we added:

Avoid username which starts with or ends with an underscore.
We don’t want two underscores together; it can be confusing to type.

As Julien says in the comments, we can simply use:

fullname.parameterize(separator: '_')

And we’ll get the same result with less code!

Avoid username duplications in the database

The next step is to prevent duplications in the database. The best way to do this is adding a unique index in the username column, so we should do it if we haven’t already done it in a previous migration. It will also make our queries faster, and we need it to avoid concurrency problems that are explained later.

class AddIndexToUsernameInUsers < ActiveRecord::Migration[5.0]
  def change
    add_index :users, :username, unique: true
  end
end

So what should it happen if the generated username, john_doe, is already in the database? With our solution, it should generate john_doe_2. But if this username is already taken too? Well, as you can imagine, the function should return john_doe_3.

The code would be something like this:

  def find_unique_username(username)
    taken_usernames = User
      .where("username LIKE ?", "#{username}%")
      .pluck(:username)

    # username if it's free
    return username if ! taken_usernames.include?(username)

    count = 2
    while true
      # username_2, username_3...
      new_username = "#{username}_#{count}"
      return new_username if ! taken_usernames.include?(new_username)
      count += 1
    end
  end

The given username variable is the one we generated previously with generate_username. We will imagine that the value is 'john_doe'.

Here, it is important thing is to minimize the number of queries we make to our database. Instead of doing a query for each username, we will do only one query for all the current usernames that start with 'john_doe'. The regexp is /^john_doe.*/, but in SQL, it is john_doe%. We use the ActiveRecord method pluck which is ideal for this situation.

First, try to check if john_doe is taken. If it is not, we’ll return it. Otherwise, we iterate generating the usernames:

john_doe_2
john_doe_3
john_doe_4

Until one is not taken.

Avoid concurrency problems with unique validation in Rails

Finally, we have to deal with the concurrency problem. If two separate requests reach our servers, at the same time, with both the same name, one of them will fail (if we added the unique index). This problem is more common that you can think at first, and it should not be ignored! Especially as it is so easy to fix.

There are a lot of pages talking about the problem of uniqueness validation, so we know we should not use it. The SQL unique index is a more appropriate tool however Rails does not deal with it nicely. If the row already exists, Rails launches an exception we would have to capture.

Fortunately, rescue unique constraint gem solves exactly this problem. Personally, I’d like a solution better integrated like Phoenix/Ecto with #check_constraint/3, but Rails has its own personality.

Anyway, it is a small gem with no dependencies, so you can add it directly or you can to “copy & paste & modify” its code in your project.

Another difficulty is to differentiate between the same error in different fields:

email is taken: we should notify the user, probably he is already registered
username is taken: we have to generate a new username again because a concurrent thread was faster

For example:

  def create_user(params)
    username = generate_username(params.fetch(:name))
    username = find_unique_username(username)
    user = User.create(params.merge(username: username))
    if user.errors.details[:username].any? { |h| h[:error] == :taken }
      # a faster thread took our username, try again!
      create_user(params)
    else
      user
    end
  end

Conclusion

The code is not very long, and it does what is expected. We could integrate it in almost any Rails project. I recommend having all this user creation code in a service object, or much better, considerate creating a form object.

However you do it, the most important is to simplify the life of our users. And this is our job, isn’t it?

Github Pages with Jekyll Plugins and Theme

2017-03-04T00:00:00+01:00

Github allows us to host a Jekyll page using Github Pages, but the support for plugins and themes is very limited. We cannot freely use the plugins that you want, and we have exactly the same problem with the themes. Only safe plugins and the next available themes can be used.

At the beginning, I was happy with those limitations. Plugins can make your life much easier sometimes but I didn’t need them at the moment. So I installed minimal mistakes theme manually and everything works.

The day I wanted to modify and to update the theme I realized that I need a simpler alternative to work with. It turned out to be much easier than I supposed at beginning. Github Pages also allows us to upload an static page instead of the source of a Jekyll page and we will use this feature to our goal.

If you have minimal mistakes theme like me, probably you can find useful the migration to gem version as first step.

Deploy manually to Github Pages

The simplest solution I found is to work with two different branches:

source will contain the website source code, which the current master branch is
master will contain only the Jekyll output, the _site folder in Jekyll project.

We will have the two branches working together in the same directory structure. To do that will execute the follow commands, but be careful: the commands will remove the history of the master branch, although it will be kept in the source branch.

# create the source branch from the master
git checkout -b source
# push this branch to git repository
git push

# build the jekyll site
bundle exec jekyll build
# move to the output directory
cd _site
# create a new git repository in this folder
git init .
# we commit everything in this folder, which is the page
git add .
git commit -m "Initial commit"
# push this commit to github pages, all the history to avoid conflicts
git remote add origin git@github.com:YOURUSERNAME.github.io
git push -u --force

Like the _site directory is ignored by git because .gitignore excludes it, we can create another git repository to be used only with the master branch.

The source branch, where we will be working most of the time, it will be host in the root of the directory. We can work in it like we were working before in the master.

To deploy new changes we only have to:

bundle exec jekyll build
cd _site
git add .
git commit -m "New version"
git push

But also we can create bin/deploy file with execution permission and the following content:

#!/usr/bin/env bash
set -e # halt script on error

bundle exec jekyll build
cd _site
rm -rf .git
git init .
git add .
git commit -m "New version"
git remote add origin git@github.com:YOURUSERNAME.github.io
git push -u --force

We avoid conflicts forcing the push at the expense of don’t have master branch history. I don’t find it very useful anyway.

I found this solution very simple to work with and very integrated. Another advantage is that we can have our source and published site desynchronized, i.e: we can be working in a new draft post and push it to the source branch, so will be saved in Github repostory but until we don’t push the master branch will not be published.

On the other hand, if we forgot by mistake to push the master branch our work will never be published. We can automate the deploy process using Travis CI, or we can, in fact, use both methods at the same time.

Testing Jekyll site

html-proofer is a gem very useful to test the output of our Jekyll site. We add to the Gemfile:

gem "html-proofer" # Add this line

Run bundle install to be installed.

And we create a new script, bin/test, to check the html generated is correct:

#!/usr/bin/env bash
set -e # halt script on error

bundle exec htmlproofer --allow-hash-href  --internal-domains YOURDOMAIN ./_site

To test it we need the execute the Jekyll server previously.

$ bin/test
Running ["LinkCheck", "ScriptCheck", "ImageCheck"] on ["./_site"] on *.html...

Checking 30 external links...
Ran on 8 files!

HTML-Proofer finished successfully.

No problem in this output! :dizzy:

Deploy automatically to Github Pages with Travis-ci and CI

If you don’t like manual deployment we have another free alternative. Travis CI is a platform for continuous integration. It allows us to build, test and deploy almost any project. It can be used for free for open sourced projects and we’ll use this possibility for our web site.

We have to connect our Github page with the Travis one and to add our Github pages repository.

After that we need those two files:

bin/build: just to build the project easier

#!/usr/bin/env bash
set -e # halt script on error

bundle exec jekyll build

.travis.yml: this is the configuration for Travis CI ```yaml language: ruby env: global:
- NOKOGIRI_USE_SYSTEM_LIBRARIES=true
before_script: ./bin/build script: ./bin/test

deploy: provider: pages github-token: $PRIVATE_TOKEN target-branch: master local-dir: _site project-name: blog email: YOUREMAIL@gmail.com name: Deployment bot

skip_cleanup: true on: branch: source ```

Remember to add executable permissions to the files in the bin directory:

chmod +x bin/*

We make use of the (private token of Github)[https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/] for public repositories. We add it to (variables in repository settings)[https://docs.travis-ci.com/user/environment-variables/#Defining-Variables-in-Repository-Settings] like PRIVATE_TOKEN.

If everything is ok, every time we push the source branch Travis will build, test and push the new version to the Github pages. If something fails it will send us an email with the error.

Start using Docker

2017-01-23T00:00:00+01:00

Docker can be an overwhelming technology at the beginning. However we don’t need to use it in every piece of our developments, even less when we are learning it. We can start using slowly.

Use docker for databases in a development environment

I think a good first step to start using it can be like a database for development purpose. If we have installed a Postgres, MySQL or MongoDB service on our development machine, usually, the information we are storing there is for testing and not important at all. In those cases, we can virtualize those services inside a docker instance without fear.

Moreover, we will have several advantages:

We can have multiple versions running at the same time without conflicts
We can use the production version without depending on your distribution packages
We can stop using them with just a command, reducing memory when need it
We can learn more things about docker in the process

Dockerizing a Postgres Database

Previous steps

Before going on, remember to uninstall your current database if you have already installed:

# pacman -Rns postgres

And of course, install docker. Pay special attention to the docker group and remember to log out correctly of your current session.

Running Postgres in a Docker container

A simple way to start could be:

$ docker run --restart=unless-stopped --name postgres -p 5432:5432 -d postgres:latest

Let’s explain the command bit by bit:

docker the main program
run is a subcommand defined in docker. It allows us to run a command in a new container
--restart=unless-stopped in case the container died for any reason, this will reboot the docker container. It also works to start the container automatically when computer boots
--name postgres a label to identify the container in an easy way, in this case, we called postgres
-p 5432:5432 exposes the port of 5432 of the container like 5432 in our real computer. This port is the default postgres port
-d sends the process to the background
postgres:latest is the image we are going to execute. In this case is the official postgres image. It is ready to execute the database automatically. latest is to use the last version available, but we could use other options like 9.6, 9.5 or even more specific 9.5.2.

And now we have an postgres running as docker container. To see the docker currently running we can use:

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
462b1bae2bf8        postgres            "/docker-entrypoint.s"   4 weeks ago         Up 2 days           0.0.0.0:5432->5432/tcp   postgres

If for any reason we want to stop it:

$ docker stop postgres

And to remove definitely we will use:

$ docker rm postgres

Connecting to the database

You can now connect to the database. The default credentials are:

host: localhost
user: postgres
password: (empty)

You can modify those values. More information in the official postgres image. Remember to change those values in your development projects. In Ruby on Rails would be something like this:

# File config/database.yml
development:
  adapter:  postgresql
  host:     localhost
  encoding: unicode
  database: project_development
  pool:     5
  username: postgres
  password:
  template: template0

Conclusion

This is it, this is the very minimum to start with docker. It exists a long way before mastering this awesome technology: docker-compose, production environment, interconnection, etc. However, we can benefit from minute one, just learning the basics.

And do you use docker to virtualize your databases in development?

Hello World

2016-12-11T00:00:00+01:00

It’s finally here!! This is my new and shiny blog!! Hooray :tada: :tada: :tada:

It has been a long wait… but better late than never.

The main topic of this small space in Internet will be programming and technical stuff. But I don’t consider myself only a developer because what it really motivates me is to create useful products for people. So I’ll also write about everything around to build a product like marketing, user experience, make it profitable and many other things that I still don’t know but I’ll try to learn and to share here.

Furthermore as this is a personal blog, I cannot ensure you that eventually I change the topic and I talk about astronomy or urban gardens, I’ve always wanted to grow potatoes!

So everything seems ready, let’s start!