Code, Love & Boards

Headless CMS fun with Phoenix LiveView and Airtable (pt. 4)

2020-07-27T07:00:00Z

This post belongs to the Headless CMS fun with Phoenix LiveView and Airtable series.

In the previous part, we talked about Phoenix LiveView and implemented all the necessary views and templates to render all the content on our website. Each live view requests its contents to Airtable on its mount function, assigning them in its state. At first, it sounds like a proper implementation, similar to what we would have done using Ecto and a database to store the contents. However, the current solution has three main issues:

Using HTTP to request content against an external service adds an overhead to the initial page load, slowing down page loads.
If, for whatever reason, the external service is down, we won't be able to display any content to our visitors.
Last but not least, Airtable is limited to 5 requests per second per base. If we exceed this rate, it will return a 429 status code, and we'll need to wait 30 seconds before a new request succeeds.

So, how can we solve these issues? The solution we are going to implement consists of adding an in-memory cache to the repository, which stores both the contents and the articles, and synchronizes every second with Airtable. This way, we remove the additional overhead, render content even if Airtable happens to be down, and we control the number of requests per second done in a single place. Let's do this!

The repository cache

Erlang/OTP/Elixir comes with a robust in-memory term storage, called ETS, out of the box. It can store large amounts of data offering constant time data access. ETS organizes data as a set of dynamic tables, consisting of tuples, containing a key and the stored term. Tables are created by a process that is its owner and deleted when this process dies. In Elixir, ETS tables are often managed using a GenServer process, and in our particular case, we need two different tables, ergo two different cache processes, one to store PhoenixCms.Content terms and the other to store PhoenixCms.Article. Knowing this, let's implement the generic GenServer cache definition:

# lib/phoenix_cms/repo/cache.ex

defmodule PhoenixCms.Repo.Cache do
  use GenServer

  @callback table_name :: atom
  @callback start_link(keyword) :: GenServer.on_start()

  @impl GenServer
  def init(name) do
    name
    |> table_for()
    |> :ets.new([:ordered_set, :protected, :named_table])

    {:ok, %{name: name}}
  end

  defp table_for(name), do: apply(name, :table_name, [])
end

This module defines the behavior that any cache process must fulfill, which is initially a table_name/0 function, which returns the ETS table name for that cache and a start_link/1 function, which defines how the cache process starts. The module also defines the generic logic of a cache process, like the init/1 function, which using name with the private table_for/1 function, gets the table name, depending on the current cache module, and creates the ETS table using the following parameters:

name, an atom representing the table's name, in our case :articles or :contents.
:ordered_set, which specifies the type of the table, in our case, an ordered set that contains a value per unique key.
:protected, which sets the access control, in our case read is available for all processes, but write operations are permitted only from the owner process, avoiding race conditions.
:named_table, makes the table accessible by name instead of the by its PID.

Finally, it sets the initial process state to a map containing the cache module name, which we'll use later on. Now let's define the specific cache modules:

# lib/phoenix_cms/article/cache.ex

defmodule PhoenixCms.Article.Cache do
  alias PhoenixCms.Repo.Cache

  @behaviour Cache

  def child_spec(opts) do
    %{
      id: __MODULE__,
      start: {__MODULE__, :start_link, [opts]},
      type: :worker,
      restart: :permanent,
      shutdown: 500
    }
  end

  @impl Cache
  def table_name, do: :articles

  @impl Cache
  def start_link(_args) do
    GenServer.start_link(Cache, __MODULE__, name: __MODULE__)
  end
end

# lib/phoenix_cms/content/cache.ex

defmodule PhoenixCms.Content.Cache do
  alias PhoenixCms.Repo.Cache

  @behaviour Cache

  def child_spec(opts) do
    %{
      id: __MODULE__,
      start: {__MODULE__, :start_link, [opts]},
      type: :worker,
      restart: :permanent,
      shutdown: 500
    }
  end

  @impl Cache
  def table_name, do: :contents

  @impl Cache
  def start_link(_args) do
    GenServer.start_link(Cache, __MODULE__, name: __MODULE__)
  end
end

Both modules fulfill the PhoenixCms.Repo.Cache behavior by implementing the table_name/0 function, which returns the corresponding table name, and start_link/1, starts a new PhoenixCms.Repo.Cache GenServer, and uses their module name to set the initial state and name the process. To start both cache processes when the application starts, let's add them to the main supervision tree:

# lib/phoenix_cms/application.ex

defmodule PhoenixCms.Application do
  use Application

  def start(_type, _args) do
    children = [
      # ...

      # Start cache processes
      PhoenixCms.Article.Cache,
      PhoenixCms.Content.Cache,

      # ...
    ]

    opts = [strategy: :one_for_one, name: PhoenixCms.Supervisor]
    Supervisor.start_link(children, opts)
  end

  # ...
end

To check that everything works fine, let's start IEX and get the cache processes info:

➜ iex -S mix
Erlang/OTP 23 [erts-11.0.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1]

Interactive Elixir (1.10.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> PhoenixCms.Article.Cache |> GenServer.whereis() |> Process.info()
[
  registered_name: PhoenixCms.Article.Cache,
  current_function: {:gen_server, :loop, 7},
  initial_call: {:proc_lib, :init_p, 5},
  status: :waiting,
  message_queue_len: 0,
  links: [#PID<0.318.0>],
  dictionary: [
    "$initial_call": {PhoenixCms.Repo.Cache, :init, 1},
    "$ancestors": [PhoenixCms.Supervisor, #PID<0.317.0>]
  ],
  trap_exit: false,
  error_handler: :error_handler,
  priority: :normal,
  group_leader: #PID<0.316.0>,
  total_heap_size: 233,
  heap_size: 233,
  stack_size: 12,
  reductions: 44,
  garbage_collection: [
    max_heap_size: %{error_logger: true, kill: true, size: 0},
    min_bin_vheap_size: 46422,
    min_heap_size: 233,
    fullsweep_after: 65535,
    minor_gcs: 0
  ],
  suspending: []
]
iex(2)> PhoenixCms.Content.Cache |> GenServer.whereis() |> Process.info()
[
  registered_name: PhoenixCms.Content.Cache,
  current_function: {:gen_server, :loop, 7},
  initial_call: {:proc_lib, :init_p, 5},
  status: :waiting,
  message_queue_len: 0,
  links: [#PID<0.318.0>],
  dictionary: [
    "$initial_call": {PhoenixCms.Repo.Cache, :init, 1},
    "$ancestors": [PhoenixCms.Supervisor, #PID<0.317.0>]
  ],
  trap_exit: false,
  error_handler: :error_handler,
  priority: :normal,
  group_leader: #PID<0.316.0>,
  total_heap_size: 233,
  heap_size: 233,
  stack_size: 12,
  reductions: 44,
  garbage_collection: [
    max_heap_size: %{error_logger: true, kill: true, size: 0},
    min_bin_vheap_size: 46422,
    min_heap_size: 233,
    fullsweep_after: 65535,
    minor_gcs: 0
  ],
  suspending: []
]

Since the processes are up, both ETS tables must have been created as well, let's confirm it:

iex(3)> :ets.info(:articles)
[
  id: #Reference<0.2785746427.4276748289.246266>,
  decentralized_counters: false,
  read_concurrency: false,
  write_concurrency: false,
  compressed: false,
  memory: 145,
  owner: #PID<0.326.0>,
  heir: :none,
  name: :articles,
  size: 0,
  node: :nonode@nohost,
  named_table: true,
  type: :ordered_set,
  keypos: 1,
  protection: :protected
]
iex(4)> :ets.info(:contents)
[
  id: #Reference<0.2785746427.4276748289.246267>,
  decentralized_counters: false,
  read_concurrency: false,
  write_concurrency: false,
  compressed: false,
  memory: 145,
  owner: #PID<0.327.0>,
  heir: :none,
  name: :contents,
  size: 0,
  node: :nonode@nohost,
  named_table: true,
  type: :ordered_set,
  keypos: 1,
  protection: :protected
]
iex(5)>

There we go! And if we check both cache processes PIDs, they should match their corresponding table owner's PID:

iex(5)> GenServer.whereis(PhoenixCms.Article.Cache)
#PID<0.326.0>
iex(6)> GenServer.whereis(PhoenixCms.Content.Cache)
#PID<0.327.0>
iex(7)>

Let's add some helper functions to the cache module to get and set data from the corresponding ETS table:

# lib/phoenix_cms/repo/cache.ex

defmodule PhoenixCms.Repo.Cache do
  # ...

  def all(cache) do
    cache
    |> table_for()
    |> :ets.tab2list()
    |> case do
      values when values != [] ->
        {:ok, Enum.map(values, &elem(&1, 1))}

      _ ->
        {:error, :not_found}
    end
  end

  def get(cache, key) do
    cache
    |> table_for()
    |> :ets.lookup(key)
    |> case do
      [{^key, value} | _] ->
        {:ok, value}

      _ ->
        {:error, :not_found}
    end
  end

  def set_all(cache, items), do: GenServer.cast(cache, {:set_all, items})

  def set(cache, id, item), do: GenServer.cast(cache, {:set, id, item})

  # ...

  @impl GenServer
  def handle_cast({:set_all, items}, %{name: name} = state)
      when is_list(items) do
    Enum.each(items, &:ets.insert(table_for(name), {&1.id, &1}))

    {:noreply, state}
  end

  def handle_cast({:set, id, item}, %{name: name} = state) do
    name
    |> table_for()
    |> :ets.insert({id, item})

    {:noreply, state}
  end
end

Let's take a closer look at them:

all/1 takes the cache module, which uses to get the table name and calls :ets.tab2list/1, which returns all the entry tuples of a given table, which maps to a list of values, or an {:error, :not_found} if empty.
get/2 receives the cache module and a key, and does the same as all/1.
set_all/2 and set/3 are different tho. Since we configured the table as protected, we can only insert data from the process which created the table. Therefore, it sends the corresponding messages to the processes using GensServer.cast/2. It implements both message callback functions, which insert all of the given entries or a given entry by its ID correspondently.

Let's refactor the repository module to include the cache in its logic, and avoid unnecessary HTTP requests when the data already exists in the cache:

# lib/phoenix_cms/repo.ex

defmodule PhoenixCms.Repo do
  alias PhoenixCms.{Article, Content}
  # ...

  def articles(skip_cache \\ false)
  def articles(false), do: all(Article)
  def articles(true), do: @adapter.all(Article)

  def contents(skip_cache \\ false)
  def contents(false), do: all(Content)
  def contents(true), do: @adapter.all(Content)

  def get_article(id), do: get(Article, id)

  defp all(entity) do
    with cache <- cache_for(entity),
         {:error, :not_found} <- Cache.all(cache),
         {:ok, items} <- @adapter.all(entity) do
      Cache.set_all(cache, items)
      {:ok, items}
    end
  end

  defp get(entity, id) do
    with cache <- cache_for(entity),
         {:error, :not_found} <- Cache.get(cache, id),
         {:ok, item} <- @adapter.get(entity, id) do
      Cache.set(cache, id, item)
      {:ok, item}
    end
  end

  defp cache_for(Article), do: PhoenixCms.Article.Cache
  defp cache_for(Content), do: PhoenixCms.Content.Cache
end

We are adding a new skip_cache parameter that when is false, instead of directly calling the adapter in the public functions, now it checks if the requested items exist in the cache, calling the adapter if not and populating them. Hence, the next request has the data already cached. When it is true, it uses the adapter directly, skipping the cache, and we'll use this variation in a minute. Let's navigate through the application checking out the logs:

>>>>>>> Homepage first visit
[info] GET /
[info] GET https://api.airtable.com/v0/appXTw8FgG3h55fk6/contents -> 200 (663.964 ms)
[info] Sent 200 in 834ms

>>>>>>> Blog page first visit
[info] GET /blog
[info] Sent 200 in 400µs
[info] GET https://api.airtable.com/v0/appXTw8FgG3h55fk6/articles -> 200 (225.874 ms)

>>>>>>> Blog detail first visit
[info] GET /blog/rec1osLptzsXfWg5g/lorem-ipsum
[info] Sent 200 in 422µs

>>>>>>> Homepage second visit
[info] GET /
[info] Sent 200 in 456µs

>>>>>>> Blog page second visit
[info] GET /blog
[info] Sent 200 in 531µs

The first time we visit the home and blog pages, it performs the corresponding HTTP requests against Airtable. However, when visiting an article page, since the article has already been cached from the previous request, it does not need to request it. The same happens when we visit both the home and blog pages a second time.

Synchronizing the caches

Although the current cache implementation covers the issues that we have using the HTTP client directly, we now face a new problem. If we update any data stored in Airtable, the changes will not be reflected in our application, as the cache is already populated. Currently, Airtable does not have any mechanism to report data changes, such as webhooks. Nevertheless, we can achieve this using services like Zapier, but it only works for new rows or row deletions, not for updates, which is not suitable for our needs. Therefore, let's build or own cache synchronization solution, which is very easy, thanks to Elixir.

The main idea is to have two new processes, one for each cache, that periodically make requests against Airtable, updating their corresponding cache if there are new changes. Let's define the new module:

# lib/phoenix_cms/repo/cache/synchronizer.ex

defmodule PhoenixCms.Repo.Cache.Synchronizer do
  alias PhoenixCms.Repo.Cache

  use GenServer

  @refresh_time :timer.seconds(1)

  def start_link(opts) do
    GenServer.start_link(__MODULE__, opts)
  end

  @impl GenServer
  def init(opts) do
    cache = Keyword.fetch!(opts, :cache)

    send(self(), :sync)

    {:ok, cache}
  end

  @impl GenServer
  def handle_info(:sync, cache) do
    with {:ok, items} <- apply(cache, :fetch_fn, []).() do
      Cache.set_all(cache, items)
    end

    schedule(cache)

    {:noreply, cache}
  end

  defp schedule(cache) do
    Process.send_after(self(), :sync, @refresh_time)
  end
end

One more time, we are using GenServer for the new process. Itsinit/1 function takes the mandatory cache key from the options, which can contain either PhoenixCms.Article.Cache or PhoenixCms.Content.Cache, and sets it as the initial state of the process. It also sends itself a :sync message, which handles in the handle_info(:sync, cache) callback, applying the :fetch_fn function to the cache module, and setting the items in the cache if the request succeeds. Finally, it calls the schedule/1 private function, which sends the :sync message again after one second. So, we now have a process that requests the corresponding data to Airbrake every second, updating the corresponding cache table. Now we need to start these processes, so let's refactor the cache module:

# lib/phoenix_cms/repo/cache.ex

defmodule PhoenixCms.Repo.Cache do
  use GenServer

  alias __MODULE__.Synchronizer

  @callback fetch_fn :: fun

  # ...

  @impl GenServer
  def init(name) do
    Process.flag(:trap_exit, true)

    name
    |> table_for()
    |> :ets.new([:ordered_set, :protected, :named_table])

    {:ok, pid} = Synchronizer.start_link(cache: name)
    ref = Process.monitor(pid)

    {:ok, %{name: name, synchronizer_ref: ref}}
  end

  # ...

  @impl GenServer
  def handle_info(
        {:DOWN, ref, :process, _object, _reason},
        %{synchronizer_ref: ref, name: name} = state
      ) do
    {:ok, pid} = Synchronizer.start_link(cache: name)
    ref = Process.monitor(pid)

    {:noreply, %{state | synchronizer_ref: ref}}
  end

  def handle_info({:EXIT, _, _}, state) do
    {:noreply, state}
  end

  # ...
end

Once the cache process initializes, it creates the corresponding ETS table as it was doing before and starts its synchronizer process. Since the synchronizer process links to the cache process, it does the following:

First of all, it traps exits, so if the synchronizer dies, it does not kill the cache process, implementing handle_info({:EXIT, _, _}, state).
Secondly, it monitors the synchronizer process, storing the monitor ref in its state, so in case the synchronizer process dies, it spawns a new one in handle_info( {:DOWN, ref, :process, _, _}, %{synchronizer_ref: ref), so that the cache keeps up to date with its source of truth, which is Airtable.

Finally, we need to implement the fetch_fn/0 function in the cache modules:

# lib/phoenix_cms/article/cache.ex

defmodule PhoenixCms.Article.Cache do
  alias PhoenixCms.{Repo, Repo.Cache}

  @behaviour Cache

  # ...

  @impl Cache
  def fetch_fn, do: fn -> Repo.articles(true) end
end

# lib/phoenix_cms/content/cache.ex

defmodule PhoenixCms.Content.Cache do
  alias PhoenixCms.{Repo, Repo.Cache}

  @behaviour Cache

  # ...

  @impl Cache
  def fetch_fn, do: fn -> Repo.contents(true) end
end

Each function calls the corresponding repository, passing true as the skip_cache parameter so that it always checks against Airtable. Let's go back to IEX and start :observer to check the application tree:

We can see both the PhoenixCms.Article.Cache and PhoenixCms.Content.Cache processes, each of them linked to their PhoenixCms.Repo.Cache.Synchronizer process. And if we check the logs, we can see the following:

iex(3)> [info] GET https://api.airtable.com/v0/appXTw8FgG3h55fk6/articles -> 200 (126.214 ms)
[info] GET https://api.airtable.com/v0/appXTw8FgG3h55fk6/contents -> 200 (121.886 ms)
[info] GET https://api.airtable.com/v0/appXTw8FgG3h55fk6/articles -> 200 (123.290 ms)
[info] GET https://api.airtable.com/v0/appXTw8FgG3h55fk6/contents -> 200 (125.372 ms)
[info] GET https://api.airtable.com/v0/appXTw8FgG3h55fk6/articles -> 200 (140.528 ms)
[info] GET https://api.airtable.com/v0/appXTw8FgG3h55fk6/contents -> 200 (116.197 ms)
[info] GET https://api.airtable.com/v0/appXTw8FgG3h55fk6/articles -> 200 (123.440 ms)
[info] GET https://api.airtable.com/v0/appXTw8FgG3h55fk6/contents -> 200 (121.408 ms)
[info] GET https://api.airtable.com/v0/appXTw8FgG3h55fk6/articles -> 200 (121.811 ms)
[info] GET https://api.airtable.com/v0/appXTw8FgG3h55fk6/contents -> 200 (118.887 ms)
[info] GET https://api.airtable.com/v0/appXTw8FgG3h55fk6/articles -> 200 (437.112 ms)

If we now change anything in Airtable, and we refresh the browser tab with our application, it should render the modifications correctly.

Real-time UI updates

We currently have everything working as planned. All the content is stored in an external service, and we get the content using an HTTP client pointing to its API. We also have a cache mechanism that auto-updates the stored data, and which prevents the repository from making additional HTTP requests. Finally, we display all the content using Phoenix LiveView, which lets us render updates in the UI in real-time. But, wait a sec! With our current implementation, that we have to refresh the browser manually to display content updates, we could have used regular Phoenix views. So what's the point of using LiveView anyways? The point is that we can broadcast changes to live views, which will render them to the visitor without having to refresh the browser whatsoever.

To broadcast changes to the view, we are going to be using Phoenix.PubSub, which comes by default with Phoenix. Let's do some refactoring to the cache module:

# lib/phoenix_cms/repo/cache.ex

defmodule PhoenixCms.Repo.Cache do
  use GenServer

  alias __MODULE__.Synchronizer

  # ...

  @callback topic :: String.t()

  @secret "cache secret"

  # ...

  @impl GenServer
  def init(name) do
    # ...

    {:ok, %{name: name, synchronizer_ref: ref, hash: ""}}
  end

  # ...

  @impl GenServer
  def handle_cast({:set_all, items}, %{name: name, hash: hash} = state)
      when is_list(items) do
    new_hash = generate_hash(items)

    if hash != new_hash do
      Enum.each(items, &:ets.insert(table_for(name), {&1.id, &1}))
      PhoenixCmsWeb.Endpoint.broadcast(apply(name, :topic, []), "update", %{})
    end

    {:noreply, %{state | hash: new_hash}}
  end

  # ...

  defp generate_hash(items) do
    :sha256
    |> :crypto.hmac(@secret, :erlang.term_to_binary(items))
    |> Base.encode64()
  end
end

First of all, we are defining a new callback function to the repository behavior, topic, which must return the topic in which we are going the broadcast the changes in the current cache. In the initial state, we are also adding a new hash empty string. While handling {:set_all, items} messages, we generate a new hash of with the items, and if the hash is different to the one stored in the state, it inserts all the items in the ETS table, like it previously did, and calls PhoenixCmsWeb.Endpoint.broadcast/3, using the :topic function from the cache module. Finally, it sets the new hash in its state. This way, we are reporting to any subscriber of topic that there are changes when the hashes are different. Moreover, it is also storing them and preventing unnecessary writes when there are no data differences.

Let's implement the topic function in both cache modules:

# lib/phoenix_cms/article/cache.ex

defmodule PhoenixCms.Article.Cache do
  alias PhoenixCms.{Repo, Repo.Cache}

  @behaviour Cache

  @topic "articles"

  # ...

  @impl Cache
  def topic, do: @topic
end

# lib/phoenix_cms/content/cache.ex

defmodule PhoenixCms.Content.Cache do
  alias PhoenixCms.{Repo, Repo.Cache}

  @behaviour Cache

  @topic "contents"

  # ...

  @impl Cache
  def topic, do: @topic
end

The last step is to subscribe to the corresponding topics in the live views:

# lib/phoenix_cms_web/live/page_live.ex

defmodule PhoenixCmsWeb.PageLive do
  use PhoenixCmsWeb, :live_view

  @topic "contents"

  @impl true
  def mount(_params, _session, socket) do
    PhoenixCmsWeb.Endpoint.subscribe(@topic)

    {:ok, assign_socket(socket)}
  end

  @impl true
  def handle_info(%{event: "update"}, socket) do
    {:noreply, assign_socket(socket)}
  end

  # ...
end

# lib/phoenix_cms_web/live/articles_live.ex

defmodule PhoenixCmsWeb.ArticlesLive do
  use PhoenixCmsWeb, :live_view

  alias PhoenixCmsWeb.LiveEncoder

  @topic "articles"

  @impl true
  def mount(_params, _session, socket) do
    PhoenixCmsWeb.Endpoint.subscribe(@topic)

    {:ok, assign_socket(socket)}
  end

  @impl true
  def handle_info(%{event: "update"}, socket) do
    {:noreply, assign_socket(socket)}
  end

  # ...
end

# lib/phoenix_cms_web/live/show_article_live.ex

defmodule PhoenixCmsWeb.ShowArticleLive do
  use PhoenixCmsWeb, :live_view

  @topic "articles"

  @impl true
  def mount(%{"id" => id}, _session, socket) do
    PhoenixCmsWeb.Endpoint.subscribe(@topic)

    {:ok, assign_socket(socket, id)}
  end

  @impl true
  def handle_info(%{event: "update"}, socket) do
    id = socket.assigns.article.id

    {:noreply, assign_socket(socket, id)}
  end

  # ...
end

In the mount/3 function, each view subscribes to the relevant topic. Once the view process adheres to the topic, it needs to handle incoming messages using handle_info/2, which reassigns the socket contents, triggering a new render in the visitor's screen. Let's jump back to the browser, change something in Airtable, and watch what happens in our application:

It works! We finally have finished our simple CMS using Phoenix and Airtable, yay! I hope you enjoyed this tutorial as much as I enjoyed writing it and implementing it. You can check the final result here, or have a look at the source code.

Happy coding!

Live demo Source code

Headless CMS fun with Phoenix LiveView and Airtable (pt. 3)

2020-07-20T06:14:00Z

This post belongs to the Headless CMS fun with Phoenix LiveView and Airtable series.

Live demo Source code

In the previous part, we generated the base application, and the Airtable API HTTP client to request both contents and blog articles. We also defined the Article and Content domain models, and implemented the repository pattern with two different adapters, one returning fake data for testing purposes, and the other using the Airtable HTTP client to request and convert the returned data into our domain. It's time for some front-end fun, so let's get cracking.

Rendering content using LiveView

One thing before continuing, though. I'm using Bulma, which is very good looking and easy to use CSS framework for the UI styles. To use it, you need to add this line in the root.html.leex template, and here you can find the CSS file with the custom styles.

What is Phoenix.LiveView? The short definition would be: a library which provides rich, real-time user experiences with server-rendered HTML, without having to write almost any JS whatsoever, only using plain Elixir. But in reality, it is a bit more complicated.

LiveView initially renders static HTML, which is fast and optimal for search and indexing engines. After the first rendering, it upgrades to a persistent connection, with its state, and is capable of listening to messages from both other processes and the browser, and update its state. Once the state is updated, it re-renders the parts of the HTML corresponding to these changes.

LiveView is currently so well integrated into Phoenix, that we can use them anywhere, including the router file as if they were controllers. Since we created the project with the --live option, we already have everything we need to start using it, so let's go ahead and edit the route file to add the three different live view that we need:

# lib/phoenix_cms_web/router.ex

defmodule PhoenixCmsWeb.Router do
  use PhoenixCmsWeb, :router

  # ...


  scope "/", PhoenixCmsWeb do
    pipe_through :browser

    live "/", PageLive
    live "/blog", ArticlesLive
    live "/blog/:id/:slug", ShowArticleLive
  end

  # ...
end

We have three different routes in our application:

/: which renders the home page using the PageLive live view.
/blog: which renders all the articles using the ArticlesLive live view.
/blog/:id/:slug: which renders a given article using the ShowArticleLive live view.

Live navigation

LiveView provides support for live navigation using the browser's pushState API, making it possible to navigate between pages without full page reloads. Let's use this feature by adding links to both the home and the blog page in the main navigation bar:

# lib/phoenix_cms_web/templates/layout/root.html.leex

<!DOCTYPE html>
<html lang="en">
  # ...

    <nav class="navbar has-shadow" role="navigation" aria-label="main navigation">
      <div class="container">
        <div class="navbar-brand">
          <%= live_patch "PhoenixCMS", to: Routes.live_path(@conn, PhoenixCmsWeb.PageLive), class: "navbar-item has-text-weight-bold has-text-link" %>
        </div>
        <div class="navbar-end">
          <%= live_patch "Blog", to: Routes.live_path(@conn, PhoenixCmsWeb.ArticlesLive), class: "navbar-item" %>
        </div>
      </div>
    </nav>

    # ...
</html>

live_patch renders a link which patches the current LiveView with the one specified in the to option, without reloading the whole page and adding a new entry in the browser's history. Now that we can navigate through our views let's implement the home page.

The PageLive live view

Let's start with the main home page:

# lib/phoenix_cms_web/live/page_live.ex

defmodule PhoenixCmsWeb.PageLive do
  use PhoenixCmsWeb, :live_view

  alias PhoenixCmsWeb.LiveEncoder

  @impl true
  def mount(_params, _session, socket) do
    {:ok, assign_socket(socket)}
  end

  # Missing assign_socket function...
end

The mount/3 function receives params, the current session, and the socket, returning it with the assigned contents. Let's implement the assign_socket/1 private function:

# lib/phoenix_cms_web/live/page_live.ex

defmodule PhoenixCmsWeb.PageLive do
  use PhoenixCmsWeb, :live_view

  # ...

  defp assign_socket(socket) do
    case fetch_contents() do
      {:ok, contents} ->
        socket
        |> assign(:page_title, "Home")
        |> assign(:contents, contents)
        |> put_flash(:error, nil)

      _ ->
        socket
        |> assign(:page_title, "Home")
        |> assign(:contents, nil)
        |> put_flash(:error, "Error fetching data")
    end
  end

  # Missing fetch_contents function...
end

Depending on the result of the fetch_contents/0 function, it assigns :contents or a flash :error. The fetch_contents/0 looks like this:

# lib/phoenix_cms_web/live/page_live.ex

defmodule PhoenixCmsWeb.PageLive do
  use PhoenixCmsWeb, :live_view

  # ...

  defp fetch_contents do
    with {:ok, contents} <- PhoenixCms.contents() do
      contents =
        contents
        |> Enum.sort_by(& &1.position)
        |> LiveEncoder.contents()

      {:ok, contents}
    end
  end
end

This function calls PhoenixCms.contents/0, which we haven't implemented yet, sorts contents by position and calls LiveEncoder.contents/1, which converts these Content structs into the payload which the live view template is expecting. When working with Pheonix apps, I like to delegate any business logic functions that need the *Web namespace from the main module, in our case PhoenixCms, acting as the public API between business logic and presentation. Let's go ahead and expose the functions that we need:

# lib/phoenix_cms.ex

defmodule PhoenixCms do
  defdelegate articles, to: PhoenixCms.Repo

  defdelegate contents, to: PhoenixCms.Repo

  defdelegate get_article(id), to: PhoenixCms.Repo
end

Now we need to implement the PhoenixCmsWeb.LiveEncoder module and convert the list of PhoenixCms.Content into the payload that the live template needs to render:

# lib/phoenix_cms_web/live/encoder.ex

defmodule PhoenixCmsWeb.LiveEncoder do
  alias PhoenixCms.Content

  def contents(items) when is_list(items) do
    {features, rest} =
      items
      |> Enum.map(&encode/1)
      |> Enum.split_with(&(&1.type == "feature"))

    rest
    |> Enum.concat([%{features: features}])
    |> List.flatten()
  end

  def encode(%Content{} = content) do
    Map.take(content, [:id, :type, :title, :content, :image, :styles])
  end
end

We want to render every content in its HTML section node, except for content with type feature, which we want to group them in the same section. Therefore, we split the contents into two different lists, extracting the ones with type feature and appending it as a map with a features key.

To render HTML in LiveView, you can either implement the render/1 callback function or create a your_view_template.html.leex template in your live view folder. Let's take the second choice:

# lib/phoenix_cms_web/live/page_live.html.leex

<%= if @contents do %>
  <%= for content <- @contents, do: render_section(content) %>
<% end %>

Iterating over the assigned contents, it calls the render_section/1 function, which we need to add to the PageLive module:

# lib/phoenix_cms_web/live/page_live.ex

defmodule PhoenixCmsWeb.PageLive do
  use PhoenixCmsWeb, :live_view

  # ...

  def render_section(%{type: "hero"} = content) do
    Phoenix.View.render(PhoenixCmsWeb.PageView, "hero.html", content: content)
  end

  def render_section(%{type: "text_and_image"} = content) do
    Phoenix.View.render(PhoenixCmsWeb.PageView, "text_and_image.html", content: content)
  end

  def render_section(%{features: content}) do
    Phoenix.View.render(PhoenixCmsWeb.PageView, "features.html", content: content)
  end
end

As we have three different content types (hero, text_and_image, and feature), we want to give them their layout and style, so we render them using different templates:

# lib/phoenix_cms_web/templates/page/hero.html.eex

<section class="hero is-link is-medium">
  <div class="hero-body">
    <div class="container">
      <header class="hero__header">
        <h1 class="title is-1 mb-6"><%= @content.title %></h1>
        <p class="subtitle is-3 mb-6"><%= @content.content %></p>
      </header>
      <figure class="image">
        <img class="" src="<%= @content.image %>" alt="Placeholder image">
      </figure>
    </div>
  </div>
</section>

# lib/phoenix_cms_web/templates/page/text_and_image.html.eex

<div class="container text-and-image">
  <div class="columns is-variable is-mobile is-8">
    <div class="column is-half">
      <header class="mb-4"><h2 class="title"><%= @content.title %></h2></header>
      <p class="subtitle"><%= @content.content %></p>
    </div>
    <div class="column is-half image-container">
      <figure class="image">
        <img src="<%= @content.image %>" alt="Placeholder image">
      </figure>
    </div>
  </div>
</div>

# lib/phoenix_cms_web/templates/page/features.html.eex

<section class="section">
  <div class="container mb-6 features">
    <header class="mb-6">
      <h2 class="title is-2">Features</h2>
    </header>
    <div class="columns is-multiline is-mobile is-8">
      <%= for item <- @content do %>
        <div class="column is-one-third feature">
          <figure class="image feature__image">
            <img src="<%= item.image %>" alt="Placeholder image">
          </figure>
          <header class="mb-4"><h4 class="title is-4"><%= item.title %></h4></header>
          <p class="subtitle"><%= item.content %></p>
        </div>
      <% end %>
    </div>
  </div>
</section>

The ArticlesLive live view

To render the articles list corresponding to the /blog route, let's implement the ArticlesLive module:

# lib/phoenix_cms_web/live/articles_live.ex

defmodule PhoenixCmsWeb.ArticlesLive do
  use PhoenixCmsWeb, :live_view

  alias PhoenixCmsWeb.LiveEncoder

  @impl true
  def mount(_params, _session, socket) do
    {:ok, assign_socket(socket)}
  end

  defp assign_socket(socket) do
    case fetch_articles() do
      {:ok, articles} ->
        socket
        |> assign(:page_title, "Blog")
        |> assign(:articles, articles)
        |> put_flash(:error, nil)

      _ ->
        socket
        |> assign(:page_title, "Blog")
        |> assign(:articles, nil)
        |> put_flash(:error, "Error fetching data")
    end
  end

  defp fetch_articles do
    with {:ok, articles} <- PhoenixCms.articles() do
      articles
      |> Enum.sort_by(& &1.published_at)
      |> LiveEncoder.articles()

      {:ok, articles}
    end
  end
end

Just like in the PageLive module, it fetches the articles using PhoenixCms.articles/0, which delegates its call to the PhoenixCms.Repo module. If everything goes fine, it encodes the items and assigns them to the socket. This step is important because since the socket process stores the assigned elements in memory, we only want to store the necessary values:

# lib/phoenix_cms_web/live/encoder.ex

defmodule PhoenixCmsWeb.LiveEncoder do
  alias PhoenixCms.{Article, Content }

  # ...

  def articles(articles) do
    Enum.map(articles, &encode/1)
  end

  def encode(%Article{} = article) do
    Map.take(article, [:id, :slug, :title, :description, :image, :author, :published_at])
  end
end

Note that we are not taking the full article content for this page, because we don't want to render it. Now let's write its template:

# lib/phoenix_cms_web/live/articles_live.html.leex

<%= if @articles  do %>
  <section class="section">
    <div class="container">
      <header class="mb-6"><h2 class="title">Blog</h2></header>
      <div class="columns is-variable is-multiline is-mobile is-8">
        <%= for article <- @articles, do: render_article(@socket, article) %>
      </div>
    </div>
  </section>
<% end %>

As we did with the contents list, we have to add the render_article/2 to the view:

# lib/phoenix_cms_web/live/articles_live.ex

defmodule PhoenixCmsWeb.ArticlesLive do
  use PhoenixCmsWeb, :live_view

  # ...


  def render_article(socket, %{id: _id, slug: _slug} = article) do
    Phoenix.View.render(PhoenixCmsWeb.PageView, "article.html", socket: socket, article: article)
  end
end

And we can't forget about its article item template:

# lib/phoenix_cms_web/templates/page/article.html.eex

<%= live_patch to: Routes.live_path(@socket, PhoenixCmsWeb.ShowArticleLive, @article.id, @article.slug), class: "column is-half article-list__article" do %>
  <img class="article__image" src="<%= @article.image %>">
  <header>
    <h3 class="title is-4"><%= @article.title %></h3>
    <h4 class="subtitle is-5"><%= @article.description %></h4>
    <div class="media">
      <div class="media-left">
        <figure class="image is-48x48">
          <img class="is-rounded avatar" src="<%= "https://avatars.dicebear.com/api/avataaars/#{@article.author}.svg" %>" alt="Placeholder image">
        </figure>
      </div>
      <div class="media-content">
        <p class="title is-6"><%= @article.author %></p>
        <p class="subtitle is-6"><%= @article.published_at %></p>
      </div>
    </div>
  </header>
<% end %>

Using the same live_patch function as in the main navigation section, we create a link around the article summary to navigate to the article detail page, in which we can read the full version of the article.

The ShowArticleLive live view

Last but not least, this LiveView renders the full version of an article:

# lib/phoenix_cms_web/live/show_article_live.ex

defmodule PhoenixCmsWeb.ShowArticleLive do
  use PhoenixCmsWeb, :live_view

  @impl true
  def mount(%{"id" => id}, _session, socket) do
    {:ok, assign_socket(socket, id)}
  end

  defp assign_socket(socket, id) do
    case PhoenixCms.get_article(id) do
      {:ok, article} ->
        socket
        |> assign(:page_title, article.title)
        |> assign(:article, article)
        |> put_flash(:error, nil)

      {:error, _} ->
        socket
        |> assign(:page_title, "Blog")
        |> assign(:article, nil)
        |> put_flash(:error, "Error fetching data")
    end
  end
end

Following the same pattern as in the previous views, it calls PhoenixCms.get_article/1 passing the article id received in its mount parameters, and assigning the result to the socket. The corresponding template looks like this:

# lib/phoenix_cms_web/live/show_article_live.html.leex

<%= if @article  do %>
  <article class="article">
    <div class="container mt-6">
      <header class="article__header">
        <h1 class="title"><%= @article.title %></h1>
        <div class="media">
          <div class="media-left">
            <figure class="image is-48x48">
              <img class="is-rounded avatar" src="<%= "https://avatars.dicebear.com/api/avataaars/#{@article.author}.svg" %>" alt="Placeholder image">
            </figure>
          </div>
          <div class="media-content">
            <p class="title is-6"><%= @article.author %></p>
            <p class="subtitle is-7"><%= @article.published_at %></p>
          </div>
        </div>
      </header>
      <figure class="image main-image">
        <img src="<%= @article.image %>">
      </figure>
      <p class="subtitle is-italic"><%= @article.description %></p>
      <section class="article__content">
        <%= raw(@article.content) %>
      </section>
    </div>
  </article>
<% end %>

Almost there

Now that we have everything ready, let's start the application and navigate through its pages, checking out the logs in the console:

iex(2)> [info] GET https://api.airtable.com/v0/appXTw8FgG3h55fk6/articles -> 200 (653.723 ms)
[info] GET /
[info] Sent 200 in 20ms
[info] GET https://api.airtable.com/v0/appXTw8FgG3h55fk6/contents -> 200 (153.722 ms)
[info] GET /blog
[info] Sent 200 in 426µs
[info] GET https://api.airtable.com/v0/appXTw8FgG3h55fk6/articles -> 200 (218.254 ms)
[info] GET /blog/rec1osLptzsXfWg5g/lorem-ipsum
[info] Sent 200 in 384µs
[info] GET https://api.airtable.com/v0/appXTw8FgG3h55fk6/articles/rec1osLptzsXfWg5g -> 200 (193.594 ms)
[info] GET /blog
[info] Sent 200 in 581µs
[info] GET https://api.airtable.com/v0/appXTw8FgG3h55fk6/articles -> 200 (211.392 ms)
[info] GET /
[info] Sent 200 in 519µs
[info] GET https://api.airtable.com/v0/appXTw8FgG3h55fk6/contents -> 200 (129.278 ms)
[info] GET /blog
[info] Sent 200 in 427µs
[info] GET https://api.airtable.com/v0/appXTw8FgG3h55fk6/articles -> 200 (224.131 ms)
[info] GET /blog/rec1osLptzsXfWg5g/lorem-ipsum
[info] Sent 200 in 381µs
[info] GET https://api.airtable.com/v0/appXTw8FgG3h55fk6/articles/rec1osLptzsXfWg5g -> 200 (118.158 ms)

As we can see, every time we visit a page, the view makes the corresponding HTTP request to get its necessary contents. Although working fine for a single user, if we had many users visiting our site, it could easily overcome Airtable's rate limit of five requests per second. Not to mention the overhead that adds making an HTTP request on every page and what would happen if Airtable is down for whatever reason. In the next and last part of the series, we will look for a solution to all these problems, by implementing an automated cache mechanism using ETS. In the meantime, you can check the end result here, or have a look at the source code.

Happy coding!

Live demo Source code

Headless CMS fun with Phoenix LiveView and Airtable (pt. 2)

2020-07-11T07:54:00Z

This post belongs to the Headless CMS fun with Phoenix LiveView and Airtable series.

Live demo Source code

In the previous part of these series, we talked about what we are going to be building and its two main parts. Today, we will focus on the Phoenix application, but you need the Airtable base to follow up on the tutorial. Therefore, if you don't have an Airtable account, sign up, and click on the Copy base link located at the top right corner of the source base. Once you have imported it into your workspace, we can continue creating the Phoenix application.

Creating the Phoenix application

Before generating the project scaffold, let's install the latest version of phx_new, which by the time I'm writing this part is v1.5.3.

mix archive.install hex phx_new 1.5.3

Now we can generate the project by running the mix phx.new with the following options:

mix phx.new phoenix_cms --no-ecto --no-gettext --no-dashboard --live

If you are not familiar with the options that we are using, here's a quick description of them:

--no-ecto: we are not using any database connection, so let's get rid of the Ecto files and configuration.
--no-gettext: we can also remove any translation-related dependency and files.
--no-dashboard: Phoenix has a brand new live dashboard where you can see all the metrics related to your application. We are going to be installing it, later on, so let's remove it for now.
--live: includes support for Phoenix LiveView, which is essential for this project.

Once the task finishes generating the project files and installing the necessary dependencies, I like to do some cleanup, removing the extra content that the generator creates for you, usually these CSS files and HTML.

The Airtable client

Before continuing any further, let's define our domain entities, which are going to map the data stored in Airtable, starting with the Content struct which represents a content section, from the contents table:

# lib/phoenix_cms/content.ex

defmodule PhoenixCms.Content do
  alias __MODULE__

  @type t :: %Content{
          id: String.t(),
          position: non_neg_integer,
          type: String.t(),
          title: String.t(),
          content: String.t(),
          image: String.t(),
          styles: String.t()
        }

  defstruct [:id, :position, :type, :title, :content, :image, :styles]
end

Let's continue by defining the Article struct, corresponding to the blog posts stored in the articles table:

# lib/phoenix_cms/article.ex

defmodule PhoenixCms.Article do
  alias __MODULE__

  @type t :: %Article{
          id: String.t(),
          slug: String.t(),
          title: String.t(),
          description: String.t(),
          image: String.t(),
          content: String.t(),
          author: String.t(),
          published_at: Date.t()
        }

  defstruct [:id, :slug, :title, :description, :image, :content, :author, :published_at]
end

The next step is to request the data to Airtable, taking advantage of its API, and convert the received data into the domain entities we have just defined. To implement the HTTP client, let's add Tesla to the project's dependencies, and install them running mix deps.get.

# mix.exs

defmodule PhoenixCms.MixProject do
  use Mix.Project

  # ...
  # ...


  defp deps do
    [
      # ...

      # Http client
      {:tesla, "~> 1.3"},
      {:hackney, "~> 1.16.0"}
    ]
  end

  # ...
end

Tesla suggests setting hackney as its default adapter, so let's go ahead and do that:

# config/config.exs

use Mix.Config

# ...

# Tesla configuration
config :tesla, adapter: Tesla.Adapter.Hackney

# ...

Once we have everything ready we can start implementing the client. When I have to use external services, such as Airtable, I like to separate any related logic in a different namespace, such as Services:

# lib/services/airtable.ex

defmodule Services.Airtable do
  # We are going to implement the public interface in a minute...

  defp client do
    middleware = [
      {Tesla.Middleware.BaseUrl, api_url() <> base_id()},
      Tesla.Middleware.JSON,
      Tesla.Middleware.Logger,
      {Tesla.Middleware.Headers, [{"authorization", "Bearer " <> api_key()}]}
    ]

    Tesla.client(middleware)
  end

  defp do_get(path) do
    client()
    |> Tesla.get(path)
    |> case do
      {:ok, %{status: 200, body: body}} ->
        {:ok, body}

      {:ok, %{status: status}} ->
        {:error, status}

      other ->
        other
    end
  end

  defp api_url, do: Application.get_env(:phoenix_cms, __MODULE__)[:api_url]

  defp api_key, do: Application.get_env(:phoenix_cms, __MODULE__)[:api_key]

  defp base_id, do: Application.get_env(:phoenix_cms, __MODULE__)[:base_id]
end

The client function returns a Tesla.Client using the following middleware:

Tesla.Middleware.BaseUrl, which sets the base URL for all the requests.
Tesla.Middleware.JSON, which encodes requests and decodes responses as JSON.
Tesla.Middleware.Logger, which logs requests and responses.
Tesla.Middleware.Headers, which sets headers for all requests, and in this particular case, the authorization header with the bearer token from Airtable.

For the base URL, we need to set both the api_url and base_id keys in the application's configuration. The same happens for api_key:

# config/config.exs

use Mix.Config

# ...


# Airtable configuration
config :phoenix_cms, Services.Airtable,
  api_key: "YOUR API KEY",
  base_id: "YOUR BASE ID",
  api_url: "https://api.airtable.com/v0/"

You can find your api_key in your Airtable account page, and the base_id in your API documentation page.

The do_get function takes a path and performs a GET request using the client. Since we don't want to deal with anything related to Tesla outside this module, it returns either a {:ok, body} or a {:error, reason} tuple. There's one thing left: to add the public interface, so let's go ahead and add two functions, one for getting all records from a table and the other for getting a table record by its ID:

# lib/services/airtable.ex

defmodule Services.Airtable do
  def all(table), do: do_get("/#{table}")

  def get(table, record_id), do: do_get("/#{table}/#{record_id}")

  # ...
end

Let's jump into iex and test the client, limiting the response to a single record:

➜ iex -S mix
Erlang/OTP 23 [erts-11.0.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1]

Interactive Elixir (1.10.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> Services.Airtable.all("contents?maxRecords=1")
[info] GET https://api.airtable.com/v0/YOUR_TABLE_ID/contents?maxRecords=1 -> 200 (614.224 ms)
[debug]
>>> REQUEST >>>
(Ommited request headers)

<<< RESPONSE <<<
(Ommited response headers)

(Ommited response payload)
{:ok,
 %{
   "records" => [
     %{
       "createdTime" => "2020-07-01T05:27:44.000Z",
       "fields" => %{
         "content" => "Lorem ipsum dolor sit amet, consectetur adipiscing elit",
         "id" => "feature_4",
         "image" => [
           %{
             "filename" => "pipe.png",
             "id" => "attJxlSNbmLRra4qx",
             "size" => 11828,
             "thumbnails" => %{
               "full" => %{
                 "height" => 3000,
                 "url" => "https://dl.airtable.com/.attachmentThumbnails/fe2e0dcd3e2a969f1816570e02dad366/7c9e2246",
                 "width" => 3000
               },
               "large" => %{
                 "height" => 512,
                 "url" => "https://dl.airtable.com/.attachmentThumbnails/2651c43ab85e28d2ba0c574f36ee7a1a/fe4a5495",
                 "width" => 512
               },
               "small" => %{
                 "height" => 36,
                 "url" => "https://dl.airtable.com/.attachmentThumbnails/0d717bf44d9552c7e25482496bc30c3c/6e29a1ad",
                 "width" => 36
               }
             },
             "type" => "image/png",
             "url" => "https://dl.airtable.com/.attachments/70ff8a20d056c7dfb677f1fc6bc79771/abea3535/pipe.png"
           }
         ],
         "position" => 10,
         "title" => "Feature 4",
         "type" => "feature"
       },
       "id" => "rec7VPdanrfUyvYnw"
     }
   ]
 }}

It works! Now let's confirm that the get/2 function works as well using the previous record ID:

iex(2)> Services.Airtable.get("contents", "rec7VPdanrfUyvYnw")
[info] GET https://api.airtable.com/v0/YOUR_TABLE_ID/contents/rec7VPdanrfUyvYnw -> 200 (6455.924 ms)
[debug]
>>> REQUEST >>>
(Ommited request headers)
<<< RESPONSE <<<
(Ommited response headers)

(Ommited response payload)
{:ok,
 %{
   "createdTime" => "2020-07-01T05:27:44.000Z",
   "fields" => %{
     "content" => "Lorem ipsum dolor sit amet, consectetur adipiscing elit",
     "id" => "feature_4",
     "image" => [
       %{
         "filename" => "pipe.png",
         "id" => "attJxlSNbmLRra4qx",
         "size" => 11828,
         "thumbnails" => %{
           "full" => %{
             "height" => 3000,
             "url" => "https://dl.airtable.com/.attachmentThumbnails/fe2e0dcd3e2a969f1816570e02dad366/7c9e2246",
             "width" => 3000
           },
           "large" => %{
             "height" => 512,
             "url" => "https://dl.airtable.com/.attachmentThumbnails/2651c43ab85e28d2ba0c574f36ee7a1a/fe4a5495",
             "width" => 512
           },
           "small" => %{
             "height" => 36,
             "url" => "https://dl.airtable.com/.attachmentThumbnails/0d717bf44d9552c7e25482496bc30c3c/6e29a1ad",
             "width" => 36
           }
         },
         "type" => "image/png",
         "url" => "https://dl.airtable.com/.attachments/70ff8a20d056c7dfb677f1fc6bc79771/abea3535/pipe.png"
       }
     ],
     "position" => 10,
     "title" => "Feature 4",
     "type" => "feature"
   },
   "id" => "rec7VPdanrfUyvYnw"
 }}

Yay! The Airtable client is ready. However, we still have to convert the returned payload into the domain entities we created previously, and for that, we are going to make use of the Repository pattern.

The Repository pattern

This pattern provides an abstraction of the data layer, which decouples it from its source or persistence layer, making it accessible through a series of straightforward functions. The basic idea is to have a public interface as the primary repository module that relies on different adapters, using the most suitable one depending on the situation or environment. The two adapters that we are going to implement are:

An HTTP adapter, powered by the Services.Airtable client, which we are going to be using while developing and in the production environment.
A fake adapter that returns hardcoded results, which we can use in our tests, prevents unnecessary HTTP requests against Airtable's API.

Let's go ahead and implement the main repository module:

# lib/phoenix_cms/repo.ex

defmodule PhoenixCms.Repo do
  alias PhoenixCms.{Article, Content}

  # Behaviour callbacks

  @type entity_types :: Article.t() | Content.t()

  @callback all(Article | Content) :: {:ok, [entity_types]} | {:error, term}
  @callback get(Article | Content, String.t()) :: {:ok, entity_types} | {:error, term}

  # Sets the adapter
  @adapter Application.get_env(:phoenix_cms, __MODULE__)[:adapter]

  # Public API functions
  def articles, do: @adapter.all(Article)

  def contents, do: @adapter.all(Content)

  def get_article(id), do: @adapter.get(Article, id)
end

In this module we are doing three different things:

First of all, it is describing the necessary callback functions that any module needs to implement to become a repository adapter. These functions are, all which receives an Article or Content atom and returns a {:ok, items} tuple on success or a {:error, reason} tuple on error.
It's also setting the current @adapter module variable from the application configuration.
Finally, it also implements three different functions, the public API of the repository, which internally use the corresponding adapter functions thanks to the previous dependency injection.

Knowing the repository interface, let's implement the HTTP adapter that relies on the Services.Airtable client that we created before:

# lib/phoenix_cms/repo/http.ex

defmodule PhoenixCms.Repo.Http do
  alias __MODULE__.Decoder
  alias PhoenixCms.{Article, Content, Repo}
  alias Services.Airtable

  @behaviour Repo

  @articles_table "articles"
  @contents_table "contents"

  @impl Repo
  def all(Article), do: do_all(@articles_table)
  def all(Content), do: do_all(@contents_table)

  @impl Repo
  def get(Article, id), do: do_get(@articles_table, id)
  def get(Content, id), do: do_get(@contents_table, id)

  defp do_all(table) do
    case Airtable.all(table) do
      {:ok, %{"records" => records}} ->
        {:ok, Decoder.decode(records)}

      {:error, 404} ->
        {:error, :not_found}

      other ->
        other
    end
  end

  defp do_get(table, id) do
    case Airtable.get(table, id) do
      {:ok, response} ->
        {:ok, Decoder.decode(response)}

      {:error, 404} ->
        {:error, :not_found}

      other ->
        other
    end
  end
end

The module implements the necessary callback functions from the Repo behavior, using the Services.Airtable client to fetch the data from the corresponding table. Since the behaviour specifies that both of these functions return Article or Contents structs, it uses a Decoder module to convert the raw HTTP response items into these domain data structures:

# lib/phoenix_cms/repo/http/decoder.ex

defmodule PhoenixCms.Repo.Http.Decoder do
  @moduledoc false

  alias PhoenixCms.{Article, Content}

  def decode(response) when is_list(response) do
    Enum.map(response, &decode/1)
  end

  def decode(%{
        "id" => id,
        "fields" =>
          %{
            "slug" => slug
          } = fields
      }) do
    %Article{
      id: id,
      slug: slug,
      title: Map.get(fields, "title", ""),
      description: Map.get(fields, "description", ""),
      image: decode_image(Map.get(fields, "image")),
      content: Map.get(fields, "content", ""),
      author: Map.get(fields, "author", ""),
      published_at: Date.from_iso8601!(Map.get(fields, "published_at"))
    }
  end

  def decode(%{
        "fields" =>
          %{
            "type" => type
          } = fields
      }) do
    %Content{
      id: Map.get(fields, "id", ""),
      position: Map.get(fields, "position", ""),
      type: type,
      title: Map.get(fields, "title", ""),
      content: Map.get(fields, "content", ""),
      image: decode_image(Map.get(fields, "image", "")),
      styles: Map.get(fields, "styles", "")
    }
  end

  defp decode_image([%{"url" => url}]), do: url
  defp decode_image(_), do: ""
end

Using pattern matching, it takes the necessary data to build the structs. Airtable does not send empty values, thus defaulting missing keys to empty strings. Let's jump back into iex and try it out:

➜ iex -S mix
Erlang/OTP 23 [erts-11.0.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1]

Interactive Elixir (1.10.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> PhoenixCms.Repo.Http.all(PhoenixCms.Article)
{:ok,
 [
   %PhoenixCms.Article{
     author: "author-1@phoenixcms.com",
     ...
     ...
  ]
}

iex(2)> PhoenixCms.Repo.Http.all(PhoenixCms.Content)
{:ok,
 [
   %PhoenixCms.Content{
     content: "Lorem ipsum dolor sit amet, consectetur adipiscing elit",
     id: "feature_4",
     ...
     ...
  ]
}

It works as expected! Let's continue with the fake adapter definition:

# lib/phoenix_cms/repo/fake.ex

defmodule PhoenixCms.Repo.Fake do
  @moduledoc false

  alias PhoenixCms.{Article, Content, Repo}

  @behaviour Repo

  @impl Repo
  def all(Content) do
    {:ok,
     [
       %PhoenixCms.Content{
         id: "contents-1",
         # ...
       },
       %PhoenixCms.Content{
         id: "contents-2",
         # ...
       }
     ]}
  end

  def all(Article) do
    {:ok,
     [
       %Article{
         id: "article-1",
         # ..
       },
       %Article{
         id: "article-2",
         # ..
       }
     ]}
  end

  def all(_), do: {:error, :not_found}

  @impl Repo
  def get(entity, id) when entity in [Article, Content] do
    with {:ok, items} <- all(entity),
         {:ok, nil} <- {:ok, Enum.find(items, &(&1.id == id))} do
      {:error, :not_found}
    end
  end

  def get(_, _), do: {:error, :not_found}
end

This is the most basic implementation that we can make. However, since we are not going to be using it during the tutorial, it's good enough.

We are missing something tho, which is configuring the adapter module we want to use in our different environments:

# config/config.exs

use Mix.Config

# ...

# Repo configuration
config :phoenix_cms, PhoenixCms.Repo, adapter: PhoenixCms.Repo.Http

I don't want to extend the articles more than the necessary, so we are not going to be implementing any tests. Nevertheless, if you're going to write your own, add the fake adapter to the test environment configuration to prevent unnecessary HTTP requests against Airtable:

# config/test.exs

use Mix.Config

# ...

# Repo configuration
config :phoenix_cms, PhoenixCms.Repo, adapter: PhoenixCms.Repo.Fake

And that's all for today. In the next part, we will focus on the front-end, rendering the Phoenix.LiveView pages using the data returned by the repository, and eventually discovering that this is not a very good solution, and thinking about a more performant one. In the meantime, you can check the end result here, or have a look at the source code.

Happy coding!

Live demo Source code

Headless CMS fun with Phoenix LiveView and Airtable (pt. 1)

2020-07-03T05:19:00Z

This post belongs to the Headless CMS fun with Phoenix LiveView and Airtable series.

Live demo Source code

Last year, I built a static website for a friend of mine who has an Italian restaurant, which has been vital for her business since then. The first year the site performed really well businesswise. However, issues have started to appear as my friend needed to change the site's content to showcase the new season products, menu, and schedule. I started implementing the changes by hand, but we suddenly realized that this was not convenient at all, since she needed to change the content on the fly without having to rely on me. After considering many different possibilities, I solved the problem by implementing a simple solution in a single weekend, thanks to Phoenix and Airtable. Here's who I did it.

Introduction

In this new tutorial, we are going to be building a headless content management system consisting of two main elements, which are an Airtable base and a Phoenix application powered by LiveView. Let's take a more in-depth look at them:

Airtable

Airtable is a really cool service based on spreadsheets grouped in bases, that act as a database. Using a very intuitive and friendly UI, it lets you design your own data structures, add attachments, handle relationships between different tables, design different data views for your tables, and much more. It also exposes all the data through a very convenient API, which is crucial for this tutorial.

But why do we need such a service? The main reason is that we want to externalize the content of our website and the management of it by our users, letting us focus only on building the presentation layer, which is a simple Phoenix application. And Airtable is perfect for that.

Here you can find the base that we are using for this tutorial, which consists of two tables:

The contents table

This table stores the sections of the main page. Its structure is straightforward, structuring the data with the following fields:

id: the ID of the section.
position: An auto-increment number that specifies the order of the section within the page.
type: the type of the section, which can have three different values:
- hero: for a hero section containing a big title and subtitle.
- text_and_image: for sections that have some text and an image.
- feature: for a section that has a list of items with some text and an icon.
title: the title of the section.
content: the main content of the section. It can store HTML.
image: the main image of the section. Stored as an attachment.
styles: any additional styles that we want to add to the section.

The articles table

This table stores all the blog articles of our website, and each article consists of the following attributes:

slug: the SEO friendly slug for the article.
title: the main title of the article.
description: the article's excerpt.
image: the main image of the article. Stored as an attachment.
content: the main content of the article. It can store HTML.
author: the email of the article's author.
published_at: the publication date of the article.

The Phoenix application

The presentation layer of our CMS is a Phoenix application which supervision tree looks something like this:

Let's have a closer look at some of its components.

Cache

Airtable has a limit of five requests per second for free accounts, so we can't just send requests on every page visit, because if we have a lot of users, the API is likely going to start returning rate limit errors. Using Erlang's ETS to store successful responses from the API helps to prevent rate limiting issues. Once a page is mounted, the data is taken from the cache instead of performing an HTTP request. However, this is not enough, because we need to keep the cache data synced with the latest changes in Airtable.

Synchronizer

To keep the cache data in sync with Airtable, it spawns a GenServer process, which periodically makes requests to the API every second, updating its stored data if needed and broadcasting the new data to the live views using PubSub. This way, we limit the number of interactions with the API to two requests per second, no matter the number of users are currently visiting our site.

LiveView

Instead of using regular views and templates, the application takes advantage of Phoenix LiveView, subscribing to specific PubSub topics, updating its data when needed, and refreshing its content on the browser without requiring a reload from the user.

Here you can see the three views that it has:

A main landing page that renders the sections stored in the contents table. These sections are the main hero section, some image and text sections, and a feature list section.
A blog page, listing all the articles stored in the articles table.
An article detail page that renders a complete article.

And I think this is all for the introduction. In the next part, we will start building the Phoenix application and implementing an HTTP client to retrieve the data stored in Airtable. In the meantime, you can check the end result here, or have a look at the source code.

Happy coding!

Live demo Source code

elmcsspatterns.io

2020-04-27T07:00:00Z

The inspiration

A couple of weeks ago, while searching for a convenient CSS pattern that I needed to implement in one of my elm projects, I stumbled upon csslayout.io and felt in love with it instantly. His author, phuocng, has done a fantastic job not only collecting such a massive collection of patterns but making them easy to find and implement.

The motivation

csslayout.io is the kind of resource that I like to keep handy while working on my front-end, as I tend to write my styles without using any CSS framework whatsoever. Moreover, for the last year, I only use elm-css to generate the CSS, which feels to me like Sass but functional and statically typed, which is just awesome. Unfortunately, there aren't any similar resources for elm-css that I'm aware of, so I couldn't resist writing my version, collecting and sharing the common patterns that I often use.

On the other hand, I've been looking for an excuse to play around with elm-spa lately, which I think is going to be one of the next big things in the elm ecosystem. If you are not familiar with elm-spa, it basically consists of an elm library and a JS client, which automagically takes care of generating all the boilerplate regarding elm single-page applications, letting you focus on the fun part. His author, ryannhg, is doing an excellent job, keep it up!

The result

So having csslayout.io as inspiration, and elm-css + elm-spa as motivation, I have started working on elmcsspatterns.io. It is still an early version, and I will probably change everything now and then, but if you are into elm and elm-css I hope you find it useful, and if not, I hope it makes you want to try them :)

Happy coding!

Site Source code

Dynamic base path for an Elm SPA

2019-06-06T07:00:00Z

While building an Elm SPA dashboard, I faced the following problem. In the local development environment, the URL to access it is http://localhost:1234, which is Parcel's default URL, and the Elm SPA gets mounted in /, so Elm navigation handles as expected any internal routes like /projects or /tasks. The problem came while deploying it into production because the base URL didn't match the root path. In other words, it looked something like https://nifty-minsky-538aab.netlify.com/private/admin/ where /private/admin/ was the base path for the application, and this path could change depending on the environment, which made Elm navigation tricky, especially while parsing URLs to get the current route. I wanted to avoid using URL fragments, so this is how I solved it.

The <base> HTML element

First of all, I needed a way to prepend the dynamic base URL to any of the internal Elm routes. After some research I found the handy <base> HTML element, which specifies the base URL to use for all relative URLs contained within a document. This means that if you set <base href="http://localhost:1234/private/admin/">, any relative link I would add like <a href="projects">Projects</a>, automatically points to http://localhost:1234/private/admin/projects, and that was exactly what I was looking for.

<!DOCTYPE html>
<html lang="en">
  <head>
    <base href="{{ BASE_URL }}">
  </head>
  <body>
    <main></main>
    <script src="./js/index.js"></script>
  </body>
</html>

Setting the href value for the current environment is easy using environment variables, depending on the technology stack you are using.

Passing the base path to the Elm application

Now that I had a way to set the base URL to all the internal links of the application, I needed a way to make Elm aware of this base path, which was pretty straightforward using flags and the baseURI property:

import { Elm } from '../src/Main.elm';

const basePath = new URL(document.baseURI).pathname;

Elm.Main.init({
  node: document.querySelector('main'),
  flags: { basePath },
});

baseURI basically returns the document's location, unless you set <base> in which case it always returns the value set. I only needed the path, therefore taking it from URL(document.baseURI).pathname and passing it to the Elm.Main.init function as a flag.

Elm routing and the base path

I always like defining the application routes as soon as possible, which helps me understand how to structure it. Moreover, in this particular case, routing was the source of the issue and the solution ifself, so let's have a look at the Route module I implemented:

-- src/Route.elm

module Route exposing
    ( Route(..)
    , fromUrl
    , toString
    )

import Url exposing (Url)
import Url.Parser as Parser exposing (Parser)


type Route
    = Home
    | Projects
    | Tasks
    | NotFound


parser : Parser (Route -> b) b
parser =
    Parser.oneOf
        [ Parser.map Home Parser.top
        , Parser.map Projects (Parser.s "projects")
        , Parser.map Tasks (Parser.s "tasks")
        ]

-- ...

This is pretty much the standard way of defining routes and their parser in Elm, and there wasn't any particular change I had to implement to make it work. However, both fromUrl and toString functions needed to be slightly different than usual:

-- src/Route.elm

-- ...


fromUrl : String -> Url -> Route
fromUrl basePath url =
    { url | path = String.replace basePath "" url.path }
        |> Parser.parse parser
        |> Maybe.withDefault NotFound


toString : Route -> String
toString route =
    case route of
        Home ->
            ""

        Projects ->
            "projects"

        Tasks ->
            "tasks"

        NotFound ->
            "not-found"

fromUrl takes a basePath and a Url parameter and returns a Route. The first parameter is the flag passed to the Elm application on its initialization, and to get the corresponding Route, we only need to remove basePath from its path and parse it as usually. Bear in mind, that this only works with URLs built using the <base> element set in the document header. Last but not least, the toString function offers a convenient way of building a relative path for a given Route.

Gluing it all together

Having the parsing of URLs solved, building the rest of the application was quite simple. Let's take a look at some of the implementation details:

-- src/Main.elm

module Main exposing (main)


import Browser exposing (Document)
import Browser.Navigation as Navigation
import Html as Html exposing (Html)
import Route exposing (Route)
import Url exposing (Url)

-- MODEL


type alias Flags =
    { basePath : String }


type alias Model =
    { flags : Flags
    , navigation : Navigation
    }


type alias Navigation =
    { key : Navigation.Key
    , route : Route
    }


init : Flags -> Url -> Navigation.Key -> ( Model, Cmd Msg )
init ({ basePath } as flags) url key =
    ( { flags = flags
      , navigation =
            { key = key
            , route = Route.fromUrl basePath url
            }
      }
    , Cmd.none
    )

-- ...

-- MAIN


main : Program Flags Model Msg
main =
    Browser.application
        { init = init
        , update = update
        , view = view
        , subscriptions = subscriptions
        , onUrlRequest = UrlRequested
        , onUrlChange = UrlChange
        }

I usually store the flags passed to the application in the model using a custom type named Flags, which in this particular example only contains basePath. I also like to store a Navigation custom element which contains a Navigation.Key, necessary for navigating, and the current route. The init function is using the previously defined Route.fromUrl function to set the current route from the browser's URL and the basePath flag. However, it also needs to set it every time the URL changes:

-- src/Main.elm

-- ...

-- UPDATE


type Msg
    = UrlRequested Browser.UrlRequest
    | UrlChange Url


update : Msg -> Model -> ( Model, Cmd Msg )
update msg ({ flags, navigation } as model) =
    case msg of
        UrlRequested urlRequest ->
-- ...

        UrlChange url ->
            ( { model
                | navigation =
                    { navigation
                        | route = Route.fromUrl flags.basePath url
                    }
              }
            , Cmd.none
            )

And this is how I created the navigation links using the Route.toString function:

Html.div
    []
    [ Html.a
        [ Html.href <| Route.toString Route.Home ]
        [ Html.text "Home" ]
    , Html.a
        [ Html.href <| Route.toString Route.Projects ]
        [ Html.text "Projects" ]
    , Html.a
        [ Html.href <| Route.toString Route.Tasks ]
        [ Html.text "Tasks" ]
    ]

And that's it; everything worked like a charm. Being honest, I tried different approaches before getting to this solution, including custom Url parsers, which is something difficult to understand for me. Have you faced the same issue? If so, I hope this solution helps you on the next occasion, and if you have solved differently, please share it :)

Happy coding!

Live demo Source code