Lotechnica

Configuring a dev environment for Phoenix package development

2025-05-05T00:00:00-07:00

When developing a hex package intended to be used with Phoenix projects, such as a component library, plug, on_mount callback, or any other piece of shared code, Phoenix needs some extra configuration to enable the great developer experience we have come to expect. By default, a Phoenix app will only reload its own code, and only prompt the browser to live reload when you make changes to files in that project. So this makes it challenging if you're making changes to a mix package in one directory, and testing with a Phoenix project in another.

This is a quick guide on a few ways I've found to configure Phoenix that makes developing a package much nicer, and enables live reloading for faster iterations while developing.

Read on for some explanation behind how Phoenix does code and live reloading, or skip to one of these solutions to bootstrap your project with.

Option 1: Creating two "sibling" projects, your package and a Phoenix project, side by side in the same parent directory. (Similar to a poncho project.)
Option 2: Creating a Phoenix project as a subdir inside your package dir, all as a monorepo.
Option 3: Using an umbrella project and publishing your library straight from its "app/" subdirectory. (Not recommended)

Understanding the Phoenix Developer Experience

There are two modules that give us the live reload experience when developing Phoenix applications, so that changes to source files appear in the browser without having to restart our server or reload the page.

For these explanations it is assumed you have a regular mix package called "my_library" in one directory, and a phoenix application called "my_app" in another directory.

The Code Reloader

The first is Phoenix.CodeReloader, which is included in the :phoenix package. This is a plug that you can see in your "my_app_web/endpoint.ex" file as:

plug Phoenix.CodeReloader

Whenever it handles a request, it uses mix to (re)compile your app before processing the rest of the plug pipeline.

It's important to note a few things:

This only recompiles elixir code (.ex and .*eex files for example), similar to running recompile in iex. It will not recompile assets such as CSS or JS, which should be done by your asset tools like esbuild and tailwind.
This will not recompile dependencies listed in your "mix.exs".
This will only recompile things when it gets a request, it does not happen as soon as you save the source file.
By default, it will only recompile your :my_app app (unless you're in an umbrella in which case it will recompile all apps in your "apps/".)
By default, in a new Phoenix app, this is only enabled in the dev mix environment. It is possible to enable it in prod, but it requires :mix being available. See the source code for a hint.

Buried deep in the docs is a clue how we can configure Phoenix.CodeReloader to recompile any app whenever it gets a request. This functionality was probably created to serve umbrella apps, but even if we're not running in an umbrella, we can use it to configure any app in our BEAM server to be reloaded on request. I'll go into more detail below in one of the demonstrated options, but in short, just give it a list of apps you want to recompile in "config/dev.exs":

# Configure Phoenix.CodeReloader to reload my_library as well as this application on every request
config :my_app, MyAppWeb.Endpoint, reloadable_apps: [:my_app, :my_library]

After this, you should see

==> my_library
Compiling 1 file (.ex)
Generated my_library app

in your logs whenever you make a request and there has been a change in the :my_library app.

The Live Reloader

The second piece of the puzzle is Phoenix.LiveReloader which is contained in the :phoenix_live_reload package. This uses the :file_system package to monitor changes to files on disk and then notify the browser when it detects changes. You can see this in your "my_app_web/endpoint.ex" file as:

socket "/phoenix/live_reload/socket", Phoenix.LiveReloader.Socket
plug Phoenix.LiveReloader

A few notes about this module:

This works for any files that change, as long as it matches the list of patterns. So css, js, images, etc, as well as elixir and template files.
It works with the browser by injecting an iframe into every page served by Phoenix that contains custom JS to connect back to the server via a Phoenix Channel websocket connection.
When a file changes, a message is pushed to the iframe code. If it's CSS, then it will attempt to inject it into the existing DOM. Otherwise, it will force a page reload.
This is only for dev environments, and should never be used in production.

Again, buried deep in the docs is a note about configuring Phoenix.LiveReloader to work with umbrella apps that we can use to configure it for broader use in non-umbrella apps. It has an option, :dirs, which is a list of directories for it to watch for changes, which is passed directly to the FileSystem module. The default is to watch [""] which is interpreted as the current working dir (CWD).

So if you want to watch more than one directory, such as your app and a dependency, that can be done with this, for example:

# Configure Phoenix.LiveReloader to watch the CWD and `my_library` paths for file changes
config :phoenix_live_reload, :dirs, ["", "../my_library"]

This will cause FileSystem to monitor all changes in the current directory as well as a sibling "my_library/" directory. However, in a typical phoenix project, notifications are filtered so the browser doesn't reload unless a source file for our app is modified. You wouldn't want your browser constantly reloading every time some build or cache files changed, after all. The default config looks something like this:

# Watch static and templates for browser reloading.
config :my_app, MyAppWeb.Endpoint,
  live_reload: [
    patterns: [
      ~r"priv/static/(?!uploads/).*(js|css|png|jpeg|jpg|gif|svg)$",
      ~r"priv/gettext/.*(po)$",
      ~r"lib/my_app_web/(?:controllers|live|components|router)/?.*\.(ex|heex)$",
    ]
  ]

It's important to note that when Phoenix.LiveReloader gets a notification of a file changing in any of the configured :dirs, the path given to it will be an absolute path.

So if we want to notify the browser when any file changes in our "my_library" dir that it's now monitoring, then we just have to add this pattern:

~r"lib/my_library.*(ex|heex)$"

This will match any path ending with this pattern, such as "/path/to/my_library/lib/my_library.ex".

Debugging tip

If you're having trouble with your patterns, you can add some debug prints to "deps/phoenix_live_reload/lib/phoenix_live_reload/channel.ex" in the handle_info({:file_event, ...}, ...) function, then force recompile it with mix deps.compile --force phoenix_live_reload.

This function is called by FileSystem every time it detects a changed file, and it's here where the server pushes a message to the browser.

Example Setups

Now let's look at 3 different ways of configuring a development setup. In each example our goal is to be able to publish a module called MyLibrary in a package named :my_library. The hex package format basically requires a "mix.exs" with a few keys defined, a list of dependencies that are required by your package, and of course some code, typically in "lib/". These dependencies will be installed along with your package when someone does a mix deps.get after adding your package to their list of dependencies, so you should be careful with these dependency requirements.

See the hex publishing guide for more information than we'll cover here.

Option 1: Sibling projects

In this scenario, we'll have a "my_library/" dir in the same parent directory as a "my_app/" Phoenix project to develop with. So the two directories will be siblings, or side-by-side.

Pros:

Your package directory will be clean of anything not directly related to it.
You can publish your Phoenix project as a separate git repository for other people to check out.
You can open these two projects in two different IDEs or text editors, and it's less likely to confuse your LSP.

Cons:

You have to open two separate IDEs, or configure an overall Workspace to edit both projects in the same IDE.
You have to commit changes to both git repositories separately, making synchronization of the two a little more involved.
Other developers have to clone both repos to set up their dev environment, and update both when there are changes.

Create the library `MyLibrary`

Change to the parent dir you want both projects to live in, such as "~/projects".
Create a new mix project with mix new my_library.
Change to the new dir with cd my_library.
Add :phoenix_live_view as a dependency to "mix.exs" so we can use Phoenix.Component. We'll use the spec ~> 1.0 to say our library works with any version of LiveView that is >= 1.0.0 and < 2.0.0.
```
defp deps do
  [
    {:phoenix_live_view, "~> 1.0"}
  ]
end
```
Run mix deps.get

Edit "lib/my_library.ex" to be:

defmodule MyLibrary do
  use Phoenix.Component

  def hello(assigns) do
    ~H"""
    <div>
      Hello, <%= @name %>!
    </div>
    """
  end
end

Edit "mix.exs" to add required hex package info in the main project/0 function:

def project do
  [
    app: :my_library,
+   description: "A library of Phoenix LiveView components",
    version: "0.1.0",
    elixir: "~> 1.18",
    start_permanent: Mix.env() == :prod,
    deps: deps(),
+   package: [
+     licenses: ["MIT"],
+     links: %{"GitHub" => "https://github.com/myuser/my_library"}
+   ]
  ]
end

Run mix hex.build to make sure that it will build a package tarball without error.

Create the phoenix app `MyApp` / `MyAppWeb`

Change back to the parent directory (e.g. "~/projects") that "my_library/" is in with cd ...
Create a new Phoenix project with mix phx.new my_app. Customize with whatever switches you want (e.g. --no-ecto, --no-mailer, etc). Say yes to "Fetch and install dependencies?".
Change to the new dir with cd my_app.
Add the library to the list of deps in "mix.exs" as a relative path. This will tell mix to use our sibling library instead of trying to fetch it from hex.pm.
```
defp deps do
  [
    {:my_library, path: "../my_library"}
  ]
end
```
We'll create a new action in the default PageController just to test with. In "lib/my_app_web/controllers/page_controller.ex" create a new function:
```
def hello(conn, %{"name" => name}) do
  render(conn, "hello.html", name: name)
end
```
Create the template "lib/my_app_web/controllers/page_html/hello.html.heex" to go along with this, which is where we're going to call our new hello component from MyLibrary:
- Phoenix < 1.8
```
<MyLibrary.hello name={@name} />
```
- Phoenix >= 1.8
```
<MyAppWeb.Layouts.app flash={@flash}>
  <MyLibrary.hello name={@name} />
</MyAppWeb.Layouts.app>
```
Add this new action to your router, right below the one for :home:
```
get "/hello/:name", PageController, :hello
```
Start up Phoenix with mix phx.server (or iex -S mix phx.server if you prefer.)
Navigate to http://localhost:4000/hello/bob. You should see "Hello, bob!".

Configuring Phoenix for Live Reloading

At this point you should have a working page that uses the function component from your library. However, make any kind of change to the function component in "my_library/lib/my_library.ex" and you'll notice a couple problems.

The browser doesn't reload on its own when you make a change.
Even hitting reload doesn't cause the change to show up.
Only stopping and starting the server will make the change appear in the browser.

So first let's get Phoenix.CodeReloader working so that MyLibrary is recompiled when the server gets a request. This is as easy as adding this to our "my_app/config/dev.exs".

# Configure Phoenix.CodeReloader to reload my_library as well as this application on every request
config :my_app, MyAppWeb.Endpoint, reloadable_apps: [:my_app, :my_library]

Restart your Phoenix server, make another change to "my_library/lib/my_library.ex", then refresh your browser. You should see the change appear, and a line about compiling "my_library" in your console.

Now that's done, let's get the browser to do the refreshing automatically, by configuring Phoenix.LiveReloader. Again, in "my_app/config/dev.exs" add this line:

# Configure Phoenix.LiveReloader to watch the CWD and `my_library` paths for file changes
config :phoenix_live_reload, :dirs, ["", "../my_library"]

This will cause Phoenix.LiveReloader to now watch our sibling "../my_library" directory in addition to the phoenix "my_app" directory. However, it will most likely be filtering out these notifications, so we'll need to also add a line to the patterns so that it includes these changes as valid.

# Watch static and templates for browser reloading.
config :my_app, MyAppWeb.Endpoint,
  live_reload: [
    patterns: [
      ~r"priv/static/(?!uploads/).*(js|css|png|jpeg|jpg|gif|svg)$",
      ~r"priv/gettext/.*(po)$",
      ~r"lib/my_app_web/(?:controllers|live|components|router)/?.*\.(ex|heex)$",
+     ~r"my_library/lib/.*(ex|heex)$"
    ]
  ]

Restart your Phoenix server one last time, again make a change to "my_library/lib/my_library.ex", and as soon as you save you should see the browser refresh and the change appear on its own. If you don't, then recheck these steps and also review the sections above on Phoenix.CodeReloader and Phoenix.LiveReloader

Bonus: Dynamically referencing the `:my_library` dependency

In the example above, we just hard-coded the path to MyLibrary in the "my_app/mix.exs" file.

defp deps do
  [
    {:my_library, path: "../my_library"}
  ]
end

This is probably fine if only you or your team will using the "my_app/" project. However, if you want to also publish "my_app/" as a demonstration of your library, for other people to clone, then we can do better.

Check for the existence of the sibling directory

Instead of just assuming "../my_library/" will exist, we can check for it, then use a relative path. But if it doesn't exist, we can instead configure it as a normal hex package. So if someone checks out "my_app/" but not "my_library/", then it will just pull the latest version from hex.pm.

First define a new function in "my_app/mix.exs".

defp my_library() do
  if File.exists?(Path.expand("../my_library/mix.exs", __DIR__)) do
    {:my_library, path: Path.expand("../my_library", __DIR__)}
  else
    {:my_library, ">= 0.0.0"}
  end
end

Then just call this in your deps list, instead of hard-coding the :my_library dependency.

defp deps do
  [
    my_library()
  ]
end

Finally, we'll configure things in "config/dev.exs" to use the discovered path for live reloading.

# Configure Phoenix.CodeReloader to reload my_library as well as this application on every request
config :my_app, MyAppWeb.Endpoint, reloadable_apps: [:my_app, :my_library]

# Configure Phoenix.LiveReloader to watch the CWD and `my_library` paths for file changes
my_library_path = Mix.Project.deps_paths()[:my_library]
config :phoenix_live_reload, :dirs, ["", my_library_path]

# Watch static and templates for browser reloading.
config :my_app, MyAppWeb.Endpoint,
  live_reload: [
    patterns: [
      ~r"priv/static/(?!uploads/).*(js|css|png|jpeg|jpg|gif|svg)$",
      ~r"priv/gettext/.*(po)$",
      ~r"lib/my_app_web/(?:controllers|live|components|router)/?.*\.(ex|heex)$",
      ~r"my_library/lib/.*(ex|heex)$"
    ]
  ]

Use an optional file to determine the dependency

There are some that would view "reaching" outside a project's working directory, to somewhere else on the developers filesystem, as a violation. I'm not going to argue one way or another, but leave it up to you. The advantage of the previous solution is that it "just works", but it requires checking for a directory next to the "my_app/" checkout. Instead, we could require the user to specifically set the path for the dependency by putting the path in a ".my_library_path" file in the root of "my_app/". If they don't we fall back to loading the dependency from hex.pm.

Same as the above example, but instead define my_library/0 as this function, which reads the path from the expected file, and uses it to set the relative dependency.

defp my_library() do
  path_file = Path.expand("./.my_library_path", __DIR__)

  if File.exists?(path_file) do
    path = path_file |> File.read!() |> String.trim()
    {:my_library, path: Path.expand(path, __DIR__)}
  else
    {:my_library, ">= 0.0.0"}
  end
end

And then echo "../my_library" > .my_library_path or similar to configure it to use the local sibling directory.

Finally configure "config/dev.exs" exactly the same as above.

Option 2: A subdir Phoenix project

In this scenario, we'll have one parent directory named "my_library/" which will be our package we intend to publish. Then to test it, we'll create a new Phoenix project in a subdirectory, so that it will be "my_library/my_app/".

Pros:

Your reference Phoenix application that you develop your library with is in the same git repository.
Synchronizing changes between the two projects is simple since you commit changes to the same monorepo.
You can edit everything in one IDE. (See fix below for code formatting.)

Cons:

Your package repository will be larger, with files and commits for both your library and phoenix app.
Having two elixir projects open in the same IDE might cause some confusion or issues, depending on your IDE.
You can't clone just the phoenix project like in Option 1, you have to clone everything (barring some advanced git techniques.)

Create the library `MyLibrary`

Create a new mix project with mix new my_library.
Change to the new dir with cd my_library.
Add :phoenix_live_view as a dependency to "mix.exs" so we can use Phoenix.Component. We'll use the spec ~> 1.0 to say our library works with any version of LiveView that is >= 1.0.0 and < 2.0.0.
```
defp deps do
  [
    {:phoenix_live_view, "~> 1.0"}
  ]
end
```
Run mix deps.get

Edit "lib/my_library.ex" to be:

defmodule MyLibrary do
  use Phoenix.Component

  def hello(assigns) do
    ~H"""
    <div>
      Hello, <%= @name %>!
    </div>
    """
  end
end

Edit "mix.exs" to add required hex package info in the main project/0 function:

def project do
  [
    app: :my_library,
+   description: "A library of Phoenix LiveView components",
    version: "0.1.0",
    elixir: "~> 1.18",
    start_permanent: Mix.env() == :prod,
    deps: deps(),
+   package: [
+     licenses: ["MIT"],
+     links: %{"GitHub" => "https://github.com/myuser/my_library"}
+   ]
  ]
end

Run mix hex.build to make sure that it will build a package tarball without error.

Create the phoenix app `MyApp` / `MyAppWeb`

While still in "my_library/", create a new Phoenix project with mix phx.new my_app. Customize with whatever switches you want (e.g. --no-ecto, --no-mailer, etc). Say yes to "Fetch and install dependencies?".
Change to the new dir with cd my_app.
Add the library to the list of deps in "mix.exs" as a relative path. This will tell mix to use our library one directory up instead of trying to fetch it from hex.pm.
```
defp deps do
  [
    {:my_library, path: "../"}
  ]
end
```
We'll create a new controller just to test with. In "lib/my_app_web/controllers/page_controller.ex" create a new function:
```
def hello(conn, %{"name" => name}) do
  render(conn, "hello.html", name: name)
end
```
Create the template "lib/my_app_web/controllers/page_html/hello.html.heex" to go along with this, which is where we're going to call our new hello component from MyLibrary:
- Phoenix < 1.8
```
<MyLibrary.hello name={@name} />
```
- Phoenix >= 1.8
```
<MyAppWeb.Layouts.app flash={@flash}>
  <MyLibrary.hello name={@name} />
</MyAppWeb.Layouts.app>
```
Add this new action to your router, right below the one for :home:
```
get "/hello/:name", PageController, :hello
```
Start up Phoenix with mix phx.server (or iex -S mix phx.server if you prefer.)
Navigate to http://localhost:4000/hello/mary. You should see "Hello, mary!".

Fixing Code Formatting

You might be getting a warning from mix format or your LSP that the files you modify in "my_library/my_app/" are not included in "my_library/.formatter.exs". You can fix that by adding this to "my_library/.formatter.exs":

[
  inputs: ["{mix,.formatter}.exs", "{config,lib,test}/**/*.{ex,exs}"],
+ subdirectories: ["my_app"]
]

You might also then get a message about "Unknown dependency :ecto_sql given to :import_deps". This is because "my_library/my_app/.formatter.exs" is trying to load formatting options from a dependency it has, but Elixir LS is looking for those dependencies in "my_library/mix.exs". The only fix I know for this is to just add that as a dependency in "my_library/mix.exs", but mark it as only: [:dev, :test] so that it isn't a dependency of your library (unless that actually is the case!) You have to include it in :test since that's the environment that ElixirLS compiles in, and :dev for running mix format on the command line from "my_library/".

defp deps do
  [
    {:ecto_sql, "~> 3.0", only: [:dev, :test]}
  ]
end

Then run mix deps.get followed by mix deps.compile in "my_library/". You may also need to restart your IDE, or even quit your IDE, delete ".elixir_ls" and restart. But hopefully then formatting will work.

Configuring Phoenix for Live Reloading

The browser doesn't reload on its own when you make a change.
Even hitting reload doesn't cause the change to show up.
Only stopping and starting the server will make the change appear in the browser.

So first let's get Phoenix.CodeReloader working so that MyLibrary is recompiled when the server gets a request. This is as easy as adding this to our "my_library/my_app/config/dev.exs".

# Configure Phoenix.CodeReloader to reload my_library as well as this application on every request
config :my_app, MyAppWeb.Endpoint, reloadable_apps: [:my_app, :my_library]

Now that's done, let's get the browser to do the refreshing automatically, by configuring Phoenix.LiveReloader. Again, in "config/dev.exs" add this line:

# Configure Phoenix.LiveReloader to watch the CWD and `my_library` paths for file changes
config :phoenix_live_reload, :dirs, ["", "../"]

This will cause Phoenix.LiveReloader to now watch our parent directory in addition to the phoenix "my_app" directory. However, it will most likely be filtering out these notifications, so we'll need to also add a line to the patterns so that it includes these changes as valid.

# Watch static and templates for browser reloading.
config :my_app, MyAppWeb.Endpoint,
  live_reload: [
    patterns: [
      ~r"priv/static/(?!uploads/).*(js|css|png|jpeg|jpg|gif|svg)$",
      ~r"priv/gettext/.*(po)$",
      ~r"lib/my_app_web/(?:controllers|live|components|router)/?.*\.(ex|heex)$",
+     ~r"lib/my_library.*(ex|heex)$"
    ]
  ]

Option 3: Umbrella

In this scenario, we'll create a new Phoenix project as an Umbrella, then create our library as one of the "apps" under it, in the "apps/" directory. When publishing our library, we will publish from "apps/my_library" instead of the root of the project, like the other options.

This is not recommended

I have to say that I do not recommend this option. Umbrellas just make things extra complex for little to no gain. I would suggest using one of the other options first, and only use this option if you really think you want to, or already have an umbrella project. This is really just here because it's possible and I feel remiss not mentioning it.

Pros:

You have one monorepo with many possibly unrelated apps in it, including your package.
Synchronizing changes between the projects is simple since you commit both to the same monorepo.
You can edit everything in one IDE.

Cons:

Umbrellas make things much more complicated.
Your library is buried inside what looks like a Phoenix Umbrella project, making it difficult for people to understand.
Your docs published to hexdocs will need to have extra work done to make source links work properly.
Running your library through CI/CD will be more complicated.
Your package repository will be larger, with files and commits for both your library and phoenix app.
Your IDE might not support umbrella projects properly.
You can't clone just the phoenix project like in Option 1, you have to clone everything (barring some advanced git techniques.)

Create the phoenix umbrella

Create a new Phoenix umbrella project with mix phx.new my_app --umbrella. Customize with whatever switches you want (e.g. --no-ecto, --no-mailer, etc.) Say yes to "Fetch and install dependencies?".
Change to the new dir with cd my_app_umbrella.

Create the library `MyLibrary`

Change to the apps dir with cd apps.
Create a new mix project with mix new my_library. Mix will detect that this is inside of an umbrella and configure the library accordingly.
Change to the new dir with cd my_library.
Add :phoenix_live_view as a dependency to "apps/my_library/mix.exs" so we can use Phoenix.Component. We'll use the spec ~> 1.0 to say our library works with any version of LiveView that is >= 1.0.0 and < 2.0.0.
```
defp deps do
  [
    {:phoenix_live_view, "~> 1.0"}
  ]
end
```
Run mix deps.get from the parent directory of the umbrella.

Edit "apps/my_library/lib/my_library.ex" to be:

defmodule MyLibrary do
  use Phoenix.Component

  def hello(assigns) do
    ~H"""
    <div>
      Hello, <%= @name %>!
    </div>
    """
  end
end

Edit "apps/my_library/mix.exs" to add required hex package info in the main project/0 function:

def project do
  [
    app: :my_library,
+   description: "A library of Phoenix LiveView components",
    version: "0.1.0",
    build_path: "../../_build",
    config_path: "../../config/config.exs",
    deps_path: "../../deps",
    lockfile: "../../mix.lock",
    elixir: "~> 1.18",
    start_permanent: Mix.env() == :prod,
    deps: deps(),
+   package: [
+     licenses: ["MIT"],
+     links: %{"GitHub" => "https://github.com/myuser/my_library"}
+   ]
  ]
end

Run mix hex.build in "apps/my_library/" to make sure that it will build a package tarball without error.

Configure the "MyAppWeb" umbrella app

Add the library to the list of deps in "apps/my_app_web/mix.exs" as an umbrella path. This tells mix that the dependency is another app in the same umbrella.
```
defp deps do
  [
    {:my_library, in_umbrella: true}
  ]
end
```
We'll create a new controller just to test with. In "apps/my_app_web/lib/my_app_web/controllers/page_controller.ex" create a new function:
```
def hello(conn, %{"name" => name}) do
  render(conn, "hello.html", name: name)
end
```
Create the template "apps/my_app_web/lib/my_app_web/controllers/page_html/hello.html.heex" to go along with this, which is where we're going to call our new hello component from MyLibrary:
- Phoenix < 1.8
```
<MyLibrary.hello name={@name} />
```
- Phoenix >= 1.8
```
<MyAppWeb.Layouts.app flash={@flash}>
  <MyLibrary.hello name={@name} />
</MyAppWeb.Layouts.app>
```
Add this new action to your router, right below the one for :home:
```
get "/hello/:name", PageController, :hello
```
Change back to the root directory of the umbrella, that has the "apps/" subdir.
Start up Phoenix with mix phx.server (or iex -S mix phx.server if you prefer.)
Navigate to http://localhost:4000/hello/arnold. You should see "Hello, arnold!".

Configuring Phoenix for Live Reloading

At this point you should have a working page that uses the function component from your library. You should even be able to make a change to "apps/my_library/lib/my_library.ex", refresh the page, and see the changes. This is because Phoenix.CodeReloader detects that it's running in an umbrella app, and by default will recompile all umbrella apps every time it gets a request. This is the only real advantage of umbrellas (but still not worth it, imo.) However, you still have to refresh the page to see the changes, so let's get Phoenix.LiveReloader working as well so it auto-refreshes.

In "config/dev.exs" add these lines, per the example in the docs.

root_path = __ENV__.file |> Path.dirname() |> Path.join("..") |> Path.expand()

# Configure Phoenix.LiveReloader to watch the `my_app_web` and `my_library` paths for file changes
config :phoenix_live_reload, :dirs, [
 Path.join([root_path, "apps", "my_library"]),
 Path.join([root_path, "apps", "my_app_web"])
]

This will cause Phoenix.LiveReloader to now watch both of those app directories for changes. However, it will most likely be filtering out these notifications, so we'll need to also add a line to the patterns so that it includes these changes as valid.

# Watch static and templates for browser reloading.
config :my_app, MyAppWeb.Endpoint,
  live_reload: [
    patterns: [
      ~r"priv/static/(?!uploads/).*(js|css|png|jpeg|jpg|gif|svg)$",
      ~r"priv/gettext/.*(po)$",
      ~r"lib/my_app_web/(?:controllers|live|components|router)/?.*\.(ex|heex)$",
+     ~r"lib/my_library.*(ex|heex)$"
    ]
  ]

Restart your Phoenix server, make a change to "apps/my_library/lib/my_library.ex", and as soon as you save you should see the browser refresh and the change appear on its own. If you don't, then recheck these steps and also review the sections above on Phoenix.CodeReloader and Phoenix.LiveReloader

Publishing your package

Now that you have a working dev environment set up, with live reloads, it's time to get cracking on your package! Remember to tweak the patterns if you need to, based on what files you want to cause live reloads when they change. This will depend on your package and what it does!

Once you're ready, follow the publishing guide to set any other options you want, build your docs, etc. With all 3 options you should be able to cd into your "my_library/" directory and mix hex.publish.

For Option 3, the umbrella, you might have some extra work. I have never done it myself, but you can refer to this post on the Elixir Forums for help, and the linked example monorepo for a few published hex packages for guidance.

If you find an issue with this guide, please hit the "Contact Me" link in the footer.

Happy packaging!

The Stock Market Game, made easy with Phoenix LiveView

2025-03-03T00:00:00-07:00

A couple years ago my wife, who is an 8th grade US History teacher, came to me for help with a spreadsheet to simulate the US stock market leading up to the crash in 1929. The spreadsheet was cumbersome, very complicated for the students, and a major pain for her to grade. The point of the lesson is to lead the students to buying stocks on huge margins to recreate the conditions that led to the collapse, but the math for calculating loans and debt was too complicated. And this wasn't a math class.

So I did the completely normal thing, and decided to implement it as a web app using Phoenix LiveView. The advantages were many, for the students and for my wife. What used to be a major time sink for her that took up huge portions of her class time, and was too complicated to be fun for the kids, turned into a 5 minute/day activity that the kids absolutely loved. The whole school is abuzz with talk of, "What's your equity dude" and "I think I'm over leveraged." With this being the third year of the lesson, kids have actually been asking all year when they were going to play the stock market game they had heard about for years.

The advantages of doing this proved to be overwhelmingly worth the time it took for me to code it all up. The lesson now just takes 5 minutes a day in each class, during which the students can make their daily buys and sells (though many are refreshing away before class to catch the new prices, with cries of joy and pain heard through the school), and also so they can see a big leaderboard on the projector for their class and the whole school.

With LiveView, they get instant feedback as they work on their worksheet for the day, with many helpful errors, and min/max hints so they know what they can do for each decision. And since all the logic and form validation is done server-side, there is no chance for them to cheat, and I didn't have to implement the same complicated form validation on both front-end and back-end.

The goal of the multi-day lesson is to push them to over-borrow, so that when the market inevitably collapses, they have a first-hand experience of what it was like to get drunk on huge numbers. They all start with merely $1000 each, but they can borrow up to 90% of a stock purchase. (Believe it or not, this was common in the early 20th century!) It is not uncommon to see numbers in the billions and trillions.

We purposely created the leaderboard (something not possible with spreadsheets before!) to foster competition. To get the biggest numbers, you should sell everything every day, if it went up in price, pay back the minimum 10% to the broker, then buy even more at 90% margin. But the kids never know how many days the game is going to go on, so it's all about pushing your luck. The riskiest players stand to make the most money, but also to lose the most.

Also, since this is the third year, previous students get involved a lot. In one example, this year a previous student was seen parading up and down the 8th grade hallway talking about how much they made last year when they won the game. She even had to vary up the stocks this year, because certain stocks have become legend for making billionaires.

So today brought the year's lesson to a close, and I thought I'd write up this quick retrospective. Today she "crashed" the market, but reducing all share prices by less than 100%. Every stock ends up higher than what it started as, but due to the successful psychological cues, the ease of borrowing money, and the push to be at the top of the leaderboards, so far every year the market has truly crashed.

Inevitably, the winners with a positive balance at the end will ask, "When am I getting paid?"

To which my wife can respond, "I can pay you as soon as every else pays me what they owe!"

Note for Students!: If you happen to find this article, and you're a student or future student, just know that all the data in the screenshots has been scrambled or fabricated. Have a good game!

Dynamic configuration for Phoenix Endpoints

2024-04-02T00:00:00-07:00

Update 2024-04-10: This article has been updated with a better Workaround that uses your generated MyappWeb.Endpoint.start_link/1. The original article called directly into Phoenix.Endpoint.Supervisor which was more brittle.

In Phoenix 1.7.11, the Endpoint.init/2 callback was deprecated. If you have defined it, then you will get this warning on startup:

warning: MyappWeb.Endpoint.init/2 is deprecated, use config/runtime.exs instead or pass
         additional options when starting the endpoint in your supervision tree
  (phoenix 1.7.11) lib/phoenix/endpoint/supervisor.ex:36: Phoenix.Endpoint.Supervisor.init/1
  (stdlib 5.2) supervisor.erl:330: :supervisor.init/1
  (stdlib 5.2) gen_server.erl:980: :gen_server.init_it/2
  (stdlib 5.2) gen_server.erl:935: :gen_server.init_it/6
  (stdlib 5.2) proc_lib.erl:241: :proc_lib.init_p_do_apply/3

The suggestions given may work fine for you. However, if you're relying on some other part of the system's supervisor tree to be up and running for the config, then those solutions will not work. The reason is that both happen before the main application supervision tree is started by the main Supervisor. The "config/runtime.exs" file is evaluated as the VM is starting up. And if you try to configure the Endpoint via the list of child_specs for the Supervisor in "lib/myapp/application.ex" then that config will be evaluated before the Supervisor starts any of the children.

Example

An example of this is if you rely on your Ecto Repo to be available to help configure how the Endpoint will run. For example, before 1.7.11 you could have the following in your "lib/myapp_web/endpoint.ex" file:

# Note: this is very simplified to illustrate the inability to use Myapp.Repo at this time.

def init(:supervisor, config) do
  # Get the hostname as configured by the user and stored in the database
  hostname = Myapp.Repo.get_by!(TenantSetting, key: "hostname")

  # Configure the Endpoint to use the hostname in generated URLs
  {:ok, Keyword.put(config, :url, scheme: "http", host: hostname, port: 4000)}
end

You could try to duplicate this functionality by passing options when starting the Endpoint in the main application supervision tree, as suggested, such as:

defmodule Myapp.Application do
  use Application

  def start(_type, _args) do
    children = [
      Myapp.Repo,
      ...

      # Attempt to dynamically configure the endpoint using the 2-tuple argument
      {MyappWeb.Endpoint, endpoint_config()}
    ]

    opts = [strategy: :one_for_one, name: Myapp.Supervisor]
    Supervisor.start_link(children, opts)
  end

  defp endpoint_config() do
    # Try to fetch dynamic run-time configuration from the DB. This will fail!
    hostname = Myapp.Repo.get_by!(TenantSetting, key: "hostname")

    [url: [scheme: "http", host: hostname, port: 4000]]
  end
end

However, since this runs before Myapp.Repo is started, it will fail with the error "(RuntimeError) could not lookup Ecto repo Myapp.Repo because it was not started or it does not exist".

Workaround

If this callback stays deprecated, this is a way to configure the endpoint dynamically before it starts up, but after the rest of the supervision tree has been started, such as after the Ecto Repo is running and available. It works by intercepting the start_link/1 call from the main Supervisor, then starting the Endpoint.Supervisor directly with options derived at that point in time.

First, create a new module: "/lib/myapp_web/endpoint/config.ex" with these contents, but with all occurances of MyappWeb replaced with your actual module name.

defmodule MyappWeb.Endpoint.Config do
  @moduledoc """
  This will start up the MyappWeb.Endpoint as normal, but only after injecting some custom
  configuration that is retrieved from the database after the rest of the application stack
  has started. This needs to be placed in the list of Application children after
  MyappWeb.Repo and any other dependent children to work correctly.

  Prior to Phoenix 1.7.11, this configuration was done in the `init/1` callback of
  MyappWeb.Endpoint. But that callback is now deprecated. The deprecation suggests to
  "use config/runtime.exs instead or pass additional options when starting the endpoint in
  your supervision tree" but the problem with both of these suggestions is that they happen
  before the rest of the supervision tree is started. "runtime.exs" is evaluated during
  vm boot, and the list of children and their options is evaluated as soon as the
  supervisor starts, but before children are started.

  So this mimics the original behavior by intercepting the `start_link/1` call from the
  Supervisor to fetch the configuration options at that point. This will happen after the
  previous children in the main Supervision tree are already started.
  """

  def child_spec(opts) do
    %{
      id: MyappWeb.Endpoint,
      start: {MyappWeb.Endpoint.Config, :start_link, [opts]},
      type: :supervisor
    }
  end

  def start_link(opts) do
    dynamic_config(opts)
    |> MyappWeb.Endpoint.start_link()
  end

  defp dynamic_config(opts) do
    hostname = Myapp.Repo.get_by!(TenantSetting, key: "hostname")
    Keyword.put(opts, :url, scheme: "http", host: hostname, port: 4000)
  end
end

Then change your main supervision tree in "lib/myapp/application.ex" to include this module in the list of children instead of the original Endpoint module:

defmodule Myapp.Application do
  use Application

  def start(_type, _args) do
    children = [
      Myapp.Repo,
      ...
      # Replace "MyappWeb.Endpoint" with:
      MyappWeb.Endpoint.Config
    ]

    opts = [strategy: :one_for_one, name: Myapp.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

Caveats

This makes some assumptions about how the use Phoenix.Enpoint macro injects child_spec/1 and start_link/1 into your MyappWeb.Endpoint file. If this is ever changed, this Config module will also need to be updated. However, since these are standard callbacks dictate by Supervisor, it's unlikely that this will happen.

Fix?

I have submitted an issue to the phoenix github repository requesting this deprecation be rolled back.

Prefix normalize.css or sanitize.css before gulp-sass

2018-03-17T00:00:00-07:00

A pretty common setup for front-end design work is to use Sass to compile .scss files into a single .css file. It's also pretty common to use a CSS Reset, such as normalize.css or sanitize.css, to reset your CSS to a common base. For that CSS Reset to work, it needs to be included before any of the app-specific CSS, so you can override it and not the other way around. You could just copy the normalize.css file into your project and link it separately in your HTML, or you could even stick it in your scss/ directory and have Sass process it unnecessarily. But what if you want to just install it from npm and have gulp include it in your output app.css file? Here's a quick and easy way to do that in your gulpfile.js.

This assumes you have a basic gulp setup with:

$ npm install -D gulp gulp-sass gulp-concat

And assuming you have a gulp.task() like the following:

var gulp = require('gulp'),
    sass = require('gulp-sass'),
    concat = require('gulp-concat');

gulp.task('css', function () {
    return gulp.src('scss/**/*.scss')
        .pipe(sass().on('error', sass.logError))
        .pipe(concat('app.css'))
        .pipe(gulp.dest('css/'));
});

Now you install normalize.css with the following:

$ npm install --save normalize.css

which will stick it in your node_modules/ subdirectory. How do you tell Gulp to just include that file, then run Sass on your *.scss files, and output it all to css/app.css? The solution is to create two streams and use the package merge-stream to merge them together before concatenating. So install merge-stream:

$ npm install -D merge-stream

Then create 2 separate streams, one for the normalize CSS (and any other NPM installed CSS files you might want to use), the other for your Sass processed files. Finally merge them, concatenate, and output. Your gulpfile.js will look like the following:

var gulp = require('gulp'),
    sass = require('gulp-sass'),
    concat = require('gulp-concat'),
    merge = require('merge-stream');

gulp.task('css', function () {
    const reset = gulp.src('node_modules/normalize.css/normalize.css');

    const scss = gulp.src('scss/**/*.scss')
        .pipe(sass().on('error', sass.logError));

    return merge(reset, scss)
        .pipe(concat('app.css'))
        .pipe(gulp.dest('css/'));
});

Your css/app.css will now contain the contents of normalize.css followed by all of your CSS compiled by Sass. Whenever you update your normalize.css package from NPM, your output will be automatically updated. Unless you pin a specific version of normalize.css which may be a good idea depending on your project.

Postgres Full-Text Search With Django

2017-05-03T00:00:00-07:00

Django has added support for Postgres's built-in full-text searching as of version 1.10. This is a great alternative to a heavier search system such as elasticsearch or SOLR, when we want decent search capabilities without having to setup and maintain another service. Postgres's full-text search is Good Enough for a lot of use cases.

With this quick guide, I'll show how to add full-text searching to a Django application. Simple use-cases are covered pretty well in the Django documentation, so I will focus on a slightly more advanced example that allows searching through multiple fields, including data through relationships, weighting fields differently, adding an index for speed, and methods to keep the search data up-to-date.

It should go without saying that this is focused on Django with a PostgreSQL database back-end. This will not work with SQLite or MySQL. I also assume familiarity with Django and a basic understanding of Postgres.

An example project is available on github that follows this guide.

Models

We will use these models as an example. It's simple data for a Blog-like application consisting of Posts with data both directly included and referenced through relationships. But most importantly, we have data we want to search through that exists as ManyToOne (author) and ManyToMany (tags)relationships.

class Author(models.Model):
    name = models.CharField(max_length=50)


class Tag(models.Model):
    name = models.CharField(max_length=20)


class Post(models.Model):
    title = models.CharField(max_length=50)
    content = models.TextField()
    author = models.ForeignKey(Author)
    tags = models.ManyToManyField(Tag)

We will use this sample data:

jim = Author.objects.create(name="Jim Blogwriter")
nancy = Author.objects.create(name="Nancy Blogaday")

databases = Tag.objects.create(name="Databases")
programming = Tag.objects.create(name="Programming")
python = Tag.objects.create(name="Python")
postgres = Tag.objects.create(name="Postgres")
django = Tag.objects.create(name="Django")

django_post = Post.objects.create(
    title="Django, the western character",
    content="Django is a character who appears in a number of spaghetti "
            "western films.",
    author=jim
)
django_post.tags.add(django)

python_post = Post.objects.create(
    title="Python is a programming language",
    content="Python is a programming language created by Guido van Rossum "
            "and first released in 1991. Django is written in Python. Python "
            "can connect to databases.",
    author=nancy
)
python_post.tags.add(django, programming, python)

postgres_post = Post.objects.create(
    title="What is Postgres",
    content="PostgreSQL, commonly Postgres, is an open-source, "
            "object-relational database (ORDBMS).",
    author=nancy
)
postgres_post.tags.add(databases, postgres)

Building Documents

The first step is going to be building Documents for our posts. Each Document will be a logical representation of a single post, including:

title
content
Author's name
All tag names

Here is an example Django query:

from django.db.models.functions import Concat
from django.db.models import TextField, Value as V
from django.contrib.postgres.aggregates import StringAgg

document=Concat(
    'title', V(' '),
    'content', V(' '),
    'author__name', V(' '),
    StringAgg('tags__name', delimiter=' '),
    output_field=TextField()
)
Post.objects.annotate(document=document).values_list('document', flat=True)

<QuerySet [
  "Django, the western character Django is a character who appears in a
    number of spaghetti western films. Jim Blogwriter Django",
  "Python is a programming language Python is a programming language
    created by Guido van Rossum and first released in 1991. Django is
    written in Python. Python can connect to databases. Nancy Blogaday
    Python Django Programming",
  "What is Postgres PostgreSQL, commonly Postgres, is an open-source,
    object-relational database (ORDBMS). Nancy Blogaday Postgres Databases"
]>

This includes all of our data for each post, separated by spaces.

Search Vectors

So now that we have our documents, we need to convert them into a format that Postgres can index and search through. Postgres calls these Vectors. Django has a class to encapsulate this functionality called SearchVector. A SearchVector can also take a weight, so we'll rewrite our query to create vectors.

from django.contrib.postgres.search import SearchVector
from django.contrib.postgres.aggregates import StringAgg

vector=SearchVector('title', weight='A') + \
       SearchVector('content', weight='C') + \
       SearchVector('author__name', weight='B') + \
       SearchVector(StringAgg('tags__name', delimiter=' '), weight='B')
Post.objects.annotate(document=vector).values_list('document', flat=True)

<QuerySet [
  "'appear':10C 'blogwrit':19B 'charact':4A,8C 'django':1A,5C,20B 'film':17C
    'jim':18B 'number':13C 'spaghetti':15C 'western':3A,16C",
  "'1991':20C 'blogaday':32B 'connect':28C 'creat':11C 'databas':30C
    'django':21C,34B 'first':17C 'guido':13C 'languag':5A,10C 'nanci':31B
    'program':4A,9C,35B 'python':1A,6C,25C,26C,33B 'releas':18C
    'rossum':15C 'van':14C 'written':23C",
  "'blogaday':18B 'common':5C 'databas':15C,20B 'nanci':17B 'object':13C
    'object-rel':12C 'open':10C 'open-sourc':9C 'ordbm':16C
    'postgr':3A,6C,19B 'postgresql':4C 'relat':14C 'sourc':11C"
]>

Each document has been normalized down to a common set of word stems. This includes dropping case, dropping common prefixes and suffixes (like 's', and 'es'), and removing common words like 'a', 'an' and 'the'. The numbers are where in the document it found that stem, and the letter is the weight. If we want to override the configuration setting for how Postgres processes the words, for example to specify a different language, then we can pass an optional config parameter to SearchVector. If not given, then it will use whatever the database is configured to use by default, most likely based on the configured locale.

Performing a search

So now that we have our documents, we can perform a search. The easiest way to do this is to filter on our documents.

vector=SearchVector('title', weight='A') + \
       SearchVector('content', weight='C') + \
       SearchVector('author__name', weight='B') + \
       SearchVector(StringAgg('tags__name', delimiter=' '), weight='B')
Post.objects.annotate(document=vector).filter(document='django')

<QuerySet [<Post: Django, the western character>,
           <Post: Python is a programming language>]>

By default, django will use the plainto_tsquery() function in Postgres to parse the query. The short of this is that it will search for documents that match all of the words. However, instead of a string we can pass a SearchQuery() instance, which can be combined with several boolean operators.

from django.contrib.postgres.search import SearchQuery

query = SearchQuery('django') & SearchQuery('program')
Post.objects.annotate(document=vector).filter(document=query)

If we use a custom config with SearchVector(), then we should use that same config with SearchQuery().

Ranking

Results are most useful if we can rank them, taking into consideration the different weights we've assigned to each part of the document. Django provides the SearchRank class for this purpose.

from django.contrib.postgres.search import SearchVector, SearchQuery, SearchRank
from django.contrib.postgres.aggregates import StringAgg

vector=SearchVector('title', weight='A') + \
       SearchVector('content', weight='C') + \
       SearchVector('author__name', weight='B') + \
       SearchVector(StringAgg('tags__name', delimiter=' '), weight='B')
query = SearchQuery('django')
Post.objects\
    .annotate(document=vector, rank=SearchRank(vector, query))\
    .filter(document=query)\
    .order_by('-rank')\
    .values_list('title', 'rank')

<QuerySet [
  ('Django, the western character', 0.665342),
  ('Python is a programming language', 0.364756)
]>

This gives us the functionality we want, but this isn't the best way to go about it if we care about performance. Every time we run this query the database has to build all the documents, for all rows in the table before it can search and rank them. It's fine for a few rows, but after more than a couple hundred it's going to start slowing down to unacceptable speeds. If our documents only contained data from a single table, we could just build a GIN index, but that won't work in this case since we pull extra data from other tables. So what we really want to do is to pre-calculate all of the documents and store them in the database.

Storing Vectors with SearchVectorField

Django provides a special field that allows us to store pre-calculated vectors in a field called SearchVectorField. We'll add that field to our Post model:

from django.contrib.postgres.search import SearchVectorField

class Post(models.Model):
    ...
    search_vector = SearchVectorField(null=True)

Then we'll migrate this to add the field.

./manage.py makemigrations
./manage.py migrate

Let's update this field manually for now.

from django.contrib.postgres.search import SearchVector, SearchQuery, SearchRank
from django.contrib.postgres.aggregates import StringAgg

vector=SearchVector('title', weight='A') + \
       SearchVector('content', weight='C') + \
       SearchVector('author__name', weight='B') + \
       SearchVector(StringAgg('tags__name', delimiter=' '), weight='B')
for post in Post.objects.annotate(document=vector):
    post.search_vector = post.document
    post.save(update_fields=['search_vector'])

Note: This will issue an UPDATE for each row in the table, which will take forever if our table has a lot of rows. If we are only including fields from a single model in our document, then something like this would be much more efficient:

vector=SearchVector('title', weight='A') + \
       SearchVector('content', weight='C')
Post.objects.update(search_vector=vector)

Django doesn't allow us to use aggregate functions with an update clause, but Postgres does, so if we really wanted, we could issue a query like this to refresh all of the documents at once:

UPDATE blog_post
SET search_vector = document.vector
FROM (
     SELECT post.id,
            setweight(to_tsvector(post.title), 'A') ||
            setweight(to_tsvector(post.content), 'C') ||
            setweight(to_tsvector(author.name), 'B') ||
            setweight(to_tsvector(COALESCE(string_agg(tag.name, ', '), '')), 'B')
              AS vector
     FROM blog_post AS post
     JOIN blog_author AS author ON author.id = post.author_id
     LEFT JOIN blog_post_tags AS post_tags ON post_tags.post_id = post.id
     LEFT JOIN blog_tag AS tag ON tag.id = post_tags.tag_id
     GROUP BY post.id, author.id
   ) AS document
WHERE blog_post.id = document.id;

Searching on search_vector

Now that we have our documents stored, we can easily search on them

from django.db.models import F
from django.contrib.postgres.search import SearchVector, SearchQuery, SearchRank

query = SearchQuery('django')
Post.objects.annotate(rank=SearchRank(F('search_vector'), query))\
    .filter(search_vector=query).order_by('-rank').values_list('title', 'rank')

<QuerySet [
  ('Django, the western character', 0.665342),
  ('Python is a programming language', 0.364756)
]>

Indexing

Now that the documents are stored in a field, we can create a GIN index to speed up searching. With Django 1.11, this is as easy as adding an indexes Meta option to our model, then create and apply a migration.

from django.contrib.postgres.indexes import GinIndex

class Post(models.Model):
    title = models.CharField(max_length=50)
    content = models.TextField()
    author = models.ForeignKey(Author)
    tags = models.ManyToManyField(Tag)
    search_vector = SearchVectorField(null=True)

    class Meta:
        indexes = [
            GinIndex(fields=['search_vector'])
        ]

For Django 1.10, we'd need to create an empty migration then add a RunSQL operation:

migrations.RunSQL(
    "CREATE INDEX blog_post_search_vector_idx ON blog_post USING gin(search_vector)",
    "DROP INDEX blog_post_search_vector_idx"
)

Updating the documents

This is great and all, but as soon as any of the data changes, the documents will be out of date, and the search results will be incorrect. The first way we could address this is with a cron or scheduled task that updates the whole table (like we did above) at regular intervals. This would be a good option for an application that processes a lot of updates, or does its updates in large batches. This way we don't add overhead to every update, and instead can more efficiently update all rows at once.

For other applications that have a slow steady stream of updates, it would be more appropriate to update every time the data changed. The upside is that the search data will always be up to date. The downside is that every update will incur extra overhead as the search_vector is calculated.

A compromise would be to queue up the update to the search_vector as an asynchronous process, so that it's updated fairly quickly, but the updates can still be done as a batch. This is outside the scope of this article, but it shouldn't be too difficult depending on the application architecture.

The best method is going to depend on the particular application. Here are a couple simple ways to update on every save.

Overriding save()

One method to update the document is to override the save() method on Post. With this method, every time the data that search relies on is updated, then the search_vector is also updated alongside it. So search results reflect data changes immediately. This does incur an overhead on every update to the database, though.

First we'll create a custom manager that adds documents to the queryset when we request it, so we can keep things DRY, and only have our SearchVectors defined in one place.

class PostManager(models.Manager):
    def with_documents(self):
        vector = SearchVector('title', weight='A') + \
                 SearchVector('content', weight='C') + \
                 SearchVector('author__name', weight='B') + \
                 SearchVector(StringAgg('tags__name', delimiter=' '), weight='B')
        return self.get_queryset().annotate(document=vector)

Now we'll update our Post model and add our custom manager and a custom save function. The idea here is to save the data to the database, then do a SELECT query that joins all the data together and then save the new search_vector. So each save will result in an UPDATE, SELECT, then another UPDATE.

from django.contrib.postgres.search import SearchVectorField, SearchVector

class Post(models.Model):
    title = models.CharField(max_length=50)
    content = models.TextField()
    author = models.ForeignKey(Author)
    tags = models.ManyToManyField(Tag)
    search_vector = SearchVectorField(null=True)

    objects = PostManager()

    def save(self, *args, **kwargs):
        super().save(*args, **kwargs)
        if 'update_fields' not in kwargs or 'search_vector' not in kwargs['update_fields']:
            instance = self._meta.default_manager.with_documents().get(pk=self.pk)
            instance.search_vector = instance.document
            instance.save(update_fields=['search_vector'])

Also, updates to authors and tags won't trigger the save(), so we'll also add signals for those that force a save() on the Post as well to update the search_vector.

from django.db.models.signals import post_save, m2m_changed
from django.dispatch import receiver

@receiver(post_save, sender=Author)
def author_changed(sender, instance, **kwargs):
    for post in instance.post_set.with_documents():
        post.search_vector = post.document
        post.save(update_fields=['search_vector'])


@receiver(m2m_changed, sender=Post.tags.through)
def post_tags_changed(sender, instance, action, **kwargs):
    if action in ('post_add', 'post_remove', 'post_clear'):
        instance.save()

Now any changes to a Post, Author or if tags are added, removed or cleared will cause the search data to be updated. If a tag is renamed, then we won't pick up on that without creating another signal handler.

Using triggers

It's also possible to install a few triggers in the database that will automatically update the search_vector whenever the data is changed. I won't go into too much detail, but they will look something like this. We can easily add these to a migration that uses a RunSQL operation to install them into our database. The idea is exactly the same as above, but since the database can do it all locally and not have to send data back and forth to Django, it will perform better.

-- Trigger on insert or update of blog.Post
CREATE OR REPLACE FUNCTION post_search_vector_trigger() RETURNS trigger AS $$
BEGIN
  SELECT setweight(to_tsvector(NEW.title), 'A') ||
         setweight(to_tsvector(NEW.content), 'C') ||
         setweight(to_tsvector(author.name), 'B') ||
         setweight(to_tsvector(COALESCE(string_agg(tag.name, ', '), '')), 'B')
  INTO NEW.search_vector
  FROM blog_post AS post
  JOIN blog_author AS author ON author.id = post.author_id
    LEFT JOIN blog_post_tags AS post_tags ON post_tags.post_id = post.id
    LEFT JOIN blog_tag AS tag ON tag.id = post_tags.tag_id
  WHERE post.id = NEW.id
  GROUP BY post.id, author.id;
  RETURN NEW;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER search_vector_update BEFORE INSERT OR UPDATE ON blog_post
  FOR EACH ROW EXECUTE PROCEDURE post_search_vector_trigger();

-- Trigger after blog.Author is updated
CREATE OR REPLACE FUNCTION author_search_vector_trigger() RETURNS trigger AS $$
BEGIN
  UPDATE blog_post SET id = id WHERE author_id = NEW.id;
  RETURN NEW;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER search_vector_update AFTER INSERT OR UPDATE ON blog_author
  FOR EACH ROW EXECUTE PROCEDURE author_search_vector_trigger();

-- Trigger after blog.Post.tags are added, deleted from a post
CREATE OR REPLACE FUNCTION tags_search_vector_trigger() RETURNS trigger AS $$
BEGIN
  IF (TG_OP = 'DELETE') THEN
    UPDATE blog_post SET id = id WHERE id = OLD.post_id;
    RETURN OLD;
  ELSE
    UPDATE blog_post SET id = id WHERE id = NEW.post_id;
    RETURN NEW;
  END IF;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER search_vector_update AFTER INSERT OR UPDATE OR DELETE ON blog_post_tags
  FOR EACH ROW EXECUTE PROCEDURE tags_search_vector_trigger();

-- Trigger after blog.Tag is updated
CREATE OR REPLACE FUNCTION tag_search_vector_trigger() RETURNS trigger AS $$
BEGIN
  UPDATE blog_post SET id = id WHERE id IN (
    SELECT post_id FROM blog_post_tags WHERE tag_id = NEW.id
  );
  RETURN NEW;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER search_vector_update AFTER UPDATE ON blog_tag
  FOR EACH ROW EXECUTE PROCEDURE tag_search_vector_trigger();

Conclusion

Now we have a working application using Postgres full-text search, and once set up, it can be mostly forgotten about. Compared to setting up elasticsearch or SOLR (even with Haystack), this is a breeze, and results are going to be good enough for most applications.

For more information and more features, such as language support, custom stemming, trigrams, accents, etc, please see the following sources:

Official PostgreSQL Full-Text Search Documentation
Official Django Postgres Search Documentation
Postgres full-text search is Good Enough: Great article on the basics of Postgres full-text search.
An example project that this blog post goes along with.

Updates to this article

10/10/17

Fixed SQL queries that were doing INNER joins on Tags, which would cause the queries to return nothing if no tags were applied. Switch to LEFT OUTER JOINs.
Added another sample Trigger to update search_vectors when tag names are changed.

Fix Django's runserver when run under Docker or PyCharm

2016-03-14T00:00:00-07:00

When running Django's runserver management command in a Docker container or under a PyCharm run configuration (in Linux at least), I very often had issues with the process not getting killed in a timely manner. The symptom is that the process doesn't die immediately after it receives a SIGTERM or SIGINT. This is very puzzling, as hitting ctrl-c in a terminal kills it immediately like it's supposed to. There doesn't seem to be any reason for it to not die.

When run in a Docker container, such as part of a Docker Compose service, the symptom is that doing a docker-compose stop takes over 10 seconds to complete. It's really annoying when you're doing a lot of iteration testing of a compose config and want to up/down a lot.

Under PyCharm, the problem is that hitting Stop on the configuration doesn't seem to work at all, and instead you have to hit the skull icon that appears when the stop fails to happen within a second or so. Not that bad, but annoying none-the-less, especially if you're switching from Run to Debug and back on a regular basis.

Either way, it's the little things that are annoying and distracting from getting your real work done. So I dug into it a little, and I think I have a fix.

The quick fix

The quick fix is to specifically listen for SIGINT and SIGTERM in your manage.py, and sys.kill() when you get them. So modify your manage.py to add a signal handler:

def sighandler(signum, frame):
    sys.exit(1)

Then install that signal handler as part of the __main__ code:

if __name__ == "__main__":
    signal.signal(signal.SIGTERM, sighandler)
    signal.signal(signal.SIGINT, sighandler)

So here is an example of a complete manage.py, based on the skeleton version you get from django-admin startproject with Django 1.9.

#!/usr/bin/env python
import os
import sys
import signal


def sighandler(signum, frame):
    sys.exit(1)

if __name__ == "__main__":
    signal.signal(signal.SIGTERM, sighandler)
    signal.signal(signal.SIGINT, sighandler)
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings")

    from django.core.management import execute_from_command_line

    execute_from_command_line(sys.argv)

The why

You know, I've looked into this a little bit, and written all kinds of test code to try to suss out exactly what is happening, but I can't really make heads or tails of it. The problem still persists if you use --noreload so that runserver doesn't use the autoreloader code, which uses threading. So it's not a clash between signals and threads, which is typically a problem. Conversely, when I try to narrow it down to issues with only the parent process getting the signal, or only the child, or it being a process group in the shell but not in Docker or PyCharm, I get conflicting results that don't seem to point to a clear explanation. I think understanding it would require a lot more time, and a lot more expertise into the bowels of CPython and the shell vs Docker's simplified environment or exactly how PyCharm execs your commands from Java. For now, having a quick fix is enough. If anyone knows what's going on, please enlighten me.

A Phoenix Plug for assigning template variables

2015-10-05T00:00:00-07:00

Here's a quick Plug for a Phoenix project that allows you to quickly and easily set template variables (assigns) in either a Pipeline or for all actions in a controller. I used this recently as a way to set a variable, admin: true, for all of my controllers under an /admin scope. I also used it to set a subsection variable for the different controllers that make up the different parts of the site so that my Layout template could mark which section of the site the user was currently on in the site navigation header.

Update 10/14/15: I went ahead and published this as a self-contained hex package, if nothing else than as an exercise in publishing things. It's available on hex.pm. Note that I renamed it to Plug.Assign to better match existing Plug names.

The Plug

The Plug itself is very straightforward since we're able to exploit the fact that Phoenix just uses Plug's assigns map that is part of the Plug.Conn struct as the holding place for template context variables. Thus we can assign our own variables as part of the Plug stack before the Controller action (which itself is just a plug) even executes. So our first step is going to be creating the plug.

As an interface, our plug will accept a map of key/value pairs, exactly like the render/3 function in a Controller/View. So as an example, it will look like:

plug Plugs.Assigns, foo: "This is foo", bar: true, baz: 5

I decided to put it in lib/plugs and named the source file assigns.ex so that my module name will be Plugs.Assigns. This way I can easily copy this plug to other projects. It could even be packaged up as a Hex package and installed as a dependency. So lib/plugs/assigns.ex will look like:

defmodule Plugs.Assigns do
    import Plug.Conn

    def init(assigns), do: assigns

    def call(conn, assigns) do
        Enum.reduce assigns, conn, fn {k, v}, conn ->
            Plug.Conn.assign(conn, k, v)
        end
    end
end

If you're not familiar with the Plug spec, two functions are required, init/1 and call/2. We don't need to do anything with the map at initialization time, so our init/1 function just returns it. However, for each request, our call/2 function will take the assigns map and add it to the conn. This could possibly be written more efficiently by directly manipulating the assigns map in the Plug.Conn struct conn, but I opted to use the interface so that it uses the public Plug.Conn.assign function, making it more compatible with possible future changes to Plug.

Using in a Pipeline

Let's say you want to assign a variable admin to either true if the current request is being routed to an /admin scope. For example, you have something like this defined in your routers.ex:

defmodule Blog.Router do
  use Blog.Web, :router

  pipeline :browser do
    plug :accepts, ["html"]
    plug :fetch_session
    plug :fetch_flash
    plug :protect_from_forgery
    plug :put_secure_browser_headers
  end

  scope "/", Blog do
    pipe_through :browser # Use the default browser stack

    get "/", PageController, :index
    get "/post/:post_id", PageController, :post
  end

  scope "/admin", Blog.Admin do
    pipe_through :browser # Use the default browser stack

    resources "/posts", PostController
    resources "/users", UserController
  end
end

Now let's say you would like to modify your layout/app.html.eex template so that it has "Admin: " prefixed to the page title, as an example.

<title><%= if assigns[:admin], do: "Admin: " %>My Blog!</title>

We would do this by defining a new pipeline in router.ex, and call it :admin:

pipeline :admin do
  plug Plugs.Assigns, %{admin: true}
end

And then we can modify our /admin scope to use it.

scope "/admin", Blog.Admin do
  pipe_through [:browser, :admin] # Use admin stack as well

  resources "/posts", PostController
  resources "/users", UserController
end

Now all requests to any controller under /admin will include the variable admin: true.

Using it to assign a variable for all Controllers

Another way to use this would be to set a variable for all action functions in a Controller. To continue our example, in our web/controllers/admin/post_controller.ex file, we could just add a plug call to our custom Plug near the top.

defmodule Blog.Admin.PostController do
  use Blog.Web, :controller

  alias Blog.Post

  plug :scrub_params, "post" when action in [:create, :update]
  plug Plugs.Assigns, subsection: :posts

  ...
end

Our corresponding template could check if the subsection variable is set to something specific to enable a visual indication of what subsection the user is currently in.

<li class="<%= if assigns[:subsection] == :posts do "active" end %>">
    <a href="<%= post_path(@conn, :index) %>">Posts</a>
</li>

If you wanted to limit the plug to only certain actions, this could be done by adding a when clause. However, do note that when you do that, the map of assigns will no longer be the last predicate, and so you need to wrap it in a map operator, %{}.

plug Plugs.Assigns, %{subsection: :posts} when action in [:index, :show]

Now the subsection variable will only be set to :posts when the user is calling the :index and :show actions.

So there's an example of how powerful the Plug interface is, and how it allows you to add so much functionality with almost no code.

A few notes on Bourbon, Neat, Bitters, Refills

2015-09-29T00:00:00-07:00

When I decided to port this blog to be statically generated, I also decided to redesign it from the ground up. Since I had recently learned about Bourbon Neat, I thought this would be a perfect opportunity to try it out. These are just a few thoughts on the process and result.

Context

First, understand that I'm a veteran Bootstrap user. It's where I learned responsive design, and it's the only solution I've ever used for responsive design. Thus, my experience with Bourbon Neat is going to be in contrast to Bootstrap.

<awesome>Semantic markup!</awesome>

Semantic markup is awesome. This is what attracted me to Bourbon Neat in the first place. I agree with the premise that HTML should be a structured document of information, and not have any styling information in it. This is, of course, impossible to get perfectly, but with Neat, I've come the closest I ever have. As an example, this is the basic document layout for the main index of this blog:

<header id="main-header">
    <ul id="main-menu" role="nav">
        ...
    </ul>
    <section id="masthead">
        <img src="logo.jpg" />
        <h1>Lotechnica</h1>
        <p>Technical writings by Nathan Shafer</p>
    </section>
</header>

<section id="index">
    <section id="articles">
        ...
    </section>
    <aside id="about">
        ...
    </aside>
</section>

<footer id="main-footer">
    <section id="links">
        ...
    </section>
    <section id="copyright">
        ...
    </section>
</footer>

The result is very clean and readable, and does not contain deep nesting of responsive .row and .col-xx-xx classes.

Performance wise, this is good because the page remains flatter, thus making it easier for the browser to render. You can see what this looks like to the right in Firefox's 3D view. Maintenance wise, this means the presentation logic for the different breakpoints are contained in one place, the .scss file. In the future it will be much easier to tweak this layout, and most likely not require any changes to the HTML.

Kitchen sink not included

The other way that Bourbon Neat differs from Bootstrap is in the base philosophy of packaging.

With Bootstrap, you're given everything by default... the grid, pretty buttons, tables, forms, and other neat things like responsive navigation, dropdowns, collapsible panels, wells, jumbotrons, alerts, tooltips, badges, etc, etc. If you don't want all of these things, then you need to remove things from the source files, or use Bootstrap's online customizer. So in other words, you start with everything, and reduce down.

I will often throw a page or site together just using the boostrap.min.css hosted on their CDN with no custom CSS. I just use all the right classes in all the right places, and get a decent looking page as a result. This is great for little projects or prototypes where I need to get something done sooner than later. However, as the project grows, and the design dictates changes, I find that I fight against Bootstrap quite a bit and am constantly overriding things in my stylesheet. The result is that the user has to download a large bootstrap stylesheet, then my stylesheet which just overrides a large portion of the bootstrap stylesheet. This is not good for performance, both for the download sizes and for the time it takes the browser to render the CSS.

With Bourbon Neat, you're given almost nothing by default. It is pure SCSS, and there is no pre-compiled version. Instead you @import the mixins you need into your stylesheet as you need it, and the resulting CSS is added. So you start with nothing and build up. The downside here, is that you don't get anything by default. Your page will look like vanilla user-agent styling. If you want pretty buttons, forms, tooltips, a responsive nav menu, etc, you're going to have to roll it yourself.

The Bourbon project does have a couple aids for this, though. The Bitters module will add a basically fancy reset and help you start your project with a bunch of typography, color, form, table, list and grid settings that you can then customize from there, but they are nowhere near as comprehensive as Bootstrap's. For some other elements, you could also look in the Refills project, as they have a bunch of components you can copy and paste into your HTML, SCSS and JS source to get some pre-built elements, such as nav, tabs, accordions, tooltips, and even some things that default Bootstrap doesn't have, such as Parallax areas and sliding panels.

Efficient results as a side effect

So with fact that you're only adding what you actually use to your stylesheet, you will end up with a much more efficient stylesheet for your site. And since you're using Sass to build it, it could be just one stylesheet. For this blog I decided to put all CSS for every page into one file, and didn't break it up by page-type or anything. This allows the browser to leverage caching better between page views. However, even though I did that, the stylesheet still came out at only 15KB, minified.

Compare this to 143KB for the minified bootstrap.min.css and bootstrap-theme.min.css that I usually throw in a project. The result is that the entire style for this whole site is only 10% of the size of just the bootstrap stylesheet. To get close to that with Bootstrap I would have needed to customize it and go through everything to throw out what I wasn't using. I haven't actually done that, but I'm willing to guess that the customized bootstrap.min.css file would still be larger than 15KB.

You want this when?

Without all the components and convenience classes of Bootstrap, you will be responsible for styling everything. You don't get anything for "free." For example, up in the header of this site is a menu of categories. With bootstrap, I just need to mark the HTML up with the proper classes in the proper layout, and boom, I have a responsive menu that collapses down to a single hamburger-button that expands the menu for mobile. With the Bourbon suite I have nothing like that. Refills currently has a nav component, but it is not responsive. So you're on your own to implement this feature. (I ended up just hiding the menu for mobile because I determined that it wasn't necessary and just muddied up the layout. All the same links are in the footer.)

The end result is that you will need to take more time designing all of the components to the site. The Bourbon suite of projects is not meant for whipping together quick projects or prototypes. It is also not very friendly for non-designers to work with, who just want to focus on the code generating the site and not worry so much about designing every little detail of the final result.

Overall more efficient for larger or more custom projects

If you are going to be working on a larger project, where you know you're going to be customizing almost everything or don't want it to look like a Bootstrap site, then the Bourbon suite is the way to go. For one, you won't be fighting against existing style rules the whole time, and instead have a cleaner base to work up from. Also, your end result is going to be a much more minimal set of CSS for the browser to download and process. And with all the other benefits, such as semantic HTML and cleaner separation of HTML and CSS, the project will most likely be easier to maintain in the long run. So in that aspect, Bourbon, Neat, Bitters and Refills are the way to go.

Final thoughts

As with most things, there is "a right tool for job." The bourbon suite does not replace Bootstrap, or even come close to competing with it for simple projects or prototypes that need to get something decent done in a short amount of time. Bourbon is also not a good fit for programmers that just need to get something done that looks decent enough.

However, for anything else, Bourbon, Bourbon Neat, Bitters and Refills will overall give you a much better result and be a better base to build on. Control freaks like me will really appreciate knowing exactly what every line of CSS is doing to the page.

In the end it all depends on the type of project you're working on, and I'm really happy that there is a choice.

If you want to see how I put it all together for a real, working example of Bourbon, you can check out the source for this blog on GitHub. Look at theme.scss, which is compiled down to theme.css.

I have released my first mobile app, AnswerCast

2015-08-28T00:00:00-07:00

I’m happy to finally announce the release of my little pet-project, AnswerCast. The app is now available on Google Play and the iTunes App Store.

In short it’s an app that enhances the experience when playing certain family/board games such as The Game of Things, Balderdash, Loaded Questions and others. Submit your answers with your smartphone or tablet to a Google Cast device which then scrambles them and displays them on your TV.

I first came up with this years ago and even wrote a prototype that allowed you to text your answers to a phone number, and then display them on a TV with a connected laptop. However, this solution was not practical for the average user. So then recently I realized I could use (or abuse) a Chromecast to do the same thing while giving me an opportunity to learn how to develop native apps, and hence the project was born.

So if you happen to be in the small niche of people who like these kinds of board games, have a Google Cast device such as a Chromecast or Nexus Player, and all of your players have an Android or iOS phone or tablet, then this is the app for you!

More info

AnchorSprite: Setting an anchor point on any Sprite

2013-06-12T00:00:00-07:00

Having done a lot of work with matrices in Gideros recently, during a discussion at the Gideros forums it occurred to me that with the Matrix class that I recently covered, I could possibly address an oft-requested feature of Gideros. The problem is that in Gideros, all Sprite objects have an anchor point of [0,0] that you can't change. So all transformations such as rotation and scaling are relative to the top left point of the Sprite. The one exception to this is with Bitmap objects, where there is a setAnchorPoint() method. But this doesn't help for other Sprite based objects, such as TextFields or Shapes.

So I whipped together a class called AnchorSprite, which when added to your project, adds a setAnchorPoint() and getAnchorPoint() method to every Sprite object and all objects that inherit from Sprite. Further, bowerandy had the great idea to make this support completely inactive until you set an anchor, that way you could avoid any overhead on sprites that still had the default anchor point.

So I'm going to explain a little bit about how this class works. I'm going to skip over most of the matrix stuff since I already covered that in detail.

Intercepting Sprite transform methods

The first step is to intercept all calls to transform methods of the Sprite class. This includes setPosition(), setScale(), setRotation() as well as all the other specific ones, setX(), setScaleY(), etc. We also need to intercept set() itself. We have to do this, because we don't want the Sprite class to actually change anything. Instead, we just take the desired changes and save them in private member variables. For example, here is how we intercept set().

-- Intercept Sprite's set and get functions for position, scale and rotation
function AnchorSprite:set(key, value)
    if value then
        if key == "x" then
           self._positionX = value
        elseif key == "y" then
           self._positionY = value
        elseif key == "scaleX" then
           self._scaleX = value
        elseif key == "scaleY" then
           self._scaleY = value
        elseif key == "rotation" then
           self._rotation = value
        else
           self._anchorBackup.set(self, key, value)
        end
        self:_applyTransforms()
    end
end

You'll notice that this is in the AnchorSprite class and not the Sprite class just yet. We don't want to intercept it until an anchor point is set, which I'll show in a bit.

So just to show another example, this is what our intercepted setPosition() method will look like.

function AnchorSprite:setPosition(x, y)
    self:set("x", x)
    self:set("y", y)
end

Applying the transforms

The thing to notice is that we're not doing anything with the values being passed in, just storing them in private member variables. This is because we need to basically keep a stack of transforms in memory, so that we can apply them all whenever any of them change. This is done in the _applyTransformsmethod, which is called every time set() is called.

-- New function to apply all transforms whenever anything changes
function AnchorSprite:_applyTransforms()

This function will do all the calculations needed to apply the transforms. It starts from scratch with a brand new Matrix.

    -- Create a new identity matrix
    local matrix = Matrix.new()

We then apply that new matrix to the sprite, thus resetting it back to completely default location, rotation and scale. This is so we can get an accurate width and height, so we know how to apply the anchor, which is specified as a percentage of the dimensions of the Sprite. We could forgo this and just have the anchor set as a specific pixel location, but I chose to do it this way to mirror the way that the Bitmap:setAnchorPoint() works. Plus it means you can change the dimensions of the Sprite and always know that the anchor point will be in the place you expect without having to calculate the anchor point yourself.

 -- Zero ourselves out so we can get accurate width and height
 self:setMatrix(matrix)

 -- Calculate the actual position of the anchor
 local anchorOffsetX = self._anchorPointX * self:getWidth()
 local anchorOffsetY = self._anchorPointY * self:getHeight()

The next set of operations apply our transformations to the Matrix, effectively concatenating them. They should all be pretty self explanatory if you've read my matrix overview. In short, each is in turn concatenated onto the matrix much like characters can be concatenated together to form a string. First we set the position, then rotation and scale.

 -- set position
 matrix:translate(self._positionX, self._positionY)

 -- concatenate rotation and scale matrices
 matrix:rotate(self._rotation)
 matrix:scaleX(self._scaleX)
 matrix:scaleY(self._scaleY)

And finally the magic part, which is to offset by the negative of the anchor point.

 -- concatenate offset to new origin in modified coordinate space
 matrix:translate(-anchorOffsetX, -anchorOffsetY)

Since we concatenate the anchor point translate last, it is applied to the new coordinate space that is a result of the previous concatenations, and not based on the original coordinate space. Thus if you have an anchor point of [.5,.5] and rotate clockwise 45 degrees, the last translate will move the whole sprite along that 45 degree angle, or basically straight up. If you were to try this with the original Sprite:setPosition, it would move based on the original coordinate space, and so end up moving the sprite up and to the left, which is not what you wanted.

Lastly we apply our built-up matrix with setMatrix(), thus causing all the transforms to get applied. This will affect the current sprite as well as any children in the scene graph.

 -- Apply the new matrix
 self:setMatrix(matrix)

This is our finished _applyTransforms() function.

-- New function to apply all transforms whenever anything changes
function AnchorSprite:_applyTransforms()
    -- Create a new identity matrix
    local matrix = Matrix.new()

    -- Zero ourselves out so we can get accurate width and height
    self:setMatrix(matrix)

    -- Calculate the actual position of the anchor
    local anchorOffsetX = self._anchorPointX * self:getWidth()
    local anchorOffsetY = self._anchorPointY * self:getHeight()

    -- set position
    matrix:translate(self._positionX, self._positionY)

    -- concatenate rotation and scale matrices
    matrix:rotate(self._rotation)
    matrix:scaleX(self._scaleX)
    matrix:scaleY(self._scaleY)

    -- concatenate offset to new origin in modified coordinate space
    matrix:translate(-anchorOffsetX, -anchorOffsetY)

    -- Apply the new matrix
    self:setMatrix(matrix)
end

Optimization

So the last bit is to make this code only get put into place when an anchor point is set. This way, after you add AnchorPoint.lua to your project, it doesn't do anything to any Sprites by default. Only those that you set an anchor point for with setAnchorPoint will start to intercept the transform methods, thus resulting in less overhead. This is why our functions so far have been in the AnchorSprite class. First step is to add the setAnchorPoint() method to the Sprite class along with our private member variables for storing the transforms.

Sprite._anchorSupport = false
Sprite._anchorBackup = {}
Sprite._anchorPointX = 0
Sprite._anchorPointY = 0
Sprite._positionX = 0
Sprite._positionY = 0
Sprite._scaleX = 1
Sprite._scaleY = 1
Sprite._rotation = 0

-- New functions in Sprite for setting the anchor
function Sprite:setAnchorPoint(x, y)
    self._anchorPointX = x
    self._anchorPointY = y or x

    if x ~= 0 or y ~= 0 and not self._anchorSupport then
    self:_enableAnchorSupport()
    elseif x == 0 and y == 0 and self._anchorSupport then
    self:_disableAnchorSupport()
    end
end

As you can see, this mainly calls a function called _enableAnchorSupport() that does all the work of modifying the Sprite class accordingly.

function Sprite:_enableAnchorSupport()
    self._anchorSupport = true

    -- Get current values
    self._positionX = self:getX()
    self._positionY = self:getY()
    self._scaleX = self:getScaleX()
    self._scaleY = self:getScaleY()
    self._rotation = self:getRotation()

    -- Override Sprite functions with AnchorSprite versions
    for k,v in pairs(AnchorSprite) do
        if type(v) == "function" then
            --print("AnchorSprite", k, v)
            if self[k] then
                -- Backup existing function
                self._anchorBackup[k] = Sprite[k]
            end

            -- Replace with AnchorSprite version
            self[k] = v
        end
    end

    self:_applyTransforms()
end

This initializes our transform variables, then does a bit of magic of modifying the Sprite class itself. If you're unfamiliar with how Lua works, the thing to remember is that almost everything in Lua is a table. All classes are just a table, and values of that table can be variables or references to functions. So after backing up the original reference in _anchorBackup (a table), we just replace the reference in Sprite's table with the one in AnchorSprites table. So now functions like AnchorSprite:_applyTransforms from earlier are now Sprite:_applyTransforms. The methods in AnchorSprite are never meant to be called while they're in AnchorSprite, and would actually throw weird errors. We make backups so that if the anchor is reset to [0,0], we can called _disableAnchorSupport() and return Sprite to the way it was.

Result

In the end, the result is that to the user, all we've done is added Sprite:setAnchorPoint() and Sprite:getAnchorPoint(), and everything just works. A sample project is available to see how it can be implemented and to show the results. To use it, simply include AnchorSprite.lua and Matrix.lua into your project.

Of course, this is hopefully just a stopgap solution until Gideros builds this functionality in. Until then, this is one way to get around the limitation. Let me know if you find it useful.

The beauty of using matrices to apply transformations to objects

2013-05-17T00:00:00-07:00

I have recently done a lot of work with matrix transformations, and I found the information on them either sparse, or over-complicated for what I needed. I had to pull together information from many places, split between mathematics focused information, programming reference, and forum posts. So I decided to hopefully write a simple but complete guide on how to understand matrix transformations for the purposes of applying them to computer graphics.

Enter the matrix

When dealing with 2D or 3D graphics programming, there will generally be a couple ways to apply transformations, such as translation (position), rotation, scaling, reflection and skewing (shearing.) There usually is a direct way, with calls to things like setPosition() and setRotation(). However, most frameworks also have another way, using matrix transformations. These are a more powerful way to affect the transformations of the object, and are computationally very quick as they only require addition and multiplication of a few values.

One big advantage is that you can string multiple transformations together one at a time, and they will be applied in the order you add them. To give a practical example, consider the SVG graphics standard for saving vector graphics. SVG files are XML files that use entities such as <rect>, <circle> and <path>. Each of those defines through its attributes how to draw a rectangle, circle, or arbitrary path of lines and curves. Each of those entities can define a "transform" attribute which will contain transformations to apply to the shape right before its drawn. Sometimes it will just be a matrix definition.

So that's simple enough, however the tricky bit comes into play when you consider that SVG also supports a <g> tag, short for "Group." These entities can contain any number of other groups or shapes in them. So you could end up with a deep hierarchy of shapes within groups within groups within groups... Each group can also define a "transform" attribute, to be applied to all groups and shapes within it, and so a shape could have many individual transforms or matrices applied to it. If you were to write an SVG parser (which I did recently and will be posting about soon,) then you would need to be able to concatenate all of those transformations together. With matrices, it's easy.

What are matrices?

A matrix is simply a two dimensional array of numbers. They are used in all kinds of scientific fields. For our purposes we just need to understand how they are defined and how to multiply (concatenate) them together. Matrices in graphics are always defined as $m \times n$ where $m$ is the number of rows, and $n$ is the number of columns. So a $2 \times 3$ matrix looks like this.

$$ A = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \\ \end{bmatrix} $$

The method in which you access the individual elements of the matrix will vary toolkit to toolkit. One common method, and the one we'll use, is to reference them by $\text{m}_{mn}$ where 'm' is just the character 'm', '$m$' is the row number and '$n$' is the column number.

$$ A = \begin{bmatrix} \text{m}_{11} & \text{m}_{12} & \text{m}_{13} \\ \text{m}_{21} & \text{m}_{22} & \text{m}_{23} \\ \end{bmatrix} $$

2D and 3D transformations matrices

First, we need to understand what dimensions our matrices are going to be in. Computer graphics generally use matrices of a size $m \times n$ where each of $m$ and $n$ are your dimension size plus one. So if you're working with 2D graphics, then you will be using 3x3 matrices, and 3D graphics use 4x4 matrices. For example, here is a 3x3 matrix used in 2D graphics and a 4x4 matrix used in 3D graphics.

3x3 2D Affine Matrix	4x4 3D Affine Matrix
$$ \begin{bmatrix} \text{m}_{11} & \text{m}_{12} & \text{tx} \\ \text{m}_{21} & \text{m}_{22} & \text{ty} \\ 0 & 0 & 1 \\ \end{bmatrix} $$	$$ \begin{bmatrix} \text{m}_{11} & \text{m}_{12} & \text{m}_{13} & \text{tx} \\ \text{m}_{21} & \text{m}_{22} & \text{m}_{23} & \text{ty} \\ \text{m}_{31} & \text{m}_{32} & \text{m}_{33} & \text{tz} \\ 0 & 0 & 0 & 1 \\ \end{bmatrix} $$

You'll notice that there are a few elements named differently, specifically, tx, ty, and tz. These elements always deal with the translation portion, so it's common to just name them that. For simplicity, we'll be dealing with just 2D 3x3 matrices from here on out, however everything here will also apply for 3D transforms, it's just you have an extra row and column to handle to work on the z-axis.

Homogeneous Matrices

The next thing we need to understand is why we have an extra row in each matrix. Transformation matrices are always "homogeneous." This is so the math is nice and tidy, and allows us to multiply them together and to individual points, which we'll get to later. We don't really need to understand it, thankfully. Just always know that you will have an extra row of zeroes and a one. The transformations we apply with these homogeneous matrices are called "affine transformations."

Identity Matrices

Just a quick note that there exists a matrix that is basically the equivalent of the number $1$, in that if you multiply a given matrix by an identity matrix, you'll always end up with the given matrix as the result. These identity matrices are used as the foundation of transform matrices, and are the reason the last row is zeroes followed by a one. An identity matrix is always zeroes except a "stripe" of ones going through it diagonally from top left to bottom right.

3x3 2D Affine Identity Matrix	4x4 3D Affine Identity Matrix
$$ \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{bmatrix} $$	$$ \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \\ \end{bmatrix} $$

Multiplication/Concatenation

To go any further we first need to tackle the most complicated part of the new functionality that we're going to need, multiplication. Two matrices can be multiplied together to create a new matrix. The process is a pretty straightforward set of operations, but there are a few rules we have to abide by.

Rule One: Order Matters!

Matrix multiplication is non-commutative, which means that unlike normal numbers, multiplying the matrices $A \times B$ does not get the same result as $B \times A$. For this reason, it's often called concatenation instead of multiplication. We'll just call it multiplication from now on. But always remember that the order matters. Generally we take our existing matrix as the operand on the left, and multiply our new matrix as the operand on the right, thus concatenating it onto the end. If it helps, think of it as concatenating strings together. If you were building a sentence one word at a time, you would always concatenate the newest word onto the end of the string as you went, otherwise it would be unreadable. It's basically the same here.

Rule Two: Matrices have to have the same inner dimensions

To multiply two matrices together, their inner dimensions must match. This is best explained by an example.

$$ A \times B = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \\ \end{bmatrix} \times \begin{bmatrix} 1 & 2 \\ 3 & 4 \\ 5 & 6 \\ \end{bmatrix} $$

This is a valid multiplication because the first matrix, A, is a 2x3 matrix, and B is a 3x2 matrix. So their inner dimensions, meaning the number of columns of A (3) is equal to the number of rows of B (3).

$$ 2 \times \fbox{$3 \times 3$} \times 2 $$

An invalid multiplication would look like:

$$ A \times B = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \\ \end{bmatrix} \times \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \\ \end{bmatrix} $$

This is because the inner dimensions don't match

$$ 2 \times \fbox{$3 \times 2$} \times 3 $$

And now you know why transformation matrices are always homogeneous. If they didn't have that extra row of zeroes and a one, you wouldn't be able to multiply them together. A 3x3 matrix can always be multiplied by a 3x3 matrix in either direction because their inner dimensions will always match.

Edit: Ok, so that's not the main reason they're homogeneous, just a nice by-product. I'm definitely not a mathematician, but as kyz on reddit pointed out, this excellent explanation was written by one, if you're really curious about the math behind it.

Rule Three: Resulting matrix size

Just a quick note that the resulting matrix after the operation will always be the same number of rows as the left matrix, and the same number of columns as the right matrix. So multiplying our 2x3 matrix by the 3x2 matrix from above will result in a 2x2 matrix. For our purposes, multiplying a 3x3 homogeneous transformation matrix by another will always result in a 3x3 result matrix. Another reason they are always homogeneous!

Multiplication

The actual act of multiplying the two matrices together involves finding the dot-product of each row of the first matrix by each column of the second matrix. As a reminder, a dot-product is where you multiply each element of an array by the corresponding element of a second array, then add the results together. A dot product of two arrays would look like.

$$ (a1, a2, a3) \cdot (b1, b2, b3) = (a1 \times b1 + a2 \times b2 + a3 \times b3) $$

With matrix multiplication we just do this multiple times for each element of the final matrix.

$$ \begin{bmatrix} \text{a}_{11} & \text{a}_{12} & \text{a}_{tx} \\ \text{a}_{21} & \text{a}_{22} & \text{a}_{ty} \\ 0 & 0 & 1 \end{bmatrix} \cdot \begin{bmatrix} \text{b}_{11} & \text{b}_{12} & \text{b}_{tx} \\ \text{b}_{21} & \text{b}_{22} & \text{b}_{ty} \\ 0 & 0 & 1 \end{bmatrix} $$

Would be

$$ \begin{bmatrix} \text{a}_{11} \times \text{b}_{11} + \text{a}_{12} \times \text{b}_{21} + \text{a}_{tx} \times 0 & \text{a}_{11} \times \text{b}_{12} + \text{a}_{12} \times \text{b}_{22} + \text{a}_{tx} \times 0 & \text{a}_{11} \times \text{b}_{tx} + \text{a}_{12} \times \text{b}_{ty} + \text{a}_{tx} \times 1 \\ \text{a}_{21} \times \text{b}_{11} + \text{a}_{22} \times \text{b}_{21} + \text{a}_{ty} \times 0 & \text{a}_{21} \times \text{b}_{12} + \text{a}_{22} \times \text{b}_{22} + \text{a}_{ty} \times 0 & \text{a}_{21} \times \text{b}_{tx} + \text{a}_{22} \times \text{b}_{ty} + \text{a}_{ty} \times 1 \\ 0 \times \text{b}_{11} + 0 \times \text{b}_{21} + 1 \times 0 & 0 \times \text{b}_{12} + 0 \times \text{b}_{22} + 1 \times 0 & 0 \times \text{b}_{tx} + 0 \times \text{b}_{ty} + 1 \times 1 \end{bmatrix} $$

Application

So now that we understand the basics of the matrices we're going to be working with, let's delve into some example code. For this blog I'm going to be using the Gideros framework for a couple reasons. First, it's what I've been using to develop a lot lately, and thus it's what I've been using matrices in, but also because it doesn't have any real built-in matrix functionality. It has a built-in Matrix class, but you can only get and set the matrix of a given sprite. It doesn't have any functionality for manipulating those matrices like most frameworks do. So we're going to write our own.

However, credit is due to ar2rsawseen from the Gideros community for putting together a first draft of matrix functions in his GiderosCodingEasy. His code is where I started, however there were a few issues that prevented them from working properly in what I would consider a normal or standard fashion. He had originally ported that code from an ActionScript project that used matrix transforms in a non-standard way. So I modified his code to work in the same way that I saw in other frameworks. I guess that's what really helped me understand what was going on as I fixed those issues. He has since integrated my changes into his code.

Matrix functionality in Gideros

In Gideros, there is a Sprite class that every visible object is based on, whether it be text, shapes, bitmaps or whatever. This sprite class has a method for getting the current transform matrix, Sprite:getMatrix(), and a method for setting a new transform matrix, Sprite:setMatrix(matrix), which is then applied to the sprite. getMatrix() returns an object of the Matrix class, and setMatrix() takes an object of the Matrix class. The Matrix class itself has many methods for getting and setting the individual elements of the 3x3 matrix, such as getM11(), setM22(), getTx(), setTy(), etc. What it is missing is any mathematical operations, specifically multiplication/concatenation, or any shortcuts to applying transforms, such as translate, rotate, scale or skew. So those are the parts we're going to fill in, and along the way gain a better understanding of how these all work together.

We're going to do this by adding our own methods to the built-in Matrix class. This is possible in Lua because nothing is protected, functions are a first-class type, and almost everything is a table (dictionary hash/array.) Matrix is just a table in the global namespace, with methods as keys and the actual functions as values. So we can just define our own methods like so.

function Matrix:newMethod(var1, var2)
  -- 'self' is automatically defined because we used the lua ':' syntactic sugar
  print("Matrix:newMethod called", self, var1, var2)
end

Once our code is included in a project, you won't be able to tell that newMethod wasn't always there.

Multiplication Method

So our first and probably most important extension to the Matrix class will be a multiply() method that takes a matrix as its argument and multiplies itself by that matrix using the formula we discussed above. We can skip the parts of the dot product that always add a 0 to avoid unnecessary calculations.

function Matrix:multiply(matrix)
  local m11 = self:getM11()*matrix:getM11() + self:getM12()*matrix:getM21()
  local m12 = self:getM11()*matrix:getM12() + self:getM12()*matrix:getM22()
  local m21 = self:getM21()*matrix:getM11() + self:getM22()*matrix:getM21()
  local m22 = self:getM21()*matrix:getM12() + self:getM22()*matrix:getM22()
  local tx  = self:getM11()*matrix:getTx()  + self:getM12()*matrix:getTy() + self:getTx()
  local ty  = self:getM21()*matrix:getTx()  + self:getM22()*matrix:getTy() + self:getTy()
  return self:setElements(m11, m12, m21, m22, tx, ty)
end

This is now our foundation upon which we can build everything else. Please note that Gideros Matrix methods take the arguments in a slightly odd order: m11, m12, m21, m22, tx, ty.

Translation Method

Translation is the simplest matrix to apply. Given a tx and ty as distance to translate x and y by, we simply build the following matrix based on an identity matrix, then multiply it.

$$ \begin{bmatrix} 1 & 0 & tx \\ 0 & 1 & ty \\ 0 & 0 & 1 \\ \end{bmatrix} $$

This is what our translate() method will look like.

function Matrix:translate(x,y)
  return self:multiply(Matrix.new(1, 0, 0, 1, x, y or x))
end

Scaling Method

Scaling is also pretty simple. Given sx and sy as the amount to scale x and y by, we build the following matrix based on an identity matrix, then multiply it.

$$ \begin{bmatrix} sx & 0 & 0 \\ 0 & sy & 0 \\ 0 & 0 & 1 \\ \end{bmatrix} $$

And here is the code that will do that.

function Matrix:scale(sx,sy)
  return self:multiply(Matrix.new(sx, 0, 0, sy or sx, 0, 0))
end

Rotation Method

Rotation is a little more involved, but still isn't bad. Given the angle a in radians, our rotation translation matrix looks like this.

$$ \begin{bmatrix} \cos(a) & -\sin(a) & 0 \\ \sin(a) & \cos(0) & 0 \\ 0 & 0 & 1 \\ \end{bmatrix} $$

Since Gideros functions always deal in degrees and not radians, we'll write our function to take the angle in degrees, then convert to radians for building the matrix.

function Matrix:rotate(deg)
  local rad = math.rad(deg)
  return self:multiply(Matrix.new(math.cos(rad), -math.sin(rad), math.sin(rad), math.cos(rad), 0, 0))
end

Skew Method

Skew is similar to rotation. Given an angle ax to skew on the x-axis, and an angle ay to skew on the y-axis, we construct the matrix as such.

$$ \begin{bmatrix} 1 & \tan(ax) & 0 \\ \tan(ay) & 1 & 0 \\ 0 & 0 & 1 \\ \end{bmatrix} $$

Similar to rotation, we'll take our arguments in degrees and convert to radians for the calculation.

function Matrix:skew(xAng,yAng)
  xAng = math.rad(xAng)
  yAng = math.rad(yAng or xAng)
  return self:multiply(Matrix.new(1, math.tan(xAng), math.tan(yAng), 1, 0, 0))
end

Reflection

Reflecting on the x or y axis is as simple as applying a negative scale in either direction.

-- Flip or mirror on the horizontal axis.
matrix:setScaleX(-1)

Applying a transformation matrix to an arbitrary point

Lastly, it comes in handy every once in a while to be able to apply a given transformation matrix to a single point. This is mainly used to translate between an original coordinate space and the transformed coordinate space. Points can be represented as a 3x1 matrix.

$$ p = \begin{bmatrix} x \\ y \\ 1 \\ \end{bmatrix} $$

This means we can multiply a 3x3 transformation matrix times a point, but not the other way, since the inner products wouldn't match. So this multiplication simply looks like this.

$$ \begin{bmatrix} \text{a}_{11} & \text{a}_{12} & \text{a}_{tx} \\ \text{a}_{21} & \text{a}_{22} & \text{a}_{ty} \\ 0 & 0 & 1 \\ \end{bmatrix} \cdot \begin{bmatrix} x \\ y \\ 1 \\ \end{bmatrix} = \begin{bmatrix} \text{a}_{11} \times x + \text{a}_{12} \times y + \text{a}_{tx} \times 1 \\ \text{a}_{21} \times x + \text{a}_{22} \times y + \text{a}_{ty} \times 1 \\ 0 \times x + 0 \times y + 1 \times 1 \\ \end{bmatrix} $$

Converting this formula to code will result in this method.

function Matrix:applyToPoint(x, y)
  local newX = self:getM11() * x + self:getM12() * y + self:getTx()
  local newY = self:getM21() * x + self:getM22() * y + self:getTy()
  return newX, newY
end

Putting it all together

Once this is all in place, the following example code is an idea of what is possible.

-- Create a rectangle Shape which inherits from Sprite sized 100 pixels wide and 100 pixels tall
local box = createBox(100,100)

-- Set a new position for the box.
box:setPosition(100,200)

-- This gets the current matrix, which includes the new position we just set
local matrix = box:getMatrix()

-- Move to our new anchor point relative to current position
matrix:translate(50,50)

-- Rotate by 45 degrees
matrix:rotate(45)

-- Scale by 150%
matrix:scale(1.5)

-- Skew the x axis by 20 degrees
matrix:skew(20, 0)

-- Put it back
matrix:translate(-50,-50)

-- Apply our new concatenated transform to the sprite
box:setMatrix(matrix)

Full code and Sample project

I have made the full code to this Matrix extension available on Github. You can view just the Matrix class extensions or a full example Gideros project that shows the code in action. The code is the same as in this post, plus a few extra convenience methods.

References

I'd like to thank the following wonderful references that I read when I was learning all of this.

Real world uses

I'll soon post followups to this post showing a couple examples of how I used this new Matrix class to implement a movable anchor for Gideros Sprites, and to parse SVG files and all their transforms. For example, here's my AnchorSprite implementation, which supports moving the anchor for a sprite from [0,0] to anywhere you want, and affects all position, rotation, scale and skew operations.

More on 3D

When applying these same techniques to 3D, most of the basics are the same, however you have bigger matrices and one more axis to handle. Matrix multiplication still works the same, the matrices are still homogeneous, identity matrices are the same. What will be different are the types of matrices you construct to do your transformations. Here is just a quick list of these matrices.

4x4 3D Translation Matrix	4x4 3D Scale Matrix
$$ \begin{bmatrix} 1 & 0 & 0 & tx \\ 0 & 1 & 0 & ty \\ 0 & 0 & 1 & tz \\ 0 & 0 & 0 & 1 \\ \end{bmatrix} $$	$$ \begin{bmatrix} sx & 0 & 0 & 0 \\ 0 & sy & 0 & 0 \\ 0 & 0 & sz & 0 \\ 0 & 0 & 0 & 1 \\ \end{bmatrix} $$

4x4 3D x-axis Rotation Matrix	4x4 3D y-axis Rotation Matrix	4x4 3D z-axis Rotation Matrix
$$ \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 & \cos(a) & -\sin(a) & 0 \\ 0 & \sin(a) & \cos(a) & 0 \\ 0 & 0 & 0 & 1 \\ \end{bmatrix} $$	$$ \begin{bmatrix} \cos(a) & 0 & \sin(a) & 0 \\ 0 & 1 & 0 & 0 \\ -\sin(a) & 0 & \cos(a) & 0 \\ 0 & 0 & 0 & 1 \\ \end{bmatrix} $$	$$ \begin{bmatrix} \cos(a) & -\sin(a) & 0 & 0 \\ \sin(a) & \cos(a) & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \\ \end{bmatrix} $$

4x4 3D xz-axis Skew Matrix	4x4 3D zx-axis Skew Matrix	4x4 3D yz-axis Skew Matrix
$$ \begin{bmatrix} 1 & 0 & 0 & 0 \\ \tan(a) & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \\ \end{bmatrix} $$	$$ \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ \tan(a) & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \\ \end{bmatrix} $$	$$ \begin{bmatrix} 1 & \tan(a) & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \\ \end{bmatrix} $$

4x4 3D zy-axis Skew Matrix	4x4 3D xy-axis Skew Matrix	4x4 3D yx-axis Skew Matrix
$$ \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & \tan(a) & 1 & 0 \\ 0 & 0 & 0 & 1 \\ \end{bmatrix} $$	$$ \begin{bmatrix} 1 & 0 & \tan(a) & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \\ \end{bmatrix} $$	$$ \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 & 1 & \tan(a) & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \\ \end{bmatrix} $$

Points in 3D are a 4x1 of $[x,y,z,1]^T$, and the math to apply them to a transform is the same as above, just one more axis to determine. Here are some further resources specific to 3D transformations.

Implementing a better Bezier class in Gideros

2013-05-03T00:00:00-07:00

Bezier curves are nothing new, and there is a lot of good code out there that implements them well. I started with Paul Burke's algorithm, mainly because someone had already ported it to Gideros. This algorithm works quite well, is fast, and is simple enough to understand. However, I thought I could figure out how to improve on my implementation at least a little, without switching to a much more complicated algorithm. My goals were to produce curves with points in all the optimal places resulting in it being smooth, but with as few points as possible. So this is what I've come up with.

First, here is my naive implementation.

Bezier = Core.class(Shape)

function Bezier:createCubicCurve(p1, p2, p3, p4, steps)
  self.points = {}
  steps = steps or 100
  for i = 0, steps do
    table.insert(self.points, self:bezier4(p1, p2, p3, p4, i/steps))
  end
end

function Bezier:bezier4(p1,p2,p3,p4,mu)
  local mum1,mum13,mu3;
  local p = {}

  mum1 = 1 - mu
  mum13 = mum1 * mum1 * mum1
  mu3 = mu * mu * mu

  p.x = mum13*p1.x + 3*mu*mum1*mum1*p2.x + 3*mu*mu*mum1*p3.x + mu3*p4.x
  p.y = mum13*p1.y + 3*mu*mum1*mum1*p2.y + 3*mu*mu*mum1*p3.y + mu3*p4.y
  --p.z = mum13*p1.z + 3*mu*mum1*mum1*p2.z + 3*mu*mu*mum1*p3.z + mu3*p4.z

  return p 
end

This gives us 101 points for the cubic curve {100,100}, {800,250}, {800,100}, {150,200}. This isn't too bad, as it gives us enough for it to be smooth, but using way too many points. Much longer curves will end up with too-few points and could appear jagged, and much shorter curves will end up with too many.

Automatic estimation of steps

First, I wanted to figure out a way to estimate the number of steps for a given curve, based on it's length instead of just always using 100. This way shorter curves would have fewer points, avoiding unnecessary overhead, and longer curves would have more points, smoothing them out. The problem is, we don't really know the length until we actually calculate it. However, we can guess based on the distances of all of the points. Basically we estimate it at 10% (by default) of the total distance between all points. This seems to work fine in my tests. If you want more points, just increase the percentage.

function Bezier:estimateSteps(p1, p2, p3, p4)
  local distance = 0
  if p1 and p2 then
    distance = distance + self:pointDistance(p1, p2)
  end
  if p2 and p3 then
    distance = distance + self:pointDistance(p2, p3)
  end
  if p3 and p4 then
    distance = distance + self:pointDistance(p3, p4)
  end

  return math.max(1, math.floor(distance * self.autoStepScale))
end

This gives us 153 points for this particular curve. This will help smooth out tight turns in curves much larger than this, but is way too many for most parts of the curve, such as the long straight parts. So by itself, this isn't much of a solution.

Reduction of points

So the last thing I wanted to do was to get rid of unnecessary points. If a part of the curve is pretty much straight, we don't need so many points to somewhat accurately describe it. After some searching around I found the Ramer-Douglas-Peucker algorithm for reducing points in a curve, along with an implementation for Corona SDK. So now we can add a reduce() method. It recursively figures out redundant points that are less than epsilon distance from a line comprised of the current segment it's looking at. After it's done, we throw away all points that aren't marked as, "keep."

function Bezier:reduce(epsilon)
  epsilon = epsilon or .1

  if #self.points > 1 then
    -- Keep first and last
    self.points[1].keep = true
    self.points[#self.points].keep = true

    -- Figure out the rest
    self:douglasPeucker(1, #self.points, epsilon)
  end

  -- Replace point list with only those that are marked to keep
  local old = self.points
  self.points = {}

  for i,point in ipairs(old) do
    if point.keep then
      table.insert(self.points, {x=point.x, y=point.y})
    end
  end
end

function Bezier:douglasPeucker(first, last, epsilon)
  local dmax = 0
  local index = 0

  for i=first+1, last-1 do
    local d = self:pointLineDistance(self.points[i], self.points[first], self.points[last])

    if d > dmax then
      index = i
      dmax = d
    end
  end

  if dmax >= epsilon then
    self.points[index].keep = true

    -- Recursive call
    self:douglasPeucker(first, index, epsilon)
    self:douglasPeucker(index, last, epsilon)
  end
end

function Bezier:pointLineDistance(p, a, b)
  -- calculates area of the triangle
  local area = math.abs(0.5 * (a.x * b.y + b.x * p.y + p.x * a.y - b.x * a.y - p.x * b.y - a.x * p.y))
  -- calculates the length of the bottom edge
  local dx = a.x - b.x
  local dy = a.y - b.y
  local bottom = math.sqrt(dx*dx + dy*dy)
  -- the triangle's height is also the distance found
  return area / bottom
end

This now gives us 34 points. It spreads them out on the straighter parts, but packs them in on the sharp corners so they will appear as smooth as the un-reduced version. So visually this is exactly what I wanted, but at what cost?

Performance

At first I expected these extra operations to slow everything down, but I figured it was worth it in certain cases. However, I was surprised to find that by far the slowest part of rendering the curve is actually having Gideros draw it as a series of connected lines. So while my reduce() function is expensive, because it reduces the number of points so much, the end result is less drawing, and so overall it's faster. Here are some benchmarks of the curve in the images above, running in the desktop player:

Parameters	#points	Creation	Drawing	Reduction	Total
100 steps, no reduce	101	0.12ms	1.94ms	0.00ms	2.07ms
Auto steps, no reduce	153	0.21ms	3.15ms	0.00ms	3.36ms
Auto steps, .1 epsilon	34	0.21ms	0.67ms	0.82ms	1.70ms

Source

The full class and a test project is available at github.

Further

If you're interested in a different approach altogether, take a look at Maxim Shemanarev's paper, "Adaptive Subdivision of Bezier Curves." It is a much more complicated algorithm, but he gets beautiful results.

Fixing Lua indenting in Sublime Text

2013-04-10T00:00:00-07:00

Updated September 18th, 2015: Added info on making the same changes to latest ST2 and ST3 build 3083

I recently started using Sublime Text to edit the Lua project I'm working on, however, it did a couple strange things with indenting. Here is how I fixed them.

First, open the file we need to edit in your Sublime Packages Lua directory.

Sublime Text 2:

Find your Packages directory by going to Preferences -> Browse Packages... in Sublime Text.
Open the Lua folder
Edit the Indent.tmPreferences file

Sublime Text 3:

Install PackageResourceViewer using Package Control or from https://github.com/skuroda/PackageResourceViewer
Run PackageResourceViewer: Open Resource
Select Lua then Indent.tmPreferences

In that file you'll see keys labeled, increaseIndentPattern and decreaseIndentPattern with accompanying regexes wrapped in <string> entities. We'll be editing those for these two fixes.

Nested curly brace indentation

First, whenever you nested curly braces, indenting any curly braces after the first one would end up with something like this:

When instead we want something like this:

The fix for this is pretty easy. We want to edit the increaseIndentPattern

<key>increaseIndentPattern</key>
<string>^\s*(else|elseif|for|(local\s+)?function|if|repeat|until|while)\b((?!end).)*$|.*\{\s*$</string>

The first half is concerned with matching keywords like "if" and "for", but without also having "end" on the same line. The latter half is the part we care about.

|\{\s*$

This matches a curly brace followed by zero or more whitespace characters then the EOL. At first glance this seems like it would work. The fix, weirdly enough, is to match anything before the curly brace as well. So we add ".*" which matches any character zero or more times. I think it might be because sublime throws out any partial matches to prevent erroneous indents. It wants the whole line to be matched. So the fix looks like:

|.*\{\s*$

This does mean that it will indent ANY line ending in a curly brace.

Strange indentation of "elseif"

The other problem is improper indentation of the "elseif" statement. Basically sublime gets really confused because it matches "else" then "elseif" and then "elseif" with something after it, and it ends up not recognizing the statement as valid, and doesn't know to unindent it. It looks like this bit of strangeness:

The problem here lies in the "decreaseIndentPattern" part.

<key>decreaseIndentPattern</key>
<string>^\s*(elseif|else|end|\})\s*$</string>

This works fine for "else" and "end" because the don't have anything after them. However, "elseif" by it's nature will have text coming after it. So to fix this, we are going to remove "elseif" from this statement, and break it out into its own sub-regex. That way "else" and "end" will continue to work like they should. Then we're going to tell it that "elseif" can have characters after it, but only if there is no "end" after the "elseif". You might have noticed that this sounds like what it does in the "increaseIndentPattern" part. So we can just copy that bit of regex to fit this need as well. We end up with:

<string>^\s*(else|end|\})\s*$|^\s*elseif\b((?!end).)*$</string>

After the "|" we put in, it matches zero or more whitespace, the word "elseif" followed by a word boundary, then it does a look-ahead to match any character that doesn't follow "end" zero or more times before finding the end-of-line. This results in proper behavior:

Fully fixed file

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>name</key>
    <string>Indent</string>
    <key>scope</key>
    <string>source.lua</string>
    <key>settings</key>
    <dict>
        <key>decreaseIndentPattern</key>
        <string>^\s*(else|end|\})\s*$|^\s*elseif\b((?!end).)*$</string>
        <key>increaseIndentPattern</key>
        <string>^\s*(else|elseif|for|(local\s+)?function|if|repeat|until|while)\b((?!end).)*$|.*\{\s*$</string>
    </dict>
    <key>uuid</key>
    <string>411468A8-E0AC-415A-9E71-E2BD091EB571</string>
</dict>
</plist>

Restart Sublime Text

Sublime Text 2:

Just restart and your changes should take effect.

Sublime Text 3:

Find your Packages directory by going to "Preferences->Browse Packages..." in Sublime Text 3. Leave this window open.
Close all Sublime Text 3 windows
Back in your file browser window, navigate up one directory then delete the "Cache" directory.
Start Sublime Text 3 and it should work now.

Lotechnica

Configuring a dev environment for Phoenix package development

Understanding the Phoenix Developer Experience

The Code Reloader

The Live Reloader

Example Setups

Option 1: Sibling projects

Create the library MyLibrary

Create the phoenix app MyApp / MyAppWeb

Configuring Phoenix for Live Reloading

Bonus: Dynamically referencing the :my_library dependency

Check for the existence of the sibling directory

Use an optional file to determine the dependency

Option 2: A subdir Phoenix project

Create the library MyLibrary

Create the phoenix app MyApp / MyAppWeb

Configuring Phoenix for Live Reloading

Option 3: Umbrella

Create the phoenix umbrella

Create the library MyLibrary

Configure the "MyAppWeb" umbrella app

Configuring Phoenix for Live Reloading

Publishing your package

The Stock Market Game, made easy with Phoenix LiveView

Dynamic configuration for Phoenix Endpoints

Example

Workaround

Caveats

Fix?

Prefix normalize.css or sanitize.css before gulp-sass

Postgres Full-Text Search With Django

Models

Building Documents

Search Vectors

Performing a search

Ranking

Storing Vectors with SearchVectorField

Searching on search_vector

Indexing

Updating the documents

Overriding save()

Using triggers

Conclusion

Updates to this article

10/10/17

Fix Django's runserver when run under Docker or PyCharm

The quick fix

The why

A Phoenix Plug for assigning template variables

The Plug

Using in a Pipeline

Using it to assign a variable for all Controllers

A few notes on Bourbon, Neat, Bitters, Refills

Context

<awesome>Semantic markup!</awesome>

Kitchen sink not included

Efficient results as a side effect

You want this when?

Overall more efficient for larger or more custom projects

Final thoughts

I have released my first mobile app, AnswerCast

AnchorSprite: Setting an anchor point on any Sprite

Intercepting Sprite transform methods

Applying the transforms

Optimization

Result

The beauty of using matrices to apply transformations to objects

Enter the matrix

What are matrices?

2D and 3D transformations matrices

Homogeneous Matrices

Identity Matrices

Multiplication/Concatenation

Rule One: Order Matters!

Rule Two: Matrices have to have the same inner dimensions

Rule Three: Resulting matrix size

Multiplication

Application

Matrix functionality in Gideros

Multiplication Method

Create the library `MyLibrary`

Create the phoenix app `MyApp` / `MyAppWeb`

Bonus: Dynamically referencing the `:my_library` dependency

Create the library `MyLibrary`

Create the phoenix app `MyApp` / `MyAppWeb`

Create the library `MyLibrary`