The universal document converter

Pandoc – written and maintained by John MacFarlane – is an relatively old project. It has grown considerably since the first version was published in 2006: at the time of writing, pandoc can read 27 different document formats and dialects, and can write 49 formats. Besides serving as a one-off document conversions tool, pandoc also frequently features as the central part of publishing pipelines. For example, Pandoc is used in static site generators and is frequently used by academic writers, due also to its excellent support for citations.

As a brief example, consider the following commands which transform Markdown input into docx, HTML, or PDF:

 -- command to convert a markdown file to docx
pandoc input-file.md --output=output-file.docx

-- convert to HTML
pandoc input-file.md --standalone --output=output-file.html

 -- convert to PDF (via LaTeX)
pandoc input-file.md --output=output-file.pdf

Many conversion tasks need to alter the default behavior or require special conversion features. This highlights the importance of good customization support for a conversion tool, one of the areas in which Lua shines.

Pandoc is unusual for a Lua-extendable program, in that it is written in Haskell. Using Haskell is very productive, but is less suitable as an extension language: its concepts are often alien to users of other languages, and shipping a full Haskell interpreter with pandoc would result in considerable bloat. Lua is an excellent choice here, as it is lightweight, simple, and beautiful. It should be noted, however, that bridging Haskell and Lua is its own can of worms and worth a separate blog post.

Pandoc's document AST

An important factor in pandoc's immense transformation powers is its use of a unifying document representation: Every input is parsed into this document AST, which is then rendered in the desired output format. While a direct conversion between any of n input and m output formats would require n * m converters, using an intermediate representation reduces complexity to n + m.

There are additional advantages to this: as we'll see, it becomes much simpler to work with a unified document representation than it would be to work with any of the input or output formats directly.

There are four main types in pandoc's document model: inlines, blocks, document metadata, and the full document.

• Inline elements represent text and text markup. Examples are Space for inter-word spaces, Str for (usually non-whitespace) text, and Emph for emphasized text.

• Blocks are elements like paragraphs, lists, code listings, and headers. They are usually rendered in lines or blocks of their own; many block elements contain lists of inline elements.

• Meta information is a simple mapping from string keys to meta values. Meta values can be thought of as a special JSON or YAML object.

• Last but not least, the Pandoc type represents a full document. A Pandoc element consists of a lists of block elements, plus additional document metadata.

Pandoc's Lua features revolve around modifying or converting these elements. The oldest use of Lua in pandoc enables the conversion of AST elements into strings as to output any document format.

Custom writers

Users can define custom writers in Lua to render any document format. Each of the aforementioned AST elements is transformed to a string by calling a Lua function of the same name as the element. E.g., this example demonstrates how emphasized text can be rendered as HTML:

function Emph(content_string)
  return '<em>' .. content_string .. '</em>'
end

A full custom writer is defined by specifying functions for all document AST elements. Example writers using this method include 2bbcode by \@lilydjwg (依 云), as well as pandoc's sample.lua. The latter is a well documented starting point for authors of new custom writers. The file can be produced by calling `pandoc --print-default-data-file=sample.lua`.

The pandoc-scholar project serves as an example for the power offered by custom writers. It is a publishing tool intended to help authors of scholarly articles and was created with custom Lua writers. The tool leans on the custom writers feature in ways that writers were not intended to be used, which resulted in the development of lua filters.

Filters

An additional benefit of a unified document type is that the document can be modified programmatically, regardless of which input and output format is chosen. Pandoc provides two interfaces for this.

JSON Filters

The first – very flexible – method is based on JSON. Pandoc can serialize the document to JSON; other programs can read and modify the document. The resulting document JSON is passed back to pandoc, thus allowing users to use any programming language capable of parsing JSON to alter the document. Many libraries for various languages have been implemented, including Haskell, Python, Ruby, and JavaScript.

The flexibility of JSON filters can also be a disadvantage, as it requires additional software and usually the full installation of a scripting language's ecosystem. Pandoc is designed to work on all major platforms and without any dependencies on other libraries and binaries. Depending on additional software can be problematic, especially for non-technical users.

Lua filters

The Lua filter system added in pandoc 2.0 not only solves the portability issue of JSON filters, but also offers better performance and more functionality. Document elements can be selectively serialized to Lua tables, modified using the full power of Lua, and will then be transferred back, thus replacing the previous values.

Lua filters operate by calling filter functions on each element of the specified name. I.e., if a Lua filter contains a function with the same name as an AST element, then this function is called for all elements of the respective type. The serialized element is passed as input to the filter function, and the function's return value is deserialized and used to replace the input element. This method is as simple as it is flexible, and fits well with the concept of immutability which is prevalent in Haskell programs: pandoc ignores modifications to the serialized object itself, it will just use the filter function's return value.

The following example filter transforms all text set in small caps into emphasized text:

function SmallCaps (element)
  return pandoc.Emph(element.content)
end

The element constructor functions in module pandoc, like pandoc.Emph in the above example, are also the central step when transforming elements from their pandoc-internal representation to Lua values. This ensures consistency in the way element values are produced, whether during serialization or through a constructor call in the filter script. The current implementation uses only strings, tables, and some metatables when constructing element values, with the goal of marking these values easy and flexible to use.

Lua filter example: macro expander

Below is the code for a simple macro expander using pandoc's Lua filter functionality. The expander replaces all macro occurrences in the given document. Macro definitions are hard-coded into the filter, but could as well be read from an external file.

-- file: macro-expander.lua

-- Macro substitutions: contains macro identifier as
-- keys and the expanded inlines as values.
local macro_substs = {
  ['{{hello}}'] = pandoc.Emph{pandoc.Str "Hello, World!"}
}

-- Replace string with macro expansion, if any.
function Str (s)
  return macro_substs[s.text] or s
end

The heart of the macro expander is the function Str. It is called on all simple strings in the document. The return value of this function is then read back into pandoc, replacing the original Str value.

Assume a Markdown file greeting.md:

Greeting: {{hello}}

We can apply the macro expander by calling

pandoc --lua-filter macro-expander.lua greeting.md

resulting in the expected expansion:

Greeting: Hello, World!

The function Str could be shortened further by dropping the trailing or s:

function Str (s) return macro_substs[s.text] end

This is a convenience feature of pandoc filters: if the function returns no value (or nil), the original value is kept unchanged. This makes filter functions easier to write and speeds up filtering, as unchanged elements don't need to be deserialized again.

What's good, and what's next

Using pandoc with Lua is a fast, flexible, and platform independent way of augmenting pandoc with additional functionality. For me personally, having the full power of Lua at ones finger tips proved to be a lot of fun, while opening unexpected document processing possibilities.

Pandoc and its Lua subsystem are under constant development. E.g., the next versions will feature more utility functions exposed via Lua modules. There is constant work to make more and more internal functions available. The next big goal is to grant scripting access to all format-output functions. However, this requires some changes to pandoc's internals. It remains a long way for pandoc to become a fully Lua-scriptable publishing platform.

If you want to learn more about Lua filters, the Lua filter docs is a good place to start. It includes up-to-date examples of Lua scripts, as well as a reference of all modules and functions accessible via Lua. Pandoc's user manual is a good resource to learn about all of pandoc features and its command line options.

Feedback is always welcome!

Acknowledgements

A big thank you to Jennifer König, Birgit Pohl, and John MacFarlane for their feedback on an earlier version of this post, and to all pandoc contributors and users, who make working on this project incredibly fun.

The arrival of Web Assembly reignited that desire for many of us. But wasm is a long way from being usable in production and still does not allow full interaction with all web apis.

Lua is a good fit for the browser

Lua is a simple language with very few concepts to understand and a clear and readable syntax. You can be proficient with it in a matter of hours. Yet it provides very useful features. To name a few:

• First class functions and closures
• A versatile data structure: the table
• Vararg expression
• Lexical scoping
• Iterators
• Coroutines (see below)

It has a sane coercion system (I'm looking at you JS!). Nobody's adding X new concepts to it each version (still looking at you JS), which makes it a stable and reliable language.

Lua is already successfully used server-side with the awesome OpenResty which powers sites like itch.io. There's even some great web frameworks like lapis.

Lua can be found in a number of widely different contexts because of how easy it is to embed it. You can write a whole program with it or simply use it as a glue language thanks to the Lua C api.

Coroutines

One of its main selling point for the browser are coroutines. Coroutines address the issue of writing asynchronous code beautifully. No more promises, generators, etc. You can just write asynchronous code as easily as regular code.

Here's a simple example (fully functional version here):

local bird1 = fetch("http://some.api.com/raven") -- fetch being a random xhr call
local bird2 = fetch("http://some.api.com/dove")

print(bird1.name, bird2.name)

A similar version of that using async/await could be:

let asynchronousFn = async function() {
    let bird1 = await fetch("http://some.api.com/raven");
    let bird2 = await fetch("http://some.api.com/dove");

    console.log(bird1.name, bird2.name);
}

These are close, but notice how, in the Lua version, you don't need to be in a async function. In JS you would have to constantly make the conscious choice of tagging a function async or not. In Lua, any function can be interrupted as long as it's running in a coroutine. Bob Nystrom wrote a great post about it here.

When you're writing asynchronous code in JS, you're in fact piling up function calls. Those callbacks all retain their upvalues which can lead to high memory usage. With coroutines your function can actually be suspended and resumed without having to descend from a pyramid of closures because they run in separate lua states.

Existing ways of using Lua in the browser

There's already a few projects out there to use Lua in the browser in some extent:

• Moonshine is a reimplementation of Lua in JS. Sadly, it's not being actively developed anymore.
• Starlight by the same author is a Lua to JS transpiler. Unfortunately coroutines can't be implemented effectively with this approach.
• lua.vm.js a port of the original Lua implementation to JS using Emscripten. It's fast and works well but its interoperability with JS is compromised by the fact that we end up with two garbage collectors trying to manage the same memory.

Fengari

Fengari (moon in greek) is the Lua VM written in Javascript with the browser as its primary target.

Fengari aims to be 100% compliant with the latest Lua semantics. It stays as close as possible to Lua's codebase and if you're familiar with it, you'll understand Fengari rather easily. 99% of Lua's scope is currently covered so you should be able to run any Lua project with minor adjustments.

With the C API (rather JS API), you can decide to write everything in Lua or to use it as a higher level language that calls your JS codebase to do the heavy lifting. This also means that you can segregate your code in separate insulated Lua states.

You can also interact with the JS side directly from your Lua code effortlessly with the fengari-interop module. It ensures that manipulating JS objects or functions always behave the way you would expect it to:

local global = js.global
local document = global.document

global:alert("Hello from Fengari")

document:getElementById("my-div").textContent = "Hi there !"

The REPL you see on fengari.io is itself written in Lua and JS is only used to create the Lua state and run the main script.

Fengari is still in development and any feedback is welcome !

Lua is a small, fast, powerful, and embeddable scripting language. It is well-suited for use in video games, application scripting, embedded devices, and nearly anywhere else a scripting language is needed. This quick reference contains a wealth of knowledge on how to program in and embed Lua, whether it is Lua 5.3, 5.2, or 5.1. This book can even be used with LuaJIT, a Just-In-Time compiler for Lua based on Lua 5.1. Lua Quick Reference groups the language's features and C API in a convenient and easy-to-use manner, while clearly marking the differences between Lua versions.

This book covers:

• Lua syntax, expressions, and statements
• Metatables and metamethods
• Object-oriented programming with Lua
• Creating and working with Lua and C Modules
• Lua's standard library and its C API
• Collaborative multi-threading in Lua and C
• How to embed and use Lua within a host
• And much more

Lua is a small, fast, powerful, and embeddable scripting language. It is well-suited for use in video games, application scripting, embedded devices, and nearly anywhere else a scripting language is needed. This quick reference contains a wealth of knowledge on how to program and embed Lua, whether it is Lua 5.3, 5.2, or 5.1. It groups the language's features and C API in a convenient and easy-to-use manner, while clearly marking the differences between Lua versions.

This book covers:

• Lua syntax, expressions, and statements
• Metatables and metamethods
• Object-oriented programming with Lua
• Creating and working with Lua and C Modules
• Lua's standard library and its C API
• Collaborative multi-threading in Lua and C
• How to embed and use Lua within a host
• And much more

Google Summer of Code is an international annual program, in which Google awards stipends to students who successfully complete a free and open-source software coding project during the summer. The program is open to university students aged 18 or over. The coding period goes from May 30 to August 29 (that is, the summer break in the Northern Hemisphere).

LuaRocks, the package manager for Lua, has been part of GSoC in the past as part of the LabLua organization (a research lab at PUC-Rio, the home of Lua). But this was the first time we applied as an independent organization, and we are very happy and thankful to be accepted.

Our global team of mentors (with people hailing from four continents!) has produced a nice list of project ideas, but students are free and encouraged to contact us with ideas of their own as well!

Applying to the program

If you are an eligible student and wish to apply, you should join the luarocks-developers mailing list and say hi at our Gitter group.

Once you find a project you are interested in, you should contact the mentor for that project by email, and this questionnaire. If your application looks appropriate, the mentor may ask you to perform some small task related to the project to assess your abilities, and discuss with you how to best present your proposal. Proposals accepted from March 20 to April 3, 2017!

Join us and come hack LuaRocks this next summer! (...or winter if you're in the Southern Hemisphere! :) )

Next

• March 5th 2017 - Lua/LuaJIT conference, Moscow

  • Annnouncement

• June 3rd 2017 - LuaConf, Rio de Janeiro

  • Website
  • Call for presentations
  • Call for sponsor

Past

• February 4-5th 2017 - Lua Devroom @ FOSDEM 2017, Brussels

  • Slides and videos
  • Video of panel: The Future of Small Languages
  • Pictures

• October 13-14th 2016 - Lua Workshop, California

  • Slides and videos

• July 9th 2016 - Lua Conf, Rio de Janeiro

  • Videos

News

• Feb 7th - Lua community events & developer relations effort
• Jan 30th - Lua 5.3.4 now available
• Jan 13th - New Lua IRC channel in French

Featured Releases

• Feb 6th - LuaWt 0.0.1 (GPL-2.0)
• Jan 30th - Ravi 0.19 (MIT / GPL)
• Jan 24th - SmithSNMP 0.8 (GPL-2.0)
• Jan 22nd - Lyaml 6.1 (MIT)
• Jan 13th - LPeg 1.0.1 (MIT)
• Jan 13th - opeth (MIT)
• Jan 5th - see.lua (MIT)

Resources

• Oxford Machine Learning lectures

In 2015 we had a Birds of a Feather meeting and in 2016 we had a real developer room for half a day. Both meetings were a big success and next year we will have half a day of a developer room again.

We are now accepting submissions of talk proposals. The format will be 30 minutes talks, including questions (so roughly 20-25 minute talks, 5-10 minutes for Q&A). The talks will be filmed, possibly streamed live, and the videos will be made available publicly under a CC-BY license. Submitting a talk for the devroom implies that you agree to be filmed under such terms.

All topics related to Lua are accepted. The deadline for submission is Thursday, 1st of December 2016.

All submission should be made using the FOSDEM even planning tool Pentabarf. When you create your event, make sure to select "Lua devroom" under "Track". If you already have a Pentabarf account from a previous year, please reuse it instead of creating a new one.

There is a mailing-list for the devroom, feel free to join and discuss talk ideas, recommend speakers, ask if anyone wants to talk about a topic you would like to know more about or discuss any other Lua devroom related topic.

If you have any questions, you can contact the devroom organizers by email:
Etiene Dalcol : dalcol [at] etiene [dot] net
Pierre Chapuis : catwell [at] archlinux [dot] us

Important dates summary:
Submission deadline: 2016-12-01
Acceptance notifications: 2016-12-11
Final schedule announcement: 2016-12-18
Devroom: 4th February 2017 (morning)

Please share this CFP with any person or group you know that could be interested!

See you at FOSDEM!

Without further ado, let's go!

A table is a mapping of keys to values. They're explained quite well in the PICO-8 manual and the Lua reference manual so I won't go into more detail. In particular you should know that t.foo is just a nicer way of writing t["foo"] and also that t:foo() is a nicer way of calling the function t.foo(t)

A metatable is a table with some specially named properties defined inside. You apply a metatable to any other table to change the way that table behaves. This can be used to:

1. define custom operations for your table (+, -, etc.)
2. define what should happen when somebody tries to look up a key that doesn't exist
3. specify how your table should be converted to a string (e.g. for printing)
4. change the way the garbage collector treats your table (e.g. tables with weak keys)

Point #2 is especially powerful because it allows you to set default values for missing properties, or specify a prototype object which contains methods shared by many tables.

You can attach a metatable to any other table using the setmetatable function.

All possible metatable events are explained on the lua-users wiki:
>>> list of metatable events <<<

which is, as far as I'm aware, the best reference for everything that metatables can be used for.

And that's really all you need to know!

Vectors Example

I'll now demonstrate how metatables could be used to make a "2D point/vector" type, with custom operators.

-- define a new metatable to be shared by all vectors
local mt = {}

-- function to create a new vector
function makevec2d(x, y)
    local t = {
        x = x,
        y = y
    }
    setmetatable(t, mt)
    return t
end

-- define some vector operations such as addition, subtraction:
function mt.__add(a, b)
    return makevec2d(
        a.x + b.x,
        a.y + b.y
    )
end

function mt.__sub(a, b)
    return makevec2d(
        a.x - b.x,
        a.y - b.y
    )
end

-- more fancy example, implement two different kinds of multiplication:
-- number*vector -> scalar product
-- vector*vector -> cross product
-- don't worry if you're not a maths person, this isn't important :)

function mt.__mul(a, b)
    if type(a) == "number" then
        return makevec2d(b.x * a, b.y * a)
    elseif type(b) == "number" then
        return makevec2d(a.x * b, a.y * b)
    end
    return a.x * b.x + a.y * b.y
end

-- check if two vectors with different addresses are equal to each other
function mt.__eq(a, b)
    return a.x == b.x and a.y == b.y
end

-- custom format when converting to a string:
function mt.__tostring(a)
    return "(" .. a.x .. ", " .. a.y .. ")"
end

Now we can use our newly defined 'vector' type like this:

local a = makevec2d(3, 4)
local b = 2 * a

print(a)      -- calls __tostring internally, so this prints "(3, 4)"
print(b)      -- (6, 8)
print(a + b)  -- (9, 12)

Pretty neat right?

Object Orientation

I mentioned that metatables can be used to define what should happen when a key lookup fails, and that this can be used to create custom methods shared by many tables. For example we might want to be able to do this:

a = makevec2d(3, 4)
a:magnitude()  -- calculate the length of the vector, returning 5

In Lua this is not always necessary, for example, we could define an ordinary function to do the job for us:

function magnitude(vec)
    return sqrt(vec.x^2 + vec.y^2)
end
magnitude(a)  -- returns 5

In fact, for PICO-8 I would recommend that approach, because it's as efficient as you can get, and it uses the least number of tokens (PICO-8 cartridges are limited in code size).

But I think it's educational to see how metatables can make it possible to use Lua in a more OOP style.

First off, we define all our methods in a table somewhere. Note, you can define them in the metatable itself (this is a common convention), but I'll put them in a different table to prevent confusion.

local methods = {}
function methods.magnitude(self)
    return sqrt(self.x^2 + self.y^2)
end

The __index property of a metatable is referred to when you try to look up a key 'k' which is not present in the original table 't'.

If __index is a function, it is called like mt.__index(t, k)
If __index is a table, a lookup is performed like mt.__index[k]

So we can add the magnitude function, along any other methods we may have defined, to all our existing vector objects by simply setting the __index property to our table of methods:

mt.__index = methods

And now, as we wanted, we can call a:magnitude()
Which is a shortcut for a.magnitude(a)
Which is a shortcut for a["magnitude"](a)

Hopefully given all this information, it's clear what's happening: We never defined a magnitude property in 'a', so when we try to lookup the string "magnitude", the lookup fails and Lua refers to the metatable's __index property instead.

Since __index is a table, it looks in there for any property called "magnitude" and finds the magnitude function that we defined. This function is then called with the parameter 'a' which we implicitly passed when we used the : operator.

Well, that's it from me! I hope somebody finds this post useful, and please let me know if there is something you don't understand, or something that I left out or could have explained better. If you'd like to see more examples of metatable usage and OOP, I recommend chapters 13, 16 and 17 of Programming in Lua.

Next

• July 9th 2016 - LuaConf, Rio de Janeiro

  • Website
  • Call for presentations

• October 13th-14th 2016 - Lua Workshop, San Francisco

  • Website

Past

• March 8th - 1st Bay Area OpenResty meetup, San Francisco

  • Meetup page
  • Slides and Videos

• January 31st - Lua Devroom @ FOSDEM 2016, Brussels

  • Photos
  • Slides and videos
  • Video of panel: The Future of Small Languages

News

• May 4th - Lua 5.3.3 now available for testing
• May 2nd - Annoy (Approximate Nearest Neighbors) now has Lua bindings
• April 28th - Haxe introduces Lua target
• April 22nd - Book "Le guide et ses applications 2ed" published (French)
• April 4th - HexChat IRC client gets Lua scripting support

Featured Releases

• April 18th - LuaCov 0.11.0 (MIT)
• April 17th - Luacheck 0.15.0 (MIT)
• April 16th - Bencoding 2.2.0 (MIT)
• April 5th - Omnia 0.3.0 (MIT)
• Mar 20th - Ravi 0.15 (MIT)
• Mar 16th - OpenResty 1.9.7.4 (BSD)
• Mar 2nd - Nozzle (MIT)
• Feb 27th - Luaposix 33.4.0 (MIT)
• Feb 13th - Lua Matchtext 0.3.0 (MIT)
• Feb 13th - Lift 0.1.0 (MIT)
• Feb 12th - Specl 14.1.6 (MIT)
• Feb 12th - Wsapi-openresty 0.0.1 (MIT)
• Feb 7th - std.prototype 1.0.1
• Jan 28th - LGSL 0.1 (GPL-3)
• Jan 24th - Lua filesize 0.1.1 (MIT)
• Jan 22nd - ZeroBrane Studio 1.3.0 (MIT)
• Jan 9th - LuaRocks 2.3.0 (MIT)

Scripting of dialogues, animations and in-game events is a hugely important aspect for any RPG, and I wanted to build a scripting system that's easy to use and doesn't require any knowledge about the rest of the game's engine, but is also powerful enough for me to extend with new functionality as needed. This post aims to show how a few simple Lua features can be combined to create a scripting environment that's pleasant to use.

First, let's take a look at the XSE language used in Pokémon modding, as this was probably my main point of inspiration. It has a very straightforward, imperative style, even though each instruction doesn't correspond to a single function in-game.

By this I mean, the whole game engine doesn't freeze just because you are talking to an NPC, however there are points at which the dialogue script cannot progress until the text animations have finished and the player has pressed the [A] button.

Another interesting tool is Yarn, a dialogue editor in which you connect nodes of text together to form complete conversations. It has variables, conditionals and custom commands which you can hook up to different parts of your engine to trigger animations and such. I'd say it's definitely worth checking out especially if you're using Unity or similar.

So how would we go about creating such a system in LÖVE without creating our own language or writing an interpreter for an existing language such as Yarn?

Part 1: Chaining Callbacks Together

The first thing we need is the ability to 'say' some text from inside a script, which boils down to setting a string and then waiting for the user to press a button before we resume execution of the script. The game should still be updating on every frame, even when text is being displayed.

In true JavaScript fashion, we could create an asynchronous API that looks a bit like this:

text = nil
callback = nil

function say(str, cb)
    text = str
    callback = cb
end

Our game logic & rendering code could look something like this:

function love.update(dt)
    if not text then
        -- player movement code
    end
end

function love.draw()
    -- code to draw the world goes here

    if text then
        love.graphics.print(text, 10, 10)
    end
end

function love.keypressed(key, isRepeat)
    if text and key == "space" then
        text = nil
        if callback then
            -- execute the next part of the script
            callback()
        end
    end
end

Then we could write a dialogue script that looks like this, potentially fetching it at runtime with a call to dofile() or something:

say("Hello there!", function ()
    say("How's it going?", function ()
        say("Well, nice talking to you!")
    end)
end)

This kind of code grows unwieldy very quickly. It's confusing for non-coders and also error prone (many places to miss out a comma or a closing bracket). You could try some variations such as giving a name to each function, but it still turns out quite unpleasant to work with because managing all those functions gets in the way of what matters: writing good dialogue and scenes. At this point we'd surely be better off writing a Yarn interpreter or using some other existing solution.

But this is not JavaScript, and we can do better!

Part 2: Using Coroutines

For the uninitiated, coroutines are chunks of code that can be jumped to much like functions. A coroutine can suspend itself (yield) at will, returning to the point at which it was called. At a later stage, the program can jump back into the coroutine and resume where it left off.

I suppose this puts them in a sort of middle ground between functions and threads. They are more powerful than functions, but you still have to manage them explicitly - you can't just leave them running in the background to do their own thing. Typically they are used to break up an intensive task into small bursts, so that the program can still function as normal (receive user input, print to console, etc.)

Hang on a minute, doesn't this sound a lot like what we want from the dialogue scripting system? Executing a single line and then suspending the script while we give control back to the game loop?

Let's see how we could achieve the same result as Part 1, only using a coroutine instead of a chain of callbacks.

text = nil
routine = nil

function say(str)
    text = str
    coroutine.yield()
    text = nil
end

function run(script)
    -- load the script and wrap it in a coroutine
    local f = loadfile(script)
    routine = coroutine.create(f)

    -- begin execution of the script
    coroutine.resume(routine)
end

The important difference here is the implementation of the say function. Instead of setting a callback for later use, we tell the current coroutine to yield. This means we can't call say directly from the main program, only from inside a coroutine. Also there is now a loader function which creates a new coroutine and tells it to run the script.

Next we need to rewrite love.keypressed to make it resume the coroutine on the press of the space bar.

function love.keypressed(key, isRepeat)
    if text and key == "space" then
        if routine and coroutine.status(routine) ~= "dead" then
            -- execute the next part of the script
            coroutine.resume(routine)
        end
    end
end

And finally, we can write a script that looks like this:

say("Hello there!")                -- the script suspends once here
say("How's it going?")             -- it suspends again here
say("Well, nice talking to you!")  -- it suspends for the 3rd time here

Part 3: Sandboxing and Advanced Usage

If we declare a global variable, 'n', we can create an NPC that remembers how many times the player has spoken to it.

say("Hey kid, I'm Mr. Red!")

if n == 0 then
    say("I don't believe we've met before!")
else
    say("You have spoken to me "..n.." times!")
end

n = n + 1

It's great that this works, because it does exactly what you would expect and it's super easy to use. However, there are some problems.

If all the variables are stored in the global environment, we risk running into naming collisions which at best will cause scripts to behave incorrectly and at worst could replace key functionality and crash the game.

Additionally, having our game's state scattered across a ton of globals makes things very difficult when we want to think about serialising the gamestate to produce a save file.

Fortunately Lua makes it easy to swap out the environment of a function for any table, using setfenv in Lua 5.1 or _ENV in Lua 5.2 or greater. We don't need to change our scripts at all, we just need to make sure that they still have access to the say function, by placing it in their environment (the game table below).

game = {}

function game.say(str)
    text = str
    coroutine.yield()
    text = nil
end

function run(script)
    local f = loadfile(script)
    setfenv(f, game)
    routine = coroutine.create(f)

    -- begin execution of the script
    coroutine.resume(routine)
end

It also might be helpful to have a script that is called once at startup, to initialise all the game variables to default values, or load them from a save file.

As far as animation goes, we can drop in a tweening solution like flux, along with a few helper functions which will allow us to pause the script until the animation completes.

game.flux = require "flux"

game.pause = coroutine.yield

function game.resume()
    coroutine.resume(routine)
end

and then we could tween a character to x = 800 with a script like this:

flux.to(myNpc, 2.0, { x = 800 }):ease("linear"):oncomplete(resume)
pause()

which yes, is a mouthful for non-coders, and it introduces an asynchronous aspect back into the scripting API. We would probably benefit from a custom animation system that's more more tailored to our game, but this hopefully goes to show how easy it is to make scripts that can interact with any other part of the engine.

What Next?

I hope I was able to teach some interesting ideas here! I wanted to share this because coroutines are something I've known about for a while, but until now I've never had a good reason to use them. I would be interested to know which other languages can be used to create a system like this.

Here are some things you might want to do next, to create a more full-featured RPG engine:

• Add lock() and release(), so it's possible to display text while the player is moving, or stop the player from moving even when there is no text.
• Add an ask(str, ...) function whereby the player can choose from a list of options (e.g. yes/no)
• Download a level editor such as Tiled, or create your own. Try attaching some scripts to game objects such as buttons and NPCs. Relevant tutorial on using Tiled with LÖVE
• Create an easy-to-use animation system with commands such as 'face X direction' or 'move N steps'
• Add character portraits so that the player knows who's speaking (this might require you to add an extra parameter to say() or some new functions)
• Consider how you would go about handling save data. How to distinguish it from data which is part of the gamestate but does not need to be saved permanently?