Ezequiel Leonardo Castaño Personal Website

Type-Safe Pipelines and Compose

2023-09-23T00:00:00-03:00

When dealing with multiple transformations that run sequentially, one could leverage the compose function to create pipelines. This article shows different way to implement them in a type-safe way. All the code was written using pure Python 3.11 and MyPy 1.5.1 and PyLance v2023.9.20

Function Composition

There are two notations for function composition one with a right-to-left direction and one with a left-to-right direction. Namely:

Right to left:

$$f∘g = f(g(x))$$

Left to right:

$$f∘g = g(f(x))$$

The traditional notation in math is the right-to-left one. Some languages like Haskell has built-in syntax for both. In Haskell the usual way to do composition is with . which is right-to-left, it is also possible to use >>> for left-to-right composition.

On the other hand, one of the most common ways to compose functions in programming is through Unix Pipes which use the | operator, which conversely is left-to-right composition.

That means that when implementing a compose or a pipe function is not trivial which orientation to choose as they are competing notations. In the context of programming the left to right orientation is more common.

All the examples here will use the left-to-right orientation.

The compose function

The compose function is a function that takes two functions f and g and returns a new function that performs g and then f. In Python, a generic implementation looks like this:

def compose(f, g):
    def inner(*args, **kwargs):
        return g(f(*args, **kwargs))
    return inner

For this to work, it is necessary that the output of f matches the input of g, or otherwise the execution may fail. To enforce this relationship a typed version of the composed can be written as follows:

from typing import Callable, ParamSpec, TypeVar

T = TypeVar("T")
U = TypeVar("U")
P = ParamSpec("P")

def compose(f: Callable[P, T], g: Callable[[T], U]) -> Callable[P, U]:
    def inner(*args: P.args, **kwargs: P.kwargs) -> U:
        return g(f(*args, **kwargs))
    return inner

This implementation uses Python 3.10's ParamSpec, which allows typing *args and **kwargs.

Having a typed version is important as one typical case is composing multiple functions and if the types are not enforced, unexpected behavior could arise.

One example of using this compose implementation is the following:

def add_one(x: int) -> int:
    return x + 1

def convert_to_string(x: int) -> str:
    return str(x)

composed_functions = compose(add_one, convert_to_string)

When using the typed version, both MyPy and Pyright can infer the resulting type:

Pyright (through Pylance)

MyPy

One important aspect of function composition is that it is not commutative, that is

$$f∘g \neq g∘f$$

. There may be cases where both are equivalent but most of the time it is not.

This is helpful when programming as the IDE can automatically detect and flag types' incompatibility. Take for example trying to add_one to the result of convert_to_string.

composed_incorrect = compose(convert_to_string, add_one)

Pyright (through Pylance)

MyPy

The error from MyPy may not be descriptive enough but it at least points in the right direction. Pyright can identify the exact cause of the problem: add_one takes an int but it receives an str as the output of convert_to_string.

Extension to multiple functions

With that generic compose function, it is possible to compose any number of functions, however, the syntax is a little convoluted. These are the three sample functions to compose, one is int to int, another one is int to str and the last one is str to str, it is also interesting to use built-in functions such as len and str instead of user-defined ones.

def add_one(x: int) -> int:
    return x + 1

def convert_to_string(x: int) -> str:
    return str(x)

def sort_characters(x: str) -> str:
    return "".join(sorted(x))

The composition of interest is the one that would be equivalent to doing:

def user_defined_composition(x: int) -> int:
    return str(len(sort_characters(convert_to_string(add_one(x)))))

It is valid Python to create a function like that, but, it is difficult to read and to change since the "application order" as well as the final "type mapping" is hardcoded. Any variation would require a different signature and invocation order.

To leverage the generic compose, one alternative would be to write:

user_defined_composition = compose(
    compose(compose(compose(add_one, convert_to_string), sort_characters), len), str
)
assert user_defined_composition(2) == "1"

Or equivalently:

user_defined_composition = (
    compose(
        compose(
            compose(
                compose(add_one, convert_to_string),
                sort_characters
            ),
            len
        ),
        str
    )
)
assert user_defined_composition(2) == "1"

Note: both approaches are valid and threw no errors when scanned with MyPy, nevertheless, Pyright is not able to fully detect the types.

Pyright:

MyPy:

Neither of the previous is any easier to read than the original one. Some languages do have a compact syntax for this use case. Haskell's . is an example of clear and concise syntax (Haskell has right to left application):

user_defined_composition = str . len . sort_characters . convert_to_string . add_one

This by no means implies that Python is not a good fit for composing functions, it just lacks built-in syntax to make it more approachable. Even though the previous example would work without issues, in the following sections easier and more practical alternatives will be introduced.

Naive Iterative Pipe

The simplest evolution of compose to multiple functions in a cleaner way would be to wrap it in a for-loop:

from typing import Any, Callable

def pipe_iterative(*functions: Callable[..., Any]) -> Callable[..., Any]:
    composed_function, *rest = functions
    for function in rest:
        composed_function = compose(composed_function, function)
    return composed_function

user_defined_composition = pipe_iterative(
    add_one,
    convert_to_string,
    sort_characters,
    len,
    str
)
assert user_defined_composition(2) == "1"

This approach, albeit it will work, loses all the type information.

Pyright:

MyPy:

At the moment, it is not possible to type *functions so that Python infers that each of the functions will have an output type that will match the next functions's input type.

It is possible though to type the pipe_iterative function if we assume that all input and output types will be the same:

from typing import Callable, TypeVar

T = TypeVar("T")

def pipe_iterative(*functions: Callable[[T], T]) -> Callable[[T], T]:
    ...

This signature is only applicable to a small subset of the possible function compositions and will throw several errors with the functions used in the previous examples.

Pyright:

In this case, the types are compatible with the pipe_iterative signature but are not what the function will expect.

MyPy

MyPy detects the conflicting types and assumes Any will be the input and output types.

Disagreeing two typing tools like this may be misleading and can cause issues down the line. Pylance's error message is much more detailed and helpful in this instance.

If this type of composition is required though, there is a cleaner way of using the functools module.

Reduced Pipe

When a function takes an input type T, returns the same type, and has one identity element, it is said to be a Monoid. It is a common word in the Functional Programming jargon. The last signature of pipe_iterative follows this pattern since it always keeps its outputs and inputs within the type T.

Note: The definition of a Monoid includes the identity element, since pipes operate over functions, the identity element is a function that can be composed without altering the result. This is trivially doable in Python:

from typing import TypeVar

T = TypeVar("T")

def identity(x: T) -> T:
    return x

There is a function that achieves the same and is built-in, the functools.reduce. The pipe function can then be rewritten as follows:

from functools import reduce
from typing import Any, Callable

def pipe(*functions: Callable[..., Any]) -> Callable[..., Any]:
    return reduce(compose, functions)

user_defined_composition = pipe(
    add_one,
    convert_to_string,
    sort_characters,
    len,
    str
)
assert user_defined_composition(2) == "1"

The same considerations could be made about the signature, this implementation can either be type-safe and only operate on a single type T or it could be used more generically as the previous pipe_iterative while losing type information.

Since the signature is the same, MyPy and Pyright show the same messages.

There is a way to circumvent this issue and that is through the use of Generics. Where types become an additional parameter. In Python, Generics can only be used in classes, therefore the next sections will migrate to an object-oriented programming style.

Why Types matter

If the previous is a working implementation, why bother with adding types? There is a rising number of articles and online material arguing in favor of adding types. A summary could be the following.

Reasons to add types:

Typed code produces fewer bugs
Typed code enables enforcement of input and output compatibility
Typed code can be checked statically, i.e. without running the code
Typed code is easier to test comprehensively
Typed code is easier to validate. See Pydantic for example
Typed code is less ambiguous

There are also some reasons not to add types, namely:

Types make the code more verbose
Types make the learning curve for a codebase steeper
~~Types are not supported in all versions~~. All stable versions support types (3.8+).
~~Some Type constructs, like ParamSpec are not available in old versions~~. Use typing_extensions if something is not supported

And it is also important to clarify some myths

Types will not make the code slower, they are ignored at runtime
Types are not checked at runtime, they are just used by tools like IDEs and Linters

Eager Monoidal Pipe

Before diving too deep into different alternatives, the first step would be to replicate what the previous pipe function achieved but now supporting different types.

from __future__ import annotations

from dataclasses import dataclass
from typing import Callable, Generic, TypeVar

T = TypeVar("T")
U = TypeVar("U")

@dataclass
class PipeEagerSameType(Generic[T]):
    x: T

    def then(self, f: Callable[[T], T]) -> PipeEagerSameType[T]:
        self.x = f(self.x)
        return self

    def __call__(self) -> T:
        return self.x

user_defined_composition = (
    PipeEagerSameType(2)
    .then(add_one)
    .then(convert_to_string)
    .then(sort_characters)
    .then(len)
    .then(str)
)
assert user_defined_composition() == "1"
assert user_defined_composition.x == "1"

The implementation is straightforward, the class stores the initial value x, and then when then is called, that value is transformed and saved. This method returns self to provide a Fluent Interface. that makes the code easier to read.

The result at the end could be obtained by either calling the instance and thus executing __call__ or by simply accessing x. The former approach makes the instance behave like a function and the latter like a plain object.

This first implementation using object-oriented programming produces similar yet different messages from both Pyright and MyPy:

Pyright:

The errors are analogous to the implementation using reduce.

MyPy:

This time the errors are much more descriptive and resemble those produced by Pyright.

Now, whenever we inspect the type, both pyright and Mypy will output the class name and the type will be encoded in the Generic, i.e. the type now is PipeEagerSameType[int] and it is the [int] part which tells the actual type. In this case, it is the final return type which is inaccurate because the assumption that all functions return the same type and the first one returns int.

The next step would be to try to keep the same easy-to-read syntax while at the same time being able to pass functions with different signatures.

The Eager in the class name is no coincidence, the previous examples using compose were not eagerly evaluated, they had a lazy evaluation strategy. This implementation though transforms the value at each step. In the case of lazy evaluation, all the transformations are defined first and evaluated only at the end. Lazy implementations for this Pipe class will be introduced later on.

Eager Multitype Pipe

Due to the way Generics work, each instance can only hold a reference to a single type T. To support multiple types there are two possible solutions, either use as many TypeVars as types needed or use as many instances as types needed. The latter is more straightforward. The implementation looks like the following:

from __future__ import annotations

from dataclasses import dataclass
from typing import Callable, Generic, TypeVar

T = TypeVar("T")
U = TypeVar("U")

@dataclass
class PipeEager(Generic[T]):
    x: T

    def then(self, f: Callable[[T], U]) -> PipeEager[U]:
        return PipeEager(f(self.x))

    def __call__(self) -> T:
        return self.x

user_defined_composition = (
    PipeEager(2)
    .then(add_one)
    .then(convert_to_string)
    .then(sort_characters)
    .then(len)
    .then(str)
)

assert user_defined_composition() == "1"
assert user_defined_composition.x == "1"

This time, instead of returning self, a new instance is created, with the updated value. Each instance now has information on the current step (through T), and the following step (through U). The following step is not defined at instantiation and therefore it is not needed to add U to the Generic.

From the outside, the API is preserved, with the additional support for functions with different types.

Both Pyright and MyPy reflect types are properly inferred:

Pyright:

MyPy:

Now, with a compatible implementation, the type has become PipeEager[str], correctly inferring that calling the instance will return a str object.

The next step is to make a Lazy implementation resemble the first ones built with compose.

Lazy Pipe with Start

To make a lazy implementation, the value should not be stored and transformed, it will be provided last. Therefore what needs to be stored and mapped are the transformations.

To achieve this, a similar approach to the Eager classes can be used, but replacing the value x with a function f:

from __future__ import annotations

from dataclasses import dataclass
from typing import Callable, Generic, ParamSpec, TypeVar

T = TypeVar("T")
U = TypeVar("U")
P = ParamSpec("P")

@dataclass
class PipeLazyWithStart(Generic[P, T]):
    f: Callable[P, T]

    def then(self, g: Callable[[T], U]) -> PipeLazyWithStart[P, U]:
        return PipeLazyWithStart(compose(self.f, g))

    def __call__(self, *args: P.args, **kwargs: P.kwargs) -> T:
        return self.f(*args, **kwargs)

user_defined_composition = (
    PipeLazyWithStart(add_one)
    .then(convert_to_string)
    .then(sort_characters)
    .then(len)
    .then(str)
)

assert user_defined_composition(2) == "1"

In this case, rather than applying a transformation to the value and storing the transformed result, an instance is created with a new transformation which is the current and the next one composed. This allows the creation of new instances each composing on the previous ones.

The API is kept somewhat similar, it now resembles the approach with pipe_iterative, where all the functions are passed first and then evaluated on a particular value.

Pyright:

MyPy:

With the lazy implementation, the types have changed slightly, now the type is PipeLazyWithStart[(x: int), str], which means that it takes a single parameter called x of type int and returns a single value of type str. The x name comes from the instance variable with the same name in the class.

In case the advantages of lazy evaluation are unclear, one of the following sections will explain the pros and cons succinctly.

The next step would be to simplify the class so that it can be instantiated without an initial function, this could be helpful for many reasons, one example is testing.

Lazy Pipe

To remove the initial function, the first impression would be to mark f as Optional, however, has lots of caveats, and MyPy and Pyright will complain. A simple alternative is to create a helper class that surrogates the initial creation of the LazyPipeline.

from dataclasses import dataclass
from typing import Callable, Generic, ParamSpec, TypeVar

T = TypeVar("T")
P = ParamSpec("P")

@dataclass
class PipeLazy(Generic[P, T]):
    def then(self, f: Callable[P, T]) -> PipeLazyWithStart[P, T]:
        return PipeLazyWithStart(f)

user_defined_composition = (
    PipeLazy[[int], int]()
    .then(add_one)
    .then(convert_to_string)
    .then(sort_characters)
    .then(len)
    .then(str)
)
assert user_defined_composition(2) == "1"

This modification makes the implementation easily generalizable, being able to pass yet uninstantiated pipelines down the line. It also forces the user to specify the first input and output types of a given pipeline rather than infering it which could help prevent bugs as things are more explicit now.

Pyright:

MyPy:

Now the type is identical to the previous one but without specifying the x, that is because, at the moment of instantiation, the PipeLazy object takes no arguments. And even after being initialized with [[int], int] it accurately updated the return type to str by following the chain of compositions.

So far this implementation checks all the boxes, however, there is yet another way to implement similar behavior leveraging less common Python features. The next section will evolve this implementation to add support for decorators and operator overloading.

Composable Objects

This section is kept to showcase some of the features of the language, but it may result in code that is harder to read.

Classes as decorators is an intermediate-advanced concept in Python, in fact, PipeLazyWithStart can already be used as a decorator since it only takes f which is a function.

The other feature of the language that can be used is operator overloading, in this case, the >> operator (also known as right shift) which resembles the >>> in Haskell for left-to-right composition. Other valid alternatives could have been the | or the & (bitwise or and bitwise and respectively). Python does not support the definition of custom operators so having a new >>> operator is not feasible.

To overload the >> operator one needs to define a __rshift__ method, which in this case will have the same functionality as the then method. The only difference would be that it would need to handle functions and objects of the same class.

The implementation looks like the following:

from __future__ import annotations

from dataclasses import dataclass
from typing import Callable, Generic, ParamSpec, TypeVar

T = TypeVar("T")
U = TypeVar("U")
P = ParamSpec("P")
Q = ParamSpec("Q")

@dataclass
class Composable(Generic[P, T]):
    f: Callable[P, T]

    def __rshift__(self, other: Callable[Q, U]) -> Composable[P, U]:
        if not isinstance(other, Composable):
            other = Composable(other)
        return Composable(compose(self.f, other.f))

    def __call__(self, *args: P.args, **kwargs: P.kwargs) -> T:
        return self.f(*args, **kwargs)

@Composable
def convert_to_string_decorated(x: int) -> str:
    return str(x)

user_defined_composition = (
    Composable(add_one)
    >> convert_to_string_decorated
    >> Composable(sort_characters)
    >> len
    >> str
)
assert user_defined_composition(2) == "1"

The resulting syntax with >> is something new, in some cases, like when building a framework or an SDK, introducing new syntax like this could have great benefit. One project that does this is Apache Airflow. However, most often it results in niche uses that, without proper documentation could make the code harder to read and maintain.

Pyright:

MyPy:

The types are identical to those of PipeLazyWithStart, however, Mypy complains about not being able to infer the type of the first parameter of compose, which is unexpected. This may have been due to the interaction between ParamSpecs and TypeVars in an edge case.

Why does Lazy Evaluation matter?

Eager vs. Lazy evaluation is a topic that would deserve an article on its own. There are tools like polars and pandas that use eager evaluation and at the same time other tools like pyspark use lazy evaluation.

The following are some key questions to help decide which is better for each case:

Speed - Either
Easy to read code - Either
Easy to test code - Either
Easy to debug code / interactive programming - Eager
Optimized transformations - Lazy
Flexibility - Lazy
Separation of transformations from data - Lazy

Conclusion

There are scenarios where certain operations need to be applied sequentially, that is a hint for the pipeline pattern which can be implemented with either an Eager or a Lazy evaluation approach.

Knowing when to use which depends on the scenario but the key aspect is to structure the code so that the final result is easy to read, easy to test and easy to maintain.

On a personal note, I believe that unless required, the LazyPipe would suffice most of the time, and in case it doesn't switching to the Eager version is a minor change that should not introduce any breaks down the line.

If dealing with a lot of error handling, these pipelines could be combined with Railway Oriented Programming to streamline processes end to end.

ANGEL - An Oppinionated Guide on how to Learn Programming (focusing on Python)

2023-07-30T00:00:00-03:00

Courses and pieces of training claiming to deliver rapid results have become increasingly popular these days. Nonetheless, amidst the abundance of online resources, it can be challenging to distinguish valuable ones from the noise. Enter ANGEL, which stands for "ANGEL's Not a Guide to Effortlessly Learning." This guide presents readers with a practical approach to the programming learning experience. While the examples will be specific to Python, the reader can apply the core content to any programming language or framework.

Indeed, writing with both potential readers in mind is essential to ensure that the content resonates with learners and teachers alike. In this article, we'll address "the learner" as the protagonist in a more impersonal manner. Whenever "the learner" is mentioned, it refers to YOU if you are someone eager to learn, and if you are a teacher, "the learner" represents YOUR learners. By striking this balance, we can provide valuable insights and guidance to everyone involved in the learning process.

Programming is not an innate ability but rather a skill that requires deliberate development. Although not impossible to learn, it may not be necessary or suitable for everyone. This guide takes a meta-analysis approach from an "angel's" point of view (pun intended). Rather than providing a rigid set of precise resources or a specific order of topics, this guide concentrates on the learning journey. It offers valuable insights and tips to help learners avoid frustration, maintain engagement, and establish a clear framework for progression.

Some key ideas should be taken into consideration while reading:

Ownership: The learning process belongs to the learner. No one can guide them better in "learning to learn" than themselves. While teachers and mentors are valuable sources of support and guidance, the learner must take charge of discovering how they learn best, being actively involved and interested in shaping their learning process to make it more effective and enjoyable.
Overshooting and Undershooting: drawing from control theory, in practical terms, it means that being too strict or too lax in one's approach can hinder progress. While understanding the advantages and disadvantages of different method is vital, excessively deviating from a structured approach may lead to confusion and challenges. The learner should be open to exploring new things in a controlled and measured way while avoiding chaos.
Same Destination: when following a roadmap to learn something, it's crucial to recognize that not everyone will take the same path. Diversity in learning styles and methods is natural and should be embraced. Each learner's unique approach doesn't necessarily indicate superiority or inferiority in learning ability. With enough time and dedication, everyone can achieve a comparable level of proficiency, although individual objectives may lead to slightly different outcomes.

The sections in this article are thoughtfully structured as follows:

Learner Profiles: This section provides background information about the learners, including their previous experiences and personal goals related to programming.
Learning Approaches: Here, the post explores how the content is structured and organized, potentially outlining different learning paths or methodologies.
Thinking Style: This section delves into how learners process information and generate insights, which can impact their learning experience.
Learning Material: Considerations for selecting appropriate learning materials. The quality and relevance of the resources chosen can significantly impact the learning journey.
Roadmap: This roadmap serves as a clear path to follow, highlighting stages for learners to progress steadily in their programming competence.
Closing Thoughts: It wraps up the guide, giving learners a sense of direction and motivation to continue their learning journey.

SPOILER ALERT: It's essential to note that this guide isn't a quick and superficial "Learn X in Y minutes" type of resource. There are no TL;DR, summaries, or skippable sections. This post is thought for those genuinely invested and wanting to thoroughly read and absorb the content.

Learner Profiles

Individuals embark on learning programming for various reasons and bring diverse backgrounds to the table. Acknowledging these differences and adjusting expectations is vital to create a roadmap that facilitates effective knowledge acquisition.

Below, some typical learner stereotypes will be examined. While it's crucial to recognize that these stereotypes are not entirely accurate and may not encompass the full breadth of learner profiles, they provide a fair simplification that allows for better planning and organization.

Other Field Professional

Learners from diverse fields like finance, biology, marketing, or any non-IT domain, often seek to learn programming, particularly Python, to enhance their skill sets. Their motivation stems from specific goals, such as automating repetitive tasks, conducting data analysis, or developing applications relevant to their respective domains. While they might not prioritize high-quality code, their main objective is to achieve efficient and effective solutions that get the job done.

For instance, individuals with a background in the stock market may be interested in learning Python for algorithmic trading, as highlighted in this FreeCodeCamp Course with 2M views .

These learners typically possess expertise in their respective fields but may have limited or no programming background. They are focused on achieving tangible results and may prioritize functionality over writing high-quality code. PyData Conferences are a testament to the widespread adoption of Python among learners from non-IT fields, where many professionals showcase how they employ Python as an effective tool in their daily work.

Tinkerers

Python has a magnetic appeal for learners who have a tinkering spirit and love to experiment with code. These learners are naturally curious and enjoy the process of experimenting with code, embarking on personal projects, creating utilities, and exploring Python's vast potential for creative expression.

A prime example of this kind of learner is someone who aims to learn Python to automate tasks in their home using Raspberry Pi and delve into the world of domotics, as showcased in the Sentdex Guide with 500K views .

While they are keen on understanding what they are doing, they might not necessarily delve deeply into the theoretical aspects of programming. Instead, the focus lies on testing multiple solutions and optimizing for performance. They prefer practical and tangible outcomes rather than diving into the intricacies of the underlying algorithms.

Career Switchers / Leverage Seeker

These learners want to transition to a new career as their primary goal, often utilizing programming as leverage to achieve this change. They may or may not have a background in programming but are especially interested in maximizing returns with minimal effort.

An example could be someone feeling locked into their current job or career path. They are not entirely comfortable with their daily job and seek an exit strategy. As a result, they are often looking for efficient and rapid ways to acquire the necessary programming skills that can open new opportunities. As this FreeCodeCamp Course with 40M views shows

A typical subtype within this category is the high school or college student who solely wants to learn what is needed to pass an exam or fulfill specific academic requirements. Their focus is on attaining the necessary skills to achieve academic success.

Theorists / Scholars

Theorists and scholars, often coming from a Computer Science or related background, approach learning Python with a specific purpose – to implement algorithms and data structures with a strong emphasis on code analysis, particularly time and space complexity.

A classic example of this type of learner is a college student attending an Algorithms and Data Structures class or someone involved in research within this field. For instance, this MIT Course with 5M views .

Theorists and scholars typically have a narrow perspective on code, emphasizing performance, correctness, and theoretical analysis. They are less concerned with "idiomatic" or conventional coding practices and may not prioritize aspects related to software project management. Instead, they focus on maximizing the efficiency and effectiveness of individual pieces of code.

Professional Developer

Professional developers who work with Python or other programming languages bring a wealth of experience and software development knowledge to the table. For them, Python is another valuable tool in their arsenal that can be beneficial in specific scenarios and alongside other languages.

An exemplary case of this type of learner is someone who has already learned Python but is now looking to deepen their understanding of design patterns to enhance the robustness of their code. As this Arjan's Software Design Playlist with 250K views shows .

These learners often come with a set of pre-existing skills and practices from their background in software development. As a result, they might not fully embrace Python's "way of doing things," but they are dedicated to maintaining high-quality code and adopting a product-oriented approach.

Learning approaches

Learning approaches can vary widely across available resources, and not all styles may be suitable for every learner, regardless of their profile. In this section, we'll explore three of the most typical learning approaches:

Bottom-Up

The Bottom-Up learning strategy is characterized by starting from foundational concepts and gradually building upon them to understand more complex ideas. This approach breaks down concepts into smaller, easy-to-digest pieces, emphasizing a solid foundation before tackling more advanced topics.

Pros

Solid Foundation: Learners gain a reliable and solid foundation in programming. By understanding the smaller building blocks, they can construct more significant and complex solutions.
Easy-to-Digest: Focusing on manageable and easy-to-understand bits ensures learners grasp the fundamentals thoroughly, preventing information overload.
Progressive Learning: New knowledge is built upon previously taught material, creating a sense of continuity and progression in the learning journey.

Cons

Time to Tangible Results: The emphasis on learning basics and building blocks may delay seeing tangible results or practical applications, leading to frustration or an unproductive sensation.
Impostor Syndrome: Learners may develop impostor syndrome , feeling that they only know the building blocks and haven't built anything substantial, underestimating their capabilities.
Fragmented Knowledge: The focus on independent parts can hinder learners from seeing the big picture and understanding how different concepts relate.
Start from Scratch: This approach often involves starting from scratch, which may not align with real-world scenarios where learners might work with existing codebases.
Long-Term Retention: As learners progress through individual components, their knowledge can become fragmented and hinder long-term retention if not periodically reviewed.

Example

Most guides, tutorials, and even university courses often follow the bottom-up learning approach. They start by teaching foundational concepts such as variables, control structures, and data structures, and then progressively move to solving larger problems by reusing the previously taught concepts.

Proponents would say "You cannot start with X unless you cover W, Y, and Z in detail first."

Detractor would say one may be "Missing the forest for the trees".

Top-Down

The top-down learning approach focuses on starting with the "big picture" and making something work even if the learner doesn't fully understand all the underlying details. From there, they build on the working solution and delve into the specifics.

Pros

Avoiding Excessive Frustration: it helps avoid excessive frustration as learners start with a known solution, reducing trivial errors like syntax mistakes or wrong operators.
Fast Working Solution: Learners can achieve a functional result relatively quickly, gaining a sense of accomplishment early on.
Project-Based Approach: The top-down approach naturally adopts a project-based learning style, resonating well with many learners who prefer practical, hands-on experiences.
Database of Examples: By starting with working solutions, learners build a database of examples that can be leveraged in future projects, reducing the need for extensive long-term recall.

Cons

Dunning-Krugger Effect: Learners may be susceptible to overestimating their competence, believing they understand more than they do, leading to a false sense of mastery.
Difficulty in Troubleshooting: When faced with errors or issues, learners may struggle to troubleshoot effectively, lacking the foundational knowledge to explore the reasons behind them.
Starting from Scratch: Learners might experience challenges when starting a project from scratch, as they have accustomed to pre-filled boilerplate code during the initial phase.
Oversimplification of Concepts: In some cases, concepts may be oversimplified, hiding their true complexities. When learners deviate too much, they may encounter daunting intricacies.

Example

A prominent resource that follows the top-down approach is FastAI's Practical Deep Learning for Coders .

Proponents would say:

X is not hard, you just need W, Y, Z.
Don't worry if you don't understand now, it will become clearer later.

Detractor would say one may be "Ignoring the devil in the details".

Experimental

The experimental learning approach is characterized by a trial-and-error mindset. Learners focus on making a specific solution work without overly concerning themselves with alternative approaches or how individual building blocks interact.

Pros

Domain Expertise: The experimental approach allows learners to explore multiple paths, creating domain-specific expertise in the process.
Hands-On Learning: Learners consolidate their knowledge through hands-on experience, gaining practical insights into problem-solving.
Avoiding Dunning-Kruger Effect: By facing errors and challenges multiple times, learners avoid the Dunning-Kruger effect and develop a more realistic assessment of their skills.
Overcoming Impostor Syndrome: Encountering both failures and successes reinforces learners' confidence, preventing impostor syndrome.
Reinforcement and Faster Recall: This approach strengthens learners' understanding of solutions for similar problems. They can recall how certain problems were solved more efficiently in the future.

Cons

Time-Consuming: The experimental approach can be time-consuming, as learners try out different approaches and solutions.
Domain-Specific Knowledge: The knowledge acquired through experimentation may be highly domain-specific and not easily applicable to other areas.
Fragmented Knowledge: Learners may end up with fragmented knowledge, understanding how pieces fit together for specific solutions but struggling to apply them in different contexts.
Bias Towards Past Solutions: Having invested many hours in trying to solve a particular problem, learners might prematurely dismiss options that were previously discarded, limiting their exploration of alternative solutions.

Example

One example of the experimental learning approach can be found in the YouTube channel "Stuff Made Here" .This channel tackles novel problems where previous solutions are suboptimal, and the creator explores multiple attempts to gain experience and refine the approach. This channel effectively combines software and hardware aspects, making it a valuable resource for hands-on learning.

Another scenario where the experimental approach is commonly used is in hackathons. Participants often explore different solutions and experiment with various technologies to find the most suitable and innovative solutions within a short timeframe.

Proponents would say "You just need to sit down and focus on it until it works".

Detractor would say one could be "A one-trick pony".

Thinking Style

Understanding the thinking style of learners is crucial in presenting information in a way that resonates with them and enhances their learning experience. People can be categorized into two main thinking styles:

Analytic Thinkers: These individuals prefer to break down complex information into smaller, more manageable details and subgroups. They excel at analyzing and dissecting concepts to understand their intricacies thoroughly. They often appreciate a bottom-up learning approach, where they start with foundational concepts and gradually build a comprehensive understanding.
Holistic Thinkers: Holistic thinkers, on the other hand, find it easier to see the grand scheme of things at once. They prefer to see how various concepts are interconnected and how they create a cohesive structure. For holistic thinkers, a top-down learning approach, starting with an overview and then delving into specifics, may be more appealing.

Additionally, information organization preference can be categorized into two styles:

Hierarchical Thinkers: Hierarchical thinkers prefer taxonomies and hierarchical structures to organize information. They find it natural to group concepts into categories, subcategories, and broader groups, similar to organizing files in a directory structure.
Networked Thinkers: Networked thinkers see concepts as independent ideas interconnected through relationships, much like social networks. They prefer to understand how various concepts relate to one another rather than organizing them hierarchically.

Authors and educators might naturally lean towards one thinking style or information organization. The alignment between the author's presentation style and the reader's thinking style can significantly impact how well the information is understood and absorbed.

If you're interested in exploring this topic further, the resources:

Learning Material

There is no one-size-fits-all approach, and diversity in learning resources can be beneficial for learners. Different individuals may prefer various formats, including videos, text-based materials, MOOCs (Massive Open Online Courses), 1:1 mentoring, interactive exercises, and more. Each format has its advantages and appeals to different learning styles and preferences.

The idea of a "visual learner" has been challenged , as the most effective learning experiences often involve a combination of various approaches. The more diverse the learning methods, the better learners can engage with the material and enhance their understanding.

Active participation is a crucial aspect of effective learning. Engaging learners and making them actively participate in the learning process can significantly improve knowledge retention and understanding. Gamification, a technique that uses game elements and mechanics in a non-gaming context, is one approach that teachers and educators can use to make learning more interactive, enjoyable, and effective, especially when dealing with complex concepts.

Attention Spans

In today's fast-paced digital age, attention spans have been decreasing, and there is a growing demand for short and easily digestible content, especially on social networking sites. However, presenting complex information in such a brief format can be challenging, as it may require oversimplifications or assume a significant amount of prior knowledge.

Python tutorials at events like PyCo often last for more than 3 hours because they aim to provide hands-on and in-depth learning experiences. Learning programming concepts and skills requires time and practice, and it cannot be condensed into quick soundbites or short videos. Programming is a skill that needs to be nurtured and honed through active engagement and practical application.

Noise to Signal Ratio

With the vast amount of information available on the internet, distinguishing between valuable and mediocre resources can be challenging, especially for complete beginners. This concept essentially refers to the ratio between high-quality, valuable resources (the signal) and less useful or irrelevant information (the noise).

For complete beginners, it's essential to exercise caution and be selective about the sources they rely on for learning. Beginners may lack the experience and knowledge to distinguish credible and accurate resources from those containing errors or outdated information.

When reviews are not available, learners can use proxy indicators to gauge a resource's value:

How many people reviewed it positively? (e.g. at Coursera/Udemy/Pluralsight)
How many people watch it? (e.g. on Youtube)
How many people like it? (e.g. on Medium)
Where was it published? (e.g. at a PyCon)
Is this official documentation?

Recommendations

Be wary of recommendations, as it is particularly tricky due to expert blindness . Expert blindness can influence how experienced individuals perceive and recommend certain materials or approaches. What works well for them at their current level of expertise might not be the most suitable choice for beginners who lack that prior knowledge.

For example, recommending a fully-fledged IDE to a beginner might overwhelm them with its complexity and hinder their learning process. While it might be an invaluable tool for an experienced programmer, it might not be the best starting point for someone just beginning their programming journey.

Assessing which learning material, roadmap, or approach to use can be subjective, and what works for one person might not work as effectively for another. Learners should take experts' advice with a grain of salt.

Learning Git is a classic example of the challenges with recommendations. Git is a powerful version control system, and learning resources are numerous. What might be a "game-changer" for one person might not be the best fit for someone else. Beginners often benefit from exploring multiple resources and finding the ones that resonate best with their learning style.

"Aha!" moments can be transformative, but they can differ for each individual due to their unique backgrounds and existing knowledge. Some learners might have breakthroughs with a specific resource, while others might find that same resource less helpful.

Self Identification

Understanding one's self-identification as a learner is crucial as it can guide the choice of learning approaches and resources that best suit individual needs. It's common for individuals to identify with one or more types of learners, and these identifications may change over time as one gains more experience and knowledge.

For example, those who identify as "Tinkerers" may naturally gravitate towards the "Experimental" approach, enjoying hands-on exploration and experimentation. "Theorists" may lean towards the "Bottom-Up" approach, delving into the theoretical aspects and code analysis. "Other Field Professionals" might prefer the "Top-Down" approach, seeking to apply Python to their specific domains. "Professional Developers" and "Career Switchers" might find benefit in various approaches, depending on their specific interests and goals.

This post is primarily oriented toward a combination of the "Theorist" and the "Professional Developer" learner types. It is not a quick "Learn Python in X days/months" guide for "Career Switchers," nor a "How to automate things with Python" tutorial for "Tinkerers." However, depending on te learner's interests and background, those identifying as an "Other Field Professional" may still find valuable insights and benefits in this guide.

Working in a Team

Working in a team can yield the best results when there is a diverse mix of profiles among team members. Assuming equal competence, having a combination of analytical and holistic thinkers can enrich the team's experience more than if everyone had the same thinking style.

As one climbs higher in the organizational hierarchy, the need for a more holistic approach becomes apparent. Higher-level positions require considering problems in the context of features, products, portfolios, strategies, and overall value and profit. C-Suite executives often work at the strategy level, where specific feature details may be abstracted away.

As organizations grow, this hierarchical nature becomes more pronounced. This also explains the distinction between Individual Contributors (IC) and Managers. Individual Contributors typically possess analytical thinking skills, capable of handling complex tasks with a single-minded focus. In contrast, Managers tend to be people-centric and adopt a holistic approach, keeping the big picture in mind while managing details.

Transitioning from an IC to a Manager role is not always a seamless process, and not everyone may find it suitable for them. Some individuals might prefer to remain in IC roles, as they excel in handling complexities and prefer focused tasks. On the other hand, those who embrace a more holistic mindset and enjoy managing teams and projects may thrive in managerial positions.

Understanding these dynamics can help shape the career outlook for learners. By recognizing their strengths and preferences, learners can make informed decisions about the type of role and career path they want to pursue.

The Roadmap

The remainder of the article will explore an approach to learning from scratch through different levels of competence. The idea is to divide the learning process into different stages and give specific recommendations for each stage. Before continuing, remember that "All models are wrong, but some are useful"

Knowledge vs Competence

The distinction between knowledge and competence is crucial when it comes to learning programming. Learning materials typically focus on imparting knowledge, which comprises information and facts about a specific topic. However, learners, employers, and everyone involved ultimately seek competence, which refers to the ability to apply that knowledge effectively in real-world situations.

A challenge arises when evaluating competency, as there are limited tools available for its assessment compared to the abundance of tools to test knowledge. Job interviews, for instance, are often time-boxed, limiting the depth of competency assessment. As a result, proxies such as quizzes, code challenges, or design questions are commonly used. Evaluating soft skills is also a valuable proxy, as they are essential for competence in certain roles, such as effective communication for a skilled software architect. To read more about this topic, have a look at Fagner Brack's article .

Passing knowledge tests, such as exams, does not automatically imply competency. Competency evaluation is a more time-consuming process that involves analyzing an individual's strengths and weaknesses in practical situations. Due to its complexity, it does not scale well for large groups of learners. As a result, most teaching processes involving multiple learners primarily assess knowledge, with some individuals developing competency to varying degrees.

Mentorships stand out as an exception, as they focus on testing competency rather than knowledge. A mentor's full attention is on a single learner, allowing for a deeper assessment of their real-world problem-solving skills.

For those seeking job opportunities, online portfolios can be a better proxy for demonstrating competency. Portfolios showcase projects that require the expected level of competence, and they offer a chance for learners to prepare and present their abilities before interviews. While this is not a foolproof method, it provides a more tangible representation of a learner's potential compared to mere completion certificates, regardless of the hours invested in a course. One exception to this is official certifications, which are carefully designed by a company to ensure professionals are competent in a specific set of tools, e.g. Microsoft, AWS, Cisco, etc.

For complete beginners, knowledge, and competency are often acquired concurrently. However, once competency is established, it can be transferred to other domains with relative ease. Learning a second programming language, for instance, becomes more accessible as learners have already grasped general problem-solving with programming, and they only need to adapt to the specific syntax of the new language. One exception is when assumptions no longer hold, for instance, Haskell (or any other pure functional programming language) has some unique features that require special competencies that may not be easily obtainable in other languages.

This roadmap focuses on how to achieve different levels of competence, for that, consider the Four Stages of Competence plus some personal contributions:

Unconscious Incompetence: one is unaware of programming and does not realize that it is a learnable skill. They might not even recognize the significance of programming in today's world.
Conscious Incompetence: individuals become aware of programming and its importance. However, they struggle to write code and acknowledge their lack of proficiency. They are conscious of their limitations and the need to improve their skills.
Conscious Competence: they can write code, but it requires concentration and deliberate thinking. They are competent programmers, but the process still demands effort and focus.
Unconscious Competence: programming becomes second nature to learners. They can write code effortlessly, with deep knowledge and skill. Their proficiency allows them to navigate complex problems seamlessly.

Additional Contributions:

Reflective Competence: Learners at this stage continually evaluate their skills and actively seek to identify their weaknesses. They are self-aware and strive to go beyond their comfort zone, taking on challenging projects and seeking opportunities for growth and improvement.
Replicative Competence: Individuals who have achieved replicative competence have a profound impact on those around them. They become mentors and role models, helping others grow their level of competence exponentially. Their expertise and support empower others to excel in their learning journeys.

Unconscious Incompetence - Take Off

This stage is a universal starting point for every learner, where one realizes, "I don't know how much I don't know." At this juncture, learners cannot even articulate what or how to learn next, and it can be a critical phase in the learning process. The risk of feeling overwhelmed, frustrated, and disappointed is high, potentially leading to abandoning the learning journey altogether.

To navigate this crucial stage successfully, I strongly recommend adopting a Bottom-Up approach. Dedicate around 100 hours to immerse oneself in pure Python without involving third party libraries, frameworks, or tools other than the language itself.

Numerous online resources are available to assist individuals in commencing their coding endeavors, providing a range of options from which to choose the most comfortable platform. Utilizing fast-feedback online environments is recommended, as they allow one to experiment with code and observe errors without encountering the intricacies of installation and setup. Avoiding such initial technical challenges can prevent premature disappointment.

Here are some good free online environments for learning Python (not sponsored):

Single File:

Multi File:

Important note: Refrain from relying on help tools such as forums, chatbots, or other guides during this stage. Instead, focus on understanding the fundamentals of Python - what things are, how they work, and why they behave as they do. Using AI Chat or similar aids at this point can be likened to learning multiplication with a calculator; it hinders the learning experience. Save these tools for later when a more solid foundation has been laid out.

This stage could be colloquially called "Take Off," as the learner leaves the ground of total inexperience and begin to graduate from being completely unaware to having a growing awareness of the Python language. Embrace this initial learning phase with determination and patience, knowing it will lay the foundation for future programming prowess.

Conscious Incompetence - Consolidation

Having spent a significant number of hours learning or possessing a background in programming, learners at this stage can now say, "I know what I don't know." It may still seem like a daunting task to achieve complete understanding, but there is a clear awareness of what lies ahead in the learning journey.

This awareness could be a result of existing programming knowledge in other languages, prior experience with Python, or expertise in other fields that serve as a good proxy to estimate the upcoming complexity. Regardless of the source, there has been significant improvement from the previous stage.

At this point, I recommend continuing with the Bottom-Up approach, but now exploring more complex topics, such as popular Python libraries, frameworks, or design patterns. Focus on well-known and reputable sources, avoiding experimental or hobbyist projects. Additionally, start engaging with some Top-Down material that integrates the use of these resources, understanding how various pieces work together.

By now, the learner should have a proper Python installation on their local PC, and dealing with installation challenges is acceptable, as long as they have learned how to install Python from scratch. However, there is no need to delve into virtual environments or dependency locking just yet; that can be left for the next stage.

For instance, if the learner aims to become a back-end developer, they could explore the following resources:

Learn how APIs work (Bottom-Up)
Learn how Flask work (Bottom-Up)
Learn how to structure a Flask app (Bottom-Up)
Develop an ToDo list with Flask (Top-Down)

The focus should remain on learning rather than building fully functional projects at this stage. Familiarize with the tools and gain intuition on how to accomplish the desired tasks. This is also a suitable time to start slowly incorporating tools like AI Chats to aid in the learning process.

The Consolidation stage may span a couple of months, involving trial and error and iterative learning, but since the basics are well understood, navigating through challenges should not be a problem. This stage is aptly named "Consolidation" as learners solidify their understanding and expand their programming horizons.

Conscious Competence - Portfolio

One has reached a stage where one understands how the tools work with a comfortable level of proficiency. The majority of the time was spent learning, and now it is time to put that knowledge into practice. Bottom-Up materials may start to feel repetitive, as they cover familiar content.

In this stage, one finds oneself spending more time reading documentation, exploring forums, and seeking answers to specific problems rather than following beginners' tutorials. One has reached the "Intermediate Level," where one possesses a significant amount of knowledge but may still be lacking some hands-on experience to fully demonstrate one's competence.

At this point, it is crucial to shift the focus to "Top-Down" resources. Start creating a portfolio of projects, building things that one understands and can confidently explain to others. Share progress on social media or through a blog, or even consider building a blog as part of a portfolio.

Some tasks may still be challenging, and the learner might rely on online tutorials to guide them. Exposure to a variety of projects will help one gain intuition about how things are done and improve one's problem-solving skills. However, try to avoid starting everything from scratch. Incorporate virtual environments and dependency-locking tools like pipenv, poetry, or similar, to better manage one's projects. This stage is also an excellent opportunity to learn Git and publish projects as public repositories on GitHub.

Most of the practical building happens at this stage, which is why it is known as the "Portfolio" stage. While it is fine to experiment with some new frameworks, the bulk of the building should be focused on well-established frameworks and tools. Building a strong portfolio during this stage will not only enhance one's competence but also provide valuable material to showcase skills to potential employers or clients.

Unconscious Competence - Independence

At this stage, the learner has reached a level of competence where building projects and navigating the programming world feel like second nature. The individual can confidently and effortlessly build projects, and their understanding of how things work is deep-rooted.

However, there is a caveat to this stage. Maintaining this level of competence requires continuous practice and engagement. While knowledge may stay with individuals longer, competency can rust over time if not consistently exercised. This stage can be risky, as it might lead to stagnation, either intentionally or unintentionally. When becoming a master at something, a comfort zone naturally forms around it, and the effort invested in reaching this level can become a barrier to moving beyond it.

At this stage, learners can consider themselves "Independent." They are likely not beginners anymore, but it is essential to avoid limiting oneself to the comfort zone. Gradually expanding horizons to include more topics and challenges is recommended. Seeking mentors with extensive knowledge and experience to provide guidance is beneficial. However, it is crucial to keep in mind that expecting others to invest time without compensation may not be realistic. Learners should be proactive in their learning journey.

This is an excellent time to continue with the Top-Down approach and start incorporating Experimental learning. Embracing new challenges and exploring unfamiliar territories is essential. Staying curious and open to learning new things is advised.

Nonetheless, this stage represents the first "stable" stage in the learning journey. Individuals might feel content and comfortable with their current skills and knowledge, and that's okay. However, remembering that staying in the comfort zone for too long can make it difficult to make significant moves in the future is important. Maintaining a balance between expertise and continuous growth is essential. An example of someone at this stage might be an expert in an older, less popular language like COBOL. They are competent and independent in their field and may not feel the need to switch to other technologies as long as COBOL is still relevant. But this situation is fragile, as the longer one stays in the same niche, the harder it becomes to transition to something different.

Always remembering that expertise is not a free ticket to rest on one's laurels is crucial. Continuing to push oneself to learn and grow, and being open to embracing new challenges and opportunities is necessary. Continuous and conscious action is necessary to prevent unconscious competence from fading into unconscious incompetence.

Reflective Competence - Research and Development

The original model of four stages of competencies does not include a fifth stage, but some authors have proposed one called "Reflective Competence." At this stage, individuals not only possess a skill to the point where they can perform it effortlessly and automatically but also can reflect on their performance and continuously improve.

In this stage, one becomes aware and understands their thought process, a concept known as metacognition. This stage includes the meta-competency of proactiveness, where individuals can self-assess, take initiative, anticipate future needs, adapt in advance, and actively seek feedback. Reflective Competence represents the highest level of individual competency.

In the technology world, where change is constant, it is crucial to stay proactive and avoid becoming outdated. To do so, I suggest continuously pushing oneself outside of the comfort zone, but not too aggressively. Engage in learning new frameworks, trying new tools and methodologies, mentoring others, starting side projects, getting involved in the community, and contributing to open-source projects. This stage is known as "Research and Development," and it's a time to enter "Experimentation" mode.

Examples of activities during this stage include:

If one has mastered relational databases, one can try learning ORMs or NoSQL databases.
If one has mastered Keras, one can explore learning FastAI or Bayesian Frameworks.
If one has mastered Pytest, one can delve into learning Hypothesis.
If one has mastered Flask, one can consider learning FastAPI.
If one has mastered Type Hints, one can try learning Pydantic.
If one has mastered their field's favorite library, one can try fixing some issues or developing new features.

Individuals in this stage can act with just small pieces of information; a simple idea may be enough to spark the whole process, and they can keep the momentum going. In the industry, these individuals are often referred to by various names such as "Leaders," "Seniors," "Tech Leads," or "Mentors/Coaches," among others. They are valued for their extensive knowledge and their ability to guide and mentor others.

Replicative Competence - Exponential Growth

The stage of Replicative Competence is not formal and is a personal contribution. Once an individual reaches Reflective Competence, there is only one way to continue growing: horizontally. This stage assumes that one does not operate in isolation and has people around them.

Individuals at this stage not only continue to grow themselves but also foster an environment for efficient, engaging, and motivating growth for everyone around them.

This involves several competencies that go beyond the technical side and may not be something desired by everyone, as previously mentioned, not everyone wants to leave the IC role. These individuals are characterized by being people who C.A.R.E.S:

C - Competence in the Domain: They are experts in the specific domain or skill they aim to multiply in others.
A - Adaptive Communication: They have effective and adaptable communication skills, ensuring clear and relatable knowledge transfer.
R - Relational Empathy: They build strong relationships and connect with others' needs and emotions.
E - Encouraging Patience: They cultivate patience and persistence to support and guide individuals through their learning journey, even during challenging moments.
S - Supportive Coaching: They provide constructive feedback, coaching, and mentorship, fostering growth and critical thinking in others.

By having an individual at this level on a team, they make it possible for other team members to grow at a rate that may not have been achievable otherwise. They enable others and, by doing so, produce "Exponential Growth" within the team.

This is the stage that any person aspiring for a leadership position should aim for. People at this level are influential, they lead by example, and they create a subculture within an organization. They are sought after, and they actively seek opportunities, not just for themselves, but for the people around them. They are a never-ending fountain of knowledge and experience, driving continuous improvement and development in those they interact with.

Self-Assessment and Criticism

Self-assessment is indeed a crucial aspect of the learning process, and it's important to acknowledge that the roadmap presented in the article may not fully capture the diverse experiences and perspectives of all learners. While the article provides a structured approach, it's essential to recognize that learning is a personal journey, and each individual may have unique preferences, goals, and constraints.

Begineers do not Care

The approach presented in the article is indeed geared towards learners who are willing to invest time and effort in their learning journey and are looking for a structured and goal-oriented plan.

The content's focus on programming and Python is meant to provide concrete examples and guidance for learners in the technology field, but the underlying principles and concepts can be adapted to various domains of learning. Emphasizing that the content is intended for individuals who want a long-term plan and are committed to their learning journey is crucial. This approach aims to empower learners to take ownership of their learning, be proactive, and set realistic goals for continuous improvement.

Indeed, the value delivered by this content lies in the long-term return and the gradual internalization of the learning process. By adopting this approach and incorporating it into their learning style, learners can develop a deeper understanding and competence in their chosen field, leading to more meaningful and sustainable growth.

Ultimately, the approach provided in the article serves as a guide and a framework, but it is up to each learner to personalize and adapt it according to their needs, preferences, and learning pace. The willingness to take ownership and actively engage in the learning process will greatly influence the overall success and fulfillment of the journey.

This is Utopian and Unrealistic

Work, family, travel, personal obligations, and various external factors can often create time constraints and obstacles that make it difficult to follow a structured learning plan consistently.

No guide can indeed address all possible angles and individual circumstances, and the approach presented in the article may not be feasible for everyone in every situation. Life is unpredictable, and there will always be unforeseen challenges and disruptions that can hinder progress.

However, the value of having a long-term goal and a structured learning approach lies in the adaptability and resilience it fosters in learners. While it may be challenging to dedicate extensive time solely to learning, even small steps taken consistently can lead to significant progress over time. It is about recognizing that progress may be gradual, and it is okay to make adjustments to fit learning into one's existing lifestyle and responsibilities.

Flexibility and adaptability are essential qualities to develop, and having a loose plan or roadmap can help individuals stay focused and motivated amid the chaos of life. It provides a sense of direction and purpose, even if the pace of learning is slower than originally intended.

Moreover, acknowledging the existence of obstacles and challenges can help learners be kinder to themselves and avoid unnecessary pressure or guilt when they face setbacks. It is essential to remember that progress is not always linear, and it is okay to take breaks, reassess goals, and adapt the learning approach as needed.

In the end, the ability to cope with the challenges of the real world and maintain a commitment to learning is a valuable competency worth cultivating. Embracing the journey as an ongoing process of growth and self-improvement can lead to a more fulfilling and meaningful learning experience.

Closing Thoughts

In conclusion, learning is a deeply personal and individualized journey. There is no one-size-fits-all approach, and it is crucial to be aware of your strengths, weaknesses, and learning style. Building a roadmap that aligns with your preferences and comfort levels can lead to a more effective and enjoyable learning experience.

While there are abundant learning resources available online, it is essential to approach them with a critical mindset. Not all information is reliable, and not all approaches will work for everyone. Seeking feedback from experienced individuals and questioning your assumptions can help you refine your learning path and avoid falling into confirmation bias.

Moreover, it is crucial to strike a balance between seeking guidance and exploring on your own. Relying solely on pre-packaged solutions may limit your growth, but venturing into uncharted territory without any guidance may lead to frustration and discouragement. Embrace a mix of structured learning and experimentation to foster a well-rounded and adaptable skill set.

Lastly, be open to challenging your beliefs and exploring different perspectives. Exposing yourself to diverse ideas and viewpoints can help you grow intellectually and creatively. Embrace a growth mindset, be receptive to feedback, and never stop seeking opportunities for learning and improvement.

In the end, the journey of learning is an ongoing and enriching process. Embrace the challenges, celebrate your progress, and keep pushing yourself to reach new levels of competence. Remember that the path to mastery is not always smooth, but it is the continuous pursuit of knowledge and self-improvement that leads to true expertise.

Python Recursion: a Trampoline from the Mutual Head to the Memoized Nested Tail

2023-05-21T00:00:00-03:00

Recursion is a key concept of programming. However, it is usually only superficially explored. There are different ways of having recursion, this post will illustrate them using Python examples, call graphs and step-by-step runs. Including cases of head, tail, nested and mutual recursion. For each case, the call graph will be shown.

Content covered in this post:

An old friend: The Factorial
Recursion
- Why Recursion?
Direct vs Indirect Recursion
Linear Recursion
- Tail Recursion
Multi Recursion
Indirect Recursion
Recursion related techniques
Conclusion

An old friend: The Factorial

Many if not all programming courses introduce the factorial function at some point. This function has great mathematical importance and yet it is simple enough to showcase how recursion works. However, the approach towards it and recursion, in general, is usually superficial.

Before digging into recursion, a procedural implementation using for-loops and while loops will be shown.

Side Note

This post abuses the fact that, in Python, when a function is defined multiple times only the last definition is used for future references. There will be many refinements over the definitions and to avoid any confusion, names will not be changed to reflect that, they all do the same. To further reinforce this idea, an assert statement will be added to show that results do not change even if the definition changes.

Run Step by Step Online

def factorial(n: int) -> int:
    """Factorial function implemented using for loops"""
    result = 1
    for i in range(1, n + 1):
        result *= i
    return result

assert [factorial(i) for i in range(7)] == [1, 1, 2, 6, 24, 120, 720]

And next the while loop equivalent:

Run Step by Step Online

def factorial(n: int) -> int:
    """Factorial function implemented using while loops"""
    result = 1
    multiplier = n
    while multiplier != 0:
        result *= multiplier
        multiplier -= 1
    return result

assert [factorial(i) for i in range(7)] == [1, 1, 2, 6, 24, 120, 720]

Between the for loop and the while loop implementation differences are visible. The for-loop approach is usually the one found in many sources online, it is short, uses only basic constructs and does the job. Whereas the while approach uses one extra variable, that being said, both are valid and share the same time and space complexity.

Another possibility, not as common as the previous ones, is a functional implementation using reduce:

Run Step by Step Online

from functools import reduce

def factorial(n: int) -> int:
    """Factorial function implemented using reduce"""
    return reduce(lambda x, y: x * y, range(1, n + 1), 1)

assert [factorial(i) for i in range(7)] == [1, 1, 2, 6, 24, 120, 720]

Since the previous implementations are non-recursive, the call graph consists of a single node:

Recursion

After introducing one of the previous definitions of the factorial function, the "recursive form" is usually presented. A recursive function is a function that calls itself. There are multiple types of recursion though, and understanding them may have a huge impact on some programming languages. Before showing what the recursive version of the factorial looks like, it is important to clarify some concepts.

Why Recursion?

Recursion in and of itself is a wide field in computer science that may attract scientists and researchers. However, it is sometimes portrayed as a difficult topic and receives less attention than other techniques.

Although some may avoid recursion altogether, there are clear and noticeable benefits of using recursive functions, such as:

Declarative style: Recursive functions are written by thinking about what the function does instead of how it does it. The iterative style usually leads the programmer to think about low-level details like indexes and pointers whereas recursion brings the whole problem into mind.
Simplicity and Readability: Recursive functions can provide a more elegant and concise solution for solving complex problems by breaking them down into simpler subproblems. The recursive approach often closely resembles the problem's definition, making the code more intuitive and easier to understand.
Divide and Conquer: Recursive functions can leverage the divide and conquer technique, where a problem is divided into smaller subproblems that are solved independently. This approach simplifies the problem-solving process by reducing complex tasks into manageable pieces.
Code Reusability: Recursive functions are often reusable. Once implemented, they can be called multiple times with different inputs, allowing for efficient and modular code. Recursive functions can be applied to various instances of the same problem, enabling code reuse and promoting good software engineering practices.
Handling Recursive Structures: Recursive functions are especially useful when dealing with recursive data structures, such as trees or linked lists. The recursive nature of the data can be mirrored in the recursive functions, making it easier to traverse, manipulate, or process such structures.
Mathematical and Algorithmic Modeling: Many mathematical and algorithmic problems are naturally defined in terms of recursion. Recursive functions provide a natural and direct way to express these problems, making the code more closely aligned with the underlying mathematical or algorithmic concepts.
Time and Space Optimization: Recursive functions can often lead to more efficient algorithms in terms of time and space complexity. By breaking down a problem into smaller subproblems, recursive functions can avoid unnecessary computations by reusing previously calculated results (memoization) or eliminating redundant iterations.

Direct vs Indirect Recursion

Most commonly, when one says "recursive function", it is meant "direct recursive function", that is, the function calls itself. The other way a function could be recursive is through "indirect recursion" where, instead of calling itself, it calls another function (or chain of functions) that will in turn call the first function.

Different types of direct recursions worth mentioning are:

Based on where the recursive call is done:

Head Recursion
Middle Recursion
Tail Recursion

Based on the number of recursive calls:

Linear Recursion
Multi Recursion (also called nonlinear or exponential recursion)
- Tree Recursion (also called bifurcating recursion)
- Nested Recursion
- General Non-Linear Recursion

Based on the number of functions involved:

Direct Recursion (a single function)
Indirect Recursion (multiple functions, also called mutual recursion)

Besides the previous classification, all recursive functions must have a termination condition or else they would enter in an infinite loop. Even though recursive functions do not need to be pure (i.e. they do not have side effects), it is common for recursive functions to be pure, this simplifies the interpretation. All the examples in this article are pure functions.

Linear Recursion

Linear recursion refers to functions where there is only one recursive call.

Based on the position of the recursive call it could be further subdivided into:

Head Recursion: recursive call is the first statement.
Middle Recursion: there are other statements before and after a recursive call.
Tail Recursion: recursive call is the last statement.

There is no difference between Middle Recursion and Head Recursion from an efficiency and algorithmic perspective. So no further exploration will not be done on those two.

When a function has more than one recursive call is called Multi Recursion, Nonlinear Recursion or Exponential Recursion. These cases will be covered in a later section.

The following is an example of a middle recursion implementation of the factorial function.

Run Step by Step Online

def factorial(n: int) -> int:
    """Middle Recursion implementation of the factorial function"""
    if n == 0:
        return 1
    return n * factorial(n - 1)

assert [factorial(i) for i in range(7)] == [1, 1, 2, 6, 24, 120, 720]

It is middle recursion because the last statement is a multiplication (*) and not the recursive call itself. Depending on the operation order it could also be considered head recursion but that difference is not relevant for most contexts.

Another way to better show why this is middle recursion is to use additional variables to store interim results:

Run Step by Step Online

def factorial(n: int) -> int:
    """Middle Recursion implementation of the factorial function"""
    if n == 0:
        return 1
    previous_factorial = factorial(n - 1)
    current_factorial = n * previous_factorial
    return current_factorial

assert [factorial(i) for i in range(7)] == [1, 1, 2, 6, 24, 120, 720]

In this more explicit implementation, it is clearer that the last logical statement is the multiplication n * previous_factorial.

The call graph in the case of linear recursive functions is a series of nodes called sequentially, hence the name:

When the last statement is the recursive call, the function is called tail recursion, which will be explored in the next section.

Tail Recursion

Tail recursion is when the return statement of the function is only a recursive call, this means that a function call could be replaced with another function call directly. Some languages (Python is not one of them), use a technique named Tail-Call Optimization, which makes this particular type of recursion very efficient.

One important clarification is that the return must not be an expression. An example of a straightforward function that can be implemented in a tail recursive way is the palindrome function:

Run Step by Step Online

def palindrome(string: str) -> bool:
    "Returns True if the given string is a palindrome. Using tail recursion."
    if len(string) < 2:
        return True

    first, *rest, last = string
    if first != last:
        return False
    return palindrome(rest)

assert palindrome("a")
assert palindrome("aa")
assert palindrome("aba")
assert not palindrome("learn")
assert palindrome("rotator")

To better illustrate the fact that the returning statement must be only a function call, the following implementation is NOT a tail recursive function because the last statement is not a function call, it is a boolean expression that requires the function call to be executed before returning. The reason is the and operator which needs a value. This implementation is therefore a middle recursion.

Run Step by Step Online

def palindrome(string: str) -> bool:
    "Returns True if the given string is a palindrome. Using middle recursion."
    if len(string) < 2:
        return True

    first, *rest, last = string
    return first == last and palindrome(rest)

assert palindrome("a")
assert palindrome("aa")
assert palindrome("aba")
assert not palindrome("learn")
assert palindrome("rotator")

Sometimes a function that is not expressed in tail-call form can be converted to that form. For example the following middle recursive function:

Run Step by Step Online

def sum_integer_up_to_n(n: int) -> int:
    """Sums all integers from zero to n. Using middle recursion"""
    if n == 0:
        return 0

    return n + sum_integer_up_to_n(n - 1)

assert sum_integer_up_to_n(1) == 1
assert sum_integer_up_to_n(3) == 6

Can be rewritten into tail-recursive form as:

Run Step by Step Online

def sum_integer_up_to_n(n: int, total: int = 0) -> int:
    """Sums all integers from zero to n. Using Tail recursion"""
    if n == 0:
        return total

    return sum_integer_up_to_n(n - 1, total=n + total)

assert sum_integer_up_to_n(1) == 1
assert sum_integer_up_to_n(3) == 6

This last version uses an additional parameter to pass the total down the call chain. This compromises readability for performance if the language implements tail-call optimization. This style of programming is widely used in languages like Prolog and some purely-functional languages.

In Python however the extra parameter can be hidden by using default values, this makes the implementation more similar to the original but it is implicitly hiding the way it truly works, which is against many coding styles. Use with caution.

In the same way as sum_integer_up_to_n, the factorial function could be re-written into a tail recursive form:

Run Step by Step Online

def factorial(n: int, result: int = 1) -> int:
    if n == 0:
        return result

    return factorial(n - 1, n * result)

assert [factorial(i) for i in range(7)] == [1, 1, 2, 6, 24, 120, 720]

When comparing head/middle with tail recursion, the way each approach works differs and can be illustrated by inspecting the execution step by step:

# Head/Middle Recursion
factorial(3)
3 * factorial(3 - 1)
3 * factorial(2)
3 * 2 * factorial(2 - 1)
3 * 2 * factorial(1)
3 * 2 * 1 * factorial(1 - 1)
3 * 2 * 1 * factorial(0)
3 * 2 * 1 * 1
6

# Tail Recursion
factorial(3)
factorial(3 - 1, 3 * 1)
factorial(2, 3)
factorial(2 - 1, 3 * 2)
factorial(1, 6)
factorial(1 - 1, 6 * 1)
factorial(0, 6)
6

Multi Recursion

When there is more than one recursive call, a function is said to be multi-recursive. Multi-recursive functions can also be middle/head recursive or tail-recursive. A special case of Multi Recursion is when the recursive call is one of the arguments, in this case, it is referred to as nested recursive.

General Non-Linear Recursion

Many functions do not follow a precise pattern and they just use multiple recursive calls as part of their definition. One such example is a function that returns the nth Fibonacci number, to call the function two successive recursive calls are used.

This is the traditional implementation:

Run Step by Step Online

def fibonacci(n: int) -> int:
    if n == 0:
        return 0
    if n == 1:
        return 1
    return fibonacci(n - 1) + fibonacci(n - 2)

assert [fibonacci(i) for i in range(7)] == [0, 1, 1, 2, 3, 5, 8]

In some cases, multi-recursive functions can be refactored into linear tail recursive functions.

Run Step by Step Online

def fibonacci(n: int, partial_result: int = 0, result: int = 1) -> int:
    if n == 0:
        return 0
    if n == 1:
        return result
    return fibonacci(n - 1, result, partial_result + result)

assert [fibonacci(i) for i in range(7)] == [0, 1, 1, 2, 3, 5, 8]

Tree Recursion

In the case of multi-recursive functions, it is possible to construct a tree of the function calls. All multi-recursive functions produce a tree, that being said, in some cases the definition leverages the divide-and-conquer strategy, minimizing the depth of the tree. One example of this is the quicksort algorithm:

Run Step by Step Online

from typing import Collection

def quicksort(numbers: Collection[float]) -> list[float]:
    if len(numbers) <= 1:
        return numbers

    first, *rest = numbers

    left = [x for x in rest if x < first]
    right = [x for x in rest if x >= first]
    return quicksort(left) + [first] + quicksort(right)

assert quicksort([2, 4, 3, 5, 0, 1]) == list(range(6))
assert quicksort(list(reversed(range(10)))) == list(range(10))

Here the function divides the input into two parts and each recursive call gets one of the parts. This strategy reduces the number of recursive calls by a great amount.

Converting non-tree recursion into tree recursion

Some functions that are originally linear recursive can be converted into tree recursive to reduce the depth of the recursive stack. This is usually easy on functions that operate on arrays.

Here is a linear recursive implementation of a function that returns the maximum value of a list:

Run Step by Step Online

from typing import Iterable

def maximum(numbers: Iterable[float]) -> float:
    first, *rest = numbers
    if not rest:
        return first

    rest_max = maximum(rest)
    return first if first > rest_max else rest_max

assert maximum([2, 4, 3, 5, 0, 1]) == 5
assert maximum(list(range(10))) == 9

This holds even if re-written into a tail-recursive form:

Run Step by Step Online

from typing import Optional, Iterable

def maximum(numbers: Iterable[float], max_value: Optional[float] = None) -> float:
    first, *rest = numbers

    if max_value is None or first > max_value:
        max_value = first

    if not rest:
        return max_value

    return maximum(rest, max_value)

assert maximum([2, 4, 3, 5, 0, 1]) == 5

Both implementations will have as many recursive calls as there are elements in the list. A similar approach to the quicksort algorithm can be used to reduce the number of calls, halving the length of the list each time. With this approach, the recursive stack will be shorter.

Run Step by Step Online

from typing import Collection

def maximum(numbers: Collection[float]) -> float:
    first, *rest = numbers
    if not rest:
        return first

    middle = len(numbers) // 2
    left_max = maximum(numbers[:middle])
    right_max = maximum(numbers[middle:])
    return left_max if left_max > right_max else right_max

assert maximum([2, 4, 3, 5, 0, 1]) == 5
assert maximum(list(range(10))) == 9

Refactoring functions this way is not always possible, for functions like nth Fibonacci, it is not trivial to use a tree approach that reduces the number of recursive calls. A known solution called Fast Doubling has been discovered. Deriving this implementation requires a lot of effort and mathematical knowledge, such an approach may not apply to other functions.

The Fast Doubling Implementation is as follows:

Run Step by Step Online

def fibonacci(n: int) -> int:
    # Fast Doubling Method
    if n == 0:
        return 0
    if n == 1:
        return 1

    is_odd = n % 2
    m = n // 2 + is_odd
    fib_m = fibonacci(m)
    fib_m_1 = fibonacci(m - 1)

    if is_odd:
        return fib_m_1**2 + fib_m**2
    return 2 * fib_m * fib_m_1 + fib_m**2

assert [fibonacci(i) for i in range(7)] == [0, 1, 1, 2, 3, 5, 8]

It is even possible to further reduce the number of recursive calls by converting the multi-recursive function into a linear recursive function by changing its structure to return two values at once:

Run Step by Step Online

def fibonacci(n: int) -> int:
    # based on: https://www.nayuki.io/page/fast-fibonacci-algorithms
    def nth_and_nth_plus_one(n: int) -> tuple[int, int]:
        if n == 0:
            return 0, 1

        fib_m, fib_m_1 = nth_and_nth_plus_one(n // 2)
        nth = fib_m * fib_m_1 * 2 - fib_m**2
        nth_plus_one = fib_m**2 + fib_m_1**2

        is_odd = n % 2
        if is_odd:
            nth_plus_two = nth + nth_plus_one
            return nth_plus_one, nth_plus_two
        return nth, nth_plus_one

    nth, _ = nth_and_nth_plus_one(n)
    return nth

assert [fibonacci(i) for i in range(7)] == [0, 1, 1, 2, 3, 5, 8]

Even though in these examples with fibonacci(4) the difference is not drastic, the number of total calls in the call graph scales in notoriously different ways for each implementation, take for example fibonacci(100):

Typical Multi Recursive Implementation: 1,146,295,688,027,634,168,203 calls ≈ 1 sextillion calls
Fast Doubles: 163 calls
Tail Recursive Implementation: 100 calls
Linear Recursive Implementation: 8 calls

Nested Recursion

One special case of multi-recursion is when the argument of the recursive call is itself a recursive call. This is not usual in software development but could arise in mathematical fields. One example is the Hofstadter G Sequence:

Run Step by Step Online

def hofstadter_g(n: int) -> int:
    if n == 0:
        return 0
    return n - hofstadter_g(hofstadter_g(n - 1))

assert [hofstadter_g(i) for i in range(7)] == [0, 1, 1, 2, 3, 3, 4]

Refactoring nested recursion into non-nested multi-recursion or linear recursion is a non-trivial task and sometimes it may be impossible.

Triple Nested Recursion

The level of nesting is not limited to just two calls, the Hofstadter H Sequence has triple nesting recursion for example:

Run Step by Step Online

def hofstadter_h(n: int) -> int:
    if n == 0:
        return 0
    return n - hofstadter_h(hofstadter_h(hofstadter_h(n - 1)))

assert [hofstadter_h(i) for i in range(6)] == [0, 1, 1, 2, 3, 4]

Nested Recursion with more than one argument

Some functions can have multiple arguments and be nested recursive. One example is the Ackermann function grows extremely fast due to this nested recursive definition and it is the simplest function proved not to be primitive recursive, meaning that it cannot be expressed in iterative form with for loops.

This function is currently used to test compilers' efficiency at handling really deep recursive functions. This is a case of a nested recursive function that is also tail recursive.

Run Step by Step Online

def ackermann(m: int, n: int) -> int:
    if m == 0:
        return n + 1
    if m > 0 and n == 0:
        return ackermann(m - 1, 1)    
    return ackermann(m - 1, ackermann(m, n - 1))

assert [ackermann(i, j) for i in range(3) for j in range(3)] == [1, 2, 3, 2, 3, 4, 3, 5, 7]

Indirect Recursion

So far, all the examples showed functions that called themselves, this is direct recursion. An alternative approach is indirect recursion, also known as mutual recursion. In this case, the same categories could be applied (linear, tail, head, nested, etc.), but the recursive call is now to another function, that other function will, in turn, call the original one.

Mutual Linear Recursion

A simple example of mutual linear tail recursion is a set of functions that determines if a number is odd or even:

Run Step by Step Online

def is_even(n: int) -> bool:
    if n == 0:
        return True
    return is_odd(n - 1)

def is_odd(n: int) -> bool:
    if n == 0:
        return False
    return is_even(n - 1)

assert [is_even(i) for i in range(6)] == [True, False, True, False, True, False]
assert [is_odd(i) for i in range(6)] == [False, True, False, True, False, True]

Of course, it is also possible to implement a function that computes the same in a non-recursive form. However, this example does not require division or modulo computation, it is much slower for big numbers.

Mutual Multi Recursion

Mutual recursion can also happen in multi-recursive functions. Take the following direct multi-recursive function that computes the nth Lucas number:

Run Step by Step Online

def lucas(n: int) -> int:
    if n == 0:
        return 2
    if n == 1:
        return 1
    return lucas(n - 1) + lucas(n - 2)

assert [lucas(i) for i in range(7)] == [2, 1, 3, 4, 7, 11, 18]

It is possible to write both the Lucas and the Fibonacci functions in a mutual recursive form:

Run Step by Step Online

def lucas(n: int) -> int:
    if n == 0:
        return 2
    if n == 1:
        return 1
    return 2 * fibonacci(n) - fibonacci(n - 1) + lucas(n - 2)

def fibonacci(n: int) -> int:
    if n == 0:
        return 0
    if n == 1:
        return 1
    return (fibonacci(n - 1) + lucas(n - 1)) // 2

assert [lucas(i) for i in range(5)] == [2, 1, 3, 4, 7]
assert [fibonacci(i) for i in range(5)] == [0, 1, 1, 2, 3]

This implementation is standalone and does not require any of the two functions to be defined in a direct recursive way. In practical terms, there is no gain as it makes the whole computation slower and less efficient, it is just for demonstration purposes.

Similarly, the sequence defined as the multiplication of the last two terms can be implemented in a direct multi-recursive form:

Run Step by Step Online

def multiply_last_two(n: int) -> int:
    if n == 0:
        return 1
    if n == 1:
        return 2
    return multiply_last_two(n - 1) * multiply_last_two(n - 2)

assert [multiply_last_two(i) for i in range(7)] == [1, 2, 2, 4, 8, 32, 256]

This again can be used to implement the Fibonacci and the multiply last two as mutually recursive functions.

Run Step by Step Online

import math

def fibonacci(n: int) -> int:
    if n == 0:
        return 0
    if n == 1:
        return 1
    return int(math.log2(multiply_last_two(n - 1) * multiply_last_two(n - 2)))

def multiply_last_two(n: int) -> int:
    if n == 0:
        return 1
    if n == 1:
        return 2
    return 2 ** (fibonacci(n - 1) + fibonacci(n - 2))

assert [fibonacci(i) for i in range(7)] == [0, 1, 1, 2, 3, 5, 8]
assert [multiply_last_two(i) for i in range(7)] == [1, 2, 2, 4, 8, 32, 256]

Mutual Nested Recursion

Mutual recursion can also appear in a nested form, as is the case of the Hofstadter Female and Male sequences which are mutually nested recursive.

Run Step by Step Online

def hofstadter_female(n: int) -> int:
    if n == 0:
        return 1
    return n - hofstadter_male(hofstadter_female(n - 1))

def hofstadter_male(n: int) -> int:
    if n == 0:
        return 0
    return n - hofstadter_female(hofstadter_male(n - 1))

assert [hofstadter_female(i) for i in range(6)] == [1, 1, 2, 2, 3, 3]
assert [hofstadter_male(i) for i in range(6)] == [0, 0, 1, 2, 2, 3]

Mutual Triple Recursion

Indirect recursion is not limited to only two functions, the following example combines the Lucas, Fibonacci and multiply last two functions in a triple mutual recursive form, where each function uses the other two and itself.

Run Step by Step Online

import math

def lucas(n: int) -> int:
    if n == 0:
        return 2
    if n == 1:
        return 1
    return (
        2 * math.log2(multiply_last_two(n - 1) * multiply_last_two(n - 2))
        - fibonacci(n - 1)
        + lucas(n - 2)
    )

def fibonacci(n: int) -> int:
    if n == 0:
        return 0
    if n == 1:
        return 1
    return (fibonacci(n - 1) + lucas(n - 1)) // 2

def multiply_last_two(n: int) -> int:
    if n == 0:
        return 1
    if n == 1:
        return 2
    return 2 ** (1.5 * fibonacci(n - 2) + 0.5 * lucas(n - 2))

assert [lucas(i) for i in range(6)] == [2, 1, 3, 4, 7, 11]
assert [fibonacci(i) for i in range(6)] == [0, 1, 1, 2, 3, 5]
assert [multiply_last_two(i) for i in range(6)] == [1, 2, 2, 4, 8, 32]

Recursion is a wide field, and due to language, compilers and general developer experience limitations, several tools and techniques have been developed to improve recursion support and performance.

Memoization

When dealing with recursive functions, it is important to keep track of the call stack and optimize it to avoid wasting resources. Many recursive functions call themselves multiple times with the same parameters in their call graph, these repeated calls can be cached (assuming the function is pure) to avoid (1) continuing traversing a recursive tree unnecessarily and (2) returning the result in constant time. This technique of caching previously computed results is called memoization.

Memoization is easy to implement in Python, both from scratch and using the standard library.

The from-scratch implementation can be written using decorators:

from functools import wraps
from typing import Callable, TypeVar, ParamSpec

T = TypeVar("T")
P = ParamSpec("P")

def memoize(function: Callable[P, T]) -> Callable[P, T]:
    cache: dict[str, T] = {}

    @wraps(function)
    def wrapper(*args: P.args, **kwargs: P.kwargs) -> T:
        nonlocal cache

        cacheable_args = str(tuple(sorted(args)) + tuple(sorted(kwargs.items())))
        if cacheable_args not in cache:
            cache[cacheable_args] = function(*args, **kwargs)
        return cache[cacheable_args]

    return wrapper

Another possibility is to use the standard library, which has support for memoization through functools.cache, this function is available in Python 3.9 or later. For older versions, it is also possible to use functools.lru_cache, which also adds the capability of setting a max limit of cached entries.

Memoization Examples

This section will show what the call graph of the above examples looks like when memoization is applied.

Take for example the following call graph for a multi-recursive implementation of fibonacci(5):

When using memoization the total number of calls is reduced significantly (from 15 calls to 9):

Run Step by Step Online

Depending on the implementation, the effect of memoization is similar to linearizing the multi-recursive function, as the tree has much fewer branches while the depth is kept the same.

If considering the Fibonacci Fast Doubles implementation of fibonacci(10):

This can also be reduced (from 15 calls to 11):

Run Step by Step Online

Memoization can also be applied to nested recursive functions such as the hofstadter_g(4):

Now memoized (from 19 calls to 9):

Run Step by Step Online

Or deeply nested recursive functions like the hofstadter_h(3):

And now memoized (from 22 to 10 calls)

Run Step by Step Online

The same applies to more complex functions like the Ackermann function with Ackermann(2, 3):

And now memoized (from 44 calls to 23):

Run Step by Step Online

Memoization can also be used for mutual recursive functions, the following examples show the mutual fibonacci-lucas recursion and the Hofstadter female-male

Multi recursive fibonacci-lucas:

And now memoized (from 26 to 13):

Run Step by Step Online

And the hofstadter female-male recursion:

And now memoized (from 15 to 11 calls)

Run Step by Step Online

Taking the fibonacci(100) example from the previous section, when incorporating the memoized approach the results change substantially:

Typical Multi Recursive Implementation: 1,146,295,688,027,634,168,203 calls ≈ 1 sextillion calls
Memoized Typical Multi-Recursive Implementation: 199 calls (0.00000000000000002% of the original)
Fast Doubles: 163 calls
Tail Recursive Implementation (Memoization has no effect): 100 calls
Memoized Fast Doubles: 29 calls (17.79% of the original)
Linear Recursive Implementation (Memoization has no effect): 8 calls

Since the tail recursive and the linear recursive implementation do not have repeated calls, memoization has no effect.

Trampolining

Another restriction that many languages (such as Python) have is call stack overflow, this happens when there are too many recursive calls in the call stack. It is possible with minimal modifications to the original functions to surpass this limitation and make the language treat the recursive function as an iteration and thus bypass the overflow. This technique is called trampoline and requires the function to be implemented in tail-recursive form.

Moreover, the function should defer the evaluation of the tail call by using an anonymous function (in Python called lambdas). This step is needed to simulate a lazy evaluation.

The following is an example of how to implement the tail recursive Fibonacci using trampolining for fibonacci(10000) which would normally cause RecursionError.

Run Step by Step Online

from __future__ import annotations

from typing import TypeVar, Callable, TypeAlias

T = TypeVar("T")
TrampolineFunction: TypeAlias = "Callable[[], T | TrampolineFunction[T]]"

def trampoline(function_or_result: T | TrampolineFunction[T]) -> T:
    if not callable(function_or_result):
        return function_or_result

    while callable(function_or_result):
        function_or_result = function_or_result()

    return function_or_result

def fibonacci(
    n: int, partial_result: int = 0, result: int = 1
) -> int | TrampolineFunction[int]:
    if n == 0:
        return 0
    if n == 1:
        return result

    return lambda: fibonacci(n - 1, result, partial_result + result)

assert str(trampoline(fibonacci(10000))).startswith("3364476487")

This way of programming has several similarities with the continuation passing style since instead of executing the command, the function defers the execution to another function which in the end runs the command. In Object Oriented Programming similar behavior could have been achieved using the Command Pattern.

This particular implementation has a significant drawback, it hinders debugging and makes it less transparent, as can be seen in the run step-by-step example. This is the reason why the author of Python rejected the proposal to incorporate tail call optimization into the language.

The implementation above using lambdas is typical of other languages like Javascript. In Python, it is possible to implement it in a different way following Guido's approach which makes debugging easier and less cryptic. On the other hand, types become a bit more convoluted.

Run Step by Step Online

from __future__ import annotations

from typing import TypeVar, Callable, TypeAlias, Iterable, cast

T = TypeVar("T")

TrampolineFunction: TypeAlias = "Callable[..., T | TrampolineResult[T] | T]"
TrampolineArgs = Iterable[T]
TrampolineResult: TypeAlias = "tuple[TrampolineFunction[T], TrampolineArgs[T]]"

def trampoline(function: TrampolineFunction[T], args: TrampolineArgs[T]) -> T:
    result = function(*args)

    while isinstance(result, Iterable):
        result = cast("TrampolineResult[T]", result)
        function, args = result
        result = function(*args)

    return result

def fibonacci(
    n: int, partial_result: int = 0, result: int = 1
) -> tuple[TrampolineFunction[int], tuple[int, int, int]] | int:
    if n == 0:
        return 0
    if n == 1:
        return result

    return fibonacci, (n - 1, result, partial_result + result)

assert str(trampoline(fibonacci, (10000,))).startswith("3364476487")

A combination of the previous two approaches can also be achieved by using partial evaluation through functools.partial. This way, the code is simpler and more similar to what one could find in other languages while still being easy to debug:

Run Step by Step Online

from __future__ import annotations

from functools import partial
from typing import TypeVar, Callable, TypeAlias

T = TypeVar("T")
TrampolineFunction: TypeAlias = "Callable[[], T | TrampolineFunction[T]]"

def trampoline(function: T | TrampolineFunction[T]) -> T:
    if not callable(function):
        return function

    while callable(function):
        function = function()

    return function

def fibonacci(
    n: int, partial_result: int = 0, result: int = 1
) -> int | TrampolineFunction[int]:
    if n == 0:
        return 0
    if n == 1:
        return result

    return partial(
        fibonacci, n=n - 1, partial_result=result, result=partial_result + result
    )

assert str(trampoline(fibonacci(10000))).startswith("3364476487")

Trampolines allow to convert of recursive function calls into iterations. There is no built-in support like with the memoized technique and due to technical limitations, it is not possible to implement them as decorators, which would reduce the changes needed on the original function. The different implementations shown should give a grasp of what is possible and how it could be applied to other functions.

It is also possible to modify the default configuration of the interpreter to allow deeper recursions though. This can be done by setting a higher value to the sys.setrecursionlimit. This method requires however access to the sys module which may not be always available or editable.

Call-By-Need

Some programming languages execute instructions following a non-strict binding strategy, that is, the parameters of a function are not evaluated before the function is called. One such strategy is called call-by-need, which only evaluates the parameters when needed in the body of the function and caches them in case they are re-used.

When using recursive functions in languages that support call-by-need (like Haskell or R), the execution could be optimized as only a subset of all the recursive calls might be evaluated, thus reducing the cost of recursive functions.

Conclusion

Recursion is a diverse topic that can be used as leverage to overcome challenging problems, improve readability and facilitate the use of recursive data structures. In Python, it is possible to use recursion with some limitations, those could be mitigated by using memoization or trampolines. Other languages may use different strategies such as call-by-need or tail-call optimization.

Every program written in iterative form can be rewritten in recursive form, it is a good exercise to practice and gain intuition about recursion and when it can be applied.

In the end, recursion is another tool, it can be used everywhere but that is not necessarily the best decision always, analyzing the context and the cost-benefits is important.

If recursion is the way to go, aim for tail recursion if possible or to leverage memoization.

Some challenges for the reader to practice recursion:

Write a function that determines if a number is prime
Write the addition, multiplication and exponentiation functions
Write a function that computes a running average of a sequence of numbers

Add Google Analytics (or any custom HTML) to Streamlit with Github Pages

2022-01-14T00:00:00-03:00

Streamlit is one of the most popular libraries for creating web apps using Python, and they also provide a service to freely host and share your apps. One of the issues with this free sharing capability is that one losses control over the app, for instance, it is no longer possible to control how the precise HTML will look like or add custom Open Graph or analytics. This post shows a simple and easy way to add all of this by using Github Pages.

This post is one part of a three posts series regarding Streamlit, the posts can be read in any order and the three are built around the same demo app:

Add Google Analytics (or any custom HTML) to Streamlit with Github Pages (this post)
Multipage, State-Persistent Apps with Streamlit
Lessons learnt After Developing Finance Web Tools with Streamlit and Altair (No HTML/CSS/JS)

TL;DR

This post covers how to upload a Streamlit app to the Streamlit Sharing platform and embed it as an iframe in a custom HTML hosted in Github Pages. This is NOT suitable for sensitive data or commercial use.

The technique used here is a hack, meaning it is not the best practice, but it will get the job done. In the end, a more robust and professional alternative will be mentioned for those interested.

An example Github Repo is available for inspiration, there is also a demo app attached to check the final result.

Initial Clarification

This post assumes the Streamlit App is hosted via Streamlit Sharing since if the app is being hosted on its own servers, there should not be a problem to add tracking, Open Graph, or any custom HTML since one has full control over the code and deployment. Instead, this post focuses on a purely free and managed solution with no servers involved.

Streamlit offers a free-forever Community plan for hosting and sharing an unlimited number of public Streamlit apps, the full plan comparison is available on their page. To use that free service one has to create a free account and upload all the code to a Github Repo (Gitlab, BitBucket, and Azure DevOps only in Paid plans at the moment).

Deployment

To deploy a Streamlit app it is important not to have any uncommitted changes, otherwise, streamlit will block the deployment. If all changes have been committed (and pushed), first run the app locally, typically with:

streamlit run python_script.py

And then go to the Hamburger Menu -> "Deploy this app"

This will redirect to the Streamlit Sign in Page and after logging in, the Deployment Options will be shown, here the repository, the branch, and the file should be specified. It is also possible to add additional parameters (and even secrets) in the "Advanced Settings", there is a tutorial to do this in the docs

Once the app is deployed it should be available at a URL with this format:

https://share.streamlit.io/GITHUB_USERNAME/REPOSITORY_NAME/MAIN_SCRIPT.py

That URL is completely functional but it has the drawbacks mentioned in the introduction, namely it is not possible to:

Add any kind of tracking/analytics
Customize the Open Graph meta tags for social media sharing
Add custom HTML

It is not possible however to alter the Javascript, the HTML, or the CSS of the streamlit app itself because it is in a domain (share.streamlit.io) to which no access is granted. Nevertheless, analytics and meta tags are not "visible" content and should therefore not disrupt the typical user experience of the app. The main goal is to provide the final user with a seamless "Streamlit Experience" while enhancing the features on the developer's end.

Github Pages

One advantage of Github repositories is that they come with a static webserver directly built-in called "Github Pages". It is disabled by default, but can be enabled by going to Settings -> Pages -> Enable. For simplicity, the master branch will be used but other branches can be used if configured.

Once enabled, the root of the repository will be served at this URL:

http://GITHUB_USERNAME.github.io/REPOSITORY_NAME

That means that an index.html can be added to the root of the repo and that file will be the landing page of the URL.

A basic HTML as the following can be used for testing:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta http-equiv="X-UA-Compatible" content="ie=edge">
    <title>My App</title>
  </head>
  <body>
    <h1>Hello World</h1>
  </body>
</html>

Embedding Streamlit

Once both the Github Pages and the Streamlit Sharing are online, it is possible to embed the latter in the former by using an iframe. The iframe`` by default will show the app in smaller dimensions but with some minimal CSS it is possible to extend theiframe` to provide the same look and feel as the streamlit app.

Moreover, it is also possible to add Google Analytics or another custom HTML/CSS/JS into the page, always checking it does not affect the user experience.

The below example shows how to add the iframe to the boilerplate shown in the previous section.

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta http-equiv="X-UA-Compatible" content="ie=edge">
    <title>My App</title>

    <style>
    body {
      margin: 0;
    }

    iframe {
      display: block;
      background: #000;
      border: none;
      height: 100vh;
      width: 100vw;
    }
    </style>

  </head>
  <body>
    <iframe src="https://share.streamlit.io/GITHUB_USERNAME/REPOSITORY_NAME/MAIN_SCRIPT.py">
        Your browser doesn't support iframes
    </iframe>
  </body>
</html>

To add Google Analytics, the JS snippet can be appended after the iframe, do remember to add the google-site-verification as meta tag:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta http-equiv="X-UA-Compatible" content="ie=edge">

    <!-- Google Verification -->
    <meta name="google-site-verification" content="GOOGLE_SITE_VERIFICATION_TOKEN" />

    <title>My App</title>

    <style>
    body {
      margin: 0;
    }

    iframe {
      display: block;
      background: #000;
      border: none;
      height: 100vh;
      width: 100vw;
    }
    </style>

  </head>
  <body>
    <iframe src="https://share.streamlit.io/GITHUB_USERNAME/REPOSITORY_NAME/MAIN_SCRIPT.py">
        Your browser doesn't support iframes
    </iframe>

    <!-- Google Analytics -->
    <script>
      window.ga = window.ga || function () {
        (ga.q = ga.q || []).push(arguments)
      };
      ga.l = +new Date;
      ga('create', 'UA_TRACKING_CODE', 'auto');
      ga('send', 'pageview');
    </script>
    <script async src='https://www.google-analytics.com/analytics.js'></script>
  </body>
</html>

With this last change and upon committing and pushing the Github Pages site should show the Streamlit app seamlessly and it should also appear in the Real Time tracking of Google Analytics.

Bonus: Custom Domains

As an additional advantage, Github Pages allows custom domain names, which, if using streamlit is a paid-only feature. To learn more about this feature, check the official docs

Do it the right way

The steps described above can and should be considered merely a hack since it is not the intended way to use the Streamlit Sharing platform.

There is indeed one other way to achieve the same, building a custom Streamlit Component that achieves the same functionality. Streamlit Components are custom Web Components that can extend the built-in Streamlit features, programming those require React knowledge though and even if the result is proper and of higher quality it will be more time-consuming.

If the iframe-trick does not work for a given use case, using Streamlit Components is a good next step.

There is documentation about Streamlit Components in the official Docs.

Other Alternatives

Some alternatives to the shown method could be:

Run a Flask server and use Jinja2 to inject the custom HTML into the served streamlit
Modify the default index.html the Streamlit library uses
Host the HTML/CSS/JS snippet and embed that inside Streamlit using st.components.v1.iframe

All of these are either less elegant or much more complex. The author recommends either the iframe-trick or custom-components.

Conclusion

Streamlit is one of the most popular dashboarding platforms in the Python Ecosystem and allows to create apps with ease, Streamlit Sharing provides a free forever hosting platform for unlimited public apps. However, it is not possible to add analytics or custom HTML out of the box. When embedded in a static site like a plain index.html and served via a free web server like Github Pages one can add these features on top of it without issues.

An example Github Repo is available for inspiration, there is also a demo app attached to check the final result.

Check the Series

If you liked this post, it is highly likely that you will also like the other in this 3 part Streamlit Series:

Add Google Analytics (or any custom HTML) to Streamlit with Github Pages (this post)
Multipage, State-Persistent Apps with Streamlit
Lessons learnt After Developing Finance Web Tools with Streamlit and Altair (No HTML/CSS/JS)

Lessons learnt After Developing Finance Web Tools with Streamlit and Altair (No HTML/CSS/JS)

2022-01-14T00:00:00-03:00

There are a lot of resources available online about personal finances, stocks, markets, cryptos, and many more economy-related topics. However the decisions one has to make should be informed in order to be good enough. This post covers the lessons learnt after developing similar tools using Streamlit and Altair. No HTML/CSS/JS were needed. This post will make the reader aware of some pros and cons of this libraries and also show an example app.

This post is one part of a three posts series regarding Streamlit, the posts can be read in any order and the three are built around the same demo app:

Lessons learnt After Developing Finance Web Tools with Streamlit and Altair (No HTML/CSS/JS) (this post)
Add Google Analytics (or any custom HTML) to Streamlit with Github Pages
Multipage, State-Persistent Apps with Streamlit

Disclaimer

The author has no formal education in economics nor any related field, these tools have been thought for personal use only and may contain bugs or inaccuracies. They are provided for free "as is".

TL;DR

This post explains how this web app was developed with a focus on Streamlit, streamlit-multipage, NumPy, pandas , and Altair. The full code is available on this Github Repo

The app covers topics like compound interest, flex term vs fixed terms, inflation adjustment, fee recovery, and asset past profitability. If the reader is already familiar with these topics, it is suggested to check the app and come back in case of questions.

Warning

This is NOT an economy/finance post, it is a technical post about the development of personal finance tools, do not expect full details about the economic part, there are plenty of other resources online. This post will focus on the Python implementation, in particular in the use of Streamlit, Streamlit-MultiPage and Altair.

Personal Finance Tools

Finance is a subject that in many cases is not taught in the depth it should, meaning there is a lot of confusion, misinformation and in many topics, scams. That is why having open-source and free tools to help taking decisions is critical nowadays.

This is a showcase of some of what is possible with Python, thanks to libraries like Streamlit that convert scripts into web apps and Altair that generate transparent vector plots, it is now easy to develop tools that everyone can use.

The State of the Art

In the particular case of investing, there are already two pages that dominate the market Investing and TradingView. Both have free and paid plans and this post does not focus on that type of apps, but rather much simpler ones.

For example, can someone confidently invest in stocks or crypto if they do not understand how inflation or compound interest work? Sure it is technically possible, but the decision is not informed. The apps developed help to fill that gap and give the first contact with terms like inflation, interest and Fixed/Flex Terms.

At the moment on the internet there are plenty of online tools to achieve the same, those, however, are typically calculators that only return a single value and/or whose code is not available. In many cases, they "refer to the formula". Those apps have a main focus on computing, not teaching, if someone does not know how interests work, giving them the final interest at the end of the period might help but it is not the whole story.

That does not mean that the apps developed are the best online by any means but, by providing the code and a more customizable UI, the user can experiment more and "learn while doing" instead of staring at formulas that seem cryptic at times.

The Challenge

Making something technically sound and user-friendly at the same time is a challenging task. The more visual a tool is, the more likely it is for the reader to understand how it is working and the underlying principle behind the tool.

Since the author has no formal education in economics or finance, it was also important to provide an easy way for others to contribute to fix bugs and add enhancements. That is why an open-source solution is a must in this case.

To the best of the author's knowledge, there are no Python-based tools like the developed, so the added-value lies not only in the tools themselves but also because the project serves as a baseline for future related projects.

It was also important to reduce the entry barrier as much as possible so that only some programming skills are needed to extend the existing apps.

The Result

The result of this is having a web app that can be easily used by anyone with or without programming/economics knowledge and it is as easy to maintain and extend as possible. The app is already online and can be used.

The next section will focus on the technology more in-depth.

The Python Implementation

The project uses Python, a programming language with a mature ecosystem of libraries like the Scipy Stack but at the same time it is an easy-to-learn language which can be taught in very few time.

That being said, to generate a fully working web app, web technologies should be used. Fortunately, there are Python libraries that allow the developer to bypass that step.

Stack

The different apps within the project use the same stack which is:

Python as the base programming language
Streamlit to handle the "web part"
Streamlit-multipage to bundle different apps together
[Altair]https://altair-viz.github.io/ for plots
Scipy Stack (Numpy, Pandas and Scipy) to run simulations faster
YFinance to download ticker information
Joblib (optional) for the persistence of state, fallbacks to pickle

The choice of the dependencies and the pros and cons will be extended in the upcoming sections. The main objective was to keep the project simple and avoid lesser-known libraries when possible.

All the code is hosted on Github and can be starred, forked or clone right away.

Simulation Challenges

The defacto standard for numerical simulations is the Scipy Stack, there have been some challenges though as some of the processes do not have a closed formula one can translate to Python.

In some cases like the Inflation Simulation a pure numpy solution could be achieved, but in others like the compound interest with recurring deposits with a different frequency than the compounding frequency a for-loop approach was used.

Nevertheless, even native for-loops being several orders of magnitude slower than NumPy, that was not the bottleneck of the app. The bottleneck was the plotting step, since the length of the data in these apps is usually under the tens of thousands which is fast enough, even when using native for-loops.

Having the app running locally would be enough for personal use however many non-technical people would have been excluded when doing so. Therefore, having the app available online is just as important as it working properly.

Deployment Process

To deploy Streamlit apps there are two alternatives:

Hosted by Streamlit Sharing (Community plan is free forever)
Self-Hosted

The self-hosted approach will involve either server or Docker configuration and thus will not be covered in this post.

The use of Streamlit Sharing and how to deploy a streamlit app there has been already covered in detail in other of the posts in this Streamlit Series. That post also covers how to achieve Google Analytics (or any custom HTML) integration in the app, although that would be only relevant for the tech-savvy.

Streamlit Review

Streamlit alone is a feature-rich, well-documented, and tested library, however, it would have been impractical to have several small streamlit apps. This is something also found in other online tools, they provide "Several calculators" each in different pages hindering accessibility and visibility.

However, streamlit does not support multi-page apps by default, which could be achieve by using streamlit-multipage , a third-party library.

DISCLAIMER: The author is not the original developer of streamlit-multipage but has contributed significantly to that library.

In one of the other posts in this Streamlit series, there is an extended explanation on how to use this library and what is possible.

The rest of this section will cover unresolved issues to consider when developing this kind of apps.

Hot reload could be harmful

By default, each time there is a change in any of the Streamlit Visual components, it triggers the execution of the whole app. This behavior might deteriorate the user experience if there are many inputs and the process takes more than a few seconds.

Having a "Run Button" is possible without any plugins but the "magic update" of streamlit is definitely a feature no one would want to lose.

Lack of skeleton elements

A related issue to the previously mentioned is that when streamlit is "Loading" or "Processing" there is no visual hint for the user that this is the case. No spinners, skeletons, or anything like that. This should not be a problem for fast apps but in the case of taking several seconds, this might result in a suboptimal user experience.

There is no straightforward way to mitigate this besides programming the loading/skeletons themselves as independent Streamlit Components.

Lack of basic HTML Elements

Streamlit combines Python and Web Technologies (in particular React) in a clever way, abstracting every web-related from the Python side. That has its costs though, thinking of custom yet basic HTML elements like a "Click me Button" that redirects to a page might be non-trivial to do and might require either custom HTML with styling or designing a React component just for that which seems excesive.

Streamlit is ideal for the use cases it was thought but deviating too much from it might increase developing time significantly and make the learning curve much steeper.

Persistence might be complex

Even though the streamlit-multipage provides persistence mechanisms, they were tested in Streamlit Sharing. This is because of how streamlit works under the hood, it runs "the whole script" with every change, making any kind of "initialization code" extremely hard to avoid repeating with each change.

There is support for some basic native persistence through st.session_state but it has not been optimized for multipage apps nor it provides two-way binding as one might be used to from Front-End frameworks.

Streamlit Conclusion

Streamlit is not the only of its kind, there is Panel, Dash, Voila, and there might be more in the future. That being said, Streamlit was the best tool for the job in this case for many reasons, it does not really on other tools (dash is heavily tiered towards plotly and voila works on Jupyter), and it has a mature and responsive community (panel is lacking in this aspect.)

Streamlit alone was initially not enough due to the lacking multipage support but using streamlit-multipage solved this without disrupting the streamlit developer experience.

Altair Review

Altair is a plotting library that aims to provide a "declarative interface" rather than a "descriptive interface", in practice, this allows for simpler code and more code re-use.

It is not the most popular tool by any means, that would be matplotlib but one of the main disadvantages of the latter is that there is no support for transparency. Transparency might be a not-so-important feature, but, since Streamlit has a light and a dark theme, a transparent plot is necessary to ensure a consistent visual design that is not compatible with Matplotlib.

On the other side, matplotlib is much faster than Altair, this might change in the future but at least when using Altair one has to limit the number of data points to 5000. That being said, the interactiveness Altair provides makes it a perfect fit for web apps.

Learning Altair requires some time but if one is already familiar with matplotlib the learning curve is a bit flatter. There are many good materials already available to learn on PyVideo.

Verbosity and Modularity

Unless using the defaults, producing highly customized plots requires many lines of code, regardless of the plotting library. The main difference with Altair is that those "extra-custom-lines" could be re-use across plots.

This is an example to add a text tooltip

def add_text(chart, field, selection):
    return (
        chart.mark_text(align="right", dx=-5, dy=-12, color="white", fontSize=18)
        .encode(text=field)
        .transform_filter(selection)
    )

This function returns an Altair object given a chart, a field, and a selection, that means that one can add the same tooltip to many plots at the same time while keeping the code centralized in a separate file.

This is an underrated feature that allows to re-use code and becomes critical when using multiple somewhat similar plots.

Altair is slow

The main disadvantage of Altair is that everything that happens is handled over to Vega-Lite, the library Altair is built on. This means that Altair is much slower than other plotting libraries, this is especially true when compared with matplotlib.

Even so, having tooltips, panning and zooming are features that are worth the wait and the final user will notice the difference because the quality of the end result is evident.

Altair Conclusion

There are many options when deciding over a plotting library, the most popular by far is Matplotlib, however, for web apps, it is lacking the features many consider basic like zooming and panning. And having the possibility to re-use code across plots is something that makes Altair a very good fit for any large software project.

There are drawbacks like the data point limit and the slowness, a balance between the two should be found and it will depend on every app.

Overall Conclusion

Developing Web Apps with Python has never been easier, Python has a strong mathematical and computational stack that makes most computations fast and efficient. It also has an ecosystem that makes it suitable for most use cases and on top of that, it is considered one of the easiest programming languages to learn.

Thanks to libraries like Streamlit and Altair, one can now develop web apps and make them available online so everyone can benefit from these amazing tools in more tangible ways.

There are some issues but the author believes this will be solved in the middle term as the libraries evolve and improve themselves.

The Personal Finance Tools is online and ready to use, the code is hosted in Github as well.

Check the Series

If you liked this post, it is highly likely that you will also like the other in this 3 part Streamlit Series:

Lessons learnt After Developing Finance Web Tools with Streamlit and Altair (No HTML/CSS/JS) (this post)
Add Google Analytics (or any custom HTML) to Streamlit with Github Pages
Multipage, State-Persistent Apps with Streamlit

Multipage, State-Persistent Apps with Streamlit

2022-01-14T00:00:00-03:00

Converting a Python script to a web app has never been easier thanks to libraries such as Streamlit. This library allows to create simple and responsive web apps with minimal effort but there is one potential limitation: No multipage support. In this post, a potential solution to this problem is presented with examples and demos.

This post is one part of a three posts series regarding Streamlit, the posts can be read in any order and the three are built around the same demo app:

Multipage, State-Persistent Apps with Streamlit (this post)
Add Google Analytics (or any custom HTML) to Streamlit with Github Pages
Lessons learnt After Developing Finance Web Tools with Streamlit and Altair (No HTML/CSS/JS)

TL;DR

There is a pip installable plugin called streamlit-multipage, this post is an extension of the README of the Github Repository.

Try it yourself by doing

pip install streamlit-multipage

An example Github Repo is available for inspiration, there is also a demo app attached to check the final result.

For an alternative, more in-depth library, check Hydralit

Why Multipage Apps?

Streamlit is an awesome tool for converting Python scripts into Webapps but, sometimes one wants to have many tiny scripts combined in a user-friendly way and, even though it is possible to develop one streamlit app for each, the process is tedious and the whole project loses maintainability.

Multipage apps are apps that can host different apps in parallel with some additional features like state persistence or customizable UI.

Streamlit-multipage

There are several solutions to these problems and to the best of the author's knowledge, these are the most popular ones:

However, for one reason or another, none of them fulfill the purpose of a "simple multipage experience".

Hydralit is a bit overkill because it is a big project deeply integrated with streamlit and one has to use import hydralit as hy instead of import streamlit as st which breaks the typical idioms built around streamlit.

Streamlit-Multiapps is not pip installable and has NumPy and pandas as dependencies, which, even if used in most cases, are not required always and are considered among the heaviest dependencies in the Python ecosystem.

Streamlit-pages is a 55 LoC script, which is too basic and provides no state persistence options.

There is a fourth option called streamlit-multipage which is pip installable and it has state persistence while keeping streamlit idioms. It is not as feature-rich as Hydralit but has a much more friendly learning curve.

In this post, the streamlit-multipage will be used and covered in depth.

To start working with this library one can simply install it via pip:

pip install streamlit-multipage

DISCLAIMER: The original development of streamlit-multipage was done by Yan Almeida, however, this blog's author contributed significantly to the library. Most of this post content will be extracted from the repository README with extended explanations.

Features

Streamlit-multipage has several appealing features that may be useful for most users.

Simplicity

Using streamlit-multipage is as simple as writing a native streamlit like this:

import streamlit as st

st.title("My Amazing App")
name = st.text_input("Your Name: ")
st.write(f"Hello {name}!")

And refactoring in a function like this

import streamlit as st
from streamlit_multipage import MultiPage


def my_page(st, **state):
    st.title("My Amazing App")
    name = st.text_input("Your Name: ")
    st.write(f"Hello {name}!")


app = MultiPage()
app.st = st

app.add_app("Hello World", my_page)

app.run()

The function has to take one positional argument, ideally called st, that is the streamlit object. The function also needs to take arbitrary keyword argument **state, this is where the state of the app will be injected.

As seen in the example, it is not necessary to use the state variable at all in the function.

In the last part of the snippet, we see the streamlit-multipage specific part which consists of instantiating a MultiPage object, passing the streamlit object and then adding the app (or page) with the add_app function.

This is ideal for short scripts but typically streamlit apps have many dozens if not hundreds of lines of code, for that the function could be moved to a different folder

Since the st object used will be passed as a parameter, it is not needed to import streamlit anywhere else in the project besides where the MultiPage object was instantiated.

Working in the functions feels like native streamlit because it is, the st parameter is not a mocked object nor is it monkey patched, providing the native streamlit experience.

State Persistence

Another important feature is inter-page persistence, this is a type of persistence that allows saving variables from one page to the other but it also allows saving the state after restarting the app.

Here is an example of this feature:

import streamlit as st
from streamlit_multipage import MultiPage


def input_page(st, **state):
    st.title("Body Mass Index")

    weight_ = state["weight"] if "weight" in state else 0.0
    weight = st.number_input("Your weight (Kg): ", value=weight_)

    height_ = state["height"] if "height" in state else 0.0
    height = st.number_input("Your height (m): ", value=height_)

    if height and weight:
        MultiPage.save({"weight": weight, "height": height})


def compute_page(st, **state):
    st.title("Body Mass Index")

    if "weight" not in state or "height" not in state:
        st.warning("Enter your data before computing. Go to the Input Page")
        return

    weight = state["weight"]
    height = state["height"]

    st.metric("BMI", round(weight / height ** 2, 2))


app = MultiPage()
app.st = st

app.add_app("Input Page", input_page)
app.add_app("BMI Result", compute_page)

app.run()

Here the app "Input Page" can pass variables to the "BMI Result" page by calling the MultiPage.save method. By default, all the variables are saved in a global namespace but different namespaces can be used as well (see next example).

To access the state on a page one has to use the state variable, which is a Python dictionary, simply checking if the necessary keys are present should be enough. It is recommended to add warning messages if there are missing keys.

If default values are desired for when the variables are not present the get method can be used as well.

def my_app(st, **state):
    my_variable = state.get("Year", 2022)

Internally the state is persisted in a pickle file which uses joblib if available, if it is not installed it gracefully fallsback to the native pickle module.

WARNING: This feature has been tested locally but not with the Streamlit Sharing Service (Deploy App Option).

Namespaces

The MultiPage.save method can take a second keyword argument namespaces which receives a list of all the namespaces the variables should be saved into. That is, a page can save a variable in multiple namespaces.

There is no efficiency or comprehension consideration in the implementation so beware of storing heavy objects in multiple namespaces at once (especially important for large NumPy arrays or tensors).

When using namespaces it is a good practice to add a namespace variable at the very top of the function for easy reference.

The example below illustrates this feature

import streamlit as st
from streamlit_multipage import MultiPage


def input_page(st, **state):
    namespace = "input"
    variables = state[namespace] if namespace in state else {}
    st.title("Tax Deduction")

    salary_ = variables["salary"] if "salary" in variables else 0.0
    salary = st.number_input("Your salary (USD): ", value=salary_)

    tax_percent_ = variables["tax_percent"] if "tax_percent" in variables else 0.0
    tax_percent = st.number_input("Taxes (%): ", value=tax_percent_)

    total = salary * (1 - tax_percent)

    if tax_percent and salary:
        MultiPage.save({"salary": salary, "tax_percent": tax_percent}, namespaces=[namespace])

    if total:
        MultiPage.save({"total": total}, namespaces=[namespace, "result"])


def compute_page(st, **state):
    namespace = "result"
    variables = state[namespace] if namespace in state else {}
    st.title("Your Salary After Taxes")

    if "total" not in variables:
        st.warning("Enter your data before computing. Go to the Input Page")
        return

    total = variables["total"]

    st.metric("Total", round(total, 2))


app = MultiPage()
app.st = st

app.add_app("Input Page", input_page)
app.add_app("Net Salary", compute_page)

app.run()

When using custom namespaces, it is important to access state[namespace], instead of the state alone, because state will contain all namespaces, including the global one.

WARNING: no variable in the global namespace should be named identically as another namespace.

Automatic Namespaces

It is also possible to avoid that altogether and use Automatic Namespaces, streamlit-multipage will filter the state variable before passing it to a function if it detects that there is a namespace with the same name of the app. That is if the namespace is the same as the name used in the add_app method.

Note: The namespace should be explicitly specified in the MultiPage.save method.

This is an example of automatic namespaces

import streamlit as st
from streamlit_multipage import MultiPage


def input_page(st, **state):
    st.title("Tax Deduction")

    salary_ = state["salary"] if "salary" in state else 0.0
    salary = st.number_input("Your salary (USD): ", value=salary_)

    tax_percent_ = state["tax_percent"] if "tax_percent" in state else 0.0
    tax_percent = st.number_input("Taxes (%): ", value=tax_percent_)

    total = salary * (1 - tax_percent)

    if tax_percent and salary:
        MultiPage.save({"salary": salary, "tax_percent": tax_percent}, namespaces=["Input Page"])

    if total:
        MultiPage.save({"total": total}, namespaces=["Net Salary"])


def compute_page(st, **state):
    st.title("Your Salary After Taxes")

    if "total" not in state:
        st.warning("Enter your data before computing. Go to the Input Page")
        return

    total = state["total"]

    st.metric("Total", round(total, 2))


app = MultiPage()
app.st = st

app.add_app("Input Page", input_page)
app.add_app("Net Salary", compute_page)

app.run()

Extensibility

When using streamlit-multipage projects can grow easily within a directory structure.

Consider the following directory structure

.
└── root/
    ├── pages/
    │   ├── __init__.py
    │   ├── input_data.py
    │   └── result.py
    └── main.py/

Here the main.py is the entry point, it imports both streamlit and streamlit-multipage, it instantiates the MultiPage object and loads the pages and adds them to the object.

Each app or page can be separated into its file and common code could be abstracted as well.

The __init__.py facilitates the import hierarchy by giving a single entry point to the pages module.

These could be the content of the files:

input_data.py

from streamlit_multipage import MultiPage


def input_page(st, **state):
    st.title("Tax Deduction")

    salary_ = state["salary"] if "salary" in state else 0.0
    salary = st.number_input("Your salary (USD): ", value=salary_)

    tax_percent_ = state["tax_percent"] if "tax_percent" in state else 0.0
    tax_percent = st.number_input("Taxes (%): ", value=tax_percent_)

    total = salary * (1 - tax_percent)

    if tax_percent and salary:
        MultiPage.save({"salary": salary, "tax_percent": tax_percent}, namespaces=["Input Page"])

    if total:
        MultiPage.save({"total": total}, namespaces=["Net Salary"])

results.py

def compute_page(st, **state):
    st.title("Your Salary After Taxes")

    if "total" not in state:
        st.warning("Enter your data before computing. Go to the Input Page")
        return

    total = state["total"]

    st.metric("Total", round(total, 2))

__init__.py

from .input_data import input_page
from .result import compute_page

pages = {
    "Input Page": input_page,
    "Net Salary": compute_page,
}

main.py

from pages import pages

import streamlit as st
from streamlit_multipage import MultiPage

app = MultiPage()
app.st = st

for app_name, app_function in pages.items():
    app.add_app(app_name, app_function)

app.run()

Structuring the application this way makes all the files much shorter and with a single responsibility. This could have been done with just streamlit but with minor modifications

Customization

It is also possible to add customizations to each app to enhance the user experience such as adding a header, a footer, additional sidebar elements, hiding the hamburger menu and customizing the default text (e.g. for localization)

This code shows how these customizations can be used:

import streamlit as st
from streamlit_multipage import MultiPage, save


def input_page(st, **state):
    """See Example on Multipage"""


def compute_page(st, **state):
    """See Example on Multipage"""


def footer(st):
    st.write("Developed by [ELC](https://elc.github.io)")


def header(st):
    st.write("This app is free to use")


def sidebar(st):
    st.button("Donate (Dummy)")


app = MultiPage()
app.st = st

app.start_button = "Go to the main page"
app.navbar_name = "Other Pages:"
app.next_page_button = "Next Chapter"
app.previous_page_button = "Previous Chapter"
app.reset_button = "Delete Cache"
app.navbar_style = "SelectBox"

app.header = header
app.footer = footer
app.navbar_extra = sidebar

app.hide_menu = True
app.hide_navigation = True

app.add_app("Input Page", input_page)
app.add_app("BMI Result", compute_page)

app.run()

Non-Functional Features

In addition to the previous functional features, there are some non-functional ones that could be interesting for some.

Single-File Project

Streamlit-multipage is a single file project, meaning that instead of pip install it and add a dependency to track, the whole project can be simply add as a library in any other project standalone and it should work out of the box.

Quality of Code

The code has been formatted with Black and Type Hints were added whenever possible to improve integration with editors and IDEs.

Limitations

The library itself has its limitations, namely:

The state is saved in a dictionary and then in a pickled file, there is no usage of st.session_state which would imply a better user experience, this has been tried but due to streamlit lacking two-way binding in the session_state, this is not possible at the moment.
There are no tests, according to this discussion, it is not straightforward to test streamlit apps nor libraries built around it, mainly because this involves testing not only Python but also Front-End code. This has not been done so far, contributions are much welcome though.
Integration with Streamlit Sharing has not been tested thoroughly, the basic multi-page app works but no state management has been tested on the managed service. All the features shown in this post have been manually tested on a locally running streamlit app.

Conclusion

Streamlit is one of the most popular tools for dashboarding and easily converting Python Scripts into Web Apps. Streamlit-Multipage is a lightweight app that can be pip installable (or drop in a standalone script) that allows for seamless integration of multi-page apps while keeping state and idioms.

The streamlit-multipage has no tests and has not been thoroughly tested on the Streamlit Sharing but for self-hosted apps should be working without issues.

An example Github Repo is available for inspiration, there is also a demo app attached to check the final result using the streamlit-multipage.

Check the Series

If you liked this post, it is highly likely that you will also like the other in this 3 part Streamlit Series:

Multipage, State-Persistent Apps with Streamlit (this post)
Add Google Analytics (or any custom HTML) to Streamlit with Github Pages
Lessons learnt After Developing Finance Web Tools with Streamlit and Altair (No HTML/CSS/JS)

Secure Live Collaboration in Jupyter Lab

2021-09-04T00:00:00-03:00

The single most requested feature for Jupyter Notebooks/Lab was live collaboration á la Google Docs. Today, this is possible but should be done with caution, since it gives access to the whole internet to run any (including malicious) code on our local PC. This is a quick guide to prevent this by implementing several layers of security.

Collaborative Notebooks

There are many ways to work collaboratively, however not all are suitable for every scenario. In the case of Jupyter Notebooks, there are three main types of collaboration, the last of which has been introduced recently.

Asincronous Multiuser: each party makes changes and then changes are merged. This is git-like collaboration and is possible with Jupyter Notebooks out of the box.
Non-Concurrent Multiuser: Several users make changes at the same time but not in the same files. This is currently possible when using Jupyter-Hub (or MyBinder) servers. There might be synchronization issues if modifying the same file.
Concurrent Multiuser: Several users make changes at the same time in any document (even the same one) and the changes are merged without overlapping. This is the Google Docs style and is the new feature of Jupyter. Hereafter this is the type of collaboration that will be referred to as live collaboration

Why should we work collaboratively?

Working collaboratively has definite some advantages, one way to show them is by some example use cases that would not be possible without this feature:

A teacher can invite their students to actively collaborate on a notebook and write comments and questions in a centralized way.
A small team can leverage a server (or powerful computer) to run the notebooks instead of each having to run the kernels on their local PCs.
Two team members can work in different parts of the same problem, like hyperparameter tuning and data visualization concurrently.
A presenter can invite a small audience to try some Jupyter app (e.g. built with widgets) and have them use it with zero-install.

Of course, this is probably not the best solution for all possible scenarios but for small teams, mentoring / teaching and showing proof of concepts, it might be worthwhile.

The results are illustrated in the following video

Your browser does not support the HTML5 video.

Risks involved

Before enabling live collaboration, it is important to understand the risks it implies. Therefore three different use cases will be illustrated:

Host a Jupyter Notebook on the local PC and work together with someone in the same local network (LAN or VPN)
Host a Jupyter Notebook on the local PC and work together with someone outside of the local network (any PC with internet access)
Host a Jupyter Notebook on a remote PC and work together with someone outside the local network

The first is mostly secure (as long as the local network can be trusted). However, when giving access to the whole internet (the second option), some security measures should be taken. The third case is too vaguely described to be explained in detail since the steps might change for servers behind a reverse proxy, servers in the cloud, or servers on-premises, furthermore, some servers require specific firewall changes. For on-home, on-premise servers steps should be similar to what is described for the second case.

Quick Solution

This article covers some of the technical details and implications of each of the steps. It is recommended to read the full article but for those who already had and/or want a quick solution, refer to the 4-command-summary at the end.

Important Disclaimer

This article assumes you have full control of the PC you are going to use, I highly recommend against using these guide on any company-owned hardware to avoid introducing security risks and/or going against the company's policies.

Moreover, I am not a security expert so I strongly recommend the reader to make their research and validate everything here written because fine details might be overlooked.

Finally, take into consideration that this tutorial is OPENING A BACKDOOR for your PC so, one can never have too many security measures. If this makes you feel uneasy, do all your experiments inside a sandbox (like a container or a virtual machine).

To further illustrate this, Ribaka Nazmara showed in Quora some dangerous yet simple code a malicious user might run:

Fill up disk:

import os 

while True: 
    os.fork()

Delete all files:

import os 

os.system("rm -rf /")

Crash the system:

import os 
os.system("%0|%0")

From Python, one can even run processes in the background with the subprocess module. These examples should illustrate why not using adequate security measures might result in disastrous outcomes.

Requirements

This tutorial requires Jupyter Lab 3.1.0 or newer and NGrok (for the second use case).

If ngrok is not installed, all the steps are explained in this 5 minute tutorial by CalmCode.

In the case of Jupyter Lab, the version required was released on the 27th of July 2021 so if Jupyterlab was not installed recently, an update will be needed.

To check your Jupyter Lab version simply run:

jupyter lab --version

Jupyter Lab Update Instructions

If using Anaconda, run:

conda update anaconda
conda update jupyterlab

For traditional pip installations, run:

pip install jupyterlab>=3.1.0

Use Case 1: Collaborative work inside LAN or VPN

Once Jupyterlab 3.1.0 or newer is installed, it is possible to enable the collaborative feature by running:

jupyter lab --collaborative

This will create a normal session available at http://localhost:8888. However, this URL is only accessible inside your local network. To access from another device (PC or Mobile) one should replace the "localhost" with the private IP address, also known as internal IP. The IP address is a network address and is not attached to specific hardware (as the MAC address), which means that each time the PC is turned on, it might have a different private IP.

To know your private IP simply run this command in a terminal

On Linux/Mac:

ifconfig -a

On Windows:

ipconfig

Private IPs are the ones following these formats:

Class A: from 10.0.0.0 to 10.255.255.255
Class B: from 172.16.0.0 to 172.31.255.255
Class C: from 192.168.0.0 to 192.168.255.255

The "Default Gateway" IP is the one for your router or networking device whereas the other one (Ethernet, IPv4, or similar) is the one of the PC.

For example, if the PC's IP address is 192.168.1.35 any person connected to the same network could access the Jupyter session by navigating to the URL http://192.168.1.35:8888 in the browser. Note: This required that the port 8888 is open in the firewall of the PC running Jupyter.

By default, Jupyter uses Token Authentication which is transparently applied for the local PC, meaning that incoming connections from other PCs will be rejected or they will be asked to provide a token.

Tokens are difficult to share and remember as they are long strings of seemingly random text, that is why they will be replaced with a user-defined password.

First Layer of Security: Jupyter Password

The Jupyter CLI provides a straightforward way to set a global password by running:

jupyter lab --generate-config
jupyter lab password

The first command generates a default config file, if already run before, it can be omitted. The second one will prompt the user to write a password and then confirm it. Note: when writing the password, no characters (not even "*") will be shown, this is done so that the number of characters of the password is kept secret, this is common practice in Linux Systems but may be new for some users.

Continuing the previous example, when a user from another PC enters the http://localhost:8888, it can access by writing the pre-defined password.

If configured correctly, the next time Jupyter is run, it will prompt for the password:

Technical Details: The password is stored as a hashed in the configuration file, so it is not stored as plain-text anywhere in the system and is a global setting for all Jupyter Lab sessions (Jupyter Notebook sessions are not affected).

Important Note: Always refer to the Official Docs in case some of the information presented here changes.

Second Layer of Security: Local TLS Encryption

Even having configured a password, the connection is not encrypted, which means that, inside the local network, the password travels as plain text through the network and thus is susceptible to man-in-the-middle attacks.

If you trust your VPN or local network, this step is optional. Nevertheless, this is mandatory when using public networks (cafés, airports, etc.).

To prevent this from happening, a protocol with encryption (like HTTPS) should be used. HTTPS is the most common approach, it uses TLS encryption so all the data sent between two nodes is encrypted, which of course includes the password that would have otherwise traveled as plain text.

Explaining how certificates (the key component of TLS encryption) work is beyond the scope of this article. Nonetheless, one can set up quite easily a self-signed certificate, the browser will show a warning saying the connection might be insecure but that is simply because the certificate does not come from a Certificate Authority, a company that is trusted to emit valid and trustworthy certificates. Instead, the certificate is emitted by the user in the local PC is not suitable for any production environment but is more enough for this use case.

To generate this certificate the simplest way is to use OpenSSL which is already installed if using Anaconda. In case Anaconda is not installed, OpenSSL should be installed independently. Using Anaconda is however the easiest way for Windows users.

Once OpenSSL is installed, navigate to the folder where Jupyter will be started and run:

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout mykey.key -out mycert.pem

Note: if the error: "Can't open {some_path}ssl/openssl.cnf for reading, No such file or directory" is thrown, see the next section to configure it using Anaconda.

Once run, it will generate two files, the mykey.key, and the mycert.pem, none of which should be added to source control. These two files will enable the Jupyter server to use HTTPS for its connections, to do so, run the following command:

jupyter lab --collaborative --certfile=mycert.pem --keyfile mykey.key

This setup should suffice but, the user should be instructed to use https://localhost:8888 (with HTTPS) explicitly because there might not be an automatic redirection to HTTPS if using HTTP. The browser might show a warning as described earlier but this should be ignored and continue anyways.

To remove the browser warning one should add the local certificate to the truststore, that, however, is beyond the scope of this article.

With the previous configuration the connection between the PC running Jupyter and the other PC is encrypted, even inside the Local Network / VPN.

Note: so far, the whole connection is done through the local network and should not be visible from outside, but in public networks, this might be different. The good practice is to never trust a public network, with the password and this connection encryption a decent level of security is achieved for local connections / VPN only.

In the following sections, this will be expanded to cover users from other networks (i.e. internet). Hence additional security measures should be taken.

Config file Missing

The configuration file for OpenSSL is not configured by default therefore one has to run:

set OPENSSL_CONF={AnacondaPath}/pkgs/openssl-1.1.1k-{some_hash}/Library/ssl/openssl.cnf

Use Case 2: Collaborative work outside the local network

It is not the most common to have multiple users in the same local network, for those cases, one should establish a connection with someone through the internet. By doing so, the local PC is open to anyone with enough knowledge to hack and get themselves in.

There is no way to guarantee 100% that no one unintended will be granted access, there are some measures one could take against though.

The first layer of security described above is a must, having at least password-protection is the bare minimum when connecting through the internet.

Moreover, in this section, several additional layers of security will be described and at the end, a quick summary will be given to help the reader get started in as little time as possible.

To set up an open connection to our computer, ngrok is a good solution. After installing it, and adding it to the Path, and providing the auth token, one can simply open connections in a single command:

ngrok http 8888

This command is enough to open the connection, and since our Jupyter environment is password-protected, there is no immediate threat. And this provides an additional layer of security out of the box: obscurity, which will be explained further in the next section.

However, if one navigates to the URL shown, a 403 Forbidden message will be displayed:

This is a security measure from Jupyter, it forbids all non-local connections:

To allow non-local connections the ServerApp.allow_remote_access should be enabled. This can be done from the Jupyter Config file created with the --generate-config command. Simply append this line to the end of the file:

c.ServerApp.allow_remote_access = True

Now after restarting the Jupyter instance, the URL provided by ngrok should be accessible:

Possible Error: Failed to complete tunnel connection

If a message like the following is shown:

This means that there is no service running at port 8888. This means that Jupyter is either not running or is being served on another port.

Local Certificate

By default, when using a port as the destination, ngrok will point to the HTTP as in http://localhost. However, when using the second layer of security and implementing local TLS the above command should be rewritten to explicitly target https://localhost as follows:

ngrok http https://localhost:8888

This is an optional extra step, only needed when the local network is not trusted (e.g. public networks). This is mandatory when used in combination with the second layer of security described earlier.

Third Layer of Security: Obscurity

Ngrok will generate a pseudo-random sequence of characters for the URL of the connection. One might think this is a disadvantage or a limited feature of the free plan (since the paid plan support custom subdomains) but, having a hard-to-guess sequence at the beginning is similar to a password, since both parties should know it beforehand to establish the connection.

https://7b42-201-212-74-100.ngrok.io

Another advantage of this approach is that if a given URL is compromised, one can simply re-run the ngrok command and get a new one with practically no downtime.

The clear disadvantage is that this method will impede any long-lasting communication mechanism with the involved users (e.g. email), and one should fall back on instant messaging methods (e.g. chats, calls, etc.). Nonetheless, if the computer running the Jupyter will be kept on for long periods (like a server), this drawback can be neglected.

That being said, obscurity on its own is not a robust security measure, it will not stand against deliberate malicious attacks but it constitutes a bare minimum layer of confidentiality.

When inspecting the console output, one can see that ngrok actually does two forwarding connections, one for the HTTP and the other for the HTTPS. For reasons already explained, if an encrypted connection is needed (as it is here because we will be sending credentials), the HTTPS should be enforced by rejecting any incoming non-HTTPS connections. Ngrok facilitates this with a simple flag as explained in the next section.

Fourth Layer of Security: Public TLS Encryption

The second layer of security introduced self-signed certificates for the connections inside the local network (LAN or VPN), ngrok supports TLS encryption with certificates coming from a Certificate Authority (CA) as part of the free plan. However, the default behavior is to allow both HTTP (without certificates) and HTTPS (with certificates).

To enforce the use of HTTPS simply run:

ngrok http -bind-tls=true 8888

The console output now shows only one forwarding connection from HTTPS and all attempts to connect to HTTP will be forwarded to HTTPS.

The main difference with the second layer of security is that ngrok uses a certificate coming from a CA and hence the browser displays no warning whatsoever, resulting in a smoother experience for the final user.

One limitation of the approach described so far is that the Jupyter password is set globally for the local PC, and in case it is compromised, it may be inconvenient to re-share the new password to all users. To mitigate this, it is possible to generate a per-session username and password. This procedure is explained in the next section.

Fifth Layer of Security: HTTP Password

The HTTP protocol implements a basic authentication system via a username-password combination. These two fields travel as plain text unless the connection is encrypted (e.g. by using HTTPS), since the ngrok connection is already configured to only accept HTTPS, the username and password will travel through the internet encrypted.

The free plan of ngrok also includes support for this authentication method and can be enabled by adding username:password to the command:

ngrok http -bind-tls=true -auth="any_username:sesion_password" 8888

These credentials will be asked upon connecting to the ngrok URL

After introducing the correct credentials, the password for the Jupyter server will be asked.

This has an unintended advantage and that is that all attempts to log in are against Ngrok servers and not our Jupyter Server. Providing a somewhat basic defense against DDoS attacks.

With everything done up to this point, there are several layers of defenses. One of the things not done yet was filtering. There are several types of filters one can implement, being the most common IP and MAC filtering. In the next section, IP filtering is explained.

Sixth Layer of Security: IP Whitelist

Ngrok provides an IP Whitelist feature, this is feature is only supported in the paid plans. Having the possibility to only allow certain IPs is useful as long as the users connecting have static IP addresses and they indeed know them.

The most common scenario is that a user does not know their IP address and even if they do it is hard to track when those are dynamically assigned (as it is with several ISPs).

In an enterprise setting, this is less convoluted since most PCs are connected through a VPN, and the VPN usually has a range of public IP addresses, thus whitelisting those makes the service only available for those in the company network.

For more information regarding this, check the official docs of ngrok.

This article has described some of the possible security measures one can take to securely connect to a Jupyter instance, the following sections will summarize the content showed in quick steps and finally, some pitfalls will be mentioned to avoid potential security risks.

Summary: Quick steps to follow

1. Update Jupyter Lab

Update Jupyter Lab to a 3.1.0 o newer with:

conda update anaconda
conda update jupyterlab

2. Set a Password

jupyter lab --generate-config
jupyter lab password

3. Enable non-local connections

Append this line to the config file generated in the previous step:

c.ServerApp.allow_remote_access = True

4. Run Ngrok

Run Ngrok with HTTPS only and providing username and password credentials

ngrok http -bind-tls=true -auth="any_username:sesion_password" 8888

To avoid manually writing this file every time, a config file can be used to automate the process. Steps to follow are shown in this video.

Share the Jupyter password via a long-lasting communication mechanism (e.g. email), then share the ngrok URL and the associated username and password via some instant messaging (e.g. chat). For extra security use two different methods for the URL and the username-password, like sharing the username-password via a screen share and the URL via chat.

6. Work Together with Live Collaboration

Now everything is set up to work collaboratively.

7. Publish your work

Whenever the project is public, it is worth sharing. When working with Jupyter Notebooks there are several ways to do it. In this survey some of the most popular options are explained.

Common Pitfalls and Questions

Everything has been configured and now the environment has at least some basic security measures. However, there are some common pitfalls to be aware of.

Explicitly Ignore your sensitive files in source control

When using source control (e.g. git), the .pem and .key should be ignored as well as any pre-configuration files from ngrok and jupyter. Otherwise, the credentials and certificates are compromised.

Choose different and hard-to-crack Passwords

It is useless to use two passwords if those are not different from each other, good passwords are easy to read, easy to remember, and hard to guess. Against what might be popular on the internet, using phrases as passwords is quite useful and effective.

Since the Jupyter password will be applied to all sessions, it should be strong, preferably more than 16 characters long, whereas the Ngrok basic auth could be something longer than 10 characters.

To check password security by using some service online

Unreasonable Trust in ngrok

In this article, ngrok is assumed to be a trusted organization, however, there are no particular reasons one should trust ngrok. This post was not sponsored by them or any other company and using the approaches here described put an amount of trust in a free service some might find uncomfortable.

For the typical user of Jupyter Notebooks, it is unlikely that the weakest point in the security pipeline is ngrok but, for the conservative, security-driven users, this is something to carefully evaluate.

Ngrok will have access to the basic auth and for them, the Jupyter password will be visible so they will have backdoor access to your PC. However, to the best of the author's knowledge, there are no issues reported as security breaches from ngrok.

Another important point is that everything is built on the free plan from ngrok, should they change the scope of the features of the free plan, some part or all the procedure here describes might be obsolete. This is another potential risk, because there is no guarantee the free service will stay free as with the same features long-term.

Not suitable for unlimited connections

Even being generous, the free plan has its limitations and the collaboration features might be limited to a handful of users (probably not more than a couple of dozens). The free plan will likely fail to support hundreds of users as in live conferences, a MOOC, or similar events.

This approach is suitable for personal/hobbyist use or small teams (< 10 people). So far there are no benchmarks available as to how many users concurrently can be working using collaborative Jupyter Lab with ngrok.

Remember no system is 100% secure

This is an approach, it only showcases one possible way of doing things and there might be better and more secure ones. However, I consider this is a good tradeoff between technical complexity and practicality.

As mentioned at the beginning, I am not a security expert and it is up to you to double-check, all the information given and assess whether it is useful for your particular scenario.

Conclusion

The new collaborative feature of Jupyter Lab is out of beta and ready to be used. Some security measures should be taken beforehand to avoid any potential breaches or risks. After understanding the underlying concepts, the new Jupyter Lab can be run and set up in just 2 commands.

Contribute¶

This content is free and promotes both free and open source software, as well as monetary and intellectual accessibility. If you want to support it, please check out the privacy-first ad at the top of this page. Helping with fixing typos, suggesting edits, or leaving feedback in the comments is also valuable. If you like this content, please share it with others who might find it useful.

If you want to contribute financially, invite me a Ko-Fi or send me donation via PayPal. If you are in Argentina, you can invite me a Cafecito.

Resources

Jupyter Official Documentation

TLS Encryption

Ngrok Official Docs

Jupyter Collaborative Feature

Jupyter Publishing Guide: From Embedded to Book

2021-08-16T00:00:00-03:00

Jupyter Notebooks are the lingua franca for Data Science. There is hardly any data scientist that has never used this IDE. However, in contrast to its popularity and usefulness, the process of sharing a Jupyter Notebook is not straightforward. This article presents a brief survey of the different publishing solutions available for personal and/or commercial projects.

Jupyter Notebooks

Jupyter notebooks are used everywhere in the Data Science ecosystem. They are present without exceptions, from Coursera Courses to projects by FAANG. It is normal then to ask, once a work has been done, how one can conveniently publish the results. The answer may not be satisfactory: "There is no well-known, commonly used, standard way to publish Jupyter Notebooks". Instead, there are co-existing solutions that target different personas:

Will the results be presented to a non-technical audience?
Will those results aim at reproducibility?
Is the focus solely on the code? Those questions require different solutions as there is no one-size-fits-all.

In this post, a showcase of solutions for the different use cases will be presented to the reader so that one can find the right tool for the job.

A general classification for the solutions could be as follows:

Share notebooks among developers
Share notebooks online
- Read-Only
- Read and Execute
Print notebooks

All solutions here assume the only hardware and software available are the ones used to work with the Jupyter Notebooks, i.e. no server or backend. Moreover, every solution presented is either open-source or free to use, both for personal and commercial use cases.

This post will follow a cookbook approach, so feel free to jump ahead to the section of interest, they are all self-contained. Each solution will contain advantages, disadvantages, and a "when to use" section.

Index

Context on .ipynb
Built-in Solutions
Read Only Solutions
Executable Solutions
Printable Solutions
- LaTeX Export
- LaTeX/PDF Jupyter Book
Honorable Mentions
What should I use?
Conclusion
Additional Resources

(Optional) A bit of Context: The .ipynb format

The format for a Jupyter notebook is JSON, but the extension used is ipynb. The reason behind using a different extension is twofold:

Not every JSON is a valid .ipynb.
The file, although plain text, is not meant to be modified manually.

When using Jupyter, the most common ways include Jupyter Notebook and Jupyter Lab, and it is through these tools that modifications are written into the underlying .json file. That ensures the file is always valid. Modifying the .json manually could make the whole notebook unreadable and such errors and difficult both to find and fix.

The official documentation for this format is available as the nbformat Python Package.

Problems with the format

This JSON-based format is also the main cause of trouble when working in teams, such as edit collaboratively and version control. That is why tools such as Yjs and nbdime were developed to solve those issues, respectively.

Developer Exchange

All that said, among developers, .json files are just ordinary everyday files, and exchanging them is nothing new. On the other hand, even if those files are easy to send, one does require a whole setup (Python + Jupyter) to properly visualize the content of such files. That is why the first and most simple way to share and publish a notebook is through the built-in Export solutions.

Using Built-Ins: Export Solutions

Jupyter, in particular JupyterLab, introduces several formats for exporting the notebook. However, in this section one will be highlighted: HTML

Advantages

It can be sent in most commonly used media such as email, instant messaging apps like WhatsApp or Telegram, corporate team communication tools like MS Teams, Slack, and Google Chat.
It is self-contained, i.e. it is a single-file export.
It shows the notebook in the most similar way possible to what is shown when using Jupyter Notebook/Lab.
It does not require any additional tools to display, not even on mobile devices.
It is compatible with Analytics solutions like Google Analytics (should be added manually).
It is copy/paste friendly for code.
It correctly shows all formulae written in LaTeX.
It is plain text and thus version control friendly.
It displays all the outputs from the executed notebook, including plots and images.
It can be easily hosted on any web server without a special back-end.
It does not require any additional tool or set up for the export, only two clicks with the default Jupyter Installation.
It can display interactive visualization tools like Altair if the output is plain HTML/JS (provided that the library supports it).
There are no dependencies on third parties.

Disadvantages

It is not editable. - Technically it is, but not in the same way a Jupyter notebook is.
It cannot be executed, i.e. it is a static file.
It has no support for a common revision mechanism (notes, comments, highlighting, etc.).

When to use

This solution is the best when the other person, for some reason, will not run the notebook. For example, when sending sample code to a fellow developer, for documentation examples, and also for simple web hosting.

Some examples of this approach can be found in the PyMC3 official documentation, where all their examples are HTML exports of Jupyter Notebooks.

Why not other export formats?

Some may argue that PDF is also a good choice. However, due to pagination, generally the content and the output is split in unpredictable ways and thus making it harder for the reader to understand, as Jupyter Notebooks were not conceived to be read in separate pages.

Asciidoc, ReStructuredText, Markdown and LaTeX require special readers and assume that the other person also knows the format. Moreover, it is not easy to display on mobile devices without any additional software.

Executable Scripts (.py) are plain text and hence they sacrifice all the effort put in formatting and styling the markdown cells. And, if that was not the case, why bothering having a Jupyter Notebook at all to begin?

Finally, Reveal.js is meant for presentations and it is a certainly useful option for that matter, but it often shows a subset of the whole notebook and it is meant for the author itself and not for third parties.

Read-Only Solutions

Even though the last section covered HTML, which is also read-only, this section will focus on non-built-it approaches. Moreover, all of the solutions presented require some configuration or additional steps, the results might be worthwhile depending on the particular scenario.

Github Publishing

Github is the most widespread solution for hosted git version control repositories, this is also the case for the whole Project Jupyter, whose repositories are hosted in this platform. With time, they included a feature that shows the Jupyter Notebook as HTML without requiring the user to do such conversion.

The view will be the same as the one with the HTML export, there are some differences to keep in mind though.

Advantages

Most of the advantages of the HTML export (see disadvantages).
Hosted online for free, can be shared with a link.
It can be downloaded as .ipynb.
Automatic support for MyBinder and Google Colab (See next sections).
Support for version control out of the box as part of git.
Works with public and private repositories and also in Github Enterprise.
It is discoverable by Github Search Engine
Generate Repository-level analytics.

Disadvantages

Requires time to load the view, it may fail for extensive notebooks.
Requires a Github account and familiarity with git in general.
Requires readers to be somewhat familiar with Github UI.
Mobile design is not optimal
Has no cache mechanism, HTML views are computed each time.
Not compatible with Analytics solutions like Google Analytics.
Not compatible with interactive visualization tools like Altair or Plotly.

When to use

Ideal for people and teams already working with Github Repositories. If not working with them, other options like Google Colab or NBViewer might be better. Another possible scenario is when old versions of particular notebooks should be re-visited, Github makes it easy and intuitive. Nonetheless, If cell execution or interactive plots are necessary, this option is not suitable.

Most Github Repositories with Jupyter Notebooks can be used as examples (there were more than 66.000 at the time of this writing). One popular instance is the Python implementation repository of the algorithms of the Pattern Recognition and Machine Learning by Christopher Bishop.

NBViewer Publishing

NBviewer is a free, online tool from the project Jupyter to display notebooks online. It combines two Python packages, NBconvert f or notebook transformations and Tornado as web service.

In this platform, notebooks are presented in HTML, the same way as with Github but, the User Interface is much more minimalistic and it is not tightly integrated with any version control system.

Advantages

Most of the advantages of the HTML export.
Hosted online for free, can be shared with a link.
It can be downloaded as .ipynb.
Automatic support for MyBinder (See next sections).
It does not require any background knowledge (e.g. as opposed to Github publishing).
It caches results, load times are fast after the first load.
The UI has less branded content and is minimal, ideal for focus.
Can load notebooks from anywhere on the web, just with the URL.
It can be seamlessly integrated with Github (Repositories and Gists).

Disadvantages

Mobile design is better than Github
Because of its cache, results may take some time to refresh when updated.
Not compatible with Analytics solutions like Google Analytics.
Not compatible with interactive visualization tools like Altair or Plotly.
No discoverability capabilities such as Github's.

When to use

This solution is similar to Github's, but it is a better fit for non-developer users (e.g. mathematicians and statisticians may not have a background with version control). Furthermore, it is commonly used for writing static books, one of the most popular ones being Probabilistic-Programming-and-Bayesian-Methods-for-Hackers. As well as other HTML-based approaches, If cell execution or interactive plots are necessary, this option is not suitable.

Static Site Generator Integration

Nowadays, there is an increasing trend in popularity for Jamstack tools, i.e. Static Site Generators. That is why there are plenty of options available in the Python ecosystem (50 at the time of this writting){: target="_blank"}. That been said the most popular ones are Pelican, Lektor and Nikola based on Github Stars.

From these three options, Nikola is the only one with native support for .ipynb files, whereas Pelican and Lektor require plugins, once installed their use should be straightforward.

Another possibility is to use FastPages, an static site generator thought specifically for Jupyter Notebooks by the FastAI team. It is not as popular as the other three but it is raising in popularity in the last couple of years.

Independently of the particular generator, the advantages and disadvantages are identical.

Note: This approach is not embedding Jupyter Notebooks in a static site, but rather using the .ipynb file itself as the source. Embedding notebooks is covered as one of the Executable solutions.

Advantages

It provides the greatest level of customization and control by designing its templates
It is compatible with Analytics solutions like Google Analytics. Should be added manually.
It is compatible with interactive, HTML-based visualization tools like Altair. Most likely require some additional configuration.
There are no dependencies on third parties
The notebook is attached to the personal brand of the site and not "hosted in the platform everyone uses".
Better discoverability through search engines when optimizing SEO.
Readers do not see a Jupyter Notebook but a web page, without the structure of the cells, this produces a more article-like look and feel.

Disadvantages

It requires a lot of setups manually. Static Gen, Templates, Plugins, etc.
Usually, the deployment requires either a manual process or configuring a Continuous Integration / Deployment Pipeline, which can be done for free but configuring it is time-consuming.
It creates a whole site, which might be excessive for simple use cases.

When to use

This approach is more suitable for individuals and or organizations who want to keep and maintain a website/blog and at the same time showcase jupyter notebooks. It requires intermediate knowledge of software engineering and is maybe not the most adequate solution for those coming from other fields.

Furthermore, it assumes a continuous stream of new content rather than simple seldom updated publications. All the discoverability capabilities of other solutions are not included here, thus requiring additional work.

That being said, this is the only solution where the writer has full control over the implementation, the look and feel, and the UI in general, allowing to use Analytics solutions as well as some interactivity.

Why not Sphinx, MkDocs, Cactus, or X?

The idea behind publishing a Jupyter Notebook is different from software documentation, which is the main focus for Sphinx and MkDocs. Cactus is another popular alternative but its last commit was in 2017, making it obsolete. Other static site generators are less known and at the time of this writing do not present any meaningful integration with Jupyter Notebooks.

A clarification should be made about Sphinx since it is used as the back-end for another solution called Jupyter-book which will be cover below.

Jupyter-Book - Static

Jupyter-Book is a solution that can be used in Static, Executable, and printable scenarios, however, the advantages and disadvantages of each are slightly different and the level of maturity is not the same either. Therefore, in this article, it will be repeated for each category.

This tool has been migrated to the Python ecosystem (it used Ruby and Jekyll in the past), and although it is not extremely popular at the moment, the project is gaining traction and seems it would be the new standard in the short-term

As an additional resource for this article, a template repository to get started with Jupyter-Book was prepared, it works for static, executable, and also printable scenarios and can help getting started.

Advantages

It creates a self-contained output, giving consistency across the content
It creates tables of contents automatically
It is compatible with Analytics solutions like Google Analytics. Should be added manually.
Many configuration options
Support for bibliography
Extended markup features via MyST
Integration with Google Colab, MyBinder, and Thebe (see below sections)
It is not necessary to execute the notebooks beforehand

Disadvantages

It creates a self-contained output, making it difficult to integrate with existing websites
It is not possible to customize the theme out of the box.
Usually, the deployment requires either a manual process or configuring a Continuous Integration / Deployment Pipeline, which can be done for free but configuring it is time-consuming.
If using one of the advanced features of MyST, the notebook might be incorrectly display using any other solution shown here

When to use

Despite having a "book" in the name, the actual layout is similar to modern software documentation. The reason being the main theme is inspired by the pandas docs. If the content consists of several Jupyter Books that can be ordered into sections, chapters, or a similar hierarchical structure, Jupyter-Book is extremely useful

As an example, the data visualization library Altair used this tool to build their official tutorial.

Executable Solutions

In the previous sections, all solutions aimed at providing a static representation of the notebook. That was the most convenient way to share a notebook. Nevertheless, in some use cases, the content is required to be executable. For example, for reproducibility results in a research review; or to showcase to students what happens when the code changes in a pedagogical setting.

The fact that the notebook is executed means that the kernel should run on a server and the input should be sent and the output should be fetched. There are free servers that allow to do that but certainly with limitations or additional steps for configuration.

In this category, two backends will be shown: Google Colab and MyBinder. And four different ways are presented on how to use them, two direct and two indirect.

Important Note: The execution of the code will be always on a sandbox environment, meaning that there are no risks of running insecure code, neither for the writer or the reader.

Jupyter Lite

Jupyter Lite is a young (First commit on the 21st March 2021) yet popular tool (1.6k stars) developed by the Jupyter team. The tool is still unofficial as per their docs but, managed to achieve something that was never done before: Jupyter in the Browser without a Backend. That means that having only a static server could bring a whole Jupyter Notebook/Lab environment.

This works thanks to Mozilla's Pyodide, this other project ships the scientific Python stack compiled to WebAssembly and thus a backend is not needed as everything happens in the front-end.

Advantages

One of the most transparent experiences for Executable Jupyter Notebooks.
It does not require a setup for the reader.
The data is persisted to the browser localstorage.
It has multiple kernels (Python and Javascript) support built-in.

Disadvantages

It requires some setup for the writer.
It is a Notebook-only environment, it does not integrate with the existing website or platform.
Configuration is not as straightforward as with other solutions.
It is still in Alpha, meaning breaking changes can be introduced without prior notice. And the performance might be unstable (crashes).
Slower than solutions with a dedicated back-end.
Since there is no dedicated Back-End, performance might be compromised in low-end PCs.
Good with the defaults but, having a custom configuration may require additional setup.

When to use

For proofs of concepts without complex dependencies, good internet connection, and modern hardware. This might be a great choice. Uses cases might be MOOCs or other events where participants need Jupyter and no dedicated server is provided.

However, it is not recommended when a stable and robust solution is needed.

Embedded Solution

Embedding a notebook means inserting some input/output cells into an existing web page (generally but not necessarily static). The aforementioned interaction with the back-end is taken care of by NBInteract, a python package that was developed for this very purpose. In this blog, there is a brief tutorial explaining how to do this step by step

Advantages

It is fully compatible with Jupyter Widgets (ipywidgets).
There is full control about dependencies and environment.
The user sees the execution inside the same page, no redirections are needed.
It is fully integrated with MyBinder (see the section below).
Several independent notebooks can be embedded in a single static page.
It is compatible with Analytics solutions like Google Analytics. Should be added manually.

Disadvantages

Requires a Github account and basic use of git.
The configuration should be done on a notebook by notebook basis.
The user has to click a button and only then the process is triggered, leading to potential delays.
The reader cannot modify the code, interactivity is limited to widgets
The server that runs the code will be shut down after some minutes of inactivity.
The user experience might be a little uncomfortable because one has to wait until the kernel is ready

When to use

This approach resembles the "Flash Application" or the "Java Applet" where a small piece of an external tool is embedded into a static website. In this case, that external tool is a Jupyter notebook.

One of the articles of this blog uses this approach extensively to showcase Ordinary Differential Equations with the aid of different ipywidgets.

Although requiring some setup, it is one of the less disruptive approaches for writers and readers alike.

Google Colaboratory

Google Colaboratory or simply Colab is a service provided for free by Google that comes with these main features:

All commonly used data science libraries are pre-installed.
It integrates seamlessly with Google Drive.
It has support for GPU out of the box.

One of the interesting features is that any public Github repo can be opened directly from Colab by changing the URL.

Advantages

Seemless import from Github.
It does not require Github at all. Any user with a Google Gmail account can use it.
Easy to share with a link.
Easy to manage permissions, the same mechanism as Google Drive.
It offers GPU support out of the box.
New dependencies can be installed if needed, yet not automatically.
Can read and write from Google Drive.
URL replacement integration with Github.
There are no set up times.

Disadvantages

The interface is custom from Google, it is different than the traditional Jupyter Lab.
The user has to navigate to a different website to execute the notebook.
If one closes the browser tab, the session is lost.
The notebook kernel is shut down after some time of inactivity.
The only possible storage is Google Drive, hence inheriting size limitations of the associated account (15GB at the time of this writing).

When to use

The fact that Colab offers GPU support is a game-changer, no other service does it for free. Some teams may choose this solution solely because of this, especially in deep learning applications.

My Binder

MyBinder or simply Binder is a free online service that creates sandbox environments to run Jupyter. It has support for both Jupyter Notebook and Jupyter Lab. However, it offers CPU-only processing capabilities.

Advantages

It is used as a base for many other services.
New dependencies can be installed if needed automatically.
No apparent limit for data while the session is alive.

Disadvantages

It takes some time to build and run.
The user has to navigate to a different website to execute the notebook.
Configuring it for the first time might require some knowledge of Linux, Docker or Conda environments.
The notebook kernel is shut down after some time of inactivity.
There are no storage capabilities, the results should be manually downloaded.

When to use

MyBinder is a backbone service and for a notebook writer, it may not be that useful since other services like Jupyter-book or the Embedded approach use it under the hood. However, it is helpful to know some of the details of the internals to see what is possible.

The typical use case for bare Binder is to have a repo with a particularly complicated setup of dependencies that cannot be used otherwise. For example, using software requiring particular C libraries or certain operating system libraries pre-installed (e.g. FFMPEG, Libpostal, etc.).

A more concrete example is creating animations with matplotlib which requires FFMPEG for some output formats, being FFMPEG not a Python library but software that should be installed on the Operating System level. The article on Times Tables covers this precise problem

Jupyter-Book - Executable

As mentioned in the static section Jupyter-Book can be used for static and executable content. In the case of executable content, it provides support for three different services:

Google Colab
MyBinder
Thebe

Since Colab and Binder were already discussed, this section will focus on Thebe.

Thebe is a service that acts as the embedded approach described above with the difference is that it is integrated with Jupyter-book since both projects are under the bigger Executable Books Project, but it is not limited to Jupyter-Book. Thebe uses MyBinder as a back-end and provides an execution environment for static sites.

Advantages

All of the advantages of using MyBinder
The user sees the execution inside the same page, no redirections are needed.
Seamless integration with Jupyter-book, no additional configuration needed.
Full support for custom dependencies through MyBinder

Disadvantages

The user experience might be a little uncomfortable because one has to wait until the kernel is ready
Configuring without using Jupyter-book might require HTML/JS knowledge

When to use

Thebe is a young project (223 Github stars at the moment of this writing) and there is not a big collection of examples nor is it a well-known tool. However, when used integrated with Jupyter-Book, it gives the reader the ability to "suddenly being able to run the code examples" which is a great user experience.

At the moment, it is the ideal solution if already using Jupyter-Book, for a similar approach using a more standalone tool, the embedded approach should be preferred.

Printable Solutions

In some cases like academic publishing, book writing, or learning material the preferred format is print paper. In this case, there are a variety of options but the choice is not dependent on the technological feasibility but rather on the target platform.

If writing a book, a publisher will likely require a specific format, if going self-publish, usually, only a PDF will be enough.

In case of requiring AsciiDoc, Markdown, or ReStructuredText, Jupyter has built-in options for exporting to these formats, then the particular publisher instructions should be followed. Since this may vary significantly in each case, this section will focus on the most used format of academia, LaTeX. LaTeX is a special format that can be easily converted to PDF, therefore the term PDF in this section will always refer to the LaTeX conversion to this format.

Regardless of the approach taken, it is important to note that LaTeX can be used for two purposes: articles and books. Typically articles are generated from a single notebook file (not exclusively though) and books are a collection of notebooks following a hierarchical structure.

For books, specific book templates should be used to give the proper format, including page numbers, headings, and such. That is why tools like Overleaf are recommended to structure the document.

Built-in Export: LaTeX

The native LaTeX export will convert the notebook to the LaTeX format. This is ideal for single file conversions and should be the preferred way to start articles based on Jupyter Notebooks.

Advantages

The same notebooks can be used for static/executable printing, no printing-exclusive modifications are needed
It is a native, straightforward solution.
All images and figures are handled automatically.

Disadvantages

If using more than one notebook, each should be exported manually (can be automated with nbconvert though).
Usually, the code should be removed manually.
It requires final editing on an external platform.

When to use

The built-in LaTeX export is suitable when dealing with single notebooks that are not connected. It might be helpful to start an academic article but in such media often the code should be removed.

If the target is not an academic conference/journal or a publisher, other tools like the static site generator or the other plain text, export might be more suitable dependent on the particular use case.

Jupyter-Book - For Printing

If already using Jupyter-Book, one possibility is to use the LaTeX target and create a .tex with all the notebooks converted to LaTeX. This approach will produce a consistent and book-like structured document. Some final formatting might still be needed in third-party tools but it is much less work than the default LaTeX export.

Advantages

The same notebooks can be used for static/executable printing, no printing-exclusive modifications are needed
It combines multiple notebooks following a predefined structure.
It can export multiple notebooks at the same time.
It creates a numbered table of contents.
It handles global bibliography.
It supports basic PDF export.
Can be integrated to generate a PDF on each commit.

Disadvantages

The support is still experimental and might change.
It requires configuring a CI/CD Pipeline.
It requires final editing on an external platform.

When to use

At the moment LaTeX and PDF are not the main focus of Jupyter-Book and the design might appear basic, however, it is indeed a good starting point for books. Using the tool for single notebooks might be excessive.

As an example, the official Jupyter-Book project generates a PDF version of their docs in the form of a book, using the LaTeX export option in the middle

Honorable Mentions

Some tools could be useful in some contexts but are not so widespread as the ones presented so far.

NBdev: is a tool developed by the FastAI team and can be used to create Python libraries from Jupyter Notebooks. The whole FastAI framework is built with it. Not in the list because building libraries from notebooks is more of a niche application and the project is still very young (just a few years).
Fastdoc: also from the FastAI team, it is a library to create publication ready books from Jupyter Notebooks. It was used to write the O'Reilly FastAI Book. Not in the list because the project is not currently active (Latest release October 2020) and it is not as widespread as Jupyter-Book.
Starboard is an online service that provides literal notebooks (not Jupyter) with Jupyter-like capabilities such as Code and Text cells. It runs completely on the browser (with Pyodide) and its goal is to make notebooks easier to share. Not in the list because it is its own format, not Jupyter (although Jupyter import is possible), and because it is in early development.
Github Codespaces: is a Github project to work online in a Visual Studio Code hosted on the browser. At the time of this writing, it can be used in any repository by pressing "." (dot), one can navigate and edit the files with the free tier. To attach a compute instance, a paid service is offered to selected beta-tester organizations. Not in the list because it does not work with Jupyter out of the box and it only has a paid solution.
Cocalc: a tool very similar to Google Colab but with a less powerful free tier plan. It does have unique features such as Chat Rooms and collaborative editing. Not in the list because there is a feature overlap with Colab and it is not as widespread.
Spell.ml: it is a service to provide workspaces to run Jupyter Notebooks. It has CPU support for free with additional paid GPU instances. Not in the list because it is not for publishing as only a single user has access to the notebooks but can be used to run long computational notebooks without needing the browser tab open as MyBinder or Colab.
Datalore: the Jetbrains dedicated IDE for Jupyter Notebooks. It has some advantages as collaborative editing but it has only a limited amount of hours for the free plan. Not in the list because it is not an open-source solution and the policy of the free plan can change without notice.
Your suggestion: if you want to add another solution to the list, send me an email with a brief description of the tool at castanoezequieleonardo at gmail.com

Choose the best tool

To choose the best tool, look for the use case that best represents your needs and see the available options

I need GPU Support

The only option is Colab.

I want to keep my readers on my site

Then use either the Embedded or Thebe.

I need Analytics

Use Jupyter-Book or Embedded approaches.

I need Cache for fast load time

Use NBviewer.

I want to create a portfolio using Jupyter Notebooks

Use a static site generator. This site was built using Pelican.

I have a very specific setup of dependencies

Use any of the MyBinder-based approaches:

MyBinder
Embedded
Jupyter-Book

Conclusion

There are many different tools to publish Jupyter notebooks and whole tutorials could be written about the features and possibilities of each. In this article, the most compelling ones were summarized to help the notebook writer to decide which tool is the most suitable for their use case.

If any of the information presented is outdated, or you have suggestions for new advantages or disadvantages of a particular solution please leave a comment below.

Below is a list of useful resources to continue learning and searching for inspiration.

Additional Resources

Awesome Lists:

Gallery of Jupyter-Books

Some Books built with Jupyter Notebooks:

Lyrics Presentation Generator

2019-05-22T00:00:00-03:00

More often than not I find myself writing power point presentations for Lyrics. This usually happens in the youth service at church and in christian camps. Since the process is quite repetitive and simple enough I consider developing a web app to generate this presentations automatically. In this article I will show it how it works and how to use it.

The Problem

As said in the introduction the problem is basically, avoid writing over an over the presentation for the lyrics

The Solution

Developing a Web App to automatically generate the power point presentation. This app doesn't require any installation and will work from any browser, but for a better user experience it is recommended to use from a desktop/tablet environment. In this article, I will dedicate the next section to explain how to use it for non-technical users and then I will explain all its components from a technical perspective

Note: I understand the importance of responsive design but to in this case in order to produce a presentation that match the text, bigger dimensions than the usual cellphone provides are needed. Besides, it is unlikely that the user will download the presentation from a mobile device.

How to use it

The app is available online, as soon as you open it, you will see something like the following:

In the screen there are 3 main areas:

Lyrics Area: is the area marked in orange, this is where the text of the lyrics will be written. To properly write the lyrics, one should follow the instructions (see below).
Language Selection: In the top right corner, there are two flags marked with green, each flag changes the language of the whole page.
Buttons: This area is marked with purple, it includes two buttons, one for generating the presentation and the other to clear the text area.

Instructions for writing the lyrics

In order to write the lyrics, the following guide should be followed:

Write each Verse with no blank lines in between.
Separate Verses with blank lines.
Separate Songs with three dashes, "---".

The size of the Lyrics Area is set to match the resulting presentation so if the text fits well in the web page, it will fit in the presentation as well.

How it works (Technical)

This web app is built using vanilla HTML5, CSS3 and Javascript. Additionally, a Javascript library is used for creating the presentation, this library is PptxGenJS.

The CSS is written inside the HTML to avoid using another request, this is done because the whole style is about 100 lines and the whole app is contained in a single html file. The Javascript is separated to keep logic and presentation layers separated.

First I will show all the code and then split it into pieces and explain part by part

The Javascript code to achieved this is the following:

function createPPT() {
    var pptx = new PptxGenJS();

    pptx.setTitle("Presentation created with ELC's Presentation Generator");

    pptx.defineSlideMaster({
        title: 'Template',
        bkgd: '000000',
        objects: [{
            'placeholder': {
                options: {
                    name: 'body',
                    type: 'body',
                    x: 0,
                    y: 0,
                    w: "100%",
                    h: "100%",
                    align: "center",
                    fontSize: 44,
                    color: 'FFFFFF',
                    fontFace: 'Verdana',
                    valign: 'middle',
                    paraSpaceAfter: '6',
                    paraSpaceBefore: '6'
                },
                text: ''
            }
        }]
    });

    var text = document.getElementById('lyrics').value

    parseLyrics(text).forEach(verse => {
        if (verse === "---") {
            createSlide(pptx, '');
            return;
        }

        createSlide(pptx, verse);
    });

    pptx.save("Presentation - Created with ELCs Presentation Generator");
}

function parseLyrics(text) {
    return text.replace(/[\r\n]{3,}/, "\n\n").split("\n\n");
}

function createSlide(pptx, text) {
    var slide = pptx.addNewSlide('Template');
    slide.addText(text.toUpperCase(), {
        placeholder: 'body'
    });
}

function clearText() {
    document.getElementById('lyrics').value = '';
}

function changeLang(locale) {
    document.webL10n.setLanguage(locale);
}

window.addEventListener('localized', function () {
    document.getElementById('lyrics').value = document.webL10n.get('verse1') + "\n\n";
    document.getElementById('lyrics').value += document.webL10n.get('verse2') + "\n\n";
    document.getElementById('lyrics').value += document.webL10n.get('separator') + "\n\n";
    document.getElementById('lyrics').value += document.webL10n.get('verse1') + "\n\n";
    document.getElementById('lyrics').value += document.webL10n.get('verse2') + "\n\n";
    document.getElementById('lyrics').value += document.webL10n.get('verse3');
}, false);

Main Function

The createPPT function is the main function which is called whenever the "Generate Presentation" Button is clicked. It performs the following actions:

Create the presentation object.
Set the Title.
Define the template
Extract the text
Add each of the slides
Save the file

Each of the actions is represented in one of the following lines. To keep it simple, the most complex tasks are explained in further detail below. The methods used for setting the title and saving are self explanatory.

function createPPT() {
    var pptx = new PptxGenJS();

    pptx.setTitle("Presentation created with ELC's Presentation Generator");

    // Template Definition

    // Text Extraction

    // Add Slide

    pptx.save("Presentation - Created with ELCs Presentation Generator");
}

Template Definition

In order to achieve a consistent style, a template should be used. In my own experience, black background with white letters gives the best results. To define a Template the method defineSlideMaster. Its parameters corresponds to achieving slides with the following characteristics:

Black Background Color: bkgd: '000000'
Assing a unique title to use when creating the slide: title: 'Template'
Give a unique identifier to a text element inside the slide: name: 'body'
Set the coordinates for the text element: x: 0, y: 0,
Set the dimensions, both width and height: w: "100%", h: "100%",
Use a big font size: fontSize: 44
Use White color for the font: color: 'FFFFFF'
Center the text horizontally and vertically: align: "center", valign: 'middle',
Set paragraph spacing: paraSpaceAfter: '6', paraSpaceBefore: '6'

When combining all the above and following the syntax provided by the official docs, the results is the following code:

// Template Definition
pptx.defineSlideMaster({
    title: 'Template',
    bkgd: '000000',
    objects: [{
        'placeholder': {
            options: {
                name: 'body',
                type: 'body',
                x: 0,
                y: 0,
                w: "100%",
                h: "100%",
                align: "center",
                fontSize: 44,
                color: 'FFFFFF',
                fontFace: 'Verdana',
                valign: 'middle',
                paraSpaceAfter: '6',
                paraSpaceBefore: '6'
            },
            text: ''
        }
    }]
});

Text Extraction

Once the template is defined, the text should be extracted, to achieved this a very common practice is used in JS. Just referenced the element by a given ID and then extract its value. In this case the element is a TextArea defined in HTML5

The resulting code is the following:

// Text Extraction

var text = document.getElementById('lyrics').value

Add Slide

After extracting the text, it should be parsed and then a new slide should be added for each verse.

The parsing function returns a list of strings, each being either a verse or a --- which basically mean "Empty Slide". Both the parseLyrics and the createSlide are explained below.

// Add Slide

parseLyrics(text).forEach(verse => {
    if (verse === "---") {
        createSlide(pptx, '');
        return;
    }

    createSlide(pptx, verse);
});

Parsing

One would never know what the user would write, and, in order to support several languages (specially those with special characters), it is important to use a separator that works universally, in this case the blank line will separate two verses and the three dashes (---) will separate songs.

In order to ignore several blank lines and then split a regular expression and then the split method is used.

function parseLyrics(text) {
    return text.replace(/[\r\n]{3,}/, "\n\n").split("\n\n");
}

Create Slide

Each slide will have a different text so to create a slide one should provide the engine (in this case the library pptx) and the text to be inserted (the text argument), then the new slide is created using the predefined template, after that the text in upper case is added using the placeholder defined in the template.

function createSlide(pptx, text) {
    var slide = pptx.addNewSlide('Template');
    slide.addText(text.toUpperCase(), {
        placeholder: 'body'
    });
}

Clear Text Button

In case the user wants to delete everything in the Lyrics Area, a button is provided, the functionality is overwriting the current content of the TextArea with an empty string.

function clearText() {
    document.getElementById('lyrics').value = '';
}

Localization

This extra feature provides client-side localization via WebL10n, each string of the web app is both available in English and Spanish, but more languages could be added in the future if needed.

function changeLang(locale) {
    document.webL10n.setLanguage(locale);
}

window.addEventListener('localized', function () {
    document.getElementById('lyrics').value = document.webL10n.get('verse1') + "\n\n";
    document.getElementById('lyrics').value += document.webL10n.get('verse2') + "\n\n";
    document.getElementById('lyrics').value += document.webL10n.get('separator') + "\n\n";
    document.getElementById('lyrics').value += document.webL10n.get('verse1') + "\n\n";
    document.getElementById('lyrics').value += document.webL10n.get('verse2') + "\n\n";
    document.getElementById('lyrics').value += document.webL10n.get('verse3');
}, false);

Embed Interactive Jupyter Notebooks in Static Websites for Free

2019-05-01T00:00:00-03:00

After a lot of research on the internet, I found no practical tutorial explaining how to embed Jupyter Notebooks in Static Websites using only free technologies. I found a way to do it using Github Gists, MyBinder and NBInteract along with IPython Widgets and I want to share it so no one has to reinvent the wheel.

This is a step by step guide to embed Jupyter Notebooks in a static website. There is also an official tutorial available in the docs.

This guide will cover some points that aren't fully explained in the official tutorial, such as:

Use Gists instead of Repositories
Reduce the size of the output HTML
Specify dependencies for Binder
Use only some cells and insert them in a blog post via IFrames

Motivation

I really like using Jupyter Notebooks for my experiments and I also like to show them in this blog, which is a pure static website built with Pelican. The problem emerged when I wanted to embed some interactive content from the notebook in the website, I thought this should have been solved long ago but it wasn't or at least it wasn't as Public as it should be.

Having the possibility to embed Jupyter notebooks in a static website provides lots of advantages, just to mention a few:

Visitors don't need to leave the site in other to see the notebook.
Visitors can actually interact with the code and see the results live even without knowing how to program.
You can provide a quick way to show the results with prior and posterior explanations without writing everything in a notebook.

The Tech Stack

In order to do this, I used a set of tools. This is my personal choice but I believed is appropriate, of course, if you have any suggestions, feel free to write a comment below.

Github Gists: To host the code and provide isolation.
NBInteract: To communicate with a JupyterHub Server.
IPython Widgets: To give interactivity to the notebook.
MyBinder: To provide a server for the computations.

The key component here is NBInteract but the other tools were necessary too.

First, I want to clarify my workflow:

I want to write an isolated notebook that the user could run via Binder (1), I want to write a blog post about what I've done in that notebook (2) and in the post, I want to insert specific cells when appropriate (3). Additionally, I want my experiments pieces (the specific cells inserted) to be sharable, meaning any could embed them easily in their website (4) and I also want to track how many times they were used and from which source (5).

These 5 objectives are accomplished in the following way:

Since my experiments are usually independent of each other, I host them in Github Gist instead of repositories. I believe it makes no sense to have a one-file Repo without a very sound reason.
I use Pelican to write the post but you can use whichever tool you are used to. I know one could write posts inside Jupyter Notebooks themselves, even books have been written this way. But for consistency reasons, I want all my posts to be in Markdown.
NBInteract is used to insert specific cells in the post. A little modification is needed to make it work with Gists instead of repositories.
Everything above should be contained in a single HTML file and embedded through an IFrame.
Integrate Google Analytics in the HTML from the previous step and add a filtered view in the admin panel to have a dedicated dashboard for this IFrames.

The full process could be separated into the following steps:

Environment Setup
Gist Creation
IFrame Creation
Testing and Adaptation

1. Environment Setup

First thing first, you need a fully working notebook. It should run all cells without errors in your local machine.

In order to take full advantage of this methodology make sure you use widgets (such as IPython's) to give control to the user. Don't rely on manually changing variables inside cells but rather provide a user-friendly interface with buttons and sliders, this will result in a much more pleasant and useful UX/UI for the reader of your blog. Besides, this way, the reader doesn't have to know programming at all!

When you have that done, you have to identify the notebook dependencies. It could be either from Conda, pip, apt or anywhere else.

There are basically four ways you can specify the dependencies for Binder

requirements.txt: only suitable for pypi-only dependencies
environment.yml: better for Conda dependencies and pypi dependencies
apt.txt: only option to specify apt dependencies.
dockerfile: specify not only the dependencies but the whole OS.

The most common approach to work with Binder is a combination of environment.yml and apt.txt. Requirements.txt usually provide very little support for customization and dockerfile is discourage from the Binder developers themselves and should only be used when no other option worked.

A more detailed explanation which many other ways can be found in the official docs

2. Gist Creation

Gists are like small repositories, they have a sort of version control called "Revisions" but there are no branches, they provide simple updates one after the other. They are especially useful for cases where a repository is too much and just a few files are needed.

Here I will explain how to create a Gist in Github, in case you already know it, you can skip to the next section, where I explain how to set the dependencies for Binder.

Creating a Gist

In order to create a Gist, you have to first have a Github Account.

Once logged in, you have to click on your profile image in the top bar

Now select "Your Gists"

The next page will change if you already have gists created but the top bar will remain the same, in the top bar select the "+" icon

Now you are in the Gist creation Template, here you have to write the Gist Name, the filename of the first file, its contents and you can additionally add files if necessary with the button below. When everything is set up, click in create public gist.

Note: You can also create private gists for free but for NBInteract and thus this guide to work, the gist should be public.

Dependency Management

You have to create a Github Gist and upload all the files needed, namely:

The repo/repository file (.ipynb files)
The dependencies files (requirements.txt, environment.yml, apt.txt, dockerfile)

Once the gist is created, it should be tested with MyBinder

In order to test it first select the Combo Box in the main page

Then select the Gist Option

And finally, write the Username/GISTID in the textbox and then click on "Launch"

Note: The first time you launch could take several minutes because it's building a docker image, this process is repeated each time you change the Gist so it is a good idea to always run your gists on Binder after you made a change.

If the build was successful, open the notebook and run all the cells, check for errors, fix them until none appear and then continue to the next step.

This step might seem to be pretty easy but there are cases (especially when doing interesting things) that may require additional settings, for example, setting the FFMPEG dependency correctly for creating animations and videos.

You can check some of my personal examples of Gists:

ODE with Python (environment.yml)
Pascal Triangle Visualization (environment.yml + apt.txt)

With this step completed, you can add a Binder Badge in your post directly to the notebook. Although the main objective is to embed specific parts, it doesn't imply the user wouldn't want access to the whole notebook in a more familiar environment such as the one Binder Provides.

Example of a binder Badge (it leads to the a Notebook about Ordinary Differential Equations):

3. IFrame Creation

This is the most important part and where NBInteract comes in. For all the previous steps we used online tools such as Github Gists and MyBinder. But for this step, nbinteract should be install.

Installation

These are the installation instructions:

pip install nbinteract

# The next two lines can be skipped for notebook version 5.3 and above
jupyter nbextension enable --py --sys-prefix widgetsnbextension
jupyter nbextension enable --py --sys-prefix bqplot

The steps for the installation could be found in its official repo, make sure to check it in case there is an update.

Usage

The nbinteract is a command line application (CLI), so first navigate to the directory where the notebook file is and then run:

nbinteract {notebook_filename}.ipynb -s {GitHub_UserName}/{Gist_ID}

This will create an HTML file with the same filename of the notebook, but as mentioned earlier, the current version of NBInteract (0.2.4 when writing this) only supports Github Repositories and not Gists.

To fix this problem, just open the HTML file and look for this snippet (it should be at the end of the file):

<script>
    (function setupNbinteract() {
        // If NbInteract hasn't loaded, wait one second and try again
        if (window.NbInteract === undefined) {
        setTimeout(setupNbinteract, 1000)
        return
        }

        var interact = new window.NbInteract({
        spec: '{GitHub_UserName}/{Gist_ID}/master',
        baseUrl: 'https://mybinder.org',
        provider: 'gh',
        })
        interact.prepare()

        window.interact = interact
    })()
</script>

Only a small change is needed, change the provider from gh to gist. It should be something like this:

<script>
    (function setupNbinteract() {
        // If NbInteract hasn't loaded, wait one second and try again
        if (window.NbInteract === undefined) {
        setTimeout(setupNbinteract, 1000)
        return
        }

        var interact = new window.NbInteract({
        spec: '{GitHub_UserName}/{Gist_ID}/master',
        baseUrl: 'https://mybinder.org',
        provider: 'gist',
        })
        interact.prepare()

        window.interact = interact
    })()
</script>

Adjustments

Now the file is already prepared to be embedded as an HTML IFrame, but although that's technically true, we might want to make other adjustments:

Create a valid HTML5 document (some closing tags may be missing).
Reduce the size of the file (by default they are quite big).
Use only a subset of cells and not the whole notebook.
Add a tracking system.

HTML5

The first adjustment is for improving SEO and compatibility, it will depend on every particular case as each notebook may need its own little changes. The W3C has an online validator to check for Markup errors and warnings

Size

The second adjustment is needed since nbinteract inserts Bootstrap and Font Awesome by default in the HTML, you can check it because the style tag covers approximately a 95% of the file. With custom styles and a careful selection of CSS classes, one can reduce the size pretty significantly. In my own experience, I manage to reduce from 14000 lines to 400 (318KB to 25KB), this is a reduction of the 90%!

Of course it may vary from user to user but since file size is crucial for a good user experience, I recommend to tweak and adapt the file to be as little as possible, and remember to use CSS in style tags and not relying on the website general CSS, this would be the only way to provide the same look and feel when the file is used as an IFrame. This way each file will be completely independent of each other.

Specific Cells

The third adjustment is more of a hack actually. NBInteract is designed so that the user would use the notebook as the entire page but if the notebook contains just some cells (each with the proper imports), one can create minimal notebooks and insert them individually in a larger pure HTML page

Just use nbinteract with a notebook file containing only the cells you are interested in. This will naturally lead to having as many .ipynb files as a group of cells you want to embed, each with an associated HTML file. It isn't a good idea to have everything in a big Notebook but neither is to have dozens of little notebooks. I believe a reasonable amount would be between 5 and 10.

Tracking System

The fourth and last adjustment will depend on the tracking system of your choice but generally, it implies adding some script tag at the end of the file with a specific tracking ID. But DO NOT add the system yet! Let's test it and then add it to avoid meaningless analytics (unless you've added your IP as an exception in the Tracking System).

4. Testing and Adaptation

Lastly, you should embed the IFrame in your website and test it, here is where you should modify the styles according to the general aesthetics of the page and check if all works as desired. Make sure the IFrame size is correct for its content (avoid scrollbars)

You can also test whether you want the input cells to be visible or only let the output to be rendered.

After everything is tested, you can now add the tracking ID for the Analytics service (Google Analytics or the one of your choice)

To make your IFrame easier to share, add an HTML snippet to specify how to embed your IFrame in other pages, see the examples below.

Examples

You can find some examples in the official docs but I've also built some that you may find interesting. You can check the following posts to see them:

Ordinary Differential Equations

Use NBInteract to numerically integrate several types of differential equations (including systems of ODEs) and change the parameters with sliders and see the results live.

If you want to embed this widget in your website, just add the following HTML:

<iframe src="https://elc.github.io/blog/iframes/ode-python/foxes-rabbits-iframe.html"></iframe>

Read the Full Article

Circular Times Table

Use NBInteract to produce a video with matplotlib animation and FFmpeg. The user can create an image and a video with the set of parameters of their choice and then save them.

If you want to embed this widget in your website, just add the following HTML:

<iframe src="https://elc.github.io/blog/iframes/times-tables/times-table-line-by-line-iframe.html"></iframe>

Read the Full Article

Additional resources

In case you want to go deeper, here are some useful resources:

Other resources to achieve similar things are:

NBDime: Tool to work with git and Jupyter Notebook for easy Diff
JupyterBook: This tool also integrates executable and interactive notebook cells but rather than embedding them into a webpage, it creates the whole website. Alternative to GitBook
Voila: Interactive renderer for creating Dashboards
SageMathCell: A way to generate embeddings online for cells of notebooks. It supports Sage, Gap, GP, HTML, Maculay2, Maxima, Octave, Python, R and Singular
ThebeLab: Thebe Lab turns your static HTML pages into interactive ones, powered by a kernel. It is an experiment attempting to rebuild Thebe with javascript APIs provided by JupyterLab.
Awesome Jupyter: A curated list of awesome Jupyter projects, libraries and resources.

Introductory Guide to Git with Visual Interface

2019-04-24T00:00:00-03:00

As a Systems Engineering student, on successive occasions, I came across version control systems (VCS for short), sometimes from subjects related to programming and other in subjects related to software engineering.

Remembering that when I had to learn all these concepts (especially how Git works), I went through several difficulties, I decided to make an introductory guide with basic concepts of Git and how to use it through software with a visual interface.

Personally, I think that to begin with, using a visual interface can be useful and didactic, however, I share the idea that understanding how the tool works from the console is fundamental for a deeper understanding of the concepts.

This tutorial will summarize the main features of a particular software called GitKraken, that I personally think is the best. This guide isn't intended in any way to be exhaustive or detailed in all the functions of GitKraken, it will rather give the user the basic knowledge so that they can use the tool and gradually explore the other features on their own.

At the end of the document some of the advanced features of Git are detailed so that the user can know what those are and what is their purpose.

This guide have independent sections so I leave the table of contents below in case you want to skip to a particular section

Introduction to Git
Fork of a repository
- 1. Find the original repository
- 2. Perform Fork
Clone a repository
Open a repository
Interface Elements
- Basic interface
- Commits and Branches Tree
Add Changes
Synchronization with remote repositories
Working in multiple branches
Pull Requests
Additional Features

Introduction to Git

Git is a version control system (VCS) technology, it arises as an alternative to SVN, Hg, and TFS and its original purpose was to control the Linux kernel versions.

Before starting with the specific vocabulary it is necessary to explain why you should (many people would say must) use VCS (git or other).

Problem: When working on a project, files undergo multiple transformations, this is especially noticeable in graphic design and software development projects. As time goes by, it's normal to detect errors and then returning to a previous stage, what is often done to accomplish this is to create periodic backups, however, this approach is impractical in environments with constant changes and increasing number of files, this problem also grows substantially when working in large teams.

Solution: Each VCS proposes a different solution, in the case of Git, when talking about changes, it is meant to be a change at line level, ie. a file changes when one of its lines changes and if several changes are made on the same line, counts as a single change. Likewise, if several lines are changed in the same file, it is considered as many changes as lines were modified. Whenever we talk about changes, we will refer to this definition of change, at a line level. This makes Git extremely useful for text files with multiple lines (such as the source files of a programming language) and little applicable to files that don't follow this format (such as executable files, images, etc.).

Areas in Git

Git works mainly separating the changes into 3 areas:

Working Area: These are the files you are currently working with.
Staging Area: These are the next changes that will be saved in the repository.
Repository: These are the saved files, usually local and on the internet.

Also, to move from one area to another, there are different transitions. To pass from the Working Area to the Staging Area a Stage is made and to pass from the Staging Area to the repository a Commit is made.

Stage vs Commit

The typical process is as follows: the user works with their files in the Working Area, once he has finished the desired changes, he can select exactly which changes he wants to add to the repository, these changes are chosen and "moved" to the Staging Area (the term move in this case is misleading because in the local system the files do not undergo any modification). Once all the changes have been moved to the Staging Area you can make a commit, the commit basically consists of "packing everything that is in the Staging Area, associating a title and a description and uploading it to the repository".

At first this way of working could give the impression that the Staging Area does not fulfill any function, however it is especially relevant when one wants to make a commit with a subset of all the changes made, for example, if changes were made in the graphical interface and in the database but one wants each "package of changes" to be in a different commit, for that, first, the user would move to the Staging Area the changes related to the database, then they would make the commit and finally they would repeat the process with the changes related to graphical interface.

NOTE: The user is not required to commit all the changes.

WARNING: One could move to the Staging Area only a subset of the modified lines in a file, that's why it's important to remember that the term "change" refers to lines and not to files.

Once the commit is made, the changes are definitively saved in the repository, the repository always exists at a local level on the user's machine and additionally, a copy is usually used remotely in some online provider, the most common are GitHub, Gitlab and Bitbucket.

Remote Repositories

When you work with a remote repository, three important terms arise: Pull, Push, Clone. As the names suggest, Pull brings the changes from the remote repository and applies them to the local repository, Push uploads the changes from the local repository to the remote repository and Clone, on the other hand, copies the remote repository and creates a local one, this operation is only done once (not to be confused with Fork).

Fork vs Clone: A Fork is an independent copy of a repository. When one makes a Clone, one intends to work in their local computer and then upload the changes to the original repository, when one makes a Fork one intends to make a custom variant and become independent (to a certain extent) from the original repository. Making a Fork replaces the process of creating the repository and to do so one must go to the original repository.

NOTE: As long as the changes do not overlap (i.e. two users have not modified the same line), the same Git system takes care of combining them. When two or more users modify the same line of the same file, a "Conflict Resolution" is necessary. It is an advanced topic however it can be required daily in environments with multiple programmers, those who use git as the only users should not worry about this aspect.

Branches

The last important concept is the branches, this is the key concept of Git, within a repository one can have multiple branches, each branch allows to have an independent Working Area, this is done to keep the changes isolated, although the Stage-Commit process is the same as the one mentioned above, one may want to experiment with a new feature but without risking the rest of the codebase that already works correctly. In the case the experimentation is successful, it can be joined (merge) to the main branch. There are multiple criteria for the creation of branches, some organizations use a branch by features, others a branch by developers, etc. The way to use the branches is usually called "Git Strategy" or "Git Workflow", is considered good practice to have a simplified scheme with the following branches:

Master: Where you have the code of the latest stable version
Release: Where all the changes that will be in the next version are concentrated and the final tests are carried out.
Develop: Where you have experimental code that is still being tested
Hot Fix: Where errors detected a posteriori are repaired (outside the tests)
Features: An independent branch for each feature you want to add to the software.

This can be visualized much better in an image. Each arrow indicates a merge operation.

Additional Software

Git is a command line application (CLI), however, there are multiple softwares with a graphical interface built on top of this console application. In this case, we are going to talk about how to use one of the most popular: GitKraken.

For the sake of simplicity, it is assumed that the user has correctly installed the Gitkraken software and that it is ready to use, therefore no initial installation and configuration steps are detailed.

Fork of a repository

As mentioned, a fork is a stand-alone copy of a repository. The repository already has its own creator and history of branches and commits but one makes a separated copy. This procedure can vary from one remote repository provider to another, here is the procedure for doing this in Github

1. Find the original repository

First, we look in Github the repository from which the Fork is wanted, in this case, the Manim repository will be used, the animation engine created by 3Blue1Brown.

As you can see in the image, the repository creator is 3b1b and the currently logged user is ELC so the owner and the creator are different users.

2. Perform Fork

To make a fork, simply click on the Fork button at the top.

Immediately afterward, GitHub will redirect us to a screen that tells us that the Fork is being created.

Once the process is finished, a repository of the same name will be created in the logged user account.

As you can see in the image above, the repository now belongs to ELC, with a small label below that indicates that the repository is a Fork. A sign that this repository is independent and belongs to the logged user is that there is a new section "Settings" that otherwise would not appear.

NOTE: This section may appear if you are a contributor to the repository, however that feature may vary from provider to provider and will not be detailed in this guide.

One last confirmation could be the Forks count which is a common number for all Forks.

From now on the evolution of this new repository can be completely independent of the original.

IMPORTANT:What is the difference between making a fork and creating an identical repository with the same files? When doing a Fork you have a common commit history with the original repository, so it is possible to combine new changes from the original to any of the Forks (merge) and from the Forks back to the original (Pull Request).

The alternative procedure of creating a repository from scratch as well as cloning from GitKraken is detailed in the following section

Clone a repository

The most traditional approach is to first create the repository on the remote server (Github or similar) and then clone it. In this case, we are going to create a repository called "gitkraken-tutorial" in Github.

1. Create a repository

This can be done from the Github homepage.

1 (Alternative). Create a repository

Or from the repository section.

2. Complete the necessary information

IMPORTANT: In order to clone, our repository must be "Initialized" therefore, it is essential to check the option "Initialize this repository with a README". The gitignore options allow git to ignore certain files (temporary files, caches, etc.) and the license allows you to choose a standard license from a list.

3. Get Repository URL

Two perspectives can be used to clone the repository from GitKraken:

Linking GitHub's account with Gitkraken's account
Use the repository URL

All git repository services attach a public URL to the repository, so this approach will be taken because it is common to all providers.

Once you follow the steps in the images, the URL will be copied to the clipboard.

4. Open the window of Clone inside the GitKraken

In the File menu, select the option "Clone Repo".

4 (Alternative 1). Open the Clone window inside the GitKraken

At any time, you can use the button located in the upper left corner.

Then choose the Clone option.

4 (Alternative 2). Open the Clone window inside the GitKraken

If no repository is open, the menu can be accessed directly from the home screen.

5. Paste the URL

Additionally, you can detail the path where the repository will be cloned and change the name of the destination folder. It is recommended to have a "Repositories" folder and clone all repositories there.

5. Notification of Success

Once the repository has been cloned, a notification should appear at the top saying that the operation was successful. One can use the "Open Now" button to open the repository immediately. In this case, it will not be done to illustrate how to open a repository from scratch.

Open a repository

Once the repository has been cloned, it is necessary to open it in order to start working.

NOTE: The repository could have been initialized instead of cloned but this scenario is unlikely as a link to the remote repository should then be made. As the approach outlined at the beginning is generally used, initialization of repositories is beyond the scope of this guide.

1. Open Window

In the File menu, select the option "Open Repo".

1 (Alternative 1). Open Window

In addition, the icon in the upper left corner can be used.

1 (Alternative 2). Open Window

If no repository is opened, you can use the start screen button.

2. Select the repository

"Open a Repository" is selected.

3. Select the folder

Navigate to the folder where the repository is located, an easy way to identify it is that you should see a folder called ".git" (This folder may not appear if you do not have Windows hidden files enabled).

Interface Elements

Basic interface

When opening a repository we will find the basic interface.

Commits and Branches Tree

When having several commits and branches, in the central part, it is possible to see the bifurcations and the unions, in the following image is shown a project with several branches (Only as an example).

Add Changes

Git tools can be separated into two categories:

Integrated with IDEs
Decoupled

Many IDEs come with their own integration with Git, some examples of this are Eclipse, Visual Studio and PyCharm. However, it is possible to use completely decoupled applications such as Gitkraken, Github Desktop or the git CLI itself.

The former have the advantage that the user does not have to leave the application in which he develops the code and tests, but has the disadvantage that he can easily confuse which is his current branch and if the system is in a consistent state, in addition, each IDE will have its special interface, and the IDE tend to change if the programming language is changed. On the other hand, decoupled tools such as Gitkraken allow the developer to use the IDE only for what was designed, programming, debugging and testing, abstracting completely from the version control system, this approach has the advantage that the user only has to worry about version control when starting to work and when making the Stage-Commit and that the same interface is used regardless of which IDE has been used, but brings as a disadvantage that you must have additional software installed.

When using Gitkraken it is possible to work as usual in the traditional IDE and when you want to add changes to the repository what will appear is something like the following:

The acronym WIP (Work in Progress) refers to the Working Area. By clicking on this box you can see on the right side the changes made separated by files and also a clear division between the Working Area and the Staging Area.

It is also important to clarify the function of three buttons:

Trash basket in the upper right corner: DELETEs all changes to the Working Area, use only if you know what you are doing.
Tree: Allows to visualize the files in the form of a tree of directories, depending on the configuration may not be selected by default, it is recommended to use it.
Stage Files/changes to commit: Generates a commit with the given title and description of all changes that are in the Staging Area.

If you click on any of the files in both the Working and Staging Area, you can see a detail of the changes:

Each line shaded with red implies that that line was in the last version and will be removed, the green shaded indicate that they were not in the last version and will be added. In Git there is no concept of modifying a line so if a modification is made, the line from the previous version will be deleted and an identical one will be added but with the changes made.

If one hovers the cursor over one of these lines, a button with a plus sign (+) will appear that will allow us to move individual lines to the Staging Area:

Gitkraken, in turn, identifies changes by sections in files and allows several sections to be added to the Staging Area.

Finally, if you want to add all the changes of a file to the Staging Area, you can position the cursor over that file and use the "Stage File" button that will appear. It is also possible to move ALL files from the Working Area to the Staging Area with the "Stage all Changes" button:

The process to remove changes from the Staging Area and pass them to the Working Area is identical, only the corresponding file must be selected first from the Staging Area, in this case the buttons are red and the button with the plus (+) becomes a minus (-):

Once you have moved the desired changes to the Staging Area, you can create the desired commit, you need to assign it a title, you can also detail an optional description:

Once the commit is done, it will appear in the main screen with the name assigned to it:

It is important to note that this generates the commit in the LOCAL repository and that these changes have not yet impacted the remote repository, Gitkraken shows us this by using a computer for the Local repository and the Github user logo to represent the remote repository. In the next section, we will see how to synchronize both repositories.

Synchronization with remote repositories

As mentioned before, to keep two repositories up to date you need to use the Pull and Push actions

Download changes from the remote repository

It is considered a good practice to always pull before uploading changes, this will bring any changes that we could not have in our local machine. In this opportunity, we will have to distinguish between the Pull action and the Fetch action.

Fetch: Check if there are changes in the remote repository and shows us what is the status of the remote with respect to the local
Pull: Makes a Fetch and applies those changes to the local repository.

Fetch is, therefore, a completely safe action, while Pull can bring changes that generate collisions with ours, although this is not frequent and is only possible in environments with multiple users, it is considered a good practice that the user resolves conflicts on his local computer and then can upload the changes. Otherwise, the remote repository could be left in an inconsistent state, halting the progress of other team members.

In the Gitkraken action bar we have the Fetch action:

To access the Pull action, choose the arrow next to Fetch:

In the case of the example repository, the Pull should be successful as there were no changes in the remote repository.

Upload changes to the remote repository

To upload changes it is necessary to first perform a Pull to resolve conflicts (if any) once resolved, you can perform a Push using the associated button in the action bar:

Once done, the easiest way to verify that it was successful is to see the commits tree.

This time both the Github user logo and the computer are together, showing that the content is synchronized.

Example of Fetch and Pull

To illustrate what a reverse situation would look like (changes were made to the remote repository and we want to download them). First, we do Fetch and we'll see something like this:

As you can see, the branch master is "advanced" and has a commit called "Update README.md" that is only in the remote repository (evidenced by the Github account logo). And our local version is "behind". In this situation we have to evaluate the changes and if there are any conflicts (including the Working Area files). It is recommended not to have anything in the Working Area at the time of Pulling, that is, that all changes are already inside commits, this simplifies the process and decreases the probability of finding conflicts.

NOTE: There are advanced situations where you can temporarily save Working Area changes in a Stash but this feature is outside the scope of this guide.

In case there are no conflicts, a Pull can be performed without problems:

The result is as expected, both the local and remote repositories are synchronized.

Working in multiple branches

Once you can work efficiently on one branch, it is highly recommended to select one Git Strategy and use multiple branches.

Create Branch

To create a branch we must first position ourselves in a base branch, this is the branch from which the new branch will bifurcate.

NOTE: There is the concept of orphan branches, which are branches that are not derived from any other branch. However, their use is usually very particular and specific and they are therefore beyond the scope of this guide.

In this case, we will create a master branch, add two commits, join it and then delete it.

In the first place, we position ourselves in master, double-clicking in "master".

Then use the Branch button on the action bar.

Then write the name of the branch and press enter:

Gitkraken allows us to see in several places that the branch was successfully created.

However, it is possible to observe that in the commits tree that the master branch does not appear, this is because it is hidden in the +1 icon:

When passing the cursor over it we can see that the branch is visible again:

This simplification of the interface is due to the fact that the content of both branches is identical (we haven't made any changes yet) and although it can be a little confusing for beginners, it is extremely useful when you have multiple branches. On the other hand, GitKraken's visual aid allows us to observe that the "add-license" branch only exists in the local context and does not yet exist in the remote repository.

Join Branches (Merge)

In order to join two branches it is necessary that they are different, for it some commits are added to the branch "add-license". After adding the commits, you get something like the following:

While the branch still exists only in the local context, it is important to see how it is "ahead" of the master branch. Although in the scheme presented at the beginning the branches are arranged parallel to each other, Gitkraken only arranges them that way when there are several changes, that is, only when the master branch has been modified and that this modification is not one of the commits of the branch with which it is aligned (add-license) in this case. To illustrate this example, a change will be added to the master branch directly (not in the add-license branch).

This way of visualizing branches is more similar to the previous view, the view changed because the commit "Add Contribution Guide" was added directly in the master.

Before joining the branches we are going to make a Push, in this case, as the branch does not exist in the remote repository, it will ask us if we want to link it with some branch of the already existing remote repository, if we leave the space blank and click on Submit, it will create a branch with the same name in the remote repository.

Now both branches are synchronized with the remote repository:

While it was not necessary to make a Push of the branch before joining it, it is considered good practice to make the history of changes public so other users can see who made the changes and how.

To merge branches one must click on the name of the source branch and drag and drop on the name of the destination branch, this will display the following menu, where we select the Merge option.

Once the option is chosen, the commits and branches tree will look similar to the next one:

As you can see the merge was successful, however, it will not yet be reflected in the remote repository, therefore, it is necessary to activate the master repository (by double-clicking on the name) and then do a Push.

Once the Push is done, both the branches and the Merge operation are synchronized in the remote and local repository.

Erase Branch

Throughout the life of a development project can be created infinity of branches, therefore, to keep the interface clean, it is a good practice to remove the branches after they joined (merge) to master (or to the branch that corresponds according to the Git Workflow established).

To remove a branch just right click on the name and select the option that removes it from both the local and remote repositories.

GitKraken will warn us that this is a destructive operation and can not be undone, select Delete

Once the branch has been successfully removed the commits and branches tree will look similar to the next one:

As you can see, the deleted branch is neither in the local nor in the remote repository, however, and this is where the utility is, it can be clearly seen as it once existed and consisted of two commits that were then attached to the master. The advantage of this functionality is that one can remove the branches and not lose the history of what was done, having a separation in the commits that allows you to easily track who made what changes and how they were made. This wouldn't be possible (or rather, it would be much more complex) if you only worked with the master branch.

Pull Requests

In a development environment, it is rare for developers to perform merge to master directly, which is why an approval and review mechanism is used before merge can be performed on critical branches. Each provider calls that process differently, in the most popular (Github) that process is called Pull Request.

A Pull Request is basically the following: I have changes made and would like to attach them to this branch. It is a common practice in the most critical branches within development projects, where the person in charge of those reviews and approvals is usually called Release Manager and is also extremely popular in Open Source projects, where you have the main repository and people who want to contribute, not having permissions to edit the repository directly, can make Pull Requests. The creator of the repository will then be able to accept or reject them.

Although Pull Requests are usually done through the provider's interface (Github in this case). They can also be done from GitKraken.

The procedure is shown for illustrative purposes only:

Right click on the branch from which you want to make the Pull Request
The Pull Request option is selected
The requested fields are completed

NOTE: Pull Requests must generally be made in a specific way that is detailed in the same repository of the software to which one wants to contribute to the example shown is trivial, only to have a notion of how the procedure would be.

Additional Features

Apart from everything mentioned in this guide, there is still a lot of ground to explore both in what Git allows and what can be done with GitKraken, some of these possibilities are listed below for the reader to investigate in more detail if desired

Have repositories within repositories. Search: Submodules and Subtree
Apply only one commit from one branch to another and not make a history union. Search: Cherry Pick Commit
Have branches completely independent of others: Search: Orphan Branch
Unify branches forgetting history. Search: Rebase
Add content to a previous commit. Search: Amend
Undo several commits while keeping the changes. Search: Soft Reset
Force changes in the remote repository. Search: Push Force
Revert changes of a specific commit. Search: Revert
Save changes to the Working Area in a temporary space. Search: Stash
Create tags to identify specific versions of the entire repository when it is in a consistent state (baseline). Search: Tags

Some of the above features can only be done from the console, i.e. they are not fully supported in GitKraken.

Aditional Resources

GitKraken Official Website
Git Official Website
Git Console Sandbox: It allows to experiment with several git console commands in a sandbox environment inside the browser.
Awesome Git: A curated list of amazingly awesome Git tools, resources and shiny things.

Multi-Floor Elevator Simulation with AnyLogic

2019-04-06T00:00:00-03:00

In my 4th year of University I had what would become one of my favorite courses, Simulation, where, as a final project, each student (or team of students) should choose a real problematic situation and create a simulation model (with AnyLogic). I choose something I was familiar with, the never-ending queue of the elevators of the University building, I asked for the blueprints and simulate several alternatives to find which was the best.

In case you never heard of what simulation is, don't worry, I will give you a quick introduction to this incredible world. For those who are already familiar with it, you can skip the introduction section or if you prefer, jump directly to the computational model

Introduction

Simulation is a way of problem-solving, it basically consists of translating a problem into a digital realm, make changes or test alternatives in this new environment and see the results. This information is later used for decision making. Saving loads of money, time and manpower.

Why is this necessary or useful?

Many processes can't be stopped or delayed, industries, for example, can just stop producing in order to test alternatives that might result in terrible outcomes! For this reason, it is safer, cheaper and much more convenient to use simulation.

What are the advantages?

Simulation Software Companies usually compare simulation with working with spreadsheets (such as Excel) in order to make decision and, although spreadsheets are really useful for some problems, when it comes to testing alternatives of the real world, a dynamic, interactive and more visual approach is preferred, especially when talking to non-technical people, who probably will be the ones making the final decision.

Another point is that Simulation Software relies (in most cases) on random number generators and thus the outcome of this software tends to be much similar to the real world. As opposed to the general analytical and "ideal" approach one takes when using spreadsheets.

How can be done?

Simulation software exists, both commercial and open-source and when a graphic user interface (GUI) is not needed, even some programming languages may be used with some additional libraries. In this article, I'll be talking about AnyLogic, a commercial software which has a free student license.

Context

I study at the "Universidad Tecnológica Nacional (UTN)" also translated as (National Technological University), it has several buildings but the biggest is the main one. There are 5 different departments, Mechanical Engineering, Chemistry Engineering, Electric Engineering, Civil Engineering, and Information Systems Engineering, each department has its own floor. The building has 5 floors and one underground level. The building is old, preserving the really high ceilings and the long stairs common in the time of construction, making the elevators a must. There are two elevators but only one goes to the fourth and fifth floor.

To better represent the distribution, the blueprints were used (with permission):

Problem

The university has approximately 1000 students (first year aside), and for two elevators in the rush hour, it is natural to think that long queues will form. In my fourth year, I had the chance to experiment and try to fix this situation, most people believe that the long queues are due to the internal logic of the elevator so I thought, what if we change that? As a result, with another two classmates, we decided to test several alternatives.

But before trying new alternatives, we had to identify how to the elevator is working now.

Right now, when the elevator is moving up, it stops in the floors requested from inside, when is moving down, it stops in the floors requested from both inside and outside (This is called Actual in our model).

This information was obtained via experimentation since there was no manual available to get the information from.

Alternatives

The alternatives we first thought was FIFO (First in, First Out), we knew it would be awful but it's a good way to determine what the worst non-cyclic is like (Cycles are really bad since it could produce infinite loops). So in order to test how good the other alternatives are we chose FIFO to represent the worst possible solution. The other alternatives are as follows:

Alternative 1 (ALT 1): The same as Actual but, when is lazy, it goes to Ground Floor.
Alternative 2 (ALT 2): The elevator stops in the floors requested from both inside and outside when moving either up or down
Alternative 3 (ALT 3): The same as ALT 2 but, when is lazy, it goes to Ground Floor.

Simplifications

Before diving into the model itself, some simplifications must be stated:

The population of interest is the students, so teachers and administration staff are not taken into account
When a student goes to the ground floor, we assume they leave the university, since there is nothing they can do there. All the offices on the ground floor are for the University staff.
The simulation only involves one of the two elevators, to get a simpler model
The simulation doesn't cover the underground floor for two reasons: 1. The chosen elevator can't go there and 2. The number of people going to that floor is very low.

Students behavior

Each student has a particular behavior and it is important to model it correctly, the following are a simplified and a detailed version of this behavior.

Simplified

The student goes into the university, enters a specific classroom and stays there for a certain amount of time and then leaves.

Detailed

The actual behavior is a bit more complex:

Every student enters the building in the Ground Floor (GF)
The students go from the GF to another floor and vice versa. It is rarely that a student travels from a floor different from GF to another floor since all the activities usually are covered in one or more classrooms on the same floor. This behavior exists but it is considered as not significant.
Students tend to tolerate longer waits when going to the GF and when going from the GF their tolerance is much lower.

All of the previous allow us to introduce a more specific problem: The GF queue. The main objective of the simulation will be reducing as much as possible both the length of the queue and the waiting time of the GF since this is the scenario where there is the lowest tolerance.

It is also necessary to mention that since the tolerance is too low, an auxiliary system should be modeled too since many of the users will "abandon" the queue and use the stairs instead.

The behavior of the students is represented in the following diagram:

Model

There are different types of simulation but for this particular situation, we choose "Agent-Based", since it is natural to think the elevators and the students as two agents interacting with each other.

When talking about Agents and how they interact, a tool called "State Machines" is used. This artifact models the possible "states" of a particular agent and "transitions" which are how the agent goes from one state to the other

State Machines are defined in the UML standard but in this case, a modified version will be used, this version is the one that AnyLogic uses. And in the software, there are 3 types of transitions:

By Delay: The transition occurs automatically after a certain delay.
By Condition: The transition occurs whenever a certain condition is true.
By Message: The transition occurs when the agent receives a specific message from an agent.

Global Conditions

In addition to the two main agents, there are global variables that affect the whole simulation such as:

Storey Height: It includes the height from the floor to the ceiling and the thickness of the ceiling.
Average Students per minute: the arrival frequency of the students to the system.
Average Mass of students (Kg): The average mass of the students, as a first approximation a triangular distribution is used to estimate the weights.
Average time to enter into the elevator (s): it is the time each person takes to enter the elevator, it is a small amount but it can accumulate when there are lots of people in the queue.

Elevator Agent

On the other hand, the elevator also has its own variables, these variables can be changed any time during the simulation but when doing an experiment, they should be set once at the beginning and kept fixed.

Capacity (Kg): Maximum capacity for the elevator, it is approximately 5 people (75Kg each) as default.
Speed (m/s): Speed at which the elevator moves
Service Policy: Determines how the Requests should be served. This is the main variable since it is where the alternatives should take effect.
Number of users inside: The number of people inside
Requests list: Collection of requests in the order they were received, used by the service policy to determine which is the next.
Actual Floor: Floor the elevator is in at the moment.
Destination Floor: Floor of the currently served Request.

The state machine of this agent is the following:

Where there are two main states:

Waiting: The elevator is stopped at a certain floor, it could be loading/unloading people, empty or full.
Operating: The Elevator starts moving until reaching the destination floor, there is a delay to open and close the doors.

Student Agent

The student agent is a bit different since there are loads of student so it has more random particularities.

The variables of this agent are:

Time to enter the elevator (s): Each person takes a different amount of time to enter the elevator.
Mass (Kg): body mass of the student, determined using a triangular distribution
Probability of using the stairs: Each user as a particular desire to use the stairs, whether it being because the queue is too long, they have already waited for too much or personal preference. This probability is inversely proportional to the number of the floor the student should go.
Speed (m/s): Each student has a particular speed, which affects their speed while using the stairs, the speed of going upstairs and downstairs are different but both proportional to the average speed of the person.
Tolerance to the waits (s): Maximum amount of time the student is willing to wait in the queue, if it is exceeded, they will abandon the queue and use the stairs.
Actual Floor: Floor the student is at the moment.
Destination Floor: Floor the student is willing to go.

The state machine of the students is the following:

Each student has 3 possible states:

Walking: When the user is neither inside the elevator or waiting in a queue
InQueue: When the user is waiting in a queue
WaitingInside: When the user is inside the elevator

The Experiment

Once everything is set up in the software, a set of experiments are run and then statistical methods are used to determine which alternative is the best (with a certain level of confidence). This post was more of the "theory" behind the model but to really understand how it works I recommend you to play with the demo available in the next section.

Among the five alternatives, it was necessary to identify which were the worsts, so both the average length and the average waiting time were calculated, each with its confidence interval. Resulting in the following:

Looking at the confidence interval of the waiting time, it is evident that FIFO and ALT2 were the worst and the other 3 options are quite similar. That's why Ranking and Selection was used to discern which was the best, increasing the sample size and the final result was the following (now the % of people leaving the queue is added):

Demo

This is the final result, the initial menu is in Spanish but the rest of the GUI is in English. In case you can't run it you can go to the screenshots section to see how it looks like and then run the model from a PC.

Also, you can embed this simulation in your website with the following snippet

<iframe src="https://elc.github.io/blog/iframes/elevator-simulation/simulation-anylogic-iframe.html"></iframe>

Conclusion

Although the results were contrary to the expected, it is worth mentioning that there are some limitations to take into consideration, which may vary the final result such as:

The university as an underground floor which wasn't modeled
The second elevator wasn't modeled
The electrical consumption of the alternatives is not considered
The speed of the elevator is considered constant

These limitations would result in a much more complex model but yield more accurate results. Also, notice that in this simulation the economical variable wasn't considered since it is tightly related to the electrical consumption.

Quote and citation

If you want to receive the source files of the model and/or quote this work, please feel free to send me an email at castanoezequielleonardo@gmail.com

Screenshots

The following are some screenshots of the running model

Main Screen

Elevator Control Panel

Student Behavior

2D View

2D View with Flow Density

Waiting Time Histogram

Number of people vs Time (Multiple graphs)

Additional Resources

AnyLogic Official Website: Here you can see the software features, case studies and examples.
AnyLogic in Three Days: Modeling and Simulation Textbook: This was the book I used to learn the basics of the software although experimentation is also required to fully understand some topics.
AnyLogic Cloud: This is the online repository of AnyLogic, where you can find several models from different topics. Free registration is required in order to see and run the models. Here is where the elevator simulation model is hosted.

Ordinary Differential Equations (ODE) with Python

2019-01-29T00:00:00-03:00

When I was at my 3rd year of University I have a complete subject about Ordinary Differential Equations and other similar topics. For that course we used Wolfram Mathematica throughout the year and I asked the teacher whether I can do it with Python, here you can see the results.

Of course in one year one solves a lot of differential equations but there were some problems that I think are really interesting and that rely on ordinary differential equations (ODE), numerical integration and reasoning. The year after, when I was in my 4th year, I had a subject called simulation where I learnt other techniques with another software and the focus was the process instead of the analytical form.

The teacher showed us several situations one can model with ODE and each has its particularities:

Economy of a Home: Piecewise Defined Derivatives
Sales of Houses and Air Conditionings: Systems of ODEs
Stock Control: Systems with oscillation (Underdamped, critically damped, overdamped and undamped)
Useful life in medicine: Start from a previous result
Predator and Prey Model: Non-linear relationships between variables

Some of them are quite well-known but the aim of this post isn't introducing these models but rather how you can solve them with Python, I will also attach the interactive widgets so you can experiment with the models and see how they behave when you change the parameters. Also, I prepared a jupyter notebook where all these models are together, you can open it online and experiment directly. Additionally if you find any of the widgets useful you can embed them in your own website, at the end of each widget the code for doing so will be provided.

This post will have some explanation of the models, and the equations, I won't focused on how to solve a ODE by hand since the focus is the computational aid one can have from open source tools.

Requirements:

Jupyter: Notebook Interface
Numpy: For array manipulation
Scipy: For numerical integration
Matplotlib: For visualization and animation

Imports and setup

Before starting with the models themselves, we need to import some libraries, the imports are the same for all models so one can simply place this snippet at the top of the script/notebook. In this snipped the style for the sliders is also included in order to have the same look and feel in all the widgets

# General Purpose
import numpy as np
from matplotlib import pyplot as plt
from scipy.integrate import odeint

# Jupyter Specifics
from IPython.display import HTML
from ipywidgets.widgets import interact, IntSlider, FloatSlider, Layout

%matplotlib inline

style = {'description_width': '150px'}
slider_layout = Layout(width='99%')

Economy of a Home

A home economy (family) has a fixed income at the beginning of some periodic interval (usually a month), there are also two kind of outcomes, fixed costs and extraordinary expenses, which are proportional to the money at the moment. Additionally, the family decides to set apart some proportion of the income just for fixed costs. This way, when the capital of the family is below this amount, the extraordinary expenses are 0.

The model has the following variables:

initial_salary [$]: the amount of money for the income
savings_ratio [%]: Proportion of the income that will be the keep just for fixed costs
fixed_costs [$]: The amount of money of fixed costs distributed per day
extraordinary_expenses [%]: Proportion of money that will be spend in extraordinary expenses.
days [days]: The time interval until the next income

Notation for the equation:

$x(t):$ Capital over time
$x(0) =$ initial_salary
$s =$ savings_ration * initial_salary
$a =$ fixed_costs
$b =$ extraordinary_expenses

With the previously defined notation, the ordinary differential equation is as follows:

$$ \frac{\mathrm{d} x}{\mathrm{d} t} = \left\{\begin{matrix} - a & x \leq s \\ - b (x - s) & x > s \end{matrix}\right. $$

Below is the resulting model in action and then the code explain part by part.

If you want to embed this widget in your blog/website, just add the following HTML:

<iframe src="https://elc.github.io/blog/iframes/ode-python/home-economy-iframe.html"></iframe>

Code

Now I will comment part by part the code used, at the end of this section, I will place the full code for easy copy/paste

Function Definition

def main(initial_salary, savings_ratio, extraordinary_expenses, fixed_costs, days):

First, we defined a function called main for easier use of interact which will produce the interactive widget. After that, all the constants are calculated, in this case saving_limit which is analogous to $s$ in the equation.

ODE Specifics

saving_limit = savings_ratio * initial_salary

def function(capital, time):
    if capital <= saving_limit:
        out_rate = 0
    else:
        out_rate = extraordinary_expenses * (capital - saving_limit)
    return -fixed_costs - out_rate

time = np.linspace(0, days, days * 10)

solution = odeint(function, initial_salary, time)

A nested function is defined (there could be better ways to do this but I find this the simplest), this function is the differential equation, it should take two parameters and return the value of $\frac{\mathrm{d} x}{\mathrm{d} t}$. The first parameter can be used as the current value of $x$ for a given $t$. For the numerical integration scipy.integrate.odeint is used and that's why this format is required. Using a nested function has the advantage that its name doesn't conflict with the outer scope of the function containing it and that it has access to all the parameters the outer function has, avoiding unnecessary indirection due to reference (If you know a better way to implement this, let me know in the comments).

Then the time variable is defined and initialized, this will be the $t$ steps. We have basically two options here:

Define the start, the end and the number of steps: For this numpy.linspace could be used
Define the start, the end and the step: For this numpy.arange could be used

Note: It is important to note that this will determined how smooth the resulting function will be. Although at first it might seem that the integration step (usually denoted with $h$) is implicitly defined here, that's not the case since, as you can see in the docs of scipy.integrate.odeint, $h$ is determined by the solver in each iteration.

Then the scipy.integrate.odeint function is called with the function defined as a first parameter, the initial conditions as the second and the window time as the third. This function returns the solution array which will be later used for the plotting.

Graphics Specifics

fig, ax = plt.subplots(figsize=(15, 10))

ax.plot((0, days), (saving_limit, saving_limit), label='Saving Limit')
ax.plot(time, solution, label='Capital(t)')

if days <= 60:
    step = 1
    rotation = "horizontal"
elif days <= 300:
    step = 5
    rotation = "vertical"
else:
    step = 10
    rotation = "vertical"

ax.set_xticklabels(np.arange(0, days + 1, step, dtype=np.int), rotation=rotation)
ax.set_xticks(np.arange(0, days + 1, step))

ax.set_yticks(np.arange(0, initial_salary * 1.1, initial_salary / 20))

ax.set_xlim([0, days])
ax.set_ylim([0, initial_salary * 1.1])
ax.set_xlabel('Days')
ax.set_ylabel('Capital $')
ax.legend(loc='best')
ax.grid()

plt.show()

Next, the graphics part, here matplotlib is used and this snipped is specific to this library but I will explain it in order to be self-contained (although I might produce a matplotlib tutorial in the future).

Note: I know there are plenty of plotting libraries, and many even more elegant and powerful than matplotlib but for this use case, where the complexity lies in the differential equation and not in the graphic itself, I believe it is more than enough. For complex graphics such as statistics or big data, I would recommend another library such as bqplot.

First we define a figure and an axes object, explicitly initializing the figure size through figsize. Then a horizontal line with the label Saving Limit is plot at the heigh of the saving_limit and after that the solution is plot with the label Capital(t).

The if-elif-else structure defines how many sticks there will be in the x-axis and if they should be place horizontal or vertically. Then the said sticks are plotted, the same occurs with the yticks. Then the limits for each axis and the labels for each axis are set and finally, the legend position is defined and the grid is added.

Then when plt.show() is called the figure is generated and showed in the screen.

Interaction Specifics

interact(main, initial_salary=IntSlider(min=0, max=25000, step=500, value=15000, description='Initial Salary', style=style, layout=slider_layout),
               savings_ration=FloatSlider(min=0, max=1, step=0.01, value=0.2, description='Savings Ratio', style=style, layout=slider_layout),
               extraordinary_expenses=FloatSlider(min=0, max=1, step=0.005, description='Extraordinary Expenses', style=style, value=0.3, layout=slider_layout),
               fixed_costs=IntSlider(min=1, max=1000, step=1, value=100, description='Fixed Costs', style=style, layout=slider_layout),
               days=IntSlider(min=1, max=600, step=5, value=30, description='Total Number of Days', style=style, layout=slider_layout)
        );

Finally, the code for the interaction, here the interact function from ipywidgets is used. A separate slider is used for each parameter with the default parameters set for a nice default visualization, with a description and the style and layout is used from the initialization cells.

Full code

def main(initial_salary, savings_ratio, extraordinary_expenses, fixed_costs, days):
    saving_limit = savings_ratio * initial_salary

    def function(capital, time):
        if capital <= saving_limit:
            out_rate = 0
        else:
            out_rate = extraordinary_expenses * (capital - saving_limit)
        return -fixed_costs - out_rate

    time = np.linspace(0, days, days * 10)

    solution = odeint(function, initial_salary, time)

    #Graphic details
    fig, ax = plt.subplots(figsize=(15, 10))

    ax.plot((0, days), (saving_limit, saving_limit), label='Saving Limit')
    ax.plot(time, solution, label='Capital(t)')

    if days <= 60:
        step = 1
        rotation = "horizontal"
    elif days <= 300:
        step = 5
        rotation = "vertical"
    else:
        step = 10
        rotation = "vertical"

    ax.set_xticklabels(np.arange(0, days + 1, step, dtype=np.int), rotation=rotation)
    ax.set_xticks(np.arange(0, days + 1, step))

    ax.set_yticks(np.arange(0, initial_salary * 1.1, initial_salary / 20))

    ax.set_xlim([0, days])
    ax.set_ylim([0, initial_salary * 1.1])
    ax.set_xlabel('Days')
    ax.set_ylabel('Capital $')
    ax.legend(loc='best')
    ax.grid()

    plt.show()

interact(main, initial_salary=IntSlider(min=0, max=25000, step=500, value=15000, description='Initial Salary', style=style, layout=slider_layout),
               savings_ratio=FloatSlider(min=0, max=1, step=0.01, value=0.2, description='Savings Ratio', style=style, layout=slider_layout),
               extraordinary_expenses=FloatSlider(min=0, max=1, step=0.005, description='Extraordinary Expenses', style=style, value=0.3, layout=slider_layout),
               fixed_costs=IntSlider(min=1, max=1000, step=1, value=100, description='Fixed Costs', style=style, layout=slider_layout),
               days=IntSlider(min=1, max=600, step=5, value=30, description='Total Number of Days', style=style, layout=slider_layout)
        );

Sales of Houses and Air Conditionings

In a city there are houses and air conditionings (AC) for sale and it's known that these are complementary goods, which means that if the sales of one increase the sales of the other will also increase. The number of houses sales is proportional to the number of houses that weren't sold yet and the number of AC sales is proportional to the houses sold that doesn't yet have an AC.

The model has the following variables:

initial_houses [houses]: Number of Houses initially sold, could be 0.
initial_ac [AC]: Number of air conditionings sold, could be 0 and should be less than initial_houses.
avg_time_house [days]: Average number of days to sell a house.
avg_time_ac [days]: Average number of days to sell an AC.
total_houses [houses]: Total number of Houses for sale.
days [days]: The time interval until the next income

Notation for the equation:

$x(t):$ Number of sold Houses
$y(t):$ Number of sold Air Conditionings
$x(0) =$ initial_houses
$y(0) =$ initial_ac
$h =$ total_houses
$a =$ avg_time_house
$b =$ avg_time_ac

With the previously defined notation, the ordinary differential equation is as follows:

$$ \left\{\begin{matrix} \frac{\mathrm{d} x}{\mathrm{d} t} =\frac{1}{a} (h - x) \\\\ \frac{\mathrm{d} y}{\mathrm{d} t} =\frac{1}{b} (x - y) \end{matrix}\right. $$

Note: The last scenario was a first-order differential equation and in this case it a system of two first-order differential equations, the package we are using, scipy.integrate.odeint can only integrate first-order differential equations but this doesn't limit the number of problems one can solve with it since any ODE of order greater than one can be [and usually is] rewritten as system of first order ODEs ¹

Widget

Embed this widget in your website:

<iframe src="https://elc.github.io/blog/iframes/ode-python/houses-air_conditioning-iframe.html"></iframe>

Code

In this scenario and the followings the code will be explain where it differs from the first example to avoid unnecesary repetition

ODE Specifics

def function(s, time):
    x, y = s
    dydt = [
        (1 / avg_time_house) * (total_houses - x),    # dx/dt: Change in the House sales
        (1 / avg_time_ac)    * (x - y)                # dx/dt: Change in the AC sales
    ]
    return dydt

time = np.linspace(0, days, days * 10)
initial_conditions = [initial_houses, initial_ac]
solution = odeint(function, initial_conditions, time)

In this case the constants avg_time_house and avg_time_ac are used directly inside the function since it is much clearer this way. To define a system of ODEs, first the initial condition should be a list, each element of the list represents the initial condition for each of the variables. The nested function also changes a bit, in order to work, the first parameter would also be a list so it is necessary to unpack it into the variables and then a variable dydt is used (the dydt name doesn't refer to the variable y but rather is the convention in scipy) to represent the system where each element is the left side of each equation (in canonical form).

Although it may differ from PEP8 (Python style for coding), I prefer to split the elements one per line to improve readability. Finally the dydt variable is returned. The definition of time is the same as previously seen and the solution variable is now a nested array containing the solution for both variables. This is specially useful when plotting.

Graphics Specifics

ax.plot(time, solution[:, 0], label='Houses(t)')
ax.plot(time, solution[:, 1], label='Air Conditionings(t)')

These are the only lines that differ in which instead of plotting the solution variable, the numpy indexing and slicing syntax is used to easily split the results from each of the variables.

Full code

def main(initial_houses, initial_ac, avg_time_house, avg_time_ac, total_houses, days):

    def function(s, time):
        x, y = s
        dydt = [
            (1 / avg_time_house) * (total_houses - x), # dx/dt: Change in the House sales
            (1 / avg_time_ac)    * (x - y)             # dx/dt: Change in the AC sales
        ]
        return dydt

    time = np.linspace(0, days, days * 10)
    initial_conditions = [initial_houses, initial_ac]
    solution = odeint(function, initial_conditions, time)

    #Graphic details
    fig, ax = plt.subplots(figsize=(15, 10))

    ax.plot(time, solution[:, 0], label='Houses(t)')
    ax.plot(time, solution[:, 1], label='Air Conditionings(t)')
    ax.plot((0, days), (total_houses, total_houses), label='Total Houses')

    if days <= 60:
        step = 1
        rotation = "horizontal"
    elif days <= 300:
        step = 5
        rotation = "vertical"
    else:
        step = 10
        rotation = "vertical"

    ax.set_xticklabels(np.arange(0, days + 1, step, dtype=np.int), rotation=rotation)
    ax.set_xticks(np.arange(0, days + 1, step))

    ax.set_yticks(np.arange(0, total_houses * 1.1, total_houses / 20))

    ax.set_xlim([0, days])
    ax.set_ylim([0, total_houses * 1.1])
    ax.set_xlabel('Months')
    ax.set_ylabel('Units')
    ax.legend(loc='best')
    ax.grid()

    plt.show()

interact(main, initial_houses=IntSlider(min=0, max=2000, step=10, value=0, description='Initial sold Houses', style=style, layout=slider_layout),
               initial_ac=IntSlider(min=0, max=2000, step=10, value=0, description='Initial sold AC', style=style, layout=slider_layout),
               total_houses=IntSlider(min=1, max=2000, step=100, value=1000, description='Total Houses', style=style, layout=slider_layout),
               avg_time_house=FloatSlider(min=0.1, max=24, step=0.1, value=2, description='Time for House', style=style, layout=slider_layout),
               avg_time_ac=FloatSlider(min=0.1, max=24, step=0.1, value=4, description='Time for AC', style=style, layout=slider_layout),
               days=IntSlider(min=1, max=360, step=10, value=30, description='Total Number of Days', style=style, layout=slider_layout),
        );

Stock Control

A company wants a desired stock, they started with an initial stock and they work with a provider which sends deliveries goods through sales orders. The company asks the provider each day a sales order proportional to the missing or surplus quantity to the desired stock and the market demand. The provider deliver the goods according to the sales order with a constant delay. Finally, the market demand is consider constant. The company wants to know how their stock and their request to the provider will behave and they will try to avoid alternating behavior (oscillation in stock).

The model has the following variables:

desired_stock [goods]: Amount of stock the company wants to have.
initial_stock [goods]: The initial amount of stock of the company.
initial_orders [goods]: initial amount of goods asked in sales orders.
stock_control [%]: Proportion of the missing/surplus stock to ask to the provider.
market_demand [goods]: The daily demand of the market
provider_delay [days]: The number of days of the provider delay.
days [days]: The total number of days the model will analyse.

Notation for the equation:

$x(t):$ Number of goods in stock
$y(t):$ Number of pending sales order sent to the provider
$x(0) =$ initial_stock
$y(0) =$ initial_orders
$s =$ desired_stock
$d =$ market_demand
$k =$ stock_control
$p =$ provider_delay

With the previously defined notation, the ordinary differential equation is as follows:

$$ \left\{\begin{matrix} \frac{\mathrm{d} x}{\mathrm{d} t} =& \frac{1}{p} y - d \\\\ \frac{\mathrm{d} y}{\mathrm{d} t} =& - \frac{1}{p} y + d + k ( s - x ) \end{matrix}\right. $$

Widget

Embed this widget in your website:

<iframe src="https://elc.github.io/blog/iframes/ode-python/stock-control-iframe.html"></iframe>

Code

ODE Specifics

def function(v0, time):
    x, y = v0
    dydt = [
        (1 / provider_delay) * y - market_demand,                                        # dx/dt: Change in Stock
      - (1 / provider_delay) * y + market_demand + stock_control * (desired_stock - x)   # dy/dt: Change in Requests
    ]
    return dydt

This time the code is only changed in the variables names and the equations but it is very similar to the previous example, the particularity of this model doesn't lie in the code but rather in the behavior, since it can show the differences between an oscillation behavior and a damped one. Going through all kinds of possibilities such as underdamped, overdamped, critically damped and undamped. I encourage you to experiment with the parameters in the widget.

Full code

def main(desired_stock, initial_stock, initial_orders, stock_control, market_demand, provider_delay, days):

    def function(v0, time):
        x, y = v0
        dydt = [
            (1 / provider_delay) * y - market_demand,                                        # dx/dt: Change in Stock
          - (1 / provider_delay) * y + market_demand + stock_control * (desired_stock - x)   # dy/dt: Change in Requests
        ]
        return dydt

    time = np.linspace(0, days, days * 10)
    initial_conditions = [initial_stock, initial_orders]
    solution = odeint(function, initial_conditions, time)

    #Graphic details
    fig, ax = plt.subplots(figsize=(15, 10))

    ax.plot(time, solution[:, 0], label='Stock(t)')
    ax.plot(time, solution[:, 1], label='Requests(t)')
    ax.plot((0, days), (desired_stock, desired_stock), label='Desired Stock')

    if days <= 60:
        step = 1
        rotation = "horizontal"
    elif days <= 300:
        step = 5
        rotation = "vertical"
    else:
        step = 10
        rotation = "vertical"

    ax.set_xticklabels(np.arange(0, days + 1, step, dtype=np.int), rotation=rotation)
    ax.set_xticks(np.arange(0, days + 1, step))

    ax.set_xlim([0, days])
    ax.set_ylim([0, max(max(solution[:, 0]), max(solution[:, 1])) * 1.05])
    ax.set_xlabel('Days')
    ax.set_ylabel('Units')
    ax.legend(loc='best')
    ax.grid()

    plt.show()


interact(main, desired_stock=IntSlider(min=1, max=100, step=1, value=4, description='Desired Stock', style=style, layout=slider_layout),
               initial_stock=IntSlider(min=1, max=100, step=1, value=8, description='Initial Stock', style=style, layout=slider_layout),
               initial_orders=IntSlider(min=1, max=100, step=1, value=14, description='Initial Requests', style=style, layout=slider_layout),
               stock_control=FloatSlider(min=0, max=2, step=0.001, value=1.5, description='Stock Control', style=style, layout=slider_layout),
               market_demand=FloatSlider(min=0, max=24, step=0.01, value=3, description='Market Demand', style=style, layout=slider_layout),
               provider_delay=FloatSlider(min=0, max=10, step=0.1, value=4, description='Provider Delay', style=style, layout=slider_layout),
               days=IntSlider(min=1, max=360, step=10, value=50, description='Total Number of Days', style=style, layout=slider_layout),
        );

Useful Life

A patient takes a certain drug with a recipe, the recipe says how many milligrams the patient should have per intake, how often the intakes should be and how many. Additionally the drug itself provides information about its useful life which helps to determine the optimal interval between intakes. It's also known that the useful life of the drug follows a natural decay.

The model has the following variables:

useful_life [hs]: Duration of the useful life of the drug
intake_mg [mg]: Amount of milligrams per intake.
intake_interval [hs]: Time interval between intakes.
intake_number [1]: Number of intakes.
hours [hours]: The total number of hours the model will analyse.

Notation for the equation:

$x(t):$ Milligrams of drug in the body
$x(0) =$ intake_mg
$a =$ useful_life

With the previously defined notation, the ordinary differential equation is as follows:

$$ \frac{\mathrm{d} x}{\mathrm{d} t} = - \frac{1}{a} ln(2) y $$

Note: the $ln(2)$ is part of the natural decay formula.

Widget

<iframe src="https://elc.github.io/blog/iframes/ode-python/useful-life-iframe.html"></iframe>

Code

ODE Specifics

def function(y, t):
    return - (np.log(2) / useful_life) * y # dy/dt: Change of mg

intake_hours = [intake_interval * i for i in range(intake_number - 1)]
initial_condition = intake_mg
times = []
solutions = []

for intake_time in intake_hours:
    time = np.arange(intake_time, intake_time + intake_interval, 0.1)
    solution = odeint(function, initial_condition, time)

    initial_condition = solution[-1] + intake_mg

    times.extend(time)
    solutions.extend(solution)

intake_time = intake_hours[-1] + intake_interval
time = np.arange(intake_time, intake_time + 10 * intake_interval, 0.1)
solution = odeint(function, initial_condition, time)
times.extend(time)
solutions.extend(solution)

This model doesn't introduces complexity in the ODE but rather in the way one can model the successive intakes. It is usually helpful to keep this 'iteration logic' independent from the ODE itself, that's why no particular changes are needed in the equations.

There are many ways for achieving this but the one I used is the following:

Create a list with all the intake hours
Create an empty list, times, for the times
Create an empty list, solutions for the solutions
Iterate through that list
1. Create a time array starting in the time iterated and ended in the time iterated + the intake_interval
2. Solve the ODE with the initial_condition and the previously created time
3. Update the value of the initial_conditions with the final solution
4. Append the solution list to the end of the solutions list
5. Append the time list to the end of the times list
Create a time array starting from the last the last time of intakes to 10 times the intake_interval
Solve the ODE again with a longer interval to see how it behaves after the iterations
Append the solution list to the end of the solutions list
Append the time list to the end of the times list
Proceed to Plotting

This approach consist in solving several ODEs, each of which receives the final solution of the previous as its initial conditions.

Full code

def main(useful_life, intake_mg, intake_interval, intake_number, hours):

    def function(y, t):
        return - (np.log(2) / useful_life) * y # dy/dt: Change of mg

    intake_hours = [intake_interval * i for i in range(intake_number - 1)]
    initial_condition = intake_mg
    times = []
    solutions = []

    for intake_time in intake_hours:
        time = np.arange(intake_time, intake_time + intake_interval, 0.1)
        solution = odeint(function, initial_condition, time)

        initial_condition = solution[-1] + intake_mg

        times.extend(time)
        solutions.extend(solution)


    intake_time = intake_hours[-1] + intake_interval
    time = np.arange(intake_time, intake_time + 10 * intake_interval, 0.1)
    solution = odeint(function, initial_condition, time)
    times.extend(time)
    solutions.extend(solution)

    #Graphic details
    fig, ax = plt.subplots(figsize=(15, 10))

    plt.plot(times, solutions, label='Concentration in the Body(t)')

    if hours <= 60:
        step = 1
        rotation = "horizontal"
    elif hours <= 300:
        step = 5
        rotation = "vertical"
    else:
        step = 10
        rotation = "vertical"

    ax.set_xticklabels(np.arange(0, hours + 1, step, dtype=np.int), rotation=rotation)
    ax.set_xticks(np.arange(0, hours + 1, step))

    ax.set_xlim([0, hours])
    ax.set_ylim([0, max(solutions) * 1.05])
    ax.set_xlabel('Hours')
    ax.set_ylabel('Concentration')
    ax.legend(loc='best')
    ax.grid()

    plt.show()

interact(main, useful_life=FloatSlider(min=0, max=24, step=0.01, value=3.8, description='Useful Life (hs)', style=style, layout=slider_layout),
               intake_mg=FloatSlider(min=0, max=1, step=0.001, value=0.01, description='Milligrams per Intake', style=style, layout=slider_layout),
               intake_interval=FloatSlider(min=0, max=24, step=0.1, value=6, description='Hours between Intakes', style=style, layout=slider_layout),
               intake_number=IntSlider(min=1, max=20, step=1, value=4, description='Number of Intakes', style=style, layout=slider_layout),
               hours=FloatSlider(min=1, max=240, step=0.5, value=40, description='Total number of Hours', style=style, layout=slider_layout),
        );

Foxes and Rabbits

This is the model known as predator–prey or Lotka–Volterra equations. There are two species, a predator and a prey, like foxes and rabbits. Each has a certain birthrate and a deathrate. The birthrate of the prey is proportional to its current population and the birthrate of the predator is proportional to both its population and the prey population. The deathrate of the prey is proportional to both its population and the predator population and the deathrate of the predator is proportional to its population.

The model has the following variables:

rabbits_birthrate [1]: Duration of the useful life of the drug
rabbits_deathrate [1]: Amout of miligrams per intake.
foxes_birthrate [1]: Time interval between intakes.
foxes_deathrate [1]: Number of intakes.
initial_rabbits [rabbits]: Number of intakes.
initial_foxes [foxes]: Number of intakes.
days [days]: The total number of days the model will analyse.

Notation for the equation:

$x(t):$ Population of Rabbits over time
$x(0) =$ initial_rabbits
$y(t):$ Population of Foxes over time
$y(0) =$ initial_foxes
$a =$ rabbits_birthrate
$b =$ rabbits_deathrate
$c =$ foxes_birthrate
$d =$ foxes_deathrate

With the previously defined notation, the ordinary differential equation is as follows:

$$ \left\{\begin{matrix} \frac{\mathrm{d} x}{\mathrm{d} t} =& a x &-& bxy\\\\ \frac{\mathrm{d} y}{\mathrm{d} t} =& cxy &-& dy \end{matrix}\right. $$

Or more concisely:

$$ \left\{\begin{matrix} \frac{\mathrm{d} x}{\mathrm{d} t} =& x (a - by)\\\\ \frac{\mathrm{d} y}{\mathrm{d} t} =& y (cx - d) \end{matrix}\right. $$

Widget

<iframe src="https://elc.github.io/blog/iframes/ode-python/foxes-rabbits-iframe.html"></iframe>

Code

ODE Specifics

def function(s, t):
    x, y = s
    dydt = [
        rabbits_birthrate * x     - rabbits_deathrate * x * y, # dx/dy: Change in Rabbits
        foxes_birthrate   * x * y - foxes_deathrate   * y      # dy/dt: Change in Foxes
    ]

    return dydt

In this case the equations are similar to the houses and air conditioning example but the main difference is that the equations are non-linear since they include $xy$, this implies that finding an analytical solution will be much harder but for the numerical integration the problem isn't any more difficult.

Graphics Specifics

fig, axes = plt.subplots(1, 2, figsize=(15, 10))

ax = axes[0]

ax.plot(time, solution[:, 0], label='Rabbits(t)')
ax.plot(time, solution[:, 1], label='foxes(t)')


::python
ax = axes[1]

ax.plot(solution[:, 0], solution[:, 1], label='Foxes vs Rabbits')

This scenario also introduces some variations in the plotting since, it is quite useful to have a parametric plot of the variables, where, instead of using the time as independent variables the two variables are plot. This is simply achieved with ax.plot(solution[:, 0], solution[:, 1], label='Foxes vs Rabbits') also, this model uses subplots and that's why the axes list is used to place the plots one next to the other. This parametric plot also reveals an interesting fact, since the parametric plot is a closed curve, it implies that the functions are periodic which really reflect the natural behavior of predators and preys.

Full code

def main(rabbits_birthrate, rabbits_deathrate, foxes_birthrate, foxes_deathrate, initial_rabbits, initial_foxes, days):

    def function(s, t):
        x, y = s
        dydt = [
            rabbits_birthrate * x     - rabbits_deathrate * x * y, # dx/dy: Change in Rabbits
            foxes_birthrate   * x * y - foxes_deathrate   * y      # dy/dt: Change in Foxes
        ]

        return dydt

    time = np.arange(0, days, 0.01)
    initial_conditions = [initial_rabbits, initial_foxes]
    solution = odeint(function, initial_conditions, time)

    #Graphic details
    fig, axes = plt.subplots(1, 2, figsize=(15, 10))

    ax = axes[0]

    ax.plot(time, solution[:, 0], label='Rabbits(t)')
    ax.plot(time, solution[:, 1], label='Foxes(t)')

    if days <= 30:
        step = 1
        rotation = "horizontal"
    elif days <= 150:
        step = 5
        rotation = "vertical"
    else:
        step = 10
        rotation = "vertical"

    ax.set_xticklabels(np.arange(0, days + 1, step, dtype=np.int), rotation=rotation)
    ax.set_xticks(np.arange(0, days + 1, step))

    ax.set_xlim([0, days])
    ax.set_ylim([0, max(max(solution[:, 0]), max(solution[:, 1])) * 1.05])
    ax.set_xlabel('Time')
    ax.set_ylabel('Population')
    ax.legend(loc='best')
    ax.grid()


    ax = axes[1]

    ax.plot(solution[:, 0], solution[:, 1], label='Foxes vs Rabbits')

    ax.set_xlim([0, max(solution[:, 0]) * 1.05])
    ax.set_ylim([0, max(solution[:, 1]) * 1.05])
    ax.set_xlabel('Rabbits')
    ax.set_ylabel('Foxes')
    ax.legend(loc='best')
    ax.grid()

    plt.tight_layout()
    plt.show()

interact(main, rabbits_birthrate=FloatSlider(min=0, max=24, step=0.01, value=1, description='Birth Rate of Rabbits', style=style, layout=slider_layout),
               rabbits_deathrate=FloatSlider(min=0, max=24, step=0.01, value=1, description='Death Rate of Rabbits', style=style, layout=slider_layout),
               foxes_birthrate=FloatSlider(min=0, max=24, step=0.01, value=1, description='Birth Rate of Foxes', style=style, layout=slider_layout),
               foxes_deathrate=FloatSlider(min=0, max=24, step=0.01, value=1, description='Death Rate of Foxes', style=style, layout=slider_layout),
               initial_rabbits=FloatSlider(min=0 , max=100, step=1, value=2, description='Initial Rabbits', style=style, layout=slider_layout),
               initial_foxes=FloatSlider(min=0 , max=100, step=1, value=1, description='Initial Foxes', style=style, layout=slider_layout),
               days=FloatSlider(min=0 ,max=365 , step=10, value=15, description='Total number of Days', style=style, layout=slider_layout),
        );

Uri M. Ascher; Linda R. Petzold (1998). Computer Methods for Ordinary Differential Equations and Differential-Algebraic Equations. SIAM. p. 5. ↩

Visualizing Math: A Guide to Creating Times Table Animations with Python

2019-01-22T00:00:00-03:00

Explore the mesmerizing patterns of times tables with Python! In this blog post, we present an alternative implementation to the famous Mathologer's video where beautiful patterns emerge from times tables. Burkard highlights how stunning patterns arise from these tables and utilizes Wolfram Mathematica to demonstrate these patterns in greater detail. This blog post aims to showcase a similar implementation using Python.

For those unfamiliar with the concept, it is recommended to view the video that served as inspiration for this post. It provides a comprehensive explanation of what Times Tables are.

This post provides an alternative approach to generate similar animations to the ones seen in the video, using Python and its libraries such as Matplotlib and IPython.

The following animations were created using Python and its supporting libraries.

Bringing this captivating video to life with Python is made easy with the help of a Jupyter Notebook. Explore the examples and discover the beauty of Times Tables for yourself by following along. With all the necessary dependencies already installed, the Jupyter Notebook is the perfect platform for experimentation and hands-on learning

In this post, multiple examples are provided, including:

A static version, as shown above An interactive, parametric version where you can experiment by adjusting values with sliders
An animated construction, line by line, where the factor and number of points remain fixed, but each line is plotted one at a time
An animated construction, point by point, where the factor and lines are fixed, but the number of points increases with each frame
An animated construction, factor by factor, where the lines and number of points remain fixed, but the factor increases, first in monochrome and then with a rainbow effect.n with the rainbow effect seen in the video.

Note: It's important to note that after exploring each scenario, both an interactive and a static representation of the code will be provided. The interactive version is built using Jupyter widgets, allowing you to adjust parameters and see the results in real-time. However, this also means that each time the sliders are moved, the full animation needs to be re-calculated, so some patience is required. On the other hand, the static representation is optimized for faster display and can be useful for low-speed connections. It should be noted that the static representation cannot be adjusted or experimented with.

Requirements:

Jupyter: Notebook Interface
Numpy: For array manipulation
Matplotlib: For visualization and animation
ffmpeg (Optional): For exporting the animation

In order to produce these images and animations, the code should be split in 4 parts:

Import all the necesary libraris
Define all the auxiliary functions
Plot the animations
Export (optional)

Initialization

It is recommended to place all imports at the beginning of the document for better organization and maintenance. This also helps to clearly identify the required tools. In this instance, there are both general purpose imports and those specific to Jupyter. To utilize the code outside of Jupyter, it may need to be modified to remove Jupyter-specific components. This ensures the code can run as a standalone script.

# General Purpose
import colorsys
from typing import Optional, List

from matplotlib import animation, rc
from matplotlib import pyplot as plt
from matplotlib.collections import LineCollection
from matplotlib.lines import Line2D
import numpy as np
import numpy.typing as npt

# Jupyter Specifics
from ipywidgets.widgets import interact, IntSlider, FloatSlider, Layout

%matplotlib inline
rc('animation', html='html5')

Basic Functions

With all the necessary imports in place, it's time to define a few functions that will bring this project to life. These functions include:

A function to calculate the points around a circle
A function to generate each of the lines
A function to plot the labels and points on the circle
A function to plot the lines on the circle

The first function, named points_around_circle, uses polar coordinates to determine a specified number of points around a circle of radius 1. The use of numpy in this calculation enhances its performance.

def points_around_circle(number: int = 100) -> npt.NDArray[np.float64]:
    theta = np.linspace(0, 2 * np.pi, number, endpoint=False)
    xs, ys = np.cos(theta), np.sin(theta)
    return np.stack((xs, ys))

The second function pertains to the generation of lines. Given a list of points, this function generates a new line.

def generate_lines_from_points(
    points: npt.NDArray[np.float64], factor: float
) -> npt.NDArray[np.float64]:
    _, size = points.shape
    index = (np.arange(size) * factor) % size

    line_starts = points.T
    line_ends = line_starts[index.astype(int)]

    return np.stack((line_ends, line_starts), axis=1)

However, the line in numpy format is not directly plotable. To translate the numpy array into a data structure compatible with matplotlib, a LineCollection is used. The function generate_line_collection not only generates the line collection, but also specifies the color based on the HSV format. This capability will come in handy when creating the final animation.

def generate_line_collection(
    points: npt.NDArray[np.float64], factor: float, color: Optional[float] = None
) -> LineCollection:
    lines = generate_lines_from_points(points, factor)
    color_ = colorsys.hsv_to_rgb(color, 1.0, 0.8) if color else None
    return LineCollection(lines, color=color_)

Static Version

With all the required functions defined, plotting a static version of the data is straightforward. Simply generate the axis object and call the functions in the appropriate order, and the image will be generated. This approach is useful for quickly experimenting with a fixed number of factors and points.

def plot_static(factor: float, number_of_points: int) -> None:
    points = points_around_circle(number=number_of_points)
    lines = generate_line_collection(points, factor)

    plt.figure(figsize=(10, 10))
    ax = plt.gca()
    ax.axis("off")
    ax.annotate(f"Points: {number_of_points}", (0.8, 0.9))
    ax.annotate(f"Factor: {factor}", (0.8, 1))
    ax.plot(*points, "-ko", markevery=1)
    ax.add_collection(lines)

factor = 2
points = 100
plot_static(factor, points)

Parametric Version

One method to update the factor and points variables is to manually change them and re-execute the cell or function. However, Jupyter offers support for interaction, allowing for a more user-friendly approach via Sliders, a built-in user interface of IPython. The function plot_parametric serves the same purpose as plot_static, with the addition of a call to plt.show() at the end to display the image. While the image is still static, the user can adjust the variables by moving the sliders.

def plot_parametric(factor: float = 2, points: int = 100) -> None:
    points = points_around_circle(number=points)
    lines = generate_line_collection(points, factor)

    plt.figure(figsize=(10, 10))
    ax = plt.gca()
    ax.axis("off")
    ax.plot(*points, "-ko", markevery=1)
    ax.add_collection(lines)
    plt.show()


factors = [21, 29, 33, 34, 49, 51, 66, 67, 73, 76, 79, 80, 86, 91, 99]
print("Try these Factors with different number of points:", *factors)

interact(
    plot_parametric,
    factor=FloatSlider(min=0, max=100, step=0.1, value=2, layout=Layout(width="99%")),
    points=IntSlider(min=0, max=300, step=25, value=100, layout=Layout(width="99%")),
)

Animate Construction Line by Line

In the next step, the focus shifts to animations. In this particular animation, the factor and the number of points remain constant while the lines change. This animation replicates the act of drawing these times tables by hand and provides a deeper understanding of the sequence in which the lines are plotted, instead of simply presenting them all together.

In Matplotlib, animations are constructed using the animate function, which returns the objects to be displayed in each frame. As a result, two functions need to be defined in this and the following animations. The first is for the animate API of Matplotlib and the second is to be integrated into the interact function of IPython. In this particular case, the function line_by_line takes a given factor, a specified number of max_points, and an interval. The first two parameters are already familiar from previous functions, while interval represents the delay between frames in milliseconds. This delay is directly related to the animation's frames per second (FPS), which is calculated as FPS = 1000 / delay.

def animate_line_by_line(
    i: int, lines: npt.NDArray[np.float64], ax: plt.Axes
) -> List[Line2D]:
    start_point, end_point = lines[i].T
    line = Line2D(start_point, end_point)
    ax.add_line(line)
    return []


def line_by_line(
    factor: float, max_points: int, interval: int
) -> animation.FuncAnimation:
    points = points_around_circle(number=max_points)
    lines = generate_lines_from_points(points, factor)

    fig = plt.figure(figsize=(10, 10))
    ax = plt.gca()
    ax.axis("off")
    ax.annotate(f"Factor: {factor}", (0.8, 1))
    ax.annotate(f"Interval: {interval}", (0.8, 0.8))
    ax.annotate(f"Points: {max_points}", (0.8, 0.9))
    ax.plot(*points, "-ko", markevery=1)
    anim = animation.FuncAnimation(
        fig,
        animate_line_by_line,
        frames=max_points - 2,
        interval=interval,
        blit=True,
        fargs=(lines, ax),
    )
    plt.close()

    return anim


interact(
    line_by_line,
    factor=FloatSlider(min=0, max=100, step=0.1, value=2, layout=Layout(width="99%")),
    max_points=IntSlider(min=1, max=200, step=1, value=100, layout=Layout(width="99%")),
    interval=IntSlider(min=5, max=500, step=5, value=75, layout=Layout(width="99%")),
)

Animate Construction Point by Point

The next animation focuses on the process of how the figure becomes clearer as the number of points increases. The factor is fixed, and all lines are plotted at once, but with each frame, the number of points increases incrementally from 0 to a specified maximum number of max_points. This perspective provides a different view on the relationship between the number of points and the clarity of the figure.

def animate_point_by_point(
    i: int, ax: plt.Axes, factor: float, interval: int, max_points: int
) -> List[Line2D]:
    points = points_around_circle(number=i + 1)
    lines = generate_line_collection(points, factor)

    ax.cla()
    ax.axis("off")
    ax.set_ylim(-1.2, 1.2)
    ax.set_xlim(-1.2, 1.2)
    ax.annotate(f"Interval: {interval}", (0.8, 0.8))
    ax.annotate(f"Points: {max_points}", (0.8, 0.9))
    ax.annotate(f"Factor: {factor}", (0.8, 1))
    ax.plot(*points, "-ko", markevery=1)
    ax.add_collection(lines)
    return []


def point_by_point(
    factor: float, interval: int, max_points: int
) -> animation.FuncAnimation:
    fig = plt.figure(figsize=(10, 10))
    ax = plt.gca()
    anim = animation.FuncAnimation(
        fig,
        animate_point_by_point,
        frames=max_points,
        interval=interval,
        blit=True,
        fargs=(ax, factor, interval, max_points),
    )
    plt.close()

    return anim


interact(
    point_by_point,
    factor=FloatSlider(min=0, max=100, step=0.1, value=2, layout=Layout(width="99%")),
    max_points=IntSlider(min=1, max=200, step=1, value=75, layout=Layout(width="99%")),
    interval=IntSlider(min=100, max=500, step=1, value=200, layout=Layout(width="99%")),
)

Animate Construction Factor by Factor

The animation displays the effect of incrementing the factor, with the number of points fixed and all lines plotted at once. To enhance the visual impact, the factor is increased in increments of 0.1 instead of 1. This results in a smoother animation. This version is monochromatic, with all lines appearing in the same color.

def animate_factor_by_factor(
    i: int, ax: plt.Axes, max_points: int, interval: int
) -> List[Line2D]:
    points = points_around_circle(number=max_points)
    lines = generate_line_collection(points, i / 10)

    ax.cla()
    ax.axis("off")
    ax.set_ylim(-1.2, 1.2)
    ax.set_xlim(-1.2, 1.2)
    ax.annotate(f"Interval: {interval}", (0.8, 0.8))
    ax.annotate(f"Points: {max_points}", (0.8, 0.9))
    ax.annotate(f"Factor: {factor}", (0.8, 1))
    ax.plot(*points, "-ko", markevery=1)
    ax.add_collection(lines)
    return []


def factor_by_factor(
    factor: float, interval: int, max_points: int
) -> animation.FuncAnimation:
    fig = plt.figure(figsize=(10, 10))
    ax = plt.gca()

    frames = int(factor * 10)
    anim = animation.FuncAnimation(
        fig,
        animate_factor_by_factor,
        frames=frames,
        interval=interval,
        blit=True,
        fargs=(ax, max_points, interval),
    )

    plt.close()

    return anim


interact(
    factor_by_factor,
    factor=FloatSlider(min=0, max=100, step=0.1, value=5, layout=Layout(width="99%")),
    max_points=IntSlider(min=1, max=200, step=1, value=100, layout=Layout(width="99%")),
    interval=IntSlider(min=50, max=500, step=25, value=100, layout=Layout(width="99%")),
)

Animate Construction Factor by Factor with Color

The final animation is similar to the previous one, with the difference being the addition of color. To achieve this, the function animate_factor_by_factor_colored is passed an additional parameter, frames, which specifies the total number of frames. The HSV color system is utilized, with fixed saturation and value, while the hue changes from 0 to 1 as the current frame (i) is divided by the total number of frames (frames), resulting in a range from 0 to 1.

def animate_factor_by_factor_colored(
    i: int, ax: plt.Axes, max_points: int, interval: int, frames: int
) -> List[Line2D]:
    points = points_around_circle(number=max_points)
    lines = generate_line_collection(points, i / 10, color=i / frames)

    ax.cla()
    ax.axis("off")
    ax.annotate(f"Interval: {interval}", (0.8, 0.8))
    ax.annotate(f"Points: {max_points}", (0.8, 0.9))
    ax.annotate(f"Factor: {factor}", (0.8, 1))
    ax.plot(*points, "-ko", markevery=1)
    ax.add_collection(lines)
    return []


def factor_by_factor_colored(
    factor: float, interval: int, max_points: int
) -> animation.FuncAnimation:
    fig = plt.figure(figsize=(10, 10))
    ax = plt.gca()

    frames = int(factor * 10)
    anim = animation.FuncAnimation(
        fig,
        animate_factor_by_factor_colored,
        frames=frames,
        interval=interval,
        blit=True,
        fargs=(ax, max_points, interval, frames),
    )

    plt.close()

    return anim


interact(
    factor_by_factor_colored,
    factor=FloatSlider(min=0, max=100, step=0.1, value=5, layout=Layout(width="99%")),
    max_points=IntSlider(min=1, max=200, step=1, value=100, layout=Layout(width="99%")),
    interval=IntSlider(min=50, max=500, step=25, value=100, layout=Layout(width="99%")),
)

Export

The generated animations can be exported as mp4 files by calling the appropriate function, storing the result in a variable and using the following code snippet. The function specific_function should be replaced with the desired animation, such as factor_by_factor_colored or animate_point_by_point, and the corresponding parameters should be included.

anim = specific_function(*args)
filename = "my_animation.mp4"

Writer = animation.writers['ffmpeg']
writer = Writer(fps=30)

anim.save(filename, writer=writer)

Pascal Triangle and Genetic Algorithm - A Visualization

2019-01-21T00:00:00-03:00

Inspired by a Wikipedia article, I replicate a way to visualize the Pascal Triangle and used the same approach in Genetic Algorithms

Once I came accross this wikipedia article about Pascal Triangle, there you can find the following animation:

I found it interesting to replicate this very animation in Python, just as a challenge and maybe later I would find a useful application for it. At the end I will add a link to the jupyter notebook so you can experiment your own variants. If you want, you can jump directly to the genetic algorithm application.

First, there are some tools we need to install:

Jupyter (Optional): Notebook Interface
Numpy: For array manipulation
Matplotlib: For visualization and animation
ffmpeg (Optional): For exporting the animation

Once everything is installed, we need to understand what is necessary, I decided to split the code in 5 parts, although it can be a single file script. The final result is the following:

Initialization

First, import everything needed, this should be the first cell in the notebook or be placed at the top of the script

import matplotlib.pyplot as plt
import numpy as np
from matplotlib import animation
from IPython.display import HTML # Only required for Jupyter

%matplotlib inline # Only required for Jupyter

Pascal Triangle

Secondly, define the proper functions to create a Pascal triangle, since for every frame only the last row is represented, a separate function that returns only that row is generated, to avoid unnecessary array manipulation. Additionally since many frames could be asked, it's important to cache the intermediate results since every pascal triangle is build based on the previous one, for this purpose a technique named memoization is used and this is the reason of why pascal_triangle is a recursive function.

def memoize(f):
    memo = {}
    def helper(x):
        if x not in memo:
            memo[x] = f(x)
        return memo[x]
    return helper

@memoize
def pascal_triangle(n):
    if n == 0:
        return [np.array([1])]
    prev = pascal_triangle(n - 1)[-1]
    middle = np.add(prev[:-1], prev[1:])
    return pascal_triangle(n - 1) + [np.array([1, *middle, 1], dtype=np.ulonglong)]

@memoize
def pascal_last_row(n):
    return pascal_triangle(n)[-1]

Animation Function

Thirdly, Matplotlib asks a function to animate which will generate each frame. For this purpose each frame is generated, adding extra white space to the sides when needed and returning a blank image for the 0 frame

def animate(i, length, ax, width):
    if i == 0:
        im = ax.imshow([[0],[0]], cmap='Greys', aspect='equal', extent=(0, width - 2, 0, length), animated=True)
        return im,

    elements = pascal_last_row(i)

    extra_space = [0] * ((width - len(elements) - 1) // 2)

    elements = extra_space + list(elements) + extra_space

    real_width = len(elements)

    if real_width % 2 == 1:
        real_width -= 1

    binary_elements = []

    for j in elements:
        row = []
        for k in bin(j)[2:].zfill(length):
            row.append(int(k))

        binary_elements.append(row)

    im = ax.imshow(np.transpose(binary_elements), cmap='Greys', aspect='equal', extent=(0, real_width, 0, length), animated=True)

    return im,

Animation Video

Finally, the main part of the script is where the crucial variables are defined such as the number of frames and the width of the window.

frames = 20
last_row = pascal_last_row(frames)
length = len(bin(max(last_row))[2:]) - 1
width = len(last_row)

aspect_ratio = width / length

frame_width = 8

fig, ax = plt.subplots(figsize=(frame_width, frame_width / aspect_ratio))

fig.subplots_adjust(left=0, bottom=0.01, right=1, top=1)

ax.set_xticks([]) 
ax.set_yticks([])

if frames % 2 == 0:
    width += 1

anim = animation.FuncAnimation(fig, animate, frames=frames, interval=1000, blit=True, fargs=(length, ax, width))

plt.close() # Avoid extra picture

HTML(anim.to_html5_video()) # Only required for Jupyter, replace this line with plt.show() for scripts

Export

As a bonus, if the animation is needed as a separate file, ffmpeg can be used through Matplotlib to generate the corresponding mp4 file

Writer = animation.writers['ffmpeg']
writer = Writer(fps=1, metadata=dict(artist='Me'), bitrate=1800)

anim.save('im.mp4', writer=writer)

Final Result

As seen at the beginning the previous code resulted in the following animation:

Notebook

Everything showed above can be executed without installing anything just by using Binder, open the gist online and experiment yourself.

Application in Genetic Algorithms

When I started this visualization challenge, I had no application in mind but then I realized the same approach could be used for visualizing the genotype of a population of chromosomes in a traditional genetic algorithm. Each colum would represent a chromosome instead of a number each row a particular gene for each chromosome, being black if the gene is 1 and white if it is 0. This approach asumes a binary genotype configuration. Then another possible improvement is to sort them by fitness, the ones with the highest fitness on the left and the ones with the lowest on the right. Sorting them could help understanding patterns in the genotype that might not be obvious in the phenotype.

This is the result of finding the maximum of a function with elitism and high mutation. Each frame represents a population the next frame represents the childs of the previous one. The source code that generates this animation can be found in my personal github repository

Juegos Bíblicos para Jóvenes y Adolescentes

2018-12-28T00:00:00-03:00

Compilado de juegos bíblicos listos para imprimir. Pensados y adaptados para jóvenes y adolescentes que buscan conocer más a Dios.

EDICIÓN 2020.18.15: Ya está disponible la app web para el Tabú Bíblico!

En el pasado yo siempre tuve una mirada despectiva de los juegos, pensé que eran innecesarios e infantiles pero fue el Señor que me sorprendió dandome la habilidad para diseñarlos y encontrarme con que resultaron ser de suma utilidad, tanto en una reunión de jóvenes como de adolescentes.

En este artículo voy a dejar todos los juegos que vaya diseñando, la idea es tener un lugar centralizado donde las personas puedan referenciar y también proveer plantillas para que puedan ser adaptados si hiciera falta. Cada juego va a tener una imagen que muestra como es, las instrucciones, un enlace a una versión en PDF lista para imprimir y recortar y un enlace a una versión de Google Docs lista para descargar y adaptar.

Adicionalmente, toda crítica, sugerencia y comentario es bien recibido, pueden dejarlo expresado en los comentarios al final de este artículo.

También te puede interesar: si participás en una actividad donde se utilizan presentationes en Power Point para pasar las letras de las canciones (campamentos, reunión de jóvenes, etc.), puede interesarte esta aplicación online para generar las presentaciones automáticamente.

Tabú Bíblico

Volver al Índice

Se arman equipos y se juega por turnos, en el turno de un equipo, se designa una persona, a esta persona se le entregan una serie de tarjetas, las tarjetas van a tener una palabra principal y una serie de palabras secundarias. El objetivo es que la persona designada pueda hacer que su equipo adivine cual es la palabra principal sin utilizar las palabras secundarias, ni sus familias de palabras, tampoco puede usar mímica ni decir cuántas letras tiene la palabra principal o usar referencias fonéticas.

La persona designada puede saltear la carta si lo cree conveniente. Por cada tarjeta adivinada en el tiempo fijado, el equipo gana un punto y por cada tarjeta salteada o penalizada el equipo pierde un punto. Luego de una cierta cantidad de rondas prefijada el equipo con más puntos gana.

La tarjetas tienen distintas dificultades y deben estar mezcladas previamente, una vez que la tarjeta fue adivinada se separa, las tarjetas salteadas o penalizadas siguen en juego.

Descargar el documento editable (Ver Instrucciones)

Descargar el PDF listo para imprimir

Ahora también disponible como app web para celulares y PCs: Disponible online aquí

Trivia Adolescente

Volver al Índice

El juego consiste en responder preguntas - para ganar - se deben responder al menos una pregunta correctamente de cada categoría. Dentro de cada categoría hay preguntas de distintas dificultades.

Las categorías son:

Historia
Geografía
Arte y Entretenimiento
Ciencia
Deportes
Biblia

Descargar el documento editable (Ver Instrucciones)

Descargar el PDF listo para imprimir

Conocé la historia

Volver al Índice

A cada jugador se le coloca una tarjeta en la espalda, cada tarjeta contiene una serie de oraciones que están vinculadas a un personaje bíblico, pueden ser eventos, lugares, pasajes o cualquier otro dato relacionado.

Los jugadores deberán preguntarles a los demás que dice su tarjeta, sin embargo sólo le dirán un dato, y para darle ese dato debe mencionar correctamente y en orden todos los anteriores. Es decir, el primer dato se lo dirá sin preguntar nada, para darle el segundo debe decir el primero, para darle el tercero debe decir el primero y el segundo.

El primer jugador que se acerque al organizador y pueda decir todos los datos correctamente y en orden será el ganador.

Descargar el documento editable (Ver Instrucciones)

Descargar el PDF listo para imprimir

Espadeo Bíblico - Versión Modificada

Volver al Índice

Este es un juego con 3 modalidales, el organizador puede alternar entre ellas o escoger la que más se adapte a sus necesidades.

Modalidad clásica: Se lee una pequeña introducción y luego la cita de un versículo y los equipos deben buscarlo, el que lo encuentre debe comenzar a leerlo inmediatamente, si es el versículo correcto, gana.
Modalidad Completar el versículo: Se lee una pequeña introducción y luego un versículo en forma incompleta, los equipos tienen la posibilidad de completarlo si lo saben de memoria, se dan unos segundos para esto. En caso de que ninguno de los equipos haya respondido, se dice cuál es la cita y se procede con la modalidad clásica, el equipo que complete el versículo antes es el que gana.
Modalidad Preguntas con Opciones: Se hace una pregunta y se dan tres citas como posibles soluciones, en todos los casos hay sólo una respuesta correcta. Cada equipo debe buscar las opciones y comenzar a leer el versículo que corresponda. El primer equipo que lea el versículo correspondiente a la pregunta es el ganador. Los equipos pueden decir la cita y el versículo de memoria si lo saben.

Descargar el documento editable (Ver Instrucciones)

Descargar el PDF listo para imprimir

Memo Paralelo

Volver al Índice

Se arman equipos y a cada equipo se le da una serie de tarjetas, cada equipo coloca sus tarjetas boca abajo. Una persona de cada equipo será la encargada de dar vuelta las tarjetas, cada tarjeta contiene una cita a un pasaje bíblico o un pasaje escrito. El objetivo del juego es agrupar las tarjetas de a pares según el tema que traten.

La persona designada sólo podrá voltear dos tarjetas a la vez, si desea voltear una tercera deberá colocar boca abajo alguna de las otras dos antes. Los miembros del equipo deben ayudar buscando los pasajes e identificar cuales son los que están relacionados.

Es importante asegurarse que antes de darles las tarjetas a los equipos cada tarjeta tenga su par. El organizador tendrá una lista de todos los pares para poder validar si los equipos resolvieron el juego correctamente, el equipo que haya encontrado más pares en el tiempo fijado será el ganador.

Descargar el documento editable (Ver Instrucciones)

Descargar el PDF listo para imprimir

Armá la Historia

Volver al Índice

A cada equipo se le da una serie de tarjetas, algunas contienen pasajes y otras contienen eventos o historias. Cada equipo deberá asociar cada tarjeta de evento con una de un pasaje y luego ordenarlas para formar la historia bíblica.

Al principio a cada equipo se le da una historia fácil y cada vez que completen una historia se le da una de mayor dificultad, sólo se considerará correcta si todas las tarjetas están en la posición adecuada. El organizador deberá decir que es incorrecta sin mencionar dónde está el error. En caso de empate el punto va para el equipo que más tarjetas correctas tenga en la historia en curso.

Descargar el documento editable (Ver Instrucciones)

Descargar el PDF listo para imprimir

Instrucciones para la versión editable

Cada juego tiene se puede acceder de dos maneras, en PDF o en formato de Google Docs (Similar al Microsoft Word). Al final de la descripción de cada juego habrá dos botones para cada forma respectivamente.

La versión en PDF no es editable pero es la más fácil de imprimir (no se cambia el formato al imprimir), en caso de requerir ediciones, éstas pueden hacerse libremente en el formato de Google Docs o Microsoft Word.

Si decidiste modificar alguno de los juegos, me encantaría que pudieras compartir las modificaciones mandándome un email a castanoezequielleonardo@gmail.com.

Para utilizar la versión de Google Docs se debe iniciar primero en la cuenta de Google (en su correo de @gmail) y luego tendrá la posibilidad de "Hacer una copia" esa copia es completamente suya y allí podrá hacer todas las modificaciones necesarias.

Por otro lado, en caso de preferir el entorno de Microsoft Word, basta con hacer click en "Descargar" y luego seleccionar el formato preferido, el de Microsoft Word es el "docx".

Step by Step Fractals with Python

2018-11-04T00:00:00-03:00

Fractals are awesome, they are built with very complex pattern and they allow you to zoom in forever! In this post we will see how easily it is to plot several kinds of fractals using a tool called L-Systems and the Python Turtle module for the step to step plotting.

In this post I'm not going to dive into too many technical details but instead I'll present a little introduction, a lot of animated examples and at the end, the code to generate your own. If you want skip the theory and see the animations, jump directly to the animated examples, if you want to see the code instead, jump directly to that section. Additionally, there will be some resources for both code and math background if you want to explore at the end.

What's a Fractal?

First lets give a "non-strict" definition of what a fractal is, it is basically a geometric figure which shows the same characteristics no matter how much you zoom in.

It isn't quite correct but for those who wanted to know the exact definition here it is:

A fractal is an object or quantity that displays self-similarity, in a somewhat technical sense, on all scales. The object need not exhibit exactly the same structure at all scales, but the same "type" of structures must appear on all scales. A plot of the quantity on a log-log graph versus scale then gives a straight line, whose slope is said to be the fractal dimension. - Math World

How can we draw Fractals with Python?

Fractals are typically hard to draw, because there is a concept which is deeply tight in them, recursion. When we talk about graphics and plotting we usually talk about pixels or vectors, but there is always a limit, fractals by definition are infinitely recursive. So when we want to plot one we should stop at some point, that's why we talk about "iterations". At each iteration the fractal becomes more and more complex, but at some point it is impossible to distinguish between to successive iterations (this happens when changes occur at individual pixel levels), so it is quite reasonable to stop there, sometimes it is quite clear what the shape is and we can stop even earlier.

A two examples for this are the Quadratic Koch Island, which with 3 iterations has a clear structure and in the other hand the Dragon Curve which has a clear structure with 8 iterations. How many iterations are needed depends highly on the specific fractal we are working with.

Certainly there are lots of plotting libraries in Python, being Matplotlib the most popular but they are usually design to plot statistical data and well known plots. Matplotlib in particular has some low level constructs that allow us to build fractals but this time we will be focusing in a usually forget module in the standard library, the Turtle Module.

Turtle Module

According to Python docs: "Turtle graphics is a popular way for introducing programming to kids. It was part of the original Logo programming language developed by Wally Feurzig and Seymour Papert in 1966."

The key here is that turtle recognizes basically 3 commands:

Move Forwards
Turn Left by angle
Turn Right by angle

Note: THe standard library provided other commands but here we are going to just those 3.

Additionally we have the option to:

Disable writing
Enable writing

This characteristics seems too simple for plotting such complex graphics as fractals but we will use another tool that uses just this little set of instructions, I'm talking about L-Systems.

L-Systems

An L-System is a way of representing recursive structures (such as fractals) as a string of characters, this is done by rewriting the string over and over. Again, the formal definition is the following:

A Lindenmayer system, also known as an L-system, is a string rewriting system that can be used to generate fractals with dimension between 1 and 2. - Math World

Once we understand what an L-System is we can produce recursive structures, but before we are able to do that we need to understand what are the pieces we need. Every L-System has:

An alphabet: The set of symbols the L-System is going to use.
An axiom: The initial string for the generation.
A set of production rules: These rules tells how each symbol should be replaced in the following iteration.

Note for computer science fans: If you ever dived into Computer Science this might sound familiar, it is actually since this is very similar to the definition of a Formal Grammar, the key difference is that in each iteration, as opposed to grammars, as many rules as possible are applied instead of just one. So L-Systems are a subset of Context-Free Grammars.

Since we are going to use Turtle to plot and L-Systems to represent what we want to plot we need to create a relationship between them.

Since the only commands we have in Turtle are the mentioned above we will assign each a symbol which will represent the alphabet

F: Move Forwards
+: Turn Right
-: Turn Left

In order to make this work, each fractal should also provide an angle, which will be the angle the turtle will turn either right or left, for simplicity reasons only one angle should be provided and the L-System should be written taking that into consideration.

The axiom and the production rules will depend on the fractal only, but the fractal should be written in a way that can be represented by these only three symbols. This introduces a limitation, we will be able to produce only one-line fractals, so some such as the Cantor Set won't be able to be produced this way, this is only a simplification, since we can introduce two other commands to move forwards without writing and analogously for the backwards movement, but to keep things simple we will keep that simplification.

Now let's move to some examples!

Animated Examples

The following examples were compiled from several places publicly available on internet. I decided to port them to Python with the help of the Turtle module, center them, adding colors and provide a way to export them in vectorial format.

Because the browser execute Python via Skulpt and Colorsys isn't supported yet, the animations will be Black & White but images with colors and the corresponding code to generate them will be provided. Each animation has a code associated which you can open and fork in Repl.it

WARNING: The animations you are about to see are quite large in size, it is recommended to see them only with a good connection. The Repl snippet may not work since it uses your resources so mobile users might not be able to see it properly.

Note: Skulpt uses YOUR BROWSER to render and to make the animation so if in any case you experience delays, lags or strange behaviour it may be a problem with the browser, replaying the animation or reloading the page should fix most issues. It may not work properly on mobile.

The examples are ordered by their complexity (my own judgement though), so the best ones are at the end.

Koch-Snowflake"

axiom = "F--F--F"
rules = {"F":"F+F--F+F"}
iterations = 4 # TOP: 7
angle = 60

Quadratic-Koch-Island

axiom = "F+F+F+F"
rules = {"F":"F-F+F+FFF-F-F+F"}
iterations = 2 # TOP: 4
angle = 90

Crystal

axiom = "F+F+F+F"
rules = {"F":"FF+F++F+F"}
iterations = 3 # TOP: 6
angle = 90

Quadratic-Snowflake

axiom = "F--F"
rules = {"F":"F-F+F+F-F"}
iterations = 4 # TOP: 6
angle = 90

Box-Fractal

axiom = "F-F-F-F"
rules = {"F":"F-F+F+F-F"}
iterations = 4 # TOP: 6
angle = 90

Levy-C-Curve

axiom = "F"
rules = {"F":"+F--F+"}
iterations = 10 # TOP: 16
angle = 45

Sierpinski-Arrowhead

axiom = "YF"
rules = {"X":"YF+XF+Y", "Y":"XF-YF-X"}
iterations = 1 # TOP: 10
angle = 60

Siepinski-Sieve

axiom = "FXF--FF--FF"
rules = {"F":"FF", "X":"--FXF++FXF++FXF--"}
iterations = 7 # TOP: 8
angle = 60

Board

axiom = "F+F+F+F"
rules = {"F":"FF+F+F+F+FF"}
iterations = 3 # TOP: 5
angle = 90

Tiles

axiom = "F+F+F+F"
rules = {"F":"FF+F-F+F+FF"}
iterations = 3 # TOP: 4
angle = 90

Rings

axiom = "F+F+F+F"
rules = {"F":"FF+F+F+F+F+F-F"}
iterations = 2 # TOP: 4
angle = 90

Cross-2

axiom = "F+F+F+F"
rules = {"F":"F+F-F+F+F"}
iterations = 3 # TOP: 6
angle = 90

Pentaplexity

axiom = "F++F++F++F++F"
rules = {"F":"F++F++F+++++F-F++F"}
iterations = 1 # TOP: 5
angle = 36

32-Segment-Curve

axiom = "F+F+F+F"
rules = {"F":"-F+F-F-F+F+FF-F+F+FF+F-F-FF+FF-FF+F+F-FF-F-F+FF-F-F+F+F-F+"}
iterations = 3 # TOP: 3
angle = 90

Peano-Gosper-Curve

axiom = "FX"
rules = {"X":"X+YF++YF-FX--FXFX-YF+", "Y":"-FX+YFYF++YF+FX--FX-Y"}
iterations = 4 # TOP: 6
angle = 60

Sierpinski-Curve

axiom = "F+XF+F+XF"
rules = {"X":"XF-F+F-XF+F+XF-F+F-X"}
iterations = 4 # TOP: 8
angle = 90

Krishna-Anklets

axiom = " -X--X"
rules = {"X":"XFX--XFX"}
iterations = 3 # TOP: 9
angle = 45

Quadratic-Gosper

axiom = "YF"
rules = {"X": "XFX-YF-YF+FX+FX-YF-YFFX+YF+FXFXYF-FX+YF+FXFX+YF-FXYF-YF-FX+FX+YFYF-", 
        "Y": "+FXFX-YF-YF+FX+FXYF+FX-YFYF-FX-YF+FXYFYF-FX-YFFX+FX+YF-YF-FX+FX+YFY"}
iterations = 2 # TOP: 3
angle = 90

Moore-Curve

axiom = "LFL-F-LFL"
rules = {"L":"+RF-LFL-FR+", "R":"-LF+RFR+FL-"}
iterations = 0 # TOP: 8
angle = 90

Hilberts-Curve

axiom = "L"
rules = {"L":"+RF-LFL-FR+", "R":"-LF+RFR+FL-"}
iterations = 8 # TOP: 9
angle = 90

Hilbert-Curve-II

axiom = "X"
rules = {"X":"XFYFX+F+YFXFY-F-XFYFX", "Y":"YFXFY-F-XFYFX+F+YFXFY"}
iterations = 4 # TOP: 6
angle = 90

Peano-Curve"

axiom = "F"
rules = {"F":"F+F-F-F-F+F+F+F-F"}
iterations = 2 # TOP: 5
angle = 90

Cross

axiom = "F+F+F+F"
rules = {"F":"F+FF++F+F"}
iterations = 3 # TOP: 6
angle = 90

Triangle

axiom = "F+F+F"
rules = {"F":"F-F+F"}
iterations = 2 # TOP: 9
angle = 120

Dragon-Curve

axiom = "FX"
rules = {"X":"X+YF+", "Y":"-FX-Y"}
iterations = 8 # TOP: 16
angle = 90

TerDragon-Curve

axiom = "F"
rules = {"F":"F-F+F"}
iterations = 5 # TOP: 10
angle = 120

Twin-Dragon-Curve

axiom = "FX+FX"
rules = {"X":"X+YF+", "Y":"-FX-Y"}
iterations = 6 # TOP: 16
angle = 90

ThreeDragon-Curve

axiom = "FX+FX+FX"
rules = {"X":"X+YF+", "Y":"-FX-Y"}
iterations = 7 # TOP: 15
angle = 90

Code

All the examples above were produced by the same code and some challenges emerged when working on it, mainly to keep the fractal centered (or at least as much as possible), dealing with colors, inversions and offsets and to export it quickly in a vectorial format. Here I will show you the most basic version, if you want to know how I dealt with those challenges, leave me a comment and I will make a Part 2.

This version plots in black and white and with no export functionalities

import turtle

def create_l_system(iters, axiom, rules):
    start_string = axiom
    if iters == 0:
        return axiom
    end_string = ""
    for _ in range(iters):
        end_string = "".join(rules[i] if i in rules else i for i in start_string)
        start_string = end_string

    return end_string


def draw_l_system(t, instructions, angle, distance):
    for cmd in instructions:
        if cmd == 'F':
            t.forward(distance)
        elif cmd == '+':
            t.right(angle)
        elif cmd == '-':
            t.left(angle)


def main(iterations, axiom, rules, angle, length=8, size=2, y_offset=0,
        x_offset=0, offset_angle=0, width=450, height=450):

    inst = create_l_system(iterations, axiom, rules)

    t = turtle.Turtle()
    wn = turtle.Screen()
    wn.setup(width, height)

    t.up()
    t.backward(-x_offset)
    t.left(90)
    t.backward(-y_offset)
    t.left(offset_angle)
    t.down()
    t.speed(0)
    t.pensize(size)
    draw_l_system(t, inst, angle, length)
    t.hideturtle()

    wn.exitonclick()

Code Explanation

import turtle

First we need to import the Turtle Module

def create_l_system(iters, axiom, rules):
    start_string = axiom
    if iters == 0:
        return axiom
    end_string = ""
    for _ in range(iters):
        end_string = "".join(rules[i] if i in rules else i for i in start_string)
        start_string = end_string

    return end_string

Then we need to generate the L-System which will be the set of instructions for the turtle. We define a function called create_l_system which receives the number of iterations, the axiom and the production rules. It starts with the axiom and uses an auxiliary variable called end_string if iteration is equal to 0 it will return the axiom since some fractals can be plot with iterations equal to 0. Rules here are assumed to be dictionaries, so the key will be unique and represents the symbol and the value represents what should be replaced with. So we join all the replacements for each symbol and we end up with the string for the next iteration.

def draw_l_system(t, instructions, angle, distance):
    for cmd in instructions:
        if cmd == 'F':
            t.forward(distance)
        elif cmd == '+':
            t.right(angle)
        elif cmd == '-':
            t.left(angle)

Then we define a draw_l_system which takes the turtle, the set of instructions (the output of the L-System), the angle for the turn right/left and the length of each individual line. It consist of a simple elif structure for each of the previously defined commands.

def main(iterations, axiom, rules, angle, length=8, size=2, y_offset=0,
        x_offset=0, offset_angle=0, width=450, height=450):

    inst = create_l_system(iterations, axiom, rules)

    t = turtle.Turtle()
    wn = turtle.Screen()
    wn.setup(width, height)

    t.up()
    t.backward(-x_offset)
    t.left(90)
    t.backward(-y_offset)
    t.left(offset_angle)
    t.down()
    t.speed(0)
    t.pensize(size)
    draw_l_system(t, inst, angle, length)
    t.hideturtle()

    wn.exitonclick()

And finally the main function, it receives all the parameters it need for the generation of the L-Systems and also y_offset, x_offset, offset_angle, width and height. The first three represent the offset of the turtle, this is merely for positioning the plot as we want inside the canvas.

The function first generates a set of instructions and stores it in inst then it initializes the turtle and the screen and it positions the turtle in the defined position, then it plots the instructions and wait for a click to close.

Special Considerations

As I mentioned earlier there are lots of limitations here, first we are not including the possibility of moving the turtle without drawing, this would require another symbol, it isn't either a symbol for going backwards, or remember previous positions. This weren't necessary for all the fractals you've seen but for some other (like fractal trees), these symbols are required.

Additional Resources

There are lots of resources in internet about fractals both from the programming and from the math perspective. I will show you two that I find really interesting: The 3Blue1Brown for math and the CodingTrain for code.

Coding Train

3Blue1Brown

Sources and References

This post was inspired by a Math World post, the Paul Broke Blog and an university assignment for Genetic Algorithms.

Configuring FastAI with Google Colab

2018-09-28T00:00:00-03:00

Nowadays there are lots of tutorials and material to learn Artificial Inteligence, Machine Learning and Deep Learning but whenever you want to do something interesting you notice you need a Nvidia GPU. In this tutorial we are going to solve this issue with a free cloud solution.

EDIT: this post is now OBSOLETE since the FastAI Team prepare specific tutorial for this an other setups, this could be found in their official course page.

WARNING: The fastai framework is migrating because of the release of PyTorch 1.0. The content of this notebook will be updated but the code might not work until the version 1.x of fastai has clear instructions about its installation.

First of all I'm going to introduce some definitions and concepts just for those that are not familiar with these topics. If you just want to jump directly to the tutorialn, scroll down to the tutorial section

Introduction

More often than not I find people who are confused about the relationship of the terms "Artificial Inteligence", "Machine Learning" and "Deep Learning". Let's try to explain it a bit (no rigorous math!). Artificial Inteligence is the broad term, lots of areas fall into this category for example, neural networks, computer vision, natural language, chatbots, sentiment analisis, etc. When people used Machine Learning they refer to more "statistical" methods such as Support Vector Machines (SVMs), Classification Trees, K-Means, Mean-Shift, etc. Another are is Deep Learning which is the narrower of all since it focuses just in one topic artificial neural networks, there are several type of neural networks and when they have several "hidden layers" we are talking about "Deep" Learning.

If you want to know more about which are the fields of Computer Science, I suggest you to watch Map of Computer Science by Domain of Science

One of the most common libraries to solve Machine Learning Problems is Scikit-Learn and they provide a beautiful roadmap to understand which algorithms are inside this field and their categorizations (there may be much more but it's enough for an introduction):

Simirlarly there is a beautiful image which shows all the types of neural networks, along with a medium post explaining them.

The problem

Generally, when talking about computers, the more you want to scale and parallelize, the simpler the algorithm should be. Complex algorithms are slow and don't usually scale well, this is why nowadays there is a lot of hype arround AI. Artificial Neural Nets were first proposed in 1943! Why nobody used them until now? Because although they are simple, they need lots of computations and CPUs aren't that good at that, they have to perform logic operation, memory management and many other things. But GPUs on the other hand were created primarily for aritmetic calculations and it is today that we have GPUs that powerful that Deep Learning and Machine Learning are indeed possible in a reasonable amount of time. Other things influenced too of course, such as cloud services, fast disks and lots of research.

This is why according to the Gartner Hype Cycle Deep Learning is at its peak:

So, just buy a GPU and you will be ready for Deep learning! Not so fast, frameworks and libraries work better on GPUs, but those should be Nvidia and support CUDA and cuDNN which is another requirement many won't meet. Additionally, the GPU is generally (unless you are building a server), the most expensive part of a PC System.

This is why finding a way to experiment with deep learning may not be free. There are several cloud based services where you can rent a GPU for hour and even seconds and they are pretty cheap, but if you just want to try without paying, those might not be a suitable option. This is where Colaboratory comes in, it's a service offered by Google which provides free GPU (up to 12 GB of Video RAM) for your computations, it also supports colaboratory editing (like Google Docs) and it can be saved in Google Drive or Github.

Tools and requirements

Developing software implies learning a programming language, usually a framework, learn arquitecture patterns, best practices and of course a lot more. But AI goes in another direction, it is math heavy (although you can build amazing stuff with high school algebra), it realies in a predifined set of algorithms and the cutting edge news usually comes in the form of a scientific paper rather than a fancy framework. Because of this, two different roles emerged, in one hand, there is the Software Developer and in the other hand there is the Data Scientist, both know data structures, algorithms, programming but their skillset is quite different. We will see which tools you will need to start digging in AI and Deep Learning.

First you will need to learn programming basics, control flow structures and decisions, it isn't needed to know OOP or Functional programming although it might be useful for some. Second, you will need to know Python, it is the most used language in the field and it seems it will continue like that and third you will need to feel confortable with a new "environment" called notebooks, particularly Jupyter Notebooks. I know some might disagree, specially in the last point, there are lots of opinions and I strongly believe that if you learn these things you will be able to watch most of video-tutorials on youtube and you will be on track without issues. You can convice yourselves looking at this survey, based on 10.153 responses:

Once you know this prerequirements, you will face a very common dilemma, which Deep Learning Framework should you use? Using a framework is recommended (although doing it manually help a lot to understand the underlying process) so for this tutorial I chose fastai because it's built over PyTorch which is known for being the most pythonic and thus the friendlier to begginers.

Tutorial

In case you want to see the final result, this is the final notebook:

Now I'm going to explain how that notebook works.

First we have to make sure GPU acceleration is enabled, to achieved this by setting "Hardware Accelerator" to GPU in the menu: Edit > Notebook settings > set "Hardware Accelerator" to GPU.

Since Fastai is not built in Colaboratory, we have to install it manually, the best way is by source since it's in rapid development and the realeses found via pip could be outdated. Additionally we can install PyTorch 3.1 with CUDA 9.1 in the same cell.

%%bash

if ! [ -d fastai ]
then
git clone https://github.com/fastai/fastai.git
fi

cd fastai

git pull

pip -q install .

pip -q install http://download.pytorch.org/whl/cu91/torch-0.3.1-cp36-cp36m-linux_x86_64.whl

pip -q install torchvision

Now we can import fastai with all its modules

from fastai.imports import *
from fastai.transforms import *
from fastai.conv_learner import *
from fastai.model import *
from fastai.dataset import *
from fastai.sgdr import *
from fastai.plots import *

And we are ready to go! You can check that the GPU is enabled with the following command:

f'Is CUDA and CUDNN enabled: {torch.cuda.is_available() and torch.backends.cudnn.enabled}'

And if you want to check exactly how much Video RAM you have you can run the following snippet:

!ln -sf /opt/bin/nvidia-smi /usr/bin/nvidia-smi
!pip3 -q install gputil
!pip3 -q install psutil
!pip3 -q install humanize

import psutil
import humanize
import os
import GPUtil as GPU

GPUs = GPU.getGPUs()
gpu = GPUs[0]
process = psutil.Process(os.getpid())

print(f"Number of GPUs: {len(GPUs)}")
print(f"Gen RAM Free: {humanize.naturalsize( psutil.virtual_memory().available )} | Proc size: {humanize.naturalsize( process.memory_info().rss)}")
print("GPU RAM Free: {0:.0f}MB | Used: {1:.0f}MB | Util {2:3.0f}% | Total {3:.0f}MB".format(gpu.memoryFree, gpu.memoryUsed, gpu.memoryUtil*100, gpu.memoryTotal))

Conclusion

It is indeed possible to do Deep Learning with GPU without paying nothing, we just have to wait until the right set of technologies is available.

Create one executable file for a Flask app with PyInstaller

2018-02-24T00:00:00-03:00

EDIT 2019.05.03: The content order was change and the ideas were express more succinctly.

Having a single executable file could be a great advantage in so many cases and nowadays more a more desktop applications are using web technologies (React Native, Cordova, Ionic, etc.). It was time for the Python stack to join the Group via PyFlaDesk, a simple script to combine Flask, Qt and PyInstaller to create desktop Apps.

I wanted to create a single executable desktop application and since PyInstaller was there for a while I thought it would be pretty easy, it turns out that after after trying lots of solutions from Stack overflow, Quora and several blogs without success, I decided to post how I managed to solve the problem.

Note: A full list of the resources mentioned is below

If you just want to read the solution, scroll down to the solution section

The Problem

Well, Flask is a WEB framework (or Microframework as it desribes itself), used to create WEBsites that will be served by a WEB server and view from a WEB browser. Problems emerge when we try to replace 'WEB' with desktop in the previous sentence, we are trying to force something built for a particular scenario with its limitations to work in a completely different one.

Motivation

Desktop applications aren't the most popular use for Python, those positions belongs to Web Development, DevOps and Data Science and, among the technologies used in the Web Development, Django and Flask are the most popular frameworks.

Using the tools one already knows for something completely different is a great advantage, in this case we want to create a desktop app with a single executable file from a web app built with Flask.

Some of the main advantages of the self-contained (also known as portable) software are:

No virtual environment required.
Knowing which software is installed in the destination machine isn't necessary.
No installation required, just copy and paste or execute from removable drive.
Easier to tell the final user where is the program and how to execute it.
Harder (yet not impossible) to reverse engineer the software.

There are some obvious disadvantages such as:

Since all dependencies are included, files could get BIG (Hello World app in Flask is about 25MB).
No version control support for binaries.
If you consume self-contained software you have to trust your provider about its safety.
Working with external files, databases and other external sources has to be strongly tested.

This are some points to consider, they could be advantages or disadvantages depending on the case:

You treat the software as a black box.
When a new version is released you just replace the file and you're done.

Flask comes into the action

If you attempt to port a web app to a desktop one without changing the framework chances are you will use a lightweight framework like Flask. Now, how do you create a desktop app? Using a desktop framework (Qt, Tk, wx, etc), the most commonly used framework for this task is Qt. The idea is to create a minimal web browser capable of rendering HTML and then, execute the flask application server and browse to the url of the server inside this browser.

But what are the differences between creating a web browser and using the systems default? Well, first of all, you assume there will be one, and that that one would be able to render all the HTML, CSS and JS you are using, that could not be the case. More often than not we found ourselves developing software for old operating systems (aka Windows XP o older).

Sounds easy and actually it can be done, I was contributing to a script to achieve this very goal PyFladesk. There are some concerns about which version of Qt is the appropriate and the convenience of one over the other.

Note: If all you use is Flask for served static or pseudo static content you could tried Frozen Flask.

Solution

After reading all the previous posts and some of the PyInstaller docs. I found that some people actually solved it! But, the solution they propose was editing the spec file, which is generated after a first run of PyInstaller. I thought that solution was a hack and not the proper way to achieve what I wanted.

So I tried to understand what the changes in the spec file did and it turned out that that changes was to copy the folders Flask uses into the file directory/file (Actually one of the proposed solutions was build and then copy paste the folders, but besides being unpractical it wouldn't work with one file builds). After a little reasearch, I came across the command line arguments to achieve the same.

Windows:

pyinstaller -w -F --add-data "templates;templates" --add-data "static;static" app.py

Linux (NOT TESTED):

pyinstaller -w -F --add-data "templates:templates" --add-data "static:static" app.py

This will create a folder dist with our executable ready to be shipped. The executable will open the main window of our app.

It first started as a contribution for the PyFladesk project and then, I realize that since Qt is quite big, our executable were big too. The example app of that repository is 70 MB (much of which was the Qt Component for displaying HTML (WebEngine)). This is reasonable taking into account that we were shipping a self contain web browser.

There is no simple solution to the size problem but some suggestions are proposed in a following section.

Tutorial

If you haven't already, install it with pip (if you use virtual environments you should install it inside it)

pip install pyinstaller

Some parameters to consider:

F - Bundles everything in a single file
w - Avoid displaying a console
--add-data - Add Folders to the directory/executable

Since Flask relies on a directory structure you should pass it to the folder, in the example case we only have two folders: templates and static, in case you use a database or some other directory structure you may adapt this.

Note: For more complex scenarios check the PyInstaller Docs

The Size Problem

Is the project using as few dependencies as possible? If yes, continue reading, if not check and then come back. Make sure you create a virtual environment for your project and execute PyInstaller from there, if the size is still big, I recommend you to check one of these:

Use Virtual Environments and install everything you need there, including PyInstaller (but nothing more!)
Check if all your dependences are really necessary, try to use the standard library when possible
Check if your biggest dependencies could be replaced with lightweight alternatives
Use one-dir option and then see what are the biggest dlls and if you can exclude them
Use the ArchiveViewer.py script that's part of PyInstaller and exclude everything you don't need

Conclusion

Using PyInstaller and Flask is not as hard as people may experience if you have the correct knowledge but it requires a bit of work to get a clean, lightweight file. However, it's possible to create executable with complex apps, with special directory structure, databases and so on, but don't expect that to be a tiny file.

Additional Resources

The following are the pages where this topic was mentioned and I couldn't find a proper answer:

Flask application built using pyinstaller not rendering index.html - jinja2.exceptions.TemplateNotFound (Stack Overflow)
Pyinstaller on Flask app, import error (Stack Overflow)
Using Pyinstaller on Python Flask Application to create Executable (Stack Overflow)
Python 2.7.12, trying to build an executable file using pyinstaller. I keep getting the below error (Stack Overflow)
Can I convert a Flask application into an executable file that runs on Windows like an .exe file? (Quora)
Is it possible to deploy/distribute Flask as an executable for desktop use? (Reddit)
Flask and pyinstaller notice (Personal Website)

Haskell pattern matching in Python

2018-02-18T00:00:00-03:00

I started to learn a bit of Haskell and one of its features amazed me: Pattern Matching. Then, unexpectably, I notice Python 3 have them too but with a different name: Extended Tuple Unpacking. Okay, it's not exactly the same but you can get quite similar functionalities if you master it. This post is inspired by this question

Pattern what?

Pattern matching is a feature of Haskell that allows us to think in terms of wholemeal programming, this means that we forget about the specific details of what we are working with. For example: Think in terms of first and last element instead of index 0 and index -1. This way of thinking has a very strong link with another common functional pattern: Recursive Functions

Is this a Python or a Haskell thing?

Haskell is much older than Python and there are other languages that implements it besides Haskell so we can't say it's from either. But it turned out to be something that useful that it was incorporated into Python in the PEP 3132 (In case you don't know, PEPs are the python enhancenment proposals, where all new features are asked and discussed).

How does it look like?

Tuple Unpacking

You probably used and make yourself familiar with typica tuple unpacking

Either because you did this:

a, b, c = (3, 4, 5)

a, b, c = (3, 4, 5)

Or this:

a = 5
b = 10
a, b = b, a

Or maybe you use it at the top of a for loop:

names = ['John', 'Peter']
for i, name in enumerate(names):
    print(f'{i}: {name}')

Extended Tuple Unpacking

These are common uses of the tuple unpacking but a more powerful feature is built on top of it: The extended version, which uses the Unpacking Argument Lists (splat)

Lets see some examples:

We will use the following list for this example:

list_ = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Which by the way can also be written as:

list_ = [i for i in range(10)]

Or even more simplified:

list_ = list(range(10))

Examples

Get the first item of a list:

first, *rest = list_

Get the last item of a list:

*rest, last = list_

Get the first and the last item of a list

first, *rest, last = list_

Caveats

Although it may seem quite simple an easy, there are lots of ways to get it wrong:

(a,b), c = "XYZ"                 # ERROR -- too many values to unpack
(a,b), c = "XY"                  # ERROR -- need more than 1 value to unpack
(a,b), (c,) = [1,2],'this'       # ERROR -- too many values to unpack
a, *b, c, *d = 1,2,3,4,5         # ERROR -- two starred expressions in assignment
(a,b),c = 1,2,3                  # ERROR -- too many values to unpack
(a,b), *c = 1,2,3                # ERROR - 'int' object is not iterable

Is there any logic behind it?

Of course there is! This is programming and for a full explanation I recommend you to read the PEP and this answer