EnVision

2026-10-19T04:38:12.977-07:00

As Electi reaches it's 9th birthday, we have gone major shifts in the industry.

Last year marked the end of scarcity in code, now everything revolves around leveraging LLM's and AI to create something.

However building something of value in a controlled environment with such immence speed requires having absolute control of the process. Since 2017 we have worked with banks, regulators, insurers and shipping groups on the buying side of AI.

Follow one product from first pilot to scale inside organisations like these, and the same foundations appear in order.

1. It starts with a job that has an owner, a deadline and a budget, like closing the month-end books. Jensen Huang points to the same place: the biggest opportunities sit inside real-world problems. The best teams stay in that one segment until their first 100 users cannot work without them. Those users outweigh 100,000 who tried the demo once.

2. Then the buyer asks the first hard question: how do you know it is right? Evals answer it, so they come before features. They also make the architecture safe to change. Plan to swap models twice a year. With a test set that shows within a day whether the new model performs better on real tasks, a swap takes two weeks. Without one, it takes a quarter. Next to evals sits the trust layer: permissions, audit logs, human in the loop. Enterprise buyers pay for control.

3. Next comes money. Inference is COGS, so measure the cost of each completed task before you set a price. Then price the outcome. Seat pricing shrinks as agents replace headcount, which means revenue falls just as the product starts working. Outcome pricing holds up when both sides share a definition of done they can audit. Covey's advice applies: begin with the end in mind.

4. After that, the product starts to compound. Every completed task feeds a loop of usage data that only you hold. The same usage earns trust across your client base, and that trust becomes your distribution. A competitor can rebuild your product in 6 weeks. Earning those relationships takes years. Eric Ries put the discipline plainly: the only way to win is to learn faster than anyone else.

Read the journey again. Every stage is an operating discipline: focus, measurement, control, pricing, learning. Grove, Covey and Horowitz wrote that playbook decades before the first LLM, and it still applies.

build the loop <-> rent the model

2026-09-04T05:44:11.058-07:00

This quote hit hard : "Nobody wants to read something that you didn’t bother writing."

also : "Artisanal, hand made slop"

Not your average LI slop:

2026-09-03T01:28:49.783-07:00

It's been a long time since i posted here.

We have moved to the agentic era, a new shitstorm of bad software is coming online.

Software that is sloppy, brittle and unrefined.

The main culprit of fragility at this current period is the agentic part.

Your agent has one version number and four things that decide what it actually does.

> Ship a change to the orchestration code, tag it, done. That covers one of the four.

>> The instructions and policy block is the second. Also yours, also the one everyone treats as the whole system, right up until the third and fourth parts move without telling you.

>> The model runtime is the third. A vendor's pinned snapshot, sitting in Active, Legacy, Deprecated, or Retired, on that vendor's own lifecycle, with a minimum notice period before it goes away. Anthropic's is 60 days. That is one company's policy. It binds nobody else's model, and it does not ask what your regression suite thinks.

>>> The fourth is the protocol and tools your agent calls out through. MCP now stamps a dated version string on every single request, which is more discipline than most teams manage on their own code. The tool schemas sitting behind that protocol still version on their own schedule, with no such stamp, and no one upstream is watching your integration tests when they change.

Most teams have a git log for the first two and nothing for the last two, then act surprised when the agent that passed every eval last month answers differently this month with no local commit to blame.

I put the rest of it in the attached deck: the release pipeline for a system that cannot be tested to certainty, what decides whether a rollback is even possible, and the four OWASP agentic-risk entries that are this problem read as a containment question.

Presentation here: https://drive.google.com/file/d/1rAghJ-Wcp57gFey1caBpA8xKWenz28eB/view?usp=drive_link

Keras/Tensorflow threshold with gradient flow

2021-01-18T01:54:00.005-08:00

Tensorflow / Keras threshold operations break the gradient flow.

There is a way to fix this by using a combination of operations.

def threshold_min_max_value(input_layer,
                            min_value=0.0,
                            max_value=1.0):
    """
    Thresholds all the values of the a tensor that exceed value to that
    max_value, and than are lower than the min_value,
    this layer retains gradient flow

    :param input_layer: the input layer
    :param min_value: minimum value to threshold to
    :param max_value: maximum value to threshold to
    :return: threshold-ed input layer
    """

    def _threshold(_x):
        ge_max_value = K.greater_equal(_x, max_value)
        ge_max_value = K.cast_to_floatx(ge_max_value)
        lt_max_value = 1.0 - ge_max_value

        le_min_value = K.less_equal(_x, min_value)
        le_min_value = K.cast_to_floatx(le_min_value)
        gt_min_value = 1.0 - le_min_value

        tmp0 = keras.layers.Multiply()([
            lt_max_value, gt_min_value, _x
        ])
        return tmp0 + (min_value * le_min_value) + (max_value * ge_max_value)

    return keras.layers.Lambda(_threshold)(input_layer)

Fix Tensorflow Object Detection Framework taking too much disk space while training

2020-12-24T04:22:00.001-08:00

The model_lib_tf2.py file contains all the training and evaluation loops.

In the function:

def eager_train_step(detection_model,
                     features,
                     labels,
                     unpad_groundtruth_tensors,
                     optimizer,
                     learning_rate,
                     add_regularization_loss=True,
                     clip_gradients_value=None,
                     global_step=None,
                     num_replicas=1.0):

We can see that every training iteration it saves a few training dataset images.

tf.compat.v2.summary.image(
    name='train_input_images',
    step=global_step,
    data=features[fields.InputDataFields.image],
    max_outputs=5)

There are three problems with this:

1. It takes A LOT of space

2. It actually slows up training

3. Images look saturated.

We can fix this easily by replacing the above snippet with this:

if global_step % 100 == 0:
    # --- get images and normalize them
    images_normalized = \
        (features[fields.InputDataFields.image] + 128.0) / 255.0
    tf.compat.v2.summary.image(
        name='train_input_images',
        step=global_step,
        data=images_normalized,
        max_outputs=5)

Gaussian Filter in Keras (code snippet)

2020-10-12T04:36:00.002-07:00

Very often we need to perform basic vision operations on a computational graph like building a Laplacian pyramid or filter a tensor with a specific precalculated filter.

Below i present a code snippet for building a fixed non-trainable gaussian filter in keras.

import keras
import numpy as np
import scipy.stats as st

def gaussian_filter_block(input_layer,
                          kernel_size=3,
                          strides=(1, 1),
                          dilation_rate=(1, 1),
                          padding="same",
                          activation=None,
                          trainable=False,
                          use_bias=False):
    """
    Build a gaussian filter block
    :return:
    """

    def _gaussian_kernel(kernlen=[21, 21], nsig=[3, 3]):
        """
        Returns a 2D Gaussian kernel array
        """
        assert len(nsig) == 2
        assert len(kernlen) == 2
        kern1d = []
        for i in range(2):
            interval = (2 * nsig[i] + 1.) / (kernlen[i])
            x = np.linspace(-nsig[i] - interval / 2., nsig[i] + interval / 2.,
                            kernlen[i] + 1)
            kern1d.append(np.diff(st.norm.cdf(x)))

        kernel_raw = np.sqrt(np.outer(kern1d[0], kern1d[1]))
        # divide by sum so they all add up to 1
        kernel = kernel_raw / kernel_raw.sum()
        return kernel

    # Initialise to set kernel to required value
    def kernel_init(shape, dtype):
        kernel = np.zeros(shape)
        kernel[:, :, 0, 0] = _gaussian_kernel([shape[0], shape[1]])
        return kernel

    return keras.layers.DepthwiseConv2D(
        kernel_size=kernel_size,
        strides=strides,
        padding=padding,
        depth_multiplier=1,
        dilation_rate=dilation_rate,
        activation=activation,
        use_bias=use_bias,
        trainable=trainable,
        depthwise_initializer=kernel_init,
        kernel_initializer=kernel_init)(input_layer)

from my open source project https://github.com/NikolasMarkou/multiscale_variational_autoencoder

Tensorflow to Onnx (tf2onnx) testing with different opsets

2020-08-24T02:29:00.003-07:00

To test tf2onnx changes with different operators set your external variable prior to calling pytest.

So for example in a windows setup this will run the tests

set TF2ONNX_TEST_OPSET=11

pytest

and in a standard linux setup

export TF2ONNX_TEST_OPSET=11

pytest

Opencv 4.3.0 options

2020-06-11T06:03:00.002-07:00

Building OpenCV 4.3.0 from source

cmake ../ -DHAVE_MKL=ON -DBUILD_opencv_python2=OFF -DBUILD_opencv_java=OFF -DMKL_WITH_TBB=ON -DWITH_OPENCL=ON -DWITH_CUDA=ON -DWITH_CUBLAS=ON -DENABLE_CXX11=ON -DOPENCV_EXTRA_MODULES_PATH=/home/arxwn/Repositories/opencv/opencv_contrib/modules -DCPU_BASELINE="SSE3" -DCPU_DISPATCH="SSE4_1;SSE4_2;AVX;FP16;AVX2;AVX512_SKX" -DOPENCV_ENABLE_NONFREE=ON

Netron an awesome neural networks visualization tool

2020-04-07T19:51:00.001-07:00

I've been working recently on optimizing models from tensorflow to onnx and finally to tensorrt and got introduced to an excellent tool for all ml engineers / data scientists working with different flavors and formats of neural networks.

It is a huge improvement over the and visualizer.

Check it out here

Compiling Tensorflow 1.15 from source

2020-03-23T23:28:00.002-07:00

Following my previous post Compiling Tensorflow under Debian Linux with GPU support and CPU extensions I was trying to build version r1.15 when i stumbled to another peculiar bug. When reaching the end of the compilation you get a lovely error in the form of

from keras.preprocessing import image as image_utils
ImportError: No module named keras.preprocessing

To fix this you you just need to install the following packages prior to compiling

pip install keras_applications==1.0.4 --no-deps
pip install keras_preprocessing==1.0.2 --no-deps
pip install h5py==2.8.0

And that's it you can build again.

bazel build -c opt --config=v1 --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-mfpmath=both --copt=-msse4.2 --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package

Software Commodification

2020-02-05T06:28:00.001-08:00

Something that people outside the don't fully grasp is the increasing pace of .

You had a Phd in CS 20 years ago ? congrats all your knowledge is a library.

You were an excellent physicist 10 years ago ? Congrats we can import and run everything you knew without even understanding half of it.

You studied Artificial Intelligence in depth 15 years ago ? A 17 year old with a weekend in can provide better solutions than you. Everything you know is getting obsolete in an increasing rate.

As people we are are used to packaging up our knowledge and re-learning new skills (tools, theories, frameworks) every couple of years, because the old have been completely commoditized.

It is an exciting but tiresome journey that it shouldn't be required by all. However as everything is becoming software there may not be a choice by most people.

Personally the increasing pace of commodification allows me to cut through the noise and add depth to my knowledge. Knowing that the low hanging fruits have been picked will clear up the space of and Machine Learning.

What if you could push your AI models to be 10 times faster?

2020-01-20T05:58:00.002-08:00

Deceptive Easy

Modern Machine Learning (ML) and especially Deep Learning (DL) have become deceptively easy. It is almost trivial to have something up and running with presentable results as the the tools hide most of the complexity and hard decisions. Whereas that might be good enough for a Proof of Concept (POC) or a Minimum Viable Product (MVP), ensuring stability, high performance and scalability is a whole different ball game.

High Sunk Cost

Many CXO’s and seniors managers find themselves trapped into subpar solutions that cannot be used effectively because they lack the technical know-how to productionalize them.

At Electi we have just release a new brochure listing the services we can offer to companies already implementing Deep Learning. Check it out here.

Performance Measures for Machine Learning

2019-12-02T11:23:00.001-08:00

Summary

In this blog spot I'm presenting a few performance measures for machine learning tasks. These performance measures come up a lot in the marketing domain.

Take at home lessons:

the measure you optimize to makes a difference
the measure you report makes a difference
use measure appropriate for problem/community
accuracy often is not sufficient/appropriate
only accuracy generalizes to >2 classes
this is not an exhaustive list of performance measures

Confusion Matrix

First we construct a confusion matrix for a binary classification problem. Given a classification function f(x)->R and a threshold T that can split the outcomes into {0, 1} we can create a confusion matrix that counts the occurrences of the predicted class given the true label.

Accuracy

Accuracy is then measured as the percentage of correct responses (True Positives + True Negatives) over the total amount of responses.

Problems with Accuracy

This measure is commonly used but it can be misleading. The problems arise from the domain we are modelling. If one of the class for example is poorly represented into the metric is meaningless as we could predict the same class always and still have a good results.

• Assumes equal cost for both kinds of errors

cost(b-type-error) = cost (c-type-error)

• is 99% accuracy good?

can be excellent, good, mediocre, poor, terrible
depends on problem

• Base Rate = accuracy of predicting predominant class

Weighted (Cost sensitive) Accuracy

A modified version of accuracy is "Weighted Accuracy" were we count the cost of misclassification.

In this scenario we aiming for a model and a threshold that can minimize the total cost.

Lift

If we are not interested in the accuracy on the entire dataset but want accurate predictions for 5%, 10% or 20% of the dataset then we can use the lift measure.

Lift measures how much better than random prediction on the fraction of the dataset predicted true (f(x) > threshold).

Precision / Recall

The Precision measure counts how many of the interest class are correct.
The Recall measure counts how many of the interest class does the model return.

In the case below the interest class is a(1).

We can change the sweep over the threshold calculate Precision/Recall multiple times and graph out what is called the Precision/Recall curve.

At each different threshold we can see a different tradeoff between the two metrics.

When the threshold is too high then c (everything is predicted as class 0) becomes zero and then the precision becomes zero.
When the threshold is too low then b (everything is predicted as class 1) becomes zero and then the recall becomes zero.

Both of these metrics are flawed in isolation and it is the eye of the modeller on which one better represents the problem.

F-Measure

The F-Measure is an attempt to merge the two measures to construct a more meaningful performance measure.

Receiver Operating Characteristic (ROC)

• Developed in WWII to statistically model false positive and false negative detections of radar operators
• Better statistical foundations than most other measures
• Standard measure in medicine and biology
• Becoming more popular in ML

Although ROC graphs are apparently simple, there are some common misconceptions and pitfalls when using them in practice.

One of the earliest adopters of ROC graphs in machine learning was Spackman (1989), who demonstrated the value of ROC curves in evaluating and comparing algorithms.

ROC graphs are conceptually simple, but there are some non-obvious complexities that arise when they are used in research.

ROC Plot

• Sweep threshold and plot

TPR vs. FPR
Sensitivity vs. 1-Specificity
P(true|true) vs. P(true|false)

• Sensitivity = a/(a+b) = Recall = LIFT numerator
• 1 - Specificity = 1 - d/(c+d)

A ROC graph depicts relative trade-offs between benefits (true positives) and costs (false positives).

The lower left point (0,0) represents the strategy of never issuing a positive classiffication.
The opposite strategy is represented by the upper right point (1,1).
The point (0,1) represents perfect classiffication.
The diagonal line y = x represents the strategy of randomly guessing a class.
A random classifier will produce an ROC point that "slides" back and forth on the diagonal based on the frequency with which it guesses the positive class. In order to get away from this diagonal into the upper triangular region, the classifier must exploit some information in the data.
Any classifier that appears in the lower right triangle performs worse than random guessing. This triangle is therefore usually empty in ROC graphs.
ROC curves have an attractive property: they are insensitive to changes in class distribution.
Any performance metric that uses values from both columns of theconfusion matrix will be inherently sensitive to class skews. Metrics such as accuracy, precision, lift and F scores use values from both
columns of the confusion matrix.
ROC graphs are based upon TP rate and FP rate, in which each dimension is a strict columnar ratio, so do not depend on class distributions.

Model Distillation

2019-10-24T06:53:00.003-07:00

Model Distillation is the process of taking a big model or ensemble of models and producing a smaller model that captures most of the performance of the original bigger model. It could be also better be described as a blind model replication method.

The reasons for doing so are:

improved run-time performance (FLOP operations)
(maybe) better generalization because of the model simplicity
you don't have access to the training of the original model.
you have access to a remotely deployed model and you want to replicate it (it happens more than you can imagine)
original model maybe is too complicated
insights that may arise from the process itself

How it works

Assume a MNIST classifier \(F_{MNIST}\) composed of an ensemble of \(N\) convolutional deep neural networks that produces a logit \(z_i\) which is then converted to a probability of an input image, \(x_i\), for each of the possible labels \(C_{0}-C_{9}\).

The distillation process will give us an \(F_{MNIST_{distilled}}\) composed of a single deep neural network that will approximate the classification results of the bigger ensemble of models.

In distillation, knowledge is transferred from the teacher model to the student by minimizing a loss function in which the target is the distribution of class probabilities predicted by the teacher model. That is - the output of a softmax function on the teacher model's logits.

Logits \(z_j\) are converted to probabilities \(P(C_i|x)\) using the softmax layer:

\(
p_i = \frac
{exp(z_i)}
{\sum_{j}exp(z_j)}
\)

However, in many cases, this probability distribution has the correct class at a very high probability, with all other class probabilities very close to 0. As such, it doesn't provide much information beyond the ground truth labels already provided in the dataset.

To tackle this issue, Hinton et al., 2015 introduced the concept of "softmax temperature". The probability \(q_i\) is computer by the logit \(z_i\) for the scalar softmax temperature \(T\):

\(
q_i = \frac
{exp(\frac{z_i}{T})}
{\sum_{j}exp(\frac{z_j}{T})}
\)

where T is a temperature that is normally set to 1. Using a higher value for T produces a softer probability distribution over classes. Softer probability distribution means that the values are somewhat diffused and a 0.999 probability may become 0.9 and the rest spread to the other classes.

In the simplest form of distillation, knowledge is transferred to the distilled model by training it on a transfer set and using a soft target distribution for each case in the transfer set that is produced by using the cumbersome model with a high temperature in its softmax. The same high temperature is
used when training the distilled model, but after it has been trained it uses a temperature of 1. When the correct labels are known for all or some of the transfer set, this method can be significantly improved by also training the distilled model to produce the correct labels. One way to do this is to use the correct labels to modify the soft targets, but we found that a better way is to simply use a weighted average of two different objective functions.

The first objective function is the cross entropy with the soft targets and this cross entropy is computed using the same high temperature in the softmax of the distilled model as was used for generating the soft targets from the cumbersome model.
The second objective function is the cross entropy with the correct labels. This is computed using exactly the same logits in softmax of the distilled model but at a temperature of 1.

This very simple operation can have a multitude of knobs and parameters to adjust but the core essence is very simple and works quite well.

ResNets inner workings and notes

2019-10-18T06:21:00.000-07:00

A residual network (or ResNet) is a standard deep neural net architecture, with state-of-the-art performance across numerous applications. The main premise of ResNets is that they allow the training of each layer to focus on fitting just the residual of the previous layer’s output and the target output. Thus, we should expect that the trained network is no worse than what we can obtain if we remove the residual layers and train a shallower network instead.

Up until 2015 we had 3 mainstream ways of training deep networks:

Greedy per layer optimization and freezing
Various flavors of Dropout
and Batch Normalization (came out early 2015, wasn't mainstream until 2016, now it is patented by google so who knows if people can use it)

The main contribution of the "Deep Residual Learning for Image Recognition, 2015" paper is a novel and smart building block for training very deep neural networks.

The "Residual Learning" or "Identity Learning" block.

A special note here:
We need at least 2 weight layers here (nonlinearities are not essential but welcome) to get the benefits of the universal function approximator that has been proven since the 90's. You can add more but you then hit the limits of information propagation and you will need residual/skip connections inside the residual block.

A stack of \(n\) residual blocks is described as follows :

\(x_0\) : input
\(x_1 = x_0 + F_1(x_0)\)
\(x_2 = x_1 + F_2(x_1)\)
\(x_3 = x_2 + F_3(x_2)\)
\(...\)
\(x_n = x_{n-1} + F_{n-1}(x_{n-1})\)

This can be re-written as :

\(x_n = x_{n-1} + F_{n-1}( x_{n-2} + F_{n-2}(x_{n-2}) \)

Which can be expanded as :

\(x_n = x_{n-1} + F_{n-1}( x_{n-2} + F_{n-2}( x_{n-3} +F_{n-3}( x_{n-4} + F_{n-4}(... + F_1(x_0)))))) \)

Now if we assume that \(F_i\) is a linear function (it is not for \(x \lt 0 \) ) but we can ignore it) :

\(
x_n =
x_{n-1} +
F_{n-1}( x_{n-2} ) +
F_{n-1}F_{n-2}(x_{n-3}) +
F_{n-1}F_{n-2}F_{n-3}(x_{n-4}) +
...
F_{n-1}F_{n-2}F_{n-3}...F_1(x_0)
\)

As we can see from the equations and the equivalent graph, there is a clear information flow from the raw data to the output. This means that the major pain point of vanishing gradient is avoided.

This design when stacked as above allows the gradient to flow through the whole network bypassing any tricky points and in essence training the deeper network properly. Information bottlenecks can be introduced by design but the gradient can flow from the loss layer up to the base layer.

Residual Networks (ResNets) have been used to train up to a 1000 layers. It was proven empirically that going deeper generalizes better than shallow wide networks. Again empirically several variations of the Residual Block are tried:

In practice the Residual Function is not 2 (convs or fully connected) weight layers but they are accompanied by a Batch Normalization layer. This helps stabilize the gradient and the co-variate shift (there are various opposing hypothesis on why batch norm works).

In the paper "Identity Mappings in Deep Residual Networks, 2016" a different layout was proposed that improved the performance. We can see an inversion of the order of the layers. The architecture, the analysis and the system design remains the same though.

This new arrangements produces better results on very deep networks for the CIFAR dataset, shown below. Also the convergence is much faster. Since they are practically the same from an engineering perspective we don't have a reason to reject the one with the better performance.

How to Build Good Software

2019-08-27T23:14:00.000-07:00

Software has characteristics that make it hard to build with traditional management techniques; effective development requires a different, more exploratory and iterative approach.

The root cause of bad software has less to do with specific engineering choices, and more to do with how development projects are managed.

The right coding language, system architecture, or interface design will vary wildly from project to project. But there are characteristics particular to software that consistently cause traditional management practices to fail, while allowing small startups to succeed with a shoestring budget:

• Reusing good software is easy; it is what allows you to build good things quickly;
• Software is limited not by the amount of resources put into building it, but by how complex it can get before it breaks down; and
• The main value in software is not the code produced, but the knowledge accumulated by the people who produced it.

Understanding these characteristics may not guarantee good outcomes, but it does help clarify why so many projects produce bad outcomes. Furthermore, these lead to some core operating principles that can dramatically improve the chances of success:

1. Start as simple as possible;
2. Seek out problems and iterate; and
3. Hire the best engineers you can.

While there are many subtler factors to consider, these principles form a foundation that lets you get started building good software.

Software should be treated not as a static product, but as a living manifestation of the development team’s collective understanding.

1. Start as Simple as Possible

Projects that set out to be a “one-stop shop” for a particular domain are often doomed. The reasoning seems sensible enough: What better way to ensure your app solves people’s problems than by having it address as many as possible? After all, this works for physical stores such as supermarkets. The difference is that while it is relatively easy to add a new item for sale once a physical store is set up, an app with twice as many features is more than twice as hard to build and much harder to use.

Building good software requires focus: starting with the simplest solution that could solve the problem. A well-made but simplistic app never has problems adding necessary features. But a big IT system that does a lot of things poorly is usually impossible to simplify and fix. Even successful “do it all” apps like WeChat, Grab, and Facebook started out with very specific functionality and only expanded after they had secured their place. Software projects rarely fail because they are too small; they fail because they get too big.

Unfortunately, keeping a project focused is very hard in practice: just gathering the requirements from all stakeholders already creates a huge list of features.

One way to manage this bloat is by using a priority list. Requirements are all still gathered, but each are tagged according to whether they are absolutely critical features, high-value additions, or nice-to-haves. This creates a much lower-tension planning process because features no longer need to be explicitly excluded. Stakeholders can then more sanely discuss which features are the most important, without worrying about something being left out of the project. This approach also makes explicit the trade-offs of having more features. Stakeholders who want to increase the priority for a feature have to also consider what features they are willing to deprioritise. Teams can start on the most critical objectives, working their way down the list as time and resources allow.

2. Seek Out Problems and Iterate

In truth, modern software is so complicated and changes so rapidly that no amount of planning will eliminate all shortcomings. Like writing a good paper, awkward early drafts are necessary to get a feel of what the final paper should be. To build good software, you need to first build bad software, then actively seek out problems to improve on your solution.

This starts with something as simple as talking to the actual people you are trying to help. The goal is to understand the root problem you want to solve and avoid jumping to a solution based just on preconceived biases. When we first started on Parking.sg, our hypothesis was that enforcement officers found it frustrating to have to keep doing the mental calculations regarding paper coupons. However, after spending just one afternoon with an experienced officer, we discovered that doing these calculations was actually quite simple for someone doing it professionally. That single conversation saved us months of potentially wasted effort and let us refocus our project on helping drivers instead.

Beware of bureaucratic goals masquerading as problem statements. “Drivers feel frustrated when dealing with parking coupons” is a problem. “We need to build an app for drivers as part of our Ministry Family Digitisation Plans” is not. “Users are annoyed at how hard it is to find information on government websites” is a problem. “As part of the Digital Government Blueprint, we need to rebuild our websites to conform to the new design service standards” is not. If our end goal is to make citizens’ lives better, we need to explicitly acknowledge the things that are making their lives worse.

Having a clear problem statement lets you experimentally test the viability of different solutions that are too hard to determine theoretically. Talking to a chatbot may not be any easier than navigating a website, and users may not want to install yet another app on their phones no matter how secure it makes the country. With software, apparently obvious solutions often have fatal flaws that do not show up until they are put to use. The aim is not yet to build the final product, but to first identify these problems as quickly and as cheaply as possible. Non-functional mock-ups to test interface designs. Semi-functional mock-ups to try different features. Prototype code, written hastily, could help garner feedback more quickly. Anything created at this stage should be treated as disposable. The desired output of this process is not the code written, but a clearer understanding of what the right thing to build is.

3. Hire the Best Engineers You Can

The key to having good engineering is having good engineers. Google, Facebook, Amazon, Netflix, and Microsoft all run a dizzying number of the largest technology systems in the world, yet, they famously have some of the most selective interview processes while still competing fiercely to recruit the strongest candidates. There is a reason that the salaries for even fresh graduates have gone up so much as these companies have grown, and it is not because they enjoy giving away money.

Both Steve Jobs and Mark Zuckerberg have said that the best engineers are at least 10 times more productive than an average engineer. This is not because good engineers write code 10 times faster. It is because they make better decisions that save 10 times the work.

A good engineer has a better grasp of existing software they can reuse, thus minimising the parts of the system they have to build from scratch. They have a better grasp of engineering tools, automating away most of the routine aspects of their own job. Automation also means freeing up humans to work on solving unexpected errors, which the best engineers are disproportionately better at. Good engineers themselves design systems that are more robust and easier to understand by others. This has a multiplier effect, letting their colleagues build upon their work much more quickly and reliably. Overall, good engineers are so much more effective not because they produce a lot more code, but because the decisions they make save you from work you did not know could be avoided.

This also means that small teams of the best engineers can often build things faster than even very large teams of average engineers. They make good use of available open source code and sophisticated cloud services, and offload mundane tasks onto automated testing and other tools, so they can focus on the creative problem-solving aspects of the job. They rapidly test different ideas with users by prioritising key features and cutting out unimportant work. This is the central thesis of the classic book “The Mythical Man-Month”: in general, adding more software engineers does not make a project go faster, it only makes it grow bigger.

Smaller teams of good engineers will also create fewer bugs and security problems than larger teams of average engineers. Similar to writing an essay, the more authors there are, the more coding styles, assumptions, and quirks there are to reconcile in the final composite product, exposing a greater surface area for potential issues to arise. In contrast, a system built by a smaller team of good engineers will be more concise, coherent, and better understood by its creators. You cannot have security without simplicity, and simplicity is rarely the result of large-scale collaborations.

The more collaborative an engineering effort, the better the engineers need to be. Problems in an engineer’s code affect not just his work but that of his colleagues as well. In large projects, bad engineers end up creating more work for one another, as errors and poor design choices snowball to create massive issues. Big projects need to be built on solid reliable code modules in an efficient design with very clear assumptions laid out. The better your engineers, the bigger your system can get before it collapses under its own weight. This is why the most successful tech companies insist on the best talent despite their massive size. The hard limit to system complexity is not the quantity of engineering effort, but its quality.

From How to Build Good Software

Preindustrial workers worked fewer hours than today's

2019-08-22T10:18:00.001-07:00

Work brings purpose to life but it can be a great hack to keep you occupied from serious business. Work is not equivalent to production and it is not bliss. It is necessary to keep one alive and can bring happiness and purpose but it needs to be on itself purposeful and with actual tangible results.

There is an overworking and underproduction crisis. People work in bullshit works shuffling paper around and they are killing themselves while at the same time they produce nothing.

To unburden ourselves from normalcy bias we have to look at different times where things were simpler and their wasn't much space for non-producing workers.

From the paper "Preindustrial workers worked fewer hours than today's"

"The contrast between capitalist and precapitalist work patterns is most striking in respect to the working year. The medieval calendar was filled with holidays. Official -- that is, church -- holidays included not only long "vacations" at Christmas, Easter, and midsummer but also numerous saints days. These were spent both in sober churchgoing and in feasting, drinking and merrymaking. In addition to official celebrations, there were often week's worth of ales -- to mark important life events (bride ales or wake ales) as well as less momentous occasions (scot ale, lamb ale, and hock ale). All told, holiday leisure time in medieval England took up probably about one-third of the year. And the English were apparently working harder than their neighbors. The ancient regime in France is reported to have guaranteed fifty-two Sundays, ninety rest days, and thirty-eight holidays. In Spain, travelers noted that holidays totaled five months per year."

From the article " On the Phenomenon of Bullshit Jobs: A Work Rant by David Graeber "

In the year 1930, John Maynard Keynes predicted that, by century's end, technology would have advanced sufficiently that countries like Great Britain or the United States would have achieved a 15-hour work week. There's every reason to believe he was right. In technological terms, we are quite capable of this. And yet it didn't happen. Instead, technology has been marshaled, if anything, to figure out ways to make us all work more. In order to achieve this, jobs have had to be created that are, effectively, pointless. Huge swathes of people, in Europe and North America in particular, spend their entire working lives performing tasks they secretly believe do not really need to be performed. The moral and spiritual damage that comes from this situation is profound. It is a scar across our collective soul. Yet virtually no one talks about it.

Why did Keynes' promised utopia—still being eagerly awaited in the '60s—never materialise? The standard line today is that he didn't figure in the massive increase in consumerism. Given the choice between less hours and more toys and pleasures, we've collectively chosen the latter. This presents a nice morality tale, but even a moment's reflection shows it can't really be true. Yes, we have witnessed the creation of an endless variety of new jobs and industries since the '20s, but very few have anything to do with the production and distribution of sushi, iPhones, or fancy sneakers.

In Bullshit Jobs, American anthropologist David Graeber posits that the productivity benefits of automation have not led to a 15-hour workweek, but instead to "bullshit jobs": "a form of paid employment that is so completely pointless, unnecessary, or pernicious that even the employee cannot justify its existence even though, as part of the conditions of employment, the employee feels obliged to pretend that this is not the case."

The author contends that more than half of societal work is pointless, both large parts of some jobs and, as he describes, five types of entirely pointless jobs:

flunkies, who serve to make their superiors feel important, e.g., receptionists, administrative assistants, door attendants
goons, who act aggressively on behalf of their employers, e.g., lobbyists, corporate lawyers, telemarketers, public relations specialists
duct tapers, who ameliorate preventable problems, e.g., programmers repairing shoddy code, airline desk staff who calm passengers whose bags don't arrive
box tickers, who use paperwork or gestures as a proxy for action, e.g., performance managers, in-house magazine journalists, leisure coordinators
taskmasters, who manage—or create extra work for—those who don't need it, e.g., middle management, leadership professionals

From the book "Bullshit Jobs"

Identity Mappings in Deep Residual Networks, 2016

2019-08-21T01:47:00.000-07:00

A very nice improvement over the original ResNet.

In this paper, we analyze the propagation formulations behind the residual building blocks, which suggest that the forward and backward signals can be directly propagated from one block to any other block, when using identity mappings as the skip connections and after-addition activation.

Companies with overpaid CEOs have markedly underperformed the S&P 500

2019-08-20T23:09:00.000-07:00

The companies with overpaid CEOs we identified in our first report have markedly underperformed the S&P 500.

Two years ago, we analyzed how these firms’ stock price performed since we originally identified their CEOs as overpaid. We found then that the 10 companies we identified as having the most overpaid CEOs, in aggregate, underperformed the S&P 500 index by an incredible 10.5 percentage points and actually destroyed shareholder value, with a negative 5.7 percent financial return. The trend continues to hold true as we measure performance to year-end 2018. Last year, these 10 firms again, in aggregate, dramatically underperformed the S&P 500 index, this time by an embarrassing 15.6 percentage points.

In analyzing almost 4 years of returns for these 10 companies we find that they lag the S&P 500 by 14.3 percentage points, posting an overall loss in value of over 11 percent.

Consistent with our 2018 report, this year we used a two-ranking methodology to identify overpaid CEOs.
1. The first is the same HIP Investor regression we’ve used every year that computes excess CEO pay assuming such pay is related to total shareholder return (TSR).
2. The second ranking identified the companies where the most shares were voted against the CEO pay package.

These two rankings were weighted 2:1, with the regression analysis being the majority. We then excluded those CEOs whose total disclosed compensation (TDC) was in the lowest third of all the S&P 500 CEO pay packages. The full list of the 100 most overpaid CEOs using this methodology is found in Appendix A. The regression analysis of predicted and excess pay performed by HIP Investor is found in Appendix C, and its methodology is more fully explained there.

From the article
https://www.asyousow.org/report/the-100-most-overpaid-ceos-2019

Machine Learning Engineering : Tests for Infrastructure

2019-08-20T01:34:00.002-07:00

An ML system often relies on a complex pipeline rather than a single running binary.

Engineering checklist:

Test the reproducibility of training
Unit test model specification code
Integration test the full ML pipeline
Test model quality before attempting to serve it
Test that a single example or training batch can be sent to the model
Test models via a canary process before they enter production serving environments
Test how quickly and safely a model can be rolled back to a previous serving version

1. Test the reproducibility of training. Train two models on the same data, and observe any differences in aggregate metrics, sliced metrics, or example-by-example predictions. Large differences due to non-determinism can exacerbate debugging and troubleshooting.

2. Unit test model specification code. Although model specifications may seem like “configuration”, such files can have bugs and need to be tested. Useful assertions include testing that training results in decreased loss and that a model can restore from a checkpoint after a mid-training job crash.

3. Integration test the full ML pipeline. A good integration test runs all the way from original data sources, through feature creation, to training, and to serving. An integration test should run both continuously as well as with new releases of models or servers, in order to catch problems well before
they reach production.

4. Test model quality before attempting to serve it. Useful tests include testing against data with known correct outputs and validating the aggregate quality, as well as comparing predictions to a previous version of the model.

5. Test that a single example or training batch can be sent to the model, and changes to internal state can be observed from training through to prediction. Observing internal state on small amounts of data is a useful debugging strategy for issues like numerical instability.

6. Test models via a canary process before they enter production serving environments. Modeling code can change more frequently than serving code, so there is a danger that an older serving system will not be able to serve a model trained from newer code. This includes testing that a model can be loaded into the production serving binaries and perform inference on production input data at all. It also includes a canary process, in which a new version is tested on a small trickle of live data.

7. Test how quickly and safely a model can be rolled back to a previous serving version. A model “roll back” procedure is useful in cases where upstream issues might result in unexpected changes to model quality. Being able to quickly revert to a previous known-good state is as crucial with ML models as with any other aspect of a serving system.

* From "What’s your ML Test Score? A rubric for ML production systems" NIPS, 2016

Machine Learning Engineering : Tests for Model Development

2019-08-14T22:15:00.003-07:00

While the field of software engineering has developed a full range of best practices for developing reliable software systems, the set of standards and practices for developing ML models in a rigorous fashion is still developing. It can be all too tempting to rely on a single-number summary metric to judge performance, perhaps masking subtle areas of unreliability. Careful testing is needed to search for potential lurking issues.

Engineering checklist:

Test that every model specification undergoes a code review and is checked in to a repository
Test the relationship between offline proxy metrics and the actual impact metrics
Test the impact of each tunable hyperparameter
Test the effect of model staleness. Concept drift is real for non stationary processes
Test against a simpler model as a baseline
Test model quality on important data slices
Test the model for implicit bias

1. Test that every model specification undergoes a code review and is checked in to a repository.
It can be tempting to avoid, but disciplined code review remains an excellent method for avoiding
silly errors and for enabling more efficient incident response and debugging.

2. Test the relationship between offline proxy metrics and the actual impact metrics. For exam-
ple, how does a one-percent improvement in accuracy or AUC translate into effects on metrics of
user satisfaction, such as click through rates? This can be measured in a small scale A/B experiment
using an intentionally degraded model.

3. Test the impact of each tunable hyperparameter. Methods such as a grid search or a more
sophisticated hyperparameter search strategy not only improve predictive performance, but also
can uncover hidden reliability issues. For example, it can be surprising to observe the impact of
massive increases in data parallelism on model accuracy.

4. Test the effect of model staleness. If predictions are based on a model trained yesterday versus
last week versus last year, what is the impact on the live metrics of interest? All models need to be
updated eventually to account for changes in the external world; a careful assessment is important to
guide such decisions.

5. Test against a simpler model as a baseline. Regularly testing against a very simple baseline
model, such as a linear model with very few features, is an effective strategy both for confirming
the functionality of the larger pipeline and for helping to assess the cost to benefit tradeoffs of more
sophisticated techniques.

6. Test model quality on important data slices. Slicing a data set along certain dimensions of
interest provides fine-grained understanding of model performance. For example, important slices
might be users by country or movies by genre. Examining sliced data avoids having fine-grained
performance issues masked by a global summary metric.

7. Test the model for implicit bias. This may be viewed as an extension of examining important data
slices, and may reveal issues that can be root-caused and addressed. For example, implicit bias might
be induced by a lack of sufficient diversity in the training data.

* From "What’s your ML Test Score? A rubric for ML production systems" NIPS, 2016

Machine Learning Engineering : Tests for Features and Data

2019-08-14T04:10:00.004-07:00

Machine learning systems differ from traditional software-based systems in that the behavior of ML systems is not specified directly in code but is learned from data. Therefore, while traditional software can rely on unit tests and integration tests of the code, here we attempt to add a sufficient set of tests of the data.

Engineering checklist:

Test that the distributions of each feature match your expectations
Test the relationship between each feature and the target
Test the cost of each feature
Test that a model does not contain any unsuitable for use feature
Test that your system maintains privacy controls across its entire data pipeline
Test all code that creates input features

1. Test that the distributions of each feature match your expectations. One example might be to test that Feature A takes on values 1 to 5, or that the two most common values of Feature B are "Harry" and "Potter" and they account for 10% of all values. This test can fail due to real external changes, which may require changes in your model.

2. Test the relationship between each feature and the target, and the pairwise correlations between individual signals. It is important to have a thorough understanding of the individual features used in a given model; this is a minimal set of tests, more exploration may be needed to develop a full understanding. These tests may be run by computing correlation coefficients, by training models with one or two features, or by training a set of models that each have one of k features individually removed.

3. Test the cost of each feature. The costs of a feature may include added inference latency and RAM usage, more upstream data dependencies, and additional expected instability incurred by relying on that feature. Consider whether this cost is worth paying when traded off against the provided improvement in model quality.

4. Test that a model does not contain any features that have been manually determined as unsuitable for use. A feature might be unsuitable when it’s been discovered to be unreliable, overly expensive, etc. Tests are needed to ensure that such features are not accidentally included (e.g. via copy-paste) into new models.

5. Test that your system maintains privacy controls across its entire data pipeline. While strict access control is typically maintained on raw data, ML systems often export and transform that data during training. Test to ensure that access control is appropriately restricted across the entire pipeline. Test the calendar time needed to develop and add a new feature to the production model. The faster a team can go from a feature idea to it running in production, the faster it can both improve the system and respond to external changes.

6. Test all code that creates input features, both in training and serving. It can be tempting to believe feature creation code is simple enough to not need unit tests, but this code is crucial for correct behavior and so its continued quality is vital.

* From "What’s your ML Test Score? A rubric for ML production systems" NIPS, 2016

What is deep learning ?

2019-08-06T20:59:00.002-07:00

Long Story Short

I like this definition, clean, consise and to the point with zero marketing fluff.

“A class of parametrized non-linear representations encoding appropriate domain knowledge (invariance and stationarity) that can be (massively) optimized efficiently using stochastic gradient descent”

What neural networks actually do ?

Inverse Problems

2019-08-05T05:12:00.001-07:00

Challenges when trying to solve inverse problems:

They may not have a unique solution.
Often, small changes in data cause arbitrarily large changes in solution.

Definition: (Hadamard, 1902)

If the solution of a problem is not unique, or it does not depend continuously on the data, then the problem is ill-posed.

Many inverse problems are also ill-posed.
For realistic problems (containing noise), we cannot hope to compute an exact solution.

Goal: Compute a meaningful approximation.
Approach: Use regularization.

Sigsoftmax: Reanalysis of the Softmax Bottleneck

2019-06-24T00:12:00.000-07:00

This paper was kind of overlooked last year (even thought it showed up on NIPS) but it caught my attention.

The paper suggests replacing the usual softmax output layer for classification (the paper makes this suggestions for NLP problems but the theory stands for all domains with large number of classes) because its very design produces an information bottleneck.

I have yet to see this applied to computer vision problems but I will implement it in Tensorflow or Caffe and report back.

It's essence boils down to the following excerpts :

The paper then goes to suggest the properties of a replacement activation layer:

and the final outcome is this:

which satisfied the properties defined earlier