This is not meant as a comprehensive guide, rather just a place to start and get things up and running. Read further below for a background and explanation.

1. Create a custom callback

## Tracking Classfrom mlflow.tracking import MlflowClientfrom mlflow.entities.run import Runfrom typing import Listclass MLFlowTracking(Callback):    "A `LearnerCallback` that tracks the loss and other metrics into MLFlow"    def __init__(self, metric_names:List[str], client:MlflowClient, run_id:Run):        self.client = client        self.run_id = run_id        self.metric_names = metric_names                def after_epoch(self):        "Compare the last value to the best up to now"        for metric_name in self.metric_names:            m_idx = list(self.recorder.metric_names[1:]).index(metric_name)            if len(self.recorder.values) > 0:                val = self.recorder.values[-1][m_idx]                self.client.log_metric(self.run_id, metric_name, np.float(val))

2. Initialize the MLflow Experiment, Client and Run

# Make sure to install mlflow using pip, conda or your favourite package managerimport mlflowfrom mlflow.tracking import MlflowClientTRACKING_URL = "<full url, e.g: http://example.com:5000>"EXPERIMENT_NAME = "example_experiment"# Create the Clientmlfclient = mlflow.tracking.MlflowClient(tracking_uri= TRACKING_URL)# Check if the experiment already exists, or create itmlfexp = mlfclient.get_experiment_by_name(EXPERIMENT_NAME)if mlfexp is None:    mlfexp_id = mlfclient.create_experiment(EXPERIMENT_NAME)    mlfexp = mlfclient.get_experiment_by_name(EXPERIMENT_NAME)    mlrun = mlfclient.create_run(experiment_id= mlfexp.experiment_id)

3. Log the experiment’s hyperparameters

# setup up a params dictionaryparams = {    'crop_size': 250,    'epochs': 20}# log each of the params items to MLflow (for that experiment's run)for k, v in params.items():    mlfclient.log_param(run_id=mlrun.info.run_uuid, key=k, value=v)    # Make sure to use params in your data loader and experiments

4. Add the custom callback to the learner

# Assuming dataloaders is already setup learn = cnn_learner(dls, resnet18, metrics=error_rate, cbs=[MLFlowTracking(metric_names=['valid_loss', 'train_loss', 'error_rate'], client=mlfclient, run_id=mlrun.info.run_uuid)])# This should start logging the metrics and losses in your environment as well as MLflowlearn.fit(params['epochs'])

FastAI is a powerful library with a set of good features to make ML fun. One of the things I was struggling with is losing track of my experiment’s configuration and to the metrics and losses they generated.

MLflow seemed like a good fit at least for what I was trying to accomplish. Most documentations around setting up MLflow and fastai are stale (covering v1). MLflow does have fast.ai autolog feature but, as per the documentations (and trials), looks like it works only for version fast.ai 1.0.61 or earlier.

The above setup, although very crude, works for the simple setup I have. It assumes access to an MLflow server with accessible URI (TRACKING_URI). One caveat is that metric values tend to be empty for the first couple of epochs and, with more understanding of fastai Callbacks and recorder, I might be able to track down.

Happy Deep Learning

## Tracking Classfrom mlflow.tracking 
import MlflowClient from mlflow.entities.run 
import Run from typing 
import List

class MLFlowTracking(Callback):
  "A `LearnerCallback` that tracks the loss and other metrics into MLFlow"
  
  def __init__(self, metric_names:List[str], client:MlflowClient, run_id:Run):
    self.client = client
    self.run_id = run_id
    self.metric_names = metric_names
    
  def after_epoch(self):
    "Compare the last value to the best up to now" 
    for metric_name in self.metric_names: 
      m_idx = list(self.recorder.metric_names[1:]).index(metric_name) 
      if len(self.recorder.values) > 0: 
        val = self.recorder.values[-1][m_idx]
        self.client.log_metric(self.run_id, metric_name, np.float(val))

How to make this more interesting and engaging? Simple. Make it in 3D. Animated. Interactive.

The above is a data pipeline animation for one of my recent projects.

I started out looking into ways, beyond just visio, diagrams or graphviz, that would demonstrate the data flow in a simple and interesting way. Animation was a natural choice, yet the available toolsets had an associated steep learning curve.

I then came across three.js which seemed like an achiveable setup and medium. However, coding all three.js objects by hand felt like an endless task.

After a bit of digging I came across spline.design tool – an awesome 3D and Animation editor, for the web, that leverages three.js and anime.js (do check it out).

Some feedback I got:  “This looks cool”, “I like it”, “It’s like watching a domino chain project”.

The article portrays the opposing views held by data science and information system departments and how each side view their responsibilities:  “Data scientists see only throw-away software that will be discarded once the breakthrough insights have been actioned by management. The [information systems] department sees only thoroughly designed and tested software that will be used in production-quality applications.”

While I acknowledge the challenges, I don’t think that pinning data science against information system (technology services) departments, or as presented, sustainability against flexibility, tells the full story.

I personally can’t imagine a situation in which data scientists at Netflix, YouTube or Tesla are blamed for not delivering on sustainability or for their ‘information systems’ counterparts are blamed for being rigid.

Data science projects

In a nutshell, data science projects are in essence, solutions development projects. Data scientists use data, tools and code to deliver a digital solution. Those solutions are, in turn, used to help make better business decisions or optimize certain functions.

They can be as simple as a stand-alone script that generates frequent reports. Or as complex as a multi-year, multi-million dollar solution intended to create a self-driving vehicle.

From a personnel perspective, data science projects are cross functional and involve multiple roles. On a smaller scale, those roles can be assumed by a single person (a data scientist/unicorn?). On a larger scale, it’s a cross functional team with business owners, product owners, data scientists, front-end developers, back-end developers, data engineers, and more.

From a solutions perspective, data science project results and outcomes must be seamlessly embedded within business workflows. Data science project results that are not easily used in making business decisions remain largely experimental and gimmicky.

Technology Services

So – why does the introduction or growth of data science practices feel challenging? Why do data scientists feel constrained and perhaps isolated with push back from their technology services departments?

In my opinion, this might just be a secondary symptom of how less-equipped technology services departments have become.

Many technology services departments have chosen, or were forced, to become cost centers that are responsible for commodity hardware or off-the-shelf services, not the business partners needed to move the business forward.

In the process of standardizing their services, some might have shed their technical talent in favor of managerial or business one.

Their ability to develop and implement technical solutions ended up with third parties at the expense of adaptability and agility and with costs that far exceed the effort.

In this type of environment, data scientists with their need to develop technical solutions or integrate their work in the larger technological landscape, face repeated pushbacks.

Access to complementing tools or talent become entangled in process and budgetary discussions, and simple technical requests become missions to mars. A ripe environment for Shadow IT.

Data Science within Non-Tech Organizations

I would like to think of the challenge as an organizational responsibility rather than an A-against-B argument.

Organizations serious about adopting data science as a competitive advantage, or even as an investment attractor, should also consider themselves as technology creators rather than just technology consumers.

Some of the things that organizations can do to promote and encourage the development of technical solutions (rather than ‘data-science projects’) are:

• Promote and encourage ways to properly deliver digital solutions, rather than just isolated ‘data-science projects’.
• Embrace principles that value people, collaboration, flexibility and working solutions.
• Adopt processes to manage knowledge projects as well as minimize the risk during delivery.
• Implement methodologies to better understand the target [internal] market and measure traction.
• Develop technical capabilities to improve their overall competitiveness and offer services or products unmatched by their peers.

For many years, those principles, strategies, processes and methodologies have been battle-tested at technically inclined organizations. What it took is a realization, direction, learning and will.

Acknowledgment

Thanks to Brett Wiens and Marc Boulet for the first read and feedback!

I’ve been lately seeing a lot of professionals (Engineers, Geologists, Accountants, Analysts, etc…) venturing into data analytics and have consequently been working with software and various technical components that proved challenging for some. I was motivated to provide some guidance around the main components used in solutions development, the associated development processes and methodologies, as well as some common trade tools.

I gave the talk below at the Untapped Energy June 2019 meetup and very much liked the energy of the crowd during the Q&A. I hope you don’t mind a cold start.

I started out this blog in 2004, back when blogging was the social media of the day. Inspired by professionals who I admired, I tried to track my own professional and personal journey.

However, in 2016 I failed to renew my hosting plan which took down all the website’s content as well as my motivation to create new ones. There were no offsite backups of the website (Yes I know). I resorted to twitter for short-form publishing and consumption and reviving the website was put on the back-burner.

I now feel the urge to create medium and long-form content. I also feel what I published from 2004 onwards speak to who I am professionally and personally.

Over the 2019 Christmas holidays I spent a bit of time in putting a publishing mechanism in place using Jekyll, and migrating the contents over using WayBack Machine’s snapshots.

So here we are, the new and improved tamersalama.com with [most of] the original contents.

Happy new year.

1. Add the SVPullToRefresh pod
2. Add the Quartz Framework

# RakefileMotion::Project::App.setup do |app| 	# … app.frameworks += [‘QuartzCore’]	app.pods do pod ‘SVPullToRefresh’ # … endend # Then in your controller def viewDidLoad 	# …. tableView.addPullToRefreshWithActionHandler    Proc.new {        loadMagicalData(tableView)        # do some other magic        # then don’t forget to:        tableView.pullToRefreshView.stopAnimating    }# …

Installation:
In your RubyMotion application’s Rakefile: {% highlight ruby %} Motion::Project::App.setup do |app| #… app.pods do # … pod ‘NUI’ end end {% endhighlight %} Then run `

“ `

Usage:
Create a theme.nss file (which will include your application’s stylesheet). Then in your “{% highlight ruby %} class AppDelegate

@@nui_settings = NUISettings.init @@nui_settings.setAutoUpdatePath(“/absolute/path/to/project/and/theme.nss”)

def application(application, didFinishLaunchingWithOptions:launchOptions)

12@nui = NUIAppearance.init# ...   end

end {% endhighlight %}

Run `

` and check your application in the simulator. Now, any changes to the <ts_code></ts_code> file will be reflected right away to the running application in the simulator. Happy styling

Applications are grown.[1]

There’s quite the range of differences between building an application and growing one, but I’ll focus here on few main ones.

Building something is associated with a resulting rigid structure and components. It takes effort to put together, but once build – it’s static. All what’s left is usage, admiration & maintenance.

On the other hand, growing something is an organic process. It rarely stops. It requires a continuous labour of love. You can reap benefits but as you do, you always have to give back, care and nurture.

Growing is more appropriate to the act of developing software. Applications doesn’t (or shouldn’t) reach a static rigid state, they always benefit from continuous assessment, reevaluation and changes. They adjust to surroundings. They evolve, transform and get bigger (but not necessarily more complex).

So – next time you’re discussing new applications – try to see how close the word ‘growing’ is to everyone.

[1] I wish I could remember where first I heard that or who said it.

Migrated Comment – Steven: That’s a good way of putting it. I’m currently in the process of “weeding” an application.

Migrated Comment – Tamer Salama:
@steven Weed: verb ?” /w?d/ ”
Remove an inferior or unwanted component of a group or collection

I bet that’s exactly what you’re doing

class Well < ActiveRecord::Base	has_many :survey_pointsendclass SurveyPoint < ActiveRecord::Base	belongs_to :wellend

Say, we’d like to implement a filtration mechanism that allows us to query the wells – and for those set of wells we’d like to conditionally present the set of survey_points for those set of filtered wells.

A method find_for_wells would returns the set of wells we’re interested in ( “ )

Using the power of AREL chaining, we could then conditionally extract the SurveyPoints using the following:

def SurveyPoint.find_for_wells(wells)	scope.where(:well_id => wells.select(:id))end 

This roughly translates into the query:

SELECT * FROM SURVEY_POINTS WHERE WELL_ID IN (SELECT ID FROM WELLS WHERE ….)

The challenge here is that if both results is large, depending on the cardinality – the database engine might not be able to make use of defined indexes and the outer query might not be optimizable.

Instead – if we use pluck instead of select:

def self.find_for_wells(wells)	scope.where(:well_id => wells.pluck(:id))end 

Rails would fire two queries instead of an inner one. Pluck would first extract the list of ids (val_1, val_2, …) in an array before passing it to the main query. This roughly translates into:

SELECT * FROM SURVEY_POINTS WHERE WELL_ID IN (val_1, val_2, val_3, ….) 

No inner query is used in that case and the main one is easy to optimize.