Fruch's Blog

Taking Github Actions for a walk in the park

2020-12-29T00:00:00+00:00

Taking Github Actions for a walk in the park

Travis started to scare me a bit

People were talking about github actions for a while now, lot of hype around it. I myself was quite happy with the things I’ve managed to get done with Travis.

But then came in this announcement from travis: “The new pricing model for travis-ci.com”. I was reading via lot of the backlash in tweeter about it, and after carefully reading the announcement itself, I was starting to think that the work I did for scylla-driver was counting on Travis too much.

Recently I’ve start seeing long queues for job triggers in travis, and I was starting to have second thoughts about picking travis. (not so long ago, less than a year ago)

The sweet setup I had in Travis

As a recap for scylla-driver, since it’s a c based package, heavily using cython. We are building python wheels for windows, mac, linux, ranging from python2.7 to python3.9, and pypy. (also using the Travis experimental aarch64 and ppc64le support)

For building the wheels we are using cibuildwheel, which is a nifty tool that magically keep all know how, on how building wheels correctly on multiple platforms. In linux, it’s using the official docker images provided by the [Python Packaging Authority ] for manylinux2014. And on windows and mac equivalents with the correct set to needed tool to get you wheel happily containing all the need dynamic libraries it depended on

cibuildwheel also comes with readily available examples for all the major CI system out there

cibuildwheel is also helping you in running unittest ontop of each built wheel (since in my case the setup is a bit complex, getting it installed and compile across the board was hard, we we’re skipping some unittest on some platforms)

Travis was also running our integration test suite, and running it on our PRs, and automatically uploading all the wheels and source distribution to Pypi.

I was working on this setup for long week, with a tight deadline for it to be ready for europython2020, thanks to @ultrabug

Github actions

For a while I was a bit unsure if it would be a match to what I had in travis. But one day our own Jenkins server was down for maintenance, and a big portion of my day was clear case of that.

I’ve started with copy-pasting cibuildwheel github actions example, and worked my way from there.

When I’ve started adding all the different unittest and platforms, I was struggling a bit with how the matrix is working, and how conditional steps works.

It ended up something like this, almost identical to what I had in travis:

name: Build and upload to PyPi

on: [push, pull_request]

env:
  ...

jobs:
  build_wheels:
    name: Build wheels $ ($)
    if: contains(github.event.pull_request.labels.*.name, 'test-build') || github.event_name == 'push' && endsWith(github.event.ref, 'scylla')
    runs-on: $
    strategy:
      fail-fast: false
      matrix:
        include:
          - os: ubuntu-18.04
            platform: x86_64

          - os: ubuntu-18.04
            platform: i686

          - os: ubuntu-18.04
            platform: PyPy

          - os: windows-latest
            platform: win32

          - os: windows-latest
            platform: win64

          - os: windows-latest
            platform: PyPy

          - os: macos-latest
            platform: all

          - os: macos-latest
            platform: PyPy

We had a nice twist that we didn’t have before, but now we can control from github PR which part of my pipeline would run using label, since github actions is so tightly integrated you can lookup almost anything related to the PR.

Github Actions - UI

This is how the final thing looks like:

logging is a bit weird to watch while it’s running, but after the fact the log are very tidy to look at

Late surprise - cross compile using qemu

With almost perfect timing I’ve run into this https://github.com/joerick/cibuildwheel/pull/482 by Pavel Savchenko

I was trying all kind of things to get aarch64 support, but this PR figure it out quite nicely, with one magic step:

- name: Set up QEMU
id: qemu
uses: docker/setup-qemu-action@v1
with:
    platforms: all

For reference the full workflow

Now we can have docker using qemu to run docker instances based on different cpu architecture, it’s horribly slow but it’s working.

And nice side effect I’ve learned how to run it locally, so I can debug those things. Something I haven’t figured when running with Travis, apparently travis was using the same trick on their experimental support for those architectures.

Summary

Github actions works out of the box, with almost zero fraction, almost anything I could think about I’ve found a readily available step I can pickup, very impressing.

One thing I haven’t yet tried out, is using self hosted runner, hopefully I won’t need it. Cause who want to babysit anther server on his own, or worse, a bunch of them.

How much fun was EuroPython 2020

2020-07-29T00:00:00+00:00

How much fun was EuroPython 2020

#pyconil got canceled

This year I’ve finally got enough courage and will, and I had 2 submissions for #pyconil. COVID-19 had other plans, and #pyconil was canceled

I’ve told @ultrabug about this (Numberly CTO, Alexys Jacob), after a few weeks he surprised me with telling me he’s gonna present scylla-driver in europython2020, the shard-aware driver we were working on in the last 6 months.

At the time it wasn’t yet ready nor publish. (Also found out that Numberly were sponsoring europython for years now) Took me a few seconds to figure that he just set me deadline without my consent…

Fast-forwarding a bit, and scylla-driver initial release came out: https://pypi.org/project/scylla-driver/3.21.0/

And my tickets for europython2020 were booked…

Discord is fun

Few days before the date, I’ve got an email with instruction to connect to the discord of europython2020, since it’s my first COVID-19 online conference, I’ve registered and login straight away.

It was really nice to be start to start talking with people and meet them a few days before

Each track got its own chat room, backed with zoom webinar and online youtube stream each talk got its own chat room, so people can promote their and answer question before and after the talks. Sponsors had rooms too, and all the sprints had rooms (we’ll get to that later on)

Talk about timing, day before europython2020, this press release came out https://www.scylladb.com/press-release/discord-chooses-scylla-core-storage-layer/

Talks — Day 1

The opening keynote was cancelled cause of technical difficulties, that lost touch with whom was supposed to talk. (it rescheduled to next day)

Elias Mistler was talking about “How to write multi-paradigm code” show casing and coding live a sudoku solver, with functional and object originated approaches, and how to mix them both. The code is available as notebook
Bojan Miletic was talking about “Django Testing on Steroid: pytest + Hypothesis” — shows the basic usage of hypothesis, I’m hear about hypothesis all over the place, maybe it will bring me one step close to use is in one of my projects
Martin Christen gave a very cool demo of “pyRT - Computer Graphics in Jupyter Notebooks for Fun and Teaching” - I love Jupyter notebook visual trick, and I love 3d. So this one I enjoyed a lot, Martin was quite a performer and funny. I strongly recommend trying out the demo
Bernat Gabor gave an excellent talk “Lessons from the Trenches: rewriting and re-releasing virtualenv” about a big refactoring virtualenv had, packaging is a subject close to my heart. This is an excellent lesson on how to approach major refactor and a reminder that it would always take more time than you’ll estimate it. I must say that this change, from a POV of a heavy user of virtualenv, wasn’t even noticed. When he was demoing a few things I’ve realized that I’ve seen it in logs, but didn’t know the big story happing behind the scenes.
In the next one, Philip Jones was building a ASGI server from scratch, show the details of the ASGI protocol — “An ASGI Server from scratch”
“A deep dive and comparison of Python drivers for Cassandra and Scylla” — Alexys great slides (which I had a peek at few days before)with lot of emojis, going down to the details of how cassandra/scylla hash ring works, and how scylla shard share nothing architecture is use by the scylla-driver fork we work on for the last few months. And he finished with data from Numberly production before/after moving to shard aware client, 2x-2.5x speed up in response time I was live tweeting the whole thing into our company slack…
“Elegant Exception Handling”
“Best practices for production-ready Docker packaging”
“Pluggable Architecture”
“Writing Zenlike Python”

Social — the missing `e` in my whisky

During the whole day I was playing cat and mouse with people, trying to bring them into the open track, to have a face to face meeting, only later after all the sessions end, some people came in, most of them were the part of the organizing team. Every one where show their beer or whisky that they were drinking. Keith Gaughan was laughing at the ice in my whisky, and also thought me a real whisky is spelled with whiskey with an e in it.

Marc-Andre Lemburg was talking about the challenges they have as organizers, one thing led to anther, and I ask how can I help. He said they need a hand with handling the tweeter account, and that he’ll hook me tomorrow with whom handle it.

After a few more rounds of whisky, I call it a day.

Talks — Day 2

TODO:

Sprints

packaging -
hypothesis -
terminusdb-client -

My DevopsDaysTLV Summary

2019-12-22T00:00:00+00:00

My DevopsDaysTLV Summary

Late birds

Sunday night my boss asks me if I want to come to #DevopsDaysTLV, reading the program, seems like intersting talks, but no ticket left for the first day. Once we got the tickets, I know what’s Early Bird tickets, now I’ve found out there’s Late Birds tickets.

Rockstar saves the day

Gil Tayar wrote on tweeter that he’s gonna fill in for someone who could make it, since she cought the flu. I was publicly devestanded that it’s gonna be on the first day, and I won’t get to meet Gil face to face.

My DM flahses Gil supprises me with giving me his +1 ticket, I title him as a rock-star, not even gussing what to come.

Day One

Gil picks up my tag, while I get lost a bit in the convention center. Following other lost nerds, got me walking in circles around the whole place.

Didn’t know too many people, Devops isn’t excatly my field (and beyond pycon.il I don’t attend too many confrences). I was supprised to see Anna from ElasticSearch too, didn’t know she’ll be there. I was really existed to meet her face to face, and not only via zoom chat.

Picking some stickers and swag, and then keynote started.

Dr. Nicole Forsfren was talking about the 2019 State of DevOps Report, it’s was nice to see that she has numbers to proof Devops is working very good for some companies, and the trend of compenies excecuting faster is growing.
Then I’ve heard Omer Lev Hevroni, tells his tale, on how they had a leakge of cerdentionals, and the joureny and tools they used to reduce the chances it would happen again. (One more tweeter figure, that become a bit more real for me, that day)
And then Gil talk about #egoprogramofobia, really proving he’s a rockstar, making the crowd sing along with him while he trys to drive fear of own code out of people.
Coming up next, I’ve heard Shlomi Noach, from github. Explianing the CAP Theorem and extending it a bit to try to give it some more reasnable close to real-life interpertention (maybe now I’ll rememeber all what each letter stands for).
Food break
Regev Golan was talking on what you need to consider while depolying an app that uses GPUs, best practicates and thing to consider while deploying.
Erik Zaadi & Dainel Korn, made a great talk about the lesson we should learn from Rick & Morty, it was really funny. Erik dressed a pickle, was a sight not easily forgotten.

And then I had to split, to take my someone of kindergarden…

Day Two

While I’m waltzing in that morning, I’ve met Maish in his new role in the AWS booth, he was talking with Florian. Isreali of me I’ve started talking hebrew and cut thier converstaion in the middle. Took me a few moments to figure it out and appologise. Few words with Florian, and I’ve decided I defentliy goona enter his talk.

My late bird boss joined in, coming a bit injerd.

Brian Murphy from GeekSeat was giving the keynote, expliaing his approch to taking decietion about new tech, and how RFC process is importent to every team.
Florian Haas — Communication for distributed teams talk was spot one. He was talking about the missuse of diffrent communication medium while my boss show me life examples on his phone for everything Florian is talking about.
Philip Krenn from elastic, packed the room, and I was standing hearing more the logistics of handling a small flood in the side of the room. I guess he’s talk was intersting enough for everyone to squize in.
Anna Bankirer was talking about object oriented desgin. It was nice but I would have jumped stright into examples, and minize the talk intro about OOD/OOP, yeah I know the theam was “back to fundmentals”.
Elazar Leibovich — talked about docker networking, and how to share a port from a running docker instance without a restart, how exactly docker networking is working is always a puzzle to people, me included. Just a week ago I’ve myself in need of a white board to explien to someone how docker instances can communicate internall, and why a host port isn’t needed for that.
Ronen Botzer from Aerospike, was getting pissed at compenies that doing a grabge benchmarks. At least he was mentioning us (scylladb), and he told me at the first day he’s gonna talk about it
Robert Barron was closing the session with a great talk about the Appllo Missions to the Moon, while demostanting that almost the pinciples we know in devops were used in those missions, Kohelet was right while saying “nothing new under the sun”.

Beers and Sufganiut for finising up.

I enjoyed it very much…

Introducing pytest-elk-reporter

2019-06-24T00:00:00+00:00

pytest-elk-reporter

tl;td: https://github.com/fruch/pytest-elk-reporter

History

Few years back I’ve wrote a post about how I’ve connected python based test to ELK setup - “ELK is fun”, it was using an xunit xml, parsing it and sending it via Logstash.

Over time I’ve learn a lot about ElasticSearch and it’s friend Kibana, using them as a tool to handle logs. and also as a backend for a search component on my previous job.

So now I know logstash isn’t needed for reporting test result, posting straight into elasticsearch is easier and gives you better control, ES is doing anything “automagiclly” anyhow nowadays.

What can it do ?

collect as much information as you can about one test results and running environment and send it to elasticsearch.
- data like git commit/sha from current directory
- jenkins environment variables, like job id and username triggered the job.
- cool tracebacks which pytest excels at.
extend - let user configure (like target address and credentials) it from code, and add data of their own to each session or test.

Tricks I’ve picked on the way

Every time I start a new package, I learn a few new things, here’s a few I’m happy about this time:

Created the package with cookiecutter-pytest-plugin
I’m using pre-commit
- black - instead of the old friend autopep8.
- pylint in both python27 and python36 to keep the code clean of silly mistakes.
Travis

build and test with Tox on all possible python versions.

also uploading wheels to pypi based on tags.

thanks to setuptools_scm version no need to edit files for setting version number.
Appveyor

I also cover windows own little issue (non found so far).
Codecov

really like that it can collect from multiple run, for example both linux and windows tests run, which helps to show the total coverage.

got me to 93% line coverage (and going down as I add more code).

The bad stuff - a.k.a things I need to improve

Documentation - I’ve mostly wrote test, zero docs.
Name - it’s a bit boring, and might conflict with other plugins.
Kibana Dashboards examples

I should really include some sample kibana dashboard to show the strength of it, it’s own test data is a bit boring.
auto generating and upload of docs

I’ve tried using Travis Deploy to github Pages, didn’t exactly nailed, or got it working as I expect, maybe I should try readthedocs.
pytest-xdist

I still need to test it with xdist, it’s going to make thing a more tricker, as usual.

What you should next

Go grab it, git it a star, fork it.

I’m waiting for your PRs, and your stories on how it’s help you tame a whole bunch of tests.

    # forgot to show how to install :\
    pip install pytest-elk-reporter

Microsoft - amazing stuff, WSL, VSCode

2017-12-07T00:00:00+00:00

WSL

I’ve took WSL for a spin today, and I’m impressed.

# doing something like this
$ apt install docker.io

Actully worked, i.e. installed. you can’t start the dockerd as failed as in #2291

But you can use the docker client, and connect a server running dockerd (or in win10 pro the native docker supported)

The direction is quite good, I can have tmux and my favorite nano working stright on my windows box like that.

Still didn’t figured out how to leave things running, like my tmux session (if at all possible)

VSCode

Anthoer supprise from microsoft (I guess it’s not that new, just new to me), is VSCode

I hated visual studio for year, and this one is fully open source based on node.js, resembles more atom.

List of goodies I have with it:

markdown previews
spellchecker
python pylint - based on my config files, and not as in Pycharm own code style checks.
node linting - again working with my config (I guess WebStorm has it too, but it’s not free)
plantuml preview - amazing for documention
terminal can use gitbash inside of it

The marketplace of plugings, is also quite nice, and sits in a very front and center place (looking at you jetbrains, suggestion is nice, but getting to the list of plugin is a bit long)

The updates are a bit more plesent then in Jetbrain products, and comes with great webcast on each major feature. (giving us a very good lesson on writing release notes :))

And also people are pushing to have it running under WSL #2293

Conclusion

I see a very bright future for microsoft steping into the core thing that the development community are looking at.

Microsoft, i’m keeping my ears and eyes open…

You got my attention !

KISS - Keep it shorter short

2016-07-08T00:00:00+00:00

Today on work on a python related mailing list someone sent this snippet of code he didn’t wrote. and that was causing some of the server he worte/manged some troubles.

Here’s the code:

    def deploy_file(full_path_to_file, full_path_in_server):
        if full_path_to_file is None or full_path_in_server is None or full_path_to_file == "" or full_path_in_server == "":
            print_error_exit(" wrong input for deploy_file")
        try:
            post_status = None
            with open(full_path_to_file, mode="rb") as f:
                print ("Deploying...:  %s" % full_path_in_server)
                file_content = f.read()
                post_status = requests.put(url= full_path_in_server, data=file_content)
                if post_status.status_code == 200 or post_status.status_code == 201 or post_status.status_code == 202:
                    print ("Success:  %s" % full_path_in_server)
                    return {"uploaded": file_to_upload }
                else:
                    print_error_exit ("Deployment to %s failed. status_code = %s, returned_text = %s" % (full_path_in_server, post_status.status_code, post_status.text))
        except Exception as ex:
            print (ex)
            print_error_exit ("Posting with url %s failed. status: %s" % (full_path_in_server, post_status))

First thought I hade was, the one wrote it is must be coming from C (or maybe Java), so let go over what’s wrong here.

Input validation

I don’t know what’s print_error_exit() is doing exactly, let’s assume it’s raising expection (hopefully it’s not calling os.exit())
So assertion with the proper text would be alsmot identical.

Checking both for None, and for empty string is futile, just assert, like this:

        assert full_path_to_file, "parameter can't be empty"
        assert full_path_in_server, "parameter can't be empty"

Using requests to stream file

Requests is cool, alsmot the defacto standard when HTTP client is need in python. it has very good documention, clearly someone didn’t read it.streaming uploads
Multiple error paths

Lot of different error handling code, one for checking the http statuses and one for catching expections.

My version

    def deploy_file(full_path_to_file, full_path_in_server):
        assert full_path_to_file, "parameter can't be empty"
        assert full_path_in_server, "parameter can't be empty"

        res = None
        try:
            with open(full_path_to_file, mode="rb") as f:
                logging.info("Deploying...:  %s", full_path_in_server)

                res = requests.put(url= full_path_in_server, data=f)

                assert res.status_code in [200, 201, 202]
                logging.info("Success:  %s", full_path_in_server)

                return {"uploaded": file_to_upload }

        except Exception as ex:
            logging.expection("deploy_file faild") # so we'll have a nice print in the log

            # casue I don't know what print_error_exit() does, we'll call it.
            # even that I would prefer to just raise the expection up, i.e. "raise ex"
            print_error_exit ("Deployment to %s failed. status_code = %s, returned_text = %s" % 
                (full_path_in_server, res.status_code if res else None, res.text if res else None))

Microsoft + Docker, really ?

2014-11-09T00:00:00+00:00

Today this blog post of Scott Guthrie from Microsoft was slash-dotted. Interesting enough, I had no idea one can actually run Docker on Windows, got me to find that there’s an installer,

From the details, I couldn’t figure if that’s mean we could actually spin a windows docker host, and run linux based dockerized apps ? even if it’s intended only for running windows based dockerized apps, it’s an interesting match which I think I’ll might find handy in the future.

Docker is really interesting, but it got a very weird edges. 3 examples:

When your host is 64bit, and you need a 32bit application. it’s not that easy to setup up, I’ve tried couple of time and failed.
Anther weird stuff I’ve run into was this issue, (don’t ask why I need to manually change /etc/hosts, It’s a sad answer) but some kind of limitations are in place, and might drive away arguing things like “Change hosts and resolv is the basic operation in daily working.”
Entering a docker you’ve created, C`mon is should be a bit easier and integrated into docker (and not that long command line to remember)

Anyhow the whole experience with docker is very nice, and I do recommend giving it a spin. (you’ll sure put this one in your toolbox/toolbelt)

Taming the logging formatter

2014-11-06T00:00:00+00:00

We love logging, and we use it all around our code. (it’s the standard/built-in/batteries included)

We have those hugh json objects flying around, we want to log them.

pprint.pformat comes handy for those cases:

import pprint
my_large_data = [ dict(data="my large inforamtion", data1="dicts, dicts, dicts,", data3=dict(a="??", b=2, c=3, d="***")),
                  dict(data="my large inforamtion", data1="how we love big dicts ?", data3=dict(a="??", b=2, c=3, d="***")),
                  dict(data="my large inforamtion", data1="how we love big dicts ? don't we ?,", data3=dict(a="???", b=2, c=3, d="***")),
                ]

logger.error("Mary \n had \n a \n little \n lamb")
logger.error(pprint.pformat(my_large_data))
logging.error(pprint.pformat(my_large_data))

but when used with complex formatter the output goes like that:

^14/11/06 00:58:39 !ERROR <t:MainThread T:root M:log.py F:<module> L:74 > Mary
 had
 a
 little
 lamb
^14/11/06 00:58:39 !ERROR <t:MainThread T:root M:log.py F:<module> L:75 > [{'data': 'my large inforamtion',
  'data1': 'dicts, dicts, dicts,',
  'data3': {'a': '??', 'b': 2, 'c': 3, 'd': '***'}},
 {'data': 'my large inforamtion',
  'data1': 'how we love big dicts ?',
  'data3': {'a': '??', 'b': 2, 'c': 3, 'd': '***'}},
 {'data': 'my large inforamtion',
  'data1': "how we love big dicts ? don't we ?,",
  'data3': {'a': '???', 'b': 2, 'c': 3, 'd': '***'}}]
^14/11/06 00:58:39 !ERROR <t:MainThread T:root M:log.py F:<module> L:76 > [{'data': 'my large inforamtion',
  'data1': 'dicts, dicts, dicts,',
  'data3': {'a': '??', 'b': 2, 'c': 3, 'd': '***'}},
 {'data': 'my large inforamtion',
  'data1': 'how we love big dicts ?',
  'data3': {'a': '??', 'b': 2, 'c': 3, 'd': '***'}},
 {'data': 'my large inforamtion',
  'data1': "how we love big dicts ? don't we ?,",
  'data3': {'a': '???', 'b': 2, 'c': 3, 'd': '***'}}]

Kinda, not really helpful:

confusing - and you loose track of where this print came from really fast (keep in mind I’m talking about humongous json object)
filtering is (almost) impossible (yeah, not without some extra regex kung-fu)

Skimmed through the logging documentations, google it a bit think someone already solved those things and I could copy paste, came out empty handed.

So here how it can be fixed:

import sys
import logging


class MultiLineFormatter(logging.Formatter):

    def format(self, record):
        """
        split the message into different line and prepend it with the formatting
        :param record:
        :return: the formatted string
        """
        output = []

        if self.usesTime():
            record.asctime = self.formatTime(record, self.datefmt)

        data = dict(**record.__dict__)

        message = record.getMessage()
        for msg in message.split('\n'):
            data['message'] = msg
            output += [self._fmt % data]

        s = "\n".join(output)

        if record.exc_info:
            # Cache the traceback text to avoid converting it multiple times
            # (it's constant anyway)
            if not record.exc_text:
                record.exc_text = self.formatException(record.exc_info)
        if record.exc_text:
            if s[-1:] != "\n":
                s = s + "\n"
            try:
                s = s + record.exc_text
            except UnicodeError:
                # Sometimes filenames have non-ASCII chars, which can lead
                # to errors when s is Unicode and record.exc_text is str
                # See issue 8924.
                # We also use replace for when there are multiple
                # encodings, e.g. UTF-8 for the filesystem and latin-1
                # for a script. See issue 13232.
                s = s + record.exc_text.decode(sys.getfilesystemencoding(),
                                               'replace')

        return s

# and two examples how to configure the root logger to use it

import logging
import logging.config

logger = logging.getLogger()
logging.basicConfig()

def configure_with_api():
    handler = logging.StreamHandler()
    formatter = MultilineFormatter(fmt='^%(asctime)s !%(levelname)s <t:%(threadName)s T:%(name)s M:%(filename)s F:%(funcName)s L:%(lineno)d > %(message)s',
                                  datefmt='%y/%m/%d %H:%M:%S')
    handler.formatter = formatter
    logger.addHandler(handler)

def configure_with_dictConfig():

    logging.config.dictConfig( {
        'version': 1,
        'disable_existing_loggers': True,
        'formatters': {
            'multiline': {
                '()' : MultilineFormatter,
                'format': '^%(asctime)s !%(levelname)s <t:%(threadName)s T:%(name)s M:%(filename)s F:%(funcName)s L:%(lineno)d > %(message)s',
                'datefmt': '%y/%m/%d %H:%M:%S'
            },
        },
        'handlers': {
            'console':{
                'class':'logging.StreamHandler',
                'formatter': 'multiline'
            },
        },
        'loggers': {
            '': {
                'handlers':['console'],
                'propagate': True,
                'level':'INFO',
            },
        }
    }
    )

and now our output is much much clearer:

^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:74 > Mary
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:74 >  had
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:74 >  a
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:74 >  little
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:74 >  lamb
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:75 > [{'data': 'my large inforamtion',
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:75 >   'data1': 'dicts, dicts, dicts,',
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:75 >   'data3': {'a': '??', 'b': 2, 'c': 3, 'd': '***'}},
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:75 >  {'data': 'my large inforamtion',
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:75 >   'data1': 'how we love big dicts ?',
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:75 >   'data3': {'a': '??', 'b': 2, 'c': 3, 'd': '***'}},
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:75 >  {'data': 'my large inforamtion',
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:75 >   'data1': "how we love big dicts ? don't we ?,",
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:75 >   'data3': {'a': '???', 'b': 2, 'c': 3, 'd': '***'}}]
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:76 > [{'data': 'my large inforamtion',
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:76 >   'data1': 'dicts, dicts, dicts,',
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:76 >   'data3': {'a': '??', 'b': 2, 'c': 3, 'd': '***'}},
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:76 >  {'data': 'my large inforamtion',
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:76 >   'data1': 'how we love big dicts ?',
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:76 >   'data3': {'a': '??', 'b': 2, 'c': 3, 'd': '***'}},
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:76 >  {'data': 'my large inforamtion',
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:76 >   'data1': "how we love big dicts ? don't we ?,",
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:76 >   'data3': {'a': '???', 'b': 2, 'c': 3, 'd': '***'}}]

Disclaimer: use with caution, written past midnight :)

ELK is fun

2014-10-30T00:00:00+00:00

As I do a lot of testing and CI in my day job, I took ELK for a test drive. (ELK stands for ElasticSearch, LogStash and Kibana)

I’ve played around with Logstash before, but it’s first time i’m trying Kibana.

I’ve look for a quick way to setup it, and there where gazillion hits in Docker Hub (I’ve forgot which one I’ve actually installed :)) EDIT: it was cyberabis/docker-elkauto.

That went quick quickly, with a small change that I’ve used port 81

Next thing I wanted to feed it with some data, good thing someone else added some key=value files to our CI just a few days ago (Thanks Mike). And I had first real data in to play with in matter of minutes

That’s was not enough for me, I wanted all the information from my tests. Seem like the junit-xml output has all the information I wanted. I hated the idea of running something after the tests are running, I wanted it to run automatic on any test I’ll run with py.test.

Took me some time to figure it out which py.test hook fits the best, and here’s how it can be done:

    # content of conftest.py
    def collect_ec_data():
        """
        if we run in EC, collect all data we need
        if not in EC, collect only commit and commit_sha
        :return: dict
        """

        # lets assume you don't really want to know what we save in our setup :)
        return data_collected

    def send_to_elk(item):
        """
        send `item` to Logstash port
        :param item: dict
        :return: None
        """
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        s.connect((TCP_IP, TCP_PORT))
        logger.info("Sending to %s:%d", TCP_IP, TCP_PORT)
        s.sendall(json.dumps(item) + '\n')
        s.close()

    def find_test_data(junit_xml_filename):
        """
        flatten all the data from junit based xml file into a list of dicts

        :param junit_xml_filename: the output from py.test --junit-xml
        :return:
        """

        tests_list = []

        xml_test = open(junit_xml_filename, 'r').read()

        et = ElementTree.fromstring(xml_test)
        for test in et.iter('testcase'):

            item = dict(**test.attrib)
            outcome = None
            failure = test.find("failure")
            error = test.find("error")
            skipped = test.find("skipped")

            if failure is not None:
                item['failure_message'] = failure.text
                outcome = 'failure'
            elif skipped is not None:
                item['skip_message'] = skipped.attrib['message']
                outcome = 'skipped'
            elif error is not None:
                item['error_message'] = error.text
                outcome = 'error'
            else:
                outcome = 'success'

            item['outcome'] = outcome
            item['run_time'] = item['time']
            # time is a built in in Logstash
            del item['time']

            tests_list.append(item)

        # filter by group, and merge result
        cumulative_keys = ['outcome', 'run_time']
        tests_list_grouped = []
        for key, group in groupby(tests_list, lambda x: (x['name'], x['classname'])):
            item = dict()
            for thing in group:
                # merge needed values
                if item.has_key('outcome'):
                    item['outcome'] += " & " + thing['outcome']

                if item.has_key('run_time'):
                    item['run_time'] += thing['run_time']

                # overwrite the rest
                for key in thing.keys():
                    if key not in cumulative_keys or not item.has_key(key):
                        item[key] = thing[key]
            tests_list_grouped.append(item)

        return tests_list_grouped

    def pytest_unconfigure():

        junit_file = pytest.config.option.xmlpath
        ec_data = collect_ec_data()
        log_dir = os.environ.get('LOG_DEST_DIR', './')

        if junit_file:  # if found xml file, send all of it
            tests_list = find_test_data(junit_file)
            for i, test in enumerate(tests_list):
                item = dict(**ec_data)
                item.update(test)

                # save data to a file
                with open(os.path.join(log_dir, 'job_info%d.json' % i), 'w+') as f:
                    f.write(json.dumps(item, sort_keys=True,
                                       indent=4, separators=(',', ': ')))

                send_to_elk(item)

Just for reference, to show how simple is the logstash configuration, almost untouched:

input {
        tcp { port => 3333 type => "text event"}
        tcp { port => 3334 type => "json event" codec => json_lines {} }
}
filter {
        if [createTime] {
                date {
                        match => [ "createTime", "ISO8601" ]
                        target => "@timestamp"
                }
        }

}
output { elasticsearch { host => localhost } }

ELK is a great combo, me like a lot…

TheLinuxGameTome is dying, lets save it

2013-04-05T00:00:00+00:00

I’ve read this story on Slashdot, and thought I might take the database as a django practice build a real web site. Who want to join me ?

keeping code clean (the way of the nazi)

2012-06-15T00:00:00+00:00

Reading some comments in Mouse vs Python, I’ve enounter this gem pep8nazi, that can help in not lettig any code that doesn’t comply to pep8 to run at all

really like the idea, question is how it will work in real life scenrio when the package you’ll use aren’t fully complient to pep8.

Tools, tools, tools

2012-04-09T00:00:00+00:00

People always complain about tools, me included. (a.k.a trying to show that their own tool set are better) from text-editor to IDE, and don’t get me starting on usability of some websites…

This guy took it a further and kind of made it into a life philosophy Bret Victor - Inventing on Principle. and kinda of proved that all our tools suck. Really liked the tools he demonstrated, especially the mario like demo.

BTW, his site is kinda of cool http://worrydream.com/ and I quote from there:

“I designed the initial user interface concepts for iPad, iPod Nano, and half a dozen experimental hardware platforms. Initiated, designed, and prototyped over seventy concept projects…”

sound like a dream job.

and now it got me thinking, do I have a principle ? (or do I need one to get such a dream job ?)

probably not… or at least not yet… but I guess I’m getting there.

Why we HATE writting plans?

2012-03-29T00:00:00+00:00

Every time I need to start working on planning documentation, I get this weird feeling that I’m wasting my time.

I’ve tried to scribe down the reasons:

Nobody is going to read it, so why actually putting so much work into it ?
things are bound to change, so it isn’t worth the paper it’s written on (paper as in bytes on my hard-drive)
I know that I won’t have time/manpower to actually do 20% of what I’m going to plan, so what the use ?
My English sucks, no one will understand what I’m writing
I never seen a good plan, that actually succeeded. So there’s no good plans…

Now I’m left with tackling each one, and trying to hack my brain into thinking it would be fun to write the plan…

Coming up next in “Fruch picking his brain”…

Highlights from pyvideo.org

2012-03-29T00:00:00+00:00

Look like people were quite busy in the latest US PyCon, I need to promise myself to actually attend one in the next coming years. too bad there no such thing in IL, to only closest thing is the meetups of PyWeb-IL (I want to attend one also…, even thought on trying to host one at my day job office)

So thanks to the people of pyvideo.org, we could actually see what was going on.

Frozen Yogort

I’ve seen “Esky: keep your frozen apps fresh”, this guy really touch the spot, on how to deliver the code to mortal that can’t handle the python interpeter.

Amazing how it can be done with three line of code (yes, and anthor two in the setup.py):

if hasattr(sys,"frozen"):
    app = esky.Esky(sys.executable,"http://example.com/downloads/")
    app.auto_update()

go fetch it from PyPi, esky

RESTful APIs for the tired

In this One, it show how easy is to add RESTful API to a django application.

Very useful if you want to keep those control freaks nerds happy, now they could write a bash script that uses curl to interact with your site.

Fast Test, Slow Test

And a one about testing, casue my job title dictates it.

Fast Test, Slow Test, by Gary Bernhardt (really liked his dotfiles repo)

I really agreed with almost every word he were saying, too bad it really is a pain actully writing unittests that hit the right spots. his saying about not writing tests to “bake” the bad desgin into the system, reminded me what i’m doing almost daily. Writing tests to match requirements or other system behivers that are actully wrong or will do more bad then good in the system.

But then, who has to power/will to change a big eco system upside down ?

Think of it, one of the tester that work at Microsoft on a new version of Windows, will come and say that the briliant people who thought and desgin something we’re wrong. He will probably give up, every before that thought will touch the first neuron in his brain…

We all want to come home in one piece ( בסך הכל רציתי לחזור הביתה בשלום)

Hello World

2012-03-29T00:00:00+00:00

So I’ve started a blog.

First thanks, to JekyllBootstrap for the code, and for Github for the hosting (כבודה של אכסניה)

Even that I uselly don’t use ruby, I’ve installed it on my win7 for running jekyll. thanks god no need to actully code in ruby for that.

I hope now I can actully share some of my insights about python and life in general. (after I fooled around a whole day on choosing the theme, and adding facebook comments)

I wish myself that I’ll find time to write.

Fruch out…

Fruch's Blog

Taking Github Actions for a walk in the park

Taking Github Actions for a walk in the park

Travis started to scare me a bit

The sweet setup I had in Travis

Github actions

Github Actions - UI

Late surprise - cross compile using qemu

Summary

How much fun was EuroPython 2020

How much fun was EuroPython 2020

#pyconil got canceled

Discord is fun

Talks — Day 1

Social — the missing e in my whisky

Talks — Day 2

Sprints

My DevopsDaysTLV Summary

My DevopsDaysTLV Summary

Late birds

Rockstar saves the day

Day One

Day Two

Introducing pytest-elk-reporter

pytest-elk-reporter

History

What can it do ?

Tricks I’ve picked on the way

The bad stuff - a.k.a things I need to improve

What you should next

Microsoft - amazing stuff, WSL, VSCode

WSL

VSCode

Conclusion

KISS - Keep it shorter short

My version

Microsoft + Docker, really ?

Taming the logging formatter

ELK is fun

TheLinuxGameTome is dying, lets save it

keeping code clean (the way of the nazi)

Tools, tools, tools

Why we HATE writting plans?

Highlights from pyvideo.org

Frozen Yogort

RESTful APIs for the tired

Fast Test, Slow Test

Hello World

Social — the missing `e` in my whisky