Ellimilial

Python - weighted random selection

Thu, 30 Jan 2014 21:14:01 +0000

I recently had a need for weighted random sampling, evaluated several approaches. This is just the eventual product, approaches stolen from several places, tested and implementing the same list(item, weight) interface (albeit non consistent in return type).

Weighted sample - no replacement


def weightedSampleNoReplacement(items, n):

        """

        Selects without replacement n random elements from a list of (item, weight) tuples.

        """

        randomised = [(item[0], random.random() * item[1]) for item in items]

        sorted_ordered = sorted(randomised, key=lambda item: item[1])[-n:]

        return [item for item, _ in sorted_ordered]

Weighted sample - replacement


def weightedSampleReplacement(items, n):

    """

        Expects an iterable in form (item,weight).

        """

        total = float(sum(weight for item, weight in items))

        index = 0

        item, weight = items[0]

        while n:

                x = total * (1 - random.random() ** (1.0 / n))

                total -= x

                while x > weight:

                        x -= weight

                        index += 1

                        item, weight = items[index]

                weight -= x

                yield item

               n -= 1

Weighted sample - hybrid

When you want to penalise the already selected items, this is the one I used for the project.


def weightedSampleHybridReplacement(items, n):

        """

        Expect an iterable in form (item,weight).

        Based on roulette-wheel implementation. A bit expensive.

        Returns a sample with replacement but each time

        an item is selected, it penalises it by cutting its weight by half.

        """

        total = float(sum(weight for item, weight in items))


        #Sort items to optimise traversal from 0.0,

        items = sorted(items, key=lambda item: item[1])


        index = 0

        item, weight = items[0]

        location = weight

        while n:

            target = random.random() * total

            while location 
                    index += 1

                    item, weight = items[index]

                    location += weight

                penalty = weight/2

                items[index] = (items[index][0], items[index][1]-penalty)

                total -= penalty


                yield item

                index = 0

                item, weight = items[0]

                location = weight

                n -= 1

Bootstrap 2 vs Bootstrap 3 differences

Mon, 29 Jul 2013 13:32:17 +0100

There has been a recent rumbling coming from the real of Twitter Bootstrap, that is, Bootstrap 3 RC 1 has been released (and made default on http://getbootstrap.com/). You can find some skeletal information about the changes here, including a link to the actual pull request, which gives you a long (and boring) list of changes.

There have been multiple improvements to the repository structure, inheritance and general maintainability of the code (glyphicons have been moved to a separate repository). The main changes from my own perspective (i.e. a person using it as a pluggable end product) are the following:

-No longer need to include bootstrap-responsive and specifying viewport size as responsive design is now included by default (thankfully!).
-The headers are much sleeker, smaller and not as much emphasised. This makes the page look much lighter but in some cases it might have gone too far (for example in modal header).
-Buttons are now much more metro-styled, made slightly larger, no more puffiness and default color changed to dark-grey.
-Buttons in the button group are no longer visibly separated until you stick your mouse pointer on top of them, when the currently selected button borders will show.
-Navbar was made slightly larger.
-The close icon inside alert blocks now matches the colour with the rest of the component instead of being light grey.
-Modals no longer use drop down/go up animation when being made visible/hidden respectively.

In short, Bootstrap 3 comparing to make 2 screams ‘sleeker, let’s go metro’. IMHO it is the right direction, making it easier for the eye although I was fond of the sturdy, large headers and sections titles and very conservative use of space. Guess we’ll have to wait for RC 2 to see what else awaits us.

Visual Studio environment theme and settings

Tue, 09 Jul 2013 18:10:04 +0100

Just a handy tip (and a recollection point for my glorious self ) how to make the Moloch of Visual Studio a bit lighter on the head.

First you should install the right theme (http://studiostyl.es/) for your lovely VS.

Configure your VAssistX (which you should really be using) Advanced->Fonts and Colours to work well with the style and not make you lose your sight too much.

Clean the unnecessary mess on the - screen fillers that you do not need from the bars - for me that means getting rid of anything apart from Standard bar unless when debugging but that’s a different story.

Polish your right mouse button so it is always ready to click on a file tab and open it in the vertical splitter which can easily improve your efficiency by 10%.

Hiding all screens - now this is something I need most of the time when trying to crack the mysteries of the lost memory allocation. A shortcut to close all the windows (and re-open them). As most of the time I work with VS 2008, this is the one that gave me the most pain. There are several ways I’ve read about, most of them centred around creating a macro and binding a key to it. I prefer something much much simpler. Associate the keys with Window.AutoHideAll and Window.ResetWindowLayout calls. It’s nothing close to any sophistication but, at least for me, it has always done the trick. To do that, go to Options->Environment->Keyboard, search for the options and fill ‘Press Shortcut Keys:’ line edit. I usually assign Ctrl+d for closing and Ctrl+Shift+d to restore the layout.

Simple, but I can’t work without those.

Scapegoat Picker

Sun, 12 May 2013 16:07:44 +0100

At my glorious company we have recently encountered an issue with code reviewing. Sounds like an easy thing to do, right? Grab a coffee, sit down with someone else’s horrors and attempt to decipher what they meant. As we are so amazing and the errors like removing an item from current position in the iterator via an obfuscated call and then calling next on it are a rarity, it turned mostly into a competition to find a good place to put an assert or possibly point out an incoherent maximum line size.

In any case, it turned out people are not as eager to take a bite at the responsibility of viewing code of their teammates as one could expect. At one point we were discussed using a web-based randomiser to choose the reviewer. As we looked at the front page of the results, I realised that none of them were quite what we needed.

We required the randomiser to preserve list of names between daily scrums.
Our randomiser should be a fast, one-click thing.
We required some minor twists to the randomisation like weighting senior people slightly more than complete juniors and making sure that the last person chosen is safe for the next round.
Most of the randomisers we found were either chesesy (in our view). We required something with, specific kind of humour applicable at our company. Slightly darker yet kind in the end. Plus we really wanted the picture of our manager with flaming eyes!

Finally, I thought, a good reason to play with Flask instead of losing hours for Django configuration. My first real time with Javascript apart from Codacademy as well. Flask-bootstrap still gives me a headache when attempting to use reflective version, though it’s pretty neat apart from that. To be honest, Flask is the way to do simple sites. It’s dead-simple, minimalistic to the bone yet session and the rest pf API is simply functional. I mean, this could really end up in a single file if I was not so much into separating views, namespaces and common functionality. Once the site was done, I thought about deploying it on Heroku, after some time struggling with bare AWS. Again, a charm, a couple of hours to fix my sillyness and it was there. All in all, ScapegoatPicker was a nice exercise for a weekend and some.

SQL Server Management Studio - Value cannot be null bug

Tue, 02 Apr 2013 15:27:53 +0100

I’m afraid this is not going to be a particularly interesting one - just an environmental glitch I spent a day trying to work out.

I recently experienced an issue with SQL Server management studio (2008 R2) in which I was presented with the following error:

Value cannot be null.
Parameter name: viewInfo (Microsoft.SqlServer.Management.SqlStudio.Explorer)

This happened after a reinstallation of SQL Server and resulted in Object Explorer tree not being populated at all and rendered the Management Studio useless. Multiple threads relating to the issue mention that this might be due to TEMP & TMP folders being improperly configured or missing recently created folders. This was not the issue in my case and the next suggestion was to repair .Net 4.0 Framework. Again, no luck there, neither did help uninstalling / reinstalling of Management Studio.

Eventually, using Process Monitor, I realised that there were errors in machine.config xml configuration file (located under “%WinDir%\Microsoft.NET\Framework\<versionNo>\Config”) – half of the file was simply missing, making the DOM invalid. There is a skeleton machine.config.default file within the same directory with which I replaced the broken file to make the application work. There was another problem with Visual Studio not starting, which boiled down to, again, broken config XML (“\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\devenv.exe.config), which I simply terminated correctly to get it going.

In case you are forced to reinstall SQL Server and stop being able to use SSMS, this is the bit that happened to me and I could not find it on any of the web forums.

Auxilary XML - the simplest data persistence flexibility pattern

Fri, 22 Feb 2013 13:02:49 +0000

There are various ways to spec out your requirements, model your domain and translate it to database design, encompassed in the lovely ERD diagrams you can show off to everyone interested. Once it’s done, you’re ready to implement, test and deploy it. Once on site, it just works and you never have to think about it again, unless, you know, you live in a real world.

When changing the code, adding new functionality, it might be the case that you need to update the data model. Making values obsolete is easy. Extending your schema by adding a single attribute - not so much. Sure, you can use ETL software and make update to your customers’ persistence layer whenever you create a patch. It’s not always possible and it’s wasting time doing it over and over again at the smallest change. Django builders know the inconvenience of having to fiddle with their store when trying to add new attributes to their models once the application is deployed.

This frustration can be avoided with an extremely simple trick. When designing your data store, make sure you you pay attention to the most likely culprits for future extension/change. Then, simply add an auxiliary text/XML attribute. When implementing small changes, you will have an extensible store for whatever additional data you need, which you can then swiftly map to the new fields when required (if ever).

Hosting blog photos on Amazon S3

Tue, 12 Feb 2013 17:15:00 +0000

I’m afraid I’m going to spoil it straight away: DON’T, JUST DON’T.

Only recently I advised my lass to try that. She wanted to have a good control over the collection and did not enjoy the size limitations that Blogger, Picassa and Flickr happen to have. Since it costs close to nothing to store the files, it made a total sense to go this route.

It was all too easy to use one of the many software pieces that let you upload files straight to S3 and then modify the permissions to let everyone access the photos.

The catch is, as those of you who, unlike myself, happen to have any idea about how the net works, the transfer. And it can scale up massively in the longer run. In this particular case, said blog had experienced traffic increase of about 44% in about 5 weeks. At the end of the month, the total transfer out of S3 amounted to 149 GB.

So make sure you always do proper estimation beforehand.

Creating own personal tech blog

Mon, 11 Feb 2013 23:48:00 +0000

Since the times of ancient Greece, meta- prefix seems to be quite valuable. Therefore, in the spirit of making the Internet a much better (value added) place, I present my step-by-step guide to creating a personal tech oriented blog by someone who is supposed to be a software guy. Here goes:

Lock down the important bits first and fast: what to write about (i.e. not abut stuff you like, but stuff that you might know a thing or two about), intended audience, taboos and how to put a proper disclaimer.

Decide that since you’ve spent most of your time hacking in Java and C++, it is simply impossible that any scripting and/or declarative language can cause you any trouble at all. If you do multi threading, distributed, real-time systems, you will not be beaten by some silly silliness made of JavaScript mumbo-jambo and whatnot. You know that you can do this, and you simply won’t go for Blogger or Wordpress because you are TOO GOOD FOR THIS.

Realise that you might have been just a tiny bit too cocky and those PHP tutorials from uni times are not really that useful anymore (even if you did remember quarter of them).

Lucky enough, you’ve taken time to play with Python and Django and you know how to use AWS to fire your own instance somewhere in the stratosphere. You even learned that there are easier ways than manually prepare your distro from scratch!

You spend a couple of days trying out different Django blog engines, being slightly spoilt for choice.

You then realise that it is not at all cost-effective to host it on an Amazon instance. Digging deeper you find out that there are ways to host static files on S3. Thankfully, there is Cactus that is the key to using Django like that.

Come into conclusion (brain size!) that some content simply cannot be static, in which case you will need to pick up a third-party plugin. Be it Facebook comments, Disqus, IntenseDebate or something along those lines.

Get back to the basics and realise that it makes little sense to dig that deep producing nothing up to that point. Since you are supposed to be a software developer there should be no shortcuts.

After a day or two of deliberation, come to the conclusion that life is too short for that and there must be a way to do it faster and better.

Ok, so you decide to go Tumblr, you might not feel hipster enough to use it but after a day you are ready to roll. If only you read about it somewhere beforehand…