Programmer's Weblog

Home Assistant Renew DHCP Lease

noreply@blogger.com (Kamil Dworakowski) — Wed, 05 Jan 2022 22:40:00 +0000

This is a workaround for a problem with Home Assistant running in a Virtual Machine and not renewing a DHCP lease thus losing the connection to the local network after the lease runs out. You can define an automation that will periodically run a script that pokes the network interface forcing a lease renewal. I did it a while ago so I hope I remember all the elements required to make the workaround work.

The rest api that allows to poke the network interface of the home assistant os is not accessible from outside of the instance. You need to install the "Terminal & SSH" add-on to get access to the os running home assistant.

Once you "hack the system" and get inside the home assistant via ssh, inside the config directory, define a script that pokes the card:

curl -X POST \

-H "Authorization: Bearer $SUPERVISOR_TOKEN" \

-H "Content-Type: application/json" \

-d '{"enabled":"True"}' \

http://supervisor/network/interface/enp0s3/update

Where enp0s3 is the name of the network interface, it might well be different in you setup.

The script needs to be inside the config directory because config directory is special, contents are not deleted on restart for one.

Once you got access to the home assistant os, it is easy to see there seems to be cron available, but I tried in vain to use it. Not only is cron not running the crontab, but also any cron configs are wiped on restart.

Instead you can define a home assistant automation that will invoke the script.

Add to config/automations.yaml:

- id: renew-dhcp-lease

alias: "Renew DHCP lease"

trigger:

- platform: time

at: "21:35:00"

action:

- service: shell_command.cycle_network

Add to config/configuration.yaml:

shell_command:

cycle_network: bash cycle-network.sh

This will trigger the action at 21:35 each day, which was good enough for me.

Xiaomi Kingsmith Walking Pad A1 Pro WPA1F Review

noreply@blogger.com (Kamil Dworakowski) — Tue, 14 Dec 2021 22:44:00 +0000

I've been using it since May this year, so for over half a year now. I'm using it as part of a walking desk. It works well for me.

The belt size is more than enough -- I've never tripped, never felt unsure footed. I'd prefer it to be smaller actually, so it had a smaller footprint.

I used the smartphone app very rarely - set the default speed at 3 km/h. I exclusively use the manual mode -- this is the only one that makes sense for a walking desk. I usually walk at 3.5 or 4 km/h, sometimes faster if I want some exercise.

The biggest problem for me is the noise the fan makes. This is the fan that cools the engine. It only switches on a minute or two after you start walking and then stays on until the treadmill stops. At lower belt speeds, up to 4km/h, it is noisier than the belt and the engine themselves.

At 3.5 km/h the treadmill is 50-51 db before the fan switches on and 52-53 db with it. The fan noise is somehow more annoying too. It is tolerable for me, but it bothers my partner when I'm using it. It mostly limits the time I'm walking to when she is away. I sometimes get away with using it when she has noise canceling headphones on.

If there was a walking pad running 3.5 km/h at 47 db I'd replace this one in a heartbeat. I would also consider one that has at least a quieter fan.

A minor problem with the treadmill is static electricity it creas on the user. I'm not sure what is causing it. I've never experienced it from gym treadmills, and it was not happening at the beginning with this one either. Now I need to be grounded or else I experience uncomfortable static discharges when touching a laptop.

Overall the benefits are bigger than the drawbacks.

Random Seatbelt Alarm While Driving

noreply@blogger.com (Kamil Dworakowski) — Sun, 21 Oct 2018 12:00:00 +0000

My car was intermittently not detecting that the seatbelt is fastened, randomly while driving. Possible solutions:

cleaning the seatbelt tongue with isopropyl alcohol -- the quickest and I was surprised it actually worked, because the tongue did not look dirty

getting a seatbelt extender

replacing the seatbelt tongue -- seems like a lot of work

disabling the seatbelt check, on Alfa Romeo it can be done only through the OBD connector, but you need a special software (alfaobd.com, 50EUR, windows, android -- I haven't tried it)

How To Replace the Accessory Drive Belt (Serpentine Belt) on Alfa Romeo Giulietta 1.4 MultiAir

noreply@blogger.com (Kamil Dworakowski) — Sun, 21 Oct 2018 11:42:00 +0000

This could apply to other cars in Fiat Chrysler group. Find the belt on the left of the engine. You can take a picture of the routing. Identify the belt tensioner by pulling on the belt and seeing which pulley yields. In my car it is the center pulley. It has a 13mm hex bolt on it. Hold it while you pull on the belt to see which way it turns as it yields. In my car it turns clockwise. I think there is a hole to lock the loosened tensioner in place, but I did not even attempt it due to difficult access. Use a ratchet and turn the tensioner with left hand and use the right hand to slide the belt from/onto the bottom pulley. It will be easier to put on a belt if it is longer rather than shorter. I got myself a rather short replacement belt and it seemed I had to turn the tensioner to the limit to put it on.

Tools used: a 1/2" ratchet with 24cm arm, a 13mm hex socket, head lamp.

Front Suspension on E-Twow Electric Scooter

noreply@blogger.com (Kamil Dworakowski) — Thu, 04 Oct 2018 14:19:00 +0000

Etwow scooters are sold under other brands too, including Frugal, SXT and UrbanGlide. I am not actually sure ETWOW is the original manufacturer or just another brand and scooters are manufactured by a yet another company.

I got myself a used one, and it came with several issues, one of which was a stuck front suspension. But first things first. This scooter has no air in tires, instead it relies on a spring front and rear suspension for comfort. In order to actually benefit from it you need to regulate it to your weight so that there is a sag when you mount it. The rear one is very easy, just loosen the screw tightening the spring. The front one has such a screw compressing the spring too, but the access is not easy. You need to detach the wheel and the steer pipe and then you need a special tool, a deep socket wrench size 10 if I remember correctly. Car mechanics usually have those if you can't find it in a store. See the photo below of how the scooter looks like being ready to loosen the tightening screw. Here is a link to the service manual for the scooter. You need to block the screw from the bottom with a 10 hex key, and turn from the top with the socket wrench. That socket wrench needs to be deep because there are four screw nuts on top of each other.

For my electric scooter that adjusting was not enough though. The suspension was stuck because the plastic hexagonal bed was too tight for the metal rod that was moving in it. A friendly mechanic polished the metal rod by hand using some sort of metal grinding tool (no machines involved) just to make it a bit thinner so that it would move freely inside the plastic bed.

Even with the suspension working the ride is not as comfortable as on a full size bike with no suspension. I don't think it is the superiority of air tires over a spring suspension, but rather a function of wheel size.

Disclaimer: Before all this work, I contacted etwow for some information on how to adjust the front suspension and the person I talked to said the front suspension is not adjustable and that the tightening screw is holding the spring in place and loosening is not advised. I chose to ignore him and adjusted it anyway. It works.

Alfa Romeo Giulietta Worn Hatch Button Badge

noreply@blogger.com (Kamil Dworakowski) — Thu, 04 Oct 2018 11:17:00 +0000

This post is about replacing a hatch button badge that looks worn, but the button works fine. It seems to be a common issue for the model brought on by the way they are attached; I explain below. You pry the aluminum logo off the button with a knife - start with stabbing it near the edge toward the center. The logo badge is attached to the plastic button underneath with an adhesive. It is a 3 millimeter thick layer of soft adhesive that is only present in the center of the button. The adhesive not covering the whole button is most likely the reason for the wear, as the damage happens near the edges where there is no adhesive to support button pressing.

I attached a replacement badge using silicone just because I had some and it seems to have a similar consistency to the original adhesive and would be similarly easy to detach should I ever need to. I tried to distribute the silicone on the whole button, not just the middle, to prevent deformation on pressing. Silicone seems to hold it fine, haven't lost it yet.

As to the logo badge, you needed a 75mm wide one, the first one I bought, a 74mm one, did not fit. The badge is the same for the front and back.

pinfruit - my mnemonic web app for memorizing numbers

noreply@blogger.com (Kamil Dworakowski) — Thu, 13 Jan 2011 01:09:00 +0000

I have created a web app to memorize numbers. It implements the mnemonic major system. The system works on the principle that it is easier to remember words/sentences/stories than numbers. The web app facilitates finding sequences of words that encode a number in the mentioned system. I called the app pinfruit.

My first take on implementing the system was incorrect. I simply associated digits to letters. According to wikipedia the mnemonic major system assigns digits to sounds, rather than the letters of the alphabet. And rightly so, I immediately noticed that it is easier to work with sound associations.

My mistake originated in that I first learned the technique in Poland, and in Polish it does not matter if you assign digits to letters or sounds because both correspond. Unfortunately, in English this is not the case, the major difficulty in learning the language.

I have used pinfruit to memorize mainly pins and phone numbers, but also bank site's user identifiers, which often come as numbers.

The trick is to create a memorable (vivid or absurd) sentence/story for a number. My old phone number: 07514590312, can be encoded with husky leader leaps midway Hanoi, school weather helps Madonna or sickly water leaps hometown. I refer you to the wikipedia article for the detailed description of the system.

From layers to hexagonal architecture

noreply@blogger.com (Kamil Dworakowski) — Sun, 15 Aug 2010 20:41:00 +0000

The traditional layered architecture consists of data, domain and presentation layers.

Separating the presentation layer from the domain layer is rather easy. The presentation layer objects query the domain for data to display on the screen. The persistence layer is another story. The fact that it appears below the domain layer prohibits any of the classes in it knowing about the domain entities.

But consider a Fetcher class for example, a class that has responsibility for querying some domain object from the database. It has the domain type in the signature and thus has to be defined in or above the domain layer. This is leaking of the data access object out of the persistence layer.

This problem has been recognized and here we have an architecture which works around it by separating out the domain objects package. But the result is not as clear and compelling as the layered architecture.

The civilisation progressed and an architecture with nothing below the domain model has been discovered. One can see this in the ddd sample.

The persistence concern is implemented as part of the infrastructure layer. Infrastructure is that vertical layer that depends on the three other layers. Domain knows nothing about the infrastructure.

Another example is the Hexagonal Architecture.

The domain sits in the core of the application, with the persistence aspect implemented by an adapter. The adapter layer corresponds to the infrastructure and the interfaces layer combined from the ddd sample architecture diagram. The important part is that the domain does not depend on anything else.

Such layering is achieved by defining service interfaces in the domain layer for persistence purposes. These abstract interfaces are implemented by the adapters in the outer layer of the application and injected into the objects that need them.

As a result nothing in the domain layer needs to import anything from the hibernate package, or whatever persistence technology is being used. The domain layer has a custom made, abstract interface to persistence service, in its own terms. The domain code can be expressed without the details of the persistence technology being used.

GOOS Book

noreply@blogger.com (Kamil Dworakowski) — Sat, 27 Mar 2010 12:53:00 +0000

Growing Object-Oriented Software, Guided by Tests by Steve Freeman and Nat Pryce is a book I was long waiting for. GOOS demonstrates the techniques and highlights the patterns as they show up in an application grown through the book. An albeit small but a real world project is being developed from scratch in a way were tests play as important a role as object-oriented design.

It is an excellent opportunity to see how two skilled developers are growing application. The authors work in TDD in a style characterised by extensive mocking to avoid breaking encapsulation. In design they use fine grain classes. All together the style demonstrated in the book is very consistent.

IronPython Dictionaries Memory Cost

noreply@blogger.com (Kamil Dworakowski) — Sun, 07 Feb 2010 18:03:00 +0000

Maintaining a mapping as a dictionary can be quite expensive in terms of memory, more than one would expect. A couple of days ago I have checked with windbg how much a single entry in a dictionary mapping pairs to ints costs. The code I measured is:

d = {(1, 2): 3}

I have executed it in IronPython 2.6.

The graph below depicts what I saw on the heap. Each box represents a single word in memory. In a 32 bit system, a word is 4 bytes.

Every object in .NET runtime has an overhead of two words. These are the pointer to its class and some house-keeping data, which I don't know much about. Allegedly some part of it is used for object locking.

The graph shows objects below buckets array, these are the only that count. A dictionary is a hashtable which is implemented with an array of buckets. Each item in the dictionary is kept in a bucket. The size of the Bucket object determines the size of an entry in the dictionary.

There are 23 boxes in the Bucket's subtree, which means its size is 23 * 4 = 92 bytes. Quite a lot. A million numbers in that dictionary would take almost 100 megabytes!

The main reason it comes out as that many is the generality of the dictionary. The fact that it can store objects of arbitrary types means that the numbers must be boxed. .NET generic collections, when specialized for ints, would store numbers in place saving a lot of space.

Extreme programming too extreme?

noreply@blogger.com (Kamil Dworakowski) — Tue, 09 Jun 2009 13:21:00 +0000

Is extreme programming as a methodology over? Has Kent Beck killed it himself? Until now he insisted that you not write a single line of code without a failing test. He was talking about his daughter who allegedly can not even imagine writing code without tests first (in his book about test driven development)!

Extreme programming is a set of good practices but the problem is the emphasis on taking them to the extreme thus ignoring the cost/benefit ratio. There is an obvious trade-off here.

It would be optimal if people wrote just the right amount of tests, not too little and not too much, to maximise the ROI. That of course is very hard to judge, but for sure the answer is not 100% coverage in all cases, as Kent kindly observes in his post.

Extreme programming is flawed as a methodology, but I want to argue that it is good for learning the craft.

People have natural inclinations for not testing, not integrating continuously but programming a feature for days not syncing with the main code base, underestimating features and than putting in long hours, etc. Extreme programming is like a correction program for these unwholesome inclinations. :)

Aristotle noted in the Nicomachean Ethics that the best way to reach the golden mean is to aim at the opposite extreme.

In the dimension of testing, for example, people will naturally tend to write too little tests, usually none. Practising extreme programming forces us to go to other extreme: from none tests to 100% coverage no matter the cost. This teaches that you can write tests you didn't previously realized, but more importantly it changes the mindset of the programmer. After practising extreme programming you will find yourself uncomfortable without the safety net of tests, and while you will not necessarily want 100% coverage you will be in a much better position to judge how much testing is really needed.

I am glad that I was a part of an extreme programming team for almost 2 years now. We might have been too extreme, but too extreme in the good direction. 4:1 ratio of tests to code that we have might be an overkill, but it is certainly better than no tests at all. I feel similarly about other practices.

Sony PRS 505 ebook reader review

noreply@blogger.com (Kamil Dworakowski) — Sat, 16 May 2009 10:47:00 +0000

I use Sony PRS 505 for more than a month now (see how it compares with other readers). It came with 100 classic books, all of which look very tempting. I have already read over 1000 pages of David Copperfield.

(BTW, the Reader shows the beginning of the first chapter of the new "IronPython in Action" book, by my colleagues Michael Foord and Christian Muirhead.)

They say that the battery life is 5000 page turns, but mine died after about 1000 pages of mentioned David Copperfield. I have been playing with it, and sometimes going back and forth, but for sure that was not 4000 page turns worth of playing. Anyway, the battery life is very good, but the 5000 page turns is misleading at best.

The device comes with software for windows. It is not really a problem for a Linux user, because you can mount the reader like a usb stick, and copy the books directly - that is how I do it at least. There is calibre project that is supposed to be heaps better software for the reader than the official one, but it doesn't start on my system and I don't see any reason to investigate -- I'm fine with cp.

I have it for over a month and think it was worth every single pound I payed for it. I write the review now, because I have finally bought an ebook on-line and put it on the reader. That feels right. Although, in the meantime I had to buy one dead tree book simply because there was no ebook available. Hopefully that will not happen often as amazon's Kindle gains popularity.

My biggest wish is that the screen was larger. Looks like the upcoming Kindle DX is going to be the right size, though it is going to have a lower resolution than the Sony Reader. I bet that other manufacturers are already working on a larger version to compete with DX.

The small size is not really right for books with code and mathematical formulae (most cs papers and technical books). I found that putting the reader into horizontal mode (buried down in the settings) helps a lot with that.

Beautiful Code: Resolver One

noreply@blogger.com (Kamil Dworakowski) — Tue, 20 Jan 2009 08:56:00 +0000

Resolver One, aka the Python Spreadsheet, has a beautiful design at its core, which I would like to present to the programmer community. No knowledge of Python is necessary to understand this article.

The goal for the application is to provide seamless integration of spreadsheet (formulae in the cells) with sequential code. In MS Excel the VBA integration is anything but seamless. In the rest of this post I explain how Resolver One integrates Python with spreadsheet functionality.

Spreadsheet

To ensure that we are on the same page, I'll pinpoint some facts about spreadsheets. In a traditional spreadsheet there are two types of content a cell can hold:

constant: like number, date or text
formula: an expression usually referencing other cells

A user can enter this content by typing to a cell. Additionally, a user can specify formatting for a cell by various GUI means; for example, make the cell's font bold by clicking a toolbar button.

How do you add sequential code to it? Resolver's solution is to turn the spreadsheet into sequential program. User's custom code merges with the rest of the spreadsheet thus expressed.

Spreadsheet as a program

In Resolver One, what you see in the grid is the result of executing the code displayed in the coding pane.

Following is the code that appears in a document created with File | New command. I have pruned it by dropping comments and not required import statements.

from Library.Workbook import Workbook

workbook = Workbook()
workbook.AddWorksheet("Sheet1")
workbook.AddWorksheet("Sheet2")
workbook.AddWorksheet("Sheet3")

Constants = {}
Formatting = {}
workbook.Populate(Constants, Formatting)

The above featured code creates a workbook with three empty worksheets.

Where did the code come from? Resolver One generated it from the model. By model I mean the data structure that remembers constants, formatting and formulae input by the user through the GUI interface.

The drill is: Resolver One generates code from the model, executes the program, takes the resulting workbook object and displays it in the tabbed grid view. This is called recalculation.

Every time the user changes the model through the GUI commands, a new recalculation is triggered. The next snippet presents code after me changing the model by typing to some cells in the grid, I want to give you a feel of the generated code. I put number 1 into A1 and formula =A1*2 into B1.

from Library.Workbook import Workbook

workbook = Workbook()
workbook.AddWorksheet("Sheet1")
workbook.AddWorksheet("Sheet2")
workbook.AddWorksheet("Sheet3")

Constants = {
    'Sheet1': {
        (1, 1): 1,
    },
}
Formatting = {}
workbook.Populate(Constants, Formatting)

workbook["Sheet1"].B1 = workbook["Sheet1"].A1*2

User Code

All the code generated from the model is divided among three uneditable sections in the coding pane:

imports and worksheet creation
constants and formatting
formulae

In addition, there are three editable sections for user code, one after each of the generated ones. The user code is "on the same rights" with the generated code which is creating the workbook to display; the merge is perfect :).

Buttons

That would have been all and buttons aren't strictly necessary to discuss here, but they fit quite well. A button can be created in the user code and set as the value of a cell. Of course a button has a click handler that a user would define as a function. This handler is not executed during the recalculation of course. So when is it executed, and more importantly what can it do?

The first question is quite easy. The handler gets called when user clicks the button. He can't click it until he can see it, which is after the recalculation is finished the workbook is displayed on the grid.

There is nothing magic in what a button can do. Python has lexical scopes, so the handler has access to anything that was in scope where the function was defined, in particular this means the workbook object. For instance, handler could create two hundred new buttons and place them all over the grid.

When a user clicks a button the handler is called after which Resolver One refreshes the grid to display any changes made to the displayed workbook. There is no recalculation, because the model didn't change. This also means, that any changes made by the button handler will disappear after the next recalculation is finished, because there will be simply different workbook object being displayed.

Note: Resolver One 1.4, next major release, will provide means for changing the model from a button handler.

Wrapup

So that is the architecture of Resolver One, brought to you by one of the programmers on the team. Every other feature in the application revolves around this foundation. Of course the model contains more stuff, for which code needs to be generated; but it all ends up in one of the three sections in coding pane. To fully grok different features in Resolver One, you need to understand recalculation.

assert the machine did not explode

noreply@blogger.com (Kamil Dworakowski) — Fri, 08 Aug 2008 07:42:00 +0000

I see it often done by novices, heck, I used to do that myself. When writing a unit test for a method cloning an object, I would put an assertion to check that the original object is left unchanged.

Well, that is wrong. This is checking against a possible bug in the implementation, in which the original object gets modified. But this is just one thing that could go wrong. There are infinitely many possible bugs like this, maybe even one that causes your machine to explode. You can't write assertions against all of them, so don't do it against any of them.

It has two advantages. First is that you stop worrying you forget assertions against some possible bug; that is, if you were worrying about that in the first place. I am sure there are people out there who feel bad about not testing against printing foul language. Second is that the tests themselves become smaller, focused on the positive behavior, therefore easier to read.

In the specific example of cloning: check that the cloned object is equal to the original but they are not the same objects, and that is it. Nothing else.

Unit tests are positive tests, they should only test what you want the code to do, not what it shouldn't do. I am not sure about the case when we fix a defect, but probably a functional test for the defect is enough.

GVim to convert code into html

noreply@blogger.com (Kamil Dworakowski) — Sat, 05 Apr 2008 23:09:00 +0000

Few are aware of the fact that gvim can generate html for the code inside the buffer. The command is :TOhtml; just type it in, and the html will promptly appear in a split window.

In the browser, the generated html should look exactly as you see it in gvim. By default it generates html that should work in old browsers too, but can be told to generate CSS stylesheets or even xhtml. Type :help TOhtml for more details.

It works only in gvim, though; not in vim. It is handy for posting snippets of code on the web.

.NET BCL source code

noreply@blogger.com (Kamil Dworakowski) — Sun, 02 Mar 2008 14:05:00 +0000

Source code for .NET Base Class Library has been made available for debugging sessions in VS2008. It would download one file at a time as they are needed. Since then John Robbins and Karem Kusmezer have create a tool to download all the source files without hassle, great work! The tool is NetMassDownloader.

Use it whilst it is still working :). Download, unpack, and run:

NetMassDownloader.exe -output bcl
    -d C:\Windows\Microsoft.NET\Framework\v2.0.50727

I executed in in Windows XP as a limited user and have seen numerous failure messages printed out. Nevertheless, 120MB of data was downloaded and I had all the files I wanted. The authors advise though to run it as a privileged process (at least on Vista).

The most interesting parts are bcl/redbits/ndp/clr/src/BCL/System and bcl/redbits/ndp/fx/src where bcl is the name of the directory you provided after -output switch. Together they are about 75MB.

It was pretty interesting to read the implementation of the classes I had been using for so long: String, Hashtable, Arraylist. They are easy to understand, with lots of comments, and of course in C#.

The classes I have read are all pretty basic and no wonder there are lots of pointers and calls into unmanaged code, but it is still readable.

One surprising thing, but obvious after knowing it, is that " s ".Trim() will not create a new string "s", but return a slice, an object that holds a reference to the original string and knows where are the beginning and the end. This is possible thanks to the immutability of strings. I bet the substring method is implemented like this too.

There are also some monster methods to be seen. One example is internal HttpListener.HandleAuthenticate spanning more than half a thousand lines.

BTW source code for mono is available at mono-project.com/AnonSVN. Source for mono's BCL is under mcs/class/corlib. It might be interesting to compare the two.

Why Vim's modes frustrate newbies

noreply@blogger.com (Kamil Dworakowski) — Sat, 01 Mar 2008 14:09:00 +0000

Some people say they hate the Vim's modes. Puzzling, because I consider the modes as the killer feature of Vim, something that makes it superior to Emacs.

Why modes rock

I spend most of the time in the normal mode. In this mode, undo is just u, delete line dd and move one word forward w. In Emacs these commands would all include ctrl, shift, alt or whatever. Inserting text is not what I spend most of the time in an editor on. Most of the time I need to quickly navigate around and make small modification to an already existing text. It is false economy to have the keys on the keyboard be bound to less often used functionality all the time.

When I reach the exact place where I need to enter some text, I press i to enter insert mode, type it, and immediately get back to normal mode (I get to normal mode by pressing ctrl-[ rather than ESC).

Why modes frustrate newbies

While pairing with people new to Vim I noticed that they leave it in whatever mode. So if they were inserting, they live it in insert mode. When they come back to Vim window (after inspecting results of a test run for example), they think they are typing normal-mode commands, but are actually in the insert mode typing text. Or the opposite situation: start typing and instead of inserting text they are rapidly killing one chunk of the file after another.

Experienced Vimers follow a convention. After doing something briefly in other mode, immediately go back to the normal mode. I never live Vim in the insert mode or any other mode than normal; when I get back to the Vim window I can always assume Vim is in the normal mode.

xmonad

noreply@blogger.com (Kamil Dworakowski) — Sun, 17 Feb 2008 15:40:00 +0000

I am running xmonad! I had some troubles configuring it though.

I am running Ubuntu Gutsy Gibbon and had to compile it from sources which turned out to be easy, but then the instructions to actually run it didn't work. I have finally got it going by following instructions in FAQ about using xmonad with a display manager.

Before I got xmonad working I tried wmii, which was readily available in repositories. After installing it (sudo apt-get install wmii) you can log out, and in gdm (session manager) change session to wmii to finally log in and enjoy wmii. wmii would be good for me had I no dual head configuration. It was treating my two monitors as one big virtual screen. This meant that a part of it was not visible to me as my left monitor is shorter than the right.

xmonad has much fewer lines of code that wmii. I have seen it a few times already, Haskell programs being very small in comparison with alternatives in other languages. wmii is proud to keep it small in under 10 000 lines of code, while xmonad is written in less then 1300 lines of Haskell. Additionally, there are 583 lines in tests. Not bad considering that xmonad turned out to be better then wmii.

I got these numbers by running the following command on ver 0.6 sources, it ignores blank lines and comments.

find . -name \*.hs -exec cat {} \; |\
    egrep -v "^ *$|^ *--" | wc

BTW xmonad is cool.

Debugging memory problems in an IronPython app

noreply@blogger.com (Kamil Dworakowski) — Mon, 04 Feb 2008 22:41:00 +0000

Introduction

This blog entry contains notes on debugging memory problems in an IronPython application on Windows. I also provide scripts that are helpful in the process. I should also mention that it is specific to IronPython 1.*.

The Debugger I was using is windbg.exe. It is a GUI application, but feels more like a command line. It is certainly more powerful than Visual Studio Debugger.

First steps

First thing after starting windbg should be loading SOS.dll, which is an extension to windbg providing commands specific to debugging .NET apps (all starting with !). There is !help command that provides description for these commands (invoke after loading sos).

 
.load sos 
!help

If you have problems it might be because of the wrong sos.dll version. There is a different version for each .net runtime. Since IronPython runs in .net 2.0 you need sos for that specific version. You can find it in the .net sdk directory and specify the full path for the load command.

Say freeze ;)

Windbg can attach to a process by id (F6 shortcut, but there is also a menu item for it). From that moment on, the debugee is freezed until you detach from it.

Before we actually do any debugging we need to have something to debug. Prepare application so that after it does some considerable leaking, there is a moment in its execution that you would attach windbg to it. This moment should be after doing thorough garbage collection to ensure that during debugging we are dealing only with alive objects. You could have code like this:

from System import GC
from System.Diagnostics import Process
from System.Threading import Thread
for _ in range(2):
    GC.Collect()
    GC.WaitForPendingFinalizers()
print "attach now to process id %d" %\
        Process.GetCurrentProcess().Id
Thread.Sleep(100000)

Next thing is to find objects that leak. In our case the biggest ones were Dictionaries so it was enough to just list all dictionary objects. Sometimes it is unclear which objects leak, this blogpost suggest a strategy for finding out leaking objects. For this task you will find the dumpdiff.py helpful.

This following listing shows output of !dumpheap command the heap with given substring in the type name. This is always the first command after attaching to a process.

!dumpheap -type System.Collections.Generic.Dictionary
...
1b489a58 000e2e00      124     
02713600 000e2e00   134720     
02b756b0 000e2e00  2503008     
24b496a0 000e2e00  2503008  
...

Now copy one of the addresses and issue gcroot command to find some of the chains of references holding the object alive. I'll take the last address. It is worth mentioning that 24b496a0 points to an array that holds the entries of the dictionary.

!gcroot 24b496a0
...
13ab6360(<Unloaded Type>)->
13ab63bc(IronPython.Runtime.FieldIdDict)->
13ab63c8(System.Collections.Generic.Dictionary`2[[IronPython.Runtime.SymbolId, IronPython],[System.Object, mscorlib]])->
13ab6424(System.Collections.Generic.Dictionary`2+Entry[[IronPython.Runtime.SymbolId, IronPython],[System.Object, mscorlib]][])->
186787d4(<Unloaded Type>)->
...

The previous listing shows only a fragment of the output of the !gcroot command. Arrows indicate directions of references. In the output of !gcroot command, the root (a static object or value on some stack) will be at the top whereas the object you invoked the !gcroot with will be towards the bottom.

Difficulties in debugging an IronPython app

Python is a dynamic language and that makes it more difficult to debug IronPython than C# for example (with windbg). There are two problems. Firstly, types defined in IronPython appear under windbg as Unloaded Type. Even if the type names would be shown (no clue why they are not), they would be C# types (or should I say .NET native types) like OldType and UserType: the guts of the IronPython implementation, not the classes defined in our IronPython code.

Secondly, the attributes of an object are not fields. They are stored in a dictionary, and to make things worse, that dictionary is not keyed by strings but by SymbolIDs. Somewhere in memory lies a table containing strings to which SymbolIDs hold indexes.

To find out the attribute name or the type name there are many commands needed. I have created two scripts to automate it. Explaining the windbg scripts is out of scope for this entry, but I think I will post about it soon.

attr script

The first script, attr, takes two addresses as arguments, one to the array holding the dictionary entries, second to the value of the attribute, and prints the name of the attribute. Below I include the same fragment of the gcroot output plus example use of the attr. It will display the name under which 186787d4 is kept in the __dict__ of 13ab6360. Note that I pass to attr the address of the array not 13ab6360.

... 13ab6360(<Unloaded Type>)->
13ab63bc(IronPython.Runtime.FieldIdDict)->
13ab63c8(System.Collections.Generic.Dictionary`2[[IronPython.Runtime.SymbolId, IronPython],[System.Object, mscorlib]])->
13ab6424(System.Collections.Generic.Dictionary`2+Entry[[IronPython.Runtime.SymbolId, IronPython],[System.Object, mscorlib]][])->
186787d4(<Unloaded Type>)->
...
$$>a<c:\path\attr 13ab6424 186787d4
Name: System.String
MethodTable: 790fc6cc
EEClass: 790fc62c
Size: 110(0x6e) bytes
(C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String: _GUIThreadTestCase__guiControl

The above listing assumes you save the attr script in c:\path, and the dollars are required, pounds will not do.

Make sure you invoke it giving an address of a dictionary entry array keyed by SymbolIds like in the above example.

So the above gives the name of one attribute. To get more attributes you can invoke !dumparray on the entry array, which will display all the values there which you can then check individually with attr script (or write another script to automate it ^^).

!dumparray -details <entryArrayAddress>

The attr script may not work if the executable you are debugging is running from a path that contains spaces. There is one constant in the script that would have to be increased to make up for that additional space.

usrtype script

The next script is less reliable that the first one, it worked half the time I invoked it on a Unloaded Type object. I think it works for UserType, so I named it usrtype. It takes just one argument which is the address of an object and returns the name of its Python type.

$$>a<c:\path\usrtype <address>

I again assume you have it saved in c:\path.

!dumpobj

Now the most frequently used command, !dumpobj. It displays the values of the fields of an object. Of course it displays only the fields defined in the .net class. It knows what the object's type is, because each object on the managed heap has the pointer to its type in its first 4 bytes (or 8 if it is 64 bit machine).

Since all the leaking I was chasing ware caused by design flaw in implementation of events in IronPython, I usually looked for object of type ReflectedEvent in the chain displayed by the !gcroot. ReflectedEvent stores the name of the event. It was usually also useful to inspect the IronPython.Runtime.Calls.Method further down the chain, from which you can get the name of the method handling the event as well as the class this method is defined in.

Tips & Tricks

I hope you will find them useful.

Testing that the memory leak is gone

So you got rid of the memory leak, but how do you test that it is gone? I mean an automated unit test!

obj = objectThatMightLeak()
weakRefToObj = WeakReference(obj)
del obj
for _ in range(2):
GC.Collect()
GC.WaitForPendingFinalizers()
assert !weakRefToObj.IsAlive

WeakReference will allow GC to collect the object, and after collection it will answer if the object was garbage collected. The del obj line is very important, ware it not there the test would have no chance of passing.

Marking objects

Let's say you suspect that some object is not being garbage collected, and its class is defined in Python. Because of the dynamic nature of Python, its class is not represented as .NET type, hence you can't simply find it with !dumpheap -type <type_name> command.

My mother works in laboratory, when she analyzes samples, she sometimes marks them with special substance to make some elements stand out. You can mark Python objects in a similar way.

from System import GopherStyleUriParser
pythonObject.tag = GopherStyleUriParser()

GopherStyleUriParser is a good enough mark, there is low probability you will have a second one on the heap, and because it is .NET type you can find it easily with !dumpheap -type Gopher. Then invoke !gcroot on it and you will find that the chain goes through your object.

Update: Previously I was recommending using ArithmeticException instead of GopherStyleUriParser for the tag; don't do this. As soon as you would get rid of the other leaks, the object would be kept alive by the instance of ArithmeticException, which captures the stack trace.

Nuking events

That one is a dirty hack. All the trouble I had with memory leaks were because of the events, how about nuking them?

The reason the IronPython team didn't just use the .net implementation of events was that they wanted to allow removing handlers by object equality, rather then by referential equality. By referential equality I mean comparing objects' references (is operator in Python).

So how about defining an object that is equal to everything?

class EqualityForAll(object):
def __eq__(_, __):
return True

That is a very fine fellow, which we can use to get all the handlers out of an event.

e = EqualityForAll()
for _ in irange(100)
control.OnClick -= e

Ok, that will kill at most 100 handlers, but there is no way you can find out how many there are. And not, it will not throw an exception if there are no more handlers. If I remember correctly from looking at the implementation code (open source is great), there are some exception being thrown under the hood, but are swallowed higher up. This makes the operation costly if there are no handlers, as the exception in .NET are much more expensive than in CPython.

Summary

I thought that finding and stopping the leaks would be easy, but it took many days. Perhaps fixing the IronPython implementation could have taken less time. Anyway, if you think that the bug should be fixed, please vote for it. It was given a low priority and it is not easy to do, so without votes it might not make it to IronPython 2.0.

Arc

noreply@blogger.com (Kamil Dworakowski) — Thu, 31 Jan 2008 01:35:00 +0000

I have just read the Arc tutorial. Here are some comments.

Since functions of one argument are so often used in Lisp programs, Arc has a special notation for them. [... _ ...] is an abbreviation for (fn (_) (... _ ...)). So our first map example could have been written

arc> (map [+ _ 10] '(1 2 3))
(11 12 13 . nil)

I know this feature from Nemerle, and miss it when programming in other languages.

Another nice one seems to be ~ operator for negating a function, though would not be nearly as useful as the last one.

arc> (map ~odd '(1 2 3 4 5)) 
(nil t nil t nil)

Code for adding an element to a hashtable does not look very natural:

arc> (= (airports "Boston") 'bos)
bos

In Python it would be:

airports["Boston"] = 'bos

Resolver One for Open Source

noreply@blogger.com (Kamil Dworakowski) — Wed, 30 Jan 2008 01:47:00 +0000

Resolver One, a new MS Excel's alternative, is currently available in two versions. First is a vanilla commercial one, whereas the second, Resolver One - Non-Commercial, requires a bit of explanation.

Resolver One - Non-Commercial is free (as in beer). Internally at Resolver we used to call it Resolver One for Open Source. Not because it comes with sources, because it doesn't, but because it could only be used to produce open source. Since then we have further relaxed the license to allow personal use.

What a user gets after installing Resolver One Non-Commercial is feature-wise the same application as in the commercial one, but with a different license. The license requires the user to make his spreadsheets open source or mark them as personal -- not for redistribution and profit.

Technically, the Non-Commercial version will not save the spreadsheet file until user decides on a license for his spreadsheet. When he tries to save the file for the first time, he will be presented with seven well known open source licenses to choose from (MIT, GPL, Creative Commons, ...), but he can also mark it as personal. If he selects one of the open-source licenses, it is included on top of the code part of the spreadsheet, making his spreadsheet free (as in speech).

Anti-pattern: static subject to observer mapping

noreply@blogger.com (Kamil Dworakowski) — Sun, 06 Jan 2008 16:39:00 +0000

Short description

When implementing observer pattern in a language with automatic garbage collection don't use static mapping from subject to observer. Doing this will cause memory leaks.

Such an implementation involves having a hashtable which keys are subjects and values are lists of observers. Then to notify about a change in a subject, get the list of its observers from the mapping and notify them one by one.

Glossary

GC - Garbage Collector.
Dead object - an object is dead when it is eligible for garbage collection (can't be reached by strong references from any root). By symmetry, an object can be called alive.
Strong reference - a reference that keeps an object alive (normal reference), as opposed to a weak reference.
Weak reference - a reference that is not taken into account by GC. In .NET there is WeakReference class to define weak references. As far as I know this is the only way to define a weak reference.

What is wrong with static subject to observer mapping

For a warm-up I will discuss the traditional implementation of observer pattern. The nodes on the diagram represent objects. An arc from one object to another means that it is possible to get from one to the another following strong references.

In the first diagram, observer can be reached from subject and vice versa. The fact that observer can be reached from subject follows from subject keeping a list of observers to notify.

I would like to point out that the arrow from observer to subject on this diagram indicates that there is, maybe indirect, way by strong references from observer to subject, this does not follow from observer pattern. However this being optional, in production code it is quite often that observers keep references to the subjects.

What happens in this case when there are no more references to both subject and observer (apart from their own cycle of references)? They can't be reached from any root, so they are dead and the memory they occupy will be released during next garbage collection.

The second diagram shows a situation where observers are kept in a weak hashtable. The dashed arc from entry to subject indicates a weak reference. This one is not that bad. If there are no references to subject and observers apart from those on the diagram, the subject can be garbage collected, there being only a weak reference to it. The observer will be kept alive by the strong reference from weakhashmap, but WeakHashMaps are usually implemented so that on adding a new entry they will remove entries which keys have been garbage collected. This prevents entries with dead keys to pile up.

The third and last diagram depicts a case where there is a reference from observer to subject. This is the situation which results in memory leaks. In this configuration the weakHashMap will keep both subject and observer alive and every other object that can be reach from either of them.

This happens because the reference from entry to the value must be strong, otherwise an observer might get garbage collected when its subject is still alive.

Known occurrences

I have first come across this anti-pattern in the paper about implementing GOF design patterns in AspectJ by Jan Hannemann and Gregor Kiczales. Gregor Kiczales is renowned for leading the team in XEROX PARC that invented aspect-oriented programming.

In this paper they have a definition of an aspect, ObserverProtocol, that keeps a WeakHashMap from subjects to list of observers. This mapping not being static per se, is kept alive throughout the lifespan of a program execution.

The second use I have seen was in IronPython's implementation of events. IronPython has it's own implementation of events because the one in .NET runtime is not sufficient for IronPython's requirements.

For IronPython programmers it means that hooking to events will cause memory leaks if source of the event can be reached from the handler (very common).

Summary

Both of these cases show how easy it is to make this mistake, therefore I have documented it as an anti-pattern for the benefit of mankind ;).

All possible FontStyles

noreply@blogger.com (Kamil Dworakowski) — Thu, 03 Jan 2008 21:44:00 +0000

That was funny. To instantiate Font in .NET 2.0 you call one of many Font's constructors. Basically you provide a font family (either an object of the class FontFamily or a name of one as a string), size and FontStyle. There is a constructor that doesn't take a FontStyle, it assumes the FontStyle.Regular as default.

A FontFamily is a collection of fonts. Arial family will have Arial Regular, Arial Bold, Arial Bold Italic...

Now, in theory there might be a family with only Bold Italic. The requirement is in case there is no style the user requested, give back a font from the family in any available style.

Unfortunatelly in FontFamily object there is no property returning available members of the family, but there is a method for checking if given FontStyle is available.

FontStyle is an enumeration with the following members.

FontStyle.Regular = 0
FontStyle.Bold = 1
FontStyle.Italic = 2
FontStyle.Underline = 4
FontStyle.Strikeout = 8

A font style can be created by bitwise or-ing the values. For example Bold Italic can be defined by FontStyle.Bold | FontStyle.Italic wihich makes 3.

How to generate all possible combinations?

for i in xrange(16):
    fontStyle = Enum.ToObject(FontStyle, i)
    if fontFamily.IsStyleAvailable(fontStyle):
        return Font(fontFamily, fontSize, fontStyle)

Integers from 0 to 15 form all valid combinations of the FontStyle. This sixteen numbers can be represented with 4 bits, 15's binary representation having them all lit (8 + 4 + 2 + 1).

BHO.CVX Trojan

noreply@blogger.com (Kamil Dworakowski) — Fri, 21 Dec 2007 21:16:00 +0000

My friend got this on his computer. The most annoying thing for him was that his antivirus software would popup a warning message about this Trojan every now and then.

The BHO probably stands for Browser Helper Object; I could find it inside the Add-ons to IE, but deactivating it didn't help. Thankfully the antivirus was yelling which file the Trojan is in, it couldn't delete it because it was in use. Even in safe mode the file was locked.

I was able to delete it from command line after booting up the computer using windows install CD and choosing Recovery mode (or sth like that, I have German Windows :) ).

Evil locale

noreply@blogger.com (Kamil Dworakowski) — Sun, 16 Dec 2007 22:09:00 +0000

.NET is trying to protect developers from making simple mistakes while providing them with all sorts of non obvious problems completely for free.

The simple mistake is trying to mutate a struct returned by a property. Line 14 in the example code raises compile time error. This line is otherwise completely fine except that it doesn't make sense -- the Start property is returning a copy of the value type Point.

1 struct Point {
2     public int x;
3     public int y;
4 }
5 class Line {
6     Point start;
7     Point end;
8     public Point Start {get {return start;}}
9     public Point End {get {return end;}}
10 }
11 public class Main {
12     public static void Main() {
13         Line l = new Line();
14         l.Start.x = 2
15     }
16 }

The example comes from here The page explains that in such a case IronPython will also fail (by design) with an error.

Now about the free errors. They are all sponsored by the evil locale. Parsing in .NET takes the operating system's locale into account. In Poland the floating point numbers are written using coma instead of a dot. So 2.3 is written as 2,3. Trying to parse your settings file containing floats on a computer with Polish locale will throw an exception if there are any dots in them.

But it is not just parsing. If you invoke ToUpper on a string it also takes locale into account. In Turkey they have dotted i and dotless ı. The letter i (the dotted one obviously) has upper case equivalent which has a dot above it (expressible in unicode). If you have good fonts you should be able to see it right here: İ.

One more thing, when dealing with DateTime think about Buddhist Era in Sri Lanka :).