Cheap Turpentine: Ori Peleg's Blog

Python: change and restore the working directory

2012-07-22T22:43:00.000+03:00

Occasionally a command should be run from a specific directory. Here's a Python helper for this:

try/finally ensures the original working directory is restored.
contextlib.contextmanager is a neat way to write things supporting the "with" statement ("Context Managers").

import os, contextlib @contextlib.contextmanager def temp_chdir(path): """ Usage: >>> with temp_chdir(gitrepo_path): ... subprocess.call('git status') """ starting_directory = os.getcwd() try: os.chdir(path) yield finally: os.chdir(starting_directory)

Hg vs. Git in the workplace

2010-09-01T01:51:00.000+03:00

This post is based on the following mail I sent to the ALT.NET Israel list. I'm discussing a Windows development environment.

I'm much better at git than hg, and I use git a lot for real work, and it's great (no ifs or buts).
I also helped originate the move to git at work, because the transition plan was much better. I evaluated hg first and loved it, but the transition plan wouldn't have been half has smooth (a better hg-svn bridge may have been written since then).

Still, here are - in order - the reasons I thing hg would fit better at my workplace (and had we found a good transition plan we'd probably be using hg instead of git):

Friendlier commands and interface. With git I feel like I need to research how to do straightforward things. Git is extremely powerful and I can do anything with it, and I'm strange so I like the arcane, but git requires more of an investment from people to use it effectively when they have other things to do. Some things are arguably easier to understand in git (rebasing, for example, vs. hg's patch queues - although that's debatable) but the whole package leans heavily to hg IMO. This is the least-loved aspect of git among the teams.
In hg, commits remember the named branch they were committed to (see here). I would LOVE to have this in git, but no cigar - we're still playing with different schemes of approximating this.
With many people working at times on several branches, and with merging done between branches, then given commit XYZ it's nontrivial in git to figure out which branch it belongs to.
This is especially true in some scenarios we encountered - e.g. human error caused an older release branch to be fast-forwarded to the head of the latest devel branch - few traces are left in the repository itself of the previous state, and we had fun a few times fixing this (can take hours). Now we have a hook to prevent this.
If we'd have cherry-picked between different branches instead of merged then this would be easy (and this is equivalent to svn's merging, only better), but we like git's merge and we want to use that.
hg has less quirky Windows support. Points to consider:
- msysgit gets sporadic releases and is usually months behind the stable git version, the latest hg versions just work.
  - This hit us a few times (bug has been fixed but not available in msysgit).
  - Current example: Git 1.7.2 has improved crlf handling but msysgit isn't there.
- Being able to "hg serve" your repository is incredibly cool, but "git daemon" doesn't work with msysgit
  - Think continuous integration that watches your local repository and runs builds with your current code, giving up-to-date feedback
  - collaborating with someone without pushing a publicly-visible branch (I've actually wanted to do that a few times)
  - developing and testing on more than 1 machine simultaneously. There are workarounds, but "git daemon" would have been perfect.
- automating hg, including hooks, run in a Windows environment (vs. git where it runs in mingw's MSYS environment)

What doesn't bother me much for the long run is tooling. Right now hg seems to have better tools for our scenarios (even including much better Trac support), but I think git's extreme popularity will cause better and better tools to be written. Some people were very sad to let go of their VisualSvn / Ankh plugins, but they manage with Git Extensions (mostly) and occasionally TortoiseGit. I was surprised at how few people use git from the command-line, I can't live without it.

jQuery Validate: Required If Visible

2010-06-28T19:34:00.000+03:00

I needed a generic "required if visible" rule for use with the jQuery Validation plugin. This is the result:

Notes:

Using required:"#myinput:visible" could have worked, but ":visible" doesn't check for hidden parents, and I didn't want to repeat the element selector.
":hidden" is false for elements with the "visibility:hidden" CSS rule, so I couldn't use ":not(:hidden)" either.
Technique for checking element visibility from Remy Sharp's blog.

MVC's RenderPartial in a WebForms page

2010-04-25T14:58:00.000+03:00

We have a mixed ASP.NET MVC/WebForms application, slowly transitioning to all-MVC. Obviously, sharing components between MVC and WebForms is important to us.

Using WebForms controls in MVC views is easy enough (minus the postbacks), but when we rendered partial views in a WebForms page using Keith Henry's code we occasionally got evil "Validation of viewstate MAC failed" errors. Microsoft doesn't think hybrid WebForms / MVC apps are relevant, but then Mauricio Scheffer came to the rescue. Instead of having the partial views subclass ViewUserControl, we now subclass Mauricio's ViewUserControlWithoutViewState - problem solved.

TeamCity 5.0 upgrade - "out of memory" error

2010-04-12T15:08:00.000+03:00

I upgraded JetBrains TeamCity to version 5.0 on Windows (for Git support, yay!). The previous version upgrades went smoothly, but this time I got an exception when loading the server:

java.sql.SQLException: out of memory

After some googling, I thought I had to:

Run ...\TeamCity\bin\tomcat6w.exe //ES//TeamCity
In the "Java" tab change the value for "Maximum memory pool"
Not set it too high

But no cigar. I realized that I probably had a corrupt HSQLDB database, and I ended up reinstalling and copying the config by hand (which was a little frustrating). I don't think there's a real reason why an embedded SQL database should be less dependable than one with a server (I know that from SQLite), so I'm guessing it's either TeamCity's or HSQLDB's fault. Even so, an "export configuration" button would be nice, and take the pain out of reinstalling TeamCity.

Encoding usernames and passwords in FTP URIs

2010-02-14T19:22:00.000+02:00

I want to encode the following into a single URI:

Username: 'jane+foo@example.com'
password: 'my password' (with a space)
ftp server: 'ftp.example.org'

The FTP URI that I expect:

ftp://jane%2bfoo%40example.com:my%20password@ftp.example.org

In .NET, HttpUtility.UrlEncode almost does what I need - it hex-encodes the special chars, but it also converts spaces to "+" which the FTP server interprets as a literal "+".

Final code that works with my tests using the FileZilla FTP Server:

string EncodeFtpUserInfoComponent(string s)
{
  // assume or assert s != null
  return HttpUtility.UrlEncode(s).Replace("+", "%20");
}

PowerResizer Keyboard Shortcuts

2010-02-02T20:04:00.001+02:00

For some reason the keyboard shortcuts / hotkeys for Federico Bastianello's PowerResizer aren't on the web, only in the "Readme.txt" installed with it. So, as a public service, here they are (for version 0.95):

2.1 - Window Docking Features and Keyboard Shortcuts

 - Move Window + A: dock a window to the left-half of the screen.
 - Move Window + S: dock a window to the right-half of the screen.
 - Move Window + W: dock a window to the top-half of the screen.
 - Move Window + Z: dock a window to the bottom-half of the screen.
 - Move Window + 1: dock a window to the top-left corner of the screen.
 - Move Window + 2: dock a window to the top-right corner of the screen.
 - Move Window + 3: dock a window to the bottom-left corner of the screen.
 - Move Window + 4: dock a window to the bottom-right corner of the screen.
 
When you resize the inner border of those windows you docked to one of the four
corners of the screen or to one of the four screen halves, PowerResizer 
changes the windows' behavior during resize in a way that all docked windows
will follow the new size of their neighboors.
It is possible to mix docked windows in halves of the screen and in corners.

2.2 - Window Moving/Resizing Features and Keyboard Shortcuts

 - Resize Window + CTRL: resize a window mantaining fixed its center point.
 - Resize Window + SHIFT: move a window using its border.
 - Move Window + 5: center a window into the screen (1/16 screen size margin).
 - Move Window + 6: center a window into the screen (1/10 screen size margin).
 - Move Window + 7: center a window into the screen (1/6 screen size margin).
 - Move Window + 8: center a window into the screen (1/4 screen size margin).
 - Move Window + 9: center a window into the screen (1/3 screen size margin).
 - Move Window + 0: maximize a window.
 
2.3 - Screen window reposition Features and Keyboard Shortcuts

 - CTRL + ALT + PAGE UP: Rotate docked windows counter-clockwise.
 - CTRL + ALT + PAGE DOWN: Rotate docked windows clockwise.
 - CTRL + ALT + HOME: Reposition windows.
 
Note: the keyboard shortcuts in that section, are system-wide shortcuts and 
they will work on windows docked on the monitor containing active window.

Console colors on Windows in .NET

2009-10-29T17:31:00.000+02:00

Changing the console output color is easy on *nix terminals with escape sequences, but has a funky API on Windows (see the color support code from Testoob for a Python example with pywin32/ctypes).

It turns out that there's a very nice .NET API for it, though (there's a nice article by Sam Allen on the Dot Net Perls site). This would be awesome for Testoob's IronPython support.

Include literal file contents in ASP.NET

2009-10-26T10:47:00.000+02:00

I was looking for something similar to Velocity's #include for ASP.NET. <% Response.WriteFile([filename]); %> works well, but a .NET control for this could be nice. Here is my attempt (MIT license):

using System.IO;
using System.Web;
using System.Web.UI;

namespace FooControls
{
  public class LiteralContentFromFile : Control
  {
    public string Path { get; set; }

    protected override void Render(HtmlTextWriter writer)
    {
      var server = HttpContext.Current.Server;
      var physicalPath = server.MapPath(Path);
      var fileContents = File.ReadAllText(physicalPath);
      writer.Write(fileContents);
    }
  }
}

Usage in an .aspx file (WebForms or ASP.NET MVC):

<%@ Register TagPrefix="foo"
  Namespace="FooControls" Assembly="FooControls" %>
...
<foo:LiteralContentFromFile runat="server"
  Path="~/path/to/file" />

Edit: subclassing System.Web.UI.Control instead of System.Web.UI.WebControls.WebControl which added unnecessary and unwanted behavior (for example, wrapped the output with <span>).

Issues API for Google Code Hosting

2009-10-22T01:43:00.000+02:00

Google have recently released an issue-management API for projects hosted on Google Code.

That's good news for Testoob - I've been wanting to import its old Trac tickets for ages!

Testoob 1.15 released

2009-10-08T02:30:00.001+02:00

I'm very happy to announce that after nearly 3 years Testoob sees another release.

Testoob is the advanced Python test runner and testing framework that spices up any existing unittest test suite.

Changes: Version 1.15 (Oct. 2009) adds better Python 2.6, IronPython, and Jython support, as well as test coverage improvements, better color support, and some new options and bugfixes. A full list of changes is available in the release announcement.

A big shout out to Ronnie van 't Westeinde - without him this version wouldn't have been possible.

Options to install:

With easy_install: "easy_install -U testoob"
One of the available packages

IDCC 2009 Registration Is Open

2009-09-09T15:54:00.000+03:00

The response for the call for content for IDCC 2009 has been impressive, and now two tracks are open with great talks.

Hurry up and register!

"Enter" submits my ASP.NET form!

2009-09-07T16:43:00.000+03:00

ASP.NET encourages web pages to be written as one big <form>, and every interaction with it to be a POST to the server ("Postback" in ASP.NET). This means pages often have multiple auto-generated <input type="submit"> elements.

On some browsers, pressing Enter causes the first submit button to be "clicked". In pages with many such buttons, that's usually not what you want. As a hack to make "Enter" do nothing on an already-built page, I added the following before any other submit button:

<div style="visibility:hidden;height:0;">
  <input type="submit" value="Dummy" onclick="return false;" />
</div>

Now this do-nothing, invisible submit button "catches" the Enter and prevents the next button from being "clicked". This requires JavaScript turned on, but ASP.NET's "Postbacks" require them anyway, so no big loss.

Implementation note: using 'display:none;' caused the browser to ignore it, so I had to resort to 'visibility:hidden;' and then 'height:0;' to impact the layout as little as I could. There could be cross-browser styling issues here, my specific case didn't need anything special.

Python: checking if an executable exists in the path

2009-08-24T15:54:00.001+03:00

I wanted to do this on Windows. Based on this StackOverflow question and on Trey Stout's and Carl Meyer's responses I made the following snippet:

def exists_in_path(cmd):
  # can't search the path if a directory is specified
  assert not os.path.dirname(cmd)

  extensions = os.environ.get("PATHEXT", "").split(os.pathsep)
  for directory in os.environ.get("PATH", "").split(os.pathsep):
    base = os.path.join(directory, cmd)
    options = [base] + [(base + ext) for ext in extensions]
    for filename in options:
      if os.path.exists(filename):
        return True
  return False

JavaScript Namespaces

2009-07-22T21:43:00.000+03:00

It's important to namespace your JavaScript code. I want one "master" namespace, and everything else under it (like YUI's namespacing approach). I fell into some potholes of browser compatibility on the way (with var foo; if (!foo) foo = {};), but here's what I ended up with:

// define master namespace
if (typeof foo === "undefined") var foo = {};

(function(){ // lexical scoping, allows "private" variables and functions

// define sub-namespace
if (!foo.bar) foo.bar = {};

// these don't leak outside
function helper() { /* ... */ } // "private" function
var counter = 0;                // "private" variable

// this function shows up globally under namespace foo.bar
foo.bar.func1 = function(x) { /* ... */ };

// not just functions
foo.bar.constant1 = 12;
foo.bar.shared_data = [];

})(); // end of lexical scoping

This way all the JS files can use the same namespace hierarchy, and can still be defined separately. No JS library necessary.

IDCC 2009, call for content

2009-07-15T11:31:00.000+03:00

The Israeli Developers Community Conference 2009 has started its call for content. It's a community conference centered on the .NET world.

If you have a session to propose, please do! The community will vote on the proposals, and the conference will conform to the participants - not the other way around.

Fresh instances in Python with lazy evaluation

2009-06-05T17:17:00.001+03:00

Testoob's --capture feature (written by Misha) replaces sys.stdout and sys.stderr for each test being run, and displays the output only if the test fails.

Leeor Aharon, of ff-activex-host fame, wanted to use it with Python's logging module, but the StreamHandler class stores a reference to the output stream on initialization - it already has a reference to sys.stdout, so replacing it won't affect it.

We originally thought of sublcassing StreamHandler and making retrieve the logger from a property, but we came up with this elegant code instead:

 
class LazyEvaluator(object):
    def __init__(self, factory):
        self.__factory = factory
    def __getattr__(self, name):
        return getattr(self.__factory(), name)

lazy_stdout = LazyEvaluator(lambda:sys.stdout)
handler = StreamHandler(strm=lazy_stdout)

The "lazy evaluator" is initialized with a factory callable, and every time it tries to access an attribute or method the object will be re-created by the factory. No changes necessary for StreamHandler, and ./alltests.py --capture works like a charm.

jQuery UI scoped themes and the Dialog widget

2009-05-27T10:47:00.001+03:00

There's a problem using scoped themes with jQuery UI's Dialog widget in the 1.6 and 1.7 releases (and possibly 1.7.1 as well).

Christoph Herold mentioned his workaround - wrapping the dialog's DOM with a scoping div (with the scoped theme's class name). I wrote a small jQuery plugin that does this and works fine so far:

// Wraps the dialog in a div with the scoping class on
// dialog open, removes the scoping class on dialog close
$.fn.dialogScopingWorkaround = function(scopingClass) {
  this.bind('dialogopen', function(event, ui) {
    $(this).parents(".ui-dialog").wrap(
      '<div class="dialogScopingWrapper '
      + scopingClass
      + '"></div>'
    );
  });
  this.bind('dialogclose', function(event, ui) {
    var wrapper = $(this).parents(".dialogScopingWrapper");
    wrapper.replaceWith(wrapper.children());
  });
  return this;
};

Example with the "smoothness" theme scoped with class "theme-smoothness":

$('#dialog').dialogScopingWorkaround("theme-smoothness").dialog();

// Must call the workaround before opening the dialog
// so the 'dialogopen' event is bound.

IE7 compatibility image wants to install IE8

2009-05-17T17:05:00.000+03:00

I use Microsoft's IE Application Compatibility virtual machine images to test web sites with IE6 and IE7. Well, today my IE7 virtual machine wanted to automatically update to IE8. Great for the web, funny for an IE7 compatibility test image :)

Ajax.NET Professional "onTimeout" exception

2009-05-14T22:38:00.001+03:00

We just upgraded to Ajax.NET Professional 9.2.17.1 and started seeing these JavaScript exceptions in file "core.ashx":

this.onTimeout is not a function

It turns out this is a known issue with synchronous calls done with AjaxPro's client-side JS. We should probably have ditched it and used regular jQuery AJAX calls on the client side (like here and here) but we haven't gotten around to that yet.

Our workaround was easy - we were making a "fire and forget" call, so it didn't need to be synchronous (we added an empty callback handler to the arg list). We could have also patched the JS ourselves and:

rebuilt AjaxPro with the patched version
made AjaxPro load the patched version
not used RegisterTypeForAjax and added the script references ourselves - pointing to the patched version, and possibly taking the opportunity to combine and minify AjaxPro's JS files.

Reddit and Digg icons on Blogger without external scripts

2009-03-17T02:31:00.000+02:00

It took me a while to be able to add reddit and Digg icons that didn't load external scripts - Blogger's templating language had me stumped. I used reddit's, Digg's, and Blogger's documentation, but I would never have gotten the "onclick" escaping right without this post.

<!-- reddit button -->
<a expr:href='"http://www.reddit.com/submit?url=" + data:post.url' expr:onclick='"window.location = \"http://www.reddit.com/submit?url=\" + encodeURIComponent(\"" + data:post.url + "\"); return false"'> <img src="http://www.reddit.com/static/spreddit1.gif" title="submit to reddit" alt="submit to reddit" border="0" /> </a>
<!-- digg button -->
<a expr:href='"http://digg.com/submit?url=" + data:post.url' expr:onclick='"window.location = \"http://digg.com/submit?url=\" + encodeURIComponent(\"" + data:post.url + "\"); return false"'> <img style="background:white;" width="16" height="16" alt="Submit Story to Digg" title="Submit Story to Digg" src="http://digg.com/img/digg-guy-icon.gif"/></a>

P.S. - if you can't find the "data:post.body" element to add the buttons after, like I couldn't, see if you forgot to check the "Expand Widget Templates" checkbox under "Layout -> Edit HTML".

Prefetching JavaScript (or anything) with jQuery

2009-03-13T00:21:00.000+02:00

While users are logging into a web site, I thought why not prefetch some JavaScript files they'll be needing on the next page?

I could load them with Ajax and this will be invisible to users (see the Even Faster Web Sites talk by Steven Souders, and his discussion of browser busy indicators).

The following seems to work, prefetching 2 files in parallel:

(function($) {
 $.ajax({ url:"/js/file1.js", cache:true, dataType:"text" });
 $.ajax({ url:"/js/file2.js", cache:true, dataType:"text" });
})(jQuery);

The 'text' dataType means jQuery won't try to evaluate the JavaScript it fetches. The 'cache' parameter defaults to 'true', but I prefer adding it.

I tested several browsers with a proxy, and the JavaScript files are cached - cool! I presume this technique would work with any resource, not just JS.

bzr svn: 'unsupported protocol'

2009-03-08T23:56:00.001+02:00

Update (Aug 2009): this is a workaround for a bug that's been fixed for a while in the stable bzr versions.

This is something I've banged my head against for a nontrivial while when working with bzr and a subversion repository requiring authentication:

> bzr push http://svnrepos.example.net/trunk
bzr: ERROR: Invalid http response for http://.../.bzr/branch-format: Unable to handle http code 401: Authorization Required

Hard-coding 'http+svn' used to work (and give a warning), but it doesn't in the latest versions:

> bzr push http+svn://svnrepos.example.net/trunk
bzr: ERROR: Unsupported protocol for url "http+svn://..."

There's a ticket in progress about this issue, but there's a simple workaround: specify the username in the URL with http://username@host:

> bzr push http://username@svnrepos.example.net/trunk

Success!

Resharper 4.5 Nightlies Available

2008-12-30T17:01:00.000+02:00

JetBrains started releasing nightlies on Dec. 26, hooray :)

Using Python for Serious Development

2008-11-29T23:19:00.000+02:00

Some time ago an old friend asked a question on the Python-IL mailing list, wondering what we thought of using Python for serious development. I think my response back then was pretty interesting, so I'm reproducing it here (edited cosmetically only). Today I would've probably put it differently, but the essence would be the same.

From: Ori Peleg
Mon, 16 Apr 2007 23:17:55 -0700

Yes, you should program in Python! (or a similar agile
language) :-)

My main points:

 - Python is excellent for refactoring (beats C++ and, yes,
   Java too)
 - Don't move an inch with unit and system tests
 - These tests are much easier to write in Python
 - Compiler-checked interfaces aren't that important
 - After a few serious successes, Python is a very serious
   contender in my company.

Now to our application's story:
We ported a large application core to Python (original was in
C++ - we wrote it too).
Our motivation was maintainability - we were sure we could
reduce that hulking core to a smaller and simpler creature.
This really did happen, to the point that the project never
went into maintenance mode - it suddenly became useful for
things we didn't predict :)

We got performance benefits as well (yes, we were surprised).
The reasons were straightforward - we had many more options
when refactoring the code, and separated a lot of logic
neatly (for example extracted DB caching to a decorator -
"AOP" in a language that needs a new term :-)

The gains we ended up getting could have been achieved in
C++, obviously, but they only became apparent after
refactoring with Python.

We also wrote a quick sample client/server application with
Twisted, which was later modified in 2 weeks by someone
different and transformed into an excellent production
application, that's still working flawlessly after 6 months
with no maintenance. All this with about 300 lines of simple,
practically stupid, "I understand this code the first time I
see it" code (actually, this is probably the reason for the
quality).

Now, every time someone says "client/server" people think
about Python and Twisted, and even a large legacy network
client is being replaced with a small Twisted client soon.

How come Python is so amazing for refactoring?
First, you need really good tests. I don't miss the compiler
when I have really good tests.
The good side is that to develop at good quality and speed
you need really good tests in any language, so you're not
losing anything.  These tests are also so much easier to
write in Python - we get more and better tests, and spend
less time writing them. (Whatever language you use for the
system, you should probably test in a language like Python).

Once these tests are in place, refactoring in Python with a
good text editor and some basic tools is so much smoother
than C++ or Java, even with Eclipse's really cool 'Refactor'
menu.
Those actions that are trivial in Eclipse ( e.g. 'Rename
Method') are a little more complicated in Python, but not by
much (remember the good tests).
On the other hand, anything that isn't in Eclipse's context
menu is absolutely horrible to do in Java, to the point that
I've met many good Java developers that don't think anything
else exists at all :) In Python, I've found it to be a snap.
And when you add all the more and less dynamic tools that can
be put to good use, I'm one happy refactorer. Decorators,
bound methods, generator expressions, and the like give me
many ways to make the code simpler that aren't nearly as
accessible in Java (or barely possible in C++).

Why aren't interfaces that important? The reason I like
interfaces is as documentation that the compiler happens to
check. It's convenient, but I don't terribly miss it in the
presence of (a) good documentation, and (b), you guessed it,
good tests :-D
I don't have much experience with zope.interface, so I don't
know how easy it is to use or how much of the benefit from
interfaces you get.
I must say about this point that I'm comparing a full
static-typing approach (e.g. Java) to a full dynamic-typing
approach (e.g. Python) - give me dynamic typing any day. A
hybrid statically- and dynamically-typed language like Groovy
could work better, I don't have the experience to say.

orip.