Cheap Turpentine: Ori Peleg's Blog

Guava's I/O Sources and Sinks on Android

2014-07-13T16:49:00.000+03:00

I love Guava's I/O wrappers - ByteSource, ByteSink, CharSource, and CharSink. They encapsulate I/O stream (or reader/writer) access in a way that makes it easy for me to open & close the stream in the same location; when I pass a stream directly, the points of opening it and reliably closing it may be far apart. They also have helper methods to sweeten some common tasks, like reading an entire char source.

Android provides an existing mechanism for accessing private files while controlling their access modes: Context.openFileOutput and Context.openFileInput. As a Source/Sink primer, here's a cool way to adapt between the two:

The usage would look like this:

AndroidPrivateFileGuavaAdapter adapter =
  new AndroidPrivateFileGuavaAdapter("filename", Context.MODE_PRIVATE, context);

 // Read all text
String contents = adapter.asCharSource().read();

 // Write some bytes
byte[] bytes = ...; // obtains bytes from somewhere
adapter.asByteSink().write(bytes);

Instantly migrate to a secure password hashing scheme

2013-11-11T22:18:00.000+02:00

So your site uses terrible password hashing scheme H1 (perhaps MD5), and you want to migrate to secure password hashing scheme H2 (preferably scrypt, but possibly bcrypt or PBKDF2).

The standard migration approach is to upgrade users as they log in and enter their password. There are insecurities[1] and workarounds[2] doing this, but there's an alternative that instantly secures everybody using your service with no distruption:

Change your password hash to H2(H1(password)).

Can be done immediately for everyone, because you know H1(password).
Critical parameters such as salt, CPU-hardness (stretching), and possibly memory-hardness (scrypt FTW) are provided by H2.
Weaknesses in H1, such as collisions, don't affect the limited use case of password hashing.

I first heard of this idea from Avner Rosenan, and there's an interesting discussion about it on crypto.stackexchange.com.

Footnotes:

[1] e.g. old password hashes kept around

[2] e.g. delete old hashes and require a password reset

Android: backup and restore an app's data during development

2013-10-21T20:59:00.000+03:00

During development, having an app with a realistic configuration and data set is really useful. Here's a trick that can help.

Connect a device with the app configured with lots of good data, turn USB debugging on, and then:
adb backup com.example.mypackage

Later, on another device (or the same device after a clean install):
adb restore backup.ab

W00t.

Versioning Android apps with git tags, using the new Gradle-based build system

2013-10-07T21:09:00.001+03:00

At Onavo we version our Android and iOS apps with git tags (inspired by Fraser Hess). No more mucking about with manifests and plists in short-lived branches.

Our release git tags look like abc1.2.3-foo_15_bar, which translates to a version name of 1.2.3-foo and a version code of 15.

The 'abc' prefix differentiates between different apps on the same repo, and is ignored.
The 'bar' suffix differentiates between identical version names and version codes, and is also ignored. We sometimes need this when a a bad tag was made and pushed and we have to retag with the proper revision, since replacing a pushed tag in git isn't wise.

Our apk filenames look like appname-flavor-release-1.2.3-rc2-68fb97b.apk

Android's Gradle-based build system makes this super sweet. Here's how we do it:

To see something different, +Jake Wharton describes an awesome approach to versioning & git hacks with Gradle.
Soon I'll hopefully write about how we use Gradle with different flavors, and some some apk signing hacks.

EDIT: I forgot to explain the mysterious tagToBuildFrom - it's for the cases when we release a version of 2 different apps from the same git revision. There are 2 git tags, and "git describe --exact HEAD" will arbitrarily choose one. We can force gradle to use a specific tag by running:

./gradlew --daemon assembleRelease -PtagToBuildFrom=myapp4.5.6_31

Python: change and restore the working directory

2012-07-22T22:43:00.000+03:00

Occasionally a command should be run from a specific directory. Here's a Python helper for this:

try/finally ensures the original working directory is restored.
contextlib.contextmanager is a neat way to write things supporting the "with" statement ("Context Managers").

import os, contextlib @contextlib.contextmanager def temp_chdir(path): """ Usage: >>> with temp_chdir(gitrepo_path): ... subprocess.call('git status') """ starting_directory = os.getcwd() try: os.chdir(path) yield finally: os.chdir(starting_directory)

Hg vs. Git in the workplace

2010-09-01T01:51:00.000+03:00

This post is based on the following mail I sent to the ALT.NET Israel list. I'm discussing a Windows development environment.

I'm much better at git than hg, and I use git a lot for real work, and it's great (no ifs or buts).
I also helped originate the move to git at work, because the transition plan was much better. I evaluated hg first and loved it, but the transition plan wouldn't have been half has smooth (a better hg-svn bridge may have been written since then).

Still, here are - in order - the reasons I thing hg would fit better at my workplace (and had we found a good transition plan we'd probably be using hg instead of git):

Friendlier commands and interface. With git I feel like I need to research how to do straightforward things. Git is extremely powerful and I can do anything with it, and I'm strange so I like the arcane, but git requires more of an investment from people to use it effectively when they have other things to do. Some things are arguably easier to understand in git (rebasing, for example, vs. hg's patch queues - although that's debatable) but the whole package leans heavily to hg IMO. This is the least-loved aspect of git among the teams.
In hg, commits remember the named branch they were committed to (see here). I would LOVE to have this in git, but no cigar - we're still playing with different schemes of approximating this.
With many people working at times on several branches, and with merging done between branches, then given commit XYZ it's nontrivial in git to figure out which branch it belongs to.
This is especially true in some scenarios we encountered - e.g. human error caused an older release branch to be fast-forwarded to the head of the latest devel branch - few traces are left in the repository itself of the previous state, and we had fun a few times fixing this (can take hours). Now we have a hook to prevent this.
If we'd have cherry-picked between different branches instead of merged then this would be easy (and this is equivalent to svn's merging, only better), but we like git's merge and we want to use that.
hg has less quirky Windows support. Points to consider:
- msysgit gets sporadic releases and is usually months behind the stable git version, the latest hg versions just work.
  - This hit us a few times (bug has been fixed but not available in msysgit).
  - Current example: Git 1.7.2 has improved crlf handling but msysgit isn't there.
- Being able to "hg serve" your repository is incredibly cool, but "git daemon" doesn't work with msysgit
  - Think continuous integration that watches your local repository and runs builds with your current code, giving up-to-date feedback
  - collaborating with someone without pushing a publicly-visible branch (I've actually wanted to do that a few times)
  - developing and testing on more than 1 machine simultaneously. There are workarounds, but "git daemon" would have been perfect.
- automating hg, including hooks, run in a Windows environment (vs. git where it runs in mingw's MSYS environment)

What doesn't bother me much for the long run is tooling. Right now hg seems to have better tools for our scenarios (even including much better Trac support), but I think git's extreme popularity will cause better and better tools to be written. Some people were very sad to let go of their VisualSvn / Ankh plugins, but they manage with Git Extensions (mostly) and occasionally TortoiseGit. I was surprised at how few people use git from the command-line, I can't live without it.

jQuery Validate: Required If Visible

2010-06-28T19:34:00.000+03:00

I needed a generic "required if visible" rule for use with the jQuery Validation plugin. This is the result:

Notes:

Using required:"#myinput:visible" could have worked, but ":visible" doesn't check for hidden parents, and I didn't want to repeat the element selector.
":hidden" is false for elements with the "visibility:hidden" CSS rule, so I couldn't use ":not(:hidden)" either.
Technique for checking element visibility from Remy Sharp's blog.

MVC's RenderPartial in a WebForms page

2010-04-25T14:58:00.000+03:00

We have a mixed ASP.NET MVC/WebForms application, slowly transitioning to all-MVC. Obviously, sharing components between MVC and WebForms is important to us.

Using WebForms controls in MVC views is easy enough (minus the postbacks), but when we rendered partial views in a WebForms page using Keith Henry's code we occasionally got evil "Validation of viewstate MAC failed" errors. Microsoft doesn't think hybrid WebForms / MVC apps are relevant, but then Mauricio Scheffer came to the rescue. Instead of having the partial views subclass ViewUserControl, we now subclass Mauricio's ViewUserControlWithoutViewState - problem solved.

TeamCity 5.0 upgrade - "out of memory" error

2010-04-12T15:08:00.000+03:00

I upgraded JetBrains TeamCity to version 5.0 on Windows (for Git support, yay!). The previous version upgrades went smoothly, but this time I got an exception when loading the server:

java.sql.SQLException: out of memory

After some googling, I thought I had to:

Run ...\TeamCity\bin\tomcat6w.exe //ES//TeamCity
In the "Java" tab change the value for "Maximum memory pool"
Not set it too high

But no cigar. I realized that I probably had a corrupt HSQLDB database, and I ended up reinstalling and copying the config by hand (which was a little frustrating). I don't think there's a real reason why an embedded SQL database should be less dependable than one with a server (I know that from SQLite), so I'm guessing it's either TeamCity's or HSQLDB's fault. Even so, an "export configuration" button would be nice, and take the pain out of reinstalling TeamCity.

Encoding usernames and passwords in FTP URIs

2010-02-14T19:22:00.000+02:00

I want to encode the following into a single URI:

Username: 'jane+foo@example.com'
password: 'my password' (with a space)
ftp server: 'ftp.example.org'

The FTP URI that I expect:

ftp://jane%2bfoo%40example.com:my%20password@ftp.example.org

In .NET, HttpUtility.UrlEncode almost does what I need - it hex-encodes the special chars, but it also converts spaces to "+" which the FTP server interprets as a literal "+".

Final code that works with my tests using the FileZilla FTP Server:

string EncodeFtpUserInfoComponent(string s)
{
  // assume or assert s != null
  return HttpUtility.UrlEncode(s).Replace("+", "%20");
}

PowerResizer Keyboard Shortcuts

2010-02-02T20:04:00.001+02:00

For some reason the keyboard shortcuts / hotkeys for Federico Bastianello's PowerResizer aren't on the web, only in the "Readme.txt" installed with it. So, as a public service, here they are (for version 0.95):

2.1 - Window Docking Features and Keyboard Shortcuts

 - Move Window + A: dock a window to the left-half of the screen.
 - Move Window + S: dock a window to the right-half of the screen.
 - Move Window + W: dock a window to the top-half of the screen.
 - Move Window + Z: dock a window to the bottom-half of the screen.
 - Move Window + 1: dock a window to the top-left corner of the screen.
 - Move Window + 2: dock a window to the top-right corner of the screen.
 - Move Window + 3: dock a window to the bottom-left corner of the screen.
 - Move Window + 4: dock a window to the bottom-right corner of the screen.
 
When you resize the inner border of those windows you docked to one of the four
corners of the screen or to one of the four screen halves, PowerResizer 
changes the windows' behavior during resize in a way that all docked windows
will follow the new size of their neighboors.
It is possible to mix docked windows in halves of the screen and in corners.

2.2 - Window Moving/Resizing Features and Keyboard Shortcuts

 - Resize Window + CTRL: resize a window mantaining fixed its center point.
 - Resize Window + SHIFT: move a window using its border.
 - Move Window + 5: center a window into the screen (1/16 screen size margin).
 - Move Window + 6: center a window into the screen (1/10 screen size margin).
 - Move Window + 7: center a window into the screen (1/6 screen size margin).
 - Move Window + 8: center a window into the screen (1/4 screen size margin).
 - Move Window + 9: center a window into the screen (1/3 screen size margin).
 - Move Window + 0: maximize a window.
 
2.3 - Screen window reposition Features and Keyboard Shortcuts

 - CTRL + ALT + PAGE UP: Rotate docked windows counter-clockwise.
 - CTRL + ALT + PAGE DOWN: Rotate docked windows clockwise.
 - CTRL + ALT + HOME: Reposition windows.
 
Note: the keyboard shortcuts in that section, are system-wide shortcuts and 
they will work on windows docked on the monitor containing active window.

Console colors on Windows in .NET

2009-10-29T17:31:00.000+02:00

Changing the console output color is easy on *nix terminals with escape sequences, but has a funky API on Windows (see the color support code from Testoob for a Python example with pywin32/ctypes).

It turns out that there's a very nice .NET API for it, though (there's a nice article by Sam Allen on the Dot Net Perls site). This would be awesome for Testoob's IronPython support.

Include literal file contents in ASP.NET

2009-10-26T10:47:00.000+02:00

I was looking for something similar to Velocity's #include for ASP.NET. <% Response.WriteFile([filename]); %> works well, but a .NET control for this could be nice. Here is my attempt (MIT license):

using System.IO;
using System.Web;
using System.Web.UI;

namespace FooControls
{
  public class LiteralContentFromFile : Control
  {
    public string Path { get; set; }

    protected override void Render(HtmlTextWriter writer)
    {
      var server = HttpContext.Current.Server;
      var physicalPath = server.MapPath(Path);
      var fileContents = File.ReadAllText(physicalPath);
      writer.Write(fileContents);
    }
  }
}

Usage in an .aspx file (WebForms or ASP.NET MVC):

<%@ Register TagPrefix="foo"
  Namespace="FooControls" Assembly="FooControls" %>
...
<foo:LiteralContentFromFile runat="server"
  Path="~/path/to/file" />

Edit: subclassing System.Web.UI.Control instead of System.Web.UI.WebControls.WebControl which added unnecessary and unwanted behavior (for example, wrapped the output with <span>).

Issues API for Google Code Hosting

2009-10-22T01:43:00.000+02:00

Google have recently released an issue-management API for projects hosted on Google Code.

That's good news for Testoob - I've been wanting to import its old Trac tickets for ages!

Testoob 1.15 released

2009-10-08T02:30:00.001+02:00

I'm very happy to announce that after nearly 3 years Testoob sees another release.

Testoob is the advanced Python test runner and testing framework that spices up any existing unittest test suite.

Changes: Version 1.15 (Oct. 2009) adds better Python 2.6, IronPython, and Jython support, as well as test coverage improvements, better color support, and some new options and bugfixes. A full list of changes is available in the release announcement.

A big shout out to Ronnie van 't Westeinde - without him this version wouldn't have been possible.

Options to install:

With easy_install: "easy_install -U testoob"
One of the available packages

IDCC 2009 Registration Is Open

2009-09-09T15:54:00.000+03:00

The response for the call for content for IDCC 2009 has been impressive, and now two tracks are open with great talks.

Hurry up and register!

"Enter" submits my ASP.NET form!

2009-09-07T16:43:00.000+03:00

ASP.NET encourages web pages to be written as one big <form>, and every interaction with it to be a POST to the server ("Postback" in ASP.NET). This means pages often have multiple auto-generated <input type="submit"> elements.

On some browsers, pressing Enter causes the first submit button to be "clicked". In pages with many such buttons, that's usually not what you want. As a hack to make "Enter" do nothing on an already-built page, I added the following before any other submit button:

<div style="visibility:hidden;height:0;">
  <input type="submit" value="Dummy" onclick="return false;" />
</div>

Now this do-nothing, invisible submit button "catches" the Enter and prevents the next button from being "clicked". This requires JavaScript turned on, but ASP.NET's "Postbacks" require them anyway, so no big loss.

Implementation note: using 'display:none;' caused the browser to ignore it, so I had to resort to 'visibility:hidden;' and then 'height:0;' to impact the layout as little as I could. There could be cross-browser styling issues here, my specific case didn't need anything special.

Python: checking if an executable exists in the path

2009-08-24T15:54:00.001+03:00

I wanted to do this on Windows. Based on this StackOverflow question and on Trey Stout's and Carl Meyer's responses I made the following snippet:

def exists_in_path(cmd):
  # can't search the path if a directory is specified
  assert not os.path.dirname(cmd)

  extensions = os.environ.get("PATHEXT", "").split(os.pathsep)
  for directory in os.environ.get("PATH", "").split(os.pathsep):
    base = os.path.join(directory, cmd)
    options = [base] + [(base + ext) for ext in extensions]
    for filename in options:
      if os.path.exists(filename):
        return True
  return False

JavaScript Namespaces

2009-07-22T21:43:00.000+03:00

It's important to namespace your JavaScript code. I want one "master" namespace, and everything else under it (like YUI's namespacing approach). I fell into some potholes of browser compatibility on the way (with var foo; if (!foo) foo = {};), but here's what I ended up with:

// define master namespace
if (typeof foo === "undefined") var foo = {};

(function(){ // lexical scoping, allows "private" variables and functions

// define sub-namespace
if (!foo.bar) foo.bar = {};

// these don't leak outside
function helper() { /* ... */ } // "private" function
var counter = 0;                // "private" variable

// this function shows up globally under namespace foo.bar
foo.bar.func1 = function(x) { /* ... */ };

// not just functions
foo.bar.constant1 = 12;
foo.bar.shared_data = [];

})(); // end of lexical scoping

This way all the JS files can use the same namespace hierarchy, and can still be defined separately. No JS library necessary.

IDCC 2009, call for content

2009-07-15T11:31:00.000+03:00

The Israeli Developers Community Conference 2009 has started its call for content. It's a community conference centered on the .NET world.

If you have a session to propose, please do! The community will vote on the proposals, and the conference will conform to the participants - not the other way around.

Fresh instances in Python with lazy evaluation

2009-06-05T17:17:00.001+03:00

Testoob's --capture feature (written by Misha) replaces sys.stdout and sys.stderr for each test being run, and displays the output only if the test fails.

Leeor Aharon, of ff-activex-host fame, wanted to use it with Python's logging module, but the StreamHandler class stores a reference to the output stream on initialization - it already has a reference to sys.stdout, so replacing it won't affect it.

We originally thought of sublcassing StreamHandler and making retrieve the logger from a property, but we came up with this elegant code instead:

 
class LazyEvaluator(object):
    def __init__(self, factory):
        self.__factory = factory
    def __getattr__(self, name):
        return getattr(self.__factory(), name)

lazy_stdout = LazyEvaluator(lambda:sys.stdout)
handler = StreamHandler(strm=lazy_stdout)

The "lazy evaluator" is initialized with a factory callable, and every time it tries to access an attribute or method the object will be re-created by the factory. No changes necessary for StreamHandler, and ./alltests.py --capture works like a charm.

jQuery UI scoped themes and the Dialog widget

2009-05-27T10:47:00.001+03:00

There's a problem using scoped themes with jQuery UI's Dialog widget in the 1.6 and 1.7 releases (and possibly 1.7.1 as well).

Christoph Herold mentioned his workaround - wrapping the dialog's DOM with a scoping div (with the scoped theme's class name). I wrote a small jQuery plugin that does this and works fine so far:

// Wraps the dialog in a div with the scoping class on
// dialog open, removes the scoping class on dialog close
$.fn.dialogScopingWorkaround = function(scopingClass) {
  this.bind('dialogopen', function(event, ui) {
    $(this).parents(".ui-dialog").wrap(
      '<div class="dialogScopingWrapper '
      + scopingClass
      + '"></div>'
    );
  });
  this.bind('dialogclose', function(event, ui) {
    var wrapper = $(this).parents(".dialogScopingWrapper");
    wrapper.replaceWith(wrapper.children());
  });
  return this;
};

Example with the "smoothness" theme scoped with class "theme-smoothness":

$('#dialog').dialogScopingWorkaround("theme-smoothness").dialog();

// Must call the workaround before opening the dialog
// so the 'dialogopen' event is bound.

IE7 compatibility image wants to install IE8

2009-05-17T17:05:00.000+03:00

I use Microsoft's IE Application Compatibility virtual machine images to test web sites with IE6 and IE7. Well, today my IE7 virtual machine wanted to automatically update to IE8. Great for the web, funny for an IE7 compatibility test image :)

Ajax.NET Professional "onTimeout" exception

2009-05-14T22:38:00.001+03:00

We just upgraded to Ajax.NET Professional 9.2.17.1 and started seeing these JavaScript exceptions in file "core.ashx":

this.onTimeout is not a function

It turns out this is a known issue with synchronous calls done with AjaxPro's client-side JS. We should probably have ditched it and used regular jQuery AJAX calls on the client side (like here and here) but we haven't gotten around to that yet.

Our workaround was easy - we were making a "fire and forget" call, so it didn't need to be synchronous (we added an empty callback handler to the arg list). We could have also patched the JS ourselves and:

rebuilt AjaxPro with the patched version
made AjaxPro load the patched version
not used RegisterTypeForAjax and added the script references ourselves - pointing to the patched version, and possibly taking the opportunity to combine and minify AjaxPro's JS files.

Reddit and Digg icons on Blogger without external scripts

2009-03-17T02:31:00.000+02:00

It took me a while to be able to add reddit and Digg icons that didn't load external scripts - Blogger's templating language had me stumped. I used reddit's, Digg's, and Blogger's documentation, but I would never have gotten the "onclick" escaping right without this post.

<!-- reddit button -->
<a expr:href='"http://www.reddit.com/submit?url=" + data:post.url' expr:onclick='"window.location = \"http://www.reddit.com/submit?url=\" + encodeURIComponent(\"" + data:post.url + "\"); return false"'> <img src="http://www.reddit.com/static/spreddit1.gif" title="submit to reddit" alt="submit to reddit" border="0" /> </a>
<!-- digg button -->
<a expr:href='"http://digg.com/submit?url=" + data:post.url' expr:onclick='"window.location = \"http://digg.com/submit?url=\" + encodeURIComponent(\"" + data:post.url + "\"); return false"'> <img style="background:white;" width="16" height="16" alt="Submit Story to Digg" title="Submit Story to Digg" src="http://digg.com/img/digg-guy-icon.gif"/></a>

P.S. - if you can't find the "data:post.body" element to add the buttons after, like I couldn't, see if you forgot to check the "Expand Widget Templates" checkbox under "Layout -> Edit HTML".