Nir Levy - Eureka

Moved to Medium

noreply@blogger.com (Nir Levy) — Tue, 4 Jul 2017 10:08:00 +0300

I'm putting all new stuff on Medium at https://medium.com/@nirlevy
See you there...

Native Fullstack Developer for DayZz

noreply@blogger.com (Nir Levy) — Sun, 18 Jun 2017 12:03:00 +0300

מפתח native fullstack

ל- dayzz דרוש/ה מפתח/ת מובייל וסרבר לפיתוח מוצר חדשני בתחום ה- health-tech. הזדמנות להתחיל מסע מאתגר משלביו הראשונים ולייצר פתרונות אשר ישפרו את חייהם של מיליוני אנשים.

דרישות טכניות

· לפחות 3 שנים בפיתוח בעולמות הווב/מובייל כולל פיתוח javascript ברמה גבוהה

· לפחות שנה בפיתוח אינטנסיבי ב-react, react-native או native android/ios

· ניסיון בפיתוח צד שרת מבוסס node.js או django/python ובמיוחד פיתוח API

· ניסיון בפיתוח מעל בסיס נתונים nosql כגון mongodb

· ניסיון בעבודה עם AWS או Google Cloud

את מי אנחנו מחפשים?

· אדם עם ראש פתוח שיכול לחשוב יצירתית ולא לפחד להביע את דעתו

· מישהו (או מישהי) שיודעים לעבוד עם צוותי מוצר באווירה דינאמית

· חשיבה מוצרית מוכוונת לקוח, והבנה של יסודות חווית משתמש

מה תעשו אצלנו?

· פיתוח react native בצד הקליינט, ויישום של חווית משתמש מעולה

· פיתוח ממשקי api ב- node.js בצד השרת, בעבודה מול mongodb

· התממשקות לרכיבים צד ג׳ כגון firebase ושירותי ענן נוספים

· פיתוח ממשקים לרכיבי חומרה חיצוניים (wearables ו-accessories)

קורות חיים ופרטים נוספים: jobs@dayzz.com

Timing django/python function calls

noreply@blogger.com (Nir Levy) — Wed, 20 Jan 2016 10:45:00 +0200

Doing some optimization for SWITCH.co.il I've come across a handy snippet to time the execution of python/django functions. Since I was doing deep-dive checking I also wanted to know the stack of the executed function. Figured someone (me) would want to use it in the future, so I'm adding it here. Not the prettiest of code, but it works.


import time
import inspect                                                
def timeme(method):
    def time_wrapper(*args, **kw):
        startTime = int(round(time.time() * 1000))
        result = method(*args, **kw)
        endTime = int(round(time.time() * 1000))
        stack=[]  
        for i in inspect.stack():
            if i[3] == 'time_wrapper': continue 
            stack.append(i[3])
        print 'Time(ms): ', ('%04d')%(endTime - startTime), method.__name__, u'->' ,u'->'.join(reversed(stack[0:5]))
        #only show 5 top stack
        return result
    return time_wrapper

#usage example
@timeme
def some_function(x,y):
    do_something
    return something

RTL Bootstrap CSS Convertor

noreply@blogger.com (Nir Levy) — Wed, 31 Jul 2013 16:01:00 +0300

Bootstrap.css is great, Bootstrap.css 3 is better. Alas, Neither have an RTL version. There are a couple of not-maintained gits floating around the web for bootstrap 2 but non for bootstrap 3

To convert bootstrap 3 to RTL you can use https://github.com/ded/R2

If you want a rather stable, working, version of RTL bootstrap.css you can use the one from my repository at https://bitbucket.org/levynir/bootstrap3-rtl. Even if you decide to brew your own version, I recommend that you read the README on this bitbucket as there are some caveats to RTLing bootstrap.

Setting I18N Hebrew in Django 1.5

noreply@blogger.com (Nir Levy) — Tue, 30 Jul 2013 14:21:00 +0300

It should be simple. But the docs are not very clear. So here is how to setup your Django site to be in Hebrew (or any other language).

The need: Make my site work in Hebrew, without taking into account Browser's Accept Language and all this stuff.

Ingredients: Working Django 1.5 project on Linux, common sense

Steps:

Install GNU gettext (even if you think you have it already)


sudo apt-get install gettext

Change in projectname/settings.py


# Language code for this installation. All choices can be found here:
# http://www.i18nguy.com/unicode/language-identifiers.html
# LANGUAGE_CODE = 'en-us'
# This will force this installation to be in Hebrew
LANGUAGE_CODE = 'he'

#...

# If you set this to False, Django will make some optimizations so as not
# to load the internationalization machinery.
USE_I18N = True

# If you set this to False, Django will not format dates, numbers and
# calendars according to the current locale.
USE_L10N = True

# If you set this to False, Django will not use timezone-aware datetimes.
USE_TZ = True

Make sure your MIDDLEWARE_CLASSES do not have LocaleMiddleware. If you do have LocaleMiddleware than Django will try to guess the right locale from the request context, which we don't want at the moment.


MIDDLEWARE_CLASSES = (
    'django.middleware.common.CommonMiddleware',
    #Comment out LocaleMiddleware if it's there to keep in the same locale all the time
    #'django.middleware.locale.LocaleMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    # Uncomment the next line for simple clickjacking protection:
    # 'django.middleware.clickjacking.XFrameOptionsMiddleware',
)

You may also need to add django.core.context_processors.i18n to your TEMPLATE_CONTEXT_PROCESSORS. (Don't do that right now, wait and see if everything is working and only add this if stuff is not working)


#add this at the end of settings.py if I18N is not working
from django.conf import global_settings
TEMPLATE_CONTEXT_PROCESSORS = global_settings.TEMPLATE_CONTEXT_PROCESSORS + (
    ".context_processors.site",
    "django.core.context_processors.i18n",
)

In your python code wrap translatable texts with ugettext_lazy or ugettext (you can and should alias them to some conventional shortcut (I use _t for ugettext and _ for ugettext_lazy, for example


from django.utils.translation import ugettext_lazy  as _
from django.utils.translation import ugettext  as _t
class Shirt(models.Model):
    COLOR_TYPES = (
                    ('B', _('Blue')),
                    ('G', _('Green')),
                    )
    
    ...
    color = models.CharField(max_length=1, choices=COLOR_TYPES)
    type(self):
        return _t('Shirt')

In your templates you need to use this (ugly and hard to read) method


{% load i18n %}
<!-- Make sure this LANGUAGE_CODE is correct: {{LANGUAGE_CODE}} -->
...
<h1>{%trans "Shirt Store"%}</h1>
...
or you can use {{_("Shirt Store")}} as an alternative method

Remember that you need to {% load i18n %} in every template that uses {%trans%}.

Now we need to create the tranlsation files themselves:


cd django-project
cd django-app
mkdir locale
django-admin.py makemessages -l he
vi locale/he/LC_MESSAGES/django.po
django-admin.py compilemessages

Rinse, repeat

Run your site and hope for the best.

Sources:

"Type 'Newtonsoft.Json.Linq.JToken' is a recursive collection data contract" While Adding Service Reference in VS2012

noreply@blogger.com (Nir Levy) — Thu, 23 Aug 2012 11:01:00 +0300

Trying to add a service reference in Visual Studio 2012 to Bing Maps SOAP service I got the following error message:

Cannot import wsdl:portType
Detail: An exception was thrown while running a WSDL import extension: System.ServiceModel.Description.DataContractSerializerMessageContractImporter
Error: Type 'Newtonsoft.Json.Linq.JToken' is a recursive collection data contract
...

Following Getting “Recursive collection data contract” when referencing a WCF service with a slightly complex method and Visual Studio 2012 Error Reference.svcmap throwing Error I've had to change the "reuse types in referenced assemblies" from "all" to everything except Newtonsoft.JSON and RestSharp.

Might not be something to write home about, but I'm sure I'll forget where it is was in a year or so.

PS. This seems to be VS2012 only issue, VS2010 worked just fine.

Tech-Ed Eilat 2010

noreply@blogger.com (Nir Levy) — Thu, 18 Nov 2010 09:35:00 +0200

So happy to be presenting at Tech-Ed Eilat 2010.
See you all at the Herods Hotel, Tue 30/11 at 11:15 where we would share our lessons from the field on enterprise web content management with SharePoint 2010.

You can read more about our session at the realcommerce web site.

IIS JNLP 404 Problem

noreply@blogger.com (Nir Levy) — Tue, 26 Jan 2010 12:18:00 +0200

If your IIS 6 returns a 404 Not Found when you request a Java Web Start jnlp file, then you probably need to add the jnlp mime-type to the IIS configuration.

IIS Settings -> Right click the machine -> mime-types -> new Extension: .jnlp MIME Type: application/x-java-jnlp-file

How to do rewrites in an IIS ISAPI

noreply@blogger.com (Nir Levy) — Sun, 4 Oct 2009 12:09:00 +0200

Note to self: If ever you need to do a rewrite in an IIS ISAPI Filter you just need to re-set the pHeaderInfo's url header. Something like this:

 
pHeaderInfo->GetHeader(pCtxt->m_pFC,
              "url",strUrl.GetBuffer(dwUrlSize+1),&dwUrlSize);
strUrl.ReleaseBuffer(); 
strNewUrl = GET_REWRITE_FOR_URL(strUrl); 
if ( strNewUrl.IsEmpty() == FALSE ) {  
    pHeaderInfo->SetHeader(pCtxt->m_pFC, 
              "url", (LPTSTR)(LPCTSTR)strNewUrl);  
} 
return SF_STATUS_REQ_NEXT_NOTIFICATION;

This is working in IIS4 and above.

Note that if you really need a good re-write/redirect filter for IIS you should look at IIRF ,but since it's not working for IIS4 i had to do this manually

Bing, Google, Nutch and Canonical URLs

noreply@blogger.com (Nir Levy) — Thu, 18 Jun 2009 09:32:00 +0300

Google announced a few months back that they started supporting canonical urls. This is a great feature that we already adopted in a couple of sites (mostly to keep tracking codes from messing with our search engine results.

Unfortunelty, altough Microsoft announced they will support this feature in Live, this was not yet implemented in Bing as of now (Jun 2009).

Nutch, on the other hand, had this on thier TODO list for 1.1.

Also, Google Search Appliance (GSA), is currently assumed by all to not support this (although nobody really knows if it does).

More about canonical URLs:
http://googlewebmastercentral.blogspot.com/2009/02/specify-your-canonical.html
http://searchengineland.com/canonical-tag-16537
http://janeandrobot.com/library/url-referrer-tracking

many group by strategies to select single rows per group

noreply@blogger.com (Nir Levy) — Mon, 15 Jun 2009 09:53:00 +0300

Xaprb gives a good explanation for selecting a single row from a group: How to select the first/least/max row per group in SQL at Xaprb.

apc futex_wait lockdown make your apache freeze over

noreply@blogger.com (Nir Levy) — Wed, 3 Jun 2009 12:58:00 +0300

We had the typical LAMP setup going on, with Drupal as the base CMS and APC for bytecode cache. We needed a good caching engine so I figured why not use APC's user cache. Well, we tried the APC Cache Drupal Module which, with minor fixes proved to work very nicely. That is, until we actually put this all thing on production.

The first thing we had was having our apache hang and not respond to any user requests. We susspected network issues, especially since netstat -na showed that all the apache processes were hanging on SYN_WAIT. However, since apache restart solved the issue i started to suspect this was something else.

To make a long (very long) story short, I got strace on our prod machines to find out that apache was either hanging on futex_lock(....FUTEXT_WAIT...) or doing infinte loops on the same functions.

To make even a longer story short, I got gdb installed on those machines and the backtrace clearly indicated that the locks were from APC user-cache calls.

We decided to abandon APC user-cache and switch to memcached which proved faster and had less lockdowns.

The funny thing is that when we talked about this over dinner the same evening a developer from another team just pointed me to this article by one of the APC leaders: (or something) How to Dismantle an APC Bomb which has been around for over a year. I am supprised and shocked that such a information is hidded so well and not mentioned anywere in the docs. Moreover, I went through the APC code again after reading this post (I went through it once when i started analysing the problem) and it seems that this is not even close to being resolved. there are no patches and no TODOs and nothing of the sort. From reading the code the entire user-cache needs a major re-write. What gives?

(this is a post i wrote a couple of months ago, never had time to finish it. ~~Unfortunately this is still not fixed afaik~~)
EDIT (07/2009): http://pecl.php.net/bugs/bug.php?id=15179 reports this to be fixed. If anyone can confirm this please send me a note so that I could update this post

GA_googleAddAttr is not defined error

noreply@blogger.com (Nir Levy) — Thu, 14 May 2009 12:02:00 +0300

If you are using Google Ad Manager you can define custom attributes for targeting.

If you are getting a javascript error GA_googleAddAttr is not defined just make sure to put the call(s) to GA_googleAddAttr in a seperate <script> line, after the GS_googleEnableAllServices.


<script type="text/javascript">
GS_googleAddAdSenseService("ca-pub-0000000000"); 
GS_googleEnableAllServices();
</script>
<script type="text/javascript">
GA_googleAddAttr("ATTRB1", "whatever");
</script>
<script type="text/javascript">
GA_googleAddSlot("ca-pub-0000000000", "slotname_216x311");
</script>

V6 DECODE64 not working properly

noreply@blogger.com (Nir Levy) — Mon, 23 Mar 2009 09:05:00 +0200

It turns out that after all these years, V6's built-in DECODE64 and ENCODE64 actually do not work properly with some binary data.


proc COMPARE_BASE64 {} {
 set code  {2lIU1ZNw5BYvKi79j4L/+GGmjAQK7uiBQG7elDdKZcE=}
 set vgn_result [DECODE64 $code];         #VGN built-in function
 set tcl_result [::base64::decode $code]; #TclLib tcl-only implementation 
 return [join [list \\
   [BIN2HEX $vgn_result] \\
   [BIN2HEX $tcl_result] \\
   [string compare $vgn_result $tcl_result ] \\
  ] "\\r\\n"]
}
proc BIN2HEX { text }  { binary scan $text H* result; return $result }
COMPARE_BASE64

--- result ----
da52145370e4162f2a2efd8f82fff861a68c040aeee881406e94374a65c1
da5214d59370e4162f2a2efd8f82fff861a68c040aeee881406ede94374a65c1
1

If you need to encode/decode base64 for use in non-vignette systems make sure you use tcllib's base64.

This was tested on Vignette's V6 but I am sure it's true for Storyserver 4.2 and V5 as well.

MySQL indexes behaving badly

noreply@blogger.com (Nir Levy) — Thu, 19 Mar 2009 22:15:00 +0200

If you have a MySQL query that for no apparent reason stops working or no longer behaves as it used to, the first thing to do is EXPLAIN it. If the explain is unreasonable (for example, using different indexes then it used to or using an index but still scanning millions or rows) then you should try to CHECK TABLE.

One thing that I accidently found out was that CHECK TABLE also updates the key statistics. This made many problems magically disappear.

A few things that should be noted:

CHECK TABLE will only update key statistics for MyISAM tables

InnoDB

OPTIMIZE TABLE

If you use OPTIMIZE TABLE to update the key statistics you should be aware that OPTIMIZE TABLE does other (good) things as well, so it usually takes much longer and it locks your table while doing it. Also, for InnoDB it actually re-builds the table using ALTER TABLE.
The CHECK TABLE have several options. Only the MEDIUM (the default) or EXTENDED update the key statistics. (i think).

So, if your indexes stop working, or just mis-behave, try to check them.

Vista fast user switching disappeared

noreply@blogger.com (Nir Levy) — Fri, 20 Feb 2009 21:33:00 +0200

My fast user switching option in Vista Home machine suddenly disappeared. I remember it was there, then it was not. I don't know why this happened, but here is how to fix it.

A rough translation:

Run regexit.exe
Go to HKEY_LOCAL_MACHINE > SOFTWARE > Microsoft > Windows > CurrentVersion > Policies > System
Look for a key named HideFastUserSwitching. If it exists, change it's value to 0 (zero). If it does not exist create it as a DWORD and set it to zero.
You might need to log off before the change takes effect.

utf8 and hebrew in tomcat

noreply@blogger.com (Nir Levy) — Sun, 8 Feb 2009 14:03:00 +0200

In tomcat 5.0 and above, if your UTF-8 request parameters are received as gibberish you might need to do the following:

In your server.xml add the URIEncoding="UTF-8" and useBodyEncodingForURI="true" to the Connector tag(s):


    <Connector 
     useBodyEncodingForURI="true" 
     URIEncoding="UTF-8" 
     acceptCount="100" 
     enableLookups="false" 
     maxSpareThreads="75" 
     maxThreads="150" 
     minSpareThreads="25" 
     port="8080" 
     redirectPort="8443" />

This should make GET requests work properly.

For some reason the above does not work for POST requests. If you ask the tomcat people they'll mumble something about W3C, RFC, and RTFM. The short way to have this work for POST requests is to write a small filter to set the request encoding properly. We are using something similar to this:


package com.realcommerce.filters;
import javax.servlet.*;
import java.io.*;
public class RequestEncodingFilter implements Filter  {
    public void init(FilterConfig filterConfig) throws ServletException {
         //Do Nothing
    }
    public void destroy() {
        //Do Nothing
    }
    public void doFilter(ServletRequest request,ServletResponse response,
        FilterChain chain) throws IOException, ServletException 
    {
        request.setCharacterEncoding("UTF-8");
        chain.doFilter(request, response);
    }
}

This made POST requests pass Hebrew (or any UTF-8) parameters properly.

slow lstat, slow php, slow drupal

noreply@blogger.com (Nir Levy) — Tue, 27 Jan 2009 23:22:00 +0200

If you are short on time go to bottom line.

Ok, the lstat() and stat() system calls system call are not really slow per se. But take a look at an apache strace -r dump from a drupal installation I was testing:


[user@host ~]# strace -o strace.load -r -s 256 -p <PID>
...
0.000081 getcwd("/mnt/var/www/html/drupal", 4096) = 32
0.000048 lstat("/mnt", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
0.000805 lstat("/mnt/var", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
0.001596 lstat("/mnt/var/www", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
0.000104 lstat("/mnt/var/www/html", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
0.000362 lstat("/mnt/var/www/html/drupal", {st_mode=S_IFDIR|0755, st_size=8192, ...}) = 0
0.000804 lstat("/mnt/var/www/html/drupal/sites", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
0.057575 lstat("/mnt/var/www/html/drupal/sites/all", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
0.000297 lstat("/mnt/var/www/html/drupal/sites/all/modules", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
0.014762 lstat("/mnt/var/www/html/drupal/sites/all/modules/Internet", {st_mode=S_IFDIR|0755, st_size=4096,
0.000214 lstat("/mnt/var/www/html/drupal/sites/all/modules/Internet/date", {st_mode=S_IFDIR|0755, st_size=4
0.000112 lstat("/mnt/var/www/html/drupal/sites/all/modules/Internet/date/date.module", {st_mode=S_IFREG|064
0.010816 open("/mnt/var/www/html/drupal/sites/all/modules/Internet/date/date.module", O_RDONLY) = 20
0.000835 fstat(20, {st_mode=S_IFREG|0644, st_size=44118, ...}) = 0
0.000069 lseek(20, 0, SEEK_CUR)    = 0
0.000063 stat("././sites/all/modules/Internet/date/date.module", {st_mode=S_IFREG|0644, st_size=44118, ...}) = 0
0.000159 close(20)                 = 0
...

As you can see, for every file included in the system there are around 10-15 lstat calls. this means that if we have ~200 files included we have ~2500 lstat calls per page. A simple sum on the strace results gave me ~8 seconds per page of lstat time, which was over a third of my test page time. see?


[user@host ~]# cat strace.single | awk '{printf " " $1 "\n " $2}' |sed 1d | sed \$d |awk '{printf $2 "+"}'  | sed s/.$/\\n/ | bc
21.204960
[user@host ~]# cat strace.single | awk '{printf " " $1 "\n " $2}' |sed 1d | sed \$d | fgrep lstat | awk '{printf $2 "+"}'  | sed s/.$/\\n/ | bc
8.361994

if you really want to understand this shell script let me know. It deserves a post by itself

Now we have 3 options: a) make lstat() faster. b) make php stop doing all those lstats(). c) make drupal include less files.

Well, (a) is not an option as our infrastructure team (aka sysadmins) insist there is nothing wrong with the filesystem and nothing can be done to enhance it's performance. Since I gave up being a sysadmin when it was still call 'sysadmin' it would be hard for me to prove them wrong (especially since they might be are right).

As (c) would require us to refactor most of our code in a very very 'not drupal way' and create very large php files, I was hoping to find a better solution.

By a rare chance of good fortune i came to learn about realpath_cache_size. Once I changed the foolish default to a reasonable 1M in php.ini, I lost most of the lstat() calls and over 10 seconds of processing time.


[user@host ~]# cat strace.2.single | awk '{printf " " $1 "\n " $2}' |sed 1d | sed \$d |awk '{printf $2 "+"}'  | sed s/.$/\\n/ | bc
9.752309
[user@host ~]# cat strace.2.single | awk '{printf " " $1 "\n " $2}' |sed 1d | sed \$d | fgrep lstat | awk '{printf $2 "+"}'  | sed s/.$/\\n/ | bc
.040935

Here is what to do in php.ini


; ...php.ini...
; Determines the size of the realpath cache to be used by PHP. This value should
; be increased on systems where PHP opens many files to reflect the quantity of
; the file operations performed.
 realpath_cache_size=1M

; Duration of time, in seconds for which to cache realpath information for a given
; file or directory. For systems with rarely changing files, consider increasing this
; value.
realpath_cache_ttl=300

For normal pages under normal load I was able to get a 10%-25% performance improvment. I think that's quite a bit for a configuration change.

IE6 freeze with MOSS 2007

noreply@blogger.com (Nir Levy) — Thu, 1 Jan 2009 09:06:00 +0200

emposha wrote about how to handle a situtation where IE6 chokes on MOSS preview pages.

Hebrew SEO: גוגל, עברית, וה"א הידיעה

noreply@blogger.com (Nir Levy) — Thu, 11 Dec 2008 15:02:00 +0200

נראה ש-google לא מתייחס נכון לה"א הידיעה.
אם נחפש לדוגמא הממשלה נקבל בתוצאות את "פורטל השרותים של ממשלת ישראל" אבל גוגל מסמן בבולד רק את המילה הממשלה ולא את המילים ממשלה או ממשלת שהן הטיות חוקיות לחלוטין.
בכיוון ההפוך, אם נחפש ממשלה נקבל תוצאות אחרות לגמרי, כאשר המילים ממשלת וממשלה מודגשות אבל המילה הממשלה אינה מוגדשת.
לפי ההבדלים בתוצאות החיפוש נראה שהבעיה קיימת לא רק בהדגשה אלא גם באינדוקס.
בדקתי גם בחיפוש לפי הילדים מול ילדים ומילים נוספות. הדבר נראה נכון גם לגבי live של מיקרוסופט (בלי לדבר על yahooשבכלל לא טרחו לתמוך במורפולוגיה עברית)
המסקנה,לצרכי אופטימיזציה במנועי החיפוש של גוגל כדאי להגדיר את מילות המפתח של האתר עם ובלי ה"א הידיעה.

drupal memcache items disappearing from the cache

noreply@blogger.com (Nir Levy) — Thu, 20 Nov 2008 23:05:00 +0200

The scenario: you are using memcached as your Drupal cache system. You do a cache_set for some item. You do a cache_get for this item. The item is there. You do a cache_get again after a few seconds/minutes, the item is not there.

You might be experiencing this issue of the memcache module: In short it means that the memcache module flushes the entire cache cluster whenever someone does a wildcard cache_clear_all. Since it's very likely that all your bins are in a single cluster you get to a situation that whenever someone-on-the-other-side-of-the-world does something that is probably completely unrelated to you, your item will still get flushed from the cache.

Note that since flush does not activly removes the items from the cache but rather marks them as expired you will not see any memcached stats that will hint to this issue.

I have commented on the issue. Here are some highlights:

...we are currently using gaolei's patch which is fast albeit expensive. We are using it on a large production system that depends heavily on memcache and we did not see any problems yet. (thank you gaolei).

We have tried to implement a lock-add-unlock scenario such as was suggested in several comments but this will be a definite nightmare for high-traffic/high-update sites...

I would suggest letting the user choose which flush mechanism to use: the current one, or the salt one. I see no reason why the module's developers should decide for me. On a site with little updates and little memory i would prefer to have the whole memcache flush; On a site with many updates and tons of memory i'm willing to sacrifice space for the sake good performance. a simple memcache_flush_method variable would do the job just fine.

MYSQL Query Optimization: Avoiding ORs

noreply@blogger.com (Nir Levy) — Mon, 10 Nov 2008 20:43:00 +0200

So, you've probably heard that using ORs in your queries is heavy and should be avoided when possible. Here is a live example. We had the following query running on a medium sized table in MYSQL (around 200K rows):


SELECT count(*)  FROM interactions p
   WHERE (
       (p.employer = 69 AND p.flag1 = 1)
       OR
       (p.employee = 69 AND p.flag2 = 0)
       )

Let's think of interactions as a table holding the interaction between employers and employees, and this query should tell the user the total number of interactions where user number 69 is involved.

The query looked pretty innoccent on an idle MYSQL server taking around 200ms, but when the server got busy the query execution time was reaching 5-6 seconds.

Altough we did have indexes on interactions.employer and interactions. employee doing EXPLAIN showed that MYSQL was not using them.

After some digging and fiddling and trying we ended up with:


SELECT count(*) as count FROM (
   SELECT 1  FROM interactions p
       WHERE (p.employer = 69 AND p.flag1 = 1)
   UNION ALL
   SELECT 1  FROM privatemsg p
       WHERE (p.employee = 69 AND p.flag2 = 0)
) t2

This query took on the busy server less then 250 milliseconds and less then 80ms on the idle server. Doing EXPLAIN showed that MYSQL was doing two queries, and using the correct index on each one.

After some more digging we noticed that if we EXPLAIN the original query on the idle server the optimizer occasionally converts it to a UNION, but for some reason this did not always happen and in any case took about twice the time then the UNION query.

To sum up the results here is a simple table

Query type	Server activity	Query Time
Using OR	Idle	200ms
Using OR	Busy	5500ms
Using UNION	Idle	78ms
Using UNION	Busy	189ms

Conclusion: always consider alternatives to OR, but make sure you check them well against real-time examples.

A Further Note: When timing queries in MYSQL alway use the optimizer hint /*! SQL_NO_CACHE */ to make sure you are not getting results from the query cache (if you have one set up).

Internet Explorer does not support http vary header

noreply@blogger.com (Nir Levy) — Wed, 10 Sep 2008 21:48:00 +0300

It was always assumed, but I just saw this quote from Microsoft:

Internet Explorer does not fully implement the VARY header per Requests for Comments (RFC) 2616. The Internet Explorer implementation of VARY is that it does not cache any data except for Vary-Useragent

Internet explorer cannot open the internet site: operation aborted

noreply@blogger.com (Nir Levy) — Tue, 9 Sep 2008 18:56:00 +0300

During an integration with a third party provider we were unfortunate enough to get the error

Internet explorer cannot open the internet site http://localhost: operation aborted

Yes, we have read http://support.microsoft.com/kb/927917 and did move the 3rd-party's script to be just before the </BODY> tag but to no avail.

After a long and hard effort by this provider they pointed out that while the view-source indeed looked like the script is just before </BODY> when looking at the DOM itself (they used Dominspector, I used the IE Developer Toolbar) it showed that the script was inside a <DIV> element.

As it turns out there was some code on the page that wrote an unclosed <DIV>. This made IE "fill in the blanks" and guess (wrongly) where it should place the closing </DIV> Fixing this javascript made the error go away on most pages. But not all.

I found out that the broken pages used some JQuery plugins (tooltips, jqmodal, etc.) to produce fancy decorations and effects. These scripts attached to $document.ready and did $('body').appendTo(...). This effectively added a couple of items to the DOM between the 3rd-party script and the </BODY>. I am still not quite sure why this should cause IE to choke but since we have a tight schedule we simply changed those scripts to add their stuff to some other elements. This indeed solved out problem.

They funny (or maybe sad) thing was that on Firefox we did not get any error but the page simply disappeared. Chrome worked just fine.

Misplaced elements with position:relative

noreply@blogger.com (Nir Levy) — Tue, 26 Aug 2008 20:41:00 +0300

We had a situation where we had three <DIV> elements in a column and the middle element had content dynamically loaded into it after the page has completed loading. When the middle element finished loading, some of the content of the last element seemed to forget its parenthood and floated nicely over the second element.

Naturally this only happened in Internet Explorer (what else).

After a lot of digging we noticed that the misplaced content of <DIV>#3 had a position:relative style (indicated as #4 in the image above). It is "well known" that IE does not handle relative positioning properly when the layout of the page change.

Luckily I chanced upon an article by Holly Bergevin et al. called On having layout which was enlightening. Understanding that IE will only respect the element's state if it have the hasLayout property quickly solved the problem.

In our specific case I used display:inline-block to force <DIV>#3 to have hasLayout set to true, which made IE respect the element's style and re-render the element's content (i.e. <DIV>#4) after re-positioning it. My only concern now is what effect this might have on rendering performance, but it will have to do for now.

Tip: you can use IE Developer Toolbar to check if an element hasLayout. It will show the hasLayout property as set to -1 if hasLayout is true.

Further credit is due to Ingo Chao who wrote relatively positioned parent and floated child – disappearance.