Codexon

WordPress Spam: Javascript antispam such as WP-SpamFree/WP-HashCash don’t work anymore

2010-08-01T23:35:53Z

Being a user of websites myself, I have always been annoyed at Captchas like this:

An annoying captcha from Google

Which is why I’ve avoided using them on Codexon, opting to use JavaScript based anti-spam like WP-SpamFree or WP-HashCash instead. Unfortunately, JavaScript anti-spam is defeated the moment spammers decide to add Javascript to their bots.

It worked well up until a couple of months ago, but there has recently been a surge of spam getting past the JavaScript anti-spam. I believe spammers are now targeting them. After installing a captcha, spam here at Codexon has dropped from 10+ a day to zero.

So if you have a website and are considering just using a convenient JavaScript solution, be wary.

2 comments | Donate

Make a Python JIT compiler without writing a single line of C or 3rd party library

2010-07-11T02:01:28Z

JIT is just a fancy term for making native code during runtime; another word for “eval” except you won’t be sneered at when you implement it.

I will show you how this can all be done in the comfort of Python without having to write a single line of C or use a 3rd party library.

from ctypes import *
 
code32 = ('\xb8\x7b\x00\x00\x00' # mov eax, 123
          '\xc3')                # ret
 
code64 = ('\x48\xc7\xc0\x7b\x00\x00\x00' # mov rax, 123
          '\xc3')                        # ret
 
def windows(code):
    PAGE_EXECUTE_READWRITE = 0x40
 
    code_buffer = create_string_buffer(code, len(code))
    code_buffer_address = addressof(code_buffer)
    windll.kernel32.VirtualProtect(code_buffer_address, len(code), PAGE_EXECUTE_READWRITE, byref(c_int()))
    return code_buffer_address
 
def linux(code):
    PROT_WRITE = 2
    PROT_EXEC  = 4
 
    libc = CDLL('libc.so.6')
    code_buffer_address = libc.valloc(len(code))
    memmove(code_buffer_address, code, len(code))
    libc.mprotect(code_buffer_address, len(code), PROT_WRITE|PROT_EXEC)
    return code_buffer_address
 
import platform
bit, os = platform.architecture()
 
code     = code32 if bit == '32bit' else code64
platform = windows if os == 'WindowsPE' else linux
 
function_prototype = CFUNCTYPE(c_int)
jitfunction = function_prototype(platform(code))
print jitfunction()

The code does the following:

Allocates space for the code.
Writes the code to memory.
Enables execution privileges on that memory.
Creates a function prototype gateway so you can use it from Python. (this feature is undocumented!)

Now that I’ve shown the basic steps, it should be simple for you to hook this up to a lexer or look up additional assembly codes to use.

2 comments | Donate

Fun with Facebook Graph

2010-05-12T22:23:18Z

Facebook has released an API to access their data through the web. Apparently you could do the same thing by logging on and using the gui, but now everyone can benefit from the lack of privacy!

Who hates their boss?

http://graph.facebook.com/search?q=I%20hate%20my%20boss

The spammers are spamming their acai berry.

http://graph.facebook.com/search?q=acai%20berry

Interesting crab infections.

http://graph.facebook.com/search?q=itchy%20crabs

Juicy medical information for insurance companies.

http://graph.facebook.com/search?q=colorectal%20surgery

Got any other interesting searches? Leave a comment below.

Google Cache of the API (no Facebook login necessary)

One comment | Donate

Is Cassandra ready yet?

2010-03-24T21:25:34Z

With Twitter, Facebook, Digg, and Reddit using the open-source Amazon Dynamo clone, Cassandra may be the winner of the NoSQL race.

But is it ready for a general purpose (excluding transactions and other obvious features) data store? There are a few kinks right now that may hamper your adoption of Cassandra.

You must restart Cassandra in order to add/delete/rename ColumnFamilies (tables)

This can be a problem if you want to run your website without interruption. A rolling restart is recommended, but that will also impact your uptime if you choose a consistency level of “ALL”. A patch to solve this problem is being worked on and is scheduled for version 0.7. The current version is 0.5

Issue 44

Getting a range of rows may return empty “tombstoned” rows

Due to the way that Cassandra is engineered, Cassandra must mark deleted rows as “tombstones” so other nodes will know that it has been deleted instead of the other node not having a copy propagated yet. Tombstones will only disappear after a costly “compact” command.

The problem is that if you want to return users 40-50, you may get 10 empty rows. To fix this you would then have to repeatedly query Cassandra until you get 10 non-empty rows. This back and forth may cause significant overhead on your network. Before it was done server-side, but apparently the early adopters preferred to return empty rows . This is a mistake in my opinion.

No distributed versioning (only last write wins)

Vector clocks support is scheduled to be added, but there is no ETA. This may be important for some applications. For example, if you have a shopping cart and you add Item A on Server 1 and Item B on Server 2 and the servers momentarily split, your shopping cart may end up with only Item B if you use “last write wins”. With vector clocks, you will get the chance to merge them.

Issue 580

No advanced queries through map/reduce (yet?)

Hadoop integration is planned, but to my knowledge, this isn’t ready yet. Your application may or may not need this feature.

Issue 342

No atomic increment/decrement

This feature is necessary for counting things like page-views accurately. There is a hack here using a 3rd party application called zookeeper.

Issue 721

Even with these current shortcomings, you may find Cassandra to be suitable for you right away. It is definitely one of the best NoSQL solutions available.

Leave a Comment | Donate

Hot code swapping for servers not written in Erlang

2010-01-20T03:40:42Z

The recent server benchmark posted here comparing Erlang, Haskell, and Python understandably upset many people. Today we will move away from polarizing benchmarks and more about features.

Even though Erlang may not be the fastest language, it still has at least one great feature: hot swapping code without restarting the server. This is something that every long running server should have, but unfortunately it comes at the cost of performance, and some languages simply are not built to support it.

However, here is a way to emulate it for a web server, thanks to techniques borrowed from Nginx.

Here are the steps:

Use the spawn family of system calls (ex. spawnl spawnvp) with the flag P_NOWAIT to execute itself. This new process has access to the open sockets.
The new process should open a predetermined file containing the file descriptors of the sockets it needs to use saved by the old process.
Start calling “accept” on the same listening sockets that the old process was accepting.
Signal the old process to stop accepting, and to quit when all requests are finished.

While Python has a rudimentary reload, it may still be preferable to use this method as it greatly simplifies the upgrade logic. Upgrading functions piecewise may lead to corruption of logic and data.

Here is an example in Python that reloads itself in a loop using this method. It saves the file descriptors for 2 connected sockets which allows the new process to signal the old one at will. Simply run the program, and then edit the print statements below to watch it change.

import os
import time
import socket
import sys
import fcntl
import _multiprocessing
from multiprocessing import Pipe
 
# global pipe variable
pipe       = None
spare_pipe = None
 
def PipeFromFD(fd):
    return _multiprocessing.Connection(fd)
 
def recordPipe(p1, p2):
    with open('fd.txt', 'w') as f:
        f.write('%s,%s' % (p1.fileno(), p2.fileno()))
 
def upgrade():
    os.spawnlp(os.P_NOWAIT, 'python', 'python', sys.argv[0], '-upgrade')
 
def upgradeBootstrap():
    global pipe, spare_pipe
    with open('fd.txt', 'r') as f:
        p1, p2 = (int(fd) for fd in f.read().split(','))
        pipe = PipeFromFD(p1)
        spare_pipe = PipeFromFD(p2)
 
    recordPipe(spare_pipe, pipe)
    pipe.send("I am upgraded!")
 
if __name__ == '__main__':    
 
    if '-upgrade' in sys.argv:
        upgradeBootstrap()
    else:
        pipe, spare_pipe = Pipe()
        recordPipe(spare_pipe, pipe)
 
    pid = os.getpid()
    print '%i: Version 1. Upgrade in 5 seconds.' % pid
    time.sleep(5)
    print '%i: Upgrading' % pid
    upgrade()
    print '%i: Message from new process: %s' % (pid, pipe.recv())
    print '%i: Upgrade Complete. Exiting' % pid

Example Output

user@test:/mnt/shared$ python upgrade.py
22930: Version 1. Upgrade in 10 seconds.
22930: Upgrading
22971: Version 2. Upgrade in 10 seconds.
22930: Message from new process: I am upgraded!
22930: Upgrade Complete. Exiting
user@test:/mnt/shared$ 22971: Upgrading
23025: Version 3. Upgrade in 10 seconds.
22971: Message from new process: I am upgraded!
22971: Upgrade Complete. Exiting
killall python

7 comments | Donate

Debunking the Erlang and Haskell hype for servers

2010-08-04T03:18:11Z

Ever since Intel and AMD have been selling multi-core cpus, the Erlang hype has been growing continuously. The number of high profile projects using Erlang were flagrantly announced over the blogosphere as the coming of the second C.

We kept hearing about: rabbitmq, couchdb, nearly all of Amazon’s AWS, Heroku’s routing grid, Facebook chat, all switching over to Erlang because it was the fast and concurrent language of the future. No other language could hold a candle to a language run by telecom giant Ericsson which was then validated by Amazon and Facebook! Or not? When I tried to find benchmarks for Erlang, they all showed otherwise.

http://muharem.wordpress.com/2007/07/31/erlang-vs-stackless-python-a-first-benchmark/

http://www.joeandmotorboat.com/2009/01/03/nginx-vs-yaws-vs-mochiweb-web-server-performance-deathmatch-part-2/

Since I initially bought into the hype, I felt compelled to test it myself to see if they made a mistake. To do this, I wrote a simple http server in Erlang, Haskell, and Python that simply outputs an HTTP reply “Pong!”. And here are the results in a graph.

The green line is the maximum req/sec possible. Higher is better.

For more details, continue reading.

Update
See comments for a faster implementation for Haskell that puts it almost on par with the Erlang.

Summary

A simple server was written in Python, Haskell, and Erlang. The server accepts any input from clients and outputs

HTTP/1.0 200 OK
Content-Length: 5
 
Pong!

and then disconnects. The servers were benchmarked with httperf compiled with increased select limit of 65535 connections. In every test, 0 errors occurred.

Python

1st place
Using a single process/thread epoll sustained the most connections per second before hitting a cpu bottleneck.
This was highly unexpected since we are comparing it with Erlang and Haskell.

Erlang

2nd place.
Having SMP/multicore enabled reduced requests/sec by a factor of 4!
Enabling kernel polling (epoll) made a negligble performance impact (less than +- 1%).
Someone suggested enabling “active” receive mode which asynchronously puts received packets in the Erlang message queue. This made a negligible difference.

Haskell

3rd place
This might be because it uses select instead of epoll. However, this did not make a difference for Erlang, so I suspect it would not for Haskell as well.
The program was compiled with -O2 with modest performance gains. Compiling with –threaded for SMP support reduced performance by a factor of 2!

Conclusion

DO NOT WRITE A SERVER IN ERLANG JUST BECAUSE YOU HEARD ERLANG IS THE FASTEST AND MOST CONCURRENT LANGUAGE.

Erlang is not “made for multicore”. Erlang only just received SMP support in 2006!

Setup

2 x 4 core Xeon E5420 @ 2.50GHz

The following steps were done to lift the usual limits that prevent default installations of Linux from being able to hammer servers. Neglecting this step will artificially cap concurrent connections at 1024 while timeout errors will increase. This is an important step that many other benchmarks have left out, and it shows in the error rate. All tests here resulted in 0 errors.

Httperf also depends on select, so the select limit was increased to 65,535 file descriptors.

edit /etc/security/limits.conf and add the lines: “* hard nofile 65535” and “* soft nofile 65535" (reboot if ulimit -n does not change)
edit /usr/include/bits/typesizes.h and change “#define __FD_SET_SIZE 1024″ to “#define __FD_SET_SIZE 65535″
compile httperf from source
Increase kernel file descriptor limit: sudo bash -c “echo “128000″ > /proc/sys/fs/file-max
Increase the backlog: sudo sysctl -w net.core.netdev_max_backlog=60000
Increase maximum connection limit: sudo sysctl -w net.core.somaxconn=250000

Software

Ubuntu 9.04 x64

Erlang: BEAM 5.6.5

Haskell: GHC 6.10.4

Python: CPython 2.6.4

Httperf: 0.9.0

Httperf was run with the following settings, where port and rate were adjusted accordingly:

httperf –port=8000 –num-conns=40000 –rate=5000

Erlang active mode

-module(echo).
-export([listen/1]).
 
-define(TCP_OPTIONS, [binary, {packet, 0}, {active, true}, {reuseaddr, true}, {backlog, 60000}]).
 
% Call echo:listen(Port) to start the service.
listen(Port) ->
    {ok, LSocket} = gen_tcp:listen(Port, ?TCP_OPTIONS),
    spawn(fun() -> accept(LSocket) end).
 
% Wait for incoming connections and spawn the echo loop when we get one.
accept(LSocket) ->
    {ok, Socket} = gen_tcp:accept(LSocket),
    Pid = spawn(fun() -> loop(Socket) end),
    gen_tcp:controlling_process(Socket, Pid),
    accept(LSocket).
 
% Echo back whatever data we receive on Socket.
loop(Socket) ->
    receive
        {tcp, Socket, Data} ->
            gen_tcp:send(Socket, "HTTP/1.0 200 OK\r\nContent-Length: 5\r\n\r\nPong!\r\n"),
            gen_tcp:close(Socket);
        {error, eaddrinuse} ->
            done
    end.

Haskell

import IO
import Control.Exception hiding (catch)
import Control.Concurrent
import Network
import System.Posix
 
main = withSocketsDo (installHandler sigPIPE Ignore Nothing >> main')
main' = listenOn (PortNumber 9900) >>= acceptConnections
 
acceptConnections sock = do
        conn@(h,host,port) <- accept sock
        forkIO $ catch (talk conn `finally` hClose h) (\e -> print e)
        acceptConnections sock
 
talk conn@(h,_,_) = hPutStrLn h "HTTP/1.0 200 OK\r\nContent-Length: 5\r\n\r\nPong!\r\n" >>
                    hFlush h >>
                    hClose h

Python

import select
import socket
 
EPOLLIN = select.EPOLLIN
EPOLLOUT = select.EPOLLOUT
 
epoll = select.epoll(60000)
connections = {}
 
class Server(object):
    def __init__(self):
        sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        sock.setblocking(0)
        sock.bind(('', 8050))
        sock.listen(60000)
        self.socket = sock
 
        fileno = sock.fileno()
        connections[fileno] = self
        epoll.register(fileno, EPOLLIN)
 
    def onInput(self):
        sock, address = self.socket.accept()
        Client(sock)
 
class Client(object):
    input  = ''
    output = "HTTP/1.0 200 OK\r\nContent-Length: 5\r\n\r\nPong!\r\n"
 
    def __init__(self, sock):
        sock.setblocking(0)
        fileno = sock.fileno()
        epoll.register(fileno, EPOLLIN|EPOLLOUT)
        connections[fileno] = self
        self.socket = sock
 
    def onInput(self):
        newdata = self.socket.recv(1024)
        if len(newdata) is 0:
            self.close()
        self.input += newdata
 
    def onOutput(self):
        sent = self.socket.send(self.output)
        self.output = self.output[sent:]
        if len(self.output) is 0:
            self.close()
 
    def close(self):
        fileno = self.socket.fileno()
        del connections[fileno]
        epoll.unregister(fileno)
        self.socket.close()
 
Server()
 
while 1:
    for fd, event in epoll.poll():
        if event &amp; EPOLLIN:
            connections[fd].onInput()
 
        if event &amp; EPOLLOUT:
            connections[fd].onOutput()

Source Code

Erlang version 1 (active mode): hello.erl

Erlang version 2 (passive mode): hello2.erl

Python: hello.py

Haskell: hello.hs

116 comments | Donate

How to create jobs in the US: Keep the corporate tax, lower consumer tax.

2010-01-06T08:58:22Z

Many Republicans believe in Reagan’s trickle down theory. They believe that by offering businesses a tax break, they will use the money to hire more people.

You don’t improve the economy by paying people to sit around and not work. You improve the economy by lowering taxes so small businesses will create more jobs.

- Representative John Linder Georgia Republican

How this frequently parroted sentiment is so popular is understandable, but… does anyone else see the problem with this argument?

If you give businesses tax breaks, they are going to horde it because they know that consumers aren’t going to have any money for them to earn.

However, if you give consumers and families a tax break, it encourages businesses to produce more, likely hiring more people in the process because that’s where the money is. Individuals and families also pay a higher tax rate than corporations that like to:

Deduct everything as a business expense or low capital gains tax. Like personal business laptops and cars.
Hide it in tax haven states or countries like Ikea and Walmart
Use it to boost company stock which can be cashed out at a rock-bottom 15% capital gains tax, the same tax rate as someone earning a paltry $8,026-32,550 in 2008.

Potential Rebuttals

But individuals will also horde money!

Individuals cannot horde as much as a corporation. Individuals need to pay rent or land tax at the minimum. A business could essentially exist without paying anything but rent.

On top of that, a “business” owner could deduct part of the cost of his own home from tax if s/he claims it was being used for business purposes.

But businesses also spend money!

While this may be true, b2b (business selling to businesses) cannot survive without b2c (business selling to consumers). You could argue that businesses could just always be exchanging goods/currency to other businesses, but someone has to provide necessary services like water, food, and shelter to individuals.

Actually, a purely b2b society would be possible. But it would consist of businesses doling out bundles of basic living needs to individuals, effectively becoming a mini-government to employees who don’t have any voting rights (unless they own a considerable amount of stock), and could usually be fired on a whim.

Leave a Comment | Donate

A Better US Income Tax

2009-12-17T01:48:31Z

People hate taxes, and everyone can agree to disagree on them. One thing most people agree on though is that it’s too complicated.

Here is a diagram of the US income tax in 2008.

US Income Tax 2008

What’s wrong with the current piecemeal tax rate?

It’s complicated.
People want to earn less if taking a raise barely puts them in the next tax bracket.
The tax rate increases faster for people earning very little compared to people earning a lot.

What’s wrong with a flat tax?

The tax rate increases more slowly as your ability to earn more money increases exponentially.
- The more money you have, the easier it is to get bargains, higher interest rates on bank accounts, and investment opportunities that allow you to own controlling shares of the next big company.
A 20% tax of someone earning $20,000 a year can cause starving and homelessness while 20% of $10 million doesn’t really make a dent.

How do we fix this?

Solution 1

Exponential Curve

T(i) = p e^(i/G)

min( max tax rate, T(i) )

p = minimum tax

e = Euler’s constant

G = growth rate = i_{high threshold} / ln (T_max / p)

This exponential curve is one such solution that eases the problem of people on the low end of the income scale getting taxed at a higher rate of change than people who are earning the most.

One potential problem with this tax curve is deciding where and how much the maximum cutoff peak is. This is something for experienced statisticians, economists, and psychologists to decide.

A possible idea would be to set the maximum cutoff at a certain percentage of the GDP and having the tax rate close to 100% at that level. It is of course considered communist to have a tax rate at around 100% somewhere, but we can probably agree that there would be revolts should anyone control 50% or more wealth in a single nation.

Solution 2

Smoothed sigmoid curve

T(i) = m / (1 + e^(-i/s+p))

m = Maximum tax rate

s = stretch factor

p = poverty offset

This smooth sigmoid curve is an improved version of the current tax curve. It is completely smooth instead of being piecemeal.

The one advantage this may have over the exponential curve is if you subscribe to the theory that it slowly becomes harder to get more money as you become richer. From anecdotal evidence, this is false, but could conceivably be true as you approach a large percentage of the GDP.

Potential Rebuttals

“But the Laffer Curve blah blah…”

The Laffer Curve is a psychological curve. It is even different in some countries than others, and may fluctuate over time. The supposed optimal maximum tax rate for the US is around 35-37% which we have thanks to Bush. But even Bush’s economic advisors have said that his tax breaks have not paid for themselves.

These formulas are highly adjustable, and experienced statisticians, economists, and psychologists should be the ones to tweak them.

“Higher tax rates will increase tax evasion”

While this may be true, this is avoiding the problem. It’s like putting a band-aid on the symptoms instead of curing the disease.

Leave a Comment | Donate

Website Contact Page Finally Fixed

2009-12-17T01:32:28Z

The contact form for this website has finally been fixed!

For some reason the WordPress mail sending function refused to work.

All thanks go to this handy plugin which uses SMTP instead of the PHP function mail.

http://www.codexon.com/contact

Leave a Comment | Donate

Wife beating and domestic violence might be human nature

2009-11-26T02:01:06Z

Recently England has mandated that children from the age of 5 would be taught “how to prevent violent relationships“.

One could explain nearly any “odd” but prevalent human behavior through evolutionary psychology. You would think it would be strange to attack someone you were living with and providing benefits. But I found it hard to be surprised when I found a paper that explained domestic abuse as mate retention behavior.

In a 2002 study by Peters, Shackleford, and Buss, around 4000 cases of wife abuse in New York was analyzed. They found that abuse was positively correlated with fecundity. Fertile and younger women were 10 times more likely to suffer domestic violence compared to older women. Large decreases in risk were found after the females were 45 years of age when menopause is likely to occur.

http://www.toddkshackelford.com/downloads/Peters-VV-2002.pdf

If domestic abuse truly serves an evolutionary purpose, schooling might not be enough to curb tendencies.

Leave a Comment | Donate