However, while I am well aware that the web is filled with various implementations of classical (i.e. class-based) object orientation in JavaScript, I can not help the feeling that they are over-engineered and over-complicated without need, quite often attempting to provide more functionality than is actually necessary. In short, I don’t feel they’re “classical” at all.

This is why I started working on my own implementation of classical OOP  in JavaScript, which is shown below. Currently this is only a proposal, although I already started using it in one of my projects. Anyway, let’s look at it first then I’ll try and explain how it works.

The Code

/** Class factory. */
function Class(members) {

    // proxy constructor
    var Proxy = function() {
        for (var item in members) {
            this[item] = members[item];
        }
    };

    // proxy inheritance
    Proxy.prototype = (members.base || Class).prototype;

    // class constructor
    var Build = members.init || function() {};

    // class inheritance
    Build.prototype = new Proxy();
    Build.prototype.base = Proxy.prototype;
    Build.prototype.constructor = Build;

    // ready
    return Build;
}

That is all, and this little function can have a whole world of nifty consequences like deep inheritance or access to the super-class via this.base (and others). But here is what happens: the basic principle at work here is the Proxy object, which essentially, contains all the members of our new class (the ones that are passed to the Class factory function as the members parameter).

So, instead of copying the given members straight into the newly created class, we put them into another object (i.e. the Proxy) and make that the prototype of our new class. This means that every class will point (via the prototype property) to an internal Proxy object holding the actual class members. The trick is that this Proxy, in turn, points to the ancestor’s Proxy object (again via the .prototype property) and so on.

The code (and further updates) may be found here.

The Concept

Here is a visual depiction of the system:

This trick enables one very important feature: classic class inheritance via the prototype property. Class members are never duplicated and the only additional objects ever created are the Proxy objects (one per class). Here are some features:

The Simplicity

You can create a class very easily, using a constructor (i.e. init()) and this class can have instance members, class members and/or static members:

var Person = Class({

    // constructor
    init: function(fname, lname, age, job) {

        // class member
        this.first_name = fname;
        this.last_name = lname;

        // instance members
        this.age = age;
        this.job = job;
    },

    // class members
    first_name: '',
    last_name: '',
});

// static member
Person.nationality = 'Romanian';

// create instance
var man = new Person('George', 'Enescu', 33, 'Classical Composer');

The Deep Inheritance

Classes can have single, but deep, inheritance (i.e. inheritance chains) where each object has direct access to the super-class via the .base property:

var A = Class({
    hello: function() {
        console.log('Hello world!');
    }
});
var B = Class({ // inherit A
    base: A,
    aloha: function() {
        console.log('Aloha world!');
    }
});
var C = Class({ // inherit B
    base: B,
    aloha: function() { // override method from B
        console.log('Goodbye world!');
    }
});

var c = new C();
c.hello(); // 'Hello world!'
c.aloha(); // 'Goodbye world!'
c.base.aloha.call(c); // 'Aloha world!'

Note: the c.base.aloha() method was invoked via call() in order to apply it onto the c object; invoking it as c.base.aloha() would have applied it onto the c.base object.

The Prototypal Concepts

The regular functionality of the prototype property is kept intact, so that changes to A.prototype will implicitly propagate to everything derived from class A (all instances, all sub-classes, all instances of sub-classes etc.):

// extend class A
A.prototype.doStuff = function() {
    console.log('Am doing some extra stuff!');
}

// check any child
c.doStuff(); // 'Am doing some extra stuff!'

[Update] This also means that you may affect the prototype of any class form an inheritance chain and it will do what you expect (i.e. propagate changes down the chain to all the children below that class).

Also the instanceof operator behaves as expected:

console.log(c instanceof A); // true
console.log(c instanceof B); // true
console.log(c instanceof C); // true

The .prototype.constructor property is also properly configured, so if you want, you can create new instances starting from an initial object, by extracting the object’s constructor:

// identical to C.prototype.constructor
var Cons = Object.getPrototypeOf(c).constructor; 

// create new object
var c2 = new Cons();

The Efficiency

Inheritance is provided without creating a new instance of the base class for every prototype. This is unnatural anyway since it calls the base constructor without arguments, which may have unwanted side-effects:

NewClass.prototype = new BaseClass(); // the weird way of prototyping

Also, class members are never copied between classes and no duplicates are ever created, thus saving memory and ensuring high access speed for the class members.

In addition, the entire code is only a few lines long (no, I’m not going to count), so it has a certain elegance to it.

The Conclusion

Maybe this technique is neither new, nor very full featured, but honestly, if I want mixins, extensible objects and other such exotic features, then why would I bother with old-school OOP classes anyway; the JavaScript prototypal approach is capable of all that, and more!

I’m not saying that this is *the* way of doing classic OOP in JavaScript, but I certainly feel it is more in-line with what I perceive to be the JavaScript mindset: keep it simple, use what you have at your disposal and don’t kill yourself trying to emulate other languages!

That being said I also think this code may be improved, so pitch in a comment or two if you like :D…

P.S.: For example, the Build() constructor may be easily modified to enforce the concept of abstract classes, effectively refusing to create the instance (i.e. via throw) if it detects that any of its members (inherited or not) is purely virtual (i.e. set to null or undefined). So a sub-class must either implement all the purely virtual members it inherits, or it will simply become abstract as well (I actually use this trick in production).

I wrote this because I needed it for a project and I did not think it was worth it to install some npm package just for this. The comment should explain most things you will need but ask me if you don’t understand something. Note that this code is not well tested (yet!) since I just wrote it this morning so post a comment when you find a bug.

Supports:

• Short and long options (i.e. ‘-t|--test‘).
• Option parameters (i.e. ‘-f /etc/resolv.conf‘).
• Mandatory options.
• Implicit help option.
• Option callback functions.
• Cumulated short options (i.e. ‘-ab‘).
• Repeatable options (i.e. ‘--add value1 --add value2‘).
• Non-option arguments.

Notes (extracted from the comments):

• Parser is case-sensitive.
• The ‘-h|–help’ option is provided implicitly.
• Successfully parsed options will be available as fields in the “options” object.
• Non-option arguments found will be available in order in the “arguments” array.
• Options and their parameters must be separated by space.
• Either one of «short» or «long» must always be provided.
• The «callback» function is optional and is invoked each time the option is encountered.
• Cumulated short options are supported (i.e. ‘-tv‘).
• If an error occurs, the process is halted and the help message is shown.
• Options with the “multiple” attribute will be cumulated into arrays (even if found only once).
• The parser does *not* test for duplicate definitions in the schema array.

See the code for everything else you need to use it (i.e. a sample schema definition). I will not waste any more time talking about it; if you like it, paste it at the top of your NodeJS script and start playing with it, e.g.:

$ node options.js -tf /some/file.txt foo bar

Also, please tell me about any improvements or fixes you produce. Thanks and… enjoy!

Why, you ask? For her: mademoiselle… Anastasia. She was born on the 5th of January, 2011 and she really wanted to say “Hi!” on daddy’s blog. Who am I to say no to such a gorgeous girl.

Anastasia giving a speech

Anastasia resting between speeches

For more pictures go here. Enjoy! (I know I do…)

6.5k+ active concurrent streams

2841.19 Mbps effective bandwidth

…end!»

Since my last post about describing the performance of Loomiere in production, quite a few things have happened. I tested a lot and found many things I would like to improve or add, I tried many hardware configurations to see what goes well with what. In short, I have a whole list of things I want to implement as soon as possible and the most important item on this list is to actually launch Loomiere to the public as soon as possible!

However, here is a quick update on the performance of Loomiere, but this time with a nice twist: it’s running on a SSD-based machine! Here are some specs:

• Intel Core I7 3GHz
• 12 GB RAM
• 6 x 256GB SSD Kingston
• 3 x 1 Gigabit Ethernet cards
• Ubuntu Lucid Lynx with 2.6.37-12-server kernel

And here are the results after only a few hours from launch (graphs follow):

• 4500+ active oncurrent connections (streams)
• 300+ MB/s of throughput (aprox. 2500 Gbps)

Concurrent connections

Network traffic: eth0

Network traffic: eth1

Network traffic: eth2

However, at this point we hit a wall: the mainboard or the kernel. I have to dig into this problem further, but from what I could gather so far, it seems that the mainboard or the Linux kernel can’t handle more punishment. Following are the CPU and memory graphs; it is clear that while the memory has no worries, the CPU is hogging! The problem lies somewhere between the I/O operations and the actual sending of data to the sockets (interrupts maybe?). More testing will follow regarding this issue.

CPU activity

Memory usage

Hmm… this is a deep one.

So, that’s it for now, I’m going to sleep. Good night .

Concurrent streams

Bandwidth utilization

This article is a quick peek into how Loomiere 2.0 is (currently) serving close to 3k active streams from a single server in production which is restricted to 1.6 Gbps of bandwidth (we are currently negotiating with our ISP to raise this limitation). All these statistics represent a single machine, in real time, over a period of 12 hours, equipped as follows:

• 2 x Xeon Dual Core (with HyperThreading) at 3.2GHz
• 16 GB RAM
• 15 x 1TB HDD (7200rpm)
• 4 x 1GB NIC’s
• This system runs on Ubuntu 10.04.1 LTS.

OK so here are the graphs:

• Cpu load.
• Network traffic (i.e. all interfaces cumulated).
• Active connections (i.e. concurrent streams).
• Memory usage.

The graphs may not yet be as stable as I would wish at the moment since I am still testing this setup as well as Loomiere itself, but I think soon I will have a more stable configuration and also some time to comment on these graphs.

Enjoy!

One great feature of LPeg is that it’s binary-safe, meaning that (unlike regular expressions) it can be safely used to parse binary data! This makes it an excellent tool for parsing binary protocols, especially network communication protocols, such as the Action Message Format (used by Adobe Flash for making remote calls and even in FLV movie files). I’ll leave it to you to explore the possibilities…

Beware that from here on, I assume that you know your way around Lua, LPeg and how they work.

The problem

That being said, this article is actually about an unusual roadblock I hit while using LPeg to build a Lua-based AMF parser, and the various solutions I found and/or came up with to overcome it (you didn’t think that I mentioned AMF before by accident, did you?).

The issue is LPeg’s implementation of repetitive patterns: in particular, its inability to match (or capture) a fixed number of occurrences of a certain pattern, although it can match a minimum or a maximum number of such occurrences, which is perfect for stream-oriented parsing (such as parsing programming languages) but insufficient for binary data.

Just to clarify, here’s a small list of LPeg patterns which correspond to the typical PCRE repetitive constructs (in each case we’re trying to match the string ‘cloth’):

For the last two cases (i.e. A and B), LPeg does not offer any simple constructs so we have to find a complex one. Let’s put aside the case B for now, and try to tackle A…

Attempt 1 (brute force)

The first solution was rather straight-forward: we explicitly compose (i.e. append) a pattern for every occurrence we need to match. I call this the brute force method (for obvious reasons). For example, to match the word ‘cloth’ exactly three times we would use the pattern lpeg.P'cloth' * lpeg.P'cloth' * lpeg.P'cloth'.

Naturally, this is quite inflexible: what if we need to match 183 occurrences? To achieve that, we use a pattern generation function. Here’s a complete working example:

local lpeg = require'lpeg'

-- Version 1.
-- Generates a big pattern where every interation has
-- a corresponding instance of the initial pattern. The
-- resulting pattern captures all occurrences in a table.
function multiply_1(item, count)
    local set = lpeg.P(true)
    while count > 0 do
        set = set * item
        count = count - 1
    end
    return lpeg.Ct(set)
end

-- Generate subject ('cloth,cloth,...').
local subject = string.rep('cloth,', 10)

-- A capture of the word 'cloth' optionally followed by a comma.
local item = lpeg.C(lpeg.P'cloth') * lpeg.P','^-1

-- Match exactly 10 items.
local result = lpeg.match(multiply_1(item, 10), subject)

-- Display capture table.
print(unpack(result))

This function adds a composition operation to the big pattern for each occurrence. One method of improving this is to reduce the amount of compositions to the a bare minimum. We do this by duplicating (i.e. growing in powers of 2) the pattern until the required size is reached (also used by LPeg’s regexp emulator):

-- Version 1(b).
-- Generates a (potentially large) pattern using only
-- the minimum possible amount of compositions.
function multiply_1b(item, count)
    local set = lpeg.P(true)
    while count >= 1 do
        if count % 2 >= 1 then
            set = set * item
        end
        item = item * item
        count = count / 2
    end
    return lpeg.Ct(set)
end

Looks good, however, this method presents a serious problem: the resulting pattern can get very large, forcing the limits of the Lua stack. It works well enough for small jobs, but it is unwise to generate more than a few hundred repetitions using this method (and even that may be pushing it). This means that the brute force method is unusable for heavy binary parsing, where we would often need to match even thousands of pattern repetitions.

We need to do better…

Attempt 2 (capture folding)

The next approach is a bit more complex and makes use of the more advanced features of LPeg: the lpeg.Cf() and lpeg.Cmt() captures.

In essence, we match at most the desired number of occurrences (using the ^-N operation described in the table at the top) and on every match (i.e. iteration) we execute a folding function (via lpeg.Cf()) that adds the item to the result table and also decreases a counter. At the end, we execute a checking function (via lpeg.Cmt()) which will test if the counter reached 0 (thus ensuring exact matching) and will fail if not.

-- Version 2.
-- Somewhat better approach, which starts with an empty table and gradually
-- folds every occurring capture of the initial pattern into this table, at
-- the same time decrementing the counter. Eventually, it also checks if the
-- counter reached 0 and fails otherwise.
function multiply_2(item, count)
    return lpeg.Cmt(

        -- Fold capture.
        lpeg.Cf(
            lpeg.Ct'' * item^-count,

            -- Append capture.
            function(set, item)
                count = count - 1
                table.insert(set, item)
                return set
            end),

        -- Final check.
        function(s, i, set)
            if count > 0 then
                return false
            end
            return i, set
        end)
end

Note that the capture starts with an empty table created from the empty string (i.e. lpeg.Ct''). This table is the first parameter given to the folding function (i.e. set); after all the iterations, it will contain all captured items and will then be given to the check-up function for evaluation. Eventually, if all goes well, it will be returned back as the final result of multiply().

Overall this method will behave much better than the first; the generated pattern is much more elegant. However, the generated pattern can still grow quite large in memory, even though it’s not that obvious (i.e. I was surprised to find that it also crashed when generating more than a few thousand iterations).

The issue is the item^-count bit. Checking the C code of the LPeg implementation, quickly reveals that whenever count is not 0 (i.e. which is a special case), the engine is forced to allocate count objects in memory, and while these objects are lighter than a complete item pattern (as in the first solution) they are still scaling-up the problem badly!

Eventually, this method only allows generating repetitive patterns of a few thousand iterations (that is as far as I was able to push it). But, what if we need to handle a rogue pattern of 35 000 occurrences? What then? No!… we still need better!

Attempt 3 (classic loop)

The only thing left to try is a (slightly) more rudimentary technique. Basically, it is the same as the second approach but with a significant twist: instead of using lpeg.Cf() to “accumulate” (i.e. fold) every occurrence of item to the result table, we just use a straight-forward loop that will advance through the input stream at every iteration.

That makes the pattern completely dynamic, i.e. totally non-proportional to the number of matched occurrences. As a result, this method allows for an unlimited number of iterations with no memory wastes.

-- Version 3.
-- Best method so far, capable of an unlimited number of iterations. It uses
-- an explicit loop to walk over all all iterations at match-time, so there
-- is no potentially huge pattern ever created. This method simply populates
-- the result table at each iteration, or fails the entire set if the count
-- is not reached precisely.
function multiply_3(item, count)
    return lpeg.Cmt(lpeg.P(true),
        function(s, i)
            local set, offset = {}, i
            for j = 1, count do
                set[j], offset = lpeg.match(item * lpeg.Cp(), s, offset)
                if not offset then
                    return false
                end
            end
            return offset, set
        end)
end

When applied to a string, this will start matching at the current position (i.e. due to the lpeg.P(true) construct). The lpeg.Cmt() capture will trigger the iteration function immediately and there, in turn, will loop for ‘count’ number of times, matching the item at every iteration and appending it to the results table. At the same time, at each iteration, the offset is advanced by capturing the position immediately following the matched item (i.e. the lpeg.Cp() construct). If anything goes wrong, then the entire match fails (i.e. via return false).

This method works flawlessly (as far as I could tell). The generated pattern is minimal (memory-wise), the source code is very small, and the function is overall very fast (only slightly slower than the first two approaches, when matching small-sized patterns). But most importantly, with this version, matching huge numbers of item repetitions is trivial (e.g. tens of thousands, millions…).

Intervals

From here, implementing the case B (i.e. multiply(X, Y)) is quite easy…

One approach would be to use multiply_3(item, X) to ensure the minimum number of occurrences (i.e. X) and then append the pattern item^-(Y-X) which will optionally match the remaining occurrences up to the maximum amount allowed. This method will work, and can also be used with the first two solutions, but will still (potentially) waste memory when (Y-X) is large.

A better (and still easy) approach would be to modify the loop inside multiply_3() to also accept an optional number of iterations (above the minimum). Here’s how…

-- Version 3(b).
-- The final variant with support for interval matching
-- based on the third capturing approach (classic loop).
function multiply_3b(item, min, max)
    return lpeg.Cmt(lpeg.P(true),
        function(s, i)
            local set, offset, attempt = {}, i, 0
            for j = 1, max or min do
                set[j], attempt = lpeg.match(item * lpeg.Cp(), s, offset)
                if not attempt then
                    if j > min then
                        break
                    else
                        return false
                    end
                end
                offset = attempt
            end
            return offset, set
        end)
end

In conclusion…

So, that’s that. Enjoy!
I would be surprised if anyone would actually read this banter.
If you did, hat’s off!

Entering lua2c.lua

Lately I became quite interested in Lua (a lot actually). It has phenomenal speed, exceptional interfacing with C and some features and libraries that just make my day (i.e. coroutines, lpeg, lua-ev and others), and since I needed to embed some Lua scripts (entirely) in a C project I’m currently working on, I ended up adapting Mike Edgar’s “bin2c.lua” script (which takes a Lua script and turns it into a C header file) to suit my needs.

Basic functionality

Specifically, this adaptation generates a function that takes a Lua state as the only argument and then runs the embedded Lua code in the given state after which it returns the status (as opposed to putting the code straight in the top-level scope of the generated file). This makes it easier to embed code in C and then invoke it, and also to apply the same code onto multiple Lua states (e.g. multiple threads).

Check further down for a short usage sample.

Options

• -c compiles to Lua bytecode for (slightly) faster loading (does not work with LuaJIT)
• -s minifies the source code before embedding to minimize wastes
• -e applies a mild (XOR-based) obfuscation to prevent embedding of plain text
• -u produces a (void) function that never fails and panics on errors

Download

So here it is, enjoy: lua2c.lua (MIT license, same as Lua)

Just to be clear

For example, given a lua script file called ‘test.lua’ (shown below) this is how we would go about embedding it into a simple C program called ‘test.c’.

-- test.lua: a small sign that we're up and running...
print("I am an embedded Lua script running inside a C binary! Hmm... cozy!")

And here is the C file (rather overly commented)…

/*
 * Include our embedded script.
 */
#include "test.h"

/*
 * Invoke the script.
 */
int main(int argc, char* argv[]) {

    // create a Lua state
    lua_State* L = lua_open();

    // load basic libraries into state
    luaL_openlibs(L);

    // invoke embedded script into state
    // by calling the generated function
    load_test_lua(L);

    // finish
    lua_close(L);
    return 0;
}

And here’s how we can compile all this into an executable. Note that I tested this on Ubuntu Lucid and the following packages had to be installed: lua5.1, liblua5.1-0-dev, liblua5.1-lpeg2, liblua5.1-bitop0. Your configuration may differ, and if you don’t have the pre-built Lua packages, you will have to install them manyally (along with the dependencies)!

# generate (unprotected, minified and encrypted) C code
lua lua2c.lua -seu test.lua > test.h

# compile C program against Lua library
gcc test.c -o test `pkg-config lua5.1 --libs --cflags`

# run program
./test
I am an embedded script running from inside a C binary! Hmm... cozy!

Let me know if it works fr you, or if you think of some cool improvement!
Cheers!

Source code: loomiere-0.2.1-tar.gz

Warning:
Any software downloaded from this website is *never* to be associated in any form with pornographic or erotic content! I.E. “Loomiere/Stream” must *never* be used to stream pornographic or erotic videos! There are plenty alternatives if you can’t help it.

Just a teaser:
Fighting the video-streaming problem has taught me very many things which, in turn, led me to realize that I might be able to use a substantially different streaming approach to achieve a massive amount of optimization in this field. So quite soon, I think I’ll have ready a brand new (and far more powerful) streamer that aims at making it possible for a single server to serve many thousands (I am still testing this) of streams simultaneously using commodity hardware. I am not yet settled on whether this project will be commercial, open-sourced or both but I hope to clear this aspect soon as well. Real-world streaming (i.e. before and after) statistics will be made available on release.

Stay tuned!

Most (not all) of these plugins are generic, meaning that they are not bound to a specific type of blog and may (or should) be used regardless of what you write.

All of these plugins are top quality (IMHO) and provide an invaluable service for any blogger. I wholeheartedly recommend that you use and support them. I think that all of these should be apart of a standard WordPress installation.

1. Akismet

Website: http://akismet.com/download/
Everyone knows this one but it deserves mention. It prevents content spam. I’ve used it for about two years and so far it did [edit] not miss! I believe it now comes bundled with WordPress out-of-the-box.

2. Bad-behaviour

Website: http://www.bad-behavior.ioerror.us/download/
Prevents link and robot spam. Again, very good service repelling e-mail address harvesters, spam bots and all kinds of evil insects.

3. WPIDS

Website: http://php-ids.org/2007/09/12/wpids-phpids-your-wordpress-the-comfy-way/
An excellent alliance between WordPress and the PHPIDS project. It monitors your website for a very wide range of security attacks and does it very well.

4. WP Total Cache

Website: http://www.w3-edge.com/wordpress-plugins/w3-total-cache/
This plugin enhances the performance of your blog and it seems to me, after trying out wp-cache and wp-super -cache that this is the best of the lot. It can even use a memcached server (if you have one).

5. FlagrantDisregard FeedBurner

Website: http://flagrantdisregard.com/feedburner/
This plugin provides the fundamental functionality of redirecting you blog feeds to FeedBurner, so you can track your subscribers properly.

6. WordPress Stats

Website: http://wordpress.org/extend/plugins/stats/
Again, a fundamental feature: basic statistical tracking of your website right in the dashboard.

7. CodeColorer

Website: http://kpumuk.info/projects/wordpress-plugins/codecolorer/
I have yet to see a better syntax highlighting plugin than this one. Full-featured, configurable, knows any language in the book and very easy to use. Mix in a good mono-spaced font and your set!

Ok so maybe there are more, but these, I find myself installing constantly on all blogs that fall into my hands (except number 7 – CodeColorer – since it is only meant for posting source code).

Enjoy… and contribute if you like!

And, why not!? I love the idea of making my source codes more readable by using some custom font (Consolas in action). Anyway, I started looking around for some fonts and was not pleased with any of them (well to be fair I did like DejaVu Sans Mono but not as much as Consolas, unfortunately).

Long story short, here is a very generous list of web-embeddable fonts which can be used freely in commercial scenarios, gathered by the awesome FontSquirrel.com website (an instant friend!). They even provide a @font-face generator interface that transforms a given font file (TTF or OTF) into the needed formats (mainly EOT for IE) and also generates the embedding CSS3 rules. You should really give it a go!

And if you are wondering how I could speak of embedding Consolas on my blog, knowing full-well the licensing policy of Microsoft in these matters, here is your answer: one can buy a license to embed Consolas on the web (I paid $17.50) from here.

You can see it in action on any of my posts that contains some source code. Like this one!

I know, I know! I also would have never believed that I would actually promote a Microsoft product on my blog, but the Consolas font deserves all the credit! It is simply unsurpassed as far as programming goes.

So, does anyone know any other web-embeddable fonts worth mentioning?