Entering lua2c.lua

Lately I became quite interested in Lua (a lot actually). It has phenomenal speed, exceptional interfacing with C and some features and libraries that just make my day (i.e. coroutines, lpeg, lua-ev and others), and since I needed to embed some Lua scripts (entirely) in a C project I’m currently working on, I ended up adapting Mike Edgar’s “bin2c.lua” script (which takes a Lua script and turns it into a C header file) to suit my needs.

Basic functionality

Specifically, this adaptation generates a function that takes a Lua state as the only argument and then runs the embedded Lua code in the given state after which it returns the status (as opposed to putting the code straight in the top-level scope of the generated file). This makes it easier to embed code in C and then invoke it, and also to apply the same code onto multiple Lua states (e.g. multiple threads).

Check the end of the post for a usage sample.

Options

• -c compiles to Lua bytecode for (slightly) faster loading (does not work with LuaJIT)
• -s minifies the source code before embedding to minimize wastes
• -e applies a mild (XOR-based) obfuscation to prevent embedding of plain text
• [UPDATE:]
  -u produces a (void) function that never fails and panics on errors

Errata

Sorry about the broken syntax highlighting. It seems that GeSHI has problems understanding the multiline comments of Lua. C’est la vie…

The code

So here it is, enjoy!

Just to be clear

For example, given a lua script file called ‘test.lua’ (shown below) this is how we would go about embedding it into a simple C program called ‘test.c’.

And here is the C file (rather overly commented)…

And here’s how we can compile all this into an executable. Note that I tested this on Ubuntu Lucid and the following packages had to be installed: lua5.1, liblua5.1-0-dev, liblua5.1-lpeg2, liblua5.1-bitop0. Your configuration may differ, and if you don’t have the pre-built Lua packages, you will have to install them manyally (along with the dependencies)!

Let me know if it works fr you, or if you think of some cool improvement!
Cheers!

Source code: loomiere-0.2.1-tar.gz

Warning:
Any software downloaded from this website is *never* to be associated in any form with pornographic or erotic content! I.E. “Loomiere/Stream” must *never* be used to stream pornographic or erotic videos! There are plenty alternatives if you can’t help it.

Just a teaser:
Fighting the video-streaming problem has taught me very many things which, in turn, led me to realize that I might be able to use a substantially different streaming approach to achieve a massive amount of optimization in this field. So quite soon, I think I’ll have ready a brand new (and far more powerful) streamer that aims at making it possible for a single server to serve many thousands (I am still testing this) of streams simultaneously using commodity hardware. I am not yet settled on whether this project will be commercial, open-sourced or both but I hope to clear this aspect soon as well. Real-world streaming (i.e. before and after) statistics will be made available on release.

Stay tuned!

Most (not all) of these plugins are generic, meaning that they are not bound to a specific type of blog and may (or should) be used regardless of what you write.

All of these plugins are top quality (IMHO) and provide an invaluable service for any blogger. I wholeheartedly recommend that you use and support them. I think that all of these should be apart of a standard WordPress installation.

1. Akismet

Website: http://akismet.com/download/
Everyone knows this one but it deserves mention. It prevents content spam. I’ve used it for about two years and so far it did [edit] not miss! I believe it now comes bundled with WordPress out-of-the-box.

2. Bad-behaviour

Website: http://www.bad-behavior.ioerror.us/download/
Prevents link and robot spam. Again, very good service repelling e-mail address harvesters, spam bots and all kinds of evil insects.

3. WPIDS

Website: http://php-ids.org/2007/09/12/wpids-phpids-your-wordpress-the-comfy-way/
An excellent alliance between WordPress and the PHPIDS project. It monitors your website for a very wide range of security attacks and does it very well.

4. WP Total Cache

Website: http://www.w3-edge.com/wordpress-plugins/w3-total-cache/
This plugin enhances the performance of your blog and it seems to me, after trying out wp-cache and wp-super -cache that this is the best of the lot. It can even use a memcached server (if you have one).

5. FlagrantDisregard FeedBurner

Website: http://flagrantdisregard.com/feedburner/
This plugin provides the fundamental functionality of redirecting you blog feeds to FeedBurner, so you can track your subscribers properly.

6. WordPress Stats

Website: http://wordpress.org/extend/plugins/stats/
Again, a fundamental feature: basic statistical tracking of your website right in the dashboard.

7. CodeColorer

Website: http://kpumuk.info/projects/wordpress-plugins/codecolorer/
I have yet to see a better syntax highlighting plugin than this one. Full-featured, configurable, knows any language in the book and very easy to use. Mix in a good mono-spaced font and your set!

Ok so maybe there are more, but these, I find myself installing constantly on all blogs that fall into my hands (except number 7 – CodeColorer – since it is only meant for posting source code).

Enjoy… and contribute if you like!

See Consolas in action

And, why not!? I love the idea of making my source codes more readable by using some custom font (Consolas anyone?). Anyway, I started looking around for some fonts and was not pleased with any of them (well to be fair I did like DejaVu Sans Mono but not as much as Consolas, unfortunately).

Long story short, here is a very generous list of web-embeddable fonts which can be used freely in commercial scenarios, gathered by the awesome FontSquirrel.com website (an instant friend!). They even provide a @font-face generator interface that transforms a given font file (TTF or OTF) into the needed formats (mainly EOT for IE) and also generates the embedding CSS3 rules. You should really give it a go!

And if you are wondering how I could speak of embedding Consolas on my blog, knowing full-well the licensing policy of Microsoft in these matters, here is your answer: one can buy a license to embed Consolas on the web (I paid $17.50) from here.

You can see it in action on any of my posts that contains some source code. Like this one!

I know, I know! I also would have never believed that I would actually promote a Microsoft product on my blog, but the Consolas font deserves all the credit! It is simply unsurpassed as far as programming goes.

So, does anyone know any other web-embeddable fonts worth mentioning?

A while ago I posted an article describing how one could parse complete URIs in Bash using the sed program. Since then, I have realized that there is a better way to do it, a much better way: via Bash built-in pattern matching! 
Here are some benefits of this improvement:

• It no longer executes external programs (i.e. sed) for pattern matching. This translates to higher speed and lower memory and CPU usages, which means that you could use this parser for much more intense URI crunching.
• The new regular expressions are drastically simplified thanks to the ${BASH_REMATCH[*]} array that is able to hold more than 9 matched sub-expressions, unlike sed that can only work with single-digit escapes: \1-\9 (yuck!).
• The parsing algorithm is contained in a single Bash function, so no external file is needed to hold the regular expressions. This also means, obviously, that the pattern file is no longer loaded from disk on every execution (so HDD is saved as well).
• The generated variables are named identically to the first version, so you should be able to upgrade your scripts to this version with absolutely minimal effort.
• [Edit]
  No eval instruction is needed (unlike in the first version), further improving performance.

Functionality

• The function takes the URI as its parameter(s) and parses it. If the parsing was successful it returns 0 (zero), if not (bad URI) it will return 1 (or any other non-zero integer).
• The URI components are placed on global variables identical to those from the first article. There is an example script further down either way so you will get the picture.

Expected URI syntax

The actual parser code

This is the actual function. You can embed this straight into your Bash script and it will be available for execution. This method greatly simplifies the usage and even debugging of this parser.

A demonstration

Ok, I wouldn’t let you hanging there without an example, would I? This example basically uses the sample URI from the first version, and parses it accordingly. The difference is that this is an actual bash script so you can paste the parser function at the top (after #!/bin/bash) and just run it to see the parser in action.

Expected output

The demonstration above should output the following lines of text.
Check to see if it matches and give me some feedback with the results!

Conclusion

So, that’s it! A far better URI parser implemented straight in Bash. However, I think it could be taken even further with your help so please give it a try and write some feedback!

Enjoy!

Now, this is obviously only a humble start, so I would like to ask the community (yes, you!) to pitch in and help grow (and correct) this list of useful principles, so that, in time, it may actually become something used by folks around the web to better their coding practices. C’mon, this will be fun, fun, fun!

I. System operations on data, memory or file descriptors

1. Use variable-length-arrays or alloca() as much as possible instead of malloc() and friends. This kind of allocation is much faster and is allocated on the current function’s stack instead of the heap, so the allocated memory is automatically released when the containing function returns
.
   [see http://en.wikipedia.org/wiki/Variable-length_array]
   [see http://www.gnu.org/s/libc/manual/html_node/Advantages-of-Alloca.html]
2. Limit kernel syscalls to an absolute minimum. All operations on file handles (including stdin, stdout and stderr) are translated (at some point) into kernel syscalls! The same goes for malloc()/free() and many others. To this end you may consider reading the following points…
3. Use mmap()/munmap() and friends whenever possible to dramatically reduce the costs of highly fragmented file i/o operations and/or memory allocations
;
   [see http://en.wikipedia.org/wiki/Memory-mapped_file]
   
4. Use readv()/writev() and friends whenever possible instead of multiple read()/write() calls to make use of scatter/gather file operations, enabling you to combine multiple read or write operations in a single kernel syscall;
   [see http://en.wikipedia.org/wiki/Vectored_I/O]
5. Use splice()/sendfile() to replace malloc()/read()/write() when copying data between file descriptors. The splice()/sendfile() calls are able to copy a segment of a given input file to an output file or internet socket just like a read()/write() instruction pair would (using a memory buffer pre-allocated with malloc()). However, splice()/sendfile() operate entirely inside the kernel, thus many syscalls are saved and no pre-allocated buffer is needed the whole copy yielding much higher performance.

II. Source generation, preprocessing, compilation

1. Avoid passing expressions or function calls as parameters to macros. The macros will simply get expanded and the expression or function call that was given as a parameter will be executed at runtime in all places where they were expanded by the macro definition; this creates a lot of useless work and opens the possibility for undefined behaviour since an expression or function will be executed multiple times rather than only once as intended.

   1 2 3 4 5

   // Example #define MAX(a, b) ((a) > (b) ? (a) : (b)) // the macro int level = MAX(get_level(), 10);         // bad usage int current_level = get_level();          // good usage int level = MAX(current_level, 10);       // ...

2. Surround macro parameters with parantheses. This prevents syntax errors when expanding expression parameters.
3. Analyze the preprocessed code. Many times your macros can expand to unexpectedly large pieces of code making the final executable much larger; analyzing the preprocessed code cand enable you to translate such macros into function calls that get executed at run-time instead of expanded at compile-time.



   1 2	# GCC: preprocess only $ gcc -E source.c > source.txt

   

This optimization can make the final executable much smaller and faster to load (and sometimes even execute) into the system memory (ergo inlining code is *not* always a better choice)!

III. Miscellaneous

1. Use environment variables or program parameters instead of configuration files whenever possible. Having configuration files, while attractive, is costly in many cases. Your program must have additional code to load and parse the configurations in memory and it also has to access the disk to read the file(s) on every run. Using parameters or environment variables avoids all these problems elegantly while still allowing you to pass various settings to the program. If you are writing a performance-critical program, that has to be small and quick to load in memory, this may save you a lot of unneeded bloat.

That’s it for now!
And, don’t be a scrooge! Tell me what you think.

Enjoy!

This shows the memory usage over one week (the switch was made on the 29th as is obvious).

…and the swap for that week:

Next we have the disk-drives load as given by iostat over one week:

Here is the TCP traffic over the interface (bare in mind that when the switch occurred, there were far more clients served simultaneously):

And finally, here is a chart showing the number of active simultaneous connections handled by the streamer over the course of one day:

Before, with psstream setup with NGINX/php-fastcgi, this server was processing 150 or (maximum 200) connections simultaneously.

So these are some first impressions, and it is plain to see that this streamer works! However, I am still not satisfied, I will continue to tinker with it and squeeze as much performance out of one server as I can.

Til’later.

ATTENTION: Due to the fact that myself and my company are not yet satisfied with some issues regarding the legal terms of Loomiere/Stream, I think it is wise to take it (temporarily) offline until we resolve our concerns! Also some of the technical characteristics have been retracted.

Are you killing psstream?

Well, yes! I am sure that many of you already know about psstream (the PHP streaming extension I made a while back). Well, many things happened since then and I came to realize I could do better; a whole lot better actually. As of now the ‘psstream’ project is officially no longer developed (see below). It will remain on the website for some time to come for archiving purposes but that is it.

But wait, why did you do it?

For some time now I have be looking into improving the streaming mechanism for a large video-sharing project run by my company. PSStream was a first effort, and it did the job but soon ran into problems. None of our servers was able to properly stream more than 150 clients simultaneously and the resources were grossly wasted, hence, Loomiere/Stream.

Care to share some caracteristics?

Sure. Unlike psstream (in which the moov parsing code from code-shop.com was used) Loomiere/Stream is written from scratch in C (and a bit of Bash) and uses a very minimalistic and lightweight approach. A lot of time was invested into optimizing this streamer; here are some delicious features I consider worth mentioning:

• Secure and configurable
  • Loomiere/Streamer is, in fact, a set of micro-streamers (mp4, flv etc.) that come together into a fast and secure HTTP server using the ipsvd and runit server tools which are very well optimized and stable. This means that I (and you) don’t have to go through the huge pain of making this server secure over time. It also means that the server’s security rules will be very flexible and easy to manage.
• Optimized to the extreme
  • Each micro-streamer is an extremely small program written in C/dietlibc (the mp4 streamer has at most 21k depending on system) and completely statically linked (no external libraries are linked when executed); this means that these processes can be started very quickly and the CPU is not strained by linking.
  • The micro-streamers have an extremely low memory footprint (mp4 has uses about 120k, never more);
  • Loomiere/Stream uses a “One process per request” approach, which is not usually very recommended on a web-server (due to the fact that it a process is usually heavier than a thread), but in the case of video streaming processes are better (IMO) than thread-pools. This is because each connection transfers a large amount of data and takes a very long time to complete, and would therefore fill-up the thread-pool rather quickly and keep it occupied for the duration of the streaming regardless of server load; here the process start-up time becomes irrelevant, and anyway the Loomiere/Strem micro-streamers startup extremely fast.
  • As a side-note, I think it would be beneficial to test a thread-based approach in the case of smooth-streaming, since in that case the requests would be short enough to make it feasible. Or maybe I could use unlimited threads to stream (i.e. use threads but not a thread pool), that would be worth trying out, I think since it is possible that less context switching will occur. I’ll have to do some tests.
• Saves bandwidth (throttling)
  • The streamer tries to send only enough data to the player so that it will always be up-to-date. This is, of course, configurable via a set of parameters.
• Versatile
  • Loomiere/Stream is able to serve MP4 and FLV files (flv, f4v, mp4, m4v, mov…) as well as image files (jpg, png, gif, bmp). Images are supported because you may have images to be served along with your movies on the same machine so you won’t have to install a second webserver to access them (however, see the recommendations further down).
  • The starting seek point can be given in bytes or seconds (only MP4 supports seconds yet, but FLV players always seek by byte position anyway).
  • The whole streaming system is very configurable via ‘/etc/loomiere/stream.conf’, and in real-time (you don’t have to restart the server for new settings to become active).
• Open source
  • Loomiere/Stream is released under the GPLv3 license with an important amendment: one can never use this software for streaming pornographic or sexual content of any kind! Aside from that, you’re clear .

Did you test this anywhere?

The Loomiere/Stream server entered full production on Friday 29th of January, 2010 at 14:31 on all the content streaming servers of the peteava.ro website. Until now it has been running flawlessly.

A random check (while I’m writing this post) on one of the most strained servers yielded 653 simutaneous clients and a workload of: 8.81, 9.03, 22.47. The old streamer served at most 150 clients with extreme difficulty (the machine was almost frozen). The bandwidth usage dropped from the initial 1300MB/s to about 900 MB/s (approximate values), even though there were many more clients served in parallel.

Everything sounds good but do you think it could be better?

I know it can! Right now the server is in beta state! I can not guarantee anything at this moment (even unofficially) since I am still testing it myself under very heavy load. I will get back with more detailed statistics (and graphs), I promise!

Fantastic, where do I get it from?

ATTENTION: Due to the fact that me and my company are not yet satisfied with some issues regarding the legal terms of Loomiere/Stream, I think it is wise to take it (temporarily) offline until we resolve our concerns!

Giving any support?

I plan to provide commercial support for all Loomiere components (just the streamer for now), so yes! Soon I promise! How soon depends on how many people will use this server and request this specifically.

Until then, the support is basically open and free to anyone who wants to get involved. Practically, if you will have a problem with it and need my help, I’ll be here, I promise!

How do we use it?

The server: I don’t have a proper documentation written yet (that will be coming) but there is a “DETAILS” file in the source archive (above) that explains (in short) what you need to do to get it working. However you should have some knowledge about how ipsvd and runit work before tackling the streamer.

The clients: That’s easy, from any flash video player, you simply use the following url syntax:

http://yourserver.com/path/to/file.mp4 [ ?start=x.xxx [ &end=y.yyy ]]

Is there anything else?

I did not do any benchmarks against the Nginx and Lighttpd modules provided by the people at code-shop.com (they have renewed their modules as well). However, I hope my streamer will perform better considering that it was written with performance in mind, as a standalone server, thus not  being required to work with existing web-servers. Maybe someone could do some tests (maybe I’ll get around to it some day).

Well, for now, give me any feedback, bug reports or feature requests as a comment here.

Remember, streaming high-volume video is a very tricky business, there are many things involved! So if you have a single server that does other things beside the video streaming, I advise you to use your main web server as a front, redirecting streaming requests to the Loomiere/Stream server or simply stream your videos directly on a different port (like 81, or 8080) so that both servers can live on the same machine.

Although Loomiere/Stream is able to stream images, I only recommend you use it in case of large images and not for very small images that are heavily accessed (which is usually the case on the web). So if you can, use another web server for small images (with caching maybe?). I recommend: thttpd, nginx or fnord but any other (even Apache) would do just fine (depending on your requirements).

That’s it for now, I gotta sleep. Enjoy!

An even better method is:

A true one-liner!

Version 2

A better method is this:

This one can also be used from the command line.

Version 1

Many times I needed to apply certain permissions recursively on a given path but with different permissions on files than on directories (i.e. I want 0644 for files and 0744 for directories). This behaviour is not provided by the chmod tool so here is a simple and effective bash function to do just that:

I’m looking for a way to improve on this since it is quite costly for large directories: for each file or directory at least two programs are executed (file and chmod) which is not very efficient! For now, it gets the job done.
Enjoy!

Here is a command-line (bash) script that uses sed to split the segments of an URI into usable variables. It also validates the given URI since malformed strings produce the text “ERROR” which can be handled accordingly:

Notice the "uri.sed" file given to sed?
It is actually responsible for the URI parsing and it contains the required regular expression rules that will produce bash code out of the given URI which, in turn, when executed, will create our final variables to play with:

After the successful execution of this piece of code the following variables will exist in the running environment:

You could play around with it a bit and tell me if you find any problems. Right now it is only a first effort but it could be improved. Cheers!