lindsaar.net - Home

Mail gets some compliments!

2009-11-02T14:45:00Z

As you know, I have put out the new mail gem… here are some of the things people have said about it:

“Wow, I find this library very exciting! Thanks for sharing.” – James Edward Gray II

“Nice work. Thanks Mikel.” – Gregory Brown

“I just finished parsing the whole Enron set (2.3gb) and the Trec 2005 set (0.5gb) again. The only issues that caused mail parser failures after the few commits on my fork are all related to bad header values. Everything else was clear sailing. I didn’t verify the correctness of the parser, just the success/failure, but still that’s ~500k emails that went through fine.” – Jim

“This is gorgeous. If you’ve got this pile of total awesome waiting for release, I’d rather jump on it.” – Eric Hodel

And only after a few short days, we have 84 people watching mail on github, compared to the few dozen on TMail.

Thanks for all your comments, keep them and your patches coming!

Mikel

Rails Unit Tests: uninitialized constant error

2009-10-29T01:05:00Z

Rails doesn’t play well with test-unit 2.x… if you try you get something like: Test::Unit::TestResult::TestResultFailureSupport on from /Library/Ruby/Gems/1.8/gems/test-unit-2.0.5/lib/test/unit/testresult.rb:28

If you are getting something like this:

mikel@baci.local ~/rails_programs/rails/actionmailer
 $ rake test
(in /Users/mikel/rails_programs/rails/actionmailer)
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby -I"lib:test" "/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake/rake_test_loader.rb" "test/adv_attr_test.rb" "test/asset_host_test.rb" "test/delivery_method_test.rb" "test/mail_helper_test.rb" "test/mail_layout_test.rb" "test/mail_render_test.rb" "test/mail_service_test.rb" "test/quoting_test.rb" "test/test_helper_test.rb" "test/tmail_test.rb" "test/url_test.rb" 
DEPRECATION WARNING: ActiveSupport::DeprecatedCallbacks has been deprecated in favor of ActiveSupport::Callbacks. (called from included at /Users/mikel/rails_programs/rails/actionmailer/lib/../../actionpack/lib/../../activesupport/lib/active_support/testing/setup_and_teardown.rb:7)
/Users/mikel/rails_programs/rails/actionmailer/lib/../../actionpack/lib/../../activesupport/lib/active_support/dependencies.rb:116:in `const_missing': uninitialized constant Test::Unit::TestResult::TestResultFailureSupport (NameError)
        from /Library/Ruby/Gems/1.8/gems/test-unit-2.0.5/lib/test/unit/testresult.rb:28
        from /Library/Ruby/Site/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
        from /Library/Ruby/Site/1.8/rubygems/custom_require.rb:31:in `require'
        from /Users/mikel/rails_programs/rails/actionmailer/lib/../../actionpack/lib/../../activesupport/lib/active_support/dependencies.rb:167:in `require'
        from /Users/mikel/rails_programs/rails/actionmailer/lib/../../actionpack/lib/../../activesupport/lib/active_support/dependencies.rb:537:in `new_constants_in'
        from /Users/mikel/rails_programs/rails/actionmailer/lib/../../actionpack/lib/../../activesupport/lib/active_support/dependencies.rb:167:in `require'
        from /Library/Ruby/Gems/1.8/gems/test-unit-2.0.5/lib/test/unit/ui/testrunnermediator.rb:9

Then you need to downgrade from test-unit 2.x to test-unit 1.2.3 or so.

12	# gem uninstall test-unit# gem install test-unit -v 1.2.3

Enjoy

New Mail gem released

2009-10-29T00:43:00Z

OK… birthday time. Mail is now a gem!

So, it is hosted on gemcutter

It is uploaded to github

Let me know how it runs!

Mikel

Mail and Bounced Emails

2009-09-19T14:31:00Z

You might have to deal with bounced emails in your application. The new mail library makes this a no brainer…

I like code more than words… so:

require 'mail'

@mail = Mail.read('/path/to/bounce_message.eml')

# You can also do Mail.new('String of email message')

@mail.bounced?
#=> true
@mail.final_recipient
#=> rfc822;mikel@dont.exist.com
@mail.action
#=> failed
@mail.error_status
#=> 5.5.0
@mail.diagnostic_code  
#=> smtp;550 Requested action not taken: mailbox unavailable 
@mail.retryable?
#=> false

You can get mail from my GitHub repository

Once we finish a few more points in the library, I’ll release a gem.

Mikel

Mail, TMail, The Future of Ruby Email Handling

2009-09-18T04:22:00Z

You may know I maintain the TMail library. Well, recently I’ve been working on the next version.

TMail was getting hard to maintain and monkey patch. It’s implementation does not take into account many little things that create a problem when trying to get Ruby 1.9 compatibility working.

Anyway, I started working on Mail… code speaks louder than words… so Mail can do the following right now:

Mail.defaults do
  smtp '127.0.0.1'
end

Mail.deliver do
  to 'nicolas@test.lindsaar.net.au'
  from 'Mikel Lindsaar <mikel@test.lindsaar.net.au>'
  subject 'First multipart email sent with Mail'
  text_part do
    body 'Here is the attachment you wanted'
  end
  html_part do
    content_type 'text/html; charset=UTF-8'
    body '<h1>Funky Title</h1><p>Here it is</p>'
  end
  add_file :filename => '/path/to/myfile.pdf'
end

And you just sent a multipart text and html email with an attachment!

Mail is my attempt to just HANDLE the problem of Ruby Email handling.

I developed it from the ground up with complete spec coverage… I am not quite at 100%... but very close. It is also a completely object oriented design and pure ruby too!

Mail tries it’s darn hardest not to crash. In fact it already checks every email in the TMail test suite and doesn’t crash once on parsing any of the emails in there.

In any case, I now have pretty much all the basic email and mime and attachment handlings working. Next up is handling multiple character sets in the header and body.

You are welcome to check it out from my GitHub account

It’s version 0.1 right now… but expect to see the multi character set support soon.

Forks and patches welcome!

Mikel

Custom Music on Hold for Asterisk

2009-07-04T16:22:00Z

I needed to change the music on hold for my Asterisk servers…

First, find your favorite music files in basically any format, get them somewhere where you can access them.

Then make sure you have ffmpeg installed, for mac users you can just do “port install ffmpeg” which can take a while as it has to install all the audio and video format libraries.

Go into the folder where you put the music files, and run the following (assuming you are on a UNIX client)

for f in `ls *.mp3` ; do FILE=$(basename $f .mp3) ;
  ffmpeg -i $FILE.mp3 -ar 8000 -ac 1 -ab 64 $FILE.wav \
  -ar 8000 -ac 1 -ab 64 -f mulaw $FILE.pcm \
  -map 0:0 -map 0:0 ;
done

This will change your “MP4” files into a PCM (ulaw) and WAV file format.

Then move the existing music on hold directory in asterisk to another name:

12	# mv /var/lib/asterisk/moh /var/lib/asterisk/moh-original# mkdir /var/lib/asterisk/moh

Then install your new music files

12	# cp ~/.pcm /var/lib/asterisk/moh/# cp ~/.wav /var/lib/asterisk/moh/

Then reload Asterisk and you should be good to go!

Mikel

Always getting an invalid authenticity token error

2009-05-03T14:59:00Z

I had a Ruby on Rails app giving a bunch of invalid authenticity token errors. I spent a while hunting down the solution until I found this write up. Very useful.

This write up was by Peter De Berdt on one of the Ruby mailing lists, I am putting it here more for my own future (maybe) reference as it is a very good solution for the situation I was running into. The entire thread is at the ruby forum site

If you have a situation where you are getting invalid authenticity tokens and are doing strange things with forms on your website with file uploads etc, and can’t for some reason just use the authenticity form helper, then this solution worked for me and should work for you too.

The solution is pretty simple to be honest:

In your view layout file, add this to the <header> section:

123	<script type="text/javascript" charset="utf-8"> window._token = '<%= form_authenticity_token -%>';</script>

In application.js, add the following:

Ajax.Base.prototype.initialize = Ajax.Base.prototype.initialize.wrap(
   function(p, options){
     p(options);
     this.options.parameters = this.options.parameters || {};
     this.options.parameters.authenticity_token = window._token || '';
   }
);

It will automatically add the authenticity token to ALL ajax requests, even those you invoke from custom code (graceful degrading and/or even delegated events for example).

A similar solution for those swapping out Prototype with JQuery has been posted here

As for file uploaders, a normal field within a form (multipart=true) will be sent as part of the form (and isn’t an ajax request in the first place) and shouldn’t be a problem. If you are using ANY other “ajax” uploader, there’s more to it. I already posted several times on how to get SWFUpload to play nicely with Rails, an overview with links to the appropriate posts can be found here.

Windows ipconfig does not show anything

2009-01-06T13:30:00Z

I had a Windows XP system (service pack 2) that would only return “Windows IP configuration” and then nothing, no indication of network disconnection nothing, well, there’s a good fix for it.

The system wouldn’t pickup an IP address from DHCP, it was like it was just confused.

You can do the following at a CMD prompt to reset the network configurations:

12	C:\> netsh winsock reset catalogC:\> netsh int ip reset reset.log

The first line resets all the WINSOCK entries to their defaults, the second line resets all the TCP/IP entries to their defaults.

The first one will ask you to reboot, don’t. Do both commands and then reboot your system.

Once I did this, the computer got it’s DHCP address again quite happily.

blogLater

Mikel

FreeBSD rc scripts

2009-01-05T20:18:00Z

I’ve always wanted a concise practical guide to rc scripting in FreeBSD, well, here it is

How to monitor a logged in professional

2008-11-24T06:37:00Z

Sometimes you will need to get a professional to log into your system to help you out. You should watch what they do, and this is the easiest way I know.

I needed to get someone from a hardware company to log into one of our phone servers to help debug a problem, but I wasn’t just about to give them full carte blanche access to my server. Especially as it is behind a firewall.

I trust these guys somewhat, I mean, if they screw up, no more customer and hello to bad PR… but at the same time, you should always watch.

So, I went hunting around, and Screen is the best answer.

Get them to log into your system and type:

$ screen -S support

Then, from your shell:

$ screen -x support

And bam, you are both sharing the same screen.

Nice and simple.

Of course, this isn’t fool proof, and it only really lets you follow what the other guy is doing. But it is a hell of a lot better than sitting there blind listening on the phone to a few hundred key strokes wondering ‘what is he up to?’

blogLater

Mikel

TMail Moves to GIT

2008-11-22T16:35:00Z

Social networking here we come!

I have moved TMail onto GIT.

You can see it here

Please fork / patch / follow / help out :)

Mikel

Funny...

2008-11-20T22:39:00Z

Some people think that rails is getting bloated… but through the hard efforts of core committers, the fluff and fat is getting sliced and trimmed down like a Döner kebab spit at a turkish festival…

And here is the perfect example seen on Rails-Contrib on freenode last night:

GitHub142: rails: David Heinemeier Hansson 2-2-stable
SHA1-9d8cc60

[19:07] GitHub142: Reduced the number of literal aliases to the
range that has actually seen personal use. With the massive
savings in overhead, I was able to fit Array#fourty_two

Here’s the commit if you are interested…

blogLater

Mikel

How to reset a sequence with PostgreSQL

2008-11-02T19:47:00Z

I would think that

12	ALTER SEQUENCE sequence_name RESTART WITH (SELECT max(id) FROM table_name);

would work, but it doesn’t. Use:

SELECT SETVAL('sequence_name', (SELECT MAX(id) FROM table_name) + 1);

instead and you will be a lot more happier.

blogLater

Mikel

OpenBSD RAID and Temp Sensors on HP Proliant DL 360 and 380 Series

2008-11-01T20:17:00Z

I have a bunch of HP DL 360 and DL 380 servers that I run OpenBSD on as gateways. This is how to monitor their RAID drive status and temperature status…

What you need to do is enable the IPMI driver.

If you read the man page, the IPMI driver provides an API like interface to the hardware it is running on.

This means you can query sysctl to get data like this:

123	$ sysctl hw.sensorshw.sensors.acpitz0.temp0=8.35 degC (zone temperature)hw.sensors.ciss0.drive0=online (sd0), OK

If you then yank one of a mirror set you get:

12	$ sysctl hw.sensors.ciss0.drive0hw.sensors.ciss0.drive0=degraded (sd0), WARNING

If you then put a new drive back into the mirror set you get:

12	$ sysctl hw.sensors.ciss0.drive0hw.sensors.ciss0.drive0=rebuilding (sd0), WARNING

Once the system has rebuilt, you get:

12	$ sysctl hw.sensors.ciss0.drive0hw.sensors.ciss0.drive0=online (sd0), OK

Which is very useful and basic data needed for monitoring systems like Nagios.

There is a catch though, the IPMI driver is not enabled by default in OpenBSD GENERIC (at least at 4.4). To enable it, you have two options.

At boot time, you can enable it by doing:

>> OpenBSD BOOT 640/31744 k [1.29]
use ? for file list, or carriage return for defaults
use hd(1,a)/bsd to boot sd0 when sd0 is also installed
Boot: -c
Booting...
=======snip=======
User Kernel Config
UKC> enable ipmi
441 ipmi0 enabled
UKC> quit

This will then boot normally and if everything worked, you’ll have an operational system that you can type the above commands in to query the status of your RAID set.

If it won’t start, then reboot without doing anything and it will go back to the way it was before, then contact openbsd misc with a copy of your dmesg.

If it all worked, you’ll probably want to enable this feature permanently, to do this, use the config(8) utility like so:

# config -u -o /bsd.new -e /bsd
(tells you about the kernel)
Enter 'help' for information
ukc> enable ipmi
441 ipmi0 enabled
ukc> quit

This then will produce an output file /bsd.new which is your new kernel.

To install the new kernel, do something like this:

# cp /bsd /bsd-original && cp -f /bsd.new /bsd

And then you can reboot to an IPMI enabled system.

Enjoy.

Terminator - Timeout without Mercy

2008-09-11T10:40:00Z

If you have been following my posts on Ruby-Talk and Ruby on Rails and even RSpec mailing list (and who wouldn’t?! I mean, aside from my mother) then you would have noticed I have been banging my head against a brick wall on the subject of System calls not being handled by the Timeout libraries in Ruby…

I’ve done a lot of work with this. I mean a lot.

So I thought I would share.

It all starts with a basic requirement I have, which is to replicate a data set from one side of the planet to the other. This involves logging into a database server remotely using ActiveRecord across some encrypted VPNs, querying some tables, and then inside of a transaction, replicating down certain data on certain rows.

The solution I have works well, I can replicate about 2-5 rows per second across the link which is more than adequate (the dataset changes upto about 100 rows per minute). The problem is that occasionally, ActiveRecord’s link to the remote database will hit a snag, then Ruby will wait, and wait, and wait to Timeout…. This can take hours, and in fact, sometimes, it never does timeout, leaving a zombie process.

To avoid multiple replication problems, I have lock files which prevent further copies of the replicator firing up if it detects that one is already running.

So, in the end, I get stuck with several replicators “running”, all hung, all waiting for a link to come back that never will, and no replication happening. In the mean time, the rows are still changing and I am getting backlogged. Leave it for a day and you are 80,000 rows backlogged and people start asking questions and I start looking for old travel tickets that I had ‘forgotten about’.

But why does ruby do this?

Shouldn’t the Timeout library protect us from such evilness?

Well, yes and no. The problem is because ruby uses Green threads. Green threads mean that there is only one real Ruby process running on the server, and it “internally” makes other “threads” that it schedules within the main ruby process’ kernel time. Green threads are very efficient (they save a lot of context switching) but the problem is that if you get a System call that ‘blocks’, then the kernel will stop servicing your main ruby thread until your system call is complete, and THIS means that the little ‘Timeout.timeout(5) { my code }’ block you put in, will never get called…

And then you get a hang… forever.

I tried a couple of handlings, the first was the SystemTimer library by Philippe Hanrigou. This entry actually has a really good and simple example of what Green threads and blocking system calls mean. If you want to brush up, it is a good read.

But SystemTimer didn’t work. It uses alarm signals which could have some problems. But in my case, it didn’t work and I still got never ending timeouts.

So I asked again on the Ruby-Talk mailing list and Ara T. Howard came back with a quick script that created little homicidal external ruby processes to kill the ruby process I was in if it didn’t “make it” in time.

The approach was actually very clever, and it works!

What he did was go “We can’t be sure that the existing Ruby process will be able to nuke itself, so instead, lets start up another Ruby instance, running on it’s own, that has the PIDs of the ruby instance that spawned it, and if the time runs out, do a system based kill TERM on the pids.”

You could say that our little ruby processes have a license to kill. Literally.

So I implemented his idea, found a couple of problems, gave back some ideas, and he coded, I spec’d, tested and described and we released a brand new gem…

The Terminator.

You can get it with;

gem install terminator

Or grab the source code.

And how do you use it? Simple:

require 'terminator'
Terminator.terminate 1 do
  sleep 2
  puts "I'll never print"
end

This will never print because the terminator times out after 1 second, which is before the sleep of 2 seconds inside the block. This will raise a Terminator::Error, which you could catch and try again if you want.

This will always work. It is because we are starting a separate process of Ruby (which has some minor overhead) that waits the specified number of seconds and then just simple does a system kernel TERM on our misbehaving process.

Why should you use it?

Well, if you are making ANY calls to external web services, external databases, OpenID, Youtube, Google Maps… anything, then you should have a fail fast policy in place and time out rapidly if these fail. As these are external system calls, they will most likely not be caught by Ruby’s timeout.rb library… and that means that your application will just hang, waiting for the call that never comes back.

It is much better to go “Ok, 2 seconds are up, no response, let’s tell the user to try again in a minute” and render an appropriate message, than have the user frustratingly whack the reload button a few hundred more times wondering why your application is not responding.

So yes, I think you should use Terminator.

Besides, how many other gems do you know that have the method ‘plot_to_kill’ ?

blogLater

Mikel