Fayland and Programming

mojo tt2 defaults layout patch

2013-04-27T00:00:00+08:00

You wrote code with open source like CPAN. You meet problems with them. You investigate, write tests then provide patches. Your patches are accepeted.

I really enjoy the way it moves.

Here is another patch for Mojolicious::Plugin::TtRenderer.

the issue is that $self->defaults(layout => 'wrapper'); is always dropping the [% content %] without reason. and with debug I see the content is really in $c->stash->{'mojo.content'}. so a simple patch is that just $c->stash->{content} ||= $c->stash->{'mojo.content'}->{content};

recently I’m writing another new app with bitcoin <-> litecoin and other virtual coins exchange. and I have a wrapper.html.tt for the whole site. but I do not want to use it in the email TT2 render.

if I put WRAPPER => 'layouts/wrapper.html.tt', in the TT2 options, it will also apply to the mail render. a better approach is to use defaults layout. then in email render, set the layout as empty or actually we need use partial => 1.

MainApp.pm

$self->plugin('tt_renderer');
$self->defaults(layout => 'wrapper');
$self->renderer->default_handler( 'tt' );

OtherController.pm

my $body = $c->render(
    template => 'emails/forgot_pass', format => 'mail',
    other_stash => $var,
    partial => 1
);

partial => 1 will set layout as null. so it will give us what we want.

Have fun!

Mojolicious TT2 patch

2013-04-12T00:00:00+08:00

I am kindy busy with adding new features to my job site

but it’s very boring to restart the app on template updates. TT2 should be able to pick up the changes automatically. it works under Dancer or Catalyst, but not Mojolicious.

I thought I should spend some time on investigating why before writing more stuff. and after a while, I have a patch for Mojolicious::Plugin::TtRenderer.

it dues to the Provider in TtRender always return 1 instead of mtime on _template_modified.

plicease accepts it quite fast with slight changes but I’m waiting for a CPAN release. :-)

here is few changes on http://findmjob.com/ recently.

tag view is splitted with job and freelance.
Location link is added like: San Francisco, CA

Happy hacking!

Dancer to Mojolicious

2013-04-10T00:00:00+08:00

The progress went smoothly because all of them are just Perl code.

old Dancer code: FindmJob::WWW

new Mojolicious code: FindmJob::WWW, FindmJob::WWW::Root and others

Few notes

hooks

from before_template_render to before_render

Dancer `forward` VS Mojolicious `before_dispatch`

we can modify the req->url->path in before_dispatch with assigning stash.

# feed.(rss|atom) and /p.2/
$self->hook( before_dispatch => sub {
    my $self = shift;

    my $p = $self->req->url->path;
    if ($p =~ s{/feed\.(rss|atom)$}{}) {
        $self->stash('is_feed' => $1);
    }
    if ($p =~ s{/p\.(\d+)(/|$)}{$2}) {
        $self->stash('page' => $1);
    }
    $self->req->url->path($p);
});

supervise

Dancer:

plackup -E production -s Starman --workers=3 -l /tmp/findmjob.sock -a /findmjob.com/www/bin/app.pl

Mojolicious:

hypnotoad -f /findmjob.com/bin/www.pl

Note -f is important here, or supervise will keep restarting the script.

above plackup using sock and hypnotoad use port. so we need update the ngnix config a bit.

and it looks like hypnotoad is working better than plackup Starman

Template name conversation

there is no rule in Dancer for the Template name. but in Mojolicious, we have to do it with $name.$format.$handler

in this case we have to rename index.tt2 to index.html.tt

Conclusion

well, I’m not saying that Dancer is worse than Mojolicous or any other words. both of them are great.

but Dancer is a bit messy with Dancer and Dancer2. actually I love Moo a lot, but I don’t want to spend time on upgrading Dancer to Dancer2 (there seems some more difference than expected.)

in the other hand, Mojolicious looks amazing, sri improves it every week and I admire/trust his professional knowledge.

Have fun.

Two new CPAN modules

2013-03-30T00:00:00+08:00

I got two new CPAN modules uploaded today.

WWW::SpinnerChief

I wrote this based on the Python code

it’s pretty good that you know how to read the Python.

Business::PayPoint

aka tips to write SOAP module in Perl.

the PHP SOAP lib is pretty good while the Perl one is a little suck.

so usually if I can’t write it at the first try in Perl, I would use PHP to find the sending request/response

$client = new SoapClient('https://www.example.com/Test?wsdl', array(
    'trace' => 1,
) );
$client->__call('TestAction', $params);
echo "RESPONSE:\n" . $client->__getLastResponse() . "\n";
echo "REQUEST HEADER:\n" . $client->__getLastRequestHeaders() . "\n";
echo "REQUEST:\n" . $client->__getLastRequest() . "\n";

After I got the correct sample request, I’ll try to use Perl library to send the same XML with trace on.

use SOAP::Lite +trace => 'all';

well, you even can’t write correct one sometimes (maybe I am a little stupid).

but I have a final trick for it. use XML::Write to genereate the request XML. then use

my $som = $soap->call($method, SOAP::Data->type('xml' => $xml));

you won’t be wrong in this case. :)

Have fun.

Net::Telnet with bitflu

2013-03-18T00:00:00+08:00

Even Net::Telnet is quite old, it’s still very powerful and simple to use.

use strict;
use warnings;
use Net::Telnet ();
use Data::Dumper;

my $host = '0';
my $port = 4001;

my $telnet = Net::Telnet->new();
unless ($telnet->open( Host => $host, Port => $port, Timeout => 30 )) {
    die "Can't connect to $host:$port\n";
}

$telnet->waitfor('/bitflu> /');

my @messages = map { chomp; $_ } $telnet->cmd(String => 'ls');
pop @messages if $messages[-1] eq 'bitflu';
print Dumper(\@messages);

$telnet->close();

there are some tips:

remove ANSI color

foreach my $msg (@messages) {
    # if you do not remove this, your regex with /^\[/ may break
    $msg =~ s/\e\[[\d;]*[a-zA-Z]//g;
}

Change Window Size

if you use term to do bitflu> ls, you’ll see full torrent name. but with the code above, you can only see few chars. here is the note to change the Window Size:

my $telnet = Net::Telnet->new();

$telnet->option_callback( sub { return; } );
$telnet->option_accept(Do => 31);

unless ($telnet->open( Host => $host, Port => $port, Timeout => 60 )) {
    die "Can't connect to $host:$port\n";
}
$telnet->waitfor('/bitflu> /');

## copied from http://blog.webdir.bg/perl-apache-realtime-output-from-script/
## Many Thanks!
$telnet->telnetmode(0);
$telnet->put(pack("C9",
                  255,                  # TELNET_IAC
                  250,                  # TELNET_SB
                  31, 0, 200, 0, 0,     # TELOPT_NAWS
                  255,                  # TELNET_IAC
                  240));                # TELNET_SE
$telnet->telnetmode(1);

Learn Python Note 1

2013-03-11T00:00:00+08:00

I’m starting learning Python a bit.

argparse

if you have args setup in different modules, like you want setup --log on logging, setup --tor --skip-tor on requesting, the combination of add_help=False and parse_known_args is pretty cool.

import argparse
parent_argparse = argparse.ArgumentParser(add_help=False)
parent_argparse.add_argument('--log', action='store', default='ERROR', dest='log', help='logging level: DEBUG, INFO, WARNING, ERROR, CRITICAL')
_args, _args_unknown = parent_argparse.parse_known_args()

afterwards, in real place, do

parser = argparse.ArgumentParser(description='Some real CLI', parents=[parent_argparse])

init.py and as

actually it’s a very cool feature. you can put some common code for all sub-modules in __init__.py like config code (to avoid write another file)

import ConfigParser
try:
    cfg = ConfigParser.ConfigParser()
    cfg.read(['/somwhere/conf/project.ini', '/somwhere/conf/project_local.ini'])
except ConfigParser.Error, e:
    logger.critical('Cannot read configuration file, reason [%s]' % e)

and as is pretty amazing.

from modulename import cfg as project_config

BeautifulSoup is another HTML::TreeBuilder

BeautifulSoup works like Perl HTML::TreeBuilder.

from bs4 import BeautifulSoup
soup = BeautifulSoup(text)

for tag in soup.find_all('span', attrs={'id': re.compile('^ctl00_contentPageContent_lbl(.*?)Value$')}):
    id = tag['id']
    id = id.replace('ctl00_contentPageContent_lbl', '').replace('Value', '')
    data[id] = tag.get_text()

something I do like in Python

def with args and default values. (Perl sub was developed too many years ago.)

you don’t need type so many {} or ().

something I do not like in Python

string interpolation is quite inconvenience. there are %, format and Template. but I really miss the $ with Perl.

I miss Perl regexp a lot. the re.compile, re.sub, re.I is too much typing.

Disclaim

I’m still a newbie. so parden me if anything is wrong. I’ll write more when I learn more. and I admit it’s pretty fun to learn Python.

Blogger local file > Jekyll migrator

2013-03-07T00:00:00+08:00

I have an old Blogger before I move to MT5, it contains my posts from 2006 to 2009. some are quite old, broken, useless and even not worth reading. but some are still very useful and it’s a part of my memory and life. (Sorry that comments are dropped that I can’t import it into Disqus.)

so I wrote a simple Perl script to convert the local HTML files into Jekyll here. the source code can be found in github.

it’s quite simple and maybe just fit for my need. but feel free to change it if you have same demand.

Have fun.

add alert note for old posts in Jekyll

2013-03-07T00:00:00+08:00

add alert note for old posts

For Jekyll Bootstrap, it’s quite simple as we can do something like below:

edit _includes/themes/twitter/post.html, add

{% assign page_year = page.date | date: "%Y" %}
{% if page_year < '2010' %}
<div class='alert alert-info'>This post may be outdated due to it was written on {{ page_year }}. The links may be broken. The code may be not working anymore. Leave comments if needed.</div>
{% endif %}

20 items in rss/atom

I have another if in the {% for post in site.posts %} for rss.xml and atom.xml

{% if forloop.index < 20 %}
...
{% endif %}

paginate

edit _config.yml

paginate: 10
paginate_path: 'page/:num'

Have fun.

MT > Jekyll perl migrator

2013-03-06T00:00:00+08:00

I bought a new domain fayland.me.

I picked Jekyll as the blog engine.

I wrote a simple Perl script to convert my old Movable Type 5 blogs to Jekyll Bootstrap.

Github code

the difference between this one and the Jekyll::MT is that it supports tags and it’s for Jekyll Bootstrap.

at last, using

rsync -arv --delete _site fay:/srv/www/fayland.me/

to rsync the site.

Have fun.

Net-Amazon-DynamoDB

2012-11-22T00:00:00+08:00

I have been playing with Net::Amazon::DynamoDB a lot recently. this article is not to comment on that service. it's just a few notes for Perl guys who're using it.

1. Moo based instead of Moose.

well, if you're not using Moose in your system, Moose may be too much for your speed. I am maintain a new branch on that which converted Moose to Moo. changes can be found at https://github.com/fayland/Net-Amazon-DynamoDB/commit/0b3b9a493e92c85461353ef1fc1da422e4d0bb48 and I'll update when the Moose version is updated.

2. custom Cache module supports.

I removed the isa => 'Cache' in the Moo version, instead, I'm using

has cache => ( isa => sub {

die "thaw/freeze/remove must be supported in cache" unless $_[0]->can('thaw') and $_[0]->can('freeze') and $_[0]->can('remove')

}, is => 'rw', predicate => 'has_cache' );

so that we can use custom layer like

package My::Dummy::Cache::FastMmap;

use Moo;
use Cache::FastMmap;

has 'cache' => (is => 'lazy');
sub _build_cache { Cache::FastMmap->new(share_file => '/tmp/mycache_fastmmap', unlink_on_exit => 0) }

# Cache::FastMmap do not have thaw/freeze sub
sub thaw {
my $self = shift;
$self->cache->get(@_);
}
sub freeze {
my $self = shift;
$self->cache->set(@_);
}
sub remove {
my $self = shift;
$self->cache->remove(@_);
}

1;

I have fixed some bugs for the cache system like not set while return is undef, delete on batch_write etc. but it is still not perfect b/c it's not supported inside batch_get_items yet. I may write patch for it later.

3. set is not array.

well, for SS or NS. it's set. it's not array. it means there is not an order inside.

if you want something like order, we can join it as string and save it. after get, split back.

Thanks.

reCaptcha and lightbox and ajaxPost

2012-09-14T00:00:00+08:00

sometimes, you want to do lightbox for Email Us or Contact Us page.

in this case, you don't want to include the recaptcha js in every page, you want to include it in the popup lightbox but here we have an issue that $(document).ready would be called while the google recaptcha js is not loaded. b/c document ready is just checking the original page instead of the lightbox. in this case, here is a common solution for every js loading (wait until that js loaded.)

<script type="text/javascript" src="https://www.google.com/recaptcha/api/js/recaptcha_ajax.js"></script>

<script type="text/javascript">
$(document).ready(function(){
var reCaptcha_timer;
reCaptcha_timer = setInterval(function(){
if (typeof(Recaptcha) != 'undefined') {
clearInterval(reCaptcha_timer);
CreateReCaptcha();
}
}, 50);
});
function CreateReCaptcha() {
Recaptcha.create("public_key_blabla", 'captcha-placeholder', {
theme: "white",
callback: Recaptcha.focus_response_field
});
}
</script>

see we have a tricky that keep check if Recaptcha is inited (which will be done when recatpcha_ajax.js is loaded).

and only after it's loaded, we clear the check, and create the catpcha.

the trick works pretty good. but here is another issue, when the lightbox is loaded and user submits the form, and if some elements are wrong or captcha is wrong, that we need show captcha once again. it will be broken. b/c Recaptcha js variable is already defined, and Recaptcha.create will not working fine b/c it has something stored for previous captcha.

in this case, another tricky is much more simpler.

<script type="text/javascript">
// so that we can reload it
if (typeof(Recaptcha) != 'undefined') Recaptcha = undefined;
</script>
<script type="text/javascript" src="https://www.google.com/recaptcha/api/js/recaptcha_ajax.js"></script>

before load the js, we reset it so it works like it's the first time we are trying to create reCaptcha.

those issue only happens on ajax lightbox load reCaptcha and ajaxPost the reCaptcha form.

stupid but works.

Thanks.

MacOSx homebrew postgresql issue note

2012-09-12T00:00:00+08:00

after brew install postgresql

then run "initdb /usr/local/var/postgres -E utf8", may got:

selecting default shared_buffers ... 400kB
creating configuration files ... ok
creating template1 database in /usr/local/var/postgres/base/1 ... FATAL: could not create shared memory segment: Cannot allocate memory
DETAIL: Failed system call was shmget(key=1, size=2138112, 03600).
HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded available memory or swap space, or exceeded your kernel's SHMALL parameter. You can either reduce the request size or reconfigure the kernel with larger SHMALL. To reduce the request size (currently 2138112 bytes), reduce PostgreSQL's shared memory usage, perhaps by reducing shared_buffers or max_connections.
The PostgreSQL documentation contains more information about shared memory configuration.
child process exited with exit code 1
initdb: removing data directory "/usr/local/var/postgres"

fixes as:

~ sudo sysctl -w kern.sysv.shmall=1024000

kern.sysv.shmall: 8192 -> 1024000

~ initdb /usr/local/var/postgres -E utf8

then everything will move smoothly.

just for note.

MongoDB and Perl

2012-04-26T00:00:00+08:00

it's really a pain to work with MongoDB in Perl. Perl has no 'type' so when you get INT value from DBI, it might be really "1" instead of int 1.

I got some data dumped from MySQL to MongoDB and found all the 'time' field is wrapped as "1335350669" instead of 1335350669.

and when you insert it from code like { time => time() }, you really have 1335350670 instead of "1335350670". it breaks the sort. it breaks the deletion. it breaks everything.

blabla, to fix that, we just need update it in mongodb like below.

PRIMARY> db.jobs.find().forEach(
function(job) {
job.time = parseInt(job.time);
db.jobs.save(job);
});

but it's still a pain. I didn't check MongoDBx::Class or Mongoose yet, but Moose's type should be able to fix it I think.

Thanks

api and android app

2012-04-14T00:00:00+08:00

I'm a fan of Android. even I like my iPad and MBP very much too.

I'd like to write an App for Android. and here is all the story.

first of all, I need write an API which supports JSONP. since it would be very simple, I don't want to mix it up with Dancer. at last, I wrote something based on webmachine-perl. I like the idea behind the webmachine. the chain design looks pretty nice. even I don't use that too much.

the code is at

https://github.com/fayland/findmjob.com/blob/master/api/app.psgi

https://github.com/fayland/findmjob.com/blob/master/api/lib/FindmJob/Resource.pm

and live demo as

http://api.findmjob.com/search?callback=jQuery171016797792096622288_1334322201618&q=perl&loc=&_=1334322214418

after API is done. now comes the android app part. Sorry that I don't know much about Java. and I failed to download appmobi/jqmobi (the download never ends here). so at last I picked up phonegap.

the progress went pretty smooth. mixed with jQuery Mobile, I have it out after few hours.

you can download it for fun from http://static.findmjob.com/FindmJob.apk

it's pretty simple, just with one JSONP request and not much different than the demo.

even there, I'd like to share the code with you:

https://github.com/fayland/findmjob.com/blob/master/mobile/android/assets/www/index.html

https://github.com/fayland/findmjob.com/blob/master/mobile/android/assets/www/mobile.coffee

it's very cool and I'm excited! (badly Google costs 25$ for submitting it and I don't want to do it for now)

Thanks.

sphinx search with varchar primary key

2012-04-08T00:00:00+08:00

usually when you index the mysql data into sphinx, you'll use id int/bigint for the primary key. but it's broken for me on http://findmjob.com/, we use uuid everywhere for the primary key.

here is the solution for it. use @id := @id + 1 for the indexer, and use sql_field_string to get the real id when matched. sample code below:

sql_query_pre = SET NAMES utf8
sql_query_pre = SET @id := 1;
sql_query = \
SELECT @id := @id + 1 AS tid, id, title, description, location, contact, inserted_at FROM job ORDER BY inserted_at DESC LIMIT 10000

sql_field_string = id

it requires latest sphinx to support sql_field_string. and the latest CPAN module too.

and Perl code for it will be normal like before. and instead you use ->{doc}, you need use the attribtues ->{id} like

my @jobids = map { $_->{id} } @{$ret->{matches}};

so now we have sphinx search supports in my new site, eg: http://findmjob.com/search/Perl.html?q=Perl

Thanks.

DBIx::Class with Moose has

2012-04-04T00:00:00+08:00

well, I don't know how to name it.

DBIx::Class is one of my favorite modules. for its structure, with DBIx::Class you can make all your code very well organized and clean. writing code with DBIx::Class means you can use it in framework like Catalyst or Mojo or Dancer, and you can use it in any perl script (cron usually). for me, DBIx::Class is the right model, TT2 is the right template, and framework is just for URL dispatch.

Moose is another module I like. Role, and clean OO.

by using DBIx::Class::Schema::Loader make_schema_at, with use_moose => 1 on (eg: https://github.com/fayland/findmjob.com/blob/master/script/make_schema_at.pl), we can generate very clean Result module with Moose.

sometimes you want make assessor based on the table column. like for each job in table, we all have a URL which is based on the id and title in the table, it's so related to those two fields, so we'd better to put it in the Result.

with DBIx::Class, it's very simple. (sample code)

package FindmJob::Schema::Result::Job;

....

use FindmJob::Utils 'seo_title';
has 'url' => ( is => 'ro', isa => 'Str', lazy_build => 1 );
sub _build_url {
my ($self) = @_;

return "/job/" . $self->id . "/" . seo_title($self->title) . ".html";
}

after that, we can always call $job->url after we ->search for ->find it. very neat. live demo like:

http://findmjob.com/job/Il2bUdB74RGuDaqxKQ5yzw/Senior-Perl-Developer.html

have fun. Thanks

findmjob.com

2012-04-02T00:00:00+08:00

maybe to earn some money (not for fun this time), I decided to write a new website http://findmjob.com/

I sit front of my computer and coded it for 2 days and here is it. it's out.

it's very simple and without much stuff yet. and the final goal is undecided. but there it is. I'm very pleased to see it in public.

since there is no reason to keep it private, I opened source it in github: https://github.com/fayland/findmjob.com

for programmer, it's very simple to write website. but it's very hard to make it a success. so suggestions are welcome.

Thanks.

better pagination url design in Dancer

2012-04-02T00:00:00+08:00

usually People do param for pager like ?page=1 or ?p=1, it maybe not that good for search engine because they may not go scrape inside. so we may come out a solution with /page=1/ or /p=1/ or even /p.1/ etc.

in Dancer, it's very tricky to do add pagination regex in all URLs. and thank God, we have 'forward' and with code like below, it becomes very simple and easy to use.

Perl code:

get qr'.*?/p\.(\d+).*?' => sub {
my $uri = request->uri;
$uri =~ s'/p\.(\d+)'';
var page => $1;
$uri =~ s/\/$//;
forward $uri;
};

get '/' => sub {
my $p = vars->{page} || 1; $p = 1 unless $p =~ /^\d+$/;

we use regex to get the page stuff, then remove it from the request uri then using forward to do a internal request.

the TT2 pager code can check at https://github.com/fayland/findmjob.com/blob/master/templates/pager.tt

live demo as: http://findmjob.com/tag/perl/ and http://findmjob.com/tag/perl/p.2/

have fun.

Net-GitHub-0.43_01

2012-03-30T00:00:00+08:00

Github is moving on with their API that "We will terminate API v1 and API v2 in 1 month on May 1st, 2012.".

I know lots of people are preferring that they want token instead of writing user/pass in the config or code.

and now we have the choice with create access_token with code instead of web flow.

it's time to kick Net::GitHub default to V3 now. so here comes the beta release. and tests/patches are welcome.

http://fayland.org/CPAN/Net-GitHub-0.43_01.tar.gz (it will be on CPAN soon too)

Tips to create access_token with script:

use Net::GitHub;
my $gh = Net::GitHub->new( login => 'fayland', pass => 'secret' );
my $oauth = $gh->oauth;
my $o = $oauth->create_authorization( {
scopes => ['user', 'public_repo', 'repo', 'gist'], # just ['public_repo']
note => 'test purpose',
} );
print $o->{token};

record the token down, and later on we can always use that token without writing down user/pass.

my $github = Net::GitHub->new(
access_token => $token, # from above
);

Thanks.

Fayland and Programming

mojo tt2 defaults layout patch

Mojolicious TT2 patch

Dancer to Mojolicious

Few notes

hooks

Dancer forward VS Mojolicious before_dispatch

supervise

Template name conversation

Conclusion

Two new CPAN modules

WWW::SpinnerChief

Business::PayPoint

Net::Telnet with bitflu

remove ANSI color

Change Window Size

Learn Python Note 1

argparse

init.py and as

BeautifulSoup is another HTML::TreeBuilder

something I do like in Python

something I do not like in Python

Disclaim

Blogger local file > Jekyll migrator

add alert note for old posts in Jekyll

add alert note for old posts

20 items in rss/atom

paginate

MT > Jekyll perl migrator

Net-Amazon-DynamoDB

reCaptcha and lightbox and ajaxPost

MacOSx homebrew postgresql issue note

MongoDB and Perl

api and android app

sphinx search with varchar primary key

DBIx::Class with Moose has

findmjob.com

better pagination url design in Dancer

Net-GitHub-0.43_01

Dancer `forward` VS Mojolicious `before_dispatch`