Blog do Fernando Meyer

Writing a domain specific language DSL with python

fmeyer — Sat, 05 Sep 2009 14:51:53 PDT

A domain-specific language is a piece of software designed to be useful for a specific task in a fixed problem domain, they’re gaining popularity because they enhance productivity and reusability of artifacts. DSLs also enable expression and validation of concepts at the level of abstraction of the problem domain, this approach is very useful when you need to describe a user interface, a business process, a database, or the flow of information.

The DSL concept isn’t new after all, special-purpose programming languages and all kinds of modeling, specification languages have always existed, but this term rise due the popularity of domain-specific model.

You can easily implement dsls using the ruby language, Java or even C# if you prefer, but this isn’t the main propose of this article. The sine qua non become visible when I was implementing a simple test case with python. Indeed, there are a lot of python BDD-like frameworks, mostly who are self claimed the silver bullet, that are mismatching a lot of basic principles, but like I said, we are talking about dsls =)

With python we can easily create a piece of software that expresses some basic desired behavior, like rspec does, but much more pythonic.

spec.py

# coding: pyspec
    class Bow:
        def shot(self):
            print "got shot"
 
        def score(self):
            return 5
 
    describe Bowling:
        it "should score 0 for gutter game":
            bowling = Bow()
            bowling.shot()
            assert that bowling.score.should_be(5)

we can easily make this test dsl a runnable piece of python code, whiteout writing incompressible regular expressions, just using the python codecs and tokenizer.

Fist of all we need to define a new encoding for pyspec – our pre defined spec file syntax – this neat hacking enables a new path to tokenize this file

tokenizer.py

import tokenize
    import codecs, cStringIO, encodings
    from encodings import utf_8
 
    class StreamReader(utf_8.StreamReader):
        def __init__(self, *args, **kwargs):
            codecs.StreamReader.__init__(self, *args, **kwargs)
            data = tokenize.untokenize(translate(self.stream.readline))
            self.stream = cStringIO.StringIO(data)
 
    def search_function(s):
        if s!='pyspec': return None
        utf8=encodings.search_function('utf8') # Assume utf8 encoding
        return codecs.CodecInfo(
            name='pyspec',
            encode = utf8.encode,
            decode = utf8.decode,
            incrementalencoder=utf8.incrementalencoder,
            incrementaldecoder=utf8.incrementaldecoder,
            streamreader=StreamReader,
            streamwriter=utf8.streamwriter)
 
    codecs.register(search_function)

Our tiny translate function defines a easy way to translate both describe and it into a traditional python class and method definition.

def method_for_it(token):
        return token.strip().replace(" ", "_").replace("\"","" ) + "(self)"
 
    def translate(readline):
        previous_name = ""
        for type, name,_,_,_ in tokenize.generate_tokens(readline):
            if type ==tokenize.NAME and name =='describe':
                yield tokenize.NAME, 'class'
            elif type ==tokenize.NAME and name =='it':
                yield tokenize.NAME, 'def'
            elif type == 3 and previous_name == 'it':
                yield 3, method_for_it(name)
            else:
                yield type,name
            previous_name = name

Clever isn’t ? Now, fell yourself free to fork this project on GitHub and finish the job =) maybe someday we can have a real BDD python framework. http://github.com/fmeyer/pydsl/tree/master

References:

Pagerank isn’t for humans

fmeyer — Sat, 29 Aug 2009 22:13:26 PDT

Transitioning from a web of links and a sequence of words to a web of content, meaning and knowledge is probably the next great moving that we are going to see on the next two or maybe three years.

I was chatting with a friend, how can we bring knowledge to the web, how can we use the web to make a really efficient human driven search engine, last week a few former Googlers made some noise about a totally new SE named cuil, I tried some queries on that page, but it does work almost like google, you must be pretty much binary to retrieve some interesting information, in another words, nothing has changed since 90’s on this area. This problem is around researcher’s minds for a long time, trust, influence, authority when applied to the web are essentially people based issues.

The propose is the content being an asset with information about what it really means running against the link/words algorithms with no explicit meaning and a simple assumption “yes … we know you’re a good reference because you have a lot of links”.

Make yourself a question, how to ask something? How do I ask for information?

You ask your close friend: “Sunday night guitar red cap TV?” when you really want to know about the Sunday night TV show where a girl with a funny red cap playing a guitar. Things does change when you bring meaning to it, thats what a human being does.

Indeed, Google is still leading this running, with several fields under extreme research, articles about Data Mining, Collective Intelligence and AI being published denotes the new approach.

Yes … things are about to change

O scrume

fmeyer — Fri, 28 Aug 2009 16:52:11 PDT

Como varias pessoas próximas a mim ja sabem eu deixei a RedHat em abril deste ano por motivos pessoais, o trabalho remoto mesmo tendo suas vantagens não conseguiu despertar em mim a mesma proatividade e cumplicidade que eu teria trabalhando com pessoas reais durante o dia-a-dia. Trabalhar em uma empresa em que eu pudesse estar fisicamente locado com um time e desenvolvendo um software incremental foram os motivos que me levaram a globo.com.

Dentro da gcom trabalhamos usando a metodologia Scrum, temos varios casos de sucesso e uma boa reputaçao no mercado brasileiro com a adoção desta metodologia. Mas o sucesso dos projetos não vem apenas do uso de uma metodologia agil, pessoas com otimas noções de negócio e tecnologia fazem o barco andar de forma gloriosa. Tive a sorte de participar logo no primeiro contato na empresa com profissionais que valorizam a qualidade de seu código, alem de entregar algo rápido e funcional também olham para o como fazer melhor, refatorar um algoritmo que não ficou bom em uma primeira abordagem, ou seja fazemos software.

Sempre brinco pelos corredores, que hoje em dia que as empresas se preocupam mais em vender a imagem que fazem o negocio de uma forma diferente das demais e deixam totalmente de lado as boas praticas de desenvolvimento que foram esculpidas durante anos de tentativas, erros e evoluções. Nasce ai o Scrume, a desculpa de usar a metodologia e/ou a abordagem agile com pessoas que não levam à paixão seu trabalho. Conversas que tive com o Philip calçado e artigos como “The Decline and Fall of Agile” veem apenas confirmar esta visão, anos de evolução sendo deixadas de lado por modismos.

Se você pensar de uma maneira racional, software vem sendo desenvolvido ao longo dos anos possibilitando avanços significativos em vários campos como biologia, astronomia e finanças para não me alongar, a não adoção do “scrum” não fez projetos como linux ou colocar o homem na lua com um tubo de 8088 por exemplo ser fadado ao fracasso. Ocorreram erros, acertos e em alguns momentos pessoas precisaram se sobrepor para trazer o bom senso a tona.

Concordo plenamente que a adoção do scrum por exemplo, aumenta a visibilidade dos problemas que ocorrem no desenvolvimento de features por termos um feedback rápido do cliente ou da organização, mas não é tendo apenas essa abordagem que fara seu projeto ser bem sucedido.

Os projetos hoje que dão certo, estão envoltos em uma gama de outros fatores que equilibram essa equação. Fica meu conselho, não é trazendo um evangelista e adotanto Scrum para sua empresa que fará seus projetos darem certo, mas se e somente se as pessoas envolvidas acreditarem no que estão fazendo.

Configuring Joseki + Pellet + TDB

fmeyer — Wed, 26 Aug 2009 20:45:21 PDT

I was looking for some documentation about using TDB and Pellet
reasoser on Jena without pet peeves, didn’t found anything usable at all, so after a few
attempts I was still fighting against an ambiguity error,
whenever Joseki tries to build a dataset, the following exception pops up

18:18:49 WARN  Configuration        :: Failed to build dataset from
description (service name: reason): cannot find a most specific type
for file:///joseki-config.ttl#dataset_reason, which has as
possibilities: ja:RDFDataset ja:InfModel.
com.hp.hpl.jena.assembler.exceptions.AmbiguousSpecificTypeException:
cannot find a most specific type for file:///Users/fmeyer/projects/semantic/
servicos/joseki/joseki-config.ttl#dataset_reason, which has as possibilities: 
ja:RDFDataset ja:InfModel.
 at com.hp.hpl.jena.assembler.assemblers.AssemblerGroup$PlainAssemblerGroup.
 open(AssemblerGroup.java:104)
 at com.hp.hpl.jena.assembler.assemblers.AssemblerGroup$ExpandingAssemblerGroup.
 open(AssemblerGroup.java:70)
 at com.hp.hpl.jena.assembler.assemblers.AssemblerBase.open(AssemblerBase.java:41)
 at com.hp.hpl.jena.assembler.assemblers.AssemblerBase.open(AssemblerBase.java:38)
 at org.joseki.DatasetDesc.newDataset(DatasetDesc.java:65)
 at org.joseki.DatasetDesc.initialize(DatasetDesc.java:60)
 at org.joseki.Configuration.processModel(Configuration.java:112)
 at org.joseki.Configuration.(Configuration.java:83)
 at org.joseki.Dispatcher.setConfiguration(Dispatcher.java:130)
 at org.joseki.Dispatcher.initServiceRegistry(Dispatcher.java:100)
 at org.joseki.Dispatcher.initServiceRegistry(Dispatcher.java:93)
 at org.joseki.RDFServer.init(RDFServer.java:79)
 at org.joseki.RDFServer.(RDFServer.java:64)
 at joseki.rdfserver.main(rdfserver.java:85)

We just need to define a GraphTDB as a subclass of a Model as follow to fix this issue

@prefix ja:     <http ://jena.hpl.hp.com/2005/11/Assembler#> .
@prefix tdb:     http><http ://jena.hpl.hp.com/2008/tdb#> .
 
## Initialize TDB.
 
[] ja:loadClass "com.hp.hpl.jena.tdb.TDB" .
tdb:DatasetTDB  rdfs:subClassOf  ja:RDFDataset .
tdb:GraphTDB    rdfs:subClassOf  ja:Model .
 
## ---- A whole dadaset managed by TDB
< #dataset_reason> rdf:type      ja:RDFDataset ;
  ja:defaultGraph < #inf_graph> .
 
< #dataset> rdf:type      ja:RDFDataset ;
  ja:defaultGraph < #def_graph> ;
  .
 
< #inf_graph> rdf:type  ja:InfModel ;
    ja:reasoner [
        ja:reasonerClass
            "org.mindswap.pellet.jena.PelletReasonerFactory" ;
       ] ;
    ja:baseModel < #tdbGraph> .
 
< #def_graph> rdf:type  ja:InfModel ;
    ja:baseModel < #tdbGraph> .
 
< #tdbGraph> rdf:type tdb:GraphTDB ;
    tdb:location "DB" ;
    .

Debug Backwards in time