El problema es que el Java POJO utiliza enumeraciones y la base de datos no puede por si sólo encargarse de su manejo.

Después de leer otra vez la guía de Jackson con las anotaciones ‘@JsonDeserialize’ ‘@JsonSerialize’ y MongoDB con Java Codecs entendí que esta es la estrategia:

• En Jackson, debemos guardar y leer las enumeraciones como cadena de caracteres. De hecho mi definición contiene una representación mas amigable que la que estaba usando. Esto lo voy a usar más tarde para un servicio web que estoy escribiendo y no tiene nada que ver con MongoDB
• En MongoDB registramos un Codec que se encarga de leer la enumeración desde una cadena de caracteres y la escribe de vuelta como una cadena de caracteres

Todo se ve más claro en código, así que allí vamos. Lo primero es anotar nuestro POJO con Jackson (sólo muestro el atributo StatusEnumType):

public class ItemType {
    @JsonDeserialize(using = StatusEnumTypeDeserializer.class)
    @JsonSerialize(using = StatusEnumTypeSerializer.class)
    @XmlElement(required = true)
    @XmlSchemaType(name = "token")
    protected StatusEnumType status;

La anotación registra el atributo con las clases que van a leer y a escribir la enumeración de forma que pueda ser entendida por nuestro ObjectMapper en Jackson.

La clase que codifica:

package com.kodegeek.cvebrowser.persistence.serializers;

import com.fasterxml.jackson.core.JsonGenerator;
import com.fasterxml.jackson.databind.JsonSerializer;
import com.fasterxml.jackson.databind.SerializerProvider;
import com.kodegeek.cvebrowser.entity.SimplePhaseEnumType;

import java.io.IOException;

public class SimplePhaseEnumTypeSerializer extends JsonSerializer {
    @Override
    public void serialize(SimplePhaseEnumType value, JsonGenerator gen, SerializerProvider serializers) throws IOException {
        if (! StringChecker.isMissing(value.value())) {
            gen.writeString(value.value());
        } else {
            gen.writeNull();
        }
    }
}

Y la que decodifica:

package com.kodegeek.cvebrowser.persistence.serializers;

import com.fasterxml.jackson.core.JsonParser;
import com.fasterxml.jackson.databind.DeserializationContext;
import com.fasterxml.jackson.databind.JsonDeserializer;
import com.kodegeek.cvebrowser.entity.SimplePhaseEnumType;

import java.io.IOException;

public class SimplePhaseEnumTypeDeserializer extends JsonDeserializer {
    @Override
    public SimplePhaseEnumType deserialize(JsonParser p, DeserializationContext ctxt) throws IOException {
        SimplePhaseEnumType value = null;
        try {
            value = SimplePhaseEnumType.fromValue(p.getValueAsString());
        } catch (IllegalArgumentException eaexp) {
            eaexp.printStackTrace();
        }
        return value;
    }
}

No hay que hacer nada más para que Jackson funcione. En cuanto a MongoDB, debemos escribir una clase que implemente “Codec”. Esta clase se encarga de codificar y descodificar nuestra enumeración:

package com.kodegeek.cvebrowser.persistence.serializers;

import com.kodegeek.cvebrowser.entity.SimplePhaseEnumType;
import org.bson.BsonReader;
import org.bson.BsonWriter;
import org.bson.codecs.Codec;
import org.bson.codecs.DecoderContext;
import org.bson.codecs.EncoderContext;

public class SimplePhaseEnumTypeCodec implements Codec{
    @Override
    public SimplePhaseEnumType decode(BsonReader reader, DecoderContext decoderContext) {
        return SimplePhaseEnumType.fromValue(reader.readString());
    }

    @Override
    public void encode(BsonWriter writer, SimplePhaseEnumType value, EncoderContext encoderContext) {
        writer.writeString(value.value());
    }

    @Override
    public Class getEncoderClass() {
        return SimplePhaseEnumType.class;
    }
}

El paso final es registrar nuestros codecs con el registro, de manera que estos sean llamados cuando procesemos este tipo de datos:

    /**
     * MongoDB could not make this any simpler ;-)
     * @return a Codec registry
     */
    public static CodecRegistry getCodecRegistry() {
        final CodecRegistry defaultCodecRegistry = MongoClient.getDefaultCodecRegistry();
        final CodecProvider pojoCodecProvider = PojoCodecProvider.builder().register(packages).build();
        final CodecRegistry cvePojoCodecRegistry = CodecRegistries.fromProviders(pojoCodecProvider);
        // Aqui e stan los nuevos codecs, el del ejemplo es "SimplePhaseEnumTypeCodec"
        final CodecRegistry customEnumCodecs = CodecRegistries.fromCodecs(
                new SimplePhaseEnumTypeCodec(),
                new StatusEnumTypeCodec(),
                new TypeEnumTypeCodec()
        );
        return CodecRegistries.fromRegistries(defaultCodecRegistry, customEnumCodecs, cvePojoCodecRegistry);
    }

No es tan fácil como parece :-). En el caso de Mongo no tiene sentido pensar en JPA (aunque hay soporte para esto), así que comencé a jugar con el API del manejado de conexiones.

La razón es que cuando tenemos un esquema previo, por ejemplo una definición que viene de XSD, no hay mucha elección en cuanto a como se ven los atributos.

Veamos por ejemplo una clase que resulta de convertir la definición de una entrada de XSD CVE (https://cve.mitre.org/schema/cve/cve_1.0.xsd).

//
// This file was generated by the JavaTM Architecture for XML Binding(JAXB) Reference Implementation, v2.2.8-b130911.1802 
// See http://java.sun.com/xml/jaxb 
// Any modifications to this file will be lost upon recompilation of the source schema. 
// Generated on: 2018.02.17 at 08:52:30 PM EST 
//


package com.kodegeek.cvebrowser.entity;

import javax.xml.bind.annotation.XmlEnum;
import javax.xml.bind.annotation.XmlEnumValue;
import javax.xml.bind.annotation.XmlType;


/**
 * Java class for simplePhaseEnumType.
 * 
 * The following schema fragment specifies the expected content contained within this class.
 * 
 * 
 * <simpleType name="simplePhaseEnumType">
 *   <restriction base="{http://www.w3.org/2001/XMLSchema}token">
 *     <enumeration value="Proposed"/>
 *     <enumeration value="Interim"/>
 *     <enumeration value="Modified"/>
 *     <enumeration value="Assigned"/>
 *   </restriction>
 * </simpleType>
 * 
 * 
 */
@XmlType(name = "simplePhaseEnumType")
@XmlEnum
public enum SimplePhaseEnumType {

    @XmlEnumValue("Proposed")
    PROPOSED("Proposed"),
    @XmlEnumValue("Interim")
    INTERIM("Interim"),
    @XmlEnumValue("Modified")
    MODIFIED("Modified"),
    @XmlEnumValue("Assigned")
    ASSIGNED("Assigned");
    private final String value;

    SimplePhaseEnumType(String v) {
        value = v;
    }

    public String value() {
        return value;
    }

    public static SimplePhaseEnumType fromValue(String v) {
        for (SimplePhaseEnumType c: SimplePhaseEnumType.values()) {
            if (c.value.equals(v)) {
                return c;
            }
        }
        throw new IllegalArgumentException(v);
    }

}

Ahora mi problema es que puedo guardar los datos (aparentemente) pero no puedo recuperarlos debido a que el codec no puede transformar la enumeración (la cual por definición no tiene constructores públicos).

Según MongoDB (PojoQuickTour,java), debería ser tan fácil como usar una base de datos con el Codec apropiado:

private final static String [] packages = {  "com.kodegeek.cvebrowser.entity" };
// ...

public static CodecRegistry getCodecRegistry() {
        final CodecRegistry defaultCodecRegistry = MongoClient.getDefaultCodecRegistry();
        final CodecProvider pojoCodecProvider = PojoCodecProvider.builder().register(packages).build();
        final CodecRegistry cvePojoCodecRegistry = CodecRegistries.fromProviders(pojoCodecProvider);
        return CodecRegistries.fromRegistries(defaultCodecRegistry, cvePojoCodecRegistry);
    }

// Later on, different place we get a database with a POJO codec
// ...
mongoClient.getDatabase("cvebrowser").withCodecRegistry(getCodecRegistry())

Sin embargo si trato de hacer una búsqueda usando el codec (Joses-iMac:CVEBrowser josevnz$ gradle test):

org.bson.codecs.configuration.CodecConfigurationException: Failed to decode 'phase'. Failed to decode 'value'. Cannot find a public constructor for 'SimplePhaseEnumType'.
	at org.bson.codecs.pojo.PojoCodecImpl.decodePropertyModel(PojoCodecImpl.java:192)
	at org.bson.codecs.pojo.PojoCodecImpl.decodeProperties(PojoCodecImpl.java:168)
	at org.bson.codecs.pojo.PojoCodecImpl.decode(PojoCodecImpl.java:122)
	at org.bson.codecs.pojo.PojoCodecImpl.decode(PojoCodecImpl.java:126)
	at com.mongodb.operation.CommandResultArrayCodec.decode(CommandResultArrayCodec.java:52)
	at com.mongodb.operation.CommandResultDocumentCodec.readValue(CommandResultDocumentCodec.java:60)
	at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:84)
	at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:41)
	at org.bson.codecs.configuration.LazyCodec.decode(LazyCodec.java:47)
	at org.bson.codecs.BsonDocumentCodec.readValue(BsonDocumentCodec.java:101)
	at com.mongodb.operation.CommandResultDocumentCodec.readValue(CommandResultDocumentCodec.java:63)
	at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:84)
	at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:41)
	at com.mongodb.connection.ReplyMessage.(ReplyMessage.java:51)
	at com.mongodb.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:301)
	at com.mongodb.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:255)
	at com.mongodb.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:98)
	at com.mongodb.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:441)
	at com.mongodb.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:80)
	at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:189)
	at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:264)
	at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:126)
	at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:118)
	at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:226)
	at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:217)
	at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:120)
	at com.mongodb.operation.FindOperation$1.call(FindOperation.java:717)
	at com.mongodb.operation.FindOperation$1.call(FindOperation.java:711)
	at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:471)
	at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:415)
	at com.mongodb.operation.FindOperation.execute(FindOperation.java:711)
	at com.mongodb.operation.FindOperation.execute(FindOperation.java:83)
	at com.mongodb.Mongo$3.execute(Mongo.java:826)
	at com.mongodb.MongoIterableImpl.execute(MongoIterableImpl.java:130)
	at com.mongodb.MongoIterableImpl.iterator(MongoIterableImpl.java:77)
	at com.mongodb.MongoIterableImpl.forEach(MongoIterableImpl.java:100)
	at com.kodegeek.cvebrowser.persistence.TestCVEMongoPojoManager.testPrintIssues(TestCVEMongoPojoManager.java:47)

“Failed to decode ‘phase’. Failed to decode ‘value’. Cannot find a public constructor for ‘SimplePhaseEnumType”. Estoy aprendiendo como enseñarle a MongoDB como guardar y leer estos valores sin que le de un infarto.

(Por cierto, estoy preguntando en StackOverFlow. Vamos a ver que tan complicado es).

La Internet cuenta con muchos lugares de donde podemos bajar juegos de datos, en este caso me decidí bajarme la lista de los nombres de bebes más populares en el estado de Nueva York entre los años 2011 y 2014, en formato JSON.

{
  "meta" : {
    "view" : {
      "id" : "25th-nujf",
      "name" : "Most Popular Baby Names by Sex and Mother's Ethnic Group, New York City",
      "attribution" : "Department of Health and Mental Hygiene (DOHMH)",
      "averageRating" : 0,
      "category" : "Health",
      "createdAt" : 1382724894,
      "description" : "The most popular baby names by sex and mother's ethnicity in New York City.",
      "displayType" : "table",
      "downloadCount" : 4328,
      "hideFromCatalog" : false,
      "hideFromDataJson" : false,
      "indexUpdatedAt" : 1465427458,
      "newBackend" : false,
...
  },
  "data" : [ [ 1, "EB6FAA1B-EE35-4D55-B07B-8E663565CCDF", 1, 1386853125, "399231", 1386853125, "399231", "{\n}", "2011", "FEMALE", "HISPANIC", "GERALDINE", "13", "75" ]
, [ 2, "2DBBA431-D26F-40A1-9375-AF7C16FF2987", 2, 1386853125, "399231", 1386853125, "399231", "{\n}", "2011", "FEMALE", "HISPANIC", "GIA", "21", "67" ]
, [ 3, "54318692-0577-4B21-80C8-9CAEFCEDA8BA", 3, 1386853125, "399231", 1386853125, "399231", "{\n}", "2011", "FEMALE", "HISPANIC", "GIANNA", "49", "42" ]
, [ 4, "17C1236A-5778-412D-8DC9-94EBC01BB9A1", 4, 1386853125, "399231", 1386853125, "399231", "{\n}", "2011", "FEMALE", "HISPANIC", "GISELLE", "38", "51" ]
, [ 5, "F53CF696-A3F4-4EC3-8DFD-10C0A111B2D8", 5, 1386853125, "399231", 1386853125, "399231", "{\n}", "2011", "FEMALE", "HISPANIC", "GRACE", "36", "53" ]
, [ 6, "6615893F-39B8-440C-98D3-5A37CCF1C44B", 6, 1386853125, "399231", 1386853125, "399231", "{\n}", "2011", "FEMALE", "HISPANIC", "GUADALUPE", "26", "62" ]
, [ 7, "CC9BE461-34B8-4BD7-BEF2-BDB23CA1ADC6", 7, 1386853125, "399231", 1386853125, "399231", "{\n}", "2011", "FEMALE", "HISPANIC", "HAILEY", "126", "8" ]
, [ 8, "4EA2FFD4-1B1D-4859-A5C2-045949E3FD36", 8, 1386853125, "399231", 1386853125, "399231", "{\n}", "2011", "FEMALE", "HISPANIC", "HALEY", "14", "74" ]
]}

Antes de importar el archivo en Mongo, hay que darle un masaje a los datos, para ignorar la metadata y para separar cada entrada en el archivo para que luzca como un documento individual, sobre el cual podremos hacer agregación.

Primero les muestro como arrancar MongoDB, importar los datos (no muestro como bajarse el archivo pero es trivial) y luego corremos unos cuantos consultas:

#!/bin/bash
export PATH=$PATH:/Users/josevnz/mongo/mongodb-osx-x86_64-3.2.10/bin
mongod --fork --logpath /Users/josevnz/mongo/mongod.log --noauth --rest --pidfilepath /Users/josevnz/mongo/mongod.pid --dbpath /Users/josevnz/mongo/data

El programa para masajear los datos, en Python:

#!/usr/bin/env python
# josevn at kodegeek.com
import json
import sys
from pprint import pprint
if len(sys.argv) < 2:
  raise ValueError("Missing JSON file...")

'''
// From this (array of arrays):
{
    "data": [
        [
            1,
            "EB6FAA1B-EE35-4D55-B07B-8E663565CCDF",
            1,
            1386853125,
            "399231",
            1386853125,
            "399231",
            "{\n}",
            "2011",
            "FEMALE",
            "HISPANIC",
            "GERALDINE",
            "13",
            "75"
        ],
        [
          ...
        ]
    ],
    "metadata" : {
...
    }
}
 // To this individual maps, no metadata and no extra attributes:
      {
       "key": 1,
       "id": "EB6FAA1B-EE35-4D55-B07B-8E663565CCDF",
       "year": "2011",
       "gender": "FEMALE",
       "ethnicity": "HISPANIC",
       "name": "GERALDINE",
       "cnt": 13
      },
      {
       ...
      }
'''
with open(sys.argv[1]) as data_file:
  data = json.load(data_file)
  for item in data["data"]:
    newitem = {}
    newitem['key'] = item[0]
    newitem['id'] = item[1]
    newitem['year'] = int(item[8])
    newitem['gender'] = item[9]
    newitem['ethnicity'] = item[10]
    newitem['name'] = item[11]
    newitem['cnt'] = int(item[12])
    print(json.dumps(newitem))

Luego importamos los datos y nos conectamos listos para correr un par de consultas:

./jsonMap.py ~/Downloads/NY.babynames.json > ~/Downloads/NY.babynames.pretty.json
mongoimport --db ny --collection babynames --file ~/Downloads/NY.babynames.pretty.json --upsert
mongo localhost:27017/ny

Me encanta que no tuve que crear una base de datos, o una tabla en SQL :-). El documento se describe sólo y ahora estoy listo para hacer consultas:

// ¿Cuantos bebes hispanos entre el 2011 y el 2014?
> db.babynames.find({ "ethnicity": "HISPANIC"}).count()
4254
// Muestre los total agrupando por raza, sexo y año de mayor  a menor
> db.babynames.aggregate([ { $group: { _id: { ethnicity: "$ethnicity", gender: "$gender", year: "$year" }, total: { $sum: "$cnt"} } }, {$sort: {total:-1}} ])
{ "_id" : { "ethnicity" : "HISPANIC", "gender" : "MALE", "year" : 2011 }, "total" : 56236 }
{ "_id" : { "ethnicity" : "WHITE NON HISPANIC", "gender" : "MALE", "year" : 2011 }, "total" : 54392 }
{ "_id" : { "ethnicity" : "WHITE NON HISPANIC", "gender" : "FEMALE", "year" : 2011 }, "total" : 45120 }
{ "_id" : { "ethnicity" : "HISPANIC", "gender" : "FEMALE", "year" : 2011 }, "total" : 41532 }
{ "_id" : { "ethnicity" : "BLACK NON HISPANIC", "gender" : "MALE", "year" : 2011 }, "total" : 24540 }
{ "_id" : { "ethnicity" : "ASIAN AND PACIFIC ISLANDER", "gender" : "MALE", "year" : 2011 }, "total" : 18580 }
{ "_id" : { "ethnicity" : "BLACK NON HISPANIC", "gender" : "FEMALE", "year" : 2011 }, "total" : 17624 }
{ "_id" : { "ethnicity" : "WHITE NON HISPANIC", "gender" : "MALE", "year" : 2014 }, "total" : 14831 }
{ "_id" : { "ethnicity" : "WHITE NON HISPANIC", "gender" : "MALE", "year" : 2013 }, "total" : 14537 }
{ "_id" : { "ethnicity" : "WHITE NON HISP", "gender" : "MALE", "year" : 2012 }, "total" : 14273 }
{ "_id" : { "ethnicity" : "HISPANIC", "gender" : "MALE", "year" : 2012 }, "total" : 13809 }
{ "_id" : { "ethnicity" : "ASIAN AND PACIFIC ISLANDER", "gender" : "FEMALE", "year" : 2011 }, "total" : 13672 }
{ "_id" : { "ethnicity" : "HISPANIC", "gender" : "MALE", "year" : 2013 }, "total" : 13312 }
{ "_id" : { "ethnicity" : "HISPANIC", "gender" : "MALE", "year" : 2014 }, "total" : 13126 }
{ "_id" : { "ethnicity" : "WHITE NON HISPANIC", "gender" : "FEMALE", "year" : 2014 }, "total" : 12884 }
{ "_id" : { "ethnicity" : "WHITE NON HISP", "gender" : "FEMALE", "year" : 2012 }, "total" : 12402 }
{ "_id" : { "ethnicity" : "WHITE NON HISPANIC", "gender" : "FEMALE", "year" : 2013 }, "total" : 12303 }
{ "_id" : { "ethnicity" : "HISPANIC", "gender" : "FEMALE", "year" : 2013 }, "total" : 9755 }
{ "_id" : { "ethnicity" : "HISPANIC", "gender" : "FEMALE", "year" : 2012 }, "total" : 9738 }
{ "_id" : { "ethnicity" : "HISPANIC", "gender" : "FEMALE", "year" : 2014 }, "total" : 9729 }
Type "it" for more
> it
{ "_id" : { "ethnicity" : "BLACK NON HISP", "gender" : "MALE", "year" : 2012 }, "total" : 5965 }
{ "_id" : { "ethnicity" : "ASIAN AND PACI", "gender" : "MALE", "year" : 2012 }, "total" : 5962 }
{ "_id" : { "ethnicity" : "BLACK NON HISPANIC", "gender" : "MALE", "year" : 2013 }, "total" : 5866 }
{ "_id" : { "ethnicity" : "BLACK NON HISPANIC", "gender" : "MALE", "year" : 2014 }, "total" : 5702 }
{ "_id" : { "ethnicity" : "ASIAN AND PACIFIC ISLANDER", "gender" : "MALE", "year" : 2014 }, "total" : 5636 }
{ "_id" : { "ethnicity" : "ASIAN AND PACIFIC ISLANDER", "gender" : "MALE", "year" : 2013 }, "total" : 5281 }
{ "_id" : { "ethnicity" : "ASIAN AND PACI", "gender" : "FEMALE", "year" : 2012 }, "total" : 4338 }
{ "_id" : { "ethnicity" : "BLACK NON HISPANIC", "gender" : "FEMALE", "year" : 2013 }, "total" : 4278 }
{ "_id" : { "ethnicity" : "BLACK NON HISPANIC", "gender" : "FEMALE", "year" : 2014 }, "total" : 4255 }
{ "_id" : { "ethnicity" : "BLACK NON HISP", "gender" : "FEMALE", "year" : 2012 }, "total" : 4243 }
{ "_id" : { "ethnicity" : "ASIAN AND PACIFIC ISLANDER", "gender" : "FEMALE", "year" : 2014 }, "total" : 4198 }
{ "_id" : { "ethnicity" : "ASIAN AND PACIFIC ISLANDER", "gender" : "FEMALE", "year" : 2013 }, "total" : 4012 }

Estoy emocionado con la herramienta, pienso usarla para ver los datos en mi trabajo de manera distinta.

package com.kodegeek;


public class StaticWeird {

    public static int count = 0;

    public static void print() {
        System.out.println(++count);
    }

    /**
     * What is the output?
     * @param args
     */
    public static void main(String [] args) {
        StaticWeird weird = new StaticWeird();
        weird.print();
        weird = null; // Magic hat trick
        weird.print(); // Expecting an NPE :-)?
    }


}

No es “NullPointerException”. Java sabe que la referencia al método es estática (static) y utiliza esa en vez de una referencia a la instancia de la clase.

¡Java tramposo!

#!/usr/bin/env python3
import sys, re

def grep(expression):
    while True:
        text = (yield)
        if re.search(expression, text):
            print(text)

def cat(file, target):
    next(target)
    with open(file, "r") as fh:
        for line in fh.readlines():
            target.send(line.strip())

def main(args):
    cat(args[0], grep(args[1]))

if __name__ == "__main__":
    main(sys.argv[1:])

En la linea 11 pueden ver que hay que preparar el otro lado de la tubería antes de enviar datos, lo cual es un fastidio; Usando ‘functools’ podemos crear una anotación a la medida, la cual llamaremos ‘pipe’ (lineas 6-12):

#!/usr/bin/env python3
# Shows how to use a function decorator to create an annotation
# josevnz@kodegeek.com
import sys, re, functools

def pipe(function):
    @functools.wraps(function)
    def wrapper(*args, **kwargs):
        target = function(*args, **kwargs)
        next(target) # Saves us the trouble of calling next on the pipe
        return target
    return wrapper

@pipe
def grep(expression):
    while True:
        text = (yield)
        if re.search(expression, text):
            print(text)

def cat(file, target):
    with open(file, "r") as fh:
        for line in fh.readlines():
            target.send(line.strip())

def main(args):
    cat(args[0], grep(args[1]))

if __name__ == "__main__":
    main(sys.argv[1:])

En la línea 14 agregamos la nueva anotación y note como eliminamos el ‘next(target)’ completamente de la función cat(file, target). No paree gran cosa, pero es una línea menos de código que poner en cada llamada.

¿Pero se pueden hacer otras cosas? Se me ocurren decoraciones como ‘count’ o ‘sorted’ para funciones que trabajan con listas, estoy aprendiendo como usar esta nueva herramienta, ya les diré como me va.

# Contar cuantos 'root' hay en el archivo de passwords del servidor Linux
cat /etc/password| egrep -i root

En Python 3 podemos hacer algo como lo siguiente:

#!/usr/bin/env python3
# pipeline.py /etc/passwd root
import sys, re

def grep(expression):
    while True:
        text = (yield) # Espere por el texto enviados por la otra co-rutina
        if re.search(expression, text):
            print(text)

def cat(file, target):
    next(target) # Initialize el otro lado de la tuberia
    with open(file, "r") as fh:
        for line in fh.readlines():
            target.send(line.strip()) # envie la linea a la co-rutina

def main(args):
    cat(args[0], grep(args[1])) # La 'tuberia' se hace de afuera hacia adentro

if __name__ == "__main__":
    main(sys.argv[1:])

Otro ejemplo, vamos a modificar el programa que lee los datos del archivo de datos de DolarToday para filtrar por fecha de inicio, final, valores máximos y mínimos. Note como el uso de los filtros es opcional y si no se aplican entonces retornan todos los datos (para mantener el ejemplo sencillo, removí el código que convierte los datos al formato binario):

#!/usr/bin/env python3
# Simple program to save the 'Dolartoday' extra official dollar rates from CSV to a custom format
# Revisited to use data pipes, couroutines
# @author Jose Vicente Nunez, josevnz@kodegeek.com
import os, sys, datetime, re, gzip, struct
from optparse import OptionParser, OptionValueError
from datetime import datetime

class DollarToday:
    
    __slot__ = ("__date", "__value")
    
    def __init__(self, ddate, value):
        if isinstance(ddate, datetime):
            self.__date = ddate
        else:
            self.__date = datetime.strptime(ddate, "%m-%d-%Y")
        self.__value = float(value)
        assert self.__value > 0.0, "{} for date {} is invalid!".format(value, self.__date)
        
    @property
    def date(self):
        return self.__date
    
    @date.setter
    def date(self, date):
        assert isinstance(date, datetime), "Invalid date {}".format(date)
        self.__date = date
    
    @property
    def value(self):
        return self.__value
    
    def __str__(self):
        return "DollarToday[date={}, value={}]".format(self.__date, self.__value)
    
    def __hash__(self):
        return str(id(self))
            
class DollarCollection(dict):
    
    def values(self):
        for dateId in sorted(self.keys()):
            yield self[dateId]
    
    def items(self):
        for dateId in self.keys():
            yield (dateId, self[dateId])
    
    def __iter__(self):
        for dateId in sorted(super().keys()):
            yield dateId

    def __str__(self):
        return "Records={},\n{}".format(len(self), ",\n".join([str(date) for date in self.values()]))

# Pipe: Read from binary file
def readBinary(file, target):
    next(target)
    FILE_MAGIC = b"DLR\x00"
    FILE_VERSION = b"\x00\x01"
    dollarStruct = struct.Struct(" FILE_VERSION:
          raise "Unsupported file version: {}, expected {}".format(version, FILE_VERSION)
        while True:
          data = fh.read(dollarStruct.size)
          if len(data) == 0:
           break
          numbers = dollarStruct.unpack(data)
          target.send(DollarToday(datetime.fromordinal(numbers[0]), numbers[1]))
    except (Exception) as err:
            raise

# Pipe: Filter elements >= minimum
def min(min, target):
    next(target)
    while True:
        dollar = (yield)
        if dollar.value >= min:
            target.send(dollar)

# Pipe: Filter elements < = max
def max(max, target):
    next(target)
    while True:
        dollar = (yield)
        if dollar.value <= max:
            target.send(dollar)

# Pipe: filter from date
def from_date(fromd, target):
        next(target)
        while True:
            dollar = (yield)
            if dollar.date >= fromd:
                target.send(dollar)

# Pipe: filter from date
def to_date(tod, target):
        next(target)
        while True:
            dollar = (yield)
            if dollar.date < = tod:
                target.send(dollar)

def adder(collection):
    while True:
        dollar = (yield)
        collection[dollar.date] = dollar

def main(options):

    dc = DollarCollection()
    
    pipeline = adder(dc) # Destination of the pipe
    # Add pipes (if needed)
    if options.min != 99999999:
        pipeline = min(options.min, pipeline)
    
    if options.max != -1:
        pipeline = max(options.min, pipeline)

    if options.fromd is not None:
        pipeline = from_date(datetime.strptime(options.fromd, "%Y-%m-%d"), pipeline)
        
    if options.tod is not None:
        pipeline = to_date(datetime.strptime(options.tod, "%Y-%m-%d"), pipeline)

    readBinary(options.report, pipeline) # Start of the pipe that reads the file contents
 
    print("{}".format(dc))

if __name__ == "__main__":
    
    usagetext = """
%prog --report binary.file [--from_date YYYYMMDD] [--to YYYYMMDD] [--min amount] [--max amount]
"""

    op = OptionParser(usage=usagetext)
    op.add_option(
                  "-p", "--report",
                  action="store",
                  dest="report",
                  help="Read the contents from the binary storage and generate a report.")
    op.add_option(
                  "-f", "--from",
                  action="store",
                  dest="fromd",
                  help="Optional filter. Start date yyyy-mm-dd")
    op.add_option(
                  "-t", "--to",
                  action="store",
                  dest="tod",
                  help="Optional filter. End date yyyy-mm-dd")
    op.add_option(
                  "-m", "--min",
                  action="store",
                  dest="min",
                  default=99999999,
                  type="float",
                  help="Optional filter. Minimal amount")
    op.add_option(
                  "-M", "--max",
                  action="store",
                  dest="max",
                  default=-1,
                  type="float",
                  help="Optional filter. Maximum amount")
    
    (options, values) = op.parse_args()
    main(options)

Una salida de ejemplo:

DolarTodayReader.py --report /Users/josevnz/Documents/dolartoday.jose --min 1050 --from 2016-02-23 --to 2016-02-24
Records=2,
DollarToday[date=2016-02-23 00:00:00, value=1060.0],
DollarToday[date=2016-02-24 00:00:00, value=1071.19]

Para cerrar, les recomiendo el siguiente tutorial: http://www.dabeaz.com/coroutines/index.html. La sintaxis es de Python 2.

Les traigo de vuelta el programa de ‘DolarToday’ el cual escribe datos en un archivo binario, pero ahora utilizando contextos:

#!/usr/bin/env python3
# Simple program to save the 'Dolartoday' extra official dollar rates from CSV to a custom format
# Revisited to use 'contexts' to avoid using finally on exception handling
# @author Jose Vicente Nunez, josevnz@kodegeek.com
import os, sys, datetime, re, gzip, struct
from optparse import OptionParser, OptionValueError
from datetime import datetime

class DollarToday:
    
    def __init__(self, ddate, value):
        if isinstance(ddate, datetime):
            self.__date = ddate
        else:
            self.__date = datetime.strptime(ddate, "%m-%d-%Y")
        self.__value = float(value)
        assert self.__value > 0.0, "{} for date {} is invalid!".format(value, self.__date)
        
    @property
    def date(self):
        return self.__date
    
    @date.setter
    def date(self, date):
        assert isinstance(date, datetime), "Invalid date {}".format(date)
        self.__date = date
    
    @property
    def value(self):
        return self.__value
    
    def __str__(self):
        return "DollarToday[date={}, value={}]".format(self.__date, self.__value)
    
    def __hash__(self):
        return str(id(self))
            
class DollarCollection(dict):
    
    # https://en.wikipedia.org/wiki/List_of_file_signatures
    __FILE_MAGIC = b"DLR\x00"
    __FILE_VERSION = b"\x00\x01"
    __dollarStruct = struct.Struct(" self.__FILE_VERSION:
                    raise "Unsupported file version: {}, expected {}".format(version, self.__FILE_VERSION)
                self.clear()
                while True:
                    data = fh.read(self.__dollarStruct.size)
                    if len(data) == 0:
                        break
                    numbers = self.__dollarStruct.unpack(data)
                    #if verbose:
                    #    print("{}".format(",".join([str(x) for x in numbers])))
                    dolar = DollarToday(
                                    datetime.fromordinal(numbers[0]),
                                    numbers[1]
                                    )
                    self[dolar.date] = dolar
        except (Exception) as err:
            raise
                
    def readCsv(self, file):
        tempMap = {}
        try:
            with open(file, "r", encoding="UTF-8") as fh:
                for line in fh.readlines():
                    # 6-23-2010       9.92
                    (date, value) = line.strip().split("\t")
                    if re.match("Fecha", date):
                        continue
                    try:
                        dolVsBol = DollarToday(date, value)
                        tempMap[dolVsBol.date] = dolVsBol
                    except (Exception) as err:
                        print(err)
            self.clear()
            self.update(tempMap)
        except (Exception) as err:
            raise

    def __str__(self):
        return "Records={},\n{}".format(len(self), ",\n".join([str(date) for date in self.values()]))

def main(options):

    verbose = options.verbose
    dc = DollarCollection()
    if options.report == None:
        dc.readCsv(options.read)
        dc.save(options.write)
    else:
        dc.readBinary(options.report, verbose)
    if verbose:
        print("{}".format(dc))

if __name__ == "__main__":
    
    usagetext = """
%prog --read csv.file --write binary.file

Or:

%prog --report binary.file

"""

    op = OptionParser(usage=usagetext)
    op.add_option(
                  "-r", "--read",
                  action="store", 
                  dest="read", 
                  help="Full path to CSV file with date,value pairs per line")
    op.add_option(
                  "-w", "--write", 
                  action="store", 
                  dest="write", 
                  help="Full path to destination file in binary format")
    op.add_option(
                  "-p", "--report", 
                  action="store", 
                  dest="report",
                  help="Read the contents of the binary storage and generate a report. Incompatible with --read and --write")
    op.add_option(
                  "-v", "--verbose", 
                  action="store_true", 
                  default=False, 
                  dest="verbose", 
                  help="Enable verbose mode")
    
    (options, values) = op.parse_args()

    main(options)

#!/usr/bin/env python3
# A little fun with the candidates for the US presidency for 2016 election year
# Revisited versions with functors, slots, abstract classes
# @author josevnz@kodegeek.com
#
import abc

class SortKey:
    
    def __init__(self, *attributes):
        self.attributes = attributes
    def __call__(self, instance):
        return [getattr(instance, attribute) for attribute in self.attributes]

class Candidate(metaclass=abc.ABCMeta):
    
    __slots__ = ("__name", "__sex", "__age")

    @abc.abstractmethod    
    def __init__(self, name, sex="F", age=21):
        self.__name = name
        self.__sex = "F" if sex not in ["M", "F"] else sex
        self.__age = 21 if not (21 < = age <= 120) else age
    
    @property
    def name(self):
        return self.__name
    
    def get_party(self):
        raise NotImplemented()
    
    party = abc.abstractproperty(get_party)
    
    @property
    def sex(self):
        return self.__sex
    
    @property
    def age(self):
        return self.__age
    
    def __hash__(self):
        return hash(id(self))
    
    def __str__(self):
        return "{}=[name={}, sex={}, age={}]".format(self.__class__.__name__,
                                                            self.__name,
                                                            self.__sex,
                                                            self.__age
                                                            )
    
    def __repr__(self):
        return "{}({}{}{}{})".format(
                                 self.__class__.__name__, 
                                 self.__name, 
                                 self.__sex, 
                                 self.__age
                                )
    
    @abc.abstractmethod
    def __bool__(self):
        raise NotImplemented()
    
    def __lt__(self, other):
        return self.__age < other.__age
    
    def __gt__(self, other):
        return self.__age > other.__age
    
    def __eq__(self, other):
        return (
                self.__age == other.__age and 
                self.__name == other.__name and
                self.__sex == other.__sex
                )

class Republican(Candidate):
    
    def __init__(self, name, sex="F", age=21):
        return super().__init__(name, sex, age)
    
    party = "Replublican Party (GOP)"
    
    def __bool__(self):
        return False
            
class Democrat(Candidate):
    
    def __init__(self, name, sex="F", age=21):
        return super().__init__(name, sex, age)
    
    party = "Democratic Party"
    
    def __bool__(self):
        return True
    
if __name__ == "__main__":
        candidates = []
        candidates.append(Democrat("Hillary Clinton", "F", 68))
        candidates.append(Republican("Donald Trump", "M", 70))
        candidates.append(Democrat("Bernie Sanders", "M", 75))
        candidates.append(Republican("Marco Rubio", "M", 45))
        candidates.sort(key=SortKey("sex", "age"), reverse=False)
        for candidate in candidates:
            isDemocrat = "yes" if candidate else "no"
            print("Candidate: {}, democrat? {}".format(candidate, isDemocrat))        

Y la salida (fíjense que definimos el orden en la linea 103, ademas de que ordenamos la lista primero para después utilizarla):

Candidate: Democrat=[name=Hillary Clinton, sex=F, age=68], democrat? yes
Candidate: Republican=[name=Marco Rubio, sex=M, age=45], democrat? no
Candidate: Republican=[name=Donald Trump, sex=M, age=70], democrat? no
Candidate: Democrat=[name=Bernie Sanders, sex=M, age=75], democrat? yes

En este caso:

• Uso un ‘slot’ para guardar sólo 3 atributos que son definidos en el constructor
  Declaro la propiedad ‘party’ como abstracta ya que quiero que el que extienda las clases se vea forzado a implementar un método adecuado que me diga de que partido viene
  También obligo al usuario a declarar el método ‘__bool__’ el cual utilizo para saber si es Democrata u otra clase (si, el ejemplo es forzado :-))

#!/usr/bin/env python3
# A little fun with the candidates for the US presidency for 2016 election year
# Revisited versions with slots, abstract classes
# @author josevnz@kodegeek.com
#
import abc

class Candidate(metaclass=abc.ABCMeta):
    
    __slots__ = ("__name", "__sex", "__age")

    @abc.abstractmethod    
    def __init__(self, name, sex="F", age=21):
        self.__name = name
        self.__sex = "F" if sex not in ["M", "F"] else sex
        self.__age = 21 if not (21 < = age <= 120) else age
    
    @property
    def name(self):
        return self.__name
    
    def get_party(self):
        raise NotImplemented()
    
    party = abc.abstractproperty(get_party)
    
    @property
    def sex(self):
        return self.__sex
    
    @property
    def age(self):
        return self.__age
    
    def __hash__(self):
        return hash(id(self))
    
    def __str__(self):
        return "{}=[name={}, sex={}, age={}]".format(self.__class__.__name__,
                                                            self.__name,
                                                            self.__sex,
                                                            self.__age
                                                            )
    
    def __repr__(self):
        return "{}({}{}{}{})".format(
                                 self.__class__.__name__, 
                                 self.__name, 
                                 self.__sex, 
                                 self.__age
                                )
    
    @abc.abstractmethod
    def __bool__(self):
        raise NotImplemented()
    
    def __lt__(self, other):
        return self.__age < other.__age 

     def __gt__(self, other): 
        return self.__age > other.__age
    
    def __eq__(self, other):
        return (
                self.__age == other.__age and 
                self.__name == other.__name and
                self.__sex == other.__sex
                )

class Republican(Candidate):
    
    def __init__(self, name, sex="F", age=21):
        return super().__init__(name, sex, age)
    
    party = "Republican Party (GOP)"
    
    def __bool__(self):
        return False
            
class Democrat(Candidate):
    
    def __init__(self, name, sex="F", age=21):
        return super().__init__(name, sex, age)
    
    party = "Democratic Party"
    
    def __bool__(self):
        return True
    
if __name__ == "__main__":
        candidates = []
        candidates.append(Democrat("Hillary Clinton", "F", 68))
        candidates.append(Republican("Donald Trump", "M", 70))
        candidates.append(Democrat("Bernie Sanders", "M", 75))
        candidates.append(Republican("Marco Rubio", "M", 45))
        for candidate in sorted(candidates, reverse=True):
            isDemocrat = "yes" if candidate else "no"
            print("Candidate: {}, democrat? {}".format(candidate, isDemocrat))