Fernando Rubbo

SCD4J v0.7.5

2015-08-23T12:15:00.001-07:00

We are proud to announce a new SCD4J release (v0.7.5)

This version includes:

The latest version of gradle (2.6)
Improvements for non console runs
Other minor stuff

Besides that, we have published a new article (in Portuguese only, sorry folks) about how to install a cluster of wildfly.

Instalando um cluster de Wildfly automaticamente

2015-08-23T11:41:00.001-07:00

Aprenda como instalar automaticamente seu ambiente de desenvolvimento usando SCD4J

[Resumo]

Este artigo é útil para mostrar ao leitor que a automação de ambientes de infraestrutura está deixando ser um diferencial para se tornar uma necessidade dentro das empresas que desenvolvem software. Além disto, também é apresentada uma ferramenta, chamada SCD4J, que possui um propósito específico para a instalação e configuração de ambientes Java a qual é capaz de simplificar esta automatização em empresas de pequeno e médio porte que optaram por entrar no mundo de entregas contínuas.

Introdução

Após o boom do Continuous Delivery, que aconteceu em meados de 2010, muito tem se falado sobre a importância da automatização de builds, das revisões de código, dos testes (unit, integration, acceptance, performance, etc) e das instalações e configurações de ambientes de TI. O objetivo principal por trás disto tudo é que estas automatizações ajudam a antecipar problemas e a criar sistemas mais confiáveis e de alta qualidade, uma vez que, a cada alteração realizada no código-fonte, uma série de verificações são aplicadas sem a necessidade de qualquer intervenção humana. Porém, na prática, o que vemos sendo utilizado pela indústria de software são apenas as estratégias que envolvem os times de desenvolvimento isoladamente, isto é, deixando de lado uma parte muito importante do processo que é a automatização dos ambientes, o que, em geral, fica a cargo da equipe de infraestrutura.

É importante ressaltar que de nada adianta ter um super time de desenvolvimento que cria funcionalidades magníficas para os ambientes de testes. Isto é, se o time não conhece a fundo o ambiente onde o sistema definitivo será instalado, existe uma possibilidade muito grande de todo este trabalho ir por água abaixo quando a passagem para produção acontecer. Além disto, a experiência nos mostra que os problemas que acontecem durante esta passagem crescem exponencialmente ao tamanho do time e ao intervalo de tempo entre releases.

O motivo por trás destas afirmações é que, em geral, cada desenvolvedor possui uma máquina instalada e configurada diferentemente. Além disto, os ambientes de testes, de homologação e de produção são, comumente, mantidos por pessoas/times diferentes o que acarreta em estruturas e configurações não padronizadas. Como este modelo é muito comum nas empresas que desenvolvem software, também é muito comum ouvir comentários do tipo: “na minha máquina funcionava”, “eu testei no ambiente de testes”, “o time de infraestrutura que não consegue seguir os passos do documento”, e assim vai.

Box 1. O problema é generalizado

Qualquer semelhança não é mera coincidência. Isto é, na verdade, um problema generalizado que a indústria de software está vivenciando atualmente.

A resposta para este tipo de problema inicialmente parece ser bastante simples: basta definir um conjunto de padrões e doutrinar as equipes para que estes padrões sejam seguidos. Porém, na prática, a doutrina e a criação de documentos de configuração (que, além de se serem longos e propícios a erros, rapidamente ficam desatualizados) não são suficientes para garantir a igualdade entre ambientes, uma vez que, a qualquer momento, um indivíduo pode, intencionalmente ou não, modificar seu ambiente sem que os outros integrantes do time fiquem sabendo.

Em resumo, a falta de padrões entre os ambientes e as configurações manuais são responsáveis por muitos dos problemas que as fábricas de software costumam vivenciar. A boa notícia é que já existem soluções consolidadas no mercado para ambos estes problemas. Estas soluções são ferramentas de automatização de ambientes de infraestrutura. Dentre as mais conhecidas podemos destacar Puppet e Chef. Porém, apesar de serem excelentes opções, estas ferramentas são soluções generalistas que acabam gerando complexidade desnecessária para sysadmins e uma curva de aprendizado muito grande para desenvolvedores Java. Infelizmente, estas dificuldades fazem com que pequenas e médias empresas acabem deixando de lado este tipo automatização.

Com o objetivo de atender este nicho carente em automatização surgiu, recentemente, uma ferramenta chamada Simple Continuous Delivery for Java (SCD4J). Diferentemente das citadas anteriormente, esta ferramenta nasceu com um propósito específico para instalação e configuração de ambientes Java. Isto é, apesar de ela ser capaz de realizar a maioria das tarefas que as demais realizam, o alvo principal são empresas de pequeno e médio porte que apostam no desenvolvimento de soluções sobre a plataforma Java e na entrega contínua de software. Desta forma, sysadmins e desenvolvedores podem colaborar para construir automaticamente ambientes mais simples e padronizados. Considerando, é claro, deste a máquina do desenvolvedor até o cluster em produção.

Porque SCD4J?

Como foi mencionado anteriormente, SCD4J é uma ferramenta de automação para instalar e configurar a infraestrutura de sistemas que rodam sobre a plataforma Java. Com apenas um comando é possível instalar clusters e aplicações em qualquer ambiente do ciclo de desenvolvimento do software. Vale destacar que esta ferramenta não é apenas outra alternativa para Puppet e Chef. Ela é, na verdade, uma solução bem mais simples uma vez que foca em um nicho específico.

As principais vantagens de se utilizar SCD4J sobre as concorrentes são:

Desenhada com um propósito bem específico;

Isto é, foco em pequenas e médias empresas que optaram por utilizar tecnologias Java e entregas contínuas.

Praticamente extingue a necessidade de pré configuração dos ambientes;

Não necessita instalar e configurar servidores coordenadores;
Não necessita instalar e configurar agentes nos clientes.

Modelo arquitetural simples e efetivo

Utiliza o conceito de ganchos (em inglês, hooks) para implementar as lógicas da automação.

Pequena curva de aprendizado para desenvolvedores Java e Groovy

A codificação dos ganchos (hooks) é realizada em Groovy.

Integração direta com repositórios de binários comumente utilizados no mundo Java

Por exemplo: Artifactory, Nexus, MavenCentral, JCentral, etc.

Construída sobre soluções modernas e bem conhecidas no mundo Java

Por exemplo: Java 8, Gradle, Velocity, Mustache, etc.

Desenvolvedores conseguem rapidamente entrar no mundo de DevOps

Uma vez que a linguagem e as ferramentas já são conhecidas, basta aprender os conceitos básicos do framework.

Box 2. DevOps

Segundo Wikpedia, DevOps é uma metodologia de desenvolvimento de software que explora a comunicação, colaboração e integração entre desenvolvedores de software e profissionais de TI (Tecnologia da Informação). DevOps é a reação à interdependência entre desenvolvimento de software e operações de TI.

Desenhando os ambientes

Antes de sair automatizando a criação dos ambientes é necessário entender qual é o modelo utilizado no desenvolvimento do software em questão. Além disto, precisamos deixar bem claro para os stakeholders que todos os ambientes devem ser o mais parecidos possível a fim de antecipar problemas e evitar surpresas em produção. Muitas vezes isto implica em mudanças na infraestrutura ou mesmo no processo de desenvolvimento.

Para fins de exemplo, iremos utilizar um modelo simples mas bastante comum no desenvolvimento ágil de software com entregas contínuas. Por motivos de tamanho e escopo, não iremos abortar todos os detalhes. Contudo, iremos exemplificar como é possível fazer tal automatização em uma visão um pouco mais macro. Para mais detalhes, sugerimos que o código-fonte do exemplo, que se encontra no final deste artigo, seja estudado.

Figura 1. Definindo os ambientes

Como é possível verificar na Figura 1, possuímos três ambientes que queremos automatizar a instalação do servidor de aplicações Wildfly. Notem que o desenvolvedor trabalha em um desktop Windows que irá rodar uma única instância do servidor. Em contra partida, a partir do ambiente de testes já possuímos um cluster com, pelo menos, dois nodos. O objetivo de se ter um cluster nesta segunda etapa é que o time de testes precisa validar o sistema em um ambiente o mais próximo possível do final. Por fim, temos o nosso ambiente de produção que nada mais é do que um cluster mais “parrudo” do que o de testes.

Mãos na massa

Após definir os ambientes que queremos automatizar, temos que codificar a instalação. Como discutimos anteriormente, iremos utilizar a ferramenta SCD4J para realizar esta tarefa. A Figura 2 abaixo mostra a estrutura básica de um projeto nesta ferramenta:

Figura 2. Estrutura do Projeto SCD4J

Como podemos perceber, um projeto de SCD4J é composto de um arquivo build.gradle, que deve ser executado para iniciar a instalação, um diretório modules, que é onde colocamos o código-fonte da automatização, e um diretório config, que deve conter as configurações que são diferenciadas entre os ambiente. Vamos então começar a analisar o arquivo build.gradle.

Listagem 1. Arquivo build.gradle - plugin e dependências

1. plugins {

2. id "com.datamaio.scd4j" version "0.7.5"

3. }

5. dependencies {

6. scd4j url('http://download.jboss.org/wildfly/8.2.0.Final/wildfly-8.2.0.Final.zip'),

7. files('dependencies/cluster-checker.war')

8. }

Como é possível perceber na Listagem 1, inicialmente devemos definir a versão plugin do SCD4J e as dependências do projeto. Notem que neste nosso exemplo existem apenas duas dependências: (1) o arquivo zip do Wildfly no site da JBoss e (2) um arquivo local que será utilizado como exemplo para testar o cluster nos diferentes ambientes. É importante ressaltar que, em casos reais, a resolução de dependências baseada em URL (arquivo zip do Wildfly no nosso exemplo) não é aconselhada uma vez que o provedor do artefato pode modificar o caminho ou mesmo a versão do arquivo. Se isto acontecer, o nosso pacote de instalação pode parar de funcionar inesperadamente. O ideal é que ambas estas dependências estejam no seu repositório de binários pré existente (como, por exemplo, no Artifactory ou no Nexus).

Listagem 2. Arquivo build.gradle - configuração do SCD4J

1. scd4j {

2. install {

3. modules "wildfly"

4. config project.hasProperty("envconfig") ? envconfig : ""

5. }

6. settings {

7. template {

8. engine "mustache"

9. }

10. }

11. }

Após a declaração do plugin e das dependências do projeto, precisamos ainda definir o que o SCD4J deve fazer quando executado. Como podemos ver na linha 3 da Listagem 2, ao executar o nosso projeto (gradlew.bat no Windows ou ./gradlew no Linux), o módulo wildfly será instalado. Além disto, também precisamos informar qual é o arquivo de configuração que deve ser utilizado. Como no nosso exemplo temos três ambientes diferentes (desenvolvimento, testes e produção) não podemos colocar um valor estático para a propriedade config (linha 4). Por este motivo, capturamos dinamicamente o valor setado em envconf e atribuímos este valor esta propriedade. Isto nos dá a liberdade de definir qual arquivo de configuração deve ser utilizado na hora do disparo da instalação.

Além das definições referentes ao que deve ser instalado, também podemos realizar outras configurações mais avançadas. Como por exemplo, podemos definir que optamos por utilizar o motor de templates mustache no lugar do padrão, que é Groovy Templates (linha 8 da mesma listagem).

Um arquivo de configuração para cada ambiente

Como mencionamos na seção anterior, a propriedade config da configuração do SCD4J deve conter o arquivo que gostaríamos de utilizar durante a instalação. Tal arquivo é, na verdade, um arquivo de propriedades onde as variáveis que requerem diferentes valores entre os ambientes são adicionadas. Por exemplo, o usuário e senha de um banco de dados devem ser diferentes entre os ambientes de desenvolvimento e produção para evitarmos incidentes. Sendo assim, para garantir a correta instalação nas diferentes máquinas, devemos ter, pelo menos, um arquivo .conf para cada ambiente.

Figura 3. Arquivos de configurações

Na Figura 3 podemos perceber que, para este exemplo, uma convenção de nomes foi criada com o intuito de facilmente identificar os ambientes e as portas HTTP nas quais os servidores serão instalados. Notem também que, em desenvolvimento, iremos instalar apenas uma instância do Wildfly. Já em testes e produção iremos instalar duas instâncias em cada máquina do cluster.

Listagem 3. Arquivo desenv-8080.conf

1. env=development

2. baseDir=/opt/scd4j-examples/WildflyClusterInstallation

3. portOffset=0

4. adminPassword=29a28757d330005880b98ead71ba2aa8

Para entender um pouco melhor como isto funciona, vamos analisar o arquivo desenv-8080.conf (Listagem 3). Na linha 1 estamos dizendo que o ambiente é de desenvolvimento (env é a única propriedade que é obrigatória e os possíveis valores são: development, testing, staging ou production). Já, na linha 2, definimos qual vai ser o diretório base onde a instalação deverá ocorrer. Na linha 3, o offset das portas do servidor e, por fim, na linha 4 a senha do console do Wildfly.

É válido ressaltar que, nos ambientes de homologação e produção é importante proteger informações sensíveis como, por exemplo, a senha que temos neste arquivo. Por motivos de escopo, não vamos cobrir estes detalhes, mas sugerimos fortemente que o leitor visite a documentação do SCD4J e leia a seção Encryption Tools. Além de não deixar aberta esta informação, o ato de criptografar uma propriedade também é uma medida de segurança. Uma vez que, quando existem propriedades criptografadas, apenas pessoas com a senha de descriptografia podem executar a instalação.

Entendendo as pré e pós condições

Em um projeto SCD4J, todos os módulos devem estar dentro do diretório modules. E, como podemos perceber na Figura 4, todos projetos possuem um arquivo gancho chamado Module.hook. Este arquivo é muito importante pois ele define o que deve ser executado antes e depois da instalação do módulo em si.

Figura 4. O módulo wildfly

Além do Module.hook, também precisamos colocar dentro do diretório do módulo todos os arquivos que devem ser instalados. Isto é, podemos ter arquivos para serem copiados, deletados ou mesmo templates para serem alterados em tempo de execução. Vamos começar visitando o arquivo Module.hook.

Listagem 4. Arquivo Module.hook - definições pre e post

1. ...

2. pre {

3. installOrStopWildfly()

4. installServerNode()

5. ...

6. }

7. post {

8. deployApp()

9. startWildfly()

10. }

11. ...

Na Listagem 4 temos as duas definições mais importantes deste arquivo. Dentro do escopo da definição pre{..} devemos programar o que deve ser realizado antes da cópia de arquivos do módulo iniciar. No nosso exemplo estamos chamando alguns métodos que serão explicados em seguida. Já, dentro da definição post{..} devemos programar o que deve ser realizado após a cópia de arquivos ser finalizada. Note que, uma vez que os hooks são escritos na linguagem Groovy, qualquer lógica programação desejada pode ser utilizada. Apenas como exemplo, vamos analisar alguns métodos implementados neste arquivo. Para mais detalhes, sugerimos ao leitor que estude o código-fonte encontrado no final deste artigo.

Listagem 5. Arquivo Module.hook - métodos auxiliares da definição pre

1. ...

2. void installOrStopWildfly(){

3. if(!exists(wildflyDir)) {

4. checkJava()

5. unzip('wildfly-8.2.0.Final.zip', baseDir)

6. } else {

7. log("Wildfly is already installed at $wildflyDir")

8. def st = status("wildfly-${httpPort}");

9. if(st.contains("running")) {

10. stop("wildfly-${httpPort}")

11. }

12. }

13. }

14.

15. void installServerNode() {

16. if(!exists(nodeDir)) {

17. cp("$wildflyDir/standalone", nodeDir)

18. }

19. }

20. ...

Antes de iniciar a instalação dos arquivos o SCD4J irá executar os métodos installOrStopWildfly() e installServerNode() (ver Listagem 4). O que o primeiro método faz (linhas 2-13 da Listagem 5) é, verificar se o servidor já está instalado e, caso negativo, descompacta a dependência do Wildfly (definida na Listagem 1) dentro do diretório escolhido. Caso o servidor já esteja instalado e esteja rodando, um sinal para parar o processo é então enviado. Note que, neste exemplo, diversos métodos utilitários (tais como: unzip, log, stop, cp, etc) são utilizados. Estes métodos são definidos na classe ModuleHook.java e, para mais detalhes e outras opções, sugerimos que a sua documentação JavaDoc seja analisada.

O segundo método (linhas 15-19 da Listagem 5) simplesmente verifica se o nodo do servidor já foi instalado anteriormente, caso negativo uma cópia do diretório standalone é realizada. Vale destacar que, por motivos de simplicidade e para garantir a maior similaridade possível entre os ambientes, não utilizamos, neste exemplo, o suporte de domain (cluster com gerência centralizada) oferecido pelo Wildfly. Contudo, apesar de algumas pessoas não saberem disto, é possível criar qualquer tipo de cluster utilizando apenas o suporte standalone do servidor.

Listagem 6. Arquivo Module.hook - métodos auxiliares da definição post

1. ...

2. void deployApp(){

3. cp("cluster-checker.war", "$nodeDir/deployments")

4. }

6. void startWildfly(){

7. if(isLinux()) {

8. ...

9. start("wildfly-${httpPort}")

10. registryOnBoot("wildfly-${httpPort}")

11. }

12. }

13. ...

Já na Listagem 6, temos os métodos chamados na definição post{..} (ver Listagem 4), isto é, após a cópia de arquivos ser finalizada. O primeiro deles, linhas 2-4, simplesmente copia a dependência local da nossa aplicação (ver Listagem 1) para dentro do diretório deployments do nodo recém instalado (completando assim a implantação do sistema que iremos utilizar para testar o nosso cluster). Por fim, o segundo método, linhas 6-12, inicia e registra (no startup da máquina) o serviço do Wildfly apenas quando estamos rodando o instalador sobre o sistema operacional Linux. Se revisitarmos o nosso modelo definido na Figura 1, iremos perceber que apenas o ambiente do desenvolvedor roda em Windows e, como neste ambiente o serviço não deve subir automaticamente, optamos em realizar esta configuração apenas em testes e em produção.

Entendendo como funciona a cópia de arquivos

Após definido o que será executado antes e depois da instalação, precisamos definir ainda quais arquivos serão instalados/configurados durante a execução do SCD4J. Para isto, vamos analisar a Figura 5.

Figura 5. Detalhes do módulo Wildfly

Note que dentro do diretório do módulo Wildfly temos, além do nosso já conhecido Module.hook, diversos outros arquivos e diretórios. Estes arquivos e diretórios serão instalados no sistema operacional assim que a execução da definição pre{..} do Module.hook finalizar. Para que esta instalação ocorra corretamente é necessário que estes arquivos sejam colocados nos diretórios corretos. Isto é, o caminho do diretório dentro do módulo deve ser o mesmo que o arquivo terá no sistema operacional após finalizar a execução do SCD4J. Para definir estes caminhos existem duas possibilidades: (1) informar o caminho completo ou (2) utilizar uma variável que pode ser setada via programação (em arquivos .hook) ou estaticamente (em arquivos .conf). Neste nosso exemplo, decidimos utilizar variáveis (@wildflyDir@ e @httpPort@) uma vez que os diretórios de instalação podem variar entre as máquinas Windows e os servidores Linux.

As configurações mais simples são as dos arquivos que precisam ser apenas copiados para o diretório de destino. Neste caso, basta colocar o arquivo no diretório correto dentro do instalador (ex. mgmt-groups.properties na Figura 5). Porém, existem casos que é só possível saber o que deve ser configurado em tempo de execução, nestes casos devemos adicionar a extensão .tmpl ao final do arquivo (ex. mgmt-users.properties.tmpl e standalone.xml.tmpl). O que o SCD4J faz ao verificar que um arquivo termina com .tmpl é resolver o template utilizando a engine configurada previamente (Listagem 2). Somente após esta resolução o arquivo é copiado para o seu destino final.

Listagem 7. Trecho do arquivo standalone.xml.tmpl

1. 1. ...

2. </subsystem>

3. {{^isDevelopment}}

4. <subsystem xmlns="urn:jboss:domain:modcluster:1.2">

5. ...

6. </subsystem>

7. {{/isDevelopment}}

8. <subsystem xmlns="urn:jboss:domain:naming:2.0">

9. ...

Apenas para elucidar a utilização de templates, temos o exemplo da Listagem 7. Como podemos perceber, em arquivos .tmpl, devemos sempre utilizar a linguagem do motor de templates definida no arquivo build.gradle; no nosso caso, mustache. Por exemplo, nesta mesma listagem, estamos dizendo que não queremos instalar a definição do subsystem de modcluster do Wildfly nas máquinas de desenvolvimento (linhas 3-7), uma vez que o desenvolvedor não precisa de um balanceador de carga para realizar as suas atividades diárias de programação.

Por fim, ainda existem situações em que precisamos adicionar ganchos (hooks) de pré e pós condições nas instalações dos arquivos. Por exemplo, veja standalone.xml.tmpl.hook na Figura 5. Este arquivo, nada mais é do que um gancho de pré e pós condição para o arquivo standalone.xml.tmpl. Isto é, dentro dele temos as mesmas definições pre{..} e post{..} que possuímos no Module.hook. A principal diferença que temos entre o hook de módulo e os hooks de arquivos é que na implementação do primeiro devemos utilizar os métodos auxiliares definidos na classe ModuleHook.java, enquanto, no segundo, os métodos auxiliares são definidos na classe FileHook.java.

Executando a instalação

Para executar o nosso projeto, primeiramente precisamos saber em qual ambiente o instalador está sendo rodado. Esta informação, apesar de parecer ingênua, é muito importante pois é necessário informar no comando o nome do arquivo de configuração correto. Com isto bem claro, basta digitar: ./gradlew -Penvconfig=wildfly/NOME_ARQUIVO.conf na linha de comando.

Box 3. Trava de segurança

É importante mencionar que o SCD4J possui uma trava de segurança que evita que, por exemplo, um arquivo de configuração de testes rode em ambientes de produção. Para saber mais como utilizar esta trava, visite a documentação da ferramenta na seção Packaging and Distributing, subseção Defining Environments.

Apenas como um exemplo, iremos enumerar os passos para rodar o nosso projeto em uma máquina Linux simulando a instalação de um ambiente de testes:

Inicialmente devemos estar logado com o usuário root no sistema operacional;
Em seguida, precisamos baixar o código-fonte do exemplo (ver link no final deste artigo);
Com o código-fonte salvo em um diretório qualquer, iremos instalar o primeiro nodo do cluster digitando o seguinte comando na raiz do projeto (diretório onde se encontra o arquivo build.gradle): ./gradlew -Penvconfig=wildfly/testing-8180.conf
Ao terminar a instalação do primeiro nodo partirmos para o segundo executando o comando: ./gradlew -Penvconfig=wildfly/testing-8181.conf
Neste momento já possuímos ambos os nodos do cluster completamente configurados e prontos para receber requisições. Nos resta então verificar se tudo aconteceu como o esperado. Para tal, precisamos abrir o navegador e acessar o sistema instalado no primeiro nodo do nosso cluster: http://localhost:8180/cluster-checker
Ao pressionar a tecla F5 algumas vezes iremos perceber que a contagem do número de requisições irá aumentar respectivamente (Figura 6).

Figura 6. Acessado a instância 8180

Uma vez que está tudo certo com a instalação do primeiro nodo, podemos verificar o segundo abrindo uma outra aba navegador e digitando: http://localhost:8181/cluster-checker.

Figura 7. Acessado a instância 8181

Neste momento, se nenhum erro ocorreu durante a instalação, iremos perceber que o número de requisições continua de onde a primeira instância parou (Figura 7). Isto comprova que a replicação de sessão web está funcionando corretamente e, portanto, que o nosso cluster está no ar rodando sem problema algum.

Considerações finais

Como foi mencionado no inicio deste artigo, a automatização de ambientes está deixando de ser um diferencial entre as empresas para se tornar um padrão dentro do mundo de desenvolvimento de software. Infelizmente, as ferramentas mais conhecidas são muito complexas e exigem uma curva de aprendizado muito maior do que as pequenas e médias empresas querem pagar. Com este intuito, surgiu uma ferramenta chamada SCD4J que visa de cobrir este gap carente em automatização no mundo Java.

No decorrer deste artigo, discutimos como realizar a instalação de um cluster de Wildfly utilizando tal ferramenta. Porém, é importante ressaltar que esta instalação é apenas um exemplo do que pode ser feito automaticamente. Além disto, não podemos esquecer que as empresas devem focar no objetivo final, que é a instalação completa dos ambientes diretamente amarrada os seu processo de entrega contínua. Isto é, a cada build realizado os ambientes devem ser re-instalados. Esta estratégia garante que não apenas o sistema seja testado inúmeras vezes mas, também, a instalação do mesmo. Esta solução, além de trazer tranquilidade e confiança para todos os integrantes do desenvolvimento, também evita surpresas na hora do going live.

Links

Documentação do SCD4J

https://github.com/scd4j/gradle-plugins/wiki

Código-fonte utilizado como exemplo neste artigo

https://github.com/scd4j/gradle-plugins/tree/master/scd4j-examples/WildflyClusterInstallation

Livro sobre Contínuos Delivery do Jez Humble

http://www.amazon.com/gp/product/0321601912

Servidor de Aplicação Wildfly

http://www.wildfly.org/

Motor de templates mustache

http://mustache.github.io

Puppet

http://puppetlabs.com/

Chef

www.chef.io

Fernando Rubbo possui cerca de 14 anos de experiência em desenvolvimento de software. Atualmente trabalha como Ads Solution Engineer no Google e é mestre em Engenharia de Software pela Universidade Federal do Rio Grande do Sul (UFRGS). Fernando já vivenciou praticamente todas as áreas do ciclo do desenvolvimento de software atuando principalmente como consultor, líder técnico, arquiteto de software e, também, professor universitário.

SCD4J: Continuous Delivery has never been so simple

2015-01-25T16:11:00.000-08:00

Simple Continuous Delivery for Java and Groovy Developer (SCD4J) is an automation platform for configuring and installing your Web IT infrastructure. With just one command you can install clusters and deploy applications in any stage of your software development process. Note that SCD4J is not another option for Puppet or Chef. It is, actually, a simpler option than those tools.

The main advantages of using SCD4J over its competitors are:

Designed with a very specific purpose in mind

Small and medium enterprises which have opted in to use Java technologies and Continuous Delivery

No environment configuration

No need to install and configure coordinator servers
No need to install agents on clients

Small learning curve for Java and Groovy Developers
Built upon modern and well known Java technologies
Smooth integration with your existing binary repositories (i.e. Artifactory, Nexus, etc)
Very simple architectural model

Use module and files hooks to implement the automation logic

Developers get closer to sysadmins once they can quickly and easily jump into the infrastructure coding

Infrastructure Automation

After the 2010’s Continuous Delivery boom, too much has been talked about how to automate builds in order to anticipate issues, to assemble higher quality software and to ensure the creation of more reliable applications. However, fewer resources on the internet focus on the infrastructure automation.

It is important to say that one of the main issues I’ve seen during all the time I’ve worked as a Software Consultant is the lack of infrastructure standards (mainly the lack of standards between different stages of the software development process) and the error prone manual configurations performed by sysadmins. Both of these problems are well addressed using an automation infrastructure tool. Unfortunately, current well known solutions are too complex for non programming sysadmins and have a huge learning curve for Java developers. Making this kind of adoption, in rushy companies, somewhat difficult.

That is where SCD4J shines. Its simple architecture and specific purpose design makes it a very good match for pairing the knowledge of sysadmins and Java developers to build your company's Web infrastructure quickly.

SCD4J: creating a new project

To start a new project, first we must create a file called build.gradle, as show in the below example:

plugins {

id "com.datamaio.scd4j" version "0.7.5"

}

scd4j {

install {

modules "my_first_module" // should be any dir into modules dir

config "my_first_config.conf" // should be any properties file into config dir

}

Then, we may run the task newproject of our build.gradle using the installed gradle build tool.

Note: We need to have Gradle 2.1 (or higher) installed just for the first run. After that, SCD4J will automatically install a Gradle wrapper. Please, see Requirements for more details.

Once the execution has finished we will have the following directory structure created:

build.gradle

config/

my_first_config.conf

modules/

my_first_module/

Module.hook
gradlew

gradlew.bat

gradle/...

The config dir contains the configuration files. In our case the my_first_config.conf file (i.e. a Java properties file) in which we must put all the variables that will be used by our modules.

The modules dir contains a new module called my_first_module. This is the place where we will implement our automation. Note that the newproject task also created a file called Module.hook, which we are going to talk a little bit more later in this article.

Finally, we can see a gradle directory and gradlew and gradlew.bat files. Those are the gradle wrapper and, from now one, you can pack this project and run it in any machine without needing to install gradle.

SCD4J: basics

Similarly to any other infrastructure automation tool you can think about SCD4J as a “smart copy tool”. In which we should add files using the full path of the target environment. For example, suppose we would like to install a file called standalone.xml into the directory /opt/wildfly/standalone/configuration, then we should simply add this file in our my_first_module as shown in the below example:

build.gradle

config/

my_first_config.conf

modules/

my_first_module/

/opt/wildfly/standalone/configuration/standalone.xml

...

Running again our project, we can notice that SCD4J has simply copied our file to the right folder. But, in real situations, we want to install slightly different standalone.xml files depending on the environment we are running on. For example, we don’t want to mess with our production database. So, if we are installing the testing environment we need to set a given database URL and password, similarly, if we are installing a development environment we need to set another URL and another password. In order accomplish this we must put those two variable in the my_first_config.conf file and tell SCD4J that this file is a template. For that, we just need append .tmpl at the end of the file name.

build.gradle

config/

my_first_config.conf

modules/

my_first_module/

/opt/wildfly/standalone/configuration/standalone.xml.tmpl

...

This way, SCD4J understands that this file will need to be modified during runtime and then we can use one of the existing template options (Groovy Template is the default) to resolve the logic you would like to implement.

It is important to say that for each environment we run our project on we must provide different config files. In order to understand how to do that, take a look at packaging and distribution session of our documentation

Ok... But what about more complex installations?!? Lets say we would like to dynamically generate http port number based on the server and environment we are running on. For those kind of logic, SCD4J offers .hook files.

build.gradle

config/

my_first_config.conf

modules/

my_first_module/

/opt/wildfly/standalone/configuration/standalone.xml.tmpl

/opt/wildfly/standalone/configuration/standalone.xml.tmpl.hook

...

As we can see in the above example, we need to create a file with exactly the same name of the original one appending .hook at end. Note that this file is a groovy script in which you can define pre and post implementations. For example:

pre {

// write here the code to execute before the file installation.

def httpPort = dynamicallyBuildHttpPort()

set(“port”, httpPort) // The port variable will be accessible in the post and in the .tmpl file

}

post {

// write here the code to execute after the file installation.

// usually changes file permission, changes file owner, rename it, etc
}

The same way we can define a hook for a file, we can also define a hook for a module. For that, we need to create a file named as Module.hook in my_first_module dir. It is important to say that the pre definition must return a constant CONTINUE or ABORT. If nothing is returned, CONTINUE is assumed (in <FileName>.hook the constant SKIP_FILE is also allowed). Take the bellow code as an example:

pre{

// put here your pre module installation logic

if( isLinux() ) {

return CONTINUE

}

return ABORT

}

post{

// put here your post module installation logic

}

The above example is saying that the module will be executed only on Linux boxes. If we run this module against any other operational system, the installation will be aborted.

It is important to say that SCD4J provides many helpful functions which can be used in the Module.hook and any <FileName>.hook. Those functions allow us to install packages, change file permissions, start/stop a service, run OS specific commands, and so one. To have a complete understanding of which helpers we can use into hooks, take a look at ModuleHook and FileHook public APIs, respectively.

Conclusion

Before finish this short introduction to SCD4J we would like to highlight the core benefiting of using it to install and configure your Web infrastructure:

Manage IT risks in a simple and efficient way
Repeatability
People's time is precious, so use it for smarter tasks
Spread delivery knowledge
Infrastructure fully documented
Documentation will never get out of date

We strongly suggest you to take a look at the project documentation and examples. There you will find more complex options and examples (such as the installation of a wildfly cluster). So that you can quickly learn how to implement real world infrastructure automation.

Thanks for the interest of making the software delivery somewhat more professional. Hope you enjoy our tool.

Sincerely, SCD4J Team

Short Story

Some years ago I was called to help a big company to improve their software quality. Shortly I discovery that the main problem was not the software itself but the process to develop and delivery it. These issues led them to a lot of difficulties to put new versions in production. That situation has generated a real war between IT and Development teams, which, of course, tried to blame on each other. It took me some years to convince those managers to make sysadmins and developers work all together. Recently, more than one year have passed, I accidentally met one of the integration guys who told me that the last time they had an issue in production was before using our tools. That is one of the reasons why I choose to rewrite and publish it as a free software. Maybe other companies may get such benefits from it.

Using JAX-RS 2.0 to build an app from scratch.

2013-12-19T16:52:00.000-08:00

Hi every one,

I would like to share with my friends my last article about JAX-RS 2.0. This article has been published this month (December) in the Brazilian Java Magazine and it shows, with a practical example, how to develop a full app using the new stuff of this spec.

Unfortunately, this time I could not share its content, but for those who live in Brazil I would recommend to buy one magazine in the nearest newsstand. :-). For those who live outside of Brazil I can only share the source code with you :-(

source code: https://github.com/fbrubbo/BrazilianJavaMagazine

Anyway.. Thanks for reading.

Running Websockets on OpenShift

2013-08-11T19:35:00.000-07:00

WebSocket is a pretty new and important technology in which it is possible to send message frames from server to clients without opening multiple HTTP connections. Although it is has been proven to be a solid and consolidated specification, running websockets in the existing cloud platforms is still a pain in the ass. The main cloud players do not provide enough support for WebSocket protocol due to the lack of application servers’ readiness and due to other issues related to the routing layer.

Knowing those difficulties, I’ve decide to post the steps to help eager people to use Red Hat OpenShift PaaS to run a websocket apps in the cloud. So, let's start from the beginning.

First of all, you must sign up in the OpenShift web site
Then you must add a new application

Pick Do-It-Yourself (DIY) cartridge

This will allow you to use a “pure Linux server” to do whatever you would like to do. In the specific case of this post, we are going to install an embedded web application in order to provide the required support for websockets.

Once DIY cartridge have been selected, you must fill out the application name and click on ‘Create Application’ button
Finally, do a ‘git clone’ according instructions

At this point, we have already prepared our Linux server on the cloud to install our application according our needs. Now, let’s make the required changes in the working copy we have just cloned from OpenShift master repository.

The changes are:

Update the shell script file which will be used by OpenShift to start our app

File: .openshift/action_hoooks/start
Content:

#!/bin/bash

# The logic to start up your application should be put in this

# script. The application will work only if it binds to

# $OPENSHIFT_DIY_IP:8080

echo "Starting WebSocketExample............. $OPENSHIFT_DIY_IP:$OPENSHIFT_DIY_PORT"

nohup java -jar $OPENSHIFT_REPO_DIR/diy/WebSocketExample.war > $OPENSHIFT_DIY_DIR/logs/server.log 2>&1 &

echo "Started WebSocketExample............. $OPENSHIFT_DIY_IP:$OPENSHIFT_DIY_PORT"

Update the shell script file which will be used by OpenShift to stop your app

File: .openshift/action_hoooks/stop
Content:

#!/bin/bash

source $OPENSHIFT_CARTRIDGE_SDK_BASH

echo "Stopping WebSocketExample............."

if [ -z "$(ps -ef | grep WebSocketExample.war | grep -v grep)" ]

then

client_result "WebSocketExample Application is already stopped"

else

echo "Killing WebSocketExample............."

kill `ps -ef | grep WebSocketExample.war | grep -v grep | awk '{ print $2 }'` > /dev/null 2>&1

echo "Stopped WebSocketExample............."

Add the embedded jetty application in the ‘Do It Yourself’ directory

Directory: diy

In this example, we have used the ‘WebSocketExample.war’ app name in order to match out our start/stop scripts.
Important note about the embedded jetty application file:

There are some peculiaridades in the implementation of the class com.embedded.JettyStarter (shown below in red) which must be changed in order to bind the embedded app to the right IP and PORT number in the OpenShift platform.
Content:

public class JettyStarter {

public static void main(String[] args) throws Exception{

ProtectionDomain domain = JettyStarter.class.getProtectionDomain();

URL location = domain.getCodeSource().getLocation();

// create a web app and configure it to the root context of the server

WebAppContext webapp = new WebAppContext();

webapp.setDescriptor("WEB-INF/web.xml");

webapp.setConfigurations(new Configuration[]{ new AnnotationConfiguration()

, new WebXmlConfiguration(), new WebInfConfiguration(), new MetaInfConfiguration()

//, new FragmentConfiguration(), new EnvConfiguration(), new PlusConfiguration()

});

webapp.setContextPath("/");

webapp.setWar(location.toExternalForm());

// starts the embedded server and bind it to openshift variables

String host = System.getenv("OPENSHIFT_DIY_IP");

String port = System.getenv("OPENSHIFT_DIY_PORT");

System.out.println(host + ":" + port);

InetSocketAddress sa = new InetSocketAddress(getByName(host), valueOf(port));

Server server = new Server(sa);

server.setHandler(webapp);

server.start();

server.join();

}

After script files have been configured and the embedded application has been put in the ‘diy’ directory, you must commit and push changes back to the master repository.

The commands are:

git commit -m 'My changes'
git push

As soon as the push finishes, the stop/start scripts will run

If you notice it didn’t run accordingly, remote access your openshift server and check if everything is ok. Remember: it is Red Hat Linux, so you should be ok checking your stuff.
Try to check if start/stop scripts have execution permission (not sure why, but I have this issue in some experiments).

ls -lt $OPENSHIFT_REPO_DIR/.openshift/action_hooks/

Finally, are done. \o/

Access 'ws://[YOUR APP].rhcloud.com:8000/example' using your favorite WebSocket tool and check it by yourself.

Important: due to issues related to the routing layer, OpenShift platform allow websockets only in the port number 8000.

Don't have a WebSocket tool? Take a look on ‘Advanced REST client’ for Chrome

The openshift configuration used in this example can be found here.

Thanks for reading

Points are all about sizing. So, avoid the word estimation!

2013-07-21T14:54:00.001-07:00

How should we punctuate a certain task/story?

It is a recurrent discussion in our company and clients. From time to time I found myself arguing about this subject. This made me write down an example I’m used to put in those discussions.

Suppose you work in a team which is responsible for making article summaries for a newspaper. Then, someone in the team heard about a technique which uses points for sizing the amount of work. The team, though, decided to implement this technique and, after some discussions, everybody agreed to make sizing meetings (the event in which each person on the team give points to articles) as simple as possible. For that, they have chosen that each English full written article page would receive 1 (one) point.

In my opinion, it was a very good start for this team. Don't you think?

Then, the first article arrived. It had 10 written pages and then the team gave it 10 points. Then, the second article came with 20 pages, but 6 of them have lots of images. So, the team discussed a little bit and they agreed to give it 17 points.

Up until now, pretty simple. No?!?

But, some Portuguese articles became to arrive. The first one was a paper with 15 full written pages. How many points do you think the team would give to this article?

Think a few moments about it. I’ll be waiting right here. ;-)

Probably you would be thinking: “as a member of this team, I would give it more than 15 points because I’m not fluent in Portuguese and, therefore, it will take me more time to finish my work”.

If you have these thoughts I would say you are not sizing. You are, actually, trying to estimate.

But... What is the difference?

Sizing

The amount of work of a specific task

Estimation

Although sizing is a kind of estimation, the word estimation usually remember the amount of time, money and resources you will spend to perform a specific task

So... If more than 15 points are not appropriated, what would be a good number?

15, of course.

But why?

When you are sizing you must not consider complexity, time, money, resources and etc. Moreover, you should not take into account that you must learn something in order to perform a defined task (which is the case in this fake example).

But, again... Why?

Time and complexity are very subjective measures

For example, suppose you have a member of your team which is fluent in Portuguese and another which has never read a word in such a language. If they take into account time and/or complexity, they will think differently while punctuating. In other words, they may never reach a consensus about the amount of work.

The amount of time, complexity and money depends on who will be performing the task

In the most of cases, during the sizing meeting, we do not know exactly who will be responsible to perform a defined task. So, we must think in a generic way. For that, the most effective strategy is to relative tasks/stories to each other (i.e. sizing).

Points help focus on team performance, not on individuals

The idea behind points is to help give focus on the overall team result/capacity, and not on a specific team member performance.

The team velocity is not possible to be measured correctly if time/complexity/etc are considered

Suppose a year ago your team has given 20 points to a 10 pages article written in Portuguese. Considering this team got some fluency in that language, nowadays an article with the same number of pages is used to receive 15 points. At first glance it seems ok to give fewer points to it because the work is finished faster. However, if you do that, the velocity (commonly used in agile methodologies) will decrease (i.e. if a year ago you were able to deliver 200 points per month, now you can make only 150 - 180). If the learning curve was not taking into account the team velocity would be increased showing they are able to do more stuff in the same amount of time.

Making it short:

Points are all about sizing. If a year ago you gave to a specific task 10 points; right now, whenever you have to repeat that task, you must give it exactly the same punctuation. Even knowing you became an expert in this subject and that you will finish it faster. Finishing it faster means you have improved your velocity and, therefore, you are getting better and better in what you do.

So, whenever you are using points, avoid the word estimation. This word gives to people an incorrect impression while punctuating.

Please, leave your comments about this subject. I love to hear other people opinions.

Thanks for reading.

Fixing Run-Jetty-Run Eclipse Plugin to Run Websockets

2013-07-14T13:15:00.000-07:00

Run-Jetty-Run is an Eclipse plugin for running and debugging web apps using Jetty container. Although it has been a great and useful tool in the past days, lately, when I was playing around with websockets, I’ve found it was not working properly as it should.

It is important to say that the current stable build of Run-Jetty-Run plugin does not support the latest Jetty version (i.e. 9.0.4), so it became necessary to get its nightly build. Once I downloaded it, I've noticed it was not up-to-date either. Actually the last nightly build is pretty old too (the build date is 01/02/2013) and, unfortunately, it only supports Jetty 9.0.0.M3.

So, If you are facing the below issue or if you are a curious person, I would suggest you to keep reading.

java.util.ServiceConfigurationError: org.eclipse.jetty.websocket.servlet.WebSocketServletFactory: Provider org.eclipse.jetty.websocket.server.WebSocketServerFactory not found

at java.util.ServiceLoader.fail(ServiceLoader.java:231)

at java.util.ServiceLoader.access$300(ServiceLoader.java:181)

at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:365)

at java.util.ServiceLoader$1.next(ServiceLoader.java:445)

at org.eclipse.jetty.websocket.servlet.WebSocketServlet.init(WebSocketServlet.java:136)

at javax.servlet.GenericServlet.init(GenericServlet.java:242)

at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:516)

at org.eclipse.jetty.servlet.ServletHolder.getServlet(ServletHolder.java:398)

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:642)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:445)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138)

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:564)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:213)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1054)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:988)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

at org.eclipse.jetty.server.Server.handle(Server.java:410)

at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:245)

at org.eclipse.jetty.server.HttpConnection$1.run(HttpConnection.java:75)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:597)

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:528)

at java.lang.Thread.run(Thread.java:722)

Once the Run-Jetty-Run team seems to not be working anymore and since there were some changes in the internal Jetty API, I’ve decided to fix it by myself and to post the result to make life simple for everyone trying to run websockets into Eclipse.

It is important to say that this fix was tested only in the Eclipse Kepler. So, in order to avoid other issues, ensure you have the latest version of Kepler IDE. After that, you can follow the following steps:

First of all, you must install the core of the Run-Jetty-Run plugin.

For that, go to Help -> Install New Software.
Then, enter the http://run-jetty-run.googlecode.com/svn/trunk/updatesite-nightly URL and install the Run Jetty Run Feature (Required)

Once the core plugin has being installed, you must install my customized support for running Jetty 9.0.4.

Download the file RunJettyRun904_byRubbo_1.0.0.201306302028.jar
Copy the downloaded jar into the /<YOU_ECLIPSE_KEPLER_HOME>/plugins
And, restart the eclipse

Finally, right click on your websocket app and choose RunAs -> Run Jetty

In Jetty tab, select the following Jetty Version: Jetty 9.0.4.by.Rubbo
In Jetty Classpath tab, add your project in the Custom Jetty Classpath

It is important to say that this step is only necessary because there is another bug in the core plugin which I didn’t want to lose my time to fix.

So, we are done!!

Click in the Run button and have fun..

Hope I’ve helped to make your life simple. ;-)

Thanks for reading.

Using WebSockets to implement Dashboards

2013-06-23T15:42:00.000-07:00

Dashboards are extremely important tools mostly used for monitoring systems and environments. The main idea of these tools is to keep clients up-to-date with arriving or changing data on the server side. For that, the most popular technique used in web applications is based on ajax polling. With ajax, the client polls the server for data every time the content of the page requires an update.

Once the idea of monitoring is to show actual data information, there are several problems with current ajax technique. The first one is scalability. The number of requests made to the server can be extremely high if the frequency of polling is set to a small value. Not only the server but also the network can become saturated with all those requests. On the other hand, if we set a high value for the pooling frequency, some information may be lost or delayed. Another problem is that the response may not contain any data. So, in these cases, ajax polling overloads the server for nothing.

Fortunately, in December 2011, a new protocol called WebSocket was defined. As the spec says: “The goal of this technology is to provide a mechanism for browser-based applications that need two-way communication with servers that does not rely on opening multiple HTTP connections”. In other words, with Websocket protocol it is possible to send message frames from server to clients, once these clients have established an initial conversation, of course.

Analyzing such a capability, it becomes clear that implementations of browser-based dashboards, chats, games, kanbans, planning pokers tools, etc should be rewritten to use WebSocket protocol. Therefore, the main idea of this post is to show how to implement a very simple dashboard using websockets.

First of all, I would refer my article about how to create and embedded war with Jetty and Gradle. To make this post short and simple, I will assume you have already read that post or that you have a good knowledge about these subjects.

Second, to start our example we will be adding support for websockets in an embedded Jetty. So, you have to create your build file called ‘build.gradle’ as shown below. Note the lines in red, which are the responsible to include the expected support.

// Tells gradle it is a war project which will be imported into eclipse wtp

apply plugin: 'war'

apply plugin: 'eclipse-wtp'

// Define souce code compatibility

sourceCompatibility = 1.7

targetCompatibility = 1.7

// Use maven repository

repositories {

mavenCentral()

}

// Fills out all dependencies which are necessary to start the embedded jetty into our war file

configurations {

embeddedJetty

}

dependencies {

embeddedJetty 'org.eclipse.jetty:jetty-servlet:+'

embeddedJetty 'org.eclipse.jetty:jetty-webapp:+'

embeddedJetty 'org.eclipse.jetty:jetty-jsp:+'
embeddedJetty 'org.eclipse.jetty:jetty-annotations:+'

// add support for jetty implementation of websockets
embeddedJetty 'org.eclipse.jetty.websocket:websocket-server:+'

compile 'org.glassfish:javax.json:+'

}

war.baseName = 'WebSockets'

war {

// unzip and add all jetty dependencies into the root of our war file

from {configurations.embeddedJetty.collect {

project.zipTree(it)

}

// remove signature and unnecessary files

exclude "META-INF/*.SF", "META-INF/*.RSA", "about.html", "about_files/**", "readme.txt", "plugin.properties", "jetty-dir.css"

// include only the classes which will be used to start Embedded Jetty

from "$buildDir/classes/main"

exclude "com/myapp/"

// tells the class to run when the generate war be executed using 'java -jar'

manifest { attributes 'Main-Class': 'com.embedded.JettyStarter' }

}

// Once you will need some basic api (e.i. servlet api) for compilation, add embeddedJetty dependencies for compilation

sourceSets.main.compileClasspath += configurations.embeddedJetty

// the same for eclipse classpath, so you can use it to edit your java files

eclipse {

classpath {

plusConfigurations += configurations.embeddedJetty

}

Unfortunately, Jetty 9 still does not support JSR 356 (JavaTM API for WebSocket). So we are going to use its current API in this example (which, by the way, is very similar to JSR 356). For that, initially we have to create a class for implementing our WebSocket as shown below.

import org.eclipse.jetty.websocket.api.Session;

import org.eclipse.jetty.websocket.api.annotations.OnWebSocketClose;

import org.eclipse.jetty.websocket.api.annotations.OnWebSocketConnect;

import org.eclipse.jetty.websocket.api.annotations.WebSocket;

@WebSocket

public class DashboardWebSocket {

@OnWebSocketClose

public void onClose(final Session session, int x, String text) {

EventGenerator.unregistry(new EventObserver(session));

}

@OnWebSocketConnect

public void onOpen(final Session session) {

EventGenerator.registry(new EventObserver(session));

}

The most important parts of the above class are the onOpen(..) and onClose(..) methods, which will be executed at any WebSocket open and close connections. In our example, each time a user decides to navigate to this app we will be registering his session as an event observer. Doing that, it is possible to broadcast to all registered sessions any event which has happened in the server side. In the same way, whenever that user closes the browser, the connection is lost and his session will be unregistered from the event observer list.

Besides the WebSocket itself, using Jetty API, it is necessary to implement a WebSocketServlet to map and registry such WebSocket. In our example, each time an URI /dashboard is used; it will be redirected for this servlet.

import org.eclipse.jetty.websocket.servlet.WebSocketServlet;

import org.eclipse.jetty.websocket.servlet.WebSocketServletFactory;

@SuppressWarnings("serial")

@WebServlet(name = "Dashboard WebSocket Servlet", urlPatterns = { "/dashboard" })

public class DashboardServet extends WebSocketServlet {

@Override

public void configure(WebSocketServletFactory factory) {

factory.register(DashboardWebSocket.class);

}

Once these steps are done, it is necessary to use the JavaScript API to open the initial connection and make it wait for new message frames from the server. Note, in the below JS code, that we have used the protocol ‘ws’ instead of ‘http’. It is necessary because WebSocket is, actually, a different protocol.

var load = function() {

var socket = new WebSocket("ws://localhost:8081/dashboard");

socket.onmessage = function(event) {

// show the event.data in your graph, table or whatever

}

</script>

In the above code, we presented a very simple implementation in which a WebSocket is opened and, on each server event, data is received as a parameter by the function onmessage(..). In below image it is possible to see a printscreen of the this app running. The points represent the event amount (for an example purpose we have used a random number from 0 to 100.000) and moment it have happened.

In short I would say WebSocket is pretty important technology for building modern web applications in which, in the near future, it will be largely used for enterprise software as well for general web applications. However, it is still complicate to assume websockets as an unique strategy to replace long polling features because there exists firewalls and proxies which does not understand/support ‘ws’ protocol yet. Besides that, in the Java world, there are still few application servers which provide support for JSR 356 standard. So, we still have to wait some more months to use the API specified in that standard (p.s.: Glassfish already provides such implementation).

If you would like to study and/or execute this example, you can download the full source code here. To start the dashbord, you have to build the war and then execute it with the following command line: ‘java -jar WebSockets.war’. This will start a Jetty server in port 8081, so you can access the http://localhost:8081/ url in the browser and see by yourself.

Moreover, if you would like to understand how websocket frames are sent to the browser, I would suggest you to open the ‘Chrome Developer tools’ and click in ‘Network’ tab. In there, you will be able to see that this example app will be making no ‘http’ requests. But only one ‘ws’ connection. Clicking in the ‘ws’ connection you will be able to see the message frames being sent back from the server as shown in the image below.

Thank you for reading!

Embedded war using Jetty and Gradle

2013-06-16T15:04:00.002-07:00

Recently I had the necessity to write an embedded war. At a first glance, I thought: “It is a pretty simple task, I’ve done it many times using Jetty”. Just some minutes later I remembered: “Wait a minute... I’ve used embedded Jetty servers into jar files, but how can I do the same using war files?”.

The answer of this question is a bit “tricky”. Actually, you have to create a war file which will have to play two different roles. The first one is responsible to start the embedded Jetty server and the second one is the web app itself. Take a look at the MyEmbeddedWar.war folder structure below to understand what I’m talking about:

MyEmbeddedWar.war

META-INF

MANIFEST.MF

containing: Main-Class: com.embedded.JettyStarter

Jetty classes (org.eclipse.jetty.*, org.apache.*, javax.*, etc)
My main class: com.embedded.JettyStarter
WEB-INF

classes

containing my app classes

web.xml

index.jsp

The blue files in the root of the war are those responsible to start the embedded Jetty server and the red ones represent the content of web app. Ok, once I’ve understood what need to be done, I’ve started to think how it could be done using gradle (for those which do not know what gradle is, I strongly recommend to take a look at http://www.gradle.org/). So, at the end of my experiments I’ve got the following:

First you need to create the file ‘build.gradle’ as shown below. I’ve added some comments on it to make this post as short and simple as possible.

// Tells gradle it is a war project which will be imported into eclipse wtp IDE

apply plugin: 'war'

apply plugin: 'eclipse-wtp'

// Define souce code compatibility

sourceCompatibility = 1.7

targetCompatibility = 1.7

// Use maven repository

repositories {

mavenCentral()

}

// Fills out all dependencies which are necessary to start the embedded jetty into our war file

configurations {

embeddedJetty

}

dependencies {

embeddedJetty 'org.eclipse.jetty:jetty-servlet:+'

embeddedJetty 'org.eclipse.jetty:jetty-webapp:+'

embeddedJetty 'org.eclipse.jetty:jetty-jsp:+'

embeddedJetty 'org.eclipse.jetty:jetty-annotations:+'

}

war.baseName = 'MyExecutableWar'

war {

// unzip and add all jetty dependencies into the root of our war file

from {configurations.embeddedJetty.collect {

project.zipTree(it)

}

// remove signature and unnecessary files

exclude "META-INF/*.SF", "META-INF/*.RSA", "about.html", "about_files/**", "readme.txt", "plugin.properties", "jetty-dir.css"

// include in the root of the war only the classes which will be used to start the embedded Jetty

from "$buildDir/classes/main"

exclude "com/myapp/"

// sets the main class to run when the generate war be executed using 'java -jar'

manifest { attributes 'Main-Class': 'com.embedded.JettyStarter' }

}

// Once you will need some basic api (servlet api, for example) for compilation, you need to add embeddedJetty dependencies for compilation

sourceSets.main.compileClasspath += configurations.embeddedJetty

// you need to do the same for eclipse classpath, so you can use it to edit your java files

eclipse {

classpath {

plusConfigurations += configurations.embeddedJetty
// for gradle 2.5 use: plusConfigurations += [configurations.embeddedJetty]

}

Then, you have to create the main class to start our embedded Jetty (in this example, I’ve called it ‘com.embedded.JettyStarter'):

public class JettyStarter {

public static void main(String[] args) throws Exception{

ProtectionDomain domain = JettyStarter.class.getProtectionDomain();

URL location = domain.getCodeSource().getLocation();

// create a web app and configure it to the root context of the server

WebAppContext webapp = new WebAppContext();

webapp.setDescriptor("WEB-INF/web.xml");

webapp.setConfigurations(new Configuration[]{ new AnnotationConfiguration()

, new WebXmlConfiguration(), new WebInfConfiguration(), new MetaInfConfiguration() });

webapp.setContextPath("/");

webapp.setWar(location.toExternalForm());

// starts the embedded server and bind it on 8081 port

Server server = new Server(8081);

server.setHandler(webapp);

server.start();

server.join();

}

Once both the build and the main file are implemented, you can forget it is an embedded war and add as many JSPs and Servlets as you want. In order to show an example of an servlet being executed, I’ve added the following TimeServlet (which simply prints the current date).

@SuppressWarnings("serial")

@WebServlet(urlPatterns = { "/time" })

public class TimeServlet extends HttpServlet {

protected void doGet(HttpServletRequest request, HttpServletResponse response)

throws ServletException, IOException

{

ServletOutputStream out = response.getOutputStream();

out.print(new Date().toString());

out.flush();

}

and the following index.jsp, which prints a jsp expression in order to show a jsp being executed.

<html>

<head>

</head>

<body>

<h1>Hello: <%= "This is a jsp expression" %></h1>

<a href="time">Click here to execute TimeServlet</a>

</body>

</html>

Now, you can build your war and execute it using the following command line: ‘java -jar MyEmbeddedWar.war’. This will start the Jetty server in port 8081, so you can access the http://localhost:8081/ url in the browser.

The source code can be found here.

Good luck!

JBoss EAP 6.1.0 is out

2013-05-21T13:31:00.001-07:00

* For advantages and disadvantages of JBoss 7 (JBoss EAP 6) read Jboss 7: a love–hate relationship. *

The new version of JBoss EAP 6.1 was released yesterday (05/20/2013 10:08) and it includes lots of bug fixes and the following new features:

Global Valves in JBoss Web

JBoss Enterprise Application Platform 5 offered the capability to configure a global valve for the web container. In 6.0 it was only possible to configure a valve on a single application basis. For JBoss Enterprise Application Platform 6.1 the global valve capability has been restored.

Improved Initial Context options for EJB Clients

JBoss Enterprise Application Platform 6.1 provides a method to configure an InitialContext with a Map and create an EjbClientContext scoped to this particular InitialContext. The method also enables this configuration without dependencies on JBoss specific classes.

Passing Additional Information to the Authorization Module

In JBoss Enterprise Application Platform 6.0 it was not possible to pass additional information to the authorization module, as Java EE Interceptors were not called until after the container interceptors, including the login module, were called. JBoss Enterprise Application Platform 6.1 has added the capability to inject Java EE Interceptors into the container interceptor chain.

Security Context Propagation and User Switching

JBoss Enterprise Application Platform 6.1 is providing Quickstarts to demonstrate to following capabilities:

ejb-security-interceptors - Demonstrates how interceptors can be used to switch the identity for EJB calls on a call by call basis.
ejb-security-interceptors - Demonstrates how Interceptors can now be injected into the server's interceptor chain to propagate security contexts from one EJB to another, and switch users of an EJB.

Enhanced System Property Substitution

JBoss Enterprise Application Platform 6.0.0 included property substitution as a capability of the attributes of the configuration, but the number of properties that were supported was limited. For JBoss Enterprise Application Platform 6.1 the number of attributes enabled for substitution has increased.

Deterministic Column Names with Hibernate

In earlier versions Hibernate was not using a deterministic algorithm to generate column aliases, limiting the benefit derived from the database cache. Aliases are now consistently named.

The most expected feature is the container interceptor. Please, see community docs for more details https://docs.jboss.org/author/display/AS72/Container+interceptors.

Here is JBoss EAP 6 component matrix :

Component	EAP 6.0.0	EAP 6.0.1	EAP 6.1.0
JBoss AS	7.1.2 Final	7.1.3 Final	7.2.0 Final
WELD	1.1.8.Final	1.1.10.Final	1.1.13.Final
Rest Easy	2.3.3.Final	2.3.4.Final	2.3.6.Final
mod_cluster	1.2.1.Final	1.2.3.Final	1.2.4.Final
Picketlink Federation	2.1.1.Final	2.1.3.Final	2.1.6.Final
JBoss Modules	1.1.2.Final	1.1.3.Final	1.2.0.Final
JBoss DMR	1.1.1.Final	1.1.1.Final	1.1.6.Final
JBoss Metadata	7.0.3.Final	7.0.4.Final	7.0.8.Final
JBoss XNIO	3.0.4.GA	3.0.7.GA	3.0.7.GA
PicketBox	4.0.9.Final	4.0.14.Final	4.0.17.Final
PicketBox Commons	1.0.0.Final	1.0.0.Final	1.1.0.Final
Hibernate Core	4.1.3.Final	4.1.6.Final	4.2.0.Final
Hibernate Infinispan (2LC)	4.1.3.Final	4.1.6.Final	4.2.0.Final
Hibernate Envers	4.1.3.Final	4.1.6.Final	4.2.0.Final
Hibernate Validator	4.2.0.Final	4.2.0.Final	4.3.1.Final
Hibernate JPA 2.0 API	1.0.1.Final	1.0.1.Final	1.0.1.Final
JBoss Web	7.0.16.Final	7.0.17.Final	7.2.0.Final
Infinispan	5.1.4.Final	5.1.8.Final	5.2.6.Final
IronJacamar (JCA 1.6)	1.0.11.Final	1.0.13.Final	1.0.17.Final
JGroups	3.0.9.Final	3.0.14.Final	3.2.7.Final
JBoss Transactions	4.16.4.Final	4.16.6.Final	4.17.4.Final
JBoss EJB Client	1.0.10.Final	1.0.11.Final	1.0.21.Final
JBoss EJB3 EXT API	2.0.0	2.0.0	2.0.0
JBoss WS API	1.0.0.GA	1.0.0.GA	1.0.0.GA
JBoss WS SPI	2.0.3.GA	2.0.4.GA	2.1.2.Final
JBoss WS Common	2.0.4.GA	2.0.4.GA	2.1.1.Final
JBoss WS Common Tools	1.0.1.GA	1.0.2.GA	1.1.0.Final
JBoss WS-CXF	4.0.4.GA	4.0.6.GA	4.1.3.Final
JBoss WS-Native (for jaxrpc only)	4.0.4.GA	4.0.6.GA	4.1.1.Final
JBoss VFS	3.1.0.Final	3.1.0.Final	3.1.0.Final
JBoss Remoting3	3.2.8.GA	3.2.14.GA	3.2.16.GA
JBoss Marshalling	1.3.14.GA	1.3.15.GA	1.3.16.GA
JBoss SASL	1.0.1.Final	1.0.3.Final	1.0.3.Final
JSF	1.2_15-b01-redhat-1	1.2_15-b01-redhat-2	1.2_15-b01-redhat-3
JSF2	2.1.7-redhat-1	2.1.13-redhat-1	2.1.19-redhat-1
JacORB	2.3.2-redhat-1	2.3.2-redhat-2	2.3.2-redhat-4
JBoss Security Negotiation	2.2.0.SP1	2.2.1.Final	2.2.5.Final
JBoss Management Console	1.3.1.Final	1.4.2.Final	1.5.2.Final
HornetQ	2.2.16.Final	2.2.23.Final	2.3.1.Final
Netty (in HornetQ and Inifnispan)	3.2.6.Final	3.2.6.Final	3.6.2.Final
JBoss Logging	3.1.1.GA	3.1.2.GA	3.1.2.GA
Apache CXF	2.4.6-redhat-1	2.4.9-redhat-2	2.6.6-redhat-3
Apache CXF XJC-Utils	2.4.0-redhat-1	2.4.0-redhat-2	2.6.0-redhat-1
Apache WSS4J (for WS)	1.6.5	1.6.7	1.6.9
Xalan	2.7.1-redhat-1	2.7.1-redhat-3	2.7.1-redhat-4
Xerces	2.9.1-redhat-1	2.9.1-redhat-3	2.9.1-redhat-4
JAXB	2.2.5	2.2.5-redhat-3	2.2.5-redhat-5
Application Platform 6.0.0 Apache Web Server	based on Apache 2.2.17	based on Apache 2.2.22	based on Apache 2.2.22

Is it possible to do Continuous Delivery using branches? We think so!

2013-04-17T18:15:00.000-07:00

In previous posts I’ve been written about why you should be doing Continuous Delivery and how to avoid branches using a very simple technique called Branch By Abstraction. As we have already discussed in these articles, there is nothing wrong with branches, the problem is all about merges. So, adopting toggles may help us to have only a unique development branch. But there are some difficulties around this adoption. After a pair of years trying to find an alternative for a client we have reached a solution which allowed us to do Continuous Delivery combining both branches, Feature Toggles and a bit of automation. The idea of this post is to expose our experience in order to allow other people to experiment and suggest improvements on this solution.

The problem

Working with branches has been shown much more complex than it should be. In short I would say: the bigger is your team, the more complex is your merge. The biggest the time between merges, the more conflicts will appear. Have done a refactoring?!? Merging will be a chaos.

Merges take us too much time and are an error prone alternative, mainly when we are talking about large development teams. So, why not using a unique development branch? Adopting an unique development branch knowing that the entire code may go to production at any moment may be an easy task for companies such as Google, Facebook, Amazon, Flicker, etc (of course, they have the best software engineering the money can pay). But, in real world (a.k.a. enterprise world) it seem to be much more complex. Why? In my experience of years developing software for enterprises it is easy to notice that, generally, the teams are formed by 1/5 experts, 2/5 developers, 2/5 junior programmers or trainees. In addition, enterprises tend to add a lot of people in their projects in order to reach "high productivity" (what compromise control and quality, of course).

So, it is extremely hard to put this kind of team working on a unique branch and request them to produce production ready code.

The solution

Although we think it would be ideal to have a team of experts using Branch By Abstraction technique in a unique development branch, we know that, in real world, it hardly will be applicable. So, we have been trying to mix branches, toggles and a bit of automation in order to find a solution which fits our needs. For the happiness of our team we have found a very simple and interesting alternative. So, let me explain it and how the idea came out.

We were used to have two development branches. The Trunk, for major developments and PS for Production Support. While developers were committing new features in Trunk, the production support team were fixing bugs and implementing small improvements in PS. Initially we were used to merge PS to Trunk as soon as a new version was released (usually one merge per week basis) and, from Trunk to PS whenever business requested. Frequently, those merges took the integration team one or more days to finalize. Sometimes, bugs were introduced and some of them would not be seen until a new release was generated from Trunk (what could take months or even years). Knowing this, we have decided to use a basic premise of continuous delivery: "if it hurts, do it more often". Thinking like that we have decided to merge once a day, but we still have problems. So, we have tried to merge twice a day. Of course it got better, but nobody likes merges and do it twice a day is a pain in the ass. Then, we question ourselves: why not perform a merge per commit base? and why not automate it? At this point we decided to develop a tool and the result was the following.

In the above image, it is possible to see two developers. Dev 1 is committing in PS and Dev 2 is committing in Trunk. Note that Dev 1 have done two commits (revision 1 and revision 2) which were merged automatically to Trunk by a tool we have developed. Then Dev 2 tried to make a commit but the version control blocked it due to a conflict. So, he must first resolve the conflict in his working copy and then commit his new feature (the revision 3 in the image) in Trunk.

Whenever the team decides this feature (

revision 3) is production ready, the developer which have committed it (in this case Dev 2) must be pushing the revision to PS in order to release it as soon as possible.

A more complex scenario where the automatic merge fail can be seen in the below picture.

In this case, when Dev 1 tries to commit revision 3, our automatic merging tool shows the conflict in RED in a dashboard and stops merging. At this point, the integration team, along with the developer which has performed the commit (in this case, Dev 1), must analyze and fix the problem. After that, the merging tool must be advised to keep doing merges. As we have seen in the first picture, every time a feature is considered production ready in Trunk it must be pushed to PS in order to be released.

Advantages

Developers have time to finalize its features (in Trunk) without being afraid of putting unfinished code in production

There are fewer cases which actually require Feature Toggles

Once they are rarely used to manage unfinished code

The team which use Feature Toggles are more prepared

Toggles are usually implemented by the production support team

The integration team has more free time to improve the whole integration and delivery process

Less issues resulted from merges

More control about what is done and what is not

Done means running in production without problems

Developers have more responsibility

Developers must be responsible from coding to monitoring production

Forces the team to have focus

New features developed in Trunk must go to production as soon as possible in order to make the automated merge tool work properly

Not so good, but not disadvantages

New features must be pushed manually from Trunk to PS

Although we think this task can also be automated, we like the idea of developers being involved/responsible up to production.

The integration team is still involved to resolve conflicts

However, the time spent in this task is nothing compared to some years ago

Conflicts must be resolved as soon as possible

In order to not accumulate automatic merges and keep the whole system working seamless

Branches must be as similar as possible

In order to avoid automatic merges conflicts. So, features must be pushed from Trunk to PS as soon as possible.

This solution is way better than others we have tried up until now. It works in real world, developer is involved from the beginning until the end, the number of Feature Toggles is reduced, the integration team has more free time to improve the process and it forces the whole team to have focus on the current development features (soon I'll be writing more about what other strategies we have used to improve focus).

Are you doing or trying to do Continuous Delivery? How do you work? What do you think of this solution? Do you like it? Do you think it may work for you? Please, leave your comments. I'll strongly appreciate them.

Jboss 7: a love–hate relationship

2013-04-07T19:55:00.000-07:00

I've been using JBoss Application Server professionally since 2002. During this period, I've also used Tomcat, Jetty, Glassfish, IBM WebSphere and the already dead OC4J. Between these options I would say my personal choice always was JBoss. Do not misunderstand me; Tomcat and Jetty are both great and lightweight web containers. But the point of this post is not to discussion anyone's preferences, but my experiences with the new version of JBoss Application Server (AS).

First of all, I would like to discuss the good points and improvements JBoss 7 has over its older versions. So, my "loves" are:

Strict Java EE 6 compliant

Reduce vendor locked once JBoss development team has chosen to implement “only” the JavaEE spec.

Completely redesigned architecture (very welcome)

Modular design

Say goodbye to microcontainers

They have created an abstraction built on top of OSGI

According to JBoss guys, it is “prepared to support Jigsaw”

Modular dependencies

Best way to manage your app dependencies

Modular classloading

Say goodbye to the UnifiedClassloader. \o/

classloading happens on demand

Two execution modes

standalone

Normally used for development

Allows to organize and manage your cluster(s) as you are already used to

domain

Allows to organize and manage your cluster(s) in an easy and centralized way

Better security support

Allows to provide separate inet interfaces for apps and management consoles

Central point of configuration

the whole configuration is centralized in only one file (or two for domain mode)

Profile definition are easier and “smarter”

It simplifies a lot the AS slimming. \o/

If you do not use a specific feature, JBoss do not load it

You can reuse profile definitions. There is no need to be coping jars around anymore. \o/

Better management solutions

Simple and easy management interfaces

Web Console (GWT) with only the most important data

Rest interface

Command line

JMX (actually I’ve never used this, there is no need any more)

A centralized management interface for domain mode

Persistent configuration through any management interface available

Remember... In older versions of JBoss, the configuration through jmx-console was transient. So, this is a very welcome feature

Faster startup

Modules were optimized to minimize the startup time

Modules and packages are indexed

Only required modules are loaded

JBoss knows your dependency graph, so there is no reason to load JMS module, for example, if you do not use it. Even though you have configured it to be loaded. Actually, it seems to work pretty well.

Parallel startup

Once JBoss 7 knows your dependency graph, it can use all cores to start your apps

To resolve correctly your dependencies in older versions of JBoss, you have to inform the order you apps must start. So there was no way to start them parallelly

Memory diet

As we have seen before, only required modules are loaded

This means, less memory usage

Less memory usage == better scalability and performance, once you will have more free memory available and less Full GC executions during runtime

Better support for integration tests through Arquillian

Besides all these good stuff, let me show my "hates":

Strict Java EE 6 compliant

Wait.. You should be thinking: “at first you said it was a good stuff, and now a bad stuff?!?!”. The reason for this is that there are functionalities which are impossible to implement using pure Java EE spec and, most of them, were removed. :-(. Some of them, community yell so much they have decided to add an alternative. Take as an example Container Interceptors which is coming in JBoss AS 7.2 / JBoss EAP 6.1

Completely redesigned architecture

Again.. the new design modular is wonderful, but for those who would like to migrate from an older versions of JBoss, will notice it is not as easy and seamless as it should be

No backward compatible in JBoss specific features

JBoss 7 has broken compatibility with everything that was not in JavaEE spec. So if you use JBoss specific features, be careful, your migration may be more complex than you think it would be

No JBoss AOP any more :-(

The migration documents are useless

The difficulties we have faced do not fit in this post, so I’m planning to write one or more posts to help community understand what they will be facing during the JBoss migration

Annoying bugs deploying applications through management consoles

Although I think the fix would be coming, it has been forcing us to restart the server at each deploy

JBoss EAP Alpha

The community version of JBoss (i.e. JBoss AS) will now be called as JBoss EAP Alpha. Why?!? Marketing purpose, of course. Imagine you saying to a client they should be using an Alpha version in production
UPDATED: Just received an e-mail from Red Hat saing

JBoss AS, is now called Wildfly. See by yourself

http://www.wildfly.org/

JBoss EAP Alpha is not as stable as the JBoss AS Final

Although they say EAP Alpha is stable as the AS Final, it is not complete true. If you take a quick look on JBoss modules, you will easily notice they are using unfinished/unstable libraries.

For example, JBoss EAP 6 Alpha is bundled with hibernate-core-4.2.0.CR1. I don’t recall any CR lib in older versions of JBoss AS.

Obviously it is very ease to people with certain knowledge to modify. But most of people will be using JBoss without any module structure change, for sure.

Domain controller is not fault tolerant

Whenever the domain controller dies, other nodes cannot be restarted or managed. If you try to restart them, they will give you an error saying they cannot connect to domain controller. Hope the fix would be coming.

Does not provide load balancing for EJB Timer

Only fault tolerance

In some clients, we were forced to keep using quartz because the huge number of jobs they have

Last, but not least. JBoss EAP support is a pain in the ass

I will make no comments about this. Just try by yourself you will understand what I’m talking about.

My advice is: be prepared, mainly if you have gastritis. :-)

Even with all these issues, I still think JBoss 7 / EAP 6 is one of the best application server available, but I think the move Red Hat is taking would be pushing the community away from them. Actually, I know people which are looking for Jboss alternatives. Some friends are working in a migration from JBoss 5.1 to TomEE, another once told me: “if I have to pay for an Application Server, I will be paying for a WebLogic”. He said that knowing Oracle contracts were more expensive than Red Hat ones.

Anyway, everybody knows that there are many good Application Server options out there (both free and paid ones). So, what do you think? Would you be using the JBoss EAP Alpha in a production environment? Please, leave your comment. I would love to hear it.

Feature toggles: good or bad?

2013-03-27T18:44:00.000-07:00

Everyone who have read at least once about continuous integration or continuous delivery knows that it is not possible or, at least it is very difficult, to do if you use more than one development branches. If you have never ever faced any problems using two or more development branches you are a fucking lucky guy or, probably, you are not used to refactor your code as you should be (in these rare cases, I strongly recommend you to take a look in this Martin Fowler & Mike Mason interview and Jez Humber Continuous Delivery book). On the other hand, if you have found yourself in an endless merging task or if you have been feeling insecure about some merging results. Do not panic! Keep reading. You are not alone. :-)

As you may already know, there is no problem with branches. The problems are all about merges. So, to overcome big and error prone merges the best strategy I would recommend is Branch by Abstraction. According to Martin Fowler, Branch by Abstraction is “a term coined by Paul Hammant to describe a technique for making structural changes to a code-base without a Feature Branch in a source-code control system”. To make that real, we are used to implement new features, or updating existing ones, using a very simple technique known as Feature Toggles. In short, a feature toggle is a way you develop your software in order to make it easy to enable or disable features through configuration changes.

This kind of strategy is widely used by companies such as Facebook, Flicker, Google, Amazon, and, of course, the ones I’ve been consulting. A very simple example of feature toggles can be seeing in the Chrome web browser by typing chrome://flags in the URL input. In this page you can enable or disable features of your browse and help Google to evolve it. ;-)

After some years of experience trying to improve clients’ delivery processes, we have found some advantages and disadvantages of using toggles instead of version control system branches. So, the main idea of this post is to share and discuss our experience with the community.

The advantages we have found are:

Ability to turn features On and Off at any moment

If you have found yourself adding a new bug in production, for example, you could just turn your feature Off.

Move forward, never backward

Once you can disable incomplete or buggy features, you can move on and fix it instead of rolling back your entire app (which in some cases is extremely painful).

Ability to put unfinished code in production

At the first glance it may sound a bit weird, but it helps you to reduce the amount of stuff you put in production at once. Small batches reduce the risk of going live (see Reducing IT risk using continuous delivery )

Reduce or eliminate the necessity to create branches in a source-code control repository

Your integration team will be very thankful, believe me ;-). This means less or no merges any more. Given them more time to making smarter tasks than merging.

Ability to make experiments that could not be done outside of production environment

For example, those intermittent issues which are impossible to reproduce outside of production.

Ability to enable a feature for only one group of users

For example, you can enable a feature for a specific region, role, user group, or any other rule you need.

Ability to enable and disable the features while the app is up and running

But you may set toggles at build time

The disadvantages are:

There are cases in which it is not possible or, at least, it is very difficult to implement a feature toggle

Examples in the Java world

Static configuration files (i.e. web.xml, ejb-jar.xml e persistence.xml),
JPA mapping,
etc.

Every toggle requires an engineering process

First, avoid doing toggles for everything.
Second, you must be sure you need it.
Third, if you realize it is extremely necessary, it is important to provide a good design for its implementation.

Toggle On means: a pretty new technical debt

Once you enable your toggle and it is working 100% in production you have a technical debt that should be removed as soon as possible
If you do not remove the technical debt your software will be harder to improve, maintain, test and deploy. This happens because your code must care about all possible situations (i.e. toggle On and Off)

Features loosing priorities

Sometimes people forget to enable a feature toggle just because that feature loses its priority. It is very ease to happen, unfortunately. At least in my experience.
Sometimes people find an issue in production and then disable the toggle, but never fix it. In this case, the toggle remains there disabled forever.

Feature toggle could easily get out of control

Sometimes people lose the control of what is enable in which environment
The number of toggles will grow faster than you think. Take care.

Finally, the discussion about this subject

Any technique has its advantages and disadvantages. However, if you have noticed, most of disadvantages are a reflect of not given the right control feature toggles should have. With a good toggle management, frequent releases and focus (do one thing at a time) it is possible to solve most of these issues.

To help overcome these difficulties, we have been fulfilling the following information for every feature toggle:

The Value

Usually enable/disable, but can be a complex data structure informing the region it is enabled, the user grup, or etc

A detailed description

Detailed information about what this feature toggle does and why it is necessary

The deadline (very important)

When this toggle should die and how you should perform the clean up

The responsible (very important)

The engineer/team responsible to monitor and remove the toggle. Usually, this responsibility is given to the one which have chosen to add it.

Besides that we have developed a dashboard, which is revised periodically, which provides all toggles information in order to get the full benefits this technique could provide us. Moreover, as Martin Fowler already said “while feature toggles are a valuable tool in the box, they are a second-best option. The best thing to do with such features is to find a way to gradually release them into production as you are building them”.

If you like or not, please leave your comment. I will be very happy to listen to other teams experiences.

Continuous Delivery: why you should be doing this?

2012-07-06T05:49:00.000-07:00

If you are not doing, or not going in the direction of doing, Continuous Delivery (CD) I would say you are going down to the wrong path. I know, I have strong opinions (see my post Are you agile? really?). But this is the only way I could see software development in large scale* work. Not saying it is an easy path (I would mainly blame the culture of avoiding/blocking changes), just saying that you should start thinking on its benefits and also you should start trying out those techniques.

It is important to mention that this post will not discuss continuous delivery strategies, techniques or patterns. For that you should read the Jezz Humble’s book and search for it on web. I’m sure you will find hundreds of wonderful materials on this subject.

So, why you should be doing continuous delivery?

Lots of people think Continuous Delivery is an extension of Continuous Integration where the focus is on "solving the last mile”. Although this is kinda of true, it necessary to look at the whole idea behind Continuous Delivery to figure out that it is much more than “the last mile”. Let’s discuss some key points:

Manage IT risks in a simple and efficient way

Automate testing is one of the key strategies in continuous delivery world. A well design pipeline includes, at least, unit testing, service testing, functional or acceptance testing, smoke testing and provisioning testing. So, going down this pipeline means you are testing every piece of your software, including infrastructure. Knowing this will happens dozens of times per day you are then reducing the risk of going live.

Respond as fast as possible whenever an issue happens

Continuous Delivery does not eliminate all bugs/issues from going live -- although I’m secure to say it reduces most of them. So, when these unpleasant things happens, CD plays in your side. Let’s see an example:

The above picture shows the time spent to figure out a problem an then the respective time spent to fix that in an traditional delivery process.

On the other hand, it is possible note in the above picture that both time to discovery and to fix a problem in production is much smaller in CD than it is in traditional processes. This happens because:

Once you goes frequently to production, you have much smaller batches, then you have much smaller problems. Simple like that.

Even if you have an issue that requires to rollback the whole installation, it is much easier to do this task in CD since you have all the software and provisioning scripts marked into SCM.

As you must already know, going back must be the last option. In CD world we used to say “go forward”. That means, fix the problem and reinstall. Your process was planned for that.

Delivery of reliable products

As discussed previously, continuous delivery depends on automated testing. So, it is all about doing regression tests many times per day. That means, whenever a developer makes a change in a module A that breaks another module B, in which he is not responsible for, it will be known as soon as the pipeline starts. Therefore, you get confidence whenever you deliver a new version of the product.

Refactoring

Another interesting point is that this amount of testing actually encourages refactoring. And, periodically doing refactoring we ensure that our software will ever be up to date. Without such strategy, delivering a new version of your software means you already have a legacy.

Reduce the cost of the product

As any software engineering knows, the cost of fixing a bug/issue grows in price according time passes (see the picture below). Continuous Delivery is all about discovering this kind of stuff as early as possible in the pipeline, reducing then the product’s cost as a whole.

Builted-in software quality

It is not possible to do continuous delivery without doing continuous integration. Therefore, all the good stuff of continuous integration is earn for free. ;-)

Repeatability

One of the main strategy behind continuous delivery is the “on click” slogan. That meas that with just one click or two (or a command or two) you are able to delivery your software -- including all the infrastructure it depends to run on top of. To make this possible, you must automate almost every step of your delivery process. Doing that, you have a fast, reliable and repeatable delivering process.

Spread delivery knowledge

In most of companies I’ve being working for, the delivery knowledge is focused in a delivery team. Some times, this team was composed by only one person. Whenever this person decide to look for new opportunities and you have an urgent deliver to do, certainly you got into trouble. Why?

Just because:

You don’t know detailed steps to delivery your software
Documentation is, usually, not up to date
When you go into the delivery process you start to see the hack people do for delivering

With Continuous delivery you just press a button. Then, after delivering, you have enough time to look into the process for understanding and, maybe, improving.

Infrastructure fully documented

Every build and delivery step is “configured” in a programmatical way. So, whenever you would like to know how infrastructure work in any environment (i.e. development, staging, production, etc), just checkout the the code from SCM and take some time to understand the logic.

Documentation will never be out of date

Even better than have fully documented environments, is that this documentation will never be out of date. Whenever it gets out off date your continuous delivery process will advise you by failing some stuff in the pipeline... Amazing, don’t you think?!?!

This is one of the key benefits from Continuous Delivery that is not so commented in papers on this area.

Learning from real data usage

"When you think about it, the concept of continuous delivery isn't really new. Creating a hypothesis, developing the hypothesis, testing it, and controlling for other variables by isolating the one that changes—that is, the scientific method—isn't exactly controversial. While Agile certainly has made strides, putting a greater emphasis on making sure that testability is a first-order concern in software delivery work, continuous delivery closes the loop" [source]

Focus on customer necessity

Most of the time customers do not know what are they needs. In other times they do not have enough data for solid decisions. Continuous Delivery and its scientific method help us to focus on actual customers need (i.e. ROI) and not in what they want.

Fast feedback, visibility and control

The build pipeline used in Continuous Delivery is all about fast feedback, visibility and control. Where anyone is able to see the current status of the project just looking at a dashboard (usually shown in a large LCD).

Dashboards play a key job in CD world. Nobody should be asking for things like: Which version is being tested? Which version is in production? Which build has failed? Why it has failed? Where can I find the log? etc.. etc.. A good Dashboad should respond this and many other question for just looking at it.

Concluding remaks

All these good stuff guarantee that the product will arrive in production as fast, as mature, as stable and as reliable as possible. Besides this, doing continuous delivery is a fun task in which every time you finish a process automation you will find another part to automate or improve. As Jezz Humble once told: "we will never be done". :-)

So, my advise is: you are already late! Start using continuous delivery.

---

* In this context large scale means more than few developers. :-)

Are you agile? really?

2012-05-19T12:49:00.002-07:00

Lately I’ve heard a lot of software companies saying they are agile. However, when you start a conversation with the representative of these companies you can easily notice that Scrum is the only word they know about being agile.

Nowadays be agile means be "modern". Developers want to work in agile projects. Customers are looking for agile companies/consultants. Even Gartner predicted in 2010 that "agile development methods would be utilized in 80% of all software development projects". So.. what is the point?

Well.. the point is:

Be agile != do Scrum

Actually, be agile is much much more than do Scrum variants. Be agile requires the team to feel and act as agile. It must be running into the veins of the developer. But, unfortunately, what we can see is that the word agile is being used as marketing purpose by many companies.

While discussing about that with a friend I’ve decided to write down my thoughts about this subject. So, in my point of view, you ARE NOT agile if:

if you don’t focuses in built-in quality

There is a principle on the Agile Manifesto that says “continuous attention to technical excellence and good design enhances agility”

Agile teams must feel pleasure delivering high quality software. The era which said that software process ensures software quality have finished a long time ago. You must focus on quality from inside-out. That is, the code must be high quality, so that the functional may fit the user expectations.

With a poor code you have: more bugs, poor performance, poor extensible strategy, unreadable code, etc.. Consequently, you have a customer less happy.

if you are not able to make your customer to work in your side

don’t need further comments..

if you do not pay technical debits

I know “agile” projects that have failed because, in only two years, the software started to deteriorate. But why? the answer is simple: because developers didn’t know what technical debt is..

If you do not automate tests (in all levels)

If there isn’t regression tests

If you do not refactor your code

If you do not enhance your design/architecture

If you do not use continuous integration

Nowadays I would say, If you do not use continuous delivery

If you do not monitor your software when it goes live

If you do not learn from your live software

If you can point a person or a developer as the responsible for a delivery failure

If you do not trust in your team

If your team is separated by sites (i.e. DBAs, testers, developers, etc)

if you do have more trainees and junior developers than senior ones

The problem is that most companies which say they are agile have more trainees than senior developer. In these companies the software quality tends to be poor. I would like to say that there is nothing wrong with trainees and junior developers. The only point here is that they don’t have enough maturity to make decisions or to enhance the software design.

As any other software methodology, trainees must added into an agile team in a planned manner.

If you try to institutionalise a process that have worked for another team

If you do not use working software as the primary measure of progress

If you do not continually improve your process

If your process do not welcome changes

If you have closed scope contracts

Be agile in closed scope projects is extremely hard. This is possible if you have a very tiny software, but with large ones practice have shown us that it is not possible because you will reject changes or make changes difficult to happens. What goes against one of the main principles of the Agile Manifesto.

if you have large iterations

One month seems to long for a delivery. One or two weeks is a nice shot

if you can not break your requisites to fit into a sprint

If you do not use MVP (Minimum Viable Product).

If you say ‘done’, when the functionality is not shippable

...

It is important to say that there is nothing wrong with a company trying to use Scrum, Kanban, XP or whatever. What I’m saying it is wrong is that there are many companies using the word agile as a marketing purpose only. They have no process, no adequate team and because they’ve started to do stand up meeting and put some notes on a blackboard they sell themselves as agile.

Just because you’ve started to use Scrum and Kanban does not means you are Agile. That means you've done the baby steps to the “right” direction. Be agile is more difficult than most of people think.. That is why there is only few companies in the entire world that REALLY are agile.

Please, leave your comments.. I will appreciate them..