Here at Shopzilla, we have written several of our internal management tools in Python. We’ve worked with several different frameworks, notably Pylons, Pyramid, web2py and Django.

Pylons/Pyramid

In the first of the Python tools, we used Pylons. At the time, the thinking was that Pylons would be easy to swallow for Java developers – it’s very similar to using Spring MVC. You have to choose your ORM, template engine, session management strategy, etc. This fit well with how our Java code exists in theory. Ultimately, we were constantly left wondering if another template engine or ORM might be better, etc. We found that we spent more time thinking about the mechanics of the behind-the-scenes than the actual business need at hand.

After a few tools in Pylons, we migrated to an early version of Pyramid for a new tool. The development experience was ultimately very similar to Pylons. Powerful, but low-level.

web2py

Our DevOps crew created a deployment and configuration management tool entirely in web2py. It is used by every engineer within our organization several times a day. It is stable, extensible and replaced a hodgepodge of perl and shell scripts and manual effort. It has enabled our engineering team to be self-sufficient, letting DevOps focus on production and “keeping the lights on.”

Django

Django required us to think a little differently about several things. Django required us to spend less time worrying about the mechanics – the ORM, available databases, the template engine, how sessions were managed, etc were already there for us. As an engineer, this is fairly counter-intuitive to how we think. We like to focus on the low-level pieces of the problem, sometimes more than the actual business goal. We think of this level as fun and entertaining.

We also had to approach schema designs slightly differently. Example: Django’s ORM can only use surrogate keys on tables. Business keys don’t work as they often are composite keys.

Django’s productivity gains really clicked when we made a very small application using only the internal administration-focused scaffolding to build a user management tool for the TaDa Merchant Tools site. Within four hours, we had something in production. When we originally talked about building this tool, we had complex UI mockups, and had scoped this out to be a two to three week project total! The tool we ended up with was much less complex, but it met our needs perfectly. This sort of immediate productivity is amazing, and provided a big benefit to our business.

Django is not restricted to it’s ORM, template engine, etc, but unless you have a real use case for a different implementation, you’ll see immediate productivity gains using what it ships with.

This is not to say that Django doesn’t have its own things to worry about before and after you get to production, however. By default, Django does not use connection pooling, so that had a performance impact for us. We added django-oraclepool to address this. It was nearly a drop-in addition, and not having to set up and tear down database connections on every request has a very obvious gain. You also have to pay close attention to what the ORM is doing; it’s not always obvious when it’s making more queries than you intend for one page, and you pay the price because they all execute serially. We added django-debug-toolbar for use during development to help track down ORM issues and general request debugging. The default is lazy loading for relationships – profile, profile, profile and hint the ORM appropriately in your queries!

Standards

All of our Python applications are built using industry-established development standards to ensure that our code is maintainable, deployable and supportable, thus adding real business value.

• virtualenv during development and deployment for isolation
• setuptools for dependency management and packaging
• Hudson to build and run tests
• flup/FastCGI behind nginx for serving in QA and production
• Unit testing via nose
• Code coverage using coverage (and the django-coverage wrapper for Django projects)

We’ve added support to our internal deployment tools for our Django and Pylons/Pyramid projects. We deploy new builds or update configurations with a few clicks. It’s as seamless as our Java projects.

I’m confident we have built a solid foundation to continue utilizing Python within Shopzilla, and it’s finding wider adoption in other aspects of the business, like QA automation, text processing, etc. Python is helping us deliver software faster, without sacrificing our quality in the progress.

At Shopzilla we began using the Oracle thick drivers a couple of years ago. We needed them to get Oracle wallets to work. We needed the wallets to avoid storing the database usernames and passwords in clear text files on our production servers, which obviously isn’t good.

However, as time went by and we analyzed various crash logs, we kept finding the thick driver code was involved. We had conversations with a variety of people at Oracle, and repeatedly heard the recommendation that we switch to the thin driver. So we set out to switch, but it wasn’t so simple. Before going to production with the thin driver we needed to meet a number of conditions:

• we had to use Oracle wallets so we didn’t expose passwords
• we needed a working connection pool to scale to production loads
• we needed working database timeouts for situations such as losing connectivity to the database

The combination documented here is the first one we found that met all our criteria.

The Basics

To get the thin driver to work with wallets, you’ll need to use a recent thin driver and 3 PKI-related jars from Oracle. To get database timeouts to work with that combination you must use the Oracle Universal Connection Pool, which was introduced with Oracle version 11.2.

As an example, here are the jars that we are currently using:

1. ojdbc6-11.2.0.2.0.jar      (latest Oracle thin driver)
2. oraclepki-11.2.0.0.jar      (for wallet support)
3. osdt_cert-11.2.0.0.jar      (for wallet support)
4. osdt_core-11.2.0.0.jar      (for wallet support)
5. ucp-11.2.0.1.0.jar      (for the connection pool)

These jar files come from various Oracle downloads.

• ojdbc6: http://www.oracle.com/technetwork/database/features/jdbc/index-091264.html
• ucp: http://www.oracle.com/technetwork/database/enterprise-edition/downloads/ucp-112010-099129.html
• oraclepki, osdt_cert, and osdt_core are not available as separate downloads. Originally I used the ones that come with JDeveloper 11g:

  http://www.oracle.com/technetwork/developer-tools/jdev/downloads/index.html

  For production we ones that the Oracle DBAs obtained from some enterprise Oracle software – unfortunately I’m not sure which one.

To test, naturally you’ll need a wallet. Ours are generated by our Oracle DBAs. Dev wallets can be generated to not be machine-specific, so you can put one in version control with the project source if desired.

Before going further, it’s worth taking the time to get a simple test app working. Here’s an example that can be modified to suit your circumstances:

import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import java.util.Properties;
import oracle.ucp.jdbc.PoolDataSource;
import oracle.ucp.jdbc.PoolDataSourceFactory;

public class WalletTest {

    private PoolDataSource pds = null;

    public WalletTest() throws SQLException {
        // Create pool-enabled data source instance.
        pds = PoolDataSourceFactory.getPoolDataSource();

        // PoolDataSource and UCP configuration
        // set the connection properties on the data source and pool properties
        // pds.setUser("scott");
        // pds.setPassword("tiger");
        pds.setURL("jdbc:oracle:thin:/@WALLET_TEST_CONNECTION_ID");  // (1)
        pds.setConnectionFactoryClassName("oracle.jdbc.pool.OracleDataSource");
        pds.setInitialPoolSize(10);
        pds.setMinPoolSize(10);
        pds.setMaxPoolSize(20);
        pds.setConnectionProperty("oracle.jdbc.ReadTimeout", "5000");

        System.setProperty("oracle.net.wallet_location",
            "(SOURCE=(METHOD=file)(METHOD_DATA=(DIRECTORY=/home/scorwin/swdev/WalletExp/wallet)))");  // (2)
    }

    public void displayPoolDetails() throws SQLException {
        System.out.println("-----------");
        System.out.println("NumberOfAvailableConnections: " + pds.getAvailableConnectionsCount());
        System.out.println("BorrowedConnectionsCount: " + pds.getBorrowedConnectionsCount());
        System.out.println("-----------");
    }

    public void run() throws SQLException {
        List<Connection> connList = new ArrayList<Connection>();

        for (int i = 0; i < 5; i++) {
            // Get a database connection from the datasource.
            Connection conn = pds.getConnection();
            System.out.println("Retrieved a connection from pool");
            displayPoolDetails();
            connList.add(conn);
        }

        Connection conn = connList.get(0);
        Statement stmt = conn.createStatement();
        ResultSet rset = stmt.executeQuery("select sysdate from dual");
        while (rset.next()){
            System.out.println("sysdate from dual: " + rset.getString(1) + " as of "
                    + new Date());
        }
        rset.close();
        stmt.close();

        // close all connections
        for (int j = 0; j < connList.size(); j++) {
            ((Connection) connList.get(j)).close();
        }

        // finally display pool details again
        displayPoolDetails();
    }

    public static void main(String[] args) {
        System.out.println("Started at " + new Date());

        try {
            WalletTest basicDemo = new WalletTest();
            basicDemo.run();
        } catch (SQLException e) {
            System.out.println("SQLException occurred: " + e.getMessage());
            e.printStackTrace();
        }

        System.out.println("Ended at " + new Date());
    }
}

(1) replace WALLET_TEST_CONNECTION_ID with a connection identifier from your wallet.

(2) replace ‘/home/scorwin/swdev/WalletExp/wallet’ with the path to your wallet, being careful not to change anything else. In particular there are exactly three closing parentheses.

Here is an example of a simple shell script that compiles and runs the test app:

#!/bin/bash

export CLASSPATH=/home/scorwin/.m2/repository/com/oracle/ojdbc6/11.2.0.2.0/ojdbc6-11.2.0.2.0.jar
CLASSPATH=$CLASSPATH:/home/scorwin/apps/oracle/Middleware/jdeveloper/modules/oracle.pki_11.1.1/oraclepki.jar
CLASSPATH=$CLASSPATH:/home/scorwin/apps/oracle/Middleware/jdeveloper/modules/oracle.osdt_11.1.1/osdt_cert.jar
CLASSPATH=$CLASSPATH:/home/scorwin/apps/oracle/Middleware/jdeveloper/modules/oracle.osdt_11.1.1/osdt_core.jar
CLASSPATH=$CLASSPATH:/home/scorwin/apps/oracle/ucp11.2.0.1.0/ucp-11.2.0.1.0.jar
echo $CLASSPATH

rm WalletTest.class
$JAVA_HOME/bin/javac -Xlint:deprecation WalletTest.java

CLASSPATH=$CLASSPATH:.
$JAVA_HOME/bin/java -Doracle.net.tns_admin=/home/scorwin/swdev/WalletExp/wallet WalletTest

 

When the app runs successfully, it will produce output like this:

Started at Mon May 16 16:54:50 PDT 2011
Retrieved a connection from pool
-----------
NumberOfAvailableConnections: 9
BorrowedConnectionsCount: 1
-----------
Retrieved a connection from pool
-----------
NumberOfAvailableConnections: 8
BorrowedConnectionsCount: 2
-----------
Retrieved a connection from pool
-----------
NumberOfAvailableConnections: 7
BorrowedConnectionsCount: 3
-----------
Retrieved a connection from pool
-----------
NumberOfAvailableConnections: 6
BorrowedConnectionsCount: 4
-----------
Retrieved a connection from pool
-----------
NumberOfAvailableConnections: 5
BorrowedConnectionsCount: 5
-----------
sysdate from dual: 2011-05-16 16:55:09 as of Mon May 16 16:55:08 PDT 2011
-----------
NumberOfAvailableConnections: 10
BorrowedConnectionsCount: 0
-----------
Ended at Mon May 16 16:55:08 PDT 2011

 

Note: so far we have not found a need to change the code to include

 Security.addProvider(new oracle.security.pki.OraclePKIProvider());

or to modify the JDK config file jre/lib/security/java.security to load this provider either.

Real-world Apps

Pass the wallet path

The wallet path must be supplied in the system property oracle.net.tns_admin. This can be passed to the JVM as a command-line parameter:

 -Doracle.net.tns_admin=/path/to/wallet/files

where “/path/to/wallet/files” ends with the name of the directory that contains cwallet.sso and the related files).

When using Tomcat we instead add

-Doracle.net.tns_admin=/path/to/wallet/files

to the end of the existing catalina.opts file. Tomcat must be restarted for this to take effect.

Assemble the JDBC URI

The JDBC URI will be of the form jdbc:oracle:thin:/@connection_identifer, for example jdbc:oracle:thin:/@PROD_CONN_ID. Note that the URI contains “thin”, not “oci”. This is vital. oci = thick driver.

Additional Configs

• when configuring the datasource, the wallet location must be specified. It contains the path to the wallet in the middle of it.
• the database read timeout must be specified. This is done by setting the OracleDataSource connection property oracle.jdbc.ReadTimeout to the desired timeout in milliseconds.
• if you’re converting an existing app, any existing username and password configs should be removed. They will be ignored and only cause confusion.

Here is an example of configuring a datasource using Spring:

<bean id="dataSource" class="oracle.ucp.jdbc.PoolDataSourceFactory"
       factory-method="getPoolDataSource">
   <property name="URL" value="${DATA_SOURCE_URI}" />
   <property name="connectionFactoryClassName" value="oracle.jdbc.pool.OracleDataSource" />
   <property name="initialPoolSize" value="${POOL_CONNECTIONS_INITIAL}" />
   <property name="minPoolSize" value="${POOL_CONNECTIONS_MIN}" />
   <property name="maxPoolSize" value="${POOL_CONNECTIONS_MAX}" />
   <property name="abandonedConnectionTimeout" value="${POOL_CONNECTIONS_MAX_WAIT_TIME}" />
   <property name="loginTimeout" value="${POOL_CONNECTIONS_LOGIN_TIMEOUT}" />
   <property name="validateConnectionOnBorrow" value="true" />
   <property name="SQLForValidateConnection" value="select 1 from dual" />
   <property name="connectionProperties">
     <props merge="default">
       <prop key="oracle.jdbc.ReadTimeout">${POOL_CONNECTIONS_READ_TIMEOUT}</prop>
       <prop key="oracle.net.wallet_location">(SOURCE=(METHOD=file)(METHOD_DATA=(DIRECTORY=${DATA_SOURCE_WALLET_PATH})))</prop>
     </props>
   </property>
 </bean>

Example property values:

DATA_SOURCE_URI=jdbc:oracle:thin:/@PROD_CONN_ID
POOL_CONNECTIONS_INITIAL=2
POOL_CONNECTIONS_MIN=2
POOL_CONNECTIONS_MAX=4
POOL_CONNECTIONS_MAX_WAIT_TIME=20    (value is in seconds)
POOL_CONNECTIONS_LOGIN_TIMEOUT=30    (value is in seconds)
POOL_CONNECTIONS_READ_TIMEOUT=20000    (value is in milliseconds)
DATA_SOURCE_WALLET_PATH=/path/to/wallet/files

Finally, test your app

1. verify that it starts up successfully.
2. perform operations that will cause a variety of database operations to be performed, and verify that they succeeded.
3. you can test the timeouts by starting a database operation and then unplugging the network cable or blocking port 1521 outgoing.

Troubleshooting

Note: some of these entries mention native code. These apply when the thin drivers must coexist with the thick drivers in the same JVM, and do not imply that the thin drivers need native code.

java.sql.SQLRecoverableException: IO Error: Socket read timed out

This is thrown when the database connection times out; it actually indicates that timeouts are working correctly.

java.sql.SQLRecoverableException: IO Error: could not resolve the connect identifier

For example java.sql.SQLRecoverableException: IO Error: could not resolve the connect identifier “WALLET_TEST_CONNECTION_ID”. This means that the wallet files which define the connection identifier (WALLET_TEST_CONNECTION_ID) were not found.

Note: you can look in manifest.txt in the wallet files to see which identifiers are contained in a particular wallet.

ClassNotFoundException: oracle.ucp.jdbc.PoolDataSourceFactory

The ucp*.jar file is missing from the classpath.

java.sql.SQLException: “invalid arguments in call”

This may indicate that an older version of ojbc*.jar was found in the classpath.

oracle.net.ns.NetException: Unknown host specified

This may mean that the mapping from connect identifier to hostname is not being found. The mapping is in the wallet file tnsnames.ora.

Check that the system property oracle.net.tns_admin has the correct value for the path to where the wallet files are. If you’re passing the value on the command line, verify that the command-line parameter “-Doracle.net.tns_admin=…” is being passed to the JVM.

ORA-00942: table or view does not exist

This is most likely due to the incorrect user being specified, one who does not have permission to see the table, or the connection being made to the wrong database. Check the connection identifier values in your URIs.

To see which connection identifiers map to which users, check the wallet file manifest.txt. If you get stuck, copy the wallet to a place that’s used by your test app and modify the test app to select from the table you’re trying to access. Then you can try different connection identifiers and see which ones work.

java.lang.UnsatisfiedLinkError: oracle.jdbc.driver.T2CConnection.getLibraryVersionNumber()

The Oracle thick drivers use code in ojdbc6.jar before calling the native code. Therefore the OCI version and thin driver version need to match exactly, for both the jars and the .so files and other native code. For example if the native code files are at version 11.2.0.1.0 and ojdbc6.jar is at version 11.2.0.2.0 this exception will be thrown.

Exception in thread “com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1″ java.lang.UnsatisfiedLinkError: no ocijdbc10 in java.library.path

This is probably due to LD_LIBRARY_PATH pointing to version 10 thick driver native code, but also check for older versions of ojdbc*.jar in the classpath.

java.sql.SQLException: Unable to start the Universal Connection Pool: oracle.ucp.UniversalConnectionPoolException: Invalid Universal Connection Pool configuration

This exception is thrown when there is something wrong with the values set for the minimum, initial, and/or maximum number of connections the pool should open. For example, if the initial number is greater than the maximum number, such as initial = 6 but max = 2, then this exception will be thrown.

java.lang.NoSuchMethodError: oracle.i18n.text.converter.CharacterConverterOGS.getInstance(I)Loracle/i18n/text/converter/CharacterConverter

Check the classpath for older versions of ojdbc*.jar.

java.sql.SQLException: ORA-01017: invalid username/password; logon denied

Check for JDBC URLs which have not been converted to use a wallet. The driver will still attempt to connect with an old URL, but then fail because the username and password are no longer being supplied.

ORA-12154: TNS:could not resolve the connect identifier specified

This can happen if the driver found a URL with a wallet connect identifier in it, but couldn’t find the files for the matching wallet.

ORA-12534: TNS:operation not supported

Saw this on a test system where the TNS_ADMIN environment variable was pointing to a directory that contained a wallet, but the sqlnet.ora file in that directory contained a entry for the wallet directory but the specified path did not exist on the file system.

]]> http://tech.shopzilla.com/2011/09/oracle-wallet-with-thin-driver-with-connection-pool-with-database-timeouts/feed/ 0 http://tech.shopzilla.com/2011/09/oracle-wallet-with-thin-driver-with-connection-pool-with-database-timeouts/ http://feedproxy.google.com/~r/ShopzillaTechBlog/~3/NhVkxCeAikA/ http://tech.shopzilla.com/2011/06/shopzillas-hadoop-hackathon-learning-to-contribute/#comments Sat, 25 Jun 2011 09:40:35 +0000 Andrew Look http://tech.shopzilla.com/?p=995 With the objective of recruiting new contributors to the Hadoop ecosystem, our most recent meetup of the LA-HUG included a 5-hour hackathon in which attendees learned to set up a development environment, checkout/build/test the Hadoop codebase, find issues to work on in the Apache JIRA system, and understand the community review process.

Aaron T. Myers and Eric Sammers of Cloudera expertly led the session of roughly 15 developers, describing how developers can make contributions to Hadoop Common, MapReduce, and HDFS. While the learning curve was steep, a number of developers were able to submit patches during the session; they are currently pending community review for inclusion into the release of Hadoop 0.23.0.

• HADOOP-7418 support for multiple slashes in the path separator
• HDFS-1322 DistributedFileSystem.mkdirs(dir, dirPermission) doesn’t set the permissions of created dir to dirPermission
• HDFS-1314 dfs.block.size accepts only absolute value
• HDFS-1321 If service port and main port are the same, there is no clear log message explaining the issue.

For those interested in contributing to project in the Hadoop ecosystem, Aaron has prepared a very helpful document, which explains how to pull down the source code, set up a development environment, run the tests, and find JIRA issues to work on.

As a final note, Eric made an exciting announcement that several independent open-source projects related to Hadoop, which Cloudera has been maintaining, have been accepted into the Apache incubator program. They are actively recruiting committers; your help will make a big difference, whether you’re a Java developer, build/release/scripting ninja, documentation-writer, web designer, bug reporter, or even an end-user who pulls down the code and runs it! The list of projects is as follows:

• Flume
• MRUnit
• BigTop
• Sqoop

Thank you to everyone who attended! We hope to see more Hadoop enthusiasts give back to this ever-growing community.


]]> http://tech.shopzilla.com/2011/06/shopzillas-hadoop-hackathon-learning-to-contribute/feed/ 0 http://tech.shopzilla.com/2011/06/shopzillas-hadoop-hackathon-learning-to-contribute/ http://feedproxy.google.com/~r/ShopzillaTechBlog/~3/rxwrN99m65A/ http://tech.shopzilla.com/2011/06/secure-that-elephant/#comments Sat, 25 Jun 2011 09:12:04 +0000 Andrew Look http://tech.shopzilla.com/?p=964
Aaron T. Myers gave a fascinating presentation on the recent integration of Kerberos authentication into the latest Cloudera distribution of Hadoop. For more details on the motivation for these changes, see the initial design documentation for this security upgrade in the JIRA attachment from 2009.

Prior to this security upgrade, the state of Hadoop security has been dismal; client-side authentication means that a malicious user could trivially:

• impersonate a privileged user and execute arbitrary Java code on the cluster
• view and edit potentially sensitive data stored on HDFS
• perform man-in-the-middle attacks by impersonating the cluster services.

As a result, Hadoop clusters could only be considered secure by maintaining strict network segmentation and by only allowing completely trusted users to access the cluster; given the fact that Hadoop is designed to execute arbitrary code, network security is still important, since any implementation of Hadoop security still makes certain assumptions (for instance, that packets going over the wire cannot be read or injected). For a more detailed analysis of Hadoop’s past vulnerabilities and ongoing security challenges, see Andrew Becherer’s BlackHat presentation.

Aaron’s slides contain some useful insight on how to integrate these new features into your own Hadoop clusters. For more details, see below:

]]> http://tech.shopzilla.com/2011/06/secure-that-elephant/feed/ 0 http://tech.shopzilla.com/2011/06/secure-that-elephant/ http://feedproxy.google.com/~r/ShopzillaTechBlog/~3/Eyh5df8e0QI/ http://tech.shopzilla.com/2011/06/why-bother-committing/#comments Sat, 25 Jun 2011 00:15:40 +0000 Miranda Robinson-Perez http://tech.shopzilla.com/?p=934 I’d been hearing murmurs about Kanban for the last six months or so but I only knew the basic concepts: it was a ‘pull method’ of Agile development that was picking up momentum and limited the amount of work the team took on at a given time. However, no one in the company was talking about it or implementing it on their teams and I hadn’t seen it in action yet. We’d toyed with maybe trying it out with one of the Bizrate teams but decided to hold off since January was rather hectic.

After a turbulent Q1 that threw our normal scrum process into disarray, it became obvious to the Bizrate operations team that we needed a change.

• A series of production emergencies made it apparent that although scrum gave us flexibility, it didn’t enable us to react to rapidly shifting business priorities. Even though we moved stories out to accommodate unscheduled work, the two week cycle wasn’t a good fit for how often our business changed – both in priorities and releases.
• Our team’s sense of commitment was constantly being eroded.  There was no way to protect the team’s focus. At the start of an iteration they’d implement a strategy to complete the body of work but the shifting priorities prevented them from executing.  The commitment went from, ‘We WILL get this done’ to ‘We’ll try, but it depends on what happens’.  What was the point of committing?
• We tried a dedicated resource to act as a fire chief for production emergencies but found often there were so many fires that tied directly to revenue that we actually needed the support of the entire team.
• Perhaps most important of all, there was so much waste. The team would start on something and have to abandon it mid way to support an emergency story.  Or, in other cases, we’d finish development & testing on a story but the effort to release was separate and could take two days or a week before the feature hit production which was becoming a source of frustration for the product team.

So when Kanban came back up in discussions it seemed like the perfect time to try something new. We knew a few things were going to have to change:

• The team would decide on a work in progress limit, or ‘WIP’. By limiting the number of things that are in progress at one time, the team works as a whole to deliver that piece of functionality and everyone owns getting those stories completed.
• Stories live on a Kanban board (Japanese for ‘sign board’) with a minimum of three columns: Ready, WIP & Done. Bizrate Ops has opted to only use these three general columns but other Kanban practitioners elect to have more.
• No more iteration planning. Kanban replaces this with small, adhoc meetings that happen as a new story is pulled in.  Only those who are going to be working on a story are involved in the conversation, cutting down on team members sitting through discussion on a story they don’t ever touch.  Same goes for demos. Retrospectives stay intact – what kind of an Agile team would we be if we never inspected & adapted?
• Priorities in the ‘Ready’ column are in constant flux. This gives the business team the flexibility to decide, as often as needed, what the most important story is while limiting the impact to the team and reducing wasted effort.

As part of implementing Kanban, the Bizrate operations team agreed we needed a new definition of ‘done’.  We decided that ‘done’ would no longer mean marked business accepted; instead it would mean live and available for use in production. This is a super exciting shift that pushes us to release things as soon as they’re ready, since a new story can’t be pulled in until we release and can see the feature in production.  If we’re at our WIP limit and there’s not any engineering to be done developers and front end engineers jump in to help execute test cases, run regression on release candidates; whatever it takes to get the story moved over to ‘Done’ so we can start on something new.

We’ve been using Kanban for the last 8 weeks and we’ve already eliminated some of our major problems with scrum.  Limiting our in progress work has greatly reduced abandoned stories.  Since applying Kanban we’ve only had to swap out one story and it came right back into the queue.  The business no longer waits for a release window; the team is eager to get a feature live so they can start something new.  We’re constantly delivering business value – as soon as a feature is blessed for production it’s deployed. Things hit production as soon as they possibly can; no more features languishing while they wait for a release window, and the team is empowered to see them all the way to done.

It’s Not All Roses

There are new obstacles to work through and we tackle these issues in our retrospectives.  Kanban makes bottlenecks and inefficiencies visible and we’re realizing that, although we’ve cut down our release preparation and deployment time, we still spend a lot of time on this step in our process.  At the outset, the team knew that adjusting the definition of done to include release to production would be painful. Adopting the mantra “if something’s painful, do it a lot”, we’re slowly but surely reducing the time taken to release. As we continue as a team, this won’t be the last time we need to adapt our process to meet our needs, but that’s the luxury of being Agile.

]]> http://tech.shopzilla.com/2011/06/why-bother-committing/feed/ 3 http://tech.shopzilla.com/2011/06/why-bother-committing/ http://feedproxy.google.com/~r/ShopzillaTechBlog/~3/dDdjgsYUMG4/ http://tech.shopzilla.com/2011/06/los-angeles-hadoop-users-group-join-us-on-622-for-the-apache-hadoop-hackathon-and-stay-afterwards-to-learn-about-hadoop-security/#comments Fri, 17 Jun 2011 21:29:58 +0000 Andrew Look http://tech.shopzilla.com/?p=931 Wednesday, June 22 will be a busy day for LA-HUG’ers!

First, we have the Hadoop Hackathon, from 1-6pm, which will be led
by Cloudera’s Aaron Myers (Hadoop Committer) and Eric Sammer (Solution
Architect). There you will learn about the submission process and how to
make legitimate contributions to Apache Hadoop – I’m sure everyone will find
this to be an engaging, interactive way to gain some experience with the
internals of Hadoop.

http://www.meetup.com/LA-HUG/events/21380931/

After the Hackathon, we’ll have a presentation from 6-8pm by Aaron Myers in
which he will discuss Hadoop Security – both how it’s implemented within
Hadoop, and how you can leverage this security model to keep your data safe.
See the following post from Cloudera’s blog for a taste of the
material: http://www.cloudera.com/blog/2011/01/configuring-security-features
-in-cdh3/

http://www.meetup.com/LA-HUG/events/17448133/

]]> http://tech.shopzilla.com/2011/06/los-angeles-hadoop-users-group-join-us-on-622-for-the-apache-hadoop-hackathon-and-stay-afterwards-to-learn-about-hadoop-security/feed/ 0 http://tech.shopzilla.com/2011/06/los-angeles-hadoop-users-group-join-us-on-622-for-the-apache-hadoop-hackathon-and-stay-afterwards-to-learn-about-hadoop-security/ http://feedproxy.google.com/~r/ShopzillaTechBlog/~3/ftcMTTP7LkU/ http://tech.shopzilla.com/2011/05/modifying-an-existing-maven-archetype/#comments Fri, 27 May 2011 01:20:28 +0000 Steve Corwin http://tech.shopzilla.com/?p=921 Shopzilla uses Maven archetypes as the starting point for a variety of things, in particular creating a new web service. Recently I had to update our web service archetype because the code it was generating wouldn’t compile. As it happens I hadn’t modified the archetype much, and not for a long time. My first attempt to modify it failed, so I searched for docs on how to modify an archetype but didn’t find anything. Everything I found stops after the first archetype creation, as if it would be perfect and never-changing. So I typed up my notes and posted them here.

Steps needed to modify an archetype:
1. checkout the archetype to a local directory
2. make sure you’re working with a SNAPSHOT version. (After modifying the archetype you need to do a Maven build on it, or your changes won’t take effect.)
3. generate from the archetype in a different directory. Make sure you specify the correct SNAPSHOT version for the archetype version. (You can generate in the same directory, but the top-level pom.xml will be modified if you do. Then you’ll have to revert the modifications somehow each time you go through this process.)
4. check the results
5. delete the existing generated code
6. edit the archetype
7. rebuild the archetype: mvn clean install
8. repeat steps 3 to 7 until happy

This much repetition deserves a shell script. Here’s an example:

# rebuild the archetype
cd ~/swdev/Service/archetype-webservice
mvn clean install

# clean the previously-generated code
cd ~/temp
rm -rf archetype-test

# generate from the archetype
#
# If you are making changes to the local archetype, make sure you're specifying the matching
# version.  Otherwise you will generate from a downloaded version, and none of your local
# changes will take effect.
#
mvn archetype:generate -DarchetypeGroupId=com.shopzilla.site.archetypes \
-DarchetypeArtifactId=shopzilla-modular-hibernate \
-DarchetypeVersion=3.2-SNAPSHOT \
-DgroupId=com.shopzilla.site.service.archetypetest \
-DartifactId=archetype-test \
-Dversion=1.0-SNAPSHOT \
-Dservice-version=1 \
-Duser=scorwin \
-DcurrentYear=2011 \
-Dfull-service-name='Test of editing archetype'

# smoke test: see if the generated code will at least build
echo "about to build"
cd ~/temp/archetype-test
mvn clean install

A side note: if you’re generating from the archetype and generation stops with a line that contains only
Y: :
hitting “y” and Enter should get it going.

]]> http://tech.shopzilla.com/2011/05/modifying-an-existing-maven-archetype/feed/ 0 http://tech.shopzilla.com/2011/05/modifying-an-existing-maven-archetype/ http://feedproxy.google.com/~r/ShopzillaTechBlog/~3/rzfhpKv35-w/ http://tech.shopzilla.com/2011/05/agilecafe-at-shopzilla-with-rally-on-thursday-may-19-2011/#comments Tue, 10 May 2011 02:13:22 +0000 Rony Sawdayi http://tech.shopzilla.com/?p=907 Already Agile? Interested in being more Agile? New to Agile? Register for the Agile Cafe on Thursday May 19th from 9am to 12pm at Shopzilla in Los Angeles and hear from experienced users on why Agile makes sense for their teams and organizations. Visit AgileCafe to register today.

At the event, you will learn how:

• To accelerate delivery and reduce defects
• To adopt and scale Agile across distributed teams
• Your organization can benefit from Agile
• Agile teams are more productive

It’s a morning dedicated to all things Agile, with an interactive format and a lively Q&A session. Bring your tough questions, your Agile mind, and share a morning with like-minded practitioners.

Register at AgileCafe. Space is limited so reserve your spot soon!

]]> http://tech.shopzilla.com/2011/05/agilecafe-at-shopzilla-with-rally-on-thursday-may-19-2011/feed/ 0 http://tech.shopzilla.com/2011/05/agilecafe-at-shopzilla-with-rally-on-thursday-may-19-2011/ http://feedproxy.google.com/~r/ShopzillaTechBlog/~3/al2vn-6V8-w/ http://tech.shopzilla.com/2011/04/smjug-seven-deadly-sins-of-solr/#comments Fri, 15 Apr 2011 18:59:02 +0000 Mik Quinlan http://tech.shopzilla.com/?p=891 Shopzilla will be hosting another SMJUG event on May 19th at 7pm. Jay Hill from Lucid Imagination will be presenting. Come along for another excellent presentation and Q&A session, this time wholly focused on Apache Solr!

Jay’s presentation of the “Seven Deadly Sins of Solr” will include;

• Sloth: embrace Solr / Lucene technology, importance of schema definition, leveraging the dismax query parser, buttoning up the JVM settings
• Greed: importance of RAM allocation, physical separation of indexing and searching concerns
• Pride: don’t reinvent the wheel!
• Lust: don’t forget the operating system!
• Envy: what features do you really need, and how big does that index really need to be?
• Gluttony: importance of schema field configuration, leveraging filter queries
• Wrath: don’t rush to production, and don’t assume you will never need to reindex your data

Jay Hill

Jay Hill of Lucid Imagination will deliver the “Seven Deadly Sins of Solr” presentation. Jay Hill has been building enterprise search applications since 2003, and has worked extensively with Autonomy IDOL, Lucene, and Solr.