Programming fun at startup

Client side autocomplete searches

2012-02-11T22:26:00.000-06:00

I was looking for an autocomplete for an addressbook kind of functionality. The requirement was that users can have FirstName,LastName,Email,Type and when user starts typing they need to be searched first by LastName,FirstName,Email and some types were to be given higher priority than others. Doing this on server side means too many calls to be sent to the server so doesnt seems like a perfect solution. Also mostly users will have <100-1000 contacts in addressbook so why not do it on client side. Looked into some autocomplete plugins but none of them were what I wanted and finally landed onto this http://code.flickr.com/blog/2009/03/18/building-fast-client-side-searches/

I have never found such a perfect solution to a complex problem before. Thanks flickr for sharing it ;).

Mysql JDBC driver and Streaming large resultset

2012-02-06T23:15:00.000-06:00

We are moving one legacy component from Berkeley db to mysql so that we can replicate it to distribute read requests and shard it to scale it. I had to dump the entire db contents and stream it over http to some other component. the db content for a single customer can range from 100K to 3-4 Million records. We are using spring JDBC to make the task of dealing with jdbc api simple. Now I was using a ResultSetExtractor to stream the resultset like this.

            ResultSetExtractor resultSet = new ResultSetExtractor() {
                @Override
                public Object extractData(ResultSet result) throws SQLException, DataAccessException {
                    while(result.next()){
                        XXXXX
                    }
                    return "";
                }
            };

getJdbcTemplate(context).getJdbcOperations().query( sql, resultSetExtractor);

But it appears that Mysql native JDBC driver loads entire resultset into memory before passing the control onto ResultSetExtractor and that was causing OOM.

By default, ResultSets are completely retrieved and stored in memory. In most cases this is the most efficient way to operate, and due to the design of the MySQL network protocol is easier to implement. If you are working with ResultSets that have a large number of rows or large values, and cannot allocate heap space in your JVM for the memory required, you can tell the driver to stream the results back one row at a time.
To enable this functionality, create a Statement instance in the following manner:
stmt = conn.createStatement(java.sql.ResultSet.TYPE_FORWARD_ONLY,
              java.sql.ResultSet.CONCUR_READ_ONLY);
stmt.setFetchSize(Integer.MIN_VALUE);
The combination of a forward-only, read-only result set, with a fetch size of Integer.MIN_VALUE serves as a signal to the driver to stream result sets row-by-row. After this, any result sets created with the statement will be retrieved row-by-row.

This is explained here http://dev.mysql.com/doc/refman/5.0/en/connector-j-reference-implementation-notes.html

It seems SimpleJdbcTemplate doesnt have a setFetchSize method and JdbcTemplate has a setFetchSize but that doesn not work when you call the query method and if you use statement then the applySettings method applies fetchSize only if its >0. So the solution is to use PreparedStatement directly

callback = new PreparedStatementCallback() {
                @Override
                public Void doInPreparedStatement(PreparedStatement pstmt) throws SQLException, DataAccessException {
                    ResultSet rs = pstmt.executeQuery();
                    resultSetExtractor.extractData(rs);
                    rs.close();
                    return null;
                }
            };
        executeStreamed(jdbcTemplate, callback, sql);

    /**
    * http://dev.mysql.com/doc/refman/5.0/en/connector-j-reference-implementation-notes.html
    * Unless you specify the statement settings as below the mysql driver is going to load all results in memory.
    *
    * @param jdbcTemplate
    * @param callback
    * @param sql
    */
    protected void executeStreamed(SimpleJdbcTemplate jdbcTemplate, PreparedStatementCallback callback, final String sql) {
        PreparedStatementCreator creator = new PreparedStatementCreator() {
            @Override
            public PreparedStatement createPreparedStatement(Connection conn) throws SQLException {
                PreparedStatement pstmt = conn.prepareStatement(sql, java.sql.ResultSet.TYPE_FORWARD_ONLY,
                        java.sql.ResultSet.CONCUR_READ_ONLY);
                pstmt.setFetchSize(Integer.MIN_VALUE);
                return pstmt;
            }
        };
        jdbcTemplate.getJdbcOperations().execute(creator, callback);
    }

self signed certificates and chrome caching

2012-02-06T22:27:00.000-06:00

Ran into an issue today where despite setting cache tags on a swf file, the chrome browser was downloading it everytime I moved from one page to other and come back to same page. This was working fine in Internet Explorer and FireFox. After spending an hour with firebug, pagespeed and fiddler I figured out that cache tags were set correctly but still chrome won't cache it. At last I found this issue reported to chrome developers http://code.google.com/p/chromium/issues/detail?id=103875 . It appears that because I was using a self signed certificate on local install, chrome wont cache any content until I use a valid certificate. So the solution is to generate a free certificate using https://www.startssl.com/ and use it locally to avoid this issue.

Sharing between Andriod phone and ubuntu

2012-02-01T22:44:00.000-06:00

I dont want to install facebook on my phone and I wanted to share some pics so had I a requirement to copy some pictures taken from my phone to my ubuntu drive. This assumes your android and laptop are on same wifi network. It seems its very easy to do it and we would use samba share to do it.

1) Install AndSMB app on andriod phone.

2) create a folder on your ubuntu server and Right click on it and say Share this folder as shown below.

3) Connect to this share from your android by using the smb app and give your username/password and leave the domain blank.

4) Now you can navigate to the samba share folder and select the files to upload.

Carrying over Log4j context in threadpool

2012-01-15T19:37:00.001-06:00

We use MDC feature of slf4j/log4j to automatically log the details about user/client that is making the request so we can trace the things happening in the thread.

    public static void setRequestId(String requestId) {
        if (requestId != null) {
            MDC.put("MDC_REQUEST_ID", requestId);
        } else {
            MDC.put("MDC_REQUEST_ID", "");
        }
    }

    public static void setUserInContext(long userId) {
        MDC.put("MDC_USER_ID", String.valueOf(userId));
    }

and the log4j appender users

    <appender name="RFA_ROOT" class="org.apache.log4j.rolling.RollingFileAppender">
        <rollingPolicy class="org.apache.log4j.rolling.TimeBasedRollingPolicy">
          <param name="FileNamePattern" value="${catalina.base}/logs/kp.log.%d{yyyy-MM-dd-HH}"/>
          <param name="ActiveFileName" value="${catalina.base}/logs/kp.log"/>
        </rollingPolicy>
        <layout class="org.apache.log4j.PatternLayout">
            <param name="ConversionPattern" value="%p %d{ISO8601} %t D-%X{MDC_DOMAIN} U-%X{MDC_USER_ID} S-%X{MDC_SYNC_CID} R-%X{MDC_REQUEST_ID} %c - %m%n"/>
        </layout>
    </appender>

This will automatically print User and request Id but when you want to get the user requests funnel through some threadpool to avoid thundering herd problem then the log4j context is not carried over and this breaks traceability. To solve this problem the easy solution is to just create your own Callable and use that in thread pool.

public abstract class MyCallable implements Callable {
    private Map contextMap = AppLogger.getCopyOfContextMap();

    @Override
    public T call() throws Exception {
        MDC.setContextMap(contextMap);
        try {
            return internalCall();
        } finally {
            MDC.clear();
        }
    }

    protected abstract T internalCall() throws Exception;

}

        MyCallable task = new MyCallable() {
            @Override
            public String internalCall() throws Exception {
...
           }
        };

and use
SpringBeanLocator.getRestRequestHelper().getRestRequestExecutors().submit(task);

This way your context would be carried over to background threads.

smartsprite to generate sprite automatically

2012-01-15T18:55:00.000-06:00

I am a lazy person and like to automate things that can be automated. We had all our sprites hand written earlier and the problem with that approach is that it was append only. Every time you have to add a new image the developer would add it at the end but if I asked to remove an image then they would give me a look like what the heck I am talking about, they would give weeks estimate to go and fix coordinates in each and every place those images were used. Net net this seems like something that should not be that hard to automate and thats where smartsprite came to rescue. A very very handy tool to automatically generate your sprite and biggest benefit is that its non-intrusive and you can start small and keep cleaning your old code and convert to this mode as and when you change that part of code.   This is the link to the tool http://csssprites.org/

So all you need to do is add some comments at the top of your existing css
/** sprite: kpSmartSprite; sprite-image: url('../images/kpSmartSprite-${md5}.png'); sprite-layout: vertical; */

and then the css rule where you add image reference you have to add the comment

.t-add {
        background-image: url("../images/add.png") !important; /** sprite-ref: kpSmartSprite; sprite-alignment: left; sprite-margin-bottom:6px;*/
}

So you can keep adding these tags. Now you can use an ant target to generate your sprite. The tool will generate the sprite image and will also change you css file to add the coordinates.

    <target name="sprite-images" description="Creates a sprite using smartsprite by reading smart sprite annotations">
        <taskdef resource="smartsprites.xml">
          <classpath>
            <fileset dir="${thirdParty}/tools/smartsprites-0.2.8/lib">
              <include name="*.jar" />
            </fileset>
          </classpath>
        </taskdef>
        <smartsprites rootdir="${war.path}/css"
                      documentrootdir=""
                      outputdir="${war.path}/css-sprite"
                      cssfileencoding="UTF-8"
                      cssfilesuffix=""
                      loglevel="INFO"
                      spritepngdepth="AUTO"
                      spritepngie6="false" />
               <move todir="${war.path}/css" failonerror="no">
            <fileset dir="${war.path}/css-sprite" />
        </move>
</target>

Advantages of working remote

2011-11-20T09:35:00.001-06:00

There are many disadvantages of working remote but there are many advantages of working remote.

You are saving time on commute so you get more time for family if you are disciplined enough to start/stop work at office times (rarely happens on all days but still better than getting stuck in traffic)
Less people coming and distracting you by just walking into cubicle to discuss office politics
Less people coming and asking you things that they can just google for themeselves.
Due to the above points you get more work done in less amount of time.
You RTFM more than usual. If you are stuck on an issue you are the only one who can find the solution as the luxary of walking into someone's cubicle for help is gone, In a sense its a double edged sword but as you are left with no choice in the end you come up as winner and you become more and more of problem solver on your own. This way you tend to research things in detail and increase your arsenal of skills.

openoffice and NFS file saving issue

2011-11-15T17:15:00.001-06:00

I recently integrated a file preview application in our application with team, so now users can preview most of the files without downloading them. The hard part was to deal with NFS issues due to locking and caching. We chose to buy v/s build for the file preview and bought some third party service. The third party service lets call it APreview had a http api where you can pass the source and target file path. It would have been much better if we could stream the input and it could stream the output but that option was not there. Because we can only pass paths to it the natural solution was to use NFS paths. So we ran into two major issues:

1)The APreview application internally uses openoffice to convert word/PPT/Xls files to pdf and then it converts pdf to swf. openoffice has some issues with writing to NFS and we could use vi and other tools to write files but openoffice would just refuse to save the file as pdf. Finally I found that commenting these two lines in /usr/lib64/openoffice.org3/program/soffice made the thing working
#SAL_ENABLE_FILE_LOCKING=1
#export SAL_ENABLE_FILE_LOCKING

you have to comment it out. setting this SAL_ENABLE_FILE_LOCKING=0 wont work and dont forget to comment out the 'export' also.

2) Serving file over NFS had issues because of NFS cache lag. The VM where APreview was installed would write the file on NFS mount but the tomcat that has to serve the file was seeing the file after 15 sec delay. We tried all sort of things and ultimately gave up on NFS to serve the file. We installed apache on the Filer-server where preview were stored and served the file over http using apache reverse proxy.

Disadvantages of working remote

2011-11-15T15:23:00.001-06:00

Just ranting it out loud, I am working remote for the past two years for a startup and there are many advantages of working remote but there are many disadvantages also.

Its hard to chase people over phone. As people wont pick up the call or they wont reply to Instant Messages. So you are stuck in your chain of thoughts cursing the monitor.
You miss the coffee talk.
You miss what's going on in the office and really dont know whats going in the company.
In meetings you could be writing code and its easy to get distracted over phone.
Some people prefer to chat over talk and thats a pain because you already miss the social connection and typing is always a pain.
You miss all company dinners and lunches.
Because you save time on travel you tend to overwork.
Worse if you are working in a different timezone then people would disturb you off hours.
Lots of time is spent in trying to screen share or do go to meeting.
People will prefer to talk to person in house and unless you are a rockstar and only you can deliver the feature mostly people will prefer in house person.
Lots of things will get discussed in adhoc passageway meeting and you would miss all of that.
Dealing with HR is a pain. As you are remote you dont have the luxary of walking into their cube and getting your work done. I am literally screwed on every work that needs to flow through HR. Communicating with them on email leads to frustration and all you can do is curse your monitor.

Also check my post on Advantages of working remote

Jesey writing an authentication filter

2011-11-04T23:13:00.002-05:00

It seems there are two ways to add authentication to Jersey REST apis

1) You can add a servlet filter.

public class RestAuthenticationFilter implements Filter {
    @Override
    public void destroy() {
        // TODO Auto-generated method stub        
    }
    @Override
    public void doFilter(ServletRequest req, ServletResponse res, FilterChain chain) throws IOException, ServletException {     
    try {
           User user = BasicAuthHelper.authenticateUser(request);
            if (user == null) {
                response.sendError(HttpServletResponse.SC_UNAUTHORIZED);
            } else {
                request.setAttribute("user", user);
                chain.doFilter(request, response);
            }
     } catch (ApplicationException e) {
            response.sendError(HttpServletResponse.SC_INTERNAL_SERVER_ERROR, e.getMessage());
     }
    }
    @Override
    public void init(FilterConfig config) throws ServletException {
    } 
}

2) You can do it using the jersey filter. You have to implement a ResourceFilterFactory and handle the auth in ContainerRequestFilter. The detailed code is below. I like the approach 1 as it give complete lifecycle control. However if you need more specifc things like accessing QueryParams or PathParams then approach 2 is the way to go


public class RestAuthFilterFactory implements ResourceFilterFactory {
    private static final AppLogger logger = AppLogger
            .getLogger(RestAuthFilterFactory.class);

    @Context
    private UriInfo uriInfo;

    @Override
    public List create(AbstractMethod method) {
        return Collections.singletonList((ResourceFilter) new Filter());
    }

    private class Filter implements ResourceFilter, ContainerRequestFilter {
        protected Filter() {
        }

        public ContainerRequestFilter getRequestFilter() {
            return this;
        }

        public ContainerResponseFilter getResponseFilter() {
            return null;
        }

        public ContainerRequest filter(ContainerRequest request) {
            logger.info("Url invoked is {}", uriInfo.getPath());
            String authHeader = request.getHeaderValue("Authorization");
            if (authHeader != null && authHeader.startsWith("Basic ")) {
                   User user = BasicAuthHelper.authenticateUser(request);
               if (user == null) {
                  throw new WebApplicationException(Response.Status.UNAUTHORIZED);
               }
            return request;
            }
              throw new WebApplicationException(Response.Status.UNAUTHORIZED);
        }
        }
}

Jersey Rest API html documentation

2011-10-30T23:00:00.000-05:00

There are various ways to generate REST api documentation but how can you do it with minimal effort. Unless you are looking into very sophisticated documentation like twitter https://dev.twitter.com/docs/api, the below should suffice. If you though however want to generate twitter like documentation then also 70% of what I highlighted would work.

Jersey by default supports WADL generation by default. WADL is machine readable rest api description simliar to what WDSL was to soap. In order to get to default wadl generated by jersey you can just call http://localhost:8080/application.wadl . This WADL is a XML document so all you need now is to render it with a stylesheet. WADL has documentation tags that can be used to document a resource byt unfortunately jersey by default doesnt generate documentation tags.

So to get real html documentation the steps are:
1) Document you rest services using Java docs for the resource class and resource method. The down side of using java docs is that you will have to write the params in the method instead of declaring them as instance vars but its much better as you can kill two birds in one shot. You get javadocs for internal devloper and html documentation for rest api consumers.

@Path("/user")
public class UserRestService extends SecureRestService {
 /**
  * Gets user by userName.
  *  
  * An example of the output json is 
  *  
  *  {"prefix":"kpatel",
  *  "userNum":11,
  *  "firstName":"Kalpesh",
  *  "lastName":"Patel",
  *  "emailAddress":"kpatel@gmail.com"}
  *
  * @param userName UserName that is to be fetched. 
  * @response.representation.200.mediaType application/json
  * @response.representation.200.example  
  * @return the user json object
  * @throws IOException
  */
 @GET
 @Produces(MediaType.APPLICATION_JSON)
 @Path("username/{username}")
 public String getUser(@PathParam("username") String userName) throws IOException {
  ....
 }
}

2) Extract Javadocs to a resource-doc file to be used by jersey wadl generator. We would use a ant task to generate this file and place it in a path that can be read from the classpath by the running application.

    <target name="check-resourcedoc-exists">
        <available file="${basedir}/build/jersey-doc-dep/resourcedoc.xml" property="jersey.resourcedoc.present"/>
    </target>
    <target name="setup-jersey-wadl-dependencies" depends="check-resourcedoc-exists" unless="jersey.resourcedoc.present">
        <delete dir="build/jersey-doc-dep" />    
        <mkdir dir="build/jersey-doc-dep" />
        <javadoc access="public" classpathref="compile.classpath">
            <fileset dir="modules/server/src/java" defaultexcludes="yes">
                <include name="**/*.java" />
            </fileset>
            <fileset dir="modules/server/build/gen-src/java" defaultexcludes="yes">
                <include name="**/*.java" />
            </fileset>
            <fileset dir="modules/ui/src/java" defaultexcludes="yes">
                <include name="**/*.java" />
            </fileset>
            <doclet name="com.sun.jersey.wadl.resourcedoc.ResourceDoclet" pathref="compile.classpath">
                <param name="-output" value="${basedir}/build/jersey-doc-dep/resourcedoc.xml" />
            </doclet>
        </javadoc>
        <copy todir="${war.path}/WEB-INF/classes">
            <fileset dir="modules/server/resource/jersey-doc-dep">
              <include name="*.xml"/>
            </fileset>
            <fileset dir="build/jersey-doc-dep">
              <include name="*.xml"/>
            </fileset>
        </copy>        
    </target>

3) Add static documentation for the whole app and any a grammar file. This has to be hand written. I won't desrcibe the grammar file in details but it can be used for jaxb examples using some custom jersey example tags .
This is the application-doc.xml file

<applicationDocs targetNamespace="http://wadl.dev.java.net/2009/02">
    
    <doc xml:lang="en" title="The doc for My app API">
        My app documentation. You need to fill this in.
    </doc>

</applicationDocs>

This is the application-grammars.xml file

<grammars xmlns="http://wadl.dev.java.net/2009/02"
    xmlns:xsd="http://www.w3.org/2001/XMLSchema"
    xmlns:xi="http://www.w3.org/1999/XML/xinclude">
    <include href="schema.xsd" />
</grammars>

4) Configure jersey WADL generator to read resourcedoc.xml file and generate documentation tags in the wadl.

Add an init-param to your jersey filter

            <init-param>
                <param-name>com.sun.jersey.config.property.WadlGeneratorConfig</param-name>
                <param-value>com.foo.bar.MyWadlGeneratorConfig</param-value>
            </init-param>

Add this class


public class MyWadlGeneratorConfig extends WadlGeneratorConfig {
    @Override
    public List configure() {
        return generator(MyWadlGenerator.class).generator(WadlGeneratorApplicationDoc.class).prop(
                "applicationDocsFile", "classpath:/application-doc.xml")
                .generator(WadlGeneratorGrammarsSupport.class).prop(
                        "grammarsFile", "classpath:/application-grammars.xml")
                .generator(WadlGeneratorResourceDocSupport.class).prop(
                        "resourceDocFile", "classpath:/resourcedoc.xml")
                .descriptions();
    }

}

5) Inject html generating stylesheet. You can use the two stylesheet
https://github.com/mnot/wadl_stylesheets/ or https://github.com/ipcsystems/wadl-stylesheet

I prefered the ipc one as I didnt had initial documentation for each method and this one looked more better.

Doing this was the one thing took most of the time. At the end I ended up writing my own WadlResource.


@Produces( { "application/vnd.sun.wadl+xml", "application/xml"})
@Singleton
@Path("application.wadl")
public final class WadlResource {

 private static final AppLogger logger = AppLogger
   .getLogger(WadlResource.class);

 private WadlApplicationContext wadlContext;

 private Application application;

 private byte[] wadlXmlRepresentation;

 public WadlResource(@Context WadlApplicationContext wadlContext) {
  this.wadlContext = wadlContext;
 }

 @GET
 public synchronized Response getWadl(@Context UriInfo uriInfo) {
  if (wadlXmlRepresentation == null) {
   String styleSheetUrl = uriInfo.getBaseUri().toString() + "wadl.xsl";
   this.application = wadlContext.getApplication(uriInfo).getApplication();
   try {
    Marshaller marshaller = wadlContext.getJAXBContext()
      .createMarshaller();
    marshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, Boolean.TRUE);
    ByteArrayOutputStream os = new ByteArrayOutputStream();
                Writer writer = new OutputStreamWriter(os);
                writer.write("\n");
                writer.write("\n");
                marshaller.setProperty(Marshaller.JAXB_FRAGMENT, Boolean.TRUE);
    marshaller.marshal(application, writer);
    writer.flush();
    writer.close();
    wadlXmlRepresentation = os.toByteArray();
    os.close();
   } catch (Exception e) {
    logger.error("Could not marshal wadl Application.", e);
    return Response.ok(application).build();
   }
  }

  return Response.ok(new ByteArrayInputStream(wadlXmlRepresentation))
    .build();
 }
}

If you want twitter like documentation where you want to categorise resource

and other stuff then all you need now is to skip step 5 and add an ant task

to generate wadl with docs and feed it to a python program that can generate

the html doc at build time.

Generating pdf out of html using Java

2011-10-15T17:23:00.000-05:00

I had this requirement of generating PDF out of HTML content. Users can add any kind of html content on a Rich text editor from few lines to a full paragraph as notes for a file and save it on the server. Now I had to notify other users when some user adds a note. Now in the past I have generated PDF for reports but that was on a structured data straight out of database so I could easily use something like itext or jasperreports but this one seems an interesting problem as user can do any free form html in the editor.

Ultimately the solution came to be:

Convert the html added by user into xhtml using JTidy
Use flying saucer's ITextRenderer (http://code.google.com/p/flying-saucer/) to generate the pdf out of xhtml.

Here is a sample code


public class TextSectionConverter {

 private String notesContent;

 public TextSectionConverter(String notesContent) {
  this.notesContent = notesContent;
 }

 public void writeAsPdf(FileOutputStream fos)
   throws Exception {
  convertToXHTML();
  ITextRenderer renderer = new ITextRenderer();
  renderer.setDocumentFromString(notesContent);
  renderer.layout();
  renderer.createPDF(fos);
 }

 void convertToXHTML() throws Exception {
  notesContent = "" + notesContent + "";
  StringWriter writer = new StringWriter();
  Tidy tidy = new Tidy();
  tidy.setTidyMark(false);
  tidy.setDocType("omit");
  tidy.setXHTML(true);
  tidy.setInputEncoding("utf-8");
  tidy.setOutputEncoding("utf-8");
  tidy.parse(new StringReader(notesContent), writer);
  writer.close();
  notesContent = writer.toString();
 }

biggest concern with moving your applications to the cloud?

2011-09-10T01:30:00.001-05:00

wow I wrote about enterprise cloud adoption and found this poll on my linkedin profile that validates my thinking of security and reliability being biggest concerns for cloud adoption by enterpries.

Enterprise customers and Cloud Adoption

2011-09-09T22:42:00.001-05:00

A big fear among enterprise customers who want to adopt cloud is that

Is my data secure?
What if this site goes down for 3-4 hours?
What if this startup shutdown business, what will happen to my data?

Recent downtimes from big companies as shown below instills a fear among mission critical busineses wanting to adopt cloud.

While moving to cloud makes sense economically but having one bad day can make people go back to traditional methods of manging own infrastructure. Think of a hospital storing all records in cloud, it cant afford a 2-3 hour downtime. There is a better solution to this and its by adopting the hybrid model and companies that will do it successfully will see far more growth in enterprise area. The idea is to have an on-premise cloud that syncs to the real cloud. This way even if the real cloud is down or goes out of business, the on-premise instance ensures business continuity.

Making Junit tests faster

2011-09-09T22:32:00.000-05:00

I had some batch of Junit tests that were taking close to 20 minutes and it was wasteful to wait everytime before checking in code, one trick I found was to use the forkmod="once" property. Using this reduce the time to 5 minute 8 sec.

<junit tempdir="build" printsummary="on" fork="yes" forkmode="once" haltonfailure="${test.haltonfailure}" failureproperty="tests.failed" showoutput="false">

Earlier Junit was forking a JVM per test and now its doing one for all test so its very fast. Here is some documentation from Junit on this property.

Controls how many Java Virtual Machines get created if you want to fork some tests. Possible values are "perTest" (the default), "perBatch" and "once". "once" creates only a single Java VM for all tests while "perTest" creates a new VM for each TestCase class. "perBatch" creates a VM for each nested and one collecting all nested s. Note that only tests with the same settings of filtertrace, haltonerror, haltonfailure, errorproperty and failureproperty can share a VM, so even if you set forkmode to "once", Ant may have to create more than a single Java VM. This attribute is ignored for tests that don't get forked into a new Java VM. since Ant 1.6.2

Selenium and ExtJS

2011-09-09T22:25:00.001-05:00

A trick to test selenium with ExtJS is to use cssSelectors. As an element can have more than one css classes and you really dont need to define any style for that css class it can be a good locator for the element and faster than XPATH on IE.

You can define cssSelector as

        tbar: {
            xtype: 'toolbar',
            items: [
                {
                    xtype: 'button',
                    text: 'Send',
                    cls: 'x-btn-text',
                    overCls: 'x-btn-noicon',
                    ctCls: 't-btn-yellow x-btn-over',
                    iconCls: 't-send seleniumSendNote',
                    ...........
                    ...........
and you can then in your test call the button click as

        driver.findElement(By.cssSelector("button.seleniumSendNote"))
                .click();

Selenium and ExtJS HtmlEditor

2011-09-09T22:19:00.003-05:00

I had to add a selenium test for a page with ExtJS HtmlEditor and selenium wont recognize it, even tests recorded with Selenium IDE wont recognize it. The reason is that HtmlEditor uses a hidden textarea and a DIV on top of it to trap keystrokes. I tried using lots of ways to set text into it but it would complain about component not visible and other stuff. Finally the only way I could do is to execute Javascript from webdriver. Here is the code that I used to set the text.


                String notes = "This is a test note from selenium";
		JavascriptExecutor js = (JavascriptExecutor) driver;
		js.executeScript("Ext.getCmp('notes').setValue('"
				+ notes + "')");

Java and lsof

2011-08-29T16:38:00.000-05:00

Update: I found that using Runtime.exec was a bad idea because if you have a 2GB VM footprint then the forked process would require 2G free memory in order to run the lsof command. We had earlier writte a simple python http rpc server that would allow us to execute native commands(like creating a hardlink or running gunzip) from Java and I changed this code to delagate to RPC call few days back.
So the new code looks like



 public void writeTopCommandOutput(Writer writer) throws IOException {
  String rpcRes = Util.doCommandRpc(rpcUrl, "", ListUtil.create("top", "-n", "2", "-b", "-d", "0.2"));
  writer.write(rpcRes);
 }

 public void writeLsofOutput(Writer writer) throws IOException {
  String pid = getJvmProcessid();
  if (pid != null) {
   pid = pid.trim();
   String rpcRes = Util.doCommandRpc(rpcUrl, "", ListUtil.create("lsof", "-p", pid));
   writer.write(rpcRes);
  }
 }

--------------------------------------
I was chasing a customer issue and some threads in threaddump showed that they are stuck doing filer I/O. Now I needed to chase what file paths these threads are accessing as few days ago this customer had expereinced same issue. So I asked operations to give me lsof output and as we are a distribured team and engineers dont have access to production machines.It always takes time to chase people and to a programmer this means lots of context switches and it derails your thought process. My goal is to eliminate as many hoops in my debugging path so I wrote a JSP to get me lsof output from java. This will be a jsp accessible through internal ips only, hurray from next release onwards one more reason to avoid Operations team in chasing issues.
Here is the method I added in jsp

    public void writeLsofOutput(Writer writer) throws IOException {
        String pid = getJvmProcessid();
        if(pid!=null){
            pid = pid.trim();
            String cmdLsof = "lsof -p " + pid;
            executeCmd(writer, cmdLsof);            
        }
    }

    private void executeCmd(Writer writer, String cmd) throws IOException {
        String output;
        Process child = Runtime.getRuntime().exec(cmd, new String []{});            
        BufferedReader stdInput = new BufferedReader(new InputStreamReader(child.getInputStream()));
        while ((output = stdInput.readLine()) != null) {
            writer.write(output + "\n");
            writer.flush();
        }
        stdInput.close();
    }
    public String getJvmProcessid() throws IOException {
        String pid = null;
        File pidFile = new File(System.getenv("CATALINA_PID"));
        if(pidFile.exists()) {
            FileInputStream fin = new FileInputStream(pidFile);
            List lines = IOUtils.readLines(fin);
            fin.close();
            pid = StringUtils.join(lines.toArray());
        }
        return pid;
    }

Programatically extracting quoted reply from an email

2011-08-13T00:11:00.000-05:00

When files are uploaded to our cloud file server, we wanted to send notification email per file with its own unique email address. I will discuss how to have so many unique email address without creating a user on mail server for each file and scale out the solution in some later blog. People can just hit reply button on the generated notification email and comment on the uploaded file. When reply email reaches back the server we want to extract the comment that user added after stripping out the quoted reply form the mail client and add the clean comment to file. Seems like an easy problem isnt it, but unfortunately there is no easy way to detect the quoted reply from an incoming email because different mail clients use different way to quote a reply. On top of it quoted reply of html emails are different than plain text quoted replies.

Angle Brackets "> xxx zzz"
"---Original Message---"
"On such-and-such day, so-and-so wrote:"
html email reply in thunderbird uses blockquote tags.
yahoo/hotmail uses some div tags

Got an brilliant idea from someone to add a hash marker in the outbound notification email so that when it comes back we can strip the text after that marker. Then I found other sites are already doing this like redmine or issueburner already does that. These guys add a marker text in outbound email like below

##### Please do not write below this line #####
Hi kalpesh,

The issue has been updated.

Updated by: Kris Katta
Comment added: this is a test comment
Kris Katta's Reply..

This is a test reply

To track the status of your request and set up a profile for yourself, follow the link below:

Now all that is left to extract the mail header so using some regex you can strip that. I have handled thunderbird and outlook and will soon add yahoo/hotmail. Below is some sample code.

/**
 * @author kpatel
 */
public class QuotedReplyExtractor {
    public static final String 
REPLY_MARKER = "--- Please reply ABOVE THIS LINE to comment on this file ---";

    private static final List patterns = new ArrayList();
    static {
        patterns
                .add(Pattern.compile(".*on.*?wrote:", Pattern.CASE_INSENSITIVE));
        patterns.add(Pattern.compile("-+original\\s+message-+\\s*",
                Pattern.CASE_INSENSITIVE));
    }

    public String stripQuotedReply(String comment) {
        int startIndex = comment.indexOf(REPLY_MARKER);
        if (startIndex > 0) {
            comment = comment.substring(0, startIndex);
        }
        for (Pattern pattern : patterns) {
            Matcher matcher = pattern.matcher(comment);
            if (matcher.find()) {
                startIndex = matcher.start();
                comment = comment.substring(0, startIndex);
            }
        }
        return comment;
    }

}

Spring MVC and Unicode characters

2011-08-10T22:29:00.000-05:00

We ran into an issue where some user tried entering Danish character ø in his first name and it was not updating properly in LDAP. First we thought its a LDAP issue but then I found that in some other page where we use DWR api the same character is getting updated in LDAP properly. Finally we nailed it down to Tomcat/Spring where even after doing

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />

the character encoding was not properly set on request.

Adding this filter solved the issue. This has to be first filter in web.xml,

otherwise it wont work, I had it earlier as the second filter and i wasted some time debugging it

	 <filter>
		<!-- Filter to handle content encoding for UTF-8 encoding, this has to be the FIRST FILTER, do not move -->
	    <filter-name>encodingFilter</filter-name>

<filter-class>org.springframework.web.filter.CharacterEncodingFilter</filter-class>
	    <init-param>
	        <param-name>encoding</param-name>
	        <param-value>UTF-8</param-value>
	    </init-param>
	    <init-param>
	        <param-name>forceEncoding</param-name>
	        <param-value>true</param-value>
	    </init-param>
	 </filter>
	 <filter-mapping>
	    <filter-name>encodingFilter</filter-name>
	    <url-pattern>/*</url-pattern>
	 </filter-mapping>

Technically all its doing is

request.setCharacterEncoding("UTF-8");
response.setCharacterEncoding("UTF-8");

Ideally this should be set by tomcat as I had the below set,

but its not working somehow so setting this filter made it work.

DWR was parsing the request body manually and thats why it was working and spring was not.

<Connector port="8080" URIEncoding="UTF-8" ......>

mysql execute immediate

2011-08-10T22:09:00.000-05:00

I had a requirement to generate a sharded schema where no of shards and no of tables in the shard were dynamic. Basically we wanted to shard one table so we ended up creating 8 schemas and each schema will hold 8 tables that are copy of the same schema. Now I didnt wanted to hand write the 64 table/schema creation statement so came up with this procedure that allows to dynamically build and execute sql queries. Oracle was so easy, mysql is a little bit verbose.

    drop procedure if exists create_rdb_tables;
    delimiter #
    create procedure create_rdb_tables()
    begin
    declare v_max int unsigned default 9;
    declare v_counteri int unsigned default 1;
    declare v_counterj int unsigned default 1;
      while v_counteri < v_max do
        while v_counterj < v_max do
            set @sql_text := concat('drop table if exists metadata_rdb_schema',v_counteri, '.metadata_rdb_t', v_counterj,';');
            prepare stmt from @sql_text;
            execute stmt;
            DEALLOCATE PREPARE stmt;
    
            set @sql_text := concat('CREATE TABLE metadata_rdb_schema',v_counteri, '.metadata_rdb_t', v_counterj, ' (user_id INT NOT NULL, object_id VARCHAR(255) NOT NULL,group_id VARCHAR(36) NOT NULL,entry_id VARCHAR(36) NOT NULL,created_time DATETIME NOT NULL, primary key(object_id))TYPE=innodb;');
            prepare stmt from @sql_text;
            execute stmt;
            DEALLOCATE PREPARE stmt;
            set v_counterj=v_counterj+1;
        end while;
       set v_counteri=v_counteri+1;
      end while;
    end #
    delimiter ;
    call create_rdb_tables();
    drop procedure if exists create_rdb_tables;

Finally it started becoming messy and kludgy so I ended up writing clean python code to generate the ddl :). This effort was a waste but I ended up learning how to do dynamic sql execution in mysql.

ProcessId 911

2011-08-09T15:11:00.000-05:00

call it good luck I got process id 911 for java. :). Hiding my laptop name to keep it anonymous.

Tomcat configurable session timeout per customer

2011-08-02T22:41:00.000-05:00

We are a cloud file provider and more geared towards enterprise customers. We have a default session timeout of 6 hours for web ui access and recently customers had a requirement that they wanted to configure a session timeout themselves. As we host multiple customers on one node, this was an interesting requirement and we were discussing all sorts of hacks until I landed on to this api HttpSession.setMaxInactiveInterval.

So now all we need to do is upon successful login, check if admin has overridden session timeout settings for this enterprise and set that on session using the above api.

Tomcat printing catalina pid

2011-08-01T17:04:00.000-05:00

If you are hosting more then one tomcats on a physical box in production then lot of times you might want to see the process id of running instance. We dump jstacks/top command output in a folder every 5 minutes and this helps in correlating it. Here is a sample code to dump tomcat pid.



 public String getJvmProcessid() throws IOException {
  String pid = null;
  File pidFile = new File(System.getenv("CATALINA_PID"));
  if(pidFile.exists()) {
   FileInputStream fin = new FileInputStream(pidFile);
   List lines = IOUtils.readLines(fin);
   fin.close();
   pid = StringUtils.join(lines.toArray());
  }
  return pid;
 }

Mysql auto audit column

2011-07-29T23:55:00.000-05:00

Thanks to my colleague Deepak, today I learnt a new thing. If you want a column in your table like lastModifiedTime that gets inserted when the row is created and also auto updated whenever someone updates the row then traditionally the only way to do this was to use a trigger. I have used oracle for 4-5 years so I thought this is not possible to do in DDL and the trigger is the only way. But Mysql has this magic. You can create a column like

lastModifiedTime TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,

And this column will have current datetime on insert and will get updated with current datetime anytime a mutation to that row occurs. Wow no more triggers and this column is self maintained so no more worry about some dba disabling the trigger and did the update to get data inserted quickly.