John Toman's Tech Ramblings

A servlet filter for HTTP PUT (and network cameras)

2011-02-06T23:28:00.000-08:00

I purchased an IP camera recently to use and test for near-real time weather pictures on the WeatherCurrents.com web site. After a fair amount of research, the model I chose was a Panasonic BB-HCM511A, which provides low-light color images at night, and power-over-ethernet (POE) capability for simplified cabling.

One of the web camera's features is the ability to push out images at regular intervals over the internet using FTP or HTTP. Since I don't want to maintain a FTP server for WeatherCurrents, I chose HTTP. The screen allows configuration of a URL, and a username and password (for basic HTTP authentication).

It doesn't allow for TLS/SSL, unfortunately, which I consider to be a security mistake.

When I tested posting an image to Apache Tomcat (version 6.0.32), I found it was attempting to upload the image with an HTTP PUT. The PUT, DELETE and TRACE commands are built in to the default servlet but are off in the default configuration.

To turn them on in Tomcat, the web.xml file in the conf directory would need to be modified:

<servlet-name>default</servlet-name>

<servlet-class>org.apache.catalina.servlets.DefaultServlet</servlet-clas

<init-param>

<param-name>debug</param-name>

<param-value>0</param-value>

</init-param>

<init-param>

<param-name>listings</param-name>

<param-value>false</param-value>

</init-param>

<init-param>

<param-name>readonly</param-name>

<param-value>false</param-name>

</init-param>

<load-on-startup>1</load-on-startup>

</servlet>

If enabled, this would have the effect of opening the floodgates to PUTs and DELETEs, and only careful control over the PUT and DELETE methods through <security-contraint> clauses in the web.xml file would prevent abuse.

In the production version of WeatherCurrents.com, Tomcat is proxied behind Apache HTTPD. Rather than let Apache handle the PUT method, I wanted programmatic control over it.

So I built a servlet filter to handle HTTP PUT request from the network camera. Since filter chains are executed before any servlet (the DefaultServlet included), this would allow fine-grained control, and avoid having to use, modify or configure the DefaultServlet in any way.

The design of the filter takes into account the following:

Only look at authenticated PUT method requests. Everything else passes through gracefully.
To prevent abuse, use an upper limit on the size of the content payload. I chose 100,000 bytes arbitrarily. The actual images, in 640x480 format from the camera, are 35k or less.
Recognizing that the system may want the uploaded picture in a specific place, I provided a way to configure a directory to place it. On Linux systems, this directory must have permissions for Tomcat to write in it.
Create a role in Tomcat for the web camera. I called it "camera".
Create a username and password for HTTP Basic authentication, and bake the authentication pattern into the web.xml file, again targeting only the PUT method.
Make sure the content that we get is a JPEG image. For this, I took and modified some code originally developed for Apache Cocoon and released under the Apache 2.0 license.

Here's what the filter looks like. This filter is a simplified version of the actual one in use in the WeatherCurrents system:

public class PostNetworkCameraFilter extends HttpFilter
{
   // Sidestep DOS attacks with large files
   private static final int MAX_BYTES_LENGTH = 100000;
   private String basePath;

   @Override
   public void init(FilterConfig config) throws ServletException
   {
   Properties props = FilterUtility.configToProperties(config);
   basePath = props.getProperty("base-directory",
   "/opt/apache-tomcat/webapps/static/cameras/");
   }

   @Override
   public void doHttpFilter(HttpServletRequest request,
   HttpServletResponse response, FilterChain chain)
   throws IOException, ServletException
   {
   Logger logger = Logger.getLogger(getClass());

   // Examine PUT requests only
   if (request.getMethod().equals("PUT"))
   {
   // Check the length of the submission. No large files get through.
   if (request.getContentLength() > MAX_BYTES_LENGTH)
   {        response.setStatus(HttpServletResponse.SC_REQUEST_ENTITY_TOO_LARGE);
   logger.error("uri: " + request.getRequestURI() + " size "
   + request.getContentLength()
   + " is larger than maximum size " + MAX_BYTES_LENGTH);
   }
   else
   {
   byte[] content = contentToBytes(request.getInputStream(),
   request.getContentLength());
   // Check for a legal JPEG submission
   ImageProperties imageProps = ImageUtils
   .getJpegProperties(new ByteArrayInputStream(content));
   if (imageProps == null)
   {
   logger.error("uri: " + request.getRequestURI()
   + " is not a valid jpeg file");
   response.setStatus(HttpServletResponse.SC_UNSUPPORTED_MEDIA_TYPE);
   }
   else
   {
   // Store the image where it can be accessed
   logger.debug("uri: " + request.getRequestURI() + " size: "
   + request.getContentLength()
   + " image properties: " + imageProps);
   storeImage(new ByteArrayInputStream(content), new File(
   basePath, request.getServletPath()));

   // Other things we might do:
   // 1. Check for an existing file and rename it

// 2. Notification for the rest of the system

}

else

{

// Pass through everything else

chain.doFilter(request, response);

}

private byte[] contentToBytes(ServletInputStream is, int length)

throws IOException

{

try

{

ByteArrayOutputStream os = new ByteArrayOutputStream(length);

try

{

byte[] buffer = new byte[4096];

for (int n; (n = is.read(buffer)) != -1;)

os.write(buffer, 0, n);

}

finally

{

os.close();

}

return os.toByteArray();

}

finally

{

is.close();

}

private void storeImage(InputStream is, File file) throws IOException,

FileNotFoundException

{

try

{

file.getParentFile().mkdirs();

OutputStream os = new FileOutputStream(file);

try

{

byte[] buffer = new byte[4096];

for (int n; (n = is.read(buffer)) != -1;)

os.write(buffer, 0, n);

}

finally

{

os.close();

}

finally

{

is.close();

}

Here's what needs to go in the web.xml file:

   
   <filter>
   <filter-name>Network Camera</filter-name>
   <filter-class>org.toman.webcam.PostNetworkCameraFilter</filter-class>
   <init-param>
   <param-name>base-directory</param-name>
   <param-value>/opt/apache-tomcat/webapps/static/cameras/</param-value>
   </init-param>
   </filter>
   <filter-mapping>
   <filter-name>Network Camera</filter-name>
   <url-pattern>*.jpg</url-pattern>
   </filter-mapping>

   
   <security-constraint>
   <display-name>HTTP PUT Authentication</display-name>
   <web-resource-collection>
   <web-resource-name>Cameras</web-resource-name>
   <url-pattern>*.jpg</url-pattern>
   <http-method>PUT</http-method>
   </web-resource-collection>
   <auth-constraint>
   <role-name>camera</role-name>
   </auth-constraint>
   </security-constraint>

   <security-role>
   <role-name>camera</role-name>
   </security-role>

   <login-config>
   <auth-method>BASIC</auth-method>
   <realm-name>Authentication Needed</realm-name>
   </login-config>

Note: the two pieces of code above are just snippets. The complete code is here, and you can use most of it under the terms of the FreeBSD license. Two classes are under the Apache 2.0 license. Gradle is used to build the source, but you can easily use Maven or Ant instead.

I haven't tried this in Jetty, or in Glashfish, but there's every reason to think it will work in those servlet containers the same way as Tomcat.

A default robots.txt files for all sites

2011-01-11T12:17:00.000-08:00

I recently registered an alternate domain, wcstat.com, in order to serve the static content for weathercurrents.com from a cookie-less domain, and to improve page load times.

I moved all my static content (images, videos, CSS files, JavaScript files) to this domain and configured it in Apache, then began the process of pointing . (I also left the static content in its old places (http://static.weathercurrents.com and http://weathercurrents.com/static/) to avoid breaking any pages. Once the move is complete, I'll follow up with a couple of mod_rewrite rules.)

Looks like I forgot the robots.txt file. Here's an excerpt from this morning's error logs:

[Tue Jan 11 01:52:48 2011] [error] [client 124.115.6.10] File does not exist: /webdirectory/static/robots.txt

Drat.

Fortunately, the solution is an easy one, solved with a default robots.txt file in the root directory for wcstat.com:

User-agent: *

Disallow:

The lesson is that everyone should have one of these with at least this in it for every unique domain they own/serve. Another best practice is to include a Sitemap: entry in it too, at the very top. I would argue that this site, since it's serving only static content referenced from dynamic pages elsewhere, does not need one of those.

I should also add that another common 404 is on favicon.ico icons. Every web site should have one of those, too.

Apache Ant and Multi-file CSS / JavaScript Minification with YUI Compressor

2011-01-02T22:39:00.000-08:00

Recently for my web site I found I needed to use a CSS / JavaScript compressor, as part of a strategy to decrease pageload times. The goals were:

Keep the original CSS and JavaScript in my source tree readable.
Convert about 15 or so CSS files (one for each neighborhood of the web site) and a half a dozen JavaScript files (versioned as 0.1, 0.2, 0.3, etc.) as part of an automated build.
Deploy the compressed (or minified) versions of all the CSS and JavaScript files to staging and then to production.

Scanning what was out there, I settled on Yahoo's YUI Compressor, a decent fit because it's available as a Java program that I could run as part of my builds, which are currently based on Apache Ant (I'm in the process now of switching to Gradle, but that's a topic for another time).

So, on to the challenge.

I started with Adam Presley's blog entry, which details how to use YUI's Ant task to minify one CSS file and one JavaScript file, and generalized it to work on a set of CSS and JavaScript files. The other pre-requisite is the ant contrib library (to pick up the foreach task for iterating over a set of files).

Assumptions:

CSS files are in a source directory (the property css.src.dir below), and are built (i.e. minified / compressed) into a second build directory (the property css.build.dir below), without changing the file name.
Ditto for the JavaScript files (compress from scripts.src.dir to scripts.build.dir).

Here's a snippet of the code I came up with:

</target>

</java>

</target>

</target>

</java>

</target>

</target>

<path>

</path>

</foreach>

<path>

</path>

</foreach>

</target>

The main tasks are minifyCSS and minifyJavaScript, which work on one file at a time. Before invoking the YUI compressor, the uptodate task checks to see if the current compressed file's build time is not as recent as the source file.

Iteration over all of the files to be minified is taken care of by the minify task.

Note that this implementation still forks a JVM for each and every file that is minified.