Atwofer

Simple Caching using Guice

2011-01-31T10:26:00.000-08:00

I am at Google now so I decided to use Guice as the framework for my dependency injections needs. I am a big fan of Spring so I was a little concerned that Guice couldn't live up to my expectations. However, after having used it for the last 6 months or so I have to say that I am a fan. Gone are those long spring XML configuration files and replaced with Guice modules.

Recently, while working on my current project we found that some requests against our server were slow. Using my Guice-ified logging framework (maybe another post?) I was able to determine that we were making way too many queries against our persistence layer. Just looking at the queries we could see that a lot of the queries were exactly the same. Immediately, this seems like a problem that could be solved by some caching. Since the main problem was too many persistence store queries per request, I thought that a simple request scoped cache could go a long way and Guice allowed me to do this very easily. The advantage of making this a request scoped cache I really don't need to worry too much about invalidation or thread safety.

The first thing I needed to do was to determine which persistence calls I want to cache. My persistence class is below and I have annotated the cacheable methods with @Cache in addition I have annotated the class with @Cacheable.

@Cacheable

@Singleton
public class {

@Cache(queryType = QueryType.LOOKUP_BY_KEY)
public Object lookup(Key key)

  // Does lookup against persistence store and returns result.
}

public Object save(Object resource)

  // Saves to the persistence store.

The annotations are defined as follows:

@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.PARAMETER, ElementType.TYPE})
public @interface Cacheable {
}

@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.PARAMETER, ElementType.METHOD})
public @interface PersistenceContextCache {
  /**
   * Used to give the interceptor a hint on how to cache the
   * result of the intercepted method.
   */
  public enum QueryType {
   FIND_BY_KEY,
   LIST,
   LOOKUP_BY_KEY,
   SEARCH
  }

  QueryType queryType();
}

Because I want to be able easily disable my caching layer and not let my caching logic invade my persistence code, I am using an interceptor to implement my caching logic.

/**
* Intercepts calls to the persistence tier and checks if we
* already have the result in the cache. If it is in the cache
* then the call to the persistence tier is skipped and the
* cached result is returned. Otherwise, we call through and
* cache the result.
*/
public class CacheInterceptor implements MethodInterceptor {
  /**
   * Provides the cache which is request scoped so we don't have
   * to concern ourselves with thread safety.
   */
  @Inject
  private Provider<Cache<CacheKey, Object>> cacheProvider;

  @Override
  public Object invoke(MethodInvocation invocation)
   throws Throwable {

   // Construct the cache key
   Cache cacheAnnotation =
   invocation.getMethod().getAnnotation(
   PersistenceContextCache.class);
   CacheKey cacheKey = getCacheKey(
   cacheAnnotation.queryType(),
   invocation.getArguments(),
   invocation.getMethod());

   Object result = cacheProvider.get().getElement(cacheKey);

   // If the result is null, then we have a cache miss so call
   // through
   if (null == result) {
   result = invocation.proceed();

   // Now add it to the cache
   cacheProvider.get().addElement(
   cacheKey, result != null ?
   result : new NullCacheEntry());
   }

   return result instanceof NullCacheEntry ? null : result;
  }
}

Now let's wire things together in a Guice module.

public class CacheModule extends AbstractModule {

  @Override
  protected void configure() {
   CacheInterceptor cacheInterceptor = new CacheInterceptor();
   // Intercept any calls to methods annotated with @Cache on
   // a class annotated with @Cacheable.
   bindInterceptor(
   Matchers.annotatedWith(Cacheable.class),
   Matchers.annotatedWith(Cache.class),
   cacheInterceptor);
  }

  @Provides
  @RequestScoped
  SoftHashMap<CacheKey, Object> provideCache() {
   return new SoftHaspMap<CacheKey, Object>();
  }
}

Some things to note in the above code:

Since my interceptor is essentially a singleton but my cache is request scoped, I need to use a provider which allows me to mix scopes.
I did not show my implementation of getCacheKey. However, it should be straight-forward to see how you can come up with a Cache Key based on the data I pass in.
I did not include my cache invalidation code but you should be able to see that you could simply add another interceptor to clear the cache on any write.
I use a NullCacheEntry object to represent a "not found" resource since a Map does not allow for null values.
My cache is actually a SoftHashMap which I do not show the implementation of. The idea is we want to be safe and not let our cache get too big and cause an OutOfMemoryException so these SoftReferences should be cleaned up if the memory gets low.

Eclipse + Pydev + Django

2010-03-21T10:13:00.000-07:00

So I decided that I wanted to learn Python. On top of that I wanted to learn Django. The catch is that I like to do everything from within Eclipse so that I can make use of the debugger. I couldn't find information about how to do all of this so instead I pieced it together here. My overall goal is to run through this Django Tutorial but this post will only deal with setting up Eclipse/Pydev and Django.

NOTE: Hopefully soon this post will be rendered useless when Pydev includes Django support.

Install Pydev

Since I use Eclipse for just about everything from Java to Ruby on Rails to ActionScript I obviously already had it installed so the first thing I needed to do was to install the Pydev Eclipse plugin. Pydev is installed just like any other plugin and instructions on how to install Pydev can be found here.

Install Django

Download Django from here then unzip it and navigate to the unzipped folder and run the following command.

python setup.py install

I am on a Mac with Python 2.6 installed so Django is installed at

/Library/Python/2.6/site-packages

Create a Django Project

Now I should be ready to create my project and the configure it to work with Eclipse/Pydev. First I need to navigate to my folder where I want to create the project and type the following.

django-admin.py startproject mysite

Now this results in the following

cd mysite/

ls

__init__.py     manage.py       settings.py     urls.py

As you can see all the project files are created at the root. Generally projects consist of more than just python files. Also, pydev likes the code to be in a "src" folder so lets move the python files to a "mysite" package in the "src" folder.

mkdir src
cd src
mkdir mysite
mv *.py src/mysite/.

Create the Pydev Project

Now I can launch Eclipse and create a new project by going to File-> New -> Pydev Project. For the directory I will select the project folder I created earlier.

Click "Finish" and now I have a Pydev project that looks like the following.

Add Django to the Project

The first thing I need to do now that I have the project is tell it where to find Django. I can do this by going to Project -> Properties. Here I need to add the installed location of Django as an External Library.

Starting My Server From Eclipse

In Eclipse I go to Run -> Run Configurations... and create a new Python Run configuration. Select my project and for the main module provide the path to the manage.py file.

Switch to the arguments tab and add the "runserver --noreload" argument.

Click "Apply" and now click "Run" and navigate to http://localhost:8000.

This is as far as I have gotten so far. Next I will actually try to code something.

Single Table Inheritance and accepts_nested_attributes_for

2009-11-02T09:42:00.000-08:00

I recently upgraded our web app at Episodic from Rails 2.0 to Rails 2.3.4. While this resulted in many issues we had to resolve that I will try to write about in later post, it does allow me to take advantage of some of the Rails 2.3.x features like accepts_nested_attributes_for. If you aren’t familiar with accepts_nested_attributes_for then check out Ryan’s post.

So I went ahead and tried to make use of this in my object model. Here are the important parts of the model.


class Episode < ActiveRecord::Base
has_many :field_values, :dependent => :destroy
accepts_nested_attributes_for :field_values, :allow_destroy => true
end 

class FieldValue < ActiveRecord::Base
:belongs_to :episode
end

class TextFieldValue < FieldValue
end

class NumberFieldValue < FieldValue
end

Since I am using Single Table Inheritance I end up with a field_values table that has a column named “type” which contain either a value of “TextFieldValue” or “NumberFieldValue”.

Also, because I enabled accepts_nested_attributes_for on my Episode class I should be able to do something like:


episode.field_values_attributes = 
[{:value => “foo”, :type => “TextFieldValue”}]

However, when I look in the DB I see that there is a new row but the “type” column is NULL even though I set it to “TextFieldValue”. When I look in the logs I see: "Can't mass-assign these protected attributes: type".

Luckily, there is a way around this. I added a setter called "value_type" to my FieldValue class.


def value_type= value_type
self.type = value_type
end

Now, when I can safely use field_values_attributes and set a type.


episode.field_values_attributes = 
[{:value => “foo”, :value_type => “TextFieldValue”}]

Handling redirects with initWithContentURL

2009-08-13T03:00:00.000-07:00

I keep running into this problem so I thought that I should post my solution. There are some classes in the iPhone SDK like the MPMoviePlayerController that have an initWithContentURL method. However, the problem is that the URL I have is one that may issue a redirect response when requested. When I provide such a URL to initWithContentURL the player tells me that it cannot play me file. To work around this problem I wrote some very simple code.

The first thing you need to do is to build a HEAD request using NSMutableURLRequest. Then create an NSURLConnection settings the delegate as the current instance.


NSMutableURLRequest *headRequest = 
[NSMutableURLRequest requestWithURL:resourceURL
cachePolicy:NSURLRequestUseProtocolCachePolicy timeoutInterval:60.0];
 
[headRequest setHTTPMethod:@"HEAD"];
 
connection = [[NSURLConnection alloc] 
initWithRequest:headRequest delegate:self];

Now we just need to implement the delegate method to play the URL from the response.


- (void)connection:(NSURLConnection *)connection 
didReceiveResponse:(NSURLResponse *)response {
 
// Now we have resolved the URL so play the movie
thePlayer = [[MPMoviePlayerController alloc] 
initWithContentURL:[response URL]];
 
[thePlayer play];
}

Obviously, this example handles one redirect but you can see that you can easily have it loop and until [response statusCode] == 200.

Streaming and playing fixed-length MP3s using the iPhone SDK

2009-06-30T08:49:00.000-07:00

The Problem

Lately, I've been doing a fair amount of coding using Objective-C and the iPhone SDK. Once I got used to the language I really started to like it. The more I used the SDK the more impressed I became with how much I could do with it. However, when a customer of ours at Episodic needed an iPhone App that could play a list of MP3s (RSS Feed to be exact) with "iPod-like" functionality, I was surprised to find that this is not something I can do with the embedded Quicktime Player. In fact, you can't even just hook into the MPMoviePlayerController. For example, there is no way to know when one songs ends so that I can start the next song since the MPMoviePlayerPlaybackDidFinishNotification notification is fired when the song ends or the user hits the done button. So after a lot of searching I came across Matt Gallagher's great post "Revisiting an old post: Streaming and playing an MP3 stream".

Modifying the AudioStreamer Class to Play Fixed-Length MP3s

So Matt's post got me part of the way there. It at least gave me an understanding of the tools I would need to use to accomplish my task. The main difference between what he had and what I needed is that he was playing an MP3 stream and I wanted to play MP3 files. The first change I made was I added a new constructor to the AudioStreamer class. This allowed me to initialized the class with a URL to a file on the file system (I'll talk about how to get the file on your file system later).

- (id)initWithFileURL:(NSURL *)aURL {

[self initWithURL:aURL];

fixedLength = YES;

return self;

}

This constructor sets a flag to indicate that we are streaming fixed-length MP3s. I then key off this flag in the openFileStream method.

if (fixedLength) {

stream = CFReadStreamCreateWithFile(

kCFAllocatorDefault, (CFURLRef)url);

} else {

CFHTTPMessageRef message = CFHTTPMessageCreateRequest(

NULL, (CFStringRef)@"GET", (CFURLRef)url,

kCFHTTPVersion1_1);

stream = CFReadStreamCreateForHTTPRequest(NULL, message);

CFRelease(message);

...

The above code simply creates the stream from the file system instead of a remote file. Because the result is still a CFReadStreamRef object the rest of the class mostly remains unchanged.

Downloading the MP3

In my controller I just need to download the MP3 file to a location on the file system. This can be done very easily by using a combination of NSURLConnection and NSFileHandle.

audioFile = [[NSFileHandle

fileHandleForWritingAtPath:downloadFileName] retain];

NSURLRequest *downloadRequest = [NSURLRequest

requestWithURL:resourceURL 

cachePolicy:NSURLRequestUseProtocolCachePolicy

timeoutInterval:60.0]; 

download = [[NSURLConnection alloc]

initWithRequest:downloadRequest delegate:self];

This code creates the file to write the data to and creates the connection to fetch the data. I know just need to handle the data.

- (void)connection:(NSURLConnection *)connection 

didReceiveData:(NSData *)data {

// Append the data to the file on the file system

[audioFile writeData:data];

downloadedLength += [data length];

 

// When we get enough of the file, then just start playing.

if (!streamer && downloadedLength >

     DEFAULT_MIN_LENGTH_TO_PLAY) {

NSLog(@"start playback for %@", downloadFileName);

[self initAndStartStreamer];

}

}

The above code is the NSURLConnection delegate method for receiving the data and writing the data to the file. Once I have "enough" of the MP3 I can start playing it by create the streamer and passing it the URL to our audioFile.

NEW: Handling EOF

One bug that I ran into since my initial post is that if I actually put this on my phone and went outside, away from my wireless network, the song would sometimes end early. What was happening is that the AudioStreamer was playing the file faster than it was being downloaded. The AudioStreamer would hit the EOF and then stop. To prevent this case I added some "throttling" to the playback loop that checks if we are too close to the end of the file and changes the state to buffering until we have enough to continue. The following code is in startInternal.

// Flag to indicate that we have gotten too close to the EOF
// before the entire file has downloaded so we need throttle

// the playback.

BOOL isThrottling = NO;{



//

// Process the run loop until playback is finished or failed.

//

BOOL isRunning = YES;

do

{

// If we are playing a fixed-length MP3 make sure we are

// not too close to the end of the file. This prevents us
// from hitting the end of the file before it is fully
// downloaded. Very useful when not on 3G since the song

// may be played faster than it is downloaded.

if (!fixedLength || self.fileDownloadComplete || 

self.fileDownloadCurrentSize > (fileDownloadBytesRead + 

(kAQBufSize * kNumAQBufs))) {



isRunning = [[NSRunLoop currentRunLoop

runMode:NSDefaultRunLoopMode

beforeDate:

  [NSDate dateWithTimeIntervalSinceNow:0.25]];



if (isThrottling) {

isThrottling = NO;

AudioQueueStart(audioQueue, NULL);

self.state = AS_PLAYING;

}

 

//

// If there are no queued buffers, we need to check here

// since the handleBufferCompleteForQueue:buffer: should
// not change the state(may not enter the synchronized

// section).

//

if (buffersUsed == 0 && self.state == AS_PLAYING)

{

self.state = AS_BUFFERING;

}

} else if (!isThrottling && self.state == AS_PLAYING) {

NSLog(@"Throttle because we are too close to EOF.");

 

self.state = AS_BUFFERING;

 

AudioQueuePause(audioQueue);

isThrottling = YES;

 

// Sleep for a few seconds

[NSThread sleepForTimeInterval:3.0];

}

 

} while (isRunning && ![self runLoopShouldExit]);

Seeking?

I was unable to figure out a way to allow for seeking. This would also provide a way for playback to pick up where it left off when the connection during the download phase. If anyone has ideas on how to support this I would love to hear about it.

Conclusion

This is still far from ideal and it really seems like this is something that should be a lot easier to do within the SDK. In the meantime, hopefully this is helpful to others.

You can download my source code here.

Getting the errorID from an ErrorEvent in ActionScript

2009-02-24T13:15:00.000-08:00

Our Episodic Video Player is written in ActionScript 3. I'm still fairly new to ActionScript but came up with a solution to a problem the other day that I thought I should share.

The player loads various files from the server. In some cases I want to display an error when one of those files can't be found. I have an error handler that listens for Error Events and displays an appropriate error message depending on the type of the error event. For example, I know that error code 2032 will generally mean in our case that the file could not be found.

Alright, so this all seems simple enough except for one problem. When I looked at the ASDocs for ErrorEvent.errorID I noticed that that property is not available for Flash Player 9. Fortunately, ErrorEvent extends TextEvent which has text property I can access. In the case of an ErrorEvent the text property contains a message that includes the error ID. Therefore, I just wrote my own getter to grab the error ID.

public function get errorID():int {
if ((text != null) && (text.indexOf("Error #") == 0) 
&& (text.length > 7)) {
// The Error IDs always seem to be 4 digits
return parseInt(errorEvent.text.substr(7, errorEvent.text.length - 7));
}

Override attribute accessor and before_type_cast

2009-02-17T12:45:00.000-08:00

Our Episodic Video Publishing Web Application is written in Ruby on Rails. I recently came across a situation where I had a model where I wanted to override it's attribute accessors in some cases. Consider a model that looks like this.

create_table "podcasts", :force => true do |t|
t.string   "name"
t.boolean  "use_show_name", :default => true
t.integer   "show_id"
end

As you can see above that each podcast belongs to a show. I also have a form for this model and in the name field I want to display the name of the podcast unless "use_show_name" is true then I want to display the name of the show that this podcast belongs to.

<%= text_field :podcast, :name, :class => "text", :disabled => @podcast.use_show_name %>

I don't want to put this logic in the ERB template but instead I want to put it in the model so that anyone that calls Podcast.name will get the appropriate value depending on the "use_show_name".

class Podcast < ActiveRecord::Base
belongs_to :show
 
def name
return self.use_show_name ? self.show.name : read_attribute(:name)
end
alias_method :name_before_type_cast, :name
end

So there a few things going on above.

We override the "name" method by simply implementing it. This works because ActiveRecord using method_missing to implement it's "name" getter but since there is now a "name" method, method_missing will not be invoked.

In the method we check "use_show_name" to see what value to return. If it is true we get the value from the show object. Otherwise, we call read_attribute to get the value of the name attribute that stored in the DB.

Lastly, we alias the name method as "name_before_type_cast" since this is the method that is called by FormHelper.text_field. ActiveRecord form helpers call the before_type_cast so that they can get the string value of the attribute but we want it to get the value from our "name" method.

Prompt User About Unsaved Changes Using Mootools

2009-01-02T06:41:00.000-08:00

Mootools has a very useful method toQueryString on the Form element class. In short, this method takes all the form fields and builds a query string generally used when sending form data to the server. However, I found another use for this method that I thought I would share.

For our Episodic Video Publishing Web Application we have a number of pages that include a form. If the user tries to navigate from a form that has unsaved changes then we want to prompt the user that they have unsaved changes and that these changes will be lost if they navigate away. There are various ways that you can detect if the user has modified the page's form but the easiest I've found is to make use of Form.toQueryString.

First, create a function that stores the form state in a hidden input.

getOrCreateInitialFormState: function(form) {
var formInitialState = null;,
var formInitialState = $('form_initial_state');
if (!formInitialState)
formInitialState = new Element('input', {
id: 'form_initial_state',
type: 'hidden',
value: form.toQueryString()
}).inject(form, 'before');
return formInitialState;
}

So when the page loads record the current form state and hook into the click event for any links on your page and check if the form state has changed.

window.addEvent('domready', function()
{
// Grab the form
var form = $('my_form');

// Store the form state in a hidden input
getOrCreateInitialFormState(form);

// Capture any clicks that would cause the user to navigate away
$$('a').addEvent('click', function(e) {
if (form.toQueryString() != getOrCreateInitialFormState(form) {
e.stop();
var result = confirm('You have unsaved changes.' +
'  Are you sure you want to navigate away from this page?');
if (result == true) {
// The user wants to navigate
window.location.href = a.get('href');
}
} 
});
});

That's it!

Using AOP to Make Calls Asynchronous in Java

2008-12-16T07:39:00.000-08:00

I am working on a project called ScoreOut that consists of an application that runs on a phone that interacts with a server via a REST API. The server is written in Java and makes heavy use of the Spring Framework. I wanted to start tracking some of the actions that a user takes on the phone and recording those actions in our analytics tables. For example, we want to record when a user logs in or requests a golf course. However, we do not want to slow down the response to the client since the customer is probably out on a golf course at that time. Therefore, it seemed to makes sense to perform any auditing asynchronously.

The server is broken up into services and I already had a service that did auditing. I added two new methods to it recordClientLogin and recordCourseRetrieval. I wanted these methods to be run asynchronously but I wanted the asychronousness (if this isn't a word it should be) to be reusable and easily enabled or disabled (easier to unit test when disabled). This immediately let me to use AOP so I wrote an interceptor.


package com.scoreout.service.interceptor;
 
import org.aopalliance.intercept.MethodInterceptor;
import org.aopalliance.intercept.MethodInvocation;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
 
/**
* Intecepts a call and makes it asynchronous. If there is an error it is
* simply logged.
*/
public class AsynchronizeInterceptor implements MethodInterceptor
{
 
/**
* Logger
*/
private static final Log LOG = LogFactory.
getLog(AsynchronizeInterceptor.class);
 
/**
* This method will always return null since the method is
* invoked asynchronously.
*
* @see org.aopalliance.intercept.MethodInterceptor
*          #invoke(org.aopalliance.intercept.MethodInvocation)
*/
public Object invoke(final MethodInvocation invocation) throws Throwable
{
Thread thread = new Thread()
{
public void run()
{
try
{
invocation.proceed();
} catch (Throwable e)
{
LOG.warn("Asynchronous method invocation failed.", e);
}
}
};
 
thread.start();
 
return null;
}
}

So this inteceptor will make any method it intercepts asynchronous running the method on a separate thread and returning immediately. So now I just needed to add the point cut to the spring application context.


<!-- Turns methods calls into asynchronous method calls -->
<bean id="asynchronizeInterceptor"
class="com.scoreout.service.interceptor.AsynchronizeInterceptor">
</bean>
 
<!-- Intercept caddie event recording method calls -->
<bean id="asynchronizePointCut"
class="org.springframework.aop.support.RegexpMethodPointcutAdvisor">
<property name="advice">
<ref local="asynchronizeInterceptor" />
</property>
<property name="patterns">
<list>
<value>com.scoreout.service.Audit.recordCourseRetrieval*</value>
<value>com.scoreout.service.Audit.recordClientLogin*</value>
</list>
</property>
</bean>

Now anytime I call com.scoreout.service.Audit.recordCourseRetrieval or com.scoreout.service.Audit.recordClientLogin the call will return immediately and the method will be run in the background. For testing I can easily make these methods synchronous by either removing them from the application context or providing a new application context that does not include this pointcut (I generally prefer the latter).

IE7 Tricks

2008-12-10T03:09:00.000-08:00

Hey, my first post and I already spent an hour setting up wordpress so I'll keep it short. Recently, we did an overhaul of our video management web application at Episodic. Like any sane web developer I did all my testing in Firefox and then found myself with one day left before pushing to production and I had yet to look at our site in IE7 (IE6 not supported).

So I RDC'd to our one Windows box and, uh oh, it don't look so good and there are a bunch of JS errors. Here were a few of the main problems that I had to solve for IE.

Trailing Commas

IE does not like trailing commas in things like object declarations. For example, you will get a very general JS error that points to a line that has nothing to do with the problem if you have the following:

var myObject = {
foo: 'bar',
baz: anotherObject,
};

Positioning and Z-Index

Our site has a lot of fancy drop-down menus. The menu is absolutely positioned within some relatively positioned element and then given a z-index so that it appears overtop of other elements on the page. However, this was a problem on IE when I noticed that my menu would actually appear behind some elements on the page. In particular, the menu would appear behind other elements that have "position:relative".

This took a while for me to figure out. Eventually it came down to this: If an absolutely positioned element has a z-index then it's containing relatively positioned element must specify a z-index (z-index:1) in order for the absolutely positioned element to be able to appear on top of other relatively positioned elements.

I did not come up with this on my own but it did take me a while to find this excellent page called "Z-Indexing with position relative and absolute".

Mootools Events Targets

I LOVE Mootools. Eventually I will do another post about some Mootools tricks I have picked up but for now I'll stay within the context of this post. I have a lot of JS code that looks like the following:

someElement.addEvent('click', function(event) {
if (event.target.get('tag') == 'li'
event.target.dispose();
});

However, the Mootools documentation says that the event target, is not extended with $ for performance reasons (which actually only seems to be true in IE and would result in JS errors). Therefore, the above case should be written as:

someElement.addEvent('click', function(event) {
if ($(event.target).get('tag') == 'li'
$(event.target).dispose();
});

To avoid writing code like this everywhere I extended the Event class as follows:

Event.implement({
getTarget: function() {
if (!this.extendedTarget)
this.extendedTarget = $(this.target);
return this.extendedTarget;
});

This left me with:

someElement.addEvent('click', function(event) {
if (event.getTarget().get('tag') == 'li'
event.getTarget().dispose();
});