Learning Google App Engine and GWT Best Practices

Dumbassing things up

2010-09-23T22:02:00.001-05:00

In a previous entry, Some management servlets, in the section You do know that anyone in the world can delete your datastore, right?, I goofed.

In my defense, the first section of that entry WAS entitled I have no idea what I'm talking about. But it turns out my ignorance knows few bounds.

web.xml, access control, and Guice

I stated that in moving my BuildDB servlet under the control of Guice, I lost the ability to use a security-constraint in the web.xml file to stop the hoi polloi from accessing our servlet. Not so.

Simply have Guice respond to a URL pattern (such as "/*") that includes the security-constraint constrained URL below it, say /admin/*. All of your servlets now are Guicified, and the ones you want protected are protected.

Yeah, shoulda figured that out the first time.

war/WEB-INF/web.xml

<web-app>
 <!-- Servlets -->
 <security-constraint>
  <web-resource-collection>
   <url-pattern>/admin/*</url-pattern>
  </web-resource-collection>
  <auth-constraint>
   <role-name>admin</role-name>
  </auth-constraint>
 </security-constraint>

 <filter>
  <filter-name>guiceFilter</filter-name>
  <filter-class>com.google.inject.servlet.GuiceFilter</filter-class>
 </filter>

 <filter-mapping>
  <filter-name>guiceFilter</filter-name>
  <url-pattern>/*</url-pattern>
 </filter-mapping>

 <listener>
  <listener-class>com.lisedex.voluntickler.server.guice.VolunticklerServletContextListener</listener-class>
 </listener>

 <!-- Default page to serve -->
 <welcome-file-list>
  <welcome-file>index.html</welcome-file>
 </welcome-file-list>
</web-app>

The observant among you will notice that our package is now com.lisedex.voluntickler instead of com.lisedex.volinfoman. It's a sexy name change, I know, and I realize that you're a bit jealous, but the domains are already registered. Sorry.

Authenticating in GWT

2010-09-04T23:42:00.000-05:00

Now that we have some basic ground work done for creating accounts and logging in, we can integrate this into our GWT application. We always need to assume that the client is hostile, that the JavaScript code has been changed, or someone is calling our RPC methods directly. None of our authentication checking should be done in the GWT client, except for stuff that decides which pages should be shown. Any method that then provides data to these pages must still do a proper permissions and authentication check before returning anything.

The code is available on github, and the tag for these revisions is add-gwt-authentication.

No more client passwords

Since we're no longer using GWT to enter the user's login information, we no longer need any of the methods we had that passed entered passwords through RPC, namely checkUserPassword() and authenticatedUser(). The password can be sent once through HTTPS on our login form, and a session cookie is set on the client. That cookie maps to a session stored in both the App Engine datastore, as well as memcache. We can use the session under App Engine to store information about the session, such as whether the user is authenticated, and no client program can change the value without going through an RPC call.

So our UserService RemoteService is stripped down to an isAuthenticated() method.

src/com/lisedex/volinfoman/server/UserServiceImpl.java

        @Override
        public boolean isAuthenticated() {
                return SessionHandler.isAuthenticated(getSession());
        }

        private HttpSession getSession() {
                return this.getThreadLocalRequest().getSession();
        }

When we call the isAuthenticated() method, if our session cookie is set, it will be passed to the server in the Cookie header.

Oops. I just made a security hole.

Writing this rattled something loose in the back of my brain, which is apparently the Login Security FAQ for GWT.

The server should not rely on the session ID that comes in the Cookie header, as doing so opens up the cross-site request forgery (XSRF or CSRF) vulnerability.

In short, if the server relies on the Cookie header for the session, and the user's session has not expired, then another site could have a link to something like http://www.vulnerablesite.com/user?deleteMe, which could be even hidden as an <img src> tag. The user's session cookie would automatically be sent with the deleteMe request, and the vulnerable server would go ahead and process the request. If we require the session to be submitted in the RPC payload, the link would have to be something like http://www.vulnerablesite.com/user?deleteMe?sessionId=787asdfasdf. Since the evil site has no way to access the user's session cookie, it can't generate a usable link. Hole closed.

So let's rewind, patch the hole, and do this again. The new code is available under the fix-csrf-hole tag on github.

Fixing the CSRF vulnerability

In looking for a pretty solution to this issue, I came across references to CSRF being addressed in GWT 2.1. The current RpcRequestBuilder in 2.0.4 adds a custom header, X-GWT-Permutation, and it looks like the code in 2.1 at least checks if this header is set. But it doesn't do anything beyond an existence test, on the idea that CSRF attacks can't add custom HTTP headers. However, apparently by using Adobe Flash, you can add custom headers. It would appear that this simple test is not enough. But since 2.1 is not yet out, I'm not sure if this is the extent of what will be checked. Assuming that Google will probably handle it correctly in the end, I decided not to spend a ton of time architecting an elegant solution.

src/com/lisedex/volinfoman/client/data/UserService.java

       boolean isAuthenticated(final String sessionId);

All of my UserService RPC methods will also pass a copy of the JSESSIONID cookie in its parameters. The servlet will then compare the passed copy with the header cookie, and if they match, we assume its a valid RPC call. There's still a possibility that someone could sniff the cookie, but since this software is not exactly financial software, I'm pretty comfortable with the level of security this adds. More elaborate security could be added by using a nonce, or maybe a hash across the session and the command name.

src/com/lisedex/volinfoman/server/UserServiceImpl.java

        public boolean isAuthenticated(final String sessionId) {
                if (validateSessionId(sessionId)) {
                        return SessionHandler.isAuthenticated(getSession());
                } else {
                        LOG.severe("Possible CSRF.  JSESSIONID cookie does " +
                           "not match included session id");
                        return false;
                }
        }

        private HttpSession getSession() {
                return this.getThreadLocalRequest().getSession();
        }

        private boolean validateSessionId(String sessionId) {
                String headerSessionId = (String) getSession().getId();
                if (sessionId != null && headerSessionId != null
                                && headerSessionId.equals(sessionId)) {
                        return true;
                }
                return false;
        }

In each RPC method, we call validateSessionId, which gets the session id from the HttpServletRequest, and compares it with the passed session id. Assuming they match, the RPC method will continue.

src/com/lisedex/volinfoman/server/SessionHandler.java

        public static final String AUTHENTICATED = "authenticated";

        public static void setAuthenticated(HttpSession session,
                        boolean authenticated) {
                session.setAttribute(AUTHENTICATED, authenticated);
        }

        public static boolean isAuthenticated(HttpSession session) {
                if (session == null)
                        return false;

                Boolean auth = (Boolean) session.getAttribute(AUTHENTICATED);
                if (auth == null) {
                        return false;
                } else {
                        return auth.booleanValue();
                }
        }

In the case of isAuthenticated, we call SessionHandler.isAuthenticated, a static method that operates on the HttpServletRequest session, and just gets the value of the authenticated attribute stored in the session table in the datastore.

In our Login servlet, we call the setAuthenticated() method to mark the session as authenticated if the login is successful, and before redirecting the user to the GWT application.

The new home page, MainPageView, is just a SimplePanel, which should allow us to do a very small application load before checking whether the session is authenticated. If it is, we can put a new widget which contains the actual layout in the panel.

src/com/lisedex/volinfoman/client/MainPagePresenter.java

    private static final String HOME_PAGE_URL = "/index.html";

    private UserServiceAsync userService = null;

    @Override
    public void bind() {
        view.setContent(new HTMLPanel(LOADING_MESSAGE));
        userService.isAuthenticated(
                Cookies.getCookie(Volinfoman.SESSION_COOKIE),
                    new AsyncCallback<Boolean>() {
                        @Override
                        public void onSuccess(Boolean result) {
                                if (result.booleanValue() == true) {
                                        view.setContent(new HTMLPanel(
                                            "<h1>AUTHENTICATED</h1>"));
                                } else {
                                        view.setContent(new HTMLPanel(
                                                NOT_AUTHENTICATED_MESSAGE));
                                        Window.Location.replace(HOME_PAGE_URL);
                                }
                        }

                        @Override
                        public void onFailure(Throwable caught) {
                                view.setContent(new HTMLPanel(RPC_FAILURE_MESSAGE));
                        }
                });
    }

We pop a loading message in as the widget in our SimplePanel, and call the RPC isAuthenticated method. Cookies.getCookie is a static method which gives us access to the browser's cookies for our site. If we get a successful response for our RPC method, and it turns out that our session is marked as authenticated, we put up another message just informing us that we authenticated successfully. In the future, we'd set the SimplePanel's widget to some sort of layout panel, and set up our application. If our session is not marked as authenticated, we redirect the browser back to the login page. The Window class gives us access to some of the user's browser's properties and events. In this case, Window.Location allows us to change the current URL.

Paying attention

This definitely shows me how, if I'm not paying attention, security lapses can easily slip right by. Writing the blog entries actually helps me reparse the stuff that I've written, and catch some of my mistakes before I move on, but even then I almost blew right past this one.

The Live HTTP Headers plugin for Firefox made testing and playing with this vulnerability really easy. I could simply catch a remote call, and then replay it with whatever modifications I wanted to make: header modifications or deletions, payload changes. It's pretty smooth. I also made some packet captures, but this plugin is far easier.

I think it's time to start making some sort of user configuration wizard, where people can set up their volunteering preferences, or, if they happen to be a volunteer coordinator, the volunteer opportunities for their organization.

Some management servlets

2010-09-04T04:03:00.001-05:00

So I got StarCraft II, and let's just say I've spent a lot more time with that than I have with this project, what with playing, watching Day9 Daily archives, reading strategy...

And I'm still in the bronze league. You may now commence laughing.

But a couple of days ago, I picked this stuff back up. Leaving it for so long is a mistake, as it always takes me a while to remember what's what, where I was, and where I was going. I messed around with the interface, trying to get a very basic layout with a working login page.

I learned a few important lessons, like wrapping widgets with SimplePanels so they could be swapped out easily with just a setWidget() as the interface required, and my previous understanding, as explained in my last entry, of bind() from mvp4g was off. But I learned a much more important lesson.

I have no idea what I'm talking about

OK, that's maybe a bit harsh, but really, I think I was coming at this with a fundamental misconception about how it'll work. I was trying to build an application where you could login, or register for the site, or request a reset password, or actually use the site for its intended purpose basically without leaving the layout and original landing page. Lunacy. Trying to fit the project to the tool, while not understanding the tool all that well, either.

At first, I realized things should work more like Gmail, where your initial login and registration could be handled on some regular pages, and then you move in to the application itself. Flipping this switch in my brain alone allowed me to start making much faster progress (obviously helped along because I had moved back to a more familiar paradigm). Taking that lesson more generally, I shouldn't force this thing into a single page where widgets are swapped in and out. If it makes more sense, I should be willing to just move to another page, or sprinkle parts of GWT throughout pages, unless there's too much of a startup penalty.

In short, I don't need to write this as if it were some monolithic desktop application.

On that note, I'm going to focus in this post on the account registration, email confirmation, App Engine cron jobs. At this time, these all use GAE, but not GWT. You'll just have to ignore the GWT client code for now, as it's in a screwy state that compiles and renders a page, but doesn't do anything remotely useful.

As always, the code is available at github under the add-static-login tag.

Moving the starting line

The first step was moving the home page, and having it basically static with a login form and a link to the registration page.

war/WEB-INF/web.xml

<!-- Default page to serve -->
        <welcome-file-list>
               <welcome-file>index.html</welcome-file>
        </welcome-file-list>

I can keep the original Volinfoman.html for later, and this automatically solves one of the issues I had been thinking about, but had put off for later: auto-complete not working in GWT login pages, and using HTTPS to submit the user's password.

The link to the registration page leads to another simple, static form, register.html. This submits to a servlet which is written using App Engine, Register.

The making of a servlet

We need to register this servlet with Guice so that injection works.

src/com/lisedex/volinfoman/server/guice/VolinfomanGuiceModule.java

bind(Register.class).in(Singleton.class);

src/com/lisedex/volinfoman/server/guice/VolinfomanServletModule.java

serve("/volinfoman/register").with(Register.class);

In our Guice module, we configure Register as a Singleton, and in Guice's servlet configuration, we let it know that the URL /volinfoman/register should be sent to Register for handling. This is, of course, the action we use for the form in register.html.

src/com/lisedex/volinfoman/server/authenticate/Register.java

public class Register extends HttpServlet {
        @Inject
        private Dao dao;

        private static final Logger log = Logger.getLogger(Register.class.getName());

        @Override
        public void doGet(HttpServletRequest req, HttpServletResponse resp)
                        throws IOException {

                PrintWriter output = resp.getWriter();

                // build HTML response page
                resp.setContentType("text/html");
                resp.setCharacterEncoding("utf-8");
                output.println("<head><title>Add initial datastore information</title></head>");
                output.println("<body>");

                String username = req.getParameter("username");
                String firstName = req.getParameter("firstName");
                String lastName = req.getParameter("lastName");
                String password = req.getParameter("password");
                String email = req.getParameter("email");

                if (!StringSafety.isSafe(username)) {
                        output.println("<span style=\"color: #ff0000;\">Username bad, please go back and enter it again</span>");
                        output.println("</body>");
                        return;
                }

                if (!StringSafety.isSafe(firstName)) {
                        output.println("<span style=\"color: #ff0000;\">First name bad, please go back and enter it again</span>");
                        output.println("</body>");
                        return;
                }

                // ... etc, etc. for input safety ...

                if (dao.getUser(username) != null) {
                        output.println("<span style=\"color: #ff0000;\">Username already exists, please go back and enter it again</span>");
                        output.println("</body>");
                        return;
                }

This first pass at the servlet is pretty ugly, with plenty of string literals, but what we're doing is laying the groundwork for the registration by collecting the user's input.

We extend javax.servlet.http.HttpServlet and override its doGet() method, since at this point we have not declared a method for our HTML form to use and it defaults to GET. Later, we'll change this to doPost() since the user is entering their password in the form, and we don't really need that showing up in the address bar of their browser. Or our server logs.

We start setting up the page that gets sent back to the user when the servlet completes, by filling out the HttpServletResponse object passed in as resp. After setting what type of page we'll be returning, we print to the object line-by-line in the order we would want to build a standard HTML page.

We need to get the user's input, so we use the getParameter method of HttpServletRequest, which parses the form data for us, and hides the logic needed to handle both GET and POST forms. After we get the input, we want to protect ourselves from things like injection attacks, so we have to check each piece of input that comes from outside of the servlet. The static StringSafety.isSafe() method right now is terribly simple, just looking for ; and & characters. Later, we'd want to do better verification, like making sure the username and password meet whatever requirements we have and that the email address looks basically valid.

If any of the input doesn't meet our standards, we spit out an error message and abort the servlet. I check each separately, so I can inform the user of which field doesn't meet our standards. Ideally, instead of forcing the user to go back, we'd re-render the form, with the error message integrated into it, but right now we're doing it the simple way. I removed some of the repetitious statements for brevity; see the code for the full version.

Finally, we make sure that the username requested doesn't already exist. This seems like an expensive way to look this up, and is also open to another account with the same username getting created between that test and the next lines of code.

User user = new User(null, username, 
                    User.STATUS_UNCONFIRMED, firstName, lastName, 
                    email, password);
                dao.putUser(user);

We put the user in the datastore, with a status of User.STATUS_UNCONFIRMED. This status indicates that the account has been created, but the user has not yet clicked the link included in the email we're about to send them. This code has a bug, in that the password is not passed through BCrypt before storage. We'll fix this as we come back and fix this class up.

Properties props = new Properties();
                Session session = Session.getDefaultInstance(props, null);

                String msgBody = "Thank you for registering a VolunteerIM "
                   + "account!  Please follow the link below to confirm "
                   + "your account:\n\n";
                Random r = new Random();
                msgBody += "http://lisedexvolinfomantest/volinfoman/emailConfirm?code="
                    + Long.toString(Math.abs(r.nextLong()), 36)
                    + "\n\n";
                msgBody += "Note: Please do not reply to this address, as "
                    + "email is thrown away.  If you did not set up a "
                    + "VolunteerIM account, please ignore this email, as the "
                    + "account will be removed automatically in a week.\n";

                try {
                    Message msg = new MimeMessage(session);
                    msg.setFrom(new InternetAddress("admin@lisedex.com", 
                        "VolunteerIM Confirmation"));
                    msg.addRecipient(Message.RecipientType.TO,
                             new InternetAddress(email, firstName + 
                                 " " + lastName));
                    msg.setSubject("VolunteerIM account confirmation");
                    msg.setText(msgBody);
                    Transport.send(msg);
                } catch (AddressException e) {
                    output.println("Bad email address.  Please try again. " +
                        e.toString() + "</body>");
                    log.info("AddressException sending confirmation email: " +
                        e.toString());
                    return;
                } catch (MessagingException e) {
                    output.println("Error sending confirmation email.  Please try again. "
                        + e.toString() + "</body>");
                    log.info("MessagingException sending confirmation email: "
                        + e.toString());
                    return;
                }

        output.println("We have sent a confirmation email to " 
             + email + ".  It should arrive shortly.  As soon as you receive " 
             + "it, please <a href=\"/\">return to the front "
             + "page to log in.</a>");
        }
}

Google's App Engine has an API for sending emails which uses the JavaMail API, but we just use a tiny subset.

We need a reference to a mail Session, which we use to build a MIME compatible email. First, we build a quick message body, which contains a link with a random string that we'll use as a confirmation code.

When we set the address that the message will come from, it has to be set as an email address that is listed in the App Engine dashboard as a collaborator on the project. I found this the hard way, after a MessagingException kept getting thrown during testing. The code worked fine on the local development server, but when deployed it choked.

Then we specify the recipient and subject, and attach the message body we already generated.

We can still get an exception when sending the email, if we use a To address App Engine doesn't like, or, as I found out, a From address that's not registered. If one of these pop up, we generate an error for the user, and exit the servlet. At this stage of the code, this will leave an UNCONFIRMED account in the datastore, with no confirmation email sent. Whooops.

If everything's gone according to plan, we let the user know that they should keep an eye out for the confirmation email.

Of course, right now, if the user were to click on the link in the email, we'd have to return a 404, since there's no servlet registered at /volinfoman/emailConfirm.

OK, do you at least have an email account you can read?

Before that, let's change the registration form to use the POST method. The GET method puts all of the user's input into the URL, which include the user's password in plaintext. So not only does it show in their address bar, it will live on forever in the server logs. Fortunately, HttpServlet makes this easy: just add method="post" to the form declaration in register.html, and change the doGet() method in Register to doPost(). The class library handles the details.

Now, we want to make sure that the email address the user entered is linked to an account they can read, like every other website in existence with individual accounts. We're sending a link with a code embedded to the user after they register, so we'll need a servlet to manage confirming the accounts. We could put the confirmation code for each user into the User class, and have the datastore index it so we could run a query, but we also want to expire accounts that haven't been confirmed after some length of time. So then we'd need to add an indexed expiration date field to the User class, and it starts to look cleaner to create a datastore table just for this purpose.

src/com/lisedex/volinfoman/shared/ConfirmationCode.java

public class ConfirmationCode implements Serializable {
        @Id
        private Long id;

        @Indexed
        private String username;

        @Indexed
        private String code;

        @Indexed
        private long expires;

        public ConfirmationCode() {
        }

        /**
         * Constructor
         * @param id Datastore primary key
         * @param username username associated with code
         * @param code confirmation code associated with username
         * @param expires expiration date for code in milliseconds
         */
        public ConfirmationCode(Long id, String username, String code, long expires) {
                setId(id);
                setUsername(username);
                setCode(code);
                setExpires(expires);
        }

        public Long getId() {
                return id;
        }

        public void setId(Long id) {
                this.id = id;
        }

        public String getUsername() {
                return username;
        }

        // ... etc, etc ... it's getters and setters all the way down
}

It's a pretty bare bones data structure, though we should probably add some input verification later to the setters. We also index all of the elements of the table. At different times we'll need to search by the code, and we'll also need to query for expiration times that are older than a certain time. Currently, the username does not need to be indexed, but we'll leave it for now.

In the earlier blog entry about Guice, I talked some about a BuildDB class which could delete all Users in the datastore, and just leave one admin account. Well, we also want the option of wiping out our confirmation code table while developing, which requires a new method in our DAO. We declare it in the Dao interface, and implement it in DaoGaeDatastore.

src/com/lisedex/volinfoman/server/DaoGaeDatastore.java

@Override
        public void deleteUser(Long id) {
                ofy().delete(User.class, id.toString());
        }

        @Override
        public void putConfirmationCode(ConfirmationCode code) {
                ofy().put(code);
        }

        @Override
        public void deleteAllConfirmationCodes() {
                ofy().delete(ofy().query(ConfirmationCode.class).fetchKeys());
        }

While we're in there, we added a couple of other methods that we'll use to work with the confirmation codes right now. We'll need a way to delete the users that we put into the datastore, if we get an exception during sending them their confirmation email. When the exception is caught, we want to throw away the new User object we put in the datastore, so the user can reuse it when they resubmit their registration. When calling this, we'll already have the User object, which includes the Id field, so we won't need to run a query; we can just straight up delete using an Objectify convenience method that builds a datastore Key from the provided class and its Id.

We'll also need to store new ConfirmationCode objects, and the deleteAllConfirmationCodes() method is pretty much ripped from the deleteAllUsers() method verbatim (just changing the class).

We also need to register the ConfirmationCode class with Objectify, so it knows how to handle it. We do this in the same place we register User: a static constructor in our DAO implementation.

ObjectifyService.register(ConfirmationCode.class);

Now, in our BuildDB servlet, we can simply call dao.deleteAllConfirmationCodes when we want, and clean out the table. "Dangerous Database Deletions" are my middle names.

Now, in our Register servlet, we want to add the confirmation code to the datastore.

src/com/lisedex/volinfoman/server/authenticate/Register.java

public static final int EXPIRATION_FIELD = Calendar.DATE;
    public static final int EXPIRATION_INCREMENT = 7;

We'll use the Java Calendar class to calculate our expiration date. This gives us flexibility to easily change the delay without having to recalculate how many milliseconds it is. The EXPIRATION_FIELD says which field we'll be incrementing, where DATE is days, but we could also use MONTH or HOUR or even YEAR. Calendar will handle the rollover and carrying the ones and the leap years and leap seconds and all of that. The EXPIRATION_INCREMENT, surprisingly enough, is how much that field will be incremented by.

Random r = new Random();
                String code = Long.toString(Math.abs(r.nextLong()), 36);
                Calendar expirationTime = Calendar.getInstance();
                expirationTime.add(EXPIRATION_FIELD, EXPIRATION_INCREMENT);

                ConfirmationCode confCode = new ConfirmationCode(null, username, 
                    code, expirationTime.getTimeInMillis());
                dao.putConfirmationCode(confCode);

We generate the confirmation code before the message body so we can insert it into the ConfirmationCode table in the datastore. Calendar.getInstance() gets a Calendar object representing the current time, and we add however much time we'd like to make our expiration time in milliseconds.

In both of the exception handlers for email transmission errors, we add a line to delete the users we just added to the datastore.

dao.deleteUser(user.getId());

You do know that anyone in the world can delete your datastore, right?

In the Guice blog entry, we moved BuildDB under control of Guice so we could inject our DAO implementation. In doing so, we lost the ability to use the App Engine user authentication to control access to the servlet. As we get further along, this becomes unacceptable, but I'm still not ready to build my own authentication mechanism into the servlet. We'll also need the authentication for protecting our cron jobs, so fixing this is not a waste.

In BuildDB, we no longer inject the Dao, we directly instantiate a DaoGaeDatastore. In the VolinfomanGuiceModule and VolinfomanServletModule, we pull out our definitions for BuildDB (and CacheStats, another servlet that I don't believe works at the moment).

war/WEB-INF/web.xml

<filter-mapping>
            <filter-name>guiceFilter</filter-name>
            <url-pattern>/volinfoman/*</url-pattern>
    </filter-mapping>


    <security-constraint>
        <web-resource-collection>
            <url-pattern>/admin/*</url-pattern>
        </web-resource-collection>
        <auth-constraint>
            <role-name>admin</role-name>
        </auth-constraint>
    </security-constraint>

    <servlet>
        <servlet-name>BuildDB</servlet-name>
        <servlet-class>com.lisedex.volinfoman.server.admin.BuildDB</servlet-class>
    </servlet>

        <servlet-mapping>
                <servlet-name>BuildDB</servlet-name>
                <url-pattern>/admin/builddb</url-pattern>
        </servlet-mapping>

In web.xml, we will change the Guice url-pattern from /* to /volinfoman/*, to allow us some granularity in defining which servlets get processed through Guice. To get BuildDB out of Guice's control, we'll need to change its URL from /volinfoman/admin/builddb to /admin/builddb.

We also set a security-constraint on the url-pattern /admin/* which requires the user have an administrator role for the application. If the user is not logged in, they will be presented with a login form from Google that allows them to log in with whatever account they've used as a collaborator for the project. If their Google account is not registered as a collaborator on the project, they will get an error and the servlet will not run. This is also used to protect cron jobs, as they will run as if run by an authenticated admin user.

Finally, we set the url-pattern that configures what URLs BuildDB processes. Since /admin/builddb matches the /admin/* security-constraint definition, it's protected by Google's authentication.

Get rid of expired confirmation codes

We want to run a cron job that will find any expired confirmation codes, and remove them. It will also need to look at the User associated with the code, and if it's still in the User.UNCONFIRMED state, the User needs to be deleted as well. The reason we double check this, is that I can imagine a time when someone emails support, and support confirms their account, but forgets to remove the associated confirmation code. If we didn't check the User's status, we could delete a confirmed account.

@Override
        public void expireCodesBefore(long now) {
                Query<ConfirmationCode> oldCodes = 
                    ofy().query(ConfirmationCode.class).
                        filter("expires <", now);
                for (ConfirmationCode code: oldCodes) {
                        User user = getUser(code.getUsername());
                        if (user != null) {
                                // make sure user is still in unconfirmed state
                                if (user.getStatus() == User.STATUS_UNCONFIRMED) {
                                        deleteUser(user.getId());
                                }
                        }

                        deleteConfirmationCode(code);
                }
        }

        @Override
        public ConfirmationCode getConfirmationCode(String code) {
                ConfirmationCode fetched = 
                    ofy().query(ConfirmationCode.class).
                        filter("code", code).get();
                return fetched;
        }

        @Override
        public void deleteConfirmationCode(ConfirmationCode code) {
                if (code != null) {
                        ofy().delete(code);
                }
        }

The expireCodesBefore() method takes a time, in milliseconds, and deletes all confirmation codes with an expiration time that comes before that time. This can be tested with a relational operator: if the expiration time is less than the time provided, it's expired.

The query does just that, and the query itself is Iterable, so we walk through it. For each expired code, we get the username associated with it, and pull in the User by that name. If the user exists, and the user's status is still unconfirmed, we delete the user. The expired confirmation code is deleted regardless of the user's state.

getConfirmationCode() will retrieve a confirmation code by the code field, instead of by Id. To do this, we need to run a query. Since it should only return one hit, we only return the first one found. If there were multiple entries with the same username, we'd only work with the first one.

Lastly, we add a method to delete confirmation codes.

In the Register servlet, we move most of the string literals scatted throughout the code into static Strings at the top of the class. This includes parameter names that are submitted by the registration form, as well as information used to build the email.

src/com/lisedex/volinfoman/server/authenticate/Register.java

User user = new User(null, username, User.STATUS_UNCONFIRMED, 
                firstName, lastName, email, null);
            dao.changeUserPassword(user, password);

We now build the User with a null password, and insert the User into the datastore using the password change method. This allows us to hash the password before storing it, fixing the bug we introduced in the early version of the code.

src/com/lisedex/volinfoman/server/cron/ExpireConfirmationCodes.java

public class ExpireConfirmationCodes extends HttpServlet {
        private Dao dao = new DaoGaeDatastore();

    private static final Logger log = Logger.getLogger(ExpireConfirmationCodes.class.getName());

        @Override
        public void doGet(HttpServletRequest req, HttpServletResponse resp)
                throws IOException {

                Calendar now = Calendar.getInstance();
                log.info("Expiring expiration codes at " + Long.toString(now.getTimeInMillis()));

                dao.expireCodesBefore(now.getTimeInMillis());
        }
}

The ExpireConfirmationCodes cron job is just a stripped down servlet based on BuildDB. Since it accepts no parameters, we simply get the current time, and pass its value in milliseconds to the DAO method we wrote above for this purpose. Note that we're directly instantiating the DaoGaeDatastore again, as we need to bypass Guice since we need to put our cron jobs behind a security-constraint.

war/WEB-INF/web.xml

<!-- Cron jobs -->
        <security-constraint>
        <web-resource-collection>
            <url-pattern>/cron/*</url-pattern>
        </web-resource-collection>
        <auth-constraint>
            <role-name>admin</role-name>
        </auth-constraint>
        </security-constraint>
  
        <servlet>
                <servlet-name>expireConfirmationCodes</servlet-name>
                <servlet-class>com.lisedex.volinfoman.server.cron.ExpireConfirmationCodes</servlet-class>
        </servlet>

        <servlet-mapping>
                <servlet-name>expireConfirmationCodes</servlet-name>
                <url-pattern>/cron/expireConfirmationCodes</url-pattern>
        </servlet-mapping>

We add /cron/* URLs to what we hide behind Google's authentication, and define the URL /cron/expireConfirmationCodes as the one which invokes our expiration servlet.

war/WEB-INF/cron.xml

<cronentries>
        <cron>
                <url>/cron/expireConfirmationCodes</url>
                <description>Expire confirmation codes every day</description>
                <schedule>every day 02:00</schedule>
        </cron>
</cronentries>

The cron job definition specifies what URL gets called at what frequency. If you use a frequency like "every 5 minutes", the job is started 5 minutes after the last job completed. For example, if the job took 1 minute to run, the first job would start at 01:00, the next run at 01:06, the next at 01:12, and so on. If you wanted the jobs to start at 01:00, 01:05, and 01:10, you would need to add the keyword synchronized to the schedule.

Lastly (aka FINALLY), we can process the confirmation link and login form

This has stretched a bit longer than I expected, but it's almost over.

src/com/lisedex/volinfoman/server/authenticate/ConfirmationCodeChecker.java

public class ConfirmationCodeChecker extends HttpServlet {
        private static final String ACCOUNT_NEW = "Sorry, the account you're trying" +
           " to confirmed has not reached the point where it can be confirmed.  " +
           "Unfortunately, the only way to resolve this is to contact our support" +
           " department, or <a href=\"/register.html\">apply for a new account</a>." +
           "  We apologize for the inconvenience.";

        private static final String ACCOUNT_INVALID = ""; // another message
        private static final String ALREADY_CONFIRMED = ""; // another message
        private static final String ACCOUNT_CLOSED = ""; // another message
        private static final String BAD_CONFIRMATION_CODE = "";
        private static final String CONFIRMATION_SUCCESS = "";

        @Inject
        private Dao dao;

        private static final Logger log = Logger
                        .getLogger(ConfirmationCodeChecker.class.getName());

        private static final String UNSAFE_ERROR = "<span style=\"color: #ff0000;\">" +
            "There is a problem with the link used to confirm your user account." +
            "  Please try clicking the link again, and if the problem continues, " +
            "return to the home page and request that the email is sent again.  " +
            "Or you can contact support.  Sorry for the inconvenience!</span>";

        @Override
        public void doGet(HttpServletRequest req, HttpServletResponse resp)
                        throws IOException {
                // standard stuff
                output.println("<head><title>VolunteerIM account confirmation</title></head>");
                output.println("<body>");

                output.println("<h2>VolunteerIM account confirmation</h2><p>");

                String username = req.getParameter("username");
                String code = req.getParameter("code");

                log.info("Confirmation request with username " + username
                                + " and code " + code);

                if (!StringSafety.isSafe(username)) {
                        output.println(UNSAFE_ERROR + "</body>");
                        return;
                }

                if (!StringSafety.isSafe(code)) {
                        output.println(UNSAFE_ERROR + "</body>");
                        return;
                }

                ConfirmationCode testCode = dao.getConfirmationCode(code);
                if ((testCode == null) || (testCode.getUsername() == null)
                                || (testCode.getCode() == null)) {
                        output.println(BAD_CONFIRMATION_CODE + "</body>");
                        return;
                }

                User fetched = dao.getUser(testCode.getUsername());

                if (fetched.getStatus() == User.STATUS_UNCONFIRMED) {
                        fetched.setStatus(User.STATUS_CONFIRMED);
                        dao.putUser(fetched);

                        // eliminate confirmation code from datastore, as we're
                        // done with it
                        dao.deleteConfirmationCode(testCode);

                        output.println(CONFIRMATION_SUCCESS + "</body>");
                        log.info("Confirmed username " + fetched.getUsername());
                        return;
                }

                if (fetched.getStatus() == User.STATUS_CLOSED) {
                        output.println(ACCOUNT_CLOSED + "</body>");
                        return;
                }

                if (fetched.getStatus() == User.STATUS_CONFIRMED) {
                        output.println(ALREADY_CONFIRMED + "</body>");
                        return;
                }

                if (fetched.getStatus() == User.STATUS_INVALID) {
                        output.println(ACCOUNT_INVALID + "</body>");
                        return;
                }

                if (fetched.getStatus() == User.STATUS_NEW) {
                        output.println(ACCOUNT_NEW + "</body>");
                        return;
                }
        }
}

This servlet is under Guice's control, so its URL is defined in VolinfomanGuiceModule and VolinfomanServletModule. In this case it's /volinfoman/emailConfirm, as specified in the email sent to the user.

We define a bunch of messages that may get sent to the user, for different states of the User status, or malformed URL parameters. We get both a username and code from the URL, which we check for bad data, and also use to cross check whether the code matches the specified username. If it does, we pull up the User, and based on its status, we return different messages.

If the User was previously unconfirmed, we tell them it's now confirmed, we mark it as such in the datastore, and they can now log in from the home page. If it was closed, we let them know they can register for a new account, or contact support to find out why it was closed. Already confirmed Users get told they can go ahead and log in, and invalid and new users are told to contact support or register new accounts. In the code above, I've stripped the messages since they're pretty long, but they're available in the code in the repository at github.

src/com/lisedex/volinfoman/server/authenticate/Login.java

public class Login extends HttpServlet {
        @Inject
        Dao dao;

        private static final Logger log = Logger.getLogger(Login.class.getName());

        @Override
        public void doPost(HttpServletRequest req, HttpServletResponse resp)
                        throws IOException {

                // standard stuff

                String username = req.getParameter("username");
                String password = req.getParameter("password");

                if ((username == null) || (password == null)) {
                        output.println("<head><title>VolunteerIM login</title></head>");
                        output.println("<body>Please fill out both username and " +
                            "password fields.</body>");
                        return;
                }

                if (!StringSafety.isSafe(username)) {
                        output.println("<head><title>VolunteerIM login</title></head>");
                        output.println("<body>Username invalid, please go back and " +
                            "try again.</body>");
                        return;
                }

                if (!StringSafety.isSafe(password)) {
                        output.println("<head><title>VolunteerIM login</title></head>");
                        output.println("<body>Password invalid, please go back and try" +
                            " again.</body>");
                        return;
                }

                if (dao.checkUserPassword(username, password)) {
                        if (dao.getUser(username).getStatus() == User.STATUS_CONFIRMED) {
                                HttpSession session = req.getSession();
                                resp.sendRedirect(resp.encodeRedirectURL("/Volinfoman.html"));
                                session.setAttribute(Session.AUTHENTICATEDUSER, username);
                                return;
                        } else {
                                output.println("<head><title>VolunteerIM login</title></head>");
                                output.println("<body>INSERT CONFIRMATION MESSAGE HERE</body>");
                                return;
                        }
                } else {
                        output.println("<head><title>VolunteerIM login</title></head>");
                        output.println("<body>Username and password do not match." +
                            "  Please try again.</body>");
                        return;
                }
        }
}

For the login form, we have a Login servlet. I've stripped out some messages and some of the boilerplate code I've been using in all of these servlets. We check the submitted username and password for safety, and then pass them to our previously (way back in the day) developed password checking code in the DAO. If it's a match, we also check the User's status to make sure it's been confirmed. If it has, we set a cookie with the HTTP session identifier, mark it in App Engine's session table as an authorized user, and redirect them to /Volinfoman.html, which is the entry point to our GWT application. Currently, there is no testing whether the browser visiting /Volinfoman.html is authenticated, so anyone can visit it directly, but this will be rectified in the next bit of code.

Whew

I realize this is way too long, and maybe showing the revisions as I was figuring this stuff out is not helpful, but I wanted to give a decent introduction to some of the code that will run the site behind the scenes. I've been focusing almost exclusively on GWT code, and as I described above, I'm realizing something that I should have realized a long time ago (except I was blinded by GWT's novelty). I should have realized that GWT's not everything.

Unable to install breakpoint due to missing line number attributes

2010-07-23T00:34:00.007-05:00

I was happily coding along, and didn't change anything in the Eclipse configuration, just a few lines of code since the last time, and hit F11 to start debugging. I had set a breakpoint to see the output of the BCrypt hashing implementation I put in.

An error dialog popped up with the message:

Unable to install breakpoint in 
com.lisedex.volinfoman.server.admin.BuildDB$$FastClassByGuice$$22710124
due to missing line number attributes.  Modify compiler
options to generate line number attributes.

Reason:
Absent Line Number Information

Crap. Well, I didn't change anything, but I thought maybe I hit some key combination that turned off line numbers? I checked Window -> Preferences -> Java -> Compiler, but sure enough, the line number checkbox was ticked. So I unticked it, applied the settings, came back and re-ticked it, but no luck. Did some quick searches, and people recommended checking that option, or adding -g to the compiler options, but that doesn't really seem to be an option here.

Added a bunch of logging to look at what I wanted to see, and went ahead and ran the program anyways, OKing through the dialog.

It stopped at my breakpoint.

I tried this in several other places in my code, and they all work. I still get the error, but it doesn't appear that the problem actually exists.

So if you see this error, my suggestion is, before wasting a bunch of time trying to fix it, see if it's actually true. You may just be able to check the box for it to not notify you, and go on with your life.

Converting to the MVP architecture

2010-07-22T16:27:00.004-05:00

If you see one thing referenced again and again in writings about GWT development, it's Ray Ryan's Google IO 2009 talk on GWT Architecture Best Practices. In the talk he describes using the MVP architecture to develop GWT applications that allow easier unit testing and clean logic separation. There's also a couple of articles about large scale application development and MVP also at Google Code. And there's a million other things you can read with a quick search.

I've used the MVC architecture for other projects, and it's treated me pretty well, but you can still end up with some code that's difficult to follow from beginning to end. This is generally because of the View cutting the Controller out of the loop in going to the Model. MVP looks like it could help resolve that, so I wanted to give it a shot while learning GWT.

Finding a framework

I also didn't want to do all of the work involved in developing my own framework to support MVP, so after looking around to see what open source projects are available, I came across gwt-presenter. Several blogs and examples talk about this framework, and early on I had decided that I'd use it for this project. But now that I'm at the stage where I want to implement it, I looked a little closer.

It doesn't seem to be in active development. The last release was in August 2009, and GWT 2.0 was not yet released. There are two branches, and the GAE/GWT development blog talks about using the "replace" branch, but I'm not sure what stage it's at.

Though this may be due to my own prejudices, I preferred something that's currently being hacked, and I found that in mvp4g. It also looks to have some more features without being bogged down, have some more complete documentation, uses JUnit with pretty high coverage, and a discussion group where the developer regularly responds to questions. So I'm giving it a shot, at least the 1.2.0 snapshot (found in the examples zip), since it supports Gin.

Moving to mvp4g

The code for this stage is available under the "add-mvp4g" tag on github, or in a zip download.

I'm going to convert the login page that we currently have working (well, working is probably a bit too strong) to using this framework. The mvp4g FAQ has some good, quick explanations of their architecture, including a class diagram that I didn't find terribly useful until I got into implementing it.

The first step, as usual, is to include the jar in your classpath, as well as commons-lang and commons-configuration. We also need to inherit it in our .gwt.xml module file.

com/lisedex/volinfoman/Volinfoman.gwt.xml

<!-- Mvp4g configuration -->
       <inherits name='com.mvp4g.Mvp4gModule'/>

We fire it up in our entry point class.

com/lisedex/volinfoman/client/Volinfoman.java

// code to initialize mvp4g, which handles setting up our
      // Ginjection
      Mvp4gModule module = (Mvp4gModule)GWT.create( Mvp4gModule.class );
      module.createAndStartModule();
      RootPanel.get().add( (Widget)module.getStartView() );

If you prefer to use a layout panel for your application, the last line can use RootLayoutPanel instead of the RootPanel.

Event bus

Most of the initial configuration is done through annotations in our event bus class, so we'll set that up next; apparently you can also set mvp4g up through an XML file, but I didn't play with that.

com/lisedex/volinfoman/client/VolinfomanEventBus.java

@Events(startView = LoginView.class, historyOnStart = true,
        ginModule=VolinfomanModule.class)
@Debug( logLevel = LogLevel.DETAILED, logger = 
        Mvp4gLoggerToGwtLogAdapter.class )
public interface VolinfomanEventBus extends EventBus {
        @Event(handlers = LoginPresenter.class)
        public void login();
}

The @Events annotation declares what View our application will start in, whether it will parse any history information embedded in the URL that starts the application (we currently don't use this, but I have it enabled anyways), and what Gin module, if any, we want it to use to configure a Ginjector. Previously we defined the Ginjector in the onModuleLoad2() method in our entry point, but for mvp4g we pull it out and let the framework handle it.

I also turn on debugging using the @Debug annotation, and set the logger to one that I wrote that forwards mvp4g's logging calls to gwt-log instead of GWT.log. I wanted all of the logging in one place. It's a very simple class, so I'm going to skip it here.

Finally, we declare an interface that extends EventBus which defines all of the events that can be raised on the bus. The events should not return anything, and can have either one or zero parameters. We don't need a parameter for the login event, since the presenter that handles it will have access to the fields the user has filled out. The @Event annotation declares which classes, separated by a comma, are registered to be notified of the event.

Presenter

com/lisedex/volinfoman/client/LoginPresenter.java

@Presenter(view = LoginView.class)
public class LoginPresenter extends BasePresenter
        <LoginPresenter.LoginViewInterface, VolinfomanEventBus> {

The Presenter will be matched with a View, so we declare that association through the @Presenter annotation, which also lets the mvp4g compiler know that this class defines a Presenter. Our presenter extends BasePresenter, which is a generic that takes the class defining our View interactions (LoginViewInterface), and our Event Bus class (VolinfomanEventBus) as parameters.

The Presenter class handles the logic that will make the interface displayed to the user actually do something. In this case, we're working with a login page, so we need to define what we need for interactions with the View. I followed the mvp4g documentation, which defines it using an inner interface. I don't see why you couldn't break it out to a separate file, but it's a bit more clear having it bound to the Presenter class. Plus it's right there to reference when writing your Presenter.

public interface LoginViewInterface {
  public String getUsername();
  public String getPassword();
  public void setMessage(String msg);
  public HasClickHandlers getLoginButton();
  public void setLoginButtonEnabled(boolean enabled);
 }

We need a few things from the user, and we don't really care how the View gets them. All we care about is that it can provide the information we need to process the login. In this case, we want to be able to get the username and password they've entered, and we also want to be able to send them messages, in case of an error. We also need to find out when the user's told the interface they've completed their data entry, so we want to bind to something that throws Click events; we don't really care if it's a button or not. I also need to be able to control whether the user can request to log in; for instance, if they've just clicked, until the server responds, I don't want them to be able to keep clicking. If the View is not implementing a button, they can use this information in whatever way makes sense. In fact, writing this, it really shouldn't be called setLoginButtonEnabled, but something like setLoginEnabled instead.

The Presenter is going to perform the logic we had in our previous DefaultHomepage class, so we need our UserService.

private UserServiceAsync userService = null;

 @InjectService
 public void setService(UserServiceAsync service) {
  this.userService = service;
 }

mvp4g supports injecting these services (both RPC and non-RPC) with the @InjectService annotation. You define them as you normally would, with both Service and the ServiceAsync version. I didn't have to change the code at all.

The bind() method is called by the framework for us to do further set up of our interactions with the View. The BasePresenter we're extending provides both view and eventBus references so we don't need to handle that.

@Override
 public void bind() {
  super.bind();
  view.getLoginButton().addClickHandler(new ClickHandler() {
   public void onClick(ClickEvent event) {
    view.setLoginButtonEnabled(false);
    eventBus.login();
   }
  });
 }

All I want right now is to know when the login button (or whatever the View is using) is clicked, and when it is, I turn off the login button and fire a login event onto the Event Bus. This will allow some flexibility later, if I need to have another class know about logins.

Finally we define the method to handle the login event. All events are handled by methods that start with "on" followed by the event name. In this case, it will be onLogin.

public void onLogin() {
 if (userService == null) {
  Log.fatal("userService is null, not injected", 
    new NullPointerException("LoginPresenter.userService"));
  view.setMessage("Fatal application error.  \"userService not injected.\"");
  view.setLoginButtonEnabled(true);
  return;
 }
  
 userService.getUser(view.getUsername(), new AsyncCallback<User>() {
  @Override
  public void onFailure(Throwable caught) {
   view.setMessage("RPC FAILED");
   view.setLoginButtonEnabled(true);
  }

  @Override
  public void onSuccess(User result) {
   if (result == null) {
    view.setMessage("NO SUCH USER");
   } else {
    view.setMessage("SUCCESS: " + result.toString());
   }
   view.setLoginButtonEnabled(true);
  }
 });
}

This is pretty much the click handler that was in DefaultHomepage. We first do a quick sanity check for our UserService, and if it's not there, there's something really wrong. Then we call our service's getUser method, and when it returns we either tell the user the user doesn't exist, or spit out the user's information. Then we turn the login button back on.

View

The View just sets up visual aspect of the interface the user sees. In this case, we're still using a UiBinder, copied from the DefaultHomepage, and then adding the methods to flesh out the View interface we built in the Presenter.

com/lisedex/volinfoman/client/LoginView.java

public class LoginView extends Composite 
 implements LoginPresenter.LoginViewInterface {

 private static LoginViewUiBinder uiBinder = GWT
  .create(LoginViewUiBinder.class);

 interface LoginViewUiBinder extends UiBinder<Widget, LoginView> {
 }

The View is going to be a Composite widget, and it will implement the View interface. We also bind the View with its associated LoginView.ui.xml file, so we can use the fields declared there to provide information to the Presenter.

@UiField
 TextBox username;
 
 @UiField
 TextBox password;
 
 @UiField
 Button sendButton;
 
 @UiField
 HTML sendStatus;
 
 public LoginView() {
  initWidget(uiBinder.createAndBindUi(this));
  
  sendButton.setText("Login");
  
  sendStatus.setStyleName("serverResponseLabelError");
  
  DeferredCommand.addCommand(new Command() {
   public void execute() {
    username.setFocus(true);
   }
  });
  username.selectAll();
 }

This code is basically the same as what was in DefaultHomepage. We bind the XML defined user interface, then we set the text for our login button. We also set the style for the place we'll send messages to the user, so text shows up red, and then we set the focus on the username field. Setting focus did not work in previous versions of our code, so after a quick search I found this workaround described in GWT issue 1849, where they discuss making this the default implementation of setFocus().

The rest of the class is just implementing the LoginViewInterface.

@Override
 public String getUsername() {
  return username.getText();
 }

 @Override
 public String getPassword() {
  return password.getText();
 }

        .... etc, etc.

They're all just basic getters and setters, so I won't include them here, but you can take a peek at the complete code if you want to see the whole thing.

Some notes

I like the way mvp4g seems to work, and reading past posts on its discussion group was useful in figuring some of it out. There's other functionality I haven't even touched yet: code-splitting through using different modules for different parts of the interface, lazy loading, and, most importantly, history. I'll be adding history as soon as I have a second page or something where history makes sense. The others seem to be optimizations which we don't need yet.

I also may want to implement the command pattern later, to have more complete bits of information passed with their event on the event bus.

I've been avoiding it long enough, but now it's time to figure out a good authentication/session design, since I don't want to require the volunteers using this to have a Google account. If I did, it would pretty much be implemented for me in the App Engine Users API.

Guice

2010-07-20T22:21:00.004-05:00

NOTE: There is an intermediate stage for the project between the last posting and this one. In it, I add a servlet that will populate the datastore with sample data, optionally deleting the datastore first. It is protected by web.xml directives that no longer work after moving the servlet to Guice, but it's available as tag "add-objectify" on github.

Using Guice to inject an appropriate Dao is kind of a no-brainer goal. Use it to inject one that uses the App Engine datastore for production, and one that's got a dummy at the back end for testing.

Prepare to be Guiced

We need to make a Data Access Object interface that the methods needing it can use. The way that makes most sense to me is to rename our current DAO class to DaoGaeDatastore (correcting the capitalization to match the Java naming conventions), and extract from that a new Dao interface.

Eclipse makes this really easy to accomplish. The Refactor->Rename... menu item handles renaming the class, as well as updating all references to it in the code to the new name. The Refactor->Extract Interface... applied to DaoGaeDatastore pulls out the interface Dao, also with no real trouble. While I was at it, I removed the extra Objectify object I had added to the DAO, and used the ofy() method implemented by its parent DAOBase.

To the interface and implementation class, I added a method named deleteAllUsers(), which does exactly what you'd expect based on the name.

com/lisedex/volinfoman/server/DaoGaeDatastore.java

@Override
        public void deleteAllUsers() {
                ofy().delete(ofy().query(User.class).fetchKeys());
        }

One of the Objectify.delete() methods accepts an iterable collection of objects that can be a mix of Keys and POJOs (Plain Old Java Object). Executing a query() on User.class will return all Users, and the Query.fetchKeys() method will return an iterable collection of their Keys. We can just pass this collection to Objectify.delete(), and all of our Users in the datastore disappear.

We can now convert our BuildDB class, which was built between the last blog entry and this one, to use Dao.deleteAllUsers(), and remove its dependency on an Objectify object.

Let the Guicing commence

We need the guice-servlet jar.

I used the Guice integration documentation to see how to get Guice set up for the App Engine environment. All servlets where we want to use Guice need to be moved from their web.xml declarations into Guice module configurations. Also, all servlets must be have the Singleton scope, so we can either use the @Singleton notation in the class definitions themselves, or follow the design philosophy we used for Gin and declare them all in a Guice module. I chose the latter for consistency.

First we build an AbstractModule called VolinfomanGuiceModule. Its sole function is to override the configure() method to declare our Guice object bindings.

com/lisedex/volinfoman/server/guice/VolinfomanGuiceModule.java

public class VolinfomanGuiceModule extends AbstractModule {
        @Override
        protected void configure() {
                // Servlets
                bind(UserServiceImpl.class).in(Singleton.class);
                bind(BuildDB.class).in(Singleton.class);
        }
}

Since all servlets need to have Singleton scope, and we're not injecting them with replacement objects, this is all the binding we need.

If you've looked at the code before now, you'll notice we're using a library called gwt-log with a remote logging servlet that's been declared in web.xml. We're not moving this to the Guice configuration because it doesn't need Guice injection right now, and it also produces a warning when it's not declared in web.xml. This warning problem is corrected in r504 of gwt-log, but I'm only using the release version.

Next, we need a ServletModule that sets up what URLs each servlet handles, so we build VolinfomanServletModule and override configureServlets().

com/lisedex/volinfoman/server/guice/VolinfomanServletModule.java

public class VolinfomanServletModule extends ServletModule {
        @Override protected void configureServlets() {
                serve("/volinfoman/user").with(UserServiceImpl.class);
                serve("/volinfoman/admin/builddb").with(BuildDB.class);
        }
}

Each serve() call tells Guice which URL to map to which servlet class. Each servlet can handle multiple URLs through multiple calls to serve().

Finally, we need a GuiceServletContextListener that provides a Guice Injector that knows about the modules so it can configure the injection properly.

com/lisedex/volinfoman/server/guice/VolinfomanGuiceServletContextListener.java

public class VolinfomanGuiceServletContextListener extends
                GuiceServletContextListener {
        @Override
        protected Injector getInjector() {
                return Guice.createInjector(
                                new VolinfomanServletModule(),
                                new VolinfomanGuiceModule());
        }
}

In web.xml we need to remove references to the servlets we're configuring through Guice, and add the configuration for Guice in their place. After taking out the BuildDB and UserServiceImpl declarations, but leaving the RemoteLoggerServiceImpl for gwt-log, we put in the Guice code.

war/WEB-INF/web.xml

<!-- Configure Guice servlet, other servlets configured in
                 VolinfomanServletModule -->
        <filter>
                <filter-name>guiceFilter</filter-name>
                <filter-class>com.google.inject.servlet.GuiceFilter</filter-class>
        </filter>

        <filter-mapping>
                <filter-name>guiceFilter</filter-name>
                <url-pattern>/*</url-pattern>
        </filter-mapping>

        <listener>
                <listener-class>com.lisedex.volinfoman.server.guice.VolinfomanGuiceServletContextListener</listener-class>
        </listener>

We set up a GuiceFilter with a URL pattern that describes which URL will be processed by Guice. We also plug our GuiceServletContextListener as a listener so Guice gets configured when the application is deployed.

Moving the BuildDB servlet to Guice configuration stops the authentication we had set up through a <security-constraint> in the servlet container. We'll have to add security in the code itself later, but for now the BuildDB servlet is open to the world.

Guiced!

All that set up for what will appear to be a pretty small payoff, but we're ready to use Guice anywhere we want in our servlets, as well as Gin in our GWT client.

We want to inject an instance of DaoGaeDatastore in place of fields declared as Dao in our servlets. First we need to configure this in VolinfomanGuiceModule.

com/lisedex/volinfoman/server/guice/VolinfomanGuiceModule.java

// Data providers
            bind(Dao.class).to(DaoGaeDatastore.class).in(Singleton.class);

After adding this to the configure() method, a DaoGaeDatastore singleton will be provided to any servlet that has a class field of type Dao that we also annotate with @Inject. So we need to inject it in UserServiceImpl.

com/lisedex/volinfoman/server/UserServiceImpl.java

@Inject
    private Dao dao;

This replaces the previous declaration for dao, and tells Guice to do the injection for that field. The rest of the class stays the same, since we were already using the Dao interface to define our interaction with the datastore.

In BuildDB we remove the local variable dao in doGet(), and use the same code as above to declare it as a class variable that needs injection.

That's it. The project is now Guicified.

Using Objectify to access the datastore

2010-07-20T17:06:00.005-05:00

There are several options to choose from when deciding how you want to access the App Engine datastore. Two options are Java Data Objects (JDO) and Java Persistence API (JPA); both are from Sun and are in the Java SDK. From searching around the web, the consensus seems to be that both are too heavyweight for most App Engine projects. They are, however, both standards, and JDO is datastore agnostic, which would make moving your application off of the App Engine platform somewhat easier. However, you still are stuck with the limitations of the GAE datastore, so it's not quite so easy bringing JDO code into your GAE project.

Open source alternatives for Google App Engine

Two GAE specific open source projects are Objectify and Twig. Out of the two, Objectify seems simpler and more aligned with what's actually happening in the datastore. As I learn GAE, I feel it's somewhat imperative to not have the details of the datastore hidden from me. It's also modeled after the GAE Python library, which seems pretty clean. And, last but not least, your data classes can be passed through GWT RPC without modification. Perhaps in the future I'll come to a different conclusion if I get tired of having to write too much in the way of nuts and bolts that Twig handles automatically. Twig also supports parallel queries and merging OR queries, both of which could be pretty nice.

Hearing about the different design philosophies straight from the horses' mouths gave me some good information I needed to make this decision, too.

The Objectify concepts documentation is a good starting point for understanding both Objectify, and the GAE datastore.

Adding Objectify to VolInfoMan

Adding Objectify to the project leads us to the first time that we need a servlet running on App Engine. The servlet will simply provide a front end to Objectify and the datastore by responding to GWT RPC calls.

The code for this state of the project is available as tag "add-object-step1" on github.

He's not any kind of program, Sark. He's a User.

The first step is to define the object we want to pass back and forth between our GWT code and the GAE servlet. To continue the login page we've been working on, we'll need a User so we can authenticate the session. Since the User will be shared between GWT and GAE, we'll put it in the shared package.

(I'm leaving out the package and import statements to conserve space)

com/lisedex/volinfoman/shared/User.java

public class User implements Serializable {
        @Id
        private Long id;
        @Indexed
        private String username;
        @Indexed
        private long status;
        @Unindexed
        private String firstName;
        @Unindexed
        private String lastName;
        @Unindexed
        private String email;
        @Unindexed
        private String password;

Objects that will go through RPC need be Serializable. Later on, I'll probably add the @Cached annotation to the class so that it will automatically be cached by GAE's memcache.

You can only run queries on the GAE datastore against fields that are indexed. More specifically, you need an index for every query you intend to run, so by default all class fields are indexed to make it easier to query on any specific field. However, according to the Objectify best practices documentation, each property that is indexed requires a separate write to the index, which won't add to latency (writes are done in parallel), but will add to the CPU time used by the application. So instead, I explicitly declare which fields are to be indexed, and which are not. The same could be achieved by declaring the whole class @Indexed and only specifying the @Unindexed, or vice versa, but I like to explicitly declare each for easier reading. The field to be used to create the Key for the object is annotated with @Id.

I put in a couple of getter/setters, and a toString() method for easy logging. It will need to be fleshed out more later.

GAE, we need to have a serious talk. - Love, GWT

Next, we need to define the interface our client uses to access the servlet. I put GWT code for handling data into com.lisedex.volinfoman.client.data, so I define a UserService interface in that package.

com/lisedex/volinfoman/client/data/UserService.java

@RemoteServiceRelativePath("user")
public interface UserService extends RemoteService {
        User getUser(Long id);
        User getUser(String username);

        void putUser(User user);
}

The important part is that we declare what URL we'll be using to access the RPC servlet using the @RemoteServiceRelativePath annotation. The interface itself extends RemoteService, which all GWT RPC client interfaces should extend, and then we add the RPC functions we want to support. I want to support grabbing a User by name or id number, or putting a User object back into the datastore.

If you're using Eclipse, it will complain about the lack of a UserServiceAsync interface, and will build it for you automatically if you let it. It's basically the same as above, but doesn't need to extend RemoteService or declare a @RemoteServiceRelativePath. You can grab the complete code from github, as shown above.

In the server package, we need a servlet class that implements the UserService interface we just defined.

com/lisedex/volinfoman/server/UserServiceImpl.java

public class UserServiceImpl extends RemoteServiceServlet implements
                UserService {

    private DAO dao = new DAO();

    public User getUser(String username) {
            return dao.getUser(username);
    }
}

The real code has stub implementations for getUser(Long id) and putUser(User user) since we're not using them for now. We want to abstract away how we'll be accessing the datastore in case we need to change it, so we put actual interaction in a Data Access Object called...DAO.

com/lisedex/volinfoman/server/DAO.java

public class DAO extends DAOBase {
        static {
                ObjectifyService.register(User.class);
        }

        private Objectify ofy;

        public DAO() {
                ofy = ObjectifyService.begin();
        }

We don't have to extend the DAOBase class, but it gives us a couple of functions that we can use without coding, such as a lazily instantiated Objectify object. Of course, I didn't know that at the time I wrote the code, so I created my own. Oops.

You need to register the classes you'll be using with Objectify, and calling ObjectifyService.begin() returns an Objectify object we'll use to interact with the datastore.

public User getUser(String username) {
                User fetched = ofy.query(User.class)
                    .filter("username", username).get();
                return fetched;
        }

        public User getOrCreateUser(String username) {
                User fetched = ofy.query(User.class)
                    .filter("username", username).get();
                if (fetched == null) {
                        fetched = new User(null, username, 
                            User.STATUS_INVALID, null, null, null, 
                            null);
                        ofy.put(fetched);
                }
                return fetched;
        }

        public void putUser(User user) {
                ofy.put(user);
        }
}

To get a User by username, we need to execute a query on the datastore for all User objects, and filter the results by the username field. If no such object is in the datastore, the get() method will return a null.

Writing the object is easier, since we already know the Key for the object, as it's encoded in the class (User.class, and Long id). Just pass the object to Objectify's put() method. If the id field in the object is of type Long, and its value when passed to put() is null, Objectify will automatically generate an id for the object. If the id is of type long or String, the developer is responsible for that themselves.

The getOrCreateUser() function should be modified to use the get() and put() methods in DAO. Another oops. And I need to add more data checking, such as if the User sent to put() is null.

Make the Login button do something

Now we need to wire this stuff into the user interface, so that when we hit the Login button, we actually go out to the datastore and try to retrieve the appropriate User.

com/lisedex/volinfoman/client/DefaultHomepage.java

class MyHandler implements ClickHandler {

    @Override
    public void onClick(ClickEvent event) {
        sendStatus.setText("Sending..." + username.getText() +
                           "/" + password.getText());
        sendButton.setEnabled(false);
        userService.getUser(username.getText(),
            new AsyncCallback<User>() {
                  public void onFailure(Throwable caught) {
                      sendStatus.setText(sendStatus.getText()
                                 + "   FAILED");
                      sendButton.setEnabled(true);
                  }

                  @Override
                  public void onSuccess(User result) {
                      if (result == null) {
                             sendStatus.setText(sendStatus.getText() 
                                 + "    NO SUCH USER");
                             sendButton.setEnabled(true);
                             return;
                      }
                      sendStatus.setText(sendStatus.getText()
                          + "    SUCCESS: " + result.toString());
                      sendButton.setEnabled(true);
                  }
              });
    }
}

sendButton.addClickHandler(new MyHandler());

We obviously have not implemented the MVP model, as all of this stuff is in our view. In the ClickHandler inner class, we call the UserService.getUser() asynchronous method with an AsyncCallback inner class that will handle what happens when the RPC call returns.

If the RPC call fails, onFailure gets called with an exception. This doesn't happen if the username doesn't exist in the User indexes; it only happens with the client is not able to access the servlet for some reason. onSuccess is called with a User object when the RPC call works. The result is null if no such User was found, otherwise it's populated with all stored fields for the object.

Next up: Guice, security

Guice is the basis for Gin, except Guice provides dependency injection for Java generally, where Gin is for GWT. DAO's are a perfect place to apply dependency injection, since we may want to use the real datastore back end, or a dummy used for testing. We'll cover that in the next installment.

Also, there's no security or authorization needed to call any of the RPC functions. Anyone that wants can read or write any User object to my datastore, even if they're not using my client application. We'll, uh, get to that sometime.

Dependency injection and Gin

2010-07-18T03:56:00.001-05:00

I was coming across the phrase "dependency injection" quite a bit while reading articles about GWT and MVP, and references to the Guice and Gin libraries coming out of Google. I figured it was some new programming paradigm and that I was hopelessly out of date not knowing what it was. The wikipedia entry didn't really help clear it up, nor did other articles I read. I kept getting the feeling that I was missing something, and that the concept wasn't really something new.

Turns out I was missing something, all right

I was missing that DI is giving a name to something that I and every other object oriented developer have been doing for years.

The short summary in this great article is what finally made me realize that I could stop trying to make it more complicated than it was: "Dependency injection means giving an object its instance variables. Really. That's it." If you're having trouble with the concept, James Shore's article is invaluable.

A conceptual example would be something like: you have an object that relies on a reference to some sort of data provider. In production, the provider will be a database back end, but during testing you want to mock up a provider that is always up and you can precisely control what errors it has.

You might solve this by constructing a factory that spits out the correct type of provider, and your object can ask that factory to give it a reference to the proper kind of provider. This works, but your testing has to construct, tear down, and reset the factory to the previous state between each test. You also have to write a lot of boilerplate code for each of these factories.

DI refers to giving the object a reference to the proper provider, which, during testing, allows you to build a new provider instance which you pass to the object during each test. As you exit the test, the object and provider are dereferenced automatically, saving you the cleanup. But in the code itself, you're now either passing provider references possibly through layer after layer of constructors, or you're building factories for the initial object which get the proper provider and pass it to the object, and then return the completed object.

Guice and Gin save you from all this repetitive factory building and wiring, and what provider each object needs can be declared in one place, or inline where the instantiated providers will be injected in the object.

I'm certainly not qualified at this time to discuss all of the nuances and capabilities of these libraries, since there are many different ways this concept could be used.

Instead, I'll talk about using Gin in this project

com/lisedex/volinfoman/Volinfoman.gwt.xml

<!-- Include Gin -->
<inherits name="com.google.gwt.inject.Inject" />

Adding those lines to your module's gwt.xml file, and adding the Gin and Guice jar files to your classpath, is all that it takes to enable the Gin annotations and compile time code generation.

If you go to the VolInfoMan project page on github, you can go to the basic-gin tag and download the source. The project is very, very basic at this point, and it's easy to find the one instance of using Gin.

As practice, I wanted to set up the home page you land on when loading the site to be capable of being changed by dependency injection. This will allow, later, changing the implementation of the page by making a new class that extends the skeleton Homepage class, and just changing the Gin binding in one place, and all references to that Homepage will be using my new implementation. Look ma, no factory!

com/lisedex/volinfoman/client/Homepage.java

package com.lisedex.volinfoman.client;

import com.google.gwt.user.client.ui.Composite;

public class Homepage extends Composite {
}

First we define our skeleton Homepage class, which will be added to our RootPanel later in onModuleLoad() (actually onModuleLoad2() due to our use of the gwt-log library).

com/lisedex/volinfoman/client/gin/VolinfomanGinjector.java

package com.lisedex.volinfoman.client.gin;

import com.google.gwt.inject.client.GinModules;
import com.google.gwt.inject.client.Ginjector;
import com.lisedex.volinfoman.client.Homepage;

@GinModules(VolinfomanModule.class)
public interface VolinfomanGinjector extends Ginjector {
 Homepage getHomepage();
}

We extend the com.google.gwt.inject.client.Ginjector class and use the @GinModules annotation to specify what class contains the binding configuration. We also define the getHomepage() injector method, which we need when using Gin in our application initialization code, because Gin is working at compile time to generate JavaScript, unlike Guice. Other dependencies that operate below our initialization code will be injected automatically, and won't require this type of method.

com/lisedex/volinfoman/client/gin/VolinfomanModule.java

package com.lisedex.volinfoman.client.gin;

import com.google.gwt.inject.client.AbstractGinModule;
import com.lisedex.volinfoman.client.DefaultHomepage;
import com.lisedex.volinfoman.client.Homepage;

public class VolinfomanModule extends AbstractGinModule {
        @Override
        protected void configure() {
                bind(Homepage.class).to(DefaultHomepage.class);
        }
}

We extend AbstractGinModule to declare our bindings for injection. Using AbstractGinModule instead of GinModule allows us to drop the binder.bind() syntax in favor of bind(). In the above code, we're declaring that when our code asks for a Homepage object, it will get a DefaultHomepage instead.

com.lisedex.volinfoman.client.DefaultHomepage actually contains no references to Gin. It extends Homepage but doesn't need to know anything about injection.

com/lisedex/volinfoman/client/Volinfoman.java

package com.lisedex.volinfoman.client;

import com.lisedex.volinfoman.client.gin.VolinfomanGinjector;
// import GWT objects here...

public class Volinfoman implements EntryPoint {
        public void onModuleLoad() {
                // we set up gwt-log here and
                // use a DeferredCommand to execute
                // onModuleLoad2() */
                // ...
        }

        private void onModuleLoad2() {
                // Create a ginjector
                VolinfomanGinjector ginjector = 
                    GWT.create(VolinfomanGinjector.class);
                // Add the homepage to the rootpanel
                RootPanel.get().add(ginjector.getHomepage());
        }
}

And our GWT entry point creates the Ginjector, and we use the injector method getHomepage() to inject an instance of DefaultHomepage, thanks to the binding we set up in VolinfomanModule. As mentioned above, we have to use an injector method due to current limitations in Gin.

That's it. Remember to grab the basic-gin tag of the source code from github for a compile-ready Eclipse project where you can immediately play with Gin.

Other Gin information

Gin supports some of the same annotations as Guice, such as @Inject, @ProvidedBy, @Singleton, @ImplementedBy, all of which allow you to move much of the binding information out to the source where the bindings occur. Much of this is to provide an alternative to using a GinModule to declare all of the bindings in one place; the latter is the option I prefer, but it's personal preference for the most part.

Some of the other bind() syntaxes are:

bind(Something.class).toProvider(SomethingProvider.class);
bind(Something.class).to(SomethingImplementation.class)
    .in(Elsewhere.class);
bind(Something.class).annotatedWith(SomethingAnnotation.class);
    .to(SomethingImplementation.class);
bind(Something.class).to(SomethingImplementation.class)
    .in(Singleton.class);

Line 1: When Gin injects an instance of a class normally, it uses the class's default constructor. If you need to give extra information to a constructor, you can use the toProvider() syntax. Gin will inject an instance of Something from the SomethingProvider wherever you need a Something class.

Line 2: If you only want a binding to apply in Elsewhere, instead of globally, you can use this syntax to specify that scope.

Line 4: You can also set up custom annotations. When you want multiple bindings for one type, you can specify which binding to use by setting up an annotation for each. The annotation and the type will uniquely identify which binding is appropriate. See the Guice wiki for more information.

Line 6: Gin's injector normally creates a new instance of each SomethingImplementation when it gets injected. However, sometimes you'll want to use the Singleton pattern, and only have one instance of SomethingImplementation running around. Remember that this is a Singleton per Ginjector, so if you have multiple Ginjectors (which, if you do, you probably need to double check your code) there will be one instance of the SomethingImplementation for each.

There's a bunch more that can be done with Gin; this is obviously only scratching the surface. Remember that Gin does the injection at compile-time, so there should be almost no overhead for using it, and it can save you a lot of repetitive coding. To understand all of the capabilities, it would behoove you to read the Guice User's Guide, the Gin tutorial, and the compatibility information between Gin and Guice. The documentation for Guice is much more thorough, but you have to know which ideas can be used in your GWT code. Guice will be used later for our server side code which runs under GAE.

More research...by watching YouTube videos

2010-07-12T19:12:00.001-05:00

I'm itching to start hacking code, and I probably will this evening, but I forced myself to spend a good bit of time yesterday doing research: I read articles at the Google App Engine site, GWT tutorials, and best practices for both GAE and GWT. Generally, what I got from those is how much I have to learn about these technologies, which is a little depressing. Also, that if I'm not really focused, I'm going to get sidetracked by reading about issues I'm not even facing yet. But the up side is that there's so much information out there, already available, that most of the problems I'll run in to someone else has already hit and written about it.

Google IO videos, or how I learned to stop multitasking and love YouTube

The Google IO talks, presented on the Google Developer's Channel on YouTube, were probably some of the most informative stuff I looked at yesterday, as well as exemplifying my tendency to get sidetracked; do I really need to watch an hour and a half of the Google Wave product launch? I'm not even going to provide a link to that. It also served to remind me that sometimes trying to multitask is inefficient, as when I'd put the videos in the corner of my screen and try to do other work, I'd completely miss huge chunks of the talk. Having to continuously rewind, and, while the video was getting back to the section I missed, going back to my regular work, only to completely miss the same section again...well, it started to border on the absurd. Biting the bullet, and telling myself that just watching the video with no distractions was actually most the efficient way to get this done, was also the way to get the most out of these videos. And there's a ton of good information there.

Max Ross easily rose to the top as one of my favorite speakers. During his videos, I would actually neglect my work and focus completely on the video. The first talk of his I watched was an explanation of schemas in the data store, since I'm not going to have complete SQL functionality while writing an application for the GAE. I also listened to his (audio only) talk about Hibernate Shards, even though I won't be needing that, most likely ever, for this particular project. Speaking to Max's skill as a speaker, my girlfriend, who actively tries not to learn about computers almost to the point of Luddism, got sucked in while I was listening. I believe that, initially, it was her mishearing the word "sharding" as "sharting," and that hooked her scatalogical mind. But she ended up putting down her book, and listening with me. By the end, she actually asked me questions that showed she grasped a lot of what Max was talking about. She's a smart girl, but I was pretty surprised.

Testing

I'm a big proponent of proper testing when I'm developing software, and though I occasionally get some push back for that, it really helps my confidence in code as a project progresses. So it was nice to hear from Max that he really believes in testing, too, during his talk about GAE Testing Techniques. Using the cloud as a test farm is pretty ingenious, so I ~~wasted~~spent some time looking at Cloud Cover test harness. The code hasn't been touched since May, but I really like the idea of testing my GAE code actually on the App Engine servers. This led to GWT testing, which reinforces the idea that the MVP model will promote not just a clean conceptual layout for the project or make it easier to implement code splitting, but will also make it much easier to have GWT test cases that will run very fast, except for the small percentage that need to test the GUI widgets themselves. This talk also pointed me to WebDriver, which is the heir apparent to Selenium. The fact that Selenium ran the testing through JavaScript always seemed a bit too abstracted from the browser itself to provide true test from the user all the way to the data store. WebDriver will plug directly in to the browser and generate actual key strokes, button clicks, and other events.

Odds and ends

I also watched a talk about some of the new stuff in Google App Engine (at least new since the 2009 Google IO conference), developing complex and scalable apps in GAE, and new query capabilities that have been added to the data store.

And I believe watching all of that in one day kind of made me a bit loopy, and I know that a lot of the specifics ended up getting pushed out by new specifics; but now I have a better idea of what's out there, so when I do run into an issue or question, I know where to look. This post is actually more for me, so I have a place to refer with all of these links. I also have a good idea of how to implement my testing, and how the project should be designed initially.

It was actually time well spent, even though I had to suppress my urge to just start coding, and I'm now in a better position to start without having to feel like I'm going to have to revisit, in the near future, every single decision I make.

And I need to get over my mancrush on Max Ross. He's so dreamy.

A short note on Cygwin git

2010-07-11T10:36:00.000-05:00

I use git for my source control, and when developing under Windows, I use the Cygwin environment for all of my command line needs. My background is primarily Unix, so I'm used to having a powerful shell prompt and scripting for a lot of tasks. I also use the git available from Cygwin instead of the one from the git site.

You will quickly run into execute permission problems and CRLF vs. LF translation problems.

Execute bit

Windows doesn't manage the execute bit very well, especially on a FAT filesystem, but I find it even on NTFS. Files created under Cygwin will be created according to your umask, usually as mode 0644, while files from Windows will show up as 0700. This isn't a real problem except that git tracks the mode of each file in the repository, so changes to the mode will show up as the file being modified. Setting core.filemode to false tells git to ignore the executable bit. If something slips in, you may have to use git update-index --chmod=-x to correct it.

core.filemode is set in your .git/config file by git init regardless of how it's set in your system or global configuration, so you will need to set it in your repository configuration.

`CRLF` vs. `LF`

Files created under Cygwin will use only LF to mark end of line, while files created under Windows applications will use CRLF. By setting core.autocrlf to true, Cygwin git will convert all CRLFs to LFs on commit operations, and all LFs to CRLFs when checking out.

I also set core.safecrlf to warn, which checks, before adding a file, that doing the crlf conversions is "safe". We want to make sure that we'll get the same file back when it's checked out. Git cannot "safely" convert files that contain both LFs and CRLFs, so a file that's been edited under both Windows and Cygwin may produce warning: LF will be replaced by CRLF in Foo.java when running git add. This is not a problem with text files, because the file will be cleaned up during its next commit so that all EOLs are the same. However, binary files will be corrupted if they are not properly detected by git. I have never seen git label a binary file as text, but if this does occur you can set the file in .gitattributes as binary. For example

   *.jpg binary

will make git consider all .jpg files as binary, and will not apply the CRLF conversion, nor will it try a text diff.

You can also set core.safecrlf to true instead of warn, in which case the error will be fatal, and you will have to correct the problem before you can add the file. You can run a text file through unix2dos, or add the binary file to .gitattributes, but since git bats a thousand, in my experience, at detecting binary files, it's a waste of time to have to manually fix all of your text files.

TL;DR

So in conclusion, if developing using git under Cygwin, I run the following commands after creating a repository:

   git config core.autocrlf true
   git config core.safecrlf warn
   git config core.filemode false

Explanations of these settings can be found on the git-config man page.

In the beginning...

2010-07-10T03:20:00.000-05:00

What's all this then?

I need to bring my web development skills up to date, instead of living in the all raw-HTML-only-static-pages past. After doing some research, I liked what I read about Google App Engine (GAE), as well as Google Web Toolkit (GWT). Especially that, right now, it will let me focus on the code instead of managing the servers.

Oh, and I can use Java for both the client and server.

I volunteer at a local non-profit, where volunteers track their time with pen and paper, and our contact information is gathered once when we sign up, and then never updated. The hours have to be manually transcribed into a spreadsheet, and it makes it hard to run reports or gather statistics. This seemed like an ideal project that shouldn't take forever, but also should get into the meat of the two products.

OK, but what's the deal with the blog?

Sometimes I work late, and have trouble remembering the next day exactly what I had tried, what research I had done and its conclusions, or just where I am in the project. I usually document this stuff in a text file. While researching best practices for GAE/GWT, I wanted a site that chronicled the creation of an application from the beginning. Initially I didn't find any. I based my knowledge of best practices on the famous and excellent Google I/O talk: GWT App Architecture Best Practices by Ray Ryan. It's invaluable.

Later, I did come across a GWT/GAE Development Blog that seems like a good introduction, though I haven't worked through much of it yet. I'm sure I'll use it as a reference as I learn.

But I want to chronicle the fits and starts, both the good and bad ideas, the failures, the dead ends, the screw ups; and then hopefully successes.

Basic project details

I haven't decided on a name yet, so I'll just call it "the project" for now. It will be hosted on github, as I like git's branching and DCVS model. I'll use Java in Eclipse Helios as the language, but still access git from the command line, since I've had plenty of problems with the git plug-in. I will probably use the Gin and Guice libraries, and base my MVP design around the gwt-presenter and gwt-dispatch libraries, though they look like they haven't been updated in a while.

For testing the pages I'll use the Google plug-in for Eclipse, as well as the Speed Tracer, Firebug, etc.

Enough procrastinating...