As you Grails die-hards know, Grails 1.1 introduced the new GrailsUnitTestCase class which is a very useful extension of GroovyTestCase with support for mocking the dynamic properties and methods that Grails injects into various types of classes (domain, controller & taglibs). Getting the team up to speed on this change was the original purpose of the presentation, but I decided to expand it to cover the full spectrum of testing capabilities in Grails 1.1 for those who might not have much exposure to it previously (including me!)

The presentation covered:

- the benefits & drawbacks of each type of testing: unit testing, integration testing, functional testing.

- details on the new GrailsUnitTestCase and it’s extensions (ControllerUnitTestCase and TagLibUnitTestCase)

- a few “gotchas” – such as using mockDomain with inheritance hierarchies

- WebTest goodness – test organization & re-use, Groovy step, passing params between Groovy & WebTest contexts, a little dive into using HtmlUnit to test AJAX

- general recommendations for what types of tests to create for each artifact (domain classes, controllers, services, etc)

Here’s a link to the presentation on SlideShare.net (where you can view it online or download it):

Happy testing!
Dave

Install memcached, which is simple if you use a Mac and have MacPorts installed.

$ sudo port install memcached

Get the memcached jar , the hibernate-memcached jar and the PostgreSQL JDBC driver jar, then  add them to the project.

$ cp memcached-2.3.1.jar hibernate-memcached-1.2.jar postgresql-8.4-701.jdbc3.jar <project_dir>/lib/

Configure memcached as the second-level cache by adding something like this to your DataSource.groovy.

hibernate {
    cache.use_second_level_cache = true
    cache.use_query_cache = true
    cache.provider_class = 'com.googlecode.hibernate.memcached.MemcachedCacheProvider'
    memcached {
        servers = "localhost:11211"
    }
}

I wrote a simple test to generate a whole bunch of domain objects and shove them in the database, then go back and fetch them all again. I set the test to run the insert/load loops in batches of 1, 10, 100, 1,000, 2,000, 4,000, 7,000, and 10,000 objects. I expected that inserting the objects would take a little longer because of the memcached overhead but that loading the objects would be faster because they could be loaded from memcached instead of going out to the database.

I tested three persistence backend configurations.

• HSQLDB, the default for new Grails projects and when running unit/integration tests. This is an in-memory database only, so data aren’t persisted permanently.
• PostgreSQL for a full-blown RDBMS setup. I set dbCreate to “create-drop” in DataSource.groovy so that each test started with an empty database.
• PostgreSQL with memcached enabled. I should point out that in this test I had both PostgreSQL and memcached running on the same machine (my laptop) as the application, so it’s really not taking advantage of memcached’s parallelism strengths.

My expectations held true for the inserts. The memcached case is a little slower but not that much.

Comparison of object insert times using HSQL, PostgreSQL, and memcached+PostgreSQL for persistence backend

When I tested the load times, I was surprised. It looked like memcached wasn’t speeding things up at all. I was also surprised to see that it took less time to load 10,000 objects from the data store than 7,000 objects. I ran the test a few times to rule out the possibility of a load spike from something else running at the same time, but I got similar results every time.

Comparison of object load times using HSQL, PostgreSQL, and memcached+PostgreSQL for persistence backend

I then tried to improve my test setup by running memcached on a second machine (my desktop, connected by gigabit ethernet to the laptop). Using the PostgreSQL + memcached configuration for each test this time, I looked at three more scenarios.

• PostgreSQL running locally on the laptop, memcached running locally on the laptop.
• PostgreSQL running locally on the laptop, memcached running remotely on the desktop.
• PostgreSQL running locally on the laptop, memcached running on both the laptop and the desktop.

There’s some overhead in going out to the network to retrieve data instead of fetching it from a process running locally, so I expected the memcached local instance to be faster than the memcached remote instance. I expected the third test, with two instances of memcached running in parallel, to be fastest because it splits the load between the two instances and frees up some CPU time on the laptop. Things didn’t play out that way, though.

Comparison of object insert times for various memcache setups

Comparison of object load times for various memcache setups

I have to be honest that these were not great tests. For one, I should really have run those insert/load loops in parallel instead of sequentially. I’m guessing that the test spent a lot of time stalled waiting for data when it could have been sending out more requests. I would also like to test with a large group of memcached servers and a remote database, perhaps even a cluster of app servers with a load balancer to try the full meal deal. The memcached FAQ explains that running everything on one machine will not show off memcached’s scalability.

However, this gave me the opportunity to get my hands dirty with memcached and learn a few things.

• Integrating memcached is so simple and the performance penalty of just running it locally during development is so small that there’s no reason not to enable it.
• It’s dead simple to install and set up. We could , for example, easily install it on a bunch of test servers and run multiple instances on different ports – one for each developer (to prevent us from clobbering each other’s data).
• Measuring performance and scalability realistically is tricky. It’s one thing to hand wave and say that running multiple instances of a database or a cache will speed things up, but it takes some work to set up a realistic simulation of high traffic application performance.
• Adding memcached doesn’t automatically turn performance up to 11. I need to learn more about what’s going on under the hood with my domain objects and how they’re being persisted through memcached.

I look forward to using memcached more in the future and learning how to really take advantage of it.

Update (2009-07-24): Ray pointed out in the comments that I should have enabled caching in my domain objects under test. I added “static mapping = { cache true }” to my domain class definition and re-ran the tests to compare the effect under 4 scenarios.

• no domain object caching (as before)
• domain object caching with memcached running locally
• domain object caching with memcached running remotely
• domain object caching with memcached running both locally and remotely

Not much difference with the inserts.

Comparison of object insert times with domain object caching enabled and disabled

However, the load times are slightly better with memcached enabled during heavy activity! The advantage disappears when I add the overhead of traversing the network, but it’s a start. Thanks for the pointer, Ray!

Comparison of object load times with domain object caching enabled and disabled

Below is a simple test case showing how to do LINQ operations on a simple array:

        public void LinqArrayTest()
        {
            String[] names = { "Burke", "Connor", "Frank",
                   "Everett", "Albert", "George",
                   "Harris", "David" };

            IEnumerable<string> query = from s in names
                               where s.Length == 5
                               orderby s
                               select s;

            List<String> results = query.ToList();

            Assert.AreEqual(3, results.Count);
            String[] expected = new String[] {"Burke", "David", "Frank"};
            for (int i = 0; i < results.Count; i++)
            {
                Assert.AreEqual(expected[i], results.ElementAt(i));
            }
        }

It is also possible to create runtime LINQ queries using lambda expressions to qualify conditional logic:

        [TestMethod()]
        public void LinqLambdaTest()
        {
            String[] items = { "cabbage", "broccoli", "carrot", "parsnip", "cauliflower" };

            IEnumerable<String> query = items;

            // additive query generation using lambda expressions
            query = query.Where(item => item.StartsWith("c"));
            query = query.Where(item => item.Length < 8);
            query = query.OrderByDescending(item => item.ToLower());

            List<String> results = query.ToList();

            Assert.AreEqual(2, results.Count);
            String[] expected = { "carrot", "cabbage" };
            for (int i = 0; i < results.Count; i++)
            {
                Assert.AreEqual(expected[i], results.ElementAt(i));
            }
        }

LINQ provides a simple, generalised query framework for operating on data collections within the .NET stack. There are framework plugins to support LINQ backed by various datastore technologies, these include: LINQ to SQL, LINQ to XML and LINQ to Entities.

Single Question Mark – nullable types

The single question mark syntax is somewhat obscure. Essentially it is short hand syntax for marking primitives as nullable. System.Nullable is a generic struct that is essentially a wrapper to signify nullability. This can be used as shown below:

// a nullable type for int
int? thing = 12;

// is syntactically equivalent to
System.Nullable<int> otherThing = 12;

// can be any valid int value OR null
thing = -12;
thing = null;

// usage
//y is set to zero
int y = thing.GetValueOrDefault();
// but this will throw an exception as thing.HasValue == false
y = thing.Value;

Double Question Mark – non null assignment precedence

This one is a little obscure but can be quite handy, it is very similar to certain assignment constructs in Perl as it allows one to use precedence to assign a variable from other potential variables and is a nice way to deal with default values.

String thing = null;
String defaultThing = "nothing";

// standard trinary syntax
String otherThing = (thing != null) ? thing : defaultThing;

// syntactic sugar for the above
String sugaredThing = thing ?? defaultThing;

// can also chain things
otherThing = null;

// the first non-null value is the result of the assignment
String anotherThing = thing ?? otherThing ?? defaultThing;

// if all potentials are null, then the assignement is null)
anotherThing = thing ?? otherThing;

In most web applications, the proportion of dynamic requests (application logic) to static requests (images, javascript, css, html, flash) is usually quite low. As a result, it can be quite taxing on your application server if most of its time is spent serving static content to users.

Looking at some real statistics over two months from one of the systems we developed shows that out of 2,522,507 total requests, only 395,742 were for dynamic application data. That is 15% of all requests were dynamic, the remainder being requests for images, stylesheets or flash components.

System architecture
A common technique in web systems is to have a lightweight web server reside between your backend web application (usually a heavyweight beast) and the world at large. This allows the lightweight web server to take load off the heavyweight backend web application leaving it to focus on generating all the fun dynamic content for your application. The application server threads do not get tied up serving static content this improving system utilisation.

A lightweight proxy server (e.g. Apache HTTPD with mod_proxy) can be configured to improve performance by serving up static content fast and ensure correct headers are set such that upstream content proxies and end user browser software will cache the static content appropriately, further improving performance for repeat visitors.

Example Implementation
We setup a system with Apache HTTPD as the lightweight proxy server which sits in front of our application servers. The two are connected via mod_proxy. This Apache module provides a bridge between the two servers so that they may seamlessly interact to provide different components for a single starting web request. Essentially, all dynamic content is generated by the app server, this may reference images and other static content which will be served by the proxy server.

An example configuration is provided below:

# create an alias for static content to be served
Alias /static/ "/location/of/content"

RewriteEngine on

# static content should not be proxied to tomcat - just serve it up directly
# we re-write the URL and rely on the ALIAS defined above for '/static/'
# Essentially, we rewrite all requests for static content to use the
# hard wired alias which should point to content on disk
RewriteCond %{REQUEST_URI}  .*\.(jpg|png|gif|swf|flv|css|js|html)$
RewriteRule ^/(.+) /static/$1 [PT,L]

# assume our Java app is deployed to the 'app' context
RewriteRule ^/app/(.+) ajp://localhost:8009/app/$1 [P]

ProxyPass /app/ ajp://java-server:8009/app/ timeout=300
ProxyPassReverse /app/ ajp://java-server:8009/app/

<Proxy *>
AddDefaultCharset off
Order deny,allow
Allow from all
</Proxy>

This particular example uses mod_proxy_ajp to connect Apache HTTPD to a Java application server. However, this technique can be used to front many different backends using the Apache mod_proxy module including applications developed targeting Perl/mod_perl, IIS/ASP.NET or even ColdFusion.

The SpringModules cache extension provides the legwork for implementing this and Spring AOP the glue. One limitation of this implementation out of the box is that it is not Locale aware. Spring nicely provides a LocaleContextHolder to enable disparate components within the app stack to be locale aware so we needed to leverage this in order to ensure our method caching was able to not only cache method invocations with the same argument signature, but also within the same Locale (not something passed as arguments) .

Below is the meat of our simple solution, a LocaleAwareCacheKeyGenerator:

    public final Serializable generateKey(MethodInvocation methodInvocation) {

        Method method = methodInvocation.getMethod();

        HashCodeCalculator hashCodeCalculator = new HashCodeCalculator();
        hashCodeCalculator.append(method.hashCode());

        // now suck in the locale
        Locale locale = LocaleContextHolder.getLocale();

        // and use the bit we care about for hashcode generation
        if (considerLanguageOnly) {
            hashCodeCalculator.append(locale.getLanguage().hashCode());
        } else {
            hashCodeCalculator.append(locale.hashCode());
        }

        // now try and sort out things for the method args
        Object[] methodArguments = methodInvocation.getArguments();

        if (methodArguments != null) {
            int methodArgumentCount = methodArguments.length;

            for (int i = 0; i < methodArgumentCount; i++) {
                Object methodArgument = methodArguments[i];
                int hash = 0;

                if (generateArgumentHashCode) {
                    hash = Reflections.reflectionHashCode(methodArgument);
                } else {
                    hash = Objects.nullSafeHashCode(methodArgument);
                }

                hashCodeCalculator.append(hash);
            }
        }

        return new HashCodeCacheKey(hashCodeCalculator.getCheckSum(), hashCodeCalculator.getHashCode());
    }

This class simply adds the Locale to the code used in the HashCodeChacheKeyGenerator supplied with Spring. So now we can easily enable locale aware AOP configured method invocation caching transparently to our application code.

Nice!

P.S. An eclipse project with code/test samples is available

We had downloaded and installed the debian package grails_1.0.4-1_all.deb from the GRAILS site, configured the GRAILS_HOME environment variable for our build target, and attempted to build it. We got this error:
Grails ended with a non null return code: 127
After some digging, it turns out that the debian package installs GRAILS but puts the actual executable in /usr/bin instead of just a symlink from the GRAILS_HOME. The Maven GRAILS plugin appears to be calling grails using the GRAILS_HOME path. Making a symlink to the grails executable in the bin dir of the GRAILS_HOME fixed this problem (although the symlink really more correctly belongs in the /usr/bin dir).

Trying to build again we came across a different problem, where the GRAILS version was showing up as null, causing build errors. After some googling I found this useful post, and re-installed GRAILS with the updated grails_1.0.4-2_all.deb package, repeating the steps for the GRAILS executable. Now we’re all up and running in Hudson! I hope this might alleviate some pain for anyone trying to troubleshoot this problem.

Enter LiquiBase, an open source database-agnostic tool for tracking, managing and applying database changes. We recently worked on a small Grails project and decided to give this tool a try and I was duly impressed.

LiquiBase functionality is built around a main changelog.xml file containing changesets representing incremental database changes to be applied to a database. LiquiBase manages which changesets have been run through a DATABASECHANGELOG table it creates in each database.

We used the grails plugin which gave us most of the full LiquiBase functionality, albeit a little less than mature. There were a couple of gotchas and bugs with the plugin, but nothing we couldn’t work around.

LiquiBase buys us the ability to store database change in source control, easily sync databases in multi-environments in an automated and controlled fashion, tag database states upon iteration-end releases, auto-generate rollback sql to tagged states, diff databases, and much more.

LiquiBase-ifying Your Application

With your database in a known state, you can use the grails plugin to create your changelog.xml. First you need to install the grails LiquiBase plugin:
grails install-plugin liquibase
Once installed simply run this from the root of your grails app:
grails generate-changelog grails-app/changelog.xml

This will generate the changelog.xml from your development database (specified in DataSource.groovy) and write it to the path specified in the command. grails-app/changelog.xml is the default path where the grails plugin will be expecting the changelog to be. If you need to read from a different database environment like staging for example, just add the -Dgrails.env=staging option to the command.

To propagate this changelog to other databases (say, test), run
grails -Dgrails.env=test migrate. This runs all the sql necessary to upate the database to match the changelog. Any new changes from here on will be appended as new changesets, and can be migrated with the same command as above. You can also use the migrate-sql command instead if want to generate the sql and run it yourself. Most LiquiBase commands come in tuples: one to generate the sql for you and one to just run the sql directly against your database.

If you’re integrating LiquiBase mid-project and already have all your databases set up, you’ll need to run sql against them to update the LiquiBase DATABASECHANGELOG table to show all the changes as run. First, make sure that the databases are all in sync. To do this, you can use the handy LiquiBase diff tool. Unfortunately, the grails plugin for this diff tool is not very robust and will only diff your development database against your test database – the plugin has those two environments hard-coded. You can either mess with your DataSource.groovy db environments to do the diff setting the two dbs in question to dev and test temporarily, or install LiquiBase itself and run the diff passing the db parameters to the diff command. Using the grails plugin you would run:
grails db-diff which will spit to the screen any differences as changesets to be applied. Strangely, there is no documentation for this command in the LiquiBase Grails plugin page.

Once your database are in sync, run
grails changelog-sync-sql with the appropriate -Dgrails.env switch to generate the sql to update the DATABASECHANGELOG table, then run the sql in your database.

LiquiBase and Continuous Integration

We use Hudson at 2Paths for our CI, and have added the simple grails -Dgrails.env=test migrate command as part of our build process to migrate the test database upon every checkin. This ensures that the test db is always the most up-to-date.

Rolling LiquiBase into our Dev Process

We’ve adopted on a trial basis the following process for database change management as part of our agile software development, taking into consideration Grails development (which uses hibernate) which can auto-generates schemas based on domain objects:

1. Generate changelog from initial schema and commit to svn
2. Rollout schema to other databases by migrating from changelog
1. if the schema already exists and it needs “LiquiBase-ifying”, generate changelog-sync-sql and run it
3. Add every schema change via new changeset from hereon in by generating changelogs via grails db-diff
1. add / change domain objects in project
2. set hiberante ddl mode to update
3. start app and grails will automatically update the db
4. generate new changelogs using db-diff (against the test db)
5. append changelogs to changelog.xml
6. run grails changelog-sync-sql to generate the sql that will mark all these new changes as ran on your db, then run the sql
7. checkin changes. this will be applied to test
4. provide explicit rollback sql for any custom sql
5. tag each iteration end with a “tagDatabase” changelog in the changelog.xml

Gotchas

We ran into some gotchas with LiquiBase. One of the first things we started doing before fully understanding how LiquiBase worked was to update existing changesets when changing the schema. LiquiBase generates checksums for each changeset to ensure they don’t change, and altering existing changelogs will cause future migrations to fail. Even though LiquiBase gives you some tools to get around this, it’s generally a better practice to just add a new changeset for every database change. There is a good blog on the LiquiBase site explaining how to deal with changing changesets.

Another gotcha is that the rollback-sql command won’t magically generate rollback scripts for hand-coded sql (obviously!) You must to generate your own rollback sql for these custom sql tags, otherwise not only will you not get rollback sql for those particular changesets, but the rollback functionality won’t spit out sql for any other changeset either until you do.

Tagging was a little fussy as well. The command line grails tag can only tag a given state once – future tags will overwrite the earlier ones unless you’re at a different changeset. I found it better to add a tag changeset explicitly in the changelog.xml. The Grails plugin for tagging also seems to be problematic with spaces in the tag name. We decided to just use underscores as a convention.

We also ran across some broken functionality:

• The grails generate-changelog command generates some extraneous information that breaks sql for data type numeric(19,0) if that type has auto-increment on. It generates this in the changelog:
  <column autoIncrement=”true” name=”id” type=”numeric()(19,0)”> (notice the extra empty brackets). We needed to manually remove these to get it to work.
• The grails rollback-to-date-sql command writes to a file with the datestamp instead of to console like the other rollbacks do, even though the docs say it writes to STDOUT (rollback-count-sql also has issues, but rollback-sql for tags works just fine). The code for these plugin commands doesn’t seem to do the right thing with the args. Time permitting, we may provide a patch for this.
• db-doc generation doesn’t work with large amount of columns in a table because it uses the table description including column names for the html file name. Too many db columns make the file names too long.

Moving Forward

Although there’s a bit of grumbling amongst the developers here that using LiquiBase with the above process is a little onerous, it buys us a lot more certainty for knowing what state our databases are in, making sure changes are in source control, and knowing what updates have or haven’t been run. It’s a whole lot easier than hand-coding a bunch of sql, and easily accommodates the possibility of migrating to other DBMS’s in the future without having to re-write DBMS-specific sql. There’s even talk of trying to go a step or two further to automate even more of the process so a developer doesn’t have to think about generating changesets upon changing of a grails domain model. We’ll see how that goes.

Our XML and JSON rendering functionality leverages the XStream project, and provides an easy mechanism for pairing aliases and their respective Java objects.
The Spring Framework configuration in the webapp project sets the rendering information for each response type. Response types are used when specifying routes, which are the Tequila representation for URL request patterns.

JSON example:

<bean id="dataPointJsonRenderer" class="org.tequilaframework.core.renderer.JsonRenderer">
<property name="responseType" value="datapoint/json"/>
<property name="contentType" value="text/plain"/>
<property name="aliases">
<map value-type="java.lang.Class">
<entry>
<key><value>dataPoints</value></key>
<value>java.util.List</value>
</entry>
<entry>
<key><value>dataPoint</value></key>
<value>com.twopaths.example.domain.DataPoint</value>
</entry>
<entry>
<key><value>time</value></key>
<value>com.twopaths.example.domain.TimeDimension</value>
</entry>
<entry>
<key><value>spatial</value></key>
<value>com.twopaths.example.domain.SpatialDimension</value>
</entry>
</map>
</property>
<!-- Example omission -->
<!-- to omit a field, you must declare the parent class as the class value, and the key is the would-be rendered value -->
<!-- <property name="omissions">
<map value-type="java.lang.Class">
<entry>
<key><value>spatial</value></key>
<value>com.twopaths.example.domain.DataPoint</value>
</entry>
</map>
</property>
-->
</bean>

XML example:

<bean id="dataPointXmlRenderer" class="org.tequilaframework.core.renderer.XmlRenderer">
<property name="responseType" value="datapoint/xml"/>
<property name="contentType" value="text/plain"/>
<property name="aliases">
<map value-type="java.lang.Class">
<entry>
<key><value>dataPoints</value></key>
<value>java.util.List</value>
</entry>
<entry>
<key><value>dataPoint</value></key>
<value>com.twopaths.example.domain.DataPoint</value>
</entry>
<entry>
<key><value>time</value></key>
<value>com.twopaths.example.domain.TimeDimension</value>
</entry>
<entry>
<key><value>spatial</value></key>
<value>com.twopaths.example.domain.SpatialDimension</value>
</entry>
</map>
</property>
<!-- Example omission -->
<!-- to omit a field, you must declare the parent class as the class value, and the key is the would-be rendered value -->
<!-- <property name="omissions">
<map value-type="java.lang.Class">
<entry>
<key><value>spatial</value></key>
<value>com.twopaths.example.domain.DataPoint</value>
</entry>
</map>
</property>
-->
</bean>

The RDF renderer leverages the Jena project and Jenabean project. Jena supported RDF formats include RDF/XML, RDF/XML-ABBREV, N-TRIPLE, TURTLE, N3, N3-PP, N3-PLAIN, and N3-TRIPLE. Because of Jena, the Tequila RDF renderer gets these formats for free. The only trick with RDF in Tequila is that you will need to use annotations in the Java classes you wish to have RDF support in. If you’re using Hibernate, then you’ll likely already have RDF rendering configured. If no rdfType is specified, the default is RDF/XML.

RDF example:

<bean id="dataDescriptionRdfRenderer" class="org.tequilaframework.core.renderer.RdfRenderer">
<property name="responseType" value="application/rdf"/>
<property name="contentType" value="text/plain"/>
<property name="domain" value="http://www.2paths.com/example"/>
<property name="rdfType" value="TURTLE"/>
</bean>

With multiple rendering capabilities, the GET request to the webapp can alter the rendering output by including the flavor RPC verb descriptor to modify the return format. For example, the URL of

http://localhost:8080/example/app/data/example?flavor=application/rdf

would return an RDF rendered output, and the original response format may have been JSON, or another response format laid out in the configuration file.

One thing that we wanted to do was have themes which override certain resource bundle variables defined in an application wide, internationalized properties file. For example, our login greeting is “Welcome” with an appropriate i18n version for various locales. We also have certain themes based on types of user accounts. Some themes require their own customized login greeting and others will rely on using messages from the default MessageSource.

Our Spring application utilizes two classes for accessing resource bundles:

• org.springframework.context.support.ResourceBundleMessageSource
• org.springframework.ui.context.support.ResourceBundleThemeSource

Both these classes provide access to specific resource bundles but do not necessarily know about each other which means the operate in isolation. Ideally we wanted to have a hierarchy of resource bundles such that a resource message defined in the bundle accessed by ResourceBundleMessageSource could be overridden by a message defined in resource bundles accessed by ResourceBundleThemeSource. Whilst the ResourceBundleMessageSource class has a setParentResourceBundle(), there is no way to inject that to the ResourceBundleThemeSource via configuration.

As the resourceful (no pun intended) java developers that we are, we decided to create our own wrapper class for theme bundle resolution. This class will use the application’s ResourceBundleMessageSource definition as a parent for its own resource message source so that we can ensure that if a message cannot be resolved by key in the theme bundle, it will then defer to the parent resource bundle (which is the application level MessageSource) and try and look up the key there. This simple class is outlined below:

public class CustomResourceBundleThemeSource
        extends ResourceBundleThemeSource {

    private MessageSource parent;

    public void setParentMessageSource(MessageSource messageSource) {
        parent = messageSource;
    }

    @Override
    protected MessageSource createMessageSource(String basename) {
        ResourceBundleMessageSource messageSource = new ResourceBundleMessageSource();
        messageSource.setBasename(basename);
        messageSource.setUseCodeAsDefaultMessage(true);
        messageSource.setParentMessageSource(parent);
        return messageSource;
    }
}

With this class, we just need the following in our Spring config class:

    <bean id="messageSource"
          class="org.springframework.context.support.ResourceBundleMessageSource">
        <property name="basenames">
            <list>
                <value>bundles/messages</value>
            </list>
        </property>
    </bean>
    <bean id="themeSource"
          class="CustomResourceBundleThemeSource">
        <property name="basenamePrefix" value="/themes/" />
        <property name="parentMessageSource" ref="messageSource" />
    </bean>

Now whenever we need to resolve resource messages, we use the Spring specific theme tags:

    <spring:theme code="login.greeting" />

And if we have a definition in our theme resource bundle we see that else we see the default definition as specified in the underlying MessageSource.

This allows us to override resource bundle messages for various themes as required and removed quite a bit of ugly JSP/JSTL logic for resolving messages conditionally.

Job done.