Install memcached, which is simple if you use a Mac and have MacPorts installed.

$ sudo port install memcached

Get the memcached jar , the hibernate-memcached jar and the PostgreSQL JDBC driver jar, then  add them to the project.

$ cp memcached-2.3.1.jar hibernate-memcached-1.2.jar postgresql-8.4-701.jdbc3.jar <project_dir>/lib/

Configure memcached as the second-level cache by adding something like this to your DataSource.groovy.

hibernate {
    cache.use_second_level_cache = true
    cache.use_query_cache = true
    cache.provider_class = 'com.googlecode.hibernate.memcached.MemcachedCacheProvider'
    memcached {
        servers = "localhost:11211"
    }
}

I wrote a simple test to generate a whole bunch of domain objects and shove them in the database, then go back and fetch them all again. I set the test to run the insert/load loops in batches of 1, 10, 100, 1,000, 2,000, 4,000, 7,000, and 10,000 objects. I expected that inserting the objects would take a little longer because of the memcached overhead but that loading the objects would be faster because they could be loaded from memcached instead of going out to the database.

I tested three persistence backend configurations.

• HSQLDB, the default for new Grails projects and when running unit/integration tests. This is an in-memory database only, so data aren’t persisted permanently.
• PostgreSQL for a full-blown RDBMS setup. I set dbCreate to “create-drop” in DataSource.groovy so that each test started with an empty database.
• PostgreSQL with memcached enabled. I should point out that in this test I had both PostgreSQL and memcached running on the same machine (my laptop) as the application, so it’s really not taking advantage of memcached’s parallelism strengths.

My expectations held true for the inserts. The memcached case is a little slower but not that much.

Comparison of object insert times using HSQL, PostgreSQL, and memcached+PostgreSQL for persistence backend

When I tested the load times, I was surprised. It looked like memcached wasn’t speeding things up at all. I was also surprised to see that it took less time to load 10,000 objects from the data store than 7,000 objects. I ran the test a few times to rule out the possibility of a load spike from something else running at the same time, but I got similar results every time.

Comparison of object load times using HSQL, PostgreSQL, and memcached+PostgreSQL for persistence backend

I then tried to improve my test setup by running memcached on a second machine (my desktop, connected by gigabit ethernet to the laptop). Using the PostgreSQL + memcached configuration for each test this time, I looked at three more scenarios.

• PostgreSQL running locally on the laptop, memcached running locally on the laptop.
• PostgreSQL running locally on the laptop, memcached running remotely on the desktop.
• PostgreSQL running locally on the laptop, memcached running on both the laptop and the desktop.

There’s some overhead in going out to the network to retrieve data instead of fetching it from a process running locally, so I expected the memcached local instance to be faster than the memcached remote instance. I expected the third test, with two instances of memcached running in parallel, to be fastest because it splits the load between the two instances and frees up some CPU time on the laptop. Things didn’t play out that way, though.

Comparison of object insert times for various memcache setups

Comparison of object load times for various memcache setups

I have to be honest that these were not great tests. For one, I should really have run those insert/load loops in parallel instead of sequentially. I’m guessing that the test spent a lot of time stalled waiting for data when it could have been sending out more requests. I would also like to test with a large group of memcached servers and a remote database, perhaps even a cluster of app servers with a load balancer to try the full meal deal. The memcached FAQ explains that running everything on one machine will not show off memcached’s scalability.

However, this gave me the opportunity to get my hands dirty with memcached and learn a few things.

• Integrating memcached is so simple and the performance penalty of just running it locally during development is so small that there’s no reason not to enable it.
• It’s dead simple to install and set up. We could , for example, easily install it on a bunch of test servers and run multiple instances on different ports – one for each developer (to prevent us from clobbering each other’s data).
• Measuring performance and scalability realistically is tricky. It’s one thing to hand wave and say that running multiple instances of a database or a cache will speed things up, but it takes some work to set up a realistic simulation of high traffic application performance.
• Adding memcached doesn’t automatically turn performance up to 11. I need to learn more about what’s going on under the hood with my domain objects and how they’re being persisted through memcached.

I look forward to using memcached more in the future and learning how to really take advantage of it.

Update (2009-07-24): Ray pointed out in the comments that I should have enabled caching in my domain objects under test. I added “static mapping = { cache true }” to my domain class definition and re-ran the tests to compare the effect under 4 scenarios.

• no domain object caching (as before)
• domain object caching with memcached running locally
• domain object caching with memcached running remotely
• domain object caching with memcached running both locally and remotely

Not much difference with the inserts.

Comparison of object insert times with domain object caching enabled and disabled

However, the load times are slightly better with memcached enabled during heavy activity! The advantage disappears when I add the overhead of traversing the network, but it’s a start. Thanks for the pointer, Ray!

Comparison of object load times with domain object caching enabled and disabled

Over the years I have used a few CI servers and tools – most notably, CruiseControl (Java and .NET) and Continuum (from the Maven crew). These systems, whilst useful, cam e with a few annoyances, most notably their slight fiddliness in getting up and running. Whislt they worked, it seemed that there were always tweaks required here and there to get things running smoothly and when things went wrong they we not that easy to correct.

After chatting with a few friends, I thought I would give Hudson a try as it seems like a new contender for CI server of the month. Spurned on by glowing feedback from said friends I though I would see how easy it would be to try and aggregate our disparate CI servers (For Java and .NET) into a single hudson installation – something advertised to be easy on the tin.

Here are the steps I followed:
- download WAR, runit
- config master/slave
- config maven app
- config .NET app
- force some builds
- too easy

A simple distributed build system that works cross platform sounded too good to be true, however it seems that hudson is well on the way to being just that. I couldn’t believe how easy it was to get a distributed build/CI system up and running and building/testing our Java and .NET apps with all reports and config managed centrally. I especially like the simple reports on build trends and the weather paradigm used for displaying build stability.

After this simple proof of concept, we moved the setup to a cluster of VMWare instances (Linux and windows) so we can now build multi-platform apps and manage things centrally for all projects. This is now our main CI platform and seems to be working well.

Hurray for progress!

The theory:
- before changing the version of the project, create the future version’s self-dependency jars.
- to do that, we need to change the antrun scripts to create new jars that will be used in the next project version. Commit to subversion and run continuum build.
- then we can change the pom to the next project version and when run in continuum it uses the jar created by the last version of the project. Maven2 then thinks it has the right jar, so the dependencies pass. The antrun scripts then fire off, and create the correct new jars that will be included in the project build.

The Implementation:
- if we’re on version 32, then change the scripts to create jars for version 33, and leave the project as version 32. Commit to subversion, then in continuum run build. Now we have project 32 that created jar 33 for us.
- The version for the project can finally be updated to 33, committed to subversion, and run successfully in continuum builds.

Now, for those of you not familiar with how to get antrun working in maven2, you need to add in the pom.xml in the plugins section:


<plugin>
<artifactId>maven-antrun-plugin</artifactId>
<executions>
<execution>
<phase>generate-resources</phase>
<configuration>
<tasks>
<echo message="To skip install modify pom to not include maven-antrun-plugin"/>
<exec
dir="${basedir}"
executable="${basedir}/installLib.sh"
failonerror="true" />
<exec
dir="${basedir}"
executable="${basedir}/installXmlbeans.sh"
failonerror="true" />
</tasks>
</configuration>
<goals>
<goal>run</goal>
</goals>
</execution>
</executions>
</plugin>


For the installXmlbeans script we have something like the following: (directory xsd contains those files to be converted to xml beans)

VERSION=33
touch src/main/xsd/*
mvn -P installer xmlbeans:xmlbeans
mvn -P installer install:install-file -Dfile=target/projectname-xmlbeans.jar -DgroupId=com.twopaths.projectname -DartifactId=projectname-xmlbeans -Dversion=$VERSION -Dpackaging=jar


Hopefully this saves other people from the same continuum self-dependency versioning hell.

-Garrett