Install memcached, which is simple if you use a Mac and have MacPorts installed.

$ sudo port install memcached

Get the memcached jar , the hibernate-memcached jar and the PostgreSQL JDBC driver jar, then  add them to the project.

$ cp memcached-2.3.1.jar hibernate-memcached-1.2.jar postgresql-8.4-701.jdbc3.jar <project_dir>/lib/

Configure memcached as the second-level cache by adding something like this to your DataSource.groovy.

hibernate {
    cache.use_second_level_cache = true
    cache.use_query_cache = true
    cache.provider_class = 'com.googlecode.hibernate.memcached.MemcachedCacheProvider'
    memcached {
        servers = "localhost:11211"
    }
}

I wrote a simple test to generate a whole bunch of domain objects and shove them in the database, then go back and fetch them all again. I set the test to run the insert/load loops in batches of 1, 10, 100, 1,000, 2,000, 4,000, 7,000, and 10,000 objects. I expected that inserting the objects would take a little longer because of the memcached overhead but that loading the objects would be faster because they could be loaded from memcached instead of going out to the database.

I tested three persistence backend configurations.

• HSQLDB, the default for new Grails projects and when running unit/integration tests. This is an in-memory database only, so data aren’t persisted permanently.
• PostgreSQL for a full-blown RDBMS setup. I set dbCreate to “create-drop” in DataSource.groovy so that each test started with an empty database.
• PostgreSQL with memcached enabled. I should point out that in this test I had both PostgreSQL and memcached running on the same machine (my laptop) as the application, so it’s really not taking advantage of memcached’s parallelism strengths.

My expectations held true for the inserts. The memcached case is a little slower but not that much.

Comparison of object insert times using HSQL, PostgreSQL, and memcached+PostgreSQL for persistence backend

When I tested the load times, I was surprised. It looked like memcached wasn’t speeding things up at all. I was also surprised to see that it took less time to load 10,000 objects from the data store than 7,000 objects. I ran the test a few times to rule out the possibility of a load spike from something else running at the same time, but I got similar results every time.

Comparison of object load times using HSQL, PostgreSQL, and memcached+PostgreSQL for persistence backend

I then tried to improve my test setup by running memcached on a second machine (my desktop, connected by gigabit ethernet to the laptop). Using the PostgreSQL + memcached configuration for each test this time, I looked at three more scenarios.

• PostgreSQL running locally on the laptop, memcached running locally on the laptop.
• PostgreSQL running locally on the laptop, memcached running remotely on the desktop.
• PostgreSQL running locally on the laptop, memcached running on both the laptop and the desktop.

There’s some overhead in going out to the network to retrieve data instead of fetching it from a process running locally, so I expected the memcached local instance to be faster than the memcached remote instance. I expected the third test, with two instances of memcached running in parallel, to be fastest because it splits the load between the two instances and frees up some CPU time on the laptop. Things didn’t play out that way, though.

Comparison of object insert times for various memcache setups

Comparison of object load times for various memcache setups

I have to be honest that these were not great tests. For one, I should really have run those insert/load loops in parallel instead of sequentially. I’m guessing that the test spent a lot of time stalled waiting for data when it could have been sending out more requests. I would also like to test with a large group of memcached servers and a remote database, perhaps even a cluster of app servers with a load balancer to try the full meal deal. The memcached FAQ explains that running everything on one machine will not show off memcached’s scalability.

However, this gave me the opportunity to get my hands dirty with memcached and learn a few things.

• Integrating memcached is so simple and the performance penalty of just running it locally during development is so small that there’s no reason not to enable it.
• It’s dead simple to install and set up. We could , for example, easily install it on a bunch of test servers and run multiple instances on different ports – one for each developer (to prevent us from clobbering each other’s data).
• Measuring performance and scalability realistically is tricky. It’s one thing to hand wave and say that running multiple instances of a database or a cache will speed things up, but it takes some work to set up a realistic simulation of high traffic application performance.
• Adding memcached doesn’t automatically turn performance up to 11. I need to learn more about what’s going on under the hood with my domain objects and how they’re being persisted through memcached.

I look forward to using memcached more in the future and learning how to really take advantage of it.

Update (2009-07-24): Ray pointed out in the comments that I should have enabled caching in my domain objects under test. I added “static mapping = { cache true }” to my domain class definition and re-ran the tests to compare the effect under 4 scenarios.

• no domain object caching (as before)
• domain object caching with memcached running locally
• domain object caching with memcached running remotely
• domain object caching with memcached running both locally and remotely

Not much difference with the inserts.

Comparison of object insert times with domain object caching enabled and disabled

However, the load times are slightly better with memcached enabled during heavy activity! The advantage disappears when I add the overhead of traversing the network, but it’s a start. Thanks for the pointer, Ray!

Comparison of object load times with domain object caching enabled and disabled

AWS Import/Export

We haven’t analyzed the pricing but it is one more nail into the “let’s shut the dev cage down and go all cloud” debate.

Largely for my own reference, here are some Unix shell tricks that have come up lately. If you’re using Linux, Mac OS X, or other Unix-like systems, these might be handy.

Redirecting output

When a process starts, it opens three file descriptors: file descriptor 0 for standard input (STDIN), file descriptor 1 for standard output (STDOUT), and file descriptor 2 for standard error (STDERR). By default, the standard output and error just print to the terminal (so you see them on the screen), but you can redirect them to a file (e.g. for logging or just to cut down on gibberish that you’re not going to read anyway) by using “>” to create/overwrite the file or “>>” to create/append the file.

I’ve used “find /” as the command in these examples because it takes a while to run and tends to produce plenty of output on STDOUT and STDERR.


# Redirect STDOUT to a file, but leave STDERR to print on screen.
# If the file exists, overwrite it.
$ find / > /some/file.txt
# Redirect STDOUT to a file, but leave STDERR to print on screen.
# If the file exists, append to the end of it. If it doesn't exist, create it.
$ find / >> /some/file.txt
# Redirect STDOUT to the null device, which is a special file that silently discards everything.
# Use this if you want to ignore STDOUT but keep an eye on STDERR.
$ find / > /dev/null
# Redirect both STDOUT and STDERR to the same file.
$ find / > /some/file.txt 2>&1
# Redirect STDOUT and STDERR to different files.
# The numbers refer to the file descriptors mentioned above.
$ find / > /path/to/log.out 2>/path/to/log.err
# ... or, more explicitly:
$ find / 1>/path/to/log.out 2>/path/to/log.err


Background processes

Often you’ll want to start something running and keep working on something else. This is frequently done by redirecting the output to a file (or the null device) by starting it in the background with the “&” keyword.


# Start a process in the background.
$ find / &
# Start a process in the background and ignore all output.
$ find / > /dev/null 2>&1


To bring a background process back to the foreground, for example to stop it with ctrl+c or to provide input if it asks a question, use the “fg” command.

If you start several background processes, use the “jobs” command to list them. That will print the job number (different from the process ID) that you can use to bring a specific one back to the foreground with e.g. “fg 3″ to bring back job 3.


$ find / > /dev/null 2>&1 &
$ cp file.dat /mnt/other_drive &
$ sleep 600 &
$ jobs
[1] Running find / > /dev/null 2>&1 &
[2]- Running cp file.dat /mnt/other_drive &
[3]+ Running sleep 600 &
$ fg 3
sleep 600


Capturing process ID for simple start/stop scripts

Putting those together, here’s a simple start/stop script that starts a command in the background and stores all output in a log file. It also uses the “$!” variable to record the process ID so it can kill the process later. Call the file “startstop.sh” and just run “startstop.sh start” to start it and “startstop.sh stop” to stop it.


#!/bin/sh
PID_FILE="test.pid"
LOG_FILE="test.log"
CMD="find /"
if [ "$1" == "start" ]; then
echo "starting!"
$CMD > $LOG_FILE 2>&1 &
echo $! > $PID_FILE;
fi
if [ "$1" == "stop" ]; then
echo "stopping!"
kill `cat $PID_FILE`;
rm $PID_FILE;
fi


We had downloaded and installed the debian package grails_1.0.4-1_all.deb from the GRAILS site, configured the GRAILS_HOME environment variable for our build target, and attempted to build it. We got this error:
Grails ended with a non null return code: 127
After some digging, it turns out that the debian package installs GRAILS but puts the actual executable in /usr/bin instead of just a symlink from the GRAILS_HOME. The Maven GRAILS plugin appears to be calling grails using the GRAILS_HOME path. Making a symlink to the grails executable in the bin dir of the GRAILS_HOME fixed this problem (although the symlink really more correctly belongs in the /usr/bin dir).

Trying to build again we came across a different problem, where the GRAILS version was showing up as null, causing build errors. After some googling I found this useful post, and re-installed GRAILS with the updated grails_1.0.4-2_all.deb package, repeating the steps for the GRAILS executable. Now we’re all up and running in Hudson! I hope this might alleviate some pain for anyone trying to troubleshoot this problem.

Enter LiquiBase, an open source database-agnostic tool for tracking, managing and applying database changes. We recently worked on a small Grails project and decided to give this tool a try and I was duly impressed.

LiquiBase functionality is built around a main changelog.xml file containing changesets representing incremental database changes to be applied to a database. LiquiBase manages which changesets have been run through a DATABASECHANGELOG table it creates in each database.

We used the grails plugin which gave us most of the full LiquiBase functionality, albeit a little less than mature. There were a couple of gotchas and bugs with the plugin, but nothing we couldn’t work around.

LiquiBase buys us the ability to store database change in source control, easily sync databases in multi-environments in an automated and controlled fashion, tag database states upon iteration-end releases, auto-generate rollback sql to tagged states, diff databases, and much more.

LiquiBase-ifying Your Application

With your database in a known state, you can use the grails plugin to create your changelog.xml. First you need to install the grails LiquiBase plugin:
grails install-plugin liquibase
Once installed simply run this from the root of your grails app:
grails generate-changelog grails-app/changelog.xml

This will generate the changelog.xml from your development database (specified in DataSource.groovy) and write it to the path specified in the command. grails-app/changelog.xml is the default path where the grails plugin will be expecting the changelog to be. If you need to read from a different database environment like staging for example, just add the -Dgrails.env=staging option to the command.

To propagate this changelog to other databases (say, test), run
grails -Dgrails.env=test migrate. This runs all the sql necessary to upate the database to match the changelog. Any new changes from here on will be appended as new changesets, and can be migrated with the same command as above. You can also use the migrate-sql command instead if want to generate the sql and run it yourself. Most LiquiBase commands come in tuples: one to generate the sql for you and one to just run the sql directly against your database.

If you’re integrating LiquiBase mid-project and already have all your databases set up, you’ll need to run sql against them to update the LiquiBase DATABASECHANGELOG table to show all the changes as run. First, make sure that the databases are all in sync. To do this, you can use the handy LiquiBase diff tool. Unfortunately, the grails plugin for this diff tool is not very robust and will only diff your development database against your test database – the plugin has those two environments hard-coded. You can either mess with your DataSource.groovy db environments to do the diff setting the two dbs in question to dev and test temporarily, or install LiquiBase itself and run the diff passing the db parameters to the diff command. Using the grails plugin you would run:
grails db-diff which will spit to the screen any differences as changesets to be applied. Strangely, there is no documentation for this command in the LiquiBase Grails plugin page.

Once your database are in sync, run
grails changelog-sync-sql with the appropriate -Dgrails.env switch to generate the sql to update the DATABASECHANGELOG table, then run the sql in your database.

LiquiBase and Continuous Integration

We use Hudson at 2Paths for our CI, and have added the simple grails -Dgrails.env=test migrate command as part of our build process to migrate the test database upon every checkin. This ensures that the test db is always the most up-to-date.

Rolling LiquiBase into our Dev Process

We’ve adopted on a trial basis the following process for database change management as part of our agile software development, taking into consideration Grails development (which uses hibernate) which can auto-generates schemas based on domain objects:

1. Generate changelog from initial schema and commit to svn
2. Rollout schema to other databases by migrating from changelog
1. if the schema already exists and it needs “LiquiBase-ifying”, generate changelog-sync-sql and run it
3. Add every schema change via new changeset from hereon in by generating changelogs via grails db-diff
1. add / change domain objects in project
2. set hiberante ddl mode to update
3. start app and grails will automatically update the db
4. generate new changelogs using db-diff (against the test db)
5. append changelogs to changelog.xml
6. run grails changelog-sync-sql to generate the sql that will mark all these new changes as ran on your db, then run the sql
7. checkin changes. this will be applied to test
4. provide explicit rollback sql for any custom sql
5. tag each iteration end with a “tagDatabase” changelog in the changelog.xml

Gotchas

We ran into some gotchas with LiquiBase. One of the first things we started doing before fully understanding how LiquiBase worked was to update existing changesets when changing the schema. LiquiBase generates checksums for each changeset to ensure they don’t change, and altering existing changelogs will cause future migrations to fail. Even though LiquiBase gives you some tools to get around this, it’s generally a better practice to just add a new changeset for every database change. There is a good blog on the LiquiBase site explaining how to deal with changing changesets.

Another gotcha is that the rollback-sql command won’t magically generate rollback scripts for hand-coded sql (obviously!) You must to generate your own rollback sql for these custom sql tags, otherwise not only will you not get rollback sql for those particular changesets, but the rollback functionality won’t spit out sql for any other changeset either until you do.

Tagging was a little fussy as well. The command line grails tag can only tag a given state once – future tags will overwrite the earlier ones unless you’re at a different changeset. I found it better to add a tag changeset explicitly in the changelog.xml. The Grails plugin for tagging also seems to be problematic with spaces in the tag name. We decided to just use underscores as a convention.

We also ran across some broken functionality:

• The grails generate-changelog command generates some extraneous information that breaks sql for data type numeric(19,0) if that type has auto-increment on. It generates this in the changelog:
  <column autoIncrement=”true” name=”id” type=”numeric()(19,0)”> (notice the extra empty brackets). We needed to manually remove these to get it to work.
• The grails rollback-to-date-sql command writes to a file with the datestamp instead of to console like the other rollbacks do, even though the docs say it writes to STDOUT (rollback-count-sql also has issues, but rollback-sql for tags works just fine). The code for these plugin commands doesn’t seem to do the right thing with the args. Time permitting, we may provide a patch for this.
• db-doc generation doesn’t work with large amount of columns in a table because it uses the table description including column names for the html file name. Too many db columns make the file names too long.

Moving Forward

Although there’s a bit of grumbling amongst the developers here that using LiquiBase with the above process is a little onerous, it buys us a lot more certainty for knowing what state our databases are in, making sure changes are in source control, and knowing what updates have or haven’t been run. It’s a whole lot easier than hand-coding a bunch of sql, and easily accommodates the possibility of migrating to other DBMS’s in the future without having to re-write DBMS-specific sql. There’s even talk of trying to go a step or two further to automate even more of the process so a developer doesn’t have to think about generating changesets upon changing of a grails domain model. We’ll see how that goes.

Also note that I’ve put together a 2Paths preferences file that remaps the home and end keys in eclipse, and adds in the 2Paths code formatting templates. To import, once you’ve got the new eclipse up and running, select import and choose preferences.

Download Eclipse 2Paths Edition v2 (155.6MB)

Fix for home and end keys
If you’re anything like me, you are used to having your home and end keys go to the start of the line of text and the end of the line, respectively. The way OS X does it drives me insane, and I’ve finally found a solution that even fixes it in firefox: KeyFixer.

KeyFixer is a free utility that remaps those keys, there are 2 versions, the standard one that remaps it in OS X and most apps, and KeyFixer for Firefox which is self explanatory. I recommend installing them both if you want your home and end keys back.

Download KeyFixer (60KB)

Download KeyFixer for Firefox (80KB)