Install memcached, which is simple if you use a Mac and have MacPorts installed.

$ sudo port install memcached

Get the memcached jar , the hibernate-memcached jar and the PostgreSQL JDBC driver jar, then  add them to the project.

$ cp memcached-2.3.1.jar hibernate-memcached-1.2.jar postgresql-8.4-701.jdbc3.jar <project_dir>/lib/

Configure memcached as the second-level cache by adding something like this to your DataSource.groovy.

hibernate {
    cache.use_second_level_cache = true
    cache.use_query_cache = true
    cache.provider_class = 'com.googlecode.hibernate.memcached.MemcachedCacheProvider'
    memcached {
        servers = "localhost:11211"
    }
}

I wrote a simple test to generate a whole bunch of domain objects and shove them in the database, then go back and fetch them all again. I set the test to run the insert/load loops in batches of 1, 10, 100, 1,000, 2,000, 4,000, 7,000, and 10,000 objects. I expected that inserting the objects would take a little longer because of the memcached overhead but that loading the objects would be faster because they could be loaded from memcached instead of going out to the database.

I tested three persistence backend configurations.

• HSQLDB, the default for new Grails projects and when running unit/integration tests. This is an in-memory database only, so data aren’t persisted permanently.
• PostgreSQL for a full-blown RDBMS setup. I set dbCreate to “create-drop” in DataSource.groovy so that each test started with an empty database.
• PostgreSQL with memcached enabled. I should point out that in this test I had both PostgreSQL and memcached running on the same machine (my laptop) as the application, so it’s really not taking advantage of memcached’s parallelism strengths.

My expectations held true for the inserts. The memcached case is a little slower but not that much.

Comparison of object insert times using HSQL, PostgreSQL, and memcached+PostgreSQL for persistence backend

When I tested the load times, I was surprised. It looked like memcached wasn’t speeding things up at all. I was also surprised to see that it took less time to load 10,000 objects from the data store than 7,000 objects. I ran the test a few times to rule out the possibility of a load spike from something else running at the same time, but I got similar results every time.

Comparison of object load times using HSQL, PostgreSQL, and memcached+PostgreSQL for persistence backend

I then tried to improve my test setup by running memcached on a second machine (my desktop, connected by gigabit ethernet to the laptop). Using the PostgreSQL + memcached configuration for each test this time, I looked at three more scenarios.

• PostgreSQL running locally on the laptop, memcached running locally on the laptop.
• PostgreSQL running locally on the laptop, memcached running remotely on the desktop.
• PostgreSQL running locally on the laptop, memcached running on both the laptop and the desktop.

There’s some overhead in going out to the network to retrieve data instead of fetching it from a process running locally, so I expected the memcached local instance to be faster than the memcached remote instance. I expected the third test, with two instances of memcached running in parallel, to be fastest because it splits the load between the two instances and frees up some CPU time on the laptop. Things didn’t play out that way, though.

Comparison of object insert times for various memcache setups

Comparison of object load times for various memcache setups

I have to be honest that these were not great tests. For one, I should really have run those insert/load loops in parallel instead of sequentially. I’m guessing that the test spent a lot of time stalled waiting for data when it could have been sending out more requests. I would also like to test with a large group of memcached servers and a remote database, perhaps even a cluster of app servers with a load balancer to try the full meal deal. The memcached FAQ explains that running everything on one machine will not show off memcached’s scalability.

However, this gave me the opportunity to get my hands dirty with memcached and learn a few things.

• Integrating memcached is so simple and the performance penalty of just running it locally during development is so small that there’s no reason not to enable it.
• It’s dead simple to install and set up. We could , for example, easily install it on a bunch of test servers and run multiple instances on different ports – one for each developer (to prevent us from clobbering each other’s data).
• Measuring performance and scalability realistically is tricky. It’s one thing to hand wave and say that running multiple instances of a database or a cache will speed things up, but it takes some work to set up a realistic simulation of high traffic application performance.
• Adding memcached doesn’t automatically turn performance up to 11. I need to learn more about what’s going on under the hood with my domain objects and how they’re being persisted through memcached.

I look forward to using memcached more in the future and learning how to really take advantage of it.

Update (2009-07-24): Ray pointed out in the comments that I should have enabled caching in my domain objects under test. I added “static mapping = { cache true }” to my domain class definition and re-ran the tests to compare the effect under 4 scenarios.

• no domain object caching (as before)
• domain object caching with memcached running locally
• domain object caching with memcached running remotely
• domain object caching with memcached running both locally and remotely

Not much difference with the inserts.

Comparison of object insert times with domain object caching enabled and disabled

However, the load times are slightly better with memcached enabled during heavy activity! The advantage disappears when I add the overhead of traversing the network, but it’s a start. Thanks for the pointer, Ray!

Comparison of object load times with domain object caching enabled and disabled

PROS

Probably the number one mentioned feature by anyone talking about CouchDB is that it’s schema-free. That means that each document is its own structure, that everything is based on a key-value pairing, where the value is typically a document, but can also be a base type. This is pretty cool when dealing with describing real world objects, as one can be more descriptive and hierarchical in storing the data. Also, updating your application’s domain model is much like evolution, as soon as you need something added to the model, just add it.

CouchDB’s RESTful HTTP service implementation is great for the many applications being created for today’s web. Send Javascript queries/requests to the service, and get JSON formatted data in return. This is great, since many front-end technologies can decipher JSON, and the format itself is even human readable if need be.

There is support for interacting with CouchDB data offline. One can access and edit data while not connected, and when back online can merge and update changes made while offline. This kind of support out of the box makes CouchDB a prime candidate for  web based applications like blogs, wikis, bug reporting, CRM, etc.

Another quality of CouchDB is its MapReduce implementation for handling large data sets. I’m going to redirect you to the author’s notes for this one: MapReduce and Google’s definition.

CONS

Transaction management is non-existent. Relational database managements systems (RDBMS) provide mechanisms for things like transaction management, particularly unique constraints and locking. CouchDB is based on being ‘eventually consistent’ and ‘available’ (See the CAP Theorem section for more), which although great for many things, doesn’t lend itself to transaction management.

Data security is also a concern. A somewhat old article describes this issue quite well: Planned Security Model For CouchDB. The CouchDB team has been working on security and validation but are still early in their development.

To a less important degree, but still worth mentioning, is that personally I really don’t find the out-of-the-box UI for CouchDB to be all that useful. It feels clunky to navigate your key/document pairs, and too feature-light for managing the database.

Conclusion

If you’re looking to create a document based web application, then this may be a worthwhile technology to use. CouchDB has a number of very impressive features. However, for now at least, I believe I’ll be staying off the couch.

Aside:

When testing out CouchDB I went through the process of manually installing the application. I had to resolve the dependencies, and deal with compilation errors on my own. I kind of wish I’d found this site earlier. Follow that link if you’re a MacOSX user and want an installer for CouchDB.

For me, this was pretty much the  only useful tutorial on getting started with CouchDB: Programming CouchDB with Javascript.

Resources:

• http://couchdb.apache.org/
• http://books.couchdb.org/relax/
• http://www.eflorenzano.com/blog/tag/couchdb/
• http://labs.google.com/papers/mapreduce.html
• http://damienkatz.net/2008/02/incremental_map.html
• http://jan.prima.de/~jan/plok/archives/142-CouchDBX-Revival.html
• http://www.automatthew.com/2008/01/planned-security-model-for-couchdb.html
• http://jchrisa.net/drl/_design/sofa/_show/post/couchdb_edge__security_and_vali
• http://jan.prima.de/~jan/plok/archives/108-Programming-CouchDB-with-Javascript.html
• http://www.infoq.com/CouchDB
• http://www.infoq.com/news/2007/11/the-rdbms-is-not-enough
• http://www.infoq.com/news/2008/11/Database-Martin-Fowler
• http://code.google.com/p/couchdb-lounge/wiki/SettingUpTwoCouchInstances

Enter LiquiBase, an open source database-agnostic tool for tracking, managing and applying database changes. We recently worked on a small Grails project and decided to give this tool a try and I was duly impressed.

LiquiBase functionality is built around a main changelog.xml file containing changesets representing incremental database changes to be applied to a database. LiquiBase manages which changesets have been run through a DATABASECHANGELOG table it creates in each database.

We used the grails plugin which gave us most of the full LiquiBase functionality, albeit a little less than mature. There were a couple of gotchas and bugs with the plugin, but nothing we couldn’t work around.

LiquiBase buys us the ability to store database change in source control, easily sync databases in multi-environments in an automated and controlled fashion, tag database states upon iteration-end releases, auto-generate rollback sql to tagged states, diff databases, and much more.

LiquiBase-ifying Your Application

With your database in a known state, you can use the grails plugin to create your changelog.xml. First you need to install the grails LiquiBase plugin:
grails install-plugin liquibase
Once installed simply run this from the root of your grails app:
grails generate-changelog grails-app/changelog.xml

This will generate the changelog.xml from your development database (specified in DataSource.groovy) and write it to the path specified in the command. grails-app/changelog.xml is the default path where the grails plugin will be expecting the changelog to be. If you need to read from a different database environment like staging for example, just add the -Dgrails.env=staging option to the command.

To propagate this changelog to other databases (say, test), run
grails -Dgrails.env=test migrate. This runs all the sql necessary to upate the database to match the changelog. Any new changes from here on will be appended as new changesets, and can be migrated with the same command as above. You can also use the migrate-sql command instead if want to generate the sql and run it yourself. Most LiquiBase commands come in tuples: one to generate the sql for you and one to just run the sql directly against your database.

If you’re integrating LiquiBase mid-project and already have all your databases set up, you’ll need to run sql against them to update the LiquiBase DATABASECHANGELOG table to show all the changes as run. First, make sure that the databases are all in sync. To do this, you can use the handy LiquiBase diff tool. Unfortunately, the grails plugin for this diff tool is not very robust and will only diff your development database against your test database – the plugin has those two environments hard-coded. You can either mess with your DataSource.groovy db environments to do the diff setting the two dbs in question to dev and test temporarily, or install LiquiBase itself and run the diff passing the db parameters to the diff command. Using the grails plugin you would run:
grails db-diff which will spit to the screen any differences as changesets to be applied. Strangely, there is no documentation for this command in the LiquiBase Grails plugin page.

Once your database are in sync, run
grails changelog-sync-sql with the appropriate -Dgrails.env switch to generate the sql to update the DATABASECHANGELOG table, then run the sql in your database.

LiquiBase and Continuous Integration

We use Hudson at 2Paths for our CI, and have added the simple grails -Dgrails.env=test migrate command as part of our build process to migrate the test database upon every checkin. This ensures that the test db is always the most up-to-date.

Rolling LiquiBase into our Dev Process

We’ve adopted on a trial basis the following process for database change management as part of our agile software development, taking into consideration Grails development (which uses hibernate) which can auto-generates schemas based on domain objects:

1. Generate changelog from initial schema and commit to svn
2. Rollout schema to other databases by migrating from changelog
1. if the schema already exists and it needs “LiquiBase-ifying”, generate changelog-sync-sql and run it
3. Add every schema change via new changeset from hereon in by generating changelogs via grails db-diff
1. add / change domain objects in project
2. set hiberante ddl mode to update
3. start app and grails will automatically update the db
4. generate new changelogs using db-diff (against the test db)
5. append changelogs to changelog.xml
6. run grails changelog-sync-sql to generate the sql that will mark all these new changes as ran on your db, then run the sql
7. checkin changes. this will be applied to test
4. provide explicit rollback sql for any custom sql
5. tag each iteration end with a “tagDatabase” changelog in the changelog.xml

Gotchas

We ran into some gotchas with LiquiBase. One of the first things we started doing before fully understanding how LiquiBase worked was to update existing changesets when changing the schema. LiquiBase generates checksums for each changeset to ensure they don’t change, and altering existing changelogs will cause future migrations to fail. Even though LiquiBase gives you some tools to get around this, it’s generally a better practice to just add a new changeset for every database change. There is a good blog on the LiquiBase site explaining how to deal with changing changesets.

Another gotcha is that the rollback-sql command won’t magically generate rollback scripts for hand-coded sql (obviously!) You must to generate your own rollback sql for these custom sql tags, otherwise not only will you not get rollback sql for those particular changesets, but the rollback functionality won’t spit out sql for any other changeset either until you do.

Tagging was a little fussy as well. The command line grails tag can only tag a given state once – future tags will overwrite the earlier ones unless you’re at a different changeset. I found it better to add a tag changeset explicitly in the changelog.xml. The Grails plugin for tagging also seems to be problematic with spaces in the tag name. We decided to just use underscores as a convention.

We also ran across some broken functionality:

• The grails generate-changelog command generates some extraneous information that breaks sql for data type numeric(19,0) if that type has auto-increment on. It generates this in the changelog:
  <column autoIncrement=”true” name=”id” type=”numeric()(19,0)”> (notice the extra empty brackets). We needed to manually remove these to get it to work.
• The grails rollback-to-date-sql command writes to a file with the datestamp instead of to console like the other rollbacks do, even though the docs say it writes to STDOUT (rollback-count-sql also has issues, but rollback-sql for tags works just fine). The code for these plugin commands doesn’t seem to do the right thing with the args. Time permitting, we may provide a patch for this.
• db-doc generation doesn’t work with large amount of columns in a table because it uses the table description including column names for the html file name. Too many db columns make the file names too long.

Moving Forward

Although there’s a bit of grumbling amongst the developers here that using LiquiBase with the above process is a little onerous, it buys us a lot more certainty for knowing what state our databases are in, making sure changes are in source control, and knowing what updates have or haven’t been run. It’s a whole lot easier than hand-coding a bunch of sql, and easily accommodates the possibility of migrating to other DBMS’s in the future without having to re-write DBMS-specific sql. There’s even talk of trying to go a step or two further to automate even more of the process so a developer doesn’t have to think about generating changesets upon changing of a grails domain model. We’ll see how that goes.

After some discussion of whether we should start development on the database or application first, we decided to tackle the database first. We had started out by re-designing the database to allow for some sorely-needed normalization of some relationships. We came up with a new normalized schema through a mixture of requirements gathering with the client and reverse-engineering the legacy database. We focussed on areas where we could generalize and abstract out relationships and left the details for last. With our new schema, we then worked on migrating the legacy data into the new schema to ensure everything was accommodated.

Using our new schema we began application development with Grails. We began having difficulties where the application was changing rapidly, necessitating corresponding changes in the schema and data migration script. We were having a disconnect between the schema changes and the application changes. It was at this point that I attended Marc’s session.

The main point that he drove home which we already knew too well was that Database refactoring is expensive. Databases have no source code, databases resist change, databases need to be coordinated with corresponding applications, and teams usually have many databases to coordinate (ie: dev, staging, production). Marc stressed the first point of there being no source code as one of the major problems. Database creation and maintenance are usually done as ‘patch scripts’.

Enter LiquiBase. Marc neglected to mention this very handy library that we decided to try out with our new project. LiquiBase is DBMS-agnostic and allows us to manage database change much more cleanly than ever before. I haven’t fully explored all of the LiquiBase functionality yet but so far am very pleased with what I have explored. There is a LiquiBase Grails plugin and also a LiquiBase IntelliJ Idea plugin that we’ve been using.

Back to Marc’s presentation. Because we’re already convinced that Database refactoring is very expensive, he made a case to do Big Picture database modeling up-front, most importantly capturing all the areas where generalizations and abstractions can be made. It is much more expensive to refactor something specific into something general than to refactor something general into something specific. Adding objects rarely if ever affects existing functionality, but removing them usually does.

Start by modeling the entities and relationships only. Don’t worry about the attributes or details – these can be fleshed out during implementation.

Marc had some Dogmas to impart:

• Use subtypes: they’re great for recording extra knowledge about a logical model
• Hate NULLs: allow no optional relationships. This usually requires using a linked entity which may seem uglier but makes a whole lot more sense
• Allow no implied participants in transactions

And then there was Pragma:

• Don’t invent when you can copy
• Use standard models. He highly recommended Len Silverston’s Data Model Resource Books which we also refer to here at 2Paths
• Reuse models when you can

So far in the presentation, it didn’t seem like we were in an “Agile” session, but finally Marc roped us back into the subject at hand. Just because we’ve done up front design doesn’t mean we have to implement everything at once. Using the up front design we can do iterative development. We don’t need to have the up front model be 100% correct – we can do iterative refinement to it as the project progresses.

For an agile implementation, revisit and refine the model at the start of each iteration. Make explicit documented decisions on each of the design decisions that are made so that down the road it’s known why certain decisions were made.

In summary, Marc reiterated that because databases are so expensive to refactor, it’s important to get as much right the first go around as possible. Getting it right first will make agile development and subsequent refactors much easier. Design for change by generalizing and abstracting relationships.

Relating Marc’s advice back to our project, I realized that we hadn’t done too badly with our process after all. We had made generalizations and abstractions where we could, getting the Big Picture logical design done up front. We then began application development based only on the stories included in the current iteration. The problems we were having in syncing the application to the database were mainly mapping to physical design issues, and not logical design issues.

In hindsight, it may have been easier if we took the logical model that we had designed and implemented this in Grails first and seen what schema Grails produced from that model. We could then have tweaked this schema and associated mappings to be the physical implementation we wanted, instead of producing the physical model in the database first.

When we had started the process of our up front design we had had many discussions about whether this was the right thing to do, and if it was really agile. It was good to have confirmation that others have thought of this too and that it IS okay to do Big Picture up front design. This will hopefully save us some pain and suffering in the future. Hopefully we’ve made our future database refactoring less expensive.

These concepts are all database-agnostic, but there are specific tools for use with Java, Hibernate, and Spring to assist in persistence strategy integration. For more information, see the wiki here:

https://dev.2paths.com/wiki/display/2pathsTECH/Persistence