Apache Cassandra
Cassandra is a fairly recent addition to the “no-sql” movement. Initially developed by Facebook, Cassandra was open sourced in 2008 and is now housed and maintained by Apache.

Cassandra is a highly scalable, eventually consistent, distributed, structured key-value store. Cassandra brings together the distributed systems technologies from Dynamo and the data model from Google’s BigTable. Like Dynamo, Cassandra is eventually consistent. Like BigTable, Cassandra provides a ColumnFamily-based data model richer than typical key/value systems.

Cassandra was open sourced by Facebook in 2008, where it was designed by Avinash Lakshman (one of the authors of Amazon’s Dynamo) and Prashant Malik ( Facebook Engineer ). In a lot of ways you can think of Cassandra as Dynamo 2.0 or a marriage of Dynamo and BigTable. Cassandra is in production use at Facebook but is still under heavy development.

Data Model
The key difference between Cassandra and many of the other key-value stores is that it possesses the concept of a “Super Column.” For more information on what exactly a super column is (and an overview of the Cassandra data model in general) read the following blog post “WTF is a SuperColumn? An Intro to the Cassandra Data Model”

Performance
Many may be curious about the performance of these key value stores. We found an article that compared Cassandra to Voldemort

Interacting with Cassandra
Cassandra is developed in java however it does not allow access via JDBC. Instead it provides access to it’s repository via the “Apache Thrift” framework which basically allows developers in any major language to gain access to a given service (in this case the Cassandra data store). For those like me who are used to JDBC, the thrift interface can be a bit confusing given that there isn’t much documentation for them. The Java client made available on the Cassandra website is rather rudimentary with no support for things like transactions, therefore many people choose to make use of a library called Hector that provides many of the features necessary for any production worthy application making use of Cassandra.

The Project
What I wanted to to do was use the Cassandra data store on a pet project that I’m working on in my spare time. I intend to use the datastore to do the following:

1. Store messages being sent from one user to another
2. Track queries made by a given user
3. Store user complaints

Installation
Installation of the server is easy enough. Install Java 1.6 if you haven’t already. Edit /bin/cassandra.in.sh and change JAVA_HOME if you run a different version of java as your default JAVA_HOME. By default the server has an RMI port set to 8080, you can change the value in the /bin/cassandra.in.sh file as well.

Configuration
Edit /bin/storage-conf.xml to define various key spaces and column families. The concept of a Keyspace is similar to the schema in a relational database, where as the ColumnFamily is analogous to a table. In defining the ColumnFamilies you are defining the key groupings and how they are sorted. Therefore, I defined the following:

  <Keyspaces>
    <Keyspace Name="Nikahfied">

      <ColumnFamily Name="Queries"
                    Comment="User queries"/>

      <ColumnFamily Name="Complaints"
                    Comment="User complaints"/>

      <ColumnFamily Name="Messages"
                    ColumnType="Super"
                    CompareWith="UTF8Type"
                    CompareSubcolumnsWith="UTF8Type"
                    RowsCached="10000"
                    KeysCached="50%"
                    Comment="Message threads between users of the system"/>

    </Keyspace>
  </Keyspaces>

I would be creating a schema named Nikahfied and three table are Queries, Complaints and Messages. As you will notice, there is no real structure to the ColumnFamilies other than what type of column and the CompareWith strategy as outlined in the “WTF is a SuperColumn? An Intro to the Cassandra Data Model” article. In this case we have one super column and two regular columns. In code I will ensure that each of the rows fore each ColumnFamily contain the same content however there is nothing forcing you to do so. As you will notice, we can configure each of the ColumnFamiles individually. Messages will need to handle heavy reads and writes where as Queries and Complaints will be more write intensive. Messages therefore have a caching scheme defined where as the others do not as they will primarily handle writes rather than reads.

When initially defining the column structure I was a bit confused. I was taking the definition of Cassandra as a “key value store” literally and was confused about there being a primary key and the secondary keys. When defining a basic Column I though it would simple have a key and a value. So things would be defined as Nikahfied.<ColumnFamily>.key = value where this is actually not the case. For a basic ColumnFamily, the structure is as follows: Nikahfied.<ColumnFamily>.<Primary key>.<Secondary Key> = value. In the case of a super column, this goes one level further Nikahfied.<ColumnFamily>.<Primary key>.<Secondary Key>.<Tertiary Key> = value. That had me puzzled for some time…upon re-reading the “WTF is a SuperColumn? An Intro to the Cassandra Data Model” article I realized my mistake so I though I’d point it out to anyone else who was confused about this construct.

Data Structures:
Queries:
The structure of queries would be simple enough. I would store each query by date/time. To track which queries the user made when.

Nikahfied.Queries.UserId.<Date/Time> = <Query>

Complaints:
There is only one type of complaint wherein a user can complain about another user. So I wanted to track who received the complaint, who complained and when. The structure of a complaint would be defined as follows:
Nikahfied.Complaints.<Violator Id>.<Complainant Id> = <Date/Time>

Messages:
If there was ever a case to use a super column, a messages would be it. A user has a number of threads that contain one or more message. A user can receive and send messages. Messages themselves are the same, I simply need to track which threads contain messages that a user sent. I also need to keep track of which threads have unread messages within them. My initial design was the following:
Nikahfied.Messages.RecipientId.threads = A JSON Object that contains Threads defined in JSON Objects themselves. A thread being a group of
.sent_ids = An array of thread id's that contain messages that the user has sent
.unread_ids = An array of thread ids that contain unread messages

However, with this structure there was no need for a super column and I was determined on making use of one . So I expanded out the thread structure as follows:

Nikahfied.Messages.RecipientId.threads.<thread id> = A JSON Object that contains an individual Thread
.sent_ids = An array of thread id's that contain messages that the user has sent
.unread_ids = An array of thread ids that contain unread messages

Now that we have the content structure defined I now need to create a hook between my application and the Cassandra key value store. For development I have decided to use the Grails Framework to speed up development. I have included the Hector library to access the framework but needed to define a CassandraService to be the central point of contact between various controllers and other services to the Cassandra key store. Both Cassandra’s Thrift interface and Hector have pretty poor documentation so I created a Grails service that makes the interaction with Cassandra a bit clearer (at least to me). Note: You will need to be using Java 1.6.

import me.prettyprint.cassandra.service.CassandraClientPool
import me.prettyprint.cassandra.service.CassandraClientPoolFactory
import me.prettyprint.cassandra.service.CassandraClient
import me.prettyprint.cassandra.service.Keyspace
import org.apache.cassandra.thrift.Column;
import org.apache.cassandra.thrift.ColumnPath;
import org.apache.cassandra.thrift.NotFoundException;

import org.apache.cassandra.thrift.SuperColumn

class CassandraService {

    boolean transactional = true

    def servers=["localhost:9160"]
    def defaultKeyspace="Nikahfied"
    private static final String NOT_FOUND = ""

    private execute(keyspaceName=defaultKeyspace,block){
        CassandraClientPool pool = CassandraClientPoolFactory.INSTANCE.get();
        CassandraClient client = pool.borrowClient(servers);

        try {
            Keyspace keyspace = client.getKeyspace(keyspaceName)
            return block(keyspace)
        } finally {
            pool.releaseClient(client);
        }
    }

    /**
     * Get a single super column
     * @param columnFamily
     * @param secondaryKey
     * @param key primary key
     * @return Matching super column
     */
    public SuperColumn getSuperColumn(String columnFamily, String key, String secondaryKey) {
        ColumnPath cp = new ColumnPath(columnFamily)
        cp.setSuper_column(secondaryKey.bytes)
        return execute {Keyspace keyspace ->
            SuperColumn sc = null;
            try {
                sc = keyspace.getSuperColumn(key, cp)
            } catch (NotFoundException nfe) {
                sc = null;
            }
            return sc
        }
    }

    /**
     * Get multiple super columns
     * @param columnFamily
     * @param secondaryKey
     * @param keys primary keys
     * @return matching super columns
     */
    public Map multigetSuperColumn(String columnFamily, List keys, String secondaryKey) {
        ColumnPath cp = new ColumnPath(columnFamily)
        cp.setSuper_column(secondaryKey.bytes)
        return execute {Keyspace keyspace ->
            Map scMap = keyspace.multigetSuperColumn(keys, cp)
            return scMap
        }
    }

    /**
     * Get multiple columns
     * @param secondaryKey secondaryKey
     * @param keys primary keys
     * @param columnFamily column family
     * @return results if any
     */
    public Map multigetColumn(List keys, String secondaryKey, String columnFamily) {
    	ColumnPath cp = new ColumnPath(columnFamily)
        cp.setColumn(secondaryKey.bytes)
        return execute {Keyspace keyspace ->
            Map cMap = keyspace.multigetColumn(keys, cp)
            return cMap
        }
    }

    /**
     * Get a single column and it's values
     * @param columnFamily
     * @param key primary key
     * @return matching column
     */
    public Column getColumn(String columnFamily, String key, String secondaryKey) {
        def cp = new ColumnPath(columnFamily)
        cp.setColumn(secondaryKey.bytes)
        return execute {Keyspace keyspace ->
            Column c = null;
            try {
                c = keyspace.getColumn(key, cp)
            } catch (NotFoundException nfe) {
                c = null;
            }
            return c
        }
    }

    /**
     * Sets the new value for this column path
     * @param cf Column family
     * @param secondaryKey secondary key
     * @param key primary key
     * @param value value
     */
    def setColumnPathValue(String cf, String key, String secondaryKey, String value){
        def cp = new ColumnPath(cf)
        cp.setColumn(secondaryKey.bytes)
        return setColumnValue(cp, key, value)
    }

    /**
     * Sets the new super column value for this column path
     * @param cf Column Family
     * @param sc Super Key Id
     * @param secondaryKey Secondary Key
     * @param key primary key
     * @param value value
     * @return the value of the Column requested
     */
    def setColumnPathValue(String cf, String sc, String key, String secondaryKey, String value){
        def cp = new ColumnPath(cf)
        cp.setSuper_column(sc.bytes)
        cp.setColumn(secondaryKey.bytes)
        return setColumnValue(cp, key, value)
    }

    /**
     * Set a regular column
     * @param cp Column path
     * @param key primary key
     * @param value value
     */
    private setColumnValue(ColumnPath cp, String key, String value){
        return execute{ Keyspace keyspace ->
            keyspace.insert(key, cp, value.bytes)
        }
    }

    /**
     * Batch insert content for a give key and it's values
     * @param key primary key
     * @param columnMap column key values
     * @param superColumnMap super column key values
     */
    public batchInsert(String key, Map<String, List> columnMap, Map<String, List> superColumnMap) {
    	return execute{ Keyspace keyspace ->
        	keyspace.batchInsert(key, columnMap, superColumnMap)
    	}
    }

}

Using this class I have created a MessageService, ComplaintsService and a QueriesService that interact with the CassandraService to retrieve and store data. There is an adjoining Message and Complaints controller where as the QueriesService is referenced by the SearchController. Hopefully this blog post will provide you with a better understanding of how to get started with Cassandra the CassandraService class should provide you with a jump start on integrating Cassandra into an existing Grails or Java project.

I once heard Stephen Covey speak years ago and the main thing I remembered is that you don’t truly learn until you teach others.  So, throughout this past year, I’ve been able to take the learnings from the materials and adapt them to our needs so that our team could improve their communication skills and continue developing their leadership qualities.

In September of this year, I reviewed the module entitled “Leadership Excellence” which involved John Maxwell, author of The 21 Irrefutable Laws of Leadership and felt our entire team, technical or otherwise, would benefit from hearing some of the key messages John was imparting about the real measures of leaders and what causes people to truly connect and be inspired to follow a leader.   I decided to take the information presented in the video segment and reorganize it to create a condensed version of the learnings and attention span of  our team (under 30 minutes).  Those that attended our lunch hour “non” Techtalk, found it insightful and and useful.

I also reviewed another module called “How to be a more Influential and Persuasive Leader” presented by Kevin Hogan. This module, although similar in terms of overall topic, had a completely different approach and point of view. It really highlighted a variety of tactics people, and also leaders in a workplace, can use to influence behaviour of others (children, employees, potential customers, and hiring managers).  To share this knowledge with our team, I decided to send out daily leadership nuggets by email (less than 1 minute) that team members could read, digest and use as they felt relevant.

A third module I reviewed called “Becoming a Better Communicator (especially in crucial conversations)” presented by Ron McMillan, was also very interesting and involved the strategies master communicators use when having “crucial conversations” with team members.  I actually prepared a half-day workshop for our Support Crew including discussion time, and exercises, and was able to use some of the supporting materials from Ron McMillan’s organization www.crucialconversations.com to analyze the team’s individual and collective strengths in having these types of tough conversation.

The entire “Lessons in Leadership” package is certainly worth the cost and I encourage you to check out http://thebusinesssource.com/ for other products.

• Parent object has a collection of Child objects
• We want the Parent’s create and edit GSPs to allow us to add/remove/update associated Child objects
• The controller should correctly persist changes to the collection of Child objects, including maintaining Child object ids so any other objects referencing them don’t get screwed up

I found a really nice solution that avoids adding a lot of code to the controller to sift out added/changed/deleted collection members. The original page seems to have disappeared, so here are copies from archive.org (easier to read) and Google cache (PDF).

I was disappointed that the original page is gone, and I found some small errors in the sample code, so I thought it would be nice to document here.

Here’s a sample project I created to go through this. Source code: one-many.tar.gz

The original example used Quest objects that can hold many Task objects. I’ll follow the Grails docs and use Author objects that can hold many Book objects.

First, create the Author class.

    import org.apache.commons.collections.list.LazyList;
    import org.apache.commons.collections.FactoryUtils;

    class Author {

        static constraints = {
        }

        String name
        List books = new ArrayList()
        static hasMany = [ books:Book ]

        static mapping = {
            books cascade:"all,delete-orphan"
        }

        def getExpandableBookList() {
            return LazyList.decorate(books,FactoryUtils.instantiateFactory(Book.class))
        }

    }

(Here’s a minor correction I had to make to the original document’s code. They declared getExpandableBookList as returning a List, but that gave unknown property errors. Using a plain def fixed that.)

This adds a bunch of useful behaviour right away. The mapping block declares that books will be deleted when they’re removed from the Author.books collection, so we don’t need to clean up anything manually. By initializing books to an empty ArrayList when an Author object is created, and by using the getExpandableBookList() method, we can easily add and remove Book objects to the Author.books collection.

Next, the Book class is pretty simple.

    class Book {

        static constraints = {
        }

        String title
        boolean _deleted

        static transients = [ '_deleted' ]

        static belongsTo = [ author:Author ]

        def String toString() {
            return title
        }

    }

Nothing too fancy here, but pay attention to the _deleted property. That’s what we’ll be using to filter out Book objects that need to be removed from the Author.book collection on updates.

For the views, I like to combine the guts of the create and edit GSPs into a template that they can both render.

    <div class="dialog">
        <table>
            <tbody>
                <tr class="prop">
                    <td valign="top" class="name"><label for="name">Name:</label></td>
                    <td valign="top" class="value ${hasErrors(bean:authorInstance,field:'name','errors')}">
                        <input type="text" id="name" name="name" value="${fieldValue(bean:authorInstance,field:'name')}"/>
                    </td>
                </tr>
                <tr class="prop">
                    <td valign="top" class="name"><label for="books">Books:</label></td>
                    <td valign="top" class="value ${hasErrors(bean:authorInstance,field:'books','errors')}">
                        <g:render template="books" model="['authorInstance':authorInstance]" />
                    </td>
                </tr>
            </tbody>
        </table>
    </div>

That uses _books.gsp to render the editable list of books.

<script type="text/javascript">
    var childCount = ${authorInstance?.books.size()} + 0;

    function addChild() {
        var htmlId = "book" + childCount;
        var deleteIcon = "${resource(dir:'images/skin', file:'database_delete.png')}";
        var templateHtml = "<div id='" + htmlId + "' name='" + htmlId + "'>\n";
        templateHtml += "<input type='text' id='expandableBookList[" + childCount + "].title' name='expandableBookList[" + childCount + "].title' />\n";
        templateHtml += "<span onClick='$(\"#" + htmlId + "\").remove();'><img src='" + deleteIcon + "' /></span>\n";
        templateHtml += "</div>\n";
        $("#childList").append(templateHtml);
        childCount++;
    }
</script>

<div id="childList">
    <g:each var="book" in="${authorInstance.books}" status="i">
        <g:render template='book' model="['book':book,'i':i]"/>
    </g:each>
</div>
<input type="button" value="Add Book" onclick="addChild();" />

And that uses _book.gsp to render the individual records. It’s a bit overkill to call out to another template for only a few lines of HTML, but that’s how the original example did it and I’ll do the same for consistency.

    <div id="book${i}">
        <g:hiddenField name='expandableBookList[${i}].id' value='${book.id}'/>
        <g:textField name='expandableBookList[${i}].title' value='${book.title}'/>
        <input type="hidden" name='expandableBookList[${i}]._deleted' id='expandableBookList[${i}]._deleted' value='false'/>
        <span onClick="$('#expandableBookList\\[${i}\\]\\._deleted').val('true'); $('#book${i}').hide()"><img src="${resource(dir:'images/skin', file:'database_delete.png')}" /></span>
    </div>

Here’s where I changed a bit more from the original example. I used jQuery because the selectors make things easy. Basically we render the books from the already-persisted author object, and keep track (using the _deleted field) of any that the user wants to remove. We also keep track of new objects to add.

One of the reasons I really liked this technique was how little impact there is on the controller. We just need to add this to the update method in AuthorController.

    def update = {
        def authorInstance = Author.get( params.id )
        if(authorInstance) {
            if(params.version) {
                // ... version locking stuff
            }
        authorInstance.properties = params
        def _toBeDeleted = authorInstance.books.findAll {it._deleted}
        if (_toBeDeleted) {
            authorInstance.books.removeAll(_toBeDeleted)
        }
        // ... etc.

The original example added similar code to the save method, but I don’t think it’s required for new objects (since they don’t have any already-persisted books to delete, only new books to create) so I only put it in the update method. I also changed it from find{} to findAll{} to guarantee that we get a list, and checked that we have objects to remove before calling the removeAll().

And it works great! Let’s look at some screenshots of the application in action.

First, we can create a new author and add some books right here instead of creating them separately and then matching them up.

Hit “Create” and it creates the Author and Book objects.

Edit the author we just created and see how we get a form that looks the same.

However, it’s worth noting that the books displayed here are the already-persisted ones, so the form is keeping track of their ids and whether we should keep them or delete them on update. Let’s delete the first one and add two more new books.

Now when we hit “Update” the controller has to be smart enough to remove that first book from the Author.books collection, then create two new Book objects and add them to the collection. And naturally, it is.

In addition to creating and destroying Book objects, we can update them. For example, let’s change the title of that first book to be the long version.

No problem!

So that’s one-to-many relationships in Grails forms. I hope it’s useful.

We recently presented at the Turning Statistics Into Knowledge conference, jointly organized by the US Census Bureau, the OECD and the World Bank, on July 15-16 in DC at the Census Bureau. Here is a pdf of it including our speaking notes, or you can click through it below, sadly without notes (goto the slideshare site to see the notes).

I promised a number of people links to non-technical to technical primers, here we go:

Non-technical

1. Very straight-forward overview, very good for sharing with non-technical people:
2. Very quick one-pager, “The “Semantic Web”: a low-fat, no-nonsense introduction” by @ajubdeen
3. Original article from Tim Berners-Lee showing his vision.
4. CNN Making sense of the ’semantic Web’
5. Tim Berners-Lee 2009 TED talk on Linked Data

Semi-technical

1. Largely non-technical, injects a few buzzwords, “Quick Linked Data Introduction” by Michael Hausenblas
2. Understanding the New Web Era: Web 3.0, Linked Data, Semantic Web by Richard MacManus, summarizing a multi-part overview by Greg Boutin

Technical

1. An excellent overview “Linked Data Tutorial” by Michael Hausenblas
2. 30 Minute Guide to RDF and Linked Data by Ian Davis and Tom Heath

After that, you’re off to the races!

(but as I said, just send us an email if you have questions or wanted to talk about it some more)

Yuri from Boats.com went through their experience of using Groovy and Grails for some real projects.

• Using Grails didn’t increase developer productivity, but it’s not the fault of Grails. Most of the holdups are non-technical – waiting for information, scope creep, redefining specs, etc.
• Many tutorials show you can whip up a CRUD application in 5 minutes, but he doesn’t know anyone in the business of writing CRUD applications. Real applications involve a lot more, and Grails is certainly useful, but it’s not a simple “grails generate-my-whole-app” command like some of the tutorials seem to indicate.
• There are lots of good plugins for UI elements, but you have to be careful to avoid Javascript hell. One plugin might use YUI, another might use jQuery, another might use Prototype, etc. If you’re not careful, your web page might suddenty balloon to over a megabyte just for the sake of some rounded corners and accordion widgets. Not really surprising, but just something to be aware of.
• Groovy and Grails add some layers of abstraction and they don’t come for free, so make sure your development machine has enough CPU and RAM (and especially disk IO speed, in my opinion) to handle it.
• Because Java is mature, there are lots of nice power tools available for code analysis, etc. Groovy/Grails doesn’t have much of that yet.
• Dynamic languages are cool and it was rewarding to try out.
• Advertising their use of Groovy/Grails helped attract talented people.

That last point was interesting. I had only thought about the stack from a technical/productivity point of view, but it turns out there’s a good human justification for trying it out. If you can bait people with shiny dynamic languages, it can pay off by increasing your developer talent pool.

Next, Andrew from SpringSource presented the Groovy plugin for Eclipse that they hope to release very soon. Here at 2Paths, about half of us are using Eclipse for Grails development and about half are using IntelliJ IDEA. Andrew walked through a demo showing off a bunch of work they’ve been doing.

• Support for lots of mixing between Java and Groovy. He declared a Groovy class in one file, then created a Java subclass in another file, and then a Groovy subclass of the Java subclass in a third file. Inheritance worked nicely and  the autocomplete magic appeared to be sufficiently magical.
• There’s some code inspection happening so autocomplete knows about things like the findBy* methods on your Groovy objects. Declare a Foo property on the Bar class and autocomplete will know about Bar.findByFoo() automatically. Lots of incremental compilation happening everywhere.
• Lots of improvements in communication between the Java and Groovy compilers. Apparently the old plugin managed the stub compilation with lots of temporary files, so it was constantly writing to and reading from the disk. The new plugin couples the two worlds better and does a lot in RAM, so it should be faster and not thrash the disk so much.
• A working debugger! Conditional breakpoints work, but you have to watch out for all the casting that happens under the hood when checking values. There are still some bugs in setting and removing breakpoints, but it looks good so far. Andrew talked about how he’s still trying to decide how much of the Groovy stack frames and object decorations to show. They show up in IDEA, so you have to wade through many incomprehensibly named caller levels between your own method calls, but it’s all there in case you really want to know.
• The plugin will be available from the same place on The Codehaus as the old plugin and it should be a standard easy install. Once they have something they can release, they’re going to throw it over the fence to be an open source project and encourage outsiders to contribute.
• They fiddle with the Eclipse compiling backend by patching in some behaviour that’s basically “if (groovy) { do new_thing } else { do normal_thing }” so the plugin will work with Eclipse 3.4.2 only. They’ve done a lot of testing to make sure their patch doesn’t hurt normal Java projects, but that version dependency could become frustrating.

Personally, I switched to IDEA because of the debugging support, which works well in both running the application and the tests. I said that we’re pretty evenly split between IDEs at 2Paths and we compare notes every morning in a contest to see whose tools are the most annoying. It looks like this new plugin may push Eclipse back a few points on the annoying scale. Good work, Andrew and company.

As you Grails die-hards know, Grails 1.1 introduced the new GrailsUnitTestCase class which is a very useful extension of GroovyTestCase with support for mocking the dynamic properties and methods that Grails injects into various types of classes (domain, controller & taglibs). Getting the team up to speed on this change was the original purpose of the presentation, but I decided to expand it to cover the full spectrum of testing capabilities in Grails 1.1 for those who might not have much exposure to it previously (including me!)

The presentation covered:

- the benefits & drawbacks of each type of testing: unit testing, integration testing, functional testing.

- details on the new GrailsUnitTestCase and it’s extensions (ControllerUnitTestCase and TagLibUnitTestCase)

- a few “gotchas” – such as using mockDomain with inheritance hierarchies

- WebTest goodness – test organization & re-use, Groovy step, passing params between Groovy & WebTest contexts, a little dive into using HtmlUnit to test AJAX

- general recommendations for what types of tests to create for each artifact (domain classes, controllers, services, etc)

Here’s a link to the presentation on SlideShare.net (where you can view it online or download it):

Happy testing!
Dave

Install memcached, which is simple if you use a Mac and have MacPorts installed.

$ sudo port install memcached

Get the memcached jar , the hibernate-memcached jar and the PostgreSQL JDBC driver jar, then  add them to the project.

$ cp memcached-2.3.1.jar hibernate-memcached-1.2.jar postgresql-8.4-701.jdbc3.jar <project_dir>/lib/

Configure memcached as the second-level cache by adding something like this to your DataSource.groovy.

hibernate {
    cache.use_second_level_cache = true
    cache.use_query_cache = true
    cache.provider_class = 'com.googlecode.hibernate.memcached.MemcachedCacheProvider'
    memcached {
        servers = "localhost:11211"
    }
}

I wrote a simple test to generate a whole bunch of domain objects and shove them in the database, then go back and fetch them all again. I set the test to run the insert/load loops in batches of 1, 10, 100, 1,000, 2,000, 4,000, 7,000, and 10,000 objects. I expected that inserting the objects would take a little longer because of the memcached overhead but that loading the objects would be faster because they could be loaded from memcached instead of going out to the database.

I tested three persistence backend configurations.

• HSQLDB, the default for new Grails projects and when running unit/integration tests. This is an in-memory database only, so data aren’t persisted permanently.
• PostgreSQL for a full-blown RDBMS setup. I set dbCreate to “create-drop” in DataSource.groovy so that each test started with an empty database.
• PostgreSQL with memcached enabled. I should point out that in this test I had both PostgreSQL and memcached running on the same machine (my laptop) as the application, so it’s really not taking advantage of memcached’s parallelism strengths.

My expectations held true for the inserts. The memcached case is a little slower but not that much.

Comparison of object insert times using HSQL, PostgreSQL, and memcached+PostgreSQL for persistence backend

When I tested the load times, I was surprised. It looked like memcached wasn’t speeding things up at all. I was also surprised to see that it took less time to load 10,000 objects from the data store than 7,000 objects. I ran the test a few times to rule out the possibility of a load spike from something else running at the same time, but I got similar results every time.

Comparison of object load times using HSQL, PostgreSQL, and memcached+PostgreSQL for persistence backend

I then tried to improve my test setup by running memcached on a second machine (my desktop, connected by gigabit ethernet to the laptop). Using the PostgreSQL + memcached configuration for each test this time, I looked at three more scenarios.

• PostgreSQL running locally on the laptop, memcached running locally on the laptop.
• PostgreSQL running locally on the laptop, memcached running remotely on the desktop.
• PostgreSQL running locally on the laptop, memcached running on both the laptop and the desktop.

There’s some overhead in going out to the network to retrieve data instead of fetching it from a process running locally, so I expected the memcached local instance to be faster than the memcached remote instance. I expected the third test, with two instances of memcached running in parallel, to be fastest because it splits the load between the two instances and frees up some CPU time on the laptop. Things didn’t play out that way, though.

Comparison of object insert times for various memcache setups

Comparison of object load times for various memcache setups

I have to be honest that these were not great tests. For one, I should really have run those insert/load loops in parallel instead of sequentially. I’m guessing that the test spent a lot of time stalled waiting for data when it could have been sending out more requests. I would also like to test with a large group of memcached servers and a remote database, perhaps even a cluster of app servers with a load balancer to try the full meal deal. The memcached FAQ explains that running everything on one machine will not show off memcached’s scalability.

However, this gave me the opportunity to get my hands dirty with memcached and learn a few things.

• Integrating memcached is so simple and the performance penalty of just running it locally during development is so small that there’s no reason not to enable it.
• It’s dead simple to install and set up. We could , for example, easily install it on a bunch of test servers and run multiple instances on different ports – one for each developer (to prevent us from clobbering each other’s data).
• Measuring performance and scalability realistically is tricky. It’s one thing to hand wave and say that running multiple instances of a database or a cache will speed things up, but it takes some work to set up a realistic simulation of high traffic application performance.
• Adding memcached doesn’t automatically turn performance up to 11. I need to learn more about what’s going on under the hood with my domain objects and how they’re being persisted through memcached.

I look forward to using memcached more in the future and learning how to really take advantage of it.

Update (2009-07-24): Ray pointed out in the comments that I should have enabled caching in my domain objects under test. I added “static mapping = { cache true }” to my domain class definition and re-ran the tests to compare the effect under 4 scenarios.

• no domain object caching (as before)
• domain object caching with memcached running locally
• domain object caching with memcached running remotely
• domain object caching with memcached running both locally and remotely

Not much difference with the inserts.

Comparison of object insert times with domain object caching enabled and disabled

However, the load times are slightly better with memcached enabled during heavy activity! The advantage disappears when I add the overhead of traversing the network, but it’s a start. Thanks for the pointer, Ray!

Comparison of object load times with domain object caching enabled and disabled

PROS

Probably the number one mentioned feature by anyone talking about CouchDB is that it’s schema-free. That means that each document is its own structure, that everything is based on a key-value pairing, where the value is typically a document, but can also be a base type. This is pretty cool when dealing with describing real world objects, as one can be more descriptive and hierarchical in storing the data. Also, updating your application’s domain model is much like evolution, as soon as you need something added to the model, just add it.

CouchDB’s RESTful HTTP service implementation is great for the many applications being created for today’s web. Send Javascript queries/requests to the service, and get JSON formatted data in return. This is great, since many front-end technologies can decipher JSON, and the format itself is even human readable if need be.

There is support for interacting with CouchDB data offline. One can access and edit data while not connected, and when back online can merge and update changes made while offline. This kind of support out of the box makes CouchDB a prime candidate for  web based applications like blogs, wikis, bug reporting, CRM, etc.

Another quality of CouchDB is its MapReduce implementation for handling large data sets. I’m going to redirect you to the author’s notes for this one: MapReduce and Google’s definition.

CONS

Transaction management is non-existent. Relational database managements systems (RDBMS) provide mechanisms for things like transaction management, particularly unique constraints and locking. CouchDB is based on being ‘eventually consistent’ and ‘available’ (See the CAP Theorem section for more), which although great for many things, doesn’t lend itself to transaction management.

Data security is also a concern. A somewhat old article describes this issue quite well: Planned Security Model For CouchDB. The CouchDB team has been working on security and validation but are still early in their development.

To a less important degree, but still worth mentioning, is that personally I really don’t find the out-of-the-box UI for CouchDB to be all that useful. It feels clunky to navigate your key/document pairs, and too feature-light for managing the database.

Conclusion

If you’re looking to create a document based web application, then this may be a worthwhile technology to use. CouchDB has a number of very impressive features. However, for now at least, I believe I’ll be staying off the couch.

Aside:

When testing out CouchDB I went through the process of manually installing the application. I had to resolve the dependencies, and deal with compilation errors on my own. I kind of wish I’d found this site earlier. Follow that link if you’re a MacOSX user and want an installer for CouchDB.

For me, this was pretty much the  only useful tutorial on getting started with CouchDB: Programming CouchDB with Javascript.

Resources:

• http://couchdb.apache.org/
• http://books.couchdb.org/relax/
• http://www.eflorenzano.com/blog/tag/couchdb/
• http://labs.google.com/papers/mapreduce.html
• http://damienkatz.net/2008/02/incremental_map.html
• http://jan.prima.de/~jan/plok/archives/142-CouchDBX-Revival.html
• http://www.automatthew.com/2008/01/planned-security-model-for-couchdb.html
• http://jchrisa.net/drl/_design/sofa/_show/post/couchdb_edge__security_and_vali
• http://jan.prima.de/~jan/plok/archives/108-Programming-CouchDB-with-Javascript.html
• http://www.infoq.com/CouchDB
• http://www.infoq.com/news/2007/11/the-rdbms-is-not-enough
• http://www.infoq.com/news/2008/11/Database-Martin-Fowler
• http://code.google.com/p/couchdb-lounge/wiki/SettingUpTwoCouchInstances

The basic conference elements were all good: it felt well-run, there was a good mix of people and the sessions were more-or-less mostly-kinda interesting. The conference itself was in San Jose, so… it was warm outside, I guess? We weren’t tempted to skip sessions because there might something more interesting to do? I’ll do breakdowns by category, going in decreasing order of importance.

Keynotes

I attended the opening keynotes on Tuesday, which started with Thomas Tague from OpenCalais letting us know that People With Money™ (my term for it) were about to descend on our sleepy semantic backwater. He was a good speaker, and his message was well-received. What I got out of his talk was that we (as software developers) need to figure out where we can succeed in this new landscape, that there are still a lot of potentially profitable niches and that opportunities for tool developers still exist. Tuesday’s second keynote was Tom Gruber from Siri, who gave a demo of their upcoming mobile “virtual personal assistant”. It was a pretty rad demo, and I’m very excited about the ecosystem of services that it might cultivate. Siri has the potential to be a killer app in both the mobile and semantic web spaces, and I’d love to see their execution match their vision.

I skipped the other two keynotes. One was a panel discussion; I find it difficult to get much out of anything in this format. The closing keynote was two gentlemen from the NY Times discussing semantic technology uptake at the Grey Lady. I probably should have gone to that one.

Sessions

I’m not going to go over every session,  because that’s a lot of writing that no one wants to read. The following sessions were ones I found worthwhile, so if you’re ever undecided about seeing these people, or say, booking them for your conference, you should just go right ahead on my say-so.

Elisa Kendall and Deborah McGuinness: Ontology 101 – a lot of content in this one. One of the key themes that I picked up on during the conference is that almost every way to incorporate semantic tech in an organization starts with understanding and developing ontologies. Another one is that you shouldn’t feel bad for thinking that ontologies are hard… there are people that devote a lot of time and effort to making them, and they still sometimes get it wrong.

John Hebeler and Matthew Fisher: Semantic Web Programming – loved this. It had a great format, with short lab-like exercises that got progressively more complicated; we started with building simple models in Jena, and ended with using the Pellet reasoner to infer new knowledge. It was so good and well-presented that I immediately went out and bought their book.

Dave Wood: Active PURLs – a great, no-nonsense introduction to PURLs and their in-development “active” variants. Dave, along with quite a few people at the conference, really came across as a true believer in the infrastructure of our current web. I had always thought people just made it up as they went a long, but no, a lot of thought and effort went into those early RFCs.

Random people: Visualizing RDF – this sort-of panel was sort-of organized on the spur of the moment. I’m glad I went, because I got to see AllegroGraph’s Gruff and someone else’s Diagramic. Cool tools, these.

Food

The catered stuff was quite good for conference food. I’m going to spread the rumour that, due to certain “economic realities” (*cough*recession*cough*), the turnout was less than the organizers were expecting, and so the catering budget (which had been decided per-head much earlier) was relatively high.

Outside the conference, found a few places in San Jose which might be worth going to if, like us, you didn’t rent a car and don’t have enough time to trek to San Fran.

Mezcal – featured mezcalitas (good) and pan-fried grasshoppers (less good). Get the lightest-coloured molé… best of their three.

Angelou’s – went here while on a burrito hunt. (To a Canadian, America is the Promised Land of the Burrito). I ordered their Super Burrito. It was Merely Okay (A Bit on the Salty Side).

Hanuman – ordered the curried duck (“Thai spicy”). It was fantastically tasty, but it was not Thai spicy.

Conclusion?

This year had a good mix of people from the business, acedemic and technical communities. As expected, there was lots of networking, not all of it involving Twitter or Cat-5. They say next year will be even more business-focussed, but since grad students love to travel, expect to see them too. And maybe us, as well.

The core competency of jQuery is it’s ability to manage and manipulate the HTML DOM. This portion of the library alone makes the small 20kb library fully worth the download. Targeting an element for assignment while manipulating anything within a document is as easy as:

var element = $('#idOfTheElement')

(API for selectors)

More than just HTML document manipulation, jQuery has a very simple and clean way to implement event triggers and handlers.

// trigger
$("#idOfElement").click(function () {
   $("#idOfElementThatIsListening").trigger('click');
});
// handler
$("#idOfElementThatIsListening").bind("click", function(e){
   alert("click event happened")
});

(API for events)

This is just  a taste of what jQuery can do for your web application development. I recommend giving it a test drive if you haven’t already. The online API and documentation is thorough and well constructed. The user community is chock-full of enthusiasts with useful plug-ins.

This will be my go-to JavaScript library for most applications for the foreseeable future.