Lately in Vancouver we’ve seen the supply of developers tighten up as demand grows. Some developers, particularly those with more senior experience, are now in some cases getting multiple offers within a day or two. Whereas last year we could usually afford to spend several weeks in our screening and evaluation process, we’re finding we need to take less than a week or some good folks commit elsewhere. They can come off a project and be available only for a short time before being able to get onto another.

Having a history of doing interesting projects–especially for not only “feel-good”, but “do-good” causes such as global aid and development–has been helpful in finding good people or in some cases, having them find us. Having a track record for successful project conclusions of course helps as well. Conversely though, we are very picky about the skill level of people we bring on. This is reflected in part by our weighting on using mostly senior or sometimes intermediate developers in the vast majority of our projects. Using a technique common in many other industries, some development shops try to leverage the junior developers so they can maximize their profit–they cost less to employ, but bill out at high profit margins. However we’ve found that it’s better to sacrifice profit and ensure our quality is high. (No surprise that “profit” doesn’t appear in 2Paths’ Core Values, Beliefs or Purpose!)

Is there a difference depending on the technology? For instance, some people in the industry say that demand is swinging more to open source and java, away from the Microsoft-y world and .NET. In our case, we’re not sure, but one thing we do know…we still have demand for development in both areas from our clients. Being technology agnostic has meant that some of our projects are open source-based, e.g. using java, whereas others may use .NET technologies. We continue to recommend the technology that best fits the client needs, regardless of the availability of resources.

In our view, those of us that focus on interesting yet challenging work and provide a great work environment will continue to find the talented folks they need.

“The plan is to have all sorts of speakers speak about all sorts of stuff. Not brands, advertising, blogging and twitter but interesting, unexpected, original things. I’m hoping to find fascinating people and to just ask them to speak about something they care about. I want to replicate the experience of clicking from one really good blog to another, ranging across sciences, arts, musics, jokes and whatever.”

—Russell Davies, founder of Interesting

The format is quite cool – a smallish number of speakers give 5 minute talks, and the rest of the night is unstructured. The speakers are from quite the range of life and the topics look extremely thought provoking.

Apache Cassandra
Cassandra is a fairly recent addition to the “no-sql” movement. Initially developed by Facebook, Cassandra was open sourced in 2008 and is now housed and maintained by Apache.

Cassandra is a highly scalable, eventually consistent, distributed, structured key-value store. Cassandra brings together the distributed systems technologies from Dynamo and the data model from Google’s BigTable. Like Dynamo, Cassandra is eventually consistent. Like BigTable, Cassandra provides a ColumnFamily-based data model richer than typical key/value systems.

Cassandra was open sourced by Facebook in 2008, where it was designed by Avinash Lakshman (one of the authors of Amazon’s Dynamo) and Prashant Malik ( Facebook Engineer ). In a lot of ways you can think of Cassandra as Dynamo 2.0 or a marriage of Dynamo and BigTable. Cassandra is in production use at Facebook but is still under heavy development.

Data Model
The key difference between Cassandra and many of the other key-value stores is that it possesses the concept of a “Super Column.” For more information on what exactly a super column is (and an overview of the Cassandra data model in general) read the following blog post “WTF is a SuperColumn? An Intro to the Cassandra Data Model”

Performance
Many may be curious about the performance of these key value stores. We found an article that compared Cassandra to Voldemort

Interacting with Cassandra
Cassandra is developed in java however it does not allow access via JDBC. Instead it provides access to it’s repository via the “Apache Thrift” framework which basically allows developers in any major language to gain access to a given service (in this case the Cassandra data store). For those like me who are used to JDBC, the thrift interface can be a bit confusing given that there isn’t much documentation for them. The Java client made available on the Cassandra website is rather rudimentary with no support for things like transactions, therefore many people choose to make use of a library called Hector that provides many of the features necessary for any production worthy application making use of Cassandra.

The Project
What I wanted to to do was use the Cassandra data store on a pet project that I’m working on in my spare time. I intend to use the datastore to do the following:

1. Store messages being sent from one user to another
2. Track queries made by a given user
3. Store user complaints

Installation
Installation of the server is easy enough. Install Java 1.6 if you haven’t already. Edit /bin/cassandra.in.sh and change JAVA_HOME if you run a different version of java as your default JAVA_HOME. By default the server has an RMI port set to 8080, you can change the value in the /bin/cassandra.in.sh file as well.

Configuration
Edit /bin/storage-conf.xml to define various key spaces and column families. The concept of a Keyspace is similar to the schema in a relational database, where as the ColumnFamily is analogous to a table. In defining the ColumnFamilies you are defining the key groupings and how they are sorted. Therefore, I defined the following:

  <Keyspaces>
    <Keyspace Name="Nikahfied">

      <ColumnFamily Name="Queries"
                    Comment="User queries"/>

      <ColumnFamily Name="Complaints"
                    Comment="User complaints"/>

      <ColumnFamily Name="Messages"
                    ColumnType="Super"
                    CompareWith="UTF8Type"
                    CompareSubcolumnsWith="UTF8Type"
                    RowsCached="10000"
                    KeysCached="50%"
                    Comment="Message threads between users of the system"/>

    </Keyspace>
  </Keyspaces>

I would be creating a schema named Nikahfied and three table are Queries, Complaints and Messages. As you will notice, there is no real structure to the ColumnFamilies other than what type of column and the CompareWith strategy as outlined in the “WTF is a SuperColumn? An Intro to the Cassandra Data Model” article. In this case we have one super column and two regular columns. In code I will ensure that each of the rows fore each ColumnFamily contain the same content however there is nothing forcing you to do so. As you will notice, we can configure each of the ColumnFamiles individually. Messages will need to handle heavy reads and writes where as Queries and Complaints will be more write intensive. Messages therefore have a caching scheme defined where as the others do not as they will primarily handle writes rather than reads.

When initially defining the column structure I was a bit confused. I was taking the definition of Cassandra as a “key value store” literally and was confused about there being a primary key and the secondary keys. When defining a basic Column I though it would simple have a key and a value. So things would be defined as Nikahfied.<ColumnFamily>.key = value where this is actually not the case. For a basic ColumnFamily, the structure is as follows: Nikahfied.<ColumnFamily>.<Primary key>.<Secondary Key> = value. In the case of a super column, this goes one level further Nikahfied.<ColumnFamily>.<Primary key>.<Secondary Key>.<Tertiary Key> = value. That had me puzzled for some time…upon re-reading the “WTF is a SuperColumn? An Intro to the Cassandra Data Model” article I realized my mistake so I though I’d point it out to anyone else who was confused about this construct.

Data Structures:
Queries:
The structure of queries would be simple enough. I would store each query by date/time. To track which queries the user made when.

Nikahfied.Queries.UserId.<Date/Time> = <Query>

Complaints:
There is only one type of complaint wherein a user can complain about another user. So I wanted to track who received the complaint, who complained and when. The structure of a complaint would be defined as follows:
Nikahfied.Complaints.<Violator Id>.<Complainant Id> = <Date/Time>

Messages:
If there was ever a case to use a super column, a messages would be it. A user has a number of threads that contain one or more message. A user can receive and send messages. Messages themselves are the same, I simply need to track which threads contain messages that a user sent. I also need to keep track of which threads have unread messages within them. My initial design was the following:
Nikahfied.Messages.RecipientId.threads = A JSON Object that contains Threads defined in JSON Objects themselves. A thread being a group of
.sent_ids = An array of thread id's that contain messages that the user has sent
.unread_ids = An array of thread ids that contain unread messages

However, with this structure there was no need for a super column and I was determined on making use of one . So I expanded out the thread structure as follows:

Nikahfied.Messages.RecipientId.threads.<thread id> = A JSON Object that contains an individual Thread
.sent_ids = An array of thread id's that contain messages that the user has sent
.unread_ids = An array of thread ids that contain unread messages

Now that we have the content structure defined I now need to create a hook between my application and the Cassandra key value store. For development I have decided to use the Grails Framework to speed up development. I have included the Hector library to access the framework but needed to define a CassandraService to be the central point of contact between various controllers and other services to the Cassandra key store. Both Cassandra’s Thrift interface and Hector have pretty poor documentation so I created a Grails service that makes the interaction with Cassandra a bit clearer (at least to me). Note: You will need to be using Java 1.6.

import me.prettyprint.cassandra.service.CassandraClientPool
import me.prettyprint.cassandra.service.CassandraClientPoolFactory
import me.prettyprint.cassandra.service.CassandraClient
import me.prettyprint.cassandra.service.Keyspace
import org.apache.cassandra.thrift.Column;
import org.apache.cassandra.thrift.ColumnPath;
import org.apache.cassandra.thrift.NotFoundException;

import org.apache.cassandra.thrift.SuperColumn

class CassandraService {

    boolean transactional = true

    def servers=["localhost:9160"]
    def defaultKeyspace="Nikahfied"
    private static final String NOT_FOUND = ""

    private execute(keyspaceName=defaultKeyspace,block){
        CassandraClientPool pool = CassandraClientPoolFactory.INSTANCE.get();
        CassandraClient client = pool.borrowClient(servers);

        try {
            Keyspace keyspace = client.getKeyspace(keyspaceName)
            return block(keyspace)
        } finally {
            pool.releaseClient(client);
        }
    }

    /**
     * Get a single super column
     * @param columnFamily
     * @param secondaryKey
     * @param key primary key
     * @return Matching super column
     */
    public SuperColumn getSuperColumn(String columnFamily, String key, String secondaryKey) {
        ColumnPath cp = new ColumnPath(columnFamily)
        cp.setSuper_column(secondaryKey.bytes)
        return execute {Keyspace keyspace ->
            SuperColumn sc = null;
            try {
                sc = keyspace.getSuperColumn(key, cp)
            } catch (NotFoundException nfe) {
                sc = null;
            }
            return sc
        }
    }

    /**
     * Get multiple super columns
     * @param columnFamily
     * @param secondaryKey
     * @param keys primary keys
     * @return matching super columns
     */
    public Map multigetSuperColumn(String columnFamily, List keys, String secondaryKey) {
        ColumnPath cp = new ColumnPath(columnFamily)
        cp.setSuper_column(secondaryKey.bytes)
        return execute {Keyspace keyspace ->
            Map scMap = keyspace.multigetSuperColumn(keys, cp)
            return scMap
        }
    }

    /**
     * Get multiple columns
     * @param secondaryKey secondaryKey
     * @param keys primary keys
     * @param columnFamily column family
     * @return results if any
     */
    public Map multigetColumn(List keys, String secondaryKey, String columnFamily) {
    	ColumnPath cp = new ColumnPath(columnFamily)
        cp.setColumn(secondaryKey.bytes)
        return execute {Keyspace keyspace ->
            Map cMap = keyspace.multigetColumn(keys, cp)
            return cMap
        }
    }

    /**
     * Get a single column and it's values
     * @param columnFamily
     * @param key primary key
     * @return matching column
     */
    public Column getColumn(String columnFamily, String key, String secondaryKey) {
        def cp = new ColumnPath(columnFamily)
        cp.setColumn(secondaryKey.bytes)
        return execute {Keyspace keyspace ->
            Column c = null;
            try {
                c = keyspace.getColumn(key, cp)
            } catch (NotFoundException nfe) {
                c = null;
            }
            return c
        }
    }

    /**
     * Sets the new value for this column path
     * @param cf Column family
     * @param secondaryKey secondary key
     * @param key primary key
     * @param value value
     */
    def setColumnPathValue(String cf, String key, String secondaryKey, String value){
        def cp = new ColumnPath(cf)
        cp.setColumn(secondaryKey.bytes)
        return setColumnValue(cp, key, value)
    }

    /**
     * Sets the new super column value for this column path
     * @param cf Column Family
     * @param sc Super Key Id
     * @param secondaryKey Secondary Key
     * @param key primary key
     * @param value value
     * @return the value of the Column requested
     */
    def setColumnPathValue(String cf, String sc, String key, String secondaryKey, String value){
        def cp = new ColumnPath(cf)
        cp.setSuper_column(sc.bytes)
        cp.setColumn(secondaryKey.bytes)
        return setColumnValue(cp, key, value)
    }

    /**
     * Set a regular column
     * @param cp Column path
     * @param key primary key
     * @param value value
     */
    private setColumnValue(ColumnPath cp, String key, String value){
        return execute{ Keyspace keyspace ->
            keyspace.insert(key, cp, value.bytes)
        }
    }

    /**
     * Batch insert content for a give key and it's values
     * @param key primary key
     * @param columnMap column key values
     * @param superColumnMap super column key values
     */
    public batchInsert(String key, Map<String, List> columnMap, Map<String, List> superColumnMap) {
    	return execute{ Keyspace keyspace ->
        	keyspace.batchInsert(key, columnMap, superColumnMap)
    	}
    }

}

Using this class I have created a MessageService, ComplaintsService and a QueriesService that interact with the CassandraService to retrieve and store data. There is an adjoining Message and Complaints controller where as the QueriesService is referenced by the SearchController. Hopefully this blog post will provide you with a better understanding of how to get started with Cassandra the CassandraService class should provide you with a jump start on integrating Cassandra into an existing Grails or Java project.

I once heard Stephen Covey speak years ago and the main thing I remembered is that you don’t truly learn until you teach others.  So, throughout this past year, I’ve been able to take the learnings from the materials and adapt them to our needs so that our team could improve their communication skills and continue developing their leadership qualities.

In September of this year, I reviewed the module entitled “Leadership Excellence” which involved John Maxwell, author of The 21 Irrefutable Laws of Leadership and felt our entire team, technical or otherwise, would benefit from hearing some of the key messages John was imparting about the real measures of leaders and what causes people to truly connect and be inspired to follow a leader.   I decided to take the information presented in the video segment and reorganize it to create a condensed version of the learnings and attention span of  our team (under 30 minutes).  Those that attended our lunch hour “non” Techtalk, found it insightful and and useful.

I also reviewed another module called “How to be a more Influential and Persuasive Leader” presented by Kevin Hogan. This module, although similar in terms of overall topic, had a completely different approach and point of view. It really highlighted a variety of tactics people, and also leaders in a workplace, can use to influence behaviour of others (children, employees, potential customers, and hiring managers).  To share this knowledge with our team, I decided to send out daily leadership nuggets by email (less than 1 minute) that team members could read, digest and use as they felt relevant.

A third module I reviewed called “Becoming a Better Communicator (especially in crucial conversations)” presented by Ron McMillan, was also very interesting and involved the strategies master communicators use when having “crucial conversations” with team members.  I actually prepared a half-day workshop for our Support Crew including discussion time, and exercises, and was able to use some of the supporting materials from Ron McMillan’s organization www.crucialconversations.com to analyze the team’s individual and collective strengths in having these types of tough conversation.

The entire “Lessons in Leadership” package is certainly worth the cost and I encourage you to check out http://thebusinesssource.com/ for other products.

• Parent object has a collection of Child objects
• We want the Parent’s create and edit GSPs to allow us to add/remove/update associated Child objects
• The controller should correctly persist changes to the collection of Child objects, including maintaining Child object ids so any other objects referencing them don’t get screwed up

I found a really nice solution that avoids adding a lot of code to the controller to sift out added/changed/deleted collection members. The original page seems to have disappeared, so here are copies from archive.org (easier to read) and Google cache (PDF).

I was disappointed that the original page is gone, and I found some small errors in the sample code, so I thought it would be nice to document here.

Here’s a sample project I created to go through this. Source code: one-many.tar.gz

The original example used Quest objects that can hold many Task objects. I’ll follow the Grails docs and use Author objects that can hold many Book objects.

First, create the Author class.

    import org.apache.commons.collections.list.LazyList;
    import org.apache.commons.collections.FactoryUtils;

    class Author {

        static constraints = {
        }

        String name
        List books = new ArrayList()
        static hasMany = [ books:Book ]

        static mapping = {
            books cascade:"all,delete-orphan"
        }

        def getExpandableBookList() {
            return LazyList.decorate(books,FactoryUtils.instantiateFactory(Book.class))
        }

    }

(Here’s a minor correction I had to make to the original document’s code. They declared getExpandableBookList as returning a List, but that gave unknown property errors. Using a plain def fixed that.)

This adds a bunch of useful behaviour right away. The mapping block declares that books will be deleted when they’re removed from the Author.books collection, so we don’t need to clean up anything manually. By initializing books to an empty ArrayList when an Author object is created, and by using the getExpandableBookList() method, we can easily add and remove Book objects to the Author.books collection.

Next, the Book class is pretty simple.

    class Book {

        static constraints = {
        }

        String title
        boolean _deleted

        static transients = [ '_deleted' ]

        static belongsTo = [ author:Author ]

        def String toString() {
            return title
        }

    }

Nothing too fancy here, but pay attention to the _deleted property. That’s what we’ll be using to filter out Book objects that need to be removed from the Author.book collection on updates.

For the views, I like to combine the guts of the create and edit GSPs into a template that they can both render.

    <div class="dialog">
        <table>
            <tbody>
                <tr class="prop">
                    <td valign="top" class="name"><label for="name">Name:</label></td>
                    <td valign="top" class="value ${hasErrors(bean:authorInstance,field:'name','errors')}">
                        <input type="text" id="name" name="name" value="${fieldValue(bean:authorInstance,field:'name')}"/>
                    </td>
                </tr>
                <tr class="prop">
                    <td valign="top" class="name"><label for="books">Books:</label></td>
                    <td valign="top" class="value ${hasErrors(bean:authorInstance,field:'books','errors')}">
                        <g:render template="books" model="['authorInstance':authorInstance]" />
                    </td>
                </tr>
            </tbody>
        </table>
    </div>

That uses _books.gsp to render the editable list of books.

<script type="text/javascript">
    var childCount = ${authorInstance?.books.size()} + 0;

    function addChild() {
        var htmlId = "book" + childCount;
        var deleteIcon = "${resource(dir:'images/skin', file:'database_delete.png')}";
        var templateHtml = "<div id='" + htmlId + "' name='" + htmlId + "'>\n";
        templateHtml += "<input type='text' id='expandableBookList[" + childCount + "].title' name='expandableBookList[" + childCount + "].title' />\n";
        templateHtml += "<span onClick='$(\"#" + htmlId + "\").remove();'><img src='" + deleteIcon + "' /></span>\n";
        templateHtml += "</div>\n";
        $("#childList").append(templateHtml);
        childCount++;
    }
</script>

<div id="childList">
    <g:each var="book" in="${authorInstance.books}" status="i">
        <g:render template='book' model="['book':book,'i':i]"/>
    </g:each>
</div>
<input type="button" value="Add Book" onclick="addChild();" />

And that uses _book.gsp to render the individual records. It’s a bit overkill to call out to another template for only a few lines of HTML, but that’s how the original example did it and I’ll do the same for consistency.

    <div id="book${i}">
        <g:hiddenField name='expandableBookList[${i}].id' value='${book.id}'/>
        <g:textField name='expandableBookList[${i}].title' value='${book.title}'/>
        <input type="hidden" name='expandableBookList[${i}]._deleted' id='expandableBookList[${i}]._deleted' value='false'/>
        <span onClick="$('#expandableBookList\\[${i}\\]\\._deleted').val('true'); $('#book${i}').hide()"><img src="${resource(dir:'images/skin', file:'database_delete.png')}" /></span>
    </div>

Here’s where I changed a bit more from the original example. I used jQuery because the selectors make things easy. Basically we render the books from the already-persisted author object, and keep track (using the _deleted field) of any that the user wants to remove. We also keep track of new objects to add.

One of the reasons I really liked this technique was how little impact there is on the controller. We just need to add this to the update method in AuthorController.

    def update = {
        def authorInstance = Author.get( params.id )
        if(authorInstance) {
            if(params.version) {
                // ... version locking stuff
            }
        authorInstance.properties = params
        def _toBeDeleted = authorInstance.books.findAll {it._deleted}
        if (_toBeDeleted) {
            authorInstance.books.removeAll(_toBeDeleted)
        }
        // ... etc.

The original example added similar code to the save method, but I don’t think it’s required for new objects (since they don’t have any already-persisted books to delete, only new books to create) so I only put it in the update method. I also changed it from find{} to findAll{} to guarantee that we get a list, and checked that we have objects to remove before calling the removeAll().

And it works great! Let’s look at some screenshots of the application in action.

First, we can create a new author and add some books right here instead of creating them separately and then matching them up.

Hit “Create” and it creates the Author and Book objects.

Edit the author we just created and see how we get a form that looks the same.

However, it’s worth noting that the books displayed here are the already-persisted ones, so the form is keeping track of their ids and whether we should keep them or delete them on update. Let’s delete the first one and add two more new books.

Now when we hit “Update” the controller has to be smart enough to remove that first book from the Author.books collection, then create two new Book objects and add them to the collection. And naturally, it is.

In addition to creating and destroying Book objects, we can update them. For example, let’s change the title of that first book to be the long version.

No problem!

So that’s one-to-many relationships in Grails forms. I hope it’s useful.

We recently presented at the Turning Statistics Into Knowledge conference, jointly organized by the US Census Bureau, the OECD and the World Bank, on July 15-16 in DC at the Census Bureau. Here is a pdf of it including our speaking notes, or you can click through it below, sadly without notes (goto the slideshare site to see the notes).

I promised a number of people links to non-technical to technical primers, here we go:

Non-technical

1. Very straight-forward overview, very good for sharing with non-technical people:
2. Very quick one-pager, “The “Semantic Web”: a low-fat, no-nonsense introduction” by @ajubdeen
3. Original article from Tim Berners-Lee showing his vision.
4. CNN Making sense of the ’semantic Web’
5. Tim Berners-Lee 2009 TED talk on Linked Data

Semi-technical

1. Largely non-technical, injects a few buzzwords, “Quick Linked Data Introduction” by Michael Hausenblas
2. Understanding the New Web Era: Web 3.0, Linked Data, Semantic Web by Richard MacManus, summarizing a multi-part overview by Greg Boutin

Technical

1. An excellent overview “Linked Data Tutorial” by Michael Hausenblas
2. 30 Minute Guide to RDF and Linked Data by Ian Davis and Tom Heath

After that, you’re off to the races!

(but as I said, just send us an email if you have questions or wanted to talk about it some more)

Yuri from Boats.com went through their experience of using Groovy and Grails for some real projects.

• Using Grails didn’t increase developer productivity, but it’s not the fault of Grails. Most of the holdups are non-technical – waiting for information, scope creep, redefining specs, etc.
• Many tutorials show you can whip up a CRUD application in 5 minutes, but he doesn’t know anyone in the business of writing CRUD applications. Real applications involve a lot more, and Grails is certainly useful, but it’s not a simple “grails generate-my-whole-app” command like some of the tutorials seem to indicate.
• There are lots of good plugins for UI elements, but you have to be careful to avoid Javascript hell. One plugin might use YUI, another might use jQuery, another might use Prototype, etc. If you’re not careful, your web page might suddenty balloon to over a megabyte just for the sake of some rounded corners and accordion widgets. Not really surprising, but just something to be aware of.
• Groovy and Grails add some layers of abstraction and they don’t come for free, so make sure your development machine has enough CPU and RAM (and especially disk IO speed, in my opinion) to handle it.
• Because Java is mature, there are lots of nice power tools available for code analysis, etc. Groovy/Grails doesn’t have much of that yet.
• Dynamic languages are cool and it was rewarding to try out.
• Advertising their use of Groovy/Grails helped attract talented people.

That last point was interesting. I had only thought about the stack from a technical/productivity point of view, but it turns out there’s a good human justification for trying it out. If you can bait people with shiny dynamic languages, it can pay off by increasing your developer talent pool.

Next, Andrew from SpringSource presented the Groovy plugin for Eclipse that they hope to release very soon. Here at 2Paths, about half of us are using Eclipse for Grails development and about half are using IntelliJ IDEA. Andrew walked through a demo showing off a bunch of work they’ve been doing.

• Support for lots of mixing between Java and Groovy. He declared a Groovy class in one file, then created a Java subclass in another file, and then a Groovy subclass of the Java subclass in a third file. Inheritance worked nicely and  the autocomplete magic appeared to be sufficiently magical.
• There’s some code inspection happening so autocomplete knows about things like the findBy* methods on your Groovy objects. Declare a Foo property on the Bar class and autocomplete will know about Bar.findByFoo() automatically. Lots of incremental compilation happening everywhere.
• Lots of improvements in communication between the Java and Groovy compilers. Apparently the old plugin managed the stub compilation with lots of temporary files, so it was constantly writing to and reading from the disk. The new plugin couples the two worlds better and does a lot in RAM, so it should be faster and not thrash the disk so much.
• A working debugger! Conditional breakpoints work, but you have to watch out for all the casting that happens under the hood when checking values. There are still some bugs in setting and removing breakpoints, but it looks good so far. Andrew talked about how he’s still trying to decide how much of the Groovy stack frames and object decorations to show. They show up in IDEA, so you have to wade through many incomprehensibly named caller levels between your own method calls, but it’s all there in case you really want to know.
• The plugin will be available from the same place on The Codehaus as the old plugin and it should be a standard easy install. Once they have something they can release, they’re going to throw it over the fence to be an open source project and encourage outsiders to contribute.
• They fiddle with the Eclipse compiling backend by patching in some behaviour that’s basically “if (groovy) { do new_thing } else { do normal_thing }” so the plugin will work with Eclipse 3.4.2 only. They’ve done a lot of testing to make sure their patch doesn’t hurt normal Java projects, but that version dependency could become frustrating.

Personally, I switched to IDEA because of the debugging support, which works well in both running the application and the tests. I said that we’re pretty evenly split between IDEs at 2Paths and we compare notes every morning in a contest to see whose tools are the most annoying. It looks like this new plugin may push Eclipse back a few points on the annoying scale. Good work, Andrew and company.

As you Grails die-hards know, Grails 1.1 introduced the new GrailsUnitTestCase class which is a very useful extension of GroovyTestCase with support for mocking the dynamic properties and methods that Grails injects into various types of classes (domain, controller & taglibs). Getting the team up to speed on this change was the original purpose of the presentation, but I decided to expand it to cover the full spectrum of testing capabilities in Grails 1.1 for those who might not have much exposure to it previously (including me!)

The presentation covered:

- the benefits & drawbacks of each type of testing: unit testing, integration testing, functional testing.

- details on the new GrailsUnitTestCase and it’s extensions (ControllerUnitTestCase and TagLibUnitTestCase)

- a few “gotchas” – such as using mockDomain with inheritance hierarchies

- WebTest goodness – test organization & re-use, Groovy step, passing params between Groovy & WebTest contexts, a little dive into using HtmlUnit to test AJAX

- general recommendations for what types of tests to create for each artifact (domain classes, controllers, services, etc)

Here’s a link to the presentation on SlideShare.net (where you can view it online or download it):

Happy testing!
Dave

Install memcached, which is simple if you use a Mac and have MacPorts installed.

$ sudo port install memcached

Get the memcached jar , the hibernate-memcached jar and the PostgreSQL JDBC driver jar, then  add them to the project.

$ cp memcached-2.3.1.jar hibernate-memcached-1.2.jar postgresql-8.4-701.jdbc3.jar <project_dir>/lib/

Configure memcached as the second-level cache by adding something like this to your DataSource.groovy.

hibernate {
    cache.use_second_level_cache = true
    cache.use_query_cache = true
    cache.provider_class = 'com.googlecode.hibernate.memcached.MemcachedCacheProvider'
    memcached {
        servers = "localhost:11211"
    }
}

I wrote a simple test to generate a whole bunch of domain objects and shove them in the database, then go back and fetch them all again. I set the test to run the insert/load loops in batches of 1, 10, 100, 1,000, 2,000, 4,000, 7,000, and 10,000 objects. I expected that inserting the objects would take a little longer because of the memcached overhead but that loading the objects would be faster because they could be loaded from memcached instead of going out to the database.

I tested three persistence backend configurations.

• HSQLDB, the default for new Grails projects and when running unit/integration tests. This is an in-memory database only, so data aren’t persisted permanently.
• PostgreSQL for a full-blown RDBMS setup. I set dbCreate to “create-drop” in DataSource.groovy so that each test started with an empty database.
• PostgreSQL with memcached enabled. I should point out that in this test I had both PostgreSQL and memcached running on the same machine (my laptop) as the application, so it’s really not taking advantage of memcached’s parallelism strengths.

My expectations held true for the inserts. The memcached case is a little slower but not that much.

Comparison of object insert times using HSQL, PostgreSQL, and memcached+PostgreSQL for persistence backend

When I tested the load times, I was surprised. It looked like memcached wasn’t speeding things up at all. I was also surprised to see that it took less time to load 10,000 objects from the data store than 7,000 objects. I ran the test a few times to rule out the possibility of a load spike from something else running at the same time, but I got similar results every time.

Comparison of object load times using HSQL, PostgreSQL, and memcached+PostgreSQL for persistence backend

I then tried to improve my test setup by running memcached on a second machine (my desktop, connected by gigabit ethernet to the laptop). Using the PostgreSQL + memcached configuration for each test this time, I looked at three more scenarios.

• PostgreSQL running locally on the laptop, memcached running locally on the laptop.
• PostgreSQL running locally on the laptop, memcached running remotely on the desktop.
• PostgreSQL running locally on the laptop, memcached running on both the laptop and the desktop.

There’s some overhead in going out to the network to retrieve data instead of fetching it from a process running locally, so I expected the memcached local instance to be faster than the memcached remote instance. I expected the third test, with two instances of memcached running in parallel, to be fastest because it splits the load between the two instances and frees up some CPU time on the laptop. Things didn’t play out that way, though.

Comparison of object insert times for various memcache setups

Comparison of object load times for various memcache setups

I have to be honest that these were not great tests. For one, I should really have run those insert/load loops in parallel instead of sequentially. I’m guessing that the test spent a lot of time stalled waiting for data when it could have been sending out more requests. I would also like to test with a large group of memcached servers and a remote database, perhaps even a cluster of app servers with a load balancer to try the full meal deal. The memcached FAQ explains that running everything on one machine will not show off memcached’s scalability.

However, this gave me the opportunity to get my hands dirty with memcached and learn a few things.

• Integrating memcached is so simple and the performance penalty of just running it locally during development is so small that there’s no reason not to enable it.
• It’s dead simple to install and set up. We could , for example, easily install it on a bunch of test servers and run multiple instances on different ports – one for each developer (to prevent us from clobbering each other’s data).
• Measuring performance and scalability realistically is tricky. It’s one thing to hand wave and say that running multiple instances of a database or a cache will speed things up, but it takes some work to set up a realistic simulation of high traffic application performance.
• Adding memcached doesn’t automatically turn performance up to 11. I need to learn more about what’s going on under the hood with my domain objects and how they’re being persisted through memcached.

I look forward to using memcached more in the future and learning how to really take advantage of it.

Update (2009-07-24): Ray pointed out in the comments that I should have enabled caching in my domain objects under test. I added “static mapping = { cache true }” to my domain class definition and re-ran the tests to compare the effect under 4 scenarios.

• no domain object caching (as before)
• domain object caching with memcached running locally
• domain object caching with memcached running remotely
• domain object caching with memcached running both locally and remotely

Not much difference with the inserts.

Comparison of object insert times with domain object caching enabled and disabled

However, the load times are slightly better with memcached enabled during heavy activity! The advantage disappears when I add the overhead of traversing the network, but it’s a start. Thanks for the pointer, Ray!

Comparison of object load times with domain object caching enabled and disabled

PROS

Probably the number one mentioned feature by anyone talking about CouchDB is that it’s schema-free. That means that each document is its own structure, that everything is based on a key-value pairing, where the value is typically a document, but can also be a base type. This is pretty cool when dealing with describing real world objects, as one can be more descriptive and hierarchical in storing the data. Also, updating your application’s domain model is much like evolution, as soon as you need something added to the model, just add it.

CouchDB’s RESTful HTTP service implementation is great for the many applications being created for today’s web. Send Javascript queries/requests to the service, and get JSON formatted data in return. This is great, since many front-end technologies can decipher JSON, and the format itself is even human readable if need be.

There is support for interacting with CouchDB data offline. One can access and edit data while not connected, and when back online can merge and update changes made while offline. This kind of support out of the box makes CouchDB a prime candidate for  web based applications like blogs, wikis, bug reporting, CRM, etc.

Another quality of CouchDB is its MapReduce implementation for handling large data sets. I’m going to redirect you to the author’s notes for this one: MapReduce and Google’s definition.

CONS

Transaction management is non-existent. Relational database managements systems (RDBMS) provide mechanisms for things like transaction management, particularly unique constraints and locking. CouchDB is based on being ‘eventually consistent’ and ‘available’ (See the CAP Theorem section for more), which although great for many things, doesn’t lend itself to transaction management.

Data security is also a concern. A somewhat old article describes this issue quite well: Planned Security Model For CouchDB. The CouchDB team has been working on security and validation but are still early in their development.

To a less important degree, but still worth mentioning, is that personally I really don’t find the out-of-the-box UI for CouchDB to be all that useful. It feels clunky to navigate your key/document pairs, and too feature-light for managing the database.

Conclusion

If you’re looking to create a document based web application, then this may be a worthwhile technology to use. CouchDB has a number of very impressive features. However, for now at least, I believe I’ll be staying off the couch.

Aside:

When testing out CouchDB I went through the process of manually installing the application. I had to resolve the dependencies, and deal with compilation errors on my own. I kind of wish I’d found this site earlier. Follow that link if you’re a MacOSX user and want an installer for CouchDB.

For me, this was pretty much the  only useful tutorial on getting started with CouchDB: Programming CouchDB with Javascript.

Resources:

• http://couchdb.apache.org/
• http://books.couchdb.org/relax/
• http://www.eflorenzano.com/blog/tag/couchdb/
• http://labs.google.com/papers/mapreduce.html
• http://damienkatz.net/2008/02/incremental_map.html
• http://jan.prima.de/~jan/plok/archives/142-CouchDBX-Revival.html
• http://www.automatthew.com/2008/01/planned-security-model-for-couchdb.html
• http://jchrisa.net/drl/_design/sofa/_show/post/couchdb_edge__security_and_vali
• http://jan.prima.de/~jan/plok/archives/108-Programming-CouchDB-with-Javascript.html
• http://www.infoq.com/CouchDB
• http://www.infoq.com/news/2007/11/the-rdbms-is-not-enough
• http://www.infoq.com/news/2008/11/Database-Martin-Fowler
• http://code.google.com/p/couchdb-lounge/wiki/SettingUpTwoCouchInstances