Apache Cassandra
Cassandra is a fairly recent addition to the “no-sql” movement. Initially developed by Facebook, Cassandra was open sourced in 2008 and is now housed and maintained by Apache.

Cassandra is a highly scalable, eventually consistent, distributed, structured key-value store. Cassandra brings together the distributed systems technologies from Dynamo and the data model from Google’s BigTable. Like Dynamo, Cassandra is eventually consistent. Like BigTable, Cassandra provides a ColumnFamily-based data model richer than typical key/value systems.

Cassandra was open sourced by Facebook in 2008, where it was designed by Avinash Lakshman (one of the authors of Amazon’s Dynamo) and Prashant Malik ( Facebook Engineer ). In a lot of ways you can think of Cassandra as Dynamo 2.0 or a marriage of Dynamo and BigTable. Cassandra is in production use at Facebook but is still under heavy development.

Data Model
The key difference between Cassandra and many of the other key-value stores is that it possesses the concept of a “Super Column.” For more information on what exactly a super column is (and an overview of the Cassandra data model in general) read the following blog post “WTF is a SuperColumn? An Intro to the Cassandra Data Model”

Performance
Many may be curious about the performance of these key value stores. We found an article that compared Cassandra to Voldemort

Interacting with Cassandra
Cassandra is developed in java however it does not allow access via JDBC. Instead it provides access to it’s repository via the “Apache Thrift” framework which basically allows developers in any major language to gain access to a given service (in this case the Cassandra data store). For those like me who are used to JDBC, the thrift interface can be a bit confusing given that there isn’t much documentation for them. The Java client made available on the Cassandra website is rather rudimentary with no support for things like transactions, therefore many people choose to make use of a library called Hector that provides many of the features necessary for any production worthy application making use of Cassandra.

The Project
What I wanted to to do was use the Cassandra data store on a pet project that I’m working on in my spare time. I intend to use the datastore to do the following:

1. Store messages being sent from one user to another
2. Track queries made by a given user
3. Store user complaints

Installation
Installation of the server is easy enough. Install Java 1.6 if you haven’t already. Edit /bin/cassandra.in.sh and change JAVA_HOME if you run a different version of java as your default JAVA_HOME. By default the server has an RMI port set to 8080, you can change the value in the /bin/cassandra.in.sh file as well.

Configuration
Edit /bin/storage-conf.xml to define various key spaces and column families. The concept of a Keyspace is similar to the schema in a relational database, where as the ColumnFamily is analogous to a table. In defining the ColumnFamilies you are defining the key groupings and how they are sorted. Therefore, I defined the following:

  <Keyspaces>
    <Keyspace Name="Nikahfied">

      <ColumnFamily Name="Queries"
                    Comment="User queries"/>

      <ColumnFamily Name="Complaints"
                    Comment="User complaints"/>

      <ColumnFamily Name="Messages"
                    ColumnType="Super"
                    CompareWith="UTF8Type"
                    CompareSubcolumnsWith="UTF8Type"
                    RowsCached="10000"
                    KeysCached="50%"
                    Comment="Message threads between users of the system"/>

    </Keyspace>
  </Keyspaces>

I would be creating a schema named Nikahfied and three table are Queries, Complaints and Messages. As you will notice, there is no real structure to the ColumnFamilies other than what type of column and the CompareWith strategy as outlined in the “WTF is a SuperColumn? An Intro to the Cassandra Data Model” article. In this case we have one super column and two regular columns. In code I will ensure that each of the rows fore each ColumnFamily contain the same content however there is nothing forcing you to do so. As you will notice, we can configure each of the ColumnFamiles individually. Messages will need to handle heavy reads and writes where as Queries and Complaints will be more write intensive. Messages therefore have a caching scheme defined where as the others do not as they will primarily handle writes rather than reads.

When initially defining the column structure I was a bit confused. I was taking the definition of Cassandra as a “key value store” literally and was confused about there being a primary key and the secondary keys. When defining a basic Column I though it would simple have a key and a value. So things would be defined as Nikahfied.<ColumnFamily>.key = value where this is actually not the case. For a basic ColumnFamily, the structure is as follows: Nikahfied.<ColumnFamily>.<Primary key>.<Secondary Key> = value. In the case of a super column, this goes one level further Nikahfied.<ColumnFamily>.<Primary key>.<Secondary Key>.<Tertiary Key> = value. That had me puzzled for some time…upon re-reading the “WTF is a SuperColumn? An Intro to the Cassandra Data Model” article I realized my mistake so I though I’d point it out to anyone else who was confused about this construct.

Data Structures:
Queries:
The structure of queries would be simple enough. I would store each query by date/time. To track which queries the user made when.

Nikahfied.Queries.UserId.<Date/Time> = <Query>

Complaints:
There is only one type of complaint wherein a user can complain about another user. So I wanted to track who received the complaint, who complained and when. The structure of a complaint would be defined as follows:
Nikahfied.Complaints.<Violator Id>.<Complainant Id> = <Date/Time>

Messages:
If there was ever a case to use a super column, a messages would be it. A user has a number of threads that contain one or more message. A user can receive and send messages. Messages themselves are the same, I simply need to track which threads contain messages that a user sent. I also need to keep track of which threads have unread messages within them. My initial design was the following:
Nikahfied.Messages.RecipientId.threads = A JSON Object that contains Threads defined in JSON Objects themselves. A thread being a group of
.sent_ids = An array of thread id's that contain messages that the user has sent
.unread_ids = An array of thread ids that contain unread messages

However, with this structure there was no need for a super column and I was determined on making use of one . So I expanded out the thread structure as follows:

Nikahfied.Messages.RecipientId.threads.<thread id> = A JSON Object that contains an individual Thread
.sent_ids = An array of thread id's that contain messages that the user has sent
.unread_ids = An array of thread ids that contain unread messages

Now that we have the content structure defined I now need to create a hook between my application and the Cassandra key value store. For development I have decided to use the Grails Framework to speed up development. I have included the Hector library to access the framework but needed to define a CassandraService to be the central point of contact between various controllers and other services to the Cassandra key store. Both Cassandra’s Thrift interface and Hector have pretty poor documentation so I created a Grails service that makes the interaction with Cassandra a bit clearer (at least to me). Note: You will need to be using Java 1.6.

import me.prettyprint.cassandra.service.CassandraClientPool
import me.prettyprint.cassandra.service.CassandraClientPoolFactory
import me.prettyprint.cassandra.service.CassandraClient
import me.prettyprint.cassandra.service.Keyspace
import org.apache.cassandra.thrift.Column;
import org.apache.cassandra.thrift.ColumnPath;
import org.apache.cassandra.thrift.NotFoundException;

import org.apache.cassandra.thrift.SuperColumn

class CassandraService {

    boolean transactional = true

    def servers=["localhost:9160"]
    def defaultKeyspace="Nikahfied"
    private static final String NOT_FOUND = ""

    private execute(keyspaceName=defaultKeyspace,block){
        CassandraClientPool pool = CassandraClientPoolFactory.INSTANCE.get();
        CassandraClient client = pool.borrowClient(servers);

        try {
            Keyspace keyspace = client.getKeyspace(keyspaceName)
            return block(keyspace)
        } finally {
            pool.releaseClient(client);
        }
    }

    /**
     * Get a single super column
     * @param columnFamily
     * @param secondaryKey
     * @param key primary key
     * @return Matching super column
     */
    public SuperColumn getSuperColumn(String columnFamily, String key, String secondaryKey) {
        ColumnPath cp = new ColumnPath(columnFamily)
        cp.setSuper_column(secondaryKey.bytes)
        return execute {Keyspace keyspace ->
            SuperColumn sc = null;
            try {
                sc = keyspace.getSuperColumn(key, cp)
            } catch (NotFoundException nfe) {
                sc = null;
            }
            return sc
        }
    }

    /**
     * Get multiple super columns
     * @param columnFamily
     * @param secondaryKey
     * @param keys primary keys
     * @return matching super columns
     */
    public Map multigetSuperColumn(String columnFamily, List keys, String secondaryKey) {
        ColumnPath cp = new ColumnPath(columnFamily)
        cp.setSuper_column(secondaryKey.bytes)
        return execute {Keyspace keyspace ->
            Map scMap = keyspace.multigetSuperColumn(keys, cp)
            return scMap
        }
    }

    /**
     * Get multiple columns
     * @param secondaryKey secondaryKey
     * @param keys primary keys
     * @param columnFamily column family
     * @return results if any
     */
    public Map multigetColumn(List keys, String secondaryKey, String columnFamily) {
    	ColumnPath cp = new ColumnPath(columnFamily)
        cp.setColumn(secondaryKey.bytes)
        return execute {Keyspace keyspace ->
            Map cMap = keyspace.multigetColumn(keys, cp)
            return cMap
        }
    }

    /**
     * Get a single column and it's values
     * @param columnFamily
     * @param key primary key
     * @return matching column
     */
    public Column getColumn(String columnFamily, String key, String secondaryKey) {
        def cp = new ColumnPath(columnFamily)
        cp.setColumn(secondaryKey.bytes)
        return execute {Keyspace keyspace ->
            Column c = null;
            try {
                c = keyspace.getColumn(key, cp)
            } catch (NotFoundException nfe) {
                c = null;
            }
            return c
        }
    }

    /**
     * Sets the new value for this column path
     * @param cf Column family
     * @param secondaryKey secondary key
     * @param key primary key
     * @param value value
     */
    def setColumnPathValue(String cf, String key, String secondaryKey, String value){
        def cp = new ColumnPath(cf)
        cp.setColumn(secondaryKey.bytes)
        return setColumnValue(cp, key, value)
    }

    /**
     * Sets the new super column value for this column path
     * @param cf Column Family
     * @param sc Super Key Id
     * @param secondaryKey Secondary Key
     * @param key primary key
     * @param value value
     * @return the value of the Column requested
     */
    def setColumnPathValue(String cf, String sc, String key, String secondaryKey, String value){
        def cp = new ColumnPath(cf)
        cp.setSuper_column(sc.bytes)
        cp.setColumn(secondaryKey.bytes)
        return setColumnValue(cp, key, value)
    }

    /**
     * Set a regular column
     * @param cp Column path
     * @param key primary key
     * @param value value
     */
    private setColumnValue(ColumnPath cp, String key, String value){
        return execute{ Keyspace keyspace ->
            keyspace.insert(key, cp, value.bytes)
        }
    }

    /**
     * Batch insert content for a give key and it's values
     * @param key primary key
     * @param columnMap column key values
     * @param superColumnMap super column key values
     */
    public batchInsert(String key, Map<String, List> columnMap, Map<String, List> superColumnMap) {
    	return execute{ Keyspace keyspace ->
        	keyspace.batchInsert(key, columnMap, superColumnMap)
    	}
    }

}

Using this class I have created a MessageService, ComplaintsService and a QueriesService that interact with the CassandraService to retrieve and store data. There is an adjoining Message and Complaints controller where as the QueriesService is referenced by the SearchController. Hopefully this blog post will provide you with a better understanding of how to get started with Cassandra the CassandraService class should provide you with a jump start on integrating Cassandra into an existing Grails or Java project.

RDF

For those who are unfamilar with RDF a short description would be:

The Resource Description Framework (RDF) is a language for representing information about resources in the World Wide Web. This Primer is designed to provide the reader with the basic knowledge required to effectively use RDF. It introduces the basic concepts of RDF and describes its XML syntax. It describes how to define RDF vocabularies using the RDF Vocabulary Description Language, and gives an overview of some deployed RDF applications. It also describes the content and purpose of other RDF specification documents.

For more information about RDF read the following primer hosted on the W3C’s website.

RDF Stores

An RDF store allows for flexible definition of content which is a shortcoming of traditional relational database. An example of this would be the need to add a field within a database. In a relational database one would have to define a new column in the database table which is not usually achievable from a program given that most systems are not foolish to allow database changes from code. However, in an RDF data-store things are stored as nodes and relationships between nodes. Therefore to add a new field simply requires that a new node be created and an establishment of a relationship between the existing nodes and the new node. This flexibility is extremely handy in cases where the data being captured by a system changes often. Add to this the creation of an RDF schema or an ontology and one has the ability to semantically query the data you have within your data-store and are able to make use of meta-data in a way that can not be done within a relational database.

I was thinking that this type of flexibility would be of great benefit for a project we hope to be working on in the new year. Given Zephira’s backing of Mulgara I thought I would investigate whether it could be ready for prime time and the level of complexity required in using it.

Mulgara

Mulgara was initiated in 2006 and was a fork of the Kowari project which died or became unsupported as of 2005. Their claim is that Mulgara has the following featured:

• Native RDF support
• Multiple databases (models) per server
• Simple SQL-like query language
• Small footprint
• Full text search functionality
• Datatype support
• Supports and tracks W3C Specifications and guidelines
• Large storage capacity
• Optimized for metadata storage and retrieval
• Multi-processor support
• Independently tuned for both 64-bit and 32-bit architectures
• Low memory requirements
• On-disk joins
• Streamed query results

The remainder of this post will cover my experiences with using Mulgara.

Rough Start

Unfortunately things didn’t start off well. Though the downloading and installation of the Mulgara server was painless there were a number of issues that I came across that would deter many from considering the use of it in a production level system:

Connectors are not included in the default download

Currently there are only a few means of accessing Mulgara to perform CRUD operations:

• A JRDF connector
• A Jena connector
• Straight RMI (remote method invocation)

In the default download from the Mulgara website, the JRDF and Jena connectors are not provided. After a lot of struggling, I resorted to downloading the source and building the server from scratch to ensure that the connectors were included within the jar files.

After getting to this stage I tried to follow the tutorials to try and connect and interact with the server. It seems that the documentation on the mulgara website is stale and that they have not been able to keep their tutorials up to date because the code provided in them does not work unless one uses an in-memory database which is not specified in the tutorial. I posted questions to the various user group email lists and receive no responses.

Once I figured those aspects out things went more smoothly though there is a significant amount of boiler plate code that needs to be created to allow for simple CRUD operations.

Creating a connection to the server

As noted above, one can not simply use JDBC to connect to the server but have an option of using the majority of the popular RDF creation frameworks such as JRDF, Jena or Sesame. I initially tired to use Jena but it seems that support for the JenaMulgara connector died quite a while ago and the connector does not work for the latest version of Mulgara. Therefore, I moved on to the JRDF connector given that the Sesame connector is quite immature.

// Create the URI of the server
java.net.URI serverURI = new java.net.URI("rmi", hostName, "/" + serverName, null);

// Create a new session factory, ensure that it’s local
SessionFactory sessionFactory = SessionFactoryFinder.newSessionFactory(serverURI, false);

// Get a local JRDF Session (local)
Object o = sessionFactory.newJRDFSession();
org.mulgara.server.JRDFSession session = (JRDFSession) sessionFactory.newJRDFSession();
[/sourcecode]

Create a model/database

The above give you a connection to the Mulgara server. To create a model/database requires the following section of code:

java.net.URI modelURI = new URI("rmi", hostName, "/" + serverName, graphName);
java.net.URI modelType = new URI("http://mulgara.org/mulgara#Model");
session.createModel(modelURI, modelType);
org.jrdf.graph.Graph graph = ClientGraph(createGraphProxy(modelURI, session));

Create nodes/relationships

In order to create RDF triples one needs to specify a subject, predicate and an object. There are three types of Java objects that can be used in this in building these relationships:

1. BlankNode – a blank node that is a node used to group other relationships. A BlankNode can be used as either a subject or an object
2. Literal – a literal value of some sort (String, number, etc.) A literal can only be used as an object and not a subject or predicate
3. URIReference – a URIReference is used to define the predicate or relationship between the subject and the object.

org.jrdf.graph.GraphElementFactory elementFactory = graph.getElementFactory();
org.jrdf.graph.BlankNode blanknode = elementFactory.createResource();
org.jrdf.graph.URIReference predicate = elementFactory.createResource(new URI());
org.jrdf.graph.Literal literal = elementFactory.createLiteral(value);

Lastly to create and insert a triple one simply needs to do the following:

org.jrdf.graph.Triple triple = elementFactory.createTriple(subject, predicate, object); // Create the triple object
graph.add(triple); // Store the triple in Mulgara server

For sample code on how to do everything from connect to query and delete see the end of this post.

Performance

I wrote a simple application that took the contents of a relational database and converted it into RDF and stored it in Mulgara. The database I was using was big but not huge, roughly …. All these numbers are based on running the application on a MacBook Pro with a 2Gh Core Duo processor and 2GB of RAM. I initially wrote the app to do inserts one at a time which was obviously inefficient but I wanted to test out the speed of an insert. Each insert of a single triple took roughly 0.17 seconds. In round 2 I started doing inserts in batch. In batch mode it seems that inserts took roughly 0.08 seconds per insert of a node. Both speeds are not particularly fast but like I said, this is running off my laptop so one can’t expect superb performance.

Conclusion

Given that RDF stores are competing with regular relational databases I would have hoped that there was an easier means of connecting to the database and that RM wasn’t being used under the hood. Given the push for RESTful interfaces within the semantic world, I was surprised to see that the Mulgara server does not have a RESTful interface which would relieve the need for RMI completely. Once I got over my dependence on the abhorrent tutorials and documentation things went fairly well though I have performance concerns when dealing with large volumes of data. The main area of concern for me is the requirement of RMI which has been well documented as being not very performant. It pains me to have to use RMI when running both the server and the application on the same server. What is nice is that the server is completely transactional and any failure results in a roll-back. The last concern I have is the fact that the user community/group around Mulgara does not seem to be very active or dedicated to ensuring that the product is supported to the extent that one can rely on getting up to date documentation and a steady stream of bug fixes.

Sample Code

The following code sample shows how to perform inserts, updates, deletes and selection.

package org.twopaths.jrdf;

import java.net.InetAddress;
import java.net.URI;
import java.net.URISyntaxException;
import java.net.UnknownHostException;

import org.jrdf.graph.BlankNode;
import org.jrdf.graph.Graph;
import org.jrdf.graph.GraphElementFactory;
import org.jrdf.graph.GraphElementFactoryException;
import org.jrdf.graph.GraphException;
import org.jrdf.graph.Literal;
import org.jrdf.graph.Triple;
import org.jrdf.graph.URIReference;
import org.jrdf.util.ClosableIterator;
import org.mulgara.client.jrdf.AbstractGraphFactory;
import org.mulgara.query.QueryException;
import org.mulgara.server.JRDFSession;
import org.mulgara.server.NonRemoteSessionException;
import org.mulgara.server.SessionFactory;
import org.mulgara.server.driver.SessionFactoryFinder;
import org.mulgara.server.driver.SessionFactoryFinderException;

public class Sandbox {
	public static void main(String[] args) {
                Graph graph = null;
		try {
			// Create the host name
			String hostname = InetAddress.getLocalHost().getCanonicalHostName();

			// Create the URI of the server
			URI serverURI = new URI("rmi", hostname, "/" + "server1", null);

			// Create a new session factory, ensure that it's local
			SessionFactory sessionFactory = SessionFactoryFinder.newSessionFactory(serverURI, false);

			// Get a local JRDF Session (local)
			Object o = sessionFactory.newJRDFSession();
			System.out.println(o.getClass().getName());
//			LocalJRDFSession session = (LocalJRDFSession) sessionFactory.newJRDFSession();
			JRDFSession session = (JRDFSession) sessionFactory.newJRDFSession();

			//create a new Model
			URI modelURI = new URI("rmi", hostname, "/" + "server1", "exampleGraph");
			URI modelType = new URI("http://mulgara.org/mulgara#Model");
			session.createModel(modelURI, modelType);

			//create a JRDF Graph for the model
//			graph = new JRDFGraph(session, modelURI);
			graph = AbstractGraphFactory.createGraph(serverURI, modelURI);

			//get the Factory
			GraphElementFactory elementFactory = graph.getElementFactory();

			//create resources
			URIReference person = elementFactory.createResource(new URI("http://example.org/staffid#85740"));
			BlankNode address = elementFactory.createResource();

			//create properties
			URIReference hasAddress = elementFactory.createResource(new URI("http://example.org/terms#address"));
			URIReference hasStreet = elementFactory.createResource(new URI("http://example.org/terms#street"));
			URIReference hasCity = elementFactory.createResource(new URI("http://example.org/terms#city"));
			URIReference hasState = elementFactory.createResource(new URI("http://example.org/terms#state"));
			URIReference hasPostCode = elementFactory.createResource(new URI("http://example.org/terms#postalCode"));

			//create values
			Literal street = elementFactory.createLiteral("1501 Grant Avenue");
			Literal city = elementFactory.createLiteral("Bedford");
			Literal state = elementFactory.createLiteral("Massachusetts");
			Literal postCode = elementFactory.createLiteral("01730");

			//create statements
			Triple addressStatement = elementFactory.createTriple(person, hasAddress, address);
			Triple streetStatement = elementFactory.createTriple(address, hasStreet, street);
			Triple cityStatement = elementFactory.createTriple(address, hasCity, city);
			Triple stateStatement = elementFactory.createTriple(address, hasState, state);
			Triple postCodeStatement = elementFactory.createTriple(address, hasPostCode, postCode);

			// Add triples to graph
			graph.add(addressStatement);
			graph.add(streetStatement);
			graph.add(cityStatement);
			graph.add(stateStatement);
			graph.add(postCodeStatement);

			//get all Triples
			Triple findAll = elementFactory.createTriple(null, null, null);
			ClosableIterator allTriples = graph.find(findAll);
			while (allTriples.hasNext()) {
				System.out.println(allTriples.next().toString());
			}

			//search for address (as a subject)
			Triple findAddress = elementFactory.createTriple(address, null, null);
			ClosableIterator addressSubject = graph.find(findAddress);
			while (addressSubject.hasNext()) {
				System.out.println(addressSubject.next().toString());
			}

			//search for the city: "Bedford"
			Triple findCity = elementFactory.createTriple(null, null, city);
			ClosableIterator bedfordCity = graph.find(findCity);
			while (bedfordCity.hasNext()) {
				System.out.println(bedfordCity.next().toString());
			}

			//search for any subject that has an address
			Triple findAddresses = elementFactory.createTriple(null, hasAddress, null);
			ClosableIterator addresses = graph.find(findAddresses);
			while (addresses.hasNext()) {
				System.out.println(addresses.next().toString());
			}

		} catch (UnknownHostException uhe) {
			uhe.printStackTrace();
		} catch (URISyntaxException urise) {
			urise.printStackTrace();
		} catch (SessionFactoryFinderException sffe) {
			sffe.printStackTrace();
		} catch (NonRemoteSessionException nrse) {
			nrse.printStackTrace();
		} catch (QueryException qe) {
			qe.printStackTrace();
		} catch (GraphException ge) {
			ge.printStackTrace();
		} catch (GraphElementFactoryException gefe) {
			gefe.printStackTrace();
		} finally {
                        try {
                            graph.close();
                        } catch (Exception e) {
                            e.printStackTrace();
                        }
                }
		System.out.println("DONE");
	}
}

• CouchDB – Document oriented database
  
• Memcached – Distributed Hashmap
• MemcacheDB – A database built as on Memcached and BerkleyDB
  
• Drizzle – Lightweight SQL Database for the cloud/web
• GearMan – Distributed processing framework
  
• MogileFS – Distributed file system
  

I intend to spend some time investigating each of these pieces of technology in the coming months with the hope of making use of them when and where applicable.

Joe Stump from Digg.com was up next and his session was awesome. He talked about the evolution of their architecture from the point he joined to what they are planning on rolling out early next year along with some of the bumps they’ve encountered. MySQL’s scalability had major issue so they moved to IDDB and are considering using MemcacheDB which is MemcacheD mixed with Berkley DB.  Joe pointed out their use of MogileFS for the management of images on digg images which I will be taking a look at for our current project. Joe, you rock, maybe you can give some pointers to Dan…then again hopefully MySpace will send somone with a clue next year.

Aditya Agarwal from facebook was great. He talked about all the different aspects of facebook and the use of PHP/MySQL being the underlying architecture and how for them it . They tend to use MySQL simply as a key value store and profile content is randomly scattered across their thousands of servers. They use LAMP for the most part but have created a framework that allows them to use services written in other languages within their application.

The discovery of the day was CouchDB. One of the lead developers/designers was sitting at the same table as me during the Digg session. Tim Bray was quite excited about the technology as well having mentioned it in his keynote on day 2. I attended one of two sessions about that covered what it’s about from a high level. It has it’s roots in Lotus Notes database design given that it was created by one of the people who worked on Lotus Notes in the past. It has some very cool features:

• RESTful so it doesn’t require any connectors
• Stores all content in JSON
• Querying the database is done via javascript functions for performing map reduce like functionality
• Very efficient and performs versioning and synchronization when distributed

Jan made some good arguments was to why it has some strengths that traditional RDB’s don’t such as no need for schema management, etc. It’s a technology that I plan on taking a serious look at and may possibly use in the future.

In conclusion, QCon was a great experience. The hosts tried their best to make sure things were as tech focused as possible. There is a need to ensure that speakers cater their presentations to the audience and keep to the subject of the presentation. The lagging economy did have an effect on the conference, there were 200+ people that had been registered that didn’t attend (you could see all of the uncollected badges at the registration desk) and if the economy improves I think that next year will be even better. It’s truly a conference for developers by developers which I truly enjoyed. It was great to meet several of the authors of my favourite tech books. I would have preferred more opportunities to network with the people at the conference, an overall guest list would have helped with that or a formal networking event given that a large percentage of the nerds there were stereotypical introverted nerds who didn’t like having a strange person walk up to them and make conversation. Maybe I came off as sleazy or creepy or a threat to homeland security . Regardless, I hope to attend next year.

Update: This blog post has been featured on the infoq.com website

Next was a panel discussion on scalability and emphasized the need to understand the update requirements of the system you are designing. They proposed that there are two types of design:

1. Design for scalability up front
2. Design for simplicity and deal with bottle-necks as they pop-up

I’m not sure if I’d ever go with option #2 but they made a good point about how many architects design for things to be bullet proof when one really doesn’t really have the requirement for it to be bulletproof. I fully agree with that principle. There was mention of several websites such as flickr that have made some very good decisions on having that happy medium

The next talk was about the Java 7 concurrency library and the need for making things concurrent given the fact that processors are not getting faster but are moving towards multiple cores. Therefore, we as developers need to ensure that we ensure our applications are making everything run in parallel (concurrent). One might feel that if you are running a website that since every request is being handled by a different thread your job is complete. However, how many of us receive enough requests 24*7 to keep all the cores busy. Therefore, we need to make sure that applications is decomposed to the extent that we are doing everything concurrently that can be done concurrently to ensure maximum CPU utilization and decreased response time and therefore an improved user experience.

The last session of interest was a panel discussion on the affect that the open-source movement has and on java. It was the consensus that the open-source saved java from mediocrity and J2EE from certain death. The focus of the discussion moved to open-source during the current economic condition. The panel felt that slashed budgets within most companies would allow open-source to flourish, at least those that are well established. The last point of interest was about how they make a decision within their companies about what to open-source and what to charge for. Bob Lee of Google said that they open-source low level tools and frameworks and charge for things they build upon them. Others such as a guy from MuleSource and Rod Johnson from Spring Source said that they charge when people are already likely to be paying other vendors for their product/services. For example, the quote of the night was from Rod Johnson

If you use MySql, tomcat and Apache as your application stack then you can use spring for free and that’s great. However, if you are using Oracle Rack for a database, BEA Weblogic for an app server then you have no right to complain when we charge you for a Spring to Oracle RACK connector.

The point being if you’re already spending money then why shouldn’t we ask to be paid for what we are offering.

Tomorrow has another set of very interesting streams and discussions. I’m looking forward to the stream called “Architectures you’ve always wondered about.”

Update: This blog post has been featured on the infoq.com website

It made me think about our policy here at 2paths, everyone codes to some extent. As an architect myself, I may not code large volumes of code but I do develop code. This makes my opinion relevant when we make design decisions. Ironically, their suggestion to fix the situation was just this, architects must be part of the dev team and need to look at the code developed/checked-in on a daily/weekly basis and in the ideal case code.

The great thing about thing about this conference is the big name people they have as speakers. The speaker to attendee ratio is 1:4. Every big name author of the various tech books are here:

• Martin Fowler
• Rod Johnson
• Kent Beck
• Geagor Hohpe
• etc…

These guys are treated link rock-stars here. At the keynote, the guy sitting next to me was giddy to be in the same room as Martin Fowler .The thing is that just because they are smart and wrote great books it doesn’t mean that they give good presentations/talks. Point in case, Kent Beck. I went to two talked by Kent and though he’s funny and witty the talks were very convoluted and definitely not to the point.  That being said, both sessions were packed…people were sitting on the floor…it was surreal.

Gregor Hohpe gave a great discussion about principles underlying the creation of appkications that use a “cloud” architecture. Cloud computing is basically the use of services from other organizations/companies within your application. Examples of this would be to have an application that uses google maps or Amazon’s EC2, etc. The main take-aways from his talk where:

1. Learn to live with uncertainty
   The services you are using are not controlled by you so you have to design for the component being down/unavailable because your customers don’t care what services you are using under the covers.
2. Keep things simple and small
3. Learn to properly design for asynchronous
4. Embrace the new programming model
5. Resist applying traditional patterns

Point 1 was the most relevant and leads to a redefinition of ACID transactions

A – Associative
Service calls/requests need to be associative. Ie (A + B) + C = A + (B + C)

C – Commutative
Service calls/requests need to be commutative. ie A + B = B + A

I – Idempotent
One needs to design services to be able to handle a request or response that is received more than once due to re-sends due to delay, etc…

D – Distributed
He admitted that this was a filler since applications built on the cloud are distributed by their very nature.

All in all day one didn’t meet my expectations entirely though there were flashes of what I expected but I have hope that tomorrow and Friday meet will be better.

Update: This blog post has been featured on the infoq.com website

Scrum is about providing transparency in the development process to all involved parties. What is emphasized is that you can’t really know how long it will take you to do something because the software development process is such that we can only provide best guesses. However, clients being clients, they want predictability. Unfortunately, we have given our clients the false expectation that we can provide predictability when we all know that there is no way we can. One can never know exactly what will take place in a project. Features, requirements, human issues, etc. are never 100% known so how can you say you can get this done by a specific date guaranteed unless you pad your date like crazy and even then there is no guarantee?

Therefore, the point of agile as a whole is to educate and inform all parties involved (especially the client) about this reality. We can reasonably say that it is feasible that we could complete the project by a given date but we can’t give a 100% guarantee and therefore, if the application is mission critical to be complete by that date that they the client should make contingency plans just in case we ca not make the given date.

Anyway, we went through an exercise that took place 1.5 days into the 2 day training course. We had gone over all the various concepts behind scrum, Ken had guided us through several exercises about the various concepts within scrum. So we began to do a mock project. We received our requirements, created highlevel stories, provided storypoints and person-day estimates for each of the stories and then proceeded to determine if we could get the work done by march 31st which was the hard deadline for the mock client.

In the end we were asked to present our estimates team by team to our mock client. This being a test, Ken started with the first team who put into practice what we had learned over the past 1.5 days.

Given what we know about the application and the requirements provided, we think we could possibly provide you with a complete solution for you by the march 31st deadline. We will need to hire a few more developers to ensure we meet the deadline but it seems possible.

Ken began to push them by asking questions and prodding us to commit to completeing by March 31st, 2009. Many resisted and other gave absolutely no resistance. However, in the end, the vast majority of the teams relented and committed to getting it done by the given date. At which point Ken took the shovel we used to dig our graves with and hit us over the head with it. Paraphrasing Kens words:

Even after one and a half days of learning and making fun of people that made stupid mistakes with just a little bit of pushing you all forgot everything! You went back to making promises that you know you can’t keep.

The room got scilent, others who stupidly choose to defend their actions were hit over the head again and again until they accepted their error. Ken himself admitted that even ave 16 years of using and teaching Scrum, he still falls back on these deeply ingrained bad habits and that it takes concious effort not to regress to old habits.

The course was the best course outside of university that I have ever taken. If you plan on doing a scrum master certification course, make sure you take it with him regardless of the cost/location. I learned an emmence amount from him and have found a larger passion on promoting the principles behind scrum to the people within our company and abbroad.

There was one vendor showcase in particular that I got a great laugh out of. I was sitting in the very front row, right in front of the presenter who was about 4 feet in front of me. Before I go on I should give you some information about myself, I am a Muslim who looks very Muslim…ie the full beard, middle eastern looking, etc… I, like all the other attendees has a name tag on with my full name written in large print which is visible from several feel away. So he starts his talk about the tool that they are selling. Mid way through the presentation he starts a demonstration on how the tool was used to prove that two people who claimed to not know each other did in fact know each other and that they were smuggling funds between them.

As he went through this presentation, he paused, saw me and my name and became really uncomfortable. You see, the names of the people he was talking about had the exact same last name as me and I don’t think he really saw me sitting there until that point. At that point he tried really really hard to come up with another example to demonstrate the tool but for the life of him he could not. I had to fight the urge not to stir the pot to see how much more uncomfortable I could make him feel.

Regardless, you know this technology is going to take off when the US department of defense is interested in it. It seems that they are dumping large sums of money into it’s development and use.

1. Linguistics.
   • the study of meaning.
   • the study of linguistic development by classifying and examining changes in meaning and form.
2. Also called significs. the branch of semiotics dealing with the relations between signs and what they denote.
3. the meaning, or an interpretation of the meaning, of a word, sign, sentence, etc.: Let’s not argue about semantics.

What we are talking about here is devising a means for a computer to be able to understand the meaning of language. Once it can understand language. When you enter a question into Google such as “What is the best way to exercise your lower abs?” google goes through and analyzes the phrase/sentence you entered into the search box and breaks it up into keywords. It then uses these keywords to search their repository for the information within a web page that best matches the keywords you entered. Therefore, in the example I provided above, it will go through and extract words like ‘best’, ‘exercises’, ‘lower’ and ‘abs’ and use these keywords to search their repository and return matches that have a high likelihood of containing the information you’re looking for. Google has little to no idea what the sentance “What is the best way to exercise your lower abs?” means.

What semantics hopes to archive is to extract information from the internet such that a computer or set of computers could perform a search understanding exactly what “What is the best way to exercise your lower abs?” means and would therefore return search results matching exactly what you are looking for. In the above example it’s fairly clear what I’m looking for but what if I entered something like “paris hilton”? What exactly am I looking for? Am I interested in the Hilton hotel located in Paris, France or am I interested in information about the trash TV celebrity? By now I’m sure you get the idea.

Needless to say, getting a computer to understand meaning isn’t an easy task and therefore the community that is hoping to make semantics a reality holds two conferences a year on the topic to gather like minds and to showcase how the technology has progressed along with any real world uses of the technology. The conference has been held for the past four years and has grown steadily each and every year. This year there were over 1100 attendees which is remarkable given the downturn in the US economy. The remainder of this post will outline the technologies of interest and some real implementations of the technology.

Technologies and Specifications

Gleaning Resource Descriptions from Dialects of Languages (GRDDL)
GRDDL is a W3C specification to allow people to make their content semantically accessible. For example, one can have a website containing all sorts of data within it that may or may not be structured. The GRDDL specification allows one to embed a link to an XSLT transform that will take the XHTML of the webpage as an input and output a set of RDF triples representing the content of your page.

Sesame
Sesame is an open source RDF framework with support for RDF Schema inferencing and querying. It can be deployed on top of a variety of storage systems (relational databases, in-memory, file systems, keyword indexers, etc.), and offers tools to developers to leverage the power of RDF and RDF Schema.

ELMO
Elmo is a toolkit for developing Semantic Web applications using Sesame. Elmo wraps Sesame, providing a dedicated API for a number of well known web ontologies including Dublin Core, RSS and FOAF. The dedicated API makes it easier to work with RDF data for the supported ontologies. Elmo also offers a set of tools related to the supported ontologies, including an RDF crawler, a FOAF smusher and a FOAF validator.

JENA
JENA is a Semantic web framework that can be used to access RDF, RDFS, OWL and SPARQL and a rule based inference engine. Included within the framework is an RDF API, an OWL API and a SPARQL query engine.

Topaz
Topaz is a RDF persistence and querying service framework.

NetKernel
1060 NetKernel is a resource oriented microkernel and RESTful application server created from the convergence and unification of the powerful fundamental concepts found in the World Wide Web and Unix. NetKernel is being used by a large number of major semantic applications.

Persistant URLs
A PURL is a Persistent Uniform Resource Locator. Functionally, a PURL is a URL. However, instead of pointing directly to the location of an Internet resource, a PURL points to an intermediate resolution service. The PURL resolution service associates the PURL with the actual URL and returns that URL to the client. The client can then complete the URL transaction in the normal fashion. In Web parlance, this is a standard HTTP redirect.

Implementations

The following are a list of applications that have been created using semantic technologies in part or as a whole. Examples of implementations such as these provide direction as to where this technology can be used to solve real world problems.

Powerset
Powerset is a website that has sucked in the content from Wikipedia and Freebase. They have taken this content and have translated it into a very very very large RDF store using NetKernel (or so I believe). Therefore, using their interface, one can search the data using semantic technology. If one is doing research on a given topic and has confidence in the data contained within Wikipedia, then using powerset to do your research would be of great use. Search result and content related to a particular item are easily retrivable using powerset. For example, if I were researching the pouring of concrete and were to enter “How do you pour concrete” you would see a result of all content that matches the information you are looking for understanding as best they can the meaning of what you are asking. Upon selecting the first result you would not only be given information about concrete and a slew of related content.

Twine
Twine is a social network focused on sharing your interests. In other words, creating a grouping of people interested in the same stuff. It is currently in private beta but hopes to go into public beta sometime in the late summer/early fall. The concept revolves around the idea that if you are interested in a certain topic, twine can offer information/articles/etc that fit within that area of interest. This is where the use of semantic technology fits in. They use semantic technology to search for and filter content related to what you are discussing and so on within your group. I was extremely disappointed with the fact that they made no attempt to showcase their technology by providing a walk-through of their application, but rather spent the entire time talking high-level about the application and showing a stupid video created about the website by some pre-teen. A real waste of an opportunity.

Garlik
Garlik is the darling of the semantic world. They are an example of where semantic technology can be used to solve real world problems. In this case, the use semantic technology to help protect your identity online. Currently they offer their services within the UK. Information is sketchy as to how exactly they do this using semantic technologies. I was really disappointed that they sent a non-techy to speak about their product but that seemed to be the running theme of the these types of companies within the conference, keep your secrets guarded regardless of whether it would further the proliferation of the technology.

Zepheira Remix
Now this tool is something that follows along the lines of Garlik in that it provides a solution to a real world problem. In this case they are focusing on sharing data. They have created a tool that significantly reduces the complexity of using semantics to share data. Remix allows you to import two files and merge, query, generate content between them with no need to understand anything about semantics or even the underlying technology. Remix features an AJAX front end that allows users to massage the data as they see fit and extract a meaningful summary of the data in an HTML format. Zepheira’s goal is to advocate for the use of semantic technologies while encouraging others to attempt to reduce the barriers of entry to harnessing the power of the technology. ie lets make it easier for people to use semantics through the use of frameworks and applications.

All in all, the conference was great. I have to admit, before attending the conference I was a skeptic about the technology but after the conference I think there are some great possibilities for this technology in the very near future. I look forward to attending next year. The only thing that I believe would improve the conference is an increase in details about the technology and/or implementation/usage of the technology rather than just skimming over things at a very very high level.