RDF

For those who are unfamilar with RDF a short description would be:

The Resource Description Framework (RDF) is a language for representing information about resources in the World Wide Web. This Primer is designed to provide the reader with the basic knowledge required to effectively use RDF. It introduces the basic concepts of RDF and describes its XML syntax. It describes how to define RDF vocabularies using the RDF Vocabulary Description Language, and gives an overview of some deployed RDF applications. It also describes the content and purpose of other RDF specification documents.

For more information about RDF read the following primer hosted on the W3C’s website.

RDF Stores

An RDF store allows for flexible definition of content which is a shortcoming of traditional relational database. An example of this would be the need to add a field within a database. In a relational database one would have to define a new column in the database table which is not usually achievable from a program given that most systems are not foolish to allow database changes from code. However, in an RDF data-store things are stored as nodes and relationships between nodes. Therefore to add a new field simply requires that a new node be created and an establishment of a relationship between the existing nodes and the new node. This flexibility is extremely handy in cases where the data being captured by a system changes often. Add to this the creation of an RDF schema or an ontology and one has the ability to semantically query the data you have within your data-store and are able to make use of meta-data in a way that can not be done within a relational database.

I was thinking that this type of flexibility would be of great benefit for a project we hope to be working on in the new year. Given Zephira’s backing of Mulgara I thought I would investigate whether it could be ready for prime time and the level of complexity required in using it.

Mulgara

Mulgara was initiated in 2006 and was a fork of the Kowari project which died or became unsupported as of 2005. Their claim is that Mulgara has the following featured:

• Native RDF support
• Multiple databases (models) per server
• Simple SQL-like query language
• Small footprint
• Full text search functionality
• Datatype support
• Supports and tracks W3C Specifications and guidelines
• Large storage capacity
• Optimized for metadata storage and retrieval
• Multi-processor support
• Independently tuned for both 64-bit and 32-bit architectures
• Low memory requirements
• On-disk joins
• Streamed query results

The remainder of this post will cover my experiences with using Mulgara.

Rough Start

Unfortunately things didn’t start off well. Though the downloading and installation of the Mulgara server was painless there were a number of issues that I came across that would deter many from considering the use of it in a production level system:

Connectors are not included in the default download

Currently there are only a few means of accessing Mulgara to perform CRUD operations:

• A JRDF connector
• A Jena connector
• Straight RMI (remote method invocation)

In the default download from the Mulgara website, the JRDF and Jena connectors are not provided. After a lot of struggling, I resorted to downloading the source and building the server from scratch to ensure that the connectors were included within the jar files.

After getting to this stage I tried to follow the tutorials to try and connect and interact with the server. It seems that the documentation on the mulgara website is stale and that they have not been able to keep their tutorials up to date because the code provided in them does not work unless one uses an in-memory database which is not specified in the tutorial. I posted questions to the various user group email lists and receive no responses.

Once I figured those aspects out things went more smoothly though there is a significant amount of boiler plate code that needs to be created to allow for simple CRUD operations.

Creating a connection to the server

As noted above, one can not simply use JDBC to connect to the server but have an option of using the majority of the popular RDF creation frameworks such as JRDF, Jena or Sesame. I initially tired to use Jena but it seems that support for the JenaMulgara connector died quite a while ago and the connector does not work for the latest version of Mulgara. Therefore, I moved on to the JRDF connector given that the Sesame connector is quite immature.

// Create the URI of the server
java.net.URI serverURI = new java.net.URI("rmi", hostName, "/" + serverName, null);

// Create a new session factory, ensure that it’s local
SessionFactory sessionFactory = SessionFactoryFinder.newSessionFactory(serverURI, false);

// Get a local JRDF Session (local)
Object o = sessionFactory.newJRDFSession();
org.mulgara.server.JRDFSession session = (JRDFSession) sessionFactory.newJRDFSession();
[/sourcecode]

Create a model/database

The above give you a connection to the Mulgara server. To create a model/database requires the following section of code:

java.net.URI modelURI = new URI("rmi", hostName, "/" + serverName, graphName);
java.net.URI modelType = new URI("http://mulgara.org/mulgara#Model");
session.createModel(modelURI, modelType);
org.jrdf.graph.Graph graph = ClientGraph(createGraphProxy(modelURI, session));

Create nodes/relationships

In order to create RDF triples one needs to specify a subject, predicate and an object. There are three types of Java objects that can be used in this in building these relationships:

1. BlankNode - a blank node that is a node used to group other relationships. A BlankNode can be used as either a subject or an object
2. Literal - a literal value of some sort (String, number, etc.) A literal can only be used as an object and not a subject or predicate
3. URIReference - a URIReference is used to define the predicate or relationship between the subject and the object.

org.jrdf.graph.GraphElementFactory elementFactory = graph.getElementFactory();
org.jrdf.graph.BlankNode blanknode = elementFactory.createResource();
org.jrdf.graph.URIReference predicate = elementFactory.createResource(new URI());
org.jrdf.graph.Literal literal = elementFactory.createLiteral(value);

Lastly to create and insert a triple one simply needs to do the following:

org.jrdf.graph.Triple triple = elementFactory.createTriple(subject, predicate, object); // Create the triple object
graph.add(triple); // Store the triple in Mulgara server

For sample code on how to do everything from connect to query and delete see the end of this post.

Performance

I wrote a simple application that took the contents of a relational database and converted it into RDF and stored it in Mulgara. The database I was using was big but not huge, roughly …. All these numbers are based on running the application on a MacBook Pro with a 2Gh Core Duo processor and 2GB of RAM. I initially wrote the app to do inserts one at a time which was obviously inefficient but I wanted to test out the speed of an insert. Each insert of a single triple took roughly 0.17 seconds. In round 2 I started doing inserts in batch. In batch mode it seems that inserts took roughly 0.08 seconds per insert of a node. Both speeds are not particularly fast but like I said, this is running off my laptop so one can’t expect superb performance.

Conclusion

Given that RDF stores are competing with regular relational databases I would have hoped that there was an easier means of connecting to the database and that RM wasn’t being used under the hood. Given the push for RESTful interfaces within the semantic world, I was surprised to see that the Mulgara server does not have a RESTful interface which would relieve the need for RMI completely. Once I got over my dependence on the abhorrent tutorials and documentation things went fairly well though I have performance concerns when dealing with large volumes of data. The main area of concern for me is the requirement of RMI which has been well documented as being not very performant. It pains me to have to use RMI when running both the server and the application on the same server. What is nice is that the server is completely transactional and any failure results in a roll-back. The last concern I have is the fact that the user community/group around Mulgara does not seem to be very active or dedicated to ensuring that the product is supported to the extent that one can rely on getting up to date documentation and a steady stream of bug fixes.

Sample Code

The following code sample shows how to perform inserts, updates, deletes and selection.

package org.twopaths.jrdf;

import java.net.InetAddress;
import java.net.URI;
import java.net.URISyntaxException;
import java.net.UnknownHostException;

import org.jrdf.graph.BlankNode;
import org.jrdf.graph.Graph;
import org.jrdf.graph.GraphElementFactory;
import org.jrdf.graph.GraphElementFactoryException;
import org.jrdf.graph.GraphException;
import org.jrdf.graph.Literal;
import org.jrdf.graph.Triple;
import org.jrdf.graph.URIReference;
import org.jrdf.util.ClosableIterator;
import org.mulgara.client.jrdf.AbstractGraphFactory;
import org.mulgara.query.QueryException;
import org.mulgara.server.JRDFSession;
import org.mulgara.server.NonRemoteSessionException;
import org.mulgara.server.SessionFactory;
import org.mulgara.server.driver.SessionFactoryFinder;
import org.mulgara.server.driver.SessionFactoryFinderException;

public class Sandbox {
	public static void main(String[] args) {
                Graph graph = null;
		try {
			// Create the host name
			String hostname = InetAddress.getLocalHost().getCanonicalHostName();

			// Create the URI of the server
			URI serverURI = new URI("rmi", hostname, "/" + "server1", null);

			// Create a new session factory, ensure that it's local
			SessionFactory sessionFactory = SessionFactoryFinder.newSessionFactory(serverURI, false);

			// Get a local JRDF Session (local)
			Object o = sessionFactory.newJRDFSession();
			System.out.println(o.getClass().getName());
//			LocalJRDFSession session = (LocalJRDFSession) sessionFactory.newJRDFSession();
			JRDFSession session = (JRDFSession) sessionFactory.newJRDFSession();

			//create a new Model
			URI modelURI = new URI("rmi", hostname, "/" + "server1", "exampleGraph");
			URI modelType = new URI("http://mulgara.org/mulgara#Model");
			session.createModel(modelURI, modelType);

			//create a JRDF Graph for the model
//			graph = new JRDFGraph(session, modelURI);
			graph = AbstractGraphFactory.createGraph(serverURI, modelURI);

			//get the Factory
			GraphElementFactory elementFactory = graph.getElementFactory();

			//create resources
			URIReference person = elementFactory.createResource(new URI("http://example.org/staffid#85740"));
			BlankNode address = elementFactory.createResource();

			//create properties
			URIReference hasAddress = elementFactory.createResource(new URI("http://example.org/terms#address"));
			URIReference hasStreet = elementFactory.createResource(new URI("http://example.org/terms#street"));
			URIReference hasCity = elementFactory.createResource(new URI("http://example.org/terms#city"));
			URIReference hasState = elementFactory.createResource(new URI("http://example.org/terms#state"));
			URIReference hasPostCode = elementFactory.createResource(new URI("http://example.org/terms#postalCode"));

			//create values
			Literal street = elementFactory.createLiteral("1501 Grant Avenue");
			Literal city = elementFactory.createLiteral("Bedford");
			Literal state = elementFactory.createLiteral("Massachusetts");
			Literal postCode = elementFactory.createLiteral("01730");

			//create statements
			Triple addressStatement = elementFactory.createTriple(person, hasAddress, address);
			Triple streetStatement = elementFactory.createTriple(address, hasStreet, street);
			Triple cityStatement = elementFactory.createTriple(address, hasCity, city);
			Triple stateStatement = elementFactory.createTriple(address, hasState, state);
			Triple postCodeStatement = elementFactory.createTriple(address, hasPostCode, postCode);

			// Add triples to graph
			graph.add(addressStatement);
			graph.add(streetStatement);
			graph.add(cityStatement);
			graph.add(stateStatement);
			graph.add(postCodeStatement);

			//get all Triples
			Triple findAll = elementFactory.createTriple(null, null, null);
			ClosableIterator allTriples = graph.find(findAll);
			while (allTriples.hasNext()) {
				System.out.println(allTriples.next().toString());
			}

			//search for address (as a subject)
			Triple findAddress = elementFactory.createTriple(address, null, null);
			ClosableIterator addressSubject = graph.find(findAddress);
			while (addressSubject.hasNext()) {
				System.out.println(addressSubject.next().toString());
			}

			//search for the city: "Bedford"
			Triple findCity = elementFactory.createTriple(null, null, city);
			ClosableIterator bedfordCity = graph.find(findCity);
			while (bedfordCity.hasNext()) {
				System.out.println(bedfordCity.next().toString());
			}

			//search for any subject that has an address
			Triple findAddresses = elementFactory.createTriple(null, hasAddress, null);
			ClosableIterator addresses = graph.find(findAddresses);
			while (addresses.hasNext()) {
				System.out.println(addresses.next().toString());
			}

		} catch (UnknownHostException uhe) {
			uhe.printStackTrace();
		} catch (URISyntaxException urise) {
			urise.printStackTrace();
		} catch (SessionFactoryFinderException sffe) {
			sffe.printStackTrace();
		} catch (NonRemoteSessionException nrse) {
			nrse.printStackTrace();
		} catch (QueryException qe) {
			qe.printStackTrace();
		} catch (GraphException ge) {
			ge.printStackTrace();
		} catch (GraphElementFactoryException gefe) {
			gefe.printStackTrace();
		} finally {
                        try {
                            graph.close();
                        } catch (Exception e) {
                            e.printStackTrace();
                        }
                }
		System.out.println("DONE");
	}
}

We had downloaded and installed the debian package grails_1.0.4-1_all.deb from the GRAILS site, configured the GRAILS_HOME environment variable for our build target, and attempted to build it. We got this error:
Grails ended with a non null return code: 127
After some digging, it turns out that the debian package installs GRAILS but puts the actual executable in /usr/bin instead of just a symlink from the GRAILS_HOME. The Maven GRAILS plugin appears to be calling grails using the GRAILS_HOME path. Making a symlink to the grails executable in the bin dir of the GRAILS_HOME fixed this problem (although the symlink really more correctly belongs in the /usr/bin dir).

Trying to build again we came across a different problem, where the GRAILS version was showing up as null, causing build errors. After some googling I found this useful post, and re-installed GRAILS with the updated grails_1.0.4-2_all.deb package, repeating the steps for the GRAILS executable. Now we’re all up and running in Hudson! I hope this might alleviate some pain for anyone trying to troubleshoot this problem.

Enter LiquiBase, an open source database-agnostic tool for tracking, managing and applying database changes. We recently worked on a small Grails project and decided to give this tool a try and I was duly impressed.

LiquiBase functionality is built around a main changelog.xml file containing changesets representing incremental database changes to be applied to a database. LiquiBase manages which changesets have been run through a DATABASECHANGELOG table it creates in each database.

We used the grails plugin which gave us most of the full LiquiBase functionality, albeit a little less than mature. There were a couple of gotchas and bugs with the plugin, but nothing we couldn’t work around.

LiquiBase buys us the ability to store database change in source control, easily sync databases in multi-environments in an automated and controlled fashion, tag database states upon iteration-end releases, auto-generate rollback sql to tagged states, diff databases, and much more.

LiquiBase-ifying Your Application

With your database in a known state, you can use the grails plugin to create your changelog.xml. First you need to install the grails LiquiBase plugin:
grails install-plugin liquibase
Once installed simply run this from the root of your grails app:
grails generate-changelog grails-app/changelog.xml

This will generate the changelog.xml from your development database (specified in DataSource.groovy) and write it to the path specified in the command. grails-app/changelog.xml is the default path where the grails plugin will be expecting the changelog to be. If you need to read from a different database environment like staging for example, just add the -Dgrails.env=staging option to the command.

To propagate this changelog to other databases (say, test), run
grails -Dgrails.env=test migrate. This runs all the sql necessary to upate the database to match the changelog. Any new changes from here on will be appended as new changesets, and can be migrated with the same command as above. You can also use the migrate-sql command instead if want to generate the sql and run it yourself. Most LiquiBase commands come in tuples: one to generate the sql for you and one to just run the sql directly against your database.

If you’re integrating LiquiBase mid-project and already have all your databases set up, you’ll need to run sql against them to update the LiquiBase DATABASECHANGELOG table to show all the changes as run. First, make sure that the databases are all in sync. To do this, you can use the handy LiquiBase diff tool. Unfortunately, the grails plugin for this diff tool is not very robust and will only diff your development database against your test database - the plugin has those two environments hard-coded. You can either mess with your DataSource.groovy db environments to do the diff setting the two dbs in question to dev and test temporarily, or install LiquiBase itself and run the diff passing the db parameters to the diff command. Using the grails plugin you would run:
grails db-diff which will spit to the screen any differences as changesets to be applied. Strangely, there is no documentation for this command in the LiquiBase Grails plugin page.

Once your database are in sync, run
grails changelog-sync-sql with the appropriate -Dgrails.env switch to generate the sql to update the DATABASECHANGELOG table, then run the sql in your database.

LiquiBase and Continuous Integration

We use Hudson at 2Paths for our CI, and have added the simple grails -Dgrails.env=test migrate command as part of our build process to migrate the test database upon every checkin. This ensures that the test db is always the most up-to-date.

Rolling LiquiBase into our Dev Process

We’ve adopted on a trial basis the following process for database change management as part of our agile software development, taking into consideration Grails development (which uses hibernate) which can auto-generates schemas based on domain objects:

1. Generate changelog from initial schema and commit to svn
2. Rollout schema to other databases by migrating from changelog
1. if the schema already exists and it needs “LiquiBase-ifying”, generate changelog-sync-sql and run it
3. Add every schema change via new changeset from hereon in by generating changelogs via grails db-diff
1. add / change domain objects in project
2. set hiberante ddl mode to update
3. start app and grails will automatically update the db
4. generate new changelogs using db-diff (against the test db)
5. append changelogs to changelog.xml
6. run grails changelog-sync-sql to generate the sql that will mark all these new changes as ran on your db, then run the sql
7. checkin changes. this will be applied to test
4. provide explicit rollback sql for any custom sql
5. tag each iteration end with a “tagDatabase” changelog in the changelog.xml

Gotchas

We ran into some gotchas with LiquiBase. One of the first things we started doing before fully understanding how LiquiBase worked was to update existing changesets when changing the schema. LiquiBase generates checksums for each changeset to ensure they don’t change, and altering existing changelogs will cause future migrations to fail. Even though LiquiBase gives you some tools to get around this, it’s generally a better practice to just add a new changeset for every database change. There is a good blog on the LiquiBase site explaining how to deal with changing changesets.

Another gotcha is that the rollback-sql command won’t magically generate rollback scripts for hand-coded sql (obviously!) You must to generate your own rollback sql for these custom sql tags, otherwise not only will you not get rollback sql for those particular changesets, but the rollback functionality won’t spit out sql for any other changeset either until you do.

Tagging was a little fussy as well. The command line grails tag can only tag a given state once - future tags will overwrite the earlier ones unless you’re at a different changeset. I found it better to add a tag changeset explicitly in the changelog.xml. The Grails plugin for tagging also seems to be problematic with spaces in the tag name. We decided to just use underscores as a convention.

We also ran across some broken functionality:

• The grails generate-changelog command generates some extraneous information that breaks sql for data type numeric(19,0) if that type has auto-increment on. It generates this in the changelog:
  <column autoIncrement=”true” name=”id” type=”numeric()(19,0)”> (notice the extra empty brackets). We needed to manually remove these to get it to work.
• The grails rollback-to-date-sql command writes to a file with the datestamp instead of to console like the other rollbacks do, even though the docs say it writes to STDOUT (rollback-count-sql also has issues, but rollback-sql for tags works just fine). The code for these plugin commands doesn’t seem to do the right thing with the args. Time permitting, we may provide a patch for this.
• db-doc generation doesn’t work with large amount of columns in a table because it uses the table description including column names for the html file name. Too many db columns make the file names too long.

Moving Forward

Although there’s a bit of grumbling amongst the developers here that using LiquiBase with the above process is a little onerous, it buys us a lot more certainty for knowing what state our databases are in, making sure changes are in source control, and knowing what updates have or haven’t been run. It’s a whole lot easier than hand-coding a bunch of sql, and easily accommodates the possibility of migrating to other DBMS’s in the future without having to re-write DBMS-specific sql. There’s even talk of trying to go a step or two further to automate even more of the process so a developer doesn’t have to think about generating changesets upon changing of a grails domain model. We’ll see how that goes.

So, needless to say it was a gong-show out there. Those coming to work from further afield naturally called to say they wouldn’t be making it in. This was Kari’s bus, stuck and blocking the road:

The snow did this neat thing on the gallery space:

The BBQ’s are buried:

The rest of the shots are in a small set “Winter at Waterfall” and the 360 is below:

So - who to give to? We wanted to make sure they would have an actual impact. We decided on two broad areas. I used to be a wildlife biologist in another life and one organization that stood out was Conservation International. They have several areas of concern - to save species, conserve landscapes and seascapes, and empower communities with responsible and effective use of natural resources. It’s not just about preserving what’s out there in its current form; it’s about making sure it’s there in a sustainable way, involving the people who live there. We donated $5,000.00 and “saved” about 333 acres.

The other organization we selected this year is Kiva whose mission is to connect people through lending for the sake of alleviating poverty. Kiva, from its partner organizations, puts forward entrepreneurs from the developing world, how much money they need and for what purpose in the hopes of connecting you to them. It’s effectively a loan that will get paid back, although not necessarily. With any luck we will though, and we can loan it to other entrepreneurs and it will keep “paying a dividend” if you will. We’re trying to choose individuals/groups who want to build their business by either setting up infrastructure or creating new opportunities for themselves. Again, we’ve put $5,000.00 towards this.

The thinking behind these choices was to live a form of creative capitalism - making the world a better place hopefully in a sustainable way. We’ve been trying to live that in our company as well, through the clients we’ve been able to work with and the ones we hope to in the future. If you agree with this notion and decide to lend or donate yourself, we hope you’ll let us know by dropping us an email (giving @ 2paths.com).

We experienced what could best be described as an illuminating and transformational year at 2Paths.   From changes to our processes and tools to the diversification of our team demographic, to teaming up with some fabulous new valued partners and clients, change abounded.  And there’s not a soul among us who would argue that it wasn’t for the better.  As our year concludes, my 2008 retrospective blog finds no shortage of things to reminisce about.   Sit back and enjoy.

From a people perspective, the team welcomed 2 new babies into our extended family.  Congratulations to Michal and Lori on the birth of baby Cohen, and congratulations to Ricky and Lin on their second addition to the family, Lucas.  I personally bought a house and got married to the love of my life, Willy the caveman, in a small Whistler wedding. Garret married the love of his life, the beautiful Priscille, in grand fashion Winnipeg-style.  2Paths gladly welcomed new members to the team Omar Khan, Sandra Widjaja, Gary Meehan, Tim Oxenford, and. Kari Braaten.  And last, and very important - we celebrated Michal Urbanski’s 5-year anniversary with the company by sending him on kite-board lessons in Squamish.  He returned exhilarted and intact, thank goodness.

Processes and tools were areas that 2Paths focused heavily on in 2008, recognizing that these will differentiate us from our competition in the future.  We honed in on our existing skills by cultivating 3 certified scrum-masters (Lorill, Geoff and Omar).   Scrum (a brief daily morning developer meeting) has been pivotal to our successful software projects.   Someone likened it to inviting your mother-in-law to live with you – she may not fix everything, but she will most certainly point our everything that needs to be fixed.  Then it’s just about the fixing.    Next, the team continued to examine open source software tools and added some more to our repertoire, including Grails, Hudson, and Groovy, to name a few.   To ensure we got the latest on emerging technologies and uses of technologies, we also sent 2Pathsians to attend conferences on Geo-spatial technology, the Semantic Web, QCon (a conference of technical depth and enterprise focus), No Fluff Just Stuff, and Agile Methodologies.

In the spring, we moved our office to the Arthur Erikson Waterfall building by Granville Island from Gastown.   In addition to loving the view, we have been enjoying lunch at Go Fish, the NoodleBox, and Sushi Go, not to mention whipping up concoctions in our gourmet kitchen.  After our first pay cheque in the new location, our Synesso Cyncra coffee machine made its proud appearance in our kitchen providing java nirvana for everyone to enjoy.  There has been talk of a group membership to Caffeine Anonymous, but so far we’ve been holding out alright.

From a team-building perspective, we did a TON of fun things together this year.  Our long-standing tradition of playing together, letting off some steam, and laughing is our real secret to strong teamwork.  Here’s the illustrious list of our 2008 fun:

-    We bonded over some spring-skiing/riding at Grouse mountain, followed by beer on the patio in the glorious sun;
-    We learned to Curl at Marpole Curling;
-    We tested our trust in each other during rock-climbing;
-    We golfed at Gleneagles and BBQed at Judi’s new home;
-    We enjoyed some fine-dining at Chambar;
-    We participated in Bike to Work week
-    We enlisted a yoga instructor to lead us through weekly hatha yoga practices on the new roof-top patio in the summer, and in our boardroom in the winter.

One of our major projects has been with the “Insert name of international organization which we will shortly be revealing” on an application that has global impact and countless social good benefits.  It gave us an opportunity to go globetrotting to the likes of Paris and London.  Better yet, it presented a chance to work in concert with partners, Gagan Diesh and Kaaren Schulz of DesignStamp, http://www.designstamp.com/,whom we’ve grown to love and cannot do without.  We were also glad to continue our solid working relationship with our valued client, Carlson Wagonlit Travel, http://www.carlsonwagonlit.com/en.  And, we played pretend, and acted like a client to ourselves, dictating our own requirements towards an internal initiative which we will may be talking about a lot more in 2009.

So, as you can see we have been very busy this year.  Not just building web applications that make complicated problems palatable and bite-size, but also in being good to ourselves and to each other.   And we look forward to an exciting 2009, filled with market improvements, more work with our valued partners and clients, and at the very least, some good conversations, hopefully with you, over a fancy-pants coffee from our cool espresso machine.  Have a safe and happy holiday season, everyone!

2Paths Team

Our XML and JSON rendering functionality leverages the XStream project, and provides an easy mechanism for pairing aliases and their respective Java objects.
The Spring Framework configuration in the webapp project sets the rendering information for each response type. Response types are used when specifying routes, which are the Tequila representation for URL request patterns.

JSON example:

<bean id="dataPointJsonRenderer" class="org.tequilaframework.core.renderer.JsonRenderer">
<property name="responseType" value="datapoint/json"/>
<property name="contentType" value="text/plain"/>
<property name="aliases">
<map value-type="java.lang.Class">
<entry>
<key><value>dataPoints</value></key>
<value>java.util.List</value>
</entry>
<entry>
<key><value>dataPoint</value></key>
<value>com.twopaths.example.domain.DataPoint</value>
</entry>
<entry>
<key><value>time</value></key>
<value>com.twopaths.example.domain.TimeDimension</value>
</entry>
<entry>
<key><value>spatial</value></key>
<value>com.twopaths.example.domain.SpatialDimension</value>
</entry>
</map>
</property>
<!-- Example omission -->
<!-- to omit a field, you must declare the parent class as the class value, and the key is the would-be rendered value -->
<!-- <property name="omissions">
<map value-type="java.lang.Class">
<entry>
<key><value>spatial</value></key>
<value>com.twopaths.example.domain.DataPoint</value>
</entry>
</map>
</property>
-->
</bean>

XML example:

<bean id="dataPointXmlRenderer" class="org.tequilaframework.core.renderer.XmlRenderer">
<property name="responseType" value="datapoint/xml"/>
<property name="contentType" value="text/plain"/>
<property name="aliases">
<map value-type="java.lang.Class">
<entry>
<key><value>dataPoints</value></key>
<value>java.util.List</value>
</entry>
<entry>
<key><value>dataPoint</value></key>
<value>com.twopaths.example.domain.DataPoint</value>
</entry>
<entry>
<key><value>time</value></key>
<value>com.twopaths.example.domain.TimeDimension</value>
</entry>
<entry>
<key><value>spatial</value></key>
<value>com.twopaths.example.domain.SpatialDimension</value>
</entry>
</map>
</property>
<!-- Example omission -->
<!-- to omit a field, you must declare the parent class as the class value, and the key is the would-be rendered value -->
<!-- <property name="omissions">
<map value-type="java.lang.Class">
<entry>
<key><value>spatial</value></key>
<value>com.twopaths.example.domain.DataPoint</value>
</entry>
</map>
</property>
-->
</bean>

The RDF renderer leverages the Jena project and Jenabean project. Jena supported RDF formats include RDF/XML, RDF/XML-ABBREV, N-TRIPLE, TURTLE, N3, N3-PP, N3-PLAIN, and N3-TRIPLE. Because of Jena, the Tequila RDF renderer gets these formats for free. The only trick with RDF in Tequila is that you will need to use annotations in the Java classes you wish to have RDF support in. If you’re using Hibernate, then you’ll likely already have RDF rendering configured. If no rdfType is specified, the default is RDF/XML.

RDF example:

<bean id="dataDescriptionRdfRenderer" class="org.tequilaframework.core.renderer.RdfRenderer">
<property name="responseType" value="application/rdf"/>
<property name="contentType" value="text/plain"/>
<property name="domain" value="http://www.2paths.com/example"/>
<property name="rdfType" value="TURTLE"/>
</bean>

With multiple rendering capabilities, the GET request to the webapp can alter the rendering output by including the flavor RPC verb descriptor to modify the return format. For example, the URL of

http://localhost:8080/example/app/data/example?flavor=application/rdf

would return an RDF rendered output, and the original response format may have been JSON, or another response format laid out in the configuration file.

• CouchDB - Document oriented database
  
• Memcached - Distributed Hashmap
• MemcacheDB - A database built as on Memcached and BerkleyDB
  
• Drizzle - Lightweight SQL Database for the cloud/web
• GearMan - Distributed processing framework
  
• MogileFS - Distributed file system
  

I intend to spend some time investigating each of these pieces of technology in the coming months with the hope of making use of them when and where applicable.

Joe Stump from Digg.com was up next and his session was awesome. He talked about the evolution of their architecture from the point he joined to what they are planning on rolling out early next year along with some of the bumps they’ve encountered. MySQL’s scalability had major issue so they moved to IDDB and are considering using MemcacheDB which is MemcacheD mixed with Berkley DB.  Joe pointed out their use of MogileFS for the management of images on digg images which I will be taking a look at for our current project. Joe, you rock, maybe you can give some pointers to Dan…then again hopefully MySpace will send somone with a clue next year.

Aditya Agarwal from facebook was great. He talked about all the different aspects of facebook and the use of PHP/MySQL being the underlying architecture and how for them it . They tend to use MySQL simply as a key value store and profile content is randomly scattered across their thousands of servers. They use LAMP for the most part but have created a framework that allows them to use services written in other languages within their application.

The discovery of the day was CouchDB. One of the lead developers/designers was sitting at the same table as me during the Digg session. Tim Bray was quite excited about the technology as well having mentioned it in his keynote on day 2. I attended one of two sessions about that covered what it’s about from a high level. It has it’s roots in Lotus Notes database design given that it was created by one of the people who worked on Lotus Notes in the past. It has some very cool features:

• RESTful so it doesn’t require any connectors
• Stores all content in JSON
• Querying the database is done via javascript functions for performing map reduce like functionality
• Very efficient and performs versioning and synchronization when distributed

Jan made some good arguments was to why it has some strengths that traditional RDB’s don’t such as no need for schema management, etc. It’s a technology that I plan on taking a serious look at and may possibly use in the future.

In conclusion, QCon was a great experience. The hosts tried their best to make sure things were as tech focused as possible. There is a need to ensure that speakers cater their presentations to the audience and keep to the subject of the presentation. The lagging economy did have an effect on the conference, there were 200+ people that had been registered that didn’t attend (you could see all of the uncollected badges at the registration desk) and if the economy improves I think that next year will be even better. It’s truly a conference for developers by developers which I truly enjoyed. It was great to meet several of the authors of my favourite tech books. I would have preferred more opportunities to network with the people at the conference, an overall guest list would have helped with that or a formal networking event given that a large percentage of the nerds there were stereotypical introverted nerds who didn’t like having a strange person walk up to them and make conversation. Maybe I came off as sleazy or creepy or a threat to homeland security ;). Regardless, I hope to attend next year.

Update: This blog post has been featured on the infoq.com website

Next was a panel discussion on scalability and emphasized the need to understand the update requirements of the system you are designing. They proposed that there are two types of design:

1. Design for scalability up front
2. Design for simplicity and deal with bottle-necks as they pop-up

I’m not sure if I’d ever go with option #2 but they made a good point about how many architects design for things to be bullet proof when one really doesn’t really have the requirement for it to be bulletproof. I fully agree with that principle. There was mention of several websites such as flickr that have made some very good decisions on having that happy medium

The next talk was about the Java 7 concurrency library and the need for making things concurrent given the fact that processors are not getting faster but are moving towards multiple cores. Therefore, we as developers need to ensure that we ensure our applications are making everything run in parallel (concurrent). One might feel that if you are running a website that since every request is being handled by a different thread your job is complete. However, how many of us receive enough requests 24*7 to keep all the cores busy. Therefore, we need to make sure that applications is decomposed to the extent that we are doing everything concurrently that can be done concurrently to ensure maximum CPU utilization and decreased response time and therefore an improved user experience.

The last session of interest was a panel discussion on the affect that the open-source movement has and on java. It was the consensus that the open-source saved java from mediocrity and J2EE from certain death. The focus of the discussion moved to open-source during the current economic condition. The panel felt that slashed budgets within most companies would allow open-source to flourish, at least those that are well established. The last point of interest was about how they make a decision within their companies about what to open-source and what to charge for. Bob Lee of Google said that they open-source low level tools and frameworks and charge for things they build upon them. Others such as a guy from MuleSource and Rod Johnson from Spring Source said that they charge when people are already likely to be paying other vendors for their product/services. For example, the quote of the night was from Rod Johnson

If you use MySql, tomcat and Apache as your application stack then you can use spring for free and that’s great. However, if you are using Oracle Rack for a database, BEA Weblogic for an app server then you have no right to complain when we charge you for a Spring to Oracle RACK connector.

The point being if you’re already spending money then why shouldn’t we ask to be paid for what we are offering.

Tomorrow has another set of very interesting streams and discussions. I’m looking forward to the stream called “Architectures you’ve always wondered about.”

Update: This blog post has been featured on the infoq.com website