Apache Cassandra
Cassandra is a fairly recent addition to the “no-sql” movement. Initially developed by Facebook, Cassandra was open sourced in 2008 and is now housed and maintained by Apache.

Cassandra is a highly scalable, eventually consistent, distributed, structured key-value store. Cassandra brings together the distributed systems technologies from Dynamo and the data model from Google’s BigTable. Like Dynamo, Cassandra is eventually consistent. Like BigTable, Cassandra provides a ColumnFamily-based data model richer than typical key/value systems.

Cassandra was open sourced by Facebook in 2008, where it was designed by Avinash Lakshman (one of the authors of Amazon’s Dynamo) and Prashant Malik ( Facebook Engineer ). In a lot of ways you can think of Cassandra as Dynamo 2.0 or a marriage of Dynamo and BigTable. Cassandra is in production use at Facebook but is still under heavy development.

Data Model
The key difference between Cassandra and many of the other key-value stores is that it possesses the concept of a “Super Column.” For more information on what exactly a super column is (and an overview of the Cassandra data model in general) read the following blog post “WTF is a SuperColumn? An Intro to the Cassandra Data Model”

Performance
Many may be curious about the performance of these key value stores. We found an article that compared Cassandra to Voldemort

Interacting with Cassandra
Cassandra is developed in java however it does not allow access via JDBC. Instead it provides access to it’s repository via the “Apache Thrift” framework which basically allows developers in any major language to gain access to a given service (in this case the Cassandra data store). For those like me who are used to JDBC, the thrift interface can be a bit confusing given that there isn’t much documentation for them. The Java client made available on the Cassandra website is rather rudimentary with no support for things like transactions, therefore many people choose to make use of a library called Hector that provides many of the features necessary for any production worthy application making use of Cassandra.

The Project
What I wanted to to do was use the Cassandra data store on a pet project that I’m working on in my spare time. I intend to use the datastore to do the following:

1. Store messages being sent from one user to another
2. Track queries made by a given user
3. Store user complaints

Installation
Installation of the server is easy enough. Install Java 1.6 if you haven’t already. Edit /bin/cassandra.in.sh and change JAVA_HOME if you run a different version of java as your default JAVA_HOME. By default the server has an RMI port set to 8080, you can change the value in the /bin/cassandra.in.sh file as well.

Configuration
Edit /bin/storage-conf.xml to define various key spaces and column families. The concept of a Keyspace is similar to the schema in a relational database, where as the ColumnFamily is analogous to a table. In defining the ColumnFamilies you are defining the key groupings and how they are sorted. Therefore, I defined the following:

  <Keyspaces>
    <Keyspace Name="Nikahfied">

      <ColumnFamily Name="Queries"
                    Comment="User queries"/>

      <ColumnFamily Name="Complaints"
                    Comment="User complaints"/>

      <ColumnFamily Name="Messages"
                    ColumnType="Super"
                    CompareWith="UTF8Type"
                    CompareSubcolumnsWith="UTF8Type"
                    RowsCached="10000"
                    KeysCached="50%"
                    Comment="Message threads between users of the system"/>

    </Keyspace>
  </Keyspaces>

I would be creating a schema named Nikahfied and three table are Queries, Complaints and Messages. As you will notice, there is no real structure to the ColumnFamilies other than what type of column and the CompareWith strategy as outlined in the “WTF is a SuperColumn? An Intro to the Cassandra Data Model” article. In this case we have one super column and two regular columns. In code I will ensure that each of the rows fore each ColumnFamily contain the same content however there is nothing forcing you to do so. As you will notice, we can configure each of the ColumnFamiles individually. Messages will need to handle heavy reads and writes where as Queries and Complaints will be more write intensive. Messages therefore have a caching scheme defined where as the others do not as they will primarily handle writes rather than reads.

When initially defining the column structure I was a bit confused. I was taking the definition of Cassandra as a “key value store” literally and was confused about there being a primary key and the secondary keys. When defining a basic Column I though it would simple have a key and a value. So things would be defined as Nikahfied.<ColumnFamily>.key = value where this is actually not the case. For a basic ColumnFamily, the structure is as follows: Nikahfied.<ColumnFamily>.<Primary key>.<Secondary Key> = value. In the case of a super column, this goes one level further Nikahfied.<ColumnFamily>.<Primary key>.<Secondary Key>.<Tertiary Key> = value. That had me puzzled for some time…upon re-reading the “WTF is a SuperColumn? An Intro to the Cassandra Data Model” article I realized my mistake so I though I’d point it out to anyone else who was confused about this construct.

Data Structures:
Queries:
The structure of queries would be simple enough. I would store each query by date/time. To track which queries the user made when.

Nikahfied.Queries.UserId.<Date/Time> = <Query>

Complaints:
There is only one type of complaint wherein a user can complain about another user. So I wanted to track who received the complaint, who complained and when. The structure of a complaint would be defined as follows:
Nikahfied.Complaints.<Violator Id>.<Complainant Id> = <Date/Time>

Messages:
If there was ever a case to use a super column, a messages would be it. A user has a number of threads that contain one or more message. A user can receive and send messages. Messages themselves are the same, I simply need to track which threads contain messages that a user sent. I also need to keep track of which threads have unread messages within them. My initial design was the following:
Nikahfied.Messages.RecipientId.threads = A JSON Object that contains Threads defined in JSON Objects themselves. A thread being a group of
.sent_ids = An array of thread id's that contain messages that the user has sent
.unread_ids = An array of thread ids that contain unread messages

However, with this structure there was no need for a super column and I was determined on making use of one . So I expanded out the thread structure as follows:

Nikahfied.Messages.RecipientId.threads.<thread id> = A JSON Object that contains an individual Thread
.sent_ids = An array of thread id's that contain messages that the user has sent
.unread_ids = An array of thread ids that contain unread messages

Now that we have the content structure defined I now need to create a hook between my application and the Cassandra key value store. For development I have decided to use the Grails Framework to speed up development. I have included the Hector library to access the framework but needed to define a CassandraService to be the central point of contact between various controllers and other services to the Cassandra key store. Both Cassandra’s Thrift interface and Hector have pretty poor documentation so I created a Grails service that makes the interaction with Cassandra a bit clearer (at least to me). Note: You will need to be using Java 1.6.

import me.prettyprint.cassandra.service.CassandraClientPool
import me.prettyprint.cassandra.service.CassandraClientPoolFactory
import me.prettyprint.cassandra.service.CassandraClient
import me.prettyprint.cassandra.service.Keyspace
import org.apache.cassandra.thrift.Column;
import org.apache.cassandra.thrift.ColumnPath;
import org.apache.cassandra.thrift.NotFoundException;

import org.apache.cassandra.thrift.SuperColumn

class CassandraService {

    boolean transactional = true

    def servers=["localhost:9160"]
    def defaultKeyspace="Nikahfied"
    private static final String NOT_FOUND = ""

    private execute(keyspaceName=defaultKeyspace,block){
        CassandraClientPool pool = CassandraClientPoolFactory.INSTANCE.get();
        CassandraClient client = pool.borrowClient(servers);

        try {
            Keyspace keyspace = client.getKeyspace(keyspaceName)
            return block(keyspace)
        } finally {
            pool.releaseClient(client);
        }
    }

    /**
     * Get a single super column
     * @param columnFamily
     * @param secondaryKey
     * @param key primary key
     * @return Matching super column
     */
    public SuperColumn getSuperColumn(String columnFamily, String key, String secondaryKey) {
        ColumnPath cp = new ColumnPath(columnFamily)
        cp.setSuper_column(secondaryKey.bytes)
        return execute {Keyspace keyspace ->
            SuperColumn sc = null;
            try {
                sc = keyspace.getSuperColumn(key, cp)
            } catch (NotFoundException nfe) {
                sc = null;
            }
            return sc
        }
    }

    /**
     * Get multiple super columns
     * @param columnFamily
     * @param secondaryKey
     * @param keys primary keys
     * @return matching super columns
     */
    public Map multigetSuperColumn(String columnFamily, List keys, String secondaryKey) {
        ColumnPath cp = new ColumnPath(columnFamily)
        cp.setSuper_column(secondaryKey.bytes)
        return execute {Keyspace keyspace ->
            Map scMap = keyspace.multigetSuperColumn(keys, cp)
            return scMap
        }
    }

    /**
     * Get multiple columns
     * @param secondaryKey secondaryKey
     * @param keys primary keys
     * @param columnFamily column family
     * @return results if any
     */
    public Map multigetColumn(List keys, String secondaryKey, String columnFamily) {
    	ColumnPath cp = new ColumnPath(columnFamily)
        cp.setColumn(secondaryKey.bytes)
        return execute {Keyspace keyspace ->
            Map cMap = keyspace.multigetColumn(keys, cp)
            return cMap
        }
    }

    /**
     * Get a single column and it's values
     * @param columnFamily
     * @param key primary key
     * @return matching column
     */
    public Column getColumn(String columnFamily, String key, String secondaryKey) {
        def cp = new ColumnPath(columnFamily)
        cp.setColumn(secondaryKey.bytes)
        return execute {Keyspace keyspace ->
            Column c = null;
            try {
                c = keyspace.getColumn(key, cp)
            } catch (NotFoundException nfe) {
                c = null;
            }
            return c
        }
    }

    /**
     * Sets the new value for this column path
     * @param cf Column family
     * @param secondaryKey secondary key
     * @param key primary key
     * @param value value
     */
    def setColumnPathValue(String cf, String key, String secondaryKey, String value){
        def cp = new ColumnPath(cf)
        cp.setColumn(secondaryKey.bytes)
        return setColumnValue(cp, key, value)
    }

    /**
     * Sets the new super column value for this column path
     * @param cf Column Family
     * @param sc Super Key Id
     * @param secondaryKey Secondary Key
     * @param key primary key
     * @param value value
     * @return the value of the Column requested
     */
    def setColumnPathValue(String cf, String sc, String key, String secondaryKey, String value){
        def cp = new ColumnPath(cf)
        cp.setSuper_column(sc.bytes)
        cp.setColumn(secondaryKey.bytes)
        return setColumnValue(cp, key, value)
    }

    /**
     * Set a regular column
     * @param cp Column path
     * @param key primary key
     * @param value value
     */
    private setColumnValue(ColumnPath cp, String key, String value){
        return execute{ Keyspace keyspace ->
            keyspace.insert(key, cp, value.bytes)
        }
    }

    /**
     * Batch insert content for a give key and it's values
     * @param key primary key
     * @param columnMap column key values
     * @param superColumnMap super column key values
     */
    public batchInsert(String key, Map<String, List> columnMap, Map<String, List> superColumnMap) {
    	return execute{ Keyspace keyspace ->
        	keyspace.batchInsert(key, columnMap, superColumnMap)
    	}
    }

}

Using this class I have created a MessageService, ComplaintsService and a QueriesService that interact with the CassandraService to retrieve and store data. There is an adjoining Message and Complaints controller where as the QueriesService is referenced by the SearchController. Hopefully this blog post will provide you with a better understanding of how to get started with Cassandra the CassandraService class should provide you with a jump start on integrating Cassandra into an existing Grails or Java project.

We recently presented at the Turning Statistics Into Knowledge conference, jointly organized by the US Census Bureau, the OECD and the World Bank, on July 15-16 in DC at the Census Bureau. Here is a pdf of it including our speaking notes, or you can click through it below, sadly without notes (goto the slideshare site to see the notes).

I promised a number of people links to non-technical to technical primers, here we go:

Non-technical

1. Very straight-forward overview, very good for sharing with non-technical people:
2. Very quick one-pager, “The “Semantic Web”: a low-fat, no-nonsense introduction” by @ajubdeen
3. Original article from Tim Berners-Lee showing his vision.
4. CNN Making sense of the ’semantic Web’
5. Tim Berners-Lee 2009 TED talk on Linked Data

Semi-technical

1. Largely non-technical, injects a few buzzwords, “Quick Linked Data Introduction” by Michael Hausenblas
2. Understanding the New Web Era: Web 3.0, Linked Data, Semantic Web by Richard MacManus, summarizing a multi-part overview by Greg Boutin

Technical

1. An excellent overview “Linked Data Tutorial” by Michael Hausenblas
2. 30 Minute Guide to RDF and Linked Data by Ian Davis and Tom Heath

After that, you’re off to the races!

(but as I said, just send us an email if you have questions or wanted to talk about it some more)

As you Grails die-hards know, Grails 1.1 introduced the new GrailsUnitTestCase class which is a very useful extension of GroovyTestCase with support for mocking the dynamic properties and methods that Grails injects into various types of classes (domain, controller & taglibs). Getting the team up to speed on this change was the original purpose of the presentation, but I decided to expand it to cover the full spectrum of testing capabilities in Grails 1.1 for those who might not have much exposure to it previously (including me!)

The presentation covered:

- the benefits & drawbacks of each type of testing: unit testing, integration testing, functional testing.

- details on the new GrailsUnitTestCase and it’s extensions (ControllerUnitTestCase and TagLibUnitTestCase)

- a few “gotchas” – such as using mockDomain with inheritance hierarchies

- WebTest goodness – test organization & re-use, Groovy step, passing params between Groovy & WebTest contexts, a little dive into using HtmlUnit to test AJAX

- general recommendations for what types of tests to create for each artifact (domain classes, controllers, services, etc)

Here’s a link to the presentation on SlideShare.net (where you can view it online or download it):

Happy testing!
Dave

Install memcached, which is simple if you use a Mac and have MacPorts installed.

$ sudo port install memcached

Get the memcached jar , the hibernate-memcached jar and the PostgreSQL JDBC driver jar, then  add them to the project.

$ cp memcached-2.3.1.jar hibernate-memcached-1.2.jar postgresql-8.4-701.jdbc3.jar <project_dir>/lib/

Configure memcached as the second-level cache by adding something like this to your DataSource.groovy.

hibernate {
    cache.use_second_level_cache = true
    cache.use_query_cache = true
    cache.provider_class = 'com.googlecode.hibernate.memcached.MemcachedCacheProvider'
    memcached {
        servers = "localhost:11211"
    }
}

I wrote a simple test to generate a whole bunch of domain objects and shove them in the database, then go back and fetch them all again. I set the test to run the insert/load loops in batches of 1, 10, 100, 1,000, 2,000, 4,000, 7,000, and 10,000 objects. I expected that inserting the objects would take a little longer because of the memcached overhead but that loading the objects would be faster because they could be loaded from memcached instead of going out to the database.

I tested three persistence backend configurations.

• HSQLDB, the default for new Grails projects and when running unit/integration tests. This is an in-memory database only, so data aren’t persisted permanently.
• PostgreSQL for a full-blown RDBMS setup. I set dbCreate to “create-drop” in DataSource.groovy so that each test started with an empty database.
• PostgreSQL with memcached enabled. I should point out that in this test I had both PostgreSQL and memcached running on the same machine (my laptop) as the application, so it’s really not taking advantage of memcached’s parallelism strengths.

My expectations held true for the inserts. The memcached case is a little slower but not that much.

Comparison of object insert times using HSQL, PostgreSQL, and memcached+PostgreSQL for persistence backend

When I tested the load times, I was surprised. It looked like memcached wasn’t speeding things up at all. I was also surprised to see that it took less time to load 10,000 objects from the data store than 7,000 objects. I ran the test a few times to rule out the possibility of a load spike from something else running at the same time, but I got similar results every time.

Comparison of object load times using HSQL, PostgreSQL, and memcached+PostgreSQL for persistence backend

I then tried to improve my test setup by running memcached on a second machine (my desktop, connected by gigabit ethernet to the laptop). Using the PostgreSQL + memcached configuration for each test this time, I looked at three more scenarios.

• PostgreSQL running locally on the laptop, memcached running locally on the laptop.
• PostgreSQL running locally on the laptop, memcached running remotely on the desktop.
• PostgreSQL running locally on the laptop, memcached running on both the laptop and the desktop.

There’s some overhead in going out to the network to retrieve data instead of fetching it from a process running locally, so I expected the memcached local instance to be faster than the memcached remote instance. I expected the third test, with two instances of memcached running in parallel, to be fastest because it splits the load between the two instances and frees up some CPU time on the laptop. Things didn’t play out that way, though.

Comparison of object insert times for various memcache setups

Comparison of object load times for various memcache setups

I have to be honest that these were not great tests. For one, I should really have run those insert/load loops in parallel instead of sequentially. I’m guessing that the test spent a lot of time stalled waiting for data when it could have been sending out more requests. I would also like to test with a large group of memcached servers and a remote database, perhaps even a cluster of app servers with a load balancer to try the full meal deal. The memcached FAQ explains that running everything on one machine will not show off memcached’s scalability.

However, this gave me the opportunity to get my hands dirty with memcached and learn a few things.

• Integrating memcached is so simple and the performance penalty of just running it locally during development is so small that there’s no reason not to enable it.
• It’s dead simple to install and set up. We could , for example, easily install it on a bunch of test servers and run multiple instances on different ports – one for each developer (to prevent us from clobbering each other’s data).
• Measuring performance and scalability realistically is tricky. It’s one thing to hand wave and say that running multiple instances of a database or a cache will speed things up, but it takes some work to set up a realistic simulation of high traffic application performance.
• Adding memcached doesn’t automatically turn performance up to 11. I need to learn more about what’s going on under the hood with my domain objects and how they’re being persisted through memcached.

I look forward to using memcached more in the future and learning how to really take advantage of it.

Update (2009-07-24): Ray pointed out in the comments that I should have enabled caching in my domain objects under test. I added “static mapping = { cache true }” to my domain class definition and re-ran the tests to compare the effect under 4 scenarios.

• no domain object caching (as before)
• domain object caching with memcached running locally
• domain object caching with memcached running remotely
• domain object caching with memcached running both locally and remotely

Not much difference with the inserts.

Comparison of object insert times with domain object caching enabled and disabled

However, the load times are slightly better with memcached enabled during heavy activity! The advantage disappears when I add the overhead of traversing the network, but it’s a start. Thanks for the pointer, Ray!

Comparison of object load times with domain object caching enabled and disabled

PROS

Probably the number one mentioned feature by anyone talking about CouchDB is that it’s schema-free. That means that each document is its own structure, that everything is based on a key-value pairing, where the value is typically a document, but can also be a base type. This is pretty cool when dealing with describing real world objects, as one can be more descriptive and hierarchical in storing the data. Also, updating your application’s domain model is much like evolution, as soon as you need something added to the model, just add it.

CouchDB’s RESTful HTTP service implementation is great for the many applications being created for today’s web. Send Javascript queries/requests to the service, and get JSON formatted data in return. This is great, since many front-end technologies can decipher JSON, and the format itself is even human readable if need be.

There is support for interacting with CouchDB data offline. One can access and edit data while not connected, and when back online can merge and update changes made while offline. This kind of support out of the box makes CouchDB a prime candidate for  web based applications like blogs, wikis, bug reporting, CRM, etc.

Another quality of CouchDB is its MapReduce implementation for handling large data sets. I’m going to redirect you to the author’s notes for this one: MapReduce and Google’s definition.

CONS

Transaction management is non-existent. Relational database managements systems (RDBMS) provide mechanisms for things like transaction management, particularly unique constraints and locking. CouchDB is based on being ‘eventually consistent’ and ‘available’ (See the CAP Theorem section for more), which although great for many things, doesn’t lend itself to transaction management.

Data security is also a concern. A somewhat old article describes this issue quite well: Planned Security Model For CouchDB. The CouchDB team has been working on security and validation but are still early in their development.

To a less important degree, but still worth mentioning, is that personally I really don’t find the out-of-the-box UI for CouchDB to be all that useful. It feels clunky to navigate your key/document pairs, and too feature-light for managing the database.

Conclusion

If you’re looking to create a document based web application, then this may be a worthwhile technology to use. CouchDB has a number of very impressive features. However, for now at least, I believe I’ll be staying off the couch.

Aside:

When testing out CouchDB I went through the process of manually installing the application. I had to resolve the dependencies, and deal with compilation errors on my own. I kind of wish I’d found this site earlier. Follow that link if you’re a MacOSX user and want an installer for CouchDB.

For me, this was pretty much the  only useful tutorial on getting started with CouchDB: Programming CouchDB with Javascript.

Resources:

• http://couchdb.apache.org/
• http://books.couchdb.org/relax/
• http://www.eflorenzano.com/blog/tag/couchdb/
• http://labs.google.com/papers/mapreduce.html
• http://damienkatz.net/2008/02/incremental_map.html
• http://jan.prima.de/~jan/plok/archives/142-CouchDBX-Revival.html
• http://www.automatthew.com/2008/01/planned-security-model-for-couchdb.html
• http://jchrisa.net/drl/_design/sofa/_show/post/couchdb_edge__security_and_vali
• http://jan.prima.de/~jan/plok/archives/108-Programming-CouchDB-with-Javascript.html
• http://www.infoq.com/CouchDB
• http://www.infoq.com/news/2007/11/the-rdbms-is-not-enough
• http://www.infoq.com/news/2008/11/Database-Martin-Fowler
• http://code.google.com/p/couchdb-lounge/wiki/SettingUpTwoCouchInstances

Graph Layout

After looking at various existing graph layout algorithms, I decided to reinvent the wheel and developed my own approach. Each node has a mass based on its size, and I simply used gravity calculations to determine the (inverse, so they repel each other) force each node apples to each other node. You can see this in action when you mouse over a node, it gets bigger and other nodes retreat from it as its mass increases.

This allows each node to position itself in a way so it is visible. It’s not perfect, but it works okay and makes the graph feel responsive to the user.

How to use it

Double click on a node to load its relationships and related nodes. A node must be an rdf node to load properly, I haven’t implemented any way to handle normal web links yet. Basically, only click on links that look like http://dbpedia.org/resource/something, or that you know are rdf.

click for demo of the wikipedia article for baseball

If you’re just interested in the pretty pictures, stop here, if you want more technical information read on.

Implementation

I just used javascript and followed the HTML5 docs for using the canvas element. One thing I ran into was the fact that Firefox doesn’t yet have a full canvas implementation, specifically it doesn’t yet render text on the canvas. I tried doing some voodoo with absolute positioning divs containing text over the canvas, and after a brief descent into insanity, I found the excellent typeface.js library which emulates the missing canvas calls for text.

As for the javascript code itself, I implemented a simple object graph that can contain vertices or edges, then a render loop just advances the object graph by a given amount of time and renders the scene to the canvas.

The back end serving up rdf was done quickly in grails using the Jena framework to do all the heavy lifting. There is some minimal inference going on using the RDF data to determine if something is an image.

Disclaimer

Keep in mind that this is a work of progress that I have not spent much time with. It has bugs, and it’s nowhere near usable for anything useful, but it serves its purpose as a proof of concept. I’m not sure if it will even run under IE, but it should run on any browser that has a decent javascript + canvas implementation.

Conclusions

The canvas element is cool, and works well.

Basically I hope you like it. Please leave any suggestions or feedback as comments. I realize it’s probably full of bugs, but that’s the nature of the beast.

The course was laid out in a novel way – as a scrum project itself! As Product Owners we were asked to compose User Stories capturing what we hoped to learn from the course, and at the end of each “iteration” within the two-day course, were asked to re-prioritize these, knowing time was tight and we may not get to all topics. This helped us to have hands-on experience with Story creation and re-prioritization, with the added bonus of us each being personally invested in the outcome instead of it being a theoretical scenario.

One of the topics I was interested in was how to adapt the scrum process for a Product Owner that is not only not on site, but is overseas and in a different time zone. At 2Paths, one of our larger projects that was recently completed involved a client overseas. This project was very successful, and we accredited much of its success to our client’s buy-in and participation to our scrum process. We had, however, modified the traditional scrum process to accommodate our client’s schedule and time availability. Because they were quite busy with other projects, and due to the large time difference, the client didn’t attend our daily scrum as is recommended. Instead we had weekly phone check-ins which augmented our regular email communication to discuss issues as needed.

I was hoping in the course to see if there would be a more recommended way for this sort of Product Owner to be involved in a project, and how we could try to more closely follow the traditional scrum process. However, the course was more geared towards an internal Product Owner. This type of Product Owner would be a lot more hands on, would physically attend the daily scrum, and spend time sitting within the dev team to facilitate clearer communication. It also seemed that many of the other course attendees had similar project roles where the Product Owner was in-house. From what I could gather, I think we handled our process as well as we could given the circumstances.

The course was well organized, and the two presenters Bryan Stallings and Skip Angel were quite knowledgeable and helpful. We had many good group discussions and participants actively participated, sharing their own war stories. I’d recommend this course for other ScrumMasters wanting to get a better sense of what is required by a Product Owner, and how they fit into the scrum process.

AWS Import/Export

We haven’t analyzed the pricing but it is one more nail into the “let’s shut the dev cage down and go all cloud” debate.

Below is a simple test case showing how to do LINQ operations on a simple array:

        public void LinqArrayTest()
        {
            String[] names = { "Burke", "Connor", "Frank",
                   "Everett", "Albert", "George",
                   "Harris", "David" };

            IEnumerable<string> query = from s in names
                               where s.Length == 5
                               orderby s
                               select s;

            List<String> results = query.ToList();

            Assert.AreEqual(3, results.Count);
            String[] expected = new String[] {"Burke", "David", "Frank"};
            for (int i = 0; i < results.Count; i++)
            {
                Assert.AreEqual(expected[i], results.ElementAt(i));
            }
        }

It is also possible to create runtime LINQ queries using lambda expressions to qualify conditional logic:

        [TestMethod()]
        public void LinqLambdaTest()
        {
            String[] items = { "cabbage", "broccoli", "carrot", "parsnip", "cauliflower" };

            IEnumerable<String> query = items;

            // additive query generation using lambda expressions
            query = query.Where(item => item.StartsWith("c"));
            query = query.Where(item => item.Length < 8);
            query = query.OrderByDescending(item => item.ToLower());

            List<String> results = query.ToList();

            Assert.AreEqual(2, results.Count);
            String[] expected = { "carrot", "cabbage" };
            for (int i = 0; i < results.Count; i++)
            {
                Assert.AreEqual(expected[i], results.ElementAt(i));
            }
        }

LINQ provides a simple, generalised query framework for operating on data collections within the .NET stack. There are framework plugins to support LINQ backed by various datastore technologies, these include: LINQ to SQL, LINQ to XML and LINQ to Entities.

Single Question Mark – nullable types

The single question mark syntax is somewhat obscure. Essentially it is short hand syntax for marking primitives as nullable. System.Nullable is a generic struct that is essentially a wrapper to signify nullability. This can be used as shown below:

// a nullable type for int
int? thing = 12;

// is syntactically equivalent to
System.Nullable<int> otherThing = 12;

// can be any valid int value OR null
thing = -12;
thing = null;

// usage
//y is set to zero
int y = thing.GetValueOrDefault();
// but this will throw an exception as thing.HasValue == false
y = thing.Value;

Double Question Mark – non null assignment precedence

This one is a little obscure but can be quite handy, it is very similar to certain assignment constructs in Perl as it allows one to use precedence to assign a variable from other potential variables and is a nice way to deal with default values.

String thing = null;
String defaultThing = "nothing";

// standard trinary syntax
String otherThing = (thing != null) ? thing : defaultThing;

// syntactic sugar for the above
String sugaredThing = thing ?? defaultThing;

// can also chain things
otherThing = null;

// the first non-null value is the result of the assignment
String anotherThing = thing ?? otherThing ?? defaultThing;

// if all potentials are null, then the assignement is null)
anotherThing = thing ?? otherThing;