Validating MongoDB’s DBRefs

As discussed on this SO question.

Ref Marks The Spot

For various reasons MongoDB doesn’t support joins, but documents can be linked using DBRefs.

For example, we’d like to build a relation between space ships and crew members.
Our ships document looks like:

{
    "_id": "someID"
    "class": "Firefly"
    "name": "Serenity"
}

And our crew document looks like:

{
    "name": "Malcolm 'Mal' Reynolds"
    "ship": DBRef("ships", "someID")
}

We link Capt. Mal to his ship “Serenity” by adding a `ship` field to the crew document; the value of the `ship` is a DBRef object composed of:

  1. The name of the collection which we reference.
  2. The ID of the item we reference.

But what happens when you encounter an inconsistency between references? Links may become invalid, typically in dev and staging environments but this could also happen in your *gasp* production environment!

Solution

There’s no official built-in way to validate DBRefs, but it is easy to manually validate them.
MongoDB is awesome in many ways; one manifest of this awesomeness is the ability to execute commands in the form of Javascripts.

So I wrote a small script – validateDBRefs.js:

//Create a generic function to extract the ID from a document
var returnIdFunc = function(doc) { return doc._id; };

//Map the collection of ships to a collection of ship IDs
var allShipIds  = db.ships.find({}, {_id: 1} ).map(returnIdFunc);

//Find all crew members with ship IDs that don't exist in the allShipIds collection
var crewWithInvalidRefs = db.crew.find({"ship.$id": {$nin: allShipIds}}).map(returnIdFunc);

print("Found the following documents with invalid DBRefs");
var length = crewWithInvalidRefs.length;
for (var i = 0; i < length; i++) {
    print(crewWithInvalidRefs[i]);
}

That can be run with:

mongo DB_NAME validateDBRefs.js

In the given form, the script will output all the crew documents that reference non-existing ships:

Found the following documents with invalid DBRefs

513c4c25589446268f62f487

513c4c26589446268f62f48a

These IDs can now be used to perform validity reporting or even cleaned up as part of an automated maintenance procedure!

Announcing the Gradle REST Plugin

I’ve recently implemented a new, very simple but very useful Gradle plugin.

the -

GRADLE REST PLUGIN

TA DA!!!

Gradle is the build tool of the future, man, and that’s no secret; I build with Gradle when I possibly can.

I faced a situation where I wanted to invoke a few web-hooks during and after my build; the web-hooks I addressed and the information I sent varied according to the state of my build.
At first I thought I should invoke the web-hooks via independent build steps of my CI server, but that also meant:

  1. Breaking the build to a number of separate Gradle processes instead of single process that will execute all the tasks.
  2. Maintaining the web-hook invocation procedures outside of my standard build descriptors.

The downside of #2 being the fact that should I want to run the complete build process (as it runs on my CI server), I would have to recreate my CI server configuration on every premises.

So what do I do? Invoke the web-hooks using Gradle! BOOM!

I can write Groovy code in my Gradle scripts, so why shouldn’t I use Groovy’s HTTP Builder to execute HTTP requests throughout the build process?
Better yet, I can write a Gradle plugin that will hide the implementations details of the HTTP client so that all HTTP requests look like simple Gradle tasks!

Cool, so how do I install this thing?

The plugin source resides in GitHub and is distributed through Bintray, so you can take a look at the project page to find out what’s the latest version.

First, we add the plugin repository and plugin dependency coördinates to the buildscript section of our build script:

buildscript {
    repositories {
        maven { url 'http://dl.bintray.com/content/noamt/gradle-plugins' }
    }

    dependencies {
        classpath 'org._10ne.gradle:rest-gradle-plugin:$LATEST_VERSION'
    }
}

Then, we add the plugin declaration to our build script:

apply plugin: 'rest'

Great, now how do I use it?

The plugin introduces a new task of the type org._10ne.gradle.rest.RestTask; this task is basically a wrapper for an HTTP client, so all we have to do is extend the task, give it the details of the request we want to make and execute!

For example, a GET request looks like:

task getServerVersion(type: org._10ne.gradle.rest.RestTask) {
    httpMethod = 'get'
    uri = 'https://server.com/api/version'
    username = 'user'
    password = 'password'
    contentType = groovyx.net.http.ContentType.JSON
}
Once again, Gradle FTW, man.

Setup Once Per Feature With Spock Extensions

Disclaimer: This example is a hack that uses un-official/non-public Spock APIs.

A proper implementation of this extensions is also included and distributed as part of my Spock Extensions module; The library can be downloaded from here and is also hosted on github.

Also, as seen on Stephen Chin’s Nighthacker :)

The Missing pieces

Prior to discovering the awesome Spock Framework, I used Test-NG for all my testing needs. Although I am now completely hooked on Spock there is one thing I miss in Test-NG , and that’s the ability to run a one-time setup procedure before the test method, regardless of iterations.

I’ll try to clarify the scenario.
Let’s take a simple test specification:

class SimpleSpec extends Specification {

    def setupSpec() {
        println 'Performing long, class wide setup'
    }

    def 'Some spec'() {
        setup:
        println 'Performing long, spec wide setup'

        expect:
        true == true
        println 'Performing assertion'
    }
}

The above specification will produce the following output:

Performing long, class wide setup
Performing long, spec wide setup
Performing assertion

So as expected, the specification will execute the long setup procedure in setupSpec, execute the long setup procedure in the spec’s setup block and then execute the assertions.

Now let’s add several iterations to our spec:

def 'Some spec'() {
        setup:
        println 'Performing long, spec wide setup'

        expect:
        true == true
        println "Performing assertion $x"

        where:
        x << [1,2,3]
}

The above specification will produce the following output:

Performing long, class wide setup
Performing long, spec wide setup
Performing assertion 1
Performing long, spec wide setup
Performing assertion 2
Performing long, spec wide setup
Performing assertion 3

In this case, we notice that the setup procedure in the spec’s setup block is executed before every iteration; In cases of long setup procedures, this can be a real waste of time and resources.

At the time of writing this blog post, Spock has no built-in way to deal with this issue. So, what can we do?

T3h Hax0rz

Good news, everyone! We can easily solve this issue using Spock’s annotation-driven extension points!
Because the API is not yet finalized, official documentation is practically nonexistent; luckily, we can rely on talented people such as Luke Daley and Evgeny Goldin to spill the beans.

Let’s draw up a plan:

  1. Create an extension annotation.
  2. Create an annotation-driven extension class.
  3. Create an event interceptor class.
The Extension Annotation

With the annotation we can select the methods that apply to our extension, specify the backing extension class and specify the name of the setup method we’d like to execute before the feature.

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)

//Specify the extension class that backs this annotation
@org.spockframework.runtime.extension.ExtensionAnnotation(SetupOnceExtension)

public @interface SetupOnceExtensionAnnotation {

    //Accept a string value that represents the name of the setup method to execute
    String value();
}
The Extension Class

Bound to our annotation, the class extends AbstractAnnotationDrivenExtension, from which we inherit different levels of annotation visitation methods: Spec, Feature, Fixture and Field. Every visitation methods is also provided with mutable metadata descriptors according to the overridden level.
Because our main goal is to execute a setup method before a feature, it’s best we override the visitFeatureAnnotation; this also gives us a chance to meddle with FeatureInfo and use it to subscribe our event interceptor.

class SetupOnceExtension extends AbstractAnnotationDrivenExtension<SetupOnceExtensionAnnotation> {

    @Override
    void visitFeatureAnnotation(SetupOnceExtensionAnnotation annotation, FeatureInfo feature) {

        //Retrieve the name of the setup method we'd like to invoke from our annotation
        def methodToInvoke = annotation.value()

        //Construct and subscribe our event interceptor
        def interceptor = new SetupOnceInterceptor(methodToInvoke: methodToInvoke)
        feature.addInterceptor(interceptor)
    }
}
The Event Interceptor

Extending the AbstractMethodInterceptor we’re given many interception points, but the one that interests us most is interceptFeatureExecution.
This point is executed right before the feature so it gives us a chance to execute our setup method.

class SetupOnceInterceptor extends AbstractMethodInterceptor {

    String methodToInvoke

    @Override
    void interceptFeatureExecution(IMethodInvocation invocation) throws Throwable {

        //Get the instance of the executed spec
        def currentlyRunningSpec = invocation.sharedInstance

        //Execute the setup method we specified in the annotation value
        currentlyRunningSpec."$methodToInvoke"()
        invocation.proceed()
    }
}
Putting It All Together

Revisiting our first Specification, we can now annotate our feature method with the SetupOnceExtensionAnnotation, create a separate setup method (‘setupMethod’ in our example) and move the code from the feature’s setup block into it.

class SimpleSpec extends Specification {

    def setupSpec() {
        println 'Performing long, class wide setup'
    }

    @SetupOnceExtensionAnnotation('setupMethod')
    def 'Some spec'() {
        expect:
        true == true
        println "Performing assertion $x"

        where:
        x << [1,2,3]
    }

    void setupMethod() {
        println 'Performing long, spec wide setup'
    }
}

Now when executed, our specification will produce the following output:

Performing long, class wide setup
Performing long, spec wide setup
Performing assertion 1
Performing assertion 2
Performing assertion 3

Issue busted!

Testing Grails JAX-RS resources with Spock

The Missing Infrastructure

Developing with Grails is a real delight, and testing my delightful Grails applications with Spock makes the experience even better.
Some of Grails’ test cases, such as the Controllers, require a specialized setup because you don’t test them in the same manner as you would test PO[GJ]Os; you must mock requests and such.

When testing the REST resources provided by the excellent JAX-RS plugin, the situation is no different; we can test the resources only through specialized cases where a proper environment has been set up.
Thankfully, the authors of the JAX-RS plugin have already provided us with great infrastructure for integration testing with JUnit.
But alas, testing with Spock requires specialized classes because Spock uses a different inheritance hierarchy, and uses different methods for setup/tear-down and sharing data between specifications; so some work is required.

The Solution

Future Solution

The good news is that I’ve already prepared a pull request that adds this functionality, and it was merged in to the project fairly quickly (thanks Martin!).

Current Solution

So how do we get by until the next version of the plugin is released? I’ll now show you how easy it is to add a temporary Spock test infrastructure to your project.
We’ll simply clone the existing JUnit integration test base and adapt it to use the facilities provided by Spock.

1. Dependencies

First and foremost, let’s add a dependency on the latest Grails Spock plugin.
We’ll edit our project’s BuildConfig.groovy (which is located at $PROJECT_HOME/grails-app/conf/BuildConfig.groovy) and add the plugin dependency configuration:

grails.project.dependency.resolution = {
    plugins {
        test ":spock:0.7"
    }
}

Note that the grails.project.dependency.resolution and plugins configuration hierarchies might already exist in the file.

2. The Base Test Specification

Once our project has been reloaded and the dependencies have been resolved, we can create our base test specification.
Let’s create a new Groovy class named RestApiIntegrationSpec within our project’s integration test folder ($PROJECT_HOME/test/integration) and add the following content:

package org._10ne.example.interaction.test

import grails.plugin.spock.IntegrationSpec
import org.codehaus.groovy.grails.commons.GrailsApplication
import org.grails.jaxrs.JaxrsController
import org.grails.jaxrs.itest.IntegrationTestEnvironment
import org.grails.jaxrs.web.JaxrsContext
import org.grails.jaxrs.web.JaxrsUtils
import spock.lang.Shared

import javax.servlet.http.HttpServletResponse

/**
 * @author Noam Y. Tenne
 */
class RestApiIntegrationSpec extends IntegrationSpec {

    @Shared
    IntegrationTestEnvironment testEnvironment

    GrailsApplication grailsApplication

    def controller

    @Shared
    def headers = ['Content-Type': 'application/json', 'Accept': 'application/json']

    def setupSpec() {
        testEnvironment = null
    }

    def setup() {
        grailsApplication.config.org.grails.jaxrs.dowriter.require.generic.collections = false
        grailsApplication.config.org.grails.jaxrs.doreader.disable = false
        grailsApplication.config.org.grails.jaxrs.dowriter.disable = false

        if (!testEnvironment) {
            testEnvironment = new IntegrationTestEnvironment(contextLocations, jaxrsImplementation, jaxrsClasses, autoDetectJaxrsClasses)
        }

        controller = new JaxrsController()
        controller.jaxrsContext = testEnvironment.jaxrsContext
    }

    void setRequestUrl(String url) {
        JaxrsUtils.setRequestUriAttribute(controller.request, url)
    }

    void setRequestMethod(String method) {
        controller.request.method = method
    }

    void setRequestContent(byte[] content) {
        controller.request.content = content
    }

    void addRequestHeader(String key, Object value) {
        controller.request.addHeader(key, value)
    }

    void resetResponse() {
        controller.response.committed = false
        controller.response.reset()
    }

    HttpServletResponse getResponse() {
        controller.response
    }

    HttpServletResponse sendRequest(String url, String method, byte[] content = ''.bytes) {
        sendRequest(url, method, [:], content)
    }

    HttpServletResponse sendRequest(String url, String method, Map<String, Object> headers, byte[] content = ''.bytes) {
        resetResponse()

        requestUrl = url
        requestMethod = method
        requestContent = content

        headers.each { entry ->
            addRequestHeader(entry.key, entry.value)
        }

        controller.handle()
        controller.response
    }

    /**
     * Implementors can define additional Spring application context locations.
     */
    String getContextLocations() {
        ''
    }

    /**
     * Returns the JAX-RS implementation to use. Default is 'jersey'.
     */
    String getJaxrsImplementation() {
        JaxrsContext.JAXRS_PROVIDER_NAME_JERSEY
    }

    /**
     * Returns the list of JAX-RS classes for testing. Auto-detected classes
     * will be added to this list later.
     */
    List getJaxrsClasses() {
        []
    }

    /**
     * Determines whether JAX-RS resources or providers are auto-detected in
     * <code>grails-app/resources</code> or <code>grails-app/providers</code>.
     *
     * @return true is JAX-RS classes should be auto-detected.
     */
    boolean isAutoDetectJaxrsClasses() {
        true
    }
}

You’ll notice that the changes we made to the original class have been fairly subtle:

  1. The file ends with “Spec” instead of “Test” – This way we adhere to Spock’s naming conventions which are depended upon the by the different build and test report frameworks.
  2. The class extends grails.plugin.spock.IntegrationSpec – This will provide us with the different Grails facilities required for the tests.
  3. Variables that are shared between specifications are annotated with @Shared instead of being made static.
  4. Procedures that must be run once before the whole test’s executed are called from the setupSpec method.
  5. Procedures that must be run once before the every specification are called from the setup method.
3. The Tests

Finally, to write our tests, all we must do is extend our RestApiIntegrationSpec class!

Have fun testing!

Truly externalizing the Grails DateSource configuration

Why externalize?

During development of a Grails application, I’ve also developed some independent POJ applications to work alongside the Grails application; because these independent applications worked on the same data as the Grails, they naturally had to use the same data source.
Wouldn’t it be great if all these applications could use the same data source configuration files?

Grails’ default configuration (datasources.groovy) clearly couldn’t stay because there is no way to share it while it’s still in the ‘grails-app/conf’ directory.
After doing a tad of research, It seemed that externalization was possible… well, sort of.

Why do I say “sort of”? Because if Grails is going to share a data source configuration, it’s going to do so on its own terms.

Issues with the existing methods

Allow me to explain.
The current options for externalization are good, but they are limited:

#1 Data source configuration from an external data source

This solution presents us with the “chicken and egg problem” and stays the same; We remain with a data source which we need to share. So, scratch this.

#2 Data source configuration from an external properties file

This is a pretty good option, right? Store the properties file in an external jar file and share it as a dependency between all the applications that need it.
But that’s still not good enough!
Because the properties are eventually loaded directly to Grails’ configuration object, the property names within the file must use Grails’ naming conventions, otherwise they won’t be recognized!
We want to share this properties file between different (not-necessarily-Grails) applications, so why should other applications have to use Grails’ naming conventions?
Also, what if we would like to use a different format other than properties? YAML, JSON, GConfig, etc.?
Scratch this one off, too.

So what can we do?

There are a number of keys to the solution.
First we must understand the order in which Grails loads its data sources.
Grails and data source settings are eventually stored in the same configuration object, and during bootstrap the data source settings are first read from the Config.groovy file (whether directly, from a different data source or from an external properties file), and finally read from the datasources.groovy (if it exists).

Another thing we should remember is that the Config.groovy is… Groovy; this means we can do whatever the hell we want within it.

GIT-R-DONE

So let’s place the data source configuration file in an external shared project, say “common-settings” and add it as a dependency to our Grails project to ensure that it’s loaded on bootstrap.

Now, let’s add a new method “getDataSourcesConfig” to our project’s Config.groovy file.
The method will locate the external properties resource and read it via Groovy’s config slurper.

private ConfigObject getDataSourcesConfig() {
    Properties dataSourcesProps = new Properties()
    InputStream resourceStream
    try {
      resourceStream = new PathMatchingResourcePatternResolver().getResource('datasources/source.properties').inputStream
      dataSourcesProps.load(resourceStream)
    } finally {
        IOUtils.closeQuietly(resourceStream)
    }
    new ConfigSlurper().parse(dataSourcesProps)

Now we can call this method from anywhere within the body of our Config.groovy file, and map our properties to Grails’ data source properties.

ConfigObject dataSourcesConfig = getDataSourcesConfig()
grails.mongo.username = dataSourcesConfig.get 'custom.datasource.username'
//etc

And there we have it.

The mystery of the missing grails-app directory

During work on different Grails projects, I’d be confronted at times with the mysterious error:

| Building WAR file…..
| Error executing script War: : $USER_HOME/.grails/2.1.1/projects/some-project/resources/grails-app does not exist.

At first I thought to myself “Oh, this is just a glitch; Let’s just manually re-create folder nd get on with it!”, but when the error returned now and then I decided that I really should get to the bottom of this.

As simple as this error is, I found it quite frustrating because some key questions remained unanswered:

  • What is this directory needed for?
  • If it doesn’t exist, why doesn’t the process create it?

I’d also noticed that the error came up when the $USER_HOME/.grails/2.1.1/projects/some-project directory was cleared, either in the case of a fresh development environment or when clearing the caches and only specific projects suffered from it.

The solution

Strangely enough, what looks like the cause of this error is that the Grails project has an empty i18n folder.
Placing a dummy ‘messages.properties’ file or removing the folder altogether will solve it.
This issue’s actually a known bug (see here and here) that hasn’t been fixed yet (as of writing this post).

Given no clear explanation, it seems as if existence of the i18n creates an expectation for a grails-app folder, but the lack of any resources to copy means that this expectation is never fulfilled.