FeathersJS and in-memory database testing

Most recently, I had to build a very simple microservice that needed to expose a REST-API and potentially also provide realtime updates using Websockets down the road.

Some quick research led me to feathers, a lightweight web application framework built on top of express and targeted towards quickly scraping together APIs. Of course there are a gazillion of alternatives like sailsjs (staying in the NodeJS and Express world) or django and it’s REST framework (when venturing into the python world) but they all felt like doing too much and requiring too much boilerplate code to quickly get me going.

I was well aware that I’d eventually hit limitations with feathers but it indeed was quick to setup and draft out the API and it also allowed to simply write tests to ensure my API worked as expected.

However, once the tests started to actually create data via the API, I started to run into problems with respect to the database drivers supported by feathers. With most frameworks, the implementation of the data processing logic is independent of the underlying database by leveraging an ORM / ODM. This also allows for easier testing as one can use a speedy (but non-persistent) in-memory database for running tests and switch to a fully-fledged, production- ready database when deploying the application.

With feathers, though the implementation of API controllers / endpoints directly depend on the underlying database and in my case, this meant a hard dependency to nedb which is great for quick development by mimicking parts of mongoDB’s API but being entirely file based. Bootstrapping an endpoint that allows CRUD operations for arbitrary messages is as easy as:

messages.js

const NeDB = require("nedb");
const service = require("feathers-nedb");

const Model = new NeDB({
    filename: "./data/messages.db",
    autoload: true,
});

app.use("/messages", service({ Model }));

Obviously, basing tests on a file-based database comes with two problems:

  1. you need to have a different database for running tests and developing / running in production
  2. to minimize the waiting time for tests to finish, it would be great to elminiate file access altogether for tests

The first issue (different filename for database) was quickly addressable using feather’s configuration system which is a thin wrapper of the excellent node-config library that does exist for the sole purpose of having environment- specific configuration. So instead of having a hardcoded filename for the database, you can just as well depend on a configuration parameter’s value:

config/default.json

{
    // ...
    "nedb": { "path": "../data" }
    // ...
}

config/test.json

{
    // ...
    "nedb": { "path": "../test/data" }
    // ...
}

messages.js

const NeDB = require("nedb");
const service = require("feathers-nedb");
const path = require("path");

module.exports = function (app) {
    const { path: dbPath } = app.get("nedb");

    const Model = new NeDB({
        filename: path.join(dbPath, "messages.db"),
        autoload: true,
    });

    app.use("/messages", service({ Model }));
};

Feathers allows to access the currently loaded configuration via app.get so we need a reference to the app but once the changes to the calling code have been implemented the changes are quite straightforward as seen above.

To then optimize the test runtime and to reduce the need for clearing the database file for each test run as proposed by the feathers testing docs, we can leverage nedb’s inMemoryOnly mode and simply add a nother parameter to our environment-specific configuration files like:

config/default.json

{
    // ...
    "nedb": { "path": "../data" }
    // ...
}

config/test.json

{
    // ...
    "nedb": { "path": "../test/data", "inMemoryOnly": true }
    // ...
}

messages.js

const NeDB = require("nedb");
const service = require("feathers-nedb");
const path = require("path");

module.exports = function (app) {
    const { path: dbPath, inMemoryOnly } = app.get("nedb");

    const Model = new NeDB({
        filename: path.join(dbPath, "messages.db"),
        autoload: true,
        inMemoryOnly,
    });

    app.use("/messages", service({ Model }));
};

That’s it, now development and production will use the file ../data/messages.db to store messages whereas tests will be run against a database that only exists in the database (in a virtual location ../test/data/messages.db).