Using Multiple Mongodb Databases with Meteor.js

Question

Is it possible for 2 Meteor.Collections to be retrieving data from 2 different mongdb database servers?

Dogs = Meteor.Collection('dogs')        // mongodb://192.168.1.123:27017/dogs
Cats = Meteor.Collection('cats')        // mongodb://192.168.1.124:27017/cats

Answer 1

Update

It is now possible to connect to remote/multiple databases:

var database = new MongoInternals.RemoteCollectionDriver("<mongo url>");
MyCollection = new Mongo.Collection("collection_name", { _driver: database });

Where <mongo_url> is a mongodb url such as mongodb://127.0.0.1:27017/meteor (with the database name)

There is one disadvantage with this at the moment: No Oplog

Old Answer

At the moment this is not possible. Each meteor app is bound to one database.

There are a few ways you can get around this but it may be more complicated that its worth:

One option - Use a separate Meteor App

In your other meteor app (example running at port 6000 on same machine). You can still have reactivity but you need to proxy inserts, removes and updates through a method call

Server:

Cats = Meteor.Collection('cats')

Meteor.publish("cats", function() {
    return Cats.find();
});

Meteor.methods('updateCat, function(id, changes) {
    Cats.update({_id: id}, {$set:changes});
});

Your current Meteor app:

var connection = DDP.connect("http://localhost:6000");

connection.subscribe("cats");
Cats = Meteor.Collection('cats', {connection: connection});

//To update a collection
Cats.call("updateCat", <cat_id>, <changes);

Another option - custom mongodb connection

This uses the node js mongodb native driver.

This is connecting to the database as if you would do in any other node js app.

There is no reactivity available and you can't use the new Meteor.Collection type collections.

var mongodb = Npm.require("mongodb"); //or var mongodb = Meteor.require("mongodb") //if you use npm package on atmosphere

var db = mongodb.Db;
var mongoclient = mongodb.MongoClient;
var Server = mongodb.Server;

var db_connection = new Db('cats', new Server("127.0.0.1", 27017, {auto_reconnect: false, poolSize: 4}), {w:0, native_parser: false});

db.open(function(err, db) {
    //Connected to db 'cats'

    db.authenticate('<db username>', '<db password>', function(err, result) {
      //Can do queries here
      db.close();
   });
});

Answer 2

The answer is YES: it is possible set up multiple Meteor.Collections to be retrieving data from different mongdb database servers.

As the answer from @Akshat, you can initialize your own MongoInternals.RemoteCollectionDriver instance, through which Mongo.Collections can be created.

But here's something more to talk about. Being contrary to @Akshat answer, I find that Oplog support is still available under such circumstance.

When initializing the custom MongoInternals.RemoteCollectionDriver, DO NOT forget to specify the Oplog url:

var driver = new MongoInternals.RemoteCollectionDriver(
    "mongodb://localhost:27017/db", 
    {
      oplogUrl: "mongodb://localhost:27017/local"
    });
var collection = new Mongo.Collection("Coll", {_driver: driver});

Under the hood

As described above, it is fairly simple to activate Oplog support. If you do want to know what happened beneath those two lines of code, you can continue reading the rest of the post.

In the constructor of RemoteCollectionDriver, an underlying MongoConnection will be created:

MongoInternals.RemoteCollectionDriver = function (
  mongo_url, options) {
  var self = this;
  self.mongo = new MongoConnection(mongo_url, options);
}; 

The tricky part is: if MongoConnection is created with oplogUrl provided, an OplogHandle will be initialized, and starts to tail the Oplog (source code):

if (options.oplogUrl && ! Package['disable-oplog']) {  
  self._oplogHandle = new OplogHandle(options.oplogUrl, self.db.databaseName);
  self._docFetcher = new DocFetcher(self);
}

As this blog has described: Meteor.publish internally calls Cursor.observeChanges to create an ObserveHandle instance, which automatically tracks any future changes occurred in the database.

Currently there are two kinds of observer drivers: the legacy PollingObserveDriver which takes a poll-and-diff strategy, and the OplogObseveDriver, which effectively use Oplog-tailing to monitor data changes. To decide which one to apply, observeChanges takes the following procedure (source code):

var driverClass = canUseOplog ? OplogObserveDriver : PollingObserveDriver;
observeDriver = new driverClass({
  cursorDescription: cursorDescription,
  mongoHandle: self,
  multiplexer: multiplexer,
  ordered: ordered,
  matcher: matcher,  // ignored by polling
  sorter: sorter,  // ignored by polling
  _testOnlyPollCallback: callbacks._testOnlyPollCallback
});

In order to make canUseOplog true, several requirements should be met. A bare minimal one is: the underlying MongoConnection instance should have a valid OplogHandle. This is the exact reason why we need to specify oplogUrl while creating MongoConnection

Answer 3

This is actually possible, using an internal interface:

var d = new MongoInternals.RemoteCollectionDriver("<mongo url>");
C = new Mongo.Collection("<collection name>", { _driver: d });