Migrate binary files data from SQL Server to MongoDB GridFS

Question

Here is my Mongo document after migrating from SQL Server to regular MongoDB collection.

{
"TicketId": 23,
"Attachments" : [ 
        {
            "_id" : 4221,
            "Name" : "profile Pic",
            "Size" : 218112,
            "Description" : "User Profile Pic",
            "Data" :{ "$binary" : "0M8R4KGxGuE.............",
            "IsPrivate" : false,
            "AttachmentType" = {
                                 "ContentType"   = "image/png",
                                 "FileExtension" = ".png"
                               },
            "CreatedByUserId" : 12,
            "CreatedDateTimeUtc" : ISODate("2012-05-21T18:40:08.570Z"),
        },
     { // Another attachment },
     { // Another attachment },
     { // Another attachment }]
}

But I have attachment which are over 16 MB size, since MongoDB document size is limited to 16 MB I cannot use this approach to save my attachments.

Looks like GridFS is a right approach to save files in MongoDB I found this answer on SO https://stackoverflow.com/a/4990536/942855 which explains how to save a new file to GridFS. But I need to be able to migrate data from SQL Server to MongoGridFS.

Also when you upload a file to GRIDFS, it seems like it generates few default fields, I was wondering how can I add additional fields to it to Map to other collections?

Or should I consider keep all info related attachment with other mapping collection and add array of gridFsInfo.Id to it for mapping?

I am using MongoDB 3.2 with MongoDB C# driver

score 0 · Accepted Answer · answered Aug 18 '16 at 18:11

This is how I ended up doing it

Connect to MongoGridFs

  MongoCredential mongoCredential = MongoCredential.CreateCredential("dbName", "userName", "password");
            var mongoServerSettings = new MongoServerSettings {Server = new MongoServerAddress("host Ip",27017),
                Credentials = new List<MongoCredential> { mongoCredential },
                ConnectionMode = ConnectionMode.Automatic, ConnectTimeout = new TimeSpan(0,0,0,30)};
            var mongoServer = new MongoServer(mongoServerSettings);
            var mongoGridFsSettings = new MongoGridFSSettings { };
            var MongoGridFs = new MongoGridFS(mongoServer, DatabaseName, mongoGridFsSettings);

In my C# SQL-to-Mongo export

ticket.Attachments = (from ta in context.TicketAttachments
                                          join a in context.Attachments on ta.AttachmentId equals a.Id
                                          join at in context.AttachmentTypes on a.TypeId equals at.Id
                                          where ta.TicketId == ticket.Id
                                          select new Domain.Model.Tickets.Attachment
                                          {
                                              Id = a.Id,
                                              // Load all fields
                                              Data = a.Data,
                                              Size = a.Size,
                                              Name = a.Name,
                                              AttachmentType = new Domain.Model.Tickets.AttachmentType()
                                              {
                                                  ContentType = at.ContentType,
                                                  FileExtension = at.FileExtension
                                              }
                                          }).ToList();

                    foreach (var attachment in ticket.Attachments)
                    {
                        Stream stream = new MemoryStream(attachment.Data);
                        MongoGridFSFileInfo mongoGridFsFileInfo = mongoDbContext.MongoGridFs.Upload(stream, attachment.Name);
                        attachment.GridFsObjectId = mongoGridFsFileInfo.Id.AsObjectId;
                    }

Finally save my ticket.Attachments object in my regular MongoCollection

Migrate binary files data from SQL Server to MongoDB GridFS

1 Answers1