Inserting large quantities in IndexedDB's objectstore blocks UI

Question

I want to save some ~35000 objects in my IndexedDB's objectstore. I am using below code to insert.

AddListings = function (x2j_list_new, callback) {   
    var transaction = db.transaction(["listings"], IDBTransaction.READ_WRITE);
    var count = 0;
    transaction.oncomplete = function (event) {
        if (callback) {
            console.log('x2jShowListing Added ' + count + '/' + x2j_list_new.length);
                callback([count, x2j_list_new.length]);
            }
    };
    transaction.onerror = function (e) {
       console.log("myError: ", e);  
       if (callback) {
          callback(false);
       }
    };
    var store = transaction.objectStore("listings");

    $.each(x2j_list_new, function (index0, item0) {
        var request = store.put(item0);
        request.onsuccess = function (event) {
            count++;
            // event.target.result  
            };
        });
    });        
};

The above code works fine, but looping and inserting over ~35000 objects makes the UI unresponsive for ~200 seconds. I thought maybe i can use WebWorkers, but IndexedDB is not available inside WebWorkers. I tried to find a way to bulk insert, couldn't find one. Any ideas of how to insert large quantities of objects without blocking the UI?

For now I am splitting the array in [chunks](http://stackoverflow.com/questions/8495687/split-array-into-chunks) of 500 and using [setInterval](http://www.kryogenix.org/days/2009/07/03/not-blocking-the-ui-in-tight-javascript-loops) instead of for loop. Now the UI responds little better than before. — surya, May 07 '12 at 15:23
So I know this is old but just wanted to know if anyone had updates on using IndexedDB in Web workers since it is supported now? — aug, Jun 12 '18 at 00:00

score 41 · Accepted Answer · answered Dec 02 '12 at 05:37

You're on the right track, but you're asking the browser to store 35,000 objects before it's had a chance to finish storing one. Here's code which asynchronously waits for one request to finish before starting the next (but using the same transaction):

    openRequest = window.indexedDB.open("MyDatabase", 1);
    openRequest.onerror = function(event) {
        console.error(event);
    };
    openRequest.onsuccess = function (event) {
        var db = openRequest.result;
        db.onerror = function(event) {
            // Generic error handler for all errors targeted at this database's requests
            console.error(event.target);
            window.alert("Database error: " + event.target.wePutrrorMessage || event.target.error.name || event.target.error || event.target.errorCode);
        };
        var transaction = db.transaction('item', "readwrite");
        var itemStore = transaction.objectStore("item");
        putNext();

        function putNext() {
            if (i<items.length) {
                itemStore.put(items[i]).onsuccess = putNext;
                ++i;
            } else {   // complete
                console.log('populate complete');
                callback();
            }
        }           
    };

To be sure that all items are **really** in the store - subscribe to `transaction.oncomplete` event, smth like: `transaction.oncomplete = callback` — Kiril, Oct 24 '14 at 20:08
I can't see how to determine the end of a transaction, one get a store from transaction then add a one or more records... How do I make the transaction end after I add all of it ? Does the transaction ends after onsuccess callback returns? Which is after all entries are added in this recursive call? I suppose so — lisak, Nov 19 '14 at 18:31

score 0 · Answer 2 · edited May 12 '19 at 00:55

You're doing everything right by using callbacks.

The Webworker API has not yet been implemented by any major browser. Interestingly, it is expected to be synchronous. The regular API is async for the exact reason you describe -- it's not supposed to block the UI thread.

Using callbacks is the way to avoid lock ups, but at 35k objects you're clearly seeing this paradigm break down. Unfortunately, IDB performance is not yet on par with WebSQL from the benchmarks I've seen.

With Chrome's LevelDB there's been some new experimental backends (FF is SQLite ) but I think your experience proves that there's some room for improvement.

score 0 · Answer 3 · answered May 07 '12 at 08:03

0

Wild guess from my side, but if WebSQL is available from what is known as a "background page", and assuming the bandwidth of messaging between the front- and backpage does not lock up the UI in the same way, maybe a background page with intra page message could be utilized?

answered May 07 '12 at 08:03

Marius Kjeldahl

6,830
3
33
37

I have a feeling that its the for loop which is causing block UI. I wish there was a way to ping browser in regular interval so it doesnot think its stuck – surya May 07 '12 at 14:49
In traditional GUI programming, you could detect when the browser is "idle" and execute another step in your big foor loop. Unfortunately, detecting the idle state is not easy in javascript, but there are hacks to simulate it. For instance http://stackoverflow.com/questions/667555/detecting-idle-time-in-javascript-elegantly . – Marius Kjeldahl May 07 '12 at 17:52

score 0 · Answer 4 · answered Aug 08 '12 at 15:23

0

I am splitting the array in chunks of 500 and using setTimeout instead of for loop. Now the UI responds little better than before

answered Aug 08 '12 at 15:23

surya

1,351
1
13
29

I don't recommend accessing IndexedDB from multiple event loop tasks. It behaves pretty weird this way – lisak Nov 19 '14 at 18:26

Inserting large quantities in IndexedDB's objectstore blocks UI

4 Answers4

Linked