Create worker thread which performs specific task in the background

Question

As per My Project,

Data has been fetched from database through a query, There is an Iterator on result set and data has been added continuously to this result set.

By iterating over Iterator object results are added to ArrayList. Once we got all the entries (more than 200000) then writing it to a file.

But as it is using more heap space of jvm ,I need to use a worker thread which runs in back ground and writes the data to the file.

As I am new to multi threading , I thought of using Executor service by creating fixed thread pool of 1 thread and whenever entries reaches the count of 50000 ,then submit those entries to executor to append them to file.

please suggest me if this approach is fine or do I need to follow any other approach.

Tom Sebastian · Answer 1 · 2015-10-29T09:14:30.183

I don't think you need a ThreadPool in order to handle single thread. You can do it by creating a single thread(pseudo code):

    List<Entry> list = new ArraList<Entry>(); // class member that will hold the entries from Result set. I Assume entry as `Entry` here
    ....
    void addEntry(Entry entry){
      list.add(entry);
      if(list.size() >= 20000){
        //assign current list to a temp list inorder to reinitialze the list for next set of entries.
        final List tempList = list;// tempList has 20000 entries!
        list =  new ArraList<Entry>();// list is reinitialized

        // initiate a thread to write tempList to file
        Thread t =  new Thread(new Runnable(){

                public void run() {
                    // stuff that will write `tempList` to file

                }});

           t.start();// start thread for writing.It will be run in background and 
                     //the calling thread (from where you called `addEntry()` )will continue to add new entries to reinitialized list
       }//end of if condition
   }

Note: You mentioned about the heap space - even if we use thread it still uses heap.

Thanks @Tom, Found solution for this. – Sreekanth R Nov 24 '15 at 04:17 — Sreekanth R, Nov 24 '15 at 04:17

score 0 · Answer 2 · edited May 23 '17 at 11:51

Executing the process in a thread will free up the main thread to do other stuff. It will not solve your heap space problem.

The heap space problem is caused by the number of entries returned from the query. You could change your query to return only a set number of rows. Process that and do the query again starting from the last row that you processed.

If you are using MS SQL, there is already an answer here on how to split your queries.

Row offset in SQL Server

score 0 · Answer 3 · answered Oct 29 '15 at 10:24

You don't need to fetch all 20000 entries before writing them to file, unless they have some dependencies to each other.

In the simplest case you can write the entries directly to file as you're fetching them, making it unnecessary to have large amounts of heap.

An advanced version of that is the producer-consumer pattern, which you can then adjust to get different speed/memory use characteristics.

score 0 · Accepted Answer · answered Nov 24 '15 at 04:15

Created worker thread which process entries in the beckground.Starting this thread before fetching entries and stopping it when finished fetching all entries,

public class WriteToOutputFile implements Runnable{
BlockingQueue<entry> queue;
File file;
volatile boolean processentries;
WriteToOutputFile(BlockingQueue queue,File file){
this.queue = queue;
this.file = file;
this.processentries= tue;
}

@override 
public void run(){
while(processentries && !queue.isEmpty()){
  entry = queue.take();

if (entry== lastentry)break;
//logic to write entries to file 
} }

public void stop(){
processentries = false;
queue.put(lastentry);
}
}

Create worker thread which performs specific task in the background

4 Answers4

Linked