2

I'm using Java Threads to implement the Weka CrossValidation (10fold CV), but having difficulty understanding where the averaging part takes place in the method EvaluateModel().

I need to make sure each thread has its own copy of the error rate for each fold, and then perform the averaging when threads finish execution.

Willi Mentzel
  • 27,862
  • 20
  • 113
  • 121
Mohsen Mahmoud
  • 31
  • 6
  • Weka has option for 10 fold cross validation. Check this https://weka.wikispaces.com/Generating+cross-validation+folds+(Java+approach)?responseToken=059d9097b76aea994da4b2ffe7affd507 – Vighanesh Gursale Mar 26 '16 at 20:17
  • yes but this doesnot provide enough details for how the ModelEvaluate method works. where exactly the division by the number of folds takes place in the (Evaluation class ) – Mohsen Mahmoud Mar 26 '16 at 20:23

1 Answers1

0

From whatever you have described here is approach you must try in multiple thread concept.

You have to create thread which will accept your model and test dataset.

EvaluateThread t1 = new EvaluateThread(threadName,model,testDataset1);
EvaluateThread t2 = new EvaluateThread(threadName,model,testDataset2);
EvaluateThread t3 = new EvaluateThread(threadName,model,testDataset3);

Then create synchronized method so that each thread can access that method independently.

Something like this 

public synchronized double calculateError(model, dataset){
       // do your stuff for e.g. calculate error
       return error;
}

Finally calculate average of error you get from each thread. For more info about synchronized method check this link.

Community
  • 1
  • 1
Vighanesh Gursale
  • 921
  • 5
  • 15
  • 31
  • Yes, thats what im doing, im implementing the Runnable interface, and in the Run method i added the crossvalidatemethod for weka, the last method in the Run is called EvaluateModel(), it return a double array for predicted Values called Predicted[]. im not sure whether this method does the averaging for the errors collected from each fold or not and what the values in that predicted array means ! – Mohsen Mahmoud Mar 26 '16 at 20:44
  • Could you please post a small snippet so that I can understand more. – Vighanesh Gursale Mar 26 '16 at 20:50
  • Okay to make it clearer, i created a new class called Eval_Thread and it implements the Runnable interface, inside the Run method of this class im doing the following Instances train = data.trainCV(numFolds, i, random); setPriors(train); Classifier copiedClassifier = AbstractClassifier.makeCopy(classifier); copiedClassifier.buildClassifier(train); Instances test = data.testCV(numFolds, i); evaluateModel(copiedClassifier, test); – Mohsen Mahmoud Mar 26 '16 at 21:44
  • According you explanation you are getting the double array (double[]), and according to weka's documentation the double[] is the prediction which is done by the classifier http://weka.sourceforge.net/doc.dev/weka/classifiers/Evaluation.html#evaluateModel(weka.classifiers.Classifier,%20weka.core.Instances,%20java.lang.Object...) Kindly clarify you want the error rate or cross validation parameters like precision recall. I am sure you are not going to average the prediction. – Vighanesh Gursale Mar 26 '16 at 21:54
  • what im trying to do is to make each thread hold a fold, and does the work separately. and at the end collect the result obtained from each thread. so if the evaluatemodel method for weka does the averaging and return a predicted array, this means i can't use this method in my Run, meaning it cant be multithreaded – Mohsen Mahmoud Mar 26 '16 at 22:32
  • Simply im working to make a multithreaded version for WEka's cross validation, so i copied and modified the weka's version for cross validation so that if im doing 5 folds im using 5 threads one for each fold and return to main thread to print the output like F measure or any other metric – Mohsen Mahmoud Mar 26 '16 at 22:35
  • public void run() { inst = new Instances(inst); inst.randomize(rand); if (inst.classAttribute().isNominal()) {inst.stratify(nf); }Instances train = inst.trainCV(nf,t_ID, rand);setPriors(train);copiedClassifier = AbstractClassifier.makeCopy(clas);copiedClassifier.buildClassifier(train);evaluateModel(copiedClassifier, test); – Mohsen Mahmoud Mar 26 '16 at 22:39