I have used a rough way to avoid this problem which helped me so much and working nicely.
i.e
Use local database (SQLite) to avoid duplicate data by make phone number to unique.
I have made one SQLite DB to handle this problem:
ContactMerger.java:
public class ContactMerger {
private static final String CONTACT_TABLE = "_contact_table";
private static final String CONTACT_ID = "_contactId";
private static final String CONTACT_NAME = "_contactName";
private static final String CONTACT_MOBILE_NUMBER = "_contactNumber";
private static final String CONTACT_DATE  = "_contactDate";
private static final int DATABASE_VERSION = 1;
private static final String DATABASE_NAME = "DB_Contact";
private final Context context;
private SQLiteDatabase ourDatabase;
private DbHelper ourHelper;
private class DbHelper extends SQLiteOpenHelper {
    public DbHelper(Context context) {
        super(context, DATABASE_NAME, null, DATABASE_VERSION);
    }
    @Override
    public void onCreate(SQLiteDatabase db) {
        // TODO Auto-generated method stub
        String contactQuery = "CREATE TABLE " + CONTACT_TABLE + " ("
                + CONTACT_ID + " INTEGER PRIMARY KEY AUTOINCREMENT, "
                + CONTACT_NAME + " TEXT NOT NULL, " + CONTACT_DATE
                + " TEXT NOT NULL, " + CONTACT_MOBILE_NUMBER
                + " TEXT NOT NULL UNIQUE);";
        db.execSQL(contactQuery);
    }
    @Override
    public void onUpgrade(SQLiteDatabase db, int oldVersion, int newVersion) {
        // TODO Auto-generated method stub
        db.execSQL("DROP TABLE IF EXISTS " + CONTACT_TABLE);
        onCreate(db);
    }
}
public ContactMerger(Context context) {
    this.context = context;
}
public ContactMerger open() throws SQLException {
    ourHelper = new DbHelper(context);
    ourDatabase = ourHelper.getWritableDatabase();
    return this;
}
public void close() {
    ourHelper.close();
}
// Insert Data to Contact Table
public long insertContacts(String name, String number, String date) throws SQLException {
    ContentValues cv = new ContentValues();
    cv.put(CONTACT_NAME, name);
    cv.put(CONTACT_DATE, date);
    cv.put(CONTACT_MOBILE_NUMBER, number);
    Log.d("Insert Data", cv.toString());
    return ourDatabase.insert(CONTACT_TABLE, null, cv);
}
//Get Contact details from Contact Table
public ArrayList<ContactHolder> getContactDetails() throws Exception{
    ArrayList<ContactHolder> contactDetails = new ArrayList<ContactHolder>();
    String[] columns = new String[] { CONTACT_ID, CONTACT_NAME, CONTACT_DATE, CONTACT_MOBILE_NUMBER };
    Cursor c = ourDatabase.query(CONTACT_TABLE, columns, null, null, null,null, null);
    int iContactName = c.getColumnIndex(CONTACT_NAME);  
    int iContactDate = c.getColumnIndex(CONTACT_DATE);  
    int iContactMobileNumber = c.getColumnIndex(CONTACT_MOBILE_NUMBER);
    for (c.moveToFirst(); !c.isAfterLast(); c.moveToNext()) {
        ContactHolder data = new ContactHolder();
        data.setName(c.getString(iContactName));
        data.setDate(c.getString(iContactDate));
        data.setNumber(c.getString(iContactMobileNumber));
        contactDetails.add(data);
    }
    return contactDetails;
 }
}
Here ContactHolder is just a getter/setter class to handle contact entities.
First I inserted all Contact information once in my MainActivity by the help of a background thread. It prevents to insert the contact info multiple times.
Something like:
private ArrayList<ContactHolder> contactHolder;
private void setCallLogs(Cursor managedCursor) {
    contactHolder = new ArrayList<ContactHolder>();
    int _number = managedCursor
            .getColumnIndex(ContactsContract.CommonDataKinds.Phone.NUMBER);
    int _name = managedCursor
            .getColumnIndex(ContactsContract.CommonDataKinds.Phone.DISPLAY_NAME);
    int _id = managedCursor
            .getColumnIndex(ContactsContract.CommonDataKinds.Phone._ID);
    while (managedCursor.moveToNext()) {
        ContactHolder holder = new ContactHolder();
        holder.setNumber(managedCursor.getString(_number));
        holder.setName(managedCursor.getString(_name));
        holder.setDate(managedCursor.getString(_id));
        contactHolder.add(holder);
    }
    Thread t = new Thread(new Runnable() {
        @Override
        public void run() {
            for(int i=0; i<contactHolder.size(); i++){
                try{
                    ContactMerger merger = new ContactMerger(HomeActivity.this);
                    merger.open();
                    merger.insertContacts(contactHolder.get(i).getName(),
                            contactHolder.get(i).getNumber(),
                            contactHolder.get(i).getdate());
                    merger.close();
                } catch(Exception e){
                     e.printStackTrace();
                }
            }
        }
    });
    t.start();  
}
At last I gtt all contact information inside an Asynctask(doInbackground()) and put in adapter/listview in its onPostExecute() method in the class I want to show. 
Here:
@Override
    protected ArrayList<ContactHolder> doInBackground(String... parameters) {
        ArrayList<ContactHolder> filterContacts = new ArrayList<ContactHolder>();
        ContactMerger merger = new ContactMerger(Aaja_Contact.this);
        merger.open();
        try {
            filterContacts = merger.getContactDetails();
        } catch (Exception e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
        merger.close();
        return filterContacts;
    }