My solution was much like @epool 's except use EventBus model.
First, create a RxBus: 
RxBus.java
public class RxBus {
    private final Subject<Object, Object> _bus = new SerializedSubject<>(PublishSubject.create());
    public void send(Object o) { _bus.onNext(o); }
    public Observable<Object> toObserverable() { return _bus; }
    public boolean hasObservers() { return _bus.hasObservers(); }
}
Then, you have two way to use RxBus. Create your custom Application class with RxBus reference or create RxBus in Activity/Fragment then pass it to adapter. I'm use the first.
MyApp.java
public class MyApp extends Application {
    private static MyApp _instance;
    private RxBus _bus;
    public static MyApp get() {  return _instance; }
    @Override
    public void onCreate() {
        super.onCreate();
        _instance = this;
        _bus = new RxBus();
    }
    public RxBus bus() { return _bus; }
}
then use 
MyApp.get().bus() 
to get RxBus instance.
The usage of RxBus in Adpater was like this:
public class MyRecyclerAdapter extends ... {
    private RxBus _bus;
    public MykRecyclerAdapter (...) {
        ....
        _bus = MyApp.get().bus();
    }
    public ViewHolder onCreateViewHolder (...) {
        _sub = RxView.longClicks(itemView)  // You can use setOnLongClickListener as the same
              .subscribe(aVoid -> {
                        if (_bus.hasObservers()) { _bus.send(new SomeEvent(...)); }
                    });      
    }
}
You can send any class with _bus.send(), which We will recieve in the Activity:
RxBus bus = MyApp.get().bus();  // get the same RxBus instance
_sub = bus.toObserverable()
            .subscribe(e -> doSomething((SomeEvent) e));
About unsubscribe. 
In MyRecyclerAdapter call _sub.unsubscribe() in clearup() methods and call _sub.unsubscribe() in Activity's onDestory().
@Override
public void onDestroy() {
    super.onDestroy();
    if (_adapter != null) {
        _adapter.cleanup();
    }
    if (_sub != null) {
         _sub.unsubscribe()
    }
}