It seems in extended selection mode IsSelected binding is buggy. Looks like only last item from selection which become out of scope is handled properly.
Demonstration:
Items 0, 1, 2 and 98, 97, 96 are selected with Control. When selecting 94 (without Control!) selection counter should be 1, but you see 3 instead. Scrolling up reveals what only one (last) item of selection out of scope was unselected.
Below is mcve:
xaml:
<ListBox ItemsSource="{Binding Items}" SelectionMode="Extended" SelectionChanged="ListBox_SelectionChanged">
    <ListBox.ItemTemplate>
        <DataTemplate>
            <TextBlock Text="{Binding Text}" />
        </DataTemplate>
    </ListBox.ItemTemplate>
    <ListBox.ItemContainerStyle>
        <Style TargetType="ListBoxItem">
            <Setter Property="IsSelected" Value="{Binding IsSelected}" />
        </Style>
    </ListBox.ItemContainerStyle>
</ListBox>
cs:
public class NotifyPropertyChanged : INotifyPropertyChanged
{
    public event PropertyChangedEventHandler PropertyChanged;
    public void OnPropertyChanged([CallerMemberName] string property = "") => PropertyChanged?.Invoke(this, new PropertyChangedEventArgs(property));
}
public class Item : NotifyPropertyChanged
{
    bool _isSelected;
    public bool IsSelected
    {
        get { return _isSelected; }
        set { _isSelected = value; }
    }
    public string Text { get; set; }
}
public class ViewModel : NotifyPropertyChanged
{
    public ObservableCollection<Item> Items { get; }
    public ViewModel()
    {
        var list = new List<Item>();
        for (int i = 0; i < 100; i++)
            list.Add(new Item() { Text = i.ToString() });
        Items = new ObservableCollection<Item>(list);
    }
}
public partial class MainWindow : Window
{
    public MainWindow()
    {
        InitializeComponent();
        DataContext = new ViewModel();
    }
    void ListBox_SelectionChanged(object sender, SelectionChangedEventArgs e)
    {
        Title = ((ViewModel)DataContext).Items.Count(item => item.IsSelected).ToString();
    }
}
A quick fix is to disable list control (ListBox or ListView) virtualization:
VirtualizingStackPanel.IsVirtualizing="False"
Question: any idea how to fix it without disabling virtualization?

 
    