It is possible to have a storyboard instantiate different subclasses of a custom view controller, though it involves a slightly unorthodox technique: overriding the alloc method for the view controller.  When the custom view controller is created, the overridden alloc method in fact returns the result of running alloc on the subclass.
I should preface the answer with the proviso that, although I have tested it in various scenarios and received no errors, I can't ensure that it will cope with more complex set ups (but I see no reason why it shouldn't work).  Also, I have not submitted any apps using this method, so there is the outside chance that it might be rejected by Apple's review process (though again I see no reason why it should).
For demonstration purposes, I have a subclass of UIViewController called TestViewController, which has a UILabel IBOutlet, and an IBAction.  In my storyboard, I have added a view controller and amended its class to TestViewController, and hooked up the IBOutlet to a UILabel and the IBAction to a UIButton.  I present the TestViewController by way of a modal segue triggered by a UIButton on the preceding viewController.

To control which class is instantiated, I have added a static variable and associated class methods so get/set the subclass to be used (I guess one could adopt other ways of determining which subclass is to be instantiated):
TestViewController.m:
#import "TestViewController.h"
@interface TestViewController ()
@end
@implementation TestViewController
static NSString *_classForStoryboard;
+(NSString *)classForStoryboard {
    return [_classForStoryboard copy];
}
+(void)setClassForStoryBoard:(NSString *)classString {
    if ([NSClassFromString(classString) isSubclassOfClass:[self class]]) {
        _classForStoryboard = [classString copy];
    } else {
        NSLog(@"Warning: %@ is not a subclass of %@, reverting to base class", classString, NSStringFromClass([self class]));
        _classForStoryboard = nil;
    }
}
+(instancetype)alloc {
    if (_classForStoryboard == nil) {
        return [super alloc];
    } else {
        if (NSClassFromString(_classForStoryboard) != [self class]) {
            TestViewController *subclassedVC = [NSClassFromString(_classForStoryboard) alloc];
            return subclassedVC;
        } else {
            return [super alloc];
        }
    }
}
For my test I have two subclasses of TestViewController: RedTestViewController and GreenTestViewController.  The subclasses each have additional properties and each override viewDidLoad to change the background colour of the view and update the text of the UILabel IBOutlet:
RedTestViewController.m:
- (void)viewDidLoad {
    [super viewDidLoad];
    // Do any additional setup after loading the view.
    self.view.backgroundColor = [UIColor redColor];
    self.testLabel.text = @"Set by RedTestVC";
}
GreenTestViewController.m:
- (void)viewDidLoad {
    [super viewDidLoad];
    self.view.backgroundColor = [UIColor greenColor];
    self.testLabel.text = @"Set by GreenTestVC";
}
On some occasions I might want to instantiate TestViewController itself, on other occasions RedTestViewController or GreenTestViewController.  In the preceding view controller, I do this at random as follows:
NSInteger vcIndex = arc4random_uniform(4);
if (vcIndex == 0) {
    NSLog(@"Chose TestVC");
    [TestViewController setClassForStoryBoard:@"TestViewController"];
} else if (vcIndex == 1) {
    NSLog(@"Chose RedVC");
    [TestViewController setClassForStoryBoard:@"RedTestViewController"];
} else if (vcIndex == 2) {
    NSLog(@"Chose BlueVC");
    [TestViewController setClassForStoryBoard:@"BlueTestViewController"];
} else {
    NSLog(@"Chose GreenVC");
    [TestViewController setClassForStoryBoard:@"GreenTestViewController"];
}
Note that the setClassForStoryBoard method checks to ensure that the class name requested is indeed a subclass of TestViewController, to avoid any mix-ups.  The reference above to BlueTestViewController is there to test this functionality.