It would be nice if you could use dot notation on the sort qualifiers, like:
[captainQuery orderByAscending:@"boat.name"];
Unfortunately the attribute dot notation applies only to includeKey:.  So the way to do this to fetch, including the related object, then sort locally using the attribute of the related objects:
// ...
[captainQuery includeKey:@"boat"];
[query findObjectsInBackgroundWithBlock:^(NSArray *captains, NSError *error) {
    NSArray *sortedCaptains = [captains sortedArrayUsingComparator: ^(PFObject *capA, PFObject *capB) {
        return [capA[@"boat"][@"name"] compare:capB[@"boat"][@"name"]];
        // or reverse param order for descending
    }];
}];
EDIT - @Logan helpfully points out that this answer doesn't scale well if your query pages through thousands of captains.  If the captain count is huge, it might be better to include a boat->captain pointer and do the following:
More roundabout-ly, you could fetch boats, orderByAscending on their name, includeKey their captains (if your boats have a back-pointer to captain), and then map over the returned (sorted) boats for their captains.