You can't define categories for protocols.  There are 2 ways around this:
- use a new formal protocol
 
- use an informal protocol and runtime checking
 
Formal Protocol
Defining a new formal protocol would look like this:
@protocol MyCustomMatrixDelegate <NSMatrixDelegate>
- (void) myNewMethod:(id)sender;
@end
Then you would make your custom class conform to <MyCustomMatrixDelegate> instead of <NSMatrixDelegate>.  If you use this approach, there's something to be aware of:  [self delegate] will likely be declared as id<NSMatrixDelegate>.  This means that you can't do [[self delegate] myNewMethod:obj], because <NSMatrixDelegate> does not declare the myNewMethod: method.
The way around this is to retype the delegate object via casting.  Maybe something like:
- (id<MyCustomMatrixDelegate>) customDelegate {
  return (id<MyCustomMatrixDelegate>)[self delegate];
}
(However, you might want to do some type checking first, like:
if ([[self delegate] conformsToProtocol:@protocol(MyCustomMatrixDelegate)]) {
  return (id<MyCustomMatrixDelegate>)[self delegate];
}
return nil;
)
And then you'd do:
[[self customDelegate] myNewMethod:obj];
Informal Protocol
This is really a fancy name for a category on NSObject:
@interface NSObject (MyCustomMatrixDelegate)
- (void) myNewMethod:(id)sender;
@end
Then you just don't implement the method.  In your class that would send the method, you'd do:
if ([[self delegate] respondsToSelector:@selector(myNewMethod:)]) {
  [[self delegate] myNewMethod:someSenderValue];
}