Um, it does work for setAccessible. See:
class A {
  private String method1() {
    return "Hello World!";
  }
}
and
import java.lang.reflect.Method;
class B {
  public static void main(String[] args) throws Exception {
    System.setSecurityManager(new SecurityManager());
    Class clazz = A.class;
    Method m = clazz.getDeclaredMethod("method1");
    m.setAccessible(true);
  }
}
Results in
Exception in thread "main" java.security.AccessControlException: access denied ("java.lang.reflect.ReflectPermission" "suppressAccessChecks")
        at java.security.AccessControlContext.checkPermission(Unknown Source)
        at java.security.AccessController.checkPermission(Unknown Source)
        at java.lang.SecurityManager.checkPermission(Unknown Source)
        at java.lang.reflect.AccessibleObject.setAccessible(Unknown Source)
        at B.main(B.java:8)
One reason it might've not worked for you is that according to comments in this post it didn't use to work in Java 1.5, but works in 6 and thereafter.
Edit: to deny it for specific jars, you need to either use a policy file, example:
// specific file
grant codeBase "file:/test/path/tools.jar" {
  // no permissions for this one
};
// default to giving all
grant {
  permission java.security.AllPermission;
};
There's two ways of specifying the policy file, either give it as additions to default, or give only those that are specified (source):
If you use
java -Djava.security.manager -Djava.security.policy==someURL SomeApp
(note the double equals) then just the specified policy file will be
  used; all the ones indicated in the security properties file will be
  ignored.
...or implement a custom security manager, which doesn't look that hard. Haven't done that myself though.