Update: since some people started a pointless discussion about “how to benchmark” I will emphasize the solution to your problem contained in my answer, now right at the beginning:
You can use invokeExact even in your reflective context where you don’t have the exact type signature by converting the MethodHandle using asType to a handle taking Object as arguments. In environments affected by the performance difference between invoke and invokeExact, using invokeExact on such a converting handle is still way faster than using invoke on a direct method handle.
Original answer:
The problem is indeed that you are not using invokeExact. Below is a little benchmark program showing the results of different ways of incrementing an int field. Using invoke instead of invokeExact leads to a performance drop below the speed of Reflection.
You receive the WrongMethodTypeException because the MethodHandle is strongly typed. It expects an exact invocation signature matching type type of the field and owner. But you can use the handle to create a new MethodHandle wrapping the necessary type conversions. Using invokeExact on that handle using a generic signature (i.e. (Object,Object)Object) will be still way more efficient than using invoke with a dynamic type conversion.
The results on my machine using 1.7.0_40 were:
direct        :   27,415ns
reflection    : 1088,462ns
method handle : 7133,221ns
mh invokeExact:   60,928ns
generic mh    :   68,025ns
and using a -server JVM yields to a baffling
direct        :   26,953ns
reflection    :  629,161ns
method handle : 1513,226ns
mh invokeExact:   22,325ns
generic mh    :   43,608ns
I don’t think that it has much real life relevance seeing a MethodHandle being faster than a direct operation but it proves that MethodHandles are not slow on Java7.
And the generic MethodHandle will still outperform Reflection (whilst using invoke does not).
import java.lang.invoke.MethodHandle;
import java.lang.invoke.MethodHandles;
import java.lang.reflect.Field;
public class FieldMethodHandle
{
  public static void main(String[] args)
  {
    final int warmup=1_000_000, iterations=1_000_000;
    for(int i=0; i<warmup; i++)
    {
      incDirect();
      incByReflection();
      incByDirectHandle();
      incByDirectHandleExact();
      incByGeneric();
    }
    long direct=0, refl=0, handle=0, invokeExact=0, genericH=0;
    for(int i=0; i<iterations; i++)
    {
      final long t0=System.nanoTime();
      incDirect();
      final long t1=System.nanoTime();
      incByReflection();
      final long t2=System.nanoTime();
      incByDirectHandle();
      final long t3=System.nanoTime();
      incByDirectHandleExact();
      final long t4=System.nanoTime();
      incByGeneric();
      final long t5=System.nanoTime();
      direct+=t1-t0;
      refl+=t2-t1;
      handle+=t3-t2;
      invokeExact+=t4-t3;
      genericH+=t5-t4;
    }
    final int result = VALUE.value;
    // check (use) the value to avoid over-optimizations
    if(result != (warmup+iterations)*5) throw new AssertionError();
    double r=1D/iterations;
    System.out.printf("%-14s:\t%8.3fns%n", "direct", direct*r);
    System.out.printf("%-14s:\t%8.3fns%n", "reflection", refl*r);
    System.out.printf("%-14s:\t%8.3fns%n", "method handle", handle*r);
    System.out.printf("%-14s:\t%8.3fns%n", "mh invokeExact", invokeExact*r);
    System.out.printf("%-14s:\t%8.3fns%n", "generic mh", genericH*r);
  }
  static class MyValueHolder
  {
    int value;
  }
  static final MyValueHolder VALUE=new MyValueHolder();
  static final MethodHandles.Lookup LOOKUP=MethodHandles.lookup();
  static final MethodHandle DIRECT_GET_MH, DIRECT_SET_MH;
  static final MethodHandle GENERIC_GET_MH, GENERIC_SET_MH;
  static final Field REFLECTION;
  static
  {
    try
    {
      REFLECTION = MyValueHolder.class.getDeclaredField("value");
      DIRECT_GET_MH = LOOKUP.unreflectGetter(REFLECTION);
      DIRECT_SET_MH = LOOKUP.unreflectSetter(REFLECTION);
      GENERIC_GET_MH = DIRECT_GET_MH.asType(DIRECT_GET_MH.type().generic());
      GENERIC_SET_MH = DIRECT_SET_MH.asType(DIRECT_SET_MH.type().generic());
    }
    catch(NoSuchFieldException | IllegalAccessException ex)
    {
      throw new ExceptionInInitializerError(ex);
    }
  }
  static void incDirect()
  {
    VALUE.value++;
  }
  static void incByReflection()
  {
    try
    {
      REFLECTION.setInt(VALUE, REFLECTION.getInt(VALUE)+1);
    }
    catch(IllegalAccessException ex)
    {
      throw new AssertionError(ex);
    }
  }
  static void incByDirectHandle()
  {
    try
    {
      Object target=VALUE;
      Object o=GENERIC_GET_MH.invoke(target);
      o=((Integer)o)+1;
      DIRECT_SET_MH.invoke(target, o);
    }
    catch(Throwable ex)
    {
      throw new AssertionError(ex);
    }
  }
  static void incByDirectHandleExact()
  {
    try
    {
      DIRECT_SET_MH.invokeExact(VALUE, (int)DIRECT_GET_MH.invokeExact(VALUE)+1);
    }
    catch(Throwable ex)
    {
      throw new AssertionError(ex);
    }
  }
  static void incByGeneric()
  {
    try
    {
      Object target=VALUE;
      Object o=GENERIC_GET_MH.invokeExact(target);
      o=((Integer)o)+1;
      o=GENERIC_SET_MH.invokeExact(target, o);
    }
    catch(Throwable ex)
    {
      throw new AssertionError(ex);
    }
  }
}