I did some benchmarking here when you know the type arguments (a non generic approach wont be very different). CreateDelegate would be the fastest approach for a property if you can't directly access it. With CreateDelegate you get a direct handle to GetGetMethod and GetSetMethod of the PropertyInfo, hence reflection is not used every time.
public static Func<S, T> BuildGetAccessor<S, T>(Expression<Func<S, T>> propertySelector)
{
    return propertySelector.GetPropertyInfo().GetGetMethod().CreateDelegate<Func<S, T>>();
}
public static Action<S, T> BuildSetAccessor<S, T>(Expression<Func<S, T>> propertySelector)
{
    return propertySelector.GetPropertyInfo().GetSetMethod().CreateDelegate<Action<S, T>>();
}
// a generic extension for CreateDelegate
public static T CreateDelegate<T>(this MethodInfo method) where T : class
{
    return Delegate.CreateDelegate(typeof(T), method) as T;
}
public static PropertyInfo GetPropertyInfo<S, T>(this Expression<Func<S, T>> propertySelector)
{
    var body = propertySelector.Body as MemberExpression;
    if (body == null)
        throw new MissingMemberException("something went wrong");
    return body.Member as PropertyInfo;
}
So now you call:
TestClass cwp = new TestClass();
var access = BuildGetAccessor((TestClass t) => t.AnyValue);
var result = access(cwp);
Or even better you can encapsulate the logic in a dedicated class to have a get and set methods on it.
Something like:
public class Accessor<S>
{
    public static Accessor<S, T> Create<T>(Expression<Func<S, T>> memberSelector)
    {
        return new GetterSetter<T>(memberSelector);
    }
    public Accessor<S, T> Get<T>(Expression<Func<S, T>> memberSelector)
    {
        return Create(memberSelector);
    }
    public Accessor()
    {
    }
    class GetterSetter<T> : Accessor<S, T>
    {
        public GetterSetter(Expression<Func<S, T>> memberSelector) : base(memberSelector)
        {
        }
    }
}
public class Accessor<S, T> : Accessor<S>
{
    Func<S, T> Getter;
    Action<S, T> Setter;
    public bool IsReadable { get; private set; }
    public bool IsWritable { get; private set; }
    public T this[S instance]
    {
        get
        {
            if (!IsReadable)
                throw new ArgumentException("Property get method not found.");
            return Getter(instance);
        }
        set
        {
            if (!IsWritable)
                throw new ArgumentException("Property set method not found.");
            Setter(instance, value);
        }
    }
    protected Accessor(Expression<Func<S, T>> memberSelector) //access not given to outside world
    {
        var prop = memberSelector.GetPropertyInfo();
        IsReadable = prop.CanRead;
        IsWritable = prop.CanWrite;
        AssignDelegate(IsReadable, ref Getter, prop.GetGetMethod());
        AssignDelegate(IsWritable, ref Setter, prop.GetSetMethod());
    }
    void AssignDelegate<K>(bool assignable, ref K assignee, MethodInfo assignor) where K : class
    {
        if (assignable)
            assignee = assignor.CreateDelegate<K>();
    }
}
Short and simple. You can carry around an instance of this class for every "class-property" pair you wish to get/set.
Usage:
Person p = new Person { Age = 23 };
var ageAccessor = Accessor<Person>(x => x.Age);
int age = ageAccessor[p]; //gets 23
ageAccessor[p] = 45; //sets 45
Bit bad use of indexers here, you may replace it with dedicated "Get" and "Set" methods, but very intuitive to me :)
To avoid having to specify type each time like,
var ageAccessor = Accessor<Person>(x => x.Age);
var nameAccessor = Accessor<Person>(x => x.Name);
var placeAccessor = Accessor<Person>(x => x.Place);
I made the base Accessor<> class instantiable, which means you can do
var personAccessor = new Accessor<Person>();
var ageAccessor = personAccessor.Get(x => x.Age);
var nameAccessor = personAccessor.Get(x => x.Name);
var placeAccessor = personAccessor.Get(x => x.Place);
Having a base Accessor<> class means you can treat them as one type, for eg,
var personAccessor = new Accessor<Person>();
var personAccessorArray = new Accessor<Person>[] 
                          {
                           personAccessor.Get(x => x.Age), 
                           personAccessor.Get(x => x.Name), 
                           personAccessor.Get(x => x.Place);
                          };