Looking through the VisualVM OQL documentation, I don't get the impression that it supports Java method calls, only Java fields. (Some of their examples include .toString(), but that's clearly the JavaScript .toString() rather than the Java one, since they use it to convert a Java String object to a JavaScript string.) So, for example, their length-of-a-string examples all use the private field count rather than the public method length(), and their length-of-a-vector example uses the private field elementCount rather than the public method size().
So the error you're getting is because ConcurrentHashMap has no field named size.
Unfortunately for your query, ConcurrentHashMap doesn't store its size in a field — that would compromise its ability to avoid blocking — so I think you'll have to write something like this:
select { map: x }
from java.util.concurrent.ConcurrentHashMap x
where sum(x.segments, 'it.count') < 10
to sum all the segment-sizes yourself. (Disclaimer: 100% completely untested.)