17

I am getting following exception when using FileChannel.map

Exception in thread "main" java.lang.IllegalArgumentException: Size exceeds Integer.MAX_VALUE
    at sun.nio.ch.FileChannelImpl.map(Unknown Source)
    at niotest.NioTest.readUsingNio(NioTest.java:38)
    at niotest.NioTest.main(NioTest.java:64)

Quickly looking into OpenJdk implementation shows that the method map(..) in FileChannelImpl takes size of type long as input. But inside the body, it compares it with Integer.MAX_VALUE and throws error if its greater than that. Why take long size as input but limit it to max integer length?

Anyone knows specific reason behind this implementation? or is it some kind of bug?

Source URL - http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/sun/nio/ch/FileChannelImpl.java

I am running this program using 64bit JRE on 64bit Windows-2k8

Jacek Laskowski
  • 72,696
  • 27
  • 242
  • 420
siddharth178
  • 265
  • 3
  • 9

2 Answers2

7

It's not an implementation specific bug. The size is defined in the FileChannel.map as long, but...

size - The size of the region to be mapped; must be non-negative and no greater than Integer.MAX_VALUE

All compliant JVM implementations will be this way. I suspect the reason is a combination of history (who would need to access a file larger than 2GB? ;) and trying to push things forward in later versions of Java (it will be easier to allow values larger than Integer.MAX than it will be to change the data type from int to long.)

A lot of people find this int-based thinking in the Java API regarding anything File very confounding and short sighted. But remember, Java start development in 1995! I'm sure 2GB seemed like a relatively safe value at the time.

Stu Thompson
  • 38,370
  • 19
  • 110
  • 156
  • The real reasons is that ByteBuffer.capacity is limited to the scope of int, NIO came in 1.4 so nothing like '95 – bestsss Jun 11 '12 at 14:17
  • 1
    @bestsss: They chose `int` over `long`, which was beyond adequate at the time. `long` has been in Java since day 0. (All that on 32-bit machines, where using a 64-bit number would have been **expensive**.) And "*yes"*, Java 1.4 was released in Feb 2002, but NIO originated with JSR 51, which formally started in Jan 2000. (In 2000 the average HDD size sold was 10GB? 20GB?) And it is not stretch of the imagination to think that the original genesis of JSR 51 started over a pint of beer between a couple of bright engineers well before that. No latter than the end of 1999, most certainly. – Stu Thompson Jun 11 '12 at 19:21
  • my recollection for HDD is like 80GB and java has been targeting solaris big iron. `File.length()` has been `long` since java inception. Choosing `int` has more to do w/ the internal optimizations and branch predictions in the CPUs. For instance `for(long i=....)` is never optimized or unrolled... and you want loops over ByteBuffer both range check eliminated AND unrolled. – bestsss Jun 11 '12 at 19:31
  • I said "average", not "max". http://en.wikipedia.org/wiki/File:Hard_drive_capacity_over_time.svg *(It's amazing how quickly the numbers seem to be so small. I'be been buying storage from the kilobyte days...It's amazing.)* I think my point stands: `int` over `long` because it made sense the time. You are welcome to split hairs over 1995 vs. 1999, but it's not an exaggeration. – Stu Thompson Jun 11 '12 at 19:45
  • Oh, and Java back then was also targeting the *"write once, run anywhere...including GUI stuff"*...not just heavy Sun kit. – Stu Thompson Jun 11 '12 at 19:49
  • very nice chart btw, i had 80GB 2002. i still think the main reason is optimization based. Having `long` for position/limit/cap means no optimizations, making ByteBuffer practically useless. I guess some expert in optimization compilers can chime in but lattice w/ long values looks horrid idea. About Sun's - it was designed and had the most in mind w/ solaris - the main marketing point of java, i.e. write anywhere - run on solaris. – bestsss Jun 11 '12 at 19:51
  • I really hope the java.nio package support larger files in next version of JDK. – shuangwhywhy Feb 16 '13 at 11:19
  • https://bugs.openjdk.java.net/browse/JDK-7147951?page=com.atlassian.jira.plugin.system.issuetabpanels:changehistory-tabpanel – Grigory Kislin Feb 28 '15 at 19:33
4

ByteBuffer's capacity is limited to Integer.MAX_VALUE, so there is no way to map anything larger than that.

Look at: MappedByteBuffer map(MapMode mode, long position, long size)
position has to be long for obvious reasons.
size is not necessary to be long but in any calculation it has to be promoted - for example position+size has to be a positive long. OS mapping indeed may use long to carry the mapping, map function (mmap) may need to map more than Integer.MAX_VALUE in order to preserve page size but ByteBuffer just can't use that.

Overall int lays very deep in java's design and there is no size_t C alike type, mass utilizing long instead of int will damper the performance. So in the end: if you need greater maps than 2GB, just use more than a single ByteBuffer.

bestsss
  • 11,796
  • 3
  • 53
  • 63