How to fix the "Illegal partition" error in hadoop?

Question

I have written a custom partitioner. When I have number of reduce tasks greater than 1, the job is failing. This is the exception which I'm getting:

 java.io.IOException: Illegal partition for weburl_compositeKey@804746b1 (-1)
 at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:930)
 at org.apache.hadoop.mapred.MapTask$OldOutputCollector.collect(MapTask.java:499)

The code which I have written is

public int getPartition(weburl_compositeKey key, Text value, int numPartitions)
{
    return (key.hashCode()) % numPartitions;
}

This the key.hashCode() equals -719988079 and mod of this value is returning -1.

Appreciate your help on this. Thanks.

harpun · Accepted Answer · 2016-01-04T19:35:58.207

24

The calculated partition number by your custom Partitioner has to be non-negative. Try:

public int getPartition(weburl_compositeKey key, Text value, int numPartitions)
{
    return (key.hashCode() & Integer.MAX_VALUE) % numPartitions;
}

edited Jan 04 '16 at 19:35

answered Feb 22 '13 at 19:38

harpun

4,022
1
36
40

This question received 3 answers - two correct and one incorrect. Too bad the only incorrect one received the best score and the accepted answer mark. – Andrew Skiba Dec 31 '15 at 09:35
@AndrewSkiba thanks for pointing this out. I corrected the answer according to Tanveer's suggestion. – harpun Jan 04 '16 at 19:36
Glad to see that. GJ! – Andrew Skiba Jan 05 '16 at 12:01

score 4 · Answer 2 · answered Jul 15 '15 at 19:56

A warning about using:

public int getPartition(weburl_compositeKey key, Text value, int numPartitions)
{
    return Math.abs(key.hashCode()) % numPartitions;
}

If you hit the case where the key.hashCode() is equal to Integer.MIN_VALUE you will still get a negative partition value. It is an oddity of Java, but Math.abs(Integer.MIN_VALUE) returns Integer.MIN_VALUE ( as in -2147483648). You are safer taking the absolute value of the modulus, as in:

public int getPartition(weburl_compositeKey key, Text value, int numPartitions)
{
    return Math.abs(key.hashCode() % numPartitions);
}

score 2 · Answer 3 · answered Jan 22 '15 at 09:25

2

Or you can use

public int getPartition(weburl_compositeKey key, Text value, int numPartitions)
{
    return (key.hashCode() & Integer.MAX_VALUE) % numPartitions;
}

answered Jan 22 '15 at 09:25

Tanveer

890
12
22

How to fix the "Illegal partition" error in hadoop?

3 Answers3