3

How can I fixed this?

/tmp/hadoop-yarn/staging/ubuntu/.staging/job_1450038005671_0025/job.jar: Under replicated BP-938294433-10.0.1.190-1450037861153:blk_1073744219_3398. Target Replicas is 10 but found 3 replica(s).

I get this when I run hadoop fsck / in my master node. I assume I should change a .xml file in conf or something similar I just don't know which file to change.

Note that dfs.replication in hdfs-site.xml is already set to 3. I don't have dfs.replication.max in my hdfs-site.xml file.

Mona Jalal
  • 34,860
  • 64
  • 239
  • 408

3 Answers3

7

The replication count for files submitted as part of your job (jars, etc) is controlled by the parameter mapreduce.client.submit.file.replication (or mapred.submit.replication in pre 2.4 clusters) in mapred-site.xml. You can adjust this down for clusters that are smaller than 10 nodes, or just ignore the message from fsck.

FWIW, there is a JIRA for this but I doubt it will ever get worked.

highlycaffeinated
  • 19,729
  • 9
  • 60
  • 91
2

You can ignore. /tmp/hadoop-yarn/staging/ubuntu/.staging/job_1450038005671_0025/job.jar, it is a job resource. dfs.replication does not have impact on job resources.

  1. Job resources such as jar files, files passed using -files (distributed cache) will be copied to HDFS using 10 as replication factor
  2. When the job is running, these job resources (code) will be copied to the container/task to process the data
  3. Once the job is completed based up on thresholds these resources will be automatically recycled.

This feature helps in implementing data locality (where code goes to data) while processing the data.

Durga Viswanath Gadiraju
  • 3,896
  • 2
  • 14
  • 21
1

HDFS configuration file hdfs-site.xml should contain dfs.replication property which describes block replication factor:

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>3</value>
  </property>
</configuration>

Default hdfs-site.xml location is /etc/hadoop/hdfs-site.xml

maxteneff
  • 1,523
  • 12
  • 28