atom feed7 messages in org.apache.hadoop.core-userRE: hadoop does not see my input file
FromSent OnAttachments
Erdong (Roger) CHENJun 1, 2007 9:10 pm 
Erdong (Roger) CHENJun 2, 2007 5:27 am 
Erdong (Roger) CHENJun 2, 2007 2:15 pm 
Erdong (Roger) CHENJun 2, 2007 3:47 pm 
Devaraj DasJun 2, 2007 11:07 pm 
Victor GaoJun 3, 2007 1:49 am 
Erdong (Roger) CHENJun 3, 2007 8:24 am 
Subject:RE: hadoop does not see my input file
From:Victor Gao (gaol@gmail.com)
Date:Jun 3, 2007 1:49:21 am
List:org.apache.hadoop.core-user

Hi, I think you should copy source files to the HDFS like this: ./bin/hadoop dfs -cp <some file> /text

And remember the path in the your wordcount command should be a path in HDFS rather than ordinary path in your local filesystem.

Besides, a tiny suggestion: turn off the firewall if possilbe. I found the firewall would cause some trouble. Good luck.

Liqi Gao

-----Original Message----- From: Erdong (Roger) CHEN [mailto:roge@gmail.com] Sent: Sunday, June 03, 2007 6:47 AM To: hado@lucene.apache.org Subject: hadoop does not see my input file

Hi all,

Could anyone help me to figure out why hadoop does not see my input file?

I have three computers rosetta8, rosetta9,and rosetta10. rosetta8 is listed in masters, rosetta9 and rosetta10 are listed in slaves. I run bin/start-dfs.sh and bin/start-mapred.sh on rosetta8. This is my hapood-site.xml. I am pretty sure that I followed the installation and configuration online and the folder /tmp/in-dir/ is not empty.

I tried the following two commands: ./bin/hadoop dfs -ls /tmp/in-dir/ Found 0 items ./bin/hadoop dfs -ls /tmp/ Found 0 items

I tried both two settings for mapred.job.tracker, local and rosetta8:50034. Both don't work.

<property> <name>mapred.job.tracker</name> <value>local</value> <value>rosetta8:50034</value> </property>

<property> <name>fs.default.name</name> <value>rosetta8:50033</value> </property>

Command that I run: ./bin/hadoop jar hadoop-0.12.3-examples.jar wordcount -m 3 -r 2 /tmp/in-dir/ /tmp/out-dir/

Error message that I get: org.apache.hadoop.mapred.InvalidInputException: Input path doesnt exist : /tmp/in-dir at org.apache.hadoop.mapred.InputFormatBase.validateInput(InputFormatBase.java: 138) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:326) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:543) at org.apache.hadoop.examples.WordCount.main(WordCount.java:148) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39 ) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl .java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver .java:71) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:143) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:40) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39 ) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl .java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.apache.hadoop.util.RunJar.main(RunJar.java:155) edc@rosetta8:~/hadoop-install/hadoop$ ./bin/hadoop jar hadoop-0.12.3-examples.jar wordcount -m 3 -r 2/afs/csail.mit.edu/u/e/edc/hadoop-install/hadoop/in-dir/ /tmp/out-dir/ ERROR: Integer expected instead of 2/afs/csail.mit.edu/u/e/edc/hadoop-install/hadoop/in-dir/ wordcount [-m <maps>] [-r <reduces>] <input> <output> edc@rosetta8:~/hadoop-install/hadoop$ ./bin/hadoop jar hadoop-0.12.3-examples.jar wordcount -m 3 -r 2 /afs/csail.mit.edu/u/e/edc/hadoop-install/hadoop/in-dir/ /tmp/out-dir/ org.apache.hadoop.mapred.InvalidInputException: Input path doesnt exist : /afs/csail.mit.edu/u/e/edc/hadoop-install/hadoop/in-dir at org.apache.hadoop.mapred.InputFormatBase.validateInput(InputFormatBase.java: 138) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:326) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:543) at org.apache.hadoop.examples.WordCount.main(WordCount.java:148) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39 ) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl .java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver .java:71) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:143) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:40) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39 ) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl .java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.apache.hadoop.util.RunJar.main(RunJar.java:155)

Erdong Chen