23 messages in org.apache.incubator.whirr-userRe: Amazon EC2 and HTTP 503 errors: R...| From | Sent On | Attachments |
|---|---|---|
| Paolo Castagna | Oct 28, 2011 8:32 am | |
| Andrei Savu | Oct 29, 2011 5:37 am | |
| Paolo Castagna | Oct 31, 2011 4:41 am | |
| Andrei Savu | Oct 31, 2011 7:31 am | |
| Paolo Castagna | Nov 1, 2011 2:52 am | |
| Andrei Savu | Nov 1, 2011 5:14 am | |
| Paolo Castagna | Nov 2, 2011 7:35 am | |
| Andrei Savu | Nov 2, 2011 7:54 am | |
| Paolo Castagna | Nov 2, 2011 7:58 am | |
| Andrei Savu | Nov 2, 2011 8:04 am | |
| Andrei Savu | Nov 2, 2011 8:11 am | |
| Andrei Savu | Nov 2, 2011 8:11 am | |
| Paolo Castagna | Nov 2, 2011 8:25 am | |
| Andrei Savu | Nov 2, 2011 8:28 am | |
| Paolo Castagna | Nov 2, 2011 8:33 am | |
| Andrei Savu | Nov 2, 2011 8:38 am | |
| Paolo Castagna | Nov 2, 2011 8:51 am | |
| Andrei Savu | Nov 2, 2011 8:56 am | |
| Paolo Castagna | Nov 2, 2011 9:03 am | |
| Andrei Savu | Nov 2, 2011 9:04 am | |
| Andrei Savu | Nov 2, 2011 9:05 am | |
| Paul Baclace | Nov 2, 2011 10:18 pm | |
| Andrei Savu | Nov 3, 2011 12:57 am |
| Subject: | Re: Amazon EC2 and HTTP 503 errors: RequestLimitExceeded | |
|---|---|---|
| From: | Paolo Castagna (cast...@googlemail.com) | |
| Date: | Nov 2, 2011 8:51:44 am | |
| List: | org.apache.incubator.whirr-user | |
Hi Andrei,
I connected to one of the instance which is not listed by the NameNode, but it is running. There are no Java processes running on that machine.
This is what I see in /tmp/logs/stderr.log:
dpkg-preconfigure: unable to re-open stdin: sun-dlj-v1-1 license has already been accepted sun-dlj-v1-1 license has already been accepted sun-dlj-v1-1 license has already been accepted update-alternatives: using /usr/lib/jvm/java-6-sun/bin/HtmlConverter to provide /usr/bin/HtmlConverter (HtmlConverter) in auto mode. update-alternatives: using /usr/lib/jvm/java-6-sun/bin/appletviewer to provide /usr/bin/appletviewer (appletviewer) in auto mode. update-alternatives: using /usr/lib/jvm/java-6-sun/bin/apt to provide /usr/bin/apt (apt) in auto mode. update-alternatives: using /usr/lib/jvm/java-6-sun/bin/extcheck to provide /usr/bin/extcheck (extcheck) in auto mode. update-alternatives: using /usr/lib/jvm/java-6-sun/bin/idlj to provide /usr/bin/idlj (idlj) in auto mode. update-alternatives: using /usr/lib/jvm/java-6-sun/bin/jar to provide /usr/bin/jar (jar) in auto mode. update-alternatives: using /usr/lib/jvm/java-6-sun/bin/jarsigner to provide /usr/bin/jarsigner (jarsigner) in auto mode. update-alternatives: using /usr/lib/jvm/java-6-sun/bin/javac to provide /usr/bin/javac (javac) in auto mode. update-alternatives: using /usr/lib/jvm/java-6-sun/bin/javadoc to provide /usr/bin/javadoc (javadoc) in auto mode. update-alternatives: using /usr/lib/jvm/java-6-sun/bin/javah to provide /usr/bin/javah (javah) in auto mode. update-alternatives: using /usr/lib/jvm/java-6-sun/bin/javap to provide /usr/bin/javap (javap) in auto mode. update-alternatives: using /usr/lib/jvm/java-6-sun/bin/jconsole to provide /usr/bin/jconsole (jconsole) in auto mode. update-alternatives: using /usr/lib/jvm/java-6-sun/bin/jdb to provide /usr/bin/jdb (jdb) in auto mode. update-alternatives: using /usr/lib/jvm/java-6-sun/bin/jhat to provide /usr/bin/jhat (jhat) in auto mode. update-alternatives: using /usr/lib/jvm/java-6-sun/bin/jinfo to provide /usr/bin/jinfo (jinfo) in auto mode. update-alternatives: using /usr/lib/jvm/java-6-sun/bin/jmap to provide /usr/bin/jmap (jmap) in auto mode. update-alternatives: using /usr/lib/jvm/java-6-sun/bin/jps to provide /usr/bin/jps (jps) in auto mode. update-alternatives: using /usr/lib/jvm/java-6-sun/bin/jrunscript to provide /usr/bin/jrunscript (jrunscript) in auto mode. update-alternatives: using /usr/lib/jvm/java-6-sun/bin/jsadebugd to provide /usr/bin/jsadebugd (jsadebugd) in auto mode. update-alternatives: using /usr/lib/jvm/java-6-sun/bin/jstack to provide /usr/bin/jstack (jstack) in auto mode. update-alternatives: using /usr/lib/jvm/java-6-sun/bin/jstat to provide /usr/bin/jstat (jstat) in auto mode. update-alternatives: using /usr/lib/jvm/java-6-sun/bin/jstatd to provide /usr/bin/jstatd (jstatd) in auto mode. update-alternatives: using /usr/lib/jvm/java-6-sun/bin/native2ascii to provide /usr/bin/native2ascii (native2ascii) in auto mode. update-alternatives: using /usr/lib/jvm/java-6-sun/bin/rmic to provide /usr/bin/rmic (rmic) in auto mode. update-alternatives: using /usr/lib/jvm/java-6-sun/bin/schemagen to provide /usr/bin/schemagen (schemagen) in auto mode. update-alternatives: using /usr/lib/jvm/java-6-sun/bin/serialver to provide /usr/bin/serialver (serialver) in auto mode. update-alternatives: using /usr/lib/jvm/java-6-sun/bin/wsgen to provide /usr/bin/wsgen (wsgen) in auto mode. update-alternatives: using /usr/lib/jvm/java-6-sun/bin/wsimport to provide /usr/bin/wsimport (wsimport) in auto mode. update-alternatives: using /usr/lib/jvm/java-6-sun/bin/xjc to provide /usr/bin/xjc (xjc) in auto mode. java version "1.6.0_26" Java(TM) SE Runtime Environment (build 1.6.0_26-b03) Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode) curl: (18) transfer closed with 22890056 bytes remaining to read curl: (22) The requested URL returned error: 404
gzip: stdin: unexpected end of file tar: Unexpected EOF in archive tar: Unexpected EOF in archive tar: Error is not recoverable: exiting now
Is it in /usr/local/hadoop/bin that I should find the shell script to start datanode and tasktracker daemons?
That directory on this instance is empty.
Paolo
On 2 November 2011 15:38, Andrei Savu <savu...@gmail.com> wrote:
Try restarting the daemons. Are they running? Are there errors in the log files in /tmp?
On Wed, Nov 2, 2011 at 5:34 PM, Paolo Castagna <cast...@googlemail.com> wrote:
Hi Andrei, this cluster is still running, I am running a distcp job to copy my data from S3 to HDFS.
The NameNode (via the Web UI) is sitll reporting:
Live Nodes : 8 Dead Nodes : 0 Decommissioning Nodes : 0
I do not see errors in the logs.
I can try to connect to one of the machines which did not join the cluster, but I am not sure what to do to make it join the cluster once I am connected to it.
Paolo
On 2 November 2011 15:29, Andrei Savu <savu...@gmail.com> wrote:
Are you seeing any errors in the logs? Can you check one of the machines that failed to join the cluster? Are you sure they've tried to join the rest of the cluster? Maybe you have to wait a bit more.
-- Andrei Savu
On Wed, Nov 2, 2011 at 5:25 PM, Paolo Castagna <cast...@googlemail.com> wrote:
Hi
On 2 November 2011 14:59, Paolo Castagna <cast...@googlemail.com> wrote:
Hi Andrei, I've just tried again, the only difference in the recipe: whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,16 hadoop-datanode+hadoop-tasktracker I saw the same exception, but now I can connect to the web UIs as usual.
Well, I spoken too soon.
The very same cluster had 17 instances, I can see all of them running via the Amazon console (i.e. I am paying for them), however the NameNode and the JobTracker see only 8 nodes. :-(
Paolo