I'm trying out clustered, load-balanced Glassfish 3.1 as a candidate for our production environment, and I'm having problems deploying an application on only SOME instances (not all), and I can't figure out what's wrong. Here's my setup: Two servers with completely identical hardware specs running Windows Server 2008 R2. Glassfish 3.1 installed on non-system drive at D:\Glassfish. Four SSH instances all within one cluster (only cluster, only domain). [code]Server 1 (SSH, with Glassfish DAS, domain domain1) Apache 2.2 Glassfish SSH instance s01instance01 (cluster aspgrp01) Glassfish SSH instance s01instance02 (cluster aspgrp01) Server 2 (SSH, no Glassfish DAS) Apache 2.2 Glassfish SSH instance s02instance01 (cluster aspgrp01) Glassfish SSH instance s02instanco02 (cluster aspgrp01)[/code] Each Apache uses mod_jk to load balance requests for the application to any of the four instances. I had no issues creating this setup. Everything worked fine. Now, when I go to deploy an application. I get this error: [code]Application deployed with name [application name]. WARNING: Command _deploy did not complete successfully on server instance s02instance01: remote failure: Failed to load the application on instance s02instance01. The application will not run properly. Please fix your application and redeploy. Exception while deploying the app [application name] : Aborting, Failed to start container com.sun.enterprise.web.WebContainer. Please see server.log for more details. WARNING: Command _deploy did not complete successfully on server instance s02instance02: remote failure: Failed to load the application on instance s02instance02. The application will not run properly. Please fix your application and redeploy. Exception while deploying the app [application name] : Aborting, Failed to start container com.sun.enterprise.web.WebContainer. Please see server.log for more details.[/code] Note that the error is ONLY for the two instances on Server 2. The application deploys fine on the two instances on Server 1, and in fact I can manually hit the application on either instance (using their unique port numbers), but it does not deploy on Server 2 instances. Because of this, I do not believe the statement "Please fix your application and redeploy" in the error. There is nothing to fix in the application. (After looking at the logs mentioned below), I went to the list of instances, and the two failed-deploy instances said "Restart Required," but the two successful-deploy instances do not. I restarted the instances, but they still said restart required. I ran "list-instances --long=true" from asadmin and got "running; requires restart [pending config changes are: _deploy D:\Glassfish\glassfish\domains\domain1\applications\_internal\[app name]\[app name]3356157984287890234.war; ]" for both of the failed instances. No number of restarts or stop-starts makes this message go away. I can't undeploy it because it says it's not deployed on those two instances. I can't deploy it again because it says it is already deployed on those two instances. I had to stop, delete and recreate the instances in order to attempt another deployment, and I got the same results with that. So, I looked at the server.log files for each of the working instances (starting with everything after I started to deploy it) and everything looks correct. But here is the (not good) log contents from server.log for one of the NOT working instances (both log files are identical, with different paths, instance names, etc.) with sensitive information masked: [code][#|2011-03-29T13:52:14.939-0500|INFO|glassfish3.1|javax.enterprise.system.core.security.com.sun.enterprise.security|_ThreadID=114;_ThreadName=Thread-1;|SEC1002: Security Manager is OFF.|#] [#|2011-03-29T13:52:15.533-0500|INFO|glassfish3.1|javax.enterprise.system.core.security.com.sun.enterprise.security|_ThreadID=114;_ThreadName=Thread-1;|SEC1010: Entering Security Startup Service|#] [#|2011-03-29T13:52:15.596-0500|INFO|glassfish3.1|javax.enterprise.system.core.security.com.sun.enterprise.security|_ThreadID=114;_ThreadName=Thread-1;|SEC1143: Loading policy provider com.sun.enterprise.security.provider.PolicyWrapper.|#] [#|2011-03-29T13:52:15.721-0500|INFO|glassfish3.1|javax.enterprise.system.core.security.com.sun.enterprise.security.auth.realm|_ThreadID=114;_ThreadName=Thread-1;|SEC1115: Realm [admin-realm] of classtype [com.sun.enterprise.security.auth.realm.file.FileRealm] successfully created.|#] [#|2011-03-29T13:52:15.736-0500|INFO|glassfish3.1|javax.enterprise.system.core.security.com.sun.enterprise.security.auth.realm|_ThreadID=114;_ThreadName=Thread-1;|SEC1115: Realm [file] of classtype [com.sun.enterprise.security.auth.realm.file.FileRealm] successfully created.|#] [#|2011-03-29T13:52:15.736-0500|INFO|glassfish3.1|javax.enterprise.system.core.security.com.sun.enterprise.security.auth.realm|_ThreadID=114;_ThreadName=Thread-1;|SEC1115: Realm [certificate] of classtype [com.sun.enterprise.security.auth.realm.certificate.CertificateRealm] successfully created.|#] [#|2011-03-29T13:52:15.752-0500|INFO|glassfish3.1|javax.enterprise.system.core.security.com.sun.enterprise.security|_ThreadID=114;_ThreadName=Thread-1;|SEC1011: Security Service(s) Started Successfully|#] [#|2011-03-29T13:52:16.127-0500|INFO|glassfish3.1|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=114;_ThreadName=Thread-1;|WEB0169: Created HTTP listener [http-listener-1] on host/port [0.0.0.0:28081]|#] [#|2011-03-29T13:52:16.143-0500|INFO|glassfish3.1|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=114;_ThreadName=Thread-1;|WEB0169: Created HTTP listener [http-listener-2] on host/port [0.0.0.0:28182]|#] [#|2011-03-29T13:52:16.143-0500|INFO|glassfish3.1|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=114;_ThreadName=Thread-1;|WEB0169: Created HTTP listener [admin-listener] on host/port [0.0.0.0:24849]|#] [#|2011-03-29T13:52:16.159-0500|WARNING|glassfish3.1|javax.enterprise.system.container.web.com.sun.enterprise.web.connector.coyote|_ThreadID=114;_ThreadName=Thread-1;|WEB0343: JK properties file D:\Glassfish\glassfish\nodes\server2\s02instance02\config\glassfish-jk.properties does not exist |#] [#|2011-03-29T13:52:16.159-0500|INFO|glassfish3.1|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=114;_ThreadName=Thread-1;|WEB0170: Apache mod_jk/jk2 attached to virtual-server [server] listening on port [20,101]|#] [#|2011-03-29T13:52:16.159-0500|SEVERE|glassfish3.1|javax.enterprise.system.core.com.sun.enterprise.v3.server|_ThreadID=114;_ThreadName=Thread-1;|Cannot start container web java.lang.IllegalArgumentException: WEB0314: virtual server server has an invalid docroot D:\Glassfish\glassfish\nodes\server2\s02instance02/docroot at com.sun.enterprise.web.WebContainer.validateDocroot(WebContainer.java:1219) at com.sun.enterprise.web.WebContainer.createHost(WebContainer.java:1158) at com.sun.enterprise.web.WebContainer.createHosts(WebContainer.java:1129) at com.sun.enterprise.web.WebContainer.postConstruct(WebContainer.java:559) at com.sun.hk2.component.AbstractCreatorImpl.inject(AbstractCreatorImpl.java:131) at com.sun.hk2.component.ConstructorCreator.initialize(ConstructorCreator.java:91) at com.sun.hk2.component.AbstractCreatorImpl.get(AbstractCreatorImpl.java:82) at com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:67) at com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.java:139) at com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:76) at org.glassfish.internal.data.EngineInfo.getContainer(EngineInfo.java:93) at com.sun.enterprise.v3.server.ApplicationLifecycle.startContainers(ApplicationLifecycle.java:955) at com.sun.enterprise.v3.server.ApplicationLifecycle.setupContainerInfos(ApplicationLifecycle.java:684) at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:368) at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:240) at org.glassfish.deployment.admin.InstanceDeployCommand.execute(InstanceDeployCommand.java:186) at com.sun.enterprise.v3.admin.CommandRunnerImpl$1.execute(CommandRunnerImpl.java:355) at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:370) at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1048) at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1200(CommandRunnerImpl.java:96) at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1247) at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1235) at com.sun.enterprise.v3.admin.AdminAdapter.doCommand(AdminAdapter.java:465) at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:222) at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:168) at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:117) at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:234) at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:822) at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:719) at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1013) at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:225) at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137) at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104) at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90) at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79) at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54) at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59) at com.sun.grizzly.ContextTask.run(ContextTask.java:71) at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532) at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513) at java.lang.Thread.run(Thread.java:662) |#] [#|2011-03-29T13:52:16.159-0500|SEVERE|glassfish3.1|javax.enterprise.system.core.com.sun.enterprise.v3.server|_ThreadID=114;_ThreadName=Thread-1;|Aborting, Failed to start container com.sun.enterprise.web.WebContainer|#] [#|2011-03-29T13:52:16.159-0500|SEVERE|glassfish3.1|javax.enterprise.system.core.com.sun.enterprise.v3.server|_ThreadID=114;_ThreadName=Thread-1;|Exception while deploying the app [application name]|#] [#|2011-03-29T13:52:16.159-0500|SEVERE|glassfish3.1|javax.enterprise.system.core.com.sun.enterprise.v3.server|_ThreadID=114;_ThreadName=Thread-1;|Aborting, Failed to start container com.sun.enterprise.web.WebContainer java.lang.Exception: Aborting, Failed to start container com.sun.enterprise.web.WebContainer at com.sun.enterprise.v3.server.ApplicationLifecycle.setupContainerInfos(ApplicationLifecycle.java:687) at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:368) at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:240) at org.glassfish.deployment.admin.InstanceDeployCommand.execute(InstanceDeployCommand.java:186) at com.sun.enterprise.v3.admin.CommandRunnerImpl$1.execute(CommandRunnerImpl.java:355) at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:370) at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1048) at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1200(CommandRunnerImpl.java:96) at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1247) at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1235) at com.sun.enterprise.v3.admin.AdminAdapter.doCommand(AdminAdapter.java:465) at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:222) at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:168) at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:117) at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:234) at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:822) at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:719) at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1013) at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:225) at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137) at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104) at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90) at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79) at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54) at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59) at com.sun.grizzly.ContextTask.run(ContextTask.java:71) at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532) at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513) at java.lang.Thread.run(Thread.java:662) |#] [#|2011-03-29T13:52:16.440-0500|SEVERE|glassfish3.1|javax.enterprise.system.tools.admin.org.glassfish.deployment.admin|_ThreadID=114;_ThreadName=Thread-1;|Exception while deploying the app [application name] : Aborting, Failed to start container com.sun.enterprise.web.WebContainer|#][/code] Does anyone have any idea what's going on here? Thanks, Nick

[Message sent by forum member 'beamerblvd']