Hi Daniel:

After further investigation, I finally narrowed down the issue. It was related to the use of security manager when running the Cascader.

java -Djava.security.manager -Djava.security.policy=my.policy

my.policy contains: grant { permission java.security.AllPermission; };

Let me describe my setup. It's actually very simple like the tutorial. - One application acts as a responder. I created and started JMXConnectorServer and DiscoveryResponder, and registered with the MBeanServer. This is done in the init() method. The reponder app has a loop that: init(), sleep(), dispose(). The dispose() method stop the DiscoveryResponder, JMXConnectorServer and unregister the MBean.

- Another application acts as a Cascader. I created CascadingService, register MBean, and add listener to handle notifiaction. On the "start" notification, I mount the subagent, in this case the above responder.

When I ran the Cascader without the security manager, it reacted very well to the responder start/stop and mounting and unmounting the responder. It worked very well even when the sleep time is at 1sec between start and stop and start again.

However, when I ran the Cascader with the security manager (as shown above), the CascadingService would occasionally miss unmount (probably missed 3 in every 10). The missed mountpoints will never be unmounted. When I used JConsole to look at the thread, I found the following message for each stale mountpoint:

Name: ClientNotifForwarder-3 State: WAITING on java.util.HashSet@10769dd Total blocked: 0 Total waited: 39

Stack trace: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:485) com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:392) com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:506) com.sun.jdmk.remote.cascading.proxy.ProxyCascadingAgent.clearProxies(ProxyCascadingAgent.java:1061) - locked com.sun.jdmk.remote.cascading.proxy.ProxyCascadingAgent@1f91f44 com.sun.jdmk.remote.cascading.proxy.ProxyCascadingAgent.cleanup(ProxyCascadingAgent.java:1048) - locked com.sun.jdmk.remote.cascading.proxy.ProxyCascadingAgent@1f91f44 com.sun.jdmk.remote.cascading.proxy.ProxyCascadingAgent.stop(ProxyCascadingAgent.java:1023) - locked com.sun.jdmk.remote.cascading.proxy.ProxyCascadingAgent@1f91f44 com.sun.jdmk.remote.cascading.proxy.ProxyCascadingAgent.stopIfClosed(ProxyCascadingAgent.java:1005) com.sun.jdmk.remote.cascading.proxy.ProxyCascadingAgent.handleJMXConnectionNotification(ProxyCascadingAgent.java:680) - locked com.sun.jdmk.remote.cascading.proxy.ProxyCascadingAgent@1f91f44 com.sun.jdmk.remote.cascading.CascadingAgent$ConnectionListener.handleNotification(CascadingAgent.java:254) javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:257) javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:322) javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:307) javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:229) com.sun.jdmk.remote.cascading.BasicMBeanServerConnectionFactory.handleConnectionNotification(BasicMBeanServerConnectionFactory.java:303) com.sun.jdmk.remote.cascading.BasicMBeanServerConnectionFactory.access$000(BasicMBeanServerConnectionFactory.java:82) com.sun.jdmk.remote.cascading.BasicMBeanServerConnectionFactory$1.handleNotification(BasicMBeanServerConnectionFactory.java:105) javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:257) javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:322) javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:307) javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:229) javax.management.remote.rmi.RMIConnector.sendNotification(RMIConnector.java:402) javax.management.remote.rmi.RMIConnector.close(RMIConnector.java:494) - locked javax.management.remote.rmi.RMIConnector@73c718 javax.management.remote.rmi.RMIConnector.close(RMIConnector.java:406) - locked javax.management.remote.rmi.RMIConnector@73c718 javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.doStop(RMIConnector.java:1603) com.sun.jmx.remote.internal.ClientCommunicatorAdmin.restart(ClientCommunicatorAdmin.java:133) com.sun.jmx.remote.internal.ClientCommunicatorAdmin.gotIOException(ClientCommunicatorAdmin.java:34) javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1422) javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.getMBeanInfo(RMIConnector.java:1039) com.sun.jdmk.remote.cascading.proxy.CascadingProxy.getMBeanInfo(CascadingProxy.java:399) com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getClassName(DefaultMBeanServerInterceptor.java:1780) com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.safeGetClassName(DefaultMBeanServerInterceptor.java:1622) com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanPermission(DefaultMBeanServerInterceptor.java:1789) com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:418) com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:403) com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:506) com.sun.jdmk.remote.cascading.proxy.ProxyCascadingAgent.hideMBean(ProxyCascadingAgent.java:860) - locked com.sun.jdmk.remote.cascading.proxy.ProxyCascadingAgent@1f91f44 com.sun.jdmk.remote.cascading.proxy.ProxyCascadingAgent.handleMBeanServerNotification(ProxyCascadingAgent.java:728) com.sun.jdmk.remote.cascading.proxy.ProxyCascadingAgent$1.handleNotification(ProxyCascadingAgent.java:239) com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:491) com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:457) com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:83)

Any advice on the use of security manager in this case?

Thank you very much for your help.

Thanks, Gabe

daniel fuchs <Dani...@Sun.COM> Sent by: Dani...@Sun.COM 04/10/2009 06:25 AM Please respond to Dani...@Sun.COM

To Gabr...@sybase.com cc dfu...@dev.java.net, use...@opendmk.dev.java.net Subject Re: OpenDMK - CascadingService

Hi Gabriel,

This is not a known issue.

You could try to connect to your application with JConsole - when you suspect it is hanging - go to the thread tab, and click on "detect deadlocks".

You can also try to switch on the trace to try to understand what is happening. Note also that if your remote application is down - because e.g. the machine is rebooting - you could have very long timeouts...

Cheers,

--daniel