|Julian Löffelhardt||Dec 30, 2002 11:36 am|
|Wagoner, Mark||Dec 30, 2002 11:48 am|
|Jerome "Lacoste (Frisurf)"||Dec 30, 2002 11:51 am|
|Julian Löffelhardt||Dec 30, 2002 12:29 pm|
|Julian Löffelhardt||Dec 30, 2002 12:29 pm|
|Paul Yunusov||Dec 30, 2002 12:30 pm|
|Justin L. Spies||Dec 30, 2002 12:34 pm|
|Remy Maucherat||Dec 30, 2002 12:39 pm|
|Craig R. McClanahan||Dec 30, 2002 12:43 pm|
|mech||Dec 30, 2002 12:50 pm|
|Torsten Fohrer||Dec 30, 2002 2:38 pm|
|Dan Payne||Dec 30, 2002 2:42 pm|
|Peiqiang Han||Dec 30, 2002 3:04 pm|
|Julian Löffelhardt||Dec 30, 2002 4:55 pm|
|Jerome "Lacoste (Frisurf)"||Dec 31, 2002 2:42 am|
|Luc Foisy||Dec 31, 2002 7:43 am|
|Goehring, Chuck Mr., RCI - San Diego||Dec 31, 2002 12:28 pm|
|Jason Pyeron||Dec 31, 2002 12:55 pm|
|Ken Anderson||Dec 31, 2002 2:06 pm|
|Brian Topping||Dec 31, 2002 2:44 pm|
|Gary Gwin||Dec 31, 2002 4:03 pm|
|Triptpal Singh Lamba||Dec 31, 2002 5:21 pm|
|Will Hartung||Jan 2, 2003 2:27 pm|
|Julian Löffelhardt||Jan 2, 2003 5:38 pm|
|Will Hartung||Jan 2, 2003 6:53 pm|
|Subject:||Re: reducing tomcat & jasper memory footprint|
|From:||Will Hartung (wil...@msoft.com)|
|Date:||Jan 2, 2003 6:53:35 pm|
From: "Julian Löffelhardt" <jul...@austria.fm> Sent: Thursday, January 02, 2003 5:38 PM Subject: Re: reducing tomcat & jasper memory footprint
Thanks for your Feedback!
I will look into your suggested approch using filters.
As of 4.0.4 tomcat keeps a reference to every generated servlet, so increasing the memory size only delays the OutOfMemoryError.
Yeah, I saw that blurb. That's unfortunate, because in truth this is the crux of your problem. It is not so bad that a popular JSP gets cached into RAM, that is a good thing. That is what RAM is for. However, in this case, even the rarest and most unpopular of your JSPs are getting cached permanently. Owie.
In fact we are using loadbalancing using apache 1.3.26, mod_jk, and 3 tomcat instances. Every night one tomcat get's restarted. Basically we use the lb-options local_worker & local_worker_only to disable a tomcat instance at midnight and restart it at 4:00.
Otherwise the big problem with restarting tomcat is loss of our session-data. Saving and restoring the data in the SESSIONS.ser file doesn't help, since the loadbalancer routes the web clients to another tomcat instance where they get another sessionId.
Are you able to isolate a particular instance/server so that current users can continue with the site, while no new users go there? If that is possible with your configuration, then you do not really have a session problem as they should be invalid by the user logging out/going away/timing out.
I thought about converting the documents into velocity templates or something like that using a postprocessor but decided against it:
1) Would be quite comlicated, due to the complex structure of the pages. 2) First I would need to "proove" that my suggested different design would work better.
To be fair, replacing JSP with Velocity, particularly a JSP generated by another program (i.e. not written directly by a CBL (Carbon Based Lifeform)) implies that the execution of the compiled JSP page is slower than the intpretation of the Velocity template (modulo the fact that Tomcat does not GC JSPs, but that is a detail for this discussion). I do not have any data either way as to which is faster.
We get about 400.000 PageImpressions/day and under high load many concepts/designs/applications just dont't "deliver" For example, we performed extensive stress-tests on our application but the tests didn't reveal the problems with too many different jsp-Files since our tests where mainly focused on performance issues and we only used about 100 pages for testing.
In truth, there is not anything really wrong with your overall design. Most complaints about JSP focus on maintainability of the JSP files, and the intermixing of code and HTML etc. Velocity implicity suffers from that same issue, though perhaps its syntax and model may be better. But these are not direct issues with your system, as your JSPs are derivative products of your CMS system. In my opinion, your use of JSPs here is no different from using something akin to the Olde Tyme Server Side Includes. The facts seem to be that save for the memory issue, you system is behaving fine, and, as you noted, your JSP solution appears to scale better.
And well it should with all of your pages cached in RAM :-).
So, I would look at the other message in the list that mentioned the Jasper class loaders hash table and perhaps changing that to support an LRU queue rather than a hash table.
I know it is a complicated and not necessarily well understood area for most developers, but it IS the source of your problem. I think that if you spend a little bit of time in that section of code, you have a better chance for success in a shorter amount of time. Also, once you load test the heck out of if and stress it as best as you can, it is probably much safer as from a logical or even implementation point of view, your application does not change at all. Ideally, it can simply be a change from a HashTable (or HashMap, whatever they are using) to an LRUMap (of somekind, this is left as an exercise for the reader :-) -- maybe you can safely use WeakReferences here), with all of the Real Hard Work (tm) done by the surrounding code.
Granted, this means you must go through the nightmare of actually BUILDING Tomcat (not for the meek).
The code in place already deals with loading class files, checking the map for loaded classes, synchronization issues, etc. So what is the difference between checking a HashMap for a class and loading it if it does not exist (in this case the first time the page is loaded), and seeing that the class is not there because it happened to drop off the end of the queue? It should not matter why the class is not in the map, but simply that it is not there.
For normal operations and classes, this is less of an issue, but your classes are really content so they suck up an extraordinary amount of memory. Also, for normal classes this can be a real issue because each time a class is loaded, it is "different" than before (I think). But JSPs are not normal classes like, say, some kind of utility class. The real issue is that there is no way to query the JVM for instances of a class to know whether it is safe to drop the actual class data from a custom loader :-/.
I glanced at the code, but you will need to talk to the Tomcat developers for details. It appears that the Jasper loader delegates to the parent class loader, but it seems to me that those PARENT loaders would be either the StandardClassLoader or WebAppClassLoader, so your change may well be contained here.
So, anyway, it may not be a trivial mod but it will solve your problem and I think it is a doable modification unless the Tomcat Dev Team cries foul and heresy. They may know some subtle problems that such a scheme will aggravate. But you would think this should not be too horrible as by design the container must be able to reload JSP classes (unless of course it does not and simply use jspClass1, jspClass2, jspClass3, etc. -- eek!).
Good Luck and let us know what you do and how it works for you.
(Look at the list, Craig has some comments regarding this...in the Memory Usage and Garbage Collection thread...)
Will Hartung (wil...@msoft.com)