RE: [Webtest] OutOfMemoryError - Greg Vatman - com.canoo.lists.webtest

10 messages in com.canoo.lists.webtestRE: [Webtest] OutOfMemoryError

From	Sent On	Attachments
Greg Vatman	23 Sep 2004 12:48
rxh...@regence.com	23 Sep 2004 13:04
Dierk Koenig	24 Sep 2004 00:17
Greg Vatman	27 Sep 2004 15:44
Luca Scheuring	28 Sep 2004 00:25
Greg Vatman	29 Sep 2004 14:58
Edgar P Dollin	30 Sep 2004 05:36
Bill Milbratz	30 Sep 2004 12:51
Greg Vatman	30 Sep 2004 17:33
Dierk Koenig	11 Oct 2004 09:11

Subject:	RE: [Webtest] OutOfMemoryError
From:	Greg Vatman (greg...@paymap.com)
Date:	09/29/2004 02:58:57 PM
List:	com.canoo.lists.webtest

After much painful debugging, I have finally discovered the problem and found
the solution.

During one of my OutOfMemoryError messages, I received some extra information
about the heap:

compacting perm gen total 65536K, used 65535K [0x14010000, 0x18010000,
0x18010000) the space 65536K, 99% used [0x14010000, 0x1800fd98, 0x1800fe00, 0x18010000)

It seems the permanent generation is spaced used by Sun's HotSpot to optimize
performance and is seperate from heap sizes set by using Xmx and Xms options.
This explains why I never reach my maximum heap size before getting
OutOfMemoryError.

Ant is loading a huge number of jars and using many classes and methods, all of
which are being dumped into the permanent space. The default size is 32M for
clients and 64M for servers. I guess that is why some people solved their issues
by using the -server option.

My solution was to use the following option in ANT_OPTS: -XX:MaxPermSize=128m

Now I'm able to run the entire suite without error.

-----Original Message----- From: webt...@gate2.canoo.com [mailto:webt...@gate2.canoo.com]On Behalf Of Luca Scheuring Sent: Tuesday, September 28, 2004 12:26 AM To: webt...@gate2.canoo.com Subject: AW: [Webtest] OutOfMemoryError

I had a very similar problem recently and when you say that you have a huge allocation in StringBuffer, it's sounds familiar to me.

first I was running my testsuite on windows where it worked fine, memory consumption around 50MB. than I tried to run the same tests on linux. there, I had the OutOfMemoryError. I saw that some strings in the report xml-file were extremely long, up to 100'000 characters. this lead to a huge report file which caused the high memory consumption. the problem was that the language settings of my linux shell was set to LANG=en_US.UTF-8. since there exist some öäü characters in my properties-files, this made string processing (don't know where exactly) freaking out. setting LANG=en_US solved the problem.

maybe you have a similar problem.

--luca

-----Ursprüngliche Nachricht----- Von: webt...@lists.canoo.com [mailto:webt...@lists.canoo.com] Im Auftrag von Greg Vatman Gesendet: Dienstag, 28. September 2004 00:44 An: webt...@gate2.canoo.com Betreff: RE: [Webtest] OutOfMemoryError

Well, I'm about out of solutions.

* I only ever used one call to taskdefs * I tried splitting the xml files out by project (1-2 test cases per project) * I set property webtest.resultreporterclass = com.canoo.webtest.reporting.PlainTextReporter * tried using -XX:+AggressiveHeap * tried setting heap sizes with -Xms and -Xmx

Nothing has helped. Memory consumption always seems to increase to around 150MB and I get an OutOfMemoryError.

Running the script through hprof shows a huge char allocation in StringBuffer:

percent live alloc'ed stack class rank self accum bytes objs bytes objs trace name 46 0.40% 43.02% 39000 238 32523352 209686 129 [C

which maps to this stack trace:

TRACE 129: java.lang.StringBuffer.expandCapacity(StringBuffer.java:202) java.lang.StringBuffer.append(StringBuffer.java:401) java.util.zip.ZipFile.getEntry(ZipFile.java:149) java.util.jar.JarFile.getEntry(JarFile.java:194) java.util.jar.JarFile.getJarEntry(JarFile.java:181)

sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:668) sun.misc.URLClassPath.getResource(URLClassPath.java:157) java.net.URLClassLoader$1.run(URLClassLoader.java:191)

java.security.AccessController.doPrivileged(AccessController.j ava:Native method)

That is alot of unused chars...

I'm out of ideas at this point. Is anyone else running a large suite with many steps? Has anyone been able to solve a similar issue? I really hoping I don't have to scrap my WebTest project.

-----Original Message----- From: webt...@gate2.canoo.com [mailto:webt...@gate2.canoo.com]On Behalf Of Dierk Koenig Sent: Friday, September 24, 2004 12:18 AM To: webt...@gate2.canoo.com Subject: RE: [Webtest] OutOfMemoryError

Hi Greg,

there are two major reasons for excessive memory consumption: 1) too many calls to taskdef 2) large result (xml) files

ad 1) every time your tests hit a taskdef a new classloader is constructed reading all webtest (httunit, etc.) classes in Memory. Try having only one taskdef. The /selftests dir that comes with the doc.zip give an example on how to do that.

ad 2) when collecting too many results in one xml file the "append" to that file first reads the file, parses it, does the append and writes it back. This can result in a large memory consumption for large suites. Solutions: a) use the PlainTextReporter if its output is good enough for your purposes b) use a new xml file for each test or sub-suite and merge them afterwards I'm afraid we don't have an example for this, yet.

In our environment 2000 Steps is an upper limit for what we put into one result.xml. A few hundred would be a typical value.

cheers Mittie

-----Original Message----- From: webt...@lists.canoo.com [mailto:webt...@lists.canoo.com]On Behalf Of Greg Vatman Sent: Donnerstag, 23. September 2004 21:49 To: webt...@gate2.canoo.com Subject: [Webtest] OutOfMemoryError

I'm attemping to run about 35 automated test cases, but I keep getting an OutOfMemoryError exception. These functional tests are pretty large and have many steps involved. I've tried increasing the JVM heap size using -Xms and -Xmx (in ANT_OPTS), but it doesn't help. The memory error always seems to occur after the 28th test (at which point a little over 2000 test steps have been executed). I've check the mail archive and checked for previous issues: * showhtmlparseroutput is set to false * &definition; is only used in the main test suite file * using the latest version of Ant (1.6.2) and WebTest (507) Is anyone having success running a large set of tests with many steps? Does anyone have any other suggestions? Thanks in advance. - Greg