19 messages in com.perforce.jamming[jamming] Batching compile actions _d...| From | Sent On | Attachments |
|---|---|---|
| Peter Björklund | 13 Sep 2003 15:40 | |
| Peter Björklund | 14 Sep 2003 02:11 | |
| Alen Ladavac | 14 Sep 2003 04:30 | |
| Peter Björklund | 14 Sep 2003 05:57 | |
| Matt Armstrong | 14 Sep 2003 07:51 | |
| Alen Ladavac | 14 Sep 2003 09:00 | |
| Alen Ladavac | 14 Sep 2003 09:40 | |
| Alen Ladavac | 14 Sep 2003 11:00 | |
| Matt Armstrong | 14 Sep 2003 13:35 | |
| Alen Ladavac | 14 Sep 2003 14:02 | |
| Matt Armstrong | 15 Sep 2003 09:41 | |
| Wallace, Richard | 17 Sep 2003 17:13 | |
| Paul Forgey | 18 Sep 2003 11:17 | |
| Wallace, Richard | 19 Sep 2003 13:43 | |
| Paul Forgey | 19 Sep 2003 14:53 | |
| Wallace, Richard | 19 Sep 2003 15:13 | |
| Paul Forgey | 19 Sep 2003 15:30 | |
| Wallace, Richard | 19 Sep 2003 16:01 | |
| Ingo Weinhold | 20 Sep 2003 02:56 |
| Subject: | [jamming] Batching compile actions _does_ make a difference! _Huge_ difference! |
|---|---|
| From: | Alen Ladavac (al...@croteam.com) |
| Date: | 09/14/2003 11:00:51 AM |
| List: | com.perforce.jamming |
Hi Matt,
I would be interested to see how your tests turned out if they were done in the same way jam might batch compile. E.g. compile groups of 10 or 20 source files together.
Why only 10 or 20? Why not all files with same flags? And even if it is only 10 files at the time, it will be much better than 1 at a time.
This is the kind of test I ran 9 months ago and did not see the performance gain you see (this was with VC++ 6.0).
Yes, I know, I've seen your posts, but but the key points are:
1) Large project. 2) Lot of includes (DX9 and windows.h are evil). 3) Must use precompiled headers, and must use the manual version (/Yc, /Yu), not automatic one (/YX) as it is not good enough. 4) There are no tricks, the compiler used by the .vcproj is same command line cl.exe, the GUI front end is just sugar coating. 5) Batching does matter. A lot.
The thing is that for a lot of includes, you _must_ have precompiled headers, and must have them tuned well. Without it, it is so slow that nothing makes a difference. But if you have precompiled headers for a large project, the .pch file is several megabytes (cca 7MB in the library B). Each invocation of the compiler must load that file. I don't know exactly what is going on, but I speculate that even though the .pch stays in file cache, compiler must parse its data and recreate its internal lists from it.
Also, I wonder if pre-compiled headers entered the picture when compiling with VC++.NET. They usually don't with jam.
Compiling from withing the .NETs GUI or from Jam is essentially the same. Except for the infamous batching.
Compare command lines below if you don't believe me and you'll see that all switches are same except that they build into different target dirs, and that Jam has some clutter specifying bogus /I. /I.. /I<default include path>.
-------- sample output from Jam building (with -dx)
D:\VS.NET2003\VC7\bin\cl /nologo /c /Fo..\..\Obj.Win32\Debug\Engine\World\ScriptingDomain.obj /MDd /W3 /WX /Gm /GS /Zi /Od /RTC1 /D WIN32 /D _DEBUG /D _WINDLL /D _MBCS /EHsc /Wp64 /D ENGINE_EXPORTS /YuEngine/StdH.h /Fp..\..\Bin.Win32\Debug\Engine.pch /Fd..\..\Bin.Win32\Debug\Engine.pdb /I. /I.. /ID:\VS.NET2003\VC7\include /I.. /Tp..\Engine\World\ScriptingDomain.cpp ScriptingDomain.cpp D:\VS.NET2003\VC7\bin\cl /nologo /c /Fo..\..\Obj.Win32\Debug\Engine\World\Simulation.obj /MDd /W3 /WX /Gm /GS /Zi /Od /RTC1 /D WIN32 /D _DEBUG /D _WINDLL /D _MBCS /EHsc /Wp64 /D ENGINE_EXPORTS /YuEngine/StdH.h /Fp..\..\Bin.Win32\Debug\Engine.pch /Fd..\..\Bin.Win32\Debug\Engine.pdb /I. /I.. /ID:\VS.NET2003\VC7\include /I.. /Tp..\Engine\World\Simulation.cpp Simulation.cpp D:\VS.NET2003\VC7\bin\cl /nologo /c /Fo..\..\Obj.Win32\Debug\Engine\World\World.obj /MDd /W3 /WX /Gm /GS /Zi /Od /RTC1 /D WIN32 /D _DEBUG /D _WINDLL /D _MBCS /EHsc /Wp64 /D ENGINE_EXPORTS /YuEngine/StdH.h /Fp..\..\Bin.Win32\Debug\Engine.pch /Fd..\..\Bin.Win32\Debug\Engine.pdb /I. /I.. /ID:\VS.NET2003\VC7\include /I.. /Tp..\Engine\World\World.cpp World.cpp
-------- etc, etc.
-------- sample output from .vcproj building
Creating command line "d:\Work\main\Sources\Engine\Debug\BAT000013.bat" Creating temporary file "d:\Work\main\Sources\Engine\Debug\RSP000014.rsp" with contents [ /Od /I "../" /D "WIN32" /D "_DEBUG" /D "ENGINE_EXPORTS" /D "_WINDLL" /Gm /EHsc /RTC1 /MDd /GS /Yu"Engine/StdH.h" /Fp"Debug/EngineD.pch" /Fo"Debug/" /Fd"Debug/vc70.pdb" /W3 /WX /c /Wp64 /Zi /TP .\World\World.cpp .\World\Simulation.cpp .\World\ScriptingDomain.cpp <snip... it goes on listing about 250 files on one invocation!!!> ] Creating command line "cl.exe @d:\Work\main\Sources\Engine\Debug\RSP000014.rsp /nologo"
-------- etc, etc.
I am interested, how did you arrange for Jam to compile in batches when you did your test?
Alen
----- Original Message ----- From: "Matt Armstrong" <ma...@lickey.com> To: "Alen Ladavac" <al...@croteam.com> Cc: <jamm...@perforce.com> Sent: Sunday, September 14, 2003 2:51 PM Subject: Re: [jamming] Batching compile actions _does_ make a difference! _Huge_ difference!
Alen Ladavac <al...@croteam.com> writes: [...]
library A: separate: 31 seconds batched: 22 seconds library B: separate: 351 seconds batched: 195 seconds
Horrors... this is almost _2 times_ slower than it should be!
"Batched" case was measured as being compiled using the Visual's proprietary .vcproj format (for non-Win32 people, .vcproj is a very weird concept of holding a list of .cpp files and their settings in an .xml file).
I would be interested to see how your tests turned out if they were done in the same way jam might batch compile. E.g. compile groups of 10 or 20 source files together. This is the kind of test I ran 9 months ago and did not see the performance gain you see (this was with VC++ 6.0).
Also, I wonder if pre-compiled headers entered the picture when compiling with VC++.NET. They usually don't with jam.