21 messages in com.perforce.jamming[jamming] Anyone else using Jam for B...| From | Sent On | Attachments |
|---|---|---|
| Glen Darling | 26 Jul 2001 18:38 | |
| David Abrahams | 31 Jul 2001 12:21 | |
| Roesler, Randy | 01 Aug 2001 12:37 | |
| David Abrahams | 01 Aug 2001 15:05 | |
| Roesler, Randy | 01 Aug 2001 16:55 | |
| Glen Darling | 01 Aug 2001 17:39 | |
| Glen Darling | 01 Aug 2001 18:22 | |
| Roesler, Randy | 01 Aug 2001 21:43 | |
| Arnt Gulbrandsen | 02 Aug 2001 03:34 | |
| Arnt Gulbrandsen | 02 Aug 2001 03:40 | |
| Patrick Frants | 02 Aug 2001 06:28 | |
| David Abrahams | 02 Aug 2001 07:23 | |
| David Abrahams | 02 Aug 2001 07:29 | |
| Glen Darling | 02 Aug 2001 07:44 | |
| Glen Darling | 02 Aug 2001 09:14 | |
| Glen Darling | 02 Aug 2001 09:52 | |
| David Abrahams | 02 Aug 2001 10:22 | |
| David Abrahams | 02 Aug 2001 11:27 | |
| Roesler, Randy | 02 Aug 2001 21:52 | |
| Glen Darling | 02 Aug 2001 23:34 | |
| Craig McPheeters | 06 Aug 2001 17:15 |
| Subject: | [jamming] Anyone else using Jam for BIG projects? |
|---|---|
| From: | Glen Darling (gdar...@cisco.com) |
| Date: | 08/02/2001 11:34:39 PM |
| List: | com.perforce.jamming |
Hi Randy,
At 09:53 PM 08/02/2001 -0700, Roesler, Randy wrote:
You seem to have a very big tree of code but very small results, if you can build that much stuff in 45 minutes.
I guess we have some fast build servers. :-) But we think that its still pretty darn slow. Jam still spends a lot of time goofing around before it gets down to business and fires off the first shell commands. Also we don't have the kind of linking pains that many similar sized projects have.
-----Original Message----- From: Glen Darling [mailto:gdar...@cisco.com]
Hmmm... I'm not sure. I just counted and we have 8,995 ".h" files (comprising 1,863,096 lines) and 9,414 ".c" files (comprising 4,663,950 lines), plus many other file types are scanned in our system (we have several other source types, with various include syntaxes). We also have around 2000 lines of code in our HeaderRules.jam code that manages this
In our environment, a whole tree build from scratch takes around 45 minutes, never hours.
We have links which take more that 30 minutes and produce executables which are larger than 30 Megs.
Our code is designed for modular delivery and linking is primarily dynamic in our environment. We package large multimegabyte entities (though I do not think any have hit 30M yet, they are growing in that direction rapidly) but they are not all cross-linked so they build very fast. That is, our packages bundle up DLLs and small executables into complete images or smaller modules destined for Cisco routers.
I think there is a better way.
For first build, build everything (or everything required for this developer's context). Generate .d files as a byproduct of the build almost for free.
In subsequent builds, determine if any .d files are out-of-date with respect to their corresponding source files -- or *any* of the included files as found in the ".d" file, if so build all those sources in addition to anything that depends on them as required of course. Because the only way to alter the header include file dependencies is to touch either the source file or one of the things it includes, you are guaranteed to catch and update whenever necessary. I am very interested in making jam able to use this technique.
You would be create a lot of .d files, which is probably OK. You would end up timestamping a lot of .cpp and .h files (or .d files), which, over NFS is relativity expensive.
We normally build locally on our build servers (we have build servers locally wherever we have engineers). Tools are mounted over nfs though (on a small handful of sites), but there are only small time penalties for grabbing the tools once each.
(I have a component called "Fake" which fakes out all of our projects build tools. This allows me to simulate a build in order to debug a new set of jamfiles without having to actually build the system.
I don't quite get the purpose of this. We use some test shells to unit test/debug parts of our jam infrastructure, which includes a number of Perl scripts and C-programs invoked during the build. There is also the "-n" option to jam of course.
Of course you wound not. A single C++ file can take 10 minutes to compile on solatis (if its 2,000 lines long) and a executable can take two hours to link (on AIX).
We haven't noticed that kind of thing (in compilation of our C++ code), and as noted above I think our linking requirements are simpler.
I also have a jam.exe which I have doctored to tell me why an target is getting rebuilt. It produces too much output to be generally usable, but it has help track down bad dependency statements)
I have hacked something similar from an opposite perspective but for the same purpose. Given a fully gristed target, it will display the complete DEPENDS (or optionally both DEPENDS and INCLUDES) graph for that target. This is very useful for understanding what's going on "inside" jam when something goes wrong. Each target is only displayed once so the tree doesn't get too gory (e.g., multiple references to a directories, for example, were ugly before I changed that).
My problem is always -- wht did jam build X. If X depends on A, B, and C, I want to know which caused X to recompile. If X includes, directly and indirecty, 400 header files, then simply priniting the dependency tree just gives you too much information.
We usually resort to the -d+3 (for non-header dependencies) or -d+6 (for headers) output for this, or we use our locally hacked jam (which I aluded to above in the previous e-mail) to dump the dependency tree and pore over it.
I'va always tried for somthing like.
Building X because A is newer because B is newer includes newer C includes newer D because C does not exist
so now I can see that D (at the very least, ws touched).
This has helped me find suff where a node depended on itself.
Our hacked jam shows the dependency tree just like the output above, but currently it does not annotate stat info. It could be extended to do that easily I think. Currently when we need that info we use the -d+6 output (i.e., make, time, make* newer, made+ old, made+ update, give that info).
Also consider this
DEPENDS a b : x ; DEPENDS x : y ; TEMPORARY x ;
DEPENDS c : a ; DEPENDS c : b ;
A bug in the handling of temporaries means that during dependency processing, only the first dependency analyzed (a,b to x) will propagate the timestamp of y to (the missing) x; The other target (a/b) will always be cosidered out of date.
I didn't know about this. Thanks.
Glen.