Jam Specifications Poll =======================

Hi Jam users,

This poll is basically a collection of questions and suggestions which I have accumulated over the last few years. Essentially, what I am getting at, is that Jam, while heading in the right direction, isn't quite there yet, and, that with the help of some guidance, we might be able to help fast-track this program to a level of usability which would make it not just the "insider-tip" which it is today, but make jam a compelling replacement for make in almost all environments.

The responses to this poll will be collated and passed on to Christopher Seiwald - so if you have any suggestions or good ideas, they may well see the light of day!

This mail is rather long, so please be patient.

The structure is as follows:

Introduction (you're reading it) Concepts (My vision) Questions (Am I right, am I wrong, can you think of a better way?)

Concepts: =========

I see a build system as something which should be almost as transparent as the operating system, it should be simple to configure, easy to use, reliable, extensible, scalable and above all, predictable. The concepts behind the workings should also be simple, but powerful.

I guess the first question to ask is "How is Jam going to be used?" - Most of the Unix projects I have seen have worked in a hierarchical structure, however, I have also see schemes which work in a modular fashion, where each module is a self contained unit. These two schemes clash badly, but both have their advantages and disadvantages, there may be other schemes - in which case, I hereby ask that you send me an e-mail describing them. The schemes should be general enough to be applied to any software application, especially large ones.

My favorite method, would have to be the hierarchical approach, (which still can be modularised in some aspects) where project specific settings are set in the top directory, and all settings apply to the current directory and all directories below it (general case). This has the advantage of providing very simple configuration files, however, the disadvantage of obscuring inherited values, shouldn't be discounted.

Basic Concepts: - Simple but unrestricted - Hierarchical Tree Structure - Settings inherited by all children of a node (recursive inclusion of Jamfiles) - No Side Effects (between directories at least) - Shadow Trees (see below) - Simple Rule definitions - Extensible - Scalable

Problems of Software Development ================================ Software developers often have problems with: - Concurrent development - To little disk space - To little time - Creating Makefiles (Jamfiles) which work (with little effort) in a variety of environments

I believe that Jam, when supporting the hierarchical approach, can assist in alleviating some of these problems.

I have often seen cases where each member of a development team, has a complete copy of the product in development in their home directories, which they update "from time to time". This approach is *SUPPORTED* by conventional make files, and leads directly to *INCONSISTENT SOFTWARE*, and a shortage of disk space. Everyone is developing with an old version of their neighbour's code.

It is a much better idea to have a central directory with the complete set of source files, in a stable state, and have developers work against this 'known state'. Updates to this central repository should only be made when the updated files have been tested and shown to be (for all intents and purposes) bug free. I strongly suggest the next step, which is to use a configuration management system (CM) (something like RCS, CVS, SCCS, perforce or any of the commercial products which control the changes being made to software in such a way that you can see who did what, when, and why).

Some Solutions ============== One thing which I find a great help is the advent of shadow trees. You can think of a shadow tree as an over-head transparency. The first directory, contains the entire source code, I tend to call this the "central repository". (By the way, the central repository may only be written by an appointed administrator, the best situation is when the administrator is actually a CM system, so that only approved changes become public). The directory tree has a known structure, which the developers duplicate in their private directories (only the directory structure, not the contents). The private directory tree can then be thought of as being "laid over" the central repository. Now when the developer writes files in his directory, the files in his directory take priority over those in the central repository. There could be as many such shadow trees as you like, you simply need to tell the build mechanism where the top of each tree is, and in what order they should be searched. (This can be done easily with a PATH style environment variable).

The advantages are many, including:

- low space usage. The full repository only needs to exist once, the developers only have the files that they are working on in their private directory.

- it fits very nicely with a CM system - developers are forced to 'get' files before they can work on them, and although there is nothing stopping them from copying the files from the central repository, the act of getting the file should encourage them to get the file from the CM system via the correct channels. (with file locking, tracking of who is editing what etc)

- it is very easy for a developer to see what he has done. The chance that old files are left lying around in his private directory, possibly having unforeseen effects on the resulting software, is greatly reduced.

- what happens in a developer's private area is entirely his own business. It is impossible for him to effect someone else's work, without checking his software in, and having it extracted in the central repository. (Although this doesn't mean that bugs can't be introduced, it means that they are unlikely to happen due to accidental file changes.)

- reduced build times, as only those files which the user has actually changed are updated. The central repository should be updated at regular intervals (once a day or so) so there is no need for the developer to create libraries for the parts of the software which he is not actually working on.

But, nothing is perfect, some catches of shadow trees are:

- included files could be found in any of many directories. This means that the build system has to construct long lists of -I options for the compiler (for C and C++ at least). This can be alleviated by well structured code, and things like having relative path names used in all #includes. If you can do this, then you only need one -I option per tree, and you (as a developer) are explicitly stating exactly which file you need.

- the files which a developer doesn't have in his private area are a little more difficult for the developer to find if he wants to reference existing code. Again, there are ways to work with this. One that springs to mind is tags. Both emacs and vi can use multiple tags files, so if the central repository has a tags file at its root, and the developer creates his own tags file for the files in his environment (via the build system), then both vi and emacs users can 'magically' find the code they are looking for in most cases. It gets a little more complex for C++, but what doesn't get more complex when you are working with C++?

Gmake has introduced the idea of the VPATH, which may appear to address this concept, however this is a clumsy hack which has little relevance to a useful shadow tree system. VPATH is just too simplistic be of any real help.

Jam's SEARCH and LOCATE variables are much better than VPATH, and indeed with a little Jambase hacking, I've managed to almost make it. Unfortunately, jam insists on building all the object files, libraries and binaries in the private area, which kind of kills a lot of the above mentioned advantages, but jam isn't finished yet :-)

This concept seems to have missed large scale work for some reason. I am yet to see a system which can really handle shadow trees in a transparent manner - one of the reasons for me writing this questionnaire is to find out if there are reasons for NOT working like this, or if there is a silent majority who would also like to see something like this happen, in which case, it's time something happened.

Concerns ========

Performance. While performance is not critical for a build system, a complete re-build should never take more than 12 hours. If this is the case, then it is very difficult for development to make progress with up-to-date binaries / libraries. I have worked on a project which took up to 36 hours to compile, and I've seen the positive effects that a well thought out system can bring. (reduced compile time to ~18 Hours, 8 hours for incremental builds, and the ability to use the results from yesterdays build, while building a new one)

Reproducibility. If it happened once, it must be able to happen again. It should be possible from the output to deduce how the build was started, and why each action was carried out. In the case of errors, it should be possible to find out from the error messages, exactly which files are involved, and exactly what has gone wrong. Using gmake as a bad example, messages like: entering directory src/ cc: main.c syntax error might be helpfull, but buried in a log file which may well be megabytes long, you might be asking yourself, exactly which src directory has been entered, and thus which main.c. It is much better (for example) when you get messages like: entering module1/gui/src cc: module1/gui/src/main.c syntax error here, both the build system and the compiler state in one hit exactly which file/directory is being accessed.

Reliability. It should be possible to build several versions of the software without interfering with each other. You may for example want a simple debugging version, and a version with profiling, and another version with optimisation turned on. There should be no reason why all of these can't exist at the same time, or why each shouldn't be able to exist on it's own.

Questions: ==========

Describe your development process. - do you have a central software repository? - how are changes to the software managed? are they recorded in a system like RCS, SCCS, CVS et. al.? Can developers change files at will, or do they first have to "check them out" etc? - how are conflicts between developers handled (not personal conflicts, but can two developers edit the same file at the same time, if they can, would it be possible for them to destroy each others changes without knowing?) - do developers share their source code, or does each work in his own private area. Is it possible for one developer to inadvertently interfere with another developers' work? - does each developer have an entire copy of the sources in his private area?

Do you prefer a hierarchical approach, or modular approach? why?

Do you prefer to enforce stringent rules to prevent inconsistencies, or should developers have free access to all source code. (By this I mean that all developers have write access to the code.)

Is the concept of a shadow tree new to you?

Does jam fit your development process the way it (jam) is?

Did you have to change Jambase to make jam useful to you? if so, what changes did you have to make?

What sort of 'logging' do you require from a build system? Do you need timing information? (how long does each module take to compile? Where is the most time taken? Can the compilation time be reduced by improving the organisation of the libraries etc?)

Have you thought of producing build reports, in which just the errors are printed, while all the 'detailed' build information remains in the log file as reference material (when things go wrong, it's often the case that a chain reaction has taken place)

What features would you most like to see in your build system?

Conclusion ==========

Well, that's about all I have time for at the moment, there are still hundreds of topics which I would like to discuss relating to build systems, both those in use today, and in the 'ideal' system. Things like file name binding, CM systems and dependencies haven't been addressed by this poll, but they are all equally as important as the above, and I hope to see them find their way into this discussion as time goes on.

I would really like to see a discussion about this, complete with gory details, so that we can pull out some positive suggestions, and avoid some nasty traps. Perhaps Jam will become that ideal system, perhaps it will be 'Cons' - it almost certainly wont be any form of 'make'.

OK, lets see what you all have to say. (reply to jamming, or e-mail to sr...@pc-plus.de)

Thanks for taking the time,

Steve

_______________________________ ._ o ________________________________ // Stephen Riehm / //\. Step...@pc-plus.de \\\\ pc-plus ' \>> | Phone: +49 89 62030 148 //// 81539 Munich, Germany \\ ` Fax: +49 89 62030 113 \\