Compiling Large Book Files Without Running out of Memory

[Dan Emory <danemory@xxxxxxxxxxxx>]

Using FrameMaker, I successfully finished today a database publishing job
for a 920-page directory of 28,000 specialist physicians who are members of
an association (this is the second year in a row I've produced it). It
consists of:

1.  An auto-generated Table of Contents (2-column layout)

2. 750+ pages of database-derived member listings (2-column layout) sorted
by geographical location (Country/State/City), with a 3-15 line
who's-who-type biography of each physician.

3. A 136-page autogenerated alphabetical index (4-column layout), in which
each physician's name (index level 1) includes up to two (index level 2)
locations (City/State or City/Country) The page number(s) where the
physician's biographical information is located is indicated for each listed
location.

4. A 25-page database-derived Resource Guide, interspersed with display ads,
listing commercial companies who provide resources for the association
members. The Guide consists of an alphabetical listing of the companies
(name, address, fax, phone, URL, email) followed by a listing of the same
companies by product/service category.

The book file for all of this information contained 14 FrameMaker files
(26MB total, including the 3.5MB index).

All of the work to produce this heavy tome was done on a 32MB Windows
platform, using UniMerge and FM+SGML 5.1.1 (FM+SGML was used not because the
book is structured, but because it is much more stable than FrameMaker, and
releases memory better).

The challenge was to compile the alphabetical index from the book file
without running out of memory. Obviously, a huge chunk of memory is required
for compiling and sorting a 2-level index of 28,000 names. During this
process, FrameMaker must open, scan, paginate, and save each file in the
book several  times, storing each file in memory while it is being scanned
and paginated. If the combination of the individual file size being scanned
and the growing chunk of memory reserved for compiling the index exceeds the
available memory, FrameMaker puts up the dreaded "Can't open file xxx"
message, and aborts the Generate/Update execution.

The secret, obviously, is conservation of  memory.  There are three
techniques for doing this, and I use all of them:
	a. If the book contains two or more generated lists/indexes, update 
           only one at a time. For example, if the book file contains both 
           a generated TOC and a generated index, execute  one Generate/Update
           pass to compile the index only, and then immediately follow with
           another Generate/Update pass to compile only the TOC.
	b. Be sure that none of the individual document files in the book 
           exceed about3MB (a 2.5MB limit is safer). This is the most
           vital of the three techniques.
	c. Although it is probably too obvious to mention, shut down all 
           other applications that use up memory.

So far, I haven't built a large book I couldn't compile successfully on my
32MB platform (but I am getting more memory just in case). Recently, I did
(for the third year in a row) a database-derived 1216-page World Wide Web
Yellow Pages book containing complete descriptions and ratings, interspersed
with home-page screen shots, of 10,000 web sites sorted alphabetically under
each of 187 categories. That one had three different generated indexes (250
pages total), which all compiled successfully using the techniques described
above.

Dan Emory
Dan Emory & Associates
FrameMaker/FrameMaker+SGML Document Design
and Database Publishing Specialists

Voice/Fax: 949-722-8971
E-Mail: danemory@primenet.com
10044 Adams Ave. #208
Huntington Beach, CA 92646


** To unsubscribe, send a message to majordomo@omsys.com **
** with "unsubscribe framers" (no quotes) in the body.   **