Re: Database Publishing Suggestions

[Dan Emory <danemory@xxxxxxxxxxxx>]

The original of this message sent to David Evans of Finite Matters had a
MIME-encoded Acrobat 3.0 file that contained an example of a high-end
database application done with UniMerge. It includes a sample of the
ready-to-print output, and the guts of the FrameMaker report template used
by UniMerge to produce that output.

Anyone who's interested in inspecting that example can contact me by email,
and I'll send it to you.

This example and its implications is further discussed below.

At 08:45 PM 2/3/99 -0500, David Evans wrote:
>------------------Snip
>
>########################################################################
>And don't you just import
>ASCII (meta-tagged) information -- remember that is what you do, NOT
>database publishing. Please don't portray tagged ASCII import as database
>publishing. Based on that definition, any information coming out of a
>database can be construed as database publishing. The definition really is
>"tagged publishing", similar to that of HTML.
>Now to be fair, please tell us the whole story.  Are you not required to
>run a database process to get an ASCII data dump? And aren't you required
>to paint those pretty little meta-tags around that data. And finally,
>import that data into Frame (Oh, by the way, many use the same thing for
>Quark, PageMaker, and Ventura. And how about RTF, it to is a meta language)
>Isn't that the process?
>########################################################################
Wow. You obviously know nothing about UniMerge. There are no "meta-tags
placed around the data" in the database extract. UniMerge uses the straight
data extract produced by an ordinary database query, sorted in the order it
will be presented in the output. The data records can be in the form of:

+ ASCII data with character-delimited fields (one record per line)
OR
+ ASCII data with fixed-column records (one record per line)
OR
+ ASCII data with one field on each line (e.g., fieldName:fieldValue) In
this case, multiple nevels of nested repeating groups can be handled by UniMerge
OR
+ Data received via a live connection to the database

The FrameMaker report template used by UniMerge is a WYSIWYG representation
of the data fields, arranged and formatted in any manner desired, and
interspersed with UniMerge commands (primarily conditional statements that
test particular record fields for various purposes, so as to decide how the
data is arranged and formatted). There is no "meta-tagging" in the report
template.

So, to summarize, "meta-tagging" is not used in the data extract, the report
template, or anywhere else for that matter.
>#########################################################################
>When talking of publishing speeds, we prefer to talk about pages per
>minute.  An extremely large Telecom (the largest commercial Frame user) has
>run as many as 72 pages per minute.
>it can slow down...to 5-6 pages per minute.  But we are talking about
>complex products -- not a "mailing list" -- real stuff.  
>#########################################################################
Well Gee Willikers, a top speed of 72 pages per minute! How about that. I'm
attaching as a PDF file a filled-out 2-page questionnaire form (the mailing
address appears in the envelope window) for a project I recently finished.
Following that is the 3-page guts of the FrameMaker report template used by
UniMerge to produce it. The database information about a particular
individual is overlayed on the form background. UniMerge produces a
1000-page MIF file (about 20 megabytes) for 500 records in less than 1
minute. It takes another 2 minutes to open the MIF file in FrameMaker and
import the form background (on left and right master pages) into all 1000
pages. So, with UniMerge, I can produce 1000 pages ready to print in 3
minutes. Not bad.

That's about 5 times fasterr than PatternStream's top performance of 72
pages/minute, and 60 times faster than PatternStream's worst performance of
5 pages per minute.

Note that the ASCII data extract is chopped into files of 500 records each.
That's so the sizes of the FrameMaker and postscript files don't become
unmanageably large (each 1000-page FrameMaker file produces a 60 MB
postscript file, which is distilled and optimized to produce a 1.28 MB PDF
file, which is further reduced to a size of .75 MB by zipping before
transmittal via FTP to the printer).

I challenge you to describe in some detail how the application shown in the
attached PDF file could be successfully implemented with PatternStream, and
how much time you estimate it would take to develop that application.

Here's an implementation clue: The whole report template shown in the PDF
file is a complex multi-column table, using many straddles and other tricks
to assure that the data always properly overlays the background form, no
matter how variable the length of the individual data fields. The background
forms on the master pages are contained in even more complex tables.

When considering the development time, you should know that the customer
required extensive checking for errors and other record anomalies, rejection
of records that had incomplete or anomalous mailing addresses, and the
logging of those errors and anomalies in a log file produced by UniMerge. I
did not include in the PDF file the parts of the report template that
perform these activities.
>########################################################################
>I'm surprised that you hold a mailing list out as your comparison -- have
>you seen MS Word?  They use something called mail merge.
>Fortunately, mailing lists are not our market. However, 30 minutes seems fair
>for PatternStream also. The only difference is that we are publishing --
>not testing.
>########################################################################
David, David, David
Now, you're deliberately twisting my words. The mailing list example I used
was specifically described by me in the text below as being trivial, and the
time required (30 minutes) to develop for UniMerge was described by me as
meaningless, and atypical of real-world database publishing projects.

Here's what I actually said:
>>But such simple applications (mailing lists) provides no meaningful
??information about capability. In a
>>real-world application of some complexity, the development time is consumed
>>primarily by an analysis of the customer's requirements, the content of the
>>database, all the possible variations among records, and exception handling.
>>Test cases must be developed for all of these issues. Then, there are all
>>the issues involved in FrameMaker template design, which typically requires
>>the development of more test cases. Then, a skeleton application of some
>>sort must be developed to run against those test cases and analyze the
>>results. After that, there is usually a back-and-forth process that goes on
>>with the customer (sending samples and getting comments back) to perfect
>>everything and get all the formatting and page layout issues resolved. As
>>any programmer will tell you, no software product can automate this process.
>###########################################################################
>WHOOPS -- should of used PatternStream.   I do agree that requirements and
>poor
>data structure can slow one down, but test cases and a skeleton applications?
>Those sound as if the belong in a laboratory.  With PatternStream, you 
>publish pages -- don't like it, fix it (again, under a minute), re-publish!
>########################################################################
The kind of glib puffery above is all I hear from your company about
PatternStream. That sales crap is a real turn-off, which is one of the main
reasons why I never sought more information. By the way, I've been to your
web site. It's like eating chinese. I couldn't even find out how much
PatternStream costs.
>###########################################################################
>>Can PatternStream handle the more common cases where a live connection is
>>not only not needed, it is absolutely out of the question? UniMerge can.
>Yes, it can, but I can't think of a case that it is "absolutely out of the
>question",
>###########################################################################
Well, let me tell you, if you believe that, you're living in a dream world.

1. The typical group or department responsible for database publishing
doesn't have much database expertise, and doesn't own or manage the
databases containing the data to be published. Moreover, most databases
weren't designed for database publishing, and typically contain quirks and
artifacts that only the database owners know about and understand. The
responsible group or department must usually rely on MIS to design the
queries that produce the sorted record output they need for database publishing.

2. MIS has its own toolset for designing complex queries, specially adapted
to the peculiaries of each database, and they're not about to replace those
tools with PatternSteam. In fact, most MIS departments would probably regard
PatternStream as a tinkertoy compared to their standard toolset. They would
argue vehemently against the use of something like PatternStream by a
non-MIS group, and some of their reasons against such use would be
legitimate (see 1 above).

3. MIS tends to resist doing anything more than the minimum. They want to do
an ASCII dump (character-delimited or fixed-column) and be done with it.
They don't understand, or even care, what the end user is trying to do with
the data.

4. The responsible group or department that does the database publishing
doesn't know or understand anything about the anomalies and artifacts in the
database, or whether those oddities are in the ASCII dump they receive. All
they can do is merge the data, and examine the output for anomalies. If such
anomalies are found, the responsible group must go back to MIS and ask them
to fix the problems in the next dump. To simplify this process, it is often
necessary for the database publishing application to contain checks for
anomalous records. When an anomaly is found, it is logged as an exception. I
have included such checks and exception logging in almost every application
I've developed. UniMerge has the capability to log those exceptions in a
separate log file that uniquely identifies each offending record. The log
can then be sent to MIS for analysis and correction.

5. The responsible group or department that does the database publishing may
be remotely located, and cannot establish a live connection to the database.
The ASCII dump gets sent to them via ftp, or is put on a disk and mailed to
them.

6. In many cases, the responsible group wants to farm the actual database
publishing activity out to a subcontractor. It would be extremely rare that
a subcontractor would have a live connection to the database. So an ASCII
dump file is ftp'ed to the subcontractor, who designs/tests the application
and produces print-ready output.
>########################################################################

Best Regards
     ____________________
     | Nullius in Verba |
     ********************
Dan Emory, Dan Emory & Associates
FrameMaker/FrameMaker+SGML Document Design & Database Publishing
Voice/Fax: 949-722-8971 E-Mail: danemory@primenet.com
10044 Adams Ave. #208, Huntington Beach, CA 92646


** To unsubscribe, send a message to majordomo@omsys.com **
** with "unsubscribe framers" (no quotes) in the body.   **