TWiki for Book Authoring

Moved MikeBarton's text from MsOfficeIntegration to here.

-- PeterThoeny - 03 May 2001

Here's and idea we plan to pursue.

We're going to prototype a potential TWiki Plugin that allows editing/reviewing/collaboration of a document in TWiki. The standard TWiki syntax would be used for most of the content. We are considering the use of DocBook to insert tags for headings, etc. We already have the capability to output RTF from DocBook mark-up. Also, we have many of our engineering document templates defined in DocBook markup.

The document will be made up of a Title Page, Table of Contents Page, and a collection of other TWiki pages, one for each section of the document. There will be a feature to "build" this collection of pages into an RTF file for printing/publishing/etc. We don't expect to round-trip from RTF back to TWiki pages.

Rather than being all on one page, a document will be split into its major sections. Each section will be its own wiki page. This is important to us because it allows many users to be adding/editing different parts of the document at the same time.

A TWiki template would be created to provide some navigation through the document's pages. Each page would have hyperlinks such as:

<<-Previous Page

TOC

Next Page->>

Which get the next/previous data from the Table of Contents.

On the Table of Contents page, there will be a control to to generate an RTF file. To generate an RTF file, we'll need to convert from WikiSyntax and DocBook tags to RTF.

-- MikeBarton - 02 May 2001

That is an interesting idea. The TextFormattingRules already offer many commands needed for collaboratively authoring a larger document or a book. The syntax could be extended where needed. What kind of markup needs to be added? Idea:

• FootNotesSyntaxRule
• What else?

The DocBook DTD seems to be a standard for storing books online, the biggest supporter is O'Reiley, all books they offer online are stored in DocBook format.

To me it would make sense to author in the TWiki syntax (because it is easy to type) and to offer conversion tools (TWikiToRtfConverter and TWikiToDocBookConverter plugins) and improved navigation. Comments?

-- PeterThoeny - 03 May 2001

I think it would be good to be able to convert TWiki syntax to DocBook format, however, DocBook is much richer, so you could get a rather awkward mix on a page.

-- JohnTalintyre - 04 May 2001

Let's work on this!

-- PeterThoeny - 30 May 2001

The attached perl module implements a table of contents management system that crosses topic boundaries. It's described in the MotorolaExtensionsToTWiki topic (which uses the tags it supports so forgive the formatting! Can't show you the full beauty of the results without it being installed....). Our tech authoring team have been using it for book authoring (albeit technical manuals). I hope it might be of some assistance.

BTW, we also have Frame->TWiki, TWiki->PDF and TWiki->(distributable HTML) routes working, though these routes use 3rd party S/W so aren't releasable.

-- CrawfordCurrie - 31 May 2001

Crawford, could you post pointers to the Frame->TWiki and other conversion tools that you use? My company uses FrameMaker for documentation and course materials, so it would be particularly useful to convert from Twiki into Frame, since we frequently capture new hints and tips in TWiki and then want to document these more formally in a user guide.

DocBook sounds like a useful format - it's also used by the Linux Documentation Project for new HOWTOs and other docs (see http://www.linuxdoc.org/), and there are open source tools available as a result. Like most XML/SGML DTDs, it focuses on semantic markup, i.e. delineating which parts of the text are headings, sections, notes, warnings, etc, so that it's easy to generate output in various formats. (See http://nis-www.lanl.gov/~rosalia/mydocs/docbook-intro/get-going.html for a useful short example, part of a DocBook tutorial.)

I think TWiki is most useful in this setup as a way of capturing the raw text quickly, ideally from engineers/developers, without the hassle of semantic markup - then the documentation people can convert this, with some automated help, into DocBook format. It could probably be used in similar ways to generate text for Frame (which may support DocBook already - when I last looked at Frame there was a 'with SGML' version that enabled SGML markup within the Frame environment.

-- RichardDonkin - 01 Jun 2001

And, if you would be so kind, I would be interested in how your TWiki2DistributedHTML plugin works.

-- MartinCleaver - 01 Jun 2001

And, if you would be so kind, I would be interested in how your route TWiki->PDF works as compared to my clumsy PrintUsingPDF hack.

For INCLUDEing many sub-topics into a bigger document, we need a simple way to adjust the headerlevels depending on the include level. Say, your simple topic has

h1 foo
   h2 bar
h1 FOO

Once you include it into a bigger doc, you want:

h1 big
   h2 foo
      h3 bar
   h2 FOO

Runoff (yeah, Digital VAXen...) had a SHIFT LEVEL +/- command to do that. Any other ideas how to achieve this? As option to the include statement?

-- PeterKlausner - 01 Jun 2001

• Frame->TWiki uses Quadralay WebWorks Publisher, a subset of which is integrated in Framemaker 6. We had to author a lot of special template files. N.B We have found WebWorks to be very flakey.
• TWiki->HTML uses the TOC order file (part of TOC.pm) and a server-side perl script. All it does is visit the pages in turn and generate the HTML using calls to the TWiki view script. Some minor link filtering is also done in the perl. The TWiki Web templates have been configured with the relevant stylesheets etc. so the HTML comes out of TWiki at near-production quality already.
• PDF uses TWiki->HTML->Acrobat; see http://www.planetpdf.com/mainpage.asp?WebPageID=751 for PDF generation from HTML using Acrobat. If anyone has managed to fully automate this route, I'd be interested.

We recently inherited a tech authoring team in Austin, who have insisted on using Frame (WYSIWYG rules). We have managed to automated the Frame->HTML build process using WebWorks, so the TWiki route is currently only used for release notes. As a result, no more development will be done on it until after our next release in August at the earliest. Oh, for a WYSIWYG TWiki editor!

-- CrawfordCurrie - 24 Jun 2001

A note about DocBook: It now comes in two flavors: SGML and XML. The SGML version is far more widely used then just by O'Reilly. Their connection with it was as a maintainer/development host, but I believe they've now passed the torch.

The XML version (probably more suited to Twikish purposes then the SGML version) is relatively new and has neither the full expressiveness of the SGML version nor the high overhead and complexitiy demanded by SGML. A WYSIWYG editor based on DocBook/XML for TWiki (or darned near anything else) would be way cool imho.

-- DavidLeBlanc - 26 Jun 2001

Rumor has it that StarOffice see http://www.sun.com/staroffice/ 6 will have a gui editor for docbook.

Is there a script for converting HTML to WIKI? Is it necessary? Has anybody tried doing that yet? What were your results?

-- DennisDaniels - 30 Jul 2001

HTML->TWikiDoc should be easy as long as you find the right order of evaluation. I would think that "inside out" would be best: Find all tags that are visual markup like <b> (bold), &lti> (italic) etc. and convert the pairs (<tag>...</tag>) to the TWiki equivilent. The do the container tags like paragraphs and divisions. Tables might be a bit sticky, especially if they're nested (eeeevil imo but useful :-)).

Some tags might not be directly convertable, but would have to have combinations of TWiki markup used... not sure what those might be though.

Of course, there will be some tags that are not only not convertable, but might be also dangerous to keep around, but I can't think of an example offhand aside from <script&gt...

One could use an XML processor like XT or create an XSLT script to do the conversion.

Anyone contemplating doing open source work with RTF should probably check out http://wvware.sourceforge.net/ for GPL software that slices and dices MS Word RTF format in many of it's (version specific) variations. This site also has pointers to other resources that deal with MS RTF.

(N.B. There is another usage of RTF that refers to an IETF RFC for rich text in mail: text/rtf (I think).)

XML"> DocBook XML

I think that moving TWiki markup in the direction of DocBook/XML would be a really good idea. Not so much as allowing docbook markup in TWiki markup (although that's a very good idea too imo), as looking do DocBook/XML for inspiration for new TWiki markup and considering any future intermediate format in terms of a DocBook/XML compatible (sub)set. Probably a subject for TWikiArchitectureNG

-- DavidLeBlanc - 30 Jul 2001

It has been a long time since I started this topic. We did pursue some work here. We never got around to any DocBook work. However, I just pulled together and posted our NavPlugin.

I expect it's pretty similar to the TocPlugin. The NavPlugin is not as capable and is oriented not towards creating a document output, but for on-line documents/presentation.

I believe that these two Plugins were developed at approximately the same time without each of us knowing about the others work. It just took me a while to get around to posting it. Hopefully the Plugins web helps keep the duplicated effort to a minimum.

-- MikeBarton - 30 Jan 2002

Folks, something of possible interest here: LDP WikiText Editing HOWTO

LDP liked the wiki idea of adding text and having the system take care of html. In their case the system translates input to docbook. The app is available at the Linux Documentation Project Downloads page.

Perhaps something like that could be used for TWiki?

-- SamKritikos - 20 May 2002

I like this very much. I have considered it for a while. We currently use DocBook using the XXE Editor which is kindo' nice but... we use Swiki for intranet and customer support. I want to be able to write documentation directly for customers and then at some point decide that "this page" is going into the manual (PDF HTML Help and Java Help)

-- RobinRosenberg - 02 Jun 2002

(Just fixed link to XXE Editor above) -- PeterMasiar - 03 Jun 2002

What about good old TeX/LaTeX?

-- JanRuzicka - 11 Jul 2002

I´m searching for an tool for online collaboration authoring. twiki seems to be a good solution for authoring, but I´m not sure how to make the document at the end. Is there any other solution as PrintUsingPDF ? I´m not a programmer, but I know people who are familar with perl, XML and PDF. Some more links to perl-Scripts for the Way twiki->html would be helpful. Better would be some kind of twiki->XML which could be imported in QuarkXpress, InDesign or Framemaker.

-- JanPeterHomann - 29 October 2002

This is EXACTLY what I am looking for. A way to easily get a close-to-production printout for customers and users. The perfect solution would be a button on the Twiki page that said "Export to RTF" or "Export to PDF", and the current twiki version would be generated for download. Does anyone have a recommendation on the best way to do this? I am a newbie coder with minor PHP skills.

Maybe a good summary and step-by-step of the best current solution to do what I described above would be very helpful to us realtive newbiew who would love to implement this. Thanks a bunch.

- RyanKnoll - 21 December 2002

I have found TWiki only three days ago, after some internal discussion of how we can create and maintain better internal (and external) documentation. I like much of what I see here, but we have an absolute requirement for reasonably-good printed output.

I found a TWiki document in PDF form, which would be good enough for our needs. Is there a tutorial I can follow to produce printed docs of similar quality?

- CarlMikkelsen - 02 January 2003

The PDF doc that you saw was probably the one attached to PrintUsingPDF, which should give you enough information though it's not a tutorial.

-- RichardDonkin - 02 Jan 2003

We use Twiki for our R&D task card process. We write out test plans for each task on the wiki. I would like to get everyone away from using MS Word for writing test plans and instead do a Twiki -> DocBook conversion using XSL. Is anyone already doing this? I dont think a twiki syntax to DocBook would be that hard?

-- NateGelbard - 27 May 2003

In a way I'm surprised an easy option hasn't been mentioned - htmldoc - it runs either using a GUI (FLTK based), or as a command line program. Page numbering, table of contents, front pages, etc. At our workplace I use it to auto-pdf all the pages in our twikis (where changed) every 15 minutes (simply takes a skin which consists of just a %TEXTAREA%. Likewise, we use CrawfordCurrie's genhtml plugin to generate plain text and downloadable (tar.gz) version.

If there's interest I can put the code up - it's pretty trivial though. (And relies on the DataAndCodeSeparation the TWikiUnixInstaller implements)

-- MichaelSparks - 27 May 2003

I think for books (or perhaps for existing books to be imported into Wiki) it's essential to have dynamic footnotes and bibliography (bibtex) support. (dynamic footnotes means latex footnoes that do the numbering seperately --- i.e. i have six footnotes but I dont have the worry about which numbers they are). So much of academic work is done in LaTeX these days... MIT Press's books and many many academic journals.. And the need for footnotes -- one need not look further than the nearest ORA book.

-- AllenBeyeRiddell - 03 Jun 2003

blueDonkey.org Collaborative Books at http://www.bluedonkey.org/cgi-bin/twiki/bin/view/Books/WebHome is an interesting site using TWiki: "Here you will find all the books that are currently being written at this site. The intention of this site is to create books that will be suitable for publication in hard copy format as well as online."

-- PeterThoeny - 04 Jun 2003

Some time in February, we went live with document production using TWiki. Starting with the TocPlugin, we extended the notion of "WebOrder" to be anything that matches the "WebOrder[a-zA-Z0-9]*" pattern. When a corresponding "WebPrint..." topic is viewed, it generates a combination HTML page for that particular WebOrder. There are Perl scripts which get the topic using wget, collect the optional title page ("TitlePage..."), extract watermark information from the title page, and generate a PDF using htmldoc. The whole thing is driven from a single Perl script which searches through all webs for "WebOrder..." topics and performs the conversion.

This has become our main method of producing internal and customer documents.

Some of the patches to TocPlugin are in codev, but they predate the title page additions. Also, I didn't know what to do with the scripts. I'd be happy to place them anywhere it would be useful.

-- CarlMikkelsen - 04 Jun 2003

What I don't understand: how does that Bluedonkey site create the books? Probably should publish my PdfPlugin really soon now(tm) to gather feedback... Like the hack from PrintUsingPDF, it is based on HTMLDOC.

But anyway, the conversion to PDF is the smaller problem. The bigger part is, how to have fairly normal online wiki pages plus get them into a reasonable book structure. Different from the described TocPlugin and the SectionalEditPlugin, which add extra syntax to be used within content pages, I tried to address this with standard TWiki means plus the new IncludeIndexPlugin:

1. have regular wiki pages; Small pages might solve the issue of FineGrainedAddressing and make NamedIncludeSections superfluos
2. have one master %INCLUDing page, like e.g. TWikiDocumentation
3. have an extra online index built from this master page

Thought: is it worth the effort to try to turn any wiki page into PDF, or is it easier to have a specific control page for a "real" book?

Still open: header levels used within one page are set fix for all including master documents. Should adapt dynamically. Somehow. Someday.

-- PeterKlausner - 11 Jun 2003

http://twikiplugins.sourceforge.net/cgi-bin/twiki/view/HowToThinkLikeAComputerScientistUsingPython/ has been setup as a place to experiment on this topic.

-- WillNorris - 13 Oct 2004

Any updates on this topic? I will be looking to manage a process of collaborative book authoring via a set of content creators/developers/organizers, I've been interested in DocBook for a while, and I'd like to integrate with a Wiki if possible. This is the most interesting thing I've seen yet.

Does http://twikiplugins.sourceforge.net/cgi-bin/twiki/view/HowToThinkLikeAComputerScientistUsingPython/ represent a working DocBook-TWiki integration? Any other representative examples?

Can any TWiki topic in Cairo be converted to DocBook? What "glueware"/plugins is/are required to do this?

What's the general state of DocBook-TWiki in general?

-- MattEngland - 11 Apr 2005

I fear there hasn't been much (any) progress on this (hope I'm wrong) matter. The best integration between a Wiki-engine and DocBook I have seen with MoinMoin (it hurts to have to admit this). Well, maybe this can be done with one of the sophisticated XmlXslPlugins.

-- FranzJosefSilli - 11 Apr 2005