cifgen.mi

\input texinfo
@setfilename thisfilename
@afourpaper
@settitle Generating ciforth's
@setchapternewpage odd

@titlepage
@title ciforth Manual
A system to generate
a ciforth together with
its documentation.


@author Albert van der Horst
Dutch Forth Workshop

@page
@c @vskip Opt plus 1fill
Copyright @copyright{{}} 2000 Dutch Forth Workshop

Permission is granted to copy with attribution.
Program is protected by the GNU Public License.
@end titlepage

@chapter Overview
forthvar({ci86.gnr}) is a system to generate ciforth in all its
configuration's.
This is a configurators manual.
For each ciforth there is a corresponding documentation;
there is however just this one documentation for the generic
system.
It is assumed that you are familiar with Forth and with ciforth
in particular.
Linux is used for a development system,
and the main tool is forthprog({m4}) , the macro preprocessor.
@pindex m4
This generates an assembler source
file and a raw documentation file
out of the single generic source, controlled by a configuration
file.
In addition there is a file with blocks, that is common to all Forth's.

For further processing you need an assembler, such as forthprog({nasm})
and one or more documentation tools, such as forthprog({info}). The raw
documentation file can be ordered only by a more sophisticated tool
than the usual forthprog({sort}), called forthprog({ssort}). The
@pindex ssort
particulars of all this depend on the actual configuration chosen. It
contains in itself some provision such that it can be loaded on
any of the ciforth systems, independantly of whether they are 16 or 32
bits or whether they can or cannot handle direct access of the video memory.

@chapter Non-technical background.
@section                        Legalese
The Forth's called ciforth are made available through the
DFW .
All publications of the DFW are available under GPL, the GNU public license.
The file COPYING containing the legal expression of these lines must accompagny it.

This forthsamp({ci86.gnr}) system is protected by GPL.
This applies to the generic source, the macro files and the Forth
source in the block file.

@subsection Copyright of the ciforth's generated by this tool.
A ciforth generated by ci86.gnr is probably not a derived work
(a thesis written in TeX is not a derived work from TeX).
So DFW separately claims copyright for the different versions of
ciforth generated by her using this tool.

The following is present in all documentation of ciforth's:
forthquotation
Because Forth is ``programming by extending the language'' the GPL
could be construed to mean that systems based on ciforth
always are legally obliged to make the source available.
But we consider this ``fair use in the Forth sense''.
forthendquotation

In addition to the GPL the DFW states the following:

forthquotation
The GPL is interpreted in the sense that a system based on ciforth
and intended to serve a particular purpose, that purpose not being a
``general purpose Forth system'', is fair use of the system, even if it
could accomplish everything ciforth could, under the condition that the
ciforth it is based on is available in accordance to the GPL rules,
and this is made known to the user of the derived system.
Consequently, for these systems the obligation to make the source available
does not apply.
forthendquotation

@section Legal matters
My extensions are GPL-ed or libary GPL-ed.
I transferred the copyright to the Dutch Forth Workshop, a foundation that
supports Forth and defends the GPL.
The original figforth is public domain and is still available.
@section                         Rationale

What you find here is a Forth for the Intel 86. It is an ISO
as of old. Complying in detail with ISO for the CORE wordset at least.

This has bee split off of a similar generic Forth, that is intended
to be the last of the fig-Forth's. This generic system is mp longer
maintained, but the last fig-Forth is, in the sense that if there
ever should be found a bug, it can be fixed. It also features
the Fig glossary, which is available
in electronic from for the first time in history. I shamelessly
copied from it.
Apart from being ISO, the Forth you have here is similar in many
respects to fig-Forth.
The motivation for having this type of Forth available follows from its
characteristics. It is available as an assembler source, and it is an
indirect threaded Forth.
An assembler source has distinct advantages for getting started from
nothing. An engineer might balk at the description of how to use a meta
compiler, but feels at ease with a (much larger) assembler manual.
(The popularity of eForth proves how popular assembler source is.
I do not think that system has much merit, besides that.)

Although speed is currently in fashion, using subroutine threaded Forth's
with optimisers, indirect threading is the preferred choice for some
applications. I did this work, because I needed it.
I have also the firm belief that an optimiser on an indirect threaded system
has more information to work with and can ultimately outperform any
other system in speed.

@section History
From the introduction to the figforth installation manual:
forthquotation
The figforth implementation project occurred because a key group of Forth
fanciers wished to make this valuable tool available on a personal computing
level. In June of 1978, we gathered a team of nine systems level
programmers, each with a particular target computer. The charter of the
group was to translate a common model of Forth into assembly language
listings for each computer. It was agreed that the group's work would be
distributed in the public domain by FIG.

We intend that our primary recipients of the Implementation Project be
computer users groups, libraries, and commercial vendors.
We expect that each will further customize for particular computers and
redistribute. No restrictions are placed on cost, but we expect faithfulness
to the model. FIG does not intend to distribute machine readable versions,
as that entails customization, revision, and customer support better
reserved for commercial vendors.

Of course, another broad group of recipients of the work is the community of
personal computer users. We hope that our publications will aid in the use
of Forth and increase the user expectation of the performance of high level
computer languages.
forthendquotation

@subsection                 Deviations of the FIG model

The first version of ciforth complied faithfully to the fig model,
at least as faithfully as is customary. Now it is ISO, which
means a lot of details are changed about how words work.
In the following we will discuss not those details, but the changes
made to the general build up of the Forth.
(In fact some details are discussed. They are of little interest and
they will move to an appendix in the next version.)

The rigid subdivision in 7 area's was never adhered to.
In particular the bootup parameters
are not up front as CP/M and MSDOS require a 100H byte reserved
area there.
There is mention of forthvar({(KEY)}) being ``implementation dependant code''
but these were not often present in
fig implementations.
This was based on the idea that there was some EPROM with console commands.
This has been replaces by calls to an operating system, that do not
comply with a simple function that could be called.
Here
the code definitions for forthcode({KEY}) itself
become implementation dependant code, but often it can written in
high level.

All documentation is now accurate but only claims to describe
ciforth.
The forthcode({RUBOUT}) key is a bonafide forthcode({USER}) variable
and now has a name.

DR0 and DR1 are removed. There is only one consecutive mass storage area, be
it a disk or a file.
The assumption in using forthcodeni({OFFSET}) was that you have two identical floppy drives
and no hard disk. That is nowadays extremely unlikely. Instead I put forthcodeni({OFFSET})
to good use to screen off a part of the floppy that must not be used (such
as an MSDOS directory or the hard disk part that contains the forth system.)

forthvar({MOVE MON BLOCK-READ BLOCK-WRITE DLIST}) are not present.
Altering OUT to influence formatting doesn't work here, nor on
any figforth I know off.

forthcode({+ORIGIN}) now points to the first user
variable instead to some not well defined start of the boot image.
The layout has not changed, so the negative offsets can be used for
the other data traditionally there.
 Their indices have not changed, but there is now a bootup
parameter for each and every user variable. A system with other boot
parameters can now be generated in an even more portable fashion by using
phrase like forthsamp({7 CELLS +ORIGIN}).

Where possible installation dependant code is using a generic call to the
operating system in particular MSDOS BIOS or LINOS. This mostly results in
the installation dependant code to be high level.

The false urban legend that one could forthcodeni({FORGET}) forthcodeni({TASK})
has been replaced by an accurate description of forthcodeni({TASK}).
The following words have been documented for the first time:
forthcodeni({FLUSH}) forthcodeni({CURRENT}) forthcodeni({2DUP}) forthcodeni({RP_comat_}) forthcodeni({U.}).

Some non-substantial deviation of the original FIG source have been made
for good reasons.

The FIG filosofy is that sectors, blocks and screens must be compatible, but
may be all different. The original 8086 FIG had one sector for a block. I
changed that in having one block for a screen. This is a boon for those
wanting to ISO-fy the sources.

The way I coded the character I/O points ahead to vectoring
forthcodeni({TYPE}) and forthcodeni({EXPECT}) rather than
forthcodeni({EMIT}) and forthcodeni({KEY}) . This way I can have the
host system handle the rub out key.

I added generic words for accessing system resources
forthcodeni({BIOS}) , forthcodeni({BDOS}) and forthcodeni({LINOS}) .
(See subsection The joy of genericity.)

Some real errors were fixed:
forthenumerate
forthitem
The redefine forthsample({NULL}) bug is fixed.
It is no longer possible to redefine this word,
that handles the refill of the forthcode({TIB}),
by typing a <ret> immediately after a defining word.
forthitem
Forgetting part of a vocabulary, other than the forthcode({FORTH}) vocabulary
no longer crashes.
forthitem
Loading a screen with characters having an 8th bit set,
no longer crashes.
forthendenumerate

@section Evolution of ciforth
The first version of
ciforth  was in fact the
figforth for the 8086
that was put in the framework of this manual.
By adding a 32 bits macro file, programming
I/O for Linux, programming I/O  with non-obsolete MSDOS calls
and a way to switch to protected mode,
this figforth came available in all ciforth configurations.
The rcs version numbers of the generic file fig86.gnr
are in the 2-branch and the latest version is available still.
(The 1-branch was experimental).
This version has however an manual not split between a generic and
a user part. But the user part of the manual forthemph({is})
generated from the generic source.

The second and third
versions of ciforth (rcs branch 3 and 4)
are generated  according to this manual.
As you see there is little pertinent information about these forth's
in this manual.
All the information you need to use it is in the user manual,
generated with that version.
The second version is a ISO compatible system.
The third version is how I think a Forth should be,
based on technical criteria and putting all compatibility issues aside.

@section                         Source
In practice the GPL
 means (note: this is an explanation and has no legal value!)

They may be
further reproduced and distributed subject to the following conditions:

The three file comprising it must be kept together and in particular
the reference section with the World Wide Web sites.

This Forth builts on figforth, for its source see the next section.
The maintainer can be reached at forthmail({ciforth@@spenarnc.xs4all.nl})

@section Acknowledgment
ciforth is based on the figforth
of Charlie Krajewski and Thomas Newman, Hayward, Ca.
This figforth (as are all figforth's) is public domain.
It is still available via taygeta. And of course kudo's to FIG.

forthurl({ftp://ftp.forth.org/pub/Forth/compilers/native/dos})

This original version is public domain according to the
following statement:

forthquotation
All publications of the Forth Interest Group are public domain. They may be
further reproduced and distributed by inclusion of this credit notice:

This publication has been made available by the Forth Interest Group,

              P. O. Box 1105,  San Carlos, Ca 94070
forthendquotation

I also want to thank  J. E. Smith, Philadelphia for another fig Intel86
implementation also still obtainable from
forthurel({http://www.simtel.net/pub/simtelnet/msdos/forth/fig86.zip})

This is a fairly good documented FIG Forth for IBM PC, but its
"Seattle Computer 8086 assembler" format makes it less practical.

@chapter Background.
If you are a Unix and a Forth guru, you can skip this chapter.
_VERBOSE_({ If you think you are,
you can read this chapter and discover you are not.})
This chapter is about pervading concepts and
how tools are used, conceptually.

@section Orthogonality
The concept of orthogonality is central to this effort.
Orthogonality means that different aspects of configuration
(int this case)
are made independant of each other.
For example, ciforth can be bootable or started by MSDOS,
it can be assembled by forthprog({nasm}) or by forthprog({MASM.EXE}) .
@pindex nasm
@pindex MASM.EXE
These two choices can be made indepedantly from each other,
and every combination ought to work.
Each choice is associated with file with macros for forthprog({m4}) ,
so ideally if you need to modify how forthprog({nasm})
assembly source is generated to you only need to change the file
forthfile({nasm.m4}).

This is, of course, as far as it goes.
Try as you may to separate all information about header layout
in the forthfile({header.m4}) configuration file,
a change to the order of the fields in a header will certainly have
it impact at certain places in the source.

@section Metacompilation is outdated
Meta compilation, the generation of a new version of a Forth system
by ``simular tools as compilation'', was invented for the cassette based
computer system of the late seventies.
There may be a motivation for using metacompilation to generate
a similar forth for a forthemph({different}) processor or system.
This would properly be called cross-compilation, by the way.
In a half-decent (or better) disk operating system like MSDOS the use
of meta-compilation is a mistake at the management level.

We want our Forth to be able to generate standalone programs anyway.
(a forthdefi({turnkey}) facility.)
So what do we need
forthenumerate
forthitem
A facility to save a running system with all what is loaded on it,
in the configuration it currently has.
forthitem
A facility to remove parts of a running system, that are not needed
for an application after it has been build. (E.g. the assembler.)
forthitem
A facility to optimise some parts of a system. (Then remove the,
possibly large, optimiser.).
forthendenumerate

If you have the first facility,
you can build a powerful Forth from a small kernel and regular source code.
If you have all of them, you can build a truely,
optimal Forth from a small kernel.
You need not ``a similar tool as compilation'',
you just need ``compilation'' .

The forthcodeni({SAVE-SYSTEM}) facility of course requires
in depth knowledge of the operating system.
This doesn't mean it is cumbersome or difficult.
Under linux we need
forthexample(
{HEX
S" newforth" BLOCK-FILE $!  BLOCK-INIT   ( Abuse block file)
HERE FENCE !   LATEST 0 +ORIGIN !  ( Extend the system )
8049000 CONSTANT SM     ( This is in depth knowledge)
U0 @   0 +ORIGIN   100    CMOVE  ( Save user variables)
HERE SM - CONSTANT FSIZE   ( Length of file, incl.header)
HERE BM - SM 44 + !    FSIZE SM 20 + ! ( Patch header)
BLOCK-HANDLE @ 8049000   FSIZE  5 LINOS ?LINUX-ERROR ( Write)
BLOCK-EXIT
})

@section How m4 is used.
The unix macroprocessor forthprog({m4}) is very powerful indeed.
@pindex m4
Testimony is that the description of its usage in here
is longer that its man-pages.
You know
forthprog({m4}) is a text substitution tool.
A macro is like a function. In the macro call the text is replaced by
the text present in the function.
Within the text the placeholders for the parameters are replaced
by the actual parameters.
In forthprog({m4}) the placeholders are forthsamp({$}{1}) ... forthsamp({$}{9}).
Parameters can be passed, and any (even multiline)
text can be given as a parameter, provided it is quoted.
We will use forthsamp({_lbracket_}) and forthsamp({_rbracket_}) (braces) througout.
This is convenient, because they are
not used in a Basic Forth system
and  they are
special anyway (e.g. for TeX).
The use of quotation is very critical at times,
and the find points are not covered in the following.
@subsection Customisation
Simple customisation can be done by forthprog({m4}) as follows:

 forthsamp({define(_lbracket_version_rbracket_,2.149)})

Within the text treated the version number is substituted.
@subsection Selection
Selection, often one of alternatives, is in general done as follows

forthsamp({_BITS16_(32)_BITS32_(64)_BITS64_(128)}) ,

which gives, of course, the size of a double number.

This is accomplished by

forthsamp({define(_lbracket__BITSxx_  _rbracket_,_lbracket_$1  _rbracket_)})

for the actual bitsize and

forthsamp({define(_lbracket__BITSxx_  _rbracket_,)})

for others.

Selections can be nested within other forthprog({m4}) macro construct.
As in
forthexample(
{{_VERBOSE_}_lbracket_({_BITS64_}(_lbracket_The possibility to cycle through all (64-bit)
numbers by {forthsamp}(_lbracket_0 0 DO ... LOOP_rbracket_) is very useful indeed._rbracket_)_rbracket_)})
Here you see at work, apart from forthmacro({_BITS64_}) , the macro forthmacro({_VERBOSE_})
that allows (if turned on)
verbosity that can help understanding but is not always appreciated.
You alse see forthmacro({forthsamp}) that is in fact
a markup to indicate we have a piece of Forth code there.

Selections can be used to throw out a block of
word definitions and their documentation as a whole.
For example words accessing I/O ports are not available in a Linux Forth,
as they would only lead to privilege violations.

The braces are essential here.
Without it the introduction of a comma somewhere in the text
results in forthprog({m4}) interpreting the remainder as a second parameter,
which it wil ignore.

@subsection A postponed markup language.
Just say forthsamp({forthcode(_lbracket_+LOOP _rbracket_)}) to indicate that you want
formatting as for ``code'' words.
Later you can decide to use
forthbreak
 forthsamp({{define}(_lbracket_forthcode _rbracket_,_lbracket__comat_code_lbracket_$}{1_rbracket__rbracket_)})
forthbreak
for forthsamp({texinfo}) or
forthbreak
forthsamp({{define}(_lbracket_forthcode _rbracket_,_lbracket_<B>$1</B> _rbracket_)})
forthbreak
for forthsamp({html})
.
@subsection Defining structures
Some macro calls must be considered to define a structure, in particular
 forthsamp({worddoc}) .
Suppose we have a list of structures, meaning that the first person is
a child of the second and third person:

parents({Alice},{Mary},{John})

parents({Fred},{Mary},{Henry})

parents({Aayilah},{Sjantil},{Bodaji})

...

With
 forthsamp({{define}({parents},{$}{2})}) we get a list of (you guessed) the mothers.

The usage of forthmacro({divert()}) can best be explained with an example in this context.

forthexample(
{{define}(_lbracket_parents_rbracket_,
_lbracket_{divert(3)dnl}
$}{2
{divert(6)dnl}
$}{3
_rbracket_)})

will give out the mothers on channel 3 and fathers on channel 6.
The output will be concatenated,
but all mothers and all fathers stay together.
For forthsamp({dnl}) see the forthprog({m4}) man-page.

@subsection Defining lists
By using an extra pair of braces you can have a list in forthprog({m4}) .
So forthsamp({_lbracket__lbracket_A_rbracket_,_lbracket_B_rbracket_,_lbracket_C_rbracket_,_lbracket_D_rbracket__rbracket_}) is
a single parameter to a macro and can
be passed to other macro's as a whole.
The outer braces are removed and
without special measures (reinstalling extra braces again)
the macro called forthemph({sees})
the comma's and concludes there are four parameters.
This is put to good use in the ``See also'' and ``Test''
fields of the forthsamp({worddoc}) structure.
These fields may have zero or more parts.

The ``Test'' field contain the tests in the odd fields, and the
expected outcome in the following even fields.

@subsection Impress the crowd
By using macro's to define other macro's, then pass the result through
forthprog({m4}) another time, severe stress can be laid upon the intelligence
of the everyday person.
The very inconvenient way nodes must be linked in texinfo even forced
me to define part of the macro in one macro and the remainder in
another.

@section How forthprog({ssort}) is used
The sorting tool forthprog({ssort}) can order multiple field records, with
different sorting criteria for each field.
The fields can be defined by regular expressions, such that
the forthsamp({worddoc}) structures can be sorted by name, or by wordset
then by name, or in about any way you want.
Because such a tool didn't exist, I had to write it.

@subsection Analysing forthsamp({worddoc})
 forthprog({ssort}) captures the structure of a forthsamp({worddoc}) as follows:

 forthsamp({^worddoc(_lbracket__comat__rbracket_,_lbracket__comat__rbracket_.*\n$worddoc})

The part between forthsamp({^}) and forthsamp({$}) matches the record.
The part after the last forthsamp({$})
is for synchronisation, to make sure the record doesn't end early.
This would result in an error ``not according to structure'': the next
line doesn't start with ``worddoc'' and so it just doesn't match the record
description.
The forthsamp({$}) is merely a separation, (newlines are indicated by forthsamp({\n}) ).
The forthsamp({.*}) matches anything, including new lines.
But it isn't greedy as in ordinary regular expressions,
because not being stopped by forthsamp({\n}) ,
it would match the whole file.
Here it tries to match as little as possible.
 forthsamp(_comat__rbracket_) is shorthand for forthsamp([^_rbracket_]*_rbracket_$)
so a ``sequence of anything except
right braces followed by a right brace''.
It also contains the forthsamp({$}) to
mark the end of a field.
@subsection Sorting fields
Once we know what the fields are,
 forthsamp({-M 1S2S }) sorts on the first field
and within that field on the second. We just use the ordinary ASCII collating
sort, indicated by forthsamp({S}) .

@chapter Structures and processes
@section The generic source file
The generic source file forthfile({ci86.gnr}) mostly
consists of Intel assembly code, with which, I assume,
you are familiar. All macro's in the following are forthfile({m4}) macro's.
Words are divided in small (<20) groups of cooperating words,
the forthdefi({wordset}). See also ``thinking Forth''.

The things that differ among assemblers, are taken care of by
simple macro's, e.g. forthmacro({_COMMENT}) starts a comment.
Most of the time they don't have parameters.

The selection of parts that go or don't go into a particular configuration
is done by multiline macro's, generally with a call on a separate line.
Such as:
forthexample({
{_HIGH_BUF_}(_lbracket_
BUF1    EQU     EM-(KBBUF+2*2)*NBUF      ;_lbracket_ FIRST DISK BUFFER_rbracket_
STRUSA  EQU     BUF1-US         ;_lbracket_ User area_rbracket_
_rbracket_);{_END_}(_lbracket_ _HIGH_BUF__rbracket_)  })
Note how comments are protected from macro expansion by quotation.
The forthmacro({_END_}) is an adornment. It expands to nothing.
So it doesn't show up in the output,
but it helps to keep the generic source organised.

The forthmacro({worddoc}) macro defines a structure with additional information
of a word.
Generally it is placed in front of the word.
The same word can be found several times in the input file,
but only one is selected in a particular configuration.
The same goes for the corresponding forthmacro({worddoc}) .
forthbreak
Its fields are:
forthenumerate
forthitem
Wordset name.
forthitem
Word name.
forthitem
Pronunciation.
This is a pure textual and pronouncable identification of the word.
It is also used in forthfile({texinfo}) that doesn't handle special characters well.
forthitem
Stack effect.
The stack effect obeys all the conventions put forth in the user manual.
forthitem
Properties.
Properties are i.a. immediate and such, and the standards with which
this word complies.
Again this is described in the user manual.
forthitem
Description.
forthitem
References.
This is a list of names of other Forth words,
that can be studied to better understand this one.
forthitem
Tests.
This is a list.
The first and all other odd members is a test,
code that can be passed to Forth.
The second and all other even members is the expected outcome of the
preceding test.
forthendenumerate
forthmacro({worddoc}) are such that a structure starts with forthsamp({worddoc( })  and
end with a forthsamp({_rbracket_)}) at the end of a line.
This means that a worddoc
can be simply skipped if it occurrs in Forth code,
by defining a word forthcode({{worddoc(}}) that reads and ignores source up to the
end sentinel.

The forthmacro({worddocchapter}) macro defines a wordset.
It has the same fields as a forthmacro({worddoc}) macro,
but most are left empty.
It is primarily used for its ``description'' field,
that is used as an overview description for the wordset in glossaries.
These macro's can be put anywhere,
but take care to exclude macro's for wordsets that are not present.

@section The process
The ultimate information about how a ciforth is generated are the makefile's :
 forthfile({Makefile}) and forthfile({test.mak}) .

The process of generating a program proceeds along the following steps:
forthenumerate
forthitem
Generated the assembler source from the generic source via a configuration file.
The file suffix indicates which assembler to use.
forthitem
Generate an object file.
forthitem
Link the object file.
forthendenumerate
Once you have an assembler file,
you can do what you want with it.
Proceeding from an assembler source file to a binary is in general straightforward.

The process of generating program's documentation
(TeX and info)
proceeds along the following steps:
forthenumerate
forthitem
Generated the raw glossary documentation
from the generic source via a configuration file.
The file suffix is forthfile({.rawdoc}) .
forthitem
Sort the forthfile({.rawdoc}) file,
such that words of a wordset appear together,
and are preceeded by a wordset documentation.
The file suffix is forthfile({.mig}) .
forthitem
Generate the glossary documentation from the forthfile({.mig})
by expanding the forthmacro({worddoc}) 's into glossary entries by
forthfile({gloss.m4}) or forthfile({glosshtml.m4}) .
This, for a second time (!), takes into account the configuration file{}_VERBOSE_(
{, to generate exactly fitting information}).
forthitem
Expand the ``postponed markup's '' in forthfile({ciforth.mi})
by the macro's from forthfile({manual.m4}) to
generate the texinfo commands.
This file include all the other forthfile({.mi}) files with postponed markup's.
forthendenumerate

The process for generating html has the postponed markups and the
expansion into glossary entries in the same file forthfile({glosshtml.m4}).
Only the documentation of the glossary enters into html ,
and the forthfile({ciforth.html}) is generated from the intermediate
file forthfile({.mig}).

Generating documents is made more complicated by the requirements for special tables.
For forthsamp({html}) we want an extra alphabetic list of all the words where we click on to
get at the glossary entry immediately.
forthbreak
In forthfile({texinfo}) we need to build complicated menu structures, that refer back and forth.
forthbreak
This is done by separate passses over the forthfile({.rawdoc})  or forthfile({.mig})
files, with other macro's.
@chapter off we go
@section Introduction

What you find here is a Forth for the Intel 86.
Not much more can be said for such a highly configurable system.
But in this section we will try to summarize the common characteristics.

It borrows some philosophy from the old figforth.
It is in fact based on it, and its documentation in first draft
copied from it.
The Forths are build from
an assembler source, and it is (in general) an
indirect threaded Forth.
The motivation for having this type of Forth available follows from its
characteristics.
An assembler source has distinct advantages for getting started from
nothing.
An engineer might balk at the description of how to use a meta
compiler,
but feels at ease with a (much larger) assembler manual.
Although speed is currently in fashion, using subroutine threaded Forth's
with optimisers, indirect threading is the preferred choice for some
applications.
Furthermore the current trend of subroutine threaded Forth's may very
well be unsuitable for 64-bits processors like the Alpha.
I did this work, because I needed it for my thesis on computer intelligence.

@subsection 32 bits
It is unusual for a forth to be configurable as 16 or 32 bit.
It turned out that the addition of forthcode({CELL+}) goes a long way toward allowing
utilities like a decompiler to be 16/32 bit clean. In the documentation
mostly reference to cells can be made. But the macro's
forthsamp({{_BITS_}})
forthsamp({{_BIT16_}}) and
forthsamp({{_BIT32_}})
can be used to signify the actual number of bits and parts to refer to 16 and 32 bits only
respectively.

@subsection System requirements

This generic version -if suitably built- runs on industry standard hardware
("PC's") : standalone, under Linux and under MSDOS/MSWINDOWS.
To build, you need a version of forthprog({nasm}) , forthprog({TASM.EXE}) or forthprog({MASM.EXE}) on your system. I
recomment forthprog({nasm}) , it is an open source assembler and available on different
platforms, at least MSDOS and Unix. It solves a lot of the design errors I
find in the Intel ways of forthprog({MASM.EXE}) . It generates a binary without a linker.
On the opposite side, e.g. Borland's forthprog({TASM.EXE}) you can buy nowaday's only as part
of a giant C++ package.
If you want to use the generic possibilities you will need a Unix system
with all of its tools. I use GNU-Linux (RedHat) and do the makes and version
control on that. If you want your bootable floppies made from linux to be
MSDOS-compatible you need mtools.

@subsection Assembler sources

The following two assembly sources generated are supplied as a service.
These are in fact just examples. You can generate different ones (see next
section.)
The file forthfile({alone.asm}) can be assembled using forthprog({nasm}) . It includes a boot
sector such that it can boot from a standard floppy on a industry standard
Intel PC. If you have the mtools set (most Linux'es have it) the Makefile
shows you how to make the floppy. On MSDOS you can use forthprog({DEBUG.EXE}) .
If you run on Linux with
 forthsamp({mtools}) , forthsamp({make boot}) will do it.
The resulting floppy will even be recognized by
MSDOS, such that you can copy block sources to it.
 forthsamp({make moreboot})
will do this from Linux, then you will have forthfile({BLOCKS.BLK})
available.
forthsamp({make allboot})
will do it all, but it needs a working forth
on Linux for doing some calculations.
Otherwise on MSDOS (I recommended version 3.3, the most stable MSDOS ever)
adapt the example forthfile({genboot.asm}) .
The file forthfile({msdos.msm}) can be assembled
using forthprog({TASM.EXE}) and forthprog({MASM.EXE}).
The resulting Forth
executable can be run off hard disk and respects the file system on it.
It uses the file forthfile({BLOCKS.BLK}) .

@subsection A generic Forth
As was mentionned before, ciforth has one single source file: the generic forthfile({ci86.gnr}) .
All advantages of assembler source would be gone, if an engineer were
confronted with conditional compilation and lots of code for other systems
he doesn't want to learn or assemblers he doesn't want to use.
So we proceed in two steps. First a clean assembler source is generated from
the generic Forth using configuration files. Then the assembler source is
processed in one of a number of ways, each way familiar to one brand of
engineers.

You can customize at a number of levels.
forthenumerate
forthitem
Configuration files have extension forthsamp({.cfg}) , these are files with forthprog({m4})
commands. They are intended to use at the highest and easiest level of
configuration and contain their own simple usage instructions.
forthitem
forthprog({m4}) files have extension forthvar({.m4}), and control one aspect of genericity, such
as the header layout or the protection mode. You definitely need to know forthprog({m4})
to use these.
forthitem
Assembler files can be customised in the traditional way by adopting
constants, or commenting in source lines. The assembler files are distinct
from the one generic source. No forthprog({m4}), you need only cope with the directives
of your assembler, and will not see any code applicable to other operating
systems or I/O systems. (It is not commented out, it is just not there.)
forthitem
You can adapt the generic source. This is difficult, but gratifying.
If you manage to ISO-fy it, the result is a lot of ISO systems, not just one.
forthendenumerate

@subsection Level 1 customization.

This is assuming you run on Unix.

By specifying what you want in a configuration file you can generate a host
of assembler listings. This is as simple as replacing ``_yes'' with ``_no'' in
configuration files.
See the examples forthfile({msdos.cfg}) and forthfile({alone.cfg}) and the Makefile.
You can find out what the options are by inspecting forthfile({prelude.m4}) .

With respect to the assembler you can choose between forthprog({nasm}) and forthprog({MASM.EXE}) , with
file extension forthfile({.asm}) and forthfile({.msm}) respectively.
The forthvar({.msm}) are acceptable by
TASM.EXE too. You can generate an equivalent forthfile({.s}) file, but this is
experimental and doesn't lead to a working forth.

With respect to the I/O (words like forthcodeni({EXPECT}) forthcodeni({R/W}) ) you can choose between
three on MSDOS.
You can use dos forthmacro({_CLASSIC_}) in the classic way as with the original. This
means that the floppy is used directly without regard for directory
structures. This uses calls that are declared obsolete.
You can use dos in a modern way. forthmacro({_MODERN_}). This allocates block in the
file with name forthfile({BLOCKS.BLK}) . This name is available in the string forthcodeni({BLOCK-FILE})
for you to change, also at run time. No (as of 2000 ) obsolete MSDOS calls
are used (Checked against MS-DOS programmers reference "covers through
version 6" ISBN 1-55615-546-8)
You can use the BIOS forthmacro({_USEBIOS_}) No MSDOS interrupts are required.

With repect to I/O on linux you can choose between c-based and native.
The c-based version may be portable to other I86 unices. The native version
of course not. All linux versions have their blocks in a file. (Accessing
a floppy in the classic way is perfectly possible -- and implementing it would
be a perfectly pointless exercise.)

With respect to the hosting you can choose between forthmacro({{_HOSTED_}}) ( forthmacro({{_HOSTED_LINUX_}}) or
 forthmacro({{_HOSTED_MSDOS_})}) and forthmacro({{_BOOTED_}}) . ( forthmacro({{_BOOTDF_}}) or forthmacro({{_BOOTHD_}}) ).
A hosted version relies on MSDOS or Linux to get the program started.
(It may or may not use MSDOS for I/O, once started.).
A forthmacro({{_BOOTED_}}) version contains a boot sector, such that
you you can make a standalone version that boots from floppy or hard disk.
A forthmacro({{_BOOTED_}}) version may very well be startable from plain DOS and its files
visible from DOS.

Of course a forthmacro({{BOOTED_}}) version that tries to use MSDOS I/O (or Linux) crashes
immediately, so not all versions are useful.

You have a choice between 16 or 32 protected mode and real mode.
Of course on Linux real mode is not an option, (but you could run the
MSDOS emulator). Protected mode Forths on MSDOS cannot be started from
virtual real mode, e.g. they will not run in a "DOS box" in Windows.

If you manage to specify conflicting options the preprocessor (forthprog({m4})) breaks
off and you can find the exit code in forthfile({postlude.m4}) . Than you can reason back
why this is a conflict. For example error 1000 indicates floppy and hard
disk i/o at the same time.
From forthfile({postlude.m4}) you see that forthmacro({{_RWFD_}}) and forthmacro({{_RWHD_}}) are
on at the same time. forthmacro({{_RWHD_}}) is turned on because you wanted to boot from hard
disk or you specified it yourself in the first place. Etc.

forthfile({postlude.m4}) does you another favor. It derives logical consequences, such
as once you decide for a forthmacro({{_REAL_}}) mode Forth, it must be forthmacro({{_BITS16_}})
and you need not specify
that_VERBOSE_({ (And yes, we could add a real mode 32 bits Forth, any volunteers?)}).
In particular forthmacro({{_LINUX_N_}}) or forthmacro({{_LINUX_C_}}) define a whole configuration.

@subsection Level 2 customization.
You are on your own here.

@subsection Level 3 customization.

So you have this assembler file,
and it looks like what you want to have, but not quite.
And of course it doesn't work.

@subsubsection My rants
The usual customisation in assembler files is possible.
If you use other than 3" floppy disks you have to specify the disk
parameters. Parameters for a 5" HD floppy are present and can be commented
in.
If you do not need a DOS-compatible floppy, you can put the image
immediately after the boot sector. A bootable hard disk version always works
like that.
You can change the default name of the forthcodeni({BLOCK-FILE}) at run time.
If you want to change the header layout, you will find that the way headers
are done via MACRO's make it more pleasant to use the generic listing.
If you may want you can use this as a starting point for generating a whole
other Forth (like me).
If you want to boot into your 20 Gbyte disk (like me), you probably have a
version 3.0 super modern LBA BIOS. There is no file system, just 20,000,000
blocks (and yes a 16 bit system would be inconvenient). If you want to use
an older system you must experiment by using the forthcodeni({BIOS}) word.
(You need not resort to assembler for experimenting.)
Then you can adapt your assembler listing.
@subsubsection FIG's rants

You may want to use the assembly code of this ciforth to
base a new Forth on. If this adversely affects the documentation
I urge you not to do that but to use the generic system.

The  following words
are traditionally
the only portion that need change between different
installations of the same computer cpu.
They cannot come close to the capabilities
of the generic system,
and should be used for minor modifications only.

There are five words that need adaptation:

@table @code
forthitem KEY
Push the next ascii value (7 bits) from the terminal keystroke to the
computation stack and execute NEXT. High 9 bits are zero. Do not echo
this character, especially a control character.
forthitem EMIT
Pop the computation stack (16 bit value). Display the low 7 bits on the
terminal device, then execute NEXT. Control characters have their
natural functions.
forthitem ?TERMINAL
For terminals with a break key, wait till released and push to the
computation stack 1 if it was found depressed; otherwise 0.
Execute NEXT. If no break key is available, sense any key depression as
a break (sense but don't wait for a key). If both the above are
unavailable, simply push 0 and execute NEXT.
forthitem CR
Execute a terminal carriage return and line feed. Execute NEXT.
forthitem R/W
This colon-definition is the standard linkage to your disc. It requests
the read or write of a disc block, be it raw disk or allocated in a file.
@end table

On primitive systems these may be jumps to ROM-code. But generally on i86
facilities like this are available using forthdefi({INT})'s a kind of traps.
These observe operating system protocols and are available as high level forth
code.
@subsubsection FIG's rants : Ram disc simulation

If disc is not available, a simulation of forthcode({BLOCK}) and
forthcode({BUFFER}) may be made in RAM.
The following definitions setup high memory as mass storage.
Referenced ``screens'' are then brought to the ``disc buffer'' area.
This is a good method
to test the start-up program even if disc may be available.

forthexample(
{HEX
4000 CONSTANT LO ( START OF BUFFER AREA )
6800 CONSTANT HI ( 10 SCREEN EQUIVALENT )
: R/W >R ( save boolean )
    B/BUF * LO + DUP
    HI > 6 ?ERROR ( range check )
    R> IF ( read ) SWAP ENDIF
    B/BUF CMOVE ; })

Insert the code field address of forthcode({R/W}) into forthcode({BLOCK}) and forthcode({BUFFER})
and proceed as if testing disc.
This forthcode({R/W}) simulates screens 0 thru 9, in the
memory area 04000H thru 067FFH.
@subsubsection FIG's rants : Debugging an assembled system.

Let us assume we have an system based on an assembler listing
and we want to debug it.

Here are the sequential steps:
forthenumerate
forthitem
Familiarize yourself with the model written in Forth, the glossary, and
specific assembly listings.
forthitem
Edit the assembly listings into your system. Set the fifth
boot-up parameters (WARNING) to 0
(warning messages are shown as simple numbers).
forthitem
Alter the terminal support code ( forthcode({KEY}) , forthcode({EMIT}) , etc,) to match your system.
Observe register protocol specific to your  implementation!
forthitem
Place a break in your debugger at the end of NEXT,
just before indirectly jumping via register W to execution.
forthsamp({W}) is the Forth name for the register
holding a code field address.
In ciforth this is the Intel 86 register AX.
If your NEXT is inline code,
for the moment replace it by a jump.
Mostly this can be done by inactivating the macro forthmacro({_NEXT32_})
through removing the line
forthsamp({_BITS32_(_lbracket_{define}(_lbracket__NEXT_rbracket_,_lbracket__NEXT32_rbracket_)_rbracket_)})
from forthfile({postlude.m4}).
forthitem
Enter the cold start at the origin.
Upon the break, check that the interpretive pointer IP points within forthcodeni({ABORT})
and W points to forthcodeni({SP!}) .
In ciforth IP is held in the Intel 86 register DI.
forthcodeni({COLD}) being a colon-definition,
the IP has been initialized on the way to NEXT and your testing will
begin in forthcodeni({COLD}) .
The purpose of forthcodeni({COLD}) is to initialize IP, SP, RP, VP,
and some user variables from the start-up
parameters at the origin.
forthitem
Continue execution one word at a time.
Clever individuals could write a simple trace routine to print IP, W, SP, RP
and the top of the stacks.
Run in this single step mode until the greeting message is printed.
Note that the interpretation is several hundred cycles to this stage!
forthitem
Execution errors may be localized by observing the above pointers when a
crash occurs.
forthitem
After the word forthcodeni({QUIT}) is executed (incrementally),
and you can input a "return" key and get ``OK'' printed, remove the break.
You may have some remaining errors, but a reset and examination of the
above registers will again localize problems.
forthitem
When the system is interpreting from the keyboard, execute forthcodeni({EMPTY-BUFFERS})
to clear the disc buffer area.
forthitem
If your disc driver differs from the assembly version, you must create
your own forthcodeni({R/W}) .
You may test the disc access by typing:
forthsamp({0 BLOCK 64 TYPE})
This should bring block zero from the disc to a buffer and
type the first 64 characters.
If forthcodeni({BLOCK}) (and forthcodeni({R/W}) ) doesn't function--happy hunting!
forthendenumerate

@subsection Level 4 customization.
Contact me if you want to contribute to the wider usability of this package.

@subsection Programs
In the file forthfile({BLOCKS.BLK}) is available a screen editor, assembler, decompiler
and tools like forthcodeni({DUMP}).
Beware! Some of the tools handle hards disks.
There are example programs and benchmarks.
Everything under screen 100 you will find
more or less working, but maybe not on your system.
Everything loaded from 8 is used by me on a regular basis and is 16/32 bits
clean.
Beware! The full screen editor doesnot work under Linux (protection).
The system doesn't load it under Linux with forthsamp({8 LOAD}) .
The program forthprog({wc}) is an example of how to use forthprog({lina}) as a scripting language.

@subsection The joy of genericity

I added generic words forthcodeni({BIOS}) , forthcodeni({BDOS}) and forthcodeni({LINOS}) .
These allow to have high level
words to handle about all ``BIOS'' and interrupt 21 calls.
Linux is better.
forthcodeni({LINOS}) handles all Linux system calls.

Genericity is acomplished by the Unix tools forthprog({m4}) . I use GNU forthprog({m4}). This is a weird
tool but powerful. Forthers probably like it.
Some implementation details are hidden in the file forthfile({header.m4}) . In particular
the way headers are built. I maybe
want to get rid of the forthcodeni({WIDTH}) and forthcodeni({TRAVERSE})
peculiarities and you may want to have the headers alligned at word bounds.
This is easily done by changes to forthfile({header.m4}) .
This kind of possibilities were in fact the motivation for this undertaking.

@subsection Web sites and availability.
A newer or improved version may be gotten from
 forthurl({http://home.hccnet.nl/a.w.m.van.der.horst/ciforth.html})
Nasm is found at

ftp://ftp.us.kernel.org/pub/software/devel/nasm/source/

forthurl({http://www.cryogen.com/Nasm/})

The FIG source this is based on is at

MASM.EXE is available from IBM, at least the version 1.0 I used.
Microsoft seems to distribute later versions.

The original fig documentation is obtainable via
 forthurl({http://home.hccnet.nl/a.w.m.van.der.horst/figforth.html})
This include the pictures.

@subsection Linux application notes ciforthc version
The linux forth called ciforthc has its i/o based on c. This may seem more
portable but it isn't. Where c is very portable on Linux, the way assembler
is linked with forth is not documented (as far as I can tell, in my version.
Linux improves overnight, so this may no longer be true.)
 forthcodeni({KEY?}) is implemented using a forthsamp({select}) system call.
The forthcodeni({EXPECT}) has not the " return if
maximum reached property", so it is not strictly conforming. This can be
done at the expense of handling each character separately. (Use forthcodeni({KEY}) to
implement forthcodeni({EXPECT}) as in the CLASSIC I/O model). This results in loosing
interruptability. Moreover Linux knows better what the forthcodeni({RUBOUT}) key should be,
although for your convenenience it is already placed in a user variable and
can be easily changed.
The c-approach allows signals to be handled in a familiar way.
By using quit, a loop can be interrupted. So ^\ results in a warm start.
A segmentation fault also results in a warm start. ^C immediately leaves.
^S/^Q can be used to hold up output and are not interpreted as a break
in e.g. WORDS.

@subsection Linux application notes lina version
The lina version is based on a single assembler source, built without
trickery and binary-portable accross Linux Intel (all systems were it has
been tried work : 1.2.13 .. 2.0.13). No run time c-libraries, no compile time
c-libraries, no libraries at all. It is built directly on the solid rock
of the system calls by ignoring a taboo c-programmers suffer from.
forthexample({
    nasm -felf lina.asm
    ld lina.o -s -o lina
    strip lina
})
It is about 20k and the dictionary space is set at 64 Mbyte.

Blocks are allocated in a file called forthfile({BLOCKS.BLK}) This name can be changed in
listing and also during run time.
 forthfile({BLOCKS.BLK}) can be changed into an editable file and back by
forthsamp({cat BLOCKS.BLK | fromblocks > blocks.frt
cat blocks.frt | toblocks > BLOCKS.BLK})

The user variable forthsamp({EM}) still is the end of the memory.
The forthsamp({M4_EM}) in the configuration files is such that it designates the relative
size, from the relocatable start.
Consequently it is not the same as the user variable.
(The relocatable start is some 128 Mbyte into the memory space.).
@subsection Bugs
See the separate test report for an indication of which and how far
versions have been tested.
forthenumerate
forthitem
Linux version.
Once you have used a SIGQUIT to interrupt a loop, forthcodeni({BYE}) no longer works.
You can exit the program by "0 0 0 1 LINOS", which is exit(0) in
c-parlance or by pressing ^C, or by killing it from some other terminal,
or by just closing the window.
You will not encounter this bug in version 2.148,
because that version crashes immediately,
due to build problems.
forthitem
forthcodeni({OUT}) may not be observed in all I/O models. Needs examination.
forthitem
   More a misfeature. The negative error numbers of Linux system calls
   can be handled by negative offset's from screen 4.
forthendenumerate

@chapter Now: release it!
To release the software use the command

forthsamp({make release VERSION=forthvar({NNN})})

where forthvar({NNN}) is as appropriate: the RCS version
number (for ciforth 3d###) for a beta version and the minor
release numbers for an offical release (ciforth has major
release 4, because fig 2 istaken, my latest fig is major
release 3, so there.)
Offical releases have the forthkey({d}) replaced with a forthkey({a})
And RCS version numbers always run in
the hundreds, official minor releases are one digit.
And during started up it tells "ciforth beta ", if it is not official.

There are basically two zip files with the same contents, and
a formidable content at that. They are
 forthfile({ci86gNNNN}) and forthfile({fgNNNN.zip.}) The forthsamp({fg}) version features 8+3 file names,
appropriate for ISO9660 or FAT (MSDOS) file systems.
@chapter Loose ends
@section list of safe customisation in the assember.
RTS TIB / Return stack area

The block stuff is easily the most crappy part of this system. What
little advantage it had, is no longer applicable. Interestingly the fig
block
system is copied verbatim into ANS.
32 bits systems can enlarge the dictionary by redefining the stacks and the
terminal input buffer positions. The block buffers are in the way. So I put
them in the dictionary space. In fact the block buffers have become the
data field of FIRST.

My changes :
forthenumerate
forthitem
There is a very simple forthcode({INCLUDED}) to compile files. This goes a long
way towards eliminating the blocks. They become a convenience rather
than a necessity.
forthitem
Blocks are always forthcode({UPDATE}) 'd immediately. This makes words
like forthcode({FLUSH}) trivial.
forthitem
forthcode({BLK}) and its ilk are defined by second guessing. They do not
dictate what is going on. This is contrary to ISO, but makes the
basic system so much cleaner. Blocks that load other blocks are forthcode({LOCK}) ed
during that time, so that forthcode({WORD}) need not constantly look
whether the rug is pulled out from under it.
forthitem
forthendenumerate

The backspace character is also in the boot-up origin parameters. It is
universally expected that "rubout" is the backspace.

Despite all the talk about user variables, I am not aware that a multi-user
fig forth existed, ever. The above block policy is not compatible with
multi-user (but the actual code is not worse than it ever was, and you can
rebuilt it with 1000 block buffers if you like.)

CUSTOMISING

The name USER reflects that more than one user could use the dictionary
and users could share the background storage, provided certain precautions
are taken. About this you can forget.
What you can do is, store back you user variables in the boot-up
parameters as follows
forthsamp({USVA @   ' USVA @   +ORIGIN !})
where forthvar({USVA}) is the user variable which value you want to keep.

Underneath the I/O model has improved. TYPE and EXPECT are drawn into
the I/O model dependant part. If at all possible CR and EMIT used TYPE
such that TYPE becomes a natural vectoring point.

Rubout is best left to the EXPECT code. Remember, a Linux itself knows the
rubout key for any of its 500+ known terminal types and isn't it nice in
MSDOS that F3 gets the previous command back for you?
If EXPECT builds up a line from separate key strokes, and if you ever want
to change the rubout key, just change the RUBOUT user variable.


@section MULTI-USER

The name forthcodeni({USER}) reflects that more than one user could use the dictionary
and users could share the background storage,
provided certain precautions are taken.
These precautions are
forthenumerate
forthitem
Variables that can be different for different users,
must be defined as an offset to an area,
that is different for each user: the forthdefi({user area}) .
forthitem
Provisions that maintains the integrity of the dictionary.
Different scratchpads for each user.
forthitem
Different stacks, user area's and terminal input buffer's for each user.
forthitem
A means to switch applications.
forthendenumerate

Almost nothing from this is realised in the figforth model.
In fact only 1, but there is the pointer to that area is
in the bootup parameters, a design error.

Still forthcode({USER}) variables seem to serve a useful purpose.
They are initialised during start up,
by changing it one could modify the system.
This however is only seemingly.
Because the initialised memory is saved anyway,
the initial values would be stored even if they
were ordinary variables.
So we are left with the disadvantage that we have to store them
back before saving the system.
Remains the advantage that they can be restored by typing forthcode({COLD}) .
This too is hardly an advantage because with the fast mass storage you would
rather type forthcode({BYE}) and forthsamp({forth}) as a much safer way
to restart your Forth.
Furthermore forthcode({COLD}) performs a buggy forthcode({FORGET}) ,
it cannot reinitalise the dictionary for a system that has active
glossaries, that remain after a boot.

dnl@node Glossary Index,,,Top
@unnumbered Program Index
This index lists programs words.
@printindex pg
dnl@node Forth Word Index,,,Top
@unnumbered Forth Word Index
This index lists forth words.
@printindex fn
@node Concept Index,,,Top
@unnumbered Concept Index
This index lists concepts.
dnl The first reference is where the concept is explained.
@printindex cp
@summarycontents
@contents
@bye