Processing TEI critical apparatus data with Lua

Unofficial ConTeXt Wiki mirror

Last modified: 2026-08-30

🚧 Under construction — This page and its subpages are currently being revised. Contributions are welcome: feel free to edit and improve them.

Guide 5 of 6 — Processing TEI critical apparatus data with Lua

Previous: Guide 4 — Encoding complex textual variation in TEI  ·  Collection overview  ·  Glossary  ·  Next: Guide 6 — Typesetting a TEI critical apparatus with ConTeXt

What you will build in this guide

Guides 1–4 produced a TEI source containing witness declarations, apparatus entries, references, responsibility, uncertainty, and other editorial distinctions.

Guide 5 does not replace that source. It constructs a controlled intermediate representation: 

TEI XML
   │
   â–¼
Lua processing
├── extract
├── normalize
├── resolve
├── check project rules
└── diagnose
   │
   â–¼
editorial records
├── source connection
├── normalized identifiers
├── resolved objects
├── explicit categories
├── validity status
└── diagnostics
   │
   â–¼
Guide 6: ConTeXt composition

You will build this layer progressively:

  1. a witness registry;
  2. normalized apparatus records;
  3. reference resolution;
  4. project-level editorial checks;
  5. errors and warnings;
  6. accepted and rejected record collections;
  7. a stable Lua–ConTeXt contract.

The functional examples use a representative subset of Guide 4 so that the processing pipeline remains readable.

Contents

Guides 1–4 progressively constructed the TEI source.

By the end of Guide 4, the document could contain: 

TEI
├── declarations
│   ├── witnesses
│   └── editorial responsibility
│
└── text
    └── apparatus
        ├── lem / rdg
        ├── @wit
        ├── @resp
        ├── omissions
        ├── documentary alterations
        ├── uncertainty
        └── grouped or complex readings

At this point the problem changes.

We no longer ask mainly: 

How do we encode another textual phenomenon?

We now ask: 

How do we process all these encoded relations consistently and safely?

Guide 5 therefore adds a new layer: 

GUIDES 1–4

TEI documentary source
        │
        â–¼
encoded witnesses and apparatus
        │
        â–¼

GUIDE 5

Lua processing                         ← NEW
        │
        â–¼
checked editorial records
        │
        â–¼

GUIDE 6

ConTeXt typesetting

The central architecture is: 

                 DOCUMENTARY SOURCE

                       TEI XML
                          │
                          â–¼
              +----------------------+
              |         Lua          |
              |                      |
              | extract              |
              | normalize            |
              | resolve              |
              | validate             |
              | diagnose             |
              +----------------------+
                          │
                          â–¼
                 EDITORIAL RECORDS
                          │
                          â–¼
                       ConTeXt
                          │
                          â–¼
               TYPOGRAPHICAL OUTPUT

Guiding principle.

TEI records the documentary evidence and scholarly relationships. Lua interprets the encoded structures according to explicit editorial rules. ConTeXt gives the resulting records their typographical form.

The three layers cooperate, but they do not have the same responsibility.

1. Where this guide fits in the collection

The six-guide progression can now be understood as: 

Guide 1   STRUCTURE
             │
             â–¼
Guide 2   IDENTITIES
             │
             â–¼
Guide 3   BASIC RELATIONS
             │
             â–¼
Guide 4   COMPLEX RELATIONS
             │
             â–¼
Guide 5   INTERPRET + VALIDATE          ← YOU ARE HERE
             │
             â–¼
Guide 6   COMPOSE

How to read this diagram. Guides 1–4 primarily construct the scholarly data model. Guide 5 changes the task from encoding to controlled interpretation and checking. Guide 6 receives the resulting records and concentrates on composition.

The first four guides primarily built the scholarly data model.

Guide 5 transforms that model into a stable intermediate representation that Guide 6 can typeset without repeatedly interpreting raw XML.

The processing system developed here will progressively:

  1. build a registry of textual witnesses;
  2. convert TEI apparatus entries into Lua records;
  3. normalize identifiers, witness lists, and textual values;
  4. resolve witness references against the registry;
  5. validate the resulting editorial records;
  6. report errors and warnings;
  7. separate accepted records from records requiring correction;
  8. make checked records available to ConTeXt.

Stage reached.

The TEI source is already rich enough to describe the edition.

Guide 5 does not replace that source. It constructs a controlled processing layer between documentary encoding and typographical composition.

2. Why Lua in an LMTX workflow?

The question should be asked explicitly: 

If ConTeXt can already process XML, why introduce Lua?

There are two complementary answers.

2.1. Lua is already part of the LMTX environment

ConTeXt LMTX runs on LuaMetaTeX. Lua is therefore not an external programming environment inserted between TEI and ConTeXt.

A useful mental map is: 

                 ConTeXt LMTX
                      │
              +-------+-------+
              │               │
              â–¼               â–¼
        ConTeXt MkXL      LuaMetaTeX
                              │
                              └── Lua is available natively
                                     │
                                     â–¼
TEI XML ─────────────────────â–º processing
                                     │
                                     â–¼
                                  ConTeXt
                                     │
                                     â–¼
                                    page

How to read this diagram. Lua does not introduce a foreign application into the workflow. It uses the programming language already available in the engine environment on which ConTeXt LMTX runs. TEI processing, editorial transformation, and typesetting can therefore remain inside one integrated toolchain.

Why Lua in an LMTX workflow? ConTeXt LMTX runs on LuaMetaTeX. Lua is therefore already available where the XML is being processed and the document is being composed. When editorial relations require reusable normalization, cross-reference lookup, project-level checks, or diagnostic reports, Lua is a natural processing layer within the same environment.

Further reference.

For the engine architecture, see the LuaMetaTeX reference manual.

For programming Lua from ConTeXt, see ConTeXt and Lua programming.

2.2. A minimal proof: Lua is already here

The following Garden example does not perform editorial processing. Its only purpose is to demonstrate that Lua code can run directly inside an LMTX document:

The important fact is architectural: there is no separate conversion program to launch merely in order to use Lua.

2.3. Direct XML processing remains useful

Lua is not required simply because the source is XML.

For a small or local transformation, ConTeXt XML setups can remain the simplest path: 

TEI XML
   │
   â–¼
ConTeXt XML setup
   │
   â–¼
typeset result

That model was sufficient for the inspection tests in Guides 1–4.

The processing problem changes when each apparatus entry begins to require operations such as: 

parse @wit
    ↓
normalize identifiers
    ↓
look up declarations
    ↓
apply project rules
    ↓
record warnings
    ↓
retain invalid objects for diagnosis
    ↓
pass only accepted records to composition

At that point, repeating the same interpretation inside every typographical setup becomes difficult to maintain.

Lua is useful, not mandatory. Direct ConTeXt XML processing remains appropriate when the mapping from XML to output is local and simple. Introduce a Lua intermediate layer when processing becomes relational, reusable, rule-driven, or diagnostic.

2.4. The editorial reason for the intermediate layer

The accumulated TEI data now contain relationships across the document: 

witness declaration
        â–²
        │ @wit
        │
apparatus reading

responsibility declaration
        â–²
        │ @resp
        │
editorial intervention

The task is no longer only to select one XML element. It is to interpret relations consistently across many elements.

The architectural choice is therefore: 

DIRECT XML → TYPESETTING
works well while processing is local and simple

                    BUT

when every apparatus entry may require
normalization + lookup + checks + diagnostics

                    ↓

TEI XML
   │
   â–¼
Lua intermediate processing
   │
   â–¼
stable editorial records
   │
   â–¼
ConTeXt

How to read this diagram. Lua is introduced because the task has changed, not because XML has become unreadable. The intermediate layer centralizes operations that would otherwise be repeated in many typographical macros.

2.5. Division of labour

The three layers remain distinct: 

TEI
├── evidence
├── documentary distinctions
└── scholarly assertions
        │
        â–¼
Lua
├── extract
├── normalize
├── resolve
├── check project rules
└── diagnose
        │
        â–¼
ConTeXt
├── typography
├── punctuation
├── spacing
├── placement
└── page architecture

How to read this diagram. TEI remains the documentary and scholarly source. Lua turns selected encoded relations into controlled editorial objects. ConTeXt determines how those objects appear on the page.

Division of labour. TEI records the scholarly claims; Lua applies documented processing rules to those claims; ConTeXt composes the result. The boundaries are deliberate even though all three layers operate inside one LMTX workflow.

Further reference.

For direct XML processing, see XML setup commands.

For Lua-based XML traversal, see Processing XML with Lua.

For the broader TEI → ConTeXt architecture, see TEI XML.

Stage reached. We now know why Lua belongs in Guide 5: it is already native to the LMTX environment, and the editorial processing has become complex enough to benefit from an explicit intermediate layer.

Part I — Understand the processing model

3. From XML syntax to editorial knowledge

3.1. Reading is not yet interpreting

Consider: 

<rdg wit="#ms-A #ed-C">λÏŒγος</rdg>

An XML parser can expose: 

element name: rdg
text:         λÏŒγος
@wit:         "#ms-A #ed-C"

But the string: 

"#ms-A #ed-C"

has not yet become editorial knowledge.

The processing path is: 

XML PARSER

wit="#ms-A #ed-C"
     │
     â–¼
character string
"#ms-A #ed-C"


LUA PROCESSING

"#ms-A #ed-C"
     │
     â–¼
split references
     │
     â–¼
 A       C
 │       │
 â–¼       â–¼
lookup  lookup
 │       │
 â–¼       â–¼
known?  known?
 │       │
 +---+---+
     │
     â–¼
resolved witness relations

Lua may need to establish:

From syntax to editorial knowledge.

XML parsing makes the encoded structure accessible. Lua processing determines how that structure is to be understood within the rules of the edition.

3.2. Normalization: equivalent notation, one internal form

Equivalent information may appear as: 

wit="#ms-A #ms-B"
wit=" #ms-A   #ms-B "
wit="#ms-A #ms-A #ms-B"

If the edition treats spacing differences and repeated references as insignificant, Lua can normalize all three to: 

{ "ms-A", "ms-B" }

The operation can be visualized as: 

RAW TEI

wit="  #ms-A   #ms-B  #ms-A "
          │
          â–¼
      trim spaces
          │
          â–¼
    "#ms-A #ms-B #ms-A"
          │
          â–¼
        split
          │
      A   B   A
          │
          â–¼
      deduplicate
          │
          â–¼
      { "ms-A", "ms-B" }

Normalization may include:

Normalization is an interpretative operation.

It defines which differences in the source are editorially significant and which are merely differences of notation.

Normalization should therefore follow documented project rules rather than being treated as an invisible programming convenience.

3.3. Reference resolution changes the status of a value

Suppose the TEI header declares: 

<witness xml:id="ms-A">
  Paris, Bibliothèque nationale de France, grec 1807
</witness>

and a reading contains: 

<rdg wit="#ms-A">...</rdg>

Before resolution: 

#ms-A
│
└── character string

After resolution: 

#ms-A
 │
 â–¼
normalize
 │
 â–¼
A
 │
 â–¼
registry lookup
 │
 â–¼
witness record A
 ├── siglum
 ├── description
 ├── kind
 └── order

Thus: 

STRING
  │
  â–¼
IDENTIFIER
  │
  â–¼
RESOLVED EDITORIAL OBJECT

How to read this diagram. The characters in @wit begin as encoded text. Normalization turns them into a controlled identifier; registry lookup turns that identifier into a known editorial object. This change of status is one of the main reasons for introducing the intermediate layer.

If no record exists, the same process produces a detectable editorial problem.

3.4. Project-level validation adds another state

A document may be well-formed XML and still contain unusable apparatus data.

TEI validation and Lua project checks are not the same thing.

Schema validation asks whether a document conforms to the selected TEI model. The Lua checks developed in this guide apply additional rules chosen for this tutorial or for a particular edition: reference integrity, controlled categories, required fields, normalization policy, and the conditions treated as errors or warnings.

A rule enforced here must therefore not be presented as a universal TEI constraint.

Further reference.

Guide 1 introduced the distinction between XML well-formedness, TEI validation, and editorial correctness.

For formal TEI conformance, see the TEI P5 chapter Using the TEI. The checks implemented below belong to the processing profile of this tutorial.

For example:

The progression is: 

XML
 │
 â–¼
readable structure
 │
 â–¼
normalized Lua record
 │
 â–¼
resolved references
 │
 â–¼
project-level editorial checking
 │
 +---------------+----------------+
 │               │                │
 â–¼               â–¼                â–¼
VALID         WARNING           INVALID
 │               │                │
 │               │                └── retained for diagnosis
 │               │
 └───────┬───────┘
         │
         â–¼
usable record

Well-formed XML is not necessarily editorially usable XML.

Syntax may be correct while references, classifications, or apparatus relationships remain incomplete or contradictory.

3.5. Separate interpretation from typesetting

Without an intermediate layer, one ConTeXt macro might have to: 

read XML
   ↓
split @wit
   ↓
remove #
   ↓
lookup witnesses
   ↓
check validity
   ↓
decide omission status
   ↓
sort sigla
   ↓
choose punctuation
   ↓
typeset

That mixes several responsibilities.

The architecture adopted here is: 

TEI XML
   │
   â–¼
Lua
├── identity
├── normalization
├── resolution
├── consistency
└── editorial status
   │
   â–¼
checked records
   │
   â–¼
ConTeXt
├── placement
├── visual hierarchy
├── spacing
├── punctuation
└── page construction

Division of labour.

Lua prepares editorially meaningful records. ConTeXt transforms those records into pages.

4. The processing pipeline

The complete conceptual workflow is cumulative: 

TEI documentary source
        │
        â–¼
1. EXTRACTION
        │
        â–¼
raw Lua values
        │
        â–¼
2. NORMALIZATION
        │
        â–¼
consistent fields
        │
        â–¼
3. REFERENCE RESOLUTION
        │
        â–¼
known editorial objects
        │
        â–¼
4. VALIDATION
        │
        ├── errors
        ├── warnings
        └── accepted records
        │
        â–¼
5. INTERMEDIATE REPRESENTATION
        │
        â–¼
records prepared for ConTeXt
        │
        â–¼
6. TYPESETTING                         ← Guide 6

How to read this diagram. Each stage answers a different question. Extraction selects relevant XML structures; normalization controls their surface form; resolution connects references to declarations; project checks test declared editorial rules; the intermediate representation becomes the stable input contract for Guide 6.

Stage Input Main question Result
Documentary encoding TEI elements and attributes What evidence and relationships are being recorded? Structured XML source
Extraction TEI XML Which structures are relevant to this task? Raw Lua values
Normalization Raw strings and content Which surface differences are insignificant? Controlled internal fields
Reference resolution Identifiers and references Which declared object does this value denote? Resolved editorial objects
Validation Normalized and resolved records Are the records usable under project rules? Errors, warnings, valid records
Intermediate representation Validated data What stable structure should typesetting receive? Records prepared for ConTeXt
Typesetting Validated records How should the information appear on the page? Printed or electronic apparatus

These are conceptual stages. A practical Lua function may perform several of them together, but the distinctions remain valuable because they show where a decision belongs and where an error should be reported.

4.1. TEI remains the documentary source

The Lua representation is deliberately smaller than the TEI document.

That does not make it a replacement source. 

TEI
│
├── documentary richness
├── scholarly distinctions
├── source relationships
└── reusable encoding
        │
        │ selective transformation
        â–¼
Lua record
│
├── fields needed now
├── normalized values
├── resolved objects
└── source connection

Where practical, a record should retain a connection to its source XML node or identifier so that diagnostics can be traced back to the encoding.

Further reference.

For ConTeXt's Lua-side XML traversal facilities, see Processing XML with Lua.

For the TEI structures being processed, return to Guides 2–4 and the TEI P5 chapter Critical Apparatus.

4.2. Lua creates task-oriented editorial objects

A witness may become: 

{
    id          = "ms-A",
    siglum      = "A",
    description = "Paris, Bibliothèque nationale de France, grec 1807",
    kind        = "manuscript",
    order       = 1,
}

An apparatus entry may become: 

{
    id       = "app-001",
    lemma    = {
        text      = "λÏŒγος",
        witnesses = { "ms-A" },
    },
    readings = {
        {
            text      = "λέξις",
            witnesses = { "ms-B", "ed-C" },
        },
    },
    valid = true,
}

The relationship is: 

TEI STRUCTURE
     │
     â–¼
identify editorial roles
     │
     â–¼
LUA RECORD
     │
     ├── predictable fields
     ├── normalized values
     └── source link

4.3. The intermediate representation is a contract

Once processing is complete, ConTeXt should be able to rely on several properties:

The intermediate representation is a contract.

Lua guarantees the editorial status and internal structure of accepted records. ConTeXt can therefore concentrate on their presentation instead of reinterpreting XML syntax at every typographical command.

Scope of the processing examples.

Guide 4 introduced responsibility, certainty, documentary substitutions, reading groups, local word-order variation, and unequal textual extent in addition to simple substitutions and omissions.

The functional MWEs below deliberately process a representative subset so that the pipeline remains readable. The same intermediate architecture can be extended with fields for @resp, @cert, documentary operations, reading groups, and other project-specific structures.

Part II — Build the processing layer

5. Stage 1 — Build a witness registry

Guide 2 declared witnesses in TEI.

Guide 5 now turns those declarations into a processing structure.

5.1. From <listWit> to a Lua registry

The TEI structure is: 

listWit
├── witness A
├── witness B
└── witness C

The processing path is: 

TEI
listWit
   │
   â–¼
collect witness declarations
   │
   â–¼
normalize identifiers
   │
   â–¼
check uniqueness
   │
   â–¼
LUA WITNESS REGISTRY
├── by_id
├── in_order
└── errors

The two main collections answer different questions: 

witnesses.by_id["ms-B"]
        │
        └── Which witness is B?

witnesses.in_order[2]
        │
        └── Which witness occupies this declared position?

Technical note.

Direct lookup and documentary order are different requirements. Keep them explicit rather than relying on one Lua table to represent both implicitly.

5.2. What a witness record contains

A record may contain: 

witnesses["A"] = {
    id          = "ms-A",
    siglum      = "A",
    kind        = "manuscript",
    description = "Paris, Bibliothèque nationale de France, grec 1807",
    order       = 1,
}

Conceptually: 

TEI witness
   │
   ├── xml:id ─────────â–º id
   ├── display value ──â–º siglum
   ├── @type ──────────â–º kind
   ├── text ───────────â–º description
   └── position ───────â–º order

The exact fields depend on the needs of the edition.

5.3. Registry lookup

Once the registry exists: 

"B"
 │
 â–¼
witnesses.by_id["ms-B"]
 │
 +───────────────+
 │               │
 â–¼               â–¼
found           absent
 │               │
 â–¼               â–¼
witness record  unresolved reference

Later apparatus processing can therefore check references without repeatedly searching the original <listWit>.

5.4. Functional MWE: Building a witness registry from TEI XML

The following example reads witness declarations from a TEI fragment, builds a Lua registry, records errors, and prints a diagnostic report.

It does not yet process apparatus entries. Its sole purpose is to establish the collection of witnesses against which later references will be resolved.

The central result of the example is not the table printed on the page. The important result is the Lua registry itself: 

witnesses.by_id
witnesses.in_order
witnesses.errors

These three fields represent different kinds of knowledge about the TEI document: 

 directly accessible;

 declared;

The source XML contains witness declarations. The Lua registry turns those declarations into a collection of identifiable, ordered, and verifiable editorial objects.

5.5. Testing missing and duplicate identifiers

The validation behaviour can be tested by adding the following declarations to the TEI buffer: 

<witness type="manuscript">
  An unidentified manuscript
</witness>

<witness xml:id="ms-A" type="manuscript">
  A second declaration using the identifier A
</witness>

The first declaration cannot be referenced because it has no xml:id. The second makes the identifier A ambiguous because that identifier is already present in the registry.

The processing layer therefore reports both conditions and excludes the problematic declarations from the usable registry.

This is an important difference between merely traversing the XML tree and constructing an editorial representation. Traversal can find both elements. Registry construction must decide whether they can function as stable objects within the edition.

5.6. Result of Stage 1

At this point the source declarations have become a controlled registry: 

TEI witness declarations
        │
        â–¼
Lua witness registry
        ├── direct lookup
        ├── preserved order
        └── declaration errors

Stage reached.

Witnesses are no longer only XML elements. They are now identifiable, ordered, and verifiable editorial objects against which later references can be checked.

6. Stage 2 — Convert apparatus entries into Lua records

We now bring in the structure constructed in Guide 3: 

app
├── lem: λÏŒγος
│   └── @wit="#ms-A"
└── rdg: λέξις
    └── @wit="#ms-B #ed-C"

The next transformation is: 

TEI APP
   │
   â–¼
extract fields
   │
   â–¼
normalize witness lists
   │
   â–¼
classify editorial roles
   │
   â–¼
LUA APPARATUS RECORD

6.1. From XML hierarchy to explicit fields

For: 

<app xml:id="app-001">
  <lem wit="#ms-A">λÏŒγος</lem>
  <rdg wit="#ms-B #ed-C">λέξις</rdg>
</app>

we want: 

record
├── id: app-001
├── lemma
│   ├── text: λÏŒγος
│   └── witnesses: { A }
│
└── readings
    └── reading 1
        ├── text: λέξις
        ├── witnesses: { B, C }
        └── kind: reading

This is not yet a typeset apparatus entry. It is an editorial object with predictable fields.

6.2. Normalize witness lists

The path is: 

@wit="#ms-B #ed-C"
      │
      â–¼
raw string
      │
      â–¼
remove # markers
      │
      â–¼
split values
      │
      â–¼
remove accidental duplicates
      │
      â–¼
{ "ms-B", "ed-C" }

An irregular source such as: 

wit="  #ms-B   #ed-C  #ms-B "

can therefore produce the same normalized list while still allowing a warning to record that B was duplicated in the original value.

6.3. Preserve editorial categories

An empty value alone is not enough to explain its meaning.

Compare: 

{
    kind = "reading",
    text = "λέξις",
}

with: 

{
    kind = "omission",
    text = "",
}

The relation is: 

TEI distinction
       │
       â–¼
explicit Lua category
       │
       â–¼
later validation + typography

Do not flatten significant distinctions.

An omission, missing data, unreadable text, and a processing failure should not all become the same empty string merely because their visible text is empty.

6.4. Functional MWE: Converting TEI apparatus entries into Lua records

The following example extends the witness-registry mechanism developed in section 4.

It reads two apparatus entries, normalizes their witness lists, preserves the order of their readings, and constructs Lua records.

The example displays the resulting records for inspection. It does not yet validate whether every witness reference is known.


The first record contains: 

 ms-A (display label A);

 ms-B and ed-C (display labels
 B and C).

The second record contains: 

 ms-A and ms-B;

 ed-C.

The empty text of the omission is therefore not treated as an ordinary empty reading. Guide 4 classified the apparatus entry with type="omission"; the Lua conversion carries that project classification into the reading record: 

kind = "omission"

This is an example of contextual interpretation: an empty <rdg/> acquires its editorial meaning from the apparatus classification rather than from emptiness alone.

6.5. Result of Stage 2

At the end of this stage, each TEI <app> element has become a structured Lua record with:

The transformation has therefore moved beyond XML traversal.

Lua has identified the editorial roles of the elements, separated their contents into explicit fields, and constructed apparatus objects that can be examined independently of the original XML syntax.

The witness identifiers contained in these records have been normalized, but they have not yet been verified against the witness registry.

That verification is the purpose of the next section.

Stage reached.

Each <app> now has a predictable Lua representation. Its witness identifiers are normalized, but they have not yet been proved to denote declared witnesses.

7. Stage 3 — Resolve references and check project rules

The two structures built so far can now be joined.

7.1. Join the witness registry and apparatus records 

WITNESS REGISTRY                    APPARATUS RECORD

by_id["ms-A"] â—„──────────────────────── A
by_id["ms-B"] â—„──────────────────────── B
by_id["ed-C"] â—„──────────────────────── C

by_id["ms-D"] ? â—„────────────────────── D
                unresolved

For each normalized identifier: 

identifier
    │
    â–¼
registry lookup
    │
 +--+--+
 │     │
 â–¼     â–¼
found  absent
 │     │
 â–¼     â–¼
resolved ERROR
object

This is the point where a character string becomes a relation to a known editorial object.

7.2. Check the project apparatus profile

Reference resolution is only one part of validation.

A project may also require additional rules beyond TEI schema conformance. The small profile used by this tutorial asks, for example: 

apparatus record
      │
      ├── lemma exists?
      ├── readings exist?
      ├── ordinary readings contain text?
      ├── empty readings classified explicitly?
      ├── witnesses declared?
      └── duplicates reported?

A record may therefore be perfectly readable as XML while being unusable as an editorial record.

7.3. Errors and warnings have different meanings 

IRREGULARITY
     │
     â–¼
can processing establish
safe editorial meaning?
     │
  +--+--+
  │     │
 yes    no
  │     │
  â–¼     â–¼
WARNING ERROR
  │     │
  â–¼     â–¼
usable invalid
record record

Typical errors include:

Typical warnings may include:

Project checks change the status of the data.

A reference is no longer merely present: it is resolved or unresolved. An apparatus entry is no longer merely readable: it is usable, usable with warnings, or invalid.

7.4. Functional MWE: Resolving references and checking apparatus records

The following example combines a witness registry with several apparatus records.

It deliberately includes:

The output separates errors from warnings and marks each apparatus record as valid or invalid.


The expected validation summary marks: 

 been declared;

 as an omission;

 B was repeated in the original @wit value.

The repeated reference in app-005 is removed during normalization. The warning nevertheless preserves knowledge of the irregularity found in the source.

7.5. Result of Stage 3

At the end of this stage, the processing layer knows more than the XML parser alone can establish.

It knows:

Project-level checking therefore produces a new processing state.

The original TEI document contains encoded claims and relationships. The Lua layer tests those claims against the declarations and rules of the edition.

A reference is no longer merely syntactically present: it is either resolved or unresolved.

An apparatus entry is no longer merely well-formed: it is either editorially usable, usable with warnings, or invalid.

The next section will combine witness registration, apparatus conversion, reference resolution, and validation in a single complete workflow.

Stage reached.

The processing layer can now distinguish accepted editorial objects from records that require correction, while preserving diagnostic information about the latter.

8. Assemble the complete workflow

The three stages can now be combined while remaining conceptually distinct.

8.1. The cumulative processing map 

                         TEI SOURCE
                             │
             +---------------+---------------+
             │                               │
             â–¼                               â–¼
     witness declarations              apparatus entries
             │                               │
             â–¼                               â–¼
       Lua registry                  Lua apparatus records
             │                               │
             │                         normalization
             │                               │
             +-------------+-----------------+
                           │
                           â–¼
                 reference resolution
                           │
                           â–¼
                      validation
                           │
                  +--------+--------+
                  │                 │
                  â–¼                 â–¼
              accepted           invalid
               records            records
                  │                 │
                  │                 └── diagnostics
                  â–¼
             ConTeXt input
                  │
                  â–¼
            TYPOGRAPHY

The important collections are: 

processing.records
    │
    └── every constructed apparatus record
        including invalid records retained for inspection

processing.accepted
    │
    └── records that have passed validation
        and may be passed safely to typesetting

This prevents bad data from silently reaching the final apparatus without throwing away the information needed to diagnose it.

8.2. Preserve normalized identifiers and resolved objects

A useful record can retain both levels: 

reading
├── witness_ids
│   └── { "ms-B", "ed-C" }
│
└── witnesses
    ├── witness record B
    └── witness record C

Thus: 

ENCODED REFERENCE
      │
      â–¼
NORMALIZED IDENTIFIER
      │
      â–¼
RESOLVED OBJECT

The intermediate representation remains traceable to the source while also being convenient for later processing.

8.3. Diagnostic reports preserve failures

The processing system should expose both success and failure: 

             ALL APPARATUS RECORDS
                     │
                  validate
                     │
          +----------+----------+
          │                     │
          â–¼                     â–¼
       accepted               invalid
          │                     │
          â–¼                     â–¼
processing.accepted      retained in records
          │                     │
          â–¼                     └── diagnostics
       ConTeXt

Preserve both success and failure.

Accepted records protect the typesetting stage. Rejected records remain traceable so that the editor can inspect and correct the source or the processing rules.

8.4. Complete functional MWE: From TEI apparatus data to checked Lua records

The following MWE performs the complete processing sequence developed in this guide.

The expected summary contains:

The records app-001 and app-002 are accepted.

In app-001, the normalized identifiers ms-B and ed-C are resolved to witness records whose reader-facing labels are B and C.

In app-002, the empty reading is accepted because its kind is explicitly recorded as omission.

The record app-003 is rejected because the reference ms-D cannot be resolved to a declared witness. Its normalized identifiers remain visible, and the successfully resolved witness B is retained, but the record is not included in: 

processing.accepted

8.5. Records prepared for ConTeXt

The collection: 

processing.records

contains every apparatus record constructed from the TEI source, including invalid records retained for inspection.

The collection: 

processing.accepted

contains only records that have passed validation and may safely be transmitted to the typesetting layer.

This distinction prevents invalid data from silently reaching the final apparatus while preserving enough information for diagnosis and correction.

A record prepared for ConTeXt now possesses:

ConTeXt will therefore not need to rediscover the meaning of @wit, search the TEI witness declarations, distinguish omissions from missing data, or decide whether unresolved references should be printed.

The next guide can begin from records whose editorial status has already been made explicit.

8.6. Result of the complete workflow

The complete workflow does not merely transfer information from one notation to another.

It successively changes the epistemic status of the data:

Stage Status of the data
TEI encoding Witnesses and variants are documented through elements, attributes, identifiers, and references.
Extraction The relevant XML structures become accessible to Lua.
Normalization Notational variations are reduced to controlled internal forms.
Reference resolution Character strings become relations to known editorial objects.
Validation Encoded claims are tested against declarations and editorial rules.
Intermediate representation The accepted records form an explicit and reusable body of editorial knowledge.

The final Lua records are therefore not neutral copies of the XML source.

They are the result of documented interpretative operations whose assumptions, successes, and failures can be inspected.

This intermediate representation is what makes a reliable separation between scholarly data processing and typographical composition possible.

Stage reached. the data has not yet been typeset, but its editorial status has been made explicit. This is the stable point at which Guide 6 can take over.

Part III — Define the boundaries

9. What Lua should not decide

The processing layer now performs substantial interpretative work. That does not mean that every decision belongs in Lua.

The three layers should remain distinguishable. This repetition of the TEI → Lua → ConTeXt architecture is intentional: here the diagram no longer explains what each layer can do, but what each layer should not silently take over from the others. 

TEI
│
├── evidence
├── documentary distinctions
└── scholarly assertions
        │
        â–¼
Lua
│
├── extract
├── normalize
├── resolve
├── validate
└── diagnose
        │
        â–¼
ConTeXt
│
├── typography
├── punctuation
├── spacing
├── placement
└── page architecture

9.1. Lua should not silently repair the TEI source

If the source contains: 

<rdg wit="#ms-D">πνοή</rdg>

but no witness D has been declared, Lua may report: 

app-003: unknown witness D in reading 1.

It should not silently invent witness D.

That would turn programmatic invention into apparently documented evidence.

Do not repair silently.

A controlled default may be legitimate when the project explicitly authorizes it, but the intervention should remain visible through a warning, generated value, or another diagnostic field.

9.2. Lua should not erase significant TEI distinctions

These should not all become: 

text = ""

without further qualification: 

omission
lacuna
unreadable text
absent data
editorial suppression
processing failure

A record should retain the relevant category: 

{
    kind = "omission",
    text = "",
}

or, where appropriate: 

{
    kind = "lacuna",
    text = "",
}

The reduction from TEI to Lua should therefore be selective rather than destructive.

9.3. Lua should not determine typography prematurely

A record may state: 

kind = "omission"

It should not normally need to state: 

print the abbreviation "om."
use an italic font
insert exactly 0.5 em space

Those are ConTeXt decisions.

The relation should remain: 

EDITORIAL CATEGORY
kind = "omission"
       │
       â–¼
ConTeXt rendering policy
       │
       ├── om.
       ├── omitted
       ├── ∅
       └── another documented convention

Keeping these decisions separate allows typography to change without reconstructing the editorial records.

9.4. Lua should not replace editorial judgment

Lua can test whether:

It cannot determine every scholarly question, for example:

Project checks make assumptions explicit; they do not replace scholarship.

Lua can test the formal consequences of editorial rules. The rules themselves still depend on scholarly judgment.

10. Prepare the Lua–ConTeXt contract

The checked Lua records now form a stable boundary between interpretation and composition.

10.1. The Lua–ConTeXt contract

A record passed to ConTeXt may now contain: 

record
├── id
├── kind
├── lemma
│   ├── text
│   ├── witness_ids
│   └── resolved witnesses
│
├── readings
│   ├── text
│   ├── kind
│   ├── witness_ids
│   └── resolved witnesses
│
├── responsibility   (when processed)
├── certainty          (when processed)
├── documentary        (when processed)
├── order
├── source
├── valid
└── diagnostics

ConTeXt therefore does not need to:

It receives records whose editorial status is already explicit.

10.2. What ConTeXt will decide

Guide 6 can now concentrate on questions such as:

The transition is: 

GUIDES 1–4
BUILD TEI
      │
      â–¼
GUIDE 5
INTERPRET + VALIDATE
      │
      â–¼
checked Lua records
      │
      â–¼
GUIDE 6
COMPOSE

10.3. Preparing complex scholarly pages

The same intermediate model can support richer passage records containing, for example: 

{
    passage_id = "passage-001",

    texts = {
        greek       = { ... },
        latin       = { ... },
        translation = { ... },
    },

    apparatus = {
        textual     = { ... },
        sources     = { ... },
        translation = { ... },
    },

    bibliography = { ... },
    alignment    = { ... },

    validation = {
        valid    = true,
        errors   = {},
        warnings = {},
    },
}

The point is not to reproduce the final page as a giant typesetting macro. The processing layer prepares structured editorial objects from which ConTeXt can construct several coordinated typographical regions.

11. What you have built

Guide 5 has introduced a conceptual shift.

The first four guides primarily built encoded scholarly structures. This guide has shown how those structures become controlled editorial objects.

11.1. The cumulative transformation 

TEI ENCODING
witnesses + apparatus
        │
        â–¼
EXTRACTION
accessible XML structures
        │
        â–¼
NORMALIZATION
controlled identifiers and values
        │
        â–¼
REFERENCE RESOLUTION
strings become known objects
        │
        â–¼
VALIDATION
claims tested against rules
        │
        â–¼
DIAGNOSTICS
successes and failures remain visible
        │
        â–¼
INTERMEDIATE REPRESENTATION
validated editorial knowledge
        │
        â–¼
ConTeXt

11.2. The status of the information changes at every stage

A witness value can move through: 

encoded string
"#ms-A #ms-B"
    │
    â–¼
normalized identifiers
{ "ms-A", "ms-B" }
    │
    â–¼
resolved objects
{ witness_A, witness_B }
    │
    â–¼
validated relation
    │
    â–¼
editorial knowledge
ready for typesetting

Normalization and validation are therefore not merely technical cleanup. They are documented interpretative operations.

11.3. Diagnostics preserve the limits of interpretation

An invalid record should not disappear without explanation.

The processing layer can preserve: 

invalid record
├── normalized fields
├── successfully resolved references
├── unresolved references
├── source connection
└── diagnostic messages

Thus failure remains inspectable and correctable.

11.4. The complete six-guide architecture is now visible 

GUIDE 1
TEI structure
    │
    â–¼
GUIDE 2
witness identities
    │
    â–¼
GUIDE 3
basic apparatus relations
    │
    â–¼
GUIDE 4
complex textual relations
    │
    â–¼
GUIDE 5
normalized + resolved + checked records
    │
    â–¼
GUIDE 6
page composition

Guide 5 complete.

The data has not yet received its final typography, but its editorial status is explicit. This is the stable point at which Guide 6 can take over.

11.5. Continue with Guide 6

The central question is no longer: 

What does this TEI data mean?

Guide 5 has established a controlled answer in the intermediate records.

The next question is: 

How should this validated editorial knowledge
be composed on the page?

Guide 6 shows how ConTeXt can turn those records into a scholarly page, including:

Next: Guide 6 — Typesetting a TEI critical apparatus with ConTeXt

Guide 5 of 6 — Processing TEI critical apparatus data with Lua

Previous: Guide 4 — Encoding complex textual variation in TEI  ·  Collection overview  ·  Glossary  ·  Next: Guide 6 — Typesetting a TEI critical apparatus with ConTeXt

Related pages