Dynamic compilation of Forth words at runtime

One very powerful feature of stack-based programming is the compilation of words at runtime. This allows you to define the compiled state of a word dynamically.

To compile a word foo (the Forth community seems to have a fabel for foo and bar in examples) that will contain the even literals from 10 to 2 descending:

10 Value token

\ decrement the token value
: decr-token ( -- )
  token 1- to token
  ;

: compile-foo
  10 0 u+do
    token 2 mod 0= if \ if token is even
      \ compile token literal value into foo
      token postpone literal
    endif
    decr-token \ decrement token value
  loop
  ; compile-only immediate

Now, compile-foo is an ìmmediate word, meaning it will be executed when it is used in compile state, that is the colon definition (= what comes between : and ;) of a word. So when doing

: foo ( C: -- ; I: -- n1 n2 n3 n4 n5 )
  [ compile-foo ] ;

every word within compile-foo that is not preceded by postpone will be executed immediately. Checking correctness with see foo yields:

see foo
: foo  
  10 8 6 4 2 ; ok

Note that [ and ] will switch between compile state and interpret state. But this is in fact not even necessary:

: foo
  compile-foo ;

generated the same compiled content of foo which is easily checked with see foo.

For only the compile time behaviour of compile-foo is defined it is tagged compile-only. This will prevent any call of compile-foo in interprete state. It is not necessary to do this, yet useful, for the error message will clearly state that the call of compile-only is an error. Otherwise there would be whatever error which would be hard to find and debug in a larger program.

Executing more complicated control structures are possible as well as calling custom defined words. Every word preceded by postpone will be compiled into the word, everything else immediately executed. Note that in order to compile literals into a word it has to be followed by postpone literal as shown in compile-foo, even when it is the result of a word, a Constant or a Value.

Using the data stack in compile state

When compiling a word like this, all data is put onto the data stack while the Forth interpreter is in compile state. Therefore, all postponed words will leave a stack effect behind (compilation tokens and such - this basically is what postpone does: keep the following word from executing, but putting its compilation tokens on the data stack) which must not be tampered with. This is especially tricky when postponeing control structures like if (and therefore also endif) inside a loop. To ensure a consistent stack effect of loop control data throughout each loop iteration, the stack effect I produce manually has to be passed by each postponed word without modifying the stack-effect produced by postpone. Elsewise the endif will not match its if and a compile error will be the effect.

Problem is in general it is not clear what postpone will leave behind. Fortunately there is the return-stack which can be used in compile state (in fact it can only be used in compile state). It must only be guaranteed that the return-stack is empty when leaving compile state.

Additionally, there is this nice effect of a stack that it is a LIFO (last in first out) data structure. So when pushing (using >r) 5 values to the return-stack they will be on it in reversed order compared to when they were on the data stack. But when poping (using r>) them back onto the data stack, there will be the same stack effect as before the push operations.

Example:

: foo
  1 2 3 5 6
  >r >r >r >r >r
  ( )
  r> r> r> r> r>
  ( 1 2 3 4 5 )
  ;

Sweet.