LuaTeX: Mimicking File System Input

In LaTeX, verbatim environments are extremely tricky. Different verbatim environments are based on distinct LaTeX magic, which makes their behavior inconsistent. The only realiable way to use a verbatim environment is to write them in the TeX source as-is, for any attempt to construct such environments programmatically usually fails. In existing packages, such problems are avoided by saving the constructed environments into files first and then use \input command to read them as TeX source files. I really dislike this solution as it induces significant I/O overhead (despite the fact that other I/O bottleneck may be more dominant). In this post, I provide a LuaTeX-based method that allows \input from Lua strings.

(This post corresponds to my question on TeX.SE.)

Using LuaTeX with verbatim

At the beginning of my investigation, I found this post, which suggestes constructing verbatim environments with LuaTeX.

\directlua{tex.print([[\unexpanded{\begin{verbatim}]]..'one\rtwo'..[[\end{verbatim}}]])}

Unfortunately, this only works with verbatim environment. When I try this on minted, which is based on fancyvrb, errors will occur.

% this won't work
\directlua{tex.print([[\unexpanded{\begin{minted}{python}]]..'one\rtwo'..[[\end{minted}}]])}

This just shows that LuaTeX’s tex.print is still subject to LaTeX’s standard parsing routine. We need a way to decouple our constructed verbatim environments from this routine. The most common solution is to use an external file. But I dislike this idea because disks are slow (maybe not anymore recently…). The file system approach uses file as an outlet to escape from TeX’s parsing routine. Since LuaTeX introduces Lua as an scripting engine apart from TeX, I think there must be a way to use Lua as the outlet.

The approach

According to LuaTeX reference, we are able to customize find_read_file and open_read_file callbacks. When we call \input{filename}, the find_read_file callback is first invoked to search for filename (potentially in working directory and $PATH). If filename is found, find_read_file callback returns the actual filename of the file; otherwise, it returns nil. If filename is found, then open_read_file callback is invoked to read the file. This callback returns a Lua table object, which contains a reader key. Here, reader is a function that returns a line from the file for each invocation. When EOF is reached, reader returns nil. Therefore, we can establish the following procedure to mimic file system input with a Lua string.

1. Construct and save contents in a Lua string.
2. Define a TeX macro as follows:

    \newcommand{\TFBInputAsFile}{
    % register custom callbacks
    \directlua{tex_file_buffer:register_callback()}
    % input some random filename, which will be processed by our own callbacks
    \input randomfile
    }
   

3. In the reader function, return the Lua string and immediately remove our own callback so that it doesn’t affect the read of other files.

The source code can be seen in the source code section.

Example usage

\documentclass{article}
\usepackage[T1]{fontenc}
\usepackage{minted}
\usepackage{tex-file-buffer} % import source code
\usepackage{datetime2}
\usepackage{expl3}
\usepackage{minted}

\begin{document}


% construct buffer

\ExplSyntaxOn

\str_clear:N \l_tmpa_str
\exp_args:Nx \TFBAppend {\c_backslash_str begin{minted}{python}} \TFBAppendCR
\exp_args:Nx \TFBAppend {print("abc")} \TFBAppendCR
\exp_args:Nx \TFBAppend {\c_backslash_str end{minted}}

\ExplSyntaxOff

\TFBUse % show buffer as plain text
\TFBInputAsFile % read buffer as file
\TFBClear % clear buffer


\par\DTMNow

\end{document} 

Output

Source code