Separate the parse, symbol, and token documentation into separate files.

Doc/lib/libtoken.tex

@@ -0,0 +1,38 @@
+\section{Standard Module \sectcode{token}}
+\label{module-token}
+\stmodindex{token}
+
+This module provides constants which represent the numeric values of
+leaf nodes of the parse tree (terminal tokens).  Refer to the file
+\file{Grammar/Grammar} in the Python distribution for the defintions
+of the names in the context of the language grammar.  The specific
+numeric values which the names map to may change between Python
+versions.
+
+This module also provides one data object and some functions.  The
+functions mirror definitions in the Python C header files.
+
+\setindexsubitem{(in module token)}
+
+
+\begin{datadesc}{tok_name}
+Dictionary mapping the numeric values of the constants defined in this
+module back to name strings, allowing more human-readable
+representation of parse trees to be generated.
+\end{datadesc}
+
+\begin{funcdesc}{ISTERMINAL}{x}
+Return true for terminal token values.
+\end{funcdesc}
+
+\begin{funcdesc}{ISNONTERMINAL}{x}
+Return true for non-terminal token values.
+\end{funcdesc}
+
+\begin{funcdesc}{ISEOF}{x}
+Return true if \var{x} is the marker indicating the end of input.
+\end{funcdesc}
+
+\begin{seealso}
+\seemodule{parser}{second example uses this module}
+\end{seealso}