Previous Table of Contents "New C Standard" commentary
token: keyword identifier constant string-literal punctuatoreach non-white-space character that cannot be one of the above
preprocessing-token: header-name identifier pp-number character-constant string-literal punctuator
771 Each preprocessing token that is converted to a token shall have the lexical form of a keyword, an identifier, a constant, a string literal, or a punctuator.
773 The categories of tokens are: keywords, identifiers, constants, string literals, and punctuators.
774 A preprocessing token is the minimal lexical element of the language in translation phases 3 through 6.
775 The categories of preprocessing tokens are: header names, identifiers, preprocessing numbers, character constants, string literals, punctuators, and single non-white-space characters that do not lexically match the other preprocessing token categories.58)
777 Preprocessing tokens can be separated by white space;
778 this consists of comments (described later), or white-space characters (space, horizontal tab, new-line, vertical tab, and form-feed), or both.
779 As described in 6.10, in certain circumstances during translation phase 4, white space (or the absence thereof) serves as more than preprocessing token separation.
780 White space may appear within a preprocessing token only as part of a header name or between the quotation characters in a character constant or string literal.
781 58) An additional category, placemarkers, is used internally in translation phase 4 (see 126.96.36.199); it cannot occur in source files.
782 If the input stream has been parsed into preprocessing tokens up to a given character, the next preprocessing token is the longest sequence of characters that could constitute a preprocessing token.
There is one exception to this rule:
The program fragment
786 EXAMPLE 2
The program fragment
787 Forward references: character constants (188.8.131.52), comments (6.4.9), expressions (6.5), floating constants (184.108.40.206), header names (6.4.7), macro replacement (6.10.3), postfix increment and decrement operators (220.127.116.11), prefix increment and decrement operators (18.104.22.168), preprocessing directives (6.10), preprocessing numbers (6.4.8), string literals (6.4.5).
Created at: 2008-01-30 02:39:41
The text from WG14/N1256 is copyright © ISO
Created at: 2008-01-30 02:39:41 The text from WG14/N1256 is copyright © ISO