Lexical grammar

From Wikipedia, the free encyclopedia
Jump to navigation Jump to search

In computer science, a lexical grammar is a formal grammar defining the syntax of tokens. The program is written using characters that are defined by the lexical structure of the language used. The character set is equivalent to the alphabet used by any written language. The lexical grammar lays down the rules governing how a character sequence is divided up into subsequences of characters, each part of which represents an individual token. This is frequently defined in terms of regular expressions.[1]

For instance, the lexical grammar for many programming languages specifies that a string literal starts with a " character and continues until a matching " is found (escaping makes this more complicated), that an identifier is an alphanumeric sequence (letters and digits, usually also allowing underscores, and disallowing initial digits), and that an integer literal is a sequence of digits. So in the following character sequence "abc" xyz1 23 the tokens are string, identifier and number (plus whitespace tokens) because the space character terminates the sequence of characters forming the identifier. Further, certain sequences are categorized as keywords – these generally have the same form as identifiers (usually alphabetical words), but are categorized separately; formally they have a different token type.[2]

Examples

Regular expressions for common lexical rules follow (for example, C).

Unescaped string literal (quote, followed by non-quotes, ending in a quote):

"[^"]*"

Escaped string literal (quote, followed by escaped characters or non-quotes, ending in a quote):

"(\.|[^\"])*"

Integer literal:

[0-9]+

Decimal integer literal (no leading zero):

[1-9][0-9]*|0

Hexadecimal integer literal:

0[Xx][0-9A-Fa-f]+

Octal integer literal:

0[0-7]+

Identifier:

[A-Za-z_][A-Za-z0-9_]*

See also

References

  1. ^ Buyya (2009). Object-oriented Programming with Java: Essentials and Applications. Tata McGraw-Hill Education. pp. 57–. ISBN 978-0-07-066908-6. 
  2. ^ James Gosling (2000). The Java Language Specification. Addison-Wesley Professional. pp. 9–. ISBN 978-0-201-31008-5. 

External links

  • ANSI C grammar, Lex specification
Retrieved from "https://en.wikipedia.org/w/index.php?title=Lexical_grammar&oldid=819601051"
This content was retrieved from Wikipedia : http://en.wikipedia.org/wiki/Lexical_grammar
This page is based on the copyrighted Wikipedia article "Lexical grammar"; it is used under the Creative Commons Attribution-ShareAlike 3.0 Unported License (CC-BY-SA). You may redistribute it, verbatim or modified, providing that you comply with the terms of the CC-BY-SA