Chapter 2: Tokens

What's in Chapter 2?

This chapter defines the basic building blocks of a C program. Understanding the concepts in this chapter will help eliminate the syntax bugs that confuse even the veteran C programmer. A simple syntax error can generate 100's of obscure compiler errors. In this chapter we will introduce some of the syntax of the language.

To understand the syntax of a C program, we divide it into tokens separated by white spaces and punctuation. Remember the white spaces include space, tab, carriage returns and line feeds. A token may be a single character or a sequence of characters that form a single item. The first step of a compiler is to process the program into a list of tokens and punctuation marks. The following example includes punctuation marks of ( ) { } ; The compiler then checks for proper syntax. And, finally, it creates object code that performs the intended operations. In the following example:

The following sequence shows the tokens and punctuation marks from the above listing:

void main ( void ) { long z ; z = 0 ; while ( 1 ) { z = z + 1 ; } }

Since tokens are the building blocks of programs, we begin our study of C language by defining its tokens.

ASCII Character Set

Like most programming languages C uses the standard ASCII character set. The following table shows the 128 standard ASCII code. One or more white space can be used to separate tokens and or punctuation marks. The white space characters in C include horizontal tab (9=0x09), the carriage return (13=0x0D), the line feed (10=0x0A), space (32=0x20).

		0	1	2	3	4	5	6	7
	0	NUL	DLE	SP	0	@	P	`	p
B	1	SOH	DC1	!	1	A	Q	a	q
I	2	STX	DC2	"	2	B	R	b	r
T	3	ETX	DC3	#	3	C	S	c	s
S	4	EOT	DC4	$	4	D	T	d	t
	5	ENQ	NAK	%	5	E	U	e	u
0	6	ACK	SYN	&	6	F	V	f	v
	7	BEL	ETB	'	7	G	W	g	w
T	8	BS	CAN	(	8	H	X	h	x
O	9	HT	EM	)	9	I	Y	i	y
	A	LF	SUB	*	:	J	Z	j	z
3	B	VT	ESC	+	;	K	[	k	{
	C	FF	FS	,	<	L	\	l	|
	D	CR	GS	-	=	M	]	m	}
	E	SO	RS	.	>	N	^	n	~
	F	S1	US	/	?	O	_	o	DEL

The first 32 (values 0 to 31 or 0x00 to 0x1F) and the last one (127=0x7F) are classified as control characters. Codes 32 to 126 (or 0x20 to 0x7E) include the "normal" characters. Normal characters are divided into

Literals

Numeric literals consist of an uninterrupted sequence of digits delimited by white spaces or special characters (operators or punctuation). Although Metrowerks does support floating point, this document will not cover it. The use of floating point requires a substantial about of program memory and execution time, therefore most applications should be implemented using integer math. Consequently the period will not appear in numbers as described in this document. For more information about numbers see the sections on decimals, octals, or hexadecimals in Chapter 3.

Character literals are written by enclosing an ASCII character in apostrophes (single quotes). We would write 'a' for a character with the ASCII value of the lowercase a (97). The control characters can also be defined as constants. For example '\t' is the tab character. For more information about character literals see the section on characters in Chapter 3.

String literals are written as a sequence of ASCII characters bounded by quotation marks (double quotes). Thus, "ABC" describes a string of characters containing the first three letters of the alphabet in uppercase. For more information about string literals see the section on strings in Chapter 3.

Keywords

There are some predefined tokens, called keywords, that have specific meaning in C programs. The reserved words we will cover in this document are:

keyword	meaning
auto	Specifies a variable as automatic (created on the stack)
break	Causes the program control structure to finish
case	One possibility within a switch statement
char	8-bit integer
const	Defines global parameter as constant in ROM, and defines a local parameter as fixed value
continue	Causes the program to go to beginning of loop
default	Used in switch statement for all other cases
do	Used for creating program loops
double	Specifies variable as double precision floating point
else	Alternative part of a conditional
extern	Defined in another module
float	Specifies variable as single precision floating point
for	Used for creating program loops
goto	Causes program to jump to specified location
if	Conditional control structure
int	32-bit integer (same as long on the ARM) It should be avoided in most cases because the implementation will vary from compiler to compiler.
long	32-bit integer
register	Specifies how to implement a local
return	Leave function
short	16-bit integer
signed	Specifies variable as signed (default)
sizeof	Built-in function returns the size of an object
static	Stored permanently in memory, accessed locally
struct	Used for creating data structures
switch	Complex conditional control structure
typedef	Used to create new data types
unsigned	Always greater than or equal to zero
void	Used in parameter list to mean no parameter
volatile	Can change implicitly outside the direct action of the software. It disables compiler optimization, forcing the compiler to fetch a new value each time
while	Used for creating program loops

Did you notice that all of the keywords in C are lowercase? Notice also that as a matter of style, I used a mixture of upper and lowercase for the names I created, and all uppercase for the I/O ports. It is a good programming practice not to use these keywords for your variable or function names.

Names

We use names to identify our variables, functions, and macros.  Names must begin with a letter or underscore and the remaining characters must be either letters or digits. We can use a mixture of upper and lower case or the underscore character to create self-explaining symbols. E.g.,

time_of_day    go_left_then_stop

TimeOfDay      GoLeftThenStop

The careful selection of names goes a long way to making our programs more readable. Names may be written with both upper and lowercase letters. The names are case sensitive. Therefore the following names are different:

The practice of naming macros in uppercase calls attention to the fact that they are not variable names but defined symbols. Remember the I/O port names are implemented as macros in the header file tm4c123gh6pm.h.

We can use the map file to get the absolute addresses for these labels, then use the debugger to observe and modify their contents.

Developing a naming convention will avoid confusion. Possible ideas to consider include:

1. Start every variable name with its type. E.g.,

2. Start every local variable with "the" or "my"

3. Start every global variable and function with associated file or module name. In the following example the names all begin with Bit_. Notice how similar this naming convention recreates the look and feel of the modularity achieved by classes in C++. E.g.,

Punctuation

Punctuation marks (semicolons, colons, commas, apostrophes, quotation marks, braces, brackets, and parentheses) are very important in C. It is one of the most frequent sources of errors for both the beginning and experienced programmers.

Semicolons

Semicolons are used as statement terminators. Strange and confusing syntax errors may be generated when you forget a semicolon, so this is one of the first things to check when trying to remove syntax errors. In this example we assume that Port B has been initialized as an output. Notice that one semicolon is placed at the end of every simple statement in the following example

Preprocessor directives do not end with a semicolon since they are not actually part of the C language proper. Preprocessor directives begin in the first column with the #and conclude at the end of the line. The following example will fill the array DataBuffer with data read from the input port (PORTB). We assume in this example that Port B has been initialized as an input. Semicolons are also used in the for loop statement (see also Chapter 6), as illustrated by

Colons

We can define a label using the colon. Although C has a goto statement, I strongly discourage its use. I believe the software is easier to understand using the block-structured control statements (if, if else, for, while, do while, and switch case.) The following example will return after the Port B input reads the same value 100 times in a row. In this example, we assume Port B has been initialized as an input. Notice that every time the current value on Port B is different from the previous value the counter is reinitialized.

Colons also terminate case, and default prefixes that appear in switch statements. For more information see the section on switch in Chapter 6. In the following example, the next stepper motor output is found (the proper sequence is 10,9,5,6). The default case is used to restart the pattern.

For both applications of the colon (goto and switch), we see that a label is created that is a potential target for a transfer of control.

Commas

Commas separate items that appear in lists. We can create multiple variables of the same type. E.g.,

Lists are also used with functions having multiple parameters (both when the function is defined and called):

Listing 2-6: Commas separate the parameters of a function

Lists can also be used in general expressions. Sometimes it adds clarity to a program if related variables are modified at the same place. The value of a list of expressions is always the value of the last expression in the list. In the following example, first thetime is incremented, thedate is decremented, then x is set to k+2.

Apostrophes

Apostrophes are used to specify character literals. For more information about character literals see the section on characters in Chapter 3. Assuming the function OutChar will print a single ASCII character, the following example will print the lower case alphabet:

Quotation marks

Quotation marks are used to specify string literals. For more information about string literals see the section on strings in Chapter 3. Example

The command Letter='A'; places the ASCII code (65) into the variable Letter. The command pt="A"; creates an ASCII string and places a pointer to it into the variable pt.

Braces

Braces {} are used throughout C programs. The most common application is for creating a compound statement. Each open brace { must be matched with a closing brace }. One approach that helps to match up braces is to use indenting. Each time an open brace is used, the source code is tabbed over. In this way, it is easy to see at a glance the brace pairs. Examples of this approach to tabbing are the Bit_Put function within Listing 2-2 and the median function in Listing 1-4.

Brackets

Square brackets enclose array dimensions (in declarations) and subscripts (in expressions). Thus,

declares an integer array named Fifo consisting of 80 words numbered from 0 through 99, and

assigns the variable PutPt to the address of the first entry of the array.

Parentheses

Parentheses enclose argument lists that are associated with function declarations and calls. They are required even if there are no arguments.

As with all programming languages, C uses parentheses to control the order in which expressions are evaluated. Thus, (11+3)/2 yields 7, whereas 11+3/2 yields 12. Parentheses are very important when writing expressions.

Operators

The special characters used as expression operators are covered in the operator section in chapter 5. There are many operators, some of which are single characters

while others require two characters

and some even require three characters

The multiple-character operators can not have white spaces or comments between the characters.

The C syntax can be confusing to the beginning programmer. For example

Go to Chapter 3 on Literals Return to Table of Contents