In C/C++ tokens are the smallest element of a program. Tokens can be split into the following
1. Keywords
Predefined tokens in a programming language having fixed meaning.
e.g:- switch, case, int,return etc.
C++ has 31 additional keywords
few of the common ones used are
bool, class,static_cast,try, catch etc
2. Identifiers
Identifiers are used to name variables, labels etc. There are certain rules for the identifiers
a. They can start only with a letter or underscore(_).
b. They must contain only digits, letters and underscore.
c. They cannot contain white space.
d. maximum up to 31 characters long.
e. white space not allowed.
3. Constants
Constants refer to fixed values. e.g:- integer constants, floating point constants, character constants, octal or hexadecimal constants and string constants.
E.g:-
char *p=”Yogi”;
int age=43;
In the above examples “Yogi” and 43 are string and integer constants.
4. Strings
They are array of characters ended with null character(\0)
eg:- char test[20]=”Yogi”;
5. Symbols
Symbols have special meaning.
a. Brackets[]- used for array subscripts.
b. Paranthesis() – used for function calls and parameter passing.
c. Braces{} – Used for separating blocks of code. A variable defined inside the brace will have the scope within the brace.
d. comma, – separates statements and parameters.
e. asterisk * – Use to dereference a pointer.
There are other set of symbols also used and each has its own meaning.
6. Operators
Operators are symbols that are used to perform with operands to create a statement.
Based on the number of operands operator can act upon, operators can be classified as
a. Unary operator – i++, i–;
b. Binary operators – Arithmetic, relational, logical, assignment, conditional, bitwise
c. Ternary operator – ?:
Source: C/C++ Tokens – GeeksforGeeks