strtok Function

Our polish postfix calculate function will be mindlessly simple. All it needs to do is push any values that the user inputs onto a stack and perform an arithmetic computation with the last two values whenever an arithmetic operator is encountered. The most difficult part in fact is separating the input string inString into its various parts.

One function which is used in C for separating strings into parts, or tokens as they are called, is the strtok function. Although it will not be particularly useful for our calculator, we describe it here, as it can be useful in certain other situations.

The function strtok is found in the C string library. It takes two arguments, both arrays of characters (i.e. arguments formally of type char *) which we will informally call strings, from now on. The first string is the string that we want to break up into tokens (in our case, that would be inString). The second string consists of a list of delimiters. These are characters which mark divisions between tokens.

When called, the function finds the first token in the string, marks the end of it with a \0 and returns a pointer to the start of that token. In other words, the return value is a string containing just the first token. Once we process that token and have finished with it, we can call strtok again. However this time we call it with a NULL pointer for the first string to indicate that we want the next token of the same original string. We proceed in this manner until strtok returns a NULL pointer, which indicates there are no more tokens left in the string to return.

If we were prepared to tell the user of our calculator to put a space between every part of the polish postfix expression they enter, i.e. between every number and arithmetic operator, etc, then we could use a space as delimiter. This would just involve us setting the second parameter of strtok to the string just consisting of a single space and a \n, i.e. the string " \n". Whilst this would make life easier for us, it makes the program less usable for the user, which is something we want to avoid.

If you do use strtok you have to be careful, since only one string can be tokenised at a time in C. It is sometimes convenient to first break a string up into substrings, then break each of these substrings up into tokens. Whilst it is tempting to try and use strtok for all of this, it does not work, since C cannot keep track of whether you want the next token from a substring, or the next token (i.e. substring) from the whole original string.

In addition, one has to be careful that the delimiters used don't contain useful information. In our case for example, we could not use the arithmetic operators (i.e. the string " +-*/\n") as a delimiter string, since when strtok is retrieving the next token, it would consider the arithmetc operators to be delimiters, not tokens, and thus it would never return them for us to process. It would merely skip over them.