Locate the next token from the provided buffer, and map a
buffer reference into token. Returns true if a token was
located, false otherwise.
Note that the buffer content is not duplicated. Instead, a
slice of the buffer is referenced by the token. You can use
Token.clone() or Token.toString().dup() to copy content per
your application needs.
Note also that there may still be one token left in a buffer
that was not terminated correctly (as in eof conditions). In
such cases, tokens are mapped onto remaining content and the
buffer will have no more readable content.
Locate the next token from the provided buffer, and map a buffer reference into token. Returns true if a token was located, false otherwise.
Note that the buffer content is not duplicated. Instead, a slice of the buffer is referenced by the token. You can use Token.clone() or Token.toString().dup() to copy content per your application needs.
Note also that there may still be one token left in a buffer that was not terminated correctly (as in eof conditions). In such cases, tokens are mapped onto remaining content and the buffer will have no more readable content.