class CodeRay::Tokens

The Tokens class represents a list of tokens returned from a Scanner. It’s actually just an Array with a few helper methods.

A token itself is not a special object, just two elements in an Array:

It looks like this:

..., '# It looks like this', :comment, ...
..., '3.1415926', :float, ...
..., '$^', :error, ...

Some scanners also yield sub-tokens, represented by special token actions, for example :begin_group and :end_group.

The Ruby scanner, for example, splits “a string” into:

[
 :begin_group, :string,
 '"',          :delimiter,
 'a string',   :content,
 '"',          :delimiter,
 :end_group,   :string
]

Tokens can be used to save the output of a Scanners in a simple Ruby object that can be send to an Encoder later:

tokens = CodeRay.scan('price = 2.59', :ruby).tokens
tokens.encode(:html)
tokens.html
CodeRay.encoder(:html).encode_tokens(tokens)

Tokens gives you the power to handle pre-scanned code very easily: You can serialize it to a JSON string and store it in a database, pass it around to encode it more than once, send it to other algorithms…