A TokenStream is a list of tokens, gathered during the parse of some entity (say a method). Entities populate these streams by being registered with the lexer. Any class can collect tokens by including TokenStream. From the outside, you use such an object by calling the #start_collecting_tokens method, followed by calls to #add_token and pop_token.
# File rdoc/tokenstream.rb, line 20 def add_token(tk) @token_stream << tk end
# File rdoc/tokenstream.rb, line 24 def add_tokens(tks) tks.each {|tk| add_token(tk)} end
# File rdoc/tokenstream.rb, line 28 def pop_token @token_stream.pop end