RDoc::TokenStream

A TokenStream is a list of tokens, gathered during the parse of some entity (say a method). Entities populate these streams by being registered with the lexer. Any class can collect tokens by including TokenStream. From the outside, you use such an object by calling the #start_collecting_tokens method, followed by calls to #add_token and pop_token.

Public Class Methods

to_html(token_stream) click to toggle source

Converts token_stream to HTML wrapping various tokens with <span> elements. The following tokens types are wrapped in spans with the given class names:

TkCONSTANT: ‘ruby-constant’
TkKW: ‘ruby-keyword’
TkIVAR: ‘ruby-ivar’
TkOp: ‘ruby-operator’
TkId: ‘ruby-identifier’
TkNode: ‘ruby-node’
TkCOMMENT: ‘ruby-comment’
TkREGEXP: ‘ruby-regexp’
TkSTRING: ‘ruby-string’
TkVal: ‘ruby-value’

Other token types are not wrapped in spans.

 
               # File rdoc/token_stream.rb, line 28
def self.to_html token_stream
  token_stream.map do |t|
    next unless t

    style = case t
            when RDoc::RubyToken::TkCONSTANT then 'ruby-constant'
            when RDoc::RubyToken::TkKW       then 'ruby-keyword'
            when RDoc::RubyToken::TkIVAR     then 'ruby-ivar'
            when RDoc::RubyToken::TkOp       then 'ruby-operator'
            when RDoc::RubyToken::TkId       then 'ruby-identifier'
            when RDoc::RubyToken::TkNode     then 'ruby-node'
            when RDoc::RubyToken::TkCOMMENT  then 'ruby-comment'
            when RDoc::RubyToken::TkREGEXP   then 'ruby-regexp'
            when RDoc::RubyToken::TkSTRING   then 'ruby-string'
            when RDoc::RubyToken::TkVal      then 'ruby-value'
            end

    text = CGI.escapeHTML t.text

    if style then
      "<span class=\"#{style}\">#{text}</span>"
    else
      text
    end
  end.join
end

Public Instance Methods

add_token(*tokens) click to toggle source

Alias for: add_tokens

add_tokens(*tokens) click to toggle source

Adds tokens to the collected tokens

 
               # File rdoc/token_stream.rb, line 58
def add_tokens(*tokens)
  tokens.flatten.each { |token| @token_stream << token }
end

Also aliased as: add_token

collect_tokens() click to toggle source

Starts collecting tokens

 
               # File rdoc/token_stream.rb, line 67
def collect_tokens
  @token_stream = []
end

Also aliased as: start_collecting_tokens

pop_token() click to toggle source

Remove the last token from the collected tokens

 
               # File rdoc/token_stream.rb, line 76
def pop_token
  @token_stream.pop
end

start_collecting_tokens() click to toggle source

Alias for: collect_tokens

token_stream() click to toggle source

Current token stream

 
               # File rdoc/token_stream.rb, line 83
def token_stream
  @token_stream
end

tokens_to_s() click to toggle source

Returns a string representation of the token stream

 
               # File rdoc/token_stream.rb, line 90
def tokens_to_s
  token_stream.map { |token| token.text }.join ''
end

Commenting is here to help enhance the documentation. For example, code samples, or clarification of the documentation.

If you have questions about Ruby or the documentation, please post to one of the Ruby mailing lists. You will get better, faster, help that way.

If you wish to post a correction of the docs, please do so, but also file bug report so that it can be corrected for the next release. Thank you.

If you want to help improve the Ruby documentation, please see Improve the docs, or visit Documenting-ruby.org.

Home Classes Methods

In Files

Methods

Class/Module Index

RDoc::TokenStream

Public Class Methods

Public Instance Methods