| Class | CodeRay::Scanners::Scanner |
| In: |
lib/coderay/scanner.rb
|
| Parent: | StringScanner |
The base class for all Scanners.
It is a subclass of Ruby‘s great StringScanner, which makes it easy to access the scanning methods inside.
It is also Enumerable, so you can use it like an Array of Tokens:
require 'coderay'
c_scanner = CodeRay::Scanners[:c].new "if (*p == '{') nest++;"
for text, kind in c_scanner
puts text if kind == :operator
end
# prints: (*==)++;
OK, this is a very simple example :) You can also use map, +any?+, find and even sort_by, if you want.
| ScanError | = | Class.new(Exception) | Raised if a Scanner fails while scanning | |
| DEFAULT_OPTIONS | = | { :stream => false } |
The default options for all scanner classes.
Define @default_options for subclasses. |
| string | -> | code |
| More mnemonic accessor name for the input string. | ||
# File lib/coderay/scanner.rb, line 72
72: def file_extension extension = nil
73: if extension
74: @file_extension = extension.to_s
75: else
76: @file_extension ||= plugin_id.to_s
77: end
78: end
If you set :stream to true in the options, the Scanner uses a TokenStream with the block as callback to handle the tokens.
Else, a Tokens object is used.
# File lib/coderay/scanner.rb, line 106
106: def initialize code='', options = {}, &block
107: @options = self.class::DEFAULT_OPTIONS.merge options
108: raise "I am only the basic Scanner class. I can't scan "\
109: "anything. :( Use my subclasses." if self.class == Scanner
110:
111: super Scanner.normify(code)
112:
113: @tokens = options[:tokens]
114: if @options[:stream]
115: warn "warning in CodeRay::Scanner.new: :stream is set, "\
116: "but no block was given" unless block_given?
117: raise NotStreamableError, self unless kind_of? Streamable
118: @tokens ||= TokenStream.new(&block)
119: else
120: warn "warning in CodeRay::Scanner.new: Block given, "\
121: "but :stream is #{@options[:stream]}" if block_given?
122: @tokens ||= Tokens.new
123: end
124:
125: setup
126: end
# File lib/coderay/scanner.rb, line 66
66: def normify code
67: code = code.to_s
68: code.force_encoding 'binary' if code.respond_to? :force_encoding
69: code.to_unix
70: end
# File lib/coderay/scanner.rb, line 182
182: def column pos = self.pos
183: return 0 if pos <= 0
184: pos - (string.rindex(?\n, pos) || 0)
185: end
Whether the scanner is in streaming mode.
# File lib/coderay/scanner.rb, line 162
162: def streaming?
163: !!@options[:stream]
164: end
# File lib/coderay/scanner.rb, line 133
133: def string= code
134: code = Scanner.normify(code)
135: super code
136: reset_instance
137: end
Scans the code and returns all tokens in a Tokens object.
# File lib/coderay/scanner.rb, line 144
144: def tokenize new_string=nil, options = {}
145: options = @options.merge(options)
146: self.string = new_string if new_string
147: @cached_tokens =
148: if @options[:stream] # :stream must have been set already
149: reset unless new_string
150: scan_tokens @tokens, options
151: @tokens
152: else
153: scan_tokens @tokens, options
154: end
155: end
Scanner error with additional status information
# File lib/coderay/scanner.rb, line 213
213: def raise_inspect msg, tokens, state = 'No state given!', ambit = 30
214: raise ScanError, "\n\n***ERROR in %s: %s (after %d tokens)\n\ntokens:\n%s\n\ncurrent line: %d column: %d pos: %d\nmatched: %p state: %p\nbol? = %p, eos? = %p\n\nsurrounding code:\n%p ~~ %p\n\n\n***ERROR***\n\n" % [
215: File.basename(caller[0]),
216: msg,
217: tokens.size,
218: tokens.last(10).map { |t| t.inspect }.join("\n"),
219: line, column, pos,
220: matched, state, bol?, eos?,
221: string[pos - ambit, ambit],
222: string[pos, ambit],
223: ]
224: end
# File lib/coderay/scanner.rb, line 207
207: def reset_instance
208: @tokens.clear unless @options[:keep_tokens]
209: @cached_tokens = nil
210: end
This is the central method, and commonly the only one a subclass implements.
Subclasses must implement this method; it must return tokens and must only use Tokens#<< for storing scanned tokens!
# File lib/coderay/scanner.rb, line 202
202: def scan_tokens tokens, options
203: raise NotImplementedError,
204: "#{self.class}#scan_tokens not implemented."
205: end