Summary

A LetterAnalyzer creates a TokenStream that splits the input up into maximal strings of characters as recognized by the current locale. If implemented in Ruby it would look like;

  class LetterAnalyzer
    def initialize(lower = true)
      @lower = lower
    end

    def token_stream(field, str)
      return LetterTokenizer.new(str, @lower)
    end
  end

As you can see it makes use of the LetterTokenizer.