Package org.languagetool.tokenizers
package org.languagetool.tokenizers
-
ClassDescriptionInterface for components that take compound words and split them into their parts.Tokenizes text into sentences.A very simple sentence tokenizer that splits on
[.!?…]
followed by whitespace or an uppercase letter.Class to tokenize sentences using rules from an SRX file.Tools for loading an SRX tokenizer file.Interface for classes that tokenize text into smaller units.Tokenizes a sentence into words.