ctranslate2::tokenizer

Trait Tokenizer

Source 
pub trait Tokenizer {
    // Required methods
    fn encode(&self, input: &str) -> Result<Vec<String>>;
    fn decode(&self, tokens: Vec<String>) -> Result<String>;
}

Required Methods§

Source

fn encode(&self, input: &str) -> Result<Vec<String>>

Encodes a given string into a sequence of tokens

Source

fn decode(&self, tokens: Vec<String>) -> Result<String>

Decodes a given sequence of tokens back into a single string

Implementors§