Byte Pair Encoding (BPE)

What is BPE

BPE is a compression technique that replaces the most recurrent byte (tokens in our case) successions of a corpus, by newly created ones. The most recurrent token successions can be replaced with new created tokens, thus decreasing the sequence length and increasing the vocabulary size. Today in the NLP field, BPE is used with almost all tokenizations to build their vocabulary, as it allows to encode rare words and segmenting unknown or composed words as sequences of sub-word units. In the case of symbolic, it has been showned to improve the performances of Transformers models while helping them to learn more isotropic embedding representations.

MidiTok allows to use BPE for symbolic music, on top of any tokenizations not based on embedding pooling! BPE is backed by the Hugging Face 🤗tokenizers Rust library for fast training and encoding. You can also use the slow 100% Python alternative, even though it will be about 30 times slower. To use BPE, you must first train your tokenizer from data (miditok.MIDITokenizer.learn_bpe()), and then convert a dataset with BPE (miditok.MIDITokenizer.apply_bpe_to_dataset()).

Tokenizers can be saved and loaded (Save / Load tokenizer).

Methods

miditok.MIDITokenizer.learn_bpe(self, vocab_size: int, iterator: Optional[Iterable] = None, tokens_paths: Optional[List[Union[Path, str]]] = None, start_from_empty_voc: bool = False, **kwargs)

Method to construct the vocabulary from BPE, backed by the 🤗tokenizers library. The data used for training can either be given through the iterator argument as an iterable object yielding strings, or by tokens_paths as a list of paths to token json files that will be loaded. You can read the Hugging Face 🤗tokenizers documentation, 🤗tokenizers API documentation and 🤗tokenizers course for more details about the iterator and input type.

The training progress bar will not appear with non-proper terminals. (cf GitHub issue )

Parameters:

  • vocab_size – size of the vocabulary to learn / build.

  • iterator – an iterable object yielding the training data, as lists of string. It can be a list or a Generator. This iterator will be passed to the BPE model for training. If None is given, you must use the tokens_paths argument. (default: None)

  • tokens_paths – paths of the token json files to load and use. (default: False)

  • start_from_empty_voc – the training will start from an empty base vocabulary. The tokenizer will then have a base vocabulary only based on the unique bytes present in the training data. If you set this argument to True, you should probably then use the tokenizer only with the training data, as new data might contain “unknown” tokens missing from the vocabulary. (default: False)

  • kwargs – any additional argument to pass to the trainer.

miditok.MIDITokenizer.apply_bpe(self, seq: Union[TokSequence, List[TokSequence]])

Applies Byte Pair Encoding (BPE) to a TokSequence, or list of TokSequences. If a list is given, BPE will be applied by batch on all sequences at the time.

Parameters:

seq – Sequence(s) to apply BPE.

miditok.MIDITokenizer.apply_bpe_to_dataset(self, dataset_path: Union[Path, str], out_path: Optional[Union[str, Path]] = None)

Applies BPE to an already tokenized dataset (with no BPE).

Parameters:

  • dataset_path – path to token files to load.

  • out_path – output directory to save. If none is given, this method will overwrite original files. (default: None)

miditok.MIDITokenizer.decode_bpe(self, seq: Union[TokSequence, List[TokSequence]])

Decodes (inplace) a sequence of tokens (miditok.TokSequence) with ids encoded with BPE. This method only modifies the .ids attribute of the input sequence(s) only and does not complete it. This method can also receive a list of sequences, in which case it will decompose BPE on each of them recursively.

Parameters:

seq – token sequence to decompose.

Slow methods

To use the slow Python BPE, just train your tokenizer with miditok.MIDITokenizer.learn_bpe_slow(), and use the methods from the previous section.

miditok.MIDITokenizer.learn_bpe_slow(self, tokens_path: Union[Path, str], vocab_size: int, out_dir: Optional[Union[str, Path]] = None, files_lim: Optional[int] = None, save_converted_samples: bool = False, print_seq_len_variation: bool = True) → Tuple[List[float], List[int], List[float]]

Method to construct the vocabulary from BPE, 100% in Python and slower than miditok.MIDITokenizer.learn_bpe(). This method will build (modify) the vocabulary by analyzing an already tokenized dataset to find the most recurrent token successions. Note that this implementation is in pure Python and will be slow if you use a large amount of tokens files. It will also not be updated in the future. We advise to use the fast miditok.MIDITokenizer.learn_bpe() method.

Parameters:

  • tokens_path – path to token files to learn the BPE combinations from.

  • vocab_size – the new vocabulary size.

  • out_dir – directory to save the tokenizer’s parameters and vocabulary after BPE learning is finished.

  • files_lim – limit of token files to use. (default: None)

  • save_converted_samples – will save in out_path the samples that have been used to create the BPE vocab. Files will keep the same name and relative path. (default: True)

  • print_seq_len_variation – prints the mean sequence length before and after BPE, and the variation in %. (default: True)

Returns:

learning metrics, as lists of: - the average number of token combinations covered by the newly created BPE tokens - the maximum number of token combinations - the average sequence length Each index in the list correspond to a learning step.