Basics

This page features the bases of MidiTok, of how tokenizers work.

Tokens and vocabulary

A token can take three forms, which we name by convention:

  • Token (string): the form describing it, e.g. Pitch_50.

  • Id (int): an unique associated integer, used as an index.

  • Byte (string): an unique associated byte, used internally for Byte Pair Encoding (BPE).

MidiTok works with TokSequence objects to output token sequences of represented by these three forms.

TokSequence

The methods of MidiTok use miditok.TokSequence objects as input and outputs. A TokSequence holds tokens as the three forms described in Byte Pair Encoding (BPE).

You can use the miditok.MIDITokenizer.complete_sequence() method to automatically fill the non-initialized attributes of a TokSequence.

class miditok.TokSequence(tokens: Optional[List[Union[str, List[str]]]] = None, ids: Optional[List[Union[int, List[int]]]] = None, bytes: Optional[str] = None, events: Optional[List[Union[Event, List[Event]]]] = None, ids_bpe_encoded: bool = False, _ids_no_bpe: Optional[List[Union[int, List[int]]]] = None)

Represents a sequence of token. A TokSequence can represent tokens by their several forms:

  • tokens (list of str): tokens as sequence of strings.

  • ids (list of int), these are the one to be fed to models.

  • events (list of Event): Event objects that can carry time or other information useful for debugging.

  • bytes (str): ids are converted into unique bytes, all joined together in a single string.

Bytes are used internally by MidiTok for Byte Pair Encoding. The ids_are_bpe_encoded attribute tells if ids is encoded with BPE.

miditok.MIDITokenizer.complete_sequence()

Vocabulary

The vocabulary of a tokenizer acts as a lookup table, linking tokens (string) to their ids (integer). The vocabulary is an attribute of the tokenizer and can be accessed with tokenizer.vocab. The vocabulary is a Python dictionary binding tokens (keys) to their ids (values). For tokenizations with embedding embedding pooling (e.g. CPWord or Octuple), tokenizer.vocab will be a list of Vocabulary objects, and the tokenizer.is_multi_vocab property will be True.

With Byte Pair Encoding: tokenizer.vocab holds all the basic tokens describing the note and time attributes of music. By analogy with text, these tokens can be seen as unique characters. After training a tokenizer with Byte Pair Encoding (BPE), a new vocabulary is built with newly created tokens from pairs of basic tokens. This vocabulary can be accessed with tokenizer.vocab_bpe, and binds tokens as bytes (string) to their associated ids (int). This is the vocabulary of the 🤗tokenizers BPE model.

MIDI Tokenizer

MidiTok features several MIDI tokenizations, all inheriting from the miditok.MIDITokenizer class. The documentation of the arguments teaches you how to create a custom tokenizer.

class miditok.MIDITokenizer(pitch_range: range = range(21, 109), beat_res: Dict[Tuple[int, int], int] = {(0, 4): 8, (4, 12): 4}, nb_velocities: int = 32, additional_tokens: Dict[str, Union[bool, int, Dict[str, Tuple], Tuple[int, int]]] = {'Chord': False, 'Program': False, 'Rest': False, 'Tempo': False, 'TimeSignature': False, 'chord_maps': {'7aug': (0, 4, 8, 11), '7dim': (0, 3, 6, 9), '7dom': (0, 4, 7, 10), '7halfdim': (0, 3, 6, 10), '7maj': (0, 4, 7, 11), '7min': (0, 3, 7, 10), '9maj': (0, 4, 7, 10, 14), '9min': (0, 4, 7, 10, 13), 'aug': (0, 4, 8), 'dim': (0, 3, 6), 'maj': (0, 4, 7), 'min': (0, 3, 7), 'sus2': (0, 2, 7), 'sus4': (0, 5, 7)}, 'chord_tokens_with_root_note': True, 'chord_unknown': False, 'nb_tempos': 32, 'programs': [-1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127], 'rest_range': (2, 8), 'tempo_range': (40, 250), 'time_signature_range': (8, 2)}, special_tokens: List[str] = ['PAD', 'BOS', 'EOS', 'MASK'], unique_track: bool = False, params: Optional[Union[str, Path]] = None)

MIDI tokenizer base class, containing common methods and attributes for all tokenizers.

Parameters:

  • pitch_range – (default: range(21, 109)) range of MIDI pitches to use. Pitches can take values between 0 and 127 (included). The General MIDI 2 (GM2) specifications indicate the recommended ranges of pitches per MIDI program (instrument). These recommended ranges can also be found in miditok.constants . In all cases, the range from 21 to 108 (included) covers all the recommended values. When processing a MIDI, the notes with pitches under or above this range can be discarded.

  • beat_res – (default: {(0, 4): 8, (4, 12): 4}) beat resolutions, as a dictionary in the form: {(beat_x1, beat_x2): beat_res_1, (beat_x2, beat_x3): beat_res_2, ...} The keys are tuples indicating a range of beats, ex 0 to 3 for the first bar, and the values are the resolution (in samples per beat) to apply to the ranges, ex 8. This allows to use Duration / TimeShift tokens of different lengths / resolutions. Note: for tokenization with Position tokens, the total number of possible positions will be set at four times the maximum resolution given (max(beat_res.values)).

  • nb_velocities – (default: 32) number of velocity bins. In the MIDI norm, velocities can take up to 128 values (0 to 127). This parameter allows to reduce the number of velocity values. The velocities of the MIDIs resolution will be downsampled to nb_velocities values, equally separated between 0 and 127.

  • additional_tokens – (default: None used) specify which additional tokens to use. Compatibilities between tokenization and additional tokens may vary. See Additional tokens for the details and available tokens.

  • special_tokens – list of special tokens. This must be given as a list of strings given only the names of the tokens. (default: ["PAD", "BOS", "EOS", "MASK"])

  • unique_track – set to True if the tokenizer works only with a unique track. Tokens will be saved as a single track. This applies to representations that natively handle multiple tracks such as Octuple, resulting in a single “stream” of tokens for all tracks. This attribute will be saved in config files of the tokenizer. (default: False)

  • params – path to a tokenizer config file. This will override other arguments and load the tokenizer based on the config file. This is particularly useful if the tokenizer learned Byte Pair Encoding. (default: None)

add_to_vocab(token: Union[str, Event], vocab_idx: Optional[int] = None, byte_: Optional[str] = None, add_to_bpe_model: bool = False)

Adds an event to the vocabulary. Its index (int) will be the length of the vocab.

Parameters:

  • token – token to add, as a formatted string of the form “Type_Value”, e.g. Pitch_80, or an Event.

  • vocab_idx – idx of the vocabulary (in case of embedding pooling). (default: None)

  • byte – unique byte associated to the token. This is used when building the vocabulary with fast BPE. If None is given, it will default to chr(id_ + CHR_ID_START) . (default: None)

  • add_to_bpe_model – the token will be added to the bpe_model vocabulary too. (default: None)

apply_bpe(seq: Union[TokSequence, List[TokSequence]])

Applies Byte Pair Encoding (BPE) to a TokSequence, or list of TokSequences. If a list is given, BPE will be applied by batch on all sequences at the time.

Parameters:

seq – Sequence(s) to apply BPE.

apply_bpe_to_dataset(dataset_path: Union[Path, str], out_path: Optional[Union[str, Path]] = None)

Applies BPE to an already tokenized dataset (with no BPE).

Parameters:

  • dataset_path – path to token files to load.

  • out_path – output directory to save. If none is given, this method will overwrite original files. (default: None)

complete_sequence(seq: TokSequence)

Completes (inplace) a miditok.TokSequence object by converting its attributes. The input sequence can miss some of its attributes (ids, tokens), but needs at least one for reference. This method will create the missing ones from the present ones. The bytes attribute will be created if the tokenizer has been trained with BPE. The events attribute will not be filled as it is only intended for debug purpose.

Parameters:

seq – input miditok.TokSequence, must have at least one attribute defined.

decode_bpe(seq: Union[TokSequence, List[TokSequence]])

Decodes (inplace) a sequence of tokens (miditok.TokSequence) with ids encoded with BPE. This method only modifies the .ids attribute of the input sequence(s) only and does not complete it. This method can also receive a list of sequences, in which case it will decompose BPE on each of them recursively.

Parameters:

seq – token sequence to decompose.

property is_multi_voc: bool

Returns a bool indicating if the tokenizer uses embedding pooling, and so have multiple vocabularies.

Returns:

True is the tokenizer uses embedding pooling else False.

learn_bpe(vocab_size: int, iterator: Optional[Iterable] = None, tokens_paths: Optional[List[Union[Path, str]]] = None, start_from_empty_voc: bool = False, **kwargs)

Method to construct the vocabulary from BPE, backed by the 🤗tokenizers library. The data used for training can either be given through the iterator argument as an iterable object yielding strings, or by tokens_paths as a list of paths to token json files that will be loaded. You can read the Hugging Face 🤗tokenizers documentation, 🤗tokenizers API documentation and 🤗tokenizers course for more details about the iterator and input type.

The training progress bar will not appear with non-proper terminals. (cf GitHub issue )

Parameters:

  • vocab_size – size of the vocabulary to learn / build.

  • iterator – an iterable object yielding the training data, as lists of string. It can be a list or a Generator. This iterator will be passed to the BPE model for training. If None is given, you must use the tokens_paths argument. (default: None)

  • tokens_paths – paths of the token json files to load and use. (default: False)

  • start_from_empty_voc – the training will start from an empty base vocabulary. The tokenizer will then have a base vocabulary only based on the unique bytes present in the training data. If you set this argument to True, you should probably then use the tokenizer only with the training data, as new data might contain “unknown” tokens missing from the vocabulary. (default: False)

  • kwargs – any additional argument to pass to the trainer.

learn_bpe_slow(tokens_path: Union[Path, str], vocab_size: int, out_dir: Optional[Union[str, Path]] = None, files_lim: Optional[int] = None, save_converted_samples: bool = False, print_seq_len_variation: bool = True) → Tuple[List[float], List[int], List[float]]

Method to construct the vocabulary from BPE, 100% in Python and slower than miditok.MIDITokenizer.learn_bpe(). This method will build (modify) the vocabulary by analyzing an already tokenized dataset to find the most recurrent token successions. Note that this implementation is in pure Python and will be slow if you use a large amount of tokens files. It will also not be updated in the future. We advise to use the fast miditok.MIDITokenizer.learn_bpe() method.

Parameters:

  • tokens_path – path to token files to learn the BPE combinations from.

  • vocab_size – the new vocabulary size.

  • out_dir – directory to save the tokenizer’s parameters and vocabulary after BPE learning is finished.

  • files_lim – limit of token files to use. (default: None)

  • save_converted_samples – will save in out_path the samples that have been used to create the BPE vocab. Files will keep the same name and relative path. (default: True)

  • print_seq_len_variation – prints the mean sequence length before and after BPE, and the variation in %. (default: True)

Returns:

learning metrics, as lists of: - the average number of token combinations covered by the newly created BPE tokens - the maximum number of token combinations - the average sequence length Each index in the list correspond to a learning step.

property len: Union[int, List[int]]

Returns the length of the vocabulary. If the tokenizer uses embedding pooling / have multiple vocabularies, it will return the list of their lengths. Use the miditok.MIDITokenizer.__len__() magic method (len(tokenizer)) to get the sum of the lengths.

Returns:

length of the vocabulary.

load_params(config_file_path: Union[str, Path])

Loads the parameters of the tokenizer from a config file.

Parameters:

config_file_path – path to the tokenizer config file (encoded as json).

static load_tokens(path: Union[str, Path]) → Union[List[Any], Dict]

Loads tokens saved as JSON files.

Parameters:

path – path of the file to load.

Returns:

the tokens, with the associated information saved with.

midi_to_tokens(midi: MidiFile, apply_bpe_if_possible: bool = True, *args, **kwargs) → List[TokSequence]

Tokenizes a MIDI file. This method returns a list of miditok.TokSequence.

If you are implementing your own tokenization by subclassing this class, override the ``_midi_to_tokens`` method. This method implement necessary MIDI preprocessing.

Parameters:

  • midi – the MIDI object to convert.

  • apply_bpe_if_possible – will apply BPE if the tokenizer’s vocabulary was learned with.

Returns:

sequences of tokens.

preprocess_midi(midi: MidiFile)

Pre-process (in place) a MIDI file to quantize its time and note attributes before tokenizing it. Its notes attribute (times, pitches, velocities) will be quantized and sorted, duplicated notes removed, as well as tempos. Empty tracks (with no note) will be removed from the MIDI object. Notes with pitches outside of self.pitch_range will be deleted.

Parameters:

midi – MIDI object to preprocess.

save_params(out_path: Union[str, Path], additional_attributes: Optional[Dict] = None)

Saves the config / parameters of the tokenizer in a json encoded file. This can be useful to keep track of how a dataset has been tokenized. Note: if you override this method, you should probably call it (super()) at the end and use the additional_attributes argument.

Parameters:

  • out_path – output path to save the file.

  • additional_attributes – any additional information to store in the config file. It can be used to override the default attributes saved in the parent method. (default: None)

save_tokens(tokens: Union[TokSequence, List, ndarray, Any], path: Union[str, Path], programs: Optional[List[Tuple[int, bool]]] = None, **kwargs)

Saves tokens as a JSON file. In order to reduce disk space usage, only the ids are saved. Use kwargs to save any additional information within the JSON file.

Parameters:

  • tokens – tokens, as list, numpy array, torch or tensorflow Tensor.

  • path – path of the file to save.

  • programs – (optional), programs of the associated tokens, should be given as a tuples (int, bool) for (program, is_drum).

  • kwargs – any additional information to save within the JSON file.

token_id_type(id_: int, vocab_id: Optional[int] = None) → str

Returns the type of the given token id.

Parameters:

  • id – token id to get the type.

  • vocab_id – index of the vocabulary associated to the token, if applicable. (default: None)

Returns:

the type of the token, as a string

token_ids_of_type(token_type: str, vocab_id: Optional[int] = None) → List[int]

Returns the list of token ids of the given type.

Parameters:

  • token_type – token type to get the associated token ids.

  • vocab_id – index of the vocabulary associated to the token, if applicable. (default: None)

Returns:

list of token ids.

tokenize_midi_dataset(midi_paths: Union[List[str], List[Path]], out_dir: Union[str, Path], validation_fn: Optional[Callable[[MidiFile], bool]] = None, data_augment_offsets=None, apply_bpe: bool = True, save_programs: bool = True, logging: bool = True)

Converts a dataset / list of MIDI files, into their token version and save them as json files The resulting Json files will have the shape (T, ), first dimension is tracks, second tokens. In order to reduce disk space usage, **only the ids are saved*. If save_programs is True, the shape will be [(T, *), (T, 2)], first dim is tokens and programs instead, for programs the first value is the program, second a bool indicating if the track is drums. The config of the tokenizer will be saved as a “config.txt” file by default.

Parameters:

  • midi_paths – paths of the MIDI files.

  • out_dir – output directory to save the converted files.

  • validation_fn – a function checking if the MIDI is valid on your requirements (e.g. time signature, minimum/maximum length, instruments …).

  • data_augment_offsets – data augmentation arguments, to be passed to the miditok.data_augmentation.data_augmentation_dataset method. Has to be given as a list / tuple of offsets pitch octaves, velocities, durations, and finally their directions (up/down). (default: None)

  • apply_bpe – will apply BPE on the dataset to save, if the vocabulary was learned with.

  • save_programs – will also save the programs of the tracks of the MIDI. (default: True)

  • logging – logs progress bar.

tokens_to_midi(tokens: Union[TokSequence, List, ndarray, Any], programs: Optional[List[Tuple[int, bool]]] = None, output_path: Optional[str] = None, time_division: Optional[int] = 384) → MidiFile

Converts multiple sequences of tokens (miditok.TokSequence) into a MIDI and saves it. NOTE: With Remi, MIDI-Like, CP Word or other tokenization that processes tracks independently, only the tempo changes of the first track in tokens will be used.

Parameters:

  • tokens – tokens to convert. Can be either a list of miditok.TokSequence, a Tensor (PyTorch and Tensorflow are supported), a numpy array or a Python list of ints. The first dimension represents tracks, unless the tokenizer handle tracks altogether as a single token sequence (e.g. Octuple, MuMIDI): tokenizer.unique_track == True.

  • programs – programs of the tracks. If none is given, will default to piano, program 0. (default: None)

  • output_path – path to save the file. (default: None)

  • time_division – MIDI time division / resolution, in ticks/beat (of the MIDI to create).

Returns:

the midi object (miditoolkit.MidiFile).

abstract tokens_to_track(tokens: Union[TokSequence, List, ndarray, Any], time_division: Optional[int] = 384, program: Optional[Tuple[int, bool]] = (0, False)) → Tuple[Instrument, List[TempoChange]]

Converts a sequence of tokens into a track object. This method is unimplemented and need to be overridden by inheriting classes. This method should be decorated with _in_as_complete_seq to receive any type of input.

Parameters:

  • tokens – tokens to convert. Can be either a miditok.TokSequence, a Tensor (PyTorch and Tensorflow are supported), a numpy array or a Python list of ints.

  • time_division – MIDI time division / resolution, in ticks/beat (of the MIDI to create).

  • program – the MIDI program of the produced track and if it drum. (default (0, False), piano)

Returns:

the miditoolkit instrument object and the possible tempo changes.

abstract track_to_tokens(track: Instrument) → TokSequence

Converts a track (miditoolkit.Instrument object) into a sequence of tokens (miditok.TokSequence). This method is unimplemented and need to be overridden by inheriting classes. For an easier implementation, use the _out_as_complete_seq decorator.

Parameters:

track – MIDI track to convert.

Returns:

miditok.TokSequence of corresponding tokens.

property vocab: Union[Dict[str, int], List[Dict[str, int]]]

Get the base vocabulary, as a dictionary linking tokens (str) to their ids (int). The different (hidden / protected) vocabulary attributes of the class are:

  • ._vocab_base : Dict[str: int] token -> id - Registers all known base tokens.

  • .__vocab_base_inv : Dict[int: str] id -> token - Inverse of ._base_vocab , to go the other way.

  • ._vocab_base_id_to_byte : Dict[int: str] id -> byte - Link ids to their associated unique bytes.

  • ._vocab_base_byte_to_token : Dict[str: str] - similar as above but for tokens.

  • ._vocab_bpe_bytes_to_tokens : Dict[str: List[str]] byte(s) -> token(s) used to decode BPE.

  • ._bpe_model.get_vocab() : Dict[str: int] byte -> id - bpe model vocabulary, based on unique bytes

Before training the tokenizer with BPE, bytes are obtained by running chr(id) . After training, if we did start from an empty vocabulary, some base tokens might be removed from ._vocab_base , if they were never found in the training samples. The base vocabulary being changed, chr(id) would then bind to incorrect bytes (on which byte succession would not have been learned). We register the original id/token/byte association in ._vocab_base_id_to_byte and ._vocab_base_byte_to_token .

Returns:

the base vocabulary.

property vocab_bpe: [<class 'str'>, <class 'int'>]

Returns the vocabulary learnt with BPE. In case the tokenizer has not been trained with BPE, it returns None. In case it was trained with the slow BPE method, it returns the base vocabulary.

Returns:

the BPE model’s vocabulary.

Additional tokens

MidiTok offers to include additional tokens on music information. You can specify them in the additional_tokens argument when creating a tokenizer.

  • Chords: indicates the presence of a chord at a certain time step. MidiTok uses a chord detection method based on onset times and duration. This allows MidiTok to detect precisely chords without ambiguity, whereas most chord detection methods in symbolic music based on chroma features can’t.

  • Rests: includes Rest tokens whenever a portion of time is silent, i.e. no note is being played. This token type is decoded as a TimeShift event. You can choose the minimum and maximum rests values to represent with the rest_range key in the additional_tokens dictionary (default is 1/2 beat to 8 beats). Note that rests shorter than one beat are only divisible by the first beat resolution, e.g. a rest of 5/8th of a beat will be a succession of Rest_0.4 and Rest_0.1, where the first number indicate the rest duration in beats and the second in samples / positions.

  • Tempos: specifies the current tempo. This allows to train a model to predict tempo changes. Tempo values are quantized accordingly to the nb_tempos and tempo_range entries in the additional_tokens dictionary (default is 32 tempos from 40 to 250).

  • Programs: used to specify an instrument / MIDI program. MidiTok only offers the possibility to include these tokens in the vocabulary for you, but won’t use them. If you need model multitrack symbolic music with other methods than Octuple / MuMIDI, MidiTok leaves you the choice / task to represent the track information the way you want. You can do it as in LakhNES or MMM.

  • Time Signature: specifies the current time signature. Only implemented with Octuple in MidiTok a.t.w.

Compatibility table of tokenizations and additional tokens.

Token type

REMI

MIDI-Like

TSD

Structured

CPWord

Octuple

MuMIDI

Chord

✅

✅

✅

✅

❌

❌

✅

Rest

✅

✅

✅

✅

❌

❌

❌

Tempo

✅

✅

✅

✅

❌

✅

✅

Program

✅

✅

✅

✅

✅

✅

✅

Time signature

❌

❌

❌

❌

❌

✅

❌

Special tokens

MidiTok offers to include some special tokens to the vocabulary. To use them, you must specify them when creating a tokenizer (constructor argument). These are:

  • pad (default True) –> PAD_None: a padding token to use when training a model with batches of sequences of unequal lengths. The padding token will be at index 0 of the vocabulary.

  • sos_eos (default False) –> SOS_None and EOS_None: “Start Of Sequence” and “End Of Sequence” tokens, designed to be placed respectively at the beginning and end of a token sequence during training. At inference, the EOS token tells when to end the generation.

  • mask (default False) –> MASK_None: a masking token, to use when pre-training a (bidirectional) model with a self-supervised objective like BERT.

  • sep (default: False) –> SEP_None: a token to use as a separation between sequences.

Note: you can use the tokenizer.special_tokens property to get the list of the special tokens of a tokenizer.

Magic methods

Magic methods allows to intuitively access to a tokenizer’s attributes and methods. We list them here with some examples.

miditok.MIDITokenizer.__call__(self, obj: Any, *args, **kwargs)

Automatically tokenizes a MIDI file, or detokenizes a sequence of tokens. This will call the miditok.MIDITokenizer.midi_to_tokens() if you provide a MIDI object, or the miditok.MIDITokenizer.tokens_to_midi() method otherwise.

Parameters:

obj – a MIDI object or sequence of tokens.

Returns:

the converted object.

tokens = tokenizer(midi)
midi2 = tokenizer(tokens)
miditok.MIDITokenizer.__getitem__(self, item: Union[int, str, Tuple[int, Union[int, str]]]) → Union[str, int]

Convert a token (int) to an event (str), or vice-versa.

Parameters:

item – a token (int) or an event (str). For embedding pooling, you must provide a tuple where the first element in the index of the vocabulary.

Returns:

the converted object.

pad_token = tokenizer["PAD_None"]
miditok.MIDITokenizer.__len__(self) → int

Returns the length of the vocabulary. If the tokenizer uses embedding pooling / have multiple vocabularies, it will return the sum of their lengths. If the vocabulary was learned with fast BPE, it will return the length of the BPE vocabulary, i.e. the proper number of possible token ids. Otherwise it will return the length of the base vocabulary. Use the miditok.MIDITokenizer.len() property (tokenizer.len) to have the list of lengths.

Returns:

length of the vocabulary.

nb_classes = len(tokenizer)
nb_classes_per_vocab = tokenizer.len  # applicable to tokenizer with embedding pooling, e.g. CPWord or Octuple
miditok.MIDITokenizer.__eq__(self, other) → bool

Checks if two tokenizers are identical. This is done by comparing their vocabularies, as they are built depending on most of their attributes.

Parameters:

other – tokenizer to compare.

Returns:

True if the vocabulary(ies) are identical, False otherwise.

if tokenizer1 == tokenizer2:
    print("The tokenizers have the same vocabulary!")

Save / Load tokenizer

You can save and load a tokenizer’s parameters and vocabulary. This is especially useful to track tokenized datasets, and to save tokenizers with vocabularies learned with Byte Pair Encoding (BPE).

miditok.MIDITokenizer.save_params(self, out_path: Union[str, Path], additional_attributes: Optional[Dict] = None)

Saves the config / parameters of the tokenizer in a json encoded file. This can be useful to keep track of how a dataset has been tokenized. Note: if you override this method, you should probably call it (super()) at the end and use the additional_attributes argument.

Parameters:

  • out_path – output path to save the file.

  • additional_attributes – any additional information to store in the config file. It can be used to override the default attributes saved in the parent method. (default: None)

miditok.MIDITokenizer.load_params(self, config_file_path: Union[str, Path])

Loads the parameters of the tokenizer from a config file.

Parameters:

config_file_path – path to the tokenizer config file (encoded as json).

Limitations

Tokenizations using Bar tokens (REMI, CPWord and MuMIDI) only considers a 4/x time signature for now. This means that each bar is considered covering 4 beats. Octuple supports it.