Keras Text Preprocessing - Saving Tokenizer object to file for scoring

Question

I've trained a sentiment classifier model using Keras library by following the below steps(broadly). Convert Text corpus into sequences using Tokenizer object/class Build a model using the model.fit() method Evaluate this model Now for scoring using this model, I was able to save the model to a file and load from a file. However I've not found a way to save the Tokenizer object to file. Without this I'll have to process the corpus every time I need to score even a single sentence. Is there a way around this? Select the correct answer from above options

Answer 1

I would suggest you to use pickle to save Tokenizer: import pickle # saving with open('tokenizer.pickle', 'wb') as handle: pickle.dump(tokenizer, handle, protocol=pickle.HIGHEST_PROTOCOL) # loading with open('tokenizer.pickle', 'rb') as handle: tokenizer = pickle.load(handle)