To first give you the background, tiktoken (the tool for counting tokens in text) has not ever been maintained adequately. It is an essential Python software for application developers who need to fit large text into the available context length. It has always…