Technical Discussion Request: Numerical Encoding of Tokenizer IDs and Faithful Text Reconstruction

🌐 链接: https://community.openai.com/t/technical-discussion-request-numerical-encoding-of-tokenizer-ids-and-faithful-text-reconstruction/1407615
🏷️ 分组: OpenAI Developer Community
🕒 时间: 2026-10-19 07:15:16
OpenAI Developer Community
Technical Discussion Request: Numerical Encoding of Tokenizer IDs and Faithful Text Reconstruction
Technical Discussion Request: Numerical Encoding of Tokenizer IDs and Faithful Text Reconstruction
Hello everyone, I am looking for a technically knowledgeable person who can help assess an experimental method I have been developing over the past several days. The project concerns the numerical representation of token IDs generated by an existing tokenizer…