I designing a Vector Database to work with Falcon 180B as the knowledge domain data, which tokenizer format is used? BPE, Wordpiece, Sentence piece? it is important as to match both.
Your need to confirm your account before you can post a new comment.
