Verified claim · AI-ML · 100% confidence
SentencePiece tokenizer introduced in paper: SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing (Kudo & Richardson, 2018).
Last verified 2026-05-16 · Methodology veritas-v0.1 · 0d47bb8eb637a2e4
Structured fields
- Subject
- SentencePiece tokenizer
- Predicate
introduced_in_paper- Object
- SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing (Kudo & Richardson, 2018)
- Confidence
- 100%
- Tags
- sentencepiece · tokenization · google · foundational · 2018
Sources (2)
[1] preprint · arXiv (Kudo, Richardson) · 2018-08-19
SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing“This paper describes SentencePiece, a language-independent subword tokenizer and detokenizer designed for Neural-based text processing, including Neural Machine Translation.”
[2] github release · Google · 2018-08-19
google/sentencepiece — official implementation
Cite this claim
Ready-to-paste citation (Markdown / plain text):
SentencePiece tokenizer introduced in paper: SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing (Kudo & Richardson, 2018). — SourceScore Claim 0d47bb8eb637a2e4 (verified 2026-05-16). https://sourcescore.org/api/v1/claims/0d47bb8eb637a2e4.jsonEmbed this claim
Drop this iframe into any blog post, docs page, or knowledge base. The widget renders the signed claim + primary source + click-through to this canonical page. CC-BY 4.0; attribution included.
<iframe src="https://sourcescore.org/embed/claim/0d47bb8eb637a2e4/" width="100%" height="360" frameborder="0" loading="lazy" title="SentencePiece tokenizer introduced in paper: SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing (Kudo & Richardson, 2018)."></iframe>Preview: open in new tab
Related claims
Other verified claims sharing tags with this one — useful for LLM retrieval graphs and citation discovery.
BERT (Bidirectional Encoder Representations from Transformers) introduced in paper: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (Devlin et al., 2018).
4c1ee70007dc89c1 · 100% confidence · shares 3 tags (foundational, 2018, google)
T5 (Text-to-Text Transfer Transformer) introduced in paper: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Raffel et al., 2019).
ef28341c3b308737 · 100% confidence · shares 2 tags (foundational, google)
Sparsely-Gated Mixture-of-Experts (MoE) introduced in paper: Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (Shazeer et al., 2017).
2d6d7f61f1db6493 · 100% confidence · shares 2 tags (foundational, google)
Switch Transformer introduced in paper: Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (Fedus et al., 2021).
3d9c14b9379038c9 · 100% confidence · shares 2 tags (foundational, google)
Chain-of-Thought prompting introduced in paper: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (Wei et al., 2022).
3af924da138ff84c · 100% confidence · shares 2 tags (foundational, google)
Programmatic access
Fetch this claim with a signed envelope for verification:
curl https://sourcescore.org/api/v1/claims/0d47bb8eb637a2e4.json