The author Paul McCann (polm) is one of my favourite programmers out there!
He’s done awesome work in the Japanese NLP space over the last decade which has really helped me in my language learning projects.
He maintains a mecab (Japanese tokenizer) wrapper for Python [0], has a book on Japanese NLP written for English speakers [1] and also worked on Spacy at one point [2].
[0] https://github.com/polm/fugashi
[1] https://www.japanesenlp.com/
[2] https://spacy.io/
ERerjiang1小时前
I think there’s evidence found for the origin of “彁” as the result of a poor scan of a newspaper article. Look up “彁 新聞” to find some japanese sources about this.
HNhnfong2小时前
Well, vast swaths of the Kangxi dictionary (which serves as "sources" for probably most of the CJK characters) are such "ghost" characters as described in the article...
The peculiar properties of CJK characters and the philosophy (apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism) under which they were implemented in Unicode probably singlehandedly forced unicode to expand beyond the BMP....
SEsedatk1小时前
Fascinating. But, I guess it's better to have superfluous invalid characters than missing real ones.
PApanzi18分钟前
Is anyone using these characters now for anything? No youth language or online slang using it?
评论
5 条预览评论 · 正在加载完整讨论请先登录 h4cker 账号，然后连接 Hacker News 后发表评论。
The author Paul McCann (polm) is one of my favourite programmers out there! He’s done awesome work in the Japanese NLP space over the last decade which has really helped me in my language learning projects. He maintains a mecab (Japanese tokenizer) wrapper for Python [0], has a book on Japanese NLP written for English speakers [1] and also worked on Spacy at one point [2]. [0] https://github.com/polm/fugashi [1] https://www.japanesenlp.com/ [2] https://spacy.io/
I think there’s evidence found for the origin of “彁” as the result of a poor scan of a newspaper article. Look up “彁 新聞” to find some japanese sources about this.
Well, vast swaths of the Kangxi dictionary (which serves as "sources" for probably most of the CJK characters) are such "ghost" characters as described in the article... The peculiar properties of CJK characters and the philosophy (apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism) under which they were implemented in Unicode probably singlehandedly forced unicode to expand beyond the BMP....
Fascinating. But, I guess it's better to have superfluous invalid characters than missing real ones.
Is anyone using these characters now for anything? No youth language or online slang using it?