If we do a boolean search (AND) for two words, then those two lists are intersected and only the links that are contained in both list are returned as results.
If we search for a single word then all links contained in the list are valid search results.
Each list contains links to all web pages where that specific word occurs. During crawling for each word a separate list of links is created. Those web search engines are based on inverted indexes. Web search engines like Google or Baidu have to index that text in a way that allows efficient and fast retrieval. Why?īut why would we want to implement word segmentation programmatically, if people can read the unsegmented text anyway?įor CJK languages without spaces between words, it’s more obvious. How much that really is we will see if we attempt to do it programmatically. Our reading speed slows down just a bit, caused by all the background processing our brain has to do. O ur brain does this somehow intuitively and unconsciously. The quick brown fox jumps over the lazy dog This was known as Scriptio continua.Īnd it seems we haven’t yet lost our capabilities: we can easily decipher For people in the West it seems obvious that words are separated by space, while in Chinese, Japanese, Korean (CJK languages), Thai and Javanese words are written without spaces between words.Įven the Classical Greek and late Classical Latin were written without those spaces.
0 kommentar(er)
