linguistic/ngramextractor

从给定文本中提取n-gram，并进行诸如去除停用词等语言预处理

维护者

详细信息

github.com/linguistic-dev/n-gram-extractor

源代码

问题

安装: 804

依赖项: 0

建议者: 0

安全: 0

星星: 3

关注者: 0

分支: 0

开放问题: 0

dev-master 2017-12-05 23:09 UTC

Requires (Dev)

GPL3 0c0d27abacd3b4209c844123c3cfe882b96e74ec

  • Mark Schatz

This package is not auto-updated.

Last update: 2024-09-29 05:48:23 UTC

README

安装

通过Composer简单安装

composer require linguistic/ngramextractor

用法

即将推出。

示例

$tokenizer = new Tokenizer();
$tokenizer->addRemovalRule('/<\/?\w+[\s\w\=\"\/\#\-\:\.\_]*>/') # Removes HTML Tags
->addRemovalRule('/[^a-z0-9]+/', ' ') # Replaces everything which is not text with a space
->setSeperator('/\s+/'); # Tokenizes text with whitespace as delimiter
$content = ""; # The text that should get tokenized
$stopwords = array(); # (optional) array of stopwords

$extractor = new NGramExtractor($content, $tokenizer, $stopwords);
$unigrams    = $extractor->getNGrams(1); # gets all n-grams in the text, n = 1

$unigramsFiltered    = NGramExtractor::limitByOccurance($extractor->getNGramCount(1, true), 3); # get unigrams and their occurance if the occurance is greater or equal 3

资源