xp-framework/tokenize

文本分词

维护者

详细信息

github.com/xp-framework/tokenize

主页

源代码

问题

安装次数: 89,805

依赖项: 7

建议者: 0

安全性: 0

星标: 0

关注者: 3

分支: 0

公开问题: 0

v9.1.0 2024-03-24 13:28 UTC

Requires

Requires (Dev)

BSD-3-Clause a38b5f76ee0d8e32f017dc91f85b98d6a1857c92

modulexp

This package is auto-updated.

Last update: 2024-08-24 20:50:28 UTC

README

Build status on GitHub XP Framework Module BSD Licence Requires PHP 7.0+ Supports PHP 8.0+ Latest Stable Version

文本分词

use text\{StringTokenizer, StreamTokenizer};
use io\File;

// Supports strings and streams
$tokens= new StringTokenizer('He asked: Can you parse this?', ' .?!,;:', true);
$tokens= new StreamTokenizer((new File('parse-me.txt'))->in(), ' .?!,;:', true);

// Can iterate using foreach...
foreach ($tokens as $token) {
  Console::writeLine($token);
}

// ...or with an iterator API
while ($tokens->hasMoreTokens()) {
  Console::writeLine($tokens->nextToken());
}

// Returns: ["He", " ", "asked", ":", " ", "Can", " ", "you", " ", "parse", " ", "this", "?"]