Tokenizer · PHP学习 · 看云

NIUCLOUD是一款SaaS管理后台框架多应用插件+云编译。上千名开发者、服务商正在积极拥抱开发者生态。欢迎开发者们免费入驻。一起助力发展！广告

# Tokenizer 标记器功能提供了与Zend Engine中嵌入的PHP标记器的接口。使用这些功能，您可以编写自己的PHP源代码分析或修改工具，而无需在词汇级别上处理语言规范。另请参阅[有关令牌](https://www.php.net/manual/en/tokens.php)的[附录](https://www.php.net/manual/en/tokens.php)。 PHP 4.4.0 开始，这些函数默认是被激活的 PHP 的 Windows 版本已内建对此扩展的支持。不需要载入额外的扩展来使用这些函数 - [预定义常量](https://www.php.net/manual/zh/tokenizer.constants.php) - [范例](https://www.php.net/manual/zh/tokenizer.examples.php) - [Tokenizer 函数](https://www.php.net/manual/zh/ref.tokenizer.php) - [token\_get\_all](https://www.php.net/manual/zh/function.token-get-all.php)— 将提供的源码按 PHP 标记进行分割 - [token\_name](https://www.php.net/manual/zh/function.token-name.php)— 获取提供的 PHP 解析器代号的符号名称这里是一个简单使用 tokenizer 的PHP脚本例子，它将读取一个PHP文件，去掉代码中全部注释，然后只打印纯代码 ``` <pre class="calibre17">``` <span class="token">/* * T_ML_COMMENT 在PHP 5中不存在. * 为了保持向后兼容性，下面的三行代码定义了它。 * * 接下来的两行只定义了PHP 5的T_DOC_COMMENT，对于PHP 4，我们将其掩码为T_ML_COMMENT */</span> <span class="token5">if</span> <span class="token3">(</span><span class="token1">!</span><span class="token4">defined</span><span class="token3">(</span><span class="token2">'T_ML_COMMENT'</span><span class="token3">)</span><span class="token3">)</span> <span class="token3">{</span> <span class="token4">define</span><span class="token3">(</span><span class="token2">'T_ML_COMMENT'</span><span class="token3">,</span> T_COMMENT<span class="token3">)</span><span class="token3">;</span> <span class="token3">}</span> <span class="token5">else</span> <span class="token3">{</span> <span class="token4">define</span><span class="token3">(</span><span class="token2">'T_DOC_COMMENT'</span><span class="token3">,</span> T_ML_COMMENT<span class="token3">)</span><span class="token3">;</span> <span class="token3">}</span> $source <span class="token1">=</span> <span class="token4">file_get_contents</span><span class="token3">(</span><span class="token2">'example.php'</span><span class="token3">)</span><span class="token3">;</span> $tokens <span class="token1">=</span> <span class="token4">token_get_all</span><span class="token3">(</span>$source<span class="token3">)</span><span class="token3">;</span> foreach <span class="token3">(</span>$tokens as $token<span class="token3">)</span> <span class="token3">{</span> <span class="token5">if</span> <span class="token3">(</span><span class="token4">is_string</span><span class="token3">(</span>$token<span class="token3">)</span><span class="token3">)</span> <span class="token3">{</span> <span class="token">// simple 1-character token</span> echo $token<span class="token3">;</span> <span class="token3">}</span> <span class="token5">else</span> <span class="token3">{</span> <span class="token">// token array</span> <span class="token4">list</span><span class="token3">(</span>$id<span class="token3">,</span> $text<span class="token3">)</span> <span class="token1">=</span> $token<span class="token3">;</span> switch <span class="token3">(</span>$id<span class="token3">)</span> <span class="token3">{</span> case T_COMMENT<span class="token3">:</span> case T_ML_COMMENT<span class="token3">:</span> <span class="token">// we've defined this</span> case T_DOC_COMMENT<span class="token3">:</span> <span class="token">// and this</span> <span class="token">// no action on comments</span> <span class="token5">break</span><span class="token3">;</span> default<span class="token3">:</span> <span class="token">// anything else -> output "as is"</span> echo $text<span class="token3">;</span> <span class="token5">break</span><span class="token3">;</span> <span class="token3">}</span> <span class="token3">}</span> <span class="token3">}</span> ``` ```