通过 
搜索
搜索teamtnt / crawler
分布式爬虫架构
dev-master
2019-06-10 19:00 UTC
Requires
- php: ^7.1.3
- fideloper/proxy: ^4.0
- guzzlehttp/guzzle: ^6.3
- laravel/framework: 5.8.*
- laravel/tinker: ^1.0
- symfony/filesystem: ^4.2
- symfony/process: ^4.2
Requires (Dev)
- beyondcode/laravel-dump-server: ^1.0
- filp/whoops: ^2.0
- fzaninotto/faker: ^1.4
- mockery/mockery: ^1.0
- nunomaduro/collision: ^3.0
- phpunit/phpunit: ^7.5
This package is auto-updated.
Last update: 2024-08-30 01:26:14 UTC
README
分布式爬虫
要求
安装
通过 Composer
composer require teamtnt/crawler
配置
每个实例都需要一个标识符。这可以在 .env 中添加
NODE_NAME="Instance 1"
域名喂入器需要以种子域名开始。之后，运行
php artisan crawler
抓取单个 URL
php artisan url:frontier www.example.com/something