Scylla一款高质量的免费代理 IP 池工具

Scylla一款高质量的免费代理 IP 池工具，仅支持 Python 3.6

大话开源项目 2018-09-17 20:58

视频尚未发布，暂时无法播放

一款高质量的免费代理 IP 池工具，仅支持 Python 3.6，特性如下：

自动化的代理 IP 爬取与验证

易用的 JSON API

简单但美观的 web 用户界面，基于 TypeScript 和 React（例如，代理的地理分布）

最少仅用一行代码即可与 Scrapy 和 requests 进行集成

等等

Get started

Installation

Install with Docker (highly recommended)

docker run -d -p 8899:8899 -p 8081:8081 -v /var/www/scylla:/var/www/scylla --name scylla wildcat/scylla:latest

Install directly via pip

pip install scylla scylla --help scylla # Run the crawler and web server for JSON API

Install from source

git clone https://github.com/imWildCat/scylla.gitcd scylla pip install -r requirements.txt npm install # or yarn installmake build-assetspython -m scylla

Usage

This is an example of running a service locally (localhost), using port 8899.

Note: You might have to wait for 1 to 2 minutes in order to get some proxy ips populated in the database for the first time you use Scylla.

JSON API

Proxy IP List

http://localhost:8899/api/v1/proxies

Optional URL parameters:

ParametersDefault valueDescriptionpage1The page numberlimit20The number of proxies shown on each pageanonymousanyShow anonymous proxies or not. Possible values：true, only anonymous proxies; false, only transparent proxieshttpsanyShow HTTPS proxies or not. Possible values：true, only HTTPS proxies; false, only HTTP proxiescountriesNoneFilter proxies for specific countries. Format example: US, or multi-countries: US,GB

Sample result:

{ "proxies": [{ "id": 599, "ip": "91.229.222.163", "port": 53281, "is_valid": true, "created_at": 1527590947, "updated_at": 1527593751, "latency": 23.0, "stability": 0.1, "is_anonymous": true, "is_https": true, "attempts": 1, "https_attempts": 0, "location": "54.0451,-0.8053", "organization": "AS57099 Boundless Networks Limited", "region": "England", "country": "GB", "city": "Malton" }, { "id": 75, "ip": "75.151.213.85", "port": 8080, "is_valid": true, "created_at": 1527590676, "updated_at": 1527593702, "latency": 268.0, "stability": 0.3, "is_anonymous": true, "is_https": true, "attempts": 1, "https_attempts": 0, "location": "32.3706,-90.1755", "organization": "AS7922 Comcast Cable Communications, LLC", "region": "Mississippi", "country": "US", "city": "Jackson" }, ... ], "count": 1025, "per_page": 20, "page": 1, "total_page": 52}

System Statistics

http://localhost:8899/api/v1/stats

Sample result:

{ "median": 181.2566407083, "valid_count": 1780, "total_count": 9528, "mean": 174.3290085201}

HTTP Forward Proxy Server

By default, Scylla will start a HTTP Forward Proxy Server on port 8081. This server will select one proxy updated recently from the database and it will be used for forward proxy. Whenever an HTTP request comes, the proxy server will select a proxy randomly.

Note: HTTPS requests are not supported at present.

The example for curl using this proxy server is shown below:

curl http://api.ipify.org -x http://127.0.0.1:8081

You could also use this feature with requests:

requests.get('http://api.ipify.org', proxies={'http': 'http://127.0.0.1:8081'})

Web UI

Open http://localhost:8899 in your browser to see the Web UI of this project.

Proxy IP List

http://localhost:8899/

Screenshot:

Globally Geographical Distribution Map

http://localhost:8899/#/geo