RuiJi.Net.Core 1.2.2

crawler framework , distributed crawler extractor.
try ruiji scraper --- chrome web crawler
https://chrome.google.com/webstore/detail/ruiji-scraper/klhahkhllngppofpkjdlbmnglnmnbbol?hl=zh-CN&authuser=0

Install-Package RuiJi.Net.Core -Version 1.2.2
dotnet add package RuiJi.Net.Core --version 1.2.2
<PackageReference Include="RuiJi.Net.Core" Version="1.2.2" />
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add RuiJi.Net.Core --version 1.2.2
The NuGet Team does not provide support for this client. Please contact its maintainers for support.
#r "nuget: RuiJi.Net.Core, 1.2.2"
#r directive can be used in F# Interactive, C# scripting and .NET Interactive. Copy this into the interactive tool or source code of the script to reference the package.
// Install RuiJi.Net.Core as a Cake Addin
#addin nuget:?package=RuiJi.Net.Core&version=1.2.2

// Install RuiJi.Net.Core as a Cake Tool
#tool nuget:?package=RuiJi.Net.Core&version=1.2.2
The NuGet Team does not provide support for this client. Please contact its maintainers for support.

RuiJi.Net Documentation http://doc.ruijihg.com/

Fork me on github https://github.com/zhupingqi/RuiJi.Net

company web http://www.iotsys.net

RuiJi.Net.Core Sample crawl use local ip automatic

var crawler = new RuiJiCrawler(); var request = new Request("https://www.baidu.com"); var response = crawler.Request(request); crawl with special ip

var crawler = new RuiJiCrawler(); var request = new Request("https://www.baidu.com"); request.Ip = "192.168.31.196"; var response = crawler.Request(request); crawl with proxy

var crawler = new RuiJiCrawler(); var request = new Request("https://www.baidu.com"); request.Proxy = new RequestProxy("223.93.172.248", 3128);

var response = crawler.Request(request); extract url

var crawler = new RuiJiCrawler(); var request = new Request("https://www.oschina.net/blog");

var response = crawler.Request(request); var content = response.Data.ToString();

var parser = new RuiJiParser(); var eb = parser.ParseExtract("css a.blog-title-link[href]\nexp https://my.oschina.net//blog/"); var result = RuiJiExtractor.Extract(content, eb.Block); extract tile

var crawler = new RuiJiCrawler(); var request = new Request("http://www.ruijihg.com/archives/category/tech/bigdata");

var response = crawler.Request(request); var content = response.Data.ToString();

var parser = new RuiJiParser(); var eb = parser.ParseExtract(@"[tile]\ncss article:html

[meta] #title css .entry-header:text

#summary css .entry-header + p:text ex /Read more »/ -e");

var result = RuiJiExtractor.Extract(content, eb.Block); extract meta

var crawler = new RuiJiCrawler(); var request = new Request("https://my.oschina.net/zhupingqi/blog/1826317");

var response = crawler.Request(request); var content = response.Data.ToString();

var parser = new RuiJiParser(); var eb = parser.ParseExtract(@"[meta] #title css h1.header:text

#author css div.blog-meta .avatar + span:text

#date css div.blog-meta > div.item:first:text regS /发布于/ 1

#words_i css div.blog-meta > div.item:eq(1):text regS / / 1

#content css #articleContent:html");

var result = RuiJiExtractor.Extract(content, eb.Block); detect mine

var crawler = new RuiJiCrawler(); var request = new Request("http://img10.jiuxian.com/2018/0111/cd51bb851410404388155b3ec2c505cf4.jpg"); var response = crawler.Request(request);

var ex = response.Extensions; run js

var request = new Request("https://gitee.com/zhupingqi/RuiJi.Net"); request.RunJS = true;

var crawler = new RuiJiCrawler();

var response = crawler.Request(request); cookie

var request = new Request("https://gitee.com/zhupingqi/RuiJi.Net"); request.Cookie = "xxxxxx";

var crawler = new RuiJiCrawler();

var response = crawler.Request(request); More Feature please visit my github

https://github.com/zhupingqi

RuiJi.Net Cluster is waitting for you RuiJi.Net is a dotnet distributed crawler framework written in c#.Major features include distribute crawler, distribute Extractor and managed cookie, support ip polling that using the server public network address and proxy server.

RuiJi.Net has extract model called RuiJi Expression,It divides the web page into block,tile and meta. you can extract web page by RuiJi Expression and save the expression with text file or database.

RuiJi.Net have more features including extract rule match by url wildcard and page feature, paging extract, url function, cookie manager and cookie channel,much selecors to clear data, ...

If you like , please star my project, It will give me more motivation to improve this project.

Javascript support If you want to run html page with javascript, please install chromium.

如果你想要使用ChromiumCrawler等待页面运行JS,请安装chromium。参考以下教程。

目前chromium版本号为555668

下载Ruiji.Net部署操作系统对应的zip

在编译后的运行目录里建立以下路径

编译后的运行目录.local-chromium{操作系统}-555668\

{操作系统}分别对应:

Linux64位系统:Linux

Windows32位系统:Win

Windows64位系统:Win64

Mac系统:Mac

将下载好的zip解压到创建好的目录中。

以linux为例 chromium所在路径为 运行目录.local-chromium\Linux-555668\chrome-linux

网盘地址:https://pan.baidu.com/s/1rsyCNnXxbobCBLZuPTiJHQ cr3e

RuiJi.Net Documentation http://doc.ruijihg.com/

Fork me on github https://github.com/zhupingqi/RuiJi.Net

company web http://www.iotsys.net

RuiJi.Net.Core Sample crawl use local ip automatic

var crawler = new RuiJiCrawler(); var request = new Request("https://www.baidu.com"); var response = crawler.Request(request); crawl with special ip

var crawler = new RuiJiCrawler(); var request = new Request("https://www.baidu.com"); request.Ip = "192.168.31.196"; var response = crawler.Request(request); crawl with proxy

var crawler = new RuiJiCrawler(); var request = new Request("https://www.baidu.com"); request.Proxy = new RequestProxy("223.93.172.248", 3128);

var response = crawler.Request(request); extract url

var crawler = new RuiJiCrawler(); var request = new Request("https://www.oschina.net/blog");

var response = crawler.Request(request); var content = response.Data.ToString();

var parser = new RuiJiParser(); var eb = parser.ParseExtract("css a.blog-title-link[href]\nexp https://my.oschina.net//blog/"); var result = RuiJiExtractor.Extract(content, eb.Block); extract tile

var crawler = new RuiJiCrawler(); var request = new Request("http://www.ruijihg.com/archives/category/tech/bigdata");

var response = crawler.Request(request); var content = response.Data.ToString();

var parser = new RuiJiParser(); var eb = parser.ParseExtract(@"[tile]\ncss article:html

[meta] #title css .entry-header:text

#summary css .entry-header + p:text ex /Read more »/ -e");

var result = RuiJiExtractor.Extract(content, eb.Block); extract meta

var crawler = new RuiJiCrawler(); var request = new Request("https://my.oschina.net/zhupingqi/blog/1826317");

var response = crawler.Request(request); var content = response.Data.ToString();

var parser = new RuiJiParser(); var eb = parser.ParseExtract(@"[meta] #title css h1.header:text

#author css div.blog-meta .avatar + span:text

#date css div.blog-meta > div.item:first:text regS /发布于/ 1

#words_i css div.blog-meta > div.item:eq(1):text regS / / 1

#content css #articleContent:html");

var result = RuiJiExtractor.Extract(content, eb.Block); detect mine

var crawler = new RuiJiCrawler(); var request = new Request("http://img10.jiuxian.com/2018/0111/cd51bb851410404388155b3ec2c505cf4.jpg"); var response = crawler.Request(request);

var ex = response.Extensions; run js

var request = new Request("https://gitee.com/zhupingqi/RuiJi.Net"); request.RunJS = true;

var crawler = new RuiJiCrawler();

var response = crawler.Request(request); cookie

var request = new Request("https://gitee.com/zhupingqi/RuiJi.Net"); request.Cookie = "xxxxxx";

var crawler = new RuiJiCrawler();

var response = crawler.Request(request); More Feature please visit my github

https://github.com/zhupingqi

RuiJi.Net Cluster is waitting for you RuiJi.Net is a dotnet distributed crawler framework written in c#.Major features include distribute crawler, distribute Extractor and managed cookie, support ip polling that using the server public network address and proxy server.

RuiJi.Net has extract model called RuiJi Expression,It divides the web page into block,tile and meta. you can extract web page by RuiJi Expression and save the expression with text file or database.

RuiJi.Net have more features including extract rule match by url wildcard and page feature, paging extract, url function, cookie manager and cookie channel,much selecors to clear data, ...

If you like , please star my project, It will give me more motivation to improve this project.

Javascript support If you want to run html page with javascript, please install chromium.

如果你想要使用ChromiumCrawler等待页面运行JS,请安装chromium。参考以下教程。

目前chromium版本号为555668

下载Ruiji.Net部署操作系统对应的zip

在编译后的运行目录里建立以下路径

编译后的运行目录.local-chromium{操作系统}-555668\

{操作系统}分别对应:

Linux64位系统:Linux

Windows32位系统:Win

Windows64位系统:Win64

Mac系统:Mac

将下载好的zip解压到创建好的目录中。

以linux为例 chromium所在路径为 运行目录.local-chromium\Linux-555668\chrome-linux

网盘地址:https://pan.baidu.com/s/1rsyCNnXxbobCBLZuPTiJHQ cr3e

Release Notes

url wildcard match

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version History

Version Downloads Last updated
1.2.2 191 8/21/2020
1.2.1 284 8/21/2020
1.2.0 225 6/22/2020
1.1.9 191 6/22/2020
1.1.8 196 6/22/2020
1.1.7 503 10/8/2018
1.1.6 433 10/6/2018
1.1.5 438 10/5/2018
1.1.4 445 10/4/2018
1.1.3 415 9/25/2018
1.1.1 424 9/25/2018
1.1.0 412 9/20/2018
1.0.6-beta 443 7/30/2018
1.0.5-beta 510 7/17/2018
1.0.4-beta 463 7/7/2018
1.0.3-beta 501 7/4/2018
1.0.2-beta 535 6/25/2018
1.0.1-beta 485 6/22/2018