• 如果您觉得本站非常有看点,那么赶紧使用Ctrl+D 收藏吧

python scrapy爬虫框架概念介绍(个人理解总结为一张图)

python wangting 1年前 (2018-09-10) 2037次浏览 31个评论
pythonscrapy是一个为了爬取网站数据,提取结构性数据而编写的应用框架
pythonscrapy的安装就不介绍了,资料很多
这里我个人总结一下,能更加快理解scrapy和快速上手一个简单的爬虫程序

首先开始一个scrapy项目

用命令: scrapy startproject 项目名
创建出来的文件如下图:红框是我的命令,蓝框是scrapy自动创建的文件
python scrapy爬虫框架概念介绍(个人理解总结为一张图)

其中各个最常用文件的用处:

mytestproject
–items.py 定义spider.py到pipelines.py中间的数据格式
–pipelines.py 处理爬取到的数据
–settings.py 配置文件,定义有几个pipelines以及他们的优先级,
–spider 爬虫目录
—-spider.py 爬取哪些页面, 如何解析爬取的到的数据,整理数据返回给pipelines.py
我画了一张简单的图介绍一下各个文件之间的关系:
一个url地址对应的数据从开始爬取–>处理结果–>入库(或写文件等) 分别是在哪个文件里做的,如下图:
python scrapy爬虫框架概念介绍(个人理解总结为一张图)
scrapy框架中的各个文件和数据流转大概就是这样,后面我会详细介绍下我爬取zhihu的各个文件的详细代码.
欢迎关注我的公众号,分享面试攻略和技术干货!
python scrapy爬虫框架概念介绍(个人理解总结为一张图)

喜欢 (0)
发表我的评论
取消评论

表情 贴图 加粗 删除线 居中 斜体 签到

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址
(31)个小伙伴在吐槽
  1. Only wanna tell that this is extremely helpful, Thanks for taking your time to write this.
    guzel porno2019-08-26 18:15 回复
  2. thank you admin
    adana porno2019-08-26 18:13 回复
  3. thank you admin
  4. Sorry for the huge review, but I'm really loving the new Zune, and hope this, as well as the excellent reviews some other people have written, will help you decide if it's the right choice for you.
    escort ankara2019-08-26 07:02 回复
  5. thank you web site admin
    micaze sinan2019-08-25 14:36 回复
  6. thank you web sites.
    ivedik porno2019-08-25 14:01 回复
  7. thank you web site admin
    micaze pornolari2019-08-25 13:31 回复
  8. thank you web sites.
    elite pornos2019-08-25 12:28 回复
  9. Chociaż bardzo się staramy, aby wyszukać dokładne ceny, niekiedy nam się to nie udaje. Wypełnij poniższy formularz, aby zaalarmować naszych programistów, którzy będą mogli zbadać problem. Dzięki Twojej pomocy wyniki wyszukiwania staną się lepsze dla wszystkich.
    porn wynajem2019-08-23 21:04 回复
  10. This reminds me of the other comment I was seeing
    past lives2019-08-23 15:18 回复
  11. google hack ın my sites go to my website
    sübyen porno2019-08-23 13:25 回复
  12. This makes me think of the other page here I found earlier
  13. This seems like the other page I was looking at
    my past lives2019-08-23 08:16 回复
  14. This page seems like the other page here I saw earlier
  15. Only wanna tell that this is extremely helpful, Thanks for taking your time to write this.
    kaliteli cici porno2019-08-23 00:16 回复
  16. You made some respectable points there. I seemed on the web for the issue and found most people will go together with along with your website.
    kaliteli cici porno2019-08-22 22:36 回复
  17. This makes me think of the other comment I found earlier
  18. This page makes me think of the other page I found earlier
    past lives2019-08-22 19:54 回复
  19. Only wanna tell that this is extremely helpful, Thanks for taking your time to write this.
    tatol childs2019-07-18 03:53 回复
  20. Thanks a bunch for sharing this with all of us you actually know what you are talking about! Bookmarked. Kindly also visit my website =). We could have a link exchange agreement between us!
    Florentino Bernardo2019-07-16 17:04 回复
  21. thank you web site admin
    top childs2019-07-15 12:35 回复
  22. great put up, very informative. I wonder why the opposite specialists of this sector do not realize this. You must proceed your writing. I am sure, you've a great readers' base already!
    Dale Vlach2019-07-07 04:16 回复
  23. We extremely appreciate your blog post. You can find so many means we could put it to good use with minimum effort with time and resources. Thank you really for helping make this post respond to many queries we have got before now.
    bye oruspi seni2019-07-02 03:36 回复
  24. There are actually a whole lot of details like that to take into consideration. That could be a nice level to convey up. I offer the thoughts above as normal inspiration but clearly there are questions just like the one you convey up the place crucial thing will likely be working in trustworthy good faith. I don?t know if greatest practices have emerged round issues like that, but I'm positive that your job is clearly recognized as a good game. Both girls and boys really feel the influence of only a moment’s pleasure, for the rest of their lives.
    Margarita Castanada2019-06-29 09:01 回复
  25. I needed to post you that tiny observation to say thanks over again regarding the extraordinary pointers you've featured here. It has been seriously open-handed of you to convey publicly exactly what a few people would have advertised as an e book to generate some dough for themselves, notably now that you could possibly have done it if you ever desired. The creative ideas also acted to become a great way to comprehend someone else have the same eagerness the same as my very own to know way more with regards to this condition. I'm certain there are many more fun times ahead for individuals who discover your website.
    Mckinley Prepotente2019-06-28 23:23 回复
  26. Great article and straight to the point. I am not sure if this is in fact the best place to ask but do you guys have any thoughts on where to hire some professional writers? Thank you :)
    Richie Rappley2019-06-28 22:26 回复
  27. Thank you for any other informative site. Where else may I am getting that type of information written in such an ideal way? I have a mission that I am just now running on, and I've been on the glance out for such information.
    Tracey Fiore2019-06-28 20:07 回复
  28. You made some first rate points there. I appeared on the web for the problem and found most people will go along with with your website.
    Brice Strothmann2019-06-28 19:28 回复
  29. I think this is among the most important info for me. And i'm glad reading your article. But wanna remark on few general things, The website style is perfect, the articles is really excellent : D. Good job, cheers
    Karl Lindmeyer2019-06-28 18:29 回复
  30. Hi there would you mind sharing which blog platform you're working with? I'm going to start my own blog in the near future but I'm having a hard time deciding between BlogEngine/Wordpress/B2evolution and Drupal. The reason I ask is because your design and style seems different then most blogs and I'm looking for something completely unique. P.S My apologies for getting off-topic but I had to ask!
    Dalton Marsalis2019-06-28 12:55 回复
  31. Hiya, I'm really glad I've found this info. Nowadays bloggers publish only about gossips and web and this is actually irritating. A good blog with exciting content, that's what I need. Thank you for keeping this web-site, I'll be visiting it. Do you do newsletters? Can't find it.
    Cedrick Krawitz2019-06-28 11:31 回复