Scrub kodu Python betiğinde argümanları kabul edemiyor

Question

Bir Python betiğinde Scrapy çalıştırmayı deniyorum. Ben yönergeleri biraz here ve here izleyin çalışılıyor

import scrapy 
from scrapy.crawler import CrawlerProcess 

class PostSpider(scrapy.Spider): 
    name = "post crawler" 
    allowed_domains = ['test.com'] 

    def __init__(self, **kwargs): 
     super(PostSpider, self).__init__(**kwargs) 

     url = kwargs.get('url') 
     print(url) 
     self.start_urls = ['https://www.test.com/wp-json/test/2.0/posts' + url] 

    def parse(self, response): 
     post = json.loads(response.body_as_unicode()) 
     post = post["content"] 
     return post 

posts = GA.retrieve(TIA.start_date, TIA.end_date, "content type auto") 

process = CrawlerProcess({ 
    'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)' 
}) 

for post in posts: 
    post_url = post[2] 
    process.crawl(PostSpider(url=post_url)) 
    process.start() 

: İlgili kod İşte. Ancak, onu işe alamıyorum. Aldığım hata mesajı:

Unhandled error in Deferred: 
2016-03-25 20:49:43 [twisted] CRITICAL: Unhandled error in Deferred: 


Traceback (most recent call last): 
    File "text_analysis.py", line 48, in <module> 
    process.crawl(PostSpider(url=post_url)) 
    File "/Users/terence/TIA/lib/python3.5/site-packages/scrapy/crawler.py", line 163, in crawl 
    return self._crawl(crawler, *args, **kwargs) 
    File "/Users/terence/TIA/lib/python3.5/site-packages/scrapy/crawler.py", line 167, in _crawl 
    d = crawler.crawl(*args, **kwargs) 
    File "/Users/terence/TIA/lib/python3.5/site-packages/twisted/internet/defer.py", line 1274, in unwindGenerator 
    return _inlineCallbacks(None, gen, Deferred()) 
--- <exception caught here> --- 
    File "/Users/terence/TIA/lib/python3.5/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks 
    result = g.send(result) 
    File "/Users/terence/TIA/lib/python3.5/site-packages/scrapy/crawler.py", line 71, in crawl 
    self.spider = self._create_spider(*args, **kwargs) 
    File "/Users/terence/TIA/lib/python3.5/site-packages/scrapy/crawler.py", line 94, in _create_spider 
    return self.spidercls.from_crawler(self, *args, **kwargs) 
    File "/Users/terence/TIA/lib/python3.5/site-packages/scrapy/spiders/__init__.py", line 50, in from_crawler 
    spider = cls(*args, **kwargs) 
    File "text_analysis.py", line 17, in __init__ 
    self.start_urls = ['https://www.techinasia.com/wp-json/techinasia/2.0/posts' + url] 
builtins.TypeError: Can't convert 'NoneType' object to str implicitly 
2016-03-25 20:49:43 [twisted] CRITICAL: 
/xiaomi-still-got-it-bitches 
2016-03-25 20:49:43 [scrapy] INFO: Enabled extensions: 
['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.logstats.LogStats'] 

Neyin yanlış olduğunu anlayamıyorum.

Answer 1

CrawlerProcess.crawl bir Spidersınıfı, bir Spider örneği bekler.

Böyle argümanları geçmelidir:

process.crawl(PostSpider, url=post_url) 

Answer 2

process.crawl() çağrısı definition olarak

process.crawl(PostSpider, url=post_url) 

tarama (crawler_or_spidercls olduğunu olmalı * args ** kwargs)

Örümcek sınıfını (örneklenen nesne değil) ilk argüman olarak bekler. Aşağıdaki tüm konumsal ve anahtar kelime argümanları (*args, **kwargs) örümcek init işlevine iletilir.