[kepler@localhost beauty_worker]$ scrapy-py35 shell http://www.epocacosmeticos.com.br/future-solution-lx-total-radiance-foundation-shiseido-base-facial/p 2016-04-26 02:31:13 [scrapy] INFO: Scrapy 1.1.0rc3 started (bot: beauty_worker) 2016-04-26 02:31:13 [scrapy] INFO: Overridden settings: {'BOT_NAME': 'beauty_worker', 'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter', 'SPIDER_MODULES': ['beauty_worker.spiders'], 'LOGSTATS_INTERVAL': 0, 'ROBOTSTXT_OBEY': True, 'AUTOTHROTTLE_MAX_DELAY': 20, 'NEWSPIDER_MODULE': 'beauty_worker.spiders', 'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0', 'AUTOTHROTTLE_ENABLED': True, 'AUTOTHROTTLE_START_DELAY': 1} 2016-04-26 02:31:13 [scrapy] INFO: Enabled extensions: ['scrapy.extensions.throttle.AutoThrottle', 'scrapy.extensions.corestats.CoreStats'] 2016-04-26 02:31:13 [scrapy] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2016-04-26 02:31:13 [scrapy] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2016-04-26 02:31:13 [scrapy] INFO: Enabled item pipelines: ['scrapy_redis.pipelines.RedisPipeline'] 2016-04-26 02:31:13 [epocacosmeticos.com.br] -> DEBUG: Reading URLs from redis list 'epocacosmeticos.com.br:start_urls' 2016-04-26 02:31:13 [scrapy.core.engine] -> INFO: Spider opened 2016-04-26 02:31:13 [scrapy.core.engine] -> DEBUG: Crawled (200) (referer: None) 2016-04-26 02:31:14 [scrapy.core.engine] -> DEBUG: Crawled (200) (referer: None) 2016-04-26 02:31:15 [root] -> DEBUG: Using default logger 2016-04-26 02:31:15 [root] -> DEBUG: Using default logger [s] Available Scrapy objects: [s] crawler [s] item {} [s] request [s] response <200 http://www.epocacosmeticos.com.br/future-solution-lx-total-radiance-foundation-shiseido-base-facial/p> [s] settings [s] spider [s] Useful shortcuts: [s] shelp() Shell help (print this help) [s] fetch(req_or_url) Fetch request (or URL) and update local objects [s] view(response) View response in a browser