Scheduling Python Scrapy on Ubuntu AWS

Asked Nov 19 '15 at 12:25

Active Nov 19 '15 at 12:25

Viewed 487 times

I have a script written with the scrapy framework on python.

I can successfully run the script by running scrapy crawl mycrawler and it runs in the console.

I can also successfully run the script via a shell script sh job.sh with scrapy crawl mycrawler.

However when it comes to CRON and scheduling the task I cannot get it to complete.

I have tried everything here Scrapy crawler in Cron job which includes multiple things about the path etc etc.

I can successfully run the job.sh from cron and can debug that starts/finishes but just gives no output for the scrapy lines at all.

Sure I am missing something simple, help much appreciated!

edited May 23 '17 at 10:27

Community

asked Nov 19 '15 at 12:25

Matt The Ninja

Why don't you run it in *Scrapy Cloud* from Scrapinghub? See: http://scrapinghub.com/scrapy-cloud/ – Andrés Pérez-Albela H. Nov 19 '15 at 12:44
Thats awesome, can you host that yourself? Can I send it to external DB from there? If not I need the answer above :P – Matt The Ninja Nov 19 '15 at 12:45
Of course! It's a great PaaS (Platform as a Service). Give it a try! – Andrés Pérez-Albela H. Nov 19 '15 at 12:46
1

Cool platform, keep up the good work :) – Matt The Ninja Nov 19 '15 at 14:55

0 Answers0