-
Notifications
You must be signed in to change notification settings - Fork 218
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WIP Fix redirection bug. #261
Conversation
Scrapy's request meta holds native string as keys. This caused to a redirected URL being added as seed again.
@rolando I think it's time to remove _request_is_redirected() method and it's call in enqueue_request(). Can you do that? If not, I could fix that. |
@sibiryakov How does it look now? I don't know what was the rationale behind adding non-redirected URLs as seeds. Let me know if it's OK and then I can squash the changes and remove the WIP status. |
Codecov Report
@@ Coverage Diff @@
## master #261 +/- ##
==========================================
- Coverage 70.15% 70.11% -0.05%
==========================================
Files 68 68
Lines 4715 4715
Branches 632 576 -56
==========================================
- Hits 3308 3306 -2
- Misses 1267 1271 +4
+ Partials 140 138 -2
Continue to review full report at Codecov.
|
return True | ||
return False | ||
self._add_pending_request(request) | ||
self.stats_manager.add_redirected_requests() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
that line is wrong, otherwise LGTM.
Could you fix the tests @rolando ? |
Closing in favor #276 |
Scrapy's request meta holds native string as keys. This caused to a
redirected URL being added as seed again.