Crawler is block if some User-agent are disallowed

[froozeify]

In my robots.txt I have a directive dedicated to disallow the crawl for some specific User-Agent

User-agent: Bytespider
Disallow: /

User-agent: *
Allow: /core/*.css$
....

The SEO tab explain it's due to the robots.txt configuration

If I remove the Bytespider directive the site is well crawled.

Would it possible to have a setting to ignore specific user-agent rules (or even ignore the robots.txt file)

[janreges]

Hi @froozeify,

this bug in robots.txt processing with multiple User-Agents in robots.txt has already been fixed in 9c2c989 and will be published in the next version in the next few days.

Also you can use --ignore-robots-txt flag. This is listed in the README.md, but is missing in the documentation website. It will be added when the next version is released.

[froozeify]

Thank you @janreges for your answer, do you know if in the gui version they'll be a checkbox ? (Or maybe I didn't found it also 😃)