Comments on: An Updated robots.txt File for Magento 2 Stores https://www.lexiconn.com/blog/2016/05/an-updated-robots-txt-file-for-magento-2-stores/ All about e-Commerce, ShopSite, and Web Hosting Thu, 16 Dec 2021 19:59:25 +0000 hourly 1 http://wordpress.org/?v=4.2.1 By: Emil Shamloo https://www.lexiconn.com/blog/2016/05/an-updated-robots-txt-file-for-magento-2-stores/#comment-605980 Fri, 18 May 2018 09:26:05 +0000 http://www.lexiconn.com/blog/?p=8478#comment-605980 Hi, recently I’ve got a lot of pub/static/…. files 404, not found by google!
Should I disallow /pub for crawling my site?
Regards

]]> By: Robert Mangiafico https://www.lexiconn.com/blog/2016/05/an-updated-robots-txt-file-for-magento-2-stores/#comment-584796 Fri, 23 Jun 2017 21:36:33 +0000 http://www.lexiconn.com/blog/?p=8478#comment-584796 Looks like Magento 2 is re-writing the URL, or the file no longer exists where you expect it to be. I’d double check via ssh/ftp that sitemap/sitemap.xml physically exists.

]]> By: alli https://www.lexiconn.com/blog/2016/05/an-updated-robots-txt-file-for-magento-2-stores/#comment-584795 Fri, 23 Jun 2017 20:54:48 +0000 http://www.lexiconn.com/blog/?p=8478#comment-584795 Here is what I am seeing

My actual sitemap file that is submitted:
Sitemap: /sitemap/sitemap.xml
This Sitemap was submitted Oct 8, 2013, and processed Jan 27, 2017.
and it points to this URL >> which is our canada website (setup as a sub folder I think under our main default US website) so this seems really strange.
http://www.heididausdesigns.com/canada/catalogsearch/result/?f=site+xml&q=xml

Our 2 errors and dates are below:
June 19 We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit. example URL restricted by robots.txt

Jan 27 Your Sitemap appears to be an HTML page. Please use a supported sitemap format instead.

Any insight you have would be awesome, thank you!

]]> By: Robert Mangiafico https://www.lexiconn.com/blog/2016/05/an-updated-robots-txt-file-for-magento-2-stores/#comment-584792 Fri, 23 Jun 2017 20:47:42 +0000 http://www.lexiconn.com/blog/?p=8478#comment-584792 Does your sitemap URL have a query string in it? I don’t see anything that would prevent it in the list above.

]]> By: alli https://www.lexiconn.com/blog/2016/05/an-updated-robots-txt-file-for-magento-2-stores/#comment-584695 Thu, 22 Jun 2017 18:54:44 +0000 http://www.lexiconn.com/blog/?p=8478#comment-584695 Our Google Webmaster tool is throwing us errors while trying to access our sitemap: it says URL restricted by robots.txt

This is our current Robots.txt file, is there an unnecessary in here that could cause this issue? Thank you so much in advance!
User-agent: *
Disallow: /index.php/
Disallow: /*?
Disallow: /checkout/
Disallow: /app/
Disallow: /lib/
Disallow: /*.php$
Disallow: /pkginfo/
Disallow: /report/
Disallow: /var/
Disallow: /catalog/
Disallow: /customer/
Disallow: /sendfriend/
Disallow: /review/
Disallow: /*SID=

]]> By: Robert Mangiafico https://www.lexiconn.com/blog/2016/05/an-updated-robots-txt-file-for-magento-2-stores/#comment-570326 Wed, 28 Sep 2016 18:23:40 +0000 http://www.lexiconn.com/blog/?p=8478#comment-570326 Yes, very true. The rules handle both cases, as many people just install to root and not /pub, which is the more secure way to deploy Magento 2. Thanks for the feedback!

]]> By: Robert Egginton https://www.lexiconn.com/blog/2016/05/an-updated-robots-txt-file-for-magento-2-stores/#comment-570312 Wed, 28 Sep 2016 11:26:03 +0000 http://www.lexiconn.com/blog/?p=8478#comment-570312 It’s worth noting that this assumes that the root of the site is the whole Magento 2 installation. This is not recommended. The root of the public site should be the pub/ folder. In which case quite a bit of these restrictions are not needed e.g. app/ and vendor/

]]> By: Steve https://www.lexiconn.com/blog/2016/05/an-updated-robots-txt-file-for-magento-2-stores/#comment-564758 Wed, 25 May 2016 11:55:13 +0000 http://www.lexiconn.com/blog/?p=8478#comment-564758 I really appreciate that you have embraced Magento 2 with your constant learning. This along with your other information is quite useful. I just updated my old robots.txt with this new one.

]]>