Robots.txt tip from Bing: Include all relevant directives if you have a Bingbot section

January 3, 2019

robots-txt-tip-from-bing-include-all-relevant-directives-if-you-have-a-bingbot-section Theme Builder Layout

Frédéric Dubut, a senior program manager at Microsoft working on Bing Search, said on Twitter Wednesday that when you create a specific section in your robots.txt file for its Bingbot crawler, you should make sure to list all the default directives in that section.

Specify directives for Bingbot. “If you create a section for Bingbot specifically, all the default directives will be ignored (except Crawl-Delay),” he said. “You MUST copy-paste the directives you want Bingbot to follow under its own section,” he added.

What does it mean? This probably means that Bing has seen a number of sites complain that Bingbot is crawling areas of their websites that they do not want to be crawled. It’s likely some webmasters assumed that if they gave Bingbot some specific instructions, it would follow the rest of the default directives not listed. Instead, if you have a section for Bingbot, it will only follow the directives you’ve specifically listed in that section of your robots.txt file. If you do not have a specific section for Bingbot, then Bingbot will follow the default directives.

Why it matters. Make sure that when you set up your robots.txt file, that all the search engine crawlers can efficiently crawl your site. If you set up specific directives for blocking, crawl delays or other directives, then make sure that all the search engine crawlers are listening to those directives. They may not listen if there are syntax issues, if you do not follow their protocols or they have issues accessing such directives.

For more on setting up a robots.txt for Bing, see the help documents.

Share this article:

eHost-square-ad Theme Builder Layout

We’re listening.

Have something to say about this article? Share it with us on Facebook, Twitter or LinkedIn:


Related Posts

smx-overtime-managing-your-online-business-reviews Theme Builder Layout

SMX Overtime: Managing your online business reviews

Last month I spoke at SMX East about scaling online reputation management – specifically reviews. At the end of my presentation, “14 Tips to Scale Reviews Across Multiple Locations,” I fielded a number of fantastic questions from session attendees and wanted to...

Get ALL Your SEO, WordPress & Divi News

Join Our Daily Roundup

SEO News and More

Subscribe ToThe Weekly SEO Trade News Updates

Get the latest SEO, SEM and SMM marketing intel, tips and tricks from one of the best SEO Gurus online. 

Every Tuesday morning we send out an aggregated email listing all new posts on SEO Trade News.

Excellent! Now check your email to confirm your subscription.