Google Adds New Googlebot Crawlers: GoogleOther-Image and GoogleOther-Video

Intro

Google has introduced two new web crawlers to the Googlebot family:

• GoogleOther-Image

• GoogleOther-Video

These new crawlers are designed to enhance the crawling of binary data, which Google states will support research and development.

Introduction to the New Crawlers

The two new crawlers, GoogleOther-Image and GoogleOther-Video, are optimized versions of GoogleOther for fetching image and video bytes respectively. GoogleOther was initially added in April 2023 to be used internally by Google teams to crawl the public web, thereby freeing up resources for the main Googlebot crawlers.

GoogleOther-Image

GoogleOther-Image is the version of GoogleOther tailored for fetching publicly accessible image URLs. According to the documentation, it will operate under the user agent tokens of GoogleOther-Image and GoogleOther. The full user agent string will be GoogleOther-Image/1.0.

GoogleOther-Video

GoogleOther-Video is the version of GoogleOther designed for fetching publicly accessible video URLs. Similar to GoogleOther-Image, it will use the user agent tokens of GoogleOther-Video and GoogleOther. The full user agent string will be GoogleOther-Video/1.0.

Purpose of the New Crawlers

Google launched these new crawlers to better support the crawling of binary data that may be used for research and development. This initiative aims to improve Google's ability to handle image and video data efficiently.

Overview of Google Crawlers

Google utilizes various types of crawlers, including:

• Googlebot: The main crawler for Google’s search products, which always respects robots.txt rules.

• Special-case crawlers: These crawlers perform specific functions (such as AdsBot) and may or may not respect robots.txt rules.

• User-triggered fetchers: Tools and product functions where the end-user triggers a fetch. For instance, Google Site Verifier or some Google Search Console tools will send Google to fetch the page based on a user action.

Google IP Address Ranges and DNS Masks

Google has listed IP address ranges and reverse DNS masks for each type of crawler:

• Googlebot: googlebot.json (crawl-–––.googlebot.com or geo-crawl-–––.geo.googlebot.com)

• Special-case crawlers: special-crawlers.json (rate-limited-proxy-–––.google.com)

• User-triggered fetchers: user-triggered-fetchers.json (–––.gae.googleusercontent.com)

Why This Matters

For those monitoring crawling activities and bot activity on their websites, it's essential to recognize these new GoogleOther crawlers as legitimate Googlebots. When you see activity from these crawlers in your log files, you can be assured that it is genuine and part of Google's efforts to improve their search and data handling capabilities.