Frequency of Scans
Per default, Monsido's web crawler scans your website on a weekly basis, ensuring that you can catch issues and errors promptly, especially as new content is added. This regular scanning frequency provides you with real-time insights into your website's performance and helps maintain its quality. However, if you're working less frequently on the site or prefer a different schedule, you have the flexibility to choose a bi-weekly or monthly scan frequency instead. To explore the available options further, you can refer to the Help Center for more details on customizing your scan frequency.
Now let's take a look at how the scans work.
Links in Monsido
How Monsido Counts Pages
Exclusions and Constraints
- Link Excludes enable you to define patterns that instruct the crawler to ignore specific URLs during the scan. Although these excluded links are still recorded as present on the page, Monsido will not check or follow them. To learn more about Link Excludes and how to implement them effectively, you can refer to our dedicated resource on the Monsido website.
Link Exclude Example 2: External Affiliate Links - Often, websites include external affiliate links that lead visitors to partner websites for specific products or services. Since these links are not part of your website's direct content, scanning them might not add significant value to your analysis. By utilizing Link Excludes, you can easily instruct the crawler to bypass these affiliate links, allowing you to focus on the core content and internal links.
- Path Constraints offer you control over the pages Monsido scans. By using regular expressions, you can include or exclude content from the scan based on specific patterns. Here are a couple of examples to illustrate how Path Constraints work:
Path Constraint Example 2: Let's say you have a result page, and you want to exclude all the results from the scan. The URL of the results page could be something like: https://domain.tld/search/results?query=test. To exclude these result pages, you can create a negative constraint like !search/results?
Example 1: Print Version of a Page: Suppose you have a webpage with the URL https://domain.tld/page_id=32. When a print version of this page is created, CMS systems often add a print parameter within the URL, resulting in a URL like https://domain.tld/page_id=32?print=yes. Although the content on these two pages is essentially the same, web crawlers and search engines treat them as separate pages. To address this, you can add a canonical tag on the print page, pointing to the primary page URL:
<link rel="canonical" href="https://domain.tld/page_id=32">
This canonical tag informs web crawlers and search engines that these pages contain duplicate content, with the URL lacking the print parameter being the primary version.
Example 2: Sortable Lists: Another scenario involves a page that displays a sortable list of items, such as a news site with a list of articles or a store with a list of products. Assume the URL https://domain.tld/list contains a list where users can sort by color, price, or size. Although the content remains the same, each sorted version of the page has a unique URL:
<link rel="canonical" href="https://domain.tld/list">
This canonical link informs search engines and web crawlers, including Monsido, that all these URLs lead to pages with the same content, with the default sort version considered the primary version.
Monsido utilizes canonical tags to exclude URLs pointing to identical content. For directions on how to instruct the scan to ignore canonical URLs, you can refer to the relevant section in the Monsido Broken Links FAQ.
How the Monsido Scan Works
Monsido's crawler also interacts with robots.txt files, allowing it to detect declared sitemaps within the file. Once a sitemap is detected, Monsido automatically scans all the links specified in the sitemap's XML, including any linked PDFs.
Types of Pages in a CMS
Equipped with this knowledge, you can now seize control of your website's analysis and ensure that the Monsido tool provides the accurate metrics and profound insights your organization requires. So, don't hesitate to embark on a journey of exploration, uncovering the endless possibilities that Monsido offers. Unleash your website's full potential and elevate it to unprecedented levels of excellence.
Want to see how your website stacks up for 2023? Get a free scan now to see what we uncover on your website.