Every day, thousands of resources are accessed on the web, either opened in browsers by website visitors or crawled by spiders. Each time this happens, the server that stores the resources receives a request to send back data. One way to speed up this process is to use cache controls such as If-Modified-Since and Last-Modified. When the server is additionally configured using cache controls, this process is made easier and faster because depending on whether the data has changed over time, or not, the server will be able to determine whether it needs to waste time and send the data again or not. By improving the efficiency of the client-server communication, you are also improving the performance of your website when it comes to crawling and indexing. The same cache controls that apply for server-browser communication, also apply when web spiders crawl your webpages. The easier and faster it is for the spiders to go through your site, the better it will be indexed. And that greatly influences and improves your SEO ranking.
What are If-Modified-Since and Last-Modified Headers? The If-Modified-Since and Last-Modified headers, are a pair of cache controls called a “conditional GET” request, and they define if the response to the GET will be different depending on whether the page has changed or not. If-Modified-Since and Last-Modified headers request and response HTTP headers, respectively. Their main use is to help improve applications’ performance and at the same time, save bandwidth. If-Modified-Since is a request header which is being sent to a server when a page needs to be opened in a browser (or crawled by the Googlebot) so the browser can learn when the resource was last modified on the server side. The responding, Last-Modified header indicates the time when the resource was last modified on the server and tells the browser whether it can use its previously cached copy or if it needs to download a new version of the resource. Last-modified response header contains the date and time at which the origin server thinks the resource has been last modified. When the cache stores an entry which includes the last-modified header, it can use it to query the server if the content has changed over time. Basically, it is used as a validation of whether a resource received and stored, is still the same. And this is done using the If-Modified-Since request header. How They Work? Let’s explore a simplified example of how and where these two headers are applied:
What’s important here is that depending on the implementation of the origin server and the nature of the original resource, the exact meaning of the Last-Modified header field may be different. For entities with dynamically included parts, it may be the most recent of the set of last-modify times for its component parts. For database gateways, it may be the last-update time stamp of the record. For virtual objects, it may be the last time the internal state changed. Additionally, the origin server must not send a last-modified response where the date is then the time when the server’s t message originates. In that case, where the resource’s last modification indicates some time in the future, the server must replace the response date, with the message origination date. When the origin server sends the last-modified response, it should obtain the value as close as possible to the time when it generates the Date value of the response. That will help the recipient to make a precise assessment of the entity’s modification time, and this is especially important if the entity changed close to the time when the response is generated. Why are If-Modified and Last-Modified headers important for SEO? The main purpose for using the cache control If-Modified-Since and Last-Modified headers is to allow efficient updating of the cached information while using a minimum amount of transaction overhead. These headers will help improve page speed and load times and improve the overall website performance and user experience. In terms of technical SEO, cache controls are very important because they improve the crawling and indexing of your website. When you are optimizing your website for crawling, try to think of the search engine crawler as web proxy cache that is trying to pre-fetch your website, and create a temporarily stored version of the pages in the index. Also, remember that there is a limited crawl budget dedicated to your site, and your goal when optimizing for SEO is to make the most of it. That’s why it is excellent if you can save your precious crawling budget and help search engines crawl the pages more efficiently and index them better by telling the spiders whether they need to go over certain pages and index them again, or not. This is especially true if you have a large website, and you change or add more content frequently. There’s certainly no need for Google to re-crawl a page that hasn’t changed. But if your pages aren’t tagged correctly, or if they don’t use cached headers, your site might not be crawled correctly. For example, your “About Us” page may be crawled more often because it’s linked in the footer of each page, which sends a signal to Googlebot that it’s an important page, when in fact it isn’t, especially when it comes to crawling and SEO ranking. So instead of outing a “noindex” tag, because you still want the page indexed, you can fix this problem by using cached headers to tell web spiders that the content of this page hasn’t changed and there’s no need to send new download requests to the server. Now, that the spiders know they can use the already indexed copy of the page, they can continue crawling on your new and more relevant pages. Another problem the often arises for people with large and complex websites when it comes to crawling, and can be solved by using If-Modified-Since and Last-Modified headers, is the increased consumption of bandwidth, which means extra cost. By using cache control headers and limiting the unnecessary crawling, you will be able to save bandwidth and ease the crawling process. If you manage to reduce the number of server request, and if you use cached content, you can greatly influence and increase the website speed. And we all know that website speed and page load time are among the most important user-experience factors. And happy visitors are good for your SEO ranking. If-Modified-Since/Last-Modified plugins for WordPress If you have a WordPress website, then it is very easy to improve the functionality and performance of your site by using some of the over 49 000 plugins that WordPress offers to its users. And thousands more are available from third-party websites like Github. The plugins are small software apps which integrate and run on top of the WordPress software, and allow you to create any kind of website, and simply improve the site’s functionality. If you are satisfied with your site’s design, and want to improve its SEO and boost performance, there are certainly plugins for that too. One of the easiest ways to utilize the power of caching is by installing a plugin to your WordPress website that will handle the server-side caching performed by your WordPress server. The top three cache plugins you could install include:
Or, you can simply go with some of the If Modified Since and Last Modified plugins that WordPress has to offer too. Conclusion Having your webpages’ perform better is very important for improving the end-user’s experience and your SEO ranking. It makes no sense to have the browser download a webpage again, or have Googlebot repeatedly go over pages in which there’s no change in the content. Caching your webpages saves a lot of time when it comes to both, opening your website in a browser, or having Googlebot crawl it. The implementation of the If-Modified-Since and Last-Modified headers translates into better SEO results. It will stop Google from going over the same, unchanged pages of your site, save your crawling budget, and lower the amount of bandwidth spent for crawling. Additionally, it will improve the experience for your first-time as well as repeating visitors by shortening page load times. With a little effort, you can significantly influence your website’s performance and SEO ranking. And whether you implement this cache control manually, or decide to use a plugin, don’t hesitate. It can only do good for your website and SEO efforts.