Why Your Website Is Slowing Down (And It’s Probably Not What You Think)

Apr 25, 2026

Like0 Likes

0 Comments

Share

• facebook
• linkedin

Why Your Website Is Slowing Down (And It’s Probably Not What You Think)

If your website has recently started slowing down, becoming unstable, or even dropping offline without a clear reason, you are not alone. We are seeing this more and more across VPS and dedicated environments. In many cases, the issue isn’t the server, the code, or the hosting platform. It comes down to traffic. More specifically, automated traffic.

If your mind went straight to AI or AI Bots you are definitely on the right track in this case! Over the past 12 to 18 months, there has been a noticeable shift in how websites are accessed. Traditional search engine crawlers are no longer the only automated visitors. AI-driven tools, data scrapers, and aggressive crawlers are now hitting websites far more frequently, and often far more aggressively.

What This Looks Like in the Real World

Before assuming you need more CPU or RAM, it’s worth checking for some common patterns.

You may be dealing with bot-driven load if you are seeing:

• Spikes in bandwidth or traffic with no increase in enquiries or sales
• Multiple IP addresses hitting the same pages at the same time
• Requests for pages or URLs that don’t exist
• Periods of slowness that come and go without explanation
• Memory or resource issues on an otherwise stable server

Individually, these can be easy to dismiss. Together, they usually point to automated traffic rather than real users.

Why This Is Happening

The way websites are accessed has changed. Search engines used to crawl in a predictable and controlled way. Now, a growing number of systems are trying to analyse, index, and extract data from websites at a much deeper level. Some of these systems behave properly. Others do not. The end result is the same. More requests, more load, and more pressure on your server. The common denominator here is that your site ends up offline, or even worse, both the website and the server it is on goes offline. Then to top it off, the absolute worse case is that the server fails to come back online until technical support can help get it back up and running again. The good news is there is always a way to fix the problem. It is just going to take a few minor, yet significant changes that you may not have setup or implemented in your site previously.

Start With What You Can Control

There is no single fix, but there are practical steps you can take that make a noticeable difference. The first cab of the rank so to speak is, create or clean up your robots.txt file. The robots.txt file is still relevant in todays world and probably more important than ever. It allows you to tell well-behaved crawlers and bots what they should and should not access.

A properly structured file helps by:

1. Reducing unnecessary crawling
2. Guiding legitimate bots to the right content
3. Preventing access to areas that don’t need to be indexed

It won’t stop everything, but it does reduce load from compliant crawlers. Another great benefit of having a robots.txt file is that you may be able to get a few extra brownie points with the search engines by having one. Knowing all to well, some people reading this wont have heard of or ever seen the contents of a robots.txt file. To help with that, I have attached for you an example robots.txt file below. In fact this is the contents of one from a live site I put together a few months back for a friend of mine:

# Allow AI search and agent use
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: FirecrawlAgent
User-agent: AndiBot
User-agent: ExaBot
User-agent: PhindBot
User-agent: YouBot
Allow: /

# Allow traditional search indexing
User-agent: Googlebot
User-agent: YandexBot
User-agent: Applebot
User-agent: Bingbot
User-agent: DuckDuckBot
User-agent: Baiduspider
User-agent: facebot
User-agent: facebookexternalhit/1.0
User-agent: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
User-agent: facebookexternalhit/1.1
User-agent: Swiftbot
User-agent: Slurp
User-agent: GoogleOther
User-agent: Google-InspectionTool
User-agent: LinkedInBot
User-agent: Twitterbot
User-agent: Pinterestbot
User-agent: sogou spider
User-agent: CCbot
User-agent: Yeti
User-agent: Screaming Frog SEO Spider
User-agent: Lumar, DeepCrawl
User-agent: MJ12Bot
User-agent: facebot
Allow: /

# Disallow AI training data collection
User-agent: GPTBot
User-agent: CCBot
User-agent: Google-Extended
Disallow: /

User-agent: *
Disallow: /cgi-bin/
Allow: /

sitemap: https://www.yourwebsiteurl.com.au/sitemap.xml

Every website, as I mentioned, the exact contents will change. You may have other files or folders that you want to make sure remain hidden or not indexed in search engines and alike. On another quick note, not all bots or crawlers will pay attention to the above contents in the robots.txt file but most reputable and/or honest ones will look for this file and respect what you have instructed them to do.

Now it is time to introduce a new friend… The “llms.txt” File

This is something many websites still do not have in place. An llms.txt file is designed to give AI-driven systems a clear view of your site. Instead of crawling everything, they are directed to key pages and important content. In simple terms, it helps avoid unnecessary requests. Without it, bots will attempt to work everything out themselves, which often means hitting more pages than they need to. With it, you reduce the amount of work they need to do, which in turn reduces load on your server.

LLMS is short for Large Language Models. While this is a bit of a side note, “AI” systems in the marketplace, as we know them are actually large language models and not true AI (a debate and topic for another day). The aim of the llms.txt file is to help the large language models and AI crawlers browsing your website/web app server get a clear of understanding, indexing and use of the content within your website/web app more efficiently. It is basically the “cheat sheet” for AI that highlights important, up to date and authoritative content on your site. This stops them from aimlessly wondering through all of the HTML on your site trying to put the pieces together.

Creating a LLMS.TXT file is something that has to be done manually as it website specific. Below is an example of what an llms.txt file would potentially look like:

– [VPS Cloud Hosting](https://www.vpsblocks.com.au/Default.aspx): Reliable Australian VPS hosting with advanced features and support.
– [VPS Network Features](https://www.vpsblocks.com.au/FeaturesNetwork.aspx): Explore VPS network features including redundancy and DDOS protection.

While it is a very simple structure, it can take a bit of time to create one. There is some free generators available online which will help you generate a llms.txt file. They can be found by completing a search, to save you a bit of time I have included a couple of the popular ones currently available:

1. Firecrawl LLMS Generator – Using the one from Firecrawl enables you to create a fairly indepth one without too much effort. It may require some adjustments on your server to get it to work but generally people have reported that it works well and without needing too much done to generate the llms.txt file.
2. llmstxtgenerator.org – As the url suggests, it is a llmstxtgenerator.org

The free options above do work well and are designed to get you started. They wont necessarily work if you have a larger site (like ours). Good news for those of you who fall in to that category, but still don’t want to create your own manually. There are paid options available from both of the sites (and many others of course) where they ultimately end up doing the heavy lifting for you. Once you have implemented the llms.txt file this will allow the AI’s visiting your website review the contents without necessarily having to load or attempt to load each and every file it can find.

Accept That Some Bots Ignore the Rules

Not every crawler will follow instructions. Some will ignore both robots.txt and llms.txt completely. This is where you need to move beyond passive control and start actively blocking unwanted traffic. For a lot of people this may not be so hard to understand or believe but there is actually some bots that think they are above every other bot. There is however a few practice protection options that you can work with to help deliver the push back they so badly deserve. They include the following options:

Fail2Ban

Fail2Ban monitors your server logs and blocks IP addresses that show signs of abuse. While commonly used to prevent brute force attacks, it can also be configured to block:

• Repeated requests to non-existent pages
• High-frequency access patterns
• Known scanning behaviour

It works well at the server level and can stop problems early. If setup correctly it can be extremely effective however it can have a negative impact if it is not configured and more importantly, tested properly.

Web Application Firewalls

A Web Application Firewall (WAF) sits in front of your server and filters traffic before it reaches you. Providers such as Cloudflare, Sucuri, and Imperva offer:

• IP reputation filtering
• Behaviour-based blocking
• Automated protection against common threats

This reduces the load hitting your infrastructure and helps maintain stability. There are also other WAF’s available in the marketplace that are just as cost effective or even better yet, free. Once again, make sure it is tested extensively before putting it on to your production/live environment. If you do not have your WAF configured properly, it can be disastrous for traffic to your website.

IP Reputation Services

Services like Project Honey Pot provide lists of known malicious IP addresses. Using these allows you to block a large portion of unwanted traffic before it becomes a problem.

The Key Takeaways

If your website is slowing down, the first instinct is often to increase resources. In some cases, that is not the issue. The real problem is the type of traffic hitting your server. Understanding that, and putting basic controls in place, will often deliver better results than simply upgrading your infrastructure. Implementing some of the basics we have mentioned

What We Are Seeing Across VPSBlocks

We regularly see environments that are technically sound but struggling under unnecessary load. Once bot traffic is identified and controlled, performance returns to normal without needing additional resources. This is not a rare scenario anymore. It is becoming the standard and keeping a head of this trend will enable you to keep the websites you host online for longer with less issue. It will likely even help you keep your site a head of your compet

Final Thought

Traffic has changed. That is the reality. If your site is not prepared for how modern crawlers behave, you will feel it in performance and uptime becoming downtime. The good news is that this is manageable with the right approach. It will seem like a massive burden for a few of you who host a larger number of websites but honestly, when you keep up with the changes and being able to help mitigate the wrong time of traffic with adjustments like the ones covered in this article, you start to realise it is not as bad as you may have first thought.

Neil has been working in the industry for approximately 25 years and joined VPSBlocks in 2024. With over years of technical expertise, Neil has been instrumental in helping businesses grow. Neil's extensive skill set encompasses server management, programming, process optimization, and internet marketing. His dedication to quality and customer satisfaction has been acknowledged by clients who have benefited from his expertise and support.

Neil