| View previous topic :: View next topic |
| Author |
Message |
Whill
Dark Lord of the Jedi (Owner/Admin)
Joined: 14 Apr 2008
Posts: 10402
Location: Columbus, Ohio, USA, Earth, The Solar System, The Milky Way Galaxy
|
 Posted: Sat Jan 08, 2022 10:04 pm Post subject: Site Outages / Call for Assistance |
 |
|
Lately, you may have experienced one or more service outages lasting up to a few minutes at the Rancor Pit. I have been informed by my web hosting company that it is because the Rancor Pit has become so massive with pages that some web spider bots crawling the site are using a lot of resources, so the host temporarily throttles the service.
Some bots are good, like the ones that crawl for the major search engines that keep Rancor Pit web search results current. Those use minor resources and aren't actually the problem. Others are just website analytic services that crawl the entire web so they have the data should you elect to subscribe to their services. I don't.
The below bots are the main offenders here. From my research so far, none of these are important to the Rancor Pit. I was shown how to block IPs from the entire domain, but I need help. I am still recovering from surgery. I am in pain, very tired, and have minor dizzy spells. In anyone could find out the range of IP addresses these bots use, please report it to me (with a web page source for the info) so that I may go in and block them all. This should eliminate the throttling. Thank you.
AhrefsBot
SemrushBot
MJ12bot
YandexBot
DotBot
_________________
*
Site Map
Forum Guidelines
Registration/Log-In Help
The Rancor Pit Library
Star Wars D6 Damage |
|
| Back to top |
|
 |
cheshire
Arbiter-General (Moderator)
Joined: 04 Jan 2004
Posts: 4849
|
| Posted: Sun Jan 09, 2022 3:20 pm Post subject: Re: Site Outages / Call for Assistance |
|
|
51.222.152.133
54.36.148.1
195.154.122.
46.229.168.0/24
The best I can figure is that this one uses a dynamic IP range, and most recommend using robots.txt to disallow.
FROM TO
100.43.64.0-100.43.79.255
100.43.80.0-100.43.80.255
100.43.81.0-100.43.81.255
100.43.82.0-100.43.82.255
100.43.83.0-100.43.83.255
This one again uses a dynamic IP range, and it looks like they'll tell you how to exclude them in the robots.txt file.
https://moz.com/help/moz-procedures/crawlers/dotbot
_________________
__________________________________
Before we take any of this too seriously, just remember that in the middle episode a little rubber puppet moves a spaceship with his mind. |
|
| Back to top |
|
|
Whill
Dark Lord of the Jedi (Owner/Admin)
Joined: 14 Apr 2008
Posts: 10402
Location: Columbus, Ohio, USA, Earth, The Solar System, The Milky Way Galaxy
|
|
| Back to top |
|
|
cheshire
Arbiter-General (Moderator)
Joined: 04 Jan 2004
Posts: 4849
|
| Posted: Sun Jan 16, 2022 4:19 pm Post subject: |
|
|
I'm imagining you in a lineup of droids saying,
"What I really need is a droid that understands the programing language of php."
Though seriously, I've always outsourced the person who does our robots.txt, but I know a gal who can probably give some pointers. I'll be having a sit down chat with her in a couple of weeks, I think. I'll see if I can bring this up.
_________________
__________________________________
Before we take any of this too seriously, just remember that in the middle episode a little rubber puppet moves a spaceship with his mind. |
|
| Back to top |
|
|
|
