Robots (Crawlers) List for UBB.threads - UPDATED 2022-07-10 formerly named "Search Engine Spiders List for UBB.threads"
About: The advantage to using this list is that Robots (Crawlers and Search Engine Spiders) get put into the correct "Robots (Crawlers)" group when viewing your forum's Who's Online page.
This translation list is not used anywhere else in UBB.threads except on the Who's Online page at /ubbthreads.php/online. Having a long list of Robots will not slow down your forums.
How to install: 1) Go to Control Panel > Display Options > Who's Online Settings. 2) Copy/paste the text from the newest list in to the "Robots (Crawlers)" box at the bottom of the page. 3) Click the "Update General Display Options" button. done.
Notes: Always use the newest list. Older lists contain old robots and Inactive crawlers. Installs prior to 7.6.2 did not include a list of robot agents. • robots_20141014-UBBT762.txt Fresh Installs of UBB.threads 7.6.2 to 7.7.3 are pre-populated with this list. • robots_20200114-UBBT774.txt Fresh Installs of UBB.threads 7.7.4 are pre-populated with this list. • robots_20200924-UBBT775.txt Fresh Installs of UBB.threads 7.7.5 are pre-populated with this list. • robots_20220710-UBBT800.txt Fresh Installs of UBB.threads 8.0.0+ are pre-populated with this list. • robots_20220710-CATEGORIES.txt is the same robots list as the stock list, plus it includes a category name for many of the robots, such as Search, Marketing, Monitoring, Link Checker, Tool, etc.
Having problems using this list on an older version of UBB.threads? Remove any blank lines from the top/bottom of your copied list.
Last edited by isaac; 07/10/202210:41 AM. Reason: updated lists for 20220710
Although the source information for you to create your own updates and conversions is in the OP, I plan on updating this post every couple of months, making your job as an UBBT forum admin much easier.
--- EDIT: user-agent-string.info no longer provides a list of user agent strings without a subscription. this means that until another resource for this data is found, this list will remain as it currently is.
In my "anonymous" list, I see a number of IPs like this: 157.55.39.xxx. Hovering over the "i" icon it shows: Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
The list above has bingbot entries, and I added one from an older list, so I have this in my search engine agents list:
If an unregistered user arrives at your site through a search using bing, they will be clasified as anonymous. if your site is being crawled by bingbot (not a live person), it should be classified and shown within the spider section.
Sounds like you have visitors finding your site through Bing. This is good!
Sounds like you have visitors finding your site through Bing. This is good!
I am pretty sure that is not the case. My Who's Online shows these anonymous guests: 157.55.39.120 157.55.39.9 157.55.39.232 157.55.39.218 157.55.39.231 157.55.39.224 Each one shows bingbot when I hover over the "i" icon, and the Referrer: part is blank.
Well, if the referrer listing is blank, there isn't much you can do; as the WoL system just parses the spider data based on what is being supplied by bots IN the referrer variable.
Well, if the referrer listing is blank, there isn't much you can do; as the WoL system just parses the spider data based on what is being supplied by bots IN the referrer variable.
The referrer listing is blank on all the Search Spider entries, and is blank on all the bingbot "guest" entries. So I don't understand ho the referrer being blank causes it to show up in the guest area.
I do appreciate your taking the time to post in this dialog. I am mostly a "grasshopper" here.
I talked with Isaac last night and evidently when the user is viewing a "cached result" from Bing there is no referrer variable passed as it's not "BingBot", but a user that's requesting data through a Bing server.
You're giving Bing waaay too much credit. Right now, I have 11 guests, 5 search spiders, and one user. Normal for this small forum with very little activity at night.
Of the 11 guests, 5 are bingbot. And 3 of them are walking this silly thread that has 99 pages. Long continuing threads confound the search spiders. Every time there is a new post, they spend a long time walking through every page in the sequence of pages.
Well, not really giving them too much credit; they force SSL for all queries now (source), so it could really be either incoming users from bing are coming in on their SSL (which is the default) or they're coming in from the cache.
SteveS, are you confident your Amazon stuff is not related caching of content within your site by AWS Cloud Computing /Route 53? https://aws.amazon.com/
The IP 72.21.217.n has been used as a proxy for a User Agent of MSIE-6, which is in itself highly deprecated. Headers can also be consistent with either a battened-down proxy or a bot.
When I try to add the list, I paste them, click save, and I get a 403 forbidden page.
But when I do something else on the page, click save, I don't.
Its your host. they are censoring the content that you send. theyre basically blocking you from typing words/phrases they do not agree with... for "security" lol
edit - If it was an initial install of ubb.threads that you did, its included as the changelog says. If this was just an upgrade you did, the robots.txt will be what came with your original version PLUS any customizations that you had done to it. ie;the content in this post is for you.
Isaac, Any chance we can get this list updated? I have a feeling the number of robots / crawlers has exploded due to all the AI floating around. My whois online list a lot more visitors than I would expect (real humans).
While we're waiting on a possible update, I took a stab at doing it myself. I took the robots / crawlers seen in the last year from the udger.com website; robots / crawlers seen since the last robot file update from the github.com website; combined them with the latest robot file, sorted and removed duplicates. Update attached. Hope this helps.
Donate to UBBDev today to help aid in Operational, Server and Script Maintenance, and Development costs.
Please also see our parent organization VNC Web Services if you're in the need of a new UBB.threads Install or Upgrade, Site/Server Migrations, or Security and Coding Services.
