Thread Like Summary
Gizmo, isaac, Morgan
Total Likes: 9
Original Post (Thread Starter)
by isaac
isaac
Robots (Crawlers) List for UBB.threads - UPDATED 2022-07-10
formerly named "Search Engine Spiders List for UBB.threads"

About:
The advantage to using this list is that Robots (Crawlers and Search Engine Spiders) get put into the correct "Robots (Crawlers)" group when viewing your forum's Who's Online page.

This translation list is not used anywhere else in UBB.threads except on the Who's Online page at /ubbthreads.php/online. Having a long list of Robots will not slow down your forums.

Details:
Robots List: 2022-07-10
Total Agent Strings: 912
Sources:
https://udger.com/resources/ua-list/crawlers (2022-07-10) and
https://github.com/monperrus/crawler-user-agents (2022-05-03)
I've simply converted, combined, and cleaned the data for use with UBB.threads.

How to install:
1) Go to Control Panel > Display Options > Who's Online Settings.
2) Copy/paste the text from the newest list in to the "Robots (Crawlers)" box at the bottom of the page.
3) Click the "Update General Display Options" button.
done.

Notes:
Always use the newest list. Older lists contain old robots and Inactive crawlers.
Installs prior to 7.6.2 did not include a list of robot agents.
robots_20141014-UBBT762.txt Fresh Installs of UBB.threads 7.6.2 to 7.7.3 are pre-populated with this list.
robots_20200114-UBBT774.txt Fresh Installs of UBB.threads 7.7.4 are pre-populated with this list.
robots_20200924-UBBT775.txt Fresh Installs of UBB.threads 7.7.5 are pre-populated with this list.
robots_20220710-UBBT800.txt Fresh Installs of UBB.threads 8.0.0+ are pre-populated with this list.
robots_20220710-CATEGORIES.txt is the same robots list as the stock list, plus it includes a category name for many of the robots, such as Search, Marketing, Monitoring, Link Checker, Tool, etc.

Having problems using this list on an older version of UBB.threads? Remove any blank lines from the top/bottom of your copied list.
Liked Replies
by isaac
isaac
Although the source information for you to create your own updates and conversions is in the OP, I plan on updating this post every couple of months, making your job as an UBBT forum admin much easier.

---
EDIT: user-agent-string.info no longer provides a list of user agent strings without a subscription. this means that until another resource for this data is found, this list will remain as it currently is.
1 member likes this
by isaac
isaac
This list has been updated 2014-10-14.
1 member likes this
by isaac
isaac
Some further reading regarding http/https referer data:
https://yoast.com/web-https/
1 member likes this
by isaac
isaac
SteveS, are you confident your Amazon stuff is not related caching of content within your site by AWS Cloud Computing /Route 53? https://aws.amazon.com/

Additional reading at:
https://en.wikipedia.org/wiki/Amazon_Route_53

The IP 72.21.217.n has been used as a proxy for a User Agent of MSIE-6, which is in itself highly deprecated. Headers can also be consistent with either a battened-down proxy or a bot.

Additional reading at:
"amazonaws.com plays host to wide variety of bad bots"
https://www.webmasterworld.com/search_engine_spiders/3828718.htm
1 member likes this
by isaac
isaac
Changelog 2020-01-14
The Robots (Crawlers) List for UBB.threads has been updated in OP
1 member likes this
by isaac
isaac
Changelog 2020-09-24
The Robots (Crawlers) List for UBB.threads has been updated in OP
1 member likes this
by isaac
isaac
This list has been updated 2022-07-09.
1 member likes this
Donate Today!
Donate via PayPal

Donate to UBBDev today to help aid in Operational, Server and Script Maintenance, and Development costs.

Please also see our parent organization VNC Web Services if you're in the need of a new UBB.threads Install or Upgrade, Site/Server Migrations, or Security and Coding Services.
Recommended Hosts
We have personally worked with and recommend the following Web Hosts:
Stable Host
bluehost
InterServer
Visit us on Facebook
Member Spotlight
Bill B
Bill B
Issaquah, WA
Posts: 87
Joined: December 2001
Forum Statistics
Forums63
Topics37,573
Posts293,925
Members13,849
Most Online5,166
Sep 15th, 2019
Today's Statistics
Currently Online
Topics Created
Posts Made
Users Online
Birthdays
Top Posters
AllenAyres 21,079
JoshPet 10,369
LK 7,394
Lord Dexter 6,708
Gizmo 5,833
Greg Hard 4,625
Top Posters(30 Days)
Top Likes Received
isaac 82
Gizmo 20
Brett 7
WebGuy 2
Morgan 2
Top Likes Received (30 Days)
None yet
The UBB.Developers Network (UBB.Dev/Threads.Dev) is ©2000-2024 VNC Web Services

 
Powered by UBB.threads™ PHP Forum Software 8.0.0
(Preview build 20240506)