Current Visitors: Robots

[Voluntarist]

Can somebody bring me up to date about this? If you go to https://ronpaulforums.com/online/ you'll see a bunch of robots listed:
- Robot: Bing
- Robot: Google
- Robot: Swordy ... just kidding
- Robot: Trendiction
- Robot: Yandex

Most seem to look at forum posts, but I noticed Tendiction looking over user profiles. These web crawlers can evidently be disabled with the robots.txt file. So I'm wondering why they haven't - particularly the Russian bot Yandex. Maybe they're paying for information??

 

[Brian4Liberty]

Can somebody bring me up to date about this? If you go to https://ronpaulforums.com/online/ you'll see a bunch of robots listed:
- Robot: Bing
- Robot: Google
- Robot: Swordy ... just kidding
- Robot: Trendiction
- Robot: Yandex

Most seem to look at forum posts, but I noticed Tendiction looking over user profiles. These web crawlers can evidently be disabled with the robots.txt file. So I'm wondering why they haven't - particularly the Russian bot Yandex. Maybe they're paying for information??

Not my area of expertise, but those are all web search engine robots. So if we want internet searches to bring back results from this forum, then we have to allow them to scan the forum.

 

[ClaytonB]

Can somebody bring me up to date about this? If you go to https://ronpaulforums.com/online/ you'll see a bunch of robots listed:
- Robot: Bing
- Robot: Google
- Robot: Swordy ... just kidding
- Robot: Trendiction
- Robot: Yandex

Most seem to look at forum posts, but I noticed Tendiction looking over user profiles. These web crawlers can evidently be disabled with the robots.txt file. So I'm wondering why they haven't - particularly the Russian bot Yandex. Maybe they're paying for information??

I use Yandex constantly to defeat the Western-hemisphere censorship. It's not until you try a search on Yandex and compare it to ANY other Western-based search engine that you will realize how much we are being censored. We are under levels of censorship that would have made the Stasi blush for shame. The entire Internet on this side of the "Iron Curtain" (we're the ones behind it now) is a 1984 Memory Hole.