Bye Bye bing

User avatar
AUbicycles
Site Admin
Site Admin
Posts: 15583
Joined: Tue Aug 23, 2005 2:14 am
Location: Sydney & Frankfurt
Contact:

Bye Bye bing

Postby AUbicycles » Fri Apr 20, 2018 6:22 am

Guess what - I think I finally managed to get bing to stop overloading this site.

They have, what could be described as, an unethical bot. This crawls the site and saves it for their search engine. Yes, bing is a search engine from microsoft. The problem was, it ignore the search engine instructions and furthermore would frequently overload the site. When it visited, the site slows down for everyone else as bing was going overboard.

Though I finally got a contact partner and although they wanted me to make the site vulnerable and share user data (which I would never do), some persistence and escalations meant that they finally agrees to stop targeting.

The result is that the website is for the community who wont need to compete against a naughty bot that doesn't add any value to anyone.


If this is all a bit cryptic, don't worry, it juts feels like something to be happy about.
Cycling is in my BNA

Arbuckle23
Posts: 1133
Joined: Sun Jan 11, 2015 7:07 pm
Location: Mornington Peninsula

Re: Bye Bye bing

Postby Arbuckle23 » Fri Apr 20, 2018 7:21 am

Well done

b4igo
Posts: 60
Joined: Sun Oct 12, 2008 4:15 pm
Location: Sydney

Re: Bye Bye bing

Postby b4igo » Fri Apr 20, 2018 7:32 am

Nice work! Appreciate all your effort to keep the site running smoothly for us all!!

User avatar
ValleyForge
Posts: 1831
Joined: Tue Jan 27, 2009 5:37 pm
Location: Brisbane

Re: Bye Bye bing

Postby ValleyForge » Fri Apr 20, 2018 9:50 pm

We need more bots to protect us from the bots.
Ha ha ha! Cookies on dowels.

User avatar
10speedsemiracer
Posts: 4904
Joined: Sat Feb 11, 2017 7:38 pm
Location: Back on the Tools .. when I'm not in the office

Re: Bye Bye bing

Postby 10speedsemiracer » Fri Apr 20, 2018 10:21 pm

Well done Chris, many thanks.
Campagnolo for show, SunTour for go

User avatar
nickobec
Posts: 2271
Joined: Fri Mar 27, 2009 12:51 am
Location: Perth or 42km south as the singlespeed flies
Contact:

Re: Bye Bye bing

Postby nickobec » Sat Apr 21, 2018 11:59 am

Had a similar problem a couple years back, found out the hard way trying to slow bingbot down that if msnbot and bingbot find a rule specific to itself it will ignore all other rules:

So my robots.txt now start with
User-agent: msnbot
Disallow: /
User-agent: bingbot
Disallow: /

User avatar
MichaelB
Posts: 14775
Joined: Tue Feb 20, 2007 9:29 am
Location: Adelaide, South Australia

Re: Bye Bye bing

Postby MichaelB » Sat Apr 21, 2018 5:49 pm

You spoke to someone !!??????

User avatar
trailgumby
Posts: 15469
Joined: Sat Jan 03, 2009 5:30 pm
Location: Northern Beaches, Sydney
Contact:

Re: Bye Bye bing

Postby trailgumby » Sat Apr 21, 2018 6:32 pm

MichaelB wrote:You spoke to someone !!??????
Probably a robot.

User avatar
Thoglette
Posts: 6605
Joined: Thu Feb 19, 2009 1:01 pm

Re: Bye Bye bing

Postby Thoglette » Sat Apr 21, 2018 7:54 pm

Well done. MS should have better manners.
Stop handing them the stick! - Dave Moulton
"People are worthy of respect, ideas are not." Peter Ellerton, UQ

User avatar
AUbicycles
Site Admin
Site Admin
Posts: 15583
Joined: Tue Aug 23, 2005 2:14 am
Location: Sydney & Frankfurt
Contact:

Re: Bye Bye bing

Postby AUbicycles » Mon Apr 23, 2018 4:07 pm

Bingbot was ignoring their specified crawl rates and disallows in robots.txt . Eventually I had to block their IPs but when they were still responsible for up to 85% of traffic when their waves started, even that is still a load.
Cycling is in my BNA

User avatar
nickobec
Posts: 2271
Joined: Fri Mar 27, 2009 12:51 am
Location: Perth or 42km south as the singlespeed flies
Contact:

Re: Bye Bye bing

Postby nickobec » Mon Apr 23, 2018 4:22 pm

AUbicycles wrote:Bingbot was ignoring their specified crawl rates and disallows in robots.txt . Eventually I had to block their IPs but when they were still responsible for up to 85% of traffic when their waves started, even that is still a load.
My problem was trying to slow their crawl rates, with a bingbot specific line in robots.txt

Which meant bingbot actually obeyed that rule, but ignored all the other rules in robots.txt

Which meant bingbot would crash the server by accessing huge number of pages normally blocked by robots.txt

So I removed the bingbot specific lines and just blocked bingbot and msnbot with
User-agent: msnbot
Disallow: /
User-agent: bingbot
Disallow: /

Have not had any issues since, and that was years ago

User avatar
AUbicycles
Site Admin
Site Admin
Posts: 15583
Joined: Tue Aug 23, 2005 2:14 am
Location: Sydney & Frankfurt
Contact:

Re: Bye Bye bing

Postby AUbicycles » Tue Apr 24, 2018 2:10 am

I went so far, in frustration, to sign up to their webmaster tools and follow their guidance and set up the site but even that didn’t improve.

Very specifically, I used their tools to force them to ignore specific directories and their error was (to the effect) “these pages are too valuable to bing so can’t be removed”
Cycling is in my BNA

eeksll
Posts: 2631
Joined: Tue Oct 20, 2009 10:36 pm

Re: Bye Bye bing

Postby eeksll » Sun Apr 29, 2018 1:43 pm

just interested,

was this something they did on their end or did you also have to do something?

User avatar
AUbicycles
Site Admin
Site Admin
Posts: 15583
Joined: Tue Aug 23, 2005 2:14 am
Location: Sydney & Frankfurt
Contact:

Re: Bye Bye bing

Postby AUbicycles » Mon Apr 30, 2018 2:39 am

At their end- they wanted me to open up the site and make it vulnerable to their flooding but I refused. Contact partner was fast in responding and polite.
Cycling is in my BNA

Jmuzz
Posts: 631
Joined: Mon Jan 29, 2018 1:42 pm

Re: Bye Bye bing

Postby Jmuzz » Fri May 04, 2018 11:17 pm

Doesn't phpBB recognise all the bots as logged in users, and therefore let you assign them restricted permissions?
Or have they gone and defeated that by going stealth these days?

Shred11
Posts: 206
Joined: Sun May 18, 2014 12:32 pm
Location: Launceston

Re: Bye Bye bing

Postby Shred11 » Sat May 05, 2018 8:56 am

Does anyone actually use Bing anyway? Once in a blue moon, I resort to trying Bing and it just throws up all sorts of rubbish search results that don't help at all. duckduckgo.com isn't a bad alternative if Google isn't giving good results.

User avatar
10speedsemiracer
Posts: 4904
Joined: Sat Feb 11, 2017 7:38 pm
Location: Back on the Tools .. when I'm not in the office

Re: Bye Bye bing

Postby 10speedsemiracer » Sat May 05, 2018 4:12 pm

I'm over Google. Duckduckgo is my preference.
Campagnolo for show, SunTour for go

User avatar
mikesbytes
Super Mod
Super Mod
Posts: 22159
Joined: Thu Dec 01, 2005 11:42 pm
Location: Tempe, Sydney
Contact:

Re: Bye Bye bing

Postby mikesbytes » Sat May 05, 2018 4:52 pm

Bing is popular in China, I'm sure there will be other demographic's where it has a client base
If the R-1 rule is broken, what happens to N+1?

User avatar
AUbicycles
Site Admin
Site Admin
Posts: 15583
Joined: Tue Aug 23, 2005 2:14 am
Location: Sydney & Frankfurt
Contact:

Re: Bye Bye bing

Postby AUbicycles » Tue May 08, 2018 6:59 am

phpbb by default can do some recognition, but doesn't actively identify and control or block. The other things is that bots can spoof - while bing wasn't spoofing, it wasn't ethical either.

Does anyone use it? well microsoft forced it on people and for those who couldn't bypass, it was used. But it was never really very important even though microsoft tried to compete and, as always will push it as far as they can. As Mikebytes suggests, there are probably some demographics who rely on it but Google was always a step ahead for quality results.
Cycling is in my BNA

User avatar
kb
Posts: 2570
Joined: Sun Nov 13, 2011 3:22 pm

Re: Bye Bye bing

Postby kb » Tue May 08, 2018 12:51 pm

I do like the use of quadtrees for mapping though.
Image

User avatar
mikesbytes
Super Mod
Super Mod
Posts: 22159
Joined: Thu Dec 01, 2005 11:42 pm
Location: Tempe, Sydney
Contact:

Re: Bye Bye bing

Postby mikesbytes » Wed May 09, 2018 8:26 pm

I suspect there are few searches done on Bing to find Australian bicycle info. A quick surf of the web has uncovered that 14% of searches in the US and Canada are done using Bing and that 85% of Bing's total searches are done by US users, however I think that statement has excluded China

https://www.aborg.com/2014/08/google-vs ... ers-using/

Most of the people using this site reside within Australia and the following page states that 98% of searches within Australia are thru Google, Yahoo and NineMSM. BUT what is underneath them? go to https://www.nine.com.au/?redirect=holdingpage and have a look at who's search it is.

BTW the remaining 2% includes specialty engines such as web wombat.
If the R-1 rule is broken, what happens to N+1?

Philistine
Posts: 501
Joined: Mon Apr 18, 2016 1:57 pm

Re: Bye Bye bing

Postby Philistine » Sun Sep 30, 2018 12:10 pm

I don't know if this is the right thread to bring this up, but here goes anyway....

I am a great believer in free speech, which includes comments that I might make that other people might not want to hear, and comments other people might make that I don't want to hear. If someone puts up a bad argument, the correct response should be to put up a better argument refuting it, rather than shutting down the person making the bad argument. I have been horrified by recent reports from the USA about Google, Facebook, etc. shadow banning and otherwise censoring people that they disagree with. I am not on Facebook and unlikely to be as long as my fundamental orifice points to the ground, but I have been using Google as a search engine. I thought it was time I voted with my feet (actually with my keyboard and mouse).

I tried Bing, then Yahoo, and found that I could not access some sites - including this one - that I use regularly. I used to type in "bna" and the forum would come up, but, when I tried this on Yahoo and Bing, I got literally dozens of sites that had the initials "bna" in their name, but which were nothing remotely to do with the cycling forum. I eventually tried Duckduckgo (horrible name, quite decent search engine), and it also gave me dozens of sites that didn't interest me, but it also gave me one which reads (paraphrasing) Google search of BNA Australian Cycling Forum, which gets me here with one extra mouse click. Several other sites that I visit are similarly labeled Google search of ***********.

Does this mean that, despite my best efforts to get rid of Google, it is refusing to go quietly? Should I bow down to my new Google overlord?

User avatar
Thoglette
Posts: 6605
Joined: Thu Feb 19, 2009 1:01 pm

Re: Bye Bye bing

Postby Thoglette » Sun Sep 30, 2018 3:22 pm

Philistine wrote: Does this mean that, despite my best efforts to get rid of Google, it is refusing to go quietly? Should I bow down to my new Google overlord?
No, you should use DuckDuckGo for the majority of searches, and use Google when nothing else works.

And the Google competitors need to stop behaving badly. For example, there's absolutely no excuse for Bing misbehaving. It's not like Microsoft is a start up with no experience in server management.bbb
Stop handing them the stick! - Dave Moulton
"People are worthy of respect, ideas are not." Peter Ellerton, UQ

User avatar
10speedsemiracer
Posts: 4904
Joined: Sat Feb 11, 2017 7:38 pm
Location: Back on the Tools .. when I'm not in the office

Re: Bye Bye bing

Postby 10speedsemiracer » Sun Sep 30, 2018 4:40 pm

Thoglette wrote: No, you should use DuckDuckGo for the majority of searches, and use Google when nothing else works....
+1
Campagnolo for show, SunTour for go

User avatar
mikesbytes
Super Mod
Super Mod
Posts: 22159
Joined: Thu Dec 01, 2005 11:42 pm
Location: Tempe, Sydney
Contact:

Re: Bye Bye bing

Postby mikesbytes » Sun Sep 30, 2018 5:08 pm

The problem is that search engines are extremely heavy on the forum's resources when it gathering the information it uses within its search engine, hence why it had to be excluded. IE nothing to do with censorship
If the R-1 rule is broken, what happens to N+1?

Who is online

Users browsing this forum: No registered users