baidu search engine hammering site

**tkgafs** · 30-10-2011, 19:45

Has anybody else had the baidu search engine hammering their sites

lots of lines similar to the following

180.76.5.90 - - [30/Oct/2011:18:21:34 +0000] "GET /somepage_or_other HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"

After many years hosting on here I have blown my bandwidth allowance

I thought it was caused by a weekly backup which I had inadvertently set to run daily, but on further investigation its this spider from China

I have set the following subnets in my ip disallow

180.76.0.0/16 and 119.63/192.0/21 as they don't seem to pay much attention to robots.txt

Interested to hear if anybody has any other ideas

Tkgafs

**DPS Computing** · 30-10-2011, 19:59

Hmmm, I can't say that I've been experiencing this myself but I'll keep an eye out for it now that you've mentioned it.

You've done the right thing there disallowing the IP range. I guess it could be a temporary thing, I'm not sure why they would decide all of a sudden to start visiting your page so frequently.... :S

**Ben Stones** · 30-10-2011, 20:29

I'll also definitely keep a look out on my website as well. With Google (and presumably Bing as well) I think you can contact them about it to stop visiting/indexing your site as much if it is costing you a lot of bandwidth, but I'm not sure of the procedure for Baidu. That said, I think the best temporary measure is what you did - which is ban the entire IP range to mitigate the issue. Try seeing if you can contact Baidu about it? Though I've never dealt with Baidu so I wouldn't know whether they'd even comply with your request to not index as much or whether they have systems in place to not visit/index your site as much, but it is definitely strange that it suddenly is occurring when it wasn't happening before.

**DPS Computing** · 30-10-2011, 23:02

Originally Posted by Ben

I'll also definitely keep a look out on my website as well. With Google (and presumably Bing as well) I think you can contact them about it to stop visiting/indexing your site as much if it is costing you a lot of bandwidth, but I'm not sure of the procedure for Baidu. That said, I think the best temporary measure is what you did - which is ban the entire IP range to mitigate the issue. Try seeing if you can contact Baidu about it? Though I've never dealt with Baidu so I wouldn't know whether they'd even comply with your request to not index as much or whether they have systems in place to not visit/index your site as much, but it is definitely strange that it suddenly is occurring when it wasn't happening before.

Isn't Baidu that search engine which the Chinese government keeps blocking for various (politically motivated) reasons?

**Ben Stones** · 30-10-2011, 23:10

Originally Posted by DPS Computing

Isn't Baidu that search engine which the Chinese government keeps blocking for various (politically motivated) reasons?

Who knows. I wouldn't be surprised if they have ordered the Baidu search engine to block specific search results or alter them for the benefit of the Chinese government. To be honest, getting into this subject, I think it is a bad thing the Chinese government is getting richer (as the economy is developing well in China) primarily because China has a mainstream manufacturing industry for developed countries around the world, and it is obvious they want to have international influence (or worse - who knows) as does the US enjoy right now. This wouldn't be a problem if they are a democracy but they aren't, which is a major concern in my opinion. They are concerned of uprisings because they want to stay in power and/or want to stay as a communism. I think the international community need to be more alerted to China because they are getting to the stage where China is rapidly becoming developed in major areas of the country, which is a benefit to the Chinese government.

This is my opinion and does not reflect the opinions or views of eUKhost

.

**DPS Computing** · 30-10-2011, 23:22

Originally Posted by Ben

Who knows. I wouldn't be surprised if they have ordered the Baidu search engine to block specific search results or alter them for the benefit of the Chinese government. To be honest, getting into this subject, I think it is a bad thing the Chinese government is getting richer (as the economy is developing well in China) primarily because China has a mainstream manufacturing industry for developed countries around the world, and it is obvious they want to have international influence (or worse - who knows) as does the US enjoy right now. This wouldn't be a problem if they are a democracy but they aren't, which is a major concern in my opinion. They are concerned of uprisings because they want to stay in power and/or want to stay as a communism. I think the international community need to be more alerted to China because they are getting to the stage where China is rapidly becoming developed in major areas of the country, which is a benefit to the Chinese government.

This is my opinion and does not reflect the opinions or views of eUKhost

.

Lol nice disclaimer at the end

.

Yeah, so I'm thinking that there a bit like the Chinese Google and not wanting to bow to ridiculous censorship so maybe you should give a go contacting them. From what I've read in the press they seem like they could be good guys

.

**Ben Stones** · 30-10-2011, 23:24

Originally Posted by DPS Computing

Lol nice disclaimer at the end

.

Yeah, so I'm thinking that there a bit like the Chinese Google and not wanting to bow to ridiculous censorship so maybe you should give a go contacting them. From what I've read in the press they seem like they could be good guys

.

I had to since I work here

- I don't want John moaning at me rofl

. Serious disclaimer this time

.

You mean Baidu are good guys? How?

You mean they comply with webmaster requests? Or because they bow to Chinese government censorship requirements?

**DPS Computing** · 30-10-2011, 23:26

Originally Posted by Ben

I had to since I work here

- I don't want John moaning at me rofl

. Serious disclaimer this time

.

You mean Baidu are good guys? How?

You mean they comply with webmaster requests? Or because they bow to Chinese government censorship requirements?

Lol. No I mean they sound like good reasonable people who aren't in anybodies pockets so they won't bow to censorship and will probably listen to webmasters requests. What I am saying is I don't think it's a rogue site out to cause harm

.

**Ben Stones** · 30-10-2011, 23:57

Originally Posted by DPS Computing

Lol. No I mean they sound like good reasonable people who aren't in anybodies pockets so they won't bow to censorship and will probably listen to webmasters requests. What I am saying is I don't think it's a rogue site out to cause harm

.

Baidu particularly not maybe like many Chinese companies I agree

.

**DPS Computing** · 31-10-2011, 00:03

Originally Posted by Ben

Baidu particularly not maybe like many Chinese companies I agree

.

Yeah, its worth a shot. and then you can report back here to tell us if they listened

.

**MattBarthorp** · 31-10-2011, 14:22

I could find a related article at eUkhosts blog. Please refer : How to block traffic from Particular country . One may block baidu spiders and crawlers on the same grounds.

**DPS Computing** · 31-10-2011, 14:24

Originally Posted by MattBarthorp

I could find a related article at eUkhosts blog. Please refer : How to block traffic from Particular country . One may block baidu spiders and crawlers on the same grounds.

Yes, if it gets really bad you might end up having to block an even wider range or maybe the whole country

.

**Ben Stones** · 01-11-2011, 13:23

Originally Posted by DPS Computing

Yes, if it gets really bad you might end up having to block an even wider range or maybe the whole country

.

lmfao

.

**DPS Computing** · 03-11-2011, 19:14

Originally Posted by Ben

lmfao

.

Or, in the most extreme cases, the whole world

.

**Bud Boy** · 20-12-2011, 19:37

Baidu is absolutely hammering most of my sites and pays no attention to robots.txt

I've tried IP range blocking but it's still going strong! No idea what to do next.

Is there no way I can completely block Baidu from all sites in my reseller account!?

I reckon 90% of my bandwidth is going to China!!!

**Bud Boy** · 22-12-2011, 20:08

Apparently support say being unable to block Baidu is a coding issue:

To summarise live chat session...

Me: so you cannot block it?
.....: I'm sorry but this comes under the coding issue
.....: We dont support scripting issues

Oh, thanks

All I asked is why the 2 IP ranges Baidu uses cannot be blocked! (Not entirely sure how this is a coding issue)

I suppose as it uses 90% of my bandwidth then it's not in euk's interests anyway. Or am I being cynical??

**Brian** · 22-12-2011, 20:36

Did you try what it said in the article about blocking the IPs in .htaccess

**Bud Boy** · 22-12-2011, 20:40

Of course.

From chat...

.....: Well, I can see you have already added the code in .htaccess and robots.txt files
.....: The Rewrite rules are supposed to block those Boat

Chat did give me 5 other links to try(!) but that's not really my point. Surely blocking such an aggressive bot should be basic hosting support? Am I wrong?

**Bud Boy** · 22-12-2011, 20:48

Considering how bandwidth hungry Baidu is I'm amazed many others haven't experienced it using all their allowance!

Anyone having bandwidth issues may be suprised how much Baidu is using. I believe Baidu uses a huge percentage of my bandwidth allowance on most of the sites in my reseller account - that really is alarming!

*Baidu is an evil bot which I nor eukhost can block*

Note: if you're interested in Chinese traffic then I accept Baidu may be of some use.

**Stan** · 23-12-2011, 09:06

Originally Posted by Bud Boy

Apparently support say being unable to block Baidu is a coding issue:

To summarise live chat session...

Me: so you cannot block it?
.....: I'm sorry but this comes under the coding issue
.....: We dont support scripting issues

Oh, thanks

All I asked is why the 2 IP ranges Baidu uses cannot be blocked! (Not entirely sure how this is a coding issue)

I suppose as it uses 90% of my bandwidth then it's not in euk's interests anyway. Or am I being cynical??

Hello,

Apologies for the inconvenience you are going through. Can you please PM me your account details and the two IP ranges so that we can ask our network teams to block them ?

Looking forward to your response.

Thank you.

Best Regards,
Stan

Thread: baidu search engine hammering site

Thread Tools

Display

baidu search engine hammering site

Thread Information

Users Browsing this Thread

Posting Permissions