Been using it for years. Works on MOST websites, though not all.
Gunner
"Confiscating wealth from those who have earned it, inherited it, or got lucky is never going to help 'the poor.' Poverty isn't caused by some people having more money than others, just as obesity isn't caused by McDonald's serving super-sized orders of French fries Poverty, like obesity, is caused by the life choices that dictate results." - John Tucci,
If you try downloading my website algebra.com, you will get into an infinite recursion through millions of pages. That's why I prevent most such bots from accessing my site. This would work only on very simple sites.
Some bots as you say - limit to 1 or 2 or 3 levels deep only.
Mart> >>
----== Posted via Pronews.Com - Unlimited-Unrestricted-Secure Usenet News==----
---= - Total Privacy via Encryption =---
you must be very smart to have such a complex and sophisticated website.
Yours was one of the sites it doesnt work well on...chuckle
Been there, tried that..
But it works quite well on most others.
Gunner
"Confiscating wealth from those who have earned it, inherited it, or got lucky is never going to help 'the poor.' Poverty isn't caused by some people having more money than others, just as obesity isn't caused by McDonald's serving super-sized orders of French fries Poverty, like obesity, is caused by the life choices that dictate results." - John Tucci,
How does your web server differentiate between a bot and a human user making http requests?
Regards,
Robin
snipped-for-privacy@gmail.com fired this volley in news:b6e6ea8f-a3fe-4f46- snipped-for-privacy@f63g2000hsf.googlegroups.com:
Duh! It doesn't. The site has links back to the place where the link began. It wouldn't appear recursive to a human user, because that person would choose where he/she viewed. The spider can't tell, and ends up in recursions it can only abort by "counting out" repeats.
LLoyd
I tend to just use wget. Helps if you've got *nix for an OS or the Cygwin utilities for windoze though.
Mark Rand RTFM
I actually have some smarts in the server that can tell a bot from a human. But httrack is blocked on the spot in any case. I am not against it, as such, but it will not work on my site.
For one, "wget" can certainly detect and ignore recursive loops.
Not a bot attempting to look human. Just bots that advertise their botness, by honest design or flawed hacking.
Yes, a bot trying to look like a human (ie supplying Referer and browser-like User-Agent, I can still detec that it is a bot).
The way I detect is is that there is a hidden link that humans cannot see, and cannot click, but bots would follow it. The hidden link is not permitted by robots.txt, so it catches all non-compliant bots.
Yes, that would be difficult to defeat.
I had a lot of troubles with httrack and other bots like this. The people who run them usually are not meaning anything bad, they just do not realize that they should not run it against dynamic sites like mine. They may not even realize that my site is dynamic because it tried to look not to be (search engine friendly and all).
I spent a very long time trying to 1) make a website which hopefully does not lead into too many infinite crawlings, and 2) detect and stop bad bots early enough. But I still get problems from time to time.
Why?
Gunner
"Confiscating wealth from those who have earned it, inherited it, or got lucky is never going to help 'the poor.' Poverty isn't caused by some people having more money than others, just as obesity isn't caused by McDonald's serving super-sized orders of French fries Poverty, like obesity, is caused by the life choices that dictate results." - John Tucci,
Whats wrong with bots harvesting your manuals?
Frankly..on dialup..I dont have the time to hit each and every manual and wait for a download to start and finish.
On sites such as yours, I run the program and go to bed.
Gunner
"Confiscating wealth from those who have earned it, inherited it, or got lucky is never going to help 'the poor.' Poverty isn't caused by some people having more money than others, just as obesity isn't caused by McDonald's serving super-sized orders of French fries Poverty, like obesity, is caused by the life choices that dictate results." - John Tucci,
NOthing.
But if you go to algebra.com, you can accidentally go into an infinite loop with various scripts.
You may need to sleep a lot longer than anticipated.
i
I use wget for this too, provided saving a few pages doesn't work out so well. There are Windows binaries available, no need for Cygwin. For example:
PolyTech Forum website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.