Amit Agarwal wrote recently really nice list of tips for well known utility wget. At the end, there is a little quiz, following wget command:
wget ‐‐span-hosts ‐‐level=inf ‐‐recursive dmoz.org
The first option ‐‐span-hosts will allow to download from all links, the second option ‐‐level=inf (or ‐‐level=0) will specify infinite retrying. With conbination of the third option ‐‐recursive we’ve crawler with root at Open Directory Project – dmoz.org.