Web Crawler

If you were designing a web crawler, how would you avoid getting into infinite loops?

Solution

How to handle cycle?

How to define different in crawling

similarity standard