# robots.txt User-agent: * Crawl-delay: 30 User-agent: Yandex Disallow: / User-agent: Baiduspider Disallow: / User-agent: 360Spider Disallow: / User-agent: dataprovider Disallow: /