# robots.txt --- index my site for search but don't archive it User-agent: * Disallow: /stats Disallow: /bin Disallow: /gif Disallow: /back Disallow: /jpeg Disallow: /test # Tell the Internet Archive not to keep a copy User-agent: ia_archiver Disallow: /