Comments on: The Whitehouse.gov Website’s Robots.txt File Has 1839 Lines In It http://danielmiessler.com/blog/the-whitehousegov-websites-robotstxt-file-has-1839-lines-in-it grep understanding Sun, 29 Jan 2012 20:44:46 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: Deepak http://danielmiessler.com/blog/the-whitehousegov-websites-robotstxt-file-has-1839-lines-in-it/comment-page-1#comment-27624 Deepak Thu, 08 Feb 2007 13:38:06 +0000 http://dmiessler.com/archives/1083#comment-27624 <p>Yup, even I noted it sometime back as an excellent sitemap. ;-)</p> Yup, even I noted it sometime back as an excellent sitemap. ;-)

]]> By: Deepak http://danielmiessler.com/blog/the-whitehousegov-websites-robotstxt-file-has-1839-lines-in-it/comment-page-1#comment-246735 Deepak Thu, 08 Feb 2007 13:38:00 +0000 http://dmiessler.com/archives/1083#comment-246735 <p>Yup, even I noted it sometime back as an excellent sitemap. ;-)</p> Yup, even I noted it sometime back as an excellent sitemap. ;-)

]]> By: ghost16825 http://danielmiessler.com/blog/the-whitehousegov-websites-robotstxt-file-has-1839-lines-in-it/comment-page-1#comment-25176 ghost16825 Fri, 26 Jan 2007 10:03:46 +0000 http://dmiessler.com/archives/1083#comment-25176 <p>Ooooh, /secret/ directories. <em>nods head</em></p> Ooooh, /secret/ directories. nods head

]]> By: ghost16825 http://danielmiessler.com/blog/the-whitehousegov-websites-robotstxt-file-has-1839-lines-in-it/comment-page-1#comment-246734 ghost16825 Fri, 26 Jan 2007 10:03:00 +0000 http://dmiessler.com/archives/1083#comment-246734 <p>Ooooh, /secret/ directories. <em>nods head</em></p> Ooooh, /secret/ directories. nods head

]]> By: sergei http://danielmiessler.com/blog/the-whitehousegov-websites-robotstxt-file-has-1839-lines-in-it/comment-page-1#comment-25038 sergei Thu, 25 Jan 2007 18:11:44 +0000 http://dmiessler.com/archives/1083#comment-25038 <p>Search in Google for 'robots.txt' shows whitehouse.gov at position 5</p> Search in Google for ‘robots.txt’ shows whitehouse.gov at position 5

]]> By: sergei http://danielmiessler.com/blog/the-whitehousegov-websites-robotstxt-file-has-1839-lines-in-it/comment-page-1#comment-246733 sergei Thu, 25 Jan 2007 18:11:00 +0000 http://dmiessler.com/archives/1083#comment-246733 <p>Search in Google for 'robots.txt' shows whitehouse.gov at position 5</p> Search in Google for ‘robots.txt’ shows whitehouse.gov at position 5

]]> By: Some Joker http://danielmiessler.com/blog/the-whitehousegov-websites-robotstxt-file-has-1839-lines-in-it/comment-page-1#comment-24755 Some Joker Tue, 23 Jan 2007 22:06:03 +0000 http://dmiessler.com/archives/1083#comment-24755 <p>Looking at most of those entries, it looks like they're excluding pages which look to be designed for text only browsers/screen readers.. nearly every directory ends in /text</p> <p>Disallow: /asia/2005/photoessay/china/text Disallow: /asia/2005/photoessay/japan/text Disallow: /asia/2005/photoessay/korea/text Disallow: /asia/2005/photoessay/mongolia/text Disallow: /asia/2005/photoessay/mrsbush1/text Disallow: /asia/2005/photoessay/mrsbush2/text</p> <p>and if you browse up one directory, you get the same story with pictures..</p> <p>I'd say it looks like they are doing it to work around for a poor file structure or possibly to keep search engines from finding duplicate text (although without pictures)</p> <p><em>shrugs</em> I'm all for pointing out when the administration does something crooked, but I can't see fault in this one.. (granted, I've only checked out 20 or so of the links.. the only one that didn't go anywhere for me was /video/text )</p> Looking at most of those entries, it looks like they’re excluding pages which look to be designed for text only browsers/screen readers.. nearly every directory ends in /text

Disallow: /asia/2005/photoessay/china/text Disallow: /asia/2005/photoessay/japan/text Disallow: /asia/2005/photoessay/korea/text Disallow: /asia/2005/photoessay/mongolia/text Disallow: /asia/2005/photoessay/mrsbush1/text Disallow: /asia/2005/photoessay/mrsbush2/text

and if you browse up one directory, you get the same story with pictures..

I’d say it looks like they are doing it to work around for a poor file structure or possibly to keep search engines from finding duplicate text (although without pictures)

shrugs I’m all for pointing out when the administration does something crooked, but I can’t see fault in this one.. (granted, I’ve only checked out 20 or so of the links.. the only one that didn’t go anywhere for me was /video/text )

]]> By: Some Joker http://danielmiessler.com/blog/the-whitehousegov-websites-robotstxt-file-has-1839-lines-in-it/comment-page-1#comment-246732 Some Joker Tue, 23 Jan 2007 22:06:00 +0000 http://dmiessler.com/archives/1083#comment-246732 <p>Looking at most of those entries, it looks like they're excluding pages which look to be designed for text only browsers/screen readers.. nearly every directory ends in /text</p> <p>Disallow: /asia/2005/photoessay/china/text Disallow: /asia/2005/photoessay/japan/text Disallow: /asia/2005/photoessay/korea/text Disallow: /asia/2005/photoessay/mongolia/text Disallow: /asia/2005/photoessay/mrsbush1/text Disallow: /asia/2005/photoessay/mrsbush2/text</p> <p>and if you browse up one directory, you get the same story with pictures..</p> <p>I'd say it looks like they are doing it to work around for a poor file structure or possibly to keep search engines from finding duplicate text (although without pictures)</p> <p><em>shrugs</em> I'm all for pointing out when the administration does something crooked, but I can't see fault in this one.. (granted, I've only checked out 20 or so of the links.. the only one that didn't go anywhere for me was /video/text )</p> Looking at most of those entries, it looks like they’re excluding pages which look to be designed for text only browsers/screen readers.. nearly every directory ends in /text

Disallow: /asia/2005/photoessay/china/text Disallow: /asia/2005/photoessay/japan/text Disallow: /asia/2005/photoessay/korea/text Disallow: /asia/2005/photoessay/mongolia/text Disallow: /asia/2005/photoessay/mrsbush1/text Disallow: /asia/2005/photoessay/mrsbush2/text

and if you browse up one directory, you get the same story with pictures..

I’d say it looks like they are doing it to work around for a poor file structure or possibly to keep search engines from finding duplicate text (although without pictures)

shrugs I’m all for pointing out when the administration does something crooked, but I can’t see fault in this one.. (granted, I’ve only checked out 20 or so of the links.. the only one that didn’t go anywhere for me was /video/text )

]]>