Top 100 Websites Listed In 200 Different Categorys
Top 100 Websites Listed In 200 Different Countries
N/A
HTTP/1.1 301 Moved Permanently Server: nginx/1.14.0 (Ubuntu) Date: Thu, 04 Mar 2021 05:37:46 GMT Content-Type: text/html Content-Length: 194 Connection: keep-alive Location: https://opensource.org/
## robots.txt## This file is to prevent the crawling and indexing of certain parts# of your site by web crawlers and spiders run by sites like Yahoo!# and Google. By telling these "robots" where not to go on your site,# you save bandwidth and server resources.## This file will be ignored unless it is at the root of your host:# Used: http://example.com/robots.txt# Ignored: http://example.com/site/robots.txt## For more information about the robots.txt standard, see:# http://www.robotstxt.org/robotstxt.htmlUser-agent: *Crawl-delay: 10# CSS, JS, ImagesAllow: /misc/*.css$Allow: /misc/*.css?Allow: /misc/*.js$Allow: /misc/*.js?Allow: /misc/*.gifAllow: /misc/*.jpgAllow: /misc/*.jpegAllow: /misc/*.pngAllow: /modules/*.css$Allow: /modules/*.css?Allow: /modules/*.js$Allow: /modules/*.js?Allow: /modules/*.gifAllow: /modules/*.jpgAllow: /modules/*.jpegAllow: /modules/*.pngAllow: /profiles/*.css$Allow: /profiles/*.css?Allow: /profiles/*.js$Allow: /profiles/*.js?Allow: /profiles/*.gifAllow: /profiles/*.jpgAllow: /profiles/*.jpegAllow: /profiles/*.pngAllow: /themes/*.css$Allow: /themes/*.css?Allow: /themes/*.js$Allow: /themes/*.js?Allow: /themes/*.gifAllow: /themes/*.jpgAllow: /themes/*.jpegAllow: /themes/*.png# DirectoriesDisallow: /includes/Disallow: /misc/Disallow: /modules/Disallow: /profiles/Disallow: /scripts/Disallow: /themes/# FilesDisallow: /CHANGELOG.txtDisallow: /cron.phpDisallow: /INSTALL.mysql.txtDisallow: /INSTALL.pgsql.txtDisallow: /INSTALL.sqlite.txtDisallow: /install.phpDisallow: /INSTALL.txtDisallow: /LICENSE.txtDisallow: /MAINTAINERS.txtDisallow: /update.phpDisallow: /UPGRADE.txtDisallow: /xmlrpc.php# Paths (clean URLs)Disallow: /admin/Disallow: /comment/reply/Disallow: /filter/tips/Disallow: /node/add/Disallow: /search/Disallow: /user/register/Disallow: /user/password/Disallow: /user/login/Disallow: /user/logout/# Paths (no clean URLs)Disallow: /?q=admin/Disallow: /?q=comment/reply/Disallow: /?q=filter/tips/Disallow: /?q=node/add/Disallow: /?q=search/Disallow: /?q=user/password/Disallow: /?q=user/register/Disallow: /?q=user/login/Disallow: /?q=user/logout/
Recently Joined
Websites Top Hits