Question about htaccess, robots.txt and crawlers.

D

dsarosh

Adept
Hello,
I have a couple of questions.

1) How long after I register a domain can I expect the search engines index my site?

2) I do not have any robots.txt or any htaccess files on my server. Is that okay? Are they required for search engines to crawl my site?

Thanks,
Sarosh.
 
Not yet because it is still almost empty.

But I wanted to know how long it would take to get indexed if:

1) I submitted the site to search engines

2) I did not submit it to sites

3) I do not use any robots or htaccess file (I have learned that robots.txt and htaccess files are not required to get the site indexed)
 
dsarosh said:
Hello,
I have a couple of questions.

1) How long after I register a domain can I expect the search engines index my site?
Click to expand...

Once you submitted your site to google, it'll start the job within a minimum of 24 hours to three days.

dsarosh said:
2) I do not have any robots.txt or any htaccess files on my server. Is that okay? Are they required for search engines to crawl my site?

Thanks,
Sarosh.
Click to expand...

I think creating robots.txt is not a big deal, so create it and upload it. Simple. No need for htaccess as i guess you're on shared server.
 
robots.txt are used by crawlers, it contains paths which "ideally" the crawlers should not read/access for caching/indexing. this text file is optional and following the robots.txt is upto the crawlers which access your site.
 
Back
Top