Learn Something About Robots.txt In An Easy Way - Save Life of Your Site | Part 1/2
SEO Episode 29:
This is 29th Seo Episode. Kye Faida Nhi Na Robots.txt Ka?
:P If you read the article published on Propakistani Ky NTS Ki Site Par 1000+ Students Ki Detail Thi PDF Main, Google Main Agr Kye Search Karta Tha Wo Show Ho Rhi Thi. Un Ka Phone Number, Home Address, CNIC etc Sab Shamil Tha Even of Women. Some people commented below in that article that they had downloaded all the PDFs. Now you can imagine how destructive it might be. Masla Ye Tha NTS walo Ny Robots.txt Main Google Ko Nhi Bola Tha Ky Is Directory Ko Disallow Kar Do. Jab Disallow Nhi Kiya to Matlab Google Ny Crwal Kiya Then Index Bhi Kar Diya Then Google Search Result Main Show Bhi Karwa Diya. Jesy Nike Walo Ny Timam PDF Block Kar Rakhi Han Nike.com
Robots.txt is an interesting file that helps a lot when it comes to telling the search engine what to crawl and what not to crawl. Matlab Kya Google Search Engine Main Show Karwana Hai Aur Kya Nhi.
I explained crawling, indexing, and spider in Episode 13.
Some people don't try to change the robots.txt because they think it would destroy their site :P Par Esa Nhi Hai.
:P Par Esa Nhi Hai.
You just need to understand the following things:
User-agent — is ka matlab konsa robot ya konsa search engine ky liye ye rule follow kry.
Disallow — ye robots ko kehta hai ye file ya direcrrty crawl na karna.
Allow — aur ye wala part crawl karna
Allow — aur ye wala part crawl karna
Crawl-delay — ye btata hai ky crawl krny sy phle thora sy wait karna. Ye Bari site ky liye hota hai becase crawal sy server par asar parta hai.
Sitemap — ye show karta hai sitemap hai kahan
# — is ky agy jo marzi likho Robot nhi pary ga jesy hum html main <!-- katy han (am learning programming :P)
:P)
* — is tarha ka kye bhi match kary to include karna. Jesy robots.txt of last.fm Ny
Disallow: /user/*/library/music/
Disallow: /de/user/*/library/music/
Disallow: /es/user/*/library/music/
Disallow: /de/user/*/library/music/
Disallow: /es/user/*/library/music/
$ — matlab ye must yahan end han. Jesy Nike Walo ny PDF Ka Kiya Howa Hai.
You know robots.txt must be in the main root folder i.e., domain.com/robots.txt. It means you can see robots.txt file of most of the websites even robots.txt of Propakistani.pk
Let's see the best example of Robots.txt
- Nike.com (Last Main Dekho Nike Ka Logo Bna Howa Hai Becz # Ky Baad Kuch Count Nhi Hota Hai :P )
- Tripadvisor (ye bnda SEO service provide karny ki effort kar rha hai :P )
Try to understand it.
Chalo Ye Ek Chota Sa Part Robots.txt Ka Copy Kiya HAi Kisi Website Ka:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-content/
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-content/
* ka kya matlab hai?
Matlab ky ye taqriban timam bot par ye formula apply ho. Taqriban? Han! Because Not all bot respects robots.txt. Is Liye Propakistani Waly Ny Har Bot Ka Specifc Naam Likh Kar User-agent main Block Kiya Howa Hai.
/wp-content/ is ka matlab /wp-content/ ki jo directary hai us ko crawl na karna.
In the next part 2, I will explain how to verify it the robots.txt file you made, will work fine or not and the file want to be blocked by Google, Asal Main Hogi Bhi Ya Nhi.
Have You Missed the Previous 29 SEO Episodes and Videos?
--
Then Go To How to Remove Spam Traffic of Secret ɢoogle com
--
Then Go To How to Remove Spam Traffic of Secret ɢoogle com
#SEOEPISODE29
Written By: Muhammad Aamir Mursleen
Written By: Muhammad Aamir Mursleen