Forum Navigation
How to set up robot.txt on my Neocities website
#1 · February 11, 2025, 12:37 pm
Hello, I have a problem understanding how to set up robot.txt and what is recommended to add there to stop AI scraper.
If you guys can explain it to me how or have a link to a beginner guide please let me know
Thank you!
#2 · February 16, 2025, 6:17 am
i think you can just directly copy & paste this into the robots.txt file in your website (yourusername.neocities.org/robots.txt, pretty sure it has to be there and not in a folder. you can see why each one in the previous link is included here)
there are a few places to learn about how it works
https://developer.mozilla.org/en-US/docs/Glossary/Robots.txt
https://en.wikipedia.org/wiki/Robots.txt
DedRoll has reacted to this post.
DedRoll
