Cynicus Rex@lemmy.ml to Privacy@lemmy.mlEnglish · 2 years agoHow to block AI Crawler Bots using robots.txt filewww.cyberciti.bizexternal-linkmessage-square64linkfedilinkarrow-up1112arrow-down132
arrow-up180arrow-down1external-linkHow to block AI Crawler Bots using robots.txt filewww.cyberciti.bizCynicus Rex@lemmy.ml to Privacy@lemmy.mlEnglish · 2 years agomessage-square64linkfedilink
minus-squareJackbyDev@programming.devlinkfedilinkEnglisharrow-up1arrow-down5·2 years agoIt’s better than saying something like “there’s no point in robots.txt because bots can disobey is” though.
minus-squarethanks_shakey_snake@lemmy.calinkfedilinkarrow-up3·2 years agoMaybe? But it’s not like that’s the only alternative thing to say, lol
minus-squareReversalHatchery@beehaw.orglinkfedilinkEnglisharrow-up2arrow-down1·edit-22 years agoIs it, though? I mean, robots.txt is the Do Not Track of the opposite side of the connection.
It’s better than saying something like “there’s no point in robots.txt because bots can disobey is” though.
Maybe? But it’s not like that’s the only alternative thing to say, lol
Is it, though?
I mean, robots.txt is the Do Not Track of the opposite side of the connection.