• bdonvr@thelemmy.club
      cake
      link
      fedilink
      arrow-up
      3
      ·
      11 months ago

      That’s not true, search indexer bots should be allowed through from what I read here.

      • interdimensionalmeme@lemmy.ml
        link
        fedilink
        arrow-up
        9
        ·
        11 months ago

        If you allow my searchxng search scraper then an AI scraper is indistinguishable.

        If you mean, “google and duckduckgo are whitelisted” then lemmy will only be searchable there, those specific whitelisted hosts. And google search index is also an AI scraper bot.