Yeah, I actually think lemmy could benefit from improved scraping and indexing. For example, it’d really help if more search engines could natively understand Lemmy’s federated nature, e.g:

• Deduplicate links by prioritising results for instances hosting the community that a post was originally submitted to.
• Include and denote cross posts by recognizing order of submission timestamp and prioritizing popularity via vote ratios, comment counts, and lurker click-through traffic.
• Do the same deduplication and prioritization across instances, but for comments as well.

Another use case besides search engines would be for internet archive projects, helping to preserve historic internet content even in the face of lemmy instances falling offline and disappearing. For example, much knowledge was lost to us due to the Twitter APIoplicips and Reddit Blackout: E.g:

• https://wiki.archiveteam.org/index.php/Warrior_projects
• https://tracker.archiveteam.org/reddit/

Most of the above will only ever be possible due to improved scraping or even federation APIs.

Yeah absolutely, I’m actually tempted to implement some kind of encompassing “all” instance that listens to all communities of all known lemmy instances (to emulate something like reddits r/all equivalent for lemmy), which I think would be pretty useful, to get an idea of the “size” of the lemmy community. Probably without any users itself (also to avoid being blacklisted from other instances).