cory/tildefriends
5
6
Fork 2
Code Issues 36 Pull Requests Actions Releases 30 Wiki Activity

Make a robots.txt file #43

New Issue
Closed
opened 2024-03-21 08:36:37 -04:00 by tasiaiso · 4 comments
tasiaiso commented 2024-03-21 08:36:37 -04:00
Contributor
Copy Link

I'll do it later

I'll do it later
Screenshot_20240321-133350.png
240 KiB
Screenshot_20240321-133350.png
cory commented 2024-03-21 09:10:41 -04:00
Owner
Copy Link

I think I'm OK having it scraped (to some extent), but we should do something so it shows up better / more usefully in search results.

I think this is hardcoded in http.js.c: https://www.tildefriends.net/robots.txt

Maybe we should do this in index.html: https://metatags.io/

I think I'm OK having it scraped (to some extent), but we should do something so it shows up better / more usefully in search results. I think this is hardcoded in http.js.c: https://www.tildefriends.net/robots.txt Maybe we should do this in index.html: https://metatags.io/
tasiaiso commented 2024-03-21 10:35:47 -04:00
Author
Contributor
Copy Link

I think I'm OK having it scraped

Definitely, but I think it should only access / and /~cory/welcome/. Anything else is a waste of bandwidth, resources and storage imo.

> I think I'm OK having it scraped Definitely, but I think it should only access `/` and `/~cory/welcome/`. Anything else is a waste of bandwidth, resources and storage imo.
cory commented 2024-03-25 12:52:17 -04:00
Owner
Copy Link

I do want public blob/wiki style apps that I want searchable.

The ssb app currently doesn't do much unless you're signed in, which is my main approach to it not being consumed by bots right now.

Maybe there's a future where apps specify their own decision for the robots meta tag: https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag

I do want public blob/wiki style apps that I want searchable. The ssb app currently doesn't do much unless you're signed in, which is my main approach to it not being consumed by bots right now. Maybe there's a future where apps specify their own decision for the robots meta tag: https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag
cory added this to the Make it a platform worth building on milestone 2024-05-15 20:37:00 -04:00
cory commented 2025-02-22 11:42:36 -05:00
Owner
Copy Link

I know I've sat on this for a while. I don't think there's anything more I want to do here right now. I added enough meta tags to make search results look acceptable. None of the stock apps do much actual work if you're not authenticated, and load/traffic from the web hasn't been a problem on tildefriends.net in practice. If an app wants to hide itself, it can always generate its own <meta name="robots" content="noindex">.

I know I've sat on this for a while. I don't think there's anything more I want to do here right now. I added enough meta tags to make search results look acceptable. None of the stock apps do much actual work if you're not authenticated, and load/traffic from the web hasn't been a problem on tildefriends.net in practice. If an app wants to hide itself, it can always generate its own `<meta name="robots" content="noindex">`.
cory closed this issue 2025-02-22 11:42:39 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone Make it a platform worth building on
Assignees
Clear assignees
cory
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: cory/tildefriends#43
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?