cory/tildefriends
4
5
Fork 1
Code Issues 33 Pull Requests 1 Actions Releases 28 Wiki Activity

Make a robots.txt file #43

New Issue
Open
opened 2024-03-21 08:36:37 -04:00 by tasiaiso · 3 comments
tasiaiso commented 2024-03-21 08:36:37 -04:00
Contributor
Copy Link

I'll do it later

I'll do it later
Screenshot_20240321-133350.png
240 KiB
Screenshot_20240321-133350.png
cory commented 2024-03-21 09:10:41 -04:00
Owner
Copy Link

I think I'm OK having it scraped (to some extent), but we should do something so it shows up better / more usefully in search results.

I think this is hardcoded in http.js.c: https://www.tildefriends.net/robots.txt

Maybe we should do this in index.html: https://metatags.io/

I think I'm OK having it scraped (to some extent), but we should do something so it shows up better / more usefully in search results. I think this is hardcoded in http.js.c: https://www.tildefriends.net/robots.txt Maybe we should do this in index.html: https://metatags.io/
tasiaiso commented 2024-03-21 10:35:47 -04:00
Author
Contributor
Copy Link

I think I'm OK having it scraped

Definitely, but I think it should only access / and /~cory/welcome/. Anything else is a waste of bandwidth, resources and storage imo.

> I think I'm OK having it scraped Definitely, but I think it should only access `/` and `/~cory/welcome/`. Anything else is a waste of bandwidth, resources and storage imo.
cory commented 2024-03-25 12:52:17 -04:00
Owner
Copy Link

I do want public blob/wiki style apps that I want searchable.

The ssb app currently doesn't do much unless you're signed in, which is my main approach to it not being consumed by bots right now.

Maybe there's a future where apps specify their own decision for the robots meta tag: https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag

I do want public blob/wiki style apps that I want searchable. The ssb app currently doesn't do much unless you're signed in, which is my main approach to it not being consumed by bots right now. Maybe there's a future where apps specify their own decision for the robots meta tag: https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag
cory added this to the Make it a platform worth building on milestone 2024-05-15 20:37:00 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
No items
No Label
Milestone
Clear milestone
No items
No Milestone
Make it a platform worth building on
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: cory/tildefriends#43
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?