show robots.txt status in frontend

CHANGELOG.md

@@ -6,6 +6,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 ### Added
+- It's now shown in the front-end if an instance wasn't crawled because of its robots.txt.
 ### Changed
 ### Deprecated
 ### Removed

README.md

@@ -70,7 +70,7 @@ You don't have to follow these instructions, but it's one way to set up a contin
   * `dokku postgres:link fediversedb phoenix`
   * `dokku postgres:link fediversedb gephi`
 5. Update the backend configuration. In particular, change the `user_agent` in [config.exs](/backend/config/config.exs) to something descriptive.
-6. Push the apps, e.g. `git push dokku@<DOMAIN>:phoenix` (from your local machine or CD pipeline)
+6. Push the apps, e.g. `git push dokku@<DOMAIN>:phoenix` (note that the first push cannot be from the CD pipeline).
 7. Set up SSL for the Phoenix app
   * `dokku letsencrypt phoenix`
   * `dokku letsencrypt:cron-job --add`

backend/lib/backend_web/controllers/instance_controller.ex

@@ -13,7 +13,7 @@ defmodule BackendWeb.InstanceController do
 
   def show(conn, %{"id" => domain}) do
     instance = Api.get_instance!(domain)
-    last_crawl = get_last_successful_crawl(domain)
+    last_crawl = get_last_crawl(domain)
     render(conn, "show.json", instance: instance, crawl: last_crawl)
   end
 

frontend/src/components/Sidebar.tsx

@@ -111,6 +111,8 @@ class SidebarImpl extends React.Component<ISidebarProps, ISidebarState> {
       return this.renderEmptyState();
     } else if (this.props.instanceDetails.status.toLowerCase().indexOf("personal instance") > -1) {
       content = this.renderPersonalInstanceErrorState();
+    } else if (this.props.instanceDetails.status.toLowerCase().indexOf("robots.txt") > -1) {
+      content = this.renderRobotsTxtState();
     } else if (this.props.instanceDetails.status !== "success") {
       content = this.renderMissingDataState();
     } else if (this.props.instanceLoadError) {
@@ -380,6 +382,16 @@ class SidebarImpl extends React.Component<ISidebarProps, ISidebarState> {
     );
   };
 
+  private renderRobotsTxtState = () => {
+    return (
+      <NonIdealState
+        icon={<span>🤖</span>}
+        title="No data"
+        description="This instance was not crawled because its robots.txt did not allow us to."
+      />
+    );
+  };
+
   private openInstanceLink = () => {
     window.open("https://" + this.props.instanceName, "_blank");
   };