Commit graph

163 commits

Author SHA1 Message Date
Tao Bror Bojlén 617e2f43ee
release 2.7.0 2019-08-18 18:14:58 +02:00
Tao Bror Bojlén f033b1eec0
add friendica crawler 2019-08-18 15:27:50 +02:00
Tao Bror Bojlén 4a7804d987
remove color-coding by activity per user 2019-08-15 16:35:40 +02:00
Tao Bror Bojlén fc836256ba
clean up ES config 2019-08-15 16:19:26 +02:00
Tao Bror Bojlén b0cdab2fbd
release 2.6.1 2019-08-10 16:32:35 +03:00
Tao Bror Bojlén 2c035892d4
add most recent crawl table 2019-08-10 16:21:22 +03:00
Tao Bror Bojlén 144a6e842f
add missing indices on crawls 2019-08-10 15:35:20 +03:00
Tao Bror Bojlén 992ed6b9bb
release 2.6.0 2019-08-10 14:56:46 +03:00
Tao Bror Bojlén e8b951485e
crawl all gnusocial instances w/o nodeinfo 2019-08-10 14:18:03 +03:00
Tao Bror Bojlén 271c67ea8e
also detect gnusocial instances w/o nodeinfo 2019-08-10 14:16:55 +03:00
Tao Bror Bojlén b2e849cf9f
add ON DELETE to relations 2019-08-09 20:30:22 +03:00
Tao Bojlén 3320e050c8 Revert "add metadata endpoint"
This reverts commit 82153b283b.
2019-08-09 16:59:51 +00:00
Tao Bror Bojlén 9a0bbbb7d9
release 2.5.0 2019-08-08 14:45:17 +03:00
Tao Bror Bojlén 186abbd9cf
more performant deletes 2019-08-08 12:44:29 +03:00
Tao Bror Bojlén 30c5bbe2b9
only save successful crawls in db 2019-08-07 22:41:19 +03:00
Tao Bror Bojlén d9b9081ec3
set infinite timeout on v expensive queries 2019-08-07 20:34:02 +03:00
Tao Bror Bojlén 012fc1d85a
only use 20 crawl workers 2019-08-07 20:17:36 +03:00
Tao Bojlén f0f3f76f6c handle crawl timestamps in db 2019-08-07 16:53:34 +00:00
Tao Bror Bojlén 1e4cebb26b
fix broken stale instance manager 2019-08-07 18:39:27 +03:00
Tao Bojlén cf9ac30b1e Fewer unnecessary crawls 2019-08-07 12:41:29 +00:00
Tao Bror Bojlén c01e324e91
increase db timeout 2019-08-07 12:41:30 +03:00
Tao Bror Bojlén e4f98a2b2f
fix accidental break of mastodon crawler 2019-08-06 21:58:02 +03:00
Tao Bojlén 75e66affe3 Add misskey crawler 2019-08-06 18:04:30 +00:00
Tao Bror Bojlén cc541b9ddf
fix search view when no results, release 2.4.1 2019-08-04 18:25:50 +03:00
Tao Bror Bojlén a4e02e6f76
release 2.4.0 2019-08-04 15:46:38 +03:00
Tao Bror Bojlén de6d997976
support searching in many more languages 2019-08-04 15:09:32 +03:00
Tao Bojlén 5a10c6c9e3 filter by instance type 2019-08-04 11:39:29 +00:00
Tao Bror Bojlén 3a50e5cb4d
release 2.3.1 2019-08-03 00:33:55 +03:00
Tao Bror Bojlén 80c1462866
improve search result relevance 2019-08-03 00:25:39 +03:00
Tao Bror Bojlén 76ebf50c40
never display opted-out instances anywhere 2019-08-03 00:09:09 +03:00
Tao Bror Bojlén cc95d19ee8
fix error in mobile instance view 2019-08-02 22:49:47 +03:00
Tao Bror Bojlén 287c7b5624
normalize instance names 2019-08-02 22:46:40 +03:00
Tao Bror Bojlén 767bd4a947
release 2.3.0 2019-08-02 19:54:32 +03:00
Tao Bror Bojlén 4c1960c3b2
add color-coding by activity per person 2019-08-02 19:37:26 +03:00
Tao Bror Bojlén 09708e74ab
update ES index on crawl 2019-08-02 19:03:21 +03:00
Tao Bror Bojlén 8229f83938
separate dockerfile copy commands for debugging 2019-08-02 17:33:31 +03:00
Tao Bror Bojlén f93345ecef
extremely hacky fix for ES configuration 2019-08-02 17:30:50 +03:00
Tao Bror Bojlén 4d191f6431
update gollum; more logging in indexer 2019-07-31 18:56:21 +03:00
Tao Bror Bojlén 42e3efee71
have config available for docker dependency compilation 2019-07-31 16:52:55 +03:00
Tao Bojlén 7db145261b quantitative color coding 2019-07-27 17:58:40 +00:00
Tao Bror Bojlén 0f01620413
perf and metrics tweaks 2019-07-27 16:22:54 +03:00
Tao Bror Bojlén ea0fae2152
don't add domains to the queue twice 2019-07-27 16:22:39 +03:00
Tao Bror Bojlén 74fc30e8a5
don't log crawl times with appsignal 2019-07-27 13:57:07 +03:00
Tao Bror Bojlén 235952b072
add ES to extra_applications 2019-07-27 13:56:00 +03:00
Tao Bojlén ab5f442074 check for spam less often 2019-07-27 10:32:42 +00:00
Tao Bojlén 79257f1cbe Setup appsignal 2019-07-27 10:13:42 +00:00
Tao Bojlén 5ca8de5dbe add full-text search 2019-07-26 22:30:11 +00:00
Tao Bror Bojlén 5e9b498db0
add missing import in user_email 2019-07-26 17:41:32 +03:00
Tao Bojlén 7b2b3876c6 feature/administration 2019-07-26 14:34:23 +00:00
Tao Bror Bojlén 2f538c9a12
crawl less often 2019-07-25 19:12:16 +03:00
Tao Bojlén 490b518587 Save base domain 2019-07-25 15:56:03 +00:00
Tao Bror Bojlén 99a6dc2023
prepare for 2.2.0 release 2019-07-24 19:37:32 +03:00
Tao Bror Bojlén a615d115fd
show instance type in details 2019-07-24 19:25:20 +03:00
Tao Bojlén 3b515c3a36 Color code graph 2019-07-24 15:51:44 +00:00
Tao Bror Bojlén 2de412bfbb
prepare for 2.1.0 release 2019-07-24 13:19:24 +03:00
Tao Bojlén e490b30ebb experimental mobile support 2019-07-23 16:32:43 +00:00
Tao Bojlén 1ff7cd7290 frontend/search/ui fixes 2019-07-23 14:08:43 +00:00
Tao Bojlén 6673a24466 Refactor graph sidebar, move to backend search 2019-07-23 12:20:34 +00:00
Tao Bojlén 39d279debd add routing support for selected instance 2019-07-21 18:05:07 +00:00
Tao Bojlén 51051d37a0 show robots.txt status in frontend 2019-07-20 17:43:37 +00:00
Tao Bror Bojlén 7aa914ab3f
prepare for 2.0.0 release 2019-07-20 13:01:56 +03:00
Tao Bojlén 2cae5dccbf check robots.txt for permission to crawl 2019-07-19 20:00:28 +00:00
Tao Bojlén 5fafffc404 frontend/cytoscape feature parity 2019-07-19 18:19:53 +00:00
Tao Bojlén 2e61b70602 Hide insular instances 2019-07-18 20:05:16 +00:00
Tao Bojlén 677b984451 add Insularity score 2019-07-18 15:20:09 +00:00
Tao Bror Bojlén 9478017eb0
fix edge generation, serve graph in cytoscape format 2019-07-18 13:21:12 +03:00
Tao Bror Bojlén 82677fcd32
add spam domain to blacklist 2019-07-18 13:21:06 +03:00
Tao Bojlén 4a87c49baf set up sobelow for backend 2019-07-17 16:16:25 +00:00
Tao Bror Bojlén d5b10a6f22
set up migrations on deploy 2019-07-17 14:47:47 +03:00
Tao Bror Bojlén b8c474ca89
fix DATABASE_URL config in phoenix 2019-07-14 14:40:54 +01:00
Tao Bojlén a37452f138 refactor/elixir backend 2019-07-14 11:47:06 +00:00
Tao Bror Bojlén 0cd19e597d
hotfix: blacklist gab.best domains 2019-07-03 12:13:43 +01:00
Tao Bror Bojlén e9b8901aa5
don't display quiet nodes on graph 2019-04-19 15:29:45 +01:00
Tao Bror Bojlén 8d00ba9372
build(react-scripts): move from react-scripts-ts to react-scripts
the latter now supports typescript natively.
2019-04-17 15:44:23 +01:00
Tao Bror Bojlén 884810fa09
set status limit to 40 per page 2019-03-08 16:05:42 +00:00
Tao Bror Bojlén 2534b45487
remove trailing slash from endpoint crawled
this was causing issues on at least one pleroma instance, for some
reason
2019-03-08 15:59:07 +00:00
Tao Bror Bojlén 1770333906
remove whitelist from git 2019-03-08 15:58:43 +00:00
Tao Bror Bojlén 3a66e7464e
fix duplicates in instance peers 2019-03-01 14:42:05 +00:00
Tao Bror Bojlén e52f581067
bump personal instance threshold to 10 2019-02-28 18:06:16 +00:00
Tao Bror Bojlén dfc115f275
lowercase instance names (fix #48) 2019-02-28 18:05:54 +00:00
Tao Bror Bojlén 2e5696962b
only scrape 50 instances in one go 2019-02-28 18:03:08 +00:00
Tao Bror Bojlén 91f326e19b
handle missing data better in API 2019-02-28 17:51:03 +00:00
Tao Bojlén 648f225772 Allow CORS from all fediverse.space subdomains 2019-02-27 09:09:23 +00:00
Tao Bojlén 54b5c9ef2a migrate to caddy server 2019-02-22 15:40:44 +00:00
Tao Bror Bojlén de1cee2015
add www subdomain to CORS whitelist 2019-02-21 14:38:23 +00:00
Tao Bror Bojlén b4ef0793e7
restore original status limit 2019-02-21 12:35:53 +00:00
Tao Bror Bojlén 795ad67e8b
dockerize gephi 2019-02-21 12:32:50 +00:00
Tao Bror Bojlén 1c1f193542
improve handling of cancelled scrape 2019-02-21 10:38:49 +00:00
Tao Bror Bojlén 1b11c70430
simplify folder structure 2019-02-21 10:26:55 +00:00
Tao Bror Bojlén e33c70fa25
improve scraping logic 2019-02-20 22:09:21 +00:00
Tao Bror Bojlén a20d8e23e3
improve scraper logging 2019-02-20 21:36:12 +00:00
Tao Bror Bojlén dfaf15fb51
fix CORS headers 2019-02-20 20:54:08 +00:00
Tao Bror Bojlén 5079875697
fix CORS for netlify branch deployments 2019-02-20 20:44:27 +00:00
Tao Bojlén cd36e7902d merge 2019-02-20 20:04:15 +00:00
Tao Bror Bojlén bb2f39e785
add subdomains to allowed hosts 2019-02-20 17:15:41 +00:00
Tao Bror Bojlén d90f0f3218
don't serve static files 2019-02-20 17:14:28 +00:00
Tao Bror Bojlén 12fb04ea3e
update django 2019-02-20 17:06:36 +00:00
Tao Bror Bojlén 3a96ee1f39
docker-ise backend 2019-02-20 16:18:12 +00:00
Tao Bojlen d383bca0d8 scraper perf improvements 2018-09-01 20:46:00 +02:00
Tao Bror Bojlén 93932c5196
add graph (#1)
* add initial graph

* add graph
2018-09-01 15:32:04 +02:00