inex/index.community
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

set status limit to 40 per page

Browse source
This commit is contained in:
Tao Bror Bojlén 2019-03-08 16:05:42 +00:00
parent 2534b45487
commit 884810fa09
No known key found for this signature in database
GPG key ID: C6EC7AAB905F9E6F
1 changed files with 1 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
backend/scraper/management/commands/scrape.py
View file

@ -25,7 +25,7 @@ SEED = 'p.a3.pm'
TIMEOUT = 20 # seconds
NUM_THREADS = 16 # roughly 40MB each
PERSONAL_INSTANCE_THRESHOLD = 10 # instances with < this many users won't be crawled
MAX_STATUSES_PER_PAGE = 100
MAX_STATUSES_PER_PAGE = 40
STATUS_SCRAPE_LIMIT = 5000
INSTANCE_SCRAPE_LIMIT = 50 # note: this does not include newly discovered instances! they will always be crawled.