I have no idea where to start with problems regarding: https://devplace.net/news/1b9fe522688a-our-scraper-stored-20-links-per-page-and-kept-every-page-in-memory

First of all, with the memory of my server, you could scrape whole devRant (it's 30+ GB but I have 64). Also, why would you do it in memory? Just create a sqlite3 database and you're done. Same comfort almost and you'll have millions of pages in 'memory' without downside.

Btw, if your scraper does not do such stuff it's kinda useless. This article is about creating a basic scraper basically.