DEV Community 1h ago

How I Got Blocked 20 Times While Web Scraping-and Found the Fastest Hidden Delivery Route: RSS

Introduction

Hello from Japan! 🇯🇵 I am a professional truck driver with more than seven years of experience in logistics, currently teaching myself Python. This story happened before I started writing my series of articles titled: “What a Truck Driver Achieved After X Hours of Learning”. It took place shortly after I began learning Python. Looking back, without this experience, puoppo_app would probably never have existed. This was the starting point of my journey into software development.

One day, I decided to build a tool that could automatically collect the latest information from a website. I used BeautifulSoup and Requests and started experimenting with web scraping. What followed was a relentless storm of: 403 Forbidden.

I tried everything I could think of:

Changing the User-Agent
Adding random delays
Adjusting request intervals
Trying different approaches

Even after more than 20 attempts, I kept getting blocked. I finally thought: Brute-force access is not going to work.

Just when I was about to give up, I discovered an alternative route. That route was an RSS feed. In this article, I will share:

How repeated scraping failures led me to RSS
How that discovery inspired my AI summarization application, puoppo

This is a behind-the-scenes story about how the project was born. This article is slightly different from my usual technical series. It is an extra chapter about the origin of puoppo.

1. The Brute-Force Route: Twenty Battles Against Blocking

At first, I underestimated the task. I thought: I am only collecting a little information. This should be quick. I was wrong.

What I Tried

Attempts 1-5 - I used ordinary web scraping. The result was immediate: 403 Forbidden. The server rejected the requests before I could collect anything.

Attempts 6-12 - I thought the website might be identifying my script as a bot. I changed the User-Agent header so the request would look like it came from a browser. This worked a few times. Then I was blocked again.

Attempts 13-20 - I considered the load on the server and added intervals using time.sleep(). I also randomized the delay between requests. Even with those precautions, the large news website's security systems-including bot detection and a web application firewall-continued blocking the requests.

In logistics terms, it felt like this: Every time I arrived at the loading area, a different security guard stopped me. I changed the way I lined up, but eventually I was banned from entering altogether.

I could not help being impressed. The security of a major website really is on another level. It was a strange moment to admire the system that was defeating me.

2. Route Optimization: RSS as the Alternative Route

After being blocked for the twentieth time, I stopped and reconsidered the problem. I asked myself: Do I really need to parse the entire HTML page and extract data from between design elements?

I carefully looked through the website footer and found a small icon. It was an RSS feed. I tried the Python library feedparser. The task that had caused so much trouble suddenly required only a few lines of code.

import feedparser

RSS_URL = "https://example.com/feed"

def fetch_latest_topics(url):
    feed = feedparser.parse(url)
    for entry in feed.entries[:5]:
        print(f"Title: {entry.title}")
        print(f"Link: {entry.link}")
        print(f"Published: {entry.published}")

The same information that had been repeatedly blocked through scraping was returned immediately in a clean, structured format. No HTML parsing. No searching through page layouts. No attempts to disguise the request as a browser. The data was already prepared for automated consumption.

However, when I looked at the titles and summaries, I had another thought: Each individual entry is extremely short. There is not much information here.

3. The Question That Changed Everything: What If I Collected 100 Entries?

An RSS feed usually provides only:

A headline
A link
A publication date
A short summary

One entry by itself may not reveal very much. You might read it and simply think: Interesting.

Then I changed my perspective. What would happen if I collected not one entry, but 100 entries? In logistics work, a single complaint or one delayed delivery report may not reveal much. However, when similar reports accumulate to 100 cases, patterns begin to appear. For example:

Delays repeatedly occur on the same route at the same time
The same type of complaint appears again and again
A particular location repeatedly causes problems

I wondered whether RSS summaries worked the same way. Each entry might contain only a small amount of information. But if I collected 100 headlines and summaries, I might be able to identify:

What writers are repeatedly trying to communicate
Topics that continue appearing
Recurring concerns
Frustrations hidden beneath the individual articles
Broader trends that are invisible in one entry

Reading and analyzing 100 entries manually would be exhausting. It would be like sorting every package by hand, one at a time. Then I had the idea: Why not ask AI to summarize and analyze them all at once?

That was the moment puoppo was born.

4. How RSS and the Gemini API Became puoppo

The idea came to me while I was working. I wrote it down immediately so I would not forget it. After returning home, I started turning the idea into an application.

The name also came from a strange chain of associations: Public opinion poll → “pu” + “op” + “po” → puoppo → Pronounced something like “puo-ppo” → That sounds a little like a pigeon → Pigeons remind me of carrier pigeons → Carrier pigeons deliver news → Pigeons are also symbols of peace → All right, that is the name! 🕊️

That is how the name puoppo was created.

The application structure was simple:

Collect article summaries from RSS using feedparser
Send the collected data to the Gemini API
Ask the AI to summarize the content and analyze trends

In logistics terms, it felt like this: Instead of manually sorting every package one by one, I introduced a system that scans and sorts the entire shipment automatically. Information that looked insignificant individually became useful when processed as a group.

The original prototype took approximately ten hours to develop. That prototype later became puoppo_app, a project I still maintain today by adding features and improving its Docker environment.

View puoppo on GitHub (placeholder - actual link not provided in source)

Conclusion: Route Design for Engineers

This experience taught me several important lessons.

1. Look for Officially Published Data Before Using Brute Force

Before scraping a website, check whether it already provides:

An official API
An RSS feed
Public datasets
Export functionality
Structured data intended for reuse

The cleanest route is often better than forcing your way through the front entrance.

2. Small Pieces of Data Become Meaningful at Scale

A single short summary may not contain much information. However, when the number of entries increases, patterns begin to emerge. The value is not always inside one record. Sometimes the value appears only when many records are viewed together.

3. Do Not Stop After Collecting Data

Collecting information is only the first step. The more important question is: How can this collected information be used? The project became valuable only when I connected RSS collection with AI summarization and trend analysis.

Being blocked more than 20 times was frustrating, but those attempts were not meaningless. Without that long detour, I might never have discovered RSS. Without RSS, I might never have imagined puoppo. Sometimes a failed route forces you to search for a better one.

For anyone currently struggling with scraping blocks, check the edges of the website for an RSS feed. There may already be an officially maintained route designed for delivering exactly the information you need. And when you collect many small pieces of data, try looking at them as one larger group. You may discover patterns that were invisible in each individual item.

Thank you for reading! If you found this article useful, a reaction would mean a lot to me.

Read on DEV Community ↗ ← Back to News