I don't know how scrapers work. I wanted one anyway.
The itch was small and specific: my university ERP makes me click through four pages to see one number, my attendance sits behind a login that expires every few hours, and course registration opens at some ungodly hour (for me) and the server falls over the second it does. I wanted a thing that would just do all this for me while I was doing something else, ideally sleeping.
The problem is that having my eyes closed turns out to be the entire difficulty, and I didn't find that out until the bots started lying to me.
I didn't want a scraper, I wanted a base
I didn't want to write one bot. I wanted a base I could clone for whatever site annoyed me next — attendance today, course registration in three weeks, some price or restock later. So the first decision wasn't code, it was picking the one boring thing everything else hangs off: drive a real browser, don't fake HTTP requests. A real browser can do anything I can do on a site because it is a browser. The tool for that is Playwright, and the reason I picked it over the older options is one feature — it ships a recorder.
That recorder matters more than it sounds, because it means I never actually write the part I don't understand. I click through the task by hand once, and it writes the Python. I don't know how to address "the third button inside the login iframe." I don't have to. I just have to know how to click it.
Once again, the best code is the code you don't write. Here it was the code I didn't know how to write in the first place.
The first thing that betrayed me worked perfectly, once
I recorded the attendance task. It ran. I was thrilled for about 3 hours.
Then it died. Same code, same site, nothing changed. I re-recorded it, it worked again, and then it died again the day after. Being new to this I assumed I'd done something wrong each time — until I actually looked at what the recorder had written down, and found this sitting in the middle of my selectors:
iframe[name="8DEF612E3BDC1AC69CD3875DA76511DB"]That 32-character blob is not an address. It's a fresh random id the site mints every time you log in. The recorder had faithfully written down this session's id, which is exactly as useful as writing down today's parking spot number and expecting your car there tomorrow. Every re-recording baked in a new one, which is why re-recording "fixed" it for exactly one day at a time.
This taught me quite a bit, and annoyed me quite a bit more, honestly.
Download, not the box with the random
name around it.
Then a browser lied to me depending on whether I was watching
Next site, the fee page, opened its report in a popup. Watching the browser do it, everything worked. Running it for real — headless, no window, the way it'd actually run at night — the popup came up completely blank and sat there forever.
Same code. The only difference was whether a window was on my screen.
It turns out that when you run Chrome "headless" it quietly uses a stripped-down build of itself, and that stripped-down build handles some popup and JavaScript things differently than the real one. So the site rendered fine when I was looking and rendered nothing when I wasn't. This is a genuinely unsettling category of bug — it only exists when you're not there to see it — and it's the exact bug an unattended bot is built to walk into.
The fix was one line telling it to use the full browser even with no window.
The page that pretended to load
Same popup, once it wasn't blank anymore, had one more trick. I told the bot, reasonably, to wait for the page to finish loading before grabbing the report. It waited. And waited. And timed out, every time, on a page that had visibly finished loading ages ago.
Some pages never actually fire the "I'm done" signal — this one opens as an empty shell and quietly stuffs the content in later, so "wait until loaded" waits for a moment that never comes. The answer isn't to wait for the page, it's to wait for the thing you actually want — poll until the Download button exists, then move. Stop trusting the page to tell you it's ready. Watch for the specific thing and ignore everything else.
The base isn't clever. It's paranoid, on purpose.
Here's the thing I didn't expect going in. I thought the hard part would be the scraping, the clicking, the selectors, the clever bits. It wasn't. The recorder writes the clever bits. The hard part was that I'm not there.
So almost all the actual engineering ended up being about that absence. Every time a task dies, it screenshots every open window, saves the page, and sends the picture to my phone with the error. Not a log I have to go read — the actual image of what the bot saw at the moment it gave up. Most of the time the answer was in the picture: a login wall I didn't expect, a cookie banner over the button, a page I'd never seen. I fixed every single one of the bugs above by looking at that screenshot, not by thinking about it.
The bots run on my old laptop in the corner that never sleeps. When my attendance shows up on my phone at 7:57AM every morning, I don't think about any of this. That's the whole point — the paranoia is upfront so the mornings are boring.
I still don't really know how scrapers work
I want to be honest about that. I couldn't write one of these selectors from scratch to save my life. What I have instead is a small base that survives me: a recorder that writes the parts I don't understand, and a wrapper around it whose entire job is to fail loudly, on my phone, with a picture, so that "3am with nobody home" is a solvable problem instead of a silent one.
Five bots run on that laptop now. I learned more about the web from the four ways they broke my heart and mind than from anything that went right.