BRANDON.T.BANDE // ROGUE0015

HOME / THOUGHTS

Scraping 16,000 Jobs: ETL vs ELT and Lessons Learnt, Finale

· 5 MIN READ · ·

  • developers
  • zimbabwe

Hi,

I'm Brandon.

In the past two articles (Part 1 and Part 2), I have been going on about the thought process and the hands-on application of the concepts I ended up adopting. To wrap up the series, I will be talking less about the hands-on tech stuff and reflecting on the lessons learnt.

The basics matter

In two cases, my work would have been easier if I had known the basics. To be specific:

  1. The approach to just get data, extract the info and then tabulate it is not as simple as A, B, C
  2. If I had known the different ways of scraping data, I wouldn't have spent a few days learning Selenium and how to extract from WordPress

I have always prided myself in being self-taught. Then I coined the term 'short-cut developer' to describe myself. I am proud that with no academic or guided foundation, I got on the internet and taught myself to code. But I am also aware of my limitation that things should not just work. They should work effectively, efficiently, accurately and a bunch of other important words if a system is to serve its purpose. Back to the story.

ETL vs ELT

Extract. Transform. Load.

I had heard about the abbreviations before. I had a rough idea, but I didn't know where they applied. I guess booksmart is not really street smart. In my flow it was something like this:

  • Getting page content about a job from the internet (5 seconds)
  • Extracting the actual fields needed using an LLM on my machine (2-6 minutes and totally offline)
  • Writing to my local database (~1 second)

So initially I would get what I want after a few minutes. That worked!!! Now imagine this at scale. To cycle through 16000 jobs would take about 533 to 1600 hours of continuous compute. That is like 22 continuous days at least. The process where I obtained the page, extracted the info (transformed it) and then left it for my colleague to load it, is ETL.

The cure to my pain

Hidden from my plain sight was a principle that applies when you have such a problem. I could just store as many jobs as possible on my personal computer and then do the transformation at my own time. That is what ELT does. In my case that was the best approach because:

  1. If my internet went down - work could still go on
  2. I could provide up-to-date data to my colleague at regular intervals

To top it off, I could run a few things at the same time. To achieve that, I ended up indexing all the pages that had job listings. Then following that index, I could load page by page to my local dump for the raw job text. At times, using concepts like threading, I was able to achieve that. Then in the background the transformation would be taking place.

Another thing that came to mind is the fact that maybe in the future, if I realise that I missed something key, I could still get the raw dump and transform it again. (I did that, interestingly, I could expand on the already loaded data)

Disclaimer

One might argue that what I did is still ETL. Technically, depending on where you draw the boundaries of extraction and transformation, they could be right. But the important difference for me was what happened after I received the data.

Instead of transforming every job immediately, I extracted the raw content, loaded it locally, and transformed it later.

Extract. Load. Transform. ELT.

A bit about scraping

I think in the first article, I talked about the different approaches that I adopted as trial and error. And the key takeaway was that different strokes for different folks, I guess. Another thing that I should address is whether this was done the right way. There are a lot of grey areas when it comes to scraping content that is in the public domain. I did this under the guise of educational research and the data is not for commercial use. I also adopted respectful scraping to avoid flooding the sources with multiple requests. And I am also open to discussing methods of respectful scraping that I can adopt in the future.

Writing this series is not merely about documenting my "genius moments", but sharing the realisation that being a short-cut developer has its own limits. Building something real makes you realise the shortfalls and at times eating your shoe, admitting and acknowledging your limits is the only way to grow. And aha, I now know where to use ETL. 18,693 job posts say so :)

Finale. Curtains close.

Let's connect via the following platforms: LinkedIn, on X as @Rogue0015, or check the Sessionize speaker profile if you want this signal live on your stage. And if you're new here, start with the talks — this blog is their echo.