Scraping University Courses (part 2)

Previous Post: Part 1

See my codes on GitHub. For explanation on codes, please see the above Part 1 post.

The final code for web scraping incorporates the data extraction from the list of universities listed below. Given certain limitations in the web pages that are available and the limitations in my capabilities I have managed to obtain the list of courses from each university. For most universities the link to each course is also obtained from the same location.

The final output can be found here -> csv

Scraping University Courses (part 1)

 As a prospective student looking at starting a degree in an university, it is important to know which universities are offering the courses that you are looking for. In this project I am planning on writing some code to scrape through university pages to get a list of all courses that they offer. As my undergraduate is from the UK, I will be focusing only on UK universities at this time.

What is RPA and why you need to get onboard right now!

Have you seen mundane and repetitive tasks at your office being automated? (It is very likely that you have!) Then you have already experienced RPA or Robotic Process Automation. There is no point in avoiding it. The technology is here now to make it happen and if you don't do it, someone else will, and this is the reason why you need to embrace it.

Photo by Photos Hobby on Unsplash

RPA is a type of business process automation. It's all about improving your productivity by freeing you up from doing repetitive work that benefits both you and your company. Whether you are a sales associate, manager or engineer there will be a part of your job that can be automated to help you focus your time and energy on what is important. This can even be related to your personal life.

Scraping Gold

 For a future project, I wanted to try and get the history of the price of gold. I did a bit of scouting and landed on the history page of goldprice.org. However, the data was in a chart and I wanted to extract it to a table.

To do this, I wrote a very short web scraping tool which scrapes the data off the chart and formats the data points as a numpy array on python.

The GitHub code can be found here.

Scraping Daraz.lk

I have always wanted to learn how to write a code for a web scraper and I finally got around to doing it while at home during lock down. Here is the process on how I got started.

GitHub site: https://github.com/iamJohnnySam/WebScraping

I came across this helpful page on towards data science and got started with learning more about the web features on daraz-Mall and how to navigate them with Beautiful Soup. Having built my initial confidence from reading some more web pages I wanted to scrape daraz.lk/daraz-mall and quickly realized I bit far more than I can handle. The web scraping was successful but the site returned blanks. This was probably because I was using Googles Colab environment to execute my code. 

I next decided to take this one step at a time.