Scraping University Courses (part 2)

Previous Post: Part 1

See my codes on GitHub. For explanation on codes, please see the above Part 1 post.

The final code for web scraping incorporates the data extraction from the list of universities listed below. Given certain limitations in the web pages that are available and the limitations in my capabilities I have managed to obtain the list of courses from each university. For most universities the link to each course is also obtained from the same location.

The final output can be found here -> csv

Scraping University Courses (part 1)

 As a prospective student looking at starting a degree in an university, it is important to know which universities are offering the courses that you are looking for. In this project I am planning on writing some code to scrape through university pages to get a list of all courses that they offer. As my undergraduate is from the UK, I will be focusing only on UK universities at this time.

Back to Basics: Building a Buggy

Although I built a handful of various types of buggies during university, I have always wanted to build another one. There is something about putting together a buggy and watching it work that is very satisfying.

I recently came across one of my old drawings from high-school of a buggy that I had wanted to build but never got around do it. So I decided to collect up some of bits and bobs and put together a very rudimentary buggy just to make my lock-down weekend a little bit exciting.

From what I remember from when I was drawing this I wanted a bunch of features on it, like;

• Pull out control panel
• Automatic docking for charging
• Use as balance-bot and
• Line following

What is RPA and why you need to get onboard right now!

Have you seen mundane and repetitive tasks at your office being automated? (It is very likely that you have!) Then you have already experienced RPA or Robotic Process Automation. There is no point in avoiding it. The technology is here now to make it happen and if you don't do it, someone else will, and this is the reason why you need to embrace it.

Photo by Photos Hobby on Unsplash

RPA is a type of business process automation. It's all about improving your productivity by freeing you up from doing repetitive work that benefits both you and your company. Whether you are a sales associate, manager or engineer there will be a part of your job that can be automated to help you focus your time and energy on what is important. This can even be related to your personal life.

Setting up WiFi communication between Raspberry Pi and ESP8266 with MQTT

One of the fundamental components in setting up your DIY smart home or IoT project is to set-up the communication method between your devices and the server.

Given the most accessible wireless network to most of us is our home WiFi, I will be using the MQTT protocol to send messages over WiFi. The main server for this will be a Raspberry Pi and the devices (or clients) are ESP8266, ESP32 or similar devices

Home-made Raspberry Pi Media Center

When you have a number of laptops, phones PCs, TVs and tabs in your house in which you listen to music or watch movies and TV shows, it can be frustrating to sync all your media between each of the devices. I have set-up a Raspberry-Pi based media server in my house to be able to access my music, photos and other media from any device connected to my WiFi.

Powered by:

Microsoft Excel Based Task Scheduler (non-preemptive earliest-deadline-first (EDF))

I do not usually post any work I do on Microsoft Excel on my blog, but recently I was working on a project which forced me to prepare an automated non-preemptive earliest-deadline-first (EDF) scheduler from excel. As I was not able to find any work done on this previously on the web, I decided to share my approach to this problem.

Disclaimer: This is by far not the best approach to a problem of this nature. There are dedicated software that handle complicated functions like this. But, this is a moderately simple method to achieve the same using Microsoft Excel.

Please also note that there may be some errors that can arise as I have not extensively tested this method outside the scope of what I specifically need this for.

Scenario:

There is a list of tasks that needs to be done among 4 people. The tasks are individually assigned and they arrive at different times. Each task takes a fixed 3 working days to complete and has it's own deadline before which it needs to be completed. The work assigned to each person must be scheduled such that the deadline is prioritized as in an non-preemptive earliest-deadline-first (EDF) scheduler.

NFC Enabled Invitations Cards (+ Analytics for a Sri Lankan Wedding)

Invitation cards are one of the key components of a big event. It provides a description of the event, time, venue, dress code and other important information the invitee needs to know. These are generally a one way mode of communication. The organizer hands out the invitation card to the invitees, the invitees make a note of the details and that's the end of that.

For my wedding I wanted to push the limits of wedding invitation cards in order to provide some ease for the invitees. In order to achieve this I pasted a cheap NFC tags to the back of each invitation card with a simple message to describe what it is. As some people do not prefer this, the NFC can also be embedded within the invitation card for a premium feel.

The purpose of using an NFC tag was so that I can allocate a unique identification code to each invitation and link it to a table number or any other information that is needed. Generally table numbers are not known when handing out the invitation cards. These are usually assigned closer to the day of the event and the guests can scan their invitation cards at the entrance on the day of the event and be presented with their table number(s). However, as I was working on this project as a hobby, a number uses for this presented itself.

Benefits of using the NFC tag.

• Added security - Officials at the event location can scan the invitation card to verify the identity of the invitee
• Avoid congestion & crowding - Invitee can avoid the queue to check the table numbers at the entrance
• Custom messages - Invitees can be displayed with a custom message on arrival
• Photo booth - Scanning the invitation can launch a photo-booth that can take the picture of the invitee and automatically email it to them.

Remotely Updatable Raspberry Pi based Dashboard

This article consists of instructions on how you run your python code on a stand-alone Raspberry pi connected to a screen while updating your code from your own personal computer.

For this project, my main intention is to have a dashboard that will be displayed on a large screen connected to a Raspberry Pi while all the codes can be updated using a computer that is not connected to the Raspberry Pi. My dashboard will be a Kivy app running on python. I will be using a Raspberry Pi 3 B to display the dashboard and I will use a Basic Onedrive account to synchronize the folder structure and the python code between my computer and the raspberry pi.

Project SpiderWeb

Project SpiderWeb is my domestic answer to the fourth industrial revolution. IoT is one of the key words that gets tossed around during these times. As there is much potential in what can be achieved with devices across the house that talk to each other and exchange and analyse data, I have started my own IoT project (SpyderWeb) - for the love of technology, of course!

“The Everything Box”

I always wanted to work on a home automation project as it is very interesting and challenging to design a circuit to control most electrical appliances in a house. It’s not possible to mess around with the electricals in my room which I stay for rent. So I decided to make an automated box. I call it “The Everything Box”.

The Everything Box will house various circuits designed for different tasks. The circuits are not limited to the inside of the box. Some of them will be wired out of the box to control appliances such as desk lamps etc. Some of the circuits that I have decided to put in the box are:

• Automated Desk Lamp and other lights
• Back up power Circuit
• Automated cooling fan to maintain temperature
• Data logging
• Music Player
• More circuits will be added in the future

To get started with the project I took an old shoe box that was lying around. I cut some holes for the basic items such as the fan and buttons and painted the box in shiny black.

 

The box currently has the switches shown, a cooling fan, a number pad and a blue el wire running along the inside top of the box to illuminate the circuits when the box is opened. I will post updates on this blog when I add new circuits to The Everything Box.

Let me know what you think of this project and also let me know of some new suggestions for the circuits in the comments below. Thank you for reading.