Scraping University Courses (part 2)

Previous Post: Part 1

See my codes on GitHub. For explanation on codes, please see the above Part 1 post.

The final code for web scraping incorporates the data extraction from the list of universities listed below. Given certain limitations in the web pages that are available and the limitations in my capabilities I have managed to obtain the list of courses from each university. For most universities the link to each course is also obtained from the same location.

The final output can be found here -> csv

Scraping University Courses (part 1)

 As a prospective student looking at starting a degree in an university, it is important to know which universities are offering the courses that you are looking for. In this project I am planning on writing some code to scrape through university pages to get a list of all courses that they offer. As my undergraduate is from the UK, I will be focusing only on UK universities at this time.

Back to Basics: Building a Buggy

Although I built a handful of various types of buggies during university, I have always wanted to build another one. There is something about putting together a buggy and watching it work that is very satisfying.

I recently came across one of my old drawings from high-school of a buggy that I had wanted to build but never got around do it. So I decided to collect up some of bits and bobs and put together a very rudimentary buggy just to make my lock-down weekend a little bit exciting.

From what I remember from when I was drawing this I wanted a bunch of features on it, like;

• Pull out control panel
• Automatic docking for charging
• Use as balance-bot and
• Line following

Scraping Gold

 For a future project, I wanted to try and get the history of the price of gold. I did a bit of scouting and landed on the history page of goldprice.org. However, the data was in a chart and I wanted to extract it to a table.

To do this, I wrote a very short web scraping tool which scrapes the data off the chart and formats the data points as a numpy array on python.

The GitHub code can be found here.

Raspberry Pi Camera Time-lapse

 Previous Post: Setting up your Raspberry Pi Camera

Once I set-up my Raspberry Pi camera, I wrote a simple script to take a picture from my camera every 2 minutes.

Setting up WiFi communication between Raspberry Pi and ESP8266 with MQTT

One of the fundamental components in setting up your DIY smart home or IoT project is to set-up the communication method between your devices and the server.

Given the most accessible wireless network to most of us is our home WiFi, I will be using the MQTT protocol to send messages over WiFi. The main server for this will be a Raspberry Pi and the devices (or clients) are ESP8266, ESP32 or similar devices

Microsoft Excel Based Task Scheduler (non-preemptive earliest-deadline-first (EDF))

I do not usually post any work I do on Microsoft Excel on my blog, but recently I was working on a project which forced me to prepare an automated non-preemptive earliest-deadline-first (EDF) scheduler from excel. As I was not able to find any work done on this previously on the web, I decided to share my approach to this problem.

Disclaimer: This is by far not the best approach to a problem of this nature. There are dedicated software that handle complicated functions like this. But, this is a moderately simple method to achieve the same using Microsoft Excel.

Please also note that there may be some errors that can arise as I have not extensively tested this method outside the scope of what I specifically need this for.

Scenario:

There is a list of tasks that needs to be done among 4 people. The tasks are individually assigned and they arrive at different times. Each task takes a fixed 3 working days to complete and has it's own deadline before which it needs to be completed. The work assigned to each person must be scheduled such that the deadline is prioritized as in an non-preemptive earliest-deadline-first (EDF) scheduler.

NFC Enabled Invitations Cards (+ Analytics for a Sri Lankan Wedding)

Invitation cards are one of the key components of a big event. It provides a description of the event, time, venue, dress code and other important information the invitee needs to know. These are generally a one way mode of communication. The organizer hands out the invitation card to the invitees, the invitees make a note of the details and that's the end of that.

For my wedding I wanted to push the limits of wedding invitation cards in order to provide some ease for the invitees. In order to achieve this I pasted a cheap NFC tags to the back of each invitation card with a simple message to describe what it is. As some people do not prefer this, the NFC can also be embedded within the invitation card for a premium feel.

The purpose of using an NFC tag was so that I can allocate a unique identification code to each invitation and link it to a table number or any other information that is needed. Generally table numbers are not known when handing out the invitation cards. These are usually assigned closer to the day of the event and the guests can scan their invitation cards at the entrance on the day of the event and be presented with their table number(s). However, as I was working on this project as a hobby, a number uses for this presented itself.

Benefits of using the NFC tag.

• Added security - Officials at the event location can scan the invitation card to verify the identity of the invitee
• Avoid congestion & crowding - Invitee can avoid the queue to check the table numbers at the entrance
• Custom messages - Invitees can be displayed with a custom message on arrival
• Photo booth - Scanning the invitation can launch a photo-booth that can take the picture of the invitee and automatically email it to them.

Remotely Updatable Raspberry Pi based Dashboard

This article consists of instructions on how you run your python code on a stand-alone Raspberry pi connected to a screen while updating your code from your own personal computer.

For this project, my main intention is to have a dashboard that will be displayed on a large screen connected to a Raspberry Pi while all the codes can be updated using a computer that is not connected to the Raspberry Pi. My dashboard will be a Kivy app running on python. I will be using a Raspberry Pi 3 B to display the dashboard and I will use a Basic Onedrive account to synchronize the folder structure and the python code between my computer and the raspberry pi.

Project SpiderWeb

Project SpiderWeb is my domestic answer to the fourth industrial revolution. IoT is one of the key words that gets tossed around during these times. As there is much potential in what can be achieved with devices across the house that talk to each other and exchange and analyse data, I have started my own IoT project (SpyderWeb) - for the love of technology, of course!

Variable Power Outlet and AC to DC converter

Variable A/C power outlet

I came up with this project mainly to control the performance of some appliances that I use on a daily basis that do not offer output control. For example, I can control the temperature of my 60W soldering iron or a dim a desk lamp that is connected to the unit. It does so by adjusting the output voltage using a simple triac circuit.