See my codes on GitHub. For explanation on codes, please see the above Part 1 post.
The final code for web scraping incorporates the data extraction from the list of universities listed below. Given certain limitations in the web pages that are available and the limitations in my capabilities I have managed to obtain the list of courses from each university. For most universities the link to each course is also obtained from the same location.
As a prospective student looking at starting a degree in an university, it is important to know which universities are offering the courses that you are looking for. In this project I am planning on writing some code to scrape through university pages to get a list of all courses that they offer. As my undergraduate is from the UK, I will be focusing only on UK universities at this time.
When I set-up the Raspberry Pi camera to take pictures every 2 minutes (see my previous post), I did not check up on the files to see if they were saving properly. My goal was to leave the Pi running for at-least 10 days to get a large amount of images to work on some image processing and convolutional neural networks.
When I did my initial test for 1 days I calculated my image file size to be very small (max of about 200kB). So, if there a photo every 2 minutes, that's about 720 photos a day and 7200 photos in 10 days. Which is only 1.4-1.5GB of space - not a problem. Since my Raspberry Pi zero was did not come with WiFi I didn't bother looking at the progress as I didn't want to disturb the camera.
So where did I go wrong? When I did my first test I had a small Python script which took a photo every 5 minutes, but since the python script tends to leave the camera on as long as it is running, I decided to write a small shell script which can also do the same thing and I scheduled the script from Crontab to run every 2 minutes, but I did not check on the new file size. Apparently running the camera this way actually saves an Image of roughly 2.5MB instead of 200kB - Oops. The rest is simple, my Raspberry Pi exceeded by initial estimates by the end of Day 1 and the file system crashed at the end of Day 2. However, I did manage to extract the files from the SD card.
But turns out this is not the reason why my file system crashed. At the end of day 2 I only had about 2.5GB of space on an 8GB SD card. I knew there was no way my files could have exceeded this limit. So why then? When I did some reading I realized that this was to do with the block size. The block size on my Raspbian OS is about 4096 bytes, which means that any file is saved on fixed blocks of 4MB. Whether my image was 200kB or 2.5MB it would have taken up 4MB to save the file. So, looking at it this way it can be easily seen that to save my 2 days worth of images it had actually taken close to 5GB of disk space.
So technically, this attempt would have been a bust either way. Time to look in to a new method of saving files.
Although I built a handful of various types of buggies during university, I have always wanted to build another one. There is something about putting together a buggy and watching it work that is very satisfying.
I recently came across one of my old drawings from high-school of a buggy that I had wanted to build but never got around do it. So I decided to collect up some of bits and bobs and put together a very rudimentary buggy just to make my lock-down weekend a little bit exciting.
From what I remember from when I was drawing this I wanted a bunch of features on it, like;
I currently have a Raspberry Pi camera set up near my window taking a picture of the weather outside my house. Next I have to classify these images to meaningful categories if I am to do anything with this data. This is going to be a very tedious task. So to make things easier for me I built myself a helper using Python and Kivy.
Have you seen mundane and repetitive tasks at your office being automated? (It is very likely that you have!) Then you have already experienced RPA or Robotic Process Automation. There is no point in avoiding it. The technology is here now to make it happen and if you don't do it, someone else will, and this is the reason why you need to embrace it.
RPA is a type of business process automation. It's all about improving your productivity by freeing you up from doing repetitive work that benefits both you and your company. Whether you are a sales associate, manager or engineer there will be a part of your job that can be automated to help you focus your time and energy on what is important. This can even be related to your personal life.
For a future project, I wanted to try and get the history of the price of gold. I did a bit of scouting and landed on the history page of goldprice.org. However, the data was in a chart and I wanted to extract it to a table.
To do this, I wrote a very short web scraping tool which scrapes the data off the chart and formats the data points as a numpy array on python.
I had a small generic Raspberry Pi Camera Rev 1.3 lying around in my parts box for some time and I have always put aside working on it. So I decided to finally take it out and one of my Raspberry Pi zeros to test out the Raspberry Pi Camera.
I have always wanted to learn how to write a code for a web scraper and I finally got around to doing it while at home during lock down. Here is the process on how I got started.
I came across this helpful page on towards data science and got started with learning more about the web features on daraz-Mall and how to navigate them with Beautiful Soup. Having built my initial confidence from reading some more web pages I wanted to scrape daraz.lk/daraz-mall and quickly realized I bit far more than I can handle. The web scraping was successful but the site returned blanks. This was probably because I was using Googles Colab environment to execute my code.
One of the fundamental components in setting up your DIY smart home or IoT project is to set-up the communication method between your devices and the server.
Given the most accessible wireless network to most of us is our home WiFi, I will be using the MQTT protocol to send messages over WiFi. The main server for this will be a Raspberry Pi and the devices (or clients) are ESP8266, ESP32 or similar devices
When you have a number of laptops, phones PCs, TVs and tabs in your house in which you listen to music or watch movies and TV shows, it can be frustrating to sync all your media between each of the devices. I have set-up a Raspberry-Pi based media server in my house to be able to access my music, photos and other media from any device connected to my WiFi.
I do not usually post any work I do on Microsoft Excel on my blog, but recently I was working on a project which forced me to prepare an automated non-preemptive earliest-deadline-first (EDF) scheduler from excel. As I was not able to find any work done on this previously on the web, I decided to share my approach to this problem.
Disclaimer: This is by far not the best approach to a problem of this nature. There are dedicated software that handle complicated functions like this. But, this is a moderately simple method to achieve the same using Microsoft Excel.
Please also note that there may be some errors that can arise as I have not extensively tested this method outside the scope of what I specifically need this for.
Scenario:
There is a list of tasks that needs to be done among 4 people. The tasks are individually assigned and they arrive at different times. Each task takes a fixed 3 working days to complete and has it's own deadline before which it needs to be completed. The work assigned to each person must be scheduled such that the deadline is prioritized as in an non-preemptive earliest-deadline-first (EDF) scheduler.
Below is a dashboard of the data I collected from the invitation scanning device made from Google Data Studio. Please note that the date's are arbitrary for privacy reasons, but the time is correct.
Social distancing is one of the top recommended ways to stop the spread of CoViD-19 disease by the WHO. It is also important to make sure that this is enforced at all times as people can be absent minded or accidentally come close to each other. This is especially true in crowded areas like hospitals and work places.
To mcontinuously onitor and enforce this I have come up with a simple ESP based device. The idea is to provide everyone working in one location with this device that will continuously monitor if there is a similar device around them. If there is another similar distance around them close to a preset distance of roughly 1 meter or 3 meters according to the regulations in the location and use of PPE, the device will light up or ring loudly alerting the wearer to move away from the location or to be extra cautious.
This is a custom dashboard that I designed for my Raspberry Pi attached to my Tabletop display which you can see here -> https://johnsamarasinghe.blogspot.com/2020/03/tabletop-screen.html. The dashboard design in Kivy with Python, allows you to view up to six projects at once, along with an about section and link to view all other projects.
This is a project that I wanted to do for a quite a while.
I had a very old laptop where the screen had come off and I wasn't able to use the laptop any more. So I decided to embed the screen to my electronics table so that I can use the screen to display some information that I want to look up while working.
Invitation cards are one of the key components of a big event. It provides a description of the event, time, venue, dress code and other important information the invitee needs to know. These are generally a one way mode of communication. The organizer hands out the invitation card to the invitees, the invitees make a note of the details and that's the end of that.
For my wedding I wanted to push the limits of wedding invitation cards in order to provide some ease for the invitees. In order to achieve this I pasted a cheap NFC tags to the back of each invitation card with a simple message to describe what it is. As some people do not prefer this, the NFC can also be embedded within the invitation card for a premium feel.
The purpose of using an NFC tag was so that I can allocate a unique identification code to each invitation and link it to a table number or any other information that is needed. Generally table numbers are not known when handing out the invitation cards. These are usually assigned closer to the day of the event and the guests can scan their invitation cards at the entrance on the day of the event and be presented with their table number(s). However, as I was working on this project as a hobby, a number uses for this presented itself.
Benefits of using the NFC tag.
Added security - Officials at the event location can scan the invitation card to verify the identity of the invitee
Avoid congestion & crowding - Invitee can avoid the queue to check the table numbers at the entrance
Custom messages - Invitees can be displayed with a custom message on arrival
Photo booth - Scanning the invitation can launch a photo-booth that can take the picture of the invitee and automatically email it to them.
This article consists of instructions on how you run your python code on a stand-alone Raspberry pi connected to a screen while updating your code from your own personal computer. For this project, my main intention is to have a dashboard that will be displayed on a large screen connected to a Raspberry Pi while all the codes can be updated using a computer that is not connected to the Raspberry Pi. My dashboard will be a Kivy app running on python. I will be using a Raspberry Pi 3 B to display the dashboard and I will use a Basic Onedrive account to synchronize the folder structure and the python code between my computer and the raspberry pi.
