Today I learned about Marion Stokes' TV archive

Today I learned about Marion Stokes, a woman from Philadelphia, Pennsylvania, who recorded hundreds of thousands of hours of television news footage spanning 35 years, from 1977 until her death in 2012. At her death, she had grown a collection of about 71,000 VHS and Betamax tapes. Her archivist and compulsive hoarding tendency also resulted in a collection of 30,000 to 40,000 books.

In 2019, a documentary film was made about her, and her archives named Recorder: The Marion Stokes Project.

Her collection was archived by the Internet Archive which is available here.

Chandrayaan-3 landed on the moon

Today, August 23, 2023, ISRO's Chandrayaan-3 lunar exploration mission successfully landed on the moon at 6:04 PM IST (5:34 AM PT). The mission was launched on a Bahubali LVM3-M4 rocket carrying a lander named Vikram and a rover named Pragyan on July 14 this year which I had logged here.

With this landing, India has become the fourth nation to make a soft landing on the moon, after the former Soviet Union, the U.S. and China. It has also become the first country to land on the lunar south pole, which is still an unexplored area.

ISRO's Tweet on the successful landing:

Chandrayaan-3 Mission:
'India,
I reached my destination
and you too!'
: Chandrayaan-3

Chandrayaan-3 has successfully
soft-landed on the moon 🌖!.

Congratulations, India!#Chandrayaan_3#Ch3

— ISRO (@isro) August 23, 2023

Below is the saved live stream of the landing on YouTube. The final phase of the descent starts at 35:13.

ISRO tweeted the first images received from the lander:

Chandrayaan-3 Mission:
Updates:

The communication link is established between the Ch-3 Lander and MOX-ISTRAC, Bengaluru.

Here are the images from the Lander Horizontal Velocity Camera taken during the descent. #Chandrayaan_3#Ch3 pic.twitter.com/ctjpxZmbom

— ISRO (@isro) August 23, 2023

Earlier on November 10, 2009, with the Chandrayaan-1 mission, ISRO made a deliberate crash landing of its Moon Impact Probe (MIP) which helped discover water molecules on the moon.

Today I Learned: All coal was created around the same time

There is an ongoing debate on whether almost all coal was made around the same time, during a brief period 360 to 300 million years ago.

Photo by Nikolay Kovalenko on Unsplash

We are using the coal that was created during this period and almost no new coal has formed since then.

Steve Mould provides an easy-to-understand explanation in this video on YouTube.

The Elk Valley Coal News has a detailed article on this here.

ChatGPT Prompts: A barebones list

With new advances in artificial intelligence and its subsets machine learning and deep learning, there are many applications based on large language models that generates human-like text. The common thing in all these applications is the need of prompts so they can come up with appropriate responses. This post contains a barebones list of prompts for ChatGPT that can be used to boost productivity.

Counter reply: Write a witty counter reply to the following statement: [insert statement here]

Meeting agenda: Draft an effective meeting agenda given [insert your points here].

Email communication: Draft a polite yet effective email response for [insert your situation].

Elevator pitch: Create an elevator pitch for [insert product or idea].

Social media captions: Create an engaging caption for this: [describe social media content].

Simplify explanations: Explain the following in simple terms: [insert text here].

Really simple explanations: ELI5: [insert text here].

Project timelines: Based on the following milestones, create a project timeline: [insert milestones].

Marketing slogans: Create a catchy marketing slogan for [insert product description].

Professional online summary: Write a professional summary for my LinkedIn profile, given these skills and experiences [insert information].

Business language: Translate this text into professional business language: [insert text]

Business strategy: Generate a business strategy for a startup in the [insert industry] sector.

Product description: Write a compelling product description for [insert product].

Efficient to-do lists: Prioritize and organize these tasks into a to-do list [insert tasks].

Optimize content for SEO: Improve the SEO of this blog post: [insert blog post].

Creative advertisements. Generate a creative advertisement concept for promoting [insert product description].

Persuasive sales copy: Generate a persuasive sales copy for [insert product description].

Optimized workflow: Create an optimized workflow based on these tasks [insert tasks].

Innovation strategies: Suggest strategies for driving innovation in the [insert industry] sector.

Team motivation: Draft a motivational message to my team about [insert topic].

Team building activities: Suggest some creative team building activities for a remote team.

Impactful presentations: Design a powerful slide deck for a presentation on [insert topic].

Fundraising proposals: Create a compelling fundraising proposal for [insert cause].

Project proposals: Draft a comprehensive project proposal for [insert project description].

Recruitment ads: Compose a job ad for the position of [insert job title].

General interview questions: Provide a professional and thoughtful response to this interview question: [insert question].

Crisis management plans. Create a crisis management plan for [insert crisis].

Miscellaneous ChatGPT Prompts

What are 5 creative things I could do with my kids' art? I don't want to throw them away, but it's also so much clutter.

Using Python and BeautifulSoup to web scrape Times of India news headlines

According to Wikipedia, "web scraping, web harvesting, or web data extraction is data scraping used for extracting data from websites." In this post, we will create a small program in Python to scrape top headlines from Times of India's news headlines page using the BeautifulSoup library.

Top news headlines

Particularly, our program will fetch the Times of India Headlines page and extract the prime news headlines on the top of the page. As of this writing, the page displays 6 headlines in that section which we want to scrape. In this screenshot of the webpage, our point of interest is the highlighted section which contains the top 6 headlines.

The programming language we will use is Python 3. Along with that, we will also use the BeautifulSoup 4 package for parsing the HTML. I will assume that you already have a system on which these prerequisites are installed and ready to run. I will also assume that you have a Python editor and compiler to compile and run the program. For the purpose of this illustration, I will use Google Colab to write and execute the python code.

Part 1: Scrape the website

To start with, we will write a simple code that fetches the data and outputs the scraped text in the editors output window. Type the following code in your Python editor. I will explain the code later in this post. A copy of this code is available at my TOITopHeadlines v1.0 repository on Github.

# This program scrapes a web page from Times of India to extract

# top headlines and prints it in the output window.

import requests

from bs4 import BeautifulSoup

def toi_topheadlines():

  url = "https://timesofindia.indiatimes.com/home/headlines"

  page_request = requests.get(url)

  page_content = page_request.content

  soup = BeautifulSoup(page_content,"html.parser")

  count = 1

  for divtag_c02 in soup.find_all('div', {'id': 'c_02'}):

    for divtag_0201 in divtag_c02.find_all('div', {'id': 'c_0201'}):

      divtag_hwdt1 = divtag_0201.find('div', {'id': 'c_headlines_wdt_1'})

      for divtag_topnl in divtag_hwdt1.find_all('div',

       {'class': 'top-newslist'}):

        for ultag in divtag_topnl.find_all('ul',{'class': 'clearfix'}):

          for litag in ultag.find_all('li'):

            for spantitle in litag.find_all('span', {'class': 'w_tle'}):

              href = spantitle.find('a')['href']

              if href.find("/", 0) == 0:

                href = "https://timesofindia.indiatimes.com" + href

                print(str(count) + ". " + spantitle.find('a')['title'] +

                      " - " + href)

                count = count + 1

if __name__ == "__main__":

  toi_topheadlines()

print("\n" + "end")

Executing the code will make it extract the HTML from the URL, parse out the required data and output the list of news headline titles and respective URLs as highlighted in the screenshot below:

News headlines scraped using Python

If you have managed to get that working, congratulations. You have scraped the top headlines and now you can use it in your own creative ways. Next, we will delve into what we did and what got us here.

Now, take a look at the portion of the source code that goes through a chain of for loops to crawl into the HTML tags. This exactly corresponds to the way that the markups are structured in the web page. You can take a look at the HTML markups by going to the browser's Developer Tools and inspecting the code behind the UI elements.

Inspecting HTML tag structure

Your program has to be tuned according to the HTML markup structure of the page that you are trying to scrape.

Part 2: Write the scraped data to a file in Google Drive

Now that our program can successfully scrape the data, in this section, we will take a step forward and write the scraped data into a JSON file in Google Drive. We will continue to use Google Colab to run the program.

For this we will mount a root folder in Google Drive and create a folder to store our files. We use a list of dictionaries and then use Python's JSON library to write the list to a JSON file. A copy of this code is available at my TOITopHeadlines v2.0 repository on Github.

# This program scrapes a web page from Times of India to extract

# top headlines and writes it to a JSON file in Google Drive.

import requests

import datetime

import json

from bs4 import BeautifulSoup

# Prepare file location

import os

from google.colab import drive

strDriveMountLoc = '/content/drive'

strDriveTargetLoc = "/content/drive/My Drive/WebScrape/DataNewsScrapeTOI"

# Mount Google Drive

drive.mount(strDriveMountLoc)

# Create a folder in the root directory

!mkdir -p "/content/drive/My Drive/WebScrape/DataNewsScrapeTOI"

def toi_topheadlines():

  # Generate output filename based on the date and time

  dt = datetime.datetime.now()

  filename = "toi_topheadlines" + dt.strftime("%Y%m%d%H%M%S") + ".json"

  url = "https://timesofindia.indiatimes.com/home/headlines"

  page_request = requests.get(url)

  page_content = page_request.content

  soup = BeautifulSoup(page_content,"html.parser")

  count = 1

  txtscraped = ""

  headlines = []

  for divtag_c02 in soup.find_all('div', {'id': 'c_02'}):

    for divtag_0201 in divtag_c02.find_all('div', {'id': 'c_0201'}):

      divtag_hwdt1 = divtag_0201.find('div', {'id': 'c_headlines_wdt_1'})

      for divtag_topnl in divtag_hwdt1.find_all('div',

       {'class': 'top-newslist'}):

        for ultag in divtag_topnl.find_all('ul',{'class': 'clearfix'}):

          for litag in ultag.find_all('li'):

            for spantitle in litag.find_all('span', {'class': 'w_tle'}):

              href = spantitle.find('a')['href']

              if href.find("/", 0) == 0:

                href = "https://timesofindia.indiatimes.com" + href

                print(str(count) + ". " + spantitle.find('a')['title'] +

                      " - " + href)

                thisheadline = {

                    "sn": count,

                    "title": spantitle.find('a')['title'],

                    "href": href

                }

                headlines.append(thisheadline)

                count = count + 1

  with open(strDriveTargetLoc + '/' + filename, "a") as f:

    f.write(json.dumps(headlines, indent=2))

if __name__ == "__main__":

  toi_topheadlines()

print("\n" + "end")

Executing the code in Google Colab will display a prompt to connect to Google Drive and then take you through a series of pages to authenticate using your Google id. Once you are past the authentication the code should execute and create a JSON file in the folder path that you chose in the program. Below you can see how a list of files would look like.

JSON files in Google Drive

The content of the JSON file would look similar to what you see below.

The JSON output

In Conclusion

A word of caution on using the web scraping method is that while many websites don't mind, there are many who don't like it. It is best to go through their terms of service to understand the limitations they apply to what you can do with the data and ensure that you are not in violation. Another important point to remember is that many websites periodically change their look and feel hence modifying the structure of the HTML. On the face of such changes, your web scraping logic may fall flat, hence web scrapers need continuous maintenance. A better way to capture and harvest such data is to use APIs published by the web sites. This demonstration is for academic purposes only.

Happy scraping!

Ohio Electricity Litigation Settlement Distribution Payouts

Customers of First Energy Ohio today received an email with the subject Ohio Electricity Litigation: Your Digital Payment is Ready stating the distribution of settlement fund against Smith v. FirstEnergy Corp., et al., Case No. 2:20-cv-3755.

For a background, this is based out of two class action lawsuits against FirstEnergy and Energy Harbor, alleging that they engaged in a bribery scheme to pass a law that increased electricity rates for some Ohio residents. The lawsuits have reached a settlement of $49 million for the affected customers. Customers who have paid any rates or fees under Ohio House Bill 6 ("HB 6") from January 1, 2020 to June 22, 2022 to Toledo Edison, Cleveland Electric, or Ohio Edison, may be eligible for a payment out of this settlement.

The official website for the class action lawsuit is here.

Below is the text of the email.

Ohio Electricity Litigation settlement distribution email

Getting the payment involves clicking on a button to receive a Virtual Prepaid Mastercard which can be added to your preferred digital wallet (Google Pay, Apple Pay, and Samsung Pay).

Virtual Prepaid Mastercard

Data Visualization using Sankey diagrams

Sankey diagrams are a great way of visualizing the flow of data, where the width of each flow is based on its quantity or relative weight. Like a picture that speaks more than a thousand words, a Sankey diagram speaks more than a thousand pie charts. Perfect use cases of these diagrams are to visualize the flow of materials and money, but there is no limit on how creatively they can be used.

Here is a sample Sankey that shows the vacation budget flowing into different heads.

Sankey diagram for a vacation budget

The example below shows a monthly budget split into multi-level heads.

Sankey diagram for a monthly budget

The example below shows inflation-adjusted revenue from music sales based on media format from 1973 till 2019. Data source RIAA.

Sankey diagram for music revenue based on media type sales

Below are some resources where you can create Sankey diagrams.

SankeyDiagram.net

Sample at SankeyDiagram.net

I kept this in the beginning of the list because it is the simplest to use. Creating Sankey diagrams at SankeyDiagram.net is very easy to use and free. Just go to the website, enter your data and see the diagram evolve in real time. It has a simple input format, a few settings and the ability to export the diagram in various formats like PNG and SVG. It automatically selects colors for you and also lets you specify colors for each flow. A useful feature here is that you can use a '?' for the program to automatically calculate the value based on the lower nodes. Thanks to Jonas Lorenz for creating and publishing this open-source tool.

SankeyMATIC.com

Sample at SankeyMATIC

SankeyMATIC.com has a UI which is loaded with lots of customizations to choose from. There are controls to help customize the size of the canvas, margins, heights and spacing of nodes, default colors, opacity, curviness etc. The data input format is simple and it provides you a choice to select predefined coloring themes. With this tool, also have total control of the opacity, font positioning, size and highlighting. It lets you export the diagram as a PNG at a size of your choice. I really miss the ability to let it auto-calculate the values of higher nodes based on the values in the lower ones.

Sankey Diagram Generator by Dénes Csala

Sample at Sankey Diagram Generator

This free Sankey diagram generator at sankey.csaladen.es provides a lot of control on the configuration part. Each flow can be defined more definitively in terms of colors and layers. The nodes can be freely moved around the canvas by dragging with the mouse pointer. The application lets you save the diagram as a PNG image. Thanks to Dénes Csala for creating and publishing this open-source tool.

Sankey using Google Charts

Sample at Google Charts

If you are okay with a bit of HTML and JavaScript programming, Google Charts has a chart library that you can embed in your web page and display interactive charts. The Sankey diagram will be rendered using HTML5/SVG technology to provide cross-browser compatibility (including VML for older IE versions) and cross platform portability to iPhones, iPads, and Android. There are loads of options to customize which I will cover in a later post. Head over to Google Charts' Sankey Diagram for more details on how to create these charts.

The simple Sankey diagram below was made with Google Charts and depicts the top 10 Wikipedia pages and their source based on page views for the year 2022. Hover your mouse pointer, or tap, over the nodes and flows to see the data. [Data source: Wikipedia] [static image sample]

The Sankey diagram below, which was made with Google Charts, shows UK studio albums released by The Beatles grouped into genres. [Data source: Wikipedia] [static image sample]

You will come across many other tools to create Sankey diagrams. The ones I chose to write about here are the ones that are freely accessible and fairly easy to use. Sankey diagrams can be used to depict much more complex scenarios of data or materials flow. If you have a comment, please drop it in the comments section below.