web scraper tutorial

This page describes how to set up a basic web scraper. The core function can be used and adjusted to perform a variety of web scraping abilities. The feature this page will describe is getting text from one or more web pages, removing all HTML and CSS tags to leave the text itself. It will then return only the text.

Github for all the code: LINK

# we only need two libraries, requests and beautifulsoup
import requests
from bs4 import BeautifulSoup as bs

def web_scraper(url, sub_urls = True):
    """Takes as input a url, then it outputs texts found on the website and optionally all the urls sub-urls.
    for instance: https://www.greatwebsite.com/about as input will get text for that url, but also
        - https://www.greatwebsite.com/about/subcategory1
        - https://www.greatwebsite.com/about/subcategory2
        - etc, it has to start with the original input url.
    run the function as follows: web_scraper('https://www.greatwebsite.com/about/').
    It needs quotation marks and start with https://www."""

    page = requests.get(url)
    soup = bs(page.text, "html.parser")
    text_set = []
    url_filtered = []

    if sub_urls:
        # loop over all urls, filter the urls based on the subURL condition to prevent duplicates
        for link in soup.find_all("a"):
            subURL = link.get("href")
            if subURL != None and subURL.startswith(url) and subURL not in url_filtered:
                url_filtered.append(subURL)
                page = requests.get(subURL)
                soup = bs(page.content, "html.parser")
                for data in soup(["style", "script"]):
                    data.decompose()
                text_set.append(" ".join(soup.stripped_strings))
    else:
        for data in soup(["style", "script"]):
            data.decompose()
        text_set.append(" ".join(soup.stripped_strings))

    return " ".join(text_set)

web_scraper('https://en.wikipedia.org/wiki/Python_(programming_language)', False)