Interacting With A Website And Getting Data Using Python

I am trying to interact with a website. For my data analysis project, I have a list of 1 million websites and I want to find the category of each website. That is why I am using th

Solution 1:

You can use BeautifulSoup, i.e.:

import requests, traceback
from bs4 import BeautifulSoup

domains = ["duckduckgo.com", "opensource.com"]
for dom in domains:
    try:
        req = requests.get(f"https://fortiguard.com/webfilter?q={dom}&version=8")
        if req.status_code == 200:
            soup = BeautifulSoup(req.text, 'html.parser')
            cat = soup.find("meta",  property="description")["content"].split(":")[1].strip()
            print(dom, cat)
    except:
        passprint(traceback.format_exc())

---

Output:

duckduckgo.com Search Engines and Portals
opensource.com Information Technology

---

Demo