Loop Dynamic Ajax Based Dropdown Values Using Python Selenium And Grab Data To Mysql Database

I am able to grab data if i apply static value of each dropdown but now i am trying to fetch dynamic values loop through each value of dropdown and store result to mysqldb. Issue

Solution 1:

First find the td tag to filter out states then get the values of option tags. Here you get all the states. For each state go to the url of that state and find districts. Districts are in the 2nd td tag so use find_next_sibling() and again get the values of the option tags. So you find all the districts too for each state. And rest of the things you already did.

import time
from bs4 import BeautifulSoup
from selenium import webdriver

url = "http://xlnindia.gov.in/frm_G_Cold_S_Query.aspx"
browser = webdriver.Chrome()
browser.get(url)

time.sleep(5)
html = browser.page_source
soup = BeautifulSoup(html, "lxml")

states = [ x["value"] for x in soup.find("td", bgcolor="#ffe0c0").find_all("option") ]

for state in states:

    browser.get(url + "?ST=" + state)
    time.sleep(5)

    html = browser.page_source
    soup = BeautifulSoup(html, "lxml")

    districts = [ x["value"] for x in soup.find("td", bgcolor="#ffe0c0").find_next_sibling().find_all("option") ]
    districts = districts[1:]

    for dist in districts:

        browser.get(url + "?ST=" + state)
        district = browser.find_element_by_id("ddldistrict")
        district.send_keys(dist)

        button = browser.find_element_by_id("btnSearch")
        button.click()

browser.close()
browser.quit()